01. Data Acquisition & Status

Overview

See the Glossary for term definitions (read count, library size, gene detection, outlier, etc.).

  • RNA-seq and clinical data from GDC via the targets pipeline
  • Open Access RNA-seq from the GDC S3 bucket (s3://gdc-mmrf-commpass-phs000748-2-open/)
  • Clinical metadata via TCGAbiolinks::GDCquery_clinic()

Data Flow

flowchart LR
  subgraph GDC["GDC Open Access"]
    R["RNA-seq<br/>STAR Counts"]
    C["Clinical<br/>Metadata"]
    T["Treatment<br/>Records"]
  end

  subgraph Clean["Data Cleaning"]
    SE["Summarized<br/>Experiment"]
    CD["Clinical<br/>DataFrame"]
    TD["Treatment<br/>DataFrame"]
  end

  subgraph Analysis
    DE["Differential<br/>Expression"]
    KM["Survival<br/>Analysis"]
    PA["Pathway<br/>Enrichment"]
    DAG["Causal<br/>DAGs"]
  end

  R --> SE
  C --> CD
  T --> TD
  SE --> DE
  CD --> KM
  TD --> KM
  DE --> PA
  SE --> KM
  CD --> DAG

  style GDC fill:#e8f5e9,stroke:#4CAF50
  style Clean fill:#e3f2fd,stroke:#2196F3
  style Analysis fill:#fce4ec,stroke:#F44336

Simplified data flow from GDC to analysis outputs.

Pipeline Configuration

Current pipeline settings including sample limit, random seed, and data paths.

The pipeline downloads RNA-seq and clinical data from the GDC portal (https://portal.gdc.cancer.gov/projects/MMRF-COMMPASS) for the MMRF-COMMPASS project.

Sample limit: 200 patients (GDC has ~900; subsetting speeds local development). In CI, this is capped at 20 samples (see R/01_data_acquisition.R) to keep build times under 10 minutes.

Random seed: 42 — ensures reproducible patient selection when random_sample = TRUE.

Data directory: data | Results directory: results

Note: This vignette was built with sample_limit = 200 patients. Numbers below reflect this subset, not the full ~900-patient CoMMpass cohort. For pipeline execution details, see issue #46 (telemetry vignette).

RNA-seq Data

RNA-seq gene expression data is downloaded from GDC as a SummarizedExperiment object containing STAR-Counts.

RNA-seq Data Summary

  • Samples: 100
  • Genes: 60,660
  • Total counts: 6,045,546,169 (sum of all read counts across all genes and samples)
  • Median counts per sample: 58,030,298 (median library size)
  • Sparsity (% zero entries): 49.1%
Generating code 
{
    if (!file.exists(raw_rnaseq)) 
        return(NULL)
    se <- readRDS(raw_rnaseq)
    if (!inherits(se, "SummarizedExperiment")) 
        return(NULL)
    counts <- get_counts_assay(se)
    log_counts <- log10(counts + 1)
    mean_expr <- rowMeans(log_counts)
    gene_meta <- as.data.frame(SummarizedExperiment::rowData(se))
    has_biotype <- "gene_type" %in% names(gene_meta)
    if (has_biotype) {
        gene_meta$biotype_group <- dplyr::case_when(gene_meta$gene_type == 
            "protein_coding" ~ "protein-coding", gene_meta$gene_type %in% 
            c("lncRNA", "processed_pseudogene", "unprocessed_pseudogene", 
                "transcribed_unprocessed_pseudogene", "transcribed_processed_pseudogene") ~ 
            "lncRNA / pseudogene", TRUE ~ "other (miRNA, snoRNA, etc.)")
        plot_df <- data.frame(mean_expr = mean_expr, biotype = gene_meta$biotype_group, 
            stringsAsFactors = FALSE)
        biotype_colors <- c(`protein-coding` = "#0066CC", `lncRNA / pseudogene` = "#DC3545", 
            `other (miRNA, snoRNA, etc.)` = "#6C757D")
        p <- ggplot2::ggplot(plot_df, ggplot2::aes(x = mean_expr, 
            fill = biotype)) + ggplot2::geom_histogram(bins = 50, 
            alpha = 0.6, position = "identity") + ggplot2::scale_fill_manual(values = biotype_colors) + 
            ggplot2::labs(title = "Distribution of Mean Gene Expression by Biotype", 
                subtitle = paste0(format(nrow(counts), big.mark = ","), 
                  " genes, ", ncol(counts), " samples"), x = "Mean log10(counts + 1)", 
                y = "Number of Genes", fill = NULL) + theme_commpass_dark()
    }
    else {
        plot_df <- data.frame(mean_expr = mean_expr)
        p <- ggplot2::ggplot(plot_df, ggplot2::aes(x = mean_expr)) + 
            ggplot2::geom_histogram(bins = 50, fill = "steelblue", 
                alpha = 0.7) + ggplot2::labs(title = "Distribution of Mean Gene Expression", 
            subtitle = paste0(format(nrow(counts), big.mark = ","), 
                " genes, ", ncol(counts), " samples"), x = "Mean log10(counts + 1)", 
            y = "Number of Genes") + theme_commpass_dark()
    }
    p
}

Clinical Data

Clinical metadata from GDC provides patient demographics, disease characteristics, and outcomes. See the data dictionary for variable definitions and the glossary for units.

Data Completeness

Missing data rates across clinical variables, highlighting fields with substantial missingness.

83 of 88 variables are fully complete. Only variables with missing data are shown below. See data dictionary for variable definitions.

Quality Control

Quality control identifies outlier samples based on library size and gene detection rate. See R/02_quality_control.R for filtering criteria.

The per-sample expression summary (Total Counts, Genes Detected) is shown in the RNA-seq Data section above. This section adds QC-specific metrics: size factors, count dispersion (MAD), and outlier flags.

Variable definitions (see also Glossary):

  • Total Counts — library size: sum of all read counts mapped to genes in one sample
  • Detected Genes — number of genes with at least 1 mapped read (count > 0)
  • Median Count — median read count across all genes in a sample (most genes have 0 or low counts, so this is typically 0)
  • MAD Count — median absolute deviation of gene counts within a single sample (measures spread of the count distribution for that sample)
  • Size Factor — library size / median library size across all samples (values near 1.0 = typical; <<1 = under-sequenced; >>1 = over-sequenced)
  • Outlier — flagged Yes if the sample falls in the bottom 5th percentile of either library size OR genes detected

QC Metrics Summary

  • Samples assessed: 100
  • Outliers flagged: 9

After Filtering

  • Samples retained: 91
  • Genes retained: 30,675
  • Genes removed: 29,985 (49.4%)

Data Sources

Results in this vignette are derived from the MMRF CoMMpass study (MMRF-COMMPASS, ~1,143 patients), downloaded via TCGAbiolinks. The pipeline runs with a configurable sample_limit (default 200; CI uses 20).

For full citations, data access tiers, and the distinction between pipeline data and synthetic test data, see the Data Sources vignette.

Recent Changes

Recent project commits with lines added, files changed, and change categories.

Reproducibility

Git Commit Info (click to expand)
Generating code 
{
    if (!requireNamespace("gert", quietly = TRUE)) 
        return(NULL)
    tryCatch({
        info <- gert::git_info()
        log1 <- gert::git_log(max = 1)
        git_df <- data.frame(Item = c("Commit Hash", "Author", 
            "Time", "Branch"), Value = c(info$commit, log1$author, 
            as.character(log1$time), info$shorthand), stringsAsFactors = FALSE)
        DT::datatable(git_df, rownames = FALSE, options = list(pageLength = 10, 
            dom = "t", scrollX = TRUE), caption = htmltools::tags$caption(style = "caption-side: top; text-align: left;", 
            "Git repository state at pipeline build time."))
    }, error = function(e) NULL)
}
Session Info (click to expand)
Show code 
sessionInfo()
#> R version 4.6.1 (2026-06-24)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 26.04 LTS
#> 
#> Matrix products: default
#> BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
#> LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.32.so;  LAPACK version 3.12.0
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> time zone: Etc/UTC
#> tzcode source: system (glibc)
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] targets_1.12.0
#> 
#> loaded via a namespace (and not attached):
#>  [1] base64url_1.4      gtable_0.3.6       jsonlite_2.0.0     dplyr_1.2.1       
#>  [5] compiler_4.6.1     tidyselect_1.2.1   callr_3.8.0        jquerylib_0.1.4   
#>  [9] scales_1.4.0       yaml_2.3.12        fastmap_1.2.0      ggplot2_4.0.3     
#> [13] R6_2.6.1           generics_0.1.4     igraph_2.3.3       knitr_1.51        
#> [17] htmlwidgets_1.6.4  backports_1.5.1    tibble_3.3.1       maketools_1.3.2   
#> [21] RColorBrewer_1.1-3 bslib_0.11.0       pillar_1.11.1      rlang_1.3.0       
#> [25] DT_0.34.0          cachem_1.1.0       xfun_0.60          S7_0.2.2          
#> [29] sass_0.4.10        sys_3.4.3          otel_0.2.0         cli_3.6.6         
#> [33] withr_3.0.3        magrittr_2.0.5     crosstalk_1.2.2    ps_1.9.3          
#> [37] grid_4.6.1         digest_0.6.39      processx_3.9.0     secretbase_1.3.0  
#> [41] lifecycle_1.0.5    prettyunits_1.2.0  vctrs_0.7.3        evaluate_1.0.5    
#> [45] glue_1.8.1         data.table_1.18.4  farver_2.1.2       codetools_0.2-20  
#> [49] buildtools_1.0.0   rmarkdown_2.31     tools_4.6.1        pkgconfig_2.0.3   
#> [53] htmltools_0.5.9