Skill Tracer
Skills › Scientific Agent Skills › bulk-rnaseq › upstream-nfcore.md

upstream-nfcore.md

123 lines · skills/bulk-rnaseq/references/upstream-nfcore.md at 5f2bc03

Open on GitHub

Most marks come from matching words in the file, so treat them as pointers.

1# Path A — nf-core/rnaseq
2
3`nf-core/rnaseq` is the field-standard, community-audited pipeline for the reads → counts stage. It chains FastQC → trimming (Trim Galore or fastp) → optional contaminant/rRNA removal → alignment + quantification (STAR+Salmon, STAR+RSEM, or HISAT2) → tximport gene/transcript count merging → extensive QC → MultiQC, with reviewed defaults and per-process containers.
4
5This file covers *how to run it and what comes out*. For the Nextflow engine itself — profiles, executors, containers (Docker/Singularity/Conda/Wave), HPC/cloud, `-resume` caching, offline/`nf-core pipelines download` — use the **`nextflow`** skill. Don't duplicate that here.
6
7Documentation/schema reviewed 2026-09-30; recipes are illustrative, not a full pipeline execution. Current stable revision: **3.27.0** (always pin with `-r`).
8

Path A — nf-core/rnaseq

nf-core/rnaseq is the field-standard, community-audited pipeline for the reads → counts stage. It chains FastQC → trimming (Trim Galore or fastp) → optional contaminant/rRNA removal → alignment + quantification (STAR+Salmon, STAR+RSEM, or HISAT2) → tximport gene/transcript count merging → extensive QC → MultiQC, with reviewed defaults and per-process containers.

This file covers how to run it and what comes out. For the Nextflow engine itself — profiles, executors, containers (Docker/Singularity/Conda/Wave), HPC/cloud, -resume caching, offline/nf-core pipelines download — use the **nextflow** skill. Don't duplicate that here.

Documentation/schema reviewed 2026-09-30; recipes are illustrative, not a full pipeline execution. Current stable revision: 3.27.0 (always pin with -r).

Raw lines 1 to 8
Names a skill nextflow :5
9## Samplesheet
11`nf-core/rnaseq` takes a CSV, not loose files. The first four headers are required in this order; `fastq_2` may be empty for single-end data:
13```csv
14sample,fastq_1,fastq_2,strandedness
15CONTROL_REP1,/data/ctrl1_R1.fastq.gz,/data/ctrl1_R2.fastq.gz,auto
16CONTROL_REP2,/data/ctrl2_R1.fastq.gz,/data/ctrl2_R2.fastq.gz,auto
17TREATED_REP1,/data/trt1_R1.fastq.gz,/data/trt1_R2.fastq.gz,auto
18TREATED_REP2,/data/trt2_R1.fastq.gz,/data/trt2_R2.fastq.gz,auto
19```
21- **sample** — sample ID. Rows that share a `sample` value are treated as the same sample sequenced over multiple lanes and are merged.
22- **fastq_1 / fastq_2** — paths or URLs to gzipped FASTQ. Leave `fastq_2` empty for single-end.
23- **strandedness** — `auto` (recommended; the pipeline infers it with Salmon and warns on mismatch), or `forward` / `reverse` / `unstranded` if you know the kit. TruSeq stranded mRNA is typically `reverse`.
25Validate before launching: `python scripts/validate_samplesheet.py --samplesheet samplesheet.csv --nfcore`.

Samplesheet

nf-core/rnaseq takes a CSV, not loose files. The first four headers are required in this order; fastq_2 may be empty for single-end data:

sample,fastq_1,fastq_2,strandedness
CONTROL_REP1,/data/ctrl1_R1.fastq.gz,/data/ctrl1_R2.fastq.gz,auto
CONTROL_REP2,/data/ctrl2_R1.fastq.gz,/data/ctrl2_R2.fastq.gz,auto
TREATED_REP1,/data/trt1_R1.fastq.gz,/data/trt1_R2.fastq.gz,auto
TREATED_REP2,/data/trt2_R1.fastq.gz,/data/trt2_R2.fastq.gz,auto
  • sample — sample ID. Rows that share a sample value are treated as the same sample sequenced over multiple lanes and are merged.
  • fastq_1 / fastq_2 — paths or URLs to gzipped FASTQ. Leave fastq_2 empty for single-end.
  • strandedness — auto (recommended; the pipeline infers it with Salmon and warns on mismatch), or forward / reverse / unstranded if you know the kit. TruSeq stranded mRNA is typically reverse.

Validate before launching: python scripts/validate_samplesheet.py --samplesheet samplesheet.csv --nfcore.

Raw lines 9 to 26
Runs code python scripts/validate_samplesheet.py :25
27## Choosing the aligner / quantifier
29Set with `--aligner` (genome alignment) or `--pseudo_aligner` (lightweight). Defaults are well chosen.
31| Option | What it does | When |
32|--------|--------------|------|
33| `--aligner star_salmon` (default) | STAR genome alignment, Salmon quantifies against the transcriptome from the BAM | The standard, defensible default — gives counts **and** a genome BAM for QC/IGV |
34| `--aligner star_rsem` | STAR + RSEM | You specifically need RSEM estimates |
35| `--aligner bowtie2_salmon` | Bowtie2 transcriptome alignment + Salmon | When that alignment route is required |
36| `--aligner hisat2` | HISAT2 alignment (no built-in transcript quant) | Lower memory than STAR |
37| `--pseudo_aligner salmon` (+ `--skip_alignment`) | Salmon reads-based selective alignment, no BAM | Fastest/lightest; you don't need a genome BAM |
39`star_salmon` is recommended unless you have a specific reason otherwise. You can also add `--pseudo_aligner salmon` alongside an aligner to get both.

Choosing the aligner / quantifier

Set with --aligner (genome alignment) or --pseudo_aligner (lightweight). Defaults are well chosen.

OptionWhat it doesWhen
--aligner star_salmon (default)STAR genome alignment, Salmon quantifies against the transcriptome from the BAMThe standard, defensible default — gives counts and a genome BAM for QC/IGV
--aligner star_rsemSTAR + RSEMYou specifically need RSEM estimates
--aligner bowtie2_salmonBowtie2 transcriptome alignment + SalmonWhen that alignment route is required
--aligner hisat2HISAT2 alignment (no built-in transcript quant)Lower memory than STAR
--pseudo_aligner salmon (+ --skip_alignment)Salmon reads-based selective alignment, no BAMFastest/lightest; you don't need a genome BAM

star_salmon is recommended unless you have a specific reason otherwise. You can also add --pseudo_aligner salmon alongside an aligner to get both.

Raw lines 27 to 40
41## Running
43```bash
44# Smoke-test first (tiny bundled data; proves the environment works)
45nextflow run nf-core/rnaseq -r 3.27.0 -profile test,docker --outdir test_results
47# Real run with an iGenomes reference key
48nextflow run nf-core/rnaseq -r 3.27.0 \
49 -profile docker \
50 --input samplesheet.csv \
51 --genome GRCh38 \
52 --aligner star_salmon \
53 --outdir results \
54 -resume
56# Or supply your own reference explicitly (more reproducible than iGenomes keys)
57nextflow run nf-core/rnaseq -r 3.27.0 \
58 -profile singularity \
59 --input samplesheet.csv \
60 --fasta /ref/genome.fa --gtf /ref/annotation.gtf \
61 --aligner star_salmon --outdir results -resume
62```
64Generate a validated, documented command + params file interactively with `nf-core pipelines launch rnaseq` (see the `nextflow` skill).

Running

# Smoke-test first (tiny bundled data; proves the environment works)
nextflow run nf-core/rnaseq -r 3.27.0 -profile test,docker --outdir test_results

# Real run with an iGenomes reference key
nextflow run nf-core/rnaseq -r 3.27.0 \
  -profile docker \
  --input samplesheet.csv \
  --genome GRCh38 \
  --aligner star_salmon \
  --outdir results \
  -resume

# Or supply your own reference explicitly (more reproducible than iGenomes keys)
nextflow run nf-core/rnaseq -r 3.27.0 \
  -profile singularity \
  --input samplesheet.csv \
  --fasta /ref/genome.fa --gtf /ref/annotation.gtf \
  --aligner star_salmon --outdir results -resume

Generate a validated, documented command + params file interactively with nf-core pipelines launch rnaseq (see the nextflow skill).

Raw lines 41 to 65
Names a skill nextflow :64
66### Useful parameters
68- `--save_reference` — keep the built STAR/Salmon indices so re-runs and other projects don't rebuild them.
69- `--trimmer trimgalore|fastp` — trimming tool (default `trimgalore`).
70- `--remove_ribo_rna` — sortmerna rRNA depletion (use if libraries weren't poly-A/ribo-depleted, or to quantify rRNA contamination).
71- `--extra_salmon_quant_args='--gcBias'` — pass tool flags through.
72- `--skip_*` (e.g. `--skip_markduplicates`, `--skip_stringtie`) — drop stages you don't need.
73- `--gencode` — set when using GENCODE (not Ensembl) annotation, so gene IDs/biotypes parse correctly.
75Pick the reference release deliberately and record it. iGenomes keys (`--genome GRCh38`) are convenient but version-pinning your own `--fasta`/`--gtf` is more reproducible.

Useful parameters

  • --save_reference — keep the built STAR/Salmon indices so re-runs and other projects don't rebuild them.
  • --trimmer trimgalore|fastp — trimming tool (default trimgalore).
  • --remove_ribo_rna — sortmerna rRNA depletion (use if libraries weren't poly-A/ribo-depleted, or to quantify rRNA contamination).
  • --extra_salmon_quant_args='--gcBias' — pass tool flags through.
  • --skip_* (e.g. --skip_markduplicates, --skip_stringtie) — drop stages you don't need.
  • --gencode — set when using GENCODE (not Ensembl) annotation, so gene IDs/biotypes parse correctly.

Pick the reference release deliberately and record it. iGenomes keys (--genome GRCh38) are convenient but version-pinning your own --fasta/--gtf is more reproducible.

Raw lines 66 to 76
77## Outputs
79Key paths under `--outdir` (for `star_salmon`):
81```
82results/
83├── multiqc/ # MultiQC report — read this first
84├── star_salmon/
85│ ├── salmon.merged.gene_counts.tsv # raw estimated gene counts (tximport, countsFromAbundance=no)
86│ ├── salmon.merged.gene_counts_length_scaled.tsv # length-scaled counts -> use for DESeq2
87│ ├── salmon.merged.gene_tpm.tsv # TPM (for visualization, NOT for DESeq2)
88│ ├── salmon.merged.gene.SummarizedExperiment.rds # SummarizedExperiment (R)
89│ ├── <SAMPLE>/ # per-sample Salmon quant dirs
90│ └── deseq2_qc/ # PCA + sample-distance plots the pipeline already made
91└── pipeline_info/ # execution report, software versions, params
92```
94**The pipeline already runs tximport**, so on Path A you do **not** need this skill's `build_counts_matrix.py`. Use the merged TSV directly. This assumes `--skip_quantification_merge` is false:
95that mode omits length-scaled merged counts and DESeq2 QC. Preserve the emitted
96`salmon.merged.tx2gene_augmented.tsv` to reproduce the actual import; it can contain
97self-mappings for orphan transcripts, which must not be mistaken for genuine gene IDs.

Outputs

Key paths under --outdir (for star_salmon):

results/
├── multiqc/             # MultiQC report — read this first
├── star_salmon/
│   ├── salmon.merged.gene_counts.tsv                 # raw estimated gene counts (tximport, countsFromAbundance=no)
│   ├── salmon.merged.gene_counts_length_scaled.tsv   # length-scaled counts -> use for DESeq2
│   ├── salmon.merged.gene_tpm.tsv                     # TPM (for visualization, NOT for DESeq2)
│   ├── salmon.merged.gene.SummarizedExperiment.rds                  # SummarizedExperiment (R)
│   ├── <SAMPLE>/                                      # per-sample Salmon quant dirs
│   └── deseq2_qc/                                     # PCA + sample-distance plots the pipeline already made
└── pipeline_info/        # execution report, software versions, params

The pipeline already runs tximport, so on Path A you do not need this skill's build_counts_matrix.py. Use the merged TSV directly. This assumes --skip_quantification_merge is false: that mode omits length-scaled merged counts and DESeq2 QC. Preserve the emitted salmon.merged.tx2gene_augmented.tsv to reproduce the actual import; it can contain self-mappings for orphan transcripts, which must not be mistaken for genuine gene IDs.

Raw lines 77 to 98
99## Handoff to PyDESeq2
101`salmon.merged.gene_counts_length_scaled.tsv` is genes × samples, with a leading `gene_id` (and usually `gene_name`) column, and non-integer values. Round to integers for PyDESeq2:
103```python
104import pandas as pd
106df = pd.read_csv("results/star_salmon/salmon.merged.gene_counts_length_scaled.tsv", sep="\t")
107df = df.drop(columns=[c for c in ["gene_name"] if c in df.columns]).set_index("gene_id")
108assert df.index.is_unique and df.columns.is_unique
109assert not df.empty and df.notna().all().all()
110assert (df >= 0).all().all() and (df < float("inf")).all().all()
111counts = df.round().astype("int64") # genes x samples, integer
112counts.to_csv("counts.csv") # hand to the pydeseq2 skill (it transposes to samples x genes)
113```
115Build `metadata.csv` (index = sample IDs matching the count columns; columns = `condition`, `batch`, …) and proceed with the **`pydeseq2`** skill. The pipeline's own `deseq2_qc/` PCA is a good first sanity check before you run your own contrasts. Length-scaled counts are appropriate to round and use directly — see `counts-and-handoff.md` for the reasoning and the alternative offset-based route.
117For 3′ tag assays, use the assay-specific original-count route without length correction;
118do not select the length-scaled file automatically. Preserve `pipeline_info/`, reference
119release/checksums, quantifier mode, metadata and explicit contrast with the count export.
121Official reviewed sources: [usage](https://nf-co.re/rnaseq/3.27.0/docs/usage),
122[outputs](https://nf-co.re/rnaseq/3.27.0/docs/output),
123[parameter schema](https://github.com/nf-core/rnaseq/blob/3.27.0/nextflow_schema.json).

Handoff to PyDESeq2

salmon.merged.gene_counts_length_scaled.tsv is genes × samples, with a leading gene_id (and usually gene_name) column, and non-integer values. Round to integers for PyDESeq2:

import pandas as pd

df = pd.read_csv("results/star_salmon/salmon.merged.gene_counts_length_scaled.tsv", sep="\t")
df = df.drop(columns=[c for c in ["gene_name"] if c in df.columns]).set_index("gene_id")
assert df.index.is_unique and df.columns.is_unique
assert not df.empty and df.notna().all().all()
assert (df >= 0).all().all() and (df < float("inf")).all().all()
counts = df.round().astype("int64")          # genes x samples, integer
counts.to_csv("counts.csv")              # hand to the pydeseq2 skill (it transposes to samples x genes)

Build metadata.csv (index = sample IDs matching the count columns; columns = condition, batch, …) and proceed with the **pydeseq2** skill. The pipeline's own deseq2_qc/ PCA is a good first sanity check before you run your own contrasts. Length-scaled counts are appropriate to round and use directly — see counts-and-handoff.md for the reasoning and the alternative offset-based route.

For 3′ tag assays, use the assay-specific original-count route without length correction; do not select the length-scaled file automatically. Preserve pipeline_info/, reference release/checksums, quantifier mode, metadata and explicit contrast with the count export.

Official reviewed sources: usage, outputs, parameter schema.

Raw lines 99 to 123
Names a skill pydeseq2 :112