Skill Tracer
Skills › Scientific Agent Skills › bulk-rnaseq › upstream-manual.md

upstream-manual.md

149 lines · skills/bulk-rnaseq/references/upstream-manual.md at 5f2bc03

Open on GitHub

Most marks come from matching words in the file, so treat them as pointers.

1# Path B — Standalone tools (reads → quant)
2
3Run each stage yourself when you want transparency, have only a few samples, or can't use Nextflow/containers. Do not assume numerical equivalence to Path A: inspect all assignment, filtering and quantification options. Quantify **every sample identically**.
4
5Install (bioconda): `conda create -n rnaseq -c conda-forge -c bioconda --strict-channel-priority fastqc fastp trim-galore "star=2.7.11b" "salmon=2.8.0" subread multiqc rseqc`.
6
7These commands are illustrative except for the Salmon 2.8.0 index/quant flags,
8which were exercised on a tiny synthetic paired-end fixture. Pin the solved environment.
9Salmon 2.x uses a different index format from C++ Salmon <=1.12.0; rebuild indexes
10and use the same binary/reference for every sample. See the [current migration guide](https://github.com/COMBINE-lab/salmon/blob/v2.8.0/MIGRATION.md).

Path B — Standalone tools (reads → quant)

Run each stage yourself when you want transparency, have only a few samples, or can't use Nextflow/containers. Do not assume numerical equivalence to Path A: inspect all assignment, filtering and quantification options. Quantify every sample identically.

Install (bioconda): conda create -n rnaseq -c conda-forge -c bioconda --strict-channel-priority fastqc fastp trim-galore "star=2.7.11b" "salmon=2.8.0" subread multiqc rseqc.

These commands are illustrative except for the Salmon 2.8.0 index/quant flags, which were exercised on a tiny synthetic paired-end fixture. Pin the solved environment. Salmon 2.x uses a different index format from C++ Salmon <=1.12.0; rebuild indexes and use the same binary/reference for every sample. See the current migration guide.

Raw lines 1 to 11
12## 0. Reference data
14You need, for your organism and a **pinned** annotation release:
15- genome FASTA (`genome.fa`) and matching annotation GTF (`annotation.gtf`) — for STAR/featureCounts.
16- transcriptome FASTA (`transcripts.fa`, cDNA) — for Salmon.
18Fetch genome, transcript FASTA and GTF from one explicit Ensembl/GENCODE release (or use the `gget` skill to locate that release). Keep genome and GTF from the **same** release.

0. Reference data

You need, for your organism and a pinned annotation release:

  • genome FASTA (genome.fa) and matching annotation GTF (annotation.gtf) — for STAR/featureCounts.
  • transcriptome FASTA (transcripts.fa, cDNA) — for Salmon.

Fetch genome, transcript FASTA and GTF from one explicit Ensembl/GENCODE release (or use the gget skill to locate that release). Keep genome and GTF from the same release.

Raw lines 12 to 19
Names a skill gget :18
20## 1. QC raw reads — FastQC
22```bash
23mkdir -p qc/raw
24fastqc -t 8 -o qc/raw reads/*.fastq.gz
25```
27Inspect per-base quality, adapter content, duplication, and over-represented sequences. Interpretation/thresholds: `design-and-qc.md`.

1. QC raw reads — FastQC

mkdir -p qc/raw
fastqc -t 8 -o qc/raw reads/*.fastq.gz

Inspect per-base quality, adapter content, duplication, and over-represented sequences. Interpretation/thresholds: design-and-qc.md.

Raw lines 20 to 28
29## 2. Trim — fastp (recommended) or Trim Galore
31`fastp` is fast and emits a JSON/HTML report MultiQC understands:
33```bash
34mkdir -p trimmed
35fastp \
36 -i reads/s1_R1.fastq.gz -I reads/s1_R2.fastq.gz \
37 -o trimmed/s1_R1.fq.gz -O trimmed/s1_R2.fq.gz \
38 --detect_adapter_for_pe --qualified_quality_phred 20 --length_required 36 \
39 --thread 4 --json qc/s1.fastp.json --html qc/s1.fastp.html
40```
42Trim Galore (wraps Cutadapt + FastQC; auto-detects adapters):
44```bash
45trim_galore --paired --cores 4 --fastqc -o trimmed reads/s1_R1.fastq.gz reads/s1_R2.fastq.gz
46```
48Aggressive quality trimming is usually unnecessary for modern data and for STAR (which soft-clips); adapter removal is the main goal. Re-run FastQC on trimmed reads to confirm.

2. Trim — fastp (recommended) or Trim Galore

fastp is fast and emits a JSON/HTML report MultiQC understands:

mkdir -p trimmed
fastp \
  -i reads/s1_R1.fastq.gz -I reads/s1_R2.fastq.gz \
  -o trimmed/s1_R1.fq.gz  -O trimmed/s1_R2.fq.gz \
  --detect_adapter_for_pe --qualified_quality_phred 20 --length_required 36 \
  --thread 4 --json qc/s1.fastp.json --html qc/s1.fastp.html

Trim Galore (wraps Cutadapt + FastQC; auto-detects adapters):

trim_galore --paired --cores 4 --fastqc -o trimmed reads/s1_R1.fastq.gz reads/s1_R2.fastq.gz

Aggressive quality trimming is usually unnecessary for modern data and for STAR (which soft-clips); adapter removal is the main goal. Re-run FastQC on trimmed reads to confirm.

Raw lines 29 to 49
50## 3a. STAR — genome alignment + gene counts
52Build the index once per genome+annotation+read-length. `--sjdbOverhang` = read length − 1 (100 is a safe default). Human needs ~30 GB RAM.
54```bash
55mkdir -p star_index
56STAR --runMode genomeGenerate --runThreadN 12 \
57 --genomeDir star_index \
58 --genomeFastaFiles genome.fa \
59 --sjdbGTFfile annotation.gtf \
60 --sjdbOverhang 100
61```
63Align each sample, asking STAR to also count reads per gene:
65```bash
66mkdir -p star
67STAR --runThreadN 12 --genomeDir star_index \
68 --readFilesIn trimmed/s1_R1.fq.gz trimmed/s1_R2.fq.gz --readFilesCommand gunzip -c \
69 --outSAMtype BAM SortedByCoordinate \
70 --quantMode GeneCounts \
71 --outFileNamePrefix star/s1.
72```
74`--quantMode GeneCounts` writes `s1.ReadsPerGene.out.tab` (4 columns, see strandedness below). The sorted BAM is useful for QC (RSeQC, IGV) and for featureCounts.

3a. STAR — genome alignment + gene counts

Build the index once per genome+annotation+read-length. --sjdbOverhang = read length − 1 (100 is a safe default). Human needs ~30 GB RAM.

mkdir -p star_index
STAR --runMode genomeGenerate --runThreadN 12 \
  --genomeDir star_index \
  --genomeFastaFiles genome.fa \
  --sjdbGTFfile annotation.gtf \
  --sjdbOverhang 100

Align each sample, asking STAR to also count reads per gene:

mkdir -p star
STAR --runThreadN 12 --genomeDir star_index \
  --readFilesIn trimmed/s1_R1.fq.gz trimmed/s1_R2.fq.gz --readFilesCommand gunzip -c \
  --outSAMtype BAM SortedByCoordinate \
  --quantMode GeneCounts \
  --outFileNamePrefix star/s1.

--quantMode GeneCounts writes s1.ReadsPerGene.out.tab (4 columns, see strandedness below). The sorted BAM is useful for QC (RSeQC, IGV) and for featureCounts.

Raw lines 50 to 75
76## 3b. Salmon — decoy-aware selective alignment
78Build a **decoy-aware** index (genome as decoy) so reads from unannotated/genomic regions don't get miscounted against transcripts:
80```bash
81# 1. Decoys = all genome sequence names; gentrome = transcriptome THEN genome (order matters)
82rg '^>' genome.fa | sed 's/^>//; s/ .*//' > decoys.txt
83cat transcripts.fa genome.fa | gzip > gentrome.fa.gz
85# 2. Index (k=31 works for reads >=75 bp; use smaller k for shorter reads)
86salmon index -t gentrome.fa.gz -d decoys.txt -i salmon_index -k 31 -p 12
87```
89Quantify each sample (`-l A` auto-detects library type/strandedness; enable bias correction):
91```bash
92salmon quant -i salmon_index -l A \
93 -1 trimmed/s1_R1.fq.gz -2 trimmed/s1_R2.fq.gz \
94 --gcBias --seqBias -p 8 \
95 -o quant/s1
96```
98Selective alignment is the default; `--validateMappings` is an accepted no-op in 2.8.0.
99For GENCODE bar-delimited transcript FASTA names, add `--gencode` at indexing so
100IDs match the GTF. Record any version stripping separately. Do not use this
101length-corrected bridge for 3′ tag assays.
103Each `quant/<sample>/quant.sf` is transcript-level; aggregate to gene level with `scripts/build_counts_matrix.py --from salmon` (needs a `tx2gene` map — see `counts-and-handoff.md`). Always check the reported mapping rate (`quant/<sample>/logs/salmon_quant.log`).

3b. Salmon — decoy-aware selective alignment

Build a decoy-aware index (genome as decoy) so reads from unannotated/genomic regions don't get miscounted against transcripts:

# 1. Decoys = all genome sequence names; gentrome = transcriptome THEN genome (order matters)
rg '^>' genome.fa | sed 's/^>//; s/ .*//' > decoys.txt
cat transcripts.fa genome.fa | gzip > gentrome.fa.gz

# 2. Index (k=31 works for reads >=75 bp; use smaller k for shorter reads)
salmon index -t gentrome.fa.gz -d decoys.txt -i salmon_index -k 31 -p 12

Quantify each sample (-l A auto-detects library type/strandedness; enable bias correction):

salmon quant -i salmon_index -l A \
  -1 trimmed/s1_R1.fq.gz -2 trimmed/s1_R2.fq.gz \
  --gcBias --seqBias -p 8 \
  -o quant/s1

Selective alignment is the default; --validateMappings is an accepted no-op in 2.8.0. For GENCODE bar-delimited transcript FASTA names, add --gencode at indexing so IDs match the GTF. Record any version stripping separately. Do not use this length-corrected bridge for 3′ tag assays.

Each quant/<sample>/quant.sf is transcript-level; aggregate to gene level with scripts/build_counts_matrix.py --from salmon (needs a tx2gene map — see counts-and-handoff.md). Always check the reported mapping rate (quant/<sample>/logs/salmon_quant.log).

Raw lines 76 to 104
Runs code scripts/build_counts_matrix.py :103
105## 3c. featureCounts — counts from a STAR BAM (alternative to STAR GeneCounts)
107```bash
108mkdir -p counts
109# Strandedness: 0 unstranded, 1 forward, 2 reverse; verify before choosing.
110featureCounts -T 8 -p --countReadPairs \
111 -a annotation.gtf -g gene_id \
112 -s 2 \
113 -o counts/featurecounts.txt \
114 star/s1.Aligned.sortedByCoord.out.bam star/s2.Aligned.sortedByCoord.out.bam
115```
117Pass all sample BAMs at once to get one matrix. Parse it with `scripts/build_counts_matrix.py --from featurecounts`.

3c. featureCounts — counts from a STAR BAM (alternative to STAR GeneCounts)

mkdir -p counts
# Strandedness: 0 unstranded, 1 forward, 2 reverse; verify before choosing.
featureCounts -T 8 -p --countReadPairs \
  -a annotation.gtf -g gene_id \
  -s 2 \
  -o counts/featurecounts.txt \
  star/s1.Aligned.sortedByCoord.out.bam star/s2.Aligned.sortedByCoord.out.bam

Pass all sample BAMs at once to get one matrix. Parse it with scripts/build_counts_matrix.py --from featurecounts.

Raw lines 105 to 118
Runs code scripts/build_counts_matrix.py :117
119## Strandedness — get this right
121The wrong setting can discard most assigned reads; there is no fixed loss fraction. Determine it once, then apply consistently:
123- **Salmon** `-l A` auto-detects and reports the library type in `quant/<sample>/lib_format_counts.json` (`ISR` = reverse-stranded paired, `ISF` = forward, `IU` = unstranded; `IS` is not a valid unstranded code).
124- Or run **RSeQC** `infer_experiment.py -r genes.bed -i s1.bam` on a STAR BAM.
126Map the result to each tool:
128| Library | Salmon `-l` | featureCounts `-s` | STAR column (in `ReadsPerGene.out.tab`) |
129|---------|-------------|--------------------|------------------------------------------|
130| Unstranded | `IU` (auto `A`) | `0` | col 2 |
131| Forward (e.g. Ligation) | `ISF` (auto `A`) | `1` | col 3 |
132| Reverse (e.g. dUTP/TruSeq stranded) | `ISR` (auto `A`) | `2` | col 4 |
134Illumina TruSeq Stranded mRNA is **reverse** (`-s 2`, STAR col 4). When in doubt, let Salmon auto-detect and match the others to it.

Strandedness — get this right

The wrong setting can discard most assigned reads; there is no fixed loss fraction. Determine it once, then apply consistently:

  • Salmon -l A auto-detects and reports the library type in quant/<sample>/lib_format_counts.json (ISR = reverse-stranded paired, ISF = forward, IU = unstranded; IS is not a valid unstranded code).
  • Or run RSeQC infer_experiment.py -r genes.bed -i s1.bam on a STAR BAM.

Map the result to each tool:

LibrarySalmon -lfeatureCounts -sSTAR column (in ReadsPerGene.out.tab)
UnstrandedIU (auto A)0col 2
Forward (e.g. Ligation)ISF (auto A)1col 3
Reverse (e.g. dUTP/TruSeq stranded)ISR (auto A)2col 4

Illumina TruSeq Stranded mRNA is reverse (-s 2, STAR col 4). When in doubt, let Salmon auto-detect and match the others to it.

Raw lines 119 to 135
136## 4. Aggregate QC — MultiQC
138```bash
139multiqc qc/ star/ quant/ counts/ -o qc/multiqc
140```
142MultiQC collates FastQC, fastp/Trim Galore, STAR, Salmon, and featureCounts logs into one report — your QC narrative for the methods section. Then build the counts matrix (`counts-and-handoff.md`) and hand off to `pydeseq2`.

4. Aggregate QC — MultiQC

multiqc qc/ star/ quant/ counts/ -o qc/multiqc

MultiQC collates FastQC, fastp/Trim Galore, STAR, Salmon, and featureCounts logs into one report — your QC narrative for the methods section. Then build the counts matrix (counts-and-handoff.md) and hand off to pydeseq2.

Raw lines 136 to 143
Names a skill pydeseq2 :142
144## Official references reviewed
146- [STAR manual](https://github.com/alexdobin/STAR/blob/master/doc/STARmanual.pdf): GeneCounts columns, index and alignment flags.
147- [Salmon 2.8.0](https://github.com/COMBINE-lab/salmon/releases/tag/v2.8.0) and its installed CLI help: index/quant flags and migration.
148- [featureCounts CLI source](https://github.com/ShiLab-Bioinformatics/subread/blob/master/src/readSummary.c): `-p` declares paired reads; `--countReadPairs` counts fragments. `--fraction` produces fractional counts and is not supported by this bridge.
149- [fastp](https://github.com/OpenGene/fastp), [Trim Galore](https://github.com/FelixKrueger/TrimGalore), [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/), [MultiQC](https://docs.seqera.io/multiqc/getting_started/running_multiqc/).

Official references reviewed

Raw lines 144 to 149