Nextflow

Overview

Nextflow is a workflow language and runtime for building reproducible, portable, scalable data pipelines. It is dominant in bioinformatics but works for any data-heavy computation. nf-core is a community curating production-grade Nextflow pipelines, reusable modules, and the nf-core tooling on top of Nextflow.

Key ideas:

• Dataflow programming: pipelines are process tasks connected by channels. Nextflow infers execution order and parallelism from data dependencies — there is no explicit scheduler to write.
• Write once, run anywhere: the same pipeline runs locally, on HPC (SLURM, SGE, LSF, PBS), and on cloud (AWS Batch, Google Batch, Azure Batch, Kubernetes) by changing config/profiles, not code.
• Reproducibility: pinned software environments and pipeline revisions, immutable inputs/references, recorded parameters and seeds. -resume is a computational cache, not scientific validation. Conda is an environment manager; Wave resolves/builds images rather than executing them.
• DSL2 is the modern, required syntax: modular process/workflow/include definitions.

This skill covers both running existing pipelines and developing your own (Nextflow language + nf-core conventions, testing with nf-test, configuration, and deployment).

When to Use This Skill

Use this skill when the user wants to:

• Run an nf-core or custom Nextflow pipeline, or debug a failing/resuming run.
• Write or modify .nf scripts, nextflow.config, profiles, or nextflow_schema.json.
• Author or test nf-core-style modules/subworkflows (main.nf, meta.yml, tests/, nf-test).
• Configure executors, containers, or resources; scale to HPC or cloud.
• Implement a scientific workflow in Nextflow or adapt an existing nf-core pipeline.
• Understand processes, channels, operators, take/emit, publishDir, ext.args, meta maps.

Setup

This review targets stable Nextflow 26.04.6, nf-core tools 4.1.0, and nf-test 0.9.5. Nextflow needs Bash 3.2+ and Java 17–26; verify java -version (a launcher on PATH does not prove a runtime is installed). The strict parser is the default in 26.04. See release notes and the 26.04 migration guide. Stable and edge documentation can differ; do not use a preview feature without its version/flag.

# Install Nextflow (self-installing launcher)
export NXF_VER=26.04.6
curl -fsSL https://get.nextflow.io -o install-nextflow.sh
# Review the installer before executing it.
bash < install-nextflow.sh
mkdir -p "$HOME/.local/bin"
mv nextflow "$HOME/.local/bin/"
export PATH="$HOME/.local/bin:$PATH"
nextflow info                                # verify

# Alternative (illustrative; confirm package availability and Java compatibility)
conda create -n nf -c conda-forge -c bioconda nextflow=26.04.6 nf-core=4.1.0

# nf-core tools (Python) for creating/linting/running nf-core assets
uv tool install "nf-core==4.1.0"
nf-core --version

Pin the engine for reproducibility: export NXF_VER=26.04.6; check the selected pipeline release’s engine constraint before upgrading. Use edge only for a required, explicitly tested feature. For air-gapped/HPC, see references/running-pipelines.md (offline mode) and references/configuration.md.

Two Modes of Work

Decide which path the user is on — it changes everything:

Quick Start

Run an nf-core pipeline

Use the selected release’s small test profile first after checking its resource/download requirements. A passing smoke test verifies that configuration and fixture, not scientific accuracy or full-scale capacity. The following RNA-seq examples are illustrative; no biological pipeline or containers were run in this review.

# 1. Confirm setup works (downloads pipeline + tiny test data)
nextflow run nf-core/rnaseq -r 3.27.0 -profile test,docker --outdir test_results

# 2. Real run: pin a revision (-r), pick a container engine, pass inputs
nextflow run nf-core/rnaseq -r 3.27.0 \
  -profile docker \
  --input samplesheet.csv \
  --fasta reference.fa --gtf annotation.gtf \
  --outdir results \
  -resume

• -profile (single dash) selects bundled config profiles; combine them comma-separated, e.g. test,docker. Choose one execution environment profile (docker, singularity, or conda); a site/executor profile can be combined with it when compatible.
• --input, --genome, --outdir (double dash) are pipeline parameters. Many nf-core pipelines take a samplesheet CSV; use the selected pipeline release’s input schema.
• -resume reuses cached results from the last run. -r <version> pins a release for reproducibility.

Use nf-core pipelines launch <name> for an interactive, schema-validated way to build the command and a -params-file. See references/running-pipelines.md.

Write a minimal pipeline

This fixed-input example was executed with Nextflow 26.04.6, including -resume. Do not interpolate unvalidated sample IDs or arbitrary text into shell commands.

#!/usr/bin/env nextflow

process SAYHELLO {
    tag "$greeting"
    publishDir "results", mode: 'copy'

    input:
    val greeting

    output:
    path "${greeting}.txt", emit: message

    script:
    """
    echo '$greeting world' > ${greeting}.txt
    """
}

workflow {
    channel.of('hello', 'bonjour', 'hola') | SAYHELLO
}

nextflow run main.nf            # add -resume on reruns

The full language (processes, channels, operators, DSL2 workflows with take/main/emit, modules) is in references/language.md.

Core Concepts at a Glance

• Process: a unit of work that runs a script (Bash by default). Declares input:, output:, directives (resources, container, publishDir, tag, errorStrategy), and a script: or exec: block (shell: is deprecated). Each task runs in its own isolated work directory (work/xx/yy…).
• Channel: the async queues that connect processes. Queue channels are streams that DSL2 broadcasts to each downstream consumer; value channels hold a single reusable value. Within one process invocation, combine one queue input with reusable values, or join keyed streams into one tuple channel first. Created with factories like channel.of, channel.fromPath, channel.fromFilePairs, channel.value.
• Operator: transforms/combines channels — map, filter, collect, groupTuple, join, combine, mix, flatten, branch, multiMap, splitCsv, view, set.
• Workflow: composes processes. DSL2 workflows can declare take: (inputs), main: (logic), emit: (named outputs) and be included as subworkflows. The unnamed workflow {} is the entry point.
• Module: a .nf file exposing processes/workflows via include { NAME } from './path' (supports as aliasing).
• Configuration: nextflow.config sets params, process directives, executor, container engines, and named profiles. Selectors withName:/withLabel: target specific processes. See references/configuration.md.
• meta map (nf-core): the convention of carrying a metadata map ([ id:'sample1', single_end:false ]) alongside files in input/output tuples so samples stay labeled through the pipeline. See references/developing.md.

nf-core tools CLI

nf-core tools 4.1.0 groups subcommands under pipelines, modules, and subworkflows. Removed bare forms such as nf-core lint now fail; use nf-core pipelines lint.

Full command reference, flags, and examples: references/nf-core-tools.md.

Essential nextflow CLI

Config, executors, caching internals, and tracing details: references/configuration.md.

Best Practices (high-value habits)

• Test the selected release first with its small profile and resource limits. Check sample identity, counts, paired reads, reference assembly/annotation compatibility and expected outputs independently of exit status.
• Pin everything: pipeline revision (-r), NXF_VER, and tool versions (containers). Don't run latest for science you'll publish.
• **Use -resume** and understand caching: a task re-runs if its inputs, script, or container change. See cache-debugging in references/configuration.md.
• Parameterize via config/params-file, not hardcoded paths. Keep params and profiles in nextflow.config.
• Declare the environment per process for real analyses. Pin image digests/platform or lock Conda dependencies; preserve reference/input checksums, module/plugin versions, configuration, seeds and run reports. Local shell-only examples are suitable for plumbing tests.
• For nf-core dev: reuse existing modules (nf-core modules install) before writing new ones; pass tool flags through ext.args (not hardcoded in the script); always include a stub: block and nf-test tests; run nf-core pipelines lint and prettier before committing.
• Right-size resources with process_low/medium/high labels and errorStrategy 'retry' with dynamic task.attempt scaling instead of one giant request.
• Use the strict parser, the default in 26.04. Prefer lowercase channel, explicit closure parameters, local def variables inside closures/process scripts, and named outputs. Check with nextflow lint; static typing remains a separate preview (nextflow.enable.types = true). Legacy operators have migration guidance in references/language.md.

Reference Files

Read the relevant file when you need depth — each is self-contained:

• references/language.md — DSL2 language: processes, directives, channels, operators, workflows (take/emit), modules, dynamic resources, error handling.
• references/configuration.md — nextflow.config, scopes, profiles, withName/withLabel selectors, executors (local/SLURM/cloud), caching/-resume internals, tracing/reports, the nextflow CLI.
• references/containers.md — Docker, Singularity/Apptainer, Podman, Conda, Wave containers; choosing and enabling engines; common gotchas.
• references/running-pipelines.md — finding/running nf-core pipelines, samplesheets, params files, reference genomes (iGenomes), offline runs, institutional configs, Seqera Platform.
• references/nf-core-tools.md — complete nf-core CLI reference (pipelines/modules/subworkflows), flags, and workflows.
• references/developing.md — authoring nf-core pipelines & modules: template layout, module main.nf/meta.yml, meta maps, ext.args/modules.config, subworkflows, resource labels, linting & Harshil alignment style.
• references/testing.md — nf-test for modules/subworkflows/pipelines: test structure, assertions, snapshots, tags, running tests, CI.

Official docs: Nextflow https://docs.seqera.io/nextflow/ · nf-core https://nf-co.re/docs/ · Training https://training.nextflow.io/