Bio Workflow Management Nf Core Pipelines

Version Compatibility

Reference examples tested with: Nextflow 24.04+, nf-core/tools 3.0+, Docker 24+ or Singularity 3.8+

Before using code patterns, verify installed versions match. If versions differ:

• CLI: <tool> --version then <tool> --help to confirm flags

If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

Note: -r <tag> pins the pipeline to an immutable release; without it nextflow run nf-core/<pipe> pulls whatever the mutable default branch is today, so results are not reproducible. Pin the container engine too (a pipeline release ships digest-pinned images; -profile docker uses them, :latest does not). Schema validation moved from the deprecated nf-validation plugin to nf-schema; a current pipeline uses nf-schema, so validate samplesheets against the pipeline's shipped schema, not a hand-written one.

nf-core Pipelines

"Run a curated community pipeline on my samples" -> Select a versioned nf-core pipeline, pin its release, choose a container profile, validate a samplesheet against the pipeline schema, point it at references, and run it, reading the aggregated MultiQC report at the end.

• CLI: nextflow run nf-core/<pipeline> -r <version> -profile <container>,<institution> --input samplesheet.csv --outdir results -resume
• CLI: nf-core pipelines list / nf-core pipelines download (browse and cache pipelines)

The governing principle: ADOPT a community pipeline before authoring a new one

For any mainstream analysis - RNA-seq, germline/somatic variant calling, ATAC-seq, ChIP-seq, methylation, amplicon/metagenomics, single-cell - a curated nf-core pipeline already encodes years of QC, edge-case handling, CI/nf-test regression tests, institutional configs for hundreds of HPCs, a standardized samplesheet+schema, and MultiQC reporting (Ewels 2020 Nat Biotechnol 38:276-278). Reinventing that in hand-written Nextflow is months of work and worse QC: the community pipeline has already found the bugs a bespoke version will rediscover, and it is maintained across every future tool update and reference build. The decision every biologist should default toward is ADOPT, not BUILD.

The corollary trap is treating "adopt" as "run once and trust the number". A community pipeline is only reproducible if the RUN is pinned: -r pins the pipeline version, the release's digest-pinned containers pin the software, --genome/reference URIs pin the reference data (Wratten et al. 2021 Nat Methods 18:1161-1168; Grüning et al. 2018 Cell Syst 6:631-635). An unpinned nextflow run nf-core/rnaseq on the default branch with a :latest engine is exactly as irreproducible as a hand-rolled script - the curation buys nothing if the invocation is loose. Author from scratch only for genuinely novel logic with no community pipeline, an unsupported combination of steps, or an institutional constraint no config can express (see workflow-management/nextflow-pipelines).

Decision: adopt an nf-core pipeline vs author a new one

Decision: which container profile by platform

Profiles are comma-separated with NO spaces and applied left-to-right (later overrides earlier), so -profile test,docker runs the test dataset under Docker, and -profile singularity,uppmax layers the institutional config over Singularity.

Decision: --genome/iGenomes vs custom references

The reference layer is a reproducibility layer in its own right: --genome GRCh38 without a recorded iGenomes snapshot pins less than an explicit --fasta/--gtf URI pair. Prefer explicit references and record their source when the result must be reproduced.

The run pattern (pin everything)

# Smoke test first: tiny public dataset proves the install + engine end to end.
nextflow run nf-core/rnaseq -r 3.14.0 -profile test,docker --outdir results_test

# Real run: -r pins the release, -profile picks the engine, --input is the samplesheet.
nextflow run nf-core/rnaseq -r 3.14.0 \
    -profile singularity \
    --input samplesheet.csv \
    --genome GRCh38 \
    --outdir results \
    -resume

• -r 3.14.0 is the pipeline REVISION (a git tag). It is MANDATORY: without it the run tracks the mutable default branch and is not reproducible.
• -profile singularity selects the container engine (comma-add an institutional config: -profile singularity,uppmax).
• Single-dash options (-r, -profile, -resume, -c, -params-file) are NEXTFLOW options; double-dash options (--input, --genome, --outdir, --max_memory) are PIPELINE parameters. Mixing up the dash count is the most common invocation error.
• Config precedence, low to high: the pipeline's built-in nextflow.config -> conf/base.config -> selected profiles -> -c custom.config -> -params-file params.yaml -> a --param on the command line. A later source overrides an earlier one.

Parameters can be supplied in a YAML/JSON file instead of long command lines, which is the reproducible-provenance form:

# params.yaml  (nextflow run nf-core/rnaseq -r 3.14.0 -profile singularity -params-file params.yaml)
input: samplesheet.csv
genome: GRCh38
outdir: results
aligner: star_salmon

The samplesheet and schema validation

Every nf-core pipeline reads a CSV samplesheet whose exact columns are defined by the pipeline's shipped schema (assets/schema_input.json) and validated by the nf-schema plugin at launch, before any compute is spent. Wrong or misordered columns fail fast with a schema error rather than hours in.

sample,fastq_1,fastq_2,strandedness
CONTROL_REP1,/data/ctrl1_R1.fastq.gz,/data/ctrl1_R2.fastq.gz,auto
CONTROL_REP2,/data/ctrl2_R1.fastq.gz,/data/ctrl2_R2.fastq.gz,auto
TREAT_REP1,/data/treat1_R1.fastq.gz,/data/treat1_R2.fastq.gz,auto

• Column names are pipeline-specific: nf-core/rnaseq uses sample,fastq_1,fastq_2,strandedness; nf-core/sarek uses patient,sample,lane,fastq_1,fastq_2. Read the pipeline's docs/usage.md for the exact schema, never guess.
• A single-end sample leaves fastq_2 empty; multiple rows sharing one sample value are merged (technical replicates / multiple lanes), which is how the pipeline knows to concatenate them.
• Absolute paths or URLs are safest; relative paths resolve against the launch directory.

Threading columns into the pipeline is handled by the META MAP convention (worth understanding when reading logs or outputs): each sample flows internally as a tuple [ meta, files ] where meta is a map like [ id:'CONTROL_REP1', single_end:false ]. The samplesheet columns become meta keys, so sample identity and pairing travel WITH the files through every step - which is why outputs and the MultiQC report are labelled by the sample value from the sheet.