Bio Workflow Management Wdl Workflows

Version Compatibility

Reference examples tested with: Cromwell 87+, miniwdl 1.12+, WDL spec 1.0/1.1/1.2

Before using code patterns, verify installed versions match. If versions differ:

• CLI: <tool> --version then <tool> --help to confirm flags

If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

Note: every WDL file must open with a version 1.0/1.1/1.2 header; omitting it selects the old draft-2 dialect with no ~{} interpolation. The first-class Directory type is a 1.2 feature, NOT 1.1; min/max/None arrived in 1.1. Cromwell is the JVM production engine that powers Terra; miniwdl is the Python engine used for local dev, static linting (miniwdl check), and readable errors. Pin every docker: by @sha256: digest, never a floating tag.

WDL Workflows

"Build a WDL pipeline for Terra/AnVIL or a GATK best-practices run" -> Declare tasks (a containerized command with typed inputs/outputs and a runtime block) and wire them in a workflow, then run on Cromwell (cloud/Terra) or miniwdl (local).

• CLI: womtool validate / womtool inputs (Cromwell toolkit), miniwdl check (static lint + ShellCheck), cromwell run / miniwdl run (execute)
• WDL: version header, task/workflow/call, scatter fan-out, runtime { docker, cpu, memory, disks }

The governing principle: the runtime block is a cost + reliability CONTRACT

WDL is not chosen on language merits; it is the language of a gravitational system - GATK Best Practices -> Cromwell -> Terra/AnVIL/BioData Catalyst -> Dockstore -> WARP (Van der Auwera & O'Connor 2020). One targets WDL because the data or the collaborators already live in that NIH-cloud ecosystem, and to run vetted GATK pipelines without reinventing them. The design bet is human readability over expressive power. The corollary that governs every real decision: on a cloud backend the engine spins up a FRESH VM per task, so the author must declare its CPU, memory, and disk. The runtime block is therefore a cost-and-reliability contract, not decoration, and three traps follow from it:

• Localization dominates cost and wall-time. The engine COPIES (localizes) every input File from object storage onto the VM's local disk before the command runs, then delocalizes outputs back. A 30 GB CRAM's transfer can dwarf the compute. Disk math, call caching, and preemptibles all exist to manage bytes moved - subset early and avoid re-localizing the same reference into every scatter shard.
• Under-sized disks kills the job LATE. A static disks: "local-disk 100 HDD" fails on the one sample bigger than guessed, after an hour of localization, with a cryptic "No space left on device". Size disk dynamically from size().
• A pipeline without pinned containers is not reproducible. docker: "gatk:latest" silently breaks reproducibility AND busts call caching, because the cache key hashes the resolved image identity. Pin by @sha256: digest.

A clean WDL over unpinned tools is not reproducible: the engine pins step order (layer 1); the author must still pin the container by digest, the reference build, and the parameters.

Decision: choose WDL, and choose its engine

Practical loop: author and lint with miniwdl check locally -> validate and scaffold inputs with womtool -> run at scale on Cromwell/Terra. When Cromwell throws a JVM stack trace, reproduce under miniwdl run for a message that points at the WDL line.

preemptible: 3 is an Int (retry on a preemptible VM up to 3 times, then fall back to on-demand), NOT a Boolean.

Task and workflow: the reference shape

A task bundles a container, typed inputs, a heredoc command with ~{} placeholders, typed outputs, and a runtime block. A workflow calls tasks and passes one call's output to the next by name.

version 1.0

task fastp {
    input {
        String sample_id
        File reads_1
        File reads_2
        Int threads = 4
    }
    # ~{} is the WDL-idiomatic placeholder; ${} collides with bash parameter expansion.
    command <<<
        fastp -i ~{reads_1} -I ~{reads_2} \
            -o ~{sample_id}_R1.fq.gz -O ~{sample_id}_R2.fq.gz \
            --json ~{sample_id}.json --thread ~{threads}
    >>>
    output {
        File trimmed_1 = "~{sample_id}_R1.fq.gz"
        File trimmed_2 = "~{sample_id}_R2.fq.gz"
    }
    runtime {
        docker: "quay.io/biocontainers/fastp@sha256:<digest>"   # digest, not :latest
        cpu: threads
        memory: "4 GB"
    }
}

workflow trim {
    input { String sample_id; File r1; File r2 }
    call fastp { input: sample_id = sample_id, reads_1 = r1, reads_2 = r2 }
    output { File out_1 = fastp.trimmed_1 }
}

Scatter: explicit parallel fan-out (no channels)

WDL parallelism is explicit: build an Array, scatter over it (implicitly parallel), and the engine auto-gathers each shard's output into an Array in input order. There is no lazy channel to drain.

scatter (idx in range(length(sample_ids))) {
    call align {
        input: sample_id = sample_ids[idx], reads = fastq_files[idx], reference = reference
    }
}
# align.bam outside the scatter is an Array[File], gathered in input order.
output { Array[File] bams = align.bam }

Bundle per-sample fields into a struct (struct SampleData { String id; File bam }) and scatter over Array[SampleData] to avoid parallel-array index bugs; see usage-guide.md.