Slurm Pipeline Manager

slurm-pipeline-manager

This skill takes an EKP Julia example (decoupled or monolithic) and produces a complete SLURM job-dependency tree inside a slurm-variant/ subdirectory. slurm-variant/ is fully self-contained — it is both the HPC submission directory and the Julia project root. The original example is never modified.

The generated tree uses a one-shot precompile job followed by a dependency-chained setup → forward_map[array] → update_ensemble loop across iterations, ending in postprocessing — so ensemble members run in parallel and every stage pays precompile cost only once.

It also supports maintenance mode: if slurm-variant/hpc_config.sh already exists, re-invoking the skill updates files in place without clobbering the user's manifest settings.

Core principle — slurm-variant/ is fully self-contained

Everything new lives in slurm-variant/. The original example directory is never modified.

This includes:

• All SLURM / shell scripts (.sbatch, .sh)
• All decoupled Julia scripts (initialize_EKP.jl, run_forward_model.jl, etc.)
• Any shared model file (model.jl or similar)
• Parameter TOML files (priors.toml)
• A Project.toml copy with any additional dependencies

slurm-variant/ is the working directory for every HPC job — users cd into it before running anything. This means JULIA_PROJECT="." in hpc_config.sh and script names have no path prefix (e.g. SETUP_SCRIPT="initialize_EKP.jl").

How the generated pipeline is structured

precompile ─(manual, run first)

setup ──afterok──► fwd_iter_0[array 1..N] ──afterok──► update_iter_0
      (afterok chain continues for each iteration)
      ──afterok──► fwd_iter_K-1[array 1..N] ──afterok──► update_iter_K-1
                                                         ──afterok──► postprocess

Key design choices:

• One precompile job only. Every run-stage job sets JULIA_PKG_PRECOMPILE_AUTO=0. Without this, 60 concurrent array tasks each attempt to precompile — thrashing the shared Julia depot and wasting time.
• Forward map as a SLURM array. --array=1-$N_ENSEMBLE gives one task per ensemble member; the iteration index travels via --export=ALL,ITERATION=$i.
• afterok + --kill-on-invalid-dep=yes everywhere. If one stage fails the whole tree is killed, not left stuck in the queue. EKP errors on < 2 successful members (commit 8f2d3fa), so partial-run continuation is not useful.
• Date-stamped output root. RUN_DATE in hpc_config.sh pins all jobs in a run to the same output directory.

Workflow

Mode 0 — Detect generate vs. maintenance

Check whether <example_dir>/slurm-variant/hpc_config.sh already exists. If so, enter maintenance mode: read the existing hpc_config.sh as the source of truth, understand the user's requested change, and edit only what is necessary inside slurm-variant/ — do not overwrite hpc_config.sh with fresh template defaults.

If slurm-variant/ does not yet exist, proceed with generate mode below.

Step 1 — Read the example and map the pipeline stages

Read every .jl file in the named example directory and determine whether the pipeline is already decoupled or monolithic.

Decoupled pipeline — separate files for each stage:

Record SETUP_SCRIPT, FORWARD_SCRIPT, UPDATE_SCRIPT, POSTPROCESS_SCRIPT.

Monolithic pipeline — a single .jl file runs the full EKP loop inline (prior → truth → EKP init → forward loop → update loop → plot). This cannot be submitted to SLURM as-is. You must split it into the decoupled roles above. Write ALL split scripts into slurm-variant/ — never create new .jl files in the original example directory.

When splitting a monolithic script:

• Put shared model code (forward map function, model definition) in a model.jl (or <name>_model.jl) inside slurm-variant/; other scripts include() it. Because @__DIR__ in Julia resolves to the script's own directory, include("model.jl") correctly finds the file inside slurm-variant/ regardless of where Julia was launched from.
• Put parameter TOML definitions in priors.toml inside slurm-variant/.
• Create slurm-variant/Project.toml as a copy of the original with any new dependencies added (typically JLD2, TOML). The original Project.toml is never touched.
• Fix the RNG race: in monolithic scripts the forward map often uses a shared module-level RNG. In the SLURM variant each array task runs in its own process, so seed each member's RNG deterministically from (iteration, member), e.g. Random.MersenneTwister(iteration * 10_000 + member).

Data-generation merge pattern: If there is a separate one-off data/truth generation script (e.g. generate_data.jl), merge it as a sequential pre-step inside setup.sbatch — run it first, then the setup/init script. Bundle them to avoid an extra SLURM job and dependency.

Adapt script args: Read the ARGS usage in each Julia script carefully — many pipelines need arguments beyond output_dir iteration member. Common extras:

• Forward map: data_path, eki_path
• Update: eki_path, priors_toml

Add any extras as variables in hpc_config.sh (e.g. DATA_PATH, EKI_PATH, TOML_PATH) and thread them through the relevant sbatch files.

Read src/TOMLInterface.jl to understand the iteration_<i>/member_<j>/ directory layout that save_parameter_ensemble and path_to_ensemble_member use.

Step 2 — Verify pipeline components and report issues

Check for the following required components and report findings honestly. Do not edit the user's original Julia files unless they confirm each fix. Present a checklist:

✓  Prior built via get_parameter_distribution / constrained_gaussian
✓  Noise Γ estimated and passed to EnsembleKalmanProcess
✓  Algorithm settings: N_ensemble=6, N_iterations=5, process=Inversion()
✓  EKP iteration loop: forward-map per member + update_ensemble!
✓  JLD2 save of eki, prior, param_dict
⚠  run_computer_model.jl copies path_to_ensemble_member locally — import from
    EKP.TOMLInterface instead to avoid drift
⚠  RNG is round-tripped through truth.jld2 per member — race condition under
    SLURM array parallelism. Fix: seed per-member RNG from (iteration, member).
⚠  save_file inconsistency: "parameters.toml" in init but "parameters" in update

Flag the following SinusoidInterface-class issues whenever you see them:

1. EKP functions duplicated in the forward-map script (path_to_ensemble_member, get_parameter_values) — risk of drift if the real API changes.
2. Mutable RNG state stored in the shared truth file and re-read/written by every member — race condition under SLURM array parallelism.
3. Inconsistent save_file argument between init and update scripts.
4. Legacy EnsembleKalmanProcess(params, y, Γ, process) positional constructor — still works in v2.7.1 but predates the Observation API.
5. Missing [compat] in Project.toml.

After presenting the checklist, ask: "Should I apply the flagged fixes? I can apply them all, apply specific ones, or leave the Julia as-is and just generate the HPC scripts."