Kubernetes Jcm Runs

Running jcm on Kubernetes

The Kubernetes site layer. Read jcm-run for the model/Hydra layer and jcm-benchmark for throughput methodology and reference numbers; this file covers only what running on a cluster adds.

Why use a cluster. Kubernetes gives a pod exclusive use of the GPUs it requests. That is the guarantee benchmarking needs and the one a shared workstation cannot offer — there, a neighbour can land on your card mid-run and quietly corrupt the timing (devbox-jcm-runs). Quota permitting, a multi-config sweep also runs in parallel, finishing in the time of its slowest member rather than the sum.

Portability. The Job shape — clone pinned SHAs, install what the image lacks, run jcm, write to a volume, refuse or survive failure — is generic. Everything cluster-specific lives in scripts/sites.py as a profile: namespace, GPU resource name and selector, storage class, PVC names, image, and the extra pip packages. A second cluster is a new dict there plus --site <name>, not a fork of the generators. Only Nautilus is defined; sites.py documents what a GKE/EKS profile would have to supply (most notably nvidia.com/gpu rather than a vendor quota bucket).

Submitting

S=.claude/skills/kubernetes-jcm-runs/scripts
python $S/mkjob.py --preset ma-t63-l47 --f32              # inspect first
python $S/mkjob.py --preset ma-t63-l47 --f32 | kubectl apply -f -
python $S/mkjob.py --sweep --f32 | kubectl apply -f -     # parallel sweep

Always look at the manifest before applying it. kubectl apply --dry-run=server -f - validates against the API without creating anything.

--sweep skips presets whose inputs no pod can reach, and says which and why. That check is derived from the preset's own file overrides, so a preset becomes runnable automatically once its data is reachable rather than needing an exclusion list edited.

What the generated Job does, and why

• Clones pinned refs at startup. The published image contains a release, so branch code — including the benchmark harness itself — has to come from git. Every ref is resolved to a SHA at generation time: cloning a branch name makes the code depend on when the pod happened to start, and a job submitted seconds before a push silently runs the previous commit.
• Installs what the image lacks. It is built from a release with pip install -e . and no extras, so anything newer or optional is added at pod start (see extra_pip in the site profile, plus per-preset additions). A hard GPU check follows every install, because a pip-induced CPU fallback would still "work" and report timings ~100× slow.
• backoffLimit: 0 for benchmarks. A failed benchmark must be inspected, not silently retried onto a different node with different timings.
• Model output to an emptyDir that dies with the pod; only reports and logs reach the PVC. A benchmark needs the timings, not the fields.
• /dev/shm as a memory-backed emptyDir. The 64 MB Kubernetes default is far too small for JAX/XLA.
• --gpu 0 — the pod sees exactly one GPU, so the harness's free-GPU gate is a no-op. Kubernetes has already guaranteed exclusivity, which is the whole reason to run here.
• Remote boundary data is prefetched before the GPU is claimed. jcm resolves hf:// paths lazily during model construction, which is after the telemetry sampler starts, so a multi-GB bundle would otherwise download inside the timed region.

Production runs — mkrun.py

Benchmarks and production runs want opposite things, so they have separate generators. Do not use mkjob.py for a run whose output you intend to keep.

python $S/mkrun.py --name pd-year | kubectl apply -f -      # 12 calendar months
python $S/mkrun.py --name aci-2yr --months 24 --physics echam-jam-aci \
    --pin jcm=abc1234 | kubectl apply -f -

A run is 12 calendar months from --start-time (default 2000-01-01) by default, under the present-day climatological AMIP forcing (the mirror's forcing_pd/ozone_pd bundles, auto PD emissions/oxidants), written as calendar-month means (<name>_monthly_YYYY-MM.nc) from daily means in 5-day chunks. --days N gives a fixed length instead; --save-chunks also keeps the per-chunk daily files (hundreds of GB for a JAM year). The completion gate reads the per-chunk Chunk K | Day N health lines, so it works without chunk files.

Eviction tolerance is the whole design. A pod is not guaranteed a node for days, so a long run must assume interruption. jcm resumes automatically when its checkpoint exists, so a restarted pod picks up at the last completed chunk instead of restarting the year — that is what makes multi-day runs viable here at all. --retries is effectively the eviction budget.

Pin the code for anything long: --pin jcm=<sha>. A run resumed days later must come back on the same code; a branch name would silently restart it on whatever has since been merged, mid-year.

Output lands under /runs/<name>/ alongside a PROVENANCE file recording the SHA of every repo, appended on each attempt so a resumed run shows its full history.

A run that stops early can still exit 0. run_chunked returns normally when the health gate trips, so Kubernetes would mark a truncated year Complete. mkrun.py therefore checks the health verdict and the day count reached, and fails the Job if either falls short.

Release-validation members and retune arms

Do not hand-build these from mkrun.py's output. The release matrix (tools/release_validation/matrix.yaml) and one-member arms of it (the #682 retune) go through the release launcher, which builds the same Job through mkrun.job_manifest() — the importable engine behind mkrun.py — from the matrix's own override list:

python tools/release_validation/launch.py --site nautilus --submit    # 7 members
python tools/release_validation/launch.py --site nautilus --members echam-jam-t63-l47 \
    --days 60 --init <state> --suffix entrpen20 \
    --extra +physics.convection.entrpen=2e-4 --submit                  # one arm
python tools/release_validation/launch.py --site nautilus --fetch --members <m> --tag <t>

It pins jcm to a pushed SHA and the image to its digest, installs that commit's own requirements in the pod (locked on the first attempt), records each launch so --resume re-emits it unchanged, and has the pod refuse a run directory that belongs to a different launch, or a fresh Job onto a run another Job started (the PBS path's check_fresh, done where the volume is visible). Workflow, the arm recipe and the fetch/score/ingest commands: tools/release_validation/README.md ("Workflow on Nautilus").

A new door that keeps a run's output should call mkrun.job_manifest() too, not copy its YAML or rewrite its jcm.main line.