Derecho Jcm Runs

Running jcm on Derecho

Layering. This is the Derecho/PBS site layer. The site-agnostic model layer (config groups, Hydra traps, stability overrides) is jcm-run, and throughput methodology is jcm-benchmark; both apply here too. The shared-workstation counterpart is devbox-jcm-runs — worth a glance for the contrast, since there GPUs are self-allocated rather than scheduled — and the Kubernetes counterpart is kubernetes-jcm-runs.

Derecho's A100s are 40 GB, half the cluster and dev-box cards. Throughput matches at equal work, but the memory ceiling does not: the reference table in jcm-benchmark is measured on 80 GB, and only its smallest configs fit here.

Generate a PBS script with scripts/mkjob.py, sanity-check the config, submit, then monitor with the patterns below. Every default here was established by a real campaign; the failure modes listed are ones that have actually happened.

1. Generate and submit

python scripts/mkjob.py --name my_run --days 30 > runs/my_run.pbs
qsub runs/my_run.pbs

Common flags (see python scripts/mkjob.py --help for all):

2. Always pre-flight before burning a queue slot

Two checks, both cheap, both catch failures that otherwise waste a job:

# (a) Hydra composition — catches +/++ prefix errors and unknown keys
JAX_PLATFORMS=cpu python -m jcm.main <exact overrides> --cfg job >/dev/null

# (b) coords constructibility — --cfg job does NOT build coords, so an
#     invalid spectral truncation only fails at runtime
JAX_PLATFORMS=cpu python -c "
from jcm.utils import get_coords
from jcm.physics.echam.echam_levels import get_echam_levels
get_coords(vertical_coords=get_echam_levels(<layers>), spectral_truncation=<T>)"

mkjob.py --check runs (a) for you and prints the command for (b).

Hydra override prefixes are a recurring trap: a key that already exists in the composed config takes no +; one that does not, requires it. run=longrun replaces the whole run group, so run.checkpoint_path needs + under it but not under the default run config.

3. Environment (baked into generated scripts)

source ~/.venvs/jaxgcm/bin/activate
export PYTHONPATH=$REPO                   # jcm worktree wins over the venv's editable install
export JAX_PLATFORMS=cuda,cpu
export MAM4_JAX_ENABLE_X64=0              # f32 MAM4 core (forward-only); f64 default is much slower
export XLA_PYTHON_CLIENT_MEM_FRACTION=0.93   # 0.85 when ngpus>1 — 0.93 starves CUDA command buffers

Overridable site paths: JCM_REPO, JCM_VENV, JAM_INPUTS, JCM_EMISSIONS, PBS_ACCOUNT, SCRATCH.

Transport is always semi-Lagrangian (the Eulerian path was removed); the venv's dinosaur must be >= 1.5.0 (requirements.txt). Without it the dycore raises a clear install-instruction error; there is no fallback.

4. Input data

reference/data_paths.md lists every data source. The default is the HF data mirror (--data mirror --era pd|pi): mkjob.py derives every bundle path from --grid — terrain, forcing, emissions, DMS, dust, plus level-resolved ozone and oxidants from bundles/<grid>_l<levels>/ — and prefetches them on the login node at generation time, baking the local cache paths into the job. Compute nodes need no internet, and every grid/level combination the mirror carries (t63/t106/t127/t255 × l47/l95) works the same way: no packaged-grid special cases, no purge-eligible scratch files, and a grid/level mismatch fails at generation, not in the queue.

--data local keeps the legacy prepared-file behaviour (JAM_INPUTS / JCM_EMISSIONS, existence-checked before qsub). Its inputs are all grid-specific — level-resolved (ozone, oxidants) or horizontally validated (emissions, DMS, dust). forcing.ozone_file: auto resolves the packaged climatology (T63L47) and then the mirror's per-grid bundle, and on a hybrid grid raises if neither resolves rather than substituting the analytic profile (~7.6x the tropospheric ozone column). Prefer the mirror.

5. PBS facts specific to this machine

• GPU account is UCSD0085 (UCSD0044 is casper-only and is rejected).
• gpu_type=a100 must be inside the select chunk, not a separate -l.
• -q main is a routing queue that lands GPU jobs in gpu; gpudev exists for short interactive-style debugging.
• qsub -v VAR=x does not reach the job environment here — generated scripts hardcode their variables.
• Keep #PBS -m abe so job mail keeps working (it was silently lost once when a script was derived by sed from one that omitted it).
• Use set -euo pipefail; without -e a failed run still reaches a trailing touch DONE and looks successful.

6. Monitoring (scripts/watch_job.sh)

scripts/watch_job.sh <jobid> <logfile> "<completion marker>"

Use it as the command of a persistent Monitor. It encodes five lessons:

1. Read the log once per check. Grep the log into a variable, then both decide and report from those same bytes. Live NFS logs give stale re-reads, which produced repeated phantom "failures" whose detail printed empty.
2. Debounce: a failure signature must persist across two checks.
3. 3-strike qstat: PBS requeues and transient qstat errors otherwise look like a vanished job.
4. File existence is not success: the driver writes chunk netCDFs before the NaN check. Verify the NaN vars: 0/N health line instead.
5. Match verdicts, not keywords: jcm.main echoes the whole composed config on stdout, so every log contains bail_on_unhealthy: true. A bare unhealthy in FAIL_RE therefore failed every clean run.

Filter Lmod's "unknown module" noise — it is harmless on these nodes. scripts/watch_job_test.py (standalone, like mkjob_test.py — pytest does not collect dotted directories) checks both halves: a clean log passes and a real verdict still fails.