Jcm Run

Hydra gotchas

• run.time_step is in MINUTES, not seconds.
• save_interval must be ≤ chunk_days. Otherwise a chunk contains zero output times and the chunk write dies with a confusing IndexError: index 0 is out of bounds for axis 0 with size 0 from predictions.to_xarray(). This is easy to hit when shortening a run for a quick test and forgetting to shorten save_interval with it.
• run/longrun.yaml has no checkpoint_path key, so adding one needs Hydra's add syntax: +run.checkpoint_path=.... Plain run.checkpoint_path=... fails with Key 'checkpoint_path' is not in struct. run/default.yaml does define it.
• Ozone: forcing.ozone_file: auto is the shipped default and resolves a packaged climatology matching the grid (jcm/data/bc/t63/ozone.nc — already on L47 levels, already S→N). Leave it alone. Confirm in the log: forcing.ozone_file=auto resolved to .../t63/ozone.nc. On a hybrid grid auto now raises rather than degrading if it resolves nothing: the analytic profile carries ~7.6× the tropospheric ozone column, a large clear-sky OLR bias and not a valid basis for any radiation comparison. The error distinguishes a missing product from a cold cache and names the remedy. For any other grid auto also consults the HF mirror's level-resolved bundles; set one explicitly with forcing.ozone_file=hf://bundles/<grid>_l<levels>/ozone_pd.nc (prefetch on a node with internet), or regenerate a packaged file with jcm.data.bc.interpolate_ozone for offline work. forcing.ozone_file=analytic takes the analytic profile deliberately; a sigma grid, for which no ozone product exists, still warns and falls back.

Watching a run

One tail -F with an alternation that covers both progress and every failure signature — a filter that only matches success is silent through a crash, which reads identically to "still running":

tail -F -n 0 run_logs/PREFIX.log 2>/dev/null | grep -E --line-buffered \
  "Saved .*_day[0-9]+\.nc|Wall: |NaN vars|unhealthy|Traceback|Error|FAILED|Killed|OOM|CUDA_ERROR|HydraException"

NaN vars: N/239 is the health-check line. Parse the count — do not grep for the bare string nan, which matches unrelated output.

specific_humidity in the saved netCDF and the health report is genuinely in g/kg, and the units: 'g/kg' label is correct — state_bridge calls dimensionalize(q, gram/kilogram) on the way out. Healthy tropical surface values are ~20-30 g/kg. (An earlier version of check_health assumed kg/kg and applied a *1000, which double-counted and tripped the q_max > 100 threshold on every chunked run; that conversion was removed. Do not reintroduce the assumption in either direction.)

Failure modes worth recognising

• Chunk write crashes after "Run completed" — a diagnostic emitted a shape data_to_xarray has no dims for. The write order is to_xarray → check_health → to_netcdf → save_checkpoint, so a crash here loses the whole chunk with no checkpoint. Fix by adding the dotted key to ComposablePhysics._EXCLUDED_OUTPUT_KEYS or registering a band coord.
• Editable installs: jcm, jax-rrtmgp and mam4-jax are installed editable, so the working tree is the running code. Check git -C <repo> rev-parse --abbrev-ref HEAD before trusting any result. The site skills cover how to A/B a library version safely (worktree + PYTHONPATH, never git checkout in a shared clone).