Jcm Local CI

Local CI for jax-gcm

Reproduces the CI gates from .github/workflows/run_test.yaml + run_linter.yaml (lint, fast, slow and extras), plus the Claude review, without GitHub Actions.

One command

scripts/local_ci.sh /path/to/worktree                  # lint here, both gates via qsub
scripts/local_ci.sh --local-fast /path/to/worktree     # ...and a fast gate on this node

Lint runs on the current node; both test gates run in a single develop-queue PBS job (select=1:ncpus=16:mem=200GB), fast then slow, sequentially. Watch the job log for FAST_EXIT=0, SLOW_EXIT=0 and the closing GATES PASSED: the job exits non-zero if either gate failed, so a job that ends green means both passed.

Every submission gets its own log, named for the tree it tests: <worktree>/jcm_ci.<tag>.<UTC timestamp>.log, where <tag> is HEAD's short sha, with +dirty.<hash> appended when the worktree differs from HEAD — tracked edits or untracked, non-ignored files, which pytest would collect too; the hash is of that difference (the tracked diff and every untracked file's contents), so two different dirty trees on one commit get different tags. The PBS job is named jcm_ci_<tag>_<timestamp> to match (punctuation replaced by _, to keep the name to characters every PBS accepts). The script prints the exact log path and the watch command when it submits:

tree: 1a2b3c4d.20260930T051200Z
log:  /path/to/worktree/jcm_ci.1a2b3c4d.20260930T051200Z.log
watch: grep -E 'FAST_EXIT|SLOW_EXIT|GATES' /path/to/worktree/jcm_ci.1a2b3c4d.20260930T051200Z.log

The job tests the worktree as it is when the job starts, which an edit or a commit made while it queued can change, so it recomputes the tag then: the log opens with submitted=<tag> tested=<tag> (and a WARNING line when they differ), and every result line carries both, GATES PASSED tree=<tag>.<timestamp> tested=<tag>.

Use the printed path, not a glob over jcm_ci.*.log: an earlier run's log in the same worktree is a verdict on an earlier tree, and a green line in it says nothing about the one just submitted. Check that the tested= tag on the line you read is the sha you pushed.

The gates go to a compute node because a login node caps you at 10 GiB (see docs/source/design/test_suite_memory.md) — well under what an -n 12 fast suite needs, so a local run there reports OOM-killed workers as unrelated test failures. --local-fast opts into an -n 2 fast run on the current node anyway, for a quick read before the job lands; it runs before the submission, never alongside it, because both would fight over the same worktree's .coverage.*.

Which dinosaur the gate tests against

The gate exists to reproduce CI, so it must run the dinosaur that pip install -e . resolves — requirements.txt pins dinosaur>=1.5.0, which carries the semi-Lagrangian transport jcm's backend requires (neuralgcm/dinosaur#135). The script therefore auto-detects nothing: a stale fork checkout sitting in $HOME would silently displace the pinned package and the gate would measure a dependency CI never sees, which is the one thing it is for.

A fork is used only when you pass JCM_DINOSAUR explicitly, and the run then says so and warns that it is not at CI parity. With no override and an installed dinosaur that lacks the SL class, the script fails before lint and names both remedies — pip install -e ., or JCM_DINOSAUR=<checkout>. (Failing is the point: without SL every model-construction test raises, and ~100 unrelated failures bury the ones that matter.)

~/.venvs/jaxgcm satisfies the pin (dinosaur 1.5.0), so gate jobs from it need no override — just run the script. JCM_DINOSAUR remains the escape hatch for a pre-release checkout, and a run using it says so and states that it is not at CI dependency parity.

The gates, individually

# 1. Lint (run_linter.yaml)
ruff check .

# 2. Push gate — fast tests, 90% coverage
JAX_PLATFORMS=cpu pytest -n 12 -m "not slow" --cov=jcm --cov-fail-under=90
coverage report --fail-under=90

# 3. PR gate — slow tests only, 80% coverage vs .coveragerc-pr
JAX_PLATFORMS=cpu pytest -n 4 -m "slow" --cov=jcm \
    --cov-config=.coveragerc-pr --cov-fail-under=80
coverage report --rcfile=.coveragerc-pr --fail-under=80

# 4. extras-tests — every test an optional extra gates, fast and slow, in a
#    SEPARATE venv (installing the extras into the coverage venv breaks
#    dependency parity for gates 2 and 3):
pip install -e ".[$(python tools/ci/optional_extras.py pip-extras)]"
python tools/ci/optional_extras.py check
JCM_REQUIRE_EXTRAS=1 JAX_PLATFORMS=cpu pytest -v -rs -m requires_extra

Gate 4 mirrors the extras-tests job. Under JCM_REQUIRE_EXTRAS=1 the session refuses to start without every extra and a selected test that skips is a failure, so a green run means every gated test ran. It is serial on purpose, as in CI: the pySES delegating-config chunk peaks at ~12.7 GB. About 13 min on the dev workstation. A test gates on an extra only through @pytest.mark.requires_extra(...); any other gate fails gates 2/3 (see tools/ci/optional_extras.py). local_ci.sh does not run it: it needs its own venv, so run the three commands yourself.

Gate 3 runs the whole slow suite in one command. CI runs the same tests as two parallel path shards (slow-tests-radiation / slow-tests-rest, defined with their partition check in tools/ci/slow_shards.py) and enforces the 80% floor on their combined coverage in slow-coverage; the local gate is equivalent.

The trailing coverage report is not redundant — it mirrors the two enforcement steps CI gained in #786, and it is the one that exits non-zero on plugin behaviour nobody controls. See the "Coverage differs by suite" note below.

Gates 2 and 3 are real compute — run them on a develop-queue node, not a login node; local_ci.sh does. The -n 4 on gate 3 is mandatory on Derecho, not an optimization: a single serial process accumulates thousands of mmap'd XLA JIT code sections and dies mid-suite with LLVM ERROR: Unable to allocate section memory! (SIGSEGV/SIGABRT — three attempts at 120–220 GB all failed identically; RAM is not the issue, per-process map count is). Splitting across workers resets the budget. GitHub's runners tolerate the serial run; Derecho's do not. (docs/source/design/test_suite_memory.md covers the related growth in retained XLA executables that the root conftest.py bounds.)

Local Claude review — run it BEFORE pushing, on the right model

The adversarial self-review jcm-dev-workflow step 3 requires before every push that changes code: Codex credits are finite, and a finding Codex makes that this review would have made is a credit burnt and a round lost. Fix or explicitly refute every finding before git push.

Which reviewer depends on the session's model. /code-review high forks the invoking session — same model, full conversation context inherited — and its finders and verifiers fan out the same way, so it is billed to that session at full context. It exhausted a Fable session limit twice on 2026-09-10 (the failure notices name the model: model sent to the API: claude-fable-5-1).

• Opus or Sonnet session: /code-review high on the branch, as before.
• Fable session: do not invoke the skill. Spawn a general-purpose agent with model: opus, give it the same scope (reference formulation, JAX hygiene, tests, docs, comment style, diff vs upstream), and have it post one review via gh pr review <PR> --comment --body-file <file>. Forward its findings to the authoring agent for fix-and-reply. This is how #776's blocker and #783's second-round findings were caught.
• If the maintainer explicitly asks for /code-review from a Fable session, say first that it will fork on Fable at full context, and confirm.

For the deep cloud variant use /code-review ultra (user-triggered, separately billed).