Codetrial Verify

Validating a CodeTrial change

One gate needs no credentials and is the thing to run:

./scripts/test.sh     # the gate CI's `check` job runs
make check            # the same, plus a live Gemini credential check

scripts/test.sh is a list of named gates that all run before it reports, so one failure does not hide the next; the summary line names every gate that failed. It covers cargo fmt --check, clippy -D warnings, cargo test, the Python unittest suites, the Node browser tests, ESLint, ruff check, shellcheck, the generated-artifact drift checks, the hook suite, cargo-audit and actionlint.

It is not offline. The fetch-vendor gate downloads missing assets named by a web/vendor/**/FETCH manifest; it cannot restore committed vendor files. actionlint publishes itself as a container, so that lane may reach for docker. Both are checksum-pinned or version-pinned; neither needs a credential.

A lane whose tool is absent skips instead of failing. Skips made directly by scripts/test.sh appear under its final not checked by this run: summary. Formatter skips do not: scripts/indent.sh prints them inline while the indent gate keeps running. Read both the final summary and the formatter output before claiming coverage; never treat the skip set or its size as fixed. The drift checks always run. That skipping is why a green local run is weaker evidence than a green CI run, and so is what this script leaves out: CI also holds a pull request's own commit messages to the rules, mutation-tests the diff in its own job, and builds release binaries for three targets.

The indent gate is the one that surprises people. scripts/indent.sh --check copies the tree, runs the whole formatter chain over the copy, and diffs: comment reflow with commentflow, then cargo fmt, ruff format and shfmt. Checking the composition rather than each tool is not a flourish. commentflow puts a blank line before a comment inside a method chain and cargo fmt takes it straight back out, so commentflow --check alone can never be satisfied on Rust. make indent runs the same script with --write, so the fix for a failure is always that one command. Never pass shfmt a style flag; it reads .editorconfig. Prettier sits beside the chain rather than in it: it shares no file with the others, so the check runs it in place with --cache, alongside the copy. Its file set is in scripts/indent.sh and includes the issue forms, which ride along to be parsed. Without npm ci it skips with a note.

Drift is the usual failure

Several trees are generated, and the gate compares the committed bytes against what the generator would write now. A failure here is not a bug in your change; it means the source moved and the output did not:

python3 scripts/gen-problems.py
python3 scripts/gen-problem-cards.py   # the problem cards in web/index.html
node scripts/gen-wire-fixtures.mjs     # browser/agent wire fixtures
node scripts/gen-recording-fixtures.mjs
python3 scripts/gen-calibration-fixtures.py

Each takes --check, which is the form the gate runs. Edit problem-bank/, never the files under web/problems/ or web/judges/. scripts/gen-problems.py --sync-study-plan refuses to write while the plan and problem-bank/ disagree and names what each side is missing, so port those first.

What is deliberately outside the gate

These exercise live server, external-service, or browser flows and are run on their own when the area they cover is touched:

One command per line: two names on one line runs the first and passes the second as an argument it ignores. The entry requirements are:

scripts/browser-check.sh              # Playwright + Chromium; rust/dispatch also need LiveKit and Gemini credentials
scripts/server-check.sh               # cargo, node, curl; starts a server unless CODETRIAL_WEB_URL is set
scripts/gemini-check.sh               # GOOGLE_API_KEY(S) in the selected CodeTrial config
scripts/parity-check.sh                # the credentialed rust browser-check prerequisites
scripts/report-parity-check.sh         # the credentialed rust browser-check prerequisites
scripts/visual-parity-check.sh         # Playwright + Chromium; no service credentials
scripts/recording-provision-check.sh   # gcloud credentials and the CODETRIAL_RECORDING_* values checked at its start
scripts/recording-integration.sh       # --help lists per-phase credentials, tools and required --phase

Read the script's validation or usage block before a credentialed run; it is the source of truth for optional modes and the complete environment-variable list.

CI additionally mutation-tests the diff: plan-mutants counts what the change is worth and mutants runs cargo-mutants over it. A surviving mutant means a line changed behavior with no test noticing, so the answer is a test, not a retry.