---
name: codetrial-verify
description: How a CodeTrial change is validated - scripts/test.sh as the credential-free gate, which generated artifacts have to be regenerated before it passes, the checks that need credentials or a browser and therefore sit outside it, the browser and mutation lanes, and how to bring the server up for a live look without taking the maintainer's port. Use before calling work done, when a gate fails on drift rather than on a bug, when adding a test, or when a fix needs to be seen working in the app.
---

# Validating a CodeTrial change

One gate needs no credentials and is the thing to run:

```sh
./scripts/test.sh     # the gate CI's `check` job runs
make check            # the same, plus a live Gemini credential check
```

`scripts/test.sh` is a list of named gates that all run before it reports, so
one failure does not hide the next; the summary line names every gate that
failed. It covers `cargo fmt --check`, `clippy -D warnings`, `cargo test`, the
Python unittest suites, the Node browser tests, ESLint, `ruff check`,
`shellcheck`, the generated-artifact drift checks, the hook suite, `cargo-audit`
and `actionlint`.

It is not offline. The `fetch-vendor` gate downloads missing assets named by a
`web/vendor/**/FETCH` manifest; it cannot restore committed vendor files.
`actionlint` publishes itself as a container, so that lane may reach for docker.
Both are checksum-pinned or version-pinned; neither needs a credential.

A lane whose tool is absent skips instead of failing. Skips made directly by
`scripts/test.sh` appear under its final `not checked by this run:` summary.
Formatter skips do not: `scripts/indent.sh` prints them inline while the
`indent` gate keeps running. Read both the final summary and the formatter
output before claiming coverage; never treat the skip set or its size as fixed.
The drift checks always run. That skipping is why a green local run is weaker
evidence than a green CI run, and so is what this script leaves out: CI also
holds a pull request's own commit messages to the rules, mutation-tests the diff
in its own job, and builds release binaries for three targets.

The `indent` gate is the one that surprises people. `scripts/indent.sh --check`
copies the tree, runs the whole formatter chain over the copy, and diffs:
comment reflow with `commentflow`, then `cargo fmt`, `ruff format` and `shfmt`.
Checking the composition rather than each tool is not a flourish. commentflow
puts a blank line before a comment inside a method chain and `cargo fmt` takes
it straight back out, so `commentflow --check` alone can never be satisfied on
Rust. `make indent` runs the same script with `--write`, so the fix for a
failure is always that one command. Never pass `shfmt` a style flag; it reads
`.editorconfig`. Prettier sits beside the chain rather than in it: it shares no
file with the others, so the check runs it in place with `--cache`, alongside
the copy. Its file set is in `scripts/indent.sh` and includes the issue forms,
which ride along to be parsed. Without `npm ci` it skips with a note.

## Drift is the usual failure

Several trees are generated, and the gate compares the committed bytes against
what the generator would write now. A failure here is not a bug in your change;
it means the source moved and the output did not:

```sh
python3 scripts/gen-problems.py
python3 scripts/gen-problem-cards.py   # the problem cards in web/index.html
node scripts/gen-wire-fixtures.mjs     # browser/agent wire fixtures
node scripts/gen-recording-fixtures.mjs
python3 scripts/gen-calibration-fixtures.py
```

Each takes `--check`, which is the form the gate runs. Edit `problem-bank/`,
never the files under `web/problems/` or `web/judges/`. `scripts/gen-problems.py
--sync-study-plan` refuses to write while the plan and `problem-bank/` disagree
and names what each side is missing, so port those first.

## What is deliberately outside the gate

These exercise live server, external-service, or browser flows and are run on
their own when the area they cover is touched:

One command per line: two names on one line runs the first and passes the
second as an argument it ignores. The entry requirements are:

```sh
scripts/browser-check.sh              # Playwright + Chromium; rust/dispatch also need LiveKit and Gemini credentials
scripts/server-check.sh               # cargo, node, curl; starts a server unless CODETRIAL_WEB_URL is set
scripts/gemini-check.sh               # GOOGLE_API_KEY(S) in the selected CodeTrial config
scripts/parity-check.sh                # the credentialed rust browser-check prerequisites
scripts/report-parity-check.sh         # the credentialed rust browser-check prerequisites
scripts/visual-parity-check.sh         # Playwright + Chromium; no service credentials
scripts/recording-provision-check.sh   # gcloud credentials and the CODETRIAL_RECORDING_* values checked at its start
scripts/recording-integration.sh       # --help lists per-phase credentials, tools and required --phase
```

Read the script's validation or usage block before a credentialed run; it is
the source of truth for optional modes and the complete environment-variable
list.

CI additionally mutation-tests the diff: `plan-mutants` counts what the change
is worth and `mutants` runs `cargo-mutants` over it. A surviving mutant means a
line changed behavior with no test noticing, so the answer is a test, not a
retry.

## The git hooks

`make hooks` installs the fast half of the gate at commit time:
`scripts/git-pre-commit.sh` runs `rustfmt`, ESLint, Prettier, `ruff` and
`shellcheck` over a checkout of the index, so an unstaged edit neither fails a
commit nor sneaks through one, plus `commentflow --check` and `shfmt -d` on
staged shell. It does not build, test or check generated-artifact drift; that
is what the gate is for. `scripts/git-commit-msg.sh` holds the message to the
rules it prints with `--rules`, `scripts/git-prepare-commit-msg.sh` splices the
template above a `commit -v` scissors line, and `scripts/git-pre-push.sh`
replays the rules over commits a rebase or an amend rewrote after the fact.
`make hooks` installs every `scripts/git-*.sh`, so adding one there installs
itself. CI runs the same list over
a pull request's own commits, so the rules bind someone who never installed the
hooks as well.

The hooks have their own suite. `scripts/test-git-hooks.sh` builds a scratch
repository, installs the hooks into it and drives every case: the messages that
must be rejected, the template splice above a `commit -v` scissors line, a
staged file failing while the same edit unstaged does not, and a push carrying a
commit that skipped the hook. It runs as the `git-hooks` gate, so editing a hook
without running it is caught here.

## Writing a test

No test code goes under `src/`; codetrial-conventions has that rule and the four
things that bite when it is applied carelessly. Which of the two kinds you are
writing follows from what the test needs to see:

- Reaching a private or `pub(crate)` item makes it a unit test. It goes under
  `tests/unit/`, mirroring the path under `src/`, declared from the `src/` file
  with `#[cfg(test)] #[path = "..."] mod tests;`.
- Reaching only the public API makes it an integration test, so it goes in
  `tests/*.rs` beside the suites already there.

Browser tests go in `tests/browser/*.test.js` under `node --test`, Python tests
in `tests/test_*.py` under `scripts/run-python-tests.py`, which runs a file's
cases across a thread pool; a suite added there must keep every case owning its
own sandbox, or it races. The golden fixtures in `tests/golden/` are the
compatibility contract: a diff there is a claim that the observable output
changed on purpose, and it belongs in the commit body.

A test that passes without running anything is the failure mode this tree has
already been bitten by, hence commits like "Prove an empty test run is not a
pass". Assert on the count as well as the content when a suite discovers its
own cases.

The quieter version is a test that runs and cannot fail: a refusal asserted
against a double that was scripted to refuse, or a hash compared against one the
test computed with the function under test. The check that separates them is
cheap and is the one to run before believing a new test: break the thing it
names, watch it fail, put it back.

## Seeing it work in the app

When a fix needs a live look, build the release binary and start the server
yourself, then say it is ready to test. Do not hand over a command to run.

```sh
make build
./target/release/codetrial web --web-addr 127.0.0.1:3100
```

Port 3000 is the maintainer's own instance. Never bind, restart or kill it;
pass `--web-addr` with another port and name that port in the report. Running
from the checkout picks up `web/` edits with no environment variable set.