Release Sample Sweep

Release sample sweep (chrome-devtools)

Before a release, every user-facing surface is exercised the way a user exercises it. This skill is that procedure written down.

Three surfaces, three drivers — all three are release gates:

When to run it

• Before cutting any release. Non-negotiable — it is the last gate before release-4x.yml.
• After a change to the Console bundle (modules/spring-boot-starter/frontend/), since the Console is both the shipped sample UI and the validation surface.
• After a change to a shared module that every sample transitively depends on (modules/cpr, modules/ai, modules/spring-boot-starter, modules/admin).
• After a dependency bump wave — two of the last three sweeps found a version-skew bug that compiled clean and only failed at runtime.

What it catches that CI does not

CI builds and tests modules; this sweep exercises packaged artifacts in a browser. The gap between those is where the real bugs live:

scripts/release-gate-samples.sh automates the boot-and-assert half of this in CI. This sweep is the browser half on top of it — the layer that sees rendering, streaming, transport headers, tool cards, and console errors.

The shape of the sweep

Step 0   Preconditions  — build everything, start Ollama, free the ports, open the ledger
Step 1a  Samples        — 33 samples: launch → drive → collect → verdict → teardown
Step 1b  Expo client    — the RN client in the iOS simulator
Step 1c  CLI            — atmosphere run/new/compose/import/checkpoint + distributions
                           ALL OF PHASE 1 IS COLLECT-ONLY. Do not fix anything mid-sweep.
Step 2   Triage         — classify every finding, rank by blast radius
Step 3   Fix            — root-cause fix + a regression test per issue, in the right
                           suite, each proven to bite
Step 4   Re-test        — the failed surfaces in full, plus the blast-radius subset
                           of already-passing ones
Step 5   Report         — vault report, CI green, memory updated

Step 1 is deliberately fix-free. Fixing mid-sweep changes the artifact under test and invalidates every sample already verified against the old one. The one exception: a defect that blocks the sweep itself from continuing — fix it, say so in the ledger, and note which already-passed samples were re-run.

Non-negotiables

1. chrome-devtools, never curl, for validation. curl is allowed only for port readiness and for headless wire protocols (A2A/MCP/REST) that serve no HTML. A "works via curl" claim skips the whole JS layer and is a false pass.
2. The Atmosphere Console is the UI. Drive /atmosphere/console/ (Spring Boot samples redirect / there). A sample that needs a bespoke page instead of the Console is itself a finding.
3. Assert the rendered element, not the payload. An image node with a src is a rendered screenshot; the same base64 in a StaticText node means nothing rendered it. "Server started", "HTTP 200", and "bytes present in the DOM" are not passes.
4. Boot the packaged artifact. java -jar (or quarkus-run.jar), never spring-boot:run / quarkus:dev. Both historical bugs above existed only at artifact level.
5. Kill by PID, never pkill -f. Never touch a port or process the sweep did not start — if a port is occupied, move to another port. The same rule covers the machine's network: never run networksetup, never take an interface down, never touch VPN/DNS/proxy settings. The host's Wi-Fi carries every session the maintainer has open, and a sweep interrupted mid-toggle can leave the machine offline indefinitely. Any assertion that needs real network loss is recorded PARTIAL with its unit coverage cited — see references/expo-sweep.md.
6. Model limitation ≠ framework bug. A small local model emitting invalid tool-call arguments is a model limitation; record it as such and prove it by re-running the same flow on a capable model before calling it a regression.
7. Never write "flaky". Reproduce it, or explain the mechanism. If neither is possible yet, it is a FAIL with an open question, not a dismissal.
8. Report honestly. PASS / PARTIAL / FAIL with one line of concrete evidence each. PARTIAL must name what was not proven and why.

Step 0 — Preconditions

git status --porcelain                    # must be clean
git rev-parse --short HEAD                # record this SHA in the ledger
grep -m1 '<version>' pom.xml              # record the version under test

./mvnw install -DskipTests -Pfastinstall  # full reactor: framework + every sample jar
./scripts/sync-console-bundle.sh --check  # the Console you will drive must be current

ollama list                               # qwen2.5:3b + qwen2.5:7b-instruct-q4_K_M
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:11434/v1/models

• LLM backend is local Ollama, keyless. Use qwen2.5:3b for streaming samples and qwen2.5:7b-instruct-q4_K_M for tool-heavy agents — 3b emits invalid tool-call arguments and Ollama answers 400. Note real-ollama is a CI-harness alias only; AiConfig matches the literal local.

• The launcher scrubs ambient LLM env (LLM_API_KEY, LLM_BASE_URL, LLM_MODE, LLM_MODEL, and the provider keys) from every sample it boots, so the sweep is reproducible on any machine. SWEEP_KEEP_ENV=1 inherits instead. If you boot a sample by hand, scrub them yourself — a maintainer's profile routinely exports these.

• Always read the resolved endpoint out of the boot log before driving:

  grep 'AI config:' target/sweep/<sample>.log

  Expect mode=local … endpoint=http://localhost:11434/v1. Anything else means the sample is not talking to Ollama and the turn's result says nothing about this build. An explicit LLM_BASE_URL outranks the mode by design, so an inherited one silently redirects a "local" run to a remote provider — that is what happened on the 2026-08-07 shakedown before the scrub existed.

• Do not use a paid key. The paid-LLM lane is retired; quota starvation is what made the 2026-06 sweep report nine samples as plumbing-only.

• Do not use embacle (embacle-server --provider claude_code) for tool-calling samples — it applies the host CLI's own configuration to responses and its tool-call fidelity is inconsistent. It is only useful to demonstrate "a capable model completes this flow cleanly", then stop it.

• Ports: the sweep runs on the 9101+ block so it never collides with the samples' own defaults or the Playwright fixture's 8080–8104. Assignments are in references/sample-matrix.md.

• Open the ledger at claude_docs/sample-sweep-<YYYY-MM-DD>.md (a gitignored symlink into the vault, so it survives context compaction). Template: assets/ledger-template.md. Write each row as you finish that sample, never in a batch at the end.

Step 1a — The per-sample loop

Work through references/sample-matrix.md in order. For each sample:

# 1. Launch (the helper refuses to boot if the port is already answering)
.claude/skills/release-sample-sweep/scripts/sweep-sample.sh start <sample> \
    --port <9101+n> --ready-path <path> --env LLM_MODE=local --env LLM_MODEL=qwen2.5:3b

2. Fresh browser page per sample — new_page on the drive URL. Never reuse the previous sample's page: stale state and leftover console noise both corrupt the evidence.
3. Snapshot — take_snapshot. Confirm the Console mounted and the transport badge reads what the matrix expects (Connected · websocket / · webtransport / · grpc / · ag-ui). A transport that silently fell back is a finding.
4. Drive the headline flow for that sample — the exact interaction is in the matrix, the mechanics per surface class are in references/driving-recipes.md.
5. Wait for the rendered result — wait_for the expected text/element, then re-take_snapshot and confirm the node type (see non-negotiable #3).
6. Collect the evidence, all three sources:
   • list_console_messages — every error and warning, verbatim
   • list_network_requests — any non-2xx/failed request
   • sweep-sample.sh warnings <sample> — server-side WARN/ERROR/exception/SLF4J Record warnings even when the sample passes. The warning inventory is half the value of the sweep and is what the next release's triage starts from.
7. Verdict + one-line evidence into the ledger:
   • PASS — headline feature observed rendered, no unexplained console error, no server exception.
   • PARTIAL — plumbing proven, headline feature not observed, reason named (missing third-party key, Docker unavailable, model limitation).
   • FAIL — feature broken, error frame, exception, or the sample won't boot.
8. Teardown — close_page, then sweep-sample.sh stop <sample>. The helper verifies the port is actually released; if it is not, stop and investigate before the next sample claims it.