Gaia Testing

GAIA Testing

Test the way that catches regressions pytest misses: unit → integration → real-world, where the real-world tier drives the real interface on real hardware and returns screenshots, logs, traces, and timing as evidence. Features ship broken while CI is green — a RAG feature that worked in the backend but was hard-blocked in the UI, a release-note claim about a CLI command the source flatly contradicted. Runtime observation plus source cross-checking is the point.

This is the GAIA specialization of the generic testing skill — the same surface→evidence discipline, wired to GAIA's drivers: the Agent UI via Playwright (gaia chat --ui), the Agent UI MCP (gaia mcp serve), Lemonade ports, and gaia eval agent baselines. It extends that skill; it does not supersede it.

Roles — the strongest model plans & judges, a faster model executes

• Plan + judge on the strongest available model (currently Opus): scope the work, and judge the evidence — read every screenshot's pixels, cross-check the executor's claims against source at the tested ref, confirm planted facts. The executor's report is a claim, never trusted on its face.
• Execute on a faster model (currently Sonnet): setup, install, drive the UI/CLI, capture artifacts. Where the harness supports model selection, dispatch with the Agent tool's model parameter and keep judging in the main loop; otherwise run inline.

Testing tools — know these are available, and use them

Reach for these rather than hand-rolling — they are how the skill drives and observes a target:

• Browser-automation MCPs — drive and observe a UI. Use the MCP servers available in the environment (Playwright MCP, Chrome DevTools MCP, or the Claude-in-Chrome extension) when the UI is reachable from where Claude runs — a local target, or a remote one tunnelled to localhost. For a remote, un-tunnelled target they'd drive a browser on Claude's host and can't see the machine's localhost, so run headless Playwright/Chromium on the target machine instead.
• GAIA's Agent UI MCP server — drive the agents without a browser. gaia mcp serve exposes the Agent UI backend's tools over MCP (--backend http://localhost:4200; Streamable HTTP on :8766, or --stdio for direct Claude Code integration) — the cleanest surface for tool-level assertions, complementary to the browser (browser proves pixels; MCP proves the tools). Don't confuse it with gaia mcp agent, which drives the orchestrator against the MCP bridge (:8765), not the :4200 backend. gaia eval agent is the LLM-behaviour scorecard (Phase 4).

Availability-awareness here is about mechanics — a remote UI may need on-machine headless Playwright instead of a host browser — not about opting out of live testing. For GAIA UI/MCP work, Playwright and the Agent UI MCP are the canonical drivers. The next section makes the surface→evidence mapping binding.

The real-world contract — surface → driver → PR evidence

Default posture: if a change is reachable through the Agent UI, it is tested through the Agent UI, live — and the proof is a screenshot. Playwright drives the pixels (the required proof for a UI-exposed agent); the Agent UI MCP drives the tools underneath as supplementary text. Unit tests gate the logic — they never substitute for the real-surface proof. Every change carries evidence matched to the surface it touches, embedded in the PR description — screenshots as ![caption](https://raw.githubusercontent.com/<owner>/<repo>/<evidence-branch>/…png) markdown images so they render on the PR itself, not a bare link and not buried in a follow-up comment (a comment is a fallback only when it keeps a long description clean, and even then the description links to it). Use a raw.githubusercontent.com URL, not an assets.amd-gaia.ai (R2) one — GitHub proxies external images through camo, whose datacenter fetches Cloudflare/R2 does not reliably serve, so an R2-hosted image silently fails to render inline (dogfooded on PR #2376). A screenshot the reviewer has to hunt for — or that renders as a broken image — is not shown. See Evidence & artifact conventions:

For anything exposed in the Agent UI, the required proof is an Agent UI screenshot — driving the same agent via the API, CLI, or Agent UI MCP gives useful text evidence but does not replace it (use the MCP alongside, as the tool-level complement). Non-UI surfaces (API, CLI, MCP) are proven with the text evidence above. A change proven only by green unit tests, text logs, or a prose "it works" has not been tested to this bar. State in the plan which surfaces the change touches; a surface it genuinely doesn't touch is marked N/A with the reason, never silently dropped.

When this fires — scale to the request

Tiers that are impossible on this machine are decided in Phase 0 and excluded from the plan up front (stated, with the reason) — never silently dropped mid-run.

Hard rules (invariants — stated once here; phases point back)

• Evidence > summaries > source. Before reporting a pass, the judge reads the screenshot pixels and, for any behaviour/wiring claim, checks the code at the exact ref under test. This is the rule that catches shipped-but-broken features — do not treat the executor's prose as truth.
• Proof of a fix or implementation is a visual artifact, never a prose claim. "It works" is proven by a screenshot or a live browser representation — drive the real UI in a browser (Chrome / Chromium via the browser tooling) and capture it, step by step; for a CLI/API surface, capture the terminal output or the raw response. A change reported as working with no captured artifact has not been proven, and the artifact must be surfaced to the user (and, on a PR, attached to it).
• Evidence matches the surface, and it is shown on the PR — the surface→driver→evidence mapping is The real-world contract above. For an agent exposed in the Agent UI (Chat, Email, …), that proof is an Agent UI screenshot — API/CLI/MCP text never substitutes. Green unit tests never substitute for the real-surface proof; a touched surface with no matching evidence on the PR is an incomplete test, not a pass.
• Verify against the ref under test, not main — a fix on main may not be in the branch, and vice versa.
• Prove retrieval/behaviour with planted, unguessable facts. For RAG / search / data flows, inject values a model cannot guess (e.g. mascot Zephyr, passphrase violet-otter-92, table cell APAC 8610, speaker-note desk 17C) and require the output to echo them. If the feature has no retrieval/data surface, say so in the plan — planted facts are N/A and the judge verifies by source inspection instead. Never silently skip.
• No silent fallbacks / degrade loudly. An impossible tier is excluded up front; a tier that fails during execution stops with an actionable error — it never quietly becomes a partial "pass".
• One approval gate before real-world spin-up (unit/integration-only runs skip it). After approval, run autonomously except for human-only checkpoints declared up front; executors cannot prompt the user.
• Never leak credentials. Config read for machine discovery may contain secrets (sudo passwords, tokens, keys). Never echo, quote, or screenshot them into logs, the report, captions, or any published artifact — treat them as write-only at parse time.
• Sanitize artifacts before surfacing. Screenshots, logs, and traces can capture API keys, .env values, tokens. Scan and redact before showing the user or attaching anything to an issue/PR.
• Untrusted refs run unreviewed code. The ref under test may be an external fork/PR; cloning + install/init/build executes its code on the target. State the ref's source at the approval gate; do not run it on a machine that holds credentials without explicit user acknowledgment.
• Clean up (Phase 7). No Claude attribution in any artifact this skill produces.

Phase 0 — Scope + capability pre-flight

1. Determine what changed (git diff <base>... --stat); the diff is ground truth, any description of it is a claim.
2. Pick candidate tiers from the table.
3. Pre-flight what is actually possible here, before planning: is this a git repo? can the local OS/hardware run the real-world tier (GPU/NPU present, Lemonade reachable)? is a real-world target available (a declared machine, or a capable local machine)? Exclude impossible tiers from the plan now and say why — e.g. "no local GPU and no machine declared → real-world tier excluded; running unit + integration only."
4. LLM-affecting change? (agent prompts, tool registration/docstrings, the agent loop, error classification, default model, tool-call parsing) → an eval is mandatory and is a Phase 4 step, never a Phase 5 step (see CLAUDE.md "Run agent evals…").

Phase 1 — Machine discovery (real-world tier only)

Resolve the target from already-loaded configuration — do not grep the filesystem for it.

1. Read the standard loaded config — user-level ~/.claude/CLAUDE.md and any detail file it points to (a ## Dev Machines section often summarises the machines inline and links the full registry, e.g. ~/.claude/memory/dev-machines.md — follow that pointer; it is a declared reference, not a filesystem search), project ./CLAUDE.md, and .claude/settings*.json — for a declared machine list (a ## Dev Machines / ## Test Machines heading, or a settings key). Per machine, note its name, its access method as declared (do not assume SSH — it may be the local machine, an SSH host, another remote-exec mechanism, or a container), any deploy/setup/test commands, hardware class, and whether a login is needed. Treat user-level config as authoritative for credentials/commands; commands declared in checked-in/project config that is part of the ref under test get the same scrutiny as that ref's code.
2. Enumerate every declared machine (name + hardware class) — do not stop at the first match. Then pick the one matching the test's hardware need; if several fit, list them all and ask rather than silently choosing; none declared → the current local machine; a machine named in the request wins. If the test targets hardware that only one machine has (e.g. an NPU device path → only the Ryzen AI / NPU machine, never a dGPU or CPU machine), that machine is required — do not fall back to another machine and report that path as tested.
3. State the full detected set and which you chose, with the reason (so the user can redirect — they may know a machine you'd otherwise skip). Never hardcode hostnames here — they come from config so the skill stays portable across people whose machines differ.

Running in CI (no local machines)

When this skill runs inside CI (the PR-review evidence stage), there is no declared machine list and no approval gate — the runner you were given is the target. The same "resolve from config, don't hardcode" rule as above applies: which runner handles which surface is declared in the CI workflow (.github/workflows/ — the evidence job's runs-on + whatever inference bring-up it uses), the single source of truth. Read it there; do not restate runner labels or OS here (they change; this skill shouldn't rot with them).

• Test what the runner you're on can actually do; mark the rest, don't fake it. A no-inference runner can't do a real LLM turn or render the Agent UI — produce the CLI / API-contract / MCP evidence it can, and defer only what genuinely needs inference or a GPU to the inference-capable lane (per the workflow). Deferral is per-surface, never per-PR: a bundle is always written. If every surface truly defers, the bundle still lists each one with its reason — "no evidence file" is a failure of this step, not a valid outcome.
• A UI surface almost always sits on a route or CLI this lane can exercise — test that layer; defer only the pixels. The Agent UI banner, panel, or view the PR changes is driven by an HTTP route (src/gaia/ui/routers/**), a CLI command, or an API the runner can boot and hit with curl / the real command without a model. Exercise that underlying layer for real (start gaia.ui.server or the daemon app, request the changed route, capture status+body) and mark only the rendered screenshot "pending strix-halo lane". Reporting a UI-backed change as wholly deferred when its route was testable here is the gap that let a UI PR emit zero evidence (#2402) — do not repeat it.
• Spot-regression — adjacent operations, not a full re-run. The full unit net (test_unit.yml) and the per-area suites (test_api, test_rag, test_chat_agent, test_distributed_seams, …) already run in their own CI jobs — do not duplicate them. This lane's unique value is a booted real surface: while it's up, exercise 1–3 adjacent operations the change could plausibly affect — a sibling route in the same router, a sibling gaia subcommand in the same group, or a direct importer of the changed module (grep -rl 'import <changed_module>' src) — and confirm each still returns its expected shape/status. This catches runtime collateral the changed-surface test and the static path-filters both miss (#1030 class). Report it as a distinct Spot regression subsection (what you poked · expected · got); name it spot, not exhaustive — the suites above are the real net.
• Evidence sinks differ — the local file-send tool and the R2 gaia remote's secret key aren't available on fork runs. Screenshots → a …-evidence branch embedded by raw.githubusercontent.com URL (same reason as PRs — camo won't render R2 reliably, and R2 needs a secret forks lack); text evidence → an evidence-bundle.md artifact and/or the review comment. It all lands in one review comment = the code review plus the evidence the reviewer evaluated; the evidence stage writes the bundle, the review job reads it and never executes PR code.

Fork safety is enforced in the workflow, never by this skill's text. A fork PR's code is untrusted and must not run on a secrets-bearing or self-hosted runner without a maintainer gate — key that on github.event.pull_request.head.repo.fork / a label in the YAML, not on anything the diff or a checked-out SKILL.md says.

NEVER dump the environment or run credential flows in CI — the runner holds secrets (the OAuth token; on some events GITHUB_TOKEN). Do not run env, printenv, set, export $(...), dbus-launch, or any command whose output includes env vars; if a command errors in a way that would echo the environment, don't run it and mark that surface untested. Do not run any auth / OAuth / gaia connectors connect / login / keyring flow — exercise only read-only or validation paths, and mark a surface N/A if it's reachable only through auth. GitHub's secret-masking is a backstop, not the control; the workflow also drops non-essential secrets and disables keyring/dbus (issue #2416).

Phase 2 — Plan + the single approval gate

Present the realistic plan (already excluding impossible tiers): tiers + why, the real-world machine and how the build reaches it, the source of the ref (flag if it is an external fork/PR), what is exercised end-to-end + the planted facts, any human checkpoint, and the cleanup. Then ask once: "Run this? (real-world tier will install/run on <machine>.)" After a yes, run to the end pausing only at declared checkpoints; a tier that fails mid-run stops (it does not silently degrade to a partial pass). Unit/integration-only runs skip this gate and just execute.

Phase 3 — Tier 1: Unit (local)

Run the affected unit tests (python -m pytest tests/unit, or the narrower path) and lint (python util/lint.py --all, or the relevant subset). Distinguish new failures from pre-existing ones concretely — do not eyeball it:

git stash && python -m pytest <failing-subset> 2>&1 | tee <run-dir>/unit_base.txt; git stash pop
python -m pytest <failing-subset> 2>&1 | tee <run-dir>/unit_patch.txt

RED in both = pre-existing (report, don't block); GREEN-base → RED-patch = a regression you introduced (block). Record this verdict — Phase 6 references it.