Testing The Gaia Agent

Testing the flagship GAIA agent through the TUI

Companion to driving-the-tui (which covers the control API mechanics). This one covers what to test and how to know it actually worked — written from a full session of driving the live agent, including every trap that cost an hour.

The ladder tests capabilities. Users have conversations.

Read this before trusting a green ladder. Every rung below is a self-contained prompt — it names its repo, its numbers, its subject. So the ladder ran green for a whole day while the TUI agent had no conversation history at all: every turn reached the model as system prompt + current question, nothing else. The bug surfaced the moment a real user typed a follow-up:

triage amd/gaia → three issues listed "cool, can you print issue 2975?" → "I need to know which repository it belongs to"

Three things hid it, and all three are worth knowing:

1. L3 looks like proof of continuity and is not. "What is my favourite colour?" passes across turns via the persistent memory store, a different mechanism entirely. Its green tick actively masked the gap.
2. The stdio test asserting turn-to-turn state passes for the wrong reason. test_the_agent_survives_between_turns asserts OBJECT state (agent.loaded_skills) survives — it does, the agent is the same object. History is not accumulated object state; nobody was appending to it.
3. The HTTP surface populates history, so any test at that layer passes. The defect was transport-specific, and only the TUI used the broken transport.

So always finish with a follow-up that cannot stand alone. Use a pronoun or a bare number and give it nothing else:

The second pair is the cheap canary — two short turns, no tools, no network. Run it first. If it answers "no fruit has been mentioned", stop: history is broken and every other result is measuring an agent with amnesia.

The one rule

A plausible answer is not a passing test. The flagship's worst failure mode is answering confidently when its tools are missing. It once produced a polished "here's how I'd triage that" paragraph while having zero GitHub tools registered. Every capability claim must be checked against ground truth from outside the agent:

# agent said: #2958, #2955, #2953
gh issue list --repo amd/gaia --limit 3 --json number,title   # must match exactly

If you cannot independently verify a result, report it as unverified. Say so plainly.

Rules about the machine — ignore these and you will measure noise

Each of these cost real hours in the session this skill came from, and each produces symptoms that look like product bugs.

1. Exactly ONE TUI at a time

Kill every existing instance before launching, and never leave a second one running:

# Windows
for p in $(tasklist //FI "IMAGENAME eq gaia-drive.exe" //FO CSV //NH | cut -d, -f2 | tr -d '"'); do
  taskkill //PID $p //F
done

Two TUIs is not merely wasteful:

• They overwrite each other's ~/.gaia/tui/control.json — same pid/port/token file — so your driver silently attaches to whichever launched last. A query you never sent appears in your transcript; keys you send land in someone else's session. This happened in both directions in one day, and each time looked like a TUI bug.
• Each spawns its own agent child, so they compete for the model and every turn slows.
• The user is memory-constrained; two instances is a real cost, not a rounding error.

GAIA_TUI_HOME isolates the discovery file so concurrent agents stop hijacking each other — it does not remove the model contention. One TUI, always.

2. Never run an eval while testing the agent

gaia eval agent and the TUI both drive the single-slot Lemonade backend. Running them together makes every turn 2-5x slower and the slowdown reads as "the agent is extremely slow" — a product complaint caused entirely by the harness. Measured on the same box, same build:

Worse, CLAUDE.md warns that concurrent runs race the model slot and can produce chaotic, meaningless failures (BLOCKED_BY_ARCHITECTURE, INFRA_ERROR, ctx-size errors) that get mistaken for regressions.

Check before you start, and check again when things feel slow:

powershell.exe -NoProfile -Command "Get-CimInstance Win32_Process -Filter \"Name='python.exe'\" | Select-Object -ExpandProperty CommandLine" | grep -iE "eval agent|ui.server"

Evals are a pre-merge gate, not a testing-session activity. When a change requires one (CLAUDE.md lists the surfaces — prompts, tool schemas, tool-call parsing), record it as outstanding and run it when the box is quiet and nobody is driving the TUI. Never run two evals at once, either.

3. Give your session a private agent log

Every agent appends to ~/.gaia/logs/gaia-agent.log. When anything else is running an agent — a parallel task, a second harness — the file interleaves, and a neighbour's tool timeout reads as your session's failure. Set GAIA_AGENT_LOG in the launcher:

$env:GAIA_AGENT_LOG = 'C:\...\gaia-tui-test\logs\agent-session.log'

Lines also carry pid:NNNN, so the shared default is still attributable when you forget. This is not hypothetical: a 180s run_shell_command timeout was nearly filed as a shell bug here before the record turned out to belong to another process. Confirm the pid in the log matches the gaia-agent.exe your TUI spawned before believing anything.

4. Point memory at a throwaway DB — ALWAYS

GAIA_MEMORY_DB is mandatory in every launcher. Without it the agent writes to the user's real ~/.gaia/memory.db, and everything you plant during a test drive becomes a permanent fact about the user:

$env:GAIA_MEMORY_DB = 'C:\...\gaia-tui-test\memory\test-memory.db'

This is the eval-runner rule applied to interactive sessions. gaia eval agent already resets state between memory scenarios (GAIA_MEMORY_ADMIN=1 + memory_clear(scope=all) in src/gaia/eval/runner.py); a TUI test drive has no such cleanup, so isolation has to come from the environment.

It is not hypothetical. A ladder run planted a persona's overdue deadline; days later a real session answered the user's "sweet!" with "Priya needs that Fernbrook deck ASAP." The user's second brain had been quietly seeded by a test.

Delete the file between runs to test the cold-start path — a warm store hides first-run bugs the same way a warm model cache hid #1655.

A bad value is fatal on purpose. GAIA_MEMORY_DB set to a directory, or set blank, raises at startup rather than falling back to the real store — a harness that believes it is isolated but is not is the whole failure mode. If the agent will not start, read the error; do not unset the variable.

GAIA_HOME selects $GAIA_HOME/memory.db when GAIA_MEMORY_DB is unset. It does not isolate the whole ~/.gaia tree: config uses GAIA_CONFIG_DIR, and logs and other state may still use the real home directory. Use a separate OS user or container when the harness needs complete isolation.

Which surface you are testing

One binary, two surfaces — always state which:

Launching bare and typing lands your text in the Hub's filter box, not a chat composer. That produced a fake bug report once.

Setup

1. Build

cd tui && go build -o bin/gaia-drive.exe ./cmd/gaia

Do not launch while a build is writing the binary — the file lock makes the launch silently fail. Build, then launch.

Go must be on PATH (export PATH="/c/Program Files/Go/bin:$PATH" on Windows). Note that gofmt -l flags nearly every file on a Windows checkout — that is CRLF, not real formatting drift. Check gofmt -d <file> | cat -A for ^M before you "fix" anything.

1b. The agent binary the TUI spawns

The TUI launches gaia-agent from PATH (catalog.go, BinaryPath). A source checkout does not have it — the console script only exists once the hub package is installed:

uv pip install --python <venv>/Scripts/python.exe \
  -e hub/agents/gaia/python -e hub/agents/chat/python --no-deps

--no-deps is mandatory: without it pip pulls amd-gaia from PyPI and the agent imports THAT instead of your worktree. Put the venv's Scripts/ on PATH in the launcher or the TUI cannot find gaia-agent.

1c. The flagship ships with NO skills

gaia_agent/skills/ holds only .gitkeep, nothing stages hub/skills/ into it, and every skills: / skill_sets: / default_skill_set: key in gaia-agent.yaml is commented out. So L5–L7 cannot pass on a clean checkout — not because the agent is broken, but because it has nothing to load.

Install the one you are testing, and copy it rather than gaia skill import — import re-stamps the tier experimental, which is not what ships, and refuses a skill whose grants (like shell:execute:gh) sit above that tier's ceiling:

cp -r hub/skills/github-triage ~/.gaia/skills/
gaia skill list      # expect: github-triage  2.1.0  community  user

Also note gh is refused until the skill that grants it is loaded — the grant is shell:execute:gh. Asking for gh first produces a confident refusal that looks like a missing-tool bug and is not one.

2. Launcher (adapt paths, keep the structure)

Create launch-tui.ps1. Every line matters:

$root = '<ABSOLUTE PATH TO YOUR WORKTREE>'
$env:PYTHONPATH = "$root\src;$root\hub\agents\chat\python;$root\hub\agents\gaia\python"
$env:GAIA_TUI_HOME = '<A PRIVATE TEMP DIR — NOT ~/.gaia/tui>'
$env:GAIA_MEMORY_DB = '<A PRIVATE TEMP FILE — NOT ~/.gaia/memory.db>'
$env:GAIA_AGENT_LOG = '<A PRIVATE TEMP FILE — NOT ~/.gaia/logs/gaia-agent.log>'
$env:PYTHONIOENCODING = 'utf-8'
$inner = "cd /d `"$root`" && tui\bin\gaia-drive.exe run gaia --control-port 8817"
Start-Process -FilePath 'cmd.exe' -ArgumentList '/k', $inner -WindowStyle Normal

Launch with:

powershell.exe -NoProfile -ExecutionPolicy Bypass -File <path>\launch-tui.ps1

PYTHONPATH is mandatory. An editable install can resolve gaia to a different worktree, and the agent then dies at import with ModuleNotFoundError: No module named 'gaia.ui.sse_translation'. Verify:

python -c "import gaia; print(gaia.__file__)"   # must be YOUR worktree

GAIA_TUI_HOME is mandatory when other agents may be running — see machine rule 1 above. It gives you a private control.json (tui/internal/control/paths.go) instead of the shared ~/.gaia/tui/control.json that agents hijack from each other. It does not excuse running two TUIs.

GAIA_MEMORY_DB is mandatory always — see machine rule 4. Omit it and your test drive writes into the user's real second brain. Verify before you type anything:

python -c "from gaia.agents.base.memory_store import resolve_memory_db_path as r; print(r())"

Do not use cmd //c start from Git Bash — MSYS mangles the arguments and no window opens. PowerShell Start-Process with a .ps1 avoids the quoting entirely.

3. Driver

Use util/tui_driver.py from the repo root (repoint CJ at your GAIA_TUI_HOME).

Why one process: process spawn costs 0.7–2.0s on a Windows/MSYS box with AV — curl --version alone measured 2051 ms. A bash driver spawning bash + 2 × python + curl per command cost ~4.8s per call. The control API itself is 3 ms. Batch every step of a test into ONE python process:

5 control calls in one process: 15 ms total

Driving correctly

• Wait for streaming:true BEFORE waiting for streaming:false. Otherwise the idle-wait matches the pre-turn idle state and returns in 0.0s, and you will report a phantom instant answer.
• Press end before every capture or you capture stale scrollback and read an old turn as the current one.
• Never sleep to wait out a turn — poll status or use /control/v1/wait.
• Set PYTHONIOENCODING=utf-8 or captures die on cp1252 for the spinner glyphs.
• Do not resize larger than the real terminal — the control API returns 409 resize_exceeds_terminal; a bigger size shreds the frame.

The capability ladder

Run in order. Stop and diagnose at the first failure — later rungs depend on earlier.

L6 is the regression canary for a bug where the skill vanished between turns. L7 is the real test: it fails silently by producing a confident non-answer.

Diagnosing L7 failure

If it deflects ("first configure the connector…") it has no tools. Check, in order:

# 1. Does it think it has tools?  (a NONE here is the smoking gun)
#    ask in the TUI: "List the exact names of every tool you can call that talks
#    to GitHub. If you have none, say NONE."

# 2. What did the loader actually register?
grep -E "Loaded skill|registered_tools" ~/.gaia/logs/gaia-agent.log | tail -5
#    "0 tool(s), 1 connector requirement(s)" + 'registered_tools': [] == no tools

# 3. Is the skill the version you think?
grep -E "version:|shell:execute|mcp:connect" ~/.gaia/skills/github-triage/SKILL.md

The installed copy at ~/.gaia/skills/<name>/SKILL.md is what the agent reads, not the repo copy. After editing the repo skill, sync it or the agent runs the old one.

Verifying the permission gate

The gh grant has three tiers, and the bug you are hunting is a command in the wrong one. Run the gate check — instant, no LLM, no TUI:

python util/tui_driver.py gate      # prints each case, its tier, and ok/WRONG

Expect 13/13 as expected. What each tier means:

The two failures worth naming, because each looks fine on a green ladder:

1. A write silently landing in ALLOW. It ran and nobody was asked. The gate check catches it; a TUI session will not, because the write succeeding looks like the feature working.
2. An escalation landing in CONFIRM. Now gh auth token has a yes button. The whole point of keeping REFUSE separate is that a prompt the user learns to approve approves that too.

Then confirm end-to-end in the TUI, with the box quiet:

Answer n unless you actually want the issue filed. If you answer y, delete the issue afterwards — and note gh issue close is itself REFUSE, so that is a manual step on github.com.