GAIA Agent Eval Scorecard

Adding an Eval Scorecard to a GAIA Agent

Adopt the release eval scorecard (docs/reference/eval-scorecard.mdx) for one hub agent. The system is harness → result payload → generator → scorecard, with a standalone presence+regression release gate. The email agent is the reference implementation — mirror it.

Core modules (do not modify; reuse):

• src/gaia/eval/release_scorecard.py — ResultPayload, compute_aggregate, render_scorecard, write_scorecard, validate_scorecard, carry_forward. Harness-agnostic (stdlib + PyYAML only).
• src/gaia/eval/scorecard_gate.py — the standalone gate (python -m gaia.eval.scorecard_gate).
• Reference adapter: hub/agents/email/python/packaging/gen_scorecard.py.

This is a phased checklist with a hard gate at the real-eval step — the scorecard MUST come from an actual eval run, never hand-authored numbers.

Phase 1 — Locate the agent's surfaces

1. Version source of truth = the version: field in <agent>/gaia-agent.yaml. Never invent a parallel scheme.

2. Canonical README (where the scorecard is linked + surfaced): for an npm-published agent it is the npm client README (e.g. hub/agents/<id>/npm/README.md), NOT a packaging/README.md. For a Python-only agent it is hub/agents/<id>/python/README.md. Confirm which by checking what release_agent_<id>.yml publishes (README: env) — the published README is the one to link.

3. doc-root = the directory holding that canonical README. The scorecard lives at <doc-root>/SCORECARD.md — a single file updated in place, versioned via the publish snapshot (same as README.md). There is no scorecards/ directory.

4. Eval vehicle: what existing harness produces this agent's accuracy metric? (email → gaia eval benchmark over tests/fixtures/email/.) If none exists, STOP and surface that — propose the minimal harness before building; do not invent numbers.

   Expect none. Email is the only agent with a corpus + harness; for every other agent this step is the long pole, not a lookup. Two traps: an agent_type in the scenario corpus may name a ChatAgent prompt profile rather than your package (so a green scenario is measuring something else), and the ground truth itself is human judgment the factory does not automate. Build the corpus + deterministic fixture harness first — see porting-agent-to-hub Phase 4 — then return here for the adapter and gate.

Phase 2 — Write the adapter (harness → payload)

Copy hub/agents/email/python/packaging/gen_scorecard.py as the template. The adapter:

• imports ONLY gaia.eval.release_scorecard (never the harness or agent package — preserve loose coupling);
• reads the harness output, builds a ResultPayload;
• populates reproduction_command with the exact shell commands to reproduce this scorecard, including all required env vars (PYTHON_KEYRING_BACKEND, GAIA_AGENT_TOOL_TIMEOUT, PYTHONPATH);
• defines "judged" explicitly and raises loudly if zero results are judged (no silent 0.0);
• records dataset size (total labeled examples) and test_cases_run (subset executed) as DISTINCT fields;
• stores repo-relative paths only (never a local absolute path — it ships in a published artifact);
• records the eval limit/config so future regression checks are comparable;
• optionally populates environment (gaia_commit, lemonade_version, model, hardware — a class descriptor, never a hostname) and breakdown (per_category accuracy + top_confusions) — additive blocks that pin the run and explain the misses without ever affecting aggregate.value. Capture live environment (git SHA, server health version) in main(), not build_payload, so the payload builder stays pure and unit-testable;
• writes to <doc-root>/SCORECARD.md (the single file; --output-dir overrides to a directory, but the filename is always SCORECARD.md).

Add an offline unit test against a committed sample harness-output fixture (see tests/fixtures/eval/email_benchmark_scorecard.json + tests/unit/eval/test_release_scorecard.py::TestEmailAdapter) so the adapter is testable without a live model.

Phase 3 — Run the REAL eval (hard gate — no hand-authored numbers)

The accuracy number must come from an actual run. For the email agent:

# Real eval needs Lemonade + the model. Prefer AMD hardware (Strix Halo / Ryzen AI);
# the [self-hosted, lemonade-eval] runner is the canonical environment.
GAIA_AGENT_TOOL_TIMEOUT=1800 \
PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring \
PYTHONPATH="$(pwd)" \
  <venv>/bin/gaia eval benchmark \
    --model Gemma-4-E4B-it-GGUF \
    --mbox-path tests/fixtures/email/_stub_inbox.mbox \
    --ground-truth tests/fixtures/email/<task>_ground_truth.json \
    --limit 220 --output-dir <persistent-dir>

PYTHONPATH="$(pwd)" \
<venv>/bin/python hub/agents/email/python/packaging/gen_scorecard.py \
    --benchmark-dir <persistent-dir> --limit 220
# → writes hub/agents/email/npm/SCORECARD.md in place

⚠️ Run evals SERIALLY — never two at once (CLAUDE.md). A scorecard refresh loop is the most likely thing to violate this: chain runs with &&, never background them with &. Concurrent runs race-evict each other's models on the single-tenant Lemonade backend and produce garbage (ctx_size errors, spurious INFRA_ERROR) that looks like a real regression. Before each run: ps aux | grep "gaia eval" | grep -v grep | wc -l must be 0.

Pass both fixture flags explicitly. ls tests/fixtures/email/ and pick the real files — ground truth is task-specific (action_items_, briefing_, drafting_, followups_, longthread_), and the CLI's --ground-truth default points at a ground_truth.json that does not exist. Omit the flag and the run prints one [WARN] and scores nothing, which reads as a successful eval right up until the scorecard is empty.

Headless gotchas (see memory project-email-benchmark-headless-gotchas):

• PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring — the email agent's calendar-connector resolution blocks forever on the macOS Keychain (and can stall on Linux SecretService) in non-interactive contexts. Without this it hangs at 0% CPU during agent construction.
• PYTHONPATH="$(pwd)" — the benchmark imports tests.fixtures.email.*; the console script doesn't add the repo root.
• GAIA_AGENT_TOOL_TIMEOUT=1800 — triage of the full corpus is one tool call (~17 min for ~100 emails on a 4B model); a lower timeout (the 180s default, or even 900s) abandons it mid-run, yielding a degenerate 0-email FAIL run.
• Write --output-dir to a persistent dir, not /tmp (cleared on session resume).
• Record honestly: if the metric is low for a known reason (e.g. a taxonomy/label mismatch), put the explanation in the adapter's methodology string and link the tracking issue — never inflate the number.