Agent Harness

Agent Harness

Most agents ship on vibes: someone tries eight prompts, the output looks good, it goes to production, and the next prompt tweak silently breaks a refusal nobody re-tested. This skill builds the harness around an agent so its behaviour becomes measurable — scenario suites with structural assertions, deterministic replay of recorded tool calls, paired regression diffing across prompt and model changes, and per-scenario cost and latency budgets. The tools here score an agent; they never invoke one, so they run offline on every commit.

When to use this skill

• An agent is going to production and the only quality evidence is manual spot-checking
• A prompt, tool schema, or model version is changing and you need to know what broke
• Two model or configuration options need a defensible comparison, not a demo
• An incident happened and you need the behaviour encoded as a permanent regression test
• Agent cost or latency is climbing across releases and nobody can point to when
• An existing eval suite reports a healthy pass rate that nobody trusts

Inputs the skill expects

• The agent's tool inventory — names, arguments, and which tools are irreversible
• Recorded transcripts per scenario: tool calls, final output, turns, latency, cost, error state
• The behavioural rules the agent must hold (refusals, escalation triggers, policy boundaries)
• Known failure history — past incidents, customer complaints, internal bug reports
• Current cost and latency expectations per interaction
• The release gate that consumes the result (CI job, review checklist, launch review)

Clarify First

Before building the harness, confirm these inputs. If any is unknown or vague, ASK — do not assume:

• Which agent actions are irreversible — determines which scenarios need tool_not_called assertions at critical severity, and what the release gate blocks on
• Whether transcripts are already recorded — decides whether workflow 1 starts from replay or from an instrumentation task first
• What the suite gates — a CI blocking check, a nightly report, or a one-off comparison; changes suite size, runtime budget, and severity strictness
• The known failure modes — past incidents seed the adversarial and refusal buckets, which is where regressions actually hide

Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions at the top of the artifact.

Workflows

Workflow 1 — Stand up a scenario suite and score a run

1. Enumerate the agent's irreversible actions; each one gets a refusal scenario.
2. Draft 20-30 scenarios across all six buckets (happy, boundary, refusal, adversarial, failure-recovery, ambiguity) using assets/scenario_authoring_checklist.md. Structural assertions first — tool called / not called / order / arguments — text assertions only on domain tokens.
3. Declare suite-wide defaults for latency, cost, and turn ceilings so every scenario is budgeted without repeating yourself.
4. Record one transcript per scenario, scrubbing PII at record time, and stamp the run with model and prompt_sha.
5. Score the run and read critical failures before the pass rate.

python3 engineering/agent-harness/scripts/scenario_runner.py \
  --suite engineering/agent-harness/assets/sample_suite.json \
  --transcripts engineering/agent-harness/assets/sample_transcripts_baseline.json \
  --strict-critical

Workflow 2 — Gate a prompt or model change on a paired regression diff

1. Score the baseline and the candidate with the same suite file, saving both as JSON reports.
2. Diff them. Read regressions and budget drift before the aggregate rate.
3. Triage every regression: intended trade, real defect, or flaky scenario (re-run the flipped scenario five times to tell the last two apart).
4. Record the decision in assets/eval_report_template.md and promote the accepted candidate report to the new baseline.

python3 engineering/agent-harness/scripts/scenario_runner.py \
  --suite engineering/agent-harness/assets/sample_suite.json \
  --transcripts engineering/agent-harness/assets/sample_transcripts_candidate.json \
  --format json > /tmp/candidate.report.json

python3 engineering/agent-harness/scripts/eval_diff.py \
  --baseline engineering/agent-harness/assets/sample_baseline_report.json \
  --candidate /tmp/candidate.report.json \
  --fail-on-regression --drift-threshold 0.15

The shipped sample data demonstrates the core lesson: both runs score 83.3%, and the candidate contains a critical prompt-injection regression. A gate on pass rate ships it; the paired diff catches it.

Workflow 3 — Establish cost and latency budgets, then track drift

1. Take the last release's accepted run as the reference.
2. Set per-scenario latency at p95 × 1.3, cost at median × 1.5, and the turn ceiling at observed max + 2. Put them in the suite defaults, overriding only where a scenario is legitimately expensive.
3. Score the current run; budget breaches surface as minor assertions, so they report without blocking.
4. Diff against the reference with a tight drift threshold to catch the slow bleed that stays inside budget.

python3 engineering/agent-harness/scripts/eval_diff.py \
  --baseline engineering/agent-harness/assets/sample_baseline_report.json \
  --candidate engineering/agent-harness/assets/sample_candidate_report.json \
  --drift-threshold 0.10 --format json

Decision frameworks

Which assertion type to reach for

Severity, and what each one gates

Can I trust this diff?

A single critical regression is actionable at n = 1. Significance testing is for aggregate movement, never for safety failures.