Agent Eval Engineering

Eval Engineering

Flow

• Inspect all inputs first: the repository, Harness, optional traces, existing Tasks and runs, existing World knowledge, and the human goal. Identify the source files and skill references that apply before proposing work.
• Create or update the small project World Knowledge Skill from reusable facts in those inputs. Use it to propose one grounded Task.
• Draft the Task Spec and World Skill together. Show both to the user, keep exact Task truth only in Task.md, and refine both until the user approves them.
• Implement the approved Task, validate its Environment and Verifier, run the real Harness, inspect the full evidence, and fix only non-agent failures.
• Reconcile the World Skill with what the run proved, then repeat this flow for the next Task.

Terms

• Task Spec: Task.md, which describes the input, relevant agent conditions, Environment, scoring, fairness, and open decisions for one Task.
• Task: the runnable instruction, Environment, and Verifier.
• Harness: the complete agent Harbor runs, including prompts, model loop, tools, hooks, memory, sessions, and adapter.
• Environment: the files, data, services, identity, permissions, network, clock, and mutable state around the Harness.
• Verifier: independent checks that score the result or mark a run invalid.
• World Knowledge Skill: a repository-local skill with reusable project-specific knowledge, references, scripts, assets, and tests that help generate Task Specs and build future Tasks.
• Spec2Task: the full loop that turns a reviewed Task Spec into an audited runnable Task. Follow Task implementation for its concrete build order and reference routing.

Reference routing

Read each reference when its decision appears:

Reusable implementation resources:

• Multi-turn runner, model user, and Harbor adapter example
• Tool-schema comparison, which compares supplied schema fragments but does not resolve external $ref targets
• Read-only SQLite state snapshot

1. Inspect inputs and existing World knowledge

Review every input the user provides before proposing a Task. Use the guidance that matches each available input:

• For a repository, Harness, traces, or dependencies, read Discovery.
• For existing Tasks and runs, inspect their instructions, Environments, Verifiers, rewards, trajectories, and final state. Read Calibration when run quality or failure causes affect the new design.
• For an existing project World Skill, read World knowledge, then check the sources and reusable methods that affect the new Task.
• For human goals and constraints, read Task design.
• For relevant benchmark examples, use the domain index and source callouts in Benchmark patterns.

Inspect the repository before asking questions that source and tests can answer. Follow the active Harness through prompts, models, tools, services, state, effects, and focused tests. Inspect existing Task instructions, parsers, Verifiers, reward paths, and run evidence.

If the user supplies traces, review complete runs or threads. Use traces to learn real requests, dependency behavior, state shapes, errors, and failure conditions. Do not treat a trace answer as independent truth.

If .agents/skills/<project>-world/SKILL.md exists, read it. Follow its routing only for knowledge relevant to the current Task. Check cited repository paths, commands, and scripts when their accuracy affects the design.

2. Propose and select a Task

Read Task design and use the index in Benchmark patterns to find the relevant domain and source callouts. Focus on that domain unless the Task crosses another one. In the first user-facing design response after inspection, propose one Task grounded in repository evidence, supplied traces, existing coverage, or a human priority. State:

• the real work and capability;
• the condition that makes the case non-trivial;
• the Environment and independent evidence it needs;
• the important failure it can detect;
• how it differs from existing Tasks; and
• the main open decision.

In the same response, show the relevant current World Skill content and the specific additions or corrections this Task suggests. If no World Skill exists, show the small initial contents that will help create this Task and future Tasks. Keep the Task's exact request, focal records, expected result, hidden truth, and exact scoring rules out of the World Skill.

Let the user revise the Task proposal and World knowledge together before implementation. Offer alternatives only when a real user choice changes the design.

3. Write and review the Task Spec and World Skill

Copy the Task template to evals/<suite>/tasks/<task-id>/Task.md. Put all Task-specific design in this one file. At the same time, create or update the project World Skill by following World knowledge. Determine the project skill location supported by the active agent and repository. .agents/skills/<project>-world/SKILL.md and .claude/skills/<project>-world/SKILL.md are common landing spots. Follow an established project convention when one exists. Otherwise, explain the proposed location and get user confirmation before creating the skill. Start from the World Skill template when needed.

Keep each Task.md beside the Harbor task it describes:

evals/<suite>/tasks/<task-id>/
├── Task.md              # human-reviewed control-plane spec
├── task.toml             # required Harbor configuration
├── instruction.md        # required agent input
├── environment/          # required Environment definition and visible state
│   ├── Dockerfile        # use this or docker-compose.yaml
│   └── docker-compose.yaml # optional; primary service must be main
├── tests/
│   ├── test.sh           # required Harbor Verifier entry point
│   ├── test_*.py         # optional Verifier helpers
│   └── fixtures/         # optional hidden Verifier data
└── solution/
    └── solve.sh          # optional reference path

Never copy or mount Task.md into the evaluated agent's workspace or image. The agent receives instruction.md and only the Environment state intended for the run.

Include:

• purpose and source evidence;
• exact input and later turns;
• only the Harness conditions relevant to this Task;
• initial state, services, access, visibility, reset, and production differences;
• required results, prohibited effects, accepted alternatives, and independent Verifier evidence;
• fairness, leakage risks, and invalid-run conditions; and
• open decisions and assumptions.

Show the full Task Spec and the World Skill changes to the user. Explain what is already in the World Skill, what this Task adds or corrects, and what stays only in Task.md. Revise both through the same back-and-forth. Mark the Task Spec approved only after explicit approval. Treat World Skill changes as accepted only after the user reviews them. If the user requests an end-to-end build without an approval pause, continue with an agent-reviewed Status: Draft and label the World Skill changes as unreviewed.

If implementation changes the request, visible information, material Environment behavior, or scoring boundary, update Task.md and show the change. Set its status back to Draft. Show the diff and require explicit reapproval before setting it to Approved again.