Evaluation

Evaluation

Running an environment in this repo — both while developing/iterating on it (smoke-testing changes) and when benchmarking a finished env. Run everything from the repo root, always via uv.

Environments are v1 tasksets: a package that exports a Taskset. Run one with the eval CLI, by taskset id: uv run eval <taskset-id>.

Setup

Editable, local install from the repo root (not from inside the env dir). Envs live under environments/, grouped into semantic group folders (e.g. environments/math/, environments/swe/):

uv pip install -e path/to/env

Editable means code edits are picked up without reinstalling — so during development you can change the env and immediately re-run it. Only re-run this install after editing pyproject.toml (e.g. new deps). v1 tasksets pin a verifiers pre-release; if the install complains, add --prerelease=if-necessary (never the blanket --prerelease=allow, which bumps unrelated deps to betas — e.g. a sympy rc + mpmath alpha that can't import together).

Harbor-based envs (SWE/terminal tasksets) must depend on verifiers[harbor], not plain verifiers — harbor is an optional extra of verifiers, and without it the eval crashes at taskset load with ModuleNotFoundError: No module named 'harbor.models'.

In a dirty worktree, use uv run --no-sync ... after the editable install so machine-local uv settings cannot rewrite the repository's uv.lock. Do not export UV_FROZEN for an eval: subprocess runtimes inherit it, and their unlocked inline uv scripts cannot start in frozen mode.

Run an eval from the repository root, not from inside its virtualenv. NLTK's import protection rejects dependencies installed beneath the current working directory. For a separate temporary venv, activate it by absolute path and run uv run --active --no-project eval ... from outside that venv.

After dependency edits, sync the environment project before testing imports:

uv sync --project path/to/env --all-extras

For gated Hugging Face datasets, verify access by downloading a small required file; repository metadata can remain public even when the authenticated account cannot read the artifacts. Keep HF_TOKEN on the evaluator. If the agent needs gated corpus files, stage only those files into its runtime and keep answer keys, rubrics, and source provenance evaluator-side.

If a fresh environment import fails because resolved package versions are incompatible, fix the environment's pyproject.toml bounds and sync again. Do not patch generated .venv files.

For OfficeQA Pro V2, authenticate the evaluator for the gated databricks/officeqa-pro-v2 dataset at revision 65a2b315780417bc50d7bfe6e5bdb904e63fda65. Use the registered Prime VM image reference prime/primeintellect/officeqa-pro-v2:65a2b3157804-v1, containing all 1,435 parsed documents at /workspace/documents. Prime VM lookup rejects the OCI digest form with VM image not found. Before evaluating, use prime images list --search officeqa --output json --plain and verify the container artifact’s fullImagePath digest is sha256:f99fb779f7f473fd7b596da7690992d0bdb20cccf89a0a4083cbeb2852c01b42 so the corpus stays aligned with the pinned question CSV.

PI-OfficeQa-Pro-v2 (pi_officeqa_pro_v2) loads 82 tasks by default with 15 repairs and 8 exclusions. It blocks Hugging Face, GitHub, their content hosts, and common mirrors through its native task network policy during sandbox execution. Other web sources remain available; evaluator-side dataset downloads and provider-hosted search services outside the sandbox are unaffected.

OfficeQA-Pro-v2 (officeqa_pro_v2) loads all 90 unmodified tasks with the original regex grader at zero numerical tolerance and unrestricted networking. Both variants reuse the same public Prime image; select the environment explicitly when reporting scores.

PI-OfficeQa-Pro-v2 uses a ReferenceJudge subclass with a packaged prompt covering numeric values, signs, units, precision, and list order, defaulting to openai/gpt-5.6-luna with medium reasoning. Configure it under --env.taskset.task.judges.0.*, independently of the solver's -m setting. Judge calls use evaluator-side Prime credentials and are recorded in trace.info["judge"] and trace.extra_usage; these scores use LLM judgment rather than the upstream deterministic scorer. The prompt grades the last <FINAL_ANSWER> block when present, otherwise the stated final answer; XML tags and harmless prose remain optional.

When a judge's id names an environment package, export its judge class in that package's __all__ so Verifiers can discover it. For judge-only checks, load saved traces with vf.Trace[YourTaskData] or vf.WireTrace so custom reference-answer fields survive deserialization, then call await task.score(trace). A bare vf.Trace can discard those fields. An in-process smoke with --no-serve can validate the task and judge when the worker pool fails; record that mode because it does not validate the served path.

For reference-judge prompts that request an explanation, end with a boxed verdict, such as \boxed{yes}. Verifiers 0.3.1 prioritizes boxed text; an earlier explanation labeled "Final answer" can otherwise make its verdict parser pick up incidental yes/no words before the actual verdict.

Smoke test

Run a 3x1 (3 tasks, 1 rollout each) in plain-log mode to confirm the env loads and scores end-to-end. Spanning a few tasks surfaces weird things (bad rows, edge-case prompts, inconsistent scoring) that a single task hides:

uv run --no-sync eval <taskset-id> -n 3 -r 1 --no-rich -v --no-push

--no-rich turns off the live dashboard (which is on by default) for plain logs, -v prints prompts/completions, and --no-push keeps development runs local. This is the inner loop while developing — re-run it after each change to verify the dataset loads, the rollout runs, and the rubric scores as expected. Fix any errors here before scaling up.

Full eval

Run all tasks (omit -n) and pick -r so the total sample is representative — usually >500 total rollouts (num_tasks × r) is a good target. For large datasets -r 1 may already clear that; for small ones, bump -r.

Recommended full-eval invocation:

uv run eval <taskset-id> -r <r> -c <N> --no-rich

The run is always saved to disk (see Inspect output) — there's no save flag.

Key flags (eval)

Most tasksets yield records as they are consumed. In Verifiers 0.3.1, -n N uses a lazy head before the runner collects those N tasks, but automationbench_env, general_agent, tau2_bench, and tau3_bench still construct their complete task lists. -s calls Taskset.shuffle() first, which must materialize the complete finite taskset, so avoid -s when a smoke test should construct only a prefix.

Configuring the env

Pass typed, dotted flags or a TOML file — there is no -a JSON blob:

uv run eval wikispeedia --env.taskset.min-path-length 5 --env.taskset.max-path-length 8
uv run eval <taskset-id> @ eval.toml          # a launch TOML is re-runnable

Common knobs: --env.agent.max-turns, --env.agent.max-total-tokens, --sampling.max-tokens, --sampling.temperature, --env.agent.harness.id, --env.agent.runtime.type. uv run eval <taskset-id> -h prints the full typed help, narrowed to the chosen taskset/harness.

TasksetConfig.system_prompt is reserved for --env.taskset.system-prompt, a path to a prompt file that overrides every task during iteration. Environment configs must not redefine it as inline text; use a distinct field such as task_system_prompt for an inline/default task prompt.

List-valued flags on union configs (e.g. --env.agent.runtime.labels on the prime runtime) must be passed as a space-separated JSON value — --env.agent.runtime.labels '["ci"]'. The bare form (... labels ci) and the =-form with JSON both parse as a string and fail validation with Input should be a valid list.

A dotted override below a nested RuntimeConfig can replace that whole runtime with the selected provider's defaults. After overriding any nested runtime field, inspect configs/resolved/eval.json and confirm its image, workdir, runtime type, and resources before treating the smoke test as representative.