Pxi Eval Dataset

pxi-eval-dataset

Produce a small, well-targeted YAML dataset that drops into evals/pxi/datasets/<name>.yaml, runs through evals/pxi/harness/run_experiment.py, and is scored by deterministic code evaluators under evals/pxi/evaluators/.

The aim is a minimal but representative set of synthetic examples — think unit tests, not a benchmark. 10–50 examples, each covering a distinct dimension. Add more only when a new example tests something no existing example does.

Every example must be scorable by deterministic / heuristic / code logic. Every example must include a non-empty splits: list. Use splits: [regression] for small committed regression suites unless the user explicitly asks for a dev or val dataset; keep regression, dev, and val disjoint.

Workflow

1. Identify the target

Confirm with the user what's under test: a specific PXI tool name (e.g. set_time_range), a skill, or a higher-level behavior. For tools:

• Read the tool definition at src/phoenix/server/agents/toolsets/external/tools/<name>.py — the ToolDefinition's description and parameters_json_schema are what the LLM actually sees.
• Search the phoenix src for any server-side implementation function that backs the tool. (External tools execute browser-side and may have none — that's fine, the docstring + schema are the spec.)
• Read src/phoenix/server/agents/toolsets/__init__.py::build_toolset and src/phoenix/server/agents/toolsets/external/__init__.py::build_external_toolset to learn the availability conditions — which ChatContext must be present for the tool to be exposed (e.g. set_spans_filter only exists when deps.contexts.project is set).

2. Survey the evaluators that currently exist

List evals/pxi/evaluators/ and read each module. For every evaluator, note:

• the evaluator name and @create_evaluator(...) decorator,
• what fields it consumes from expected (e.g. expected.tools.required, expected.tool_call_args[<tool>]),
• its score / label semantics and any helpful failure metadata,
• the class of assertion it supports (tool selection, tool arguments, assistant text, multi-call sequencing, ...).

Also peek at evals/pxi/tests/test_evaluators.py for canonical input shapes.

Do NOT hard-code knowledge of which evaluators exist — re-read the directory every time. The set will grow.

3. Match the target to the evaluators — pause if a gap exists

Decide whether the behavior under test can be scored by what's there today.

• Yes → continue. The chosen evaluator names go in the dataset's evaluators: field (required, top-level). The runner uses ONLY the listed evaluators; unrelated ones are not invoked, so the dashboard stays free of vacuous passes from evaluators that don't apply to this dataset. Available names live in evals/pxi/evaluators/__init__.py (EVALUATORS_BY_NAME).

• No → stop and summarize the gap to the user. Propose the shape of a new evaluator:

  • name (snake_case),
  • the expected.<field> it would read,
  • score / label semantics,
  • whether it's tool-call-shaped, text-shaped, structural, etc.

  Then ask: "Should I add this evaluator before we generate examples?" If yes:

  • implement it under evals/pxi/evaluators/<file>.py,
  • add unit-test coverage to evals/pxi/tests/test_evaluators.py,
  • export it from evals/pxi/evaluators/__init__.py,
  • then continue.

  If no, scope the dataset down to assertions the existing evaluators can score, and tell the user what coverage that costs.

4. Enumerate coverage dimensions

Walk this checklist for the target. For each row, write down which queries you'll add to cover it. Skip rows that don't apply (e.g. booleans on a tool with no boolean field).

• Parameter coverage — every required field, every optional field, every meaningful field combination.
• Value coverage — every enum literal; for strings: empty, whitespace, special characters, very long; for numbers: zero, negative, boundary; for booleans: both polarities.
• Combination coverage — fields that interact (e.g. a filter condition paired with a scope toggle, two args that must agree).
• Negative coverage — queries where the tool should NOT be called.
• Ambiguity coverage — borderline queries that test correctness under uncertainty.

Polarity and difficulty targets are stated once in step 5 below; aim for those across the whole dataset, not within each row.

If you cannot fill at least 10 rows, the target is probably too narrow — confirm scope with the user.

5. Draft queries

For each coverage dimension, write one or more queries that feel like they came from a real Phoenix user. Mix across:

• Voice: imperative ("show me LLM spans"), declarative ("I want to see only errors"), question ("what spans took over 5s?"), fragment ("LLM only").
• Polish: clean prose, terse fragments, casual typos, abbreviations, incomplete sentences.
• Personas: new user setting up Phoenix on a new project; engineer debugging an agent or RAG pipeline; PM exploring trace quality; AI engineer writing evals; annotator marking traces; researcher exploring trends. Sample across them — don't sound like one author.
• Difficulty: ~30% obvious, ~50% moderate, ~20% ambiguous / tricky.
• Polarity: at least 30% negative (tool should NOT be called).

Anti-patterns to avoid:

• Don't paraphrase the tool's docstring.
• Don't use technical jargon a real user wouldn't reach for.
• Don't write 20 minor variations of the same intent.

6. Annotate expected outputs — subprocess annotation

Ground truth must be generated in a fresh subprocess, not inline. When annotation happens in the same context that drafted the queries, the agent builds a prior from its own examples: it anchors on condition patterns it established early, collapses ambiguous cases into overconfident annotations, and fills in tool_call_args based on what earlier examples "look like" rather than what the tool spec actually says. This is context rot — annotation signal weakens as the dataset grows.

The remedy: annotate each example in a separate subprocess whose context contains the full PXI toolset, the author's design intent for the query, and the schema contract. Each annotation is an independent reasoning trace from the toolset spec.

Prepare once, reuse across examples. Before launching annotation subprocesses, extract the full toolset spec once (see section 1 below) and cache it. Every annotation subprocess across the dataset reuses the same toolset block; only the query and design-intent block change per-example.

Subprocess context (four sections, in this order)

1. Full toolset spec — the complete toolset the PXI agent sees at runtime, NOT just the tool under test. For each tool, include the name, description, and parameters_json_schema. The annotator has to decide whether the right answer is the focal tool, a different tool, or no tool at all — and cannot do that without seeing every tool's spec. This is the single biggest hedge against annotator hallucination: with only one tool visible, the annotator will force-fit it to ambiguous or negative queries.

   Source the toolset from build_toolset and build_external_toolset in src/phoenix/server/agents/toolsets/ (and src/phoenix/server/agents/toolsets/external/). The eval harness configures ChatContext such that all external tools are available — see evals/pxi/harness/agent_task.py. List every tool that survives that filtering, not just the focal one.

   Mark the focal tool explicitly at the top of this section (★ FOCAL TOOL: <name> — this dataset evaluates this tool's behavior). The annotator's job is still to reason from the spec for whichever tool best fits the query, but the highlight tells them which tool's evaluator family will judge the answer.

2. Evaluator contract — provide the dataset-specific expected-block schema derived from the selected evaluators (what each expected field means, matching semantics, and which fields may be omitted or expressed as variants). Add the explicit instruction: if you are unsure about a value, omit the key rather than guessing.

3. Query and design intent — the input.query string, plus the dataset author's framing if assigned:

   • difficulty: obvious | moderate | tricky
   • polarity: positive | negative
   • category: <coverage dimension> (the taxonomy from step 4 — parameter, value, combination, negative, ambiguity, plus any finer-grained category like span_kind, status, latency, negative_chitchat, negative_wrong_tool, etc.)
   • Any notes: the author wrote

   Frame this for the annotator as the dataset author's intent, NOT ground truth. The annotator should reason from the toolset spec and may disagree with the author — that disagreement is one of the most valuable signals this protocol produces, because it surfaces mis-classified queries.

4. Task instruction (goal-oriented, adapt only the schema reference):

   "Your goal: determine the correct expected: block for this dataset example — the expected output needed to validate whether the PXI agent took the right actions in response to the user's query, given the full toolset spec, evaluation goal, evaluator set, and expected-block schema.

   Constraints:

   • Follow the expected-block schema provided for this dataset. The schema is determined by the evaluation goal and selected evaluators; do not assume every dataset validates only tool selection or tool arguments.
   • When the schema includes tool expectations, consider every tool in the toolset spec, not just the focal one. The right answer may be the focal tool, a different tool, or no tool at all.
   • Use only values directly implied by the relevant spec and the query. Omit any key whose value the spec doesn't determine — don't invent a plausible default.
   • The author's design intent (difficulty, polarity, category, notes) is a hint, not ground truth. If your reading of the spec contradicts the author's labeled polarity, output what the spec implies and record the disagreement in metadata.notes.
   • Return exactly one best expected: block. If multiple outputs seem valid, choose the single strongest answer and mention the alternatives in your brief reasoning; the orchestrator decides later whether variants belong in the final example.

   Output format (exactly two parts, in this order):

   1. Brief reasoning — one or two sentences explaining which expected output you concluded should be used and why. Just enough that an orchestrator comparing three independent annotations can see where you agree or diverge with the others.
   2. The expected: block as YAML matching the schema reference provided in this prompt. No other prose."

   This frames the task as a goal with constraints, not a procedural recipe. Cross-model fan-out is only useful if each annotator reasons independently — prescribing the exact reasoning steps collapses that diversity. The brief-reasoning requirement gives the orchestrator enough signal to adjudicate disagreements without dictating how each annotator gets there.

Expected-block schema and annotation rules

Before launching annotation subprocesses, define the expected-block schema for this dataset from the evaluation goal and selected evaluators. Include that schema verbatim in every annotation prompt. For example, a dataset that validates tool selection and tool arguments may use expected.tools and expected.tool_call_args, while another dataset may validate response text, refusal behavior, or some other evaluator-specific field.

When the schema includes tool expectations, use these conventions:

• Use tools.required for the must-be-called tool(s).
• Use tools.exact_match: true for strict-no-extras cases.
• For a pure-negative case where no tool should be called, set tools.required: [] and tools.exact_match: true. Do not enumerate every tool in tools.forbidden; the tool list can grow and make that brittle. Do not supply tool_call_args.
• Use tool_call_args[<tool>] only when argument correctness matters; leave it off for "any args are fine" cases — especially ambiguous cases where you want to assert the tool fires but not pin its args.

How many opinions, and which models

Collect three independent opinions for every dataset example: Sonnet, Opus, and Codex. Even obvious and pure-negative examples get the full fan-out so the dataset has a consistent annotation provenance and easy-to-audit agreement metadata.

Cross-model diversity — different families, different training, and (for Codex) different agent runtime — catches single-model biases that seed-level diversity misses.

How each opinion is invoked. The Claude Code Agent tool is Anthropic-only, so the OpenAI-side opinion comes from Codex via a Bash wrapper. Using Codex (rather than a one-shot Chat Completions call) gives the OpenAI annotator the same kind of environment access a Claude subagent has — sandbox-constrained file reads, repo navigation, the ability to double-check its reading of the toolset against the actual source. This keeps the three opinions genuinely symmetric.

The Codex wrapper at .agents/skills/pxi-eval-dataset/scripts/annotate_via_codex.sh reads a prompt on stdin and writes Codex's final message to stdout. It runs codex exec with --sandbox read-only (the annotator can read the repo but cannot modify anything), --ephemeral (no session persistence between invocations), and --skip-git-repo-check. Authentication is whatever Codex is already configured with (codex login status); no OPENAI_API_KEY plumbing required. Example invocation:

cat /tmp/annotation_prompt.txt | .agents/skills/pxi-eval-dataset/scripts/annotate_via_codex.sh
# Or pin a specific model:
cat /tmp/annotation_prompt.txt | .agents/skills/pxi-eval-dataset/scripts/annotate_via_codex.sh --model o3

Run all three opinions in parallel. Send a single message containing two Agent tool calls (Sonnet, Opus) and one Bash tool call (the Codex wrapper) — they will run concurrently. Do not share context between them.

Orchestrator subprocess (required)

After the three annotation subprocesses return, run a fourth fresh subprocess as the orchestrator. Its job is to merge the candidates into a single expected: block, with authority to drop spec-invalid proposals. Use a separate subprocess so the orchestrator sees anonymized candidate blocks and cannot be biased by knowing which model produced which annotation.

Orchestrator subprocess context:

• The full toolset spec (same as the annotation subprocess — focal tool marked, all other available tools included). The orchestrator needs this to judge "spec-valid" claims, especially for negative cases where the right answer is a non-focal tool.
• The query and design intent block (same as the annotation subprocess — difficulty, polarity, category, notes). The orchestrator should also be willing to override the author's polarity if the toolset spec and the majority of annotation opinions disagree with it; flag any such override with metadata.notes: 'orchestrator overrode author polarity'.
• All three candidate expected: blocks (anonymized — do not tell the orchestrator which model produced which proposal; this prevents reputation bias)
• The dataset-specific expected-block schema

Orchestrator decision rules:

• Merge portions of the expected: block where the models agree.
• Drop any value that is invalid under the tool spec, evaluator contract, or dataset-specific expected-block schema. Justify each rejection against a specific clause.
• Where models disagree but multiple outputs are valid and the schema supports variants, include variants only for the specific fields where variants are meaningful.
• Where variants do not make sense for the field, use the majority answer.
• Where there is no valid majority and the schema allows omission, omit the disputed optional field rather than over-constraining the expected output.
• For tool-call argument variants, keep at most three variants. If there are many valid argument shapes, prefer omitting tool_call_args[<tool>] so the evaluator checks tool selection without pinning arbitrary arguments.

The orchestrator must justify any "spec-invalid" rejection against a specific clause of the tool spec. It does NOT vote: it adjudicates against the spec.