Phoenix Evals New Metric

Creating a New Built-in Classification Evaluator

A built-in evaluator is a YAML config (source of truth) that gets compiled into Python and TypeScript code, wrapped in evaluator classes, benchmarked, and documented. The whole pipeline is linear — follow these steps in order.

Step 0: Gather Requirements

Before writing anything, clarify with the user:

1. What does this evaluator measure? Get a one-sentence description of the quality dimension.
2. What input data is available? This determines the template placeholders (e.g., {{input}}, {{output}}, {{reference}}, {{tool_definitions}}). If the user is vague, ask follow-up questions — the placeholders are the contract between the evaluator and the caller.
3. What labels make sense? Binary is most common (e.g., correct/incorrect, faithful/unfaithful), but some metrics use more. Labels map to scores.
4. Should this appear in the dataset experiments UI? If yes, it needs the promoted_dataset_evaluator label. Currently only correctness, tool_selection, and tool_invocation have this — some may new evaluators don't need it.

Step 1: Create the YAML Config

Create prompts/classification_evaluator_configs/{NAME}_CLASSIFICATION_EVALUATOR_CONFIG.yaml.

Read an existing config to match the current schema. Start with CORRECTNESS_CLASSIFICATION_EVALUATOR_CONFIG.yaml for a simple example, or TOOL_SELECTION_CLASSIFICATION_EVALUATOR_CONFIG.yaml if your evaluator needs structured span data.

Key Decision Points

choices — Maps label strings to numeric scores. For binary evaluators, use positive/negative labels (e.g., correct: 1.0 / incorrect: 0.0). The labels you pick here flow through to the Python class, TS factory, and benchmarks.

optimization_direction — Use maximize when the positive label is the desired outcome (most evaluators). Use minimize only if the metric measures something undesirable (e.g., hallucination). This affects how Phoenix displays the metric in the UI.

labels — Optional list. Add promoted_dataset_evaluator only if this evaluator should appear in the dataset experiments UI sidebar.

substitutions — Only needed if the evaluator is a promoted_dataset_evaluator and works with structured span data (tool definitions, tool calls, message arrays). These reference formatter snippets defined in prompts/formatters/server.yaml. Read that file if you need substitutions — it defines what structured data formats are available. Most evaluators that only use simple text fields (input, output, reference) don't need substitutions.

Prompt Writing Tips

• Be explicit about what makes each label correct — the LLM judge needs a clear rubric
• Separate concerns: if evaluating X, explicitly state you're NOT evaluating Y
• Wrap inputs in XML-style tags (e.g., <context>, <output>) for clear data formatting
• Tell the judge to reason before deciding — this improves accuracy
• Use {{placeholder}} (Mustache syntax) for template variables

Step 2: Compile Prompts

make codegen-prompts

This generates code in three places:

• packages/phoenix-evals/src/phoenix/evals/__generated__/classification_evaluator_configs/ (Python)
• src/phoenix/__generated__/classification_evaluator_configs/ (Python, server copy)
• js/packages/phoenix-evals/src/__generated__/default_templates/ (TypeScript)

Verify the generated files look correct before moving on.

Step 3: Create the Python Evaluator

Create packages/phoenix-evals/src/phoenix/evals/metrics/{name}.py.

Read correctness.py in that directory — it's the canonical example. Your evaluator follows the same pattern: subclass ClassificationEvaluator, pull constants from the generated config, define a Pydantic input schema with fields matching your template placeholders.

After creating the file, add it to the exports in metrics/__init__.py — both the import and the __all__ list. Read the current __init__.py to see the existing pattern.

Step 4: Create the TypeScript Evaluator

Create js/packages/phoenix-evals/src/llm/create{Name}Evaluator.ts.

Read createCorrectnessEvaluator.ts — it's the canonical example. The pattern is a factory function that wraps createClassificationEvaluator with defaults from the generated config.

Then:

1. Add the export to js/packages/phoenix-evals/src/llm/index.ts
2. Add a vitest test under js/packages/phoenix-evals/test/llm/ — read createFaithfulnessEvaluator.test.ts there for the test pattern

Step 5: Build JS

cd js && pnpm build

Fix any TypeScript errors before proceeding.

Step 6: Write the Benchmark

Create js/benchmarks/evals-benchmarks/src/{name}.eval.ts.

Read existing benchmarks in that directory to match the current patterns:

• tool_invocation.eval.ts — multi-category analysis and aggregate metrics
• aggregateMetrics.ts — shared macro precision/recall/F1 accumulation