Phoenix Harbor

Phoenix for Harbor

Use the Phoenix Harbor plugin to record Harbor agent evaluations as versioned Phoenix datasets, experiments, runs, scores, and ATIF traces.

Harbor runs agents and verifiers. Phoenix records and compares their results. Do not describe Phoenix as executing Harbor tasks or recalculating Harbor rewards.

Requirements

• Python 3.12 or newer
• Harbor 0.21.0 or newer
• arize-phoenix-client installed with the harbor extra
• Phoenix server 15.0 or newer

Install the client and Harbor in the same Python environment:

uv pip install "arize-phoenix-client[harbor]"

Choose the trace mode

Use atif unless the agent has no ATIF trajectory or the user does not want traces. ATIF is the default. It reads trajectory files after the final trial attempt, so the sandbox needs no Phoenix endpoint, credentials, instrumentation, or outbound network access.

Use null to record datasets, experiments, runs, and evaluations without traces:

--plugin-kwarg trace_mode=null

Live OpenTelemetry Protocol (OTLP) support is deferred to a follow-up. This release accepts atif or null, and does not link live OpenTelemetry agent traces to experiment runs.

Set the Phoenix destination

Set the endpoint for the Phoenix plugin process. ATIF agents and their sandboxes do not connect to Phoenix. Prefer environment variables so credentials do not enter shell history or job configuration:

export PHOENIX_COLLECTOR_ENDPOINT=http://localhost:6006
export PHOENIX_API_KEY=your-api-key

Omit PHOENIX_API_KEY when the Phoenix instance does not require authentication. The endpoint and api_key plugin kwargs override these values when the user asks for per-job settings.

Add the plugin

Add --plugin arize-phoenix to the user's existing harbor run command. Preserve their dataset, agent, model, environment, concurrency, retry, and task selections.

harbor run \
  -d terminal-bench/terminal-bench-2 \
  -a terminus-2 \
  -m openai/gpt-5-mini \
  --plugin arize-phoenix \
  --yes

Do not invent or replace Harbor settings that are unrelated to Phoenix.

Predict the Phoenix records

Use this mapping when explaining a job or checking its results:

Each single-step or multi-step Harbor task becomes one Phoenix dataset example. A multi-step example input includes its ordered step names and instructions. Phoenix examples keep output empty unless the task declares a reference file.

The plugin records only the terminal physical attempt for a logical trial. An attempt that Harbor will retry does not create a Phoenix run. Completion order does not define repetition numbers.

When a saved terminal ATIF trajectory ends with a user-facing textual agent turn, the plugin records it in chat-message format so Phoenix experiment comparisons render it as Markdown. Structured messages contribute their text parts in order; media parts are omitted. The output stays empty for missing or invalid trajectories, terminal tool calls, media-only turns, and state-only tasks. Multi-step tasks use the last attempted step, and continued trajectories use the terminal continuation. Output extraction still runs with trace_mode=null; that setting disables trace creation, not result display.

Successful runs written by older plugin versions keep their legacy Harbor metadata output because Phoenix runs are immutable. Resume recognizes that exact legacy shape and reuses the run.

Add optional reference outputs

A task can declare a checked-in reference file in its root task.toml:

[metadata.arize-phoenix]
reference_output_path = "tests/expected.json"

The plugin reads this UTF-8 JSON file during setup. A JSON string becomes {"messages": [{"role": "assistant", "content": "the reference text"}]}. A JSON object is stored unchanged as the example output. Other top-level types must be wrapped in an object. No setting means an empty output, even if an expected.json file exists.

Paths are relative to the downloaded task root on the Harbor host. Absolute paths, .. components, and symlinks escaping that root are rejected. Missing, unreadable, or invalid configured files fail setup before trials run. Keep references beside verifier assets, out of the agent workspace.

This works for local datasets, direct tasks, and published tasks. Multi-step tasks have one reference for the whole task, usually its expected final result. Reference-content changes create a dataset version; existing experiments keep their original version. References do not change Harbor's grading.

Name the dataset

The plugin infers a Phoenix dataset name for each supported single-source job. Provide dataset=<name> only when a job contains several direct tasks, which have no shared collection name, or when you want to customize the dataset's display name in Phoenix.

The inferred names are:

To name several direct tasks or override an inferred name, add this setting to the Harbor command:

--plugin-kwarg dataset=release-candidate-tasks

Stop and explain the constraint if the job has any unsupported source shape:

• more than one configured dataset;
• both a configured dataset and direct tasks;
• several direct tasks without dataset=<name>;
• duplicate task IDs; or
• a regrade job or another job derived from previous Harbor results.

The plugin synchronizes the complete resolved task set at job start. An unchanged set reuses the dataset version. A task addition, removal, or content change creates a version. Existing experiments stay pinned to their creation-time version.

Name experiments

The default template is:

{job.name} · {agent.name} · {agent.model}

For one agent configuration, an exact name is valid:

--plugin-kwarg experiment_name=release-candidate

For several agent configurations, use experiment_name_template. Available fields are:

• {job.name}
• {job.id}
• {dataset.name}
• {agent.name}
• {agent.model}
• {agent.short_digest}

Agent names do not need to be unique. Two agents with the same name but different effective configurations each get an experiment. If rendered names collide, the plugin appends the short agent digest. Stable identity comes from the Harbor job ID and effective agent configuration, not the display name.