Opik Compare

Compare — Candidate vs Baseline Over a Suite

Definition of done: two experiments on the same suite — a baseline and a candidate — read back item by item, with the per-metric deltas, the cases that went pass → fail (regressions) and fail → pass (fixes), the worst rows, a note on whether the runs are comparable, and the Opik compare-view link. If only one run exists, the done state is "baseline created — rerun after the change". If the suite can't be run or read, stop at the first genuine blocker and return exactly one next step. Aggregate scores alone are not a comparison; a verdict is not this skill's job.

Operate: run the candidate the same way the baseline was run, read both back from Opik rather than from the run's console output, name the specific cases that changed — and change no application code. The only file this skill writes is a throwaway runner outside the repo.

Inputs

The entry point is /opik-compare <suite> (run the suite now as the candidate, compare against the latest prior run), /opik-compare <experiment-A> <experiment-B> (read two existing runs, no new run), or /opik-compare right after /opik-test (that suite). Infer the rest; treat these as optional overrides:

• suite (default: <project>-regressions, or the one /opik-test just wrote to) · baseline (default: the most recent earlier experiment on the suite) · candidate name (default: candidate-<short git sha>) · judge model for assertions (default: the suite's / the baseline's) · runs per item (default: the suite's policy).

Ask only at a genuine, non-inferable blocker (see Blockers).

Activation — the only in-scope work

1. Resolve the suite and the baseline

The SDK calls: references/sdk-snippets.md (Resolve the suite and the baseline). Newest first is not guaranteed — sort the prior experiments by created_at yourself. Baseline = the most recent prior experiment on this suite, unless the user names one. No prior experiment → this run is the baseline (step 3 still runs; status baseline_created). Two explicit experiments → skip step 3, go to step 4.

Confirm Opik is reachable: if ~/.opik.config exists or OPIK_API_KEY is set, use it. Otherwise → Blocker.

2. Build the task adapter (code candidates)

The suite's items say what to call: each item description written by /opik-test ends in Entrypoint: <root span name>. Grep the repo for that function, import it, and wrap it:

def task(item: dict) -> dict:
    return {"input": item["input"], "output": str(entrypoint(item["input"]))}

Write the runner as a temp file outside the repo (or a scratch path the user names) — never into the codebase, never committed. It needs the app's provider credentials; if they're absent → Blocker (the app can't answer the items). Never point the runner at a production entrypoint that writes, sends, or spends.

Prompt candidates (the change is a prompt version, not code): there is no adapter — run server-side instead: references/sdk-snippets.md (Prompt candidates).

3. Run the candidate

Same suite, same version, same judge model, same runs-per-item as the baseline — vary only the thing under test. The run_tests call, and the guard for a judge that fails without raising: references/sdk-snippets.md (Run the candidate). Pass generate_report=False: the default writes opik_test_suite_reports/ into the user's repo. Do not read scores off result and stop — step 4 reads both runs from Opik so baseline and candidate go through the same path.

4. Read both runs back (SDK-first, MCP when connected)

The SDK read of both runs, and the side-by-side join: references/sdk-snippets.md (Read both runs back). get_experiment_by_name is deprecated — resolve names with get_experiments_by_name and pick by id. When the MCP is connected it does the whole of this step in one call: list('dataset_item', experiment_ids=['<baseline_id>', '<candidate_id>']) returns the cases side by side with each run's score, the per-case delta, the worst trace to open, and a warning when the two runs covered different cases or different dataset versions. Prefer it over the SDK join in references/sdk-snippets.md, which stays here for runs the MCP cannot reach. list('experiment', …) and read('experiment', id) still find and sanity-check the two runs.

5. Answer the questions, in this order

1. Comparable? Same suite version (dataset_version / item count), same judge model, same runs-per-item. If not, say so first — the deltas below are then indicative, not measured.
2. Which cases broke — passed in baseline, not in candidate. Each with its input (one line), the failing assertion, its reason, and the candidate trace link.
3. Which cases got fixed — the reverse.
4. Net effect per metric — mean of each feedback_scores name in both runs, and the pass rate; report baseline → candidate (delta).
5. Worst rows — lowest candidate scores; are they the same rows as the baseline's worst?
6. App or judge? A flip whose reason cites the output is the app; a flip on an unchanged output with a wavering reason is the judge — flag it, and suggest runs_per_item: 3 for that item rather than trusting one run.
7. Cost vs quality — total_estimated_cost / duration on the candidate traces vs baseline, when the traces carry them.
8. Flaky cases — items that flip across repeated runs of the same code (the suite's execution policy exposes runs_passed/runs_total).
9. Trend — when more than two runs exist, the pass rate across the last few, so a one-step delta has context.

Report what the numbers say. Do not decide ship or hold — that is /opik-verify, which applies an explicit release policy to these numbers.