Opik Online Eval

Online Eval — Take a Judge Live

Definition of done: an online evaluation rule exists on the project, is enabled, and has scored at least one new trace — confirmed by reading a fresh trace's feedback score, not by the create call returning. The rule scores one failure mode, samples at a rate the project's volume can afford, carries a cost cap, and maps its variables to the fields the traces actually have. If traffic hasn't arrived yet, the done state is "rule live, unverified — watch this filter". If the rule can't be created, stop at the first genuine blocker and return exactly one next step.

Operate: one failure mode per rule, sample before you scale, cap the spend, verify on a real trace — and change no application code. This skill writes to Opik only.

Inputs

The entry point is /opik-online-eval <what to score> ("hallucinations", "the refund-window answer", "refusals"), /opik-online-eval right after /opik-evaluate validated a judge (take that judge live), or /opik-online-eval <project>. Infer the rest; treat these as optional overrides:

• project (default: configured) · scope (default: trace; span for one model call, thread for whole conversations) · sampling rate (default: 1.0 under ~1k traces/day, else 0.1–0.2) · filters (default: none; typical: an environment or tag filter) · judge model · cost cap (default: set one) · variable mapping (default: input → input, output → output).

Ask only at a genuine, non-inferable blocker (see Blockers).

Activation — the only in-scope work

1. Resolve project, judge, and scope

Confirm Opik is reachable (~/.opik.config or OPIK_API_KEY; otherwise → Blocker). Resolve the project id from a trace or by name. Decide the judge:

• A judge /opik-evaluate already validated → reuse its prompt and output schema verbatim.
• A named metric (hallucination, answer relevance, moderation) → a minimal binary judge for that one failure mode (../opik-evaluate/references/write-judge-prompt.md).
• A mechanical check (JSON valid, contains a disclaimer, latency budget) → a Python metric rule, not a judge.

Scope: trace by default; span when the check is about one LLM/tool call; thread when the check needs the whole conversation (thread rules wait for the thread to go inactive — 15 min by default).

2. Look at the traces before mapping variables

Read three recent production traces and note the real shape of input and output. Experiment runs land in the same project with a different input shape (their metadata carries test_suite_experiment_id), so skip those — filtering on the app's entrypoint is the reliable way: client.search_traces(project_name=…, max_results=3, filter_string='name = "<entrypoint>"'). (OQL has no is_empty for metadata.* keys.) Variables are plain field paths, dot-notation for nested keys (output.answer, input.messages) — never {{ }} templates. A wrong path is the most common reason a rule silently scores nothing.

3. Check what already runs

import opik

client = opik.Opik()
existing = client.rest_client.automation_rule_evaluators.find_evaluators(project_id="<project_id>")

Same name or same failure mode already there → don't create a second one; report exists (offer to adjust sampling/enable). When the hosted MCP is connected, list('online_rule', project_id=…) and list('score_name', project_id=…) show the same, with each rule's type, enabled flag, and sampling rate.

4. Create the rule

There is no high-level SDK wrapper; use the REST client. LLM-as-judge, trace scope:

from opik.rest_api.types import (
    AutomationRuleEvaluatorWrite_LlmAsJudge,
    LlmAsJudgeCodeWrite,
    LlmAsJudgeModelParametersWrite,
    LlmAsJudgeMessageWrite,
    LlmAsJudgeOutputSchemaWrite,
)

rule = AutomationRuleEvaluatorWrite_LlmAsJudge(
    action="evaluator",  # required literal; the model rejects the payload without it
    name="refund_window_correct",  # becomes the feedback-score name on every scored trace — use underscores, not hyphens: OQL parses `feedback_scores.a-b` as an operator
    project_ids=["<project_id>"],
    sampling_rate=0.2,  # fraction of SDK-logged traces scored
    enabled=True,
    filters=[],  # e.g. [{"field": "tags", "operator": "contains", "value": "production"}]
    code=LlmAsJudgeCodeWrite(
        model=LlmAsJudgeModelParametersWrite(name="<judge model>", temperature=0.0),
        messages=[
            LlmAsJudgeMessageWrite(
                role="USER", content="<the validated judge prompt using {{input}} and {{output}}>"
            )
        ],
        variables={"input": "input", "output": "output"},  # field paths from step 2
        schema_=[
            LlmAsJudgeOutputSchemaWrite(
                name="refund_window_correct",
                type="BOOLEAN",
                description="True if the response states 5-7 business days",
            )
        ],
        max_cost_usd=5.0,  # per-rule spend cap — set it
    ),
)
created = client.rest_client.automation_rule_evaluators.create_automation_rule_evaluator(
    request=rule
)

Notes: on Opik Cloud without your own provider key, model.name="opik-free-model" uses the workspace's built-in free provider; the Python attribute is schema_ (wire name schema); sampling_rate applies to production traces only (experiment traces are always scored in full); trigger_scope defaults to production. Span and thread variants: AutomationRuleEvaluatorWrite_SpanLlmAsJudge, AutomationRuleEvaluatorWrite_TraceThreadLlmAsJudge. Python metric: AutomationRuleEvaluatorWrite_UserDefinedMetricPython with code={"metric": "<python source defining a BaseMetric>", "arguments": {"output": "output"}}.

Endpoint, if scripting outside Python: POST /v1/private/automations/evaluators/ with the same body.

5. Verify on a real trace

import time

for _ in range(12):  # ~2 min
    scored = client.search_traces(
        project_name="<project>",
        max_results=1,
        filter_string="feedback_scores.refund_window_correct is_not_empty",
    )
    if scored:
        break
    time.sleep(10)

If the score name already contains a hyphen (an existing rule), double-quote the key or the OQL parser reads the hyphen as an operator: filter_string='feedback_scores."refund-window-correct" is_not_empty'. A scored trace → live. None, and the project had no new traces in the window → live_unverified with the filter to watch. None, but traces did arrive → read the rule's logs (get_evaluator_logs_by_id(id)) — a variable-path error or model failure shows there; fix and re-verify.

6. Report

Rule name/id, scope, sampling, cost cap, the score name, the verification trace link, and one next step (see Output). Natural next steps: /opik-diagnose will now surface low scores on this name; an alert on the score threshold; or, if the judge wasn't validated first, "validate it against 20 human labels (/opik-evaluate) before anyone acts on it".