Opik Test

What it does

When your agent uses it

• Turn this into a test
• Add a regression case for this trace
• Make sure this doesnt happen again
• Capture this failure

Example prompts

• “turn this into a test”
• “add a regression case for this trace”
• “make sure this doesn”
• “/opik-test”

Requirements

• Python 3
• A credential in OPIK_API_KEY
• Compatibility (from SKILL.md): Tested with Claude Code; works with any Agent Skills-compatible host (Cursor, VS Code Copilot, Codex). Requires a Python or TypeScript project with Opik configured and at least one trace, or a described input/expected pair. Install the `opik` skill alongside this one — it holds the shared test-suite and dataset references; without it, this skill falls back to the public docs.
• Pre-approved tools (allowed-tools): Read, Grep, Glob, Bash

Workflow steps

6 steps, taken from the step headings in SKILL.md.

1. 1Resolve the case
2. 2Read the trace (SDK-first)
3. 3Write the assertions
4. 4Store the case (create or append)
5. 5Verify
6. 6Report

What it can do on your machine

Read from SKILL.md and the folder at commit e0c2057. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves these tools, so the agent can use them without asking each time:

  • Read
  • Grep
  • Glob
  • Bash

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Ships script files (Python), which the agent can run.

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  Links to these hosts (documentation or services it may open):

  • comet.com

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names these keys or tokens, usually read from environment variables:

  • OPIK_API_KEY

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

• Compatibility

  Tested with Claude Code; works with any Agent Skills-compatible host (Cursor, VS Code Copilot, Codex). Requires a Python or TypeScript project with Opik configured and at least one trace, or a described input/expected pair. Install the `opik` skill alongside this one — it holds the shared test-suite and dataset references; without it, this skill falls back to the public docs.

  From compatibility in the SKILL.md frontmatter.

Context cost

Opik Test loads about 2.8k tokens when it runs. Until then it costs about 159 tokens; SKILL.md has 1,223 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~159

When it runs · the whole SKILL.md, loaded when a task matches

~2.8k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

• NotePre-approves every shell command (allowed-tools: Bash)SKILL.md

  allowed-tools: Read, Grep, Glob, Bash

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from comet-ml/opik-mcp at commit e0c2057, republished under its Apache-2.0 licence (© comet-ml). 1,223 words, ~2,783 tokens.

name

opik-test

description

Turn a failing Opik trace (or a described failure) into a repeatable regression check — a test-suite item with the trace's input and one or two binary assertions — so a fix can be verified by the compare skill. Works over the SDK; uses the MCP write tool when connected. Returns the suite, the item, and the assertion. Use for "turn this into a test", "add a regression case for this trace", "make sure this doesn't happen again", "capture this failure", "add this to the test suite". Not for running the suite (use compare), building an evaluation from scratch (use evaluate), or explaining why the trace failed (use explain).

allowed-tools

Read, Grep, Glob, Bash

compatibility

Tested with Claude Code; works with any Agent Skills-compatible host (Cursor, VS Code Copilot, Codex). Requires a Python or TypeScript project with Opik configured and at least one trace, or a described input/expected pair. Install the `opik` skill alongside this one — it holds the shared test-suite and dataset references; without it, this skill falls back to the public docs.

metadata.last_updated

2026-09-15

metadata.source_commit

2.0.0

metadata.argument-hint

[trace id, or a description of the failure to capture]

import opik

client = opik.Opik()

tid = "<trace_id>"
trace = client.get_trace_content(
    tid
)  # TracePublic: .input, .output, .project_id — NOT .project_name (accessing it raises)
project = client.rest_client.projects.get_project_by_id(trace.project_id).name
spans = client.search_spans(
    project_name=project, trace_id=tid
)  # root span (no parent_span_id) names the entrypoint the compare skill will call
# Pass project_name: without it search_spans looks in the configured default project and returns nothing.

suite = client.get_or_create_test_suite(
    name="<project>-regressions",
    project_name=project,
    tags=["regression"],
)

# Dedupe on the source trace before inserting.
existing = suite.get_items(filter_string=f'data.source_trace_id = "{tid}"')
if not existing:
    suite.insert(
        [
            {
                "data": {
                    "input": trace.input,  # verbatim from the trace
                    "source_trace_id": tid,
                    # "expected_output": "...",           # only when the correct answer is exact
                },
                "assertions": ["<positive assertion>", "<optional negative assertion>"],
                "description": "Regression from trace <tid>: <one-line failure>. Entrypoint: <root span name>",
            }
        ]
    )

© comet-ml, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 8 other files in src/opik_mcp/skills/opik-test of comet-ml/opik-mcp.

• SKILL.md
• evals/.gitignore
• evals/HARNESS.md
• evals/cases.yaml
• evals/fixtures/toolbug/agent.py
• evals/fixtures/toolbug/pyproject.toml
• evals/grader.py
• evals/metrics.py
• evals/run_evals.py

Open the folder on GitHubat commit e0c2057