New: 97,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Agent skill

Opik Evaluate

by comet-ml in comet-ml/opik-mcp

Build an LLM evaluation and run it against the app, returning an Opik experiment with scores and its link.

Apache-2.0Auto-check: notesAI & LLM Engineering

Install Opik Evaluate

skills CLI
$ npx skills add comet-ml/opik-mcp --skill opik-evaluate -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install comet-ml/opik-mcp opik-evaluate --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/comet-ml/opik-mcp.git skills-src && mkdir -p .claude/skills && cp -r skills-src/src/opik_mcp/skills/opik-evaluate .claude/skills/opik-evaluate && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
opik-evaluate
GitHub stars
219
Token cost
~2.5k tokens
SKILL.md length
1,099 words
Files
19 (incl. references)
Skills in repo
10
Repo updated
First seen
Licence
Apache-2.0

At a glance

Build an LLM evaluation and run it against the app, returning an Opik experiment with scores and its link.

  • Works in 8 steps: Resolve the target → Ground it in failures (error analysis) → Choose the shape → …
  • Evaluate my agent
  • SKILL.md covers Inputs, Activation — the only in-scope…, Blockers and Output, plus 4 more sections
  • Runs Python scripts from its folder; needs OPIK_API_KEY and OPENAI_API_KEY

What it does

Opik Evaluate is an agent skill from comet-ml/opik-mcp. Build an LLM evaluation and run it against the app, returning an Opik experiment with scores and its link. Picks a test suite with judge assertions or a dataset with metrics, sources cases from traces or synthetic data, scores heuristics-first then one-failure-mode judges, runs client-side via the SDK or server-side for prompt-only targets, and reads the scores back. Covers RAG evaluation, error analysis, writing and validating LLM judges against human labels, and auditing an existing eval pipeline. Use for…

Its SKILL.md is about 2.5k tokens, which your agent loads only when the skill is triggered. The skill folder holds 22 other files, including reference files (for example `evals/HARNESS.md`, `evals/cases.yaml` and `evals/fixtures/support/agent.py`). Compatibility notes: Tested with Claude Code; works with any Agent Skills-compatible host (Cursor, VS Code Copilot, Codex). Requires a Python or TypeScript project with Opik…

It sits in AI & LLM Engineering, covering LLM evaluation and Test generation. It works with Model Context Protocol. The repository describes itself as: Model Context Protocol (MCP) server for Opik, the open-source LLM observability and evaluation platform, built by Comet. Read traces, log scores, and manage prompts from Claude… The licence is Apache-2.0.

When your agent uses it

  • Evaluate my agent
  • Measure quality
  • Write an LLM judge for hallucinations
  • Audit our evaluation pipeline

Example prompts

  • “evaluate my agent”
  • “measure quality”
  • “build an eval”
  • “/opik-evaluate”

Requirements

  • Python 3
  • A credential in OPIK_API_KEY
  • A credential in OPENAI_API_KEY
  • Compatibility (from SKILL.md): Tested with Claude Code; works with any Agent Skills-compatible host (Cursor, VS Code Copilot, Codex). Requires a Python or TypeScript project with Opik configured. Install the `opik` skill alongside this one — it holds the shared test-suite, dataset, and metric references; without it, this skill falls back to the public docs.
  • Pre-approved tools (allowed-tools): Read, Grep, Glob, Bash, Write

Workflow steps

8 steps, taken from the step headings in SKILL.md.

  1. Resolve the target
  2. Ground it in failures (error analysis)
  3. Choose the shape
  4. Build the cases
  5. Define the scoring
  6. Run it
  7. Read the scores back
  8. Report

What it can do on your machine

Read from SKILL.md and the folder at commit f1dd464. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves these tools, so the agent can use them without asking each time:

    • Read
    • Grep
    • Glob
    • Bash
    • Write

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Ships script files (Python, from the files we listed), which the agent can run.

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    Links to these hosts (documentation or services it may open):

    • comet.com

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names these keys or tokens, usually read from environment variables:

    • OPIK_API_KEY
    • OPENAI_API_KEY

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

  • Compatibility

    Tested with Claude Code; works with any Agent Skills-compatible host (Cursor, VS Code Copilot, Codex). Requires a Python or TypeScript project with Opik configured. Install the `opik` skill alongside this one — it holds the shared test-suite, dataset, and metric references; without it, this skill falls back to the public docs.

    From compatibility in the SKILL.md frontmatter.

Context cost

Opik Evaluate loads about 2.5k tokens when it runs, and up to ~14k if it reads all its reference files. Until then it costs about 220 tokens; SKILL.md has 1,099 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~220
When it runs · the whole SKILL.md, loaded when a task matches
~2.5k
With references · SKILL.md plus every file in references/, read only if the agent opens them
~14k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check: notes

The automated check noted patterns worth knowing about, such as sudo or a known installer.

  • NotePre-approves every shell command (allowed-tools: Bash)SKILL.md
    allowed-tools: Read, Grep, Glob, Bash, Write

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from comet-ml/opik-mcp at commit f1dd464, republished under its Apache-2.0 licence (© comet-ml). 1,099 words, ~2,518 tokens.

Download SKILL.mdSave it as .claude/skills/opik-evaluate/SKILL.md (or your agent's skills folder). This skill also uses 18 other files; get the full folder from GitHub.
name
opik-evaluate
description
Build an LLM evaluation and run it against the app, returning an Opik experiment with scores and its link. Picks a test suite with judge assertions or a dataset with metrics, sources cases from traces or synthetic data, scores heuristics-first then one-failure-mode judges, runs client-side via the SDK or server-side for prompt-only targets, and reads the scores back. Covers RAG evaluation, error analysis, writing and validating LLM judges against human labels, and auditing an existing eval pipeline. Use for "evaluate my agent", "measure quality", "build an eval", "write an LLM judge for hallucinations", "audit our evaluation pipeline", "how good is my RAG", "set up evals for this". Not for before/after on an existing suite (use compare), one regression case (use test), scoring production traffic (use online-eval), or the ship/hold decision (use verify).
allowed-tools
Read, Grep, Glob, Bash, Write
compatibility
Tested with Claude Code; works with any Agent Skills-compatible host (Cursor, VS Code Copilot, Codex). Requires a Python or TypeScript project with Opik configured. Install the `opik` skill alongside this one — it holds the shared test-suite, dataset, and metric references; without it, this skill falls back to the public docs.
metadata.last_updated
2026-09-15
metadata.source_commit
2.0.0
metadata.argument-hint
[optional: what to evaluate, or a dataset/suite name]

Evaluate — Build an Evaluation and Run It

Definition of done: an experiment in Opik with scores, its link, and a score summary the user can act on — produced by running the user's app (or prompt) over a set of cases with scoring that targets real failure modes. If the evaluation can't be built or run, stop at the first genuine blocker and return exactly one next step. A dataset with no run, a judge prompt with no scores, or a plan in prose is not success.

Operate: ground the cases in what actually goes wrong, score with code before judges, judge one failure mode at a time, run once end to end, read the scores back from Opik — and change no application code. The only file this skill writes is a runner outside the repo.

Inputs

The entry point is /opik-evaluate (evaluate the app in this repo), /opik-evaluate <what> ("the RAG answers", "the refund flow"), or /opik-evaluate <dataset-or-suite> (run against existing cases). Infer the rest; treat these as optional overrides:

  • cases source (default: existing suite/dataset → recent traces → synthetic) · scoring (default: heuristics where an expected output exists, else one judge per failure mode) · approach (default: test suite for agents; dataset + evaluate() when you need built-in metrics or exact matching) · judge model · sample size (default: 20–50 items).

Ask only at a genuine, non-inferable blocker (see Blockers).

Activation — the only in-scope work

1. Resolve the target

Confirm Opik is reachable (~/.opik.config or OPIK_API_KEY; otherwise → Blocker). Find the entrypoint to evaluate (the function a trace's root span names, or the one the user points at). Check what already exists — client.get_test_suites(project_name=…), client.get_datasets() — and reuse before creating.

Have an eval already? Audit it first (references/eval-audit.md): unvalidated judges, no error analysis, vanity metrics. Fix the worst gap, then run.

2. Ground it in failures (error analysis)

Read real traces before writing any scorer — client.search_traces(project_name=…, max_results=100) (errors, low scores, long durations first). Categorize what goes wrong and how often (references/error-analysis.md). No traces yet → generate cases (references/generate-synthetic-data.md) and say so in the report.

3. Choose the shape
SituationUse
Agent / chatbot, expectations are behaviors ("mentions Paris", "declines legal advice")Test suite — items + string assertions checked by a judge; opik.run_tests()
Exact expected outputs, or built-in metrics (Hallucination, AnswerRelevance, RAG ContextPrecision/ContextRecall)Dataset + evaluate() with scoring_metrics
The thing under test is a prompt version, not codeServer-side: client.rest_client.experiments.execute_experiment(...) — no runner

Prefer the test suite for agents; it is what /opik-test and /opik-compare operate on.

4. Build the cases

The test-suite calls: references/sdk-snippets.md (Build the cases). For the dataset path: client.get_or_create_dataset(name, project_name) then dataset.insert([{"input": …, "expected_output": …}]). Store inputs verbatim; keep source_trace_id so cases trace back to production.

5. Define the scoring
  1. Code first. Equals, Contains, RegexMatch, IsJson, JsonSchemaMatch, LevenshteinRatio from opik.evaluation.metrics whenever the check is mechanical.
  2. Then one judge per failure mode, binary pass/fail, from step 2's categories (references/write-judge-prompt.md). As a suite assertion, or as a GEval / custom BaseMetric on the dataset path.
  3. RAG: score retrieval and generation separately (references/evaluate-rag.md).
  4. Before trusting a judge on anything that matters, calibrate it against a few human labels (references/validate-evaluator.md) — TPR/TNR, not accuracy.
6. Run it

Write the task adapter as a temp file outside the repo (needs the app's provider credentials — absent → Blocker). Never run a production entrypoint that writes, sends, or spends. The run_tests and evaluate() calls: references/sdk-snippets.md (Run it). Pass generate_report=False: the default writes opik_test_suite_reports/ into the user's repo. project_name matters: datasets, suites, prompts, and experiments are project-scoped, and it must match the tracing project if the app uses @track. Set it when creating the dataset or suite — evaluate() inherits the dataset's project, and its own project_name kwarg is deprecated (the SDK warns and ignores it).

Judge credential guard: if the LLM judge (suite assertions, or an LLM metric) has no provider key, run_tests/evaluate do not raise — every item scores 0 with scoring_failed=True and a "Missing credentials" reason, and the experiment is still created. Check for that before reporting; it is a Blocker ("set the judge's provider key and rerun"), not a result.

Show full SKILL.md (437 more words)Show less
7. Read the scores back

The SDK read and the aggregates: references/sdk-snippets.md (Read the scores back). On the dataset path res.aggregate_evaluation_scores().aggregated_scores gives per-metric statistics directly. Name the worst items and the failure mode each hit — that is the actionable part. (get_experiment_by_name is deprecated; use get_experiments_by_name / get_experiment_by_id.)

8. Report

Experiment link, the score table, the three worst items with their reasons, what the cases were grounded in (traces vs synthetic), and one next step (see Output). This run is the baseline /opik-compare will compare against.

Blockers

Stop at the earliest blocker and return exactly one next step:

  • "Run opik configure, then rerun /opik-evaluate."
  • "Which function should I evaluate? Point me at the entrypoint (e.g. answer(question))."
  • "The runner needs a provider credential — set OPENAI_API_KEY (or the relevant key) and rerun."
  • "No traces and no cases yet — give me 5–10 example inputs (with expected behavior if known), or say synthetic and I'll generate them."

Output

User-facing: a short human message — the experiment link, the score table, the worst items with reasons, the case source, and the single next step. Not a raw dump of every item.

Underneath (for composition / evals), one shape, with its invariants: references/output-shape.md.

Examples

Worked runs (agent from traces, RAG dataset path, prompt only, blocked): references/examples.md.

Key principles

  • Error analysis before evaluators. Never write a scorer without reading traces first.
  • Code checks before LLM judges. Heuristic metrics wherever the check is mechanical.
  • Binary pass/fail, one failure mode per judge. Holistic judges give unactionable verdicts.
  • Validate judges against human labels before they gate anything (TPR/TNR).
  • Always pass project_name where the object is created. To get_or_create_dataset, get_or_create_test_suite, create_prompt. evaluate() and run_tests() inherit it from the dataset/suite (the evaluate(project_name=…) kwarg is deprecated).
  • Read results from Opik, not from stdout — so /opik-compare reads the same numbers later.

Anti-patterns

Building a judge before reading a single trace; a "quality 1–10" judge; scoring with a judge what Equals could check; a dataset with no run; reporting an aggregate without naming the worst cases; writing the runner into the repo; editing app code to make the eval pass; skipping project_name; deprecated get_experiment_by_name.

References

Methodology, in this skill's own references: references/eval-audit.md (audit an existing pipeline), references/error-analysis.md (failure categorization from traces), references/generate-synthetic-data.md (dimension-based inputs), references/write-judge-prompt.md (binary judges), references/validate-evaluator.md (TPR/TNR calibration), references/evaluate-rag.md (retrieval vs generation).

SDK detail lives in the opik skill, installed beside this one — paths relative to this file: ../opik/references/evaluation-test-suites.md (suites, run_tests, results), ../opik/references/evaluation-datasets.md (evaluate(), 60+ metrics, OQL, datasets from traces), ../opik/references/production.md (search_traces). If your host lays skills out differently, locate the opik skill's references/ directory.

If the opik skill isn't installed, say so in the report and use https://www.comet.com/docs/opik/ rather than working from memory.

© comet-ml, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 18 other files (references) in src/opik_mcp/skills/opik-evaluate of comet-ml/opik-mcp.

  • SKILL.md
  • evals/.gitignore
  • evals/HARNESS.md
  • evals/cases.yaml
  • evals/fixtures/support/agent.py
  • evals/fixtures/support/pyproject.toml
  • evals/fixtures/support/seed.py
  • evals/grader.py
  • evals/metrics.py
  • evals/run_evals.py
  • references/error-analysis.md
  • references/eval-audit.md
  • references/evaluate-rag.md
  • references/examples.md
  • references/generate-synthetic-data.md
  • references/output-shape.md
  • references/sdk-snippets.md
  • … and 2 more

Open the folder on GitHubat commit f1dd464

Compare with similar skills

Opik Evaluate next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Opik Evaluate compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Opik Evaluate this skillcomet-ml/opik-mcp219—~2.5kAutomated safety check: NotesApache-2.0
Quality FlywheelGoogleCloudPlatform/vertex-ai-samples791—~2kAutomated safety check: PassApache-2.0
LLM Eval Pipeline Auditai-evals-course/evals-skills1.5k—~2.5kAutomated safety check: PassApache-2.0
Synthetic Eval Data Generatorai-evals-course/evals-skills1.5k—~1.4kAutomated safety check: PassApache-2.0
Braintrust Agent Evalsgithits-com/githits-cli114—~3.1kAutomated safety check: PassApache-2.0
Compliance Drift Evalsucsandman/DashClaw310—~1.8kAutomated safety check: PassMIT

Similar skills

  • Quality Flywheel

    GoogleCloudPlatform/vertex-ai-samples

    Evaluate and improve GenAI models and agents using the Google GenAI Evaluation SDK.

    791 GitHub stars~2k tokensUpdated yesterday
    AI & LLM EngineeringAuto-check passed

  • LLM Eval Pipeline Audit

    ai-evals-course/evals-skills

    Inspects an LLM evaluation setup for missing error analysis, unvalidated judges and vanity metrics, and ranks the problems by impact with fixes.

    1.5k GitHub stars~2.5k tokensUpdated 13 days ago
    AI & LLM EngineeringAuto-check passed

  • Synthetic Eval Data Generator

    ai-evals-course/evals-skills

    Builds diverse synthetic test inputs for LLM pipeline evaluation by defining failure-focused dimensions, drafting tuples with you and turning them into realistic queries.

    1.5k GitHub stars~1.4k tokensUpdated 13 days ago
    AI & LLM EngineeringAuto-check passed

  • Braintrust Agent Evals

    githits-com/githits-cli

    Inspect, query, compare, or explicitly export GitHits agent-eval history in Braintrust using the repository's verified workflow.

    114 GitHub stars~3.1k tokensUpdated yesterday
    AI & LLM EngineeringAuto-check passed

  • Compliance Drift Evals

    ucsandman/DashClaw

    Set up compliance exports, drift detection, evaluations, scoring, and learning analytics

    310 GitHub stars~1.8k tokensUpdated yesterday
    AI & LLM EngineeringAuto-check passed

  • Suede AI Eval

    JasonColapietro/suede-creator-skills

    Suede AI eval design and coverage audit: AI-SPEC, failure-mode rubric with severity scoring, concrete pass/fail eval cases, coverage and infrastructure scores, and mechanical acceptance gates.

    127 GitHub stars~3.3k tokensUpdated 3 days ago
    AI & LLM EngineeringAuto-check passed

More from comet-ml/opik-mcp

All 10 skills in this repo

  • Opik

    comet-ml/opik-mcp

    Reference for the Opik SDK — tracing, span types, framework integrations, threads, and the prompt library (Python, TypeScript, REST).

    219 GitHub stars~2.1k tokensUpdated yesterday
    Auto-check passed

  • Opik Compare

    comet-ml/opik-mcp

    Run a candidate against the baseline over an Opik test suite and read the numbers back — which cases broke, which got fixed, the per-metric deltas, worst rows, and whether the two runs are…

    219 GitHub stars~2.6k tokensUpdated yesterday
    Auto-check: notes

  • Opik Diagnose

    comet-ml/opik-mcp

    Surface the Opik traces worth a developer's attention, ranked by signal — Diagnostics issues first, then errors, failed tool calls, latency, regressions, and low online-eval scores.

    219 GitHub stars~2.7k tokensUpdated yesterday
    Auto-check: notes

  • Opik Instrument

    comet-ml/opik-mcp

    Add Opik tracing to an existing app and verify a real trace lands.

    219 GitHub stars~2.7k tokensUpdated yesterday
    Auto-check: notes

  • Opik Optimize

    comet-ml/opik-mcp

    Improve a prompt with the Opik Agent Optimizer — resolve the prompt, a dataset, and a metric, pick the algorithm, run a bounded optimization, check the gain on held-out data, and save the winner as…

    219 GitHub stars~2.6k tokensUpdated yesterday
    Auto-check: notes

  • Opik Verify

    comet-ml/opik-mcp

    Decide ship or hold for a candidate from the compare skill's numbers, against an explicit release policy — regressions, pass rate, safety-tagged cases, subgroup consistency, latency and cost…

    219 GitHub stars~2.8k tokensUpdated yesterday
    Auto-check: notes

Works with

Categories

Questions about Opik Evaluate

What does Opik Evaluate do?

Build an LLM evaluation and run it against the app, returning an Opik experiment with scores and its link. Opik Evaluate is an agent skill from comet-ml/opik-mcp. Build an LLM evaluation and run it against the app, returning an Opik experiment with scores and its link.

When should I use Opik Evaluate?

Opik Evaluate fits situations like: evaluate my agent; measure quality; write an LLM judge for hallucinations; audit our evaluation pipeline.

How do I install Opik Evaluate in Claude Code?

Run `npx skills add comet-ml/opik-mcp --skill opik-evaluate -a claude-code`. Or copy the skill folder (src/opik_mcp/skills/opik-evaluate in comet-ml/opik-mcp) into .claude/skills/opik-evaluate in your project. Claude Code loads it when a task matches its description.

How do I install Opik Evaluate in Codex?

Run `npx skills add comet-ml/opik-mcp --skill opik-evaluate -a codex`. Or copy the skill folder (src/opik_mcp/skills/opik-evaluate in comet-ml/opik-mcp) into .agents/skills/opik-evaluate in your project. Codex loads it when a task matches its description.

Can I use Opik Evaluate in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add comet-ml/opik-mcp --skill opik-evaluate -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/opik-evaluate, .gemini/skills/opik-evaluate, .github/skills/opik-evaluate and .opencode/skills/opik-evaluate in your project.

What does Opik Evaluate need to run?

Going by SKILL.md and its folder, Opik Evaluate needs Python for the scripts in its folder and credentials named OPIK_API_KEY and OPENAI_API_KEY. Our summary lists: Python 3; A credential in OPIK_API_KEY; A credential in OPENAI_API_KEY. Its frontmatter pre-approves these tools: Read, Grep, Glob, Bash, Write. Compatibility (from SKILL.md): Tested with Claude Code; works with any Agent Skills-compatible host (Cursor, VS Code Copilot, Codex). Requires a Python or TypeScript project with Opik configured. Install the `opik` skill alongside this one — it holds the shared test-suite, dataset, and metric references; without it, this skill falls back to the public docs..

Does Opik Evaluate access the network?

SKILL.md names 1 domain. As links in the text: comet.com. This is read from the text; nothing was executed.

Is Opik Evaluate safe to install?

Our automated static check of SKILL.md found notes only (pre-approves every shell command (allowed-tools: bash)), nothing it rates as a warning. It is not a guarantee. Review the folder before installing.

What licence does Opik Evaluate use?

Opik Evaluate is published under the Apache-2.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Opik Evaluate use?

About 2.5k tokens (SKILL.md is roughly 10k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full. Its references folder adds about 12k tokens, read only when the agent opens those files.

What are the alternatives to Opik Evaluate?

Skills that share tags, products or a category with Opik Evaluate: Quality Flywheel (GoogleCloudPlatform/vertex-ai-samples, 791 stars), LLM Eval Pipeline Audit (ai-evals-course/evals-skills, 1.5k stars), Synthetic Eval Data Generator (ai-evals-course/evals-skills, 1.5k stars) and Braintrust Agent Evals (githits-com/githits-cli, 114 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Opik Evaluate?

comet-ml (a GitHub organization) maintains it in comet-ml/opik-mcp, which has 219 GitHub stars. The repository holds 10 skills in this directory. The repository was last updated on October 6, 2026.

Source: comet-ml/opik-mcp on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.