New: 98,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Agent skill

Meta Eval

by agentscope-ai in agentscope-ai/OpenJudge

A skill your agent uses when the user wants to build an evaluation system for an LLM/agent application but doesn't know where to start — they have traces, prompts, RAG pipelines, or nothing at all.

Apache-2.0Auto-check passedAI & LLM Engineering

Install Meta Eval

skills CLI
$ npx skills add agentscope-ai/OpenJudge --skill meta-eval -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install agentscope-ai/OpenJudge meta-eval --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/agentscope-ai/OpenJudge.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/eval_pipeline/00-meta-eval .claude/skills/meta-eval && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
meta-eval
GitHub stars
868
Token cost
~2.5k tokens
SKILL.md length
1,009 words
Files
1
Skills in repo
19
Repo updated
First seen
Licence
Apache-2.0

At a glance

A skill your agent uses when the user wants to build an evaluation system for an LLM/agent application but doesn't know where to start — they have traces, prompts, RAG pipelines, or nothing at all.

  • Works in 4 steps: Ask 4 diagnostic questions — data,… → Match triage table — map user scenario… → Recommend sub-skill — tell the user… → …
  • The user wants to build an evaluation system for an LLM/agent application but doesnt know where to start — they have traces
  • SKILL.md covers Checklist, Diagnostic Questions, Triage Table and Output, plus 5 more sections
  • Instructions only: no scripts, shell commands, URLs or credentials in SKILL.md

What it does

Meta Eval is an agent skill from agentscope-ai/OpenJudge. Use when the user wants to build an evaluation system for an LLM/agent application but doesn't know where to start — they have traces, prompts, RAG pipelines, or nothing at all. Also use when the user mentions evaluation, eval, benchmarking, testing LLM quality, measuring agent performance, assessing RAG accuracy, or wants to compare prompts/models. This skill is the entry router: it asks diagnostic questions then recommends which sub-skill (local workflow) to use next.

Its SKILL.md is about 2.5k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.

It sits in AI & LLM Engineering, covering Retrieval-augmented generation. The repository describes itself as: OpenJudge: A Unified Framework for Holistic Evaluation and Quality Rewards. The licence is Apache-2.0.

When your agent uses it

  • The user wants to build an evaluation system for an LLM/agent application but doesnt know where to start — they have traces
  • The user mentions evaluation
  • Testing LLM quality
  • Measuring agent performance

Example prompts

  • “/meta-eval”

Workflow steps

4 steps, taken from the first numbered list in SKILL.md.

  1. Ask 4 diagnostic questions — data, labels, stakes, domain knowledge
  2. Match triage table — map user scenario to sub-skill
  3. Recommend sub-skill — tell the user which workflow to use and why
  4. Record routing decision — write a brief summary of what was diagnosed and recommended

What it can do on your machine

Read from SKILL.md and the folder at commit d1e0642. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    No scripts in the folder and no shell commands in SKILL.md.

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    No URLs in SKILL.md.

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Meta Eval loads about 2.5k tokens when it runs. Until then it costs about 121 tokens; SKILL.md has 1,009 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~121
When it runs · the whole SKILL.md, loaded when a task matches
~2.5k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from agentscope-ai/OpenJudge at commit d1e0642, republished under its Apache-2.0 licence (© agentscope-ai). 1,009 words, ~2,513 tokens.

Download SKILL.mdSave it as .claude/skills/meta-eval/SKILL.md (or your agent's skills folder).
name
meta-eval
description
Use when the user wants to build an evaluation system for an LLM/agent application but doesn't know where to start — they have traces, prompts, RAG pipelines, or nothing at all. Also use when the user mentions evaluation, eval, benchmarking, testing LLM quality, measuring agent performance, assessing RAG accuracy, or wants to compare prompts/models. This skill is the entry router: it asks diagnostic questions then recommends which sub-skill (local workflow) to use next.
<HARD-GATE>
NO sub-skill recommendation WITHOUT identifying data_form + label_status (these two pick the entry workflow).
ALWAYS give a provisional recommendation once data_form + label_status are known, even if stakes/user_prior are still unknown — then ask the remaining questions to refine the downstream path. Do not withhold the route while waiting on stakes.
</HARD-GATE>

Meta Eval

Entry router for the eval skill collection. You diagnose what the user has and route them to the right sub-skill. You don't do evaluation yourself — you're the triage desk.

Each sub-skill is self-contained: it carries inline the data shapes, statistics, and data principles it needs, so it can be installed and used on its own.

Checklist

You MUST create a task for each item and complete them in order:

  1. Ask 4 diagnostic questions — data, labels, stakes, domain knowledge
  2. Match triage table — map user scenario to sub-skill
  3. Recommend sub-skill — tell the user which workflow to use and why
  4. Record routing decision — write a brief summary of what was diagnosed and recommended

Diagnostic Questions

Ask these 4 questions (all at once — don't drip-feed):

To route you to the right evaluation skill, I need to understand your situation:

1. What data do you have?
   a) Agent traces / production logs
   b) Product spec / design docs
   c) Nothing yet — starting from scratch

2. Do you have human labels?
   a) Yes, ≥50 labeled examples
   b) Some, but fewer than 50
   c) None

3. What are the stakes?
   a) Low — internal experimentation, exploring options
   b) Production — customer-facing, quality matters
   c) Regulated — compliance requirements, audit trail needed

4. How well do you know this evaluation domain?
   a) Very well — have clear standards and criteria
   b) Somewhat — general idea but need structure
   c) Not well — exploring what "good" even means

Shortcut rule: data_form + label_status already determine the entry workflow (see triage table). The moment those two are clear — even if stakes and domain knowledge are not — give the provisional recommendation AND ask the remaining questions in the same message. stakes and user_prior refine the downstream path (how much calibration rigor, how fast a path), not the entry point. Never make the user wait a round-trip for a route you can already determine.

Example: "no logs, no labels" → recommend 08-bootstrap now, and ask stakes/domain to tune the roadmap. Don't reply with only the questionnaire.

Triage Table

Match the user's situation to a sub-skill:

These are local workflows under skills/eval_pipeline/, not packages to install — "use" a workflow means open and follow that sub-skill.

User says / hasUse workflowWhat it does
"I have agent traces / production logs"01-eval-designExtract eval dimensions from traces → design dataset in OpenJudge format
"I have principles/criteria but need test data"01-eval-designStratified sampling + adversarial generation → OpenJudge dataset
"I have principles but don't know which graders to use"02-metric-designSelect OpenJudge graders by output type → generate executable pipeline code
"I changed my prompt, is it better?"06-prompt-regressionA/B comparison with PairwiseAnalyzer, win rates + statistical significance
"I have a RAG system"05-rag-evalRetrieval + generation separation, hallucination detection, diagnostic matrix
"I have a judge + labels, want to check accuracy"03-align-humanTPR/TNR calibration, kappa agreement, human-reduction roadmap
"I want to do safety/security testing"07-redteamAttack surface analysis, jailbreak/injection generation, harmfulness grading
"I've run multiple skills, want a comprehensive report"04-eval-reportCross-skill analysis, maturity dashboard, prioritized actions
"Nothing — starting from scratch"08-bootstrapZero-shot grader generation via SimpleRubricsGenerator, v0 in 30 minutes
None of the above match—Say "this scenario isn't covered yet" and suggest filing an issue

Output

After diagnosis, respond with:

Diagnosis: data=[data_form] | labels=[label_status] | stakes=[value or "asking"] | domain=[value or "asking"]

Recommended workflow: `[skill-name]`   (provisional if stakes/domain unknown)

Why: [one sentence explaining the routing decision from data_form + label_status]

What this workflow will do: [one sentence about the output — e.g., "produces an
OpenJudge-compatible dataset with stratified sampling"]

To refine the path, also tell me: [stakes / domain knowledge, if still unknown]

Recommend exactly ONE workflow as the immediate next step. Do NOT list a second workflow as a current action — that splits the user's focus. If they ask "what comes after," point them to the Canonical Workflow below as a map for later, explicitly framed as "once you finish [recommended workflow]," not as a second thing to do now.

A ? marks a field you are still asking about. Give the recommendation now; refine later.

Show full SKILL.md (469 more words)Show less

Canonical Workflow (the standard lifecycle)

Most evaluation builds follow this order. Use it to sequence sub-skills and to state preconditions — recommend the next workflow only when its inputs exist.

1. 00-meta-eval        route to the right entry workflow
2. entry point:
     - have traces/spec  → 01-eval-design   (build the dataset)
     - nothing at all    → 08-bootstrap      (uncalibrated v0 + roadmap to labels)
3. 02-metric-design    select graders, build the GradingRunner pipeline
4. RUN the evaluation  (produces scores; needed before any A/B or calibration)
5. 03-align-human      ONLY once ≥50 human labels exist — calibrate before any
                       production gate. Production stakes REQUIRE this step.
6. scenario module (as needed):
     - 05-rag-eval          retrieval vs generation diagnosis
     - 06-prompt-regression REQUIRES paired baseline+candidate outputs on shared
                            queries — do not route here before both prompts have
                            been run and their outputs collected
     - 07-redteam           policy-first safety + over-refusal
7. 04-eval-report      synthesize maturity + ship readiness

Precondition rules to enforce when routing:

  • Do not recommend 06-prompt-regression until the user has run both the baseline and candidate prompts and has their outputs paired by query. Comparing prompts that haven't been run yet is impossible.
  • Do not call anything "production-ready" without labels + 03-align-human calibration. If stakes are production/regulated and no labels exist, the path MUST explicitly include two steps before any ship decision: (1) collect ≥50 human labels, (2) run 03-align-human to calibrate. State both steps every time production is in scope — an unlabeled system is never production-ready, no matter how good the scores look.
  • Have traces but no labels is the common case: go 01-eval-design → 02-metric-design → run → collect labels → 03-align-human. Do not jump to bootstrap (you have data) or to prompt-regression (no paired outputs yet).

Red Flags — STOP and Re-evaluate

If you catch yourself thinking:

  • "I can skip the questions, the scenario is obvious" → STOP. Even obvious cases have hidden constraints (stakes, label availability) that change the routing.
  • "I'll just recommend bootstrap, it's always safe" → STOP. Bootstrap is for "nothing" scenarios. If the user has data, they need a data-aware skill.
  • "The user didn't mention stakes, so it's probably low" → STOP. Assuming low stakes when they might be production is how uncalibrated judges slip through.
  • "I'll recommend multiple skills at once" → STOP. Recommend one at a time. Users can chain skills but the entry point should be singular.

All of these mean: Stop. Return to the diagnostic questions.

Rationalization Defense

You might thinkReality
"This is just a simple eval question""Simple" questions hide complex trade-offs. The 4 questions catch them.
"They obviously need X"Stake levels and label availability change the answer. Low stakes → fast path. Production → must calibrate.
"I'll figure it out as we go"Routing to the wrong skill wastes more time than 4 questions.
"The triage table covers everything"It covers common paths. If nothing matches, say so — don't force-fit.

Common Mistakes

  • Recommending bootstrap when the user has data. Bootstrap's SimpleRubricsGenerator is zero-shot and ignores existing labels/traces. If the user has data, use 01-eval-design.
  • Skipping the stakes question. Low-stakes scenarios can use fast paths (skip calibration). Production scenarios need hybrid mode + calibration. Regulated needs audit trails.
  • Recommending metric-design before eval-design. Without a dataset, graders have nothing to grade. Design the test data first, then the metrics.
  • Not suggesting follow-up skills. Every skill has a natural next step. Mention it so the user knows the path forward.

What This Skill Doesn't Cover

  • Running evaluations directly (sub-skills do that)
  • Continuous production monitoring (MLOps domain — use Arize, Braintrust, Datadog LLM)
  • Benchmark leaderboards with versioned public releases
  • Real-time signal stacks embedded in agent harnesses (engineering system design)

© agentscope-ai, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

Just SKILL.md in skills/eval_pipeline/00-meta-eval of agentscope-ai/OpenJudge.

Open the folder on GitHubat commit d1e0642

Compare with similar skills

Meta Eval next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Meta Eval compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Meta Eval this skillagentscope-ai/OpenJudge868—~2.5kAutomated safety check: PassApache-2.0
Chroma Vector DatabaseOrchestra-Research/AI-Research-SKILLs13k8 repos~2.3kAutomated safety check: PassMIT
LLM Application DevMoizIbnYousaf/ai-agent-skills1.1k2 repos~1.3kAutomated safety check: PassMIT
Senior Prompt Engineermaslennikov-ig/claude-code-orchestrator-kit2604 repos~1.4kAutomated safety check: PassCustom licence
MCP Local RAGshinpr/mcp-local-rag407—~4.4kAutomated safety check: PassMIT
Ms Agent Framework RAGshuyu-labs/WebCode278—~1.1kAutomated safety check: PassCustom licence

Similar skills

  • Chroma Vector Database

    Orchestra-Research/AI-Research-SKILLs

    Shows how to store documents and embeddings in Chroma, query them by similarity with metadata filters, and persist them to disk for RAG and semantic search projects.

    13k GitHub starsUsed in 8 repos~2.3k tokens
    AI & LLM EngineeringAuto-check passed

  • LLM Application Dev

    MoizIbnYousaf/ai-agent-skills

    Building applications with Large Language Models - prompt engineering, RAG patterns, and LLM integration.

    1.1k GitHub starsUsed in 2 repos~1.3k tokens
    AI & LLM EngineeringAuto-check passed

  • Senior Prompt Engineer

    maslennikov-ig/claude-code-orchestrator-kit

    Provides reference guides and Python scripts for prompt optimization, RAG evaluation, and agent orchestration when building or tuning LLM systems.

    260 GitHub starsUsed in 4 repos~1.4k tokens
    AI & LLM EngineeringAuto-check passed

  • MCP Local RAG

    shinpr/mcp-local-rag

    Searches, saves, and maintains a local document index through a local RAG MCP server.

    407 GitHub stars~4.4k tokensUpdated today
    AI & LLM EngineeringAuto-check passed

  • Ms Agent Framework RAG

    shuyu-labs/WebCode

    Comprehensive guide for building Agentic RAG systems using Microsoft Agent Framework in C.

    278 GitHub stars~1.1k tokensUpdated 3 mo ago
    AI & LLM EngineeringAuto-check passed

  • Local RAG Search

    nkapila6/mcp-local-rag

    Efficiently perform web searches using the mcp-local-rag server with semantic similarity ranking.

    134 GitHub starsUsed in 1 repo~1.6k tokens
    AI & LLM EngineeringAuto-check passed

More from agentscope-ai/OpenJudge

All 19 skills in this repo

  • Align Human

    agentscope-ai/OpenJudge

    A skill your agent uses when the user has a judge/grader and human-labeled data, and wants to measure how well the judge agrees with humans, detect systematic biases, determine whether automatic…

    868 GitHub stars~3.1k tokensUpdated 27 days ago
    Auto-check passed

  • Prompt Regression

    agentscope-ai/OpenJudge

    A skill your agent uses when the user has changed a prompt (system prompt, RAG template, agent instruction, etc.) and wants to know whether the candidate is better or worse than the baseline.

    868 GitHub stars~2.8k tokensUpdated 27 days ago
    Auto-check passed

  • RAG Eval

    agentscope-ai/OpenJudge

    A skill your agent uses when the user has a RAG (Retrieval-Augmented Generation) system and wants to evaluate its quality — separating retrieval issues from generation issues.

    868 GitHub stars~2.4k tokensUpdated 27 days ago
    Auto-check passed

  • 01 Auto Arena

    agentscope-ai/OpenJudge

    Automatically evaluate and compare multiple AI models or agents without pre-existing test data.

    868 GitHub starsUsed in 1 repo~2.5k tokens
    Auto-check passed

  • Eval Design

    agentscope-ai/OpenJudge

    A skill your agent uses when the user needs to design evaluation datasets, create test cases, stratify samples, generate adversarial examples, extract eval dimensions from traces/specs, or build a…

    868 GitHub stars~2.8k tokensUpdated 27 days ago
    Auto-check: warnings

  • 01 Graders And Pipeline

    agentscope-ai/OpenJudge

    Build custom LLM evaluation pipelines using the OpenJudge framework.

    868 GitHub starsUsed in 1 repo~1.3k tokens
    Auto-check passed

Categories

Questions about Meta Eval

What does Meta Eval do?

A skill your agent uses when the user wants to build an evaluation system for an LLM/agent application but doesn't know where to start — they have traces, prompts, RAG pipelines, or nothing at all. Meta Eval is an agent skill from agentscope-ai/OpenJudge. Use when the user wants to build an evaluation system for an LLM/agent application but doesn't know where to start — they have traces, prompts, RAG pipelines, or nothing at all.

When should I use Meta Eval?

Meta Eval fits situations like: the user wants to build an evaluation system for an LLM/agent application but doesnt know where to start — they have traces; the user mentions evaluation; testing LLM quality; measuring agent performance.

How do I install Meta Eval in Claude Code?

Run `npx skills add agentscope-ai/OpenJudge --skill meta-eval -a claude-code`. Or copy the skill folder (skills/eval_pipeline/00-meta-eval in agentscope-ai/OpenJudge) into .claude/skills/meta-eval in your project. Claude Code loads it when a task matches its description.

How do I install Meta Eval in Codex?

Run `npx skills add agentscope-ai/OpenJudge --skill meta-eval -a codex`. Or copy the skill folder (skills/eval_pipeline/00-meta-eval in agentscope-ai/OpenJudge) into .agents/skills/meta-eval in your project. Codex loads it when a task matches its description.

Can I use Meta Eval in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add agentscope-ai/OpenJudge --skill meta-eval -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/meta-eval, .gemini/skills/meta-eval, .github/skills/meta-eval and .opencode/skills/meta-eval in your project.

What does Meta Eval need to run?

SKILL.md names no scripts, command-line tools or credentials: Meta Eval is instructions for the agent only.

Does Meta Eval access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Meta Eval safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Meta Eval use?

Meta Eval is published under the Apache-2.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Meta Eval use?

About 2.5k tokens (SKILL.md is roughly 10k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Meta Eval?

Skills that share tags, products or a category with Meta Eval: Chroma Vector Database (Orchestra-Research/AI-Research-SKILLs, 13k stars), LLM Application Dev (MoizIbnYousaf/ai-agent-skills, 1.1k stars), Senior Prompt Engineer (maslennikov-ig/claude-code-orchestrator-kit, 260 stars) and MCP Local RAG (shinpr/mcp-local-rag, 407 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Meta Eval?

agentscope-ai (a GitHub organization) maintains it in agentscope-ai/OpenJudge, which has 868 GitHub stars. The repository holds 19 skills in this directory. The repository was last updated on September 11, 2026.

Source: agentscope-ai/OpenJudge on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.