Skill Judge
shareAI-lab/lab-skills
Evaluate Agent Skill design quality with an opinionated, practice-derived rubric informed by public specifications and examples.
Design reproducible evaluations for AI agents with representative task sets, explicit rubrics, appropriate graders, baselines, regression gates, and failure analysis.
Install Agent Evaluation
$ npx skills add seb1n/awesome-ai-agent-skills --skill agent-evaluation -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install seb1n/awesome-ai-agent-skills agent-evaluation --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/seb1n/awesome-ai-agent-skills.git skills-src && mkdir -p .claude/skills && cp -r skills-src/agent-engineering/agent-evaluation .claude/skills/agent-evaluation && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "agent-evaluation" agent skill from https://github.com/seb1n/awesome-ai-agent-skills/tree/main/agent-engineering/agent-evaluation into .claude/skills/agent-evaluation/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "agent-evaluation", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/seb1n/awesome-ai-agent-skills/tree/main/agent-engineering/agent-evaluationType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add seb1n/awesome-ai-agent-skills --skill agent-evaluation -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install seb1n/awesome-ai-agent-skills agent-evaluation --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/seb1n/awesome-ai-agent-skills.git skills-src && mkdir -p .agents/skills && cp -r skills-src/agent-engineering/agent-evaluation .agents/skills/agent-evaluation && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "agent-evaluation" agent skill from https://github.com/seb1n/awesome-ai-agent-skills/tree/main/agent-engineering/agent-evaluation into .agents/skills/agent-evaluation/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "agent-evaluation", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add seb1n/awesome-ai-agent-skills --skill agent-evaluation -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install seb1n/awesome-ai-agent-skills agent-evaluation --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/seb1n/awesome-ai-agent-skills.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/agent-engineering/agent-evaluation .cursor/skills/agent-evaluation && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "agent-evaluation" agent skill from https://github.com/seb1n/awesome-ai-agent-skills/tree/main/agent-engineering/agent-evaluation into .cursor/skills/agent-evaluation/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "agent-evaluation", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/seb1n/awesome-ai-agent-skills.git --path agent-engineering/agent-evaluation--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add seb1n/awesome-ai-agent-skills --skill agent-evaluation -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install seb1n/awesome-ai-agent-skills agent-evaluation --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/seb1n/awesome-ai-agent-skills.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/agent-engineering/agent-evaluation .gemini/skills/agent-evaluation && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "agent-evaluation" agent skill from https://github.com/seb1n/awesome-ai-agent-skills/tree/main/agent-engineering/agent-evaluation into .gemini/skills/agent-evaluation/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "agent-evaluation", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install seb1n/awesome-ai-agent-skills agent-evaluationInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add seb1n/awesome-ai-agent-skills --skill agent-evaluation -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/seb1n/awesome-ai-agent-skills.git skills-src && mkdir -p .github/skills && cp -r skills-src/agent-engineering/agent-evaluation .github/skills/agent-evaluation && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "agent-evaluation" agent skill from https://github.com/seb1n/awesome-ai-agent-skills/tree/main/agent-engineering/agent-evaluation into .github/skills/agent-evaluation/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "agent-evaluation", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add seb1n/awesome-ai-agent-skills --skill agent-evaluation -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install seb1n/awesome-ai-agent-skills agent-evaluation --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/seb1n/awesome-ai-agent-skills.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/agent-engineering/agent-evaluation .opencode/skills/agent-evaluation && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "agent-evaluation" agent skill from https://github.com/seb1n/awesome-ai-agent-skills/tree/main/agent-engineering/agent-evaluation into .opencode/skills/agent-evaluation/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "agent-evaluation", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
Facts
- Skill name
agent-evaluation- GitHub stars
- 206
- Token cost
- ~1.4k tokens
- SKILL.md length
- 693 words
- Files
- 4 (incl. scripts, references)
- Skills in repo
- 92
- Repo updated
- First seen
- Licence
- MIT
At a glance
Design reproducible evaluations for AI agents with representative task sets, explicit rubrics, appropriate graders, baselines, regression gates, and failure analysis.
- Works in 5 steps: An evaluation brief with scope, risks,… → A dataset manifest with provenance,… → A scoring specification with rubrics,… → …
- Defining agent quality
- SKILL.md covers Use when, Inputs, Output contract and Workflow, plus 4 more sections
- Runs Python scripts from its folder; calls python3
What it does
Agent Evaluation is an agent skill from seb1n/awesome-ai-agent-skills. Design reproducible evaluations for AI agents with representative task sets, explicit rubrics, appropriate graders, baselines, regression gates, and failure analysis. Use when defining agent quality, comparing prompts or models, validating a release, measuring tool-use reliability, investigating regressions, or deciding whether an agent is ready for production.
Its SKILL.md is about 1.4k tokens, which your agent loads only when the skill is triggered. The skill folder holds 6 other files, including scripts and reference files (for example `agents/openai.yaml`, `references/evaluation-patterns.md` and `scripts/aggregate_results.py`).
It sits in Education, covering Agent evaluation and testing, Quizzes and assessments and Root cause analysis. The repository describes itself as: 103 ready-to-use AI agent skills for Claude Code, OpenAI Codex, Gemini CLI, Cursor, GitHub Copilot, Windsurf, and other Agent Skills-compatible tools. Complete SKILL.md… The licence is MIT.
When your agent uses it
- Defining agent quality
- Comparing prompts
- Validating a release
- Measuring tool-use reliability
Example prompts
- “/agent-evaluation”
Requirements
- Python 3
Workflow steps
5 steps, taken from the first numbered list in SKILL.md.
- An evaluation brief with scope, risks, hypotheses, and frozen system versions.
- A dataset manifest with provenance, categories, splits, and contamination controls.
- A scoring specification with rubrics, graders, thresholds, and tie-breaking rules.
- Reproducible run settings, aggregate results, uncertainty, and baseline deltas.
- A failure taxonomy, representative cases, evidence limits, and a decision memo for the accountable release owner.
What it can do on your machine
Read from SKILL.md and the folder at commit 75865a5. It shows what the files ask for, not the result of running them.
Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
Ships 1 file in scripts/ (Python), which the agent can run.
Shell commands in SKILL.md call:
python3
From the folder's file list and the shell code blocks in SKILL.md.
Network
No URLs in SKILL.md.
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Context cost
Agent Evaluation loads about 1.4k tokens when it runs, and up to ~2.3k if it reads all its reference files. Until then it costs about 95 tokens; SKILL.md has 693 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
Safety
Auto-check passed
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); the scripts in this folder are not scanned.
SKILL.md
The full file from seb1n/awesome-ai-agent-skills at commit 75865a5, republished under its MIT licence (© seb1n). 693 words, ~1,423 tokens.
Download SKILL.mdSave it as
.claude/skills/agent-evaluation/SKILL.md (or your agent's skills folder). This skill also uses 3 other files; get the full folder from GitHub.- name
- agent-evaluation
- description
- Design reproducible evaluations for AI agents with representative task sets, explicit rubrics, appropriate graders, baselines, regression gates, and failure analysis. Use when defining agent quality, comparing prompts or models, validating a release, measuring tool-use reliability, investigating regressions, or deciding whether an agent is ready for production.
Agent Evaluation
Build evidence that can inform a release owner, not a showcase of favorable examples or a safety certification.
Use when
- Define quality before building or changing an agent.
- Compare prompts, models, tools, memory strategies, or orchestration patterns.
- Convert production failures into regression cases.
- Establish a repeatable release gate or human-review plan.
Inputs
Collect the agent objective, users, supported tasks, unacceptable outcomes, current baseline, execution environment, available traces, and evaluation budget. State assumptions when an input is unavailable.
Output contract
Produce:
- An evaluation brief with scope, risks, hypotheses, and frozen system versions.
- A dataset manifest with provenance, categories, splits, and contamination controls.
- A scoring specification with rubrics, graders, thresholds, and tie-breaking rules.
- Reproducible run settings, aggregate results, uncertainty, and baseline deltas.
- A failure taxonomy, representative cases, evidence limits, and a decision memo for the accountable release owner.
Workflow
- Define the unit under test and the decision the evaluation must support. Separate model quality from tool, retrieval, policy, and infrastructure failures.
- Convert user goals and risks into observable criteria. Include task success, safety, latency, cost, and escalation quality only when relevant.
- Build representative cases from real distributions where permitted. Add boundary, long-tail, malformed-input, tool-failure, and adversarial cases. Keep a holdout set isolated from prompt iteration.
- Select the least subjective reliable grader. Prefer deterministic checks for structured facts, rubric-bound model graders for semantic quality, and blinded human review for high-impact or disputed cases. Read evaluation-patterns.md when selecting graders or gates.
- Freeze prompts, model versions, tools, data snapshots, seeds when supported, retries, and timeouts. Run the candidate and baseline under equivalent conditions; repeat stochastic cases.
- Inspect case-level failures before trusting aggregates. Slice results by task, risk, language, tool, and user cohort where sample sizes permit.
- Set a release gate that combines minimum critical-case performance, non-regression against baseline, and operational limits. Label underpowered results as inconclusive.
- Save failed production-like cases as regression fixtures without exposing private data.
Use python3 scripts/aggregate_results.py results.jsonl --score-min 0 --score-max 1 --require-passed to validate identities and declared score bounds, then produce a descriptive summary. Add --baseline old --candidate new when records use those variant labels. Add --output summary.json for an atomic file write; the destination must be a new path or regular file and cannot alias the input through spelling, resolution, a symlink, or a hard link. The script reports explicit observed, missing, and total pass-rate denominators plus approximate uncertainty; it does not certify safety, representativeness, significance, or release readiness.
Show full SKILL.md (279 more words)Show less
Safety and permissions
- Do not send private prompts, customer data, credentials, or proprietary outputs to an external grader without authorization and an approved retention policy.
- Do not run evaluations against production systems, spend paid API budget, or trigger state-changing tools without explicit permission.
- Require qualified human review for medical, legal, financial, employment, safety, or access-control decisions.
- Treat grader scores as evidence, not ground truth; disclose model-grader identity and conflicts.
Verification
- Confirm every release criterion maps to at least one case and every critical risk has a negative test.
- Verify train, development, and holdout cases do not overlap semantically or by source identifier.
- Re-run a sample manually and compare grader decisions against the rubric.
- Check that baseline and candidate used identical conditions and that reported denominators identify all missing pass labels, failures, and timeouts. Use
--require-passedwhen every record must contribute to the pass-rate denominator. - Confirm the report preserves case-level evidence needed to reproduce material claims.
Failure handling
- If representative data is missing, run a clearly labeled exploratory evaluation and request data before setting a production gate.
- If graders disagree, tighten the rubric, blind the comparison, and adjudicate a stratified sample.
- If results are unstable, increase repetitions, isolate nondeterministic dependencies, and report confidence intervals or ranges.
- If a critical case fails, block the release regardless of the overall average until an authorized owner accepts the risk.
Example
For “compare two versions of a customer-support agent,” define resolution correctness, citation fidelity, policy compliance, escalation judgment, latency, and cost; create normal, ambiguous, multilingual, prompt-injection, and unavailable-tool cases; blind the version labels; run both versions three times; aggregate by case category; inspect regressions; and return a ship, hold, or limited-rollout decision with evidence.
© seb1n, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Files
SKILL.md and 3 other files (scripts, references) in agent-engineering/agent-evaluation of seb1n/awesome-ai-agent-skills.
- SKILL.md
- agents/openai.yaml
- references/evaluation-patterns.md
- scripts/aggregate_results.py
Open the folder on GitHubat commit 75865a5
Compare with similar skills
Agent Evaluation next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Agent Evaluation this skillseb1n/awesome-ai-agent-skills | 206 | — | ~1.4k | Automated safety check: Pass | MIT | |
| Skill JudgeshareAI-lab/lab-skills | 314 | — | ~1.9k | Automated safety check: Pass | Apache-2.0 | |
| Create Custom GraderNVIDIA/SkillEvaluator | 544 | — | ~2.1k | Automated safety check: Pass | Apache-2.0 | |
| Diagnosing Superpowers Sessionsobra/superpowers | 296k | 3 repos | ~1.7k | Automated safety check: Pass | MIT | |
| Copilot Session Failure Analysisdotnet/maui | 23k | — | ~3.4k | Automated safety check: Pass | MIT | |
| A-Evolve Agent Improvementaiming-lab/AutoResearchClaw | 15k | — | ~1.8k | Automated safety check: Pass | MIT |
Similar skills
- Official
Create Custom Grader
NVIDIA/SkillEvaluator
A skill your agent uses when converting an existing benchmark, rubric, verifier, task YAML/JSON, or domain check into SkillEvaluator BYOG/BYOT custom evaluation.
Diagnosing Superpowers Sessions
obra/superpowers
Investigates a session where Superpowers went wrong, reads the transcripts on disk and produces an evidence-cited report, optionally prepared as a bug report for the maintainers.
- Official
Copilot Session Failure Analysis
dotnet/maui
Mines local Copilot CLI session logs for dotnet/maui to rank costly or failing runs, tag recurring failure modes, propose repo edits and emit guard evals.
A-Evolve Agent Improvement
aiming-lab/AutoResearchClaw
Diagnoses where an agent failed across runs and turns the findings into new skills, system prompt patches and knowledge entries, using the A-Evolve loop.
Claude Code Self-Assessment Advisor
lhfer/claude-howto-zh-cn
Quizzes you on Claude Code in a quick or deep mode, scores your level across 10 topics and recommends what to learn next, in Chinese.
More from seb1n/awesome-ai-agent-skills
All 92 skills in this repoAgent Red Teaming
seb1n/awesome-ai-agent-skills
Plan, execute, document, and retest authorized security assessments of AI agents and multi-agent workflows using safe adversarial cases, synthetic identities, canaries, and evidence-based findings.
Eu AI Act Readiness
seb1n/awesome-ai-agent-skills
Build a preliminary, evidence-based EU AI Act readiness assessment across AI-system inventory, territorial scope, operator roles, prohibited-practice screening, risk classification, transparency…
Human In The Loop
seb1n/awesome-ai-agent-skills
Design and verify auditable human oversight, approval gates, escalation paths, and safe state transitions for AI agent workflows.
MCP Server Building
seb1n/awesome-ai-agent-skills
Design, implement, harden, and verify Model Context Protocol (MCP) servers with precise tool contracts, least-privilege authorization, safe transports, structured errors, and interoperability tests.
PDF Processing
seb1n/awesome-ai-agent-skills
Inspect, extract, OCR, create, merge, split, reorder, rotate, annotate, fill, redact, compress, secure, and verify PDF documents while preserving source files and visual fidelity.
Skill Supply Chain Audit
seb1n/awesome-ai-agent-skills
Audit agent skills, plugins, prompts, manifests, scripts, dependencies, and bundled assets for provenance, prompt-injection, permission, execution, exfiltration, persistence, and update risk.
Related
Categories
Questions about Agent Evaluation
What does Agent Evaluation do?
Design reproducible evaluations for AI agents with representative task sets, explicit rubrics, appropriate graders, baselines, regression gates, and failure analysis. Agent Evaluation is an agent skill from seb1n/awesome-ai-agent-skills. Design reproducible evaluations for AI agents with representative task sets, explicit rubrics, appropriate graders, baselines, regression gates, and failure analysis.
When should I use Agent Evaluation?
Agent Evaluation fits situations like: defining agent quality; comparing prompts; validating a release; measuring tool-use reliability.
How do I install Agent Evaluation in Claude Code?
Run `npx skills add seb1n/awesome-ai-agent-skills --skill agent-evaluation -a claude-code`. Or copy the skill folder (agent-engineering/agent-evaluation in seb1n/awesome-ai-agent-skills) into .claude/skills/agent-evaluation in your project. Claude Code loads it when a task matches its description.
How do I install Agent Evaluation in Codex?
Run `npx skills add seb1n/awesome-ai-agent-skills --skill agent-evaluation -a codex`. Or copy the skill folder (agent-engineering/agent-evaluation in seb1n/awesome-ai-agent-skills) into .agents/skills/agent-evaluation in your project. Codex loads it when a task matches its description.
Can I use Agent Evaluation in Cursor, Gemini CLI or GitHub Copilot?
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add seb1n/awesome-ai-agent-skills --skill agent-evaluation -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/agent-evaluation, .gemini/skills/agent-evaluation, .github/skills/agent-evaluation and .opencode/skills/agent-evaluation in your project.
What does Agent Evaluation need to run?
Going by SKILL.md and its folder, Agent Evaluation needs Python for the scripts in its folder and the command-line tools its instructions call (python3). Our summary lists: Python 3.
Does Agent Evaluation access the network?
SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.
Is Agent Evaluation safe to install?
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. The check reads SKILL.md only: the scripts in the folder are not scanned, so read them before running anything.
What licence does Agent Evaluation use?
Agent Evaluation is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
How many tokens does Agent Evaluation use?
About 1.4k tokens (SKILL.md is roughly 5.7k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full. Its references folder adds about 870 tokens, read only when the agent opens those files.
What are the alternatives to Agent Evaluation?
Skills that share tags, products or a category with Agent Evaluation: Skill Judge (shareAI-lab/lab-skills, 314 stars), Create Custom Grader (NVIDIA/SkillEvaluator, 544 stars), Diagnosing Superpowers Sessions (obra/superpowers, 296k stars) and Copilot Session Failure Analysis (dotnet/maui, 23k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
Who maintains Agent Evaluation?
seb1n (a GitHub user) maintains it in seb1n/awesome-ai-agent-skills, which has 206 GitHub stars. The repository holds 92 skills in this directory. The repository was last updated on August 9, 2026.
Source: seb1n/awesome-ai-agent-skills on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.