Advanced Evaluation
aiskillstore/marketplace
This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise…
Design an evaluation plan for an LLM or AI feature before shipping it.
Install AI Eval Plan
$ npx skills add mohitagw15856/pm-claude-skills --skill ai-eval-plan -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install mohitagw15856/pm-claude-skills ai-eval-plan --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/mohitagw15856/pm-claude-skills.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/ai-eval-plan .claude/skills/ai-eval-plan && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "ai-eval-plan" agent skill from https://github.com/mohitagw15856/pm-claude-skills/tree/main/skills/ai-eval-plan into .claude/skills/ai-eval-plan/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "ai-eval-plan", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/mohitagw15856/pm-claude-skills/tree/main/skills/ai-eval-planType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add mohitagw15856/pm-claude-skills --skill ai-eval-plan -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install mohitagw15856/pm-claude-skills ai-eval-plan --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/mohitagw15856/pm-claude-skills.git skills-src && mkdir -p .agents/skills && cp -r skills-src/skills/ai-eval-plan .agents/skills/ai-eval-plan && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "ai-eval-plan" agent skill from https://github.com/mohitagw15856/pm-claude-skills/tree/main/skills/ai-eval-plan into .agents/skills/ai-eval-plan/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "ai-eval-plan", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add mohitagw15856/pm-claude-skills --skill ai-eval-plan -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install mohitagw15856/pm-claude-skills ai-eval-plan --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/mohitagw15856/pm-claude-skills.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/skills/ai-eval-plan .cursor/skills/ai-eval-plan && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "ai-eval-plan" agent skill from https://github.com/mohitagw15856/pm-claude-skills/tree/main/skills/ai-eval-plan into .cursor/skills/ai-eval-plan/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "ai-eval-plan", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/mohitagw15856/pm-claude-skills.git --path skills/ai-eval-plan--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add mohitagw15856/pm-claude-skills --skill ai-eval-plan -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install mohitagw15856/pm-claude-skills ai-eval-plan --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/mohitagw15856/pm-claude-skills.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/skills/ai-eval-plan .gemini/skills/ai-eval-plan && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "ai-eval-plan" agent skill from https://github.com/mohitagw15856/pm-claude-skills/tree/main/skills/ai-eval-plan into .gemini/skills/ai-eval-plan/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "ai-eval-plan", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install mohitagw15856/pm-claude-skills ai-eval-planInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add mohitagw15856/pm-claude-skills --skill ai-eval-plan -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/mohitagw15856/pm-claude-skills.git skills-src && mkdir -p .github/skills && cp -r skills-src/skills/ai-eval-plan .github/skills/ai-eval-plan && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "ai-eval-plan" agent skill from https://github.com/mohitagw15856/pm-claude-skills/tree/main/skills/ai-eval-plan into .github/skills/ai-eval-plan/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "ai-eval-plan", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add mohitagw15856/pm-claude-skills --skill ai-eval-plan -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install mohitagw15856/pm-claude-skills ai-eval-plan --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/mohitagw15856/pm-claude-skills.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/skills/ai-eval-plan .opencode/skills/ai-eval-plan && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "ai-eval-plan" agent skill from https://github.com/mohitagw15856/pm-claude-skills/tree/main/skills/ai-eval-plan into .opencode/skills/ai-eval-plan/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "ai-eval-plan", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
Facts
- Skill name
ai-eval-plan- GitHub stars
- 1.4k
- Token cost
- ~996 tokens
- SKILL.md length
- 536 words
- Files
- 1
- Skills in repo
- 1,348
- Repo updated
- First seen
- Licence
- MIT
At a glance
Design an evaluation plan for an LLM or AI feature before shipping it.
- Asked how to evaluate a prompt/model/agent
- SKILL.md covers Required Inputs, Output Format, Quality Checks and Anti-Patterns, plus 2 more sections
- Instructions only: no scripts, shell commands, URLs or credentials in SKILL.md
- Set up an eval harness
What it does
AI Eval Plan is an agent skill from mohitagw15856/pm-claude-skills. Design an evaluation plan for an LLM or AI feature before shipping it. Use when asked how to evaluate a prompt/model/agent, set up an eval harness, define quality metrics for an AI feature, or build a regression gate. Produces an eval plan — task definition, datasets, metrics & rubrics, baselines, automated + human evals, a pass bar, and a regression gate.
Its SKILL.md is about 1000 tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.
It sits in Education, covering LLM evaluation and Quizzes and assessments. The repository describes itself as: 1255 professional Agent Skills for Claude, ChatGPT, Gemini, Cursor & Codex — PRDs, postmortems, leases, medical bills, layoffs, go-bags, new countries. Plain markdown, MIT, in… The licence is MIT.
When your agent uses it
- Asked how to evaluate a prompt/model/agent
- Set up an eval harness
- Define quality metrics for an AI feature
- Build a regression gate
Example prompts
- “/ai-eval-plan”
What it can do on your machine
Read from SKILL.md and the folder at commit 1cbf1f0. It shows what the files ask for, not the result of running them.
Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
No scripts in the folder and no shell commands in SKILL.md.
From the folder's file list and the shell code blocks in SKILL.md.
Network
No URLs in SKILL.md.
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Context cost
AI Eval Plan loads about 996 tokens when it runs. Until then it costs about 93 tokens; SKILL.md has 536 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
Safety
Auto-check passed
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
SKILL.md
The full file from mohitagw15856/pm-claude-skills at commit 1cbf1f0, republished under its MIT licence (© mohitagw15856). 536 words, ~996 tokens.
Download SKILL.mdSave it as
.claude/skills/ai-eval-plan/SKILL.md (or your agent's skills folder).- name
- ai-eval-plan
- description
- Design an evaluation plan for an LLM or AI feature before shipping it. Use when asked how to evaluate a prompt/model/agent, set up an eval harness, define quality metrics for an AI feature, or build a regression gate. Produces an eval plan — task definition, datasets, metrics & rubrics, baselines, automated + human evals, a pass bar, and a regression gate.
AI Eval Plan Skill
You can't improve an AI feature you can't measure, and "it looks good in the demo" is not measurement. This skill produces an evaluation plan that turns a fuzzy quality goal into a repeatable, gated test — so a prompt change that quietly makes outputs worse can't ship.
Required Inputs
Ask for these only if they aren't already provided:
- The feature & task — what the model does and what "good output" means to a user.
- Failure modes that matter — what bad looks like (hallucination, wrong format, unsafe, off-tone, too slow).
- Available data — any real examples, logs, or labelled cases; or note there are none yet.
- Who judges quality — automated checks, an LLM judge, human raters, or a mix.
- The decision this gates — ship/no-ship, model selection, or prompt iteration.
Output Format
Eval Plan: [feature]
1. What we're measuring — the task, and a one-line definition of a good vs. bad response.
2. Eval dataset
- Cases: how many, where they come from (real logs > synthetic), and how they're split (smoke set vs. full set).
- Coverage: the slices/scenarios that must be represented (edge cases, adversarial, each major input type).
- Golden answers / references: present or not, and how they were created.
3. Metrics & rubric
- Per-dimension scores — define each dimension (e.g. correctness, grounding, format, safety, tone) on an explicit 1–5 rubric with anchor descriptions, not vibes.
- Automated checks — deterministic assertions first (valid JSON, contains required fields, no PII, latency budget).
- LLM-as-judge — the judge prompt, the rubric it applies, and how you guard against its bias (calibrate against human labels on a sample).
- Human eval — when it's required (safety, subjective quality) and the rater instructions.
4. Baselines — what each candidate is compared against (current prompt, previous model, a plain-prompt control).
5. The bar — the explicit threshold to ship (e.g. "≥4.2 avg correctness, 0 safety failures, p95 < 3s") and what happens if it's missed.
6. Regression gate — how this runs in CI on every change, and the score-drop threshold that blocks a merge.
Show full SKILL.md (209 more words)Show less
Quality Checks
- Each metric has an explicit rubric with anchors — not just a name
- Deterministic/automated checks are used wherever possible before reaching for an LLM judge
- The LLM judge is calibrated against human labels on at least a sample
- The eval set includes adversarial and edge cases, not just happy-path examples
- There is a single, explicit numeric bar for the ship decision
- The plan specifies how it runs as a regression gate, not just a one-time check
Anti-Patterns
- Do not rely on a single overall score — a feature can pass on average while failing every safety case
- Do not trust an LLM judge you haven't calibrated against humans — it has its own blind spots and biases
- Do not eval only on happy-path inputs — the failures live in the edges and the adversarial cases
- Do not let the eval set leak into the prompt/few-shot examples — that's training on the test set
- Do not define the pass bar after seeing the scores — set the threshold before you run, or it means nothing
Based On
LLM evaluation practice — task-grounded rubrics, LLM-as-judge with human calibration, and regression-gated CI evals.
Example Trigger Phrases
- "How to evaluate a prompt/model/agent?"
- "Set up an eval harness."
- "Define quality metrics for an AI feature."
- "Build a regression gate."
© mohitagw15856, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Files
Just SKILL.md in skills/ai-eval-plan of mohitagw15856/pm-claude-skills.
Open the folder on GitHubat commit 1cbf1f0
Compare with similar skills
AI Eval Plan next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| AI Eval Plan this skillmohitagw15856/pm-claude-skills | 1.4k | — | ~996 | Automated safety check: Pass | MIT | |
| Advanced Evaluationaiskillstore/marketplace | 433 | 3 repos | ~4.2k | Automated safety check: Pass | None | |
| Woo AI Smokewoocommerce/woocommerce-ios | 358 | — | ~7.4k | Automated safety check: Notes | GPL-2.0 | |
| Design AI BenchmarkingAperivue/medsci-skills | 333 | — | ~2.4k | Automated safety check: Pass | MIT | |
| Agentic Evaluation Frameworkborghei/Claude-Skills | 891 | — | ~1.9k | Automated safety check: Pass | MIT | |
| Task Creatorbenchflow-ai/benchflow | 356 | — | ~4.5k | Automated safety check: Pass | Apache-2.0 |
Similar skills
Woo AI Smoke
woocommerce/woocommerce-ios
Evaluate WooAIAssistant against a structured scenario suite with hard invariants + LLM-as-judge rubric scoring.
Design AI Benchmarking
Aperivue/medsci-skills
A skill your agent uses when designing a study that benchmarks AI systems against a human-expert panel, before data collection.
Agentic Evaluation Framework
borghei/Claude-Skills
This skill should be used when the user asks to "evaluate LLM output quality", "set up LLM-as-judge", "build an eval rubric", "compare model outputs pairwise", or "measure agent quality".
Task Creator
benchflow-ai/benchflow
SkillsBench task authoring — walk a contributor from idea to submission-ready task following CONTRIBUTING.md and the task-implementation rubric.
- Official
Create Custom Grader
NVIDIA/SkillEvaluator
A skill your agent uses when converting an existing benchmark, rubric, verifier, task YAML/JSON, or domain check into SkillEvaluator BYOG/BYOT custom evaluation.
More from mohitagw15856/pm-claude-skills
All 1,348 skills in this repoCar Tco
mohitagw15856/pm-claude-skills
Compare the total cost of car ownership across buy-new, buy-used, lease, and keep-your-current-car — depreciation, insurance, maintenance ramp, and fuel over a real horizon, not just the monthly…
Cs Health Scorecard
mohitagw15856/pm-claude-skills
Build a customer health scorecard for a specific account. An agent skill from mohitagw15856/pm-claude-skills.
Exit Waterfall
mohitagw15856/pm-claude-skills
Compute who gets what at each exit price from a cap table — liquidation preferences, conversion points, and where the founders' share collapses.
Feature Prioritisation
mohitagw15856/pm-claude-skills
Apply prioritisation frameworks (RICE, MoSCoW, Kano, ICE, Opportunity Scoring) to rank features and backlog items.
Fire Number
mohitagw15856/pm-claude-skills
Compute a financial-independence (FIRE) target and years-to-reach with every assumption labeled as an assumption — plus a sensitivity table instead of a single false-precision answer.
Freelance Rate
mohitagw15856/pm-claude-skills
Derive a freelance day/hourly rate backwards from target income, honest billable utilization, overhead, and the self-employment tax premium — the arithmetic that proves a rate is not salary÷2000.
Related
Categories
Questions about AI Eval Plan
What does AI Eval Plan do?
Design an evaluation plan for an LLM or AI feature before shipping it. AI Eval Plan is an agent skill from mohitagw15856/pm-claude-skills. Design an evaluation plan for an LLM or AI feature before shipping it.
When should I use AI Eval Plan?
AI Eval Plan fits situations like: asked how to evaluate a prompt/model/agent; set up an eval harness; define quality metrics for an AI feature; build a regression gate.
How do I install AI Eval Plan in Claude Code?
Run `npx skills add mohitagw15856/pm-claude-skills --skill ai-eval-plan -a claude-code`. Or copy the skill folder (skills/ai-eval-plan in mohitagw15856/pm-claude-skills) into .claude/skills/ai-eval-plan in your project. Claude Code loads it when a task matches its description.
How do I install AI Eval Plan in Codex?
Run `npx skills add mohitagw15856/pm-claude-skills --skill ai-eval-plan -a codex`. Or copy the skill folder (skills/ai-eval-plan in mohitagw15856/pm-claude-skills) into .agents/skills/ai-eval-plan in your project. Codex loads it when a task matches its description.
Can I use AI Eval Plan in Cursor, Gemini CLI or GitHub Copilot?
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add mohitagw15856/pm-claude-skills --skill ai-eval-plan -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/ai-eval-plan, .gemini/skills/ai-eval-plan, .github/skills/ai-eval-plan and .opencode/skills/ai-eval-plan in your project.
What does AI Eval Plan need to run?
SKILL.md names no scripts, command-line tools or credentials: AI Eval Plan is instructions for the agent only.
Does AI Eval Plan access the network?
SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.
Is AI Eval Plan safe to install?
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
What licence does AI Eval Plan use?
AI Eval Plan is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
How many tokens does AI Eval Plan use?
About 996 tokens (SKILL.md is roughly 4k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
What are the alternatives to AI Eval Plan?
Skills that share tags, products or a category with AI Eval Plan: Advanced Evaluation (aiskillstore/marketplace, 433 stars), Woo AI Smoke (woocommerce/woocommerce-ios, 358 stars), Design AI Benchmarking (Aperivue/medsci-skills, 333 stars) and Agentic Evaluation Framework (borghei/Claude-Skills, 891 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
Who maintains AI Eval Plan?
mohitagw15856 (a GitHub user) maintains it in mohitagw15856/pm-claude-skills, which has 1,434 GitHub stars. The repository holds 1,348 skills in this directory. The repository was last updated on October 9, 2026.
Source: mohitagw15856/pm-claude-skills on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.