Eval Guide
microsoft/eval-guide
Eval enablement accelerator — help customers think through "what does good look like" for their AI agent, then generate a structured eval plan and test cases they can use immediately.
[Beta] Creates permanent eval cases from promoted learnings and runs regression checks against them.
Install Eval Creator
$ npx skills add pskoett/pskoett-ai-skills --skill eval-creator -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install pskoett/pskoett-ai-skills eval-creator --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/pskoett/pskoett-ai-skills.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/eval-creator .claude/skills/eval-creator && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "eval-creator" agent skill from https://github.com/pskoett/pskoett-ai-skills/tree/main/skills/eval-creator into .claude/skills/eval-creator/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-creator", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/pskoett/pskoett-ai-skills/tree/main/skills/eval-creatorType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add pskoett/pskoett-ai-skills --skill eval-creator -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install pskoett/pskoett-ai-skills eval-creator --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/pskoett/pskoett-ai-skills.git skills-src && mkdir -p .agents/skills && cp -r skills-src/skills/eval-creator .agents/skills/eval-creator && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "eval-creator" agent skill from https://github.com/pskoett/pskoett-ai-skills/tree/main/skills/eval-creator into .agents/skills/eval-creator/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-creator", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add pskoett/pskoett-ai-skills --skill eval-creator -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install pskoett/pskoett-ai-skills eval-creator --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/pskoett/pskoett-ai-skills.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/skills/eval-creator .cursor/skills/eval-creator && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "eval-creator" agent skill from https://github.com/pskoett/pskoett-ai-skills/tree/main/skills/eval-creator into .cursor/skills/eval-creator/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-creator", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/pskoett/pskoett-ai-skills.git --path skills/eval-creator--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add pskoett/pskoett-ai-skills --skill eval-creator -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install pskoett/pskoett-ai-skills eval-creator --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/pskoett/pskoett-ai-skills.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/skills/eval-creator .gemini/skills/eval-creator && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "eval-creator" agent skill from https://github.com/pskoett/pskoett-ai-skills/tree/main/skills/eval-creator into .gemini/skills/eval-creator/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-creator", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install pskoett/pskoett-ai-skills eval-creatorInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add pskoett/pskoett-ai-skills --skill eval-creator -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/pskoett/pskoett-ai-skills.git skills-src && mkdir -p .github/skills && cp -r skills-src/skills/eval-creator .github/skills/eval-creator && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "eval-creator" agent skill from https://github.com/pskoett/pskoett-ai-skills/tree/main/skills/eval-creator into .github/skills/eval-creator/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-creator", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add pskoett/pskoett-ai-skills --skill eval-creator -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install pskoett/pskoett-ai-skills eval-creator --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/pskoett/pskoett-ai-skills.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/skills/eval-creator .opencode/skills/eval-creator && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "eval-creator" agent skill from https://github.com/pskoett/pskoett-ai-skills/tree/main/skills/eval-creator into .opencode/skills/eval-creator/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-creator", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
Facts
- Skill name
eval-creator- GitHub stars
- 311
- Token cost
- ~2.6k tokens
- SKILL.md length
- 887 words
- Files
- 1
- Skills in repo
- 24
- Repo updated
- First seen
- Licence
- None found
At a glance
[Beta] Creates permanent eval cases from promoted learnings and runs regression checks against them.
- Works in 6 steps: Check precondition — if not met, mark as… → Execute verification method → Compare result to expected → …
- A learning is promoted and has a clear pass/fail condition
- SKILL.md covers When to Use, Eval Directory Structure, Creating an Eval Case and Running Evals, plus 6 more sections
- Calls python
What it does
Eval Creator is an agent skill from pskoett/pskoett-ai-skills. [Beta] Creates permanent eval cases from promoted learnings and runs regression checks against them. Turns failures into test cases that prevent silent regression. This is the outer loop's regress-test step. Use when a learning is promoted and has a clear pass/fail condition, or on cadence to verify promoted rules still hold.
Its SKILL.md is about 2.6k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.
It sits in Agent Workflows, covering Test generation and LLM evaluation.
When your agent uses it
- A learning is promoted and has a clear pass/fail condition
- On cadence to verify promoted rules still hold
Example prompts
- “/eval-creator”
Requirements
- Python 3
Workflow steps
6 steps, taken from the first numbered list in SKILL.md.
- Check precondition — if not met, mark as skip
- Execute verification method
- Compare result to expected
- Check evidence strength — a presence/structural method cannot satisfy a
- Update last-run and last-result in the eval case file
- Update EVAL_INDEX.md with the result
What it can do on your machine
Read from SKILL.md and the folder at commit 5a836dc. It shows what the files ask for, not the result of running them.
Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
Shell commands in SKILL.md call:
python
From the folder's file list and the shell code blocks in SKILL.md.
Network
No URLs in SKILL.md.
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Context cost
Eval Creator loads about 2.6k tokens when it runs. Until then it costs about 85 tokens; SKILL.md has 887 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
Safety
Auto-check passed
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
SKILL.md
Without a licence we can't republish the file, so here is its outline and opening line. It has 887 words (~2,598 tokens).
“Turns promoted learnings into permanent eval cases. Runs regression checks to verify promoted rules hold. This is the outer loop's regress-test step.”
- name
- eval-creator
Files
Just SKILL.md in skills/eval-creator of pskoett/pskoett-ai-skills.
Open the folder on GitHubat commit 5a836dc
Compare with similar skills
Eval Creator next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Eval Creator this skillpskoett/pskoett-ai-skills | 311 | — | ~2.6k | Automated safety check: Pass | None | |
| Eval Guidemicrosoft/eval-guide | 138 | — | ~22k | Automated safety check: Warn | MIT | |
| Synthetic Eval Data Generatorai-evals-course/evals-skills | 1.5k | — | ~1.4k | Automated safety check: Pass | Apache-2.0 | |
| Axiom Eval Writeropenclaw/clawhub | 9.5k | — | ~4.1k | Automated safety check: Warn | MIT | |
| SDK AI Bot Run EvaluationAzure/azure-sdk-tools | 134 | — | ~1.1k | Automated safety check: Notes | MIT | |
| Testinginbrainfun/inbrain | 142 | 1 repos | ~2k | Automated safety check: Pass | Custom licence |
Similar skills
- Official
Synthetic Eval Data Generator
ai-evals-course/evals-skills
Builds diverse synthetic test inputs for LLM pipeline evaluation by defining failure-focused dimensions, drafting tuples with you and turning them into realistic queries.
Axiom Eval Writer
openclaw/clawhub
Scaffolds evaluation suites for the Axiom AI SDK: eval files, scorers, flag schemas and axiom.config.ts, generated from plain descriptions of an AI capability.
- Official
SDK AI Bot Run Evaluation
Azure/azure-sdk-tools
Run Azure SDK QA bot evaluations on curated datasets locally, including a single test case.
Testing
inbrainfun/inbrain
Skill validation framework PLUS daily test-suite health and regression intelligence.
Quality Flywheel
GoogleCloudPlatform/vertex-ai-samples
Evaluate and improve GenAI models and agents using the Google GenAI Evaluation SDK.
More from pskoett/pskoett-ai-skills
All 24 skills in this repoSelf Improvement
pskoett/pskoett-ai-skills
Captures learnings, errors, corrections, and feature requests to enable continuous improvement.
Self Improvement
pskoett/pskoett-ai-skills
Captures learnings, errors, corrections, and feature requests to enable continuous improvement.
Self Healing
pskoett/pskoett-ai-skills
Active runtime recovery for coding agents: when something breaks mid-task, diagnose the root cause, write a fix, VERIFY by re-running the broken thing, then file a HEAL- entry to .learnings/HEALS.md…
Skill Tester
pskoett/pskoett-ai-skills
Validates all interactive skills in this repo against the Agent Skills spec, project conventions, and structural requirements.
Skill Tester CI
pskoett/pskoett-ai-skills
Validates all CI skills in this repo. An agent skill from pskoett/pskoett-ai-skills.
Context Decay
pskoett/pskoett-ai-skills
Revalidate persistent agent knowledge by classifying why it can become stale and how quickly it changes, then retain, revise, externalize, or retire it.
Related
Questions about Eval Creator
What does Eval Creator do?
[Beta] Creates permanent eval cases from promoted learnings and runs regression checks against them. Eval Creator is an agent skill from pskoett/pskoett-ai-skills. [Beta] Creates permanent eval cases from promoted learnings and runs regression checks against them.
When should I use Eval Creator?
Eval Creator fits situations like: A learning is promoted and has a clear pass/fail condition; on cadence to verify promoted rules still hold.
How do I install Eval Creator in Claude Code?
Run `npx skills add pskoett/pskoett-ai-skills --skill eval-creator -a claude-code`. Or copy the skill folder (skills/eval-creator in pskoett/pskoett-ai-skills) into .claude/skills/eval-creator in your project. Claude Code loads it when a task matches its description.
How do I install Eval Creator in Codex?
Run `npx skills add pskoett/pskoett-ai-skills --skill eval-creator -a codex`. Or copy the skill folder (skills/eval-creator in pskoett/pskoett-ai-skills) into .agents/skills/eval-creator in your project. Codex loads it when a task matches its description.
Can I use Eval Creator in Cursor, Gemini CLI or GitHub Copilot?
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add pskoett/pskoett-ai-skills --skill eval-creator -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/eval-creator, .gemini/skills/eval-creator, .github/skills/eval-creator and .opencode/skills/eval-creator in your project.
What does Eval Creator need to run?
Going by SKILL.md and its folder, Eval Creator needs the command-line tools its instructions call (python). Our summary lists: Python 3.
Does Eval Creator access the network?
SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.
Is Eval Creator safe to install?
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
What licence does Eval Creator use?
No licence was found for Eval Creator or its repository. Without one, default copyright applies: ask the author before reusing or redistributing it.
How many tokens does Eval Creator use?
About 2.6k tokens (SKILL.md is roughly 10k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
What are the alternatives to Eval Creator?
Skills that share tags, products or a category with Eval Creator: Eval Guide (microsoft/eval-guide, 138 stars), Synthetic Eval Data Generator (ai-evals-course/evals-skills, 1.5k stars), Axiom Eval Writer (openclaw/clawhub, 9.5k stars) and SDK AI Bot Run Evaluation (Azure/azure-sdk-tools, 134 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
Who maintains Eval Creator?
pskoett (a GitHub user) maintains it in pskoett/pskoett-ai-skills, which has 311 GitHub stars. The repository holds 24 skills in this directory. The repository was last updated on October 5, 2026.
Source: pskoett/pskoett-ai-skills on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.