Agent skill

Skill Forge Eval

by AgriciDaniel in AgriciDaniel/skill-forge

Run evaluation pipelines on Claude Code skills to test triggering accuracy, workflow correctness, and output quality.

Install Skill Forge Eval

skills CLI

$ npx skills add AgriciDaniel/skill-forge --skill skill-forge-eval -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI

$ gh skill install AgriciDaniel/skill-forge skill-forge-eval --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy

$ git clone --depth 1 https://github.com/AgriciDaniel/skill-forge.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/skill-forge-eval .claude/skills/skill-forge-eval && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Paste into Claude Code

Install the "skill-forge-eval" agent skill from https://github.com/AgriciDaniel/skill-forge/tree/main/skills/skill-forge-eval into .claude/skills/skill-forge-eval/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "skill-forge-eval", then confirm the skill loads.

Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

$skill-installer

$skill-installer install https://github.com/AgriciDaniel/skill-forge/tree/main/skills/skill-forge-eval

Type this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.

skills CLI

$ npx skills add AgriciDaniel/skill-forge --skill skill-forge-eval -a codex

Project install goes to .agents/skills/; add -g for ~/.codex/skills/.

GitHub CLI

$ gh skill install AgriciDaniel/skill-forge skill-forge-eval --agent codex

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/AgriciDaniel/skill-forge.git skills-src && mkdir -p .agents/skills && cp -r skills-src/skills/skill-forge-eval .agents/skills/skill-forge-eval && rm -rf skills-src

Use ~/.agents/skills/ instead of .agents/skills for a personal install.

Codex skills documentation · loads skills from .agents/skills/

Paste into Codex

Install the "skill-forge-eval" agent skill from https://github.com/AgriciDaniel/skill-forge/tree/main/skills/skill-forge-eval into .agents/skills/skill-forge-eval/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "skill-forge-eval", then confirm the skill loads.

Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

skills CLI

$ npx skills add AgriciDaniel/skill-forge --skill skill-forge-eval -a cursor

Project install goes to .agents/skills/; add -g for ~/.cursor/skills/.

GitHub CLI

$ gh skill install AgriciDaniel/skill-forge skill-forge-eval --agent cursor

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/AgriciDaniel/skill-forge.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/skills/skill-forge-eval .cursor/skills/skill-forge-eval && rm -rf skills-src

Use ~/.cursor/skills/ instead of .cursor/skills for a personal install.

Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/

Paste into Cursor

Install the "skill-forge-eval" agent skill from https://github.com/AgriciDaniel/skill-forge/tree/main/skills/skill-forge-eval into .cursor/skills/skill-forge-eval/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "skill-forge-eval", then confirm the skill loads.

Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

gemini skills install

$ gemini skills install https://github.com/AgriciDaniel/skill-forge.git --path skills/skill-forge-eval

--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.

skills CLI

$ npx skills add AgriciDaniel/skill-forge --skill skill-forge-eval -a gemini-cli

Project install goes to .agents/skills/; add -g for ~/.gemini/skills/.

GitHub CLI

$ gh skill install AgriciDaniel/skill-forge skill-forge-eval --agent gemini-cli

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/AgriciDaniel/skill-forge.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/skills/skill-forge-eval .gemini/skills/skill-forge-eval && rm -rf skills-src

Use ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.

Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/

Paste into Gemini CLI

Install the "skill-forge-eval" agent skill from https://github.com/AgriciDaniel/skill-forge/tree/main/skills/skill-forge-eval into .gemini/skills/skill-forge-eval/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "skill-forge-eval", then confirm the skill loads.

Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

GitHub CLI

$ gh skill install AgriciDaniel/skill-forge skill-forge-eval

Installs for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).

skills CLI

$ npx skills add AgriciDaniel/skill-forge --skill skill-forge-eval -a github-copilot

Project install goes to .agents/skills/; add -g for ~/.copilot/skills/.

Manual copy

$ git clone --depth 1 https://github.com/AgriciDaniel/skill-forge.git skills-src && mkdir -p .github/skills && cp -r skills-src/skills/skill-forge-eval .github/skills/skill-forge-eval && rm -rf skills-src

Use ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.

GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/

Paste into GitHub Copilot

Install the "skill-forge-eval" agent skill from https://github.com/AgriciDaniel/skill-forge/tree/main/skills/skill-forge-eval into .github/skills/skill-forge-eval/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "skill-forge-eval", then confirm the skill loads.

GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

skills CLI

$ npx skills add AgriciDaniel/skill-forge --skill skill-forge-eval -a opencode

OpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.

GitHub CLI

$ gh skill install AgriciDaniel/skill-forge skill-forge-eval --agent opencode

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/AgriciDaniel/skill-forge.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/skills/skill-forge-eval .opencode/skills/skill-forge-eval && rm -rf skills-src

Use ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.

OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/

Paste into OpenCode

Install the "skill-forge-eval" agent skill from https://github.com/AgriciDaniel/skill-forge/tree/main/skills/skill-forge-eval into .opencode/skills/skill-forge-eval/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "skill-forge-eval", then confirm the skill loads.

OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

Facts

Skill name: skill-forge-eval
GitHub stars: 177
Token cost: ~1.7k tokens
SKILL.md length: 494 words
Files: 1
Skills in repo: 9
Repo updated: Apr 10, 2026
First seen: Oct 7, 2026
Licence: MIT

At a glance

Run evaluation pipelines on Claude Code skills to test triggering accuracy, workflow correctness, and output quality.

Works in 7 steps: Define Eval Set → Set Up Workspace → Execute Eval Runs → …
User says eval skill
SKILL.md covers Process, Advanced: Blind Comparison, Error Handling and Quality Gates
Calls python

What it does

Skill Forge Eval is an agent skill from AgriciDaniel/skill-forge. Run evaluation pipelines on Claude Code skills to test triggering accuracy, workflow correctness, and output quality. Spawns executor, grader, comparator, and analyzer sub-agents for parallel evaluation. Generates evalmetadata.json, grading.json, and feedback reports. Use when user says "eval skill", "test skill", "run evals", "evaluate skill", "skill evals", "test skill quality", "run skill tests", or "skill evaluation".

Its SKILL.md is about 1.7k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.

It sits in Agent Workflows, covering Agent evaluation and testing, LLM evaluation and Subagents. The repository describes itself as: Ultimate Claude Code skill creator — design, scaffold, build, review, evolve, and publish production-grade AI agent skills. The licence is MIT.

When your agent uses it

User says eval skill
Test skill quality
Run skill tests
Skill evaluation

Example prompts

“eval skill”
“test skill”
“run evals”
“/skill-forge-eval”

Requirements

Python 3

Workflow steps

7 steps, taken from the step headings in SKILL.md.

Define Eval Set
Set Up Workspace
Execute Eval Runs
Grade Results
Aggregate and Analyze
Present Results
Collect Feedback

What it can do on your machine

Read from SKILL.md and the folder at commit 2872ee9. It shows what the files ask for, not the result of running them.

Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
Shell commands in SKILL.md call:
- python
From the folder's file list and the shell code blocks in SKILL.md.
Network
No URLs in SKILL.md.
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Skill Forge Eval loads about 1.7k tokens when it runs. Until then it costs about 111 tokens; SKILL.md has 494 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it: ~111
When it runs · the whole SKILL.md, loaded when a task matches: ~1.7k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from AgriciDaniel/skill-forge at commit 2872ee9, republished under its MIT licence (© AgriciDaniel). 494 words, ~1,703 tokens.

Download SKILL.mdSave it as .claude/skills/skill-forge-eval/SKILL.md (or your agent's skills folder).

name: skill-forge-eval
description: Run evaluation pipelines on Claude Code skills to test triggering accuracy, workflow correctness, and output quality. Spawns executor, grader, comparator, and analyzer sub-agents for parallel evaluation. Generates eval_metadata.json, grading.json, and feedback reports. Use when user says "eval skill", "test skill", "run evals", "evaluate skill", "skill evals", "test skill quality", "run skill tests", or "skill evaluation".

Skill Evaluation Pipeline

Run structured evaluations against Claude Code skills to verify triggering, correctness, and quality using a multi-agent pipeline.

Process

Step 1: Define Eval Set

Accept eval definitions from:

Path to eval set JSON: evals/evals.json or user-specified file
Inline prompts: User provides eval queries directly
Auto-generated: Generate from skill description (see Step 1b)

Eval set JSON schema:

json

{
  "skill_name": "my-skill",
  "skill_path": "./my-skill",
  "evals": [
    {
      "eval_id": 0,
      "eval_name": "descriptive-name",
      "prompt": "The user's task prompt",
      "input_files": [],
      "assertions": [
        {
          "name": "output-has-score",
          "check": "Output contains a numeric score between 0-100",
          "weight": 1.0
        }
      ],
      "should_trigger": true
    }
  ]
}

Step 1b: Auto-Generate Eval Set

If no eval set exists, generate one:

Read the skill's SKILL.md description and instructions
Run python scripts/generate_eval_set.py <skill-path> to produce a starter set
Present the generated set to the user for review and editing
User approves or modifies before proceeding

Step 2: Set Up Workspace

Create the eval workspace outside the skill directory to avoid confusing eval artifacts with skill files. Use a sibling directory or a dedicated location:

eval-workspace/
  iteration-1/
    eval-0/
      eval_metadata.json        # Assertions and config for this eval
      with_skill/
        outputs/                # Skill execution outputs
        timing.json             # Token count + duration
        grading.json            # Assertion results + evidence
      baseline/
        outputs/
        timing.json
        grading.json
    eval-1/
      eval_metadata.json
      with_skill/
        outputs/
        timing.json
        grading.json
      baseline/
        outputs/
        timing.json
        grading.json
    benchmark.json              # Aggregated metrics
    benchmark.md                # Human-readable report

For each eval directory, create eval_metadata.json from the eval set entry:

json

{
  "eval_id": 0,
  "eval_name": "descriptive-name",
  "prompt": "The user's task prompt",
  "assertions": [...],
  "should_trigger": true
}

Step 3: Execute Eval Runs

For each eval in the set, spawn two parallel runs:

With-skill run (delegate to agents/skill-forge-executor.md):

Execute this task:
- Skill path: <path-to-skill>
- Task: <eval prompt>
- Input files: <eval files if any, or "none">
- Save outputs to: <workspace>/iteration-<N>/eval-<ID>/with_skill/outputs/
- Outputs to save: <what the assertions check>

Baseline run (delegate to agents/skill-forge-executor.md):

For new skills: run without the skill loaded
For improved skills: run with the previous version (snapshot it first)

Save timing data to timing.json in each run directory:

json

{
  "total_tokens": 84852,
  "duration_ms": 23332,
  "total_duration_seconds": 23.3
}

Step 4: Grade Results

Delegate to agents/skill-forge-grader.md for each completed run:

Grade against assertions defined in eval_metadata.json
Save results to grading.json per run:

json

{
  "eval_id": 0,
  "run_type": "with_skill",
  "assertions": [
    {
      "name": "output-has-score",
      "passed": true,
      "evidence": "Found score: 87/100 on line 14"
    }
  ],
  "pass_rate": 1.0
}

Step 5: Aggregate and Analyze

Run python scripts/aggregate_benchmark.py <workspace>/iteration-<N> --skill-name <name>
This produces benchmark.json and benchmark.md with:
- Pass rate per eval (with_skill vs baseline)
- Average time and token usage
- Improvement ratio (with_skill / baseline)
Delegate to agents/skill-forge-analyzer.md to:
- Surface patterns that aggregate stats might hide
- Identify consistently failing assertion types
- Flag regressions from previous iterations

Step 6: Present Results

Generate a summary report:

markdown

# Eval Report: [skill-name] — Iteration [N]

## Overall
| Metric | With Skill | Baseline | Delta |
|--------|-----------|----------|-------|
| Pass Rate | X% | Y% | +Z% |
| Avg Time | Xs | Ys | -Zs |
| Avg Tokens | X | Y | -Z |

## Per-Eval Results
| Eval | With Skill | Baseline | Status |
|------|-----------|----------|--------|
| eval-0 | PASS | FAIL | Improved |
| eval-1 | PASS | PASS | Maintained |

## Patterns & Insights
[From analyzer agent]

## Recommendations
[Specific improvements based on failures]

Show full SKILL.md (204 more words)Show less

Step 7: Collect Feedback

Save user feedback to feedback.json:

json

{
  "reviews": [
    {
      "run_id": "eval-0-with_skill",
      "feedback": "the chart is missing axis labels",
      "timestamp": "2026-03-06T12:00:00Z"
    }
  ],
  "status": "complete"
}

Pass feedback to /skill-forge evolve for the next iteration.

For rigorous A/B testing between skill versions:

Delegate to agents/skill-forge-comparator.md
Pass two directories: eval-<ID>/with_skill/outputs/ and eval-<ID>/baseline/outputs/
Comparator assigns random labels (Version A / Version B) so it cannot know which is new
Rates each output on assertion criteria from eval_metadata.json
Returns preference scores without knowing which is "new" vs "old"

Error Handling

Executor timeout: If a run exceeds 5 minutes, terminate and mark as "timed_out": true in timing.json
Executor failure: If a run crashes, save the error to error.txt in the run directory and continue with remaining evals
Grading failure: If grading cannot determine pass/fail, mark assertion as "passed": null with evidence explaining why
Missing files: If timing.json or grading.json is missing after a run, flag the eval as incomplete in the report
Partial completion: Always aggregate and report whatever results are available — do not block on one failed eval

Quality Gates

Before marking an eval run as complete:

All evals executed (with_skill + baseline)
Timing data captured for every run
All assertions graded with evidence
Benchmark aggregated with pass rate, time, tokens
Analyzer patterns documented
Results presented to user

Files

Just SKILL.md in skills/skill-forge-eval of AgriciDaniel/skill-forge.

Open the folder on GitHubat commit 2872ee9

Compare with similar skills

Skill Forge Eval next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Skill Forge Eval compared with similar skills
Skill	Stars	Used in	Tokens	Auto-check	Licence	Repo updated
Skill Forge Eval this skillAgriciDaniel/skill-forge	177	—	~1.7k	Automated safety check: Pass	MIT	6 mo ago
Kayba Pipelinekayba-ai/agentic-context-engine	2.6k	—	~1.4k	Automated safety check: Pass	Apache-2.0	13 days ago
Harness Evaltech-leads-club/agent-skills	7k	—	~3.9k	Automated safety check: Pass	CC-BY-4.0	17 days ago
Waza Skill Evaluatormicrosoft/waza	1.4k	—	~2k	Automated safety check: Pass	MIT	yesterday
Eval Answermalloydata/publisher	116	—	~4.3k	Automated safety check: Pass	MIT	today
Waza Interactivemicrosoft/waza	1.4k	—	~1.3k	Automated safety check: Pass	MIT	yesterday

Similar skills

Kayba Pipeline
kayba-ai/agentic-context-engine
End-to-end agent evaluation and improvement pipeline. An agent skill from kayba-ai/agentic-context-engine.
2.6k GitHub stars~1.4k tokensUpdated 13 days ago
Agent WorkflowsAuto-check passed
Harness Eval
tech-leads-club/agent-skills
Evaluates a repository's agent harness (AGENTS.md, rules, skills) for broken paths, redundant instructions and usefulness, and stops at reports.
7k GitHub stars~3.9k tokensUpdated 17 days ago
Agent WorkflowsAuto-check passed
Waza Skill Evaluator
microsoft/waza
Official
Evaluates agent skills with a Go CLI that runs YAML-defined benchmarks, compares runs and scores the quality of SKILL.md frontmatter.
1.4k GitHub stars~2k tokensUpdated yesterday
Agent WorkflowsAuto-check passed
Eval Answer
malloydata/publisher
Score one analytical answer against a verified golden, and score which of the entities the golden depends on retrieval delivered to the answerer.
116 GitHub stars~4.3k tokensUpdated today
Agent WorkflowsAuto-check passed
Waza Interactive
microsoft/waza
Official
Walks you through creating, running and reading waza evals for an agent skill, then proposes concrete fixes when tasks fail or the score is low.
1.4k GitHub stars~1.3k tokensUpdated yesterday
Agent WorkflowsAuto-check passed
Skill Creator
ZS520L/HanakoPro
Create new skills, modify and improve existing skills, and measure skill performance.
102 GitHub stars~7.4k tokensUpdated 4 mo ago
Agent WorkflowsAuto-check passed

Questions about Skill Forge Eval

What does Skill Forge Eval do?

Run evaluation pipelines on Claude Code skills to test triggering accuracy, workflow correctness, and output quality. Skill Forge Eval is an agent skill from AgriciDaniel/skill-forge. Run evaluation pipelines on Claude Code skills to test triggering accuracy, workflow correctness, and output quality.

When should I use Skill Forge Eval?

Skill Forge Eval fits situations like: user says eval skill; test skill quality; run skill tests; skill evaluation.

How do I install Skill Forge Eval in Claude Code?

Run `npx skills add AgriciDaniel/skill-forge --skill skill-forge-eval -a claude-code`. Or copy the skill folder (skills/skill-forge-eval in AgriciDaniel/skill-forge) into .claude/skills/skill-forge-eval in your project. Claude Code loads it when a task matches its description.

How do I install Skill Forge Eval in Codex?

Run `npx skills add AgriciDaniel/skill-forge --skill skill-forge-eval -a codex`. Or copy the skill folder (skills/skill-forge-eval in AgriciDaniel/skill-forge) into .agents/skills/skill-forge-eval in your project. Codex loads it when a task matches its description.

Can I use Skill Forge Eval in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add AgriciDaniel/skill-forge --skill skill-forge-eval -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/skill-forge-eval, .gemini/skills/skill-forge-eval, .github/skills/skill-forge-eval and .opencode/skills/skill-forge-eval in your project.

What does Skill Forge Eval need to run?

Going by SKILL.md and its folder, Skill Forge Eval needs the command-line tools its instructions call (python). Our summary lists: Python 3.

Does Skill Forge Eval access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Skill Forge Eval safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Skill Forge Eval use?

Skill Forge Eval is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Skill Forge Eval use?

About 1.7k tokens (SKILL.md is roughly 6.8k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Skill Forge Eval?

Skills that share tags, products or a category with Skill Forge Eval: Kayba Pipeline (kayba-ai/agentic-context-engine, 2.6k stars), Harness Eval (tech-leads-club/agent-skills, 7k stars), Waza Skill Evaluator (microsoft/waza, 1.4k stars) and Eval Answer (malloydata/publisher, 116 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Skill Forge Eval?

AgriciDaniel (a GitHub user) maintains it in AgriciDaniel/skill-forge, which has 177 GitHub stars. The repository holds 9 skills in this directory. The repository was last updated on April 10, 2026.

Source: AgriciDaniel/skill-forge on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.

Install Skill Forge Eval

Facts

At a glance

What it does

When your agent uses it

Example prompts

Requirements

Workflow steps

What it can do on your machine

Tool permissions

Runs code

Network

Credentials

Context cost

Safety

SKILL.md

Skill Evaluation Pipeline

Process

Step 1: Define Eval Set

Step 1b: Auto-Generate Eval Set

Step 2: Set Up Workspace

Step 3: Execute Eval Runs

Step 4: Grade Results

Step 5: Aggregate and Analyze

Step 6: Present Results

Step 7: Collect Feedback

Advanced: Blind Comparison

Error Handling

Quality Gates

Files

Compare with similar skills

Similar skills

More from AgriciDaniel/skill-forge

Related

Questions about Skill Forge Eval