Agent skill

Skill Forge Benchmark

by AgriciDaniel in AgriciDaniel/skill-forge

Benchmark Claude Code skill performance with variance analysis, tracking pass rate, execution time, and token usage across iterations.

MITAuto-check passedAI & LLM Engineering

Install Skill Forge Benchmark

skills CLI

$ npx skills add AgriciDaniel/skill-forge --skill skill-forge-benchmark -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI

$ gh skill install AgriciDaniel/skill-forge skill-forge-benchmark --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy

$ git clone --depth 1 https://github.com/AgriciDaniel/skill-forge.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/skill-forge-benchmark .claude/skills/skill-forge-benchmark && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Paste into Claude Code

Install the "skill-forge-benchmark" agent skill from https://github.com/AgriciDaniel/skill-forge/tree/main/skills/skill-forge-benchmark into .claude/skills/skill-forge-benchmark/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "skill-forge-benchmark", then confirm the skill loads.

Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

$skill-installer

$skill-installer install https://github.com/AgriciDaniel/skill-forge/tree/main/skills/skill-forge-benchmark

Type this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.

skills CLI

$ npx skills add AgriciDaniel/skill-forge --skill skill-forge-benchmark -a codex

Project install goes to .agents/skills/; add -g for ~/.codex/skills/.

GitHub CLI

$ gh skill install AgriciDaniel/skill-forge skill-forge-benchmark --agent codex

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/AgriciDaniel/skill-forge.git skills-src && mkdir -p .agents/skills && cp -r skills-src/skills/skill-forge-benchmark .agents/skills/skill-forge-benchmark && rm -rf skills-src

Use ~/.agents/skills/ instead of .agents/skills for a personal install.

Codex skills documentation · loads skills from .agents/skills/

Paste into Codex

Install the "skill-forge-benchmark" agent skill from https://github.com/AgriciDaniel/skill-forge/tree/main/skills/skill-forge-benchmark into .agents/skills/skill-forge-benchmark/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "skill-forge-benchmark", then confirm the skill loads.

Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

skills CLI

$ npx skills add AgriciDaniel/skill-forge --skill skill-forge-benchmark -a cursor

Project install goes to .agents/skills/; add -g for ~/.cursor/skills/.

GitHub CLI

$ gh skill install AgriciDaniel/skill-forge skill-forge-benchmark --agent cursor

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/AgriciDaniel/skill-forge.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/skills/skill-forge-benchmark .cursor/skills/skill-forge-benchmark && rm -rf skills-src

Use ~/.cursor/skills/ instead of .cursor/skills for a personal install.

Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/

Paste into Cursor

Install the "skill-forge-benchmark" agent skill from https://github.com/AgriciDaniel/skill-forge/tree/main/skills/skill-forge-benchmark into .cursor/skills/skill-forge-benchmark/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "skill-forge-benchmark", then confirm the skill loads.

Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

gemini skills install

$ gemini skills install https://github.com/AgriciDaniel/skill-forge.git --path skills/skill-forge-benchmark

--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.

skills CLI

$ npx skills add AgriciDaniel/skill-forge --skill skill-forge-benchmark -a gemini-cli

Project install goes to .agents/skills/; add -g for ~/.gemini/skills/.

GitHub CLI

$ gh skill install AgriciDaniel/skill-forge skill-forge-benchmark --agent gemini-cli

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/AgriciDaniel/skill-forge.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/skills/skill-forge-benchmark .gemini/skills/skill-forge-benchmark && rm -rf skills-src

Use ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.

Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/

Paste into Gemini CLI

Install the "skill-forge-benchmark" agent skill from https://github.com/AgriciDaniel/skill-forge/tree/main/skills/skill-forge-benchmark into .gemini/skills/skill-forge-benchmark/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "skill-forge-benchmark", then confirm the skill loads.

Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

GitHub CLI

$ gh skill install AgriciDaniel/skill-forge skill-forge-benchmark

Installs for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).

skills CLI

$ npx skills add AgriciDaniel/skill-forge --skill skill-forge-benchmark -a github-copilot

Project install goes to .agents/skills/; add -g for ~/.copilot/skills/.

Manual copy

$ git clone --depth 1 https://github.com/AgriciDaniel/skill-forge.git skills-src && mkdir -p .github/skills && cp -r skills-src/skills/skill-forge-benchmark .github/skills/skill-forge-benchmark && rm -rf skills-src

Use ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.

GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/

Paste into GitHub Copilot

Install the "skill-forge-benchmark" agent skill from https://github.com/AgriciDaniel/skill-forge/tree/main/skills/skill-forge-benchmark into .github/skills/skill-forge-benchmark/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "skill-forge-benchmark", then confirm the skill loads.

GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

skills CLI

$ npx skills add AgriciDaniel/skill-forge --skill skill-forge-benchmark -a opencode

OpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.

GitHub CLI

$ gh skill install AgriciDaniel/skill-forge skill-forge-benchmark --agent opencode

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/AgriciDaniel/skill-forge.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/skills/skill-forge-benchmark .opencode/skills/skill-forge-benchmark && rm -rf skills-src

Use ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.

OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/

Paste into OpenCode

Install the "skill-forge-benchmark" agent skill from https://github.com/AgriciDaniel/skill-forge/tree/main/skills/skill-forge-benchmark into .opencode/skills/skill-forge-benchmark/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "skill-forge-benchmark", then confirm the skill loads.

OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

Facts

Skill name: skill-forge-benchmark
GitHub stars: 177
Token cost: ~1.4k tokens
SKILL.md length: 314 words
Files: 1
Skills in repo: 9
Repo updated: Apr 10, 2026
First seen: Oct 7, 2026
Licence: MIT

At a glance

Benchmark Claude Code skill performance with variance analysis, tracking pass rate, execution time, and token usage across iterations.

Works in 6 steps: Define Benchmark Configuration → Execute Benchmark Runs → Aggregate Results → …
User says benchmark skill
SKILL.md covers Process, Error Handling and Integration with Other…
Calls python

What it does

Skill Forge Benchmark is an agent skill from AgriciDaniel/skill-forge. Benchmark Claude Code skill performance with variance analysis, tracking pass rate, execution time, and token usage across iterations. Runs multiple trials per eval for statistical reliability, aggregates results into benchmark.json, and generates comparison reports between skill versions. Use when user says "benchmark skill", "measure skill performance", "skill metrics", "compare skill versions", "skill performance", "track skill improvement", "skill regression test", or "skill A/B test".

Its SKILL.md is about 1.4k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.

It sits in AI & LLM Engineering, covering LLM evaluation and A/B testing. The repository describes itself as: Ultimate Claude Code skill creator — design, scaffold, build, review, evolve, and publish production-grade AI agent skills. The licence is MIT.

When your agent uses it

User says benchmark skill
Measure skill performance
Compare skill versions
Skill performance

Example prompts

“benchmark skill”
“measure skill performance”
“skill metrics”
“/skill-forge-benchmark”

Requirements

Python 3

Workflow steps

6 steps, taken from the step headings in SKILL.md.

Define Benchmark Configuration
Execute Benchmark Runs
Aggregate Results
Compare with Previous Iterations
Generate Benchmark Report
Threshold Gating

What it can do on your machine

Read from SKILL.md and the folder at commit 2872ee9. It shows what the files ask for, not the result of running them.

Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
Shell commands in SKILL.md call:
- python
From the folder's file list and the shell code blocks in SKILL.md.
Network
No URLs in SKILL.md.
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Skill Forge Benchmark loads about 1.4k tokens when it runs. Until then it costs about 129 tokens; SKILL.md has 314 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it: ~129
When it runs · the whole SKILL.md, loaded when a task matches: ~1.4k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from AgriciDaniel/skill-forge at commit 2872ee9, republished under its MIT licence (© AgriciDaniel). 314 words, ~1,367 tokens.

Download SKILL.mdSave it as .claude/skills/skill-forge-benchmark/SKILL.md (or your agent's skills folder).

name: skill-forge-benchmark
description: Benchmark Claude Code skill performance with variance analysis, tracking pass rate, execution time, and token usage across iterations. Runs multiple trials per eval for statistical reliability, aggregates results into benchmark.json, and generates comparison reports between skill versions. Use when user says "benchmark skill", "measure skill performance", "skill metrics", "compare skill versions", "skill performance", "track skill improvement", "skill regression test", or "skill A/B test".

Skill Benchmarking & Performance Tracking

Measure and compare skill performance across iterations with statistical rigor using multiple trials, variance analysis, and trend tracking.

Process

Step 1: Define Benchmark Configuration

Accept configuration as:

Existing eval set: Path to evals/evals.json (from /skill-forge eval)
Benchmark config: Custom config with trial count and thresholds

Benchmark config schema:

json

{
  "skill_name": "my-skill",
  "skill_path": "./my-skill",
  "eval_set_path": "./evals/evals.json",
  "trials_per_eval": 3,
  "baseline_type": "no_skill",
  "previous_benchmark": null,
  "thresholds": {
    "min_pass_rate": 0.8,
    "max_avg_tokens": 100000,
    "max_avg_duration_seconds": 120,
    "min_improvement_ratio": 1.0
  }
}

Step 2: Execute Benchmark Runs

For each eval, run trials_per_eval times (default: 3) to get reliable metrics:

Execute with-skill runs (3x per eval)
Execute baseline runs (3x per eval)
Capture per-run: pass/fail, token count, duration
Save each run's timing.json and grading.json

Use agents/skill-forge-executor.md for parallel execution where possible.

Step 3: Aggregate Results

Run python scripts/aggregate_benchmark.py <workspace>/iteration-<N> --skill-name <name>:

Output benchmark.json schema:

json

{
  "skill_name": "my-skill",
  "iteration": 1,
  "timestamp": "2026-03-06T12:00:00Z",
  "summary": {
    "total_evals": 10,
    "with_skill": {
      "pass_rate": 0.87,
      "pass_rate_std": 0.05,
      "avg_tokens": 45000,
      "avg_duration_seconds": 34.2
    },
    "baseline": {
      "pass_rate": 0.60,
      "pass_rate_std": 0.08,
      "avg_tokens": 62000,
      "avg_duration_seconds": 52.1
    },
    "improvement_ratio": 1.45,
    "token_savings_ratio": 0.73,
    "time_savings_ratio": 0.66
  },
  "per_eval": [
    {
      "eval_id": 0,
      "eval_name": "basic-trigger",
      "with_skill": {"pass_rate": 1.0, "avg_tokens": 30000, "avg_duration_seconds": 20.1},
      "baseline": {"pass_rate": 0.67, "avg_tokens": 50000, "avg_duration_seconds": 45.0},
      "trials": 3
    }
  ],
  "thresholds_met": {
    "min_pass_rate": true,
    "max_avg_tokens": true,
    "max_avg_duration_seconds": true,
    "min_improvement_ratio": true
  }
}

Step 4: Compare with Previous Iterations

If previous_benchmark is provided or prior iteration-<N-1> exists:

Load previous benchmark.json
Calculate delta per metric:
- Pass rate change
- Token usage change
- Duration change
- New regressions (evals that passed before but fail now)
- New improvements (evals that failed before but pass now)

Step 5: Generate Benchmark Report

markdown

# Benchmark Report: [skill-name]

## Iteration [N] vs [N-1]

### Summary
| Metric | Current | Previous | Delta | Threshold | Status |
|--------|---------|----------|-------|-----------|--------|
| Pass Rate | 87% | 78% | +9% | >= 80% | PASS |
| Avg Tokens | 45K | 52K | -13% | <= 100K | PASS |
| Avg Time | 34s | 41s | -17% | <= 120s | PASS |
| Improvement | 1.45x | 1.30x | +0.15x | >= 1.0x | PASS |

### Regressions (Action Required)
| Eval | Previous | Current | Notes |
|------|----------|---------|-------|
| eval-5 | PASS | FAIL | Output missing required section |

### Improvements
| Eval | Previous | Current | Notes |
|------|----------|---------|-------|
| eval-3 | FAIL | PASS | Error handling now works |

### Per-Eval Detail
[Full breakdown table]

### Variance Analysis
| Eval | Pass Rate | Std Dev | Trials | Reliability |
|------|-----------|---------|--------|-------------|
| eval-0 | 100% | 0.00 | 3 | High |
| eval-1 | 67% | 0.47 | 3 | Low (investigate) |

### Recommendations
[Based on regressions, low-reliability evals, and threshold failures]

Step 6: Threshold Gating

If any threshold fails:

Flag as FAIL with specific threshold details
List which evals caused the failure
Recommend running /skill-forge evolve to address issues
Do NOT approve for publish until thresholds pass

Error Handling

Flaky trials: If a trial times out or crashes, exclude it from variance calculation and note "trials_completed" vs "trials_requested" in per-eval results
Insufficient trials: If fewer than 2 trials complete for an eval, flag variance as "unreliable" in the report
Missing baseline: If baseline runs fail entirely, report with-skill results only and skip improvement_ratio
Threshold edge cases: If pass_rate equals the threshold exactly, treat as PASS

Integration with Other Sub-Skills

skill-forge-eval: Provides the eval set and grading infrastructure
skill-forge-evolve: Receives benchmark failures as improvement targets
skill-forge-publish: Requires benchmark pass (score >= thresholds) before publish
skill-forge-review: Can include benchmark summary in review report

Files

Just SKILL.md in skills/skill-forge-benchmark of AgriciDaniel/skill-forge.

Open the folder on GitHubat commit 2872ee9

Compare with similar skills

Skill Forge Benchmark next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Skill Forge Benchmark compared with similar skills
Skill	Stars	Used in	Tokens	Auto-check	Licence	Repo updated
Skill Forge Benchmark this skillAgriciDaniel/skill-forge	177	—	~1.4k	Automated safety check: Pass	MIT	6 mo ago
LLM Evaluationdavila7/claude-code-templates	32k	13 repos	~3.5k	Automated safety check: Pass	MIT	today
Spec Optimizeleo-kuang-ai/spec-first	107	—	~13k	Automated safety check: Pass	MIT	13 days ago
Explore Runlllllllama/RigorPilot-Skills	497	2 repos	~833	Automated safety check: Pass	MIT	14 days ago
Agent Eval Engineeringlangchain-ai/langchain-skills	1.3k	—	~4k	Automated safety check: Pass	MIT	2 days ago
Skill Conductorsmixs/skill-conductor	179	—	~6.6k	Automated safety check: Pass	MIT	2 mo ago

Similar skills

LLM Evaluation
davila7/claude-code-templates
Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing.
32k GitHub starsUsed in 13 repos~3.5k tokens
AI & LLM EngineeringAuto-check passed
Spec Optimize
leo-kuang-ai/spec-first
Run metric-driven iterative optimization loops. An agent skill from leo-kuang-ai/spec-first.
107 GitHub stars~13k tokensUpdated 13 days ago
AI & LLM EngineeringAuto-check passed
Explore Run
lllllllama/RigorPilot-Skills
Rigor Improve / Rigor Explore run leaf skill for bounded exploratory evidence in deep learning research repositories.
497 GitHub starsUsed in 2 repos~833 tokens
AI & LLM EngineeringAuto-check passed
Agent Eval Engineering
langchain-ai/langchain-skills
Official
Builds agent evaluations in stages: inspect the repository and traces, agree a Task Spec with you, then build, audit and run a Harbor task with an independent verifier.
1.3k GitHub stars~4k tokensUpdated 2 days ago
AI & LLM EngineeringAuto-check passed
Skill Conductor
smixs/skill-conductor
Create, edit, evaluate, and package agent skills. An agent skill from smixs/skill-conductor.
179 GitHub stars~6.6k tokensUpdated 2 mo ago
AI & LLM EngineeringAuto-check passed
Evaluate RAG
ai-evals-course/evals-skills
Guides evaluation of a RAG system by diagnosing failures in traces, building a retrieval test set and scoring retrieval and generation separately.
1.5k GitHub stars~1.9k tokensUpdated 13 days ago
AI & LLM EngineeringAuto-check passed

Questions about Skill Forge Benchmark

What does Skill Forge Benchmark do?

Benchmark Claude Code skill performance with variance analysis, tracking pass rate, execution time, and token usage across iterations. Skill Forge Benchmark is an agent skill from AgriciDaniel/skill-forge. Benchmark Claude Code skill performance with variance analysis, tracking pass rate, execution time, and token usage across iterations.

When should I use Skill Forge Benchmark?

Skill Forge Benchmark fits situations like: user says benchmark skill; measure skill performance; compare skill versions; skill performance.

How do I install Skill Forge Benchmark in Claude Code?

Run `npx skills add AgriciDaniel/skill-forge --skill skill-forge-benchmark -a claude-code`. Or copy the skill folder (skills/skill-forge-benchmark in AgriciDaniel/skill-forge) into .claude/skills/skill-forge-benchmark in your project. Claude Code loads it when a task matches its description.

How do I install Skill Forge Benchmark in Codex?

Run `npx skills add AgriciDaniel/skill-forge --skill skill-forge-benchmark -a codex`. Or copy the skill folder (skills/skill-forge-benchmark in AgriciDaniel/skill-forge) into .agents/skills/skill-forge-benchmark in your project. Codex loads it when a task matches its description.

Can I use Skill Forge Benchmark in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add AgriciDaniel/skill-forge --skill skill-forge-benchmark -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/skill-forge-benchmark, .gemini/skills/skill-forge-benchmark, .github/skills/skill-forge-benchmark and .opencode/skills/skill-forge-benchmark in your project.

What does Skill Forge Benchmark need to run?

Going by SKILL.md and its folder, Skill Forge Benchmark needs the command-line tools its instructions call (python). Our summary lists: Python 3.

Does Skill Forge Benchmark access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Skill Forge Benchmark safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Skill Forge Benchmark use?

Skill Forge Benchmark is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Skill Forge Benchmark use?

About 1.4k tokens (SKILL.md is roughly 5.5k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Skill Forge Benchmark?

Skills that share tags, products or a category with Skill Forge Benchmark: LLM Evaluation (davila7/claude-code-templates, 32k stars), Spec Optimize (leo-kuang-ai/spec-first, 107 stars), Explore Run (lllllllama/RigorPilot-Skills, 497 stars) and Agent Eval Engineering (langchain-ai/langchain-skills, 1.3k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Skill Forge Benchmark?

AgriciDaniel (a GitHub user) maintains it in AgriciDaniel/skill-forge, which has 177 GitHub stars. The repository holds 9 skills in this directory. The repository was last updated on April 10, 2026.

Source: AgriciDaniel/skill-forge on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.

Install Skill Forge Benchmark

Facts

At a glance

What it does

When your agent uses it

Example prompts

Requirements

Workflow steps

What it can do on your machine

Tool permissions

Runs code

Network

Credentials

Context cost

Safety

SKILL.md

Skill Benchmarking & Performance Tracking

Process

Step 1: Define Benchmark Configuration

Step 2: Execute Benchmark Runs

Step 3: Aggregate Results

Step 4: Compare with Previous Iterations

Step 5: Generate Benchmark Report

Step 6: Threshold Gating

Error Handling

Integration with Other Sub-Skills

Files

Compare with similar skills

Similar skills

More from AgriciDaniel/skill-forge

Related

Questions about Skill Forge Benchmark