Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles.

MITAuto-check passedAI & LLM Engineering

Install Eval Harness

skills CLI
$ npx skills add cloudnative-co/claude-code-starter-kit --skill eval-harness -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install cloudnative-co/claude-code-starter-kit eval-harness --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/cloudnative-co/claude-code-starter-kit.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/eval-harness .claude/skills/eval-harness && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
eval-harness
GitHub stars
151
Used in
10 other repos
Token cost
~1.3k tokens
SKILL.md length
235 words
Files
1
Skills in repo
8
Repo updated
First seen
Licence
MIT

At a glance

Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles.

  • Works in 7 steps: Code-Based Grader → Model-Based Grader → Human Grader → …
  • Tasks that involve LLM evaluation
  • SKILL.md covers Philosophy, Eval Types, Grader Types and Eval Workflow, plus 4 more sections
  • Calls npm and claude

What it does

Eval Harness is an agent skill from cloudnative-co/claude-code-starter-kit. Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles.

Its SKILL.md is about 1.3k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.

It sits in AI & LLM Engineering, covering LLM evaluation. The repository describes itself as: One-command setup of a complete Claude Code development environment with interactive wizard. The licence is MIT.

When your agent uses it

  • Tasks that involve LLM evaluation

Example prompts

  • “/eval-harness”

Workflow steps

7 steps, taken from the step headings in SKILL.md.

  1. Code-Based Grader
  2. Model-Based Grader
  3. Human Grader
  4. Define (Before Coding)
  5. Implement
  6. Evaluate
  7. Report

What it can do on your machine

Read from SKILL.md and the folder at commit f00e7ce. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Shell commands in SKILL.md call:

    • npm
    • claude

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    No URLs in SKILL.md. Its commands use npm, which can reach the network depending on how they are called.

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Eval Harness loads about 1.3k tokens when it runs. Until then it costs about 30 tokens; SKILL.md has 235 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~30
When it runs · the whole SKILL.md, loaded when a task matches
~1.3k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from cloudnative-co/claude-code-starter-kit at commit f00e7ce, republished under its MIT licence (© cloudnative-co). 235 words, ~1,259 tokens.

Download SKILL.mdSave it as .claude/skills/eval-harness/SKILL.md (or your agent's skills folder).
name
eval-harness
description
Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles.
when_to_use
Use when the user wants formal evals, pass/fail regression checks, or eval-driven development for AI behavior.

Eval Harness Skill

A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles.

Philosophy

Eval-Driven Development treats evals as the "unit tests of AI development":

  • Define expected behavior BEFORE implementation
  • Run evals continuously during development
  • Track regressions with each change

Eval Types

Capability Evals

Test if Claude can do something it couldn't before:

markdown
[CAPABILITY EVAL: feature-name]
Task: Description of what Claude should accomplish
Success Criteria:
  - [ ] Criterion 1
  - [ ] Criterion 2
  - [ ] Criterion 3
Expected Output: Description of expected result
Regression Evals

Ensure changes don't break existing functionality:

markdown
[REGRESSION EVAL: feature-name]
Baseline: git SHA (or a /checkpoint milestone name)
Tests:
  - existing-test-1: PASS/FAIL
  - existing-test-2: PASS/FAIL
  - existing-test-3: PASS/FAIL
Result: X/Y passed (previously Y/Y)

Grader Types

1. Code-Based Grader

Deterministic checks using code:

bash
# Check if file contains expected pattern
grep -q "export function handleAuth" src/auth.ts && echo "PASS" || echo "FAIL"

# Check if tests pass
npm test -- --testPathPattern="auth" && echo "PASS" || echo "FAIL"

# Check if build succeeds
npm run build && echo "PASS" || echo "FAIL"
2. Model-Based Grader

Have a separate instance (subagent) review open-ended outputs against a checklist:

markdown
[MODEL GRADER PROMPT]
Review the following code change and answer each question with YES/NO plus evidence:
1. Does it solve the stated problem?
2. Is it well-structured?
3. Are edge cases handled?
4. Is error handling appropriate?

Verdict: PASS/FAIL
Reasoning: [explanation]
3. Human Grader

Flag for manual review:

markdown
[HUMAN REVIEW REQUIRED]
Change: Description of what changed
Reason: Why human review is needed
Risk Level: LOW/MEDIUM/HIGH

Note: statistical metrics like pass@k require running the same task k times independently. If you truly need them, implement an automated script (e.g., a headless claude -p loop), not manual bookkeeping in an interactive session.

Eval Workflow

1. Define (Before Coding)
markdown
## EVAL DEFINITION: feature-xyz

### Capability Evals
1. Can create new user account
2. Can validate email format
3. Can hash password securely

### Regression Evals
1. Existing login still works
2. Session management unchanged
3. Logout flow intact
2. Implement

Write code to pass the defined evals.

3. Evaluate
bash
# Run capability evals
[Run each capability eval, record PASS/FAIL]

# Run regression evals
npm test -- --testPathPattern="existing"

# Generate report
4. Report
markdown
EVAL REPORT: feature-xyz
========================

Capability Evals:
  create-user:     PASS (attempts: 1)
  validate-email:  PASS (attempts: 2)
  hash-password:   PASS (attempts: 1)
  Overall:         3/3 passed

Regression Evals:
  login-flow:      PASS
  session-mgmt:    PASS
  logout-flow:     PASS
  Overall:         3/3 passed

Status: READY FOR REVIEW

Integration Patterns

Pre-Implementation
/eval define feature-name

Creates eval definition file at .claude/evals/feature-name.md

During Implementation
/eval check feature-name

Runs current evals and reports status

Post-Implementation
/eval report feature-name

Generates full eval report

Eval Storage

Store evals in project:

.claude/
  evals/
    feature-xyz.md      # Eval definition
    feature-xyz.log     # Eval run history
    baseline.json       # Regression baselines

Best Practices

  1. Define evals BEFORE coding - Forces clear thinking about success criteria
  2. Run evals frequently - Catch regressions early
  3. Use code graders when possible - Deterministic > probabilistic
  4. Human review for security - Never fully automate security checks
  5. Keep evals fast - Slow evals don't get run
  6. Version evals with code - Evals are first-class artifacts

Example: Adding Authentication

markdown
## EVAL: add-authentication

### Phase 1: Define (10 min)
Capability Evals:
- [ ] User can register with email/password
- [ ] User can login with valid credentials
- [ ] Invalid credentials rejected with proper error
- [ ] Sessions persist across page reloads
- [ ] Logout clears session

Regression Evals:
- [ ] Public routes still accessible
- [ ] API responses unchanged
- [ ] Database schema compatible

### Phase 2: Implement (varies)
[Write code]

### Phase 3: Evaluate
Run: /eval check add-authentication

### Phase 4: Report
EVAL REPORT: add-authentication
==============================
Capability: 5/5 passed
Regression: 3/3 passed
Status: SHIP IT

© cloudnative-co, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

Just SKILL.md in skills/eval-harness of cloudnative-co/claude-code-starter-kit.

Open the folder on GitHubat commit f00e7ce

Used in 10 other repositories

We found 15 copies of this SKILL.md (exact, near-identical or edited) in other folders, from 10 other GitHub owners. This page covers the copy in cloudnative-co/claude-code-starter-kit, which our catalogue first saw on October 7, 2026.

Compare with similar skills

Eval Harness next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Eval Harness compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Eval Harness this skillcloudnative-co/claude-code-starter-kit15110 repos~1.3kAutomated safety check: PassMIT
LLM Benchmarking with lm-evaluation-harnessOrchestra-Research/AI-Research-SKILLs13k8 repos~3kAutomated safety check: PassMIT
Azure AI Projects Python SDKmicrosoft/skills3.1k6 repos~2.8kAutomated safety check: PassMIT
Fine-Tuning ExpertJeffallan/claude-skills12k1 repos~1.7kAutomated safety check: PassMIT
Looperksimback/looper710—~2.7kAutomated safety check: NotesMIT
Hugging Face Local Model Evalshuggingface/skills11k2 repos~1.6kAutomated safety check: PassApache-2.0

Similar skills

  • LLM Benchmarking with lm-evaluation-harness

    Orchestra-Research/AI-Research-SKILLs

    Runs lm-evaluation-harness to benchmark language models on academic suites such as MMLU, GSM8K and HumanEval, compare models and track training checkpoints.

    13k GitHub starsUsed in 8 repos~3k tokens
    AI & LLM EngineeringAuto-check passed
  • Official

    Reference for building on Microsoft Foundry with the azure-ai-projects Python SDK: project clients, versioned agents, evaluations, connections, datasets and indexes.

    3.1k GitHub starsUsed in 6 repos~2.8k tokens
    AI & LLM EngineeringAuto-check passed
  • Fine-Tuning Expert

    Jeffallan/claude-skills

    Guides LLM fine-tuning with LoRA and QLoRA through Hugging Face PEFT, from dataset validation and training checks to adapter merging, quantization and deployment.

    12k GitHub starsUsed in 1 repo~1.7k tokens
    AI & LLM EngineeringAuto-check passed
  • Looper

    ksimback/looper

    Scaffold a well-designed agent loop with best-practice coaching and a cross-model review council.

    710 GitHub stars~2.7k tokensUpdated 1 mo ago
    AI & LLM EngineeringAuto-check: notes
  • Official

    Runs evaluations of Hugging Face Hub models on local hardware with inspect-ai or lighteval, and helps choose between vLLM, Transformers and accelerate backends.

    11k GitHub starsUsed in 2 repos~1.6k tokens
    AI & LLM EngineeringAuto-check passed
  • Agent Eval Engineering

    langchain-ai/langchain-skills

    Official

    Builds agent evaluations in stages: inspect the repository and traces, agree a Task Spec with you, then build, audit and run a Harbor task with an independent verifier.

    1.3k GitHub stars~4k tokensUpdated 2 days ago
    AI & LLM EngineeringAuto-check passed

More from cloudnative-co/claude-code-starter-kit

All 8 skills in this repo
  • TDD Workflow

    cloudnative-co/claude-code-starter-kit

    A skill your agent uses when the user explicitly asks for TDD or a tests-first workflow, or when developing a new feature with a test coverage requirement.

    151 GitHub stars~1.1k tokensUpdated today
    Auto-check passed
  • Web Content Extraction

    cloudnative-co/claude-code-starter-kit

    Web取得の標準前処理レイヤー。URL・公式ドキュメント・ブログ・ニュース・OSSページを読むときは、原則として毎回このSkillでDefuddleを使い本文をMarkdown/JSON化してから読む。生HTMLのまま要約・分析・比較・レビューしない。トリガー例: URLを読む/Webページ要約/OSS調査/公式Doc確認/記事解析/競合サイト確認/Web一次情報確認。

    151 GitHub stars~1.2k tokensUpdated today
    Auto-check passed
  • Project Guidelines Example

    cloudnative-co/claude-code-starter-kit

    Example project-specific skill template. An agent skill from cloudnative-co/claude-code-starter-kit.

    151 GitHub stars~446 tokensUpdated today
    Auto-check passed
  • Prompt Patterns

    cloudnative-co/claude-code-starter-kit

    Practical prompt patterns and techniques for effective Claude Code usage.

    151 GitHub stars~887 tokensUpdated today
    Auto-check passed
  • Verification Loop

    cloudnative-co/claude-code-starter-kit

    Comprehensive verification system for Claude Code sessions. An agent skill from cloudnative-co/claude-code-starter-kit.

    151 GitHub stars~733 tokensUpdated today
    Auto-check passed
  • Cloudnative Writing Baseline

    cloudnative-co/claude-code-starter-kit

    日本語の業務文書を作成・修正するときに使用する共通品質基準。事実性、確度、論理、簡潔さ、自然な日本語を守る。提案書、報告、技術説明、議事録、メール、Slack、要約、レビューに適用する。創作、広告コピー、コードや構造化データだけの生成には使用しない。

    151 GitHub stars~1.1k tokensUpdated today
    Auto-check passed

Questions about Eval Harness

What does Eval Harness do?

Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles. Eval Harness is an agent skill from cloudnative-co/claude-code-starter-kit. Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles.

When should I use Eval Harness?

Eval Harness fits situations like: tasks that involve LLM evaluation.

How do I install Eval Harness in Claude Code?

Run `npx skills add cloudnative-co/claude-code-starter-kit --skill eval-harness -a claude-code`. Or copy the skill folder (skills/eval-harness in cloudnative-co/claude-code-starter-kit) into .claude/skills/eval-harness in your project. Claude Code loads it when a task matches its description.

How do I install Eval Harness in Codex?

Run `npx skills add cloudnative-co/claude-code-starter-kit --skill eval-harness -a codex`. Or copy the skill folder (skills/eval-harness in cloudnative-co/claude-code-starter-kit) into .agents/skills/eval-harness in your project. Codex loads it when a task matches its description.

Can I use Eval Harness in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add cloudnative-co/claude-code-starter-kit --skill eval-harness -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/eval-harness, .gemini/skills/eval-harness, .github/skills/eval-harness and .opencode/skills/eval-harness in your project.

What does Eval Harness need to run?

Going by SKILL.md and its folder, Eval Harness needs the command-line tools its instructions call (npm and claude).

Does Eval Harness access the network?

SKILL.md contains no URLs. Its commands use npm, which can reach the network depending on how they are called. This is read from the text; nothing was executed.

Is Eval Harness safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Eval Harness use?

Eval Harness is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Eval Harness use?

About 1.3k tokens (SKILL.md is roughly 5k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Eval Harness?

Skills that share tags, products or a category with Eval Harness: LLM Benchmarking with lm-evaluation-harness (Orchestra-Research/AI-Research-SKILLs, 13k stars), Azure AI Projects Python SDK (microsoft/skills, 3.1k stars), Fine-Tuning Expert (Jeffallan/claude-skills, 12k stars) and Looper (ksimback/looper, 710 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Eval Harness?

cloudnative-co (a GitHub organization) maintains it in cloudnative-co/claude-code-starter-kit, which has 151 GitHub stars. The repository holds 8 skills in this directory. The repository was last updated on October 7, 2026.

Source: cloudnative-co/claude-code-starter-kit on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.