LLM Benchmarking with lm-evaluation-harness
Orchestra-Research/AI-Research-SKILLs
Runs lm-evaluation-harness to benchmark language models on academic suites such as MMLU, GSM8K and HumanEval, compare models and track training checkpoints.
Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles.
$ npx skills add cloudnative-co/claude-code-starter-kit --skill eval-harness -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install cloudnative-co/claude-code-starter-kit eval-harness --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/cloudnative-co/claude-code-starter-kit.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/eval-harness .claude/skills/eval-harness && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "eval-harness" agent skill from https://github.com/cloudnative-co/claude-code-starter-kit/tree/main/skills/eval-harness into .claude/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/cloudnative-co/claude-code-starter-kit/tree/main/skills/eval-harnessType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add cloudnative-co/claude-code-starter-kit --skill eval-harness -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install cloudnative-co/claude-code-starter-kit eval-harness --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/cloudnative-co/claude-code-starter-kit.git skills-src && mkdir -p .agents/skills && cp -r skills-src/skills/eval-harness .agents/skills/eval-harness && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "eval-harness" agent skill from https://github.com/cloudnative-co/claude-code-starter-kit/tree/main/skills/eval-harness into .agents/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add cloudnative-co/claude-code-starter-kit --skill eval-harness -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install cloudnative-co/claude-code-starter-kit eval-harness --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/cloudnative-co/claude-code-starter-kit.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/skills/eval-harness .cursor/skills/eval-harness && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "eval-harness" agent skill from https://github.com/cloudnative-co/claude-code-starter-kit/tree/main/skills/eval-harness into .cursor/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/cloudnative-co/claude-code-starter-kit.git --path skills/eval-harness--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add cloudnative-co/claude-code-starter-kit --skill eval-harness -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install cloudnative-co/claude-code-starter-kit eval-harness --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/cloudnative-co/claude-code-starter-kit.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/skills/eval-harness .gemini/skills/eval-harness && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "eval-harness" agent skill from https://github.com/cloudnative-co/claude-code-starter-kit/tree/main/skills/eval-harness into .gemini/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install cloudnative-co/claude-code-starter-kit eval-harnessInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add cloudnative-co/claude-code-starter-kit --skill eval-harness -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/cloudnative-co/claude-code-starter-kit.git skills-src && mkdir -p .github/skills && cp -r skills-src/skills/eval-harness .github/skills/eval-harness && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "eval-harness" agent skill from https://github.com/cloudnative-co/claude-code-starter-kit/tree/main/skills/eval-harness into .github/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add cloudnative-co/claude-code-starter-kit --skill eval-harness -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install cloudnative-co/claude-code-starter-kit eval-harness --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/cloudnative-co/claude-code-starter-kit.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/skills/eval-harness .opencode/skills/eval-harness && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "eval-harness" agent skill from https://github.com/cloudnative-co/claude-code-starter-kit/tree/main/skills/eval-harness into .opencode/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
eval-harnessFormal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles.
Eval Harness is an agent skill from cloudnative-co/claude-code-starter-kit. Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles.
Its SKILL.md is about 1.3k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.
It sits in AI & LLM Engineering, covering LLM evaluation. The repository describes itself as: One-command setup of a complete Claude Code development environment with interactive wizard. The licence is MIT.
7 steps, taken from the step headings in SKILL.md.
Read from SKILL.md and the folder at commit f00e7ce. It shows what the files ask for, not the result of running them.
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Shell commands in SKILL.md call:
npmclaudeFrom the folder's file list and the shell code blocks in SKILL.md.
No URLs in SKILL.md. Its commands use npm, which can reach the network depending on how they are called.
From URLs in SKILL.md, links to its own repository left out.
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Eval Harness loads about 1.3k tokens when it runs. Until then it costs about 30 tokens; SKILL.md has 235 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
The full file from cloudnative-co/claude-code-starter-kit at commit f00e7ce, republished under its MIT licence (© cloudnative-co). 235 words, ~1,259 tokens.
.claude/skills/eval-harness/SKILL.md (or your agent's skills folder).A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles.
Eval-Driven Development treats evals as the "unit tests of AI development":
Test if Claude can do something it couldn't before:
[CAPABILITY EVAL: feature-name]
Task: Description of what Claude should accomplish
Success Criteria:
- [ ] Criterion 1
- [ ] Criterion 2
- [ ] Criterion 3
Expected Output: Description of expected resultEnsure changes don't break existing functionality:
[REGRESSION EVAL: feature-name]
Baseline: git SHA (or a /checkpoint milestone name)
Tests:
- existing-test-1: PASS/FAIL
- existing-test-2: PASS/FAIL
- existing-test-3: PASS/FAIL
Result: X/Y passed (previously Y/Y)Deterministic checks using code:
# Check if file contains expected pattern
grep -q "export function handleAuth" src/auth.ts && echo "PASS" || echo "FAIL"
# Check if tests pass
npm test -- --testPathPattern="auth" && echo "PASS" || echo "FAIL"
# Check if build succeeds
npm run build && echo "PASS" || echo "FAIL"Have a separate instance (subagent) review open-ended outputs against a checklist:
[MODEL GRADER PROMPT]
Review the following code change and answer each question with YES/NO plus evidence:
1. Does it solve the stated problem?
2. Is it well-structured?
3. Are edge cases handled?
4. Is error handling appropriate?
Verdict: PASS/FAIL
Reasoning: [explanation]Flag for manual review:
[HUMAN REVIEW REQUIRED]
Change: Description of what changed
Reason: Why human review is needed
Risk Level: LOW/MEDIUM/HIGHNote: statistical metrics like pass@k require running the same task k times independently. If you truly need them, implement an automated script (e.g., a headless claude -p loop), not manual bookkeeping in an interactive session.
## EVAL DEFINITION: feature-xyz
### Capability Evals
1. Can create new user account
2. Can validate email format
3. Can hash password securely
### Regression Evals
1. Existing login still works
2. Session management unchanged
3. Logout flow intactWrite code to pass the defined evals.
# Run capability evals
[Run each capability eval, record PASS/FAIL]
# Run regression evals
npm test -- --testPathPattern="existing"
# Generate reportEVAL REPORT: feature-xyz
========================
Capability Evals:
create-user: PASS (attempts: 1)
validate-email: PASS (attempts: 2)
hash-password: PASS (attempts: 1)
Overall: 3/3 passed
Regression Evals:
login-flow: PASS
session-mgmt: PASS
logout-flow: PASS
Overall: 3/3 passed
Status: READY FOR REVIEW/eval define feature-nameCreates eval definition file at .claude/evals/feature-name.md
/eval check feature-nameRuns current evals and reports status
/eval report feature-nameGenerates full eval report
Store evals in project:
.claude/
evals/
feature-xyz.md # Eval definition
feature-xyz.log # Eval run history
baseline.json # Regression baselines## EVAL: add-authentication
### Phase 1: Define (10 min)
Capability Evals:
- [ ] User can register with email/password
- [ ] User can login with valid credentials
- [ ] Invalid credentials rejected with proper error
- [ ] Sessions persist across page reloads
- [ ] Logout clears session
Regression Evals:
- [ ] Public routes still accessible
- [ ] API responses unchanged
- [ ] Database schema compatible
### Phase 2: Implement (varies)
[Write code]
### Phase 3: Evaluate
Run: /eval check add-authentication
### Phase 4: Report
EVAL REPORT: add-authentication
==============================
Capability: 5/5 passed
Regression: 3/3 passed
Status: SHIP IT© cloudnative-co, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Just SKILL.md in skills/eval-harness of cloudnative-co/claude-code-starter-kit.
Open the folder on GitHubat commit f00e7ce
We found 15 copies of this SKILL.md (exact, near-identical or edited) in other folders, from 10 other GitHub owners. This page covers the copy in cloudnative-co/claude-code-starter-kit, which our catalogue first saw on October 7, 2026.
Eval Harness next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Eval Harness this skillcloudnative-co/claude-code-starter-kit | 151 | 10 repos | ~1.3k | Automated safety check: Pass | MIT | |
| LLM Benchmarking with lm-evaluation-harnessOrchestra-Research/AI-Research-SKILLs | 13k | 8 repos | ~3k | Automated safety check: Pass | MIT | |
| Azure AI Projects Python SDKmicrosoft/skills | 3.1k | 6 repos | ~2.8k | Automated safety check: Pass | MIT | |
| Fine-Tuning ExpertJeffallan/claude-skills | 12k | 1 repos | ~1.7k | Automated safety check: Pass | MIT | |
| Looperksimback/looper | 710 | — | ~2.7k | Automated safety check: Notes | MIT | |
| Hugging Face Local Model Evalshuggingface/skills | 11k | 2 repos | ~1.6k | Automated safety check: Pass | Apache-2.0 |
Orchestra-Research/AI-Research-SKILLs
Runs lm-evaluation-harness to benchmark language models on academic suites such as MMLU, GSM8K and HumanEval, compare models and track training checkpoints.
microsoft/skills
Reference for building on Microsoft Foundry with the azure-ai-projects Python SDK: project clients, versioned agents, evaluations, connections, datasets and indexes.
Jeffallan/claude-skills
Guides LLM fine-tuning with LoRA and QLoRA through Hugging Face PEFT, from dataset validation and training checks to adapter merging, quantization and deployment.
ksimback/looper
Scaffold a well-designed agent loop with best-practice coaching and a cross-model review council.
huggingface/skills
Runs evaluations of Hugging Face Hub models on local hardware with inspect-ai or lighteval, and helps choose between vLLM, Transformers and accelerate backends.
langchain-ai/langchain-skills
Builds agent evaluations in stages: inspect the repository and traces, agree a Task Spec with you, then build, audit and run a Harbor task with an independent verifier.
cloudnative-co/claude-code-starter-kit
A skill your agent uses when the user explicitly asks for TDD or a tests-first workflow, or when developing a new feature with a test coverage requirement.
cloudnative-co/claude-code-starter-kit
Web取得の標準前処理レイヤー。URL・公式ドキュメント・ブログ・ニュース・OSSページを読むときは、原則として毎回このSkillでDefuddleを使い本文をMarkdown/JSON化してから読む。生HTMLのまま要約・分析・比較・レビューしない。トリガー例: URLを読む/Webページ要約/OSS調査/公式Doc確認/記事解析/競合サイト確認/Web一次情報確認。
cloudnative-co/claude-code-starter-kit
Example project-specific skill template. An agent skill from cloudnative-co/claude-code-starter-kit.
cloudnative-co/claude-code-starter-kit
Practical prompt patterns and techniques for effective Claude Code usage.
cloudnative-co/claude-code-starter-kit
Comprehensive verification system for Claude Code sessions. An agent skill from cloudnative-co/claude-code-starter-kit.
cloudnative-co/claude-code-starter-kit
日本語の業務文書を作成・修正するときに使用する共通品質基準。事実性、確度、論理、簡潔さ、自然な日本語を守る。提案書、報告、技術説明、議事録、メール、Slack、要約、レビューに適用する。創作、広告コピー、コードや構造化データだけの生成には使用しない。
Categories
Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles. Eval Harness is an agent skill from cloudnative-co/claude-code-starter-kit. Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles.
Eval Harness fits situations like: tasks that involve LLM evaluation.
Run `npx skills add cloudnative-co/claude-code-starter-kit --skill eval-harness -a claude-code`. Or copy the skill folder (skills/eval-harness in cloudnative-co/claude-code-starter-kit) into .claude/skills/eval-harness in your project. Claude Code loads it when a task matches its description.
Run `npx skills add cloudnative-co/claude-code-starter-kit --skill eval-harness -a codex`. Or copy the skill folder (skills/eval-harness in cloudnative-co/claude-code-starter-kit) into .agents/skills/eval-harness in your project. Codex loads it when a task matches its description.
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add cloudnative-co/claude-code-starter-kit --skill eval-harness -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/eval-harness, .gemini/skills/eval-harness, .github/skills/eval-harness and .opencode/skills/eval-harness in your project.
Going by SKILL.md and its folder, Eval Harness needs the command-line tools its instructions call (npm and claude).
SKILL.md contains no URLs. Its commands use npm, which can reach the network depending on how they are called. This is read from the text; nothing was executed.
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
Eval Harness is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
About 1.3k tokens (SKILL.md is roughly 5k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
Skills that share tags, products or a category with Eval Harness: LLM Benchmarking with lm-evaluation-harness (Orchestra-Research/AI-Research-SKILLs, 13k stars), Azure AI Projects Python SDK (microsoft/skills, 3.1k stars), Fine-Tuning Expert (Jeffallan/claude-skills, 12k stars) and Looper (ksimback/looper, 710 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
cloudnative-co (a GitHub organization) maintains it in cloudnative-co/claude-code-starter-kit, which has 151 GitHub stars. The repository holds 8 skills in this directory. The repository was last updated on October 7, 2026.
Source: cloudnative-co/claude-code-starter-kit on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.