LLM Benchmarking with lm-evaluation-harness
Orchestra-Research/AI-Research-SKILLs
Runs lm-evaluation-harness to benchmark language models on academic suites such as MMLU, GSM8K and HumanEval, compare models and track training checkpoints.
Claude Codeセッションの正式な評価フレームワークで、評価駆動開発（EDD）の原則を実装します. An agent skill from affaan-m/ECC.
Install Eval Harness
$ npx skills add affaan-m/ECC --skill eval-harness -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install affaan-m/ECC eval-harness --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/affaan-m/ECC.git skills-src && mkdir -p .claude/skills && cp -r skills-src/docs/ja-JP/skills/eval-harness .claude/skills/eval-harness && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "eval-harness" agent skill from https://github.com/affaan-m/ECC/tree/main/docs/ja-JP/skills/eval-harness into .claude/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/affaan-m/ECC/tree/main/docs/ja-JP/skills/eval-harnessType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add affaan-m/ECC --skill eval-harness -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install affaan-m/ECC eval-harness --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/affaan-m/ECC.git skills-src && mkdir -p .agents/skills && cp -r skills-src/docs/ja-JP/skills/eval-harness .agents/skills/eval-harness && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "eval-harness" agent skill from https://github.com/affaan-m/ECC/tree/main/docs/ja-JP/skills/eval-harness into .agents/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add affaan-m/ECC --skill eval-harness -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install affaan-m/ECC eval-harness --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/affaan-m/ECC.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/docs/ja-JP/skills/eval-harness .cursor/skills/eval-harness && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "eval-harness" agent skill from https://github.com/affaan-m/ECC/tree/main/docs/ja-JP/skills/eval-harness into .cursor/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/affaan-m/ECC.git --path docs/ja-JP/skills/eval-harness--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add affaan-m/ECC --skill eval-harness -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install affaan-m/ECC eval-harness --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/affaan-m/ECC.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/docs/ja-JP/skills/eval-harness .gemini/skills/eval-harness && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "eval-harness" agent skill from https://github.com/affaan-m/ECC/tree/main/docs/ja-JP/skills/eval-harness into .gemini/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install affaan-m/ECC eval-harnessInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add affaan-m/ECC --skill eval-harness -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/affaan-m/ECC.git skills-src && mkdir -p .github/skills && cp -r skills-src/docs/ja-JP/skills/eval-harness .github/skills/eval-harness && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "eval-harness" agent skill from https://github.com/affaan-m/ECC/tree/main/docs/ja-JP/skills/eval-harness into .github/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add affaan-m/ECC --skill eval-harness -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install affaan-m/ECC eval-harness --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/affaan-m/ECC.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/docs/ja-JP/skills/eval-harness .opencode/skills/eval-harness && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "eval-harness" agent skill from https://github.com/affaan-m/ECC/tree/main/docs/ja-JP/skills/eval-harness into .opencode/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
Facts
- Skill name
eval-harness- GitHub stars
- 276k
- Used in
- 2 other repos
- Token cost
- ~884 tokens
- SKILL.md length
- 84 words
- Files
- 1
- Skills in repo
- 683
- Repo updated
- First seen
- Licence
- MIT
At a glance
Claude Codeセッションの正式な評価フレームワークで、評価駆動開発（EDD）の原則を実装します. An agent skill from affaan-m/ECC.
- Works in 7 steps: コードベース評価者 → モデルベース評価者 → 人間評価者 → …
- Tasks that involve LLM evaluation
- SKILL.md covers 哲学, 評価タイプ, 評価者タイプ and メトリクス, plus 5 more sections
- Calls npm
What it does
Eval Harness is an agent skill from affaan-m/ECC. Claude Codeセッションの正式な評価フレームワークで、評価駆動開発（EDD）の原則を実装します
Its SKILL.md is about 880 tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.
It sits in AI & LLM Engineering, covering LLM evaluation. The repository describes itself as: The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond. The licence is MIT.
When your agent uses it
- Tasks that involve LLM evaluation
Example prompts
- “/eval-harness”
Workflow steps
7 steps, taken from the step headings in SKILL.md.
What it can do on your machine
Read from SKILL.md and the folder at commit 4eb71d9. It shows what the files ask for, not the result of running them.
Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
Shell commands in SKILL.md call:
npm
From the folder's file list and the shell code blocks in SKILL.md.
Network
No URLs in SKILL.md. Its commands use npm, which can reach the network depending on how they are called.
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Context cost
Eval Harness loads about 884 tokens when it runs. Until then it costs about 16 tokens; SKILL.md has 84 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
Safety
Auto-check passed
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
SKILL.md
The full file from affaan-m/ECC at commit 4eb71d9, republished under its MIT licence (© affaan-m). 84 words, ~884 tokens.
Download SKILL.mdSave it as
.claude/skills/eval-harness/SKILL.md (or your agent's skills folder).- name
- eval-harness
- description
- Claude Codeセッションの正式な評価フレームワークで、評価駆動開発（EDD）の原則を実装します
- tools
- Read, Write, Edit, Bash, Grep, Glob
Eval Harnessスキル
Claude Codeセッションの正式な評価フレームワークで、評価駆動開発（EDD）の原則を実装します。
哲学
評価駆動開発は評価を「AI開発のユニットテスト」として扱います：
- 実装前に期待される動作を定義
- 開発中に継続的に評価を実行
- 変更ごとにリグレッションを追跡
- 信頼性測定にpass@kメトリクスを使用
評価タイプ
能力評価
Claudeが以前できなかったことができるようになったかをテスト：
[CAPABILITY EVAL: feature-name]
タスク: Claudeが達成すべきことの説明
成功基準:
- [ ] 基準1
- [ ] 基準2
- [ ] 基準3
期待される出力: 期待される結果の説明リグレッション評価
変更が既存の機能を破壊しないことを確認：
[REGRESSION EVAL: feature-name]
ベースライン: SHAまたはチェックポイント名
テスト:
- existing-test-1: PASS/FAIL
- existing-test-2: PASS/FAIL
- existing-test-3: PASS/FAIL
結果: X/Y 成功（以前は Y/Y）評価者タイプ
1. コードベース評価者
コードを使用した決定論的チェック：
# ファイルに期待されるパターンが含まれているかチェック
grep -q "export function handleAuth" src/auth.ts && echo "PASS" || echo "FAIL"
# テストが成功するかチェック
npm test -- --testPathPattern="auth" && echo "PASS" || echo "FAIL"
# ビルドが成功するかチェック
npm run build && echo "PASS" || echo "FAIL"2. モデルベース評価者
Claudeを使用して自由形式の出力を評価：
[MODEL GRADER PROMPT]
次のコード変更を評価してください：
1. 記述された問題を解決していますか？
2. 構造化されていますか？
3. エッジケースは処理されていますか？
4. エラー処理は適切ですか？
スコア: 1-5（1=不良、5=優秀）
理由: [説明]3. 人間評価者
手動レビューのためにフラグを立てる：
[HUMAN REVIEW REQUIRED]
変更内容: 何が変更されたかの説明
理由: 人間のレビューが必要な理由
リスクレベル: LOW/MEDIUM/HIGHメトリクス
pass@k
「k回の試行で少なくとも1回成功」
- pass@1: 最初の試行での成功率
- pass@3: 3回以内の成功
- 一般的な目標: pass@3 > 90%
pass^k
「k回の試行すべてが成功」
- より高い信頼性の基準
- pass^3: 3回連続成功
- クリティカルパスに使用
評価ワークフロー
1. 定義（コーディング前）
## 評価定義: feature-xyz
### 能力評価
1. 新しいユーザーアカウントを作成できる
2. メール形式を検証できる
3. パスワードを安全にハッシュ化できる
### リグレッション評価
1. 既存のログインが引き続き機能する
2. セッション管理が変更されていない
3. ログアウトフローが維持されている
### 成功メトリクス
- 能力評価で pass@3 > 90%
- リグレッション評価で pass^3 = 100%2. 実装
定義された評価に合格するコードを書く。
3. 評価
# 能力評価を実行
[各能力評価を実行し、PASS/FAILを記録]
# リグレッション評価を実行
npm test -- --testPathPattern="existing"
# レポートを生成4. レポート
評価レポート: feature-xyz
========================
能力評価:
create-user: PASS (pass@1)
validate-email: PASS (pass@2)
hash-password: PASS (pass@1)
全体: 3/3 成功
リグレッション評価:
login-flow: PASS
session-mgmt: PASS
logout-flow: PASS
全体: 3/3 成功
メトリクス:
pass@1: 67% (2/3)
pass@3: 100% (3/3)
ステータス: レビュー準備完了統合パターン
実装前
/eval define feature-name.claude/evals/feature-name.mdに評価定義ファイルを作成
実装中
/eval check feature-name現在の評価を実行してステータスを報告
実装後
/eval report feature-name完全な評価レポートを生成
評価の保存
プロジェクト内に評価を保存：
.claude/
evals/
feature-xyz.md # 評価定義
feature-xyz.log # 評価実行履歴
baseline.json # リグレッションベースラインベストプラクティス
- コーディング前に評価を定義 - 成功基準について明確に考えることを強制
- 頻繁に評価を実行 - リグレッションを早期に検出
- 時間経過とともにpass@kを追跡 - 信頼性のトレンドを監視
- 可能な限りコード評価者を使用 - 決定論的 > 確率的
- セキュリティは人間レビュー - セキュリティチェックを完全に自動化しない
- 評価を高速に保つ - 遅い評価は実行されない
- コードと一緒に評価をバージョン管理 - 評価はファーストクラスの成果物
例：認証の追加
## EVAL: add-authentication
### フェーズ 1: 定義（10分）
能力評価:
- [ ] ユーザーはメール/パスワードで登録できる
- [ ] ユーザーは有効な資格情報でログインできる
- [ ] 無効な資格情報は適切なエラーで拒否される
- [ ] セッションはページリロード後も持続する
- [ ] ログアウトはセッションをクリアする
リグレッション評価:
- [ ] 公開ルートは引き続きアクセス可能
- [ ] APIレスポンスは変更されていない
- [ ] データベーススキーマは互換性がある
### フェーズ 2: 実装（可変）
[コードを書く]
### フェーズ 3: 評価
Run: /eval check add-authentication
### フェーズ 4: レポート
評価レポート: add-authentication
==============================
能力: 5/5 成功（pass@3: 100%）
リグレッション: 3/3 成功（pass^3: 100%）
ステータス: 出荷可能© affaan-m, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Files
Just SKILL.md in docs/ja-JP/skills/eval-harness of affaan-m/ECC.
Open the folder on GitHubat commit 4eb71d9
Used in 2 other repositories
We found 2 copies of this SKILL.md (exact, near-identical or edited) in other folders, from 2 other GitHub owners. This page covers the copy in affaan-m/ECC, which our catalogue first saw on October 7, 2026.
Compare with similar skills
Eval Harness next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Eval Harness this skillaffaan-m/ECC | 276k | 2 repos | ~884 | Automated safety check: Pass | MIT | |
| LLM Benchmarking with lm-evaluation-harnessOrchestra-Research/AI-Research-SKILLs | 13k | 8 repos | ~3k | Automated safety check: Pass | MIT | |
| Hugging Face Local Model Evalshuggingface/skills | 11k | 2 repos | ~1.6k | Automated safety check: Pass | Apache-2.0 | |
| Looperksimback/looper | 710 | — | ~2.7k | Automated safety check: Notes | MIT | |
| Agent Eval Engineeringlangchain-ai/langchain-skills | 1.3k | — | ~4k | Automated safety check: Pass | MIT | |
| Quality FlywheelGoogleCloudPlatform/vertex-ai-samples | 792 | — | ~2k | Automated safety check: Pass | Apache-2.0 |
Similar skills
- Official
Hugging Face Local Model Evals
huggingface/skills
Runs evaluations of Hugging Face Hub models on local hardware with inspect-ai or lighteval, and helps choose between vLLM, Transformers and accelerate backends.
Looper
ksimback/looper
Scaffold a well-designed agent loop with best-practice coaching and a cross-model review council.
- Official
Agent Eval Engineering
langchain-ai/langchain-skills
Builds agent evaluations in stages: inspect the repository and traces, agree a Task Spec with you, then build, audit and run a Harbor task with an independent verifier.
Quality Flywheel
GoogleCloudPlatform/vertex-ai-samples
Evaluate and improve GenAI models and agents using the Google GenAI Evaluation SDK.
Eval Harness
cloudnative-co/claude-code-starter-kit
Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles.
More from affaan-m/ECC
All 683 skills in this repoSkill Stocktake
affaan-m/ECC
Audits your installed Claude skills and commands for quality, with a quick mode for recently changed skills and a full mode that evaluates all of them through subagents.
VideoDB Video Search and Editing
affaan-m/ECC
Ingests, indexes, searches, edits and monitors video, audio and live streams through the VideoDB Python SDK, returning stream links, clips and timestamps.
Docs Governance
affaan-m/ECC
Route broad documentation-governance requests to existing ECC skills and run an opt-in, read-only audit of mapped documentation roles, links, ADR indexes, and evidence references.
Rules Distillation
affaan-m/ECC
Scans installed skills for principles that recur across them and proposes rule-file changes: append, revise, add a section, create a file or leave as covered.
Master Agreement Generator
affaan-m/ECC
Builds DRAFT counterparty agreements from one markdown template and a small JSON spec per party, with clauses picked by the party's role.
Skill Compliance Checker
affaan-m/ECC
Measures whether agents actually follow a skill, rule or agent definition by generating scenarios at three strictness levels and scoring tool-call traces.
Related
Categories
Questions about Eval Harness
What does Eval Harness do?
Claude Codeセッションの正式な評価フレームワークで、評価駆動開発（EDD）の原則を実装します. An agent skill from affaan-m/ECC. Eval Harness is an agent skill from affaan-m/ECC.
When should I use Eval Harness?
Eval Harness fits situations like: tasks that involve LLM evaluation.
How do I install Eval Harness in Claude Code?
Run `npx skills add affaan-m/ECC --skill eval-harness -a claude-code`. Or copy the skill folder (docs/ja-JP/skills/eval-harness in affaan-m/ECC) into .claude/skills/eval-harness in your project. Claude Code loads it when a task matches its description.
How do I install Eval Harness in Codex?
Run `npx skills add affaan-m/ECC --skill eval-harness -a codex`. Or copy the skill folder (docs/ja-JP/skills/eval-harness in affaan-m/ECC) into .agents/skills/eval-harness in your project. Codex loads it when a task matches its description.
Can I use Eval Harness in Cursor, Gemini CLI or GitHub Copilot?
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add affaan-m/ECC --skill eval-harness -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/eval-harness, .gemini/skills/eval-harness, .github/skills/eval-harness and .opencode/skills/eval-harness in your project.
What does Eval Harness need to run?
Going by SKILL.md and its folder, Eval Harness needs the command-line tools its instructions call (npm).
Does Eval Harness access the network?
SKILL.md contains no URLs. Its commands use npm, which can reach the network depending on how they are called. This is read from the text; nothing was executed.
Is Eval Harness safe to install?
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
What licence does Eval Harness use?
Eval Harness is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
How many tokens does Eval Harness use?
About 884 tokens (SKILL.md is roughly 3.5k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
What are the alternatives to Eval Harness?
Skills that share tags, products or a category with Eval Harness: LLM Benchmarking with lm-evaluation-harness (Orchestra-Research/AI-Research-SKILLs, 13k stars), Hugging Face Local Model Evals (huggingface/skills, 11k stars), Looper (ksimback/looper, 710 stars) and Agent Eval Engineering (langchain-ai/langchain-skills, 1.3k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
Who maintains Eval Harness?
affaan-m (a GitHub user) maintains it in affaan-m/ECC, which has 276,111 GitHub stars. The repository holds 683 skills in this directory. The repository was last updated on October 10, 2026.
Source: affaan-m/ECC on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.