LLM Benchmarking with lm-evaluation-harness
Orchestra-Research/AI-Research-SKILLs
Runs lm-evaluation-harness to benchmark language models on academic suites such as MMLU, GSM8K and HumanEval, compare models and track training checkpoints.
“克劳德代码会话的正式评估框架，实施评估驱动开发（EDD）原则”
Install Eval Harness
$ npx skills add affaan-m/ECC --skill eval-harness -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install affaan-m/ECC eval-harness --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/affaan-m/ECC.git skills-src && mkdir -p .claude/skills && cp -r skills-src/docs/zh-CN/skills/eval-harness .claude/skills/eval-harness && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "eval-harness" agent skill from https://github.com/affaan-m/ECC/tree/main/docs/zh-CN/skills/eval-harness into .claude/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/affaan-m/ECC/tree/main/docs/zh-CN/skills/eval-harnessType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add affaan-m/ECC --skill eval-harness -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install affaan-m/ECC eval-harness --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/affaan-m/ECC.git skills-src && mkdir -p .agents/skills && cp -r skills-src/docs/zh-CN/skills/eval-harness .agents/skills/eval-harness && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "eval-harness" agent skill from https://github.com/affaan-m/ECC/tree/main/docs/zh-CN/skills/eval-harness into .agents/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add affaan-m/ECC --skill eval-harness -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install affaan-m/ECC eval-harness --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/affaan-m/ECC.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/docs/zh-CN/skills/eval-harness .cursor/skills/eval-harness && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "eval-harness" agent skill from https://github.com/affaan-m/ECC/tree/main/docs/zh-CN/skills/eval-harness into .cursor/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/affaan-m/ECC.git --path docs/zh-CN/skills/eval-harness--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add affaan-m/ECC --skill eval-harness -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install affaan-m/ECC eval-harness --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/affaan-m/ECC.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/docs/zh-CN/skills/eval-harness .gemini/skills/eval-harness && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "eval-harness" agent skill from https://github.com/affaan-m/ECC/tree/main/docs/zh-CN/skills/eval-harness into .gemini/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install affaan-m/ECC eval-harnessInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add affaan-m/ECC --skill eval-harness -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/affaan-m/ECC.git skills-src && mkdir -p .github/skills && cp -r skills-src/docs/zh-CN/skills/eval-harness .github/skills/eval-harness && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "eval-harness" agent skill from https://github.com/affaan-m/ECC/tree/main/docs/zh-CN/skills/eval-harness into .github/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add affaan-m/ECC --skill eval-harness -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install affaan-m/ECC eval-harness --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/affaan-m/ECC.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/docs/zh-CN/skills/eval-harness .opencode/skills/eval-harness && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "eval-harness" agent skill from https://github.com/affaan-m/ECC/tree/main/docs/zh-CN/skills/eval-harness into .opencode/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
Facts
- Skill name
eval-harness- GitHub stars
- 277k
- Used in
- 3 other repos
- Token cost
- ~916 tokens
- SKILL.md length
- 155 words
- Files
- 1
- Skills in repo
- 683
- Repo updated
- First seen
- Licence
- MIT
At a glance
- Works in 7 steps: 基于代码的评分器 → 基于模型的评分器 → 人工评分器 → …
- SKILL.md covers 何时激活, 理念, 评估类型 and 评分器类型, plus 7 more sections
- Calls npm
About this skill
Eval Harness is a skill in affaan-m/ECC (277k stars). Its SKILL.md is about 916 tokens, and copies of it appear in 3 other owners' repositories. Licence: MIT.
Workflow steps
7 steps, taken from the step headings in SKILL.md.
What it can do on your machine
Read from SKILL.md and the folder at commit 2d515e4. It shows what the files ask for, not the result of running them.
Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
Shell commands in SKILL.md call:
npm
From the folder's file list and the shell code blocks in SKILL.md.
Network
No URLs in SKILL.md. Its commands use npm, which can reach the network depending on how they are called.
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Context cost
Eval Harness loads about 916 tokens when it runs. Until then it costs about 11 tokens; SKILL.md has 155 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
Safety
Auto-check passed
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
SKILL.md
The full file from affaan-m/ECC at commit 2d515e4, republished under its MIT licence (© affaan-m). 155 words, ~916 tokens.
Download SKILL.mdSave it as
.claude/skills/eval-harness/SKILL.md (or your agent's skills folder).- name
- eval-harness
- description
- 克劳德代码会话的正式评估框架，实施评估驱动开发（EDD）原则
- origin
- ECC
- tools
- Read, Write, Edit, Bash, Grep, Glob
Eval Harness 技能
一个用于 Claude Code 会话的正式评估框架，实现了评估驱动开发 (EDD) 原则。
何时激活
- 为 AI 辅助工作流程设置评估驱动开发 (EDD)
- 定义 Claude Code 任务完成的标准（通过/失败）
- 使用 pass@k 指标衡量代理可靠性
- 为提示或代理变更创建回归测试套件
- 跨模型版本对代理性能进行基准测试
理念
评估驱动开发将评估视为 "AI 开发的单元测试"：
- 在实现 之前 定义预期行为
- 在开发过程中持续运行评估
- 跟踪每次更改的回归情况
- 使用 pass@k 指标来衡量可靠性
评估类型
能力评估
测试 Claude 是否能完成之前无法完成的事情：
[能力评估：功能名称]
任务：描述 Claude 应完成的工作
成功标准：
- [ ] 标准 1
- [ ] 标准 2
- [ ] 标准 标准 3
预期输出：对预期结果的描述
回归评估
确保更改不会破坏现有功能：
[回归评估：功能名称]
基线：SHA 或检查点名称
测试：
- 现有测试-1：通过/失败
- 现有测试-2：通过/失败
- 现有测试-3：通过/失败
结果：X/Y 通过（之前为 Y/Y）
评分器类型
1. 基于代码的评分器
使用代码进行确定性检查：
# Check if file contains expected pattern
grep -q "export function handleAuth" src/auth.ts && echo "PASS" || echo "FAIL"
# Check if tests pass
npm test -- --testPathPattern="auth" && echo "PASS" || echo "FAIL"
# Check if build succeeds
npm run build && echo "PASS" || echo "FAIL"2. 基于模型的评分器
使用 Claude 来评估开放式输出：
[MODEL GRADER PROMPT]
评估以下代码变更：
1. 它是否解决了所述问题？
2. 它的结构是否良好？
3. 是否处理了边界情况？
4. 错误处理是否恰当？
评分：1-5 (1=差，5=优秀)
推理：[解释]
3. 人工评分器
标记为需要手动审查：
[HUMAN REVIEW REQUIRED]
变更：对更改内容的描述
原因：为何需要人工审核
风险等级：低/中/高
指标
pass@k
"k 次尝试中至少成功一次"
- pass@1：首次尝试成功率
- pass@3：3 次尝试内成功率
- 典型目标：pass@3 > 90%
pass^k
"所有 k 次试验都成功"
- 更高的可靠性门槛
- pass^3：连续 3 次成功
- 用于关键路径
评估工作流程
1. 定义（编码前）
## 评估定义：功能-xyz
### 能力评估
1. 可以创建新用户账户
2. 可以验证电子邮件格式
3. 可以安全地哈希密码
### 回归评估
1. 现有登录功能仍然有效
2. 会话管理未改变
3. 注销流程完整
### 成功指标
- 能力评估的 pass@3 > 90%
- 回归评估的 pass^3 = 100%
2. 实现
编写代码以通过已定义的评估。
3. 评估
# Run capability evals
[Run each capability eval, record PASS/FAIL]
# Run regression evals
npm test -- --testPathPattern="existing"
# Generate report4. 报告
评估报告：功能-xyz
========================
能力评估：
创建用户： 通过（通过@1）
验证邮箱： 通过（通过@2）
哈希密码： 通过（通过@1）
总计： 3/3 通过
回归评估：
登录流程： 通过
会话管理： 通过
登出流程： 通过
总计： 3/3 通过
指标：
通过@1： 67% (2/3)
通过@3： 100% (3/3)
状态：准备就绪，待审核
集成模式
实施前
/eval define feature-name在 .claude/evals/feature-name.md 处创建评估定义文件
实施过程中
/eval check feature-name运行当前评估并报告状态
实施后
/eval 报告 功能名称生成完整的评估报告
评估存储
将评估存储在项目中：
.claude/
evals/
feature-xyz.md # Eval定义
feature-xyz.log # Eval运行历史
baseline.json # 回归基线最佳实践
- 在编码前定义评估 - 强制清晰地思考成功标准
- 频繁运行评估 - 及早发现回归问题
- 随时间跟踪 pass@k - 监控可靠性趋势
- 尽可能使用代码评分器 - 确定性 > 概率性
- 对安全性进行人工审查 - 永远不要完全自动化安全检查
- 保持评估快速 - 缓慢的评估不会被运行
- 评估与代码版本化 - 评估是一等工件
示例：添加身份验证
## EVAL：添加身份验证
### 第 1 阶段：定义 (10 分钟)
能力评估：
- [ ] 用户可以使用邮箱/密码注册
- [ ] 用户可以使用有效凭证登录
- [ ] 无效凭证被拒绝并显示适当的错误
- [ ] 会话在页面重新加载后保持
- [ ] 登出操作清除会话
回归评估：
- [ ] 公共路由仍可访问
- [ ] API 响应未改变
- [ ] 数据库模式兼容
### 第 2 阶段：实施 (时间不定)
[编写代码]
### 第 3 阶段：评估
运行：/eval check add-authentication
### 第 4 阶段：报告
评估报告：添加身份验证
==============================
能力：5/5 通过 (pass@3: 100%)
回归：3/3 通过 (pass^3: 100%)
状态：可以发布
产品评估 (v1.8)
当单元测试无法单独捕获行为质量时，使用产品评估。
评分器类型
- 代码评分器（确定性断言）
- 规则评分器（正则表达式/模式约束）
- 模型评分器（LLM 作为评判者的评估准则）
- 人工评分器（针对模糊输出的人工裁定）
pass@k 指南
pass@1：直接可靠性pass@3：受控重试下的实际可靠性pass^3：稳定性测试（所有 3 次运行必须通过）
推荐阈值：
- 能力评估：pass@3 >= 0.90
- 回归评估：对于发布关键路径，pass^3 = 1.00
评估反模式
- 将提示过度拟合到已知的评估示例
- 仅测量正常路径输出
- 在追求通过率时忽略成本和延迟漂移
- 在发布关卡中允许不稳定的评分器
最小评估工件布局
.claude/evals/<feature>.md定义.claude/evals/<feature>.log运行历史docs/releases/<version>/eval-summary.md发布快照
© affaan-m, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Files
Just SKILL.md in docs/zh-CN/skills/eval-harness of affaan-m/ECC.
Open the folder on GitHubat commit 2d515e4
Used in 3 other repositories
We found 3 copies of this SKILL.md (exact, near-identical or edited) in other folders, from 3 other GitHub owners. This page covers the copy in affaan-m/ECC, which our catalogue first saw on October 7, 2026.
Compare with similar skills
Eval Harness next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Eval Harness this skillaffaan-m/ECC | 277k | 3 repos | ~916 | Automated safety check: Pass | MIT | |
| LLM Benchmarking with lm-evaluation-harnessOrchestra-Research/AI-Research-SKILLs | 13k | 8 repos | ~3k | Automated safety check: Pass | MIT | |
| Hugging Face Local Model Evalshuggingface/skills | 11k | 2 repos | ~1.6k | Automated safety check: Pass | Apache-2.0 | |
| Looperksimback/looper | 710 | — | ~2.7k | Automated safety check: Notes | MIT | |
| Agent Eval Engineeringlangchain-ai/langchain-skills | 1.3k | — | ~4k | Automated safety check: Pass | MIT | |
| Quality FlywheelGoogleCloudPlatform/vertex-ai-samples | 792 | — | ~2k | Automated safety check: Pass | Apache-2.0 |
Similar skills
- Official
Hugging Face Local Model Evals
huggingface/skills
Runs evaluations of Hugging Face Hub models on local hardware with inspect-ai or lighteval, and helps choose between vLLM, Transformers and accelerate backends.
Looper
ksimback/looper
Scaffold a well-designed agent loop with best-practice coaching and a cross-model review council.
- Official
Agent Eval Engineering
langchain-ai/langchain-skills
Builds agent evaluations in stages: inspect the repository and traces, agree a Task Spec with you, then build, audit and run a Harbor task with an independent verifier.
Quality Flywheel
GoogleCloudPlatform/vertex-ai-samples
Evaluate and improve GenAI models and agents using the Google GenAI Evaluation SDK.
Eval Harness
cloudnative-co/claude-code-starter-kit
Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles.
More from affaan-m/ECC
All 682 skills in this repoSkill Stocktake
affaan-m/ECC
Audits your installed Claude skills and commands for quality, with a quick mode for recently changed skills and a full mode that evaluates all of them through subagents.
VideoDB Video Search and Editing
affaan-m/ECC
Ingests, indexes, searches, edits and monitors video, audio and live streams through the VideoDB Python SDK, returning stream links, clips and timestamps.
Docs Governance
affaan-m/ECC
Route broad documentation-governance requests to existing ECC skills and run an opt-in, read-only audit of mapped documentation roles, links, ADR indexes, and evidence references.
Rules Distillation
affaan-m/ECC
Scans installed skills for principles that recur across them and proposes rule-file changes: append, revise, add a section, create a file or leave as covered.
Master Agreement Generator
affaan-m/ECC
Builds DRAFT counterparty agreements from one markdown template and a small JSON spec per party, with clauses picked by the party's role.
Frontend Design Direction
affaan-m/ECC
Set an ECC-specific frontend design direction for production UI work.
Related
Categories
Questions about Eval Harness
How do I install Eval Harness in Claude Code?
Run `npx skills add affaan-m/ECC --skill eval-harness -a claude-code`. Or copy the skill folder (docs/zh-CN/skills/eval-harness in affaan-m/ECC) into .claude/skills/eval-harness in your project. Claude Code loads it when a task matches its description.
How do I install Eval Harness in Codex?
Run `npx skills add affaan-m/ECC --skill eval-harness -a codex`. Or copy the skill folder (docs/zh-CN/skills/eval-harness in affaan-m/ECC) into .agents/skills/eval-harness in your project. Codex loads it when a task matches its description.
Can I use Eval Harness in Cursor, Gemini CLI or GitHub Copilot?
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add affaan-m/ECC --skill eval-harness -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/eval-harness, .gemini/skills/eval-harness, .github/skills/eval-harness and .opencode/skills/eval-harness in your project.
What does Eval Harness need to run?
Going by SKILL.md and its folder, Eval Harness needs the command-line tools its instructions call (npm).
Does Eval Harness access the network?
SKILL.md contains no URLs. Its commands use npm, which can reach the network depending on how they are called. This is read from the text; nothing was executed.
Is Eval Harness safe to install?
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
What licence does Eval Harness use?
Eval Harness is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
How many tokens does Eval Harness use?
About 916 tokens (SKILL.md is roughly 3.7k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
What are the alternatives to Eval Harness?
Skills that share tags, products or a category with Eval Harness: LLM Benchmarking with lm-evaluation-harness (Orchestra-Research/AI-Research-SKILLs, 13k stars), Hugging Face Local Model Evals (huggingface/skills, 11k stars), Looper (ksimback/looper, 710 stars) and Agent Eval Engineering (langchain-ai/langchain-skills, 1.3k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
Who maintains Eval Harness?
affaan-m (a GitHub user) maintains it in affaan-m/ECC, which has 276,673 GitHub stars. The repository holds 683 skills in this directory. The repository was last updated on October 11, 2026.
Source: affaan-m/ECC on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.