LLM Benchmarking with lm-evaluation-harness
Orchestra-Research/AI-Research-SKILLs
Runs lm-evaluation-harness to benchmark language models on academic suites such as MMLU, GSM8K and HumanEval, compare models and track training checkpoints.
평가 주도 개발(EDD) 원칙을 구현하는 Claude Code 세션용 공식 평가 프레임워크. An agent skill from affaan-m/ECC.
Install Eval Harness
$ npx skills add affaan-m/ECC --skill eval-harness -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install affaan-m/ECC eval-harness --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/affaan-m/ECC.git skills-src && mkdir -p .claude/skills && cp -r skills-src/docs/ko-KR/skills/eval-harness .claude/skills/eval-harness && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "eval-harness" agent skill from https://github.com/affaan-m/ECC/tree/main/docs/ko-KR/skills/eval-harness into .claude/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/affaan-m/ECC/tree/main/docs/ko-KR/skills/eval-harnessType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add affaan-m/ECC --skill eval-harness -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install affaan-m/ECC eval-harness --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/affaan-m/ECC.git skills-src && mkdir -p .agents/skills && cp -r skills-src/docs/ko-KR/skills/eval-harness .agents/skills/eval-harness && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "eval-harness" agent skill from https://github.com/affaan-m/ECC/tree/main/docs/ko-KR/skills/eval-harness into .agents/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add affaan-m/ECC --skill eval-harness -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install affaan-m/ECC eval-harness --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/affaan-m/ECC.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/docs/ko-KR/skills/eval-harness .cursor/skills/eval-harness && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "eval-harness" agent skill from https://github.com/affaan-m/ECC/tree/main/docs/ko-KR/skills/eval-harness into .cursor/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/affaan-m/ECC.git --path docs/ko-KR/skills/eval-harness--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add affaan-m/ECC --skill eval-harness -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install affaan-m/ECC eval-harness --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/affaan-m/ECC.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/docs/ko-KR/skills/eval-harness .gemini/skills/eval-harness && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "eval-harness" agent skill from https://github.com/affaan-m/ECC/tree/main/docs/ko-KR/skills/eval-harness into .gemini/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install affaan-m/ECC eval-harnessInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add affaan-m/ECC --skill eval-harness -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/affaan-m/ECC.git skills-src && mkdir -p .github/skills && cp -r skills-src/docs/ko-KR/skills/eval-harness .github/skills/eval-harness && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "eval-harness" agent skill from https://github.com/affaan-m/ECC/tree/main/docs/ko-KR/skills/eval-harness into .github/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add affaan-m/ECC --skill eval-harness -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install affaan-m/ECC eval-harness --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/affaan-m/ECC.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/docs/ko-KR/skills/eval-harness .opencode/skills/eval-harness && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "eval-harness" agent skill from https://github.com/affaan-m/ECC/tree/main/docs/ko-KR/skills/eval-harness into .opencode/skills/eval-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "eval-harness", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
Facts
- Skill name
eval-harness- GitHub stars
- 277k
- Used in
- 2 other repos
- Token cost
- ~1.2k tokens
- SKILL.md length
- 393 words
- Files
- 1
- Skills in repo
- 683
- Repo updated
- First seen
- Licence
- MIT
At a glance
평가 주도 개발(EDD) 원칙을 구현하는 Claude Code 세션용 공식 평가 프레임워크. An agent skill from affaan-m/ECC.
- Works in 7 steps: 코드 기반 채점자 → 모델 기반 채점자 → 사람 채점자 → …
- Tasks that involve LLM evaluation
- SKILL.md covers 활성화 시점, 철학, 평가 유형 and 채점자 유형, plus 7 more sections
- Calls npm
What it does
Eval Harness is an agent skill from affaan-m/ECC. 평가 주도 개발(EDD) 원칙을 구현하는 Claude Code 세션용 공식 평가 프레임워크
Its SKILL.md is about 1.2k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.
It sits in AI & LLM Engineering, covering LLM evaluation. The repository describes itself as: The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond. The licence is MIT.
When your agent uses it
- Tasks that involve LLM evaluation
Example prompts
- “/eval-harness”
Workflow steps
7 steps, taken from the step headings in SKILL.md.
What it can do on your machine
Read from SKILL.md and the folder at commit 2d515e4. It shows what the files ask for, not the result of running them.
Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
Shell commands in SKILL.md call:
npm
From the folder's file list and the shell code blocks in SKILL.md.
Network
No URLs in SKILL.md. Its commands use npm, which can reach the network depending on how they are called.
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Context cost
Eval Harness loads about 1.2k tokens when it runs. Until then it costs about 16 tokens; SKILL.md has 393 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
Safety
Auto-check passed
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
SKILL.md
The full file from affaan-m/ECC at commit 2d515e4, republished under its MIT licence (© affaan-m). 393 words, ~1,247 tokens.
Download SKILL.mdSave it as
.claude/skills/eval-harness/SKILL.md (or your agent's skills folder).- name
- eval-harness
- description
- 평가 주도 개발(EDD) 원칙을 구현하는 Claude Code 세션용 공식 평가 프레임워크
- origin
- ECC
- tools
- Read, Write, Edit, Bash, Grep, Glob
평가 하네스 스킬
Claude Code 세션을 위한 공식 평가 프레임워크로, 평가 주도 개발(EDD) 원칙을 구현합니다.
활성화 시점
- AI 지원 워크플로우에 평가 주도 개발(EDD) 설정 시
- Claude Code 작업 완료에 대한 합격/불합격 기준 정의 시
- pass@k 메트릭으로 에이전트 신뢰성 측정 시
- 프롬프트 또는 에이전트 변경에 대한 회귀 테스트 스위트 생성 시
- 모델 버전 간 에이전트 성능 벤치마킹 시
철학
평가 주도 개발은 평가를 "AI 개발의 단위 테스트"로 취급합니다:
- 구현 전에 예상 동작 정의
- 개발 중 지속적으로 평가 실행
- 각 변경 시 회귀 추적
- 신뢰성 측정을 위해 pass@k 메트릭 사용
평가 유형
기능 평가
Claude가 이전에 할 수 없었던 것을 할 수 있는지 테스트:
[CAPABILITY EVAL: feature-name]
Task: Description of what Claude should accomplish
Success Criteria:
- [ ] Criterion 1
- [ ] Criterion 2
- [ ] Criterion 3
Expected Output: Description of expected result회귀 평가
변경 사항이 기존 기능을 손상시키지 않는지 확인:
[REGRESSION EVAL: feature-name]
Baseline: SHA or checkpoint name
Tests:
- existing-test-1: PASS/FAIL
- existing-test-2: PASS/FAIL
- existing-test-3: PASS/FAIL
Result: X/Y passed (previously Y/Y)채점자 유형
1. 코드 기반 채점자
코드를 사용한 결정론적 검사:
# Check if file contains expected pattern
grep -q "export function handleAuth" src/auth.ts && echo "PASS" || echo "FAIL"
# Check if tests pass
npm test -- --testPathPattern="auth" && echo "PASS" || echo "FAIL"
# Check if build succeeds
npm run build && echo "PASS" || echo "FAIL"2. 모델 기반 채점자
Claude를 사용하여 개방형 출력 평가:
[MODEL GRADER PROMPT]
Evaluate the following code change:
1. Does it solve the stated problem?
2. Is it well-structured?
3. Are edge cases handled?
4. Is error handling appropriate?
Score: 1-5 (1=poor, 5=excellent)
Reasoning: [explanation]3. 사람 채점자
수동 검토 플래그:
[HUMAN REVIEW REQUIRED]
Change: Description of what changed
Reason: Why human review is needed
Risk Level: LOW/MEDIUM/HIGH메트릭
pass@k
"k번 시도 중 최소 한 번 성공"
- pass@1: 첫 번째 시도 성공률
- pass@3: 3번 시도 내 성공
- 일반적인 목표: pass@3 > 90%
pass^k
"k번 시행 모두 성공"
- 신뢰성에 대한 더 높은 기준
- pass^3: 3회 연속 성공
- 핵심 경로에 사용
평가 워크플로우
1. 정의 (코딩 전)
## EVAL DEFINITION: feature-xyz
### Capability Evals
1. Can create new user account
2. Can validate email format
3. Can hash password securely
### Regression Evals
1. Existing login still works
2. Session management unchanged
3. Logout flow intact
### Success Metrics
- pass@3 > 90% for capability evals
- pass^3 = 100% for regression evals2. 구현
정의된 평가를 통과하기 위한 코드 작성.
3. 평가
# Run capability evals
[Run each capability eval, record PASS/FAIL]
# Run regression evals
npm test -- --testPathPattern="existing"
# Generate report4. 보고서
EVAL REPORT: feature-xyz
========================
Capability Evals:
create-user: PASS (pass@1)
validate-email: PASS (pass@2)
hash-password: PASS (pass@1)
Overall: 3/3 passed
Regression Evals:
login-flow: PASS
session-mgmt: PASS
logout-flow: PASS
Overall: 3/3 passed
Metrics:
pass@1: 67% (2/3)
pass@3: 100% (3/3)
Status: READY FOR REVIEW통합 패턴
구현 전
/eval define feature-name.claude/evals/feature-name.md에 평가 정의 파일 생성
구현 중
/eval check feature-name현재 평가를 실행하고 상태 보고
구현 후
/eval report feature-name전체 평가 보고서 생성
평가 저장소
프로젝트에 평가 저장:
.claude/
evals/
feature-xyz.md # 평가 정의
feature-xyz.log # 평가 실행 이력
baseline.json # 회귀 베이스라인Show full SKILL.md (172 more words)Show less
모범 사례
- 코딩 전에 평가 정의 - 성공 기준에 대한 명확한 사고를 강제
- 자주 평가 실행 - 회귀를 조기에 포착
- 시간에 따른 pass@k 추적 - 신뢰성 추세 모니터링
- 가능하면 코드 채점자 사용 - 결정론적 > 확률적
- 보안에는 사람 검토 - 보안 검사를 완전히 자동화하지 말 것
- 평가를 빠르게 유지 - 느린 평가는 실행되지 않음
- 코드와 함께 평가 버전 관리 - 평가는 일급 산출물
예시: 인증 추가
## EVAL: add-authentication
### Phase 1: 정의 (10분)
Capability Evals:
- [ ] User can register with email/password
- [ ] User can login with valid credentials
- [ ] Invalid credentials rejected with proper error
- [ ] Sessions persist across page reloads
- [ ] Logout clears session
Regression Evals:
- [ ] Public routes still accessible
- [ ] API responses unchanged
- [ ] Database schema compatible
### Phase 2: 구현 (가변)
[Write code]
### Phase 3: 평가
Run: /eval check add-authentication
### Phase 4: 보고서
EVAL REPORT: add-authentication
==============================
Capability: 5/5 passed (pass@3: 100%)
Regression: 3/3 passed (pass^3: 100%)
Status: SHIP IT제품 평가 (v1.8)
행동 품질을 단위 테스트만으로 포착할 수 없을 때 제품 평가를 사용하세요.
채점자 유형
- 코드 채점자 (결정론적 어서션)
- 규칙 채점자 (정규식/스키마 제약 조건)
- 모델 채점자 (LLM 심사위원 루브릭)
- 사람 채점자 (모호한 출력에 대한 수동 판정)
pass@k 가이드
pass@1: 직접 신뢰성pass@3: 제어된 재시도 하에서의 실용적 신뢰성pass^3: 안정성 테스트 (3회 모두 통과해야 함)
권장 임계값:
- 기능 평가: pass@3 >= 0.90
- 회귀 평가: 릴리스 핵심 경로에 pass^3 = 1.00
평가 안티패턴
- 알려진 평가 예시에 프롬프트 과적합
- 정상 경로 출력만 측정
- 합격률을 쫓으면서 비용과 지연 시간 변동 무시
- 릴리스 게이트에 불안정한 채점자 허용
최소 평가 산출물 레이아웃
.claude/evals/<feature>.md정의.claude/evals/<feature>.log실행 이력docs/releases/<version>/eval-summary.md릴리스 스냅샷
© affaan-m, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Files
Just SKILL.md in docs/ko-KR/skills/eval-harness of affaan-m/ECC.
Open the folder on GitHubat commit 2d515e4
Used in 2 other repositories
We found 2 copies of this SKILL.md (exact, near-identical or edited) in other folders, from 2 other GitHub owners. This page covers the copy in affaan-m/ECC, which our catalogue first saw on October 7, 2026.
Compare with similar skills
Eval Harness next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Eval Harness this skillaffaan-m/ECC | 277k | 2 repos | ~1.2k | Automated safety check: Pass | MIT | |
| LLM Benchmarking with lm-evaluation-harnessOrchestra-Research/AI-Research-SKILLs | 13k | 8 repos | ~3k | Automated safety check: Pass | MIT | |
| Hugging Face Local Model Evalshuggingface/skills | 11k | 2 repos | ~1.6k | Automated safety check: Pass | Apache-2.0 | |
| Looperksimback/looper | 710 | — | ~2.7k | Automated safety check: Notes | MIT | |
| Agent Eval Engineeringlangchain-ai/langchain-skills | 1.3k | — | ~4k | Automated safety check: Pass | MIT | |
| Quality FlywheelGoogleCloudPlatform/vertex-ai-samples | 792 | — | ~2k | Automated safety check: Pass | Apache-2.0 |
Similar skills
- Official
Hugging Face Local Model Evals
huggingface/skills
Runs evaluations of Hugging Face Hub models on local hardware with inspect-ai or lighteval, and helps choose between vLLM, Transformers and accelerate backends.
Looper
ksimback/looper
Scaffold a well-designed agent loop with best-practice coaching and a cross-model review council.
- Official
Agent Eval Engineering
langchain-ai/langchain-skills
Builds agent evaluations in stages: inspect the repository and traces, agree a Task Spec with you, then build, audit and run a Harbor task with an independent verifier.
Quality Flywheel
GoogleCloudPlatform/vertex-ai-samples
Evaluate and improve GenAI models and agents using the Google GenAI Evaluation SDK.
Eval Harness
cloudnative-co/claude-code-starter-kit
Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles.
More from affaan-m/ECC
All 682 skills in this repoSkill Stocktake
affaan-m/ECC
Audits your installed Claude skills and commands for quality, with a quick mode for recently changed skills and a full mode that evaluates all of them through subagents.
VideoDB Video Search and Editing
affaan-m/ECC
Ingests, indexes, searches, edits and monitors video, audio and live streams through the VideoDB Python SDK, returning stream links, clips and timestamps.
Docs Governance
affaan-m/ECC
Route broad documentation-governance requests to existing ECC skills and run an opt-in, read-only audit of mapped documentation roles, links, ADR indexes, and evidence references.
Rules Distillation
affaan-m/ECC
Scans installed skills for principles that recur across them and proposes rule-file changes: append, revise, add a section, create a file or leave as covered.
Master Agreement Generator
affaan-m/ECC
Builds DRAFT counterparty agreements from one markdown template and a small JSON spec per party, with clauses picked by the party's role.
Frontend Design Direction
affaan-m/ECC
Set an ECC-specific frontend design direction for production UI work.
Related
Categories
Questions about Eval Harness
What does Eval Harness do?
평가 주도 개발(EDD) 원칙을 구현하는 Claude Code 세션용 공식 평가 프레임워크. An agent skill from affaan-m/ECC. Eval Harness is an agent skill from affaan-m/ECC.
When should I use Eval Harness?
Eval Harness fits situations like: tasks that involve LLM evaluation.
How do I install Eval Harness in Claude Code?
Run `npx skills add affaan-m/ECC --skill eval-harness -a claude-code`. Or copy the skill folder (docs/ko-KR/skills/eval-harness in affaan-m/ECC) into .claude/skills/eval-harness in your project. Claude Code loads it when a task matches its description.
How do I install Eval Harness in Codex?
Run `npx skills add affaan-m/ECC --skill eval-harness -a codex`. Or copy the skill folder (docs/ko-KR/skills/eval-harness in affaan-m/ECC) into .agents/skills/eval-harness in your project. Codex loads it when a task matches its description.
Can I use Eval Harness in Cursor, Gemini CLI or GitHub Copilot?
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add affaan-m/ECC --skill eval-harness -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/eval-harness, .gemini/skills/eval-harness, .github/skills/eval-harness and .opencode/skills/eval-harness in your project.
What does Eval Harness need to run?
Going by SKILL.md and its folder, Eval Harness needs the command-line tools its instructions call (npm).
Does Eval Harness access the network?
SKILL.md contains no URLs. Its commands use npm, which can reach the network depending on how they are called. This is read from the text; nothing was executed.
Is Eval Harness safe to install?
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
What licence does Eval Harness use?
Eval Harness is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
How many tokens does Eval Harness use?
About 1.2k tokens (SKILL.md is roughly 5k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
What are the alternatives to Eval Harness?
Skills that share tags, products or a category with Eval Harness: LLM Benchmarking with lm-evaluation-harness (Orchestra-Research/AI-Research-SKILLs, 13k stars), Hugging Face Local Model Evals (huggingface/skills, 11k stars), Looper (ksimback/looper, 710 stars) and Agent Eval Engineering (langchain-ai/langchain-skills, 1.3k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
Who maintains Eval Harness?
affaan-m (a GitHub user) maintains it in affaan-m/ECC, which has 276,673 GitHub stars. The repository holds 683 skills in this directory. The repository was last updated on October 11, 2026.
Source: affaan-m/ECC on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.