LLM Benchmarking with lm-evaluation-harness
Orchestra-Research/AI-Research-SKILLs
Runs lm-evaluation-harness to benchmark language models on academic suites such as MMLU, GSM8K and HumanEval, compare models and track training checkpoints.
Benchmark AI models across 60+ academic evaluation suites and metrics
Install AI Model Benchmarking
$ npx skills add wentorai/research-plugins --skill ai-model-benchmarking -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install wentorai/research-plugins ai-model-benchmarking --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/wentorai/research-plugins.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/domains/ai-ml/ai-model-benchmarking .claude/skills/ai-model-benchmarking && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "ai-model-benchmarking" agent skill from https://github.com/wentorai/research-plugins/tree/main/skills/domains/ai-ml/ai-model-benchmarking into .claude/skills/ai-model-benchmarking/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "ai-model-benchmarking", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/wentorai/research-plugins/tree/main/skills/domains/ai-ml/ai-model-benchmarkingType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add wentorai/research-plugins --skill ai-model-benchmarking -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install wentorai/research-plugins ai-model-benchmarking --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/wentorai/research-plugins.git skills-src && mkdir -p .agents/skills && cp -r skills-src/skills/domains/ai-ml/ai-model-benchmarking .agents/skills/ai-model-benchmarking && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "ai-model-benchmarking" agent skill from https://github.com/wentorai/research-plugins/tree/main/skills/domains/ai-ml/ai-model-benchmarking into .agents/skills/ai-model-benchmarking/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "ai-model-benchmarking", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add wentorai/research-plugins --skill ai-model-benchmarking -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install wentorai/research-plugins ai-model-benchmarking --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/wentorai/research-plugins.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/skills/domains/ai-ml/ai-model-benchmarking .cursor/skills/ai-model-benchmarking && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "ai-model-benchmarking" agent skill from https://github.com/wentorai/research-plugins/tree/main/skills/domains/ai-ml/ai-model-benchmarking into .cursor/skills/ai-model-benchmarking/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "ai-model-benchmarking", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/wentorai/research-plugins.git --path skills/domains/ai-ml/ai-model-benchmarking--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add wentorai/research-plugins --skill ai-model-benchmarking -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install wentorai/research-plugins ai-model-benchmarking --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/wentorai/research-plugins.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/skills/domains/ai-ml/ai-model-benchmarking .gemini/skills/ai-model-benchmarking && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "ai-model-benchmarking" agent skill from https://github.com/wentorai/research-plugins/tree/main/skills/domains/ai-ml/ai-model-benchmarking into .gemini/skills/ai-model-benchmarking/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "ai-model-benchmarking", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install wentorai/research-plugins ai-model-benchmarkingInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add wentorai/research-plugins --skill ai-model-benchmarking -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/wentorai/research-plugins.git skills-src && mkdir -p .github/skills && cp -r skills-src/skills/domains/ai-ml/ai-model-benchmarking .github/skills/ai-model-benchmarking && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "ai-model-benchmarking" agent skill from https://github.com/wentorai/research-plugins/tree/main/skills/domains/ai-ml/ai-model-benchmarking into .github/skills/ai-model-benchmarking/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "ai-model-benchmarking", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add wentorai/research-plugins --skill ai-model-benchmarking -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install wentorai/research-plugins ai-model-benchmarking --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/wentorai/research-plugins.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/skills/domains/ai-ml/ai-model-benchmarking .opencode/skills/ai-model-benchmarking && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "ai-model-benchmarking" agent skill from https://github.com/wentorai/research-plugins/tree/main/skills/domains/ai-ml/ai-model-benchmarking into .opencode/skills/ai-model-benchmarking/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "ai-model-benchmarking", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
Facts
- Skill name
ai-model-benchmarking- GitHub stars
- 298
- Used in
- 1 other repo
- Token cost
- ~2k tokens
- SKILL.md length
- 457 words
- Files
- 1
- Skills in repo
- 405
- Repo updated
- First seen
- Licence
- MIT
At a glance
Benchmark AI models across 60+ academic evaluation suites and metrics
- Tasks that involve LLM evaluation
- SKILL.md covers Overview, The lm-evaluation-harness, Benchmark Selection by… and Core Benchmarks Deep Dive, plus 4 more sections
- Calls pip
What it does
AI Model Benchmarking is an agent skill from wentorai/research-plugins. Benchmark AI models across 60+ academic evaluation suites and metrics
Its SKILL.md is about 2k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.
It sits in AI & LLM Engineering, covering LLM evaluation. The repository describes itself as: 350+ academic research skills, MCP configs, and plugins for Research-Claw and AI agents. The licence is MIT.
When your agent uses it
- Tasks that involve LLM evaluation
Example prompts
- “/ai-model-benchmarking”
Requirements
- Python 3
What it can do on your machine
Read from SKILL.md and the folder at commit bf44b3c. It shows what the files ask for, not the result of running them.
Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
Shell commands in SKILL.md call:
pip
From the folder's file list and the shell code blocks in SKILL.md.
Network
Links to these hosts (documentation or services it may open):
github.comhuggingface.coarxiv.orgcrfm.stanford.educhat.lmsys.org
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Context cost
AI Model Benchmarking loads about 2k tokens when it runs. Until then it costs about 23 tokens; SKILL.md has 457 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
Safety
Auto-check passed
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
SKILL.md
The full file from wentorai/research-plugins at commit bf44b3c, republished under its MIT licence (© wentorai). 457 words, ~2,030 tokens.
Download SKILL.mdSave it as
.claude/skills/ai-model-benchmarking/SKILL.md (or your agent's skills folder).- name
- ai-model-benchmarking
- description
- Benchmark AI models across 60+ academic evaluation suites and metrics
AI Model Benchmarking Guide
Overview
Rigorous evaluation is the backbone of machine learning research. A model is only as credible as its evaluation protocol: which benchmarks were used, how metrics were computed, whether results are reproducible, and how they compare to baselines. The proliferation of LLMs has made this both more important and more complex, with over 60 established benchmarks and a rapidly evolving landscape.
This guide covers the practical side of model benchmarking: how to use the EleutherAI Language Model Evaluation Harness (lm-evaluation-harness), how to select benchmarks for different research claims, how to avoid common evaluation pitfalls, and how to present results for publication. The focus is on academic rigor rather than leaderboard chasing.
Whether you are evaluating a fine-tuned model for a paper, comparing architectures for an ablation study, or reviewing a submitted manuscript's evaluation section, these patterns will help ensure the evaluation is sound.
The lm-evaluation-harness
The EleutherAI lm-evaluation-harness is the de facto standard for LLM evaluation in academic research, supporting 60+ tasks and used by most major LLM papers.
Installation and Basic Usage
# Install
pip install lm-eval
# Run a single benchmark
lm_eval --model hf \
--model_args pretrained=meta-llama/Llama-2-7b-hf \
--tasks mmlu \
--batch_size auto \
--output_path results/llama2-7b/
# Run multiple benchmarks
lm_eval --model hf \
--model_args pretrained=meta-llama/Llama-2-7b-hf \
--tasks mmlu,hellaswag,arc_challenge,winogrande,truthfulqa_mc2 \
--batch_size auto \
--num_fewshot 5 \
--output_path results/llama2-7b/Programmatic API
import lm_eval
results = lm_eval.simple_evaluate(
model="hf",
model_args="pretrained=meta-llama/Llama-2-7b-hf",
tasks=["mmlu", "hellaswag", "arc_challenge"],
num_fewshot=5,
batch_size="auto",
device="cuda",
)
# Access results
for task, metrics in results["results"].items():
print(f"{task}: {metrics}")Benchmark Selection by Research Claim
| Research Claim | Required Benchmarks | Why |
|---|---|---|
| General knowledge | MMLU, ARC, TriviaQA | Broad factual coverage |
| Reasoning | GSM8K, BBH, ARC-Challenge | Multi-step logical reasoning |
| Coding | HumanEval, MBPP, DS-1000 | Code generation and understanding |
| Instruction following | MT-Bench, AlpacaEval, IFEval | Open-ended instruction quality |
| Safety | TruthfulQA, ToxiGen, BBQ | Truthfulness, toxicity, bias |
| Multilingual | MGSM, XWinograd, FLORES | Cross-lingual transfer |
| Long context | SCROLLS, LongBench, RULER | Long document understanding |
| Domain-specific | MedQA, LegalBench, SciQ | Professional domain knowledge |
Core Benchmarks Deep Dive
MMLU (Massive Multitask Language Understanding)
- 57 subjects: STEM, humanities, social sciences, professional
- 14,042 questions, multiple choice (4 options)
- Standard: 5-shot evaluation
- Metric: Accuracy (macro-averaged across subjects)
- Citation: Hendrycks et al., 2021
Score interpretation:
< 30%: Below random (model is miscalibrated)
30-40%: Near random (4 choices = 25% baseline)
40-60%: Basic knowledge
60-70%: Strong general knowledge
70-80%: Expert-level for most subjects
> 80%: State-of-the-art (as of 2024)GSM8K (Grade School Math)
- 8,792 grade school math word problems
- Requires multi-step arithmetic reasoning
- Standard: 8-shot chain-of-thought
- Metric: Exact match on final numerical answer
- Citation: Cobbe et al., 2021
Common pitfalls:
- Regex matching for final answer extraction
- Calculator use vs. pure model computation
- Reporting with vs. without chain-of-thoughtHumanEval (Code Generation)
- 164 Python programming problems
- Function signature + docstring -> implementation
- Metric: pass@k (k=1 standard, k=10 and k=100 also reported)
- Citation: Chen et al., 2021
pass@k computation (unbiased estimator):
pass@k = 1 - C(n-c, k) / C(n, k)
where n = total samples, c = correct samplesShow full SKILL.md (193 more words)Show less
Evaluation Pitfalls
| Pitfall | Problem | Solution |
|---|---|---|
| Data contamination | Benchmark data in training set | Use canary strings, report contamination analysis |
| Prompt sensitivity | Results vary with prompt format | Report results across 3+ prompt variants |
| Few-shot selection | Cherry-picked examples boost scores | Use fixed random seed for example selection |
| Metric gaming | Optimizing for specific metrics | Report multiple metrics, include calibration |
| Incomplete reporting | Only showing best results | Report mean and std across seeds |
| Version mismatch | Different benchmark versions | Pin exact dataset version and commit hash |
Contamination Detection
def check_contamination(training_data: list, benchmark_data: list, n: int = 13) -> dict:
"""
Check for n-gram overlap between training data and benchmark.
13-gram overlap is the standard threshold (GPT-4 technical report).
"""
from collections import defaultdict
def extract_ngrams(text, n):
words = text.lower().split()
return set(tuple(words[i:i+n]) for i in range(len(words) - n + 1))
# Build training n-gram index
train_ngrams = set()
for text in training_data:
train_ngrams.update(extract_ngrams(text, n))
# Check benchmark items
contaminated = []
for i, item in enumerate(benchmark_data):
item_ngrams = extract_ngrams(item, n)
overlap = item_ngrams & train_ngrams
if overlap:
contaminated.append({
"index": i,
"overlap_count": len(overlap),
"overlap_ratio": len(overlap) / max(len(item_ngrams), 1),
})
return {
"total_items": len(benchmark_data),
"contaminated_items": len(contaminated),
"contamination_rate": len(contaminated) / len(benchmark_data),
"details": contaminated,
}Reporting Results for Publication
Standard Results Table Format
| Model | Params | MMLU | GSM8K | HumanEval | ARC-C | HellaSwag | Avg |
|-------|--------|------|-------|-----------|-------|-----------|-----|
| Baseline | 7B | 45.2 | 12.3 | 15.8 | 42.1 | 72.3 | 37.5 |
| Ours | 7B | 52.1 (+6.9) | 28.7 (+16.4) | 22.0 (+6.2) | 48.9 (+6.8) | 76.1 (+3.8) | 45.6 |
| Ours (ablation A) | 7B | 49.8 | 24.1 | 19.5 | 46.2 | 74.8 | 42.9 |
All results: 5-shot for MMLU, 8-shot CoT for GSM8K, 0-shot for HumanEval,
25-shot for ARC-C, 10-shot for HellaSwag. Mean of 3 seeds reported.Best Practices
- Always report the exact evaluation framework version (e.g.,
lm-eval v0.4.2). - Use the same number of few-shot examples as the original benchmark paper.
- Report standard deviations across at least 3 random seeds.
- Include a contamination analysis for any new model trained on web data.
- Compare against published numbers using the same evaluation code -- do not mix results from different frameworks.
- Report inference details: precision (fp16/bf16/int8), context length, decoding strategy.
References
- lm-evaluation-harness -- EleutherAI's evaluation framework (12,000+ stars)
- Open LLM Leaderboard -- Hugging Face community leaderboard
- MMLU paper -- Hendrycks et al., 2021
- Holistic Evaluation of Language Models (HELM) -- Stanford CRFM
- Chatbot Arena -- Human preference-based evaluation (LMSYS)
© wentorai, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Files
Just SKILL.md in skills/domains/ai-ml/ai-model-benchmarking of wentorai/research-plugins.
Open the folder on GitHubat commit bf44b3c
Used in 1 other repository
We found 1 copy of this SKILL.md (exact, near-identical or edited) in other folders, from 1 other GitHub owner. This page covers the copy in wentorai/research-plugins, which our catalogue first saw on October 7, 2026.
Compare with similar skills
AI Model Benchmarking next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| AI Model Benchmarking this skillwentorai/research-plugins | 298 | 1 repos | ~2k | Automated safety check: Pass | MIT | |
| LLM Benchmarking with lm-evaluation-harnessOrchestra-Research/AI-Research-SKILLs | 13k | 8 repos | ~3k | Automated safety check: Pass | MIT | |
| Hugging Face Local Model Evalshuggingface/skills | 11k | 2 repos | ~1.6k | Automated safety check: Pass | Apache-2.0 | |
| Looperksimback/looper | 710 | — | ~2.7k | Automated safety check: Notes | MIT | |
| Agent Eval Engineeringlangchain-ai/langchain-skills | 1.3k | — | ~4k | Automated safety check: Pass | MIT | |
| Quality FlywheelGoogleCloudPlatform/vertex-ai-samples | 792 | — | ~2k | Automated safety check: Pass | Apache-2.0 |
Similar skills
- Official
Hugging Face Local Model Evals
huggingface/skills
Runs evaluations of Hugging Face Hub models on local hardware with inspect-ai or lighteval, and helps choose between vLLM, Transformers and accelerate backends.
Looper
ksimback/looper
Scaffold a well-designed agent loop with best-practice coaching and a cross-model review council.
- Official
Agent Eval Engineering
langchain-ai/langchain-skills
Builds agent evaluations in stages: inspect the repository and traces, agree a Task Spec with you, then build, audit and run a Harbor task with an independent verifier.
Quality Flywheel
GoogleCloudPlatform/vertex-ai-samples
Evaluate and improve GenAI models and agents using the Google GenAI Evaluation SDK.
Eval Harness
cloudnative-co/claude-code-starter-kit
Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles.
More from wentorai/research-plugins
All 405 skills in this repoAbstract Writing Guide
wentorai/research-plugins
Craft structured research abstracts that maximize clarity and journal acceptance
Academic Citation Manager
wentorai/research-plugins
Manage academic citations across BibTeX, APA, MLA, and Chicago formats
Academic Paper Summarizer
wentorai/research-plugins
Summarize academic papers with structured extraction of key elements
Academic Study Methods
wentorai/research-plugins
Evidence-based study techniques for academic learning and retention
Academic Tone Guide
wentorai/research-plugins
Adjust writing tone and register for academic audiences and venues
Academic Translation Guide
wentorai/research-plugins
Academic translation, post-editing, and Chinglish correction guide
Related
Categories
Questions about AI Model Benchmarking
What does AI Model Benchmarking do?
Benchmark AI models across 60+ academic evaluation suites and metrics. AI Model Benchmarking is an agent skill from wentorai/research-plugins.
When should I use AI Model Benchmarking?
AI Model Benchmarking fits situations like: tasks that involve LLM evaluation.
How do I install AI Model Benchmarking in Claude Code?
Run `npx skills add wentorai/research-plugins --skill ai-model-benchmarking -a claude-code`. Or copy the skill folder (skills/domains/ai-ml/ai-model-benchmarking in wentorai/research-plugins) into .claude/skills/ai-model-benchmarking in your project. Claude Code loads it when a task matches its description.
How do I install AI Model Benchmarking in Codex?
Run `npx skills add wentorai/research-plugins --skill ai-model-benchmarking -a codex`. Or copy the skill folder (skills/domains/ai-ml/ai-model-benchmarking in wentorai/research-plugins) into .agents/skills/ai-model-benchmarking in your project. Codex loads it when a task matches its description.
Can I use AI Model Benchmarking in Cursor, Gemini CLI or GitHub Copilot?
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add wentorai/research-plugins --skill ai-model-benchmarking -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/ai-model-benchmarking, .gemini/skills/ai-model-benchmarking, .github/skills/ai-model-benchmarking and .opencode/skills/ai-model-benchmarking in your project.
What does AI Model Benchmarking need to run?
Going by SKILL.md and its folder, AI Model Benchmarking needs the command-line tools its instructions call (pip). Our summary lists: Python 3.
Does AI Model Benchmarking access the network?
SKILL.md names 5 domains. As links in the text: github.com, huggingface.co, arxiv.org, crfm.stanford.edu and chat.lmsys.org. This is read from the text; nothing was executed.
Is AI Model Benchmarking safe to install?
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
What licence does AI Model Benchmarking use?
AI Model Benchmarking is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
How many tokens does AI Model Benchmarking use?
About 2k tokens (SKILL.md is roughly 8.1k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
What are the alternatives to AI Model Benchmarking?
Skills that share tags, products or a category with AI Model Benchmarking: LLM Benchmarking with lm-evaluation-harness (Orchestra-Research/AI-Research-SKILLs, 13k stars), Hugging Face Local Model Evals (huggingface/skills, 11k stars), Looper (ksimback/looper, 710 stars) and Agent Eval Engineering (langchain-ai/langchain-skills, 1.3k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
Who maintains AI Model Benchmarking?
wentorai (a GitHub user) maintains it in wentorai/research-plugins, which has 298 GitHub stars. The repository holds 405 skills in this directory. The repository was last updated on June 19, 2026.
Source: wentorai/research-plugins on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.