LLM Benchmarking with lm-evaluation-harness
Orchestra-Research/AI-Research-SKILLs
Runs lm-evaluation-harness to benchmark language models on academic suites such as MMLU, GSM8K and HumanEval, compare models and track training checkpoints.
Investigate a single failing eval from the convex-evals system.
Install Analyze Eval
$ npx skills add get-convex/convex-evals --skill analyze-eval -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install get-convex/convex-evals analyze-eval --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/get-convex/convex-evals.git skills-src && mkdir -p .claude/skills && cp -r skills-src/.cursor/skills/analyze-eval .claude/skills/analyze-eval && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "analyze-eval" agent skill from https://github.com/get-convex/convex-evals/tree/main/.cursor/skills/analyze-eval into .claude/skills/analyze-eval/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "analyze-eval", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/get-convex/convex-evals/tree/main/.cursor/skills/analyze-evalType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add get-convex/convex-evals --skill analyze-eval -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install get-convex/convex-evals analyze-eval --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/get-convex/convex-evals.git skills-src && mkdir -p .agents/skills && cp -r skills-src/.cursor/skills/analyze-eval .agents/skills/analyze-eval && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "analyze-eval" agent skill from https://github.com/get-convex/convex-evals/tree/main/.cursor/skills/analyze-eval into .agents/skills/analyze-eval/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "analyze-eval", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add get-convex/convex-evals --skill analyze-eval -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install get-convex/convex-evals analyze-eval --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/get-convex/convex-evals.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/.cursor/skills/analyze-eval .cursor/skills/analyze-eval && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "analyze-eval" agent skill from https://github.com/get-convex/convex-evals/tree/main/.cursor/skills/analyze-eval into .cursor/skills/analyze-eval/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "analyze-eval", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/get-convex/convex-evals.git --path .cursor/skills/analyze-eval--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add get-convex/convex-evals --skill analyze-eval -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install get-convex/convex-evals analyze-eval --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/get-convex/convex-evals.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/.cursor/skills/analyze-eval .gemini/skills/analyze-eval && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "analyze-eval" agent skill from https://github.com/get-convex/convex-evals/tree/main/.cursor/skills/analyze-eval into .gemini/skills/analyze-eval/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "analyze-eval", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install get-convex/convex-evals analyze-evalInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add get-convex/convex-evals --skill analyze-eval -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/get-convex/convex-evals.git skills-src && mkdir -p .github/skills && cp -r skills-src/.cursor/skills/analyze-eval .github/skills/analyze-eval && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "analyze-eval" agent skill from https://github.com/get-convex/convex-evals/tree/main/.cursor/skills/analyze-eval into .github/skills/analyze-eval/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "analyze-eval", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add get-convex/convex-evals --skill analyze-eval -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install get-convex/convex-evals analyze-eval --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/get-convex/convex-evals.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/.cursor/skills/analyze-eval .opencode/skills/analyze-eval && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "analyze-eval" agent skill from https://github.com/get-convex/convex-evals/tree/main/.cursor/skills/analyze-eval into .opencode/skills/analyze-eval/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "analyze-eval", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
Facts
- Skill name
analyze-eval- GitHub stars
- 129
- Token cost
- ~1.1k tokens
- SKILL.md length
- 512 words
- Files
- 1
- Skills in repo
- 5
- Repo updated
- First seen
- Licence
- Apache-2.0
At a glance
Investigate a single failing eval from the convex-evals system.
- Works in 4 steps: Extract the eval ID from the URL → Fetch the eval → Analyze the failure → …
- The user shares a visualizer URL pointing to a specific eval
- SKILL.md covers When to use, Step 1: Extract the eval ID…, Step 2: Fetch the eval and Step 3: Analyze the failure, plus 1 more section
- Calls curl, jq and npx; reaches convex-evals.netlify.app and fabulous-panther-525.convex.cloud
What it does
Analyze Eval is an agent skill from get-convex/convex-evals. Investigate a single failing eval from the convex-evals system. Use when the user shares a visualizer URL pointing to a specific eval, asks about a specific failing eval, or references a specific eval ID.
Its SKILL.md is about 1.1k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.
It sits in AI & LLM Engineering, covering LLM evaluation. The licence is Apache-2.0.
When your agent uses it
- The user shares a visualizer URL pointing to a specific eval
- Asks about a specific failing eval
- References a specific eval ID
Example prompts
- “/analyze-eval”
Requirements
- Node.js
Workflow steps
4 steps, taken from the step headings in SKILL.md.
What it can do on your machine
Read from SKILL.md and the folder at commit 68f5c0e. It shows what the files ask for, not the result of running them.
Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
Shell commands in SKILL.md call:
curljqnpx
From the folder's file list and the shell code blocks in SKILL.md.
Network
Hosts in commands or code, which the agent is likely to contact:
convex-evals.netlify.appfabulous-panther-525.convex.cloud
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Context cost
Analyze Eval loads about 1.1k tokens when it runs. Until then it costs about 54 tokens; SKILL.md has 512 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
Safety
Auto-check passed
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
SKILL.md
The full file from get-convex/convex-evals at commit 68f5c0e, republished under its Apache-2.0 licence (© get-convex). 512 words, ~1,113 tokens.
Download SKILL.mdSave it as
.claude/skills/analyze-eval/SKILL.md (or your agent's skills folder).- name
- analyze-eval
- description
- Investigate a single failing eval from the convex-evals system. Use when the user shares a visualizer URL pointing to a specific eval, asks about a specific failing eval, or references a specific eval ID.
Analyze Eval
When to use
- User shares a URL like
https://convex-evals.netlify.app/experiment/.../run/$runId/$category/$evalId - User asks "why did this eval fail?" or "what went wrong with this eval?"
- User references a specific eval ID
Step 1: Extract the eval ID from the URL
The visualizer URL pattern is:
/experiment/$experimentId/run/$runId/$category/$evalId?tab=steps$runId— the Convex document ID for the run (e.g.jn7922j1w29pdxm76bj9ps0enx80mg9e)$evalId— the Convex document ID for the specific eval (e.g.jh73jvjz2n00gfeve1dt5h963s80mbc6)
You need the runId and the evalId to query.
Step 2: Fetch the eval
The debug action debug:getEvalDebugInfo is internal, so it needs npx convex run --prod, and agents usually hit team SSO ("Single-sign on login is required"). Use the public production queries over HTTP instead. They need no login. Work in a temp directory:
URL=https://fabulous-panther-525.convex.cloud
curl -s $URL/api/query -H 'Content-Type: application/json' \
-d '{"path":"runs:getRunDetails","args":{"runId":"<runId>"}}' > run.json
jq '.value | {model, provider, experiment, status: .status.kind}' run.json
jq '.value.evals[] | select(._id == "<evalId>")' run.json > eval.jsonGet a download URL for the model output (status.outputStorageId in eval.json) and for the eval source (evalSourceStorageId):
curl -s $URL/api/query -H 'Content-Type: application/json' \
-d '{"path":"runs:getOutputUrl","args":{"storageId":"<storageId>"}}' | jq -r .valueDownload each with curl -s -o, then unzip the output into output/ and the source into source/. That gives you:
| Source | Contents |
|---|---|
eval.json | evalPath, category, name, status (pass/fail + failure reason), task text |
eval.json steps | Array of step results: filesystem, install, deploy, tsc, eslint, tests. Each is passed, failed or skipped, with a failure reason |
| run metadata | Model slug, provider, experiment (null means default), run status |
output/ | The model's generated files |
source/ | The eval source (answer dir, grader, TASK.txt, etc.) |
If you have an eval ID but no run ID, ask for the visualizer URL, or give Mike this command to run and paste back:
cd evalScores && npx convex run --prod debug:getEvalDebugInfo '{"evalId": "<evalId>"}'Show full SKILL.md (270 more words)Show less
Step 3: Analyze the failure
With the data returned, compare:
- Which step failed? Check
stepsfor the first entry withstatus.kind === "failed". ThefailureReasonfield has the error message. - What did the model generate? Look at
output/for the model's code. - What was expected? Look at
source/for the answer directory and grader test files. - What was the task? Check
taskin eval.json for the TASK.txt content.
Common failure patterns:
- eslint fail. Check the failure reason for the specific lint rule violated. Compare the model output against the answer to spot the lint issue.
- tsc fail. TypeScript compilation error. Check the failure reason for the specific type error.
- convex dev fail. Schema or function definition issues that prevent Convex from deploying.
- tests fail. The grader tests didn't pass. Compare
output/againstsource/(look for files likegrader.test.tsoranswer/) to understand what the tests expected.
Step 4: Classify and report findings
Classify the failure as one of:
- MODEL_FAULT: The model genuinely got it wrong
- OVERLY_STRICT: The eval/lint/test requirements are unreasonable for what was asked
- AMBIGUOUS_TASK: The task description is unclear and the model's interpretation was reasonable
- KNOWN_GAP: A known limitation of this eval that affects all models (e.g. the Convex API returns fields the model can't predict without being told)
Summarize:
- The eval name, model, and experiment
- Which step failed and the exact error
- The classification and reasoning
- The relevant code from the model output that caused the failure
- What the correct code should look like (from the answer/eval source)
- Whether any action is recommended (config change, task clarification, etc.)
© get-convex, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Files
Just SKILL.md in .cursor/skills/analyze-eval of get-convex/convex-evals.
Open the folder on GitHubat commit 68f5c0e
Compare with similar skills
Analyze Eval next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Analyze Eval this skillget-convex/convex-evals | 129 | — | ~1.1k | Automated safety check: Pass | Apache-2.0 | |
| LLM Benchmarking with lm-evaluation-harnessOrchestra-Research/AI-Research-SKILLs | 13k | 8 repos | ~3k | Automated safety check: Pass | MIT | |
| Azure AI Projects Python SDKmicrosoft/skills | 3.1k | 6 repos | ~2.8k | Automated safety check: Pass | MIT | |
| Fine-Tuning ExpertJeffallan/claude-skills | 12k | 1 repos | ~1.7k | Automated safety check: Pass | MIT | |
| Looperksimback/looper | 710 | — | ~2.7k | Automated safety check: Notes | MIT | |
| Hugging Face Local Model Evalshuggingface/skills | 11k | 2 repos | ~1.6k | Automated safety check: Pass | Apache-2.0 |
Similar skills
- Official
Azure AI Projects Python SDK
microsoft/skills
Reference for building on Microsoft Foundry with the azure-ai-projects Python SDK: project clients, versioned agents, evaluations, connections, datasets and indexes.
Fine-Tuning Expert
Jeffallan/claude-skills
Guides LLM fine-tuning with LoRA and QLoRA through Hugging Face PEFT, from dataset validation and training checks to adapter merging, quantization and deployment.
Looper
ksimback/looper
Scaffold a well-designed agent loop with best-practice coaching and a cross-model review council.
- Official
Hugging Face Local Model Evals
huggingface/skills
Runs evaluations of Hugging Face Hub models on local hardware with inspect-ai or lighteval, and helps choose between vLLM, Transformers and accelerate backends.
- Official
Agent Eval Engineering
langchain-ai/langchain-skills
Builds agent evaluations in stages: inspect the repository and traces, agree a Task Spec with you, then build, audit and run a Harbor task with an independent verifier.
More from get-convex/convex-evals
Add Eval
get-convex/convex-evals
Design, implement, validate, and calibrate a new eval for the convex-evals suite.
Add Model
get-convex/convex-evals
Add a new model to the convex-evals coding leaderboard, and optionally the decision benchmark, through a PR, then dispatch its baseline runs.
Analyze Run
get-convex/convex-evals
Analyze all failures in a convex-evals run, spawning parallel sub-agents to investigate each failure and producing a report with classifications and recommendations.
Validate Guidelines
get-convex/convex-evals
Empirically verify guideline changes by running before/after eval runs across multiple models and ensuring no regressions.
Related
Categories
Questions about Analyze Eval
What does Analyze Eval do?
Investigate a single failing eval from the convex-evals system. Analyze Eval is an agent skill from get-convex/convex-evals. Investigate a single failing eval from the convex-evals system.
When should I use Analyze Eval?
Analyze Eval fits situations like: the user shares a visualizer URL pointing to a specific eval; asks about a specific failing eval; references a specific eval ID.
How do I install Analyze Eval in Claude Code?
Run `npx skills add get-convex/convex-evals --skill analyze-eval -a claude-code`. Or copy the skill folder (.cursor/skills/analyze-eval in get-convex/convex-evals) into .claude/skills/analyze-eval in your project. Claude Code loads it when a task matches its description.
How do I install Analyze Eval in Codex?
Run `npx skills add get-convex/convex-evals --skill analyze-eval -a codex`. Or copy the skill folder (.cursor/skills/analyze-eval in get-convex/convex-evals) into .agents/skills/analyze-eval in your project. Codex loads it when a task matches its description.
Can I use Analyze Eval in Cursor, Gemini CLI or GitHub Copilot?
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add get-convex/convex-evals --skill analyze-eval -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/analyze-eval, .gemini/skills/analyze-eval, .github/skills/analyze-eval and .opencode/skills/analyze-eval in your project.
What does Analyze Eval need to run?
Going by SKILL.md and its folder, Analyze Eval needs the command-line tools its instructions call (curl, jq and npx). Our summary lists: Node.js.
Does Analyze Eval access the network?
SKILL.md names 2 domains. In commands or code: convex-evals.netlify.app and fabulous-panther-525.convex.cloud; the agent is likely to contact these when it follows the instructions. This is read from the text; nothing was executed.
Is Analyze Eval safe to install?
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
What licence does Analyze Eval use?
Analyze Eval is published under the Apache-2.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
How many tokens does Analyze Eval use?
About 1.1k tokens (SKILL.md is roughly 4.5k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
What are the alternatives to Analyze Eval?
Skills that share tags, products or a category with Analyze Eval: LLM Benchmarking with lm-evaluation-harness (Orchestra-Research/AI-Research-SKILLs, 13k stars), Azure AI Projects Python SDK (microsoft/skills, 3.1k stars), Fine-Tuning Expert (Jeffallan/claude-skills, 12k stars) and Looper (ksimback/looper, 710 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
Who maintains Analyze Eval?
get-convex (a GitHub organization) maintains it in get-convex/convex-evals, which has 129 GitHub stars. The repository holds 5 skills in this directory. The repository was last updated on October 8, 2026.
Source: get-convex/convex-evals on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.