Evaluate RAG
ai-evals-course/evals-skills
Guides evaluation of a RAG system by diagnosing failures in traces, building a retrieval test set and scoring retrieval and generation separately.
Evaluate retrieval and citation behavior for RAG pipelines from deterministic JSONL fixtures.
Install RAG Evaluation Harness
$ npx skills add davepoon/buildwithclaude --skill rag-evaluation-harness -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install davepoon/buildwithclaude rag-evaluation-harness --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/davepoon/buildwithclaude.git skills-src && mkdir -p .claude/skills && cp -r skills-src/plugins/all-skills/skills/rag-evaluation-harness .claude/skills/rag-evaluation-harness && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "rag-evaluation-harness" agent skill from https://github.com/davepoon/buildwithclaude/tree/main/plugins/all-skills/skills/rag-evaluation-harness into .claude/skills/rag-evaluation-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "rag-evaluation-harness", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/davepoon/buildwithclaude/tree/main/plugins/all-skills/skills/rag-evaluation-harnessType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add davepoon/buildwithclaude --skill rag-evaluation-harness -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install davepoon/buildwithclaude rag-evaluation-harness --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/davepoon/buildwithclaude.git skills-src && mkdir -p .agents/skills && cp -r skills-src/plugins/all-skills/skills/rag-evaluation-harness .agents/skills/rag-evaluation-harness && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "rag-evaluation-harness" agent skill from https://github.com/davepoon/buildwithclaude/tree/main/plugins/all-skills/skills/rag-evaluation-harness into .agents/skills/rag-evaluation-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "rag-evaluation-harness", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add davepoon/buildwithclaude --skill rag-evaluation-harness -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install davepoon/buildwithclaude rag-evaluation-harness --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/davepoon/buildwithclaude.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/plugins/all-skills/skills/rag-evaluation-harness .cursor/skills/rag-evaluation-harness && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "rag-evaluation-harness" agent skill from https://github.com/davepoon/buildwithclaude/tree/main/plugins/all-skills/skills/rag-evaluation-harness into .cursor/skills/rag-evaluation-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "rag-evaluation-harness", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/davepoon/buildwithclaude.git --path plugins/all-skills/skills/rag-evaluation-harness--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add davepoon/buildwithclaude --skill rag-evaluation-harness -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install davepoon/buildwithclaude rag-evaluation-harness --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/davepoon/buildwithclaude.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/plugins/all-skills/skills/rag-evaluation-harness .gemini/skills/rag-evaluation-harness && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "rag-evaluation-harness" agent skill from https://github.com/davepoon/buildwithclaude/tree/main/plugins/all-skills/skills/rag-evaluation-harness into .gemini/skills/rag-evaluation-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "rag-evaluation-harness", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install davepoon/buildwithclaude rag-evaluation-harnessInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add davepoon/buildwithclaude --skill rag-evaluation-harness -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/davepoon/buildwithclaude.git skills-src && mkdir -p .github/skills && cp -r skills-src/plugins/all-skills/skills/rag-evaluation-harness .github/skills/rag-evaluation-harness && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "rag-evaluation-harness" agent skill from https://github.com/davepoon/buildwithclaude/tree/main/plugins/all-skills/skills/rag-evaluation-harness into .github/skills/rag-evaluation-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "rag-evaluation-harness", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add davepoon/buildwithclaude --skill rag-evaluation-harness -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install davepoon/buildwithclaude rag-evaluation-harness --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/davepoon/buildwithclaude.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/plugins/all-skills/skills/rag-evaluation-harness .opencode/skills/rag-evaluation-harness && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "rag-evaluation-harness" agent skill from https://github.com/davepoon/buildwithclaude/tree/main/plugins/all-skills/skills/rag-evaluation-harness into .opencode/skills/rag-evaluation-harness/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "rag-evaluation-harness", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
Facts
- Skill name
rag-evaluation-harness- GitHub stars
- 3.6k
- Token cost
- ~813 tokens
- SKILL.md length
- 352 words
- Files
- 4 (incl. scripts)
- Skills in repo
- 246
- Repo updated
- First seen
- Licence
- MIT
At a glance
Evaluate retrieval and citation behavior for RAG pipelines from deterministic JSONL fixtures.
- An agent needs offline Recall@K
- SKILL.md covers Input Contract, Run an Evaluation, Interpret the Report and Verification
- Runs JavaScript scripts from its folder; calls node
- Reciprocal rank
What it does
RAG Evaluation Harness is an agent skill from davepoon/buildwithclaude. Evaluate retrieval and citation behavior for RAG pipelines from deterministic JSONL fixtures. Use when an agent needs offline Recall@K, reciprocal rank, context precision, citation coverage, citation validity, diagnostics, Markdown/JSON reports, or threshold-gated evaluation in CI.
Its SKILL.md is about 810 tokens, which your agent loads only when the skill is triggered. The skill folder holds 5 other files, including scripts.
It sits in AI & LLM Engineering, covering LLM evaluation, Retrieval-augmented generation and Citation management. The repository describes itself as: A single hub to find Claude Skills, Agents, Commands, Hooks, Plugins, and Marketplace collections to extend Claude Code, Claude Desktop, Agent SDK and OpenClaw. The licence is MIT.
When your agent uses it
- An agent needs offline Recall@K
- Reciprocal rank
- Context precision
- Citation coverage
Example prompts
- “/rag-evaluation-harness”
Requirements
- Node.js
What it can do on your machine
Read from SKILL.md and the folder at commit 616deb5. It shows what the files ask for, not the result of running them.
Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
Ships 2 files in scripts/ (JavaScript), which the agent can run.
Shell commands in SKILL.md call:
node
From the folder's file list and the shell code blocks in SKILL.md.
Network
No URLs in SKILL.md.
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Context cost
RAG Evaluation Harness loads about 813 tokens when it runs. Until then it costs about 76 tokens; SKILL.md has 352 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
Safety
Auto-check passed
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); the scripts in this folder are not scanned.
SKILL.md
The full file from davepoon/buildwithclaude at commit 616deb5, republished under its MIT licence (© davepoon). 352 words, ~813 tokens.
Download SKILL.mdSave it as
.claude/skills/rag-evaluation-harness/SKILL.md (or your agent's skills folder). This skill also uses 3 other files; get the full folder from GitHub.- name
- rag-evaluation-harness
- description
- Evaluate retrieval and citation behavior for RAG pipelines from deterministic JSONL fixtures. Use when an agent needs offline Recall@K, reciprocal rank, context precision, citation coverage, citation validity, diagnostics, Markdown/JSON reports, or threshold-gated evaluation in CI.
- category
- ai-ml
RAG Evaluation Harness
Use this skill to measure a retrieval-and-citation contract without making model calls or network requests. The bundled evaluator compares explicit document IDs, so it is suitable for repeatable local checks and CI gates.
Input Contract
Provide one JSON object per line with a unique string id and three arrays of document IDs:
{"id":"question-1","relevant_document_ids":["doc-a"],"retrieved_document_ids":["doc-b","doc-a"],"cited_document_ids":["doc-a"]}Blank lines are ignored. Invalid JSON, missing arrays, non-string IDs, and duplicate case IDs fail with the JSONL line number. Keep the fixture's relevance labels and citation IDs explicit; do not infer them from answer text.
Run an Evaluation
Set the installed skill directory and run the standard-library-only evaluator:
SKILL_DIR="<absolute path to the installed rag-evaluation-harness skill>"
node "$SKILL_DIR/scripts/evaluate-rag.mjs" "$SKILL_DIR/examples/sample-evaluation.jsonl" \\
--k 3 --format markdownUse --format json for CI or downstream tooling. Add any of these optional thresholds (each must be between 0 and 1):
--min-recall
--min-mrr
--min-context-precision
--min-citation-coverage
--min-citation-validityThe process exits 0 when all requested thresholds pass, 1 when a threshold fails, and 2 for invalid arguments or input. Threshold failures are written to stderr while the complete report remains on stdout.
Show full SKILL.md (193 more words)Show less
Interpret the Report
Recall@K: relevant IDs found in the first K unique retrieved IDs divided by all relevant IDs.Reciprocal rank: inverse rank of the first relevant retrieved ID, or zero when none is found.Context precision@K: relevant IDs in the first K unique retrieved IDs divided by the number of retrieved IDs considered.Citation coverage: relevant IDs cited divided by all relevant IDs.Citation validity: cited IDs that were retrieved divided by all cited IDs.
The summary is a macro average. Recall and citation coverage are null for cases with no relevant IDs and are excluded from their macro denominators. Other empty denominators are reported as zero. Diagnostics call out empty retrieval, absent relevance labels, duplicate retrieved IDs, citations that were not retrieved, and missing citations.
These are ID-level proxy metrics. They do not establish semantic answer quality, entailment, attribution correctness, or groundedness. Pair them with a separate answer-quality evaluation when those properties matter.
Verification
Run the focused tests and the repository skill validator:
node --test "$SKILL_DIR/scripts/evaluate-rag.test.mjs"
node scripts/validate-skills.jsThe evaluator is deterministic and offline. It reads only the supplied JSONL file and never executes retrieved content, calls an MCP server, accesses credentials, or mutates the input.
© davepoon, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Files
SKILL.md and 3 other files (scripts) in plugins/all-skills/skills/rag-evaluation-harness of davepoon/buildwithclaude.
- SKILL.md
- examples/sample-evaluation.jsonl
- scripts/evaluate-rag.mjs
- scripts/evaluate-rag.test.mjs
Open the folder on GitHubat commit 616deb5
Compare with similar skills
RAG Evaluation Harness next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| RAG Evaluation Harness this skilldavepoon/buildwithclaude | 3.6k | — | ~813 | Automated safety check: Pass | MIT | |
| Evaluate RAGai-evals-course/evals-skills | 1.5k | — | ~1.9k | Automated safety check: Pass | Apache-2.0 | |
| Sciverseopendatalab/Sciverse-Agent-Tools | 119 | — | ~3k | Automated safety check: Pass | Custom licence | |
| Jd Gap Analysisstarkyru/learn-ai | 107 | — | ~1.9k | Automated safety check: Pass | MIT | |
| RAG Cite Sourceslyonzin/knowledge-rag | 292 | — | ~1.4k | Automated safety check: Pass | MIT | |
| Penguin SDKPrism-Shadow/penguin-harness | 2.5k | — | ~11k | Automated safety check: Pass | Apache-2.0 |
Similar skills
Sciverse
opendatalab/Sciverse-Agent-Tools
A skill your agent uses when the user needs academic paper retrieval — searching scientific literature by author/year/journal, finding paper chunks for RAG-style citations, or expanding original…
Jd Gap Analysis
starkyru/learn-ai
Analyze a job description (pasted text OR a URL) and find the AI/ML/GenAI topics it requires that this learn-ai course does NOT yet cover.
RAG Cite Sources
lyonzin/knowledge-rag
Every technical claim drawn from the local corpus must ship with a source citation formatted as path:line or path:section.
Penguin SDK
Prism-Shadow/penguin-harness
A skill your agent uses whenever the user wants to build an agent application — their own program with an embedded agent, such as an AI app, an agentic app or a RAG app.
Agent Eval
ericrisco/rsc-harness
A skill your agent uses when measuring whether an LLM or agent system actually got better and gating merges on it: golden sets, fixing an inflated LLM-as-judge, scoring RAG (faithfulness, contextual…
More from davepoon/buildwithclaude
All 246 skills in this repoiOS Hig Design Guide
davepoon/buildwithclaude
Build, update, and apply iOS design specifications using Apple Human Interface Guidelines (HIG) source data.
Video Downloader
davepoon/buildwithclaude
Download YouTube videos with customizable quality and format options.
Qwen Vision
davepoon/buildwithclaude
A skill your agent uses when the user asks to "analyze video", "watch this video", "what happens in this video", "describe this clip", "review this footage", "classify these videos", "compare…
Atlas Cloud Media
davepoon/buildwithclaude
Discover Atlas Cloud image and video models, inspect their live schemas, and submit one confirmed media generation request with bounded GET polling.
Browser Extension Launch
davepoon/buildwithclaude
面向没有编程经验的用户，把想法做成可试用的浏览器插件，并完成检查、商店材料、审核提交和上线验证；也用于继续已有插件、排错和发布新版。用户说“帮我做个插件”“把插件上架”“继续我的插件”时使用。普通网站开发、仅查询插件知识不触发。
Slack Gif Creator
davepoon/buildwithclaude
Toolkit for creating animated GIFs optimized for Slack, with validators for size constraints and composable animation primitives.
Related
Categories
Questions about RAG Evaluation Harness
What does RAG Evaluation Harness do?
Evaluate retrieval and citation behavior for RAG pipelines from deterministic JSONL fixtures. RAG Evaluation Harness is an agent skill from davepoon/buildwithclaude. Evaluate retrieval and citation behavior for RAG pipelines from deterministic JSONL fixtures.
When should I use RAG Evaluation Harness?
RAG Evaluation Harness fits situations like: an agent needs offline Recall@K; reciprocal rank; context precision; citation coverage.
How do I install RAG Evaluation Harness in Claude Code?
Run `npx skills add davepoon/buildwithclaude --skill rag-evaluation-harness -a claude-code`. Or copy the skill folder (plugins/all-skills/skills/rag-evaluation-harness in davepoon/buildwithclaude) into .claude/skills/rag-evaluation-harness in your project. Claude Code loads it when a task matches its description.
How do I install RAG Evaluation Harness in Codex?
Run `npx skills add davepoon/buildwithclaude --skill rag-evaluation-harness -a codex`. Or copy the skill folder (plugins/all-skills/skills/rag-evaluation-harness in davepoon/buildwithclaude) into .agents/skills/rag-evaluation-harness in your project. Codex loads it when a task matches its description.
Can I use RAG Evaluation Harness in Cursor, Gemini CLI or GitHub Copilot?
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add davepoon/buildwithclaude --skill rag-evaluation-harness -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/rag-evaluation-harness, .gemini/skills/rag-evaluation-harness, .github/skills/rag-evaluation-harness and .opencode/skills/rag-evaluation-harness in your project.
What does RAG Evaluation Harness need to run?
Going by SKILL.md and its folder, RAG Evaluation Harness needs JavaScript for the scripts in its folder and the command-line tools its instructions call (node). Our summary lists: Node.js.
Does RAG Evaluation Harness access the network?
SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.
Is RAG Evaluation Harness safe to install?
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. The check reads SKILL.md only: the scripts in the folder are not scanned, so read them before running anything.
What licence does RAG Evaluation Harness use?
RAG Evaluation Harness is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
How many tokens does RAG Evaluation Harness use?
About 813 tokens (SKILL.md is roughly 3.3k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
What are the alternatives to RAG Evaluation Harness?
Skills that share tags, products or a category with RAG Evaluation Harness: Evaluate RAG (ai-evals-course/evals-skills, 1.5k stars), Sciverse (opendatalab/Sciverse-Agent-Tools, 119 stars), Jd Gap Analysis (starkyru/learn-ai, 107 stars) and RAG Cite Sources (lyonzin/knowledge-rag, 292 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
Who maintains RAG Evaluation Harness?
davepoon (a GitHub user) maintains it in davepoon/buildwithclaude, which has 3,605 GitHub stars. The repository holds 246 skills in this directory. The repository was last updated on October 9, 2026.
Source: davepoon/buildwithclaude on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.