Chroma Vector Database
Orchestra-Research/AI-Research-SKILLs
Shows how to store documents and embeddings in Chroma, query them by similarity with metadata filters, and persist them to disk for RAG and semantic search projects.
Evaluates RAG pipeline quality across retrieval (precision, recall, MRR) and generation (groundedness, hallucination rate).
$ npx skills add Mathews-Tom/armory --skill rag-auditor -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install Mathews-Tom/armory rag-auditor --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/Mathews-Tom/armory.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/rag-auditor .claude/skills/rag-auditor && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "rag-auditor" agent skill from https://github.com/Mathews-Tom/armory/tree/main/skills/rag-auditor into .claude/skills/rag-auditor/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "rag-auditor", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/Mathews-Tom/armory/tree/main/skills/rag-auditorType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add Mathews-Tom/armory --skill rag-auditor -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install Mathews-Tom/armory rag-auditor --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/Mathews-Tom/armory.git skills-src && mkdir -p .agents/skills && cp -r skills-src/skills/rag-auditor .agents/skills/rag-auditor && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "rag-auditor" agent skill from https://github.com/Mathews-Tom/armory/tree/main/skills/rag-auditor into .agents/skills/rag-auditor/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "rag-auditor", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add Mathews-Tom/armory --skill rag-auditor -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install Mathews-Tom/armory rag-auditor --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/Mathews-Tom/armory.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/skills/rag-auditor .cursor/skills/rag-auditor && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "rag-auditor" agent skill from https://github.com/Mathews-Tom/armory/tree/main/skills/rag-auditor into .cursor/skills/rag-auditor/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "rag-auditor", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/Mathews-Tom/armory.git --path skills/rag-auditor--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add Mathews-Tom/armory --skill rag-auditor -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install Mathews-Tom/armory rag-auditor --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/Mathews-Tom/armory.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/skills/rag-auditor .gemini/skills/rag-auditor && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "rag-auditor" agent skill from https://github.com/Mathews-Tom/armory/tree/main/skills/rag-auditor into .gemini/skills/rag-auditor/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "rag-auditor", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install Mathews-Tom/armory rag-auditorInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add Mathews-Tom/armory --skill rag-auditor -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/Mathews-Tom/armory.git skills-src && mkdir -p .github/skills && cp -r skills-src/skills/rag-auditor .github/skills/rag-auditor && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "rag-auditor" agent skill from https://github.com/Mathews-Tom/armory/tree/main/skills/rag-auditor into .github/skills/rag-auditor/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "rag-auditor", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add Mathews-Tom/armory --skill rag-auditor -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install Mathews-Tom/armory rag-auditor --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/Mathews-Tom/armory.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/skills/rag-auditor .opencode/skills/rag-auditor && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "rag-auditor" agent skill from https://github.com/Mathews-Tom/armory/tree/main/skills/rag-auditor into .opencode/skills/rag-auditor/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "rag-auditor", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
rag-auditorEvaluates RAG pipeline quality across retrieval (precision, recall, MRR) and generation (groundedness, hallucination rate).
RAG Auditor is an agent skill from Mathews-Tom/armory. Evaluates RAG pipeline quality across retrieval (precision, recall, MRR) and generation (groundedness, hallucination rate). Triggers on: "audit RAG pipeline", "RAG quality", "hallucination detection", "why is RAG failing", "grounding check". NOT for general architecture audits, use architecture-reviewer.
Its SKILL.md is about 2.4k tokens, which your agent loads only when the skill is triggered. The skill folder holds 7 other files, including reference files (for example `evals/cases.yaml`, `references/diagnostic-queries.md` and `references/failure-taxonomy.md`).
It sits in AI & LLM Engineering, covering Retrieval-augmented generation. The repository describes itself as: Curated, production-grade skills for AI coding agents. Battle-tested workflows for developers who use AI seriously. The licence is MIT.
6 steps, taken from the step headings in SKILL.md.
Read from SKILL.md and the folder at commit 4594fb7. It shows what the files ask for, not the result of running them.
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
No scripts in the folder and no shell commands in SKILL.md.
From the folder's file list and the shell code blocks in SKILL.md.
No URLs in SKILL.md.
From URLs in SKILL.md, links to its own repository left out.
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
RAG Auditor loads about 2.4k tokens when it runs, and up to ~7.6k if it reads all its reference files. Until then it costs about 79 tokens; SKILL.md has 728 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
The full file from Mathews-Tom/armory at commit 4594fb7, republished under its MIT licence (© Mathews-Tom). 728 words, ~2,380 tokens.
.claude/skills/rag-auditor/SKILL.md (or your agent's skills folder). This skill also uses 5 other files; get the full folder from GitHub.Systematic RAG pipeline evaluation across the full retrieval-generation chain: designs evaluation query sets, measures retrieval metrics (Precision@K, Recall@K, MRR), evaluates generation quality (groundedness, completeness, hallucination rate), diagnoses component-level failures, and recommends targeted improvements.
| File | Contents | Load When |
|---|---|---|
references/retrieval-metrics.md | Precision@K, Recall@K, MRR, NDCG definitions and calculation | Always |
references/generation-metrics.md | Groundedness, completeness, hallucination detection methods | Generation evaluation needed |
references/failure-taxonomy.md | RAG failure categories: retrieval, generation, chunking, embedding | Failure diagnosis needed |
references/diagnostic-queries.md | Designing evaluation query sets, known-answer questions, difficulty levels | Evaluation setup |
Document the RAG pipeline configuration:
Create a diverse set of test queries:
| Query Type | Purpose | Count |
|---|---|---|
| Known-answer (factoid) | Measure retrieval + generation accuracy | 10+ |
| Multi-hop | Require combining info from multiple chunks | 5+ |
| Unanswerable | Not in the corpus — should abstain | 3+ |
| Ambiguous | Multiple valid interpretations | 3+ |
| Recent/updated | Test freshness | 2+ |
For each query, document the expected answer and the source chunk(s).
For each test query, measure:
For each test query with retrieved context:
For every incorrect or low-quality response, classify the root cause:
| Failure Type | Diagnosis | Indicator |
|---|---|---|
| Retrieval failure | Relevant chunks not retrieved | Low Recall@K |
| Ranking failure | Relevant chunk retrieved but ranked low | Low MRR, high Recall |
| Chunk boundary issue | Answer split across chunk boundaries | Partial matches in multiple chunks |
| Embedding mismatch | Query semantics don't match chunk embeddings | Relevant chunk has low similarity score |
| Generation failure | Correct context but wrong answer | High retrieval scores, low groundedness |
| Hallucination | Model invents facts not in context | Claims not traceable to any chunk |
| Over-abstention | Model refuses to answer when context is sufficient | Unanswered with relevant context present |
Based on failure analysis, recommend specific improvements:
| Failure Pattern | Recommendation |
|---|---|
| Chunk boundary issues | Increase overlap, try semantic chunking |
| Low Precision@K | Reduce K, add reranking stage |
| Low Recall@K | Increase K, try hybrid search |
| Embedding mismatch | Try different embedding model, add query expansion |
| Hallucination | Strengthen grounding instruction in prompt, reduce temperature |
| Over-abstention | Soften abstention criteria in prompt |
## RAG Audit Report
### Pipeline Configuration
| Component | Value |
|-----------|-------|
| Documents | {N} ({format}) |
| Chunking | {strategy}, {size} tokens, {overlap}% overlap |
| Embedding | {model} ({dimensions}d) |
| Retrieval | {method}, K={N} |
| Generation | {model}, temperature={T} |
### Evaluation Dataset
- **Total queries:** {N}
- **Known-answer:** {N}
- **Multi-hop:** {N}
- **Unanswerable:** {N}
### Retrieval Quality
| Metric | Score | Target | Status |
|--------|-------|--------|--------|
| Precision@{K} | {score} | {target} | {Pass/Fail} |
| Recall@{K} | {score} | {target} | {Pass/Fail} |
| MRR | {score} | {target} | {Pass/Fail} |
### Generation Quality
| Metric | Score | Target | Status |
|--------|-------|--------|--------|
| Groundedness | {score} | {target} | {Pass/Fail} |
| Completeness | {score} | {target} | {Pass/Fail} |
| Hallucination rate | {score} | {target} | {Pass/Fail} |
| Abstention accuracy | {score} | {target} | {Pass/Fail} |
### Failure Analysis
| # | Query | Failure Type | Root Cause | Recommendation |
|---|-------|-------------|------------|----------------|
| 1 | {query} | {type} | {cause} | {fix} |
### Recommendations (Priority Order)
1. **{Recommendation}** — addresses {N} failures, expected impact: {description}
2. **{Recommendation}** — addresses {N} failures, expected impact: {description}
### Sample Failures
#### Query: "{query}"
- **Expected:** {answer}
- **Retrieved chunks:** {chunk summaries with relevance scores}
- **Generated:** {response}
- **Issue:** {diagnosis}| Problem | Resolution |
|---|---|
| No known-answer queries available | Help design them from the document corpus. Pick 10 facts and formulate questions. |
| Pipeline access not available | Work from recorded inputs/outputs. Post-hoc evaluation is possible with query-context-response triples. |
| Corpus is too large to review | Sample-based evaluation. Select representative documents and generate queries from them. |
| Multiple failure types co-exist | Address retrieval failures first. Generation quality cannot exceed retrieval quality. |
Push back if:
© Mathews-Tom, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
SKILL.md and 5 other files (references) in skills/rag-auditor of Mathews-Tom/armory.
Open the folder on GitHubat commit 4594fb7
RAG Auditor next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| RAG Auditor this skillMathews-Tom/armory | 328 | — | ~2.4k | Automated safety check: Pass | MIT | |
| Chroma Vector DatabaseOrchestra-Research/AI-Research-SKILLs | 13k | 7 repos | ~2.3k | Automated safety check: Pass | MIT | |
| Senior Prompt Engineermaslennikov-ig/claude-code-orchestrator-kit | 260 | 3 repos | ~1.4k | Automated safety check: Pass | Custom licence | |
| MCP Local RAGshinpr/mcp-local-rag | 411 | — | ~4.4k | Automated safety check: Pass | MIT | |
| Ms Agent Framework RAGshuyu-labs/WebCode | 278 | — | ~1.1k | Automated safety check: Pass | Custom licence | |
| Local RAG Searchnkapila6/mcp-local-rag | 134 | 1 repos | ~1.6k | Automated safety check: Pass | MIT |
Orchestra-Research/AI-Research-SKILLs
Shows how to store documents and embeddings in Chroma, query them by similarity with metadata filters, and persist them to disk for RAG and semantic search projects.
maslennikov-ig/claude-code-orchestrator-kit
Provides reference guides and Python scripts for prompt optimization, RAG evaluation, and agent orchestration when building or tuning LLM systems.
shinpr/mcp-local-rag
Searches, saves, and maintains a local document index through a local RAG MCP server.
shuyu-labs/WebCode
Comprehensive guide for building Agentic RAG systems using Microsoft Agent Framework in C.
nkapila6/mcp-local-rag
Efficiently perform web searches using the mcp-local-rag server with semantic similarity ranking.
iternal-technologies-partners/blockify-agentic-data-optimization
Process documents with Blockify API to create optimized IdeaBlocks for RAG.
Mathews-Tom/armory
Architecture reviews across 7 dimensions (structural, scalability, enterprise readiness, performance, security, ops, data) with scored reports.
Mathews-Tom/armory
Turn concepts into static HTML visuals exported as PNG or SVG files via HTML/CSS/SVG.
Mathews-Tom/armory
A skill your agent uses when analyzing an existing video URL or local recording: "watch this video", "analyze youtube video", "summarize this video", "youtube transcript", "find this moment", "what…
Mathews-Tom/armory
Deep code simplification and refactoring preserving behavior across Python, Go, TypeScript, Rust.
Mathews-Tom/armory
Turn concepts into animated explainer videos using Manim (Python) with MP4/GIF output, audio overlay, multi-scene composition.
Mathews-Tom/armory
Maps the unresolved architecture, policy, and scope decisions that must be answered before planning can start: one durable decision ticket per question on the issue tracker, typed and blocker-linked…
Categories
Evaluates RAG pipeline quality across retrieval (precision, recall, MRR) and generation (groundedness, hallucination rate). RAG Auditor is an agent skill from Mathews-Tom/armory. Evaluates RAG pipeline quality across retrieval (precision, recall, MRR) and generation (groundedness, hallucination rate).
RAG Auditor fits situations like: : audit RAG pipeline; hallucination detection; why is RAG failing; grounding check.
Run `npx skills add Mathews-Tom/armory --skill rag-auditor -a claude-code`. Or copy the skill folder (skills/rag-auditor in Mathews-Tom/armory) into .claude/skills/rag-auditor in your project. Claude Code loads it when a task matches its description.
Run `npx skills add Mathews-Tom/armory --skill rag-auditor -a codex`. Or copy the skill folder (skills/rag-auditor in Mathews-Tom/armory) into .agents/skills/rag-auditor in your project. Codex loads it when a task matches its description.
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add Mathews-Tom/armory --skill rag-auditor -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/rag-auditor, .gemini/skills/rag-auditor, .github/skills/rag-auditor and .opencode/skills/rag-auditor in your project.
SKILL.md names no scripts, command-line tools or credentials: RAG Auditor is instructions for the agent only.
SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
RAG Auditor is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
About 2.4k tokens (SKILL.md is roughly 9.5k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full. Its references folder adds about 5.2k tokens, read only when the agent opens those files.
Skills that share tags, products or a category with RAG Auditor: Chroma Vector Database (Orchestra-Research/AI-Research-SKILLs, 13k stars), Senior Prompt Engineer (maslennikov-ig/claude-code-orchestrator-kit, 260 stars), MCP Local RAG (shinpr/mcp-local-rag, 411 stars) and Ms Agent Framework RAG (shuyu-labs/WebCode, 278 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
Mathews-Tom (a GitHub user) maintains it in Mathews-Tom/armory, which has 328 GitHub stars. The repository holds 80 skills in this directory. The repository was last updated on October 6, 2026.
Source: Mathews-Tom/armory on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.