Hybrid Search Implementation
wshobson/agents
Shows how to run vector and keyword search side by side and merge their results, so retrieval catches both meaning and exact terms in RAG and search systems.
Generate a search quality benchmark for the AI Registry. An agent skill from agentic-community/mcp-gateway-registry.
Install Search Benchmark
$ npx skills add agentic-community/mcp-gateway-registry --skill search-benchmark -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install agentic-community/mcp-gateway-registry search-benchmark --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/agentic-community/mcp-gateway-registry.git skills-src && mkdir -p .claude/skills && cp -r skills-src/.claude/skills/search-benchmark .claude/skills/search-benchmark && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "search-benchmark" agent skill from https://github.com/agentic-community/mcp-gateway-registry/tree/main/.claude/skills/search-benchmark into .claude/skills/search-benchmark/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "search-benchmark", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/agentic-community/mcp-gateway-registry/tree/main/.claude/skills/search-benchmarkType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add agentic-community/mcp-gateway-registry --skill search-benchmark -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install agentic-community/mcp-gateway-registry search-benchmark --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/agentic-community/mcp-gateway-registry.git skills-src && mkdir -p .agents/skills && cp -r skills-src/.claude/skills/search-benchmark .agents/skills/search-benchmark && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "search-benchmark" agent skill from https://github.com/agentic-community/mcp-gateway-registry/tree/main/.claude/skills/search-benchmark into .agents/skills/search-benchmark/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "search-benchmark", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add agentic-community/mcp-gateway-registry --skill search-benchmark -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install agentic-community/mcp-gateway-registry search-benchmark --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/agentic-community/mcp-gateway-registry.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/.claude/skills/search-benchmark .cursor/skills/search-benchmark && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "search-benchmark" agent skill from https://github.com/agentic-community/mcp-gateway-registry/tree/main/.claude/skills/search-benchmark into .cursor/skills/search-benchmark/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "search-benchmark", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/agentic-community/mcp-gateway-registry.git --path .claude/skills/search-benchmark--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add agentic-community/mcp-gateway-registry --skill search-benchmark -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install agentic-community/mcp-gateway-registry search-benchmark --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/agentic-community/mcp-gateway-registry.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/.claude/skills/search-benchmark .gemini/skills/search-benchmark && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "search-benchmark" agent skill from https://github.com/agentic-community/mcp-gateway-registry/tree/main/.claude/skills/search-benchmark into .gemini/skills/search-benchmark/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "search-benchmark", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install agentic-community/mcp-gateway-registry search-benchmarkInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add agentic-community/mcp-gateway-registry --skill search-benchmark -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/agentic-community/mcp-gateway-registry.git skills-src && mkdir -p .github/skills && cp -r skills-src/.claude/skills/search-benchmark .github/skills/search-benchmark && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "search-benchmark" agent skill from https://github.com/agentic-community/mcp-gateway-registry/tree/main/.claude/skills/search-benchmark into .github/skills/search-benchmark/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "search-benchmark", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add agentic-community/mcp-gateway-registry --skill search-benchmark -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install agentic-community/mcp-gateway-registry search-benchmark --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/agentic-community/mcp-gateway-registry.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/.claude/skills/search-benchmark .opencode/skills/search-benchmark && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "search-benchmark" agent skill from https://github.com/agentic-community/mcp-gateway-registry/tree/main/.claude/skills/search-benchmark into .opencode/skills/search-benchmark/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "search-benchmark", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
Facts
- Skill name
search-benchmark- GitHub stars
- 962
- Token cost
- ~1.9k tokens
- SKILL.md length
- 746 words
- Files
- 1
- Skills in repo
- 17
- Repo updated
- First seen
- Licence
- Apache-2.0
At a glance
Generate a search quality benchmark for the AI Registry. An agent skill from agentic-community/mcp-gateway-registry.
- Works in 4 steps: Check for Existing Ground Truth → Run Benchmark → Review Report → …
- You want to measure search quality after changes to the scoring algorithm
- SKILL.md covers Prerequisites, Input, Workflow and Output Format, plus 4 more sections
- Calls curl, uv and python3; reaches d2xl2zfuhgc4l0.cloudfront.net
What it does
Search Benchmark is an agent skill from agentic-community/mcp-gateway-registry. Generate a search quality benchmark for the AI Registry. Generates ground truth from the registry's assets, runs 100+ queries against the semantic search API, evaluates results using NDCG@10/MRR/Recall, and produces a markdown report. Use when you want to measure search quality after changes to the scoring algorithm, embedding model, or indexed content.
Its SKILL.md is about 1.9k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.
It sits in AI & LLM Engineering, covering Embeddings. The repository describes itself as: Enterprise-ready MCP Gateway & Registry that centralizes AI development tools with secure OAuth authentication, dynamic tool discovery, and unified access for both autonomous AI… The licence is Apache-2.0.
When your agent uses it
- You want to measure search quality after changes to the scoring algorithm
- Embedding model
- Indexed content
Example prompts
- “/search-benchmark”
Requirements
- Python 3
Workflow steps
4 steps, taken from the step headings in SKILL.md.
What it can do on your machine
Read from SKILL.md and the folder at commit ec3a197. It shows what the files ask for, not the result of running them.
Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
Shell commands in SKILL.md call:
curluvpython3
From the folder's file list and the shell code blocks in SKILL.md.
Network
Hosts in commands or code, which the agent is likely to contact:
d2xl2zfuhgc4l0.cloudfront.net
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Context cost
Search Benchmark loads about 1.9k tokens when it runs. Until then it costs about 93 tokens; SKILL.md has 746 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
Safety
Auto-check passed
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
SKILL.md
The full file from agentic-community/mcp-gateway-registry at commit ec3a197, republished under its Apache-2.0 licence (© agentic-community). 746 words, ~1,914 tokens.
Download SKILL.mdSave it as
.claude/skills/search-benchmark/SKILL.md (or your agent's skills folder).- name
- search-benchmark
- description
- Generate a search quality benchmark for the AI Registry. Generates ground truth from the registry's assets, runs 100+ queries against the semantic search API, evaluates results using NDCG@10/MRR/Recall, and produces a markdown report. Use when you want to measure search quality after changes to the scoring algorithm, embedding model, or indexed content.
- license
- Apache-2.0
- metadata.author
- mcp-gateway-registry
- metadata.version
- 1.0
Search Benchmark Skill
Measure semantic search quality against a deployed AI Registry. Generates a ground truth dataset from the registry's own assets, runs queries against the live API, evaluates results using standard information retrieval metrics (NDCG@10, MRR, Recall@10), and produces a markdown report.
Prerequisites
- Registry URL - The base URL of the deployed registry (e.g.,
https://d2xl2zfuhgc4l0.cloudfront.net) - JWT Token - A valid admin token in
.tokenfile (get from "Get JWT Token" button in registry UI) - Registry must have assets indexed - At least some servers, agents, or skills registered
The .token file supports both raw JWT format and the full JSON response from the registry UI.
Input
/search-benchmark [REGISTRY_URL] [TOKEN_FILE]- REGISTRY_URL - Base URL of the registry to benchmark (default: reads from user or uses
http://localhost) - TOKEN_FILE - Path to the token file (default:
.token)
Workflow
Step 1: Check for Existing Ground Truth
Check if a ground truth dataset already exists:
ls tests/fixtures/search_dataset/ground_truth.json 2>/dev/nullIf the file exists, report how many queries it contains and ask the user: "A ground truth dataset already exists (N queries). Do you want to use it or generate a new one from this registry?"
- If use existing: skip to Step 2
- If generate new: proceed to Step 1b
Step 1b: Generate Expert Ground Truth
This is NOT a simple programmatic generation. You must deeply analyze the registry's assets and craft queries like a search expert. Follow this process:
1b.1: Fetch all assets as JSON
TOKEN=$(python3 -c "
import json
with open('{TOKEN_FILE}') as f:
raw = f.read().strip()
if raw.startswith('{'):
data = json.loads(raw)
print(data.get('tokens',{}).get('access_token') or data.get('access_token',''))
else:
print(raw.replace('Bearer ',''))
")
curl -s -H "Authorization: Bearer $TOKEN" "{REGISTRY_URL}/api/servers?limit=2000" > /tmp/servers.json
curl -s -H "Authorization: Bearer $TOKEN" "{REGISTRY_URL}/api/agents?limit=2000" > /tmp/agents.json
curl -s -H "Authorization: Bearer $TOKEN" "{REGISTRY_URL}/api/skills?limit=2000" > /tmp/skills.json1b.2: Analyze the assets deeply
Read the JSON dumps. Filter out stress-test and security-pending assets. For each real asset, understand:
- What it does (from description)
- What tools it has (tool names and descriptions)
- What tags it uses
- How users would naturally search for it
1b.3: Craft 100 queries across these categories
| Category | Count | Strategy |
|---|---|---|
| exact-name | 10 | Product/server/agent names as queries |
| semantic | 15 | Natural language paraphrases with NO keyword overlap |
| tool-precision | 10 | Exact tool names as queries |
| tag-based | 10 | Tag vocabulary combinations |
| multi-entity | 10 | Queries where correct answers span servers + agents + skills |
| conflict | 10 | Ambiguous terms where vector and keyword disagree |
| no-answer | 10 | Queries completely outside the dataset (no match exists) |
| agent-focused | 10 | Queries specifically targeting agents |
| skill-focused | 10 | Queries specifically targeting skills |
| tricky | 15 | Edge cases, short queries, non-English, adversarial |
1b.4: For each query, define expected results
{
"query": "the search query",
"category": "one of the categories above",
"description": "what this query tests and why",
"expected": [
{"path": "/exact-path-from-registry", "grade": 3, "reason": "why this should match"},
{"path": "/another-path", "grade": 2, "reason": "why this is relevant but not perfect"}
]
}Grade scale: 3 = perfect match, 2 = highly relevant, 1 = somewhat relevant. For no-answer queries, expected should be an empty list.
1b.5: Validate all paths exist
Every path in expected results must exist in the fetched assets. Verify before saving.
1b.6: Save
Write the ground truth to tests/fixtures/search_dataset/ground_truth.json.
Tell the user how many queries were created per category and that they should review it.
Step 2: Run Benchmark
Run all queries against the live semantic search API and generate a report:
uv run python scripts/benchmark_search.py \
--url {REGISTRY_URL} \
--token-file {TOKEN_FILE} \
--queries tests/fixtures/search_dataset/generated_ground_truth.jsonOutput:
tests/fixtures/search_dataset/benchmark_results.json(raw results)tests/fixtures/search_dataset/benchmark_results.md(markdown report)
Show full SKILL.md (290 more words)Show less
Step 3: Review Report
Show the user the key metrics from the report:
- Quality Metrics - NDCG@10 > 0.7 is good, > 0.8 is excellent
- Score Health - Saturated scores at 1.0 should be < 15% (if higher, scoring formula may have issues)
- Quality by Category - Identify weak areas (e.g., agent-focused queries underperforming)
- Per-query results - Check specific queries that scored 0.0 (complete miss) or low NDCG
Open the report in the editor for the user to review.
Step 4: Compare Before/After (Optional)
If asked to compare two runs (e.g., before and after a scoring algorithm change):
uv run python scripts/benchmark_search.py \
--compare tests/fixtures/search_dataset/benchmark_results.json other_results.jsonOutput Format
The report includes:
- Registry metadata: version, database backend, server/agent/skill counts
- Quality Metrics: NDCG@10, MRR, Recall@10 averaged across all queries
- Score Health: saturation analysis (unique scores, % at 1.0, range)
- Quality by Category: breakdown per query type
- Per-query results: top 5 results with scores and ground truth comparison
Interpreting Metrics
| Metric | Good | Excellent | Poor |
|---|---|---|---|
| NDCG@10 | > 0.65 | > 0.80 | < 0.50 |
| MRR | > 0.70 | > 0.85 | < 0.50 |
| Recall@10 | > 0.75 | > 0.90 | < 0.60 |
| Score saturation | < 15% | < 5% | > 30% |
Example
$ /search-benchmark https://d2xl2zfuhgc4l0.cloudfront.net .tokenThis will:
- Generate ~90 queries from the registry's assets
- Run each query against the semantic search API
- Produce a report showing search quality metrics
- Open the report for review
Troubleshooting
- 401 errors: Token expired. Get a fresh one from the registry UI "Get JWT Token" button.
- 0 assets found: Registry has no servers/agents/skills registered, or token lacks permissions.
- Low NDCG on "description-derived" category: Expected for programmatically generated queries. Add hand-curated queries for better evaluation.
- High saturation (>30% at 1.0): Scoring formula may be using the legacy additive method. Set
SEARCH_FUSION_METHOD=rrfon the registry.
Related
- Search Evaluation Harness README - Full documentation on the evaluation system
- Search Quality Benchmark (reference) - Sample results from development registry
- Hybrid Search Architecture - How scoring works
© agentic-community, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Files
Just SKILL.md in .claude/skills/search-benchmark of agentic-community/mcp-gateway-registry.
Open the folder on GitHubat commit ec3a197
Compare with similar skills
Search Benchmark next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Search Benchmark this skillagentic-community/mcp-gateway-registry | 962 | — | ~1.9k | Automated safety check: Pass | Apache-2.0 | |
| Hybrid Search Implementationwshobson/agents | 40k | 9 repos | ~497 | Automated safety check: Pass | MIT | |
| Gemini Live APIgoogle/skills | 21k | — | ~2.5k | Automated safety check: Notes | Apache-2.0 | |
| Turnstile Spinswyxio/skills | 172 | — | ~987 | Automated safety check: Pass | MIT | |
| Multimodal Embedding Serving Devopen-edge-platform/edge-ai-libraries | 168 | — | ~1.2k | Automated safety check: Pass | Apache-2.0 | |
| Storing And Querying Vectorsaws/agent-toolkit-for-aws | 2.8k | — | ~1.9k | Automated safety check: Pass | Apache-2.0 |
Similar skills
- Official
Gemini Live API
google/skills
Generates a Gemini LiveAPI client service class in the user's chosen programming language.
Turnstile Spin
swyxio/skills
Add or repair Cloudflare Turnstile on an existing web form by creating or reusing a widget, embedding it, wiring mandatory server-side Siteverify in the existing backend, and validating the result.
Multimodal Embedding Serving Dev
open-edge-platform/edge-ai-libraries
Develop the Multimodal Embedding Serving microservice itself — Poetry install, run the existing tests, navigate the wrapper/registry/handler architecture, add a new model family, and build the image…
- Official
Storing And Querying Vectors
aws/agent-toolkit-for-aws
Store and query vector embeddings using Amazon S3 Vectors, a cost-effective long-term vector storage service with its own API namespace (s3vectors).
- Official
Debugging Signals Pipeline
PostHog/posthog-foss
Debug the signals pipeline locally end-to-end. An agent skill from PostHog/posthog-foss.
More from agentic-community/mcp-gateway-registry
All 17 skills in this repoExplainer
agentic-community/mcp-gateway-registry
Explain a GitHub issue or pull request at 100, 200, and 300 level.
Debug
agentic-community/mcp-gateway-registry
Debug issues in the MCP Gateway Registry using first-principles thinking.
Infra Sync
agentic-community/mcp-gateway-registry
Keep Terraform and CDK infrastructure in sync. An agent skill from agentic-community/mcp-gateway-registry.
Writing
agentic-community/mcp-gateway-registry
Write prose people will actually read. An agent skill from agentic-community/mcp-gateway-registry.
Agentcore Register
agentic-community/mcp-gateway-registry
Given an MCP server URL, probe the server via curl to discover its metadata and tools, then generate a markdown file with copy-pasteable content for each field in the Amazon Bedrock AgentCore…
Benchmark Report
agentic-community/mcp-gateway-registry
Generate a benchmark report from stress test results (registration, API performance, search concurrency).
Related
Categories
Questions about Search Benchmark
What does Search Benchmark do?
Generate a search quality benchmark for the AI Registry. An agent skill from agentic-community/mcp-gateway-registry. Search Benchmark is an agent skill from agentic-community/mcp-gateway-registry. Generate a search quality benchmark for the AI Registry.
When should I use Search Benchmark?
Search Benchmark fits situations like: you want to measure search quality after changes to the scoring algorithm; embedding model; indexed content.
How do I install Search Benchmark in Claude Code?
Run `npx skills add agentic-community/mcp-gateway-registry --skill search-benchmark -a claude-code`. Or copy the skill folder (.claude/skills/search-benchmark in agentic-community/mcp-gateway-registry) into .claude/skills/search-benchmark in your project. Claude Code loads it when a task matches its description.
How do I install Search Benchmark in Codex?
Run `npx skills add agentic-community/mcp-gateway-registry --skill search-benchmark -a codex`. Or copy the skill folder (.claude/skills/search-benchmark in agentic-community/mcp-gateway-registry) into .agents/skills/search-benchmark in your project. Codex loads it when a task matches its description.
Can I use Search Benchmark in Cursor, Gemini CLI or GitHub Copilot?
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add agentic-community/mcp-gateway-registry --skill search-benchmark -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/search-benchmark, .gemini/skills/search-benchmark, .github/skills/search-benchmark and .opencode/skills/search-benchmark in your project.
What does Search Benchmark need to run?
Going by SKILL.md and its folder, Search Benchmark needs the command-line tools its instructions call (curl, uv and python3). Our summary lists: Python 3.
Does Search Benchmark access the network?
SKILL.md names 1 domain. In commands or code: d2xl2zfuhgc4l0.cloudfront.net; the agent is likely to contact it when it follows the instructions. This is read from the text; nothing was executed.
Is Search Benchmark safe to install?
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
What licence does Search Benchmark use?
Search Benchmark is published under the Apache-2.0 licence (declared in SKILL.md). It allows redistribution, so the full SKILL.md is shown on this page.
How many tokens does Search Benchmark use?
About 1.9k tokens (SKILL.md is roughly 7.7k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
What are the alternatives to Search Benchmark?
Skills that share tags, products or a category with Search Benchmark: Hybrid Search Implementation (wshobson/agents, 40k stars), Gemini Live API (google/skills, 21k stars), Turnstile Spin (swyxio/skills, 172 stars) and Multimodal Embedding Serving Dev (open-edge-platform/edge-ai-libraries, 168 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
Who maintains Search Benchmark?
agentic-community (a GitHub organization) maintains it in agentic-community/mcp-gateway-registry, which has 962 GitHub stars. The repository holds 17 skills in this directory. The repository was last updated on October 6, 2026.
Source: agentic-community/mcp-gateway-registry on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.