Agent skill

Create Metric Plugin

by NomaDamas in NomaDamas/AutoRAG-Research

Guide developers through creating a custom evaluation metric plugin for AutoRAG-Research.

Apache-2.0Auto-check: notesDevelopment

Install Create Metric Plugin

skills CLI

$ npx skills add NomaDamas/AutoRAG-Research --skill create-metric-plugin -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI

$ gh skill install NomaDamas/AutoRAG-Research create-metric-plugin --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy

$ git clone --depth 1 https://github.com/NomaDamas/AutoRAG-Research.git skills-src && mkdir -p .claude/skills && cp -r skills-src/.agents/skills/create-metric-plugin .claude/skills/create-metric-plugin && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Paste into Claude Code

Install the "create-metric-plugin" agent skill from https://github.com/NomaDamas/AutoRAG-Research/tree/main/.agents/skills/create-metric-plugin into .claude/skills/create-metric-plugin/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "create-metric-plugin", then confirm the skill loads.

Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

$skill-installer

$skill-installer install https://github.com/NomaDamas/AutoRAG-Research/tree/main/.agents/skills/create-metric-plugin

Type this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.

skills CLI

$ npx skills add NomaDamas/AutoRAG-Research --skill create-metric-plugin -a codex

Project install goes to .agents/skills/; add -g for ~/.codex/skills/.

GitHub CLI

$ gh skill install NomaDamas/AutoRAG-Research create-metric-plugin --agent codex

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/NomaDamas/AutoRAG-Research.git skills-src && mkdir -p .agents/skills && cp -r skills-src/.agents/skills/create-metric-plugin .agents/skills/create-metric-plugin && rm -rf skills-src

Use ~/.agents/skills/ instead of .agents/skills for a personal install.

Codex skills documentation · loads skills from .agents/skills/

Paste into Codex

Install the "create-metric-plugin" agent skill from https://github.com/NomaDamas/AutoRAG-Research/tree/main/.agents/skills/create-metric-plugin into .agents/skills/create-metric-plugin/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "create-metric-plugin", then confirm the skill loads.

Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

skills CLI

$ npx skills add NomaDamas/AutoRAG-Research --skill create-metric-plugin -a cursor

Project install goes to .agents/skills/; add -g for ~/.cursor/skills/.

GitHub CLI

$ gh skill install NomaDamas/AutoRAG-Research create-metric-plugin --agent cursor

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/NomaDamas/AutoRAG-Research.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/.agents/skills/create-metric-plugin .cursor/skills/create-metric-plugin && rm -rf skills-src

Use ~/.cursor/skills/ instead of .cursor/skills for a personal install.

Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/

Paste into Cursor

Install the "create-metric-plugin" agent skill from https://github.com/NomaDamas/AutoRAG-Research/tree/main/.agents/skills/create-metric-plugin into .cursor/skills/create-metric-plugin/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "create-metric-plugin", then confirm the skill loads.

Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

gemini skills install

$ gemini skills install https://github.com/NomaDamas/AutoRAG-Research.git --path .agents/skills/create-metric-plugin

--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.

skills CLI

$ npx skills add NomaDamas/AutoRAG-Research --skill create-metric-plugin -a gemini-cli

Project install goes to .agents/skills/; add -g for ~/.gemini/skills/.

GitHub CLI

$ gh skill install NomaDamas/AutoRAG-Research create-metric-plugin --agent gemini-cli

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/NomaDamas/AutoRAG-Research.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/.agents/skills/create-metric-plugin .gemini/skills/create-metric-plugin && rm -rf skills-src

Use ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.

Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/

Paste into Gemini CLI

Install the "create-metric-plugin" agent skill from https://github.com/NomaDamas/AutoRAG-Research/tree/main/.agents/skills/create-metric-plugin into .gemini/skills/create-metric-plugin/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "create-metric-plugin", then confirm the skill loads.

Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

GitHub CLI

$ gh skill install NomaDamas/AutoRAG-Research create-metric-plugin

Installs for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).

skills CLI

$ npx skills add NomaDamas/AutoRAG-Research --skill create-metric-plugin -a github-copilot

Project install goes to .agents/skills/; add -g for ~/.copilot/skills/.

Manual copy

$ git clone --depth 1 https://github.com/NomaDamas/AutoRAG-Research.git skills-src && mkdir -p .github/skills && cp -r skills-src/.agents/skills/create-metric-plugin .github/skills/create-metric-plugin && rm -rf skills-src

Use ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.

GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/

Paste into GitHub Copilot

Install the "create-metric-plugin" agent skill from https://github.com/NomaDamas/AutoRAG-Research/tree/main/.agents/skills/create-metric-plugin into .github/skills/create-metric-plugin/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "create-metric-plugin", then confirm the skill loads.

GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

skills CLI

$ npx skills add NomaDamas/AutoRAG-Research --skill create-metric-plugin -a opencode

OpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.

GitHub CLI

$ gh skill install NomaDamas/AutoRAG-Research create-metric-plugin --agent opencode

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/NomaDamas/AutoRAG-Research.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/.agents/skills/create-metric-plugin .opencode/skills/create-metric-plugin && rm -rf skills-src

Use ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.

OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/

Paste into OpenCode

Install the "create-metric-plugin" agent skill from https://github.com/NomaDamas/AutoRAG-Research/tree/main/.agents/skills/create-metric-plugin into .opencode/skills/create-metric-plugin/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "create-metric-plugin", then confirm the skill loads.

OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

Facts

Skill name: create-metric-plugin
GitHub stars: 149
Token cost: ~859 tokens
SKILL.md length: 237 words
Files: 1
Skills in repo: 7
Repo updated: Aug 9, 2026
First seen: Oct 7, 2026
Licence: Apache-2.0

At a glance

Guide developers through creating a custom evaluation metric plugin for AutoRAG-Research.

Works in 4 steps: Scaffold → Implement the metric function → Understanding retrieval_gt (AND/OR group… → …
Building a new evaluation metric
SKILL.md covers Workflow, Key Files and Examples
Calls pip

What it does

Create Metric Plugin is an agent skill from NomaDamas/AutoRAG-Research. Guide developers through creating a custom evaluation metric plugin for AutoRAG-Research. Covers both retrieval metrics (recall, precision, etc.) and generation metrics (BLEU, ROUGE, etc.). Walks through scaffolding, implementing metric functions with @metric decorators, writing configs, testing, and installing. Use when building a new evaluation metric.

Its SKILL.md is about 860 tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.

It sits in Development, covering Project scaffolding. The repository describes itself as: Automate your RAG research. The licence is Apache-2.0.

When your agent uses it

Building a new evaluation metric
Tasks that involve Project scaffolding

Example prompts

“/create-metric-plugin”

Requirements

Python 3
Pre-approved tools (allowed-tools): Bash, Read, Write, Edit

Workflow steps

4 steps, taken from the step headings in SKILL.md.

Scaffold
Implement the metric function
Understanding retrieval_gt (AND/OR group structure)
Wire up config and install

What it can do on your machine

Read from SKILL.md and the folder at commit a473cf0. It shows what the files ask for, not the result of running them.

Tool permissions
Pre-approves these tools, so the agent can use them without asking each time:
- Bash
- Read
- Write
- Edit
From allowed-tools in the SKILL.md frontmatter.
Runs code
Shell commands in SKILL.md call:
- pip
From the folder's file list and the shell code blocks in SKILL.md.
Network
No URLs in SKILL.md. Its commands use pip, which can reach the network depending on how they are called.
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Create Metric Plugin loads about 859 tokens when it runs. Until then it costs about 94 tokens; SKILL.md has 237 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it: ~94
When it runs · the whole SKILL.md, loaded when a task matches: ~859

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check: notes

The automated check noted patterns worth knowing about, such as sudo or a known installer.

NotePre-approves every shell command (allowed-tools: Bash)SKILL.md
```
allowed-tools: Bash, Read, Write, Edit
```

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from NomaDamas/AutoRAG-Research at commit a473cf0, republished under its Apache-2.0 licence (© NomaDamas). 237 words, ~859 tokens.

Download SKILL.mdSave it as .claude/skills/create-metric-plugin/SKILL.md (or your agent's skills folder).

name: create-metric-plugin
description: Guide developers through creating a custom evaluation metric plugin for AutoRAG-Research. Covers both retrieval metrics (recall, precision, etc.) and generation metrics (BLEU, ROUGE, etc.). Walks through scaffolding, implementing metric functions with @metric decorators, writing configs, testing, and installing. Use when building a new evaluation metric.
allowed-tools: Bash, Read, Write, Edit

Create Metric Plugin

Workflow

1. Scaffold

bash

# For retrieval metric:
autorag-research plugin create my_metric --type=metric_retrieval

# For generation metric:
autorag-research plugin create my_metric --type=metric_generation

Read the generated metric.py, pyproject.toml, YAML config, and test file to understand the structure.

2. Implement the metric function

Use the @metric decorator (per-input) or @metric_loop decorator (batch) from autorag_research.evaluation.metrics.util. Both validate that required fields are non-None before calling.

@metric(fields_to_check=[...]) — function receives a single MetricInput, returns float
@metric_loop(fields_to_check=[...]) — function receives list[MetricInput], returns list[float]

See autorag_research/schema.py for the full MetricInput dataclass definition.

3. Understanding `retrieval_gt` (AND/OR group structure)

For retrieval metrics, metric_input.retrieval_gt uses a nested list structure with AND/OR semantics:

retrieval_gt: list[list[str]]

Example: [["A", "B"], ["C"]]
  → Means: (A OR B) AND C
  → Each inner list is an OR group (any item satisfies the group)
  → Outer list is AND (ALL groups must be satisfied for complete retrieval)

This is critical for multi-hop queries where multiple evidence pieces are needed. Your metric must handle this structure correctly — don't just flatten it into a single set unless your metric semantics allow it.

Examples:

[["doc1"]] — single required document
[["doc1", "doc2"], ["doc3"]] — need (doc1 OR doc2) AND doc3
[["doc1"], ["doc2"], ["doc3"]] — need doc1 AND doc2 AND doc3

See retrieval_ndcg in autorag_research/evaluation/metrics/retrieval.py for a real implementation that handles AND/OR groups with graded relevance.

4. Wire up config and install

The generated config class just needs get_metric_func() to return your metric function. If your metric takes extra kwargs, override get_metric_kwargs().

bash

cd my_metric_plugin
pip install -e .   # or: uv pip install -e .
cd .. && autorag-research plugin sync

Verify: ls configs/metrics/retrieval/my_metric.yaml (or metrics/generation/)

Key Files

Purpose	Path
Base config classes	`autorag_research/config.py` → `BaseRetrievalMetricConfig`, `BaseGenerationMetricConfig`
MetricInput schema	`autorag_research/schema.py`
Metric decorators	`autorag_research/evaluation/metrics/util.py` → `@metric`, `@metric_loop`
Plugin entry point discovery	`autorag_research/plugin_registry.py`

Examples

Study these existing implementations for patterns:

autorag_research/evaluation/metrics/retrieval.py — Recall, Precision, F1, NDCG, MRR, MAP (all handle AND/OR groups)
autorag_research/evaluation/metrics/generation.py — BLEU, ROUGE, BERTScore, SemScore
YAML configs: configs/metrics/retrieval/f1.yaml, configs/metrics/generation/rouge.yaml

Files

Just SKILL.md in .agents/skills/create-metric-plugin of NomaDamas/AutoRAG-Research.

Open the folder on GitHubat commit a473cf0

Compare with similar skills

Create Metric Plugin next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Create Metric Plugin compared with similar skills
Skill	Stars	Used in	Tokens	Auto-check	Licence	Repo updated
Create Metric Plugin this skillNomaDamas/AutoRAG-Research	149	—	~859	Automated safety check: Notes	Apache-2.0	2 mo ago
Nx Generatenomcopter/react-mosaic	4.8k	7 repos	~1.9k	Automated safety check: Pass	Custom licence	10 days ago
PonytailDavidObando/gsharp	565	8 repos	~1.7k	Automated safety check: Pass	MIT	today
Run Nx Generatornrwl/nx	29k	2 repos	~592	Automated safety check: Notes	MIT	today
Conductor Setupgemini-cli-extensions/conductor	3.8k	—	~4.2k	Automated safety check: Pass	Apache-2.0	1 mo ago
Mirage VFS Adapter Authoringstrukto-ai/mirage	3.7k	—	~2.5k	Automated safety check: Pass	Apache-2.0	today

Similar skills

Nx Generate
nomcopter/react-mosaic
Generate code using nx generators. An agent skill from nomcopter/react-mosaic.
4.8k GitHub starsUsed in 7 repos~1.9k tokens
DevelopmentAuto-check passed
Ponytail
DavidObando/gsharp
Forces the laziest solution that actually works, simplest, shortest, most minimal.
565 GitHub starsUsed in 8 repos~1.7k tokens
DevelopmentAuto-check passed
Run Nx Generator
nrwl/nx
Run Nx generators with prioritization for workspace-plugin generators.
29k GitHub starsUsed in 2 repos~592 tokens
DevelopmentAuto-check: notes
Conductor Setup
gemini-cli-extensions/conductor
Scaffolds the project and sets up the Conductor environment.
3.8k GitHub stars~4.2k tokensUpdated 1 mo ago
DevelopmentAuto-check passed
Mirage VFS Adapter Authoring
strukto-ai/mirage
Builds or extends a custom Mirage virtual filesystem adapter for an API, database, object store or app data, with a working mount configuration and filesystem tests.
3.7k GitHub stars~2.5k tokensUpdated today
DevelopmentAuto-check passed
Siteboon Backend Module Standards
siteboon/claudecodeui
Enforces this repository's TypeScript backend module architecture under server/: feature folders, barrel exports, and where shared types and utilities belong.
14k GitHub stars~1.2k tokensUpdated yesterday
DevelopmentAuto-check passed

Questions about Create Metric Plugin

What does Create Metric Plugin do?

Guide developers through creating a custom evaluation metric plugin for AutoRAG-Research. Create Metric Plugin is an agent skill from NomaDamas/AutoRAG-Research. Guide developers through creating a custom evaluation metric plugin for AutoRAG-Research.

When should I use Create Metric Plugin?

Create Metric Plugin fits situations like: building a new evaluation metric; tasks that involve Project scaffolding.

How do I install Create Metric Plugin in Claude Code?

Run `npx skills add NomaDamas/AutoRAG-Research --skill create-metric-plugin -a claude-code`. Or copy the skill folder (.agents/skills/create-metric-plugin in NomaDamas/AutoRAG-Research) into .claude/skills/create-metric-plugin in your project. Claude Code loads it when a task matches its description.

How do I install Create Metric Plugin in Codex?

Run `npx skills add NomaDamas/AutoRAG-Research --skill create-metric-plugin -a codex`. Or copy the skill folder (.agents/skills/create-metric-plugin in NomaDamas/AutoRAG-Research) into .agents/skills/create-metric-plugin in your project. Codex loads it when a task matches its description.

Can I use Create Metric Plugin in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add NomaDamas/AutoRAG-Research --skill create-metric-plugin -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/create-metric-plugin, .gemini/skills/create-metric-plugin, .github/skills/create-metric-plugin and .opencode/skills/create-metric-plugin in your project.

What does Create Metric Plugin need to run?

Going by SKILL.md and its folder, Create Metric Plugin needs the command-line tools its instructions call (pip). Our summary lists: Python 3. Its frontmatter pre-approves these tools: Bash, Read, Write, Edit.

Does Create Metric Plugin access the network?

SKILL.md contains no URLs. Its commands use pip, which can reach the network depending on how they are called. This is read from the text; nothing was executed.

Is Create Metric Plugin safe to install?

Our automated static check of SKILL.md found notes only (pre-approves every shell command (allowed-tools: bash)), nothing it rates as a warning. It is not a guarantee. Review the folder before installing.

What licence does Create Metric Plugin use?

Create Metric Plugin is published under the Apache-2.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Create Metric Plugin use?

About 859 tokens (SKILL.md is roughly 3.4k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Create Metric Plugin?

Skills that share tags, products or a category with Create Metric Plugin: Nx Generate (nomcopter/react-mosaic, 4.8k stars), Ponytail (DavidObando/gsharp, 565 stars), Run Nx Generator (nrwl/nx, 29k stars) and Conductor Setup (gemini-cli-extensions/conductor, 3.8k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Create Metric Plugin?

NomaDamas (a GitHub organization) maintains it in NomaDamas/AutoRAG-Research, which has 149 GitHub stars. The repository holds 7 skills in this directory. The repository was last updated on August 9, 2026.

Source: NomaDamas/AutoRAG-Research on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.

Install Create Metric Plugin

Facts

At a glance

What it does

When your agent uses it

Example prompts

Requirements

Workflow steps

What it can do on your machine

Tool permissions

Runs code

Network

Credentials

Context cost

Safety

SKILL.md

Create Metric Plugin

Workflow

1. Scaffold

2. Implement the metric function

3. Understanding retrieval_gt (AND/OR group structure)

4. Wire up config and install

Key Files

Examples

Files

Compare with similar skills

Similar skills

More from NomaDamas/AutoRAG-Research

Related

Questions about Create Metric Plugin

3. Understanding `retrieval_gt` (AND/OR group structure)