New: 98,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Agent skill

Bmad Eval Runner

by redhat-cop in redhat-cop/vault-config-operator

Run a skill's evals and report results. An agent skill from redhat-cop/vault-config-operator.

Apache-2.0Auto-check passedAI & LLM Engineering

Install Bmad Eval Runner

skills CLI
$ npx skills add redhat-cop/vault-config-operator --skill bmad-eval-runner -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install redhat-cop/vault-config-operator bmad-eval-runner --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/redhat-cop/vault-config-operator.git skills-src && mkdir -p .claude/skills && cp -r skills-src/.agents/skills/bmad-eval-runner .claude/skills/bmad-eval-runner && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
bmad-eval-runner
GitHub stars
167
Token cost
~2.1k tokens
SKILL.md length
1,054 words
Files
13 (incl. scripts, references, assets)
Skills in repo
19
Repo updated
First seen
Licence
Apache-2.0

At a glance

Run a skill's evals and report results. An agent skill from redhat-cop/vault-config-operator.

  • Works in 7 steps: Resolve config the way… → If --headless was passed, set… → Resume check: glob the output dir for an… → …
  • The user wants to evaluate a skill
  • SKILL.md covers The four modes, Args, On activation and Run execution, plus 2 more sections
  • Runs Python scripts from its folder; calls python3

What it does

Bmad Eval Runner is an agent skill from redhat-cop/vault-config-operator. Run a skill's evals and report results. Use when the user wants to evaluate a skill, run evals, benchmark a skill, validate triggers, optimize a description, or grade skill outputs.

Its SKILL.md is about 2.1k tokens, which your agent loads only when the skill is triggered. The skill folder holds 16 other files, including scripts, reference files and assets (for example `assets/adapter-claude-code.json`, `references/description-optimization.md` and `references/eval-format.md`).

It sits in AI & LLM Engineering, covering LLM evaluation. The repository describes itself as: An operator to support Haschicorp Vault configuration workflows from within Kubernetes. The licence is Apache-2.0.

When your agent uses it

  • The user wants to evaluate a skill
  • Benchmark a skill
  • Validate triggers
  • Optimize a description

Example prompts

  • “/bmad-eval-runner”

Requirements

  • Python 3

Workflow steps

7 steps, taken from the first numbered list in SKILL.md.

  1. Resolve config the way bmad-workflow-builder does ({project-root}/_bmad/config.yaml then config.user.yaml, falling back to…
  2. If --headless was passed, set {headless_mode}=true, skip every confirmation below, pick the safest defaults, and proceed.
  3. Resume check: glob the output dir for an in-progress run's .memlog.md. If one exists and matches this skill, read it once to rebuild…
  4. Locate the skill and verify /SKILL.md exists. Halt with a clear error if it does not.
  5. Resolve the adapter config per the discovery rules in references/platform-adapter.md (explicit --adapter, BMAD_EVAL_ADAPTER, adapter.json…
  6. Discover the cases file. Look at --evals first, then /evals/, then /../../evals//, then /evals//, then anywhere under /evals/. Take the…
  7. Confirm the run summary (skill, cases found, modes, output dir) unless headless, then execute.

What it can do on your machine

Read from SKILL.md and the folder at commit 99762c2. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Ships 6 files in scripts/ (Python), which the agent can run.

    Shell commands in SKILL.md call:

    • python3

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    No URLs in SKILL.md.

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Bmad Eval Runner loads about 2.1k tokens when it runs, and up to ~10k if it reads all its reference files. Until then it costs about 50 tokens; SKILL.md has 1,054 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~50
When it runs · the whole SKILL.md, loaded when a task matches
~2.1k
With references · SKILL.md plus every file in references/, read only if the agent opens them
~10k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); the scripts in this folder are not scanned.

SKILL.md

The full file from redhat-cop/vault-config-operator at commit 99762c2, republished under its Apache-2.0 licence (© redhat-cop). 1,054 words, ~2,055 tokens.

Download SKILL.mdSave it as .claude/skills/bmad-eval-runner/SKILL.md (or your agent's skills folder). This skill also uses 12 other files; get the full folder from GitHub.
name
bmad-eval-runner
description
Run a skill's evals and report results. Use when the user wants to evaluate a skill, run evals, benchmark a skill, validate triggers, optimize a description, or grade skill outputs.

Skill Eval Runner

You run a skill's evals and report what they say. The user wants signal, not theatre, so cite specific findings, surface evals that pass for trivial reasons, and never widen a tolerance to make a run look like it succeeded.

The runner is platform-agnostic. Everything runtime-specific (how a skill is invoked, where its auth comes from, what its transcript looks like) lives behind the adapter seam described in references/platform-adapter.md. No model name is hardcoded anywhere in this skill.

The four modes

Each mode answers a different question about a skill. Pick the one that matches what the user is asking, or run several.

ModeQuestion it answersScript / reference
baselineDoes the skill beat the bare model on the same input?references/eval-format.md, scripts/run_evals.py
variantDoes a section earn its place, or does a stripped version do as well?references/eval-format.md, scripts/run_evals.py
qualityDoes the output meet the named rubric?references/grader.md, references/eval-format.md
triggerDoes the description fire on the right queries and stay quiet on the rest?references/platform-adapter.md, scripts/run_triggers.py

Baseline runs every case twice — once with the skill staged into the clean working directory and once with nothing staged — so the bare model is measured as the long-term floor under identical conditions. Variant runs the full skill against a stripped smallest-version of itself to settle whether a section is doing real work. Quality grades one config's output against a rubric with the read-only grader. Trigger measures real firing through the adapter and can optimize the description across rounds; the optimization loop lives in references/description-optimization.md.

A case is input + rubric + optional state_prefix + optional fixture files. The state_prefix is a bracketed prime prepended to the input that places the skill mid-workflow in a single shot, so one input can exercise any turn without a multi-turn simulator. The full case format and the strong-versus-weak expectation taxonomy are in references/eval-format.md.

Args

  • Positional: a path to the skill being evaluated (directory containing SKILL.md).
  • --evals <path>: explicit path to the cases file. If omitted, discover.
  • --mode baseline|variant|quality|trigger: which mode to run. May be repeated.
  • --variant-path <path>: for variant mode, the stripped or prior-version skill to compare against.
  • --project-root <path>: root of the project the skill belongs to. Default: walk up from the skill path looking for _bmad/ or .git/.
  • --output-dir <path>: where run folders are written. Default: {bmad_builder_reports}/eval-runs/ if configured, else ~/bmad-evals/.
  • --runs <n>: repeats per case for the variance benchmark. Default: 1 for a single check, higher when the user wants a stable mean.
  • --headless / -H: non-interactive; emit final JSON only.

These map directly onto the script CLIs below; anything not listed there (case subsets, timeouts, workers) is in the script docstrings.

On activation

  1. Resolve config the way bmad-workflow-builder does ({project-root}/_bmad/config.yaml then config.user.yaml, falling back to bmb/config.yaml). Resolve {user_name}, {communication_language}, {bmad_builder_reports} and apply them through the session.

  2. If --headless was passed, set {headless_mode}=true, skip every confirmation below, pick the safest defaults, and proceed.

  3. Resume check: glob the output dir for an in-progress run's .memlog.md. If one exists and matches this skill, read it once to rebuild state, then continue append-only. Capture decisions and direction changes into the run's memlog through scripts/memlog.py as they land.

  4. Locate the skill and verify <skill-path>/SKILL.md exists. Halt with a clear error if it does not.

  5. Resolve the adapter config per the discovery rules in references/platform-adapter.md (explicit --adapter, BMAD_EVAL_ADAPTER, adapter.json beside the cases file). When nothing is configured and the current runtime is Claude Code, use {skill-root}/assets/adapter-claude-code.json.

  6. Discover the cases file. Look at --evals first, then <skill-path>/evals/, then <skill-path>/../../evals/<skill-name>/, then <project-root>/evals/<skill-name>/, then anywhere under <project-root>/evals/. Take the first match. If nothing is found, halt and say so; the runner does not invent cases.

  7. Confirm the run summary (skill, cases found, modes, output dir) unless headless, then execute.

Show full SKILL.md (430 more words)Show less

Run execution

Each case runs in a clean working directory with the skill under test staged into it and an environment built from scratch, so the host shell config, prior runs, and ancestor instruction files do not bias the result. The isolation contract lives in references/platform-adapter.md; there is no container, no terminal emulation, and no credential staging.

For baseline, variant, and quality modes:

python3 {skill-root}/scripts/run_evals.py \
  --cases <cases-file> --skill-path <skill> --output-dir <dir> \
  --mode quality|baseline|variant [--variant-path <skill>] \
  [--adapter <adapter.json>] [--runs N]

The script stages the skill and any case fixtures, applies any state_prefix to the input, runs each config (baseline = skill staged AND bare; variant = skill AND --variant-path), and writes <run-dir>/<config>/<case-id>/. It captures timing and token counts the moment each invocation completes and writes them to timing.json immediately, so a later crash never loses the measurement.

For trigger mode:

python3 {skill-root}/scripts/run_triggers.py \
  --skill-path <skill> --queries <queries-file> --output-dir <dir> \
  [--adapter <adapter.json>] [--runs-per-query N]

It stages a synthetic skill where the runtime discovers skills, sends each query through the adapter, and detects the skill-load tool call. Each query runs several times for stability. When the user wants to optimize the description rather than just measure it, follow references/description-optimization.md.

For quality mode, spawn the grader described in references/grader.md per case, passing the case's rubric, transcript path, artifacts dir (the case's cwd/), and a grading_path of <case-folder>/grading.json. The grader writes that file, gives no partial credit, and flags weak or non-discriminating assertions; relay that feedback. If a grader subagent errors, mark that case grading_error — never substitute a default verdict.

When --runs is greater than one, call python3 {skill-root}/scripts/aggregate_benchmark.py --baseline <run-dir>/<config-a> --variant <run-dir>/<config-b> to produce the mean, sample standard deviation, min, max, and the delta between configs (--runs <run-dir>/<config> for a single config's spread).

When a run fails or comes back weak and the user wants the skill improved from the results, follow references/self-improvement.md.

Artifacts

Every run writes a dated run folder under the output dir, and those artifacts are permanent. Each case folder holds its prompt, transcript, the cwd/ with any files the skill wrote, timing.json, and grading.json when quality mode ran. Never delete, overwrite, or rotate a run folder; disk usage is the user's call. The run's .memlog.md records the decisions and deltas so a resumed or audited run reads back cleanly.

Tell the user where the run folder is when you finish.

Outcomes

  • The run reflects the skill's behavior in a clean working directory, not the behavior of the host shell with its memories and configs.
  • Timing and token counts land on disk the moment they are measured.
  • Failures cite specific expectations with evidence, and a pass that looks superficial is flagged rather than papered over.
  • A baseline run that the skill no longer wins points to retiring the skill, not patching it.

© redhat-cop, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 12 other files (scripts, references, assets) in .agents/skills/bmad-eval-runner of redhat-cop/vault-config-operator.

  • SKILL.md
  • assets/adapter-claude-code.json
  • references/description-optimization.md
  • references/eval-format.md
  • references/grader.md
  • references/platform-adapter.md
  • references/self-improvement.md
  • scripts/aggregate_benchmark.py
  • scripts/memlog.py
  • scripts/run_evals.py
  • scripts/run_triggers.py
  • scripts/tests/test_env_isolation.py
  • scripts/tests/test_trigger_detection.py

Open the folder on GitHubat commit 99762c2

Compare with similar skills

Bmad Eval Runner next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Bmad Eval Runner compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Bmad Eval Runner this skillredhat-cop/vault-config-operator167—~2.1kAutomated safety check: PassApache-2.0
LLM Benchmarking with lm-evaluation-harnessOrchestra-Research/AI-Research-SKILLs13k8 repos~3kAutomated safety check: PassMIT
Azure AI Projects Python SDKmicrosoft/skills3.1k6 repos~2.8kAutomated safety check: PassMIT
Fine-Tuning ExpertJeffallan/claude-skills12k1 repos~1.7kAutomated safety check: PassMIT
Looperksimback/looper710—~2.7kAutomated safety check: NotesMIT
Hugging Face Local Model Evalshuggingface/skills11k2 repos~1.6kAutomated safety check: PassApache-2.0

Similar skills

  • LLM Benchmarking with lm-evaluation-harness

    Orchestra-Research/AI-Research-SKILLs

    Runs lm-evaluation-harness to benchmark language models on academic suites such as MMLU, GSM8K and HumanEval, compare models and track training checkpoints.

    13k GitHub starsUsed in 8 repos~3k tokens
    AI & LLM EngineeringAuto-check passed

  • Azure AI Projects Python SDK

    microsoft/skills

    Official

    Reference for building on Microsoft Foundry with the azure-ai-projects Python SDK: project clients, versioned agents, evaluations, connections, datasets and indexes.

    3.1k GitHub starsUsed in 6 repos~2.8k tokens
    AI & LLM EngineeringAuto-check passed

  • Fine-Tuning Expert

    Jeffallan/claude-skills

    Guides LLM fine-tuning with LoRA and QLoRA through Hugging Face PEFT, from dataset validation and training checks to adapter merging, quantization and deployment.

    12k GitHub starsUsed in 1 repo~1.7k tokens
    AI & LLM EngineeringAuto-check passed

  • Looper

    ksimback/looper

    Scaffold a well-designed agent loop with best-practice coaching and a cross-model review council.

    710 GitHub stars~2.7k tokensUpdated 2 mo ago
    AI & LLM EngineeringAuto-check: notes

  • Hugging Face Local Model Evals

    huggingface/skills

    Official

    Runs evaluations of Hugging Face Hub models on local hardware with inspect-ai or lighteval, and helps choose between vLLM, Transformers and accelerate backends.

    11k GitHub starsUsed in 2 repos~1.6k tokens
    AI & LLM EngineeringAuto-check passed

  • Agent Eval Engineering

    langchain-ai/langchain-skills

    Official

    Builds agent evaluations in stages: inspect the repository and traces, agree a Task Spec with you, then build, audit and run a Harbor task with an independent verifier.

    1.3k GitHub stars~4k tokensUpdated 3 days ago
    AI & LLM EngineeringAuto-check passed

More from redhat-cop/vault-config-operator

All 19 skills in this repo

  • Bmad Bmb Setup

    redhat-cop/vault-config-operator

    Sets up BMad Builder module in a project. An agent skill from redhat-cop/vault-config-operator.

    167 GitHub starsUsed in 2 repos~2.1k tokens
    Auto-check passed

  • Bmad Agent Builder

    redhat-cop/vault-config-operator

    Builds, edits or analyzes Agent Skills through conversational discovery.

    167 GitHub starsUsed in 1 repo~1.4k tokens
    Auto-check passed

  • Bmad Workflow Builder

    redhat-cop/vault-config-operator

    Builds, converts, and analyzes workflows and skills. An agent skill from redhat-cop/vault-config-operator.

    167 GitHub starsUsed in 1 repo~1.4k tokens
    Auto-check passed

  • Bmad Init

    redhat-cop/vault-config-operator

    Initialize BMad project configuration and load config variables.

    167 GitHub stars~1.2k tokensUpdated 2 days ago
    Auto-check passed

  • Bmad Module Builder

    redhat-cop/vault-config-operator

    Plans, creates, and validates BMad modules. An agent skill from redhat-cop/vault-config-operator.

    167 GitHub stars~643 tokensUpdated 2 days ago
    Auto-check passed

  • Bmad Workflow Builder

    redhat-cop/vault-config-operator

    Builds, edits, and analyzes workflows and skills. An agent skill from redhat-cop/vault-config-operator.

    167 GitHub stars~1.1k tokensUpdated 2 days ago
    Auto-check passed

Categories

Questions about Bmad Eval Runner

What does Bmad Eval Runner do?

Run a skill's evals and report results. An agent skill from redhat-cop/vault-config-operator. Bmad Eval Runner is an agent skill from redhat-cop/vault-config-operator. Run a skill's evals and report results.

When should I use Bmad Eval Runner?

Bmad Eval Runner fits situations like: the user wants to evaluate a skill; benchmark a skill; validate triggers; optimize a description.

How do I install Bmad Eval Runner in Claude Code?

Run `npx skills add redhat-cop/vault-config-operator --skill bmad-eval-runner -a claude-code`. Or copy the skill folder (.agents/skills/bmad-eval-runner in redhat-cop/vault-config-operator) into .claude/skills/bmad-eval-runner in your project. Claude Code loads it when a task matches its description.

How do I install Bmad Eval Runner in Codex?

Run `npx skills add redhat-cop/vault-config-operator --skill bmad-eval-runner -a codex`. Or copy the skill folder (.agents/skills/bmad-eval-runner in redhat-cop/vault-config-operator) into .agents/skills/bmad-eval-runner in your project. Codex loads it when a task matches its description.

Can I use Bmad Eval Runner in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add redhat-cop/vault-config-operator --skill bmad-eval-runner -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/bmad-eval-runner, .gemini/skills/bmad-eval-runner, .github/skills/bmad-eval-runner and .opencode/skills/bmad-eval-runner in your project.

What does Bmad Eval Runner need to run?

Going by SKILL.md and its folder, Bmad Eval Runner needs Python for the scripts in its folder and the command-line tools its instructions call (python3). Our summary lists: Python 3.

Does Bmad Eval Runner access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Bmad Eval Runner safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. The check reads SKILL.md only: the scripts in the folder are not scanned, so read them before running anything.

What licence does Bmad Eval Runner use?

Bmad Eval Runner is published under the Apache-2.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Bmad Eval Runner use?

About 2.1k tokens (SKILL.md is roughly 8.2k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full. Its references folder adds about 7.9k tokens, read only when the agent opens those files.

What are the alternatives to Bmad Eval Runner?

Skills that share tags, products or a category with Bmad Eval Runner: LLM Benchmarking with lm-evaluation-harness (Orchestra-Research/AI-Research-SKILLs, 13k stars), Azure AI Projects Python SDK (microsoft/skills, 3.1k stars), Fine-Tuning Expert (Jeffallan/claude-skills, 12k stars) and Looper (ksimback/looper, 710 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Bmad Eval Runner?

redhat-cop (a GitHub organization) maintains it in redhat-cop/vault-config-operator, which has 167 GitHub stars. The repository holds 19 skills in this directory. The repository was last updated on October 7, 2026.

Source: redhat-cop/vault-config-operator on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.