New: 98,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Agent skill

Benchmark

by serejaris in serejaris/personal-corp-os

A skill your agent uses when choosing between engines or models for an agent pipeline and the answer must come from measurement on your own data, not from marketing pages: speech recognition for…

MITAuto-check passedAI & LLM Engineering

Install Benchmark

skills CLI
$ npx skills add serejaris/personal-corp-os --skill benchmark -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install serejaris/personal-corp-os benchmark --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/serejaris/personal-corp-os.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/benchmark .claude/skills/benchmark && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
benchmark
GitHub stars
229
Token cost
~1.8k tokens
SKILL.md length
947 words
Files
12 (incl. scripts)
Skills in repo
36
Repo updated
First seen
Licence
MIT

At a glance

A skill your agent uses when choosing between engines or models for an agent pipeline and the answer must come from measurement on your own data, not from marketing pages: speech recognition for…

  • Works in 8 steps: Isolation. Run in a dedicated container… → Keys only from env. The container gets a… → Same harness for every model. LLM steps… → …
  • Choosing between engines
  • SKILL.md covers When to use, Rules, Steps and Engines and providers, plus 4 more sections
  • Runs Python scripts from its folder; calls python3 and claude; needs ELEVENLABS_API_KEY and OPENROUTER_API_KEY

What it does

Benchmark is an agent skill from serejaris/personal-corp-os. Use when choosing between engines or models for an agent pipeline and the answer must come from measurement on your own data, not from marketing pages: speech recognition for meeting recordings, the model that turns a transcript into notes, the critic model that checks it. Runs every variant in an isolated container, measures time, cost per hour of input (USD and RUB, with price source and date) and quality (WER/CER and course terms for ASR; code checks plus a judge of a different model for LLM steps), writes…

Its SKILL.md is about 1.8k tokens, which your agent loads only when the skill is triggered. The skill folder holds 13 other files, including scripts (for example `README.md`, `README.ru.md` and `examples/adapter_example.py`).

It sits in AI & LLM Engineering, covering Speech recognition and synthesis and LLM evaluation. The repository describes itself as: Personal Corp OS — управление личной компанией через AI-агентов: задачи вне головы, отделы вместо памяти, недельное ретро. Открытые скиллы для Claude Code и Codex. The licence is MIT.

When your agent uses it

  • Choosing between engines
  • Models for an agent pipeline and the answer must come from measurement on your own data
  • Not from marketing pages: speech recognition for meeting recordings
  • The model that turns a transcript into notes

Example prompts

  • “benchmark the pipeline”
  • “compare ASR engines”
  • “/benchmark”

Requirements

  • Python 3
  • A credential in ELEVENLABS_API_KEY
  • A credential in OPENROUTER_API_KEY

Workflow steps

8 steps, taken from the first numbered list in SKILL.md.

  1. Isolation. Run in a dedicated container (LXC/VM), not on a laptop and not next to production: the timing is clean and the keys live only…
  2. Keys only from env. The container gets a root-only env file; scripts read variable names from the task and never print values. Error…
  3. Same harness for every model. LLM steps run through claude -p (Claude Code headless) against Anthropic-compatible endpoints; only the…
  4. Judge is never the author's model. llm.py refuses such a variant.
  5. No answer leaks. The author must not see the published result of the same input (a «form reference» that is the answer itself). Give it a…
  6. Prices carry source and date. prices.json: every item has source and checked; FX from the central bank of the day. A number without a…
  7. What was not run is a row too. Missing key, no balance, provider closed to new clients: write the reason and the list price, do not drop…
  8. Private input stays private. Transcripts and recordings go to a private repo; the analytics page shows only aggregate numbers.

What it can do on your machine

Read from SKILL.md and the folder at commit 95e36c3. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Ships 6 files in scripts/ (Python), which the agent can run.

    Shell commands in SKILL.md call:

    • python3
    • claude

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    No URLs in SKILL.md.

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names these keys or tokens, usually read from environment variables:

    • ELEVENLABS_API_KEY
    • OPENROUTER_API_KEY
    • ZAI_API_KEY

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Benchmark loads about 1.8k tokens when it runs. Until then it costs about 182 tokens; SKILL.md has 947 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~182
When it runs · the whole SKILL.md, loaded when a task matches
~1.8k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); the scripts in this folder are not scanned.

SKILL.md

The full file from serejaris/personal-corp-os at commit 95e36c3, republished under its MIT licence (© serejaris). 947 words, ~1,849 tokens.

Download SKILL.mdSave it as .claude/skills/benchmark/SKILL.md (or your agent's skills folder). This skill also uses 11 other files; get the full folder from GitHub.
name
benchmark
description
Use when choosing between engines or models for an agent pipeline and the answer must come from measurement on your own data, not from marketing pages: speech recognition for meeting recordings, the model that turns a transcript into notes, the critic model that checks it. Runs every variant in an isolated container, measures time, cost per hour of input (USD and RUB, with price source and date) and quality (WER/CER and course terms for ASR; code checks plus a judge of a different model for LLM steps), writes results.json/csv and one static analytics page. Triggers on "бенчмарк", "сравни движки", "сколько стоит час записи", "какую модель взять для пайплайна", "benchmark the pipeline", "compare ASR engines".

Benchmark

Measure a pipeline on your own input before you pick an engine. One task file lists the variants; the scripts run them one by one in an isolated container and write numbers you can defend: time, cost per hour of input with a dated price source, quality against a reference.

Shape of a pipeline this skill knows:

input (recording) → ASR engine → transcript → author model (notes, chapters) → code checks → judge model

Each arrow is a slot. A variant fills one slot and keeps the rest fixed.

When to use

  • «ElevenLabs, Whisper or a Russian engine: what does an hour of our meetings cost and who makes fewer mistakes»;
  • «move the notes step from Claude to GLM: does quality hold»;
  • before a pipeline goes unattended: pick the critic that finds the most real problems.

Not for: load testing, latency SLOs of a live service, model evals on public datasets.

Rules

  1. Isolation. Run in a dedicated container (LXC/VM), not on a laptop and not next to production: the timing is clean and the keys live only there. Egress: provider APIs; private network closed except the services the task needs (for example a self-hosted ASR).
  2. Keys only from env. The container gets a root-only env file; scripts read variable names from the task and never print values. Error bodies are redacted (some providers echo the key back).
  3. Same harness for every model. LLM steps run through claude -p (Claude Code headless) against Anthropic-compatible endpoints; only the model changes. Tools, prompt, permissions stay the same.
  4. Judge is never the author's model. llm.py refuses such a variant.
  5. No answer leaks. The author must not see the published result of the same input (a «form reference» that is the answer itself). Give it a sibling of the same kind.
  6. Prices carry source and date. prices.json: every item has source and checked; FX from the central bank of the day. A number without a source does not go into the report.
  7. What was not run is a row too. Missing key, no balance, provider closed to new clients: write the reason and the list price, do not drop the row.
  8. Private input stays private. Transcripts and recordings go to a private repo; the analytics page shows only aggregate numbers.

Steps

  1. Task file (task.example.json): input audio, language, terms list, reference transcript, ASR variants, LLM variants with judges, not_run rows with reasons.
  2. Container. Create it by your infra rules (the section «Container» below). Install ffmpeg, Python venv with httpx jiwer faster-whisper, Node and @anthropic-ai/claude-code. Put keys into /etc/bench.env (root 600) through a pipe from your secret store.
  3. ASR: python3 scripts/asr.py --task task.json --out runs/<name> (engines run sequentially; ElevenLabs credits are read before and after).
  4. LLM: write an adapter for your pipeline (contract in scripts/llm.py), then python3 scripts/llm.py --task task.json --out runs/<name> --stage author on the container and --stage judge where the judge's key lives (a subscription login on another machine is fine: the judge only reads files).
  5. Score: python3 scripts/score.py --task task.json --out runs/<name> --prices prices.json.
  6. Page: python3 scripts/report.py --results runs/<name>/results.json --out report.html --notes notes.md.
  7. Method note next to the results: input, reference origin, what was not run and why, known biases.
Show full SKILL.md (409 more words)Show less

Engines and providers

SlotEngine keyTestedNeeds
ASRelevenlabs (Scribe, optional keyterms)yesELEVENLABS_API_KEY
ASRopenai_compat (/v1/audio/transcriptions: self-hosted GigaAM, speaches, cloud)yesserver URL, optional key
ASRfaster_whisper (CPU, int8)yesmodel download once
ASRopenrouter_audio (chat with input_audio, chunked)request path onlyOPENROUTER_API_KEY with balance
LLMzai (GLM via api.z.ai/api/anthropic)yesZAI_API_KEY
LLManthropic-local (the login on this machine)yesClaude Code logged in
LLMopenrouter, deepseek, anthropic-api, anthropic-oauthnot yetkey with balance

Yandex SpeechKit and SaluteSpeech have no adapter yet: add one to ENGINES in asr.py (input: audio path, output: text, raw response, usage) when you have a key.

Metrics

ASR (against the reference, after lower case, ё→е, punctuation and speaker tags removed):

  • wer, cer: word and character error rate;
  • wer_termnorm: WER after known term distortions are fixed in both texts;
  • term_accuracy: share of term mentions written canonically, canon / (canon + known distortions), no reference needed;
  • term_recall: canonical term mentions vs. the reference;
  • asr_pairwise_wer: engines against each other, independent of the reference;
  • rtf: wall time / audio time.

LLM: code checks before and after the fix round, fix rounds, judges' verdicts with critical / major / minor counts, tokens, turns, wall time.

Cost per hour of input: list price (api_price), tokens × list price (tokens), provider-reported cost (reported), or the CPU-seconds share of your server's month (server_share) for self-hosted engines.

Bias checklist (write it into the method note)

  • Where the reference came from. A reference made by engine X favours X in WER; say so and read term_accuracy and pairwise WER next to it.
  • Numbers: engines write «65» or «шестьдесят пять»; WER counts both as errors.
  • Self-hosted time depends on the host's load at run time; note CPU threads and neighbours.
  • One recording is one sample. Treat differences under a couple of WER points as noise.

Container

The skill does not create infrastructure. Follow your infra rules; minimum:

  • a separate container for benchmarks, not a production or user container;
  • no inbound exposure; no SSH if the host can exec into it;
  • outbound: internet for APIs; private networks closed except named services;
  • keys in a root-only env file, services started with EnvironmentFile=, run as an unprivileged user;
  • CPU weight below production neighbours (for example cpuunits lower than default).

Files

  • scripts/asr.py, scripts/llm.py, scripts/claude_step.py, scripts/score.py, scripts/report.py, scripts/common.py
  • task.example.json: a task with every engine type
  • prices.example.json: price table with sources, checked 2026-09-30
  • examples/adapter_example.py: a minimal LLM adapter (author → checks → one fix → judge); run end to end with GLM-5.3-Flash as author and GLM-5.3 as judge

© serejaris, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 11 other files (scripts) in skills/benchmark of serejaris/personal-corp-os.

  • SKILL.md
  • README.md
  • README.ru.md
  • examples/adapter_example.py
  • prices.example.json
  • scripts/asr.py
  • scripts/claude_step.py
  • scripts/common.py
  • scripts/llm.py
  • scripts/report.py
  • scripts/score.py
  • task.example.json

Open the folder on GitHubat commit 95e36c3

Compare with similar skills

Benchmark next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Benchmark compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Benchmark this skillserejaris/personal-corp-os229—~1.8kAutomated safety check: PassMIT
Cxas Agent FoundryGoogleCloudPlatform/cxas-scrapi107—~2.4kAutomated safety check: PassApache-2.0
TriageTalAter/annyang6.8k1 repos~810Automated safety check: NotesMIT
LLM Benchmarking with lm-evaluation-harnessOrchestra-Research/AI-Research-SKILLs13k8 repos~3kAutomated safety check: PassMIT
Azure AI Projects Python SDKmicrosoft/skills3.1k6 repos~2.8kAutomated safety check: PassMIT
Yichen Asrmcncarl/yichen-skills4.3k—~780Automated safety check: PassCustom licence

Similar skills

  • Cxas Agent Foundry

    GoogleCloudPlatform/cxas-scrapi

    End-to-end GECX/CXAS/CES conversational agent lifecycle -- build agents from requirements (PRD-to-agent), create and run evals (goldens, simulations, tool tests, callback tests), debug failures, and…

    107 GitHub stars~2.4k tokensUpdated today
    DevelopmentAuto-check passed

  • Triage

    TalAter/annyang

    Triage and close GitHub issues on TalAter/annyang. An agent skill from TalAter/annyang.

    6.8k GitHub starsUsed in 1 repo~810 tokens
    AI & LLM EngineeringAuto-check: notes

  • LLM Benchmarking with lm-evaluation-harness

    Orchestra-Research/AI-Research-SKILLs

    Runs lm-evaluation-harness to benchmark language models on academic suites such as MMLU, GSM8K and HumanEval, compare models and track training checkpoints.

    13k GitHub starsUsed in 8 repos~3k tokens
    AI & LLM EngineeringAuto-check passed

  • Azure AI Projects Python SDK

    microsoft/skills

    Official

    Reference for building on Microsoft Foundry with the azure-ai-projects Python SDK: project clients, versioned agents, evaluations, connections, datasets and indexes.

    3.1k GitHub starsUsed in 6 repos~2.8k tokens
    AI & LLM EngineeringAuto-check passed

  • Yichen Asr

    mcncarl/yichen-skills

    逸尘自用的统一音视频转写入口，在 StepFun Step ASR 与火山引擎豆包 ASR 之间按输出需求、安全边界和可用状态路由。用于本地音频或视频的纯文本转写、时间戳、SRT 字幕、口播粗剪，以及转写前体检；用户明确指定服务商时不得静默切换。Use when a local audio or video file needs transcription and the correct…

    4.3k GitHub stars~780 tokensUpdated 4 days ago
    AI & LLM EngineeringAuto-check passed

  • Fine-Tuning Expert

    Jeffallan/claude-skills

    Guides LLM fine-tuning with LoRA and QLoRA through Hugging Face PEFT, from dataset validation and training checks to adapter merging, quantization and deployment.

    12k GitHub starsUsed in 1 repo~1.7k tokens
    AI & LLM EngineeringAuto-check passed

More from serejaris/personal-corp-os

All 36 skills in this repo

  • Weekly Planning

    serejaris/personal-corp-os

    A skill your agent uses when the user is transitioning from a completed retro into a weekly plan, choosing weekly outcomes, scheduling a full ISO week, or asking for "план на неделю", "weekly…

    229 GitHub stars~2.4k tokensUpdated yesterday
    Auto-check passed

  • Gh Issues

    serejaris/personal-corp-os

    A skill your agent uses when creating, searching, updating, or managing GitHub issues via CLI.

    229 GitHub stars~1.8k tokensUpdated yesterday
    Auto-check passed

  • Product Data Audit

    serejaris/personal-corp-os

    A skill your agent uses when auditing a product, business, or project ecosystem — analyzing data sources, decision loops, bottlenecks, and implementation contours.

    229 GitHub stars~1.3k tokensUpdated yesterday
    Auto-check passed

  • Tg Bot Ops

    serejaris/personal-corp-os

    A skill your agent uses when operating, debugging, deploying, or monitoring a Telegram bot or Telegram-to-agent gateway.

    229 GitHub stars~1.6k tokensUpdated yesterday
    Auto-check: notes

  • Audit Agent Rules

    serejaris/personal-corp-os

    Audits the agent rules in the current folder (AGENTS.md, nested AGENTS.md files) and the skill descriptions the agent sees at start, then reports what to cut, move or rewrite and edits only after…

    229 GitHub stars~2.3k tokensUpdated yesterday
    Auto-check passed

  • Make Landing

    serejaris/personal-corp-os

    Создаёт несколько вариантов дизайна 2D-поверхности (лендинг, герой, обложка, слайды): свой визуальный референс и автор на вариант, полный design.md с UTC/SHA-256 до кода, проверка в браузере…

    229 GitHub stars~2.1k tokensUpdated yesterday
    Auto-check passed

Categories

Questions about Benchmark

What does Benchmark do?

A skill your agent uses when choosing between engines or models for an agent pipeline and the answer must come from measurement on your own data, not from marketing pages: speech recognition for…. Benchmark is an agent skill from serejaris/personal-corp-os. Use when choosing between engines or models for an agent pipeline and the answer must come from measurement on your own data, not from marketing pages: speech recognition for meeting recordings, the model that turns a transcript into notes, the critic model that checks it.

When should I use Benchmark?

Benchmark fits situations like: choosing between engines; models for an agent pipeline and the answer must come from measurement on your own data; not from marketing pages: speech recognition for meeting recordings; the model that turns a transcript into notes.

How do I install Benchmark in Claude Code?

Run `npx skills add serejaris/personal-corp-os --skill benchmark -a claude-code`. Or copy the skill folder (skills/benchmark in serejaris/personal-corp-os) into .claude/skills/benchmark in your project. Claude Code loads it when a task matches its description.

How do I install Benchmark in Codex?

Run `npx skills add serejaris/personal-corp-os --skill benchmark -a codex`. Or copy the skill folder (skills/benchmark in serejaris/personal-corp-os) into .agents/skills/benchmark in your project. Codex loads it when a task matches its description.

Can I use Benchmark in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add serejaris/personal-corp-os --skill benchmark -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/benchmark, .gemini/skills/benchmark, .github/skills/benchmark and .opencode/skills/benchmark in your project.

What does Benchmark need to run?

Going by SKILL.md and its folder, Benchmark needs Python for the scripts in its folder, the command-line tools its instructions call (python3 and claude) and credentials named ELEVENLABS_API_KEY, OPENROUTER_API_KEY and ZAI_API_KEY. Our summary lists: Python 3; A credential in ELEVENLABS_API_KEY; A credential in OPENROUTER_API_KEY.

Does Benchmark access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Benchmark safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. The check reads SKILL.md only: the scripts in the folder are not scanned, so read them before running anything.

What licence does Benchmark use?

Benchmark is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Benchmark use?

About 1.8k tokens (SKILL.md is roughly 7.4k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Benchmark?

Skills that share tags, products or a category with Benchmark: Cxas Agent Foundry (GoogleCloudPlatform/cxas-scrapi, 107 stars), Triage (TalAter/annyang, 6.8k stars), LLM Benchmarking with lm-evaluation-harness (Orchestra-Research/AI-Research-SKILLs, 13k stars) and Azure AI Projects Python SDK (microsoft/skills, 3.1k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Benchmark?

serejaris (a GitHub user) maintains it in serejaris/personal-corp-os, which has 229 GitHub stars. The repository holds 36 skills in this directory. The repository was last updated on October 7, 2026.

Source: serejaris/personal-corp-os on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.