New: 98,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Agent skill

Caliper Harness Smoke Test

by edonadei in edonadei/caliper

Runs caliper's smoke evals against the real agent CLIs after a harness or MCP change, with a dry-run plan, failure triage and a report to attach to the PR.

MITAuto-check passedTesting & QA

Install Caliper Harness Smoke Test

skills CLI
$ npx skills add edonadei/caliper --skill smoke-harness -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install edonadei/caliper smoke-harness --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/edonadei/caliper.git skills-src && mkdir -p .claude/skills && cp -r skills-src/.agents/skills/smoke-harness .claude/skills/smoke-harness && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
smoke-harness
GitHub stars
209
Token cost
~664 tokens
SKILL.md length
374 words
Files
2 (incl. scripts)
Skills in repo
3
Repo updated
First seen
Licence
MIT

At a glance

Runs caliper's smoke evals against the real agent CLIs after a harness or MCP change, with a dry-run plan, failure triage and a report to attach to the PR.

  • Works in 4 steps: Plan. From the repo root, with the dev… → Run the same command without --dry-run.… → Read every failure before touching code.… → …
  • Checking a backend harness change against the actual agent CLIs before opening a PR
  • SKILL.md covers Steps and Adding a smoke spec
  • Runs Python scripts from its folder; calls python

What it does

Unit tests fake the agent CLIs, while the smoke evals in `tests/*-smoke.eval.yaml` run the real ones, so they are the proof that a harness change works end to end. They call paid models, which is why CI skips them and why only the backends your change reaches should run. The agent first runs `scripts/smoke.py --dry-run`, which picks the backends the branch touched and skips any whose CLI cannot be found. A backend skipped for a missing CLI is reported as unverified, never as a pass.

The real run executes each spec once and checks the saved results JSON for one passing attempt per task, a transcript, no hook failures and the right backend. Exit code 0 means every check passed, 1 lists failures and 2 means nothing ran. The agent reads every failure before changing code and sorts it as your change, a flake that earns one rerun, or an environment issue such as a CLI not logged in or a busy port. The final report holds the plan, a result line per spec and anything unverified, and a new smoke spec needs an entry in the script's SPECS list.

When your agent uses it

  • Checking a backend harness change against the actual agent CLIs before opening a PR
  • Verifying MCP wiring changes end to end
  • Writing the evidence section of a PR that touches the runner, attempt, workdir or judge

Example prompts

  • “I changed the codex harness. Plan and run the smoke evals for the backends it reaches.”
  • “Dry-run the smoke harness for this branch and tell me which backends will be skipped.”
  • “The mcp-header-smoke run failed. Work out whether it is my change, a flake or the environment.”

Requirements

  • The caliper dev environment
  • Installed and logged-in agent CLIs for the backends under test
  • Access to paid models, since the evals call them

Workflow steps

4 steps, taken from the first numbered list in SKILL.md.

  1. Plan. From the repo root, with the dev environment active
  2. Run the same command without --dry-run. Each spec runs once (--k 1) and the script checks its saved results JSON: one attempt per task…
  3. Read every failure before touching code. Each line names the task, the outcome, and the judge's or assert's evidence; the full results…
  4. Report the plan, the result line per spec, and anything unverified. That report is the PR's evidence for a harness change.

What it can do on your machine

Read from SKILL.md and the folder at commit 290ba6c. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Ships 1 file in scripts/ (Python), which the agent can run.

    Shell commands in SKILL.md call:

    • python

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    No URLs in SKILL.md.

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Caliper Harness Smoke Test loads about 664 tokens when it runs. Until then it costs about 55 tokens; SKILL.md has 374 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~55
When it runs · the whole SKILL.md, loaded when a task matches
~664

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); the scripts in this folder are not scanned.

SKILL.md

The full file from edonadei/caliper at commit 290ba6c, republished under its MIT licence (© edonadei). 374 words, ~664 tokens.

Download SKILL.mdSave it as .claude/skills/smoke-harness/SKILL.md (or your agent's skills folder). This skill also uses 1 other file; get the full folder from GitHub.
name
smoke-harness
description
Smoke-test caliper's harness against the real agent CLIs. Use after changing a backend harness, MCP wiring, or anything an attempt runs through (runner, attempt, workdir, judge), and before opening that PR.

Smoke the harness

Unit tests fake the agent CLIs. The smoke evals in tests/*-smoke.eval.yaml run the real ones, so they are the evidence that a harness change works end to end. They call paid models, which is why CI skips them and why you run only the backends your change reaches.

Steps

  1. Plan. From the repo root, with the dev environment active:

    bash
    python .agents/skills/smoke-harness/scripts/smoke.py --dry-run

    With no arguments the script picks the backends this branch touched (a backend's own harness file reaches that backend; shared caliper/ code reaches all of them) and skips any whose CLI caliper cannot find (its *_CLI_PATH override, install locations, then PATH). Name backends to override: smoke.py codex pi, or pin a model with claude-code:claude-haiku-4-5-20251001. Done when the plan lists a run for every backend your change reaches. A backend skipped for a missing CLI is unverified: say so in your report and in the PR, never as a pass.

  2. Run the same command without --dry-run. Each spec runs once (--k 1) and the script checks its saved results JSON: one attempt per task, every attempt pass, a transcript saved, no hook failures, the requested backend recorded, user customizations off. Done when the script exits. Exit 0 means every check passed; 1 lists each failing check under its spec; 2 means nothing could run.

  3. Read every failure before touching code. Each line names the task, the outcome, and the judge's or assert's evidence; the full results JSON sits in the directory printed on the last line. Sort each failure into one bucket:

    • Your change: fix it and rerun from step 2.
    • Flake: an infra_error/timeout, or a judge misreading a correct transcript. Rerun that backend once; a second failure is not a flake.
    • Environment: CLI not logged in, the network down for mcp-remote-smoke, port 8765 taken for mcp-header-smoke. Report it as unverified.

    Done when every failure sits in a bucket and every your change failure passes on rerun.

  4. Report the plan, the result line per spec, and anything unverified. That report is the PR's evidence for a harness change.

Show full SKILL.md (31 more words)Show less

Adding a smoke spec

A new tests/*-smoke.eval.yaml needs an entry in SPECS in scripts/smoke.py naming the backends it runs under, matching the "Run it" comment at the top of the spec.

© edonadei, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 1 other file (scripts) in .agents/skills/smoke-harness of edonadei/caliper.

  • SKILL.md
  • scripts/smoke.py

Open the folder on GitHubat commit 290ba6c

Compare with similar skills

Caliper Harness Smoke Test next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Caliper Harness Smoke Test compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Caliper Harness Smoke Test this skilledonadei/caliper209—~664Automated safety check: PassMIT
OpenHarness End-to-End EvalsHKUDS/OpenHarness16k1 repos~2.1kAutomated safety check: NotesMIT
Qwen Code E2E TestingQwenLM/qwen-code28k—~2.1kAutomated safety check: PassApache-2.0
Senpi Agent QA Harnesscode-yeongyu/senpi474—~2.7kAutomated safety check: NotesMIT
Exploring The WizardPostHog/wizard197—~2.2kAutomated safety check: PassMIT
Edt MCP TestingDitriXNew/EDT-MCP296—~2.1kAutomated safety check: PassAGPL-3.0

Similar skills

  • OpenHarness End-to-End Evals

    HKUDS/OpenHarness

    Validates OpenHarness features by running real multi-turn agent loops with live LLM calls against an unfamiliar codebase, checking actual tool execution.

    16k GitHub starsUsed in 1 repo~2.1k tokens
    Testing & QAAuto-check: notes

  • Qwen Code E2E Testing

    QwenLM/qwen-code

    Guides end-to-end testing of the Qwen Code CLI in headless mode with real model calls, MCP test servers and inspection of raw API traffic.

    28k GitHub stars~2.1k tokensUpdated today
    Testing & QAAuto-check passed

  • Senpi Agent QA Harness

    code-yeongyu/senpi

    Checks changes to the senpi coding agent by driving the real CLI from source in an isolated sandbox, over RPC, terminal UI, mock model and CLI smoke channels.

    474 GitHub stars~2.7k tokensUpdated today
    Testing & QAAuto-check: notes

  • Exploring The Wizard

    PostHog/wizard

    Official

    Drive the PostHog wizard headlessly against a throwaway app through wizard-ci MCP tools, inspect decisions, and capture the real TUI.

    197 GitHub stars~2.2k tokensUpdated yesterday
    Testing & QAAuto-check passed

  • Edt MCP Testing

    DitriXNew/EDT-MCP

    How to manually e2e-test each EDT-MCP server tool against a live EDT workbench + TestConfiguration.

    296 GitHub stars~2.1k tokensUpdated 3 days ago
    Testing & QAAuto-check passed

  • Reticle Runtime Verification

    reticlehq/reticle

    Installs Reticle's dev-only SDK in a running web app and verifies user-facing changes by driving a real flow, returning a verdict with the file and line to fix.

    1.2k GitHub stars~2.8k tokensUpdated yesterday
    Testing & QAAuto-check: notes

More from edonadei/caliper

  • Evaluate Skill with Caliper

    edonadei/caliper

    Runs and interprets a skill's Caliper eval: how often it succeeds over repeated attempts, whether it triggers at all, and whether it beats the agent without it.

    209 GitHub stars~2k tokensUpdated yesterday
    Auto-check: notes

  • Grill Skill Eval Designer

    edonadei/caliper

    Interviews you about what a skill should do, writes an eval spec from your answers, then loops through run, diagnose and improve until the skill is ready to ship.

    209 GitHub stars~2.1k tokensUpdated yesterday
    Auto-check: notes

Works with

Categories

Questions about Caliper Harness Smoke Test

What does Caliper Harness Smoke Test do?

Runs caliper's smoke evals against the real agent CLIs after a harness or MCP change, with a dry-run plan, failure triage and a report to attach to the PR. yaml` run the real ones, so they are the proof that a harness change works end to end. They call paid models, which is why CI skips them and why only the backends your change reaches should run.

When should I use Caliper Harness Smoke Test?

Caliper Harness Smoke Test fits situations like: checking a backend harness change against the actual agent CLIs before opening a PR; verifying MCP wiring changes end to end; writing the evidence section of a PR that touches the runner, attempt, workdir or judge.

How do I install Caliper Harness Smoke Test in Claude Code?

Run `npx skills add edonadei/caliper --skill smoke-harness -a claude-code`. Or copy the skill folder (.agents/skills/smoke-harness in edonadei/caliper) into .claude/skills/smoke-harness in your project. Claude Code loads it when a task matches its description.

How do I install Caliper Harness Smoke Test in Codex?

Run `npx skills add edonadei/caliper --skill smoke-harness -a codex`. Or copy the skill folder (.agents/skills/smoke-harness in edonadei/caliper) into .agents/skills/smoke-harness in your project. Codex loads it when a task matches its description.

Can I use Caliper Harness Smoke Test in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add edonadei/caliper --skill smoke-harness -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/smoke-harness, .gemini/skills/smoke-harness, .github/skills/smoke-harness and .opencode/skills/smoke-harness in your project.

What does Caliper Harness Smoke Test need to run?

Going by SKILL.md and its folder, Caliper Harness Smoke Test needs Python for the scripts in its folder and the command-line tools its instructions call (python). Our summary lists: The caliper dev environment; Installed and logged-in agent CLIs for the backends under test; Access to paid models, since the evals call them.

Does Caliper Harness Smoke Test access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Caliper Harness Smoke Test safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. The check reads SKILL.md only: the scripts in the folder are not scanned, so read them before running anything.

What licence does Caliper Harness Smoke Test use?

Caliper Harness Smoke Test is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Caliper Harness Smoke Test use?

About 664 tokens (SKILL.md is roughly 2.7k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Caliper Harness Smoke Test?

Skills that share tags, products or a category with Caliper Harness Smoke Test: OpenHarness End-to-End Evals (HKUDS/OpenHarness, 16k stars), Qwen Code E2E Testing (QwenLM/qwen-code, 28k stars), Senpi Agent QA Harness (code-yeongyu/senpi, 474 stars) and Exploring The Wizard (PostHog/wizard, 197 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Caliper Harness Smoke Test?

edonadei (a GitHub user) maintains it in edonadei/caliper, which has 209 GitHub stars. The repository holds 3 skills in this directory. The repository was last updated on October 10, 2026.

Source: edonadei/caliper on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.