Benchflow

What it does

When your agent uses it

• Asked to benchmark an AI coding agent
• Run a benchmark suite
• View trajectories
• Compare agent performance

Example prompts

• “/benchflow”

Requirements

• Python 3
• A Bash shell
• Docker
• A credential in GEMINI_API_KEY
• A credential in ANTHROPIC_API_KEY
• Pre-approved tools (allowed-tools): Read, Write, Edit, Bash

Workflow steps

5 steps, taken from the first numbered list in SKILL.md.

1. 1Check if benchflow is installed: uv tool list | grep benchflow
2. 2Check if API keys are set (GEMINI_API_KEY, ANTHROPIC_API_KEY, etc.)
3. 3Check available agents: bench agent list
4. 4Show recent eval results if any exist under jobs/ (the default --jobs-dir)
5. 5Point to next action based on state

What it can do on your machine

Read from SKILL.md and the folder at commit e965eee. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves these tools, so the agent can use them without asking each time:

  • Read
  • Write
  • Edit
  • Bash

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Ships script files (Shell, from the files we listed), which the agent can run.

  Shell commands in SKILL.md call:

  • uv

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  Links to these hosts (documentation or services it may open):

  • acpx.sh

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names these keys or tokens, usually read from environment variables:

  • GEMINI_API_KEY
  • ANTHROPIC_API_KEY
  • OPENAI_API_KEY
  • DAYTONA_API_KEY
  • LLM_API_KEY

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Benchflow loads about 1.9k tokens when it runs, and up to ~5.4k if it reads all its reference files. Until then it costs about 58 tokens; SKILL.md has 451 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~58

When it runs · the whole SKILL.md, loaded when a task matches

~1.9k

With references · SKILL.md plus every file in references/, read only if the agent opens them

~5.4k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

• NotePre-approves every shell command (allowed-tools: Bash)SKILL.md

  allowed-tools: Read, Write, Edit, Bash

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from benchflow-ai/benchflow at commit e965eee, republished under its Apache-2.0 licence (© benchflow-ai). 451 words, ~1,891 tokens.

name

benchflow

description

Run agent benchmarks, create tasks, analyze results, and manage agents using BenchFlow. Use when asked to benchmark an AI coding agent, run a benchmark suite, create tasks, view trajectories, or compare agent performance.

allowed-tools

Read, Write, Edit, Bash

user-invocable

true

bench eval run \
  --tasks-dir <task-path> \
  --agent gemini \
  --model gemini-3.1-flash-lite-preview \
  --sandbox daytona

import asyncio
import benchflow as bf
from benchflow import RolloutConfig, Scene
from benchflow._utils.benchmark_repos import resolve_source

async def main():
    config = RolloutConfig(
        task_path=resolve_source("benchflow-ai/skillsbench", path="tasks/edit-pdf"),
        scenes=[Scene.single(agent="gemini", model="gemini-3.1-flash-lite-preview")],
        environment="daytona",
    )
    result = await bf.run(config)
    print(f"Reward: {result.rewards}, Tools: {result.n_tool_calls}")

asyncio.run(main())

bench eval run \
  --source-repo benchflow-ai/skillsbench \
  --source-path tasks \
  --agent gemini \
  --model gemini-3.1-flash-lite-preview \
  --sandbox daytona \
  --concurrency 64

bench eval run --config benchmarks/harvey-lab/harvey-lab-gemini-flash-lite.yaml

source:
  repo: benchflow-ai/skillsbench
  path: tasks
agent: gemini
model: gemini-3.1-flash-lite-preview
environment: daytona
concurrency: 64
max_retries: 1

bench eval metrics jobs/      # aggregate pass-rate / tokens / cost (add --json to pipe)
bench eval list jobs/         # per-rollout table

rollout-dir/
├── result.json              # rewards, agent, timing
├── prompts.json             # prompts sent
├── trajectory/
│   └── acp_trajectory.jsonl # tool calls + agent thoughts
└── verifier/
    ├── reward.txt           # reward value
    └── ctrf.json            # test results

bench tasks init my-task                       # native task.md format (default)
bench tasks init my-task --no-pytest --no-oracle
bench tasks check tasks/my-task                # structural validation

my-task/
├── task.md            # YAML frontmatter (config) + prompt body
├── environment/
│   └── Dockerfile     # sandbox setup
├── verifier/
│   ├── test.sh        # verifier entrypoint -> writes /logs/verifier/reward.txt
│   └── test_outputs.py
└── oracle/            # optional reference solution (solve.sh)

bench skills list                                   # discover skills on disk
bench skills eval skills/citation-management \
  --agent claude-agent-acp                          # score a skill against its evals/evals.json

bench hub check          # inventory/structurally-check representative Harbor-registry tasks

bench agent list

bench eval run --tasks-dir tasks/edit-pdf --agent acpx/gemini --model gemini-3.1-flash-lite-preview --sandbox daytona

import asyncio
from benchflow.evaluation import Evaluation

async def main():
    for config_path in [
        "benchmarks/harvey-lab/harvey-lab-gemini-flash-lite.yaml",
        "benchmarks/harvey-lab/harvey-lab-harness-parity.yaml",
    ]:
        result = await Evaluation.from_yaml(config_path).run()
        print(f"{config_path}: {result.passed}/{result.total} ({result.score:.1%})")

asyncio.run(main())

# Install benchflow from PyPI. BenchFlow CLI releases require Python 3.12+.
uv tool install --python 3.12 --upgrade benchflow
# (or from source: uv sync --extra dev --locked)
export GEMINI_API_KEY=...     # or ANTHROPIC_API_KEY, OPENAI_API_KEY, etc.
export DAYTONA_API_KEY=...    # for cloud sandboxes

COPY skills /root/.claude/skills

bench eval run \
  --tasks-dir task-dir \
  --agent claude-agent-acp \
  --sandbox daytona \
  --skills-dir skills/ \
  --skill-mode with-skill

© benchflow-ai, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 18 other files (references) in .agents/skills/benchflow of benchflow-ai/benchflow.

• SKILL.md
• references/create-task.md
• references/dogfood.md
• references/review-and-test.md
• references/smoke-test.md
• tasks/benchflow-knowledge/environment/Dockerfile
• tasks/benchflow-knowledge/instruction.md
• tasks/benchflow-knowledge/task.toml
• tasks/benchflow-knowledge/tests/test.sh
• tasks/create-simple-task/environment/Dockerfile
• tasks/create-simple-task/instruction.md
• tasks/create-simple-task/task.toml
• tasks/create-simple-task/tests/test.sh
• … and 6 more

Open the folder on GitHubat commit e965eee