Octocode Benchmark Runner

What it does

When your agent uses it

• Comparing Octocode against a gh-based baseline on a research question
• Running a blind-judged benchmark pass across several baselines
• Validating that a benchmark campaign's character counts are correct

Example prompts

• “Run the Octocode vs plain gh matchup on the questions in compare/github-questions.”
• “Judge this pass's answers for question 4 and record the verdict.”
• “Validate the latest benchmark campaign's logs byte-for-byte.”

Requirements

• The gh CLI and the octocode, rtk and Headroom tools being compared
• Python scripts in compare/bin/ for logging and validation

Workflow steps

4 steps, taken from the first numbered list in SKILL.md.

1. 1Preflight — verify + pin every arm; a failure invalidates the run.
2. 2Answer — 2 isolated runners (anchor + baseline) per question/pass, leanest-legal path, each appends a ## Q section to answers/-p.md.
3. 3Judge — after both sections exist, one blind judge reasons to a verdict, then scores.
4. 4Summarize — validate logs, aggregate paired stats, update the rollup.

What it can do on your machine

Read from SKILL.md and the folder at commit c265e3f. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Ships 1 file in scripts/, which the agent can run.

  Shell commands in SKILL.md call:

  • python3
  • npx
  • gh
  • bash

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  No URLs in SKILL.md. Its commands use npx and gh, which can reach the network depending on how they are called.

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Octocode Benchmark Runner loads about 2.1k tokens when it runs, and up to ~15k if it reads all its reference files. Until then it costs about 137 tokens; SKILL.md has 721 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~137

When it runs · the whole SKILL.md, loaded when a task matches

~2.1k

With references · SKILL.md plus every file in references/, read only if the agent opens them

~15k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); the scripts in this folder are not scanned.

SKILL.md

The full file from bgauryy/octocode at commit c265e3f, republished under its MIT licence (© bgauryy). 721 words, ~2,067 tokens.

name

octocode-benchmark

description

Use when planning, running, grading, or reporting the by-hand Octocode research benchmark — pairwise matchups (Octocode anchor vs one baseline: gh+RTK, gh+Headroom, or plain gh) over markdown questions, with a fresh isolated runner agent per (question, arm, pass), one blind judge per question grading two answers X/Y in randomized order, and an orchestrator that summarizes accuracy/quality/workflow/characters. Results measured in total characters through the model (model-in delivered + model-out commands/args + final answer).

cd packages/octocode-benchmark
export HR_PY="$HOME/.local/share/uv/tools/headroom-ai/bin/python"   # Headroom arm only

# Phase 0 — preflight (non-zero exit = fix before running)
bash skills/octocode-benchmark/scripts/check-prereqs.sh 18.2.2

# set up a campaign dir
CAMP="campaigns/run-$(date -u +%H%M%S)-$(date -u +%Y-%m-%d)"; mkdir -p "$CAMP/answers" "$CAMP/judge"

# Phase 1 — answer (spawn ONE isolated agent per arm; never mix arms in an agent).
# Each research call sets its per-question log, e.g. octocode Q4:
OCTO_LOG="$CAMP/octocode-p1-Q4.jsonl" ./compare/bin/octoc1822 ghGetFileContent \
  --queries '{"owner":"axios","repo":"axios","path":"lib/adapters/http.js","matchString":"follow-redirects"}'
# baseline (gh+RTK) Q4:
RTK_LOG="$CAMP/rtk-p1-Q4.jsonl" ./compare/bin/rtkm search code --repo axios/axios follow-redirects --limit 20
# log the final answer, then append a "## Q4" section (Answer + Research steps) to answers/<arm>-p1.md
python3 compare/bin/record_answer.py --log "$CAMP/octocode-p1-Q4.jsonl" --question Q4 --file answer.txt

# Phase 2 — judge: build the blind packet, then one reasoning-first verdict per question
python3 compare/bin/build_blind_packet.py --help          # X/Y randomized, tool identity redacted

# Phase 3 — validate + aggregate + report
python3 compare/bin/sumlog.py --strict "$CAMP/octocode-p1-Q4.jsonl"
python3 compare/bin/validate_campaign.py "$CAMP" --question-count 30
python3 compare/bin/per_question_summary.py --out results/PER_QUESTION_SUMMARY.md --json results/per_question_summary.json

© bgauryy, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 23 other files (scripts, references) in packages/octocode-benchmark/skills/octocode-benchmark of bgauryy/octocode.

• SKILL.md
• README.md
• references/BENCHMARK.md
• references/INSTRUCTIONS.md
• references/JUDGING.md
• references/REPORT_TEMPLATE.md
• references/RUNNER.md
• references/RUNNER_TOOL_CONTEXT.md
• references/SCORING.md
• references/aggregation-and-stats.md
• references/example-verdict.md
• references/matchup-readme.md
• references/primer-gh-headroom.md
• references/primer-gh-rtk.md
• references/primer-gh.md
• references/primer-octocode.md
• references/run-phases.md
• references/run-preflight.md
• references/run-with-agents.md
• scripts
• … and 4 more

Open the folder on GitHubat commit c265e3f