Skippy Bench

skippy-bench

Use this skill for performance, orchestration, and report-oriented checks. Use skippy-correctness when the question is pass/fail exactness. All reportable benchmark runs need metrics-server. run, focused-runtime, and local-single start a collector by default; endpoint-driving commands such as chat-corpus and eval run require --metrics-http to point at an already-running metrics-server and should use --metrics-run-id matching the target endpoint's Skippy run id.

Benchmark-managed Skippy server runs must use a release skippy-serving build. Run just release-build before run, focused-runtime, local-single, or local split binary benchmarks, and use target/release/skippy-server (the SkippyBench default). Do not use target/debug/skippy-server for performance or full-corpus validation; SkippyBench rejects that path because debug builds can create false timeout and throughput failures.

Current Repo Shape

Standalone skippy-bench may not be present in this mesh checkout yet. Confirm available packages before using old source-repo commands:

cargo metadata --no-deps --format-version 1 | jq -r '.packages[].name' | sort

Useful current checks:

cargo test -p skippy-serving --lib
cargo test -p mesh-llm-host-runtime --lib inference::skippy

When benchmark harnesses are imported, keep reporting separate from request-path serving. Stage runtimes emit telemetry; benchmark/report tooling owns reports.

External Agent Evals

Use skippy-bench eval for external agent/coding benchmark harnesses. The local SkippyBench corpora are for runtime behavior, cache behavior, transport stress, and perf regression; they are not the source of agent benchmark claims.

Core pack:

skippy-bench eval list
skippy-bench eval info terminal-bench
skippy-bench eval sync --pack core
skippy-bench eval doctor
skippy-bench eval run speed-bench \
  --base-url http://127.0.0.1:9337/v1 \
  --model org/repo:Q4_K_M \
  --endpoint-concurrency 1 \
  --metrics-http http://127.0.0.1:18080 \
  --metrics-run-id run-local-qwen

--timeout-secs is passed to the native harness as its request/task timeout where supported. It is not a full-run dataset limit. Use --harness-timeout-secs only when you need a hard wall-clock cap for an operator/debug run; omit it for canonical full-dataset validation. --endpoint-concurrency must match the target endpoint's serve-openai --generation-concurrency value. SkippyBench keeps native harness request concurrency equal to that value; adapter-specific request concurrency overrides such as SWE_BENCH_PRO_NUM_WORKERS and MCP_ATLAS_COMPLETION_CONCURRENCY must match it or eval run fails before starting the upstream harness. Do not run multiple LLM workers against a single-lane Skippy endpoint when validating full corpora.

Core eval ids:

• speed-bench — llama.cpp SPEED-Bench client for OpenAI-compatible serving latency/throughput. Run the upstream qualitative benchmark across all categories with no Skippy-owned sample limit.
• terminal-bench — Terminal-Bench CLI via terminal-bench-core==0.1.1.
• swe-bench-pro — Scale SWE-Bench Pro OS repo; uses the upstream data and SWE-agent patch generation/evaluation flow rather than a Skippy-owned mini benchmark.
• mcp-atlas — Scale MCP-Atlas native harness. eval run starts the MCP agent environment and completion service when their localhost ports are not already live, then runs the upstream completion script with --no-filter so all Hugging Face dataset rows are attempted, plus the upstream scoring path, without Skippy-specific task limits or tool_choice overrides.

Use-case routing:

Optional future packs are intentionally not wired yet:

• repo-generation: NL2RepoBench.
• tool-expanded: Toolathlon / Tool-Decathlon.

Keep sync/install opt-in. Do not make normal just build or cargo build download external harnesses, datasets, or Docker images. eval sync checks out the fetched upstream ref directly, and eval run records the resolved harness SHA as harness_commit in run.json. Preserve both behaviors so benchmark evidence remains reproducible even when definitions use floating upstream refs.

Terminal-Bench should be installed with uv tool install --python 3.12 terminal-bench; Python 3.14 currently breaks the tb Typer CLI. Treat Docker as ready only when skippy-bench eval doctor reports that the daemon can start a container; docker info alone is insufficient. skippy-bench eval run performs the same prerequisite checks before launching a native harness. Do not add Skippy-owned task filters, dataset limits, compatibility shims, response-format substitutions, or tool_choice overrides to external evals unless the user explicitly asks for a noncanonical experiment.