Run Bitrouter Benchmark

Run BitRouter Benchmark

Core rule

Let Harbor own tasks, trials, environments, concurrency, scoring, artifacts, and resume. BitRouter is the model endpoint. Ask only for choices that cannot be discovered, then obtain one explicit confirmation of the resolved run before paid or mutating actions. Detected config and credentials are never permission. Treat every path, command, port, profile, region, service, package manager, and repository layout as specific to the current user's environment; discover or ask for it instead of inheriting assumptions from the skill author's machine.

1. Identify the run

Before discovery, ask for every missing choice together in one short message and stop. Do not infer an unspecified choice from local state:

1. Harbor dataset with version, local path, repository, or job config.
2. Agent: Codex, Claude Code, Terminus 2, or another installed Harbor agent.
3. Routing config: matching official OSS template (default), existing user config, or a new custom config.
4. Endpoint: existing/managed BitRouter, or BitRouter OSS to operate on AWS.
5. Intent: private score, routing comparison, or official publication.

If the user chooses a new custom config, ask one follow-up for the entry preset/model, reachable provider/model targets, fallback, and credential source names. Do not require a baseline when the user only wants one routed score. Do not ask AWS questions when neither BitRouter AWS deployment nor a Harbor AWS/EC2 environment is selected. Treat any as permission for that named choice only; never default a different missing choice. Ask all remaining Stage 1 choices together.

2. Inspect before asking again

Inspect without external mutation:

• how Harbor is launched (PATH command, absolute executable, environment launcher, or container), then its version, run help, agents, environments, and the selected dataset or benchmark-owned job config;
• how BitRouter is accessed and managed: endpoint, executable or service / container identity, config path or mount, and source/release provenance;
• current benchmark run and submission rules;
• BitRouter endpoint provenance and health metadata where available;
• selected config source and every provider/model/fallback reachable from its entry route, plus broader exposure from inherited defaults or credentials;
• AWS profile or assume-role name only if BitRouter AWS deployment or a Harbor AWS/EC2 environment was selected.

Use read-only host discovery first. A command absent from PATH is unresolved, not proof that the software is uninstalled. After discovery, batch all access methods or paths that remain genuinely unresolved into one conditional prompt. If installation or repair is needed and no preference was supplied, propose a pinned method appropriate to the confirmed host/runtime in the Stage 2 plan; do not add a separate preference round. Never assume this repository is checked out.

Use the official template from the same stable release/tag/commit as the selected BitRouter binary. If AWS deployment has no preselected binary, propose the latest stable OSS release and its matching template as the default. Use main, a development branch, or another worktree only when the user selects its exact repository, ref, and path. If binary or endpoint provenance is unavailable, say unverified; never invent a template match. If the selected release has no compatible template, ask for an existing or custom config instead of silently using starter output.

Classify the result:

• Official unchanged — exact template.
• Official-derived deployment patch — only listen/auth/database/path changes; preserve and review the diff.
• Derived from official — provider, model, fallback, preset, tier, or policy changed; preserve the diff.
• Custom — user-owned or built from scratch.

Load the bitrouter skill when available for current BitRouter CLI, config, provider, and harness facts. Otherwise inspect the selected binary and source; do not copy stale details from this benchmark skill.

3. Confirm one resolved plan

Present one compact plan containing:

• benchmark identity/revision and official-submission status;
• Harbor agent, environment, attempts, benchmark-owned settings, and explicit concurrency or the effective Harbor default;
• endpoint provenance and config classification/ref/path;
• Harbor model string, BitRouter entry preset/model, every reachable downstream provider/model/fallback, and credential source names (never values), plus any broader inherited catalog or credential exposure as a separate disclosure;
• bounded agent-specific smoke payload, maximum requests, expected cost, the whole benchmark's provider-spend estimate or ceiling, and retained artifacts;
• installs, service/config/secret-store changes, plus AWS account/principal, region, resources, cost, and—when Harbor uses EC2—the resolved controller-to- sandbox address/SSH path, bootstrap egress, proposed temporary rule specifications and stopping condition, and cleanup only when applicable;
• output path and upload destination, visibility, and data sensitivity.

Require one explicit confirmation before installation, service start, secret-store write, paid provider request, AWS mutation, benchmark launch, or upload. Reconfirm provider/model targets even when they came from existing config. Read agents.md for agent protocol and smoke checks; read aws.md when operating BitRouter or a Harbor EC2 environment on AWS.