Eval Agentic Launch

eval-agentic-launch

Launch agentic Harbor evals via the unified eval listener (eval/unified_eval_listener.py). Cluster-agnostic; read .agents/ops/<cluster>/ops.md first for the cluster's sbatch script, gpu-mem ceiling, concurrency, cert/tunnel, conda env, paths, Daytona eval-org key, and whether --pre-download is needed.

Front door: python -m hpc.launch --job_type eval_listener …. Runs the listener's main() in-process after the launcher preamble (detect_hpc + set_environment → DCFT/EXPERIMENTS_DIR/PYTHONPATH + hosted-vllm/Supabase keys + chdir to repo root), so no manual source hpc/dotenv/<cluster>.env / export PYTHONPATH / cd is needed. Forwards the listener's ~50 flags verbatim (strips only --job_type eval_listener). The raw python eval/unified_eval_listener.py … fallback still works (same public API) but you own the preamble — if you ever see FATAL: WORKDIR=... is not the OpenThoughts-Agent repo root, you used the raw script from the wrong place; switch to the front door.

⚠ Secrets from $DC_AGENT_SECRET_ENV, never hardcoded in a script/config/commit. The eval sbatch sources it (~/secrets.env; TACC $SCRATCH/keys.env) and reads the two Daytona eval-org keys: DAYTONA_API_KEY (org1) + DAYTONA_DATA_API_KEY (org2), 3:1-weighted (3/4 org2). Fails loudly (:?) if either is unset. A literal dtn_… key committed anywhere is a leak — rotate/revoke it, don't just fix-forward.

1. Select the models

• Priority list (default): a file in eval/lists/ (models_8b_*.txt, models_32b.txt, models_131k.txt). Launch with --require-priority-list --priority-file eval/lists/<file>.
• Find unevaled models — scripts/database/query_unevaled_models.py (resolves benchmark families via the Supabase duplicate_of field, e.g. dev_set_v2 ⊇ DCAgent_dev_set_v2/dev_set_v2_2.0x/openthoughts-tblite):

  python scripts/database/query_unevaled_models.py --benchmark <fam> --size <8|32> -o eval/lists/<file>.txt -v
  # needs SUPABASE_URL + SUPABASE_SERVICE_ROLE_KEY

--require-priority-list is LOAD-BEARING

--priority-file alone only changes sort order; the filter "skip models not in the list" lives behind --require-priority-list (unified_eval_listener.py ~L978). Without it the listener submits evals for every unevaled model in the lookback window (routinely 700+). Always pass both. If launched without it by accident: kill the listener before it leaves pre-download (submission is after Pre-downloading…), then confirm via squeue/sacct --starttime=now-Nmin. The listener python is a child of the sshd: …@notty session and survives the local ssh client dying — pkill -9 -f unified_eval_listener.py on the cluster to stop it.

Benchmark = a --preset (one, not both with --datasets)

tb2=terminal_bench_2, v2=dev_set_v2, dev=dev_set_71_tasks, swebench, bfcl, aider. --preset swebench is the random-100 subset (DCAgent/swebench_verified_eval_set → swebench-verified-random-100-folders, n_concurrent 32), not the full set.

"ID evals" — launch all three legs

Each leg is a separate listener invocation (different n_concurrent/harbor-config, don't combine into one --datasets):

"Run the ID evals" = fire one listener per leg (§4) + the §5 infra check on each. (Full SWE-bench-verified and other benchmarks are OOD.) Scoring side (crud-otagent-supabase) uses the same 3-member set; dev_set_v2 is partial-credit → counts toward the ID mean but excluded from the ID SE and model-vs-model ranking.

Re-eval / parity test → --force-eval

By default the listener Skips any model with a Finished+metrics row (reason=job finished) — correct for cohort fill, but blocks a deliberate re-run. --force-eval bypasses that dedup and submits a fresh sandbox_jobs row (doesn't touch the existing row → no metrics-clearing, works across users). Pair with --require-priority-list + a single-model --priority-file so only the intended model is forced. --stale-started-hours does NOT override a Finished row (only re-ages Started). Distinct from --force-reeval (resume-path flag, see eval-agentic-cleanup check 4).

2. Harbor config + timeout multiplier (config-by-size — usually nothing to do)

Do NOT pass --harbor-config for standard terminus-2 evals. The listener selects the canonical config by model size and sets EVAL_HARBOR_CONFIG per-model:

Size is read from the largest \dB token in the HF name. The multiplier flows as EVAL_TIMEOUT_MULTIPLIER and is recorded in the Pending row so dedup matches what ran. The deprecated eval_ctx*_non_it* / ctx32k_non_it_16x_eval_.yaml configs carry stale *-drop-ei metrics → JobConfig ValidationError.

Resolution order (first wins): (1) explicit --harbor-config / preset harbor_config — overrides size selection for every model (use for 131k context / openhands_* installed-harness); (2) per-model timeout_multiplier: in the registry (for names with no size token, e.g. a Qwen3-8B named laion/GLM-4_7-swesmith-…); (3) size-based table above. For a one-off harbor jobs start, point --config at the 8B/32B file.

3. Pinggy tunnel — installed-harness ONLY (not terminus-2)

Skip for the default terminus-2 agent (every eval_ctx*/*_non_it* config; all --presets). Do NOT pass --pinggy_* / consume a pair.

Installed harnesses (opencode / openhands_*) run in the Daytona sandbox and call back out to the served model over a public pinggy tunnel. The full recipe for the opencode installed-harness + pinggy ID-eval — the config (eval_opencode_ctx32k.yaml), config-delivery mechanism (--config-yaml on TACC vs --harbor-config), the --pinggy_persistent_url/--pinggy_token flags + pairs 8/9/10, the sbatch installed-agent tunnel/routing branch, the vllm/ provider, -Thinking- model specifics, and the pinggy infra checks — lives in .agents/projects/harbor/ops.md → "Agentic ID-eval via the opencode (installed) harness + pinggy". The privileged URL/token bank is in .agents/secret.md / notes/ot-agent/pinggy_bank.md (never inline it). Resume of an installed-harness eval also needs the tunnel — see eval-agentic-cleanup check 4.

4. Launch (in tmux — listener is long-running)

Concurrent-submit guard: ONE listener enqueues many legs; do NOT fire N concurrent --once processes. A multi-leg refill is one invocation that submits each leg internally with a 1s submission_delay (unified_eval_listener.py L3204–3205). Firing N listener processes near-simultaneously on the login node races conda's lazily-imported plugin registry → a circular-import at activation. If multiple listener processes are truly required (incompatible n_concurrent), stagger them ~30–45s apart — never & them together. (Per-job conda activate inside the sbatch runs on independent compute nodes and never races.)

# inside tmux. The front door does the preamble (no manual source/PYTHONPATH).
python -m hpc.launch --job_type eval_listener \
  --cluster-config <cluster-name> \
  --preset <preset> \
  --require-priority-list --priority-file eval/lists/<file>.txt \
  --config-yaml dcagent_eval_config_no_override.yaml \
  [--agent-kwarg 'extra_body={"chat_template_kwargs":{"enable_thinking":true}}'] [--agent-parser json] [--max-output-tokens 16384] \
  [--pre-download] [--force-reeval] [--pinggy_persistent_url <URL> --pinggy_token <TOKEN>] \
  --once --verbose 2>&1 | tee eval/<cluster>/logs/<preset>_listener_$(date +%Y%m%d_%H%M%S).log
# Raw-script fallback (you own the preamble): from repo root, export PYTHONPATH="$PWD:${PYTHONPATH:-}", run python eval/unified_eval_listener.py … with the same flags.

• --cluster-config takes a bare cluster name (leonardo, tacc) resolved from hpc.hpc's eval_cluster_view (a .yaml path still works as back-compat). Supplies sbatch_script/hardware/conda_envs/paths — so you no longer pass --sbatch-script/--n-concurrent/--gpu-memory-util.
• Per-model serve config (conda_env, tensor_parallel_size, data_parallel_size, max_model_len, limit_mm_per_prompt, max_output_tokens) comes from the shared registry by default — no flag. The cluster yaml's hardware_profile: (e.g. gh200) selects the per-cluster recipe; a per-cluster intrinsic delta is name@<profile>, a hardware delta is variants: {<profile>: {…}}. Confirm it loaded: listener logs Model-config registry ENABLED + Loaded model registry: N model config(s) + Using conda env '<env>' for <model>. --baseline-model-configs is deprecated (opt-out of the registry).
• Edit per-model serve config in model_config/<org>/<slug>.yaml, NOT the generated eval/configs/model_configs.yaml (auto-generated, carries a # do NOT hand-edit banner). Regenerate with python scripts/generate_eval_registry.py (drift gate: --check).
• Thinking is per-model authoritative (sourced from the registry via agent_kwargs: [extra_body={…enable_thinking:true}]); presets never carry thinking; there is no --enable-thinking flag. Override with --agent-kwarg 'extra_body={"chat_template_kwargs":{"enable_thinking":true}}' (precedence: CLI > registry > preset).