Eval Agentic Launch Iris

eval-agentic-launch-iris

📍 Iris orientation — read first. Read the Iris tools catalog (.agents/ops/iris/ops.md) and the Iris ops directory (.agents/ops/iris/ — CoreWeave GPU in ops.md, TPU marin in ops.md) before acting.

Launch → monitor → manual cleanup of an eval job via eval/cloud/launch_eval_iris.py (Iris analog of the SkyPilot launch_eval_cloud.py). For datagen/tracegen use datagen-launch-iris instead.

🚪 Iris/cloud launchers bypass hpc.launch entirely — the python -m hpc.launch --job_type eval_listener SLURM front door does NOT apply here. The Iris launcher resolves per-model serve config from model_config/ (via model_config/resolver.py + hpc/model_config_apply.py): merges/forwards the model's agent_kwargs, and applies serve intrinsics (max_model_len / limit_mm / extra_args) on the worker. It also applies n_attempts from the preset (CLI-overridable via --n_attempts) and ignores SLURM-only fields plus tp_size (TPU chip count), harbor_config (CLI-required), agent_name (from harbor config). It prints [eval-iris] preset <name>: applied {…}; ignored {…}. Precedence: explicit CLI/--preset > model_config/. Edit the source at model_config/<org>/<slug>.yaml (not the generated registry).

Required info

1. model — model id for --model (HF id or GCS/served path), OR pass --datagen_config <yaml> (model inferred from its engine.model).
2. dataset — for standard benchmarks, use --preset <name> (below), which selects the dataset. Pass an explicit dataset only for a custom benchmark or to override a preset:
   • --dataset <harbor slug> — harbor resolves/snapshots it.
   • --dataset_path <tasks dir | HF dataset id> (mutually exclusive with --dataset). A bare HF id has exactly one /, no leading ./,/,~; the worker's run_eval.py resolves it (snapshot_download + convert_parquet_to_tasks) — the launch host does NOT.
3. harbor_config — REQUIRED, an eval harbor YAML from hpc/harbor_yaml/eval/:
   • dcagent_eval_defaults.yaml — DEFAULT. Iris-adapted port of the eval team's canonical config (hpc/harbor_yaml/eval/configs/dcagent_eval_config.yaml, the SLURM listener's EVAL_CONFIG_YAML): terminus-2, timeout_multiplier: 1.0, n_attempts: 3, agent max_timeout_sec: 7200, verifier max_timeout_sec: 14400. Iris numbers match the eval team's SLURM numbers. Only deviation: force_build: true (Iris builds sandboxes at runtime).
   • eval_ctx32k.yaml / eval_ctx131k.yaml — terminus-2 with timeout_multiplier: 8.0 (8GB/4GB sandbox). Extended-budget mode — only for deliberate 8× timeout. Don't use for normal reg eval.
   • eval_openhands_ctx32k_* / eval_mini_swe_ctx32k.yaml / swe_agent_ctx32k_eval_.yaml — alternate harnesses (OpenHands / mini-SWE / SWE-agent). Only when reproducing a paper's harness.

Presets (--preset, shared with the SLURM listener)

--preset <name> pulls run defaults from eval/presets/ (one YAML per preset, same catalog the SLURM eval/unified_eval_listener.py consumes). Choices: aider, bfcl, financeagent, gaia, medagentbench, swebench, swebench_full, tb2, v1, v2. Precedence: explicit CLI flags ALWAYS override preset values.

What the Iris launcher does with each preset field:

• Applied: datasets[0] → --dataset_path (bare HF id, resolved on the worker) when neither --dataset nor --dataset_path was passed (extra datasets skipped, logged); n_concurrent → --n_concurrent when not passed.
• Applied (agent kwargs, mapped as the SLURM eval/jupiter/eval_harbor.sbatch does): agent_parser → harbor --agent-kwarg parser=<value> (e.g. swebench → parser=xml) unless you passed a parser=; each preset agent_kwargs list entry → its own --agent-kwarg key=value (your --agent_kwarg with the same key overrides).
• Thinking (NOT a preset property): the launcher resolves the model's agent_kwargs from model_config/, so thinking IS auto-applied per-model for models carrying agent_kwargs: [extra_body={…enable_thinking:true}]. For a model with no model_config/ entry, thinking falls back to the served model's chat-template default (Qwen3 = ON). For a default-OFF template model not in model_config/ (e.g. Qwen3.5/3.6), pass --agent_kwarg 'extra_body={"chat_template_kwargs":{"enable_thinking":true}}' (the live nested form vLLM applies; a bare enable_thinking=true is DEAD — terminus-2 has no such param). There is no --enable-thinking flag.
• Ignored (SLURM/vLLM-serve-only): slurm_time, vllm_max_retries, gpu_memory_util, sbatch_script, check_hf_exists, log_suffix, error_threshold, config_yaml, agent_envs, auto_snapshot.

--preset composes with --harbor_config (required), --model, --upload_to_database, etc.

Core evals

The standard/core evals are presets — launch by name (preset sets dataset, concurrency, parser; do not pass --dataset*):

Presets do not set thinking — see the note above. Both require --harbor_config hpc/harbor_yaml/eval/dcagent_eval_defaults.yaml (terminus-2 @ 32k, eval-team default budget, the Cat 1 "reg eval" harness per docs/EVAL_GUIDE.md), fit a v6e-4 for an 8B model, and should launch with --upload_to_database. For full parity with the eval team's SLURM runs, pass --n_concurrent 128 (their CLI default). (terminal-bench 2.0 also exists as slug --dataset terminal-bench@2.0; prefer the preset.)

Snapshots — eval is the exception

Eval does NOT pre-build Daytona snapshots and does NOT call hpc/snapshot_manager.ensure_snapshots. Eval harbor configs set environment.force_build: true — harbor builds each task's sandbox at runtime on the worker, no launch-host prebuild, no 60-snapshot cap, no SnapshotCapExceeded. Datagen is the opposite (force_build: false → pre-builds; see datagen-launch-iris).

Always run eval out of the MAIN Daytona org (DAYTONA_API_KEY, carried via --secrets-env). Do NOT use DAYTONA_B_KEY / DAYTONA_RL_API_KEY / DAYTONA_DATA_API_KEY (other workloads).

Prerequisites

Launch from the py3.12 otagent conda env, source "$DC_AGENT_SECRET_ENV" (see .agents/secret.md; pass --secrets-env), and git pull the marin checkout if the iris client is reported too old. Harbor env defaults to daytona (the only sandbox backend that works on iris workers).

Launch

cd /Users/benjaminfeuer/Documents/OpenThoughts-Agent
source /Users/benjaminfeuer/miniconda3/etc/profile.d/conda.sh && conda activate otagent
source "${DC_AGENT_SECRET_ENV:?set DC_AGENT_SECRET_ENV to the secrets file first}"
TS=$(date +%Y%m%d-%H%M%S)
python eval/cloud/launch_eval_iris.py \
  --preset <name> \                                  # e.g. swebench, tb2 — seeds dataset + concurrency + parser
  --harbor_config hpc/harbor_yaml/eval/dcagent_eval_defaults.yaml \  # eval-team defaults (timeout_multiplier 1.0); use eval_ctx32k.yaml only for 8x budget
  --model <hf-or-gcs-model-id> \
  --tpu v6e-4 --preemptible \
  --job_name "eval-<model-slug>-<bench>-${TS}" \
  --secrets-env "$DC_AGENT_SECRET_ENV" \
  --upload_to_database \
  --no-wait
# Custom benchmark (no preset): drop --preset and pass --dataset <harbor-slug>
# or --dataset_path <tasks dir | HF id>, plus --n_concurrent <N>.

CoreWeave H100x8 GPU eval (single node, cw-us-east-02a)

Pass --gpu H100x8 (mutually exclusive with --tpu) for one CoreWeave H100x8 node. The launcher defaults to the gpu-8x OT-Agent image, the cw-us-east-02a iris config, the datagen extra (not datagen-tpu), and skips the TPU iris-serve/patch_tpu_inference path. export KUBECONFIG=~/.kube/coreweave-iris-gpu first. Single-node only — do NOT pass --replicas > 1 (task sharding + shared multi-node vLLM not implemented for GPU eval); --gpu is limited to H100x8. Use a model known to serve on the runtime (Qwen/Qwen3-32B works).

Daytona/OpenCode against a separately served CoreWeave model: use native federated ingress, not the peer controller's public host. Submit the serving job through Marin with --target-cluster cw-us-east-02a and forward the Marin login; wait until iris --cluster=marin endpoints list <endpoint> --exact shows the mirrored peer endpoint, then mint the scoped URL at Marin and pass only https://iris.oa.dev/proxy/t/<token>/<endpoint>/v1 to the eval. A token minted with iris --cluster=cw-us-east-02a endpoints mint is peer-signed and cannot authorize the Marin parent route; iris-cw-us-east-02a.oa.dev is IP-locked and not Daytona-reachable. scripts/iris/launch_external_opencode_eval.py is the one-command Grug profile: it submits the parent-delegated serve, waits for the mirrored ready endpoint, mints at Marin, and submits the durable-S3 Harbor eval. Its defaults use the established Grug serve topology and CoreWeave S3 root; use explicit flags only to override that profile or attach an existing endpoint.

export KUBECONFIG=~/.kube/coreweave-iris-gpu
python eval/cloud/launch_eval_iris.py \
  --preset swebench \
  --harbor_config hpc/harbor_yaml/eval/dcagent_eval_defaults.yaml \
  --model Qwen/Qwen3-32B \
  --gpu H100x8 --replicas 1 \
  --n_concurrent 3 --n_attempts 1 \
  --harbor_extra_arg=--n-tasks=3 --harbor_extra_arg=--max-retries=0 \  # small subset for fast iteration
  --job_name "eval-<slug>-cw-gpu-${TS}" \
  --secrets-env "$DC_AGENT_SECRET_ENV" \
  --upload_to_database --no-wait