Eval Agentic Cleanup

eval-agentic-cleanup

Always start with the read-only audit (§0): it verifies the four conditions that actually matter, then run only the remediations it flags. "The score is in the leaderboard" does NOT mean the eval is complete — the auto-harvest path can land a score while silently skipping trace upload, or record a bogus 0 from an eval that never reached the model.

0. Read-only completeness + health AUDIT — run first (idempotent, ZERO writes)

Reads only (sacct, run-dir result.json, HF API, Supabase reads). Emits a per-eval verdict + recommended action per check; run only the flagged remediations.

Pre-gate — EXEMPT: grid / throughput / OOM-test measurement runs targeting DCAgent2/* → skip (upload only production winners).

Output one row per eval: ✅/⚠️ per check + the single recommended next action. Re-run the audit after any remediation to confirm ✅.

No-reward trial classes (VNC / infra errors) — bias direction

• VerificationNotCompletedError (harbor 9203989f) / pre-fix AgentTimeoutError WITH NO reward = the verifier never produced a result. Missing because the agent burned its budget → true score ≈ 0 (SWE-bench ~pass/fail). Harbor's JobStats mean DROPS these from the denominator → biased UP; the DB/leaderboard harvest counts them as 0 → unbiased. Either way retriable: resume on a healthy sandbox — the payoff is usually recovering real passes hidden as 0s.
• Infra errors (DaytonaAuthenticationError/DaytonaValidationError/etc.) = failure uncorrelated with solvability → dropping is ~unbiased (lowers N). Re-run for completeness only; do NOT impute 0.
• AgentTimeoutError WITH a reward = verifier scored it → passthrough VALID 0. Don't filter/re-run.
• Discriminate by reward presence, not the error label.

<90% COMPLETE → RESUME, do NOT REGISTER (and DE-REGISTER if it slipped through)

An eval whose VALID-trial count is < ~90% of planned n_rep × benchmark_size (swebench-100=300, tb2=267, dev_set_v2=300) is INCOMPLETE → RESUME to completion (Check-4 resume), NOT register. A partial score silently MIS-RANKS the model. If the auto-harvest already registered a partial (Pending/Started, or a prematurely-Finished partial) → DE-REGISTER that ONE row (cross-user FK-safe per §Remediations); it re-registers cleanly on resume. ≥90% / a ~99% tail-short is fine to finalize. Never run §1's register/upload on a <90% eval. Skip a candidate whose slurm job is still RUNNING.

>10% error-fraction HARD GATE — the authoritative metric

Compute via the ONE utility scripts/database/eval_guardrail.py (guardrail_counts(stats, planned).invalid_error_count), the Python port of the leaderboard guardrail (OT-Agent-Leaderboard server/storage.ts:416-449). It EXCLUDES the benign passthrough error types — never hand-restate the BENIGN set (that's how sibling copies drifted). error-fraction = invalid_error_count / total_attempted; a row is an incomplete partial (→ de-register + resume) iff > 10%. There is NO "borderline"/"close enough" discretion: 10.1% fails like 30%. Do NOT re-derive as a naive "no-reward / valid-reward<90%" count — that folds in the benign passthroughs and over-flags near-complete evals. Exception: an explicit, logged user grandfather of a specific already-pushed row.

Run this gate after deduplication. Duplicate attempts inflate the denominator and can falsely turn a broken run into a pass.

0a. Duplicate-contaminated trials — deduplicate before every later step

An eval is duplicate-contaminated when total trials exceed n_rep_eval × benchmark_size or any task has more than n_rep_eval attempts. Do not upload, score, register, resume, or delete the source run before deduplication.

Use the bundled deterministic script. Run its dry-run first. If after = benchmark_size × n_rep_eval, apply and re-run §0. If fewer unique attempts remain, apply the deduplication, then follow check 4 and resume the missing replicas in the same run:

python scripts/harbor/dedupe_harbor_trials.py \
  --job-dir trace_jobs/<RUN_TAG> --n-rep <n_rep_eval>
python scripts/harbor/dedupe_harbor_trials.py \
  --job-dir trace_jobs/<RUN_TAG> --n-rep <n_rep_eval> --apply

Selection is score-blind: globally collapse copied paths with the same Harbor trial_name before grouping by task, then keep the earliest numeric-reward attempts and the earliest no-reward attempts only if needed to reach n_rep_eval. Never choose by reward value. This preserves completed original attempts, lets successful infrastructure retries replace unscored attempts, and cannot cherry-pick passes. The reusable scripts/harbor/dedupe_harbor_trials.py utility removes only surplus trial directories, writes dedupe_manifest.json, and rebuilds result.json plus harbor_result.json.

For object-store runs, stage the exact run prefix locally, deduplicate there, verify the parsed matrix and aggregate, then replace only that run's harbor_trials tree and aggregate files. If result files are staged before large trial artifacts, exclude every manifest directory prefix and file path from the original object listing when hydrating retained artifacts; otherwise a stale pre-delete listing can copy a removed nested result back into the local tree. A second invocation must report before == after and remove nothing.

Check 4 — counting VALID trials (parse, never file-count) — STANDARD report element

Trial dirs at eval_jobs/eval-<safe_model>_<safe_dataset>/<task>__<id>/result.json (depth-1; the run-dir-root result.json is the aggregate — exclude via the */ glob). A trial is VALID iff reward is finite; else ERRORED. One pass over all evals (otagent env, read-only, ~1 min):

import glob, json
EJ='/e/data1/datasets/playground/ot-baf/eval_jobs'
EVALS=[('eval-laion_<model>_<safe_dataset>', 300), ...]  # (run-tag, n_rep*bench_size) per eval
for tag, exp in EVALS:
    valid=err=0
    for f in glob.glob(f'{EJ}/{tag}/*/result.json'):          # */ = per-trial only, skips root aggregate
        try: d=json.load(open(f))
        except Exception: err+=1; continue
        r=((d.get('verifier_result') or {}).get('rewards') or {}).get('reward')
        if isinstance(r,(int,float)): valid+=1   # numeric reward = VALID (incl. AgentTimeout passthrough)
        else: err+=1                             # no reward = ERRORED (hard infra/exception)
    print(f'{tag}: valid={valid} err={err} total={valid+err} / exp {exp}')

Always include the valid/errored/expected matrix in the report. Flag any eval with ERRORED rate ≳10–15% as a re-run candidate.

Check 4 — resuming errored trials (resume_chunked.py)

Resume re-runs only the errored trials of the SAME run dir (keeps valid trials). Standard driver: eval/resume_chunked.py → one unified_eval_listener.py --resume-only --force-reeval per chunk. Restate the same sizing/yaml/conda-env/Pinggy from the original fire or Harbor errors on the config.json conflict.

• --force-reeval (NOT --force-eval) — resume-mode flag; distinct from launch-time --force-eval (which bypasses dedup in should_start_job). Bypasses the DB status check AND the active_pairs resume filter (needed when an active job for the same pair uses a different scaffold).
• --once + --batch-size — in --once mode, freshly-submitted JIDs fold into active_ids as they submit, so the sliding-window --batch-size chain takes effect.

Cat-3 swe-agent (installed-harness) resume — MUST start a Pinggy tunnel. Cat-3 = preferred-harness reproduction (dcagent_eval_config_swe_agent.yaml, swe-agent/openhands/mini-swe-agent) where the sandboxed agent calls back to the served model over a public Pinggy tunnel. The resume branch of eval/jupiter/eval_harbor.sbatch starts the Pinggy SSH tunnel (gated on EVAL_PINGGY_URL) and exports OPENAI_API_BASE/OPENAI_BASE_URL (+ openhands LLM_BASE_URL/LLM_API_KEY, mini-swe-agent MSWEA_*/HOSTED_VLLM_API_BASE). A Cat-3 resume MUST pass the four passthroughs (note hyphens — distinct from launch-mode --pinggy_persistent_url/--pinggy_token):

# otagent env, in tmux. One Pinggy pair PER invocation (one chunk per free pair for multi-model batches).
python eval/resume_chunked.py \
  --csv /tmp/resume_cands.csv --preset swebench --org eval \
  --tp-size 2 --dp-size 2 --timeout-multiplier 16.0 \
  --jobs-dir <EVAL_JOBS_DIR> --conda-env <env> \
  --tag-prefix <orig_run_tag_prefix> \
  --pinggy-url <free-pair URL, e.g. dadccqeqqf.a.pinggy.link> \
  --pinggy-token <free-pair token> \
  --config-yaml dcagent_eval_config_swe_agent.yaml \
  --agent-parser '' \
  --chunk-size 4 --sleep-between 120

• --config-yaml = Harbor config (scaffold/parser); --agent-parser '' = disable parser for swe-agent.
• Pinggy pair URL+token privileged — read a FREE pair from .agents/secret.md/pinggy_bank.md.
• terminus-2 resumes leave EVAL_PINGGY_URL empty → tunnel block is a no-op (they use the normal sbatch path); only Cat-3 installed-harness resumes set the Pinggy flags.

swe-agent retry-policy (dcagent_eval_config_swe_agent.yaml, aligned to M1 SERA-32B 48.67%): ContextLengthExceededError/BadRequestError/AgentEnvironmentTimeoutError/SummarizationTimeout are RETRIED; VerifierOutputParseError/RewardFileEmptyError/RewardFileNotFoundError are terminal. A resume scored under this policy is NOT comparable to one scored under the old drifted config.

Offline-first pre-download: sbatch tries snapshot_download(local_files_only=True) first (HF cache may be read-only when HEAD advanced past the cached snapshot), falls back to online. Pinggy SSH-out is prefixed with proxychains to egress from no-internet nodes. (etag_timeout=120, 600s hard cap on the pre-download wedge.)

Remediations — the §0 audit scopes which to run (re-run §0 after to confirm ✅)

HF trace upload + Supabase registration are normal, sanctioned operations — run the ones the audit flags.

Cross-user FK safety — REQUIRED before EVERY write (update / delete)

The same (model × benchmark) pair can have MULTIPLE sandbox_jobs rows (reruns, resumes, sibling model_ids, AND other users' rows). Before ANY mutation (sandbox_jobs/sandbox_trial_model_usage/models/benchmarks): query ALL rows for the pair, disambiguate by created_at + username, assert the target row is yours (username == me; your usernames: feuer1/bfeuer00/penfever/benjaminfeuer). Scope every update/delete to id+username. NEVER mutate/delete another user's row. When you can't tell which row is yours → STOP and surface, don't guess. (An INSERT of a new own row never touches theirs — see §1.)