Analyze Job History Iris

analyze-job-history-iris

📍 Iris orientation — read first. Before acting on anything in this skill, read the Iris tools catalog (.agents/ops/iris/ops.md) and the Iris ops directory (.agents/ops/iris/ — the CoreWeave GPU particulars in ops.md, the TPU marin particulars in ops.md). They carry the binding access/preamble/gotchas and the helper-script inventory the steps below rely on.

scripts/iris/analyze_iris_harbor_job.py pulls an Iris job's complete log from the finelog store (parquet, queried by SQL) — the live deployment ∪ the GCS archive, deduped on the monotonic seq — then computes: §1 preemption count + time-to-preempt, §2 per-cycle trace progress (from harbor GCS output), §3 serving throughput. It writes a markdown report to --output and a JSON sidecar to <output>.json. Always read the sidecar with python — never eyeball the markdown.

This is now FAST and COMPLETE (the old "it's slow, page it" recipe is gone)

The analyzer used to paginate iris job logs by time windows — minutes per job, 15+ min on a multi-day job. It now queries finelog directly: seconds for a training/small job, ~2–3 min for a 60 h / 16M-row datagen job. So:

• You do NOT need the foreground-15-min-wait discipline, and you do NOT need a patient subagent. Just run it inline. (A subagent is still fine for parallelism across many jobs — see the bottom — but the old "don't background it / don't yield" warnings no longer apply.)
• Completeness is guaranteed or it fails loud. It enumerates every attempt/generation from the controller SQLite (task_attempts ⋈ tasks ⋈ jobs), fetches live ∪ GCS, and asserts each attempt window is covered. Any uncovered window > --max-coverage-gap-seconds (default 600) raises and refuses to write a "successful" report. The sidecar carries logs_complete (bool) + missing_windows (list).

Prerequisites (one-time)

1. Run under the marin venv python — it must import finelog / rigging / duckdb: /Users/benjaminfeuer/Documents/marin/.venv/bin/python (NOT the otagent env).
2. IAP login for the live finelog half (covers the recent, not-yet-archived "L0" tail — essential for RUNNING jobs). One-time, cached at ~/.config/marin/iap/marin.json:

   OAUTHLIB_RELAX_TOKEN_SCOPE=1 /Users/benjaminfeuer/Documents/marin/.venv/bin/marin-login login marin

   (The OAUTHLIB_RELAX_TOKEN_SCOPE=1 works around Google reordering the OAuth scopes — without it the login tracebacks at the final token-parse.) If the token expires, the analyzer's live fetch fails and the coverage check fails loud (it won't silently return the GCS-only fragment) — just re-run the login.

Command

/Users/benjaminfeuer/Documents/marin/.venv/bin/python \
  /Users/benjaminfeuer/Documents/OpenThoughts-Agent/scripts/iris/analyze_iris_harbor_job.py \
  <job_id> --output /tmp/$(basename <job_id>)_history.md --resync

• --resync re-fetches; omit it to re-parse the cached merged log (/tmp/iris_history_<job>.filtered.log + <...>.coverage.json) instantly.
• --max-coverage-gap-seconds N (default 600) — the max allowed empty run inside an attempt window before it's a coverage failure. --allow-incomplete — opt out of the strict raise (records logs_complete=false
  • missing_windows and proceeds); use ONLY when you knowingly accept a fragment.
• Cluster. Default marin/TPU. For CoreWeave pass --cluster cw-us-east-02a (the finelog config name == the cluster name; it resolves cw-us-east-02a automatically). Still run under the marin venv python. ⚠ CoreWeave needs R2 archive creds, NOT IAP. On cw the live half uses a k8s tunnel (no marin-login), but the archive half reads s3://marin-na/finelog/cw-us-east-02a (R2 — this literal is the historical store; the resolved root comes from marin_prefix(), see .agents/ops/iris/ops.md §rendezvous) — creds the Mac lacks, so the run crashes FileNotFoundError: The specified bucket does not exist unless you first source them from the iris-ns secret. Procedure: .agents/ops/iris/ops.md. Also: GPU-RL jobs have no harbor trial sidecars, so §2 is empty and most of the value is gone — for GPU-RL use rl-job-health-deep-dive instead. This analyzer is for harbor-shaped jobs (datagen / agentic eval).
• Completeness sanity: the run prints [enumerate] N attempt(s), [merge] live=… + gcs=… -> deduped=…, and [coverage] COMPLETE (or INCOMPLETE + the gaps). Confirm logs_complete: true in the sidecar before trusting the stats.

Parse the sidecar (exact keys)

/Users/benjaminfeuer/Documents/marin/.venv/bin/python - /tmp/$(basename <job_id>)_history.md.json <<'PY'
import json, sys
d = json.load(open(sys.argv[1]))
g = d.get("serving_summary", {}).get("gen_tps", {}) or {}
r = d.get("serving_summary", {}).get("running", {}) or {}
cyc = d.get("cycles", []) or []
served = [c for c in cyc if c.get("did_serve")]
tfs = served[0]["time_to_first_serve_s"] if served else None
ne, tot = d.get("non_empty_trials"), d.get("total_trial_dirs")
rate = (ne / tot) if (ne is not None and tot) else None
exc = sorted((d.get("harbor_exception_stats") or {}).items(), key=lambda kv: -kv[1])[:5]
print(f"logs_complete    : {d.get('logs_complete')}  missing={len(d.get('missing_windows') or [])}")
print(f"runtime_h        : {round(d.get('total_runtime_s',0)/3600, 2)}")
print(f"preemptions      : {d.get('iris_preemption_count')}  (from_log={d.get('preempt_count_from_log')})")
print(f"state            : {d.get('state')}")
print(f"cycles           : total={len(cyc)} served={len(served)}")
print(f"t_first_serve_s  : {tfs}")
print(f"gen_tps          : n={g.get('n')} mean={round(g.get('mean',0),1)} peak={round(g.get('max',0),1)} median={round(g.get('median',0),1)}")
print(f"running          : mean={round(r.get('mean',0),1)} peak={round(r.get('max',0),1)}")
print(f"saturation_rate  : {d.get('serving_summary',{}).get('saturation_rate')}")
print(f"productive trials: {ne}/{tot} = {round(rate*100,1) if rate is not None else None}%")
print(f"harbor counts    : completed={d.get('harbor_n_completed')} errored={d.get('harbor_n_errored')} running={d.get('harbor_n_running')} pending={d.get('harbor_n_pending')} total={d.get('harbor_n_total_trials')}")
print(f"harbor_updated_at: {d.get('harbor_updated_at')}  (started {d.get('harbor_started_at')})")
print(f"top exceptions   : {exc}")
PY

Check logs_complete first — if it's false, the stats are computed over a fragment; investigate the missing_windows (usually a stale IAP token → re-login) before trusting the numbers. The sidecar stores total_runtime_s (seconds — divide by 3600), not runtime_h; gen_tps/running expose max (use as "peak"), not peak; harbor_exception_stats is a {name: count} dict. S1 datagen baseline ≈ gen mean 400 / peak 1115 tok/s; short-task datasets run lower — judge health by the productive trial rate (non_empty/total), not tok/s alone.