Monitor Cron Sweep

monitor-cron-sweep

Deploy each cron sweep to produce ONE comprehensive update across all active clusters.

⚠ STEP 0 — READ .agents/ops/<cluster>/ops.md FIRST, every sweep, for each cluster you'll touch. It carries the binding gotchas: the GPFS find/du ban (stat-walks stall SSH for minutes — locate logs via scontrol show job <id> -o StdOut=/%Z + depth-1 ls), the login01 fork-saturation false-drain (re-check via login02/03/04), cleanup-isn't-done-until-rm'd, the SIF/Ray-actor/NCCL debugging tooling (ptrace blocked → faulthandler; the opCount dead false-positive), the sig53/EDQUOT traps, and shell idioms (sacct -S now-Nhours, simple single-string ssh).

Local clone = ground truth. Any code/config fix this sweep performs — or dispatches a subagent to perform — is edited in the local Mac checkout → commit → push → git pull on the cluster. NEVER hand-edit, git commit, or leave divergent/untracked changes on a cluster; no patch-by-rsync (vLLM built from source per-cluster). Bake this rule into EVERY subagent prompt you dispatch.

Tables → the monitor-job-tables skill (box-drawing ┌─┬─┐, NOT markdown), bucketed RL · SFT · Datagen · Eval · Catch-all, with the mandatory metric columns, thresholds, and benign-noise-vs-real-fault rules per bucket. This skill is the process that fills those tables. Cluster particulars (ssh, paths, RL concurrency cap, gpu-mem ceiling, dotenv) live in .agents/ops/<cluster>/ops.md — no cluster-specific values inlined here.

1. Gather (per cluster)

Scope = Leonardo + CoreWeave(iris) + TACC(Vista) — all three each sweep. Jupiter is SKIPPED (MDC downtime until ~2026-07-12 — re-add as a 4th cluster when it returns); Perlmutter DROPPED 2026-06-05 (do NOT ssh). Leonardo + TACC are SLURM (squeue/sacct); CoreWeave is a k8s/iris controller backend with NO ssh — state-poll the iris lifecycle.

SLURM clusters (Leonardo, TACC):

• squeue -u <user> -t RUNNING + sacct -u <user> -S <-Nh> -X (terminal states since last sweep). TACC <user>=penfever via ssh TACCVista; Leonardo <user>/ssh → ops/leonardo.
• Validate squeue succeeded before trusting a 0-count. A slurmctld timeout prints slurm_load_jobs error: Socket timed out with NO job lines → a naive grep -c reads 0 → false "drained". Treat an errored squeue as UNKNOWN (keep waiting); prefer a positive done-signal via sacct -j <ids> --format=State (slurmdbd survives slurmctld outages). login01 fork-saturation is a second false-empty cause → re-check via login02/03/04. Mandatory before any destructive datagen consolidate+delete.

Leonardo gather/triage → ops/leonardo/ops.md ("Sweep / gather particulars"). The Leonardo layer (GPFS find/du ban, scontrol log location, eval log-path trap, standard-eval results-JSON shape, active campaigns, flawed_summ POLICY.md/STATE.md, HF-upload sbatch-tunnel, step-ca cert, $WORK vs $SCRATCH_FAST) lives there. Cross-cluster rules that DO apply: AgentTimeoutError / ContextLengthExceeded are EXPECTED passthrough exceptions in agentic eval (still scored) — never the cause of a hang. Drive named campaigns (flawed_summ et al.) off their OWN tracker docs — do NOT restate their rules here (restated rules drift): each sweep READ the campaign's POLICY.md/STATE.md and drive off them. Eval launch/listener mechanics live in eval-agentic-launch + .agents/projects/ot-agent/, not here.

CoreWeave(iris) — STATE-POLL, not squeue, not a log-string watch:

• export KUBECONFIG=~/.kube/coreweave-iris-gpu in the same shell FIRST (the Mac default kubeconfig points at a DIFFERENT cluster → wrong-context "0 pods/not found"). Use the otagent-env iris binary /Users/benjaminfeuer/miniconda3/envs/otagent/bin/iris (the marin .venv iris has a broken kubernetes import). All iris/kubectl calls SYNCHRONOUS (never background).
• Per active job, poll the authoritative lifecycle: PY=/Users/benjaminfeuer/miniconda3/envs/otagent/bin/python; $PY scripts/iris/iris_ops.py /benjaminfeuer/<job> --once --json and/or iris --cluster=cw-us-east-02a job summary --json (authoritative). iris … query over the jobs table (state 1=PENDING 2=BUILDING 3=RUNNING) lists live jobs. Treat "running-but-0-pods / record disappeared" as TERMINAL — the silent-wedge signature (a clean kill/eviction/preempt emits no terminal log line + reaps pods). Log-content greps (scripts/iris/analyze_iris_harbor_job.py, sel_rows/EPDIAG) are for SCIENCE/throughput ONLY, never liveness. Full log: iris … job logs --since-ms <submitted_at_ms> --no-tail (finelog keeps the whole log; only --tail caps lines).
• RL bring-up signals (fresh launches): gang/leafgroup Kueue admission (pods SchedulingGated until atomically admitted = normal), apply_ep/mesh-load, weights resolving. shm_broadcast …60s + a transient ghcr-EOF ImagePullBackOff self-heal are BENIGN bring-up noise. --max-retries ≥1 re-brings-up the gang on a transient HF-weight-resolution flake.

TACC(Vista): ssh TACCVista (ControlMaster live, hardened single-string ssh). salloc is BLOCKED → sbatch; uv/builds go in a CPU -p gg sbatch, never the shared login node. Compute nodes have FULL internet → NO proxy/SOCKS/step-ca cert (contrast Leonardo). GPUs are NOT a SLURM gres (whole-node alloc); RealMemory misreported. Agentic eval runs through the front door python -m hpc.launch --job_type eval_listener --cluster-config tacc (sbatch_script=eval/tacc/eval_harbor.sbatch, eval_jobs_dir=/scratch/10635/penfever/eval_jobs) — newly integrated, currently validated by a canary, so sanity-check the canary's traces uploaded + registered before relying on it. Harvest finished TACC evals the same way as Leonardo (eval-agentic-cleanup if auto-register failed).

Cross-cluster liveness + inode checks (apply per cluster):

• LIVENESS — RUNNING is NOT proof of progress (catches silent wedges). A job can hold its allocation for hours while hung (engine deadlock, NCCL stall, generation-buffer wedge) — squeue still says RUNNING. For EVERY RUNNING job, stat -c%y <StdOut> and compare log mtime to "now": if a job has emitted NOTHING materially longer than its expected cadence (RL step / SFT log interval / eval trial — minutes, not hours), treat as a suspected silent hang → investigate (tail the log, grep ray-worker logs for EngineDead/NCCL-timeout/Watchdog/RPC-timeout around the last-output timestamp). A multi-hour-stale log on a multi-node job is a wedge burning nodes → diagnose + (with permission, since it's RUNNING) kill+relaunch. Never report a RUNNING job as "healthy" without confirming its log is live. Put the log-mtime ("last output N min ago") in the table.
• INODE HEADROOM (each sweep) — jutil project dataquota -p <project> + df -i. Inodes (file COUNT), not bytes, are the binding constraint; the shared datasets project on /e/data1/datasets (where …/playground/ot-baf lives) runs chronically near/over its soft limit. (DORMANT while Jupiter is down, re-arm when it returns; Leonardo's bind is disk quota, not inodes → ops/leonardo; CoreWeave artifacts go to HF/R2, no POSIX tree to reap.) See ops/jupiter/ops.md → "Inode allocations" (#inode-allocations). At/over the inode soft limit → sweep red-flag → trigger the cleanup-reclaim step (§4).

2. Bucket every job by type

By job-name prefix / run-tag: rl__* → RL, sft__* → SFT, datagen__* → Datagen, eval-* / eval run-tags → Eval, everything else (consolidate, pretokenize, hf_upload, SIF build, DCP/CP/GPU-CI smoke, measurement/grid probes) → Catch-all.

3. Render per monitor-job-tables (unify cross-cluster per type)

Unify all clusters' runs of a type into ONE table. Extraction pointers:

• RL — Step (.out tqdm Training Step Progress: N/M or trainer/global_step) + reward/grad/entropy/ TIS from the WANDB_MIRROR lines (chain-restart logs may have step but not the dict — scan the chain's logs). Apply the collapse-signal rule. For any RL job in a NEW/UNTESTED setting (new config/geometry/model/image, a "debug"/"smoke-test" run, or the first launch after a code/config change), the table row is NOT enough — dispatch a subagent armed with rl-job-health-deep-dive this tick to deep-probe it (sync trace_jobs + logs, live-poll GPUs vs the serving LUT, read the literal rollouts) → a KILL/NO-KILL recommendation. State-poll + metrics can read "healthy" on a run that is silently dead (weight-sync garbage, engine-starvation wedge, all-reward-0). Carry the verdict into §4; the supervisor owns the actual kill.
• SFT — Step + {'loss','grad_norm'} from the .out (NOT trainer_log.jsonl); total steps from the config/banner.
• Datagen — chunks done/total (squeue+sacct) + result.json count + avg_turns (realness gate: ≈1.0 = dead) + exc%.
• Eval — result.json/total + pass-rate + top exception + the 4 infra checks (eval-agentic-launch §4 for greps).
• Catch-all — one line each: State / Elapsed / human note.