Rl Job Health Deep Dive

⚠ Do not add comments to YAMLs. Report your recommendations directly to the supervisor.

rl-job-health-deep-dive

Probe one RL job when it is new or untested (config, geometry, model, image, debug/smoke launch, or first launch after a change), or looks starved or wedged. This distinguishes genuine progress from a silent death.

You are a SUBAGENT — you do NOT execute the kill (standing guardrail: never kill a RUNNING job without explicit permission). When genuinely uncertain, prefer NO-KILL + escalate — a wrongly-killed healthy run wastes a whole bring-up; a wrongly-kept dead one wastes one sweep. But if you couldn't get the evidence, the answer is ERROR, not a hedged NO-KILL (see §0).

STEP 0 (DO THIS FIRST) — PULL THE LOGS LOCAL, THEN ANALYZE FROM FILES (mandatory; no live-grep dance)

Before any gate, sync logs locally with this utility and analyze those files. Do not make repeated live iris job logs or kubectl exec greps the primary method.

set -a; source /Users/benjaminfeuer/Documents/secrets.env; set +a
PY=/Users/benjaminfeuer/miniconda3/envs/otagent/bin/python
$PY /Users/benjaminfeuer/Documents/MarinSkyRL/infra/sync_rl_logs.py \
    /benjaminfeuer/<job> --cluster <cw-us-east-02a|cw-rno2a> [--run run-<ts>] [--dest DIR] [--trace-jobs]
# → ./<slug>-<run>/finelog.log   +   ./<slug>-<run>/ray_session_logs/
#   (+ ./<slug>-<run>/<slug>_trace_jobs.tar.gz when --trace-jobs is passed — the Harbor rollout
#      artifacts streamed into ONE archive; opt-in, since trace_jobs can be very large)

It pulls both required sources:

• finelog.log = the aggregated controller/job stream — this is where the terminating exception / NCCL timeout / store->get(...) got error / Worker rank N received signal lives. The per-actor ray logs are frequently traceback-less (Ray's C++ core logs); a hang's root cause routinely appears ONLY in the finelog. Read this FIRST.
• ray_session_logs/ = the per-actor worker-*.out/.err + python-*/raylet/gcs — for per-rank NCCL Init COMPLETE/nranks, weight-sync group construction, py-spy correlation, and the failing rank's own stderr.

Analyze files with grep or Python. For NCCL/weight-sync structure, run experiments/active/tasktrove-dq-sweep-opencode/artifacts/parse_collectives.py on ray_session_logs/. The utility is idempotent and selects kubeconfig by cluster.

Hand the supervisor the local file paths and quoted lines. Live probes are reserved for a py-spy on a still-running wedge; all other analysis is file-based.

§0. THE CONTRACT — evidence-or-ERROR (the load-bearing rule)

Return VERDICT: ERROR whenever you could not obtain the evidence: a required tool failed (auth / PATH / resolver bug / timeout), a log can't be fetched or parsed, you can't separate policy-mesh from engine GPUs, or two authoritative signals disagree and you can't reconcile them. Stop — give (a) the exact command you ran, (b) its exact failure output, (c) what evidence is therefore missing, (d) what you did establish. Do NOT emit KILL/NO-KILL, do NOT default to NO-KILL, NEVER substitute a plausible guess for a missing measurement.

No gate verdict without its named, quoted artifact:

⛔ Engine under-subscription is NEVER Daytona / duty-cycle / tools

When engines are under-subscribed or idle (Running low, Waiting=0, KV≈0, ⅓-TDP power) and the generation buffer stalls, measure the gen→dispatch→train pipeline live:

• RolloutCoordinator dispatch cores — are the num_coordinators (K) coordinator processes CPU/GIL-pegged on submit_batch/gather/post-gather shaping? (py-spy / top them.) → dispatcher-bound → the lever is K, not n/npgw.
• staleness/backpressure — is generation throttled by max_staleness_steps because policy_train is the slow phase (training-bound)?
• issued-vs-scheduled — are generate() calls reaching the engine but Waiting stays 0 (engines drain instantly)? → the bottleneck is upstream dispatch rate, not engine capacity.

(The saturation-READ tuple + "SM-util% is a trap" in marinskyrl's "Saturating vLLM engines" section is still valid; its CAUSAL "raise concurrency to beat the duty cycle" story is REFUTED — see the caveat there.) Diagnose while the job is ALIVE (py-spy dies with it). A verdict on an unmeasured starvation is ERROR-quality.

Resources you MUST use (this skill points; these docs are the truth)

Read the relevant pointers first; this skill does not restate their contents.

• Cluster access, poll mechanics, log fetch, GPU-poll, node headroom, Daytona lifecycle, the tool failure modes → .agents/ops/<cluster>/:
  • CoreWeave → ops/iris/ops.md — §Access (kubeconfig per cluster: East vs cw-rno2a), §Observability (the state-poll primitive iris_ops.py, JobState codes, finelog fetch, and the Poll/tooling pitfalls: rno2a job summary flakiness, the *_ms query columns, the analyze_iris_harbor_job --config/RL-output-dir limits, no server-side job logs grep), §Scheduling (gang/Kueue admission + node-headroom math), §Daytona (orgs, sandbox lifecycle, concurrency headroom), §Monitoring & debugging practices (incl. py-spy). Node shape → ops/iris/ops.md.
  • Leonardo → ops/leonardo/ops.md; TACC → ops/tacc/ops.md.
  • Log-volume discipline (memory iris-log-resource-discipline): state-poll for liveness, bounded/filtered fetch for metrics — never dump a long log into the Mac. Exact bounded-fetch commands in ops §Observability.
• What the logs/config MEAN — the trainer/engine vocabulary, benign-vs-fault line, known failure modes, config semantics → .agents/projects/<dep>/:
  • projects/marinskyrl/marinskyrl.md — the phase-Timer/step vocabulary, [MoE-PATH] grouped_mm-vs-for-loop, the colocated-engine/rank-0-logging deception, engine saturation (n_concurrent_trials scaling), the 80B GDN-GIL/HeartbeatMonitor death + FlashQLA, SKYRL_W13_RELOAD_BRACKET token-salad, SKYRL_R3_RESIDENT, the benign prob_diff_mean artifact, config-schema (Hydra struct) rules, runtime knobs. Read the section matching what you're chasing — do not guess a log line's meaning.
  • projects/vllm/vllm.md — the serve engine: MoE/DCP/R3 flags, enforce_eager (CUDA-graphs) throughput cliff, serving-throughput expectations, the benign engine heartbeats (shm_broadcast … 60/600s).
  • projects/harbor/vllm/daytona/ — the rollout/trial layout, verifier/reward path, passthrough exceptions, sandbox failure modes.
• Capture + poll scripts (don't hand-roll): scripts/iris/analyze_coreweave_rl_job_live.sh (CoreWeave artifact pull), scripts/iris/iris_ops.py (state-poll), scripts/iris/analyze_iris_harbor_job.py (finelog science — mind the RL-job/rno2a limits in the ops note). The per-rung bring-up ladder is rl-agentic-launch-iris §8.

§1. Inputs + capture

Inputs (from the dispatch; most derivable): cluster, job id, pod-name substring, model + size (dense vs MoE active-B), the run's stage, and what's "new/untested" about it (scrutinize that hardest).

Check restart burn first: record B burned / K max, prior terminal errors, and whether failures repeat. Repeated identical crashes are deterministically doomed; a recovered transient is benign. See ops/<cluster>/ for mechanics.

Capture the artifacts = STEP 0's sync_rl_logs.py (finelog + ray_session_logs) — already done before you reach this gate. Do NOT hand-roll a kubectl/R2 sync or a live iris job logs grep; work from the local files STEP 0 pulled. (scripts/iris/analyze_coreweave_rl_job_live.sh is still the tool for the per-trial opencode.txt/rollout view when you need rollout-quality detail beyond the logs.) What the phase-Timers / step counter / [MoE-PATH] markers mean → projects/marinskyrl. The phase Timers are the progress truth — not a trace count, not a progress bar. (0 trials at +15 min on a long-episode arm is normal, not "done" and not "dead.")

⚡ ON ANY DEATH/WEDGE — READ BOTH finelog.log AND ray_session_logs/ (STEP 0 pulled both). The root cause hides in ONE of them; which one varies. Two failure classes, two homes:

• The finelog carries the aggregated terminating exception / NCCL timeout — e.g. store->get('...') got error: wait timeout after 1800000ms + the collective_rpc/broadcastUniqueNCCLID traceback of a het weight-sync bootstrap hang, or a Worker rank N received signal. The per-actor ray logs frequently do NOT carry this (worker-*.err may hold only a benign metrics-RPC line). Read finelog.log FIRST (grep it for got error/timeout/Traceback/received signal/Init COMPLETE).
• The ray_session_logs carry the per-actor detail: a vLLM EngineCore fatal, a Ray ObjectLostError/OwnerDiedError, an actor crash, per-rank NCCL Init COMPLETE/nranks, weight-sync group construction. Grep worker-*.out/worker-*.err (NOT python-core-*.log, Ray's traceback-less C++ core logs); for NCCL/weight-sync structure run parse_collectives.py on the dir (STEP 0). A colocated-engine job that dies exit=0 with NO finelog traceback (only Killed ray::IDLE/raylet/gcs) has its killer here.
• A "clean" exit=0 teardown with node RAM low is NOT a verdict — it's a missing-evidence ERROR until you've read BOTH local files. Worked examples: keep1-v22 (silent ~34-min death = vLLM EngineDeadError reshape_and_cache_flash … Meta tensors at first-step weight-sync, found in ray_session_logs); keep1-ncclnet (het weight-sync bootstrap hang = store->get('/skyrl/skyrl//cuda//0') 1800000ms timeout via collective_rpc, found ONLY in the finelog). Deeper background → ops/iris/ops.md §object-store / RAY ACTOR LOGS.

§2. Gate A — Liveness (alive, or zombie/wedged/dead?)

Liveness = authoritative state poll plus log freshness, never one log-string grep. Use ops/iris/… §Observability, then compare captured-log freshness with expected cadence and read wedge/death signatures.

⚠ Multi-mesh RL hides a wedged policy behind live engines (the colocated-engine deception). On any FSDP×EP×CP RL job the vLLM engines + RolloutCoordinator are SEPARATE actors from the policy mesh — a hung policy collective can read running / fresh-heartbeat / high-util while doing nothing. The three signals that don't lie (Ray actor-death logs, the NCCL watchdog, is the TRAINER/DRIVER log ADVANCING with policy-rank GPUs separated from engine GPUs) and the exact strings to grep for each — projects/marinskyrl (colocated-engine deception + rank-0-logging trap) and projects/vllm (benign engine noise vs real EngineDeadError). Do not call Gate A PASS on a multi-mesh job without clearing all three.

Gate A verdict: DEAD/TERMINAL (state-poll failed/absent, 0 pods) → nothing to kill; report the root-cause traceback + transient-vs-deterministic. WEDGED (running + a real hang signature + stale logs, no benign explanation) → lean KILL (but if it's a starvation wedge, capture the live py-spy first — §0). A py-spy barrier-snapshot + a lone NCCL Watchdog … ran for N ms LOG LINE is NOT a wedge by itself — a real tripped watchdog ABORTS the process, so require pod-restarts==0 + an actual abort/terminal state + stalled FRESH logs (all nodes) and reconcile the cited timeout against the run's timeline before calling wedge (the CW py-spy command + this caveat: ops/iris/… §Monitoring & debugging practices). ALIVE + fresh → Gate B.