Monitor Job Tables

monitor-job-tables

Read .agents/ops/<cluster>/ops.md first, every sweep. It is the source of truth for which clusters are active, how to locate logs safely, login-node caveats, and current known log noise. This skill deliberately names no cluster as active or down — that changes, and a stale list here produces confidently wrong reports.

Locate logs by cluster type, never by guessing a path:

• SLURM clusters — resolve the log path from the scheduler (scontrol show job <id> -o, fields StdOut= / %Z workdir). Never find/du on a parallel filesystem.
• Kubernetes/iris clusters — there is no scheduler .out and no path to stat. Liveness is a state poll of the job lifecycle, never a log-string grep. Use the iris job-summary/state helpers documented in the ops file; pull metrics from the job logs. "running-but-0-pods" or a record that has disappeared is TERMINAL — that is the silent-wedge signature. Keep iris/kubectl calls synchronous.

Verify a log path EXISTS before concluding "dead." A scheduler's StdOut= may name a file that was never created while the real live log sits in the same workdir under a different name. If the scheduler path is absent, ls the workdir for any *_<jobid>.out and read that. Absence at the scheduler path is a path mismatch, not a death.

A failed log fetch is indistinguishable from an idle job. An API or kubelet error can land in the same stream as the logs and parse as "no metrics". Check the line count before concluding anything about a job's state, and retry once against a floor.

Report every active and recently-terminated job, bucketed by type, in the formats below. Unify cross-cluster runs of the same type into ONE table. Give a separate table for jobs still filling their generation buffer (no metrics yet). Five buckets: RL · SFT · Datagen · Eval · Catch-all.

Cross-cutting (every bucket):

• Chain-restart TIMEOUTs are normal, NOT failures — when a walltime-limited job TIMEOUTs and its afterany successor is RUNNING/PENDING, report it as a normal restart and name the successor.
• Completion → matching cleanup skill: RL by flavor — agentic → rl-agentic-job-cleanup, standard non-agentic → rl-standard-job-cleanup; SFT → sft-job-cleanup; datagen → datagen-job-cleanup; eval → eval-agentic-cleanup. Object-store-backed RL routes the same way but leaves no on-disk trial tree to reap. On shared-filesystem clusters, cleanup is not done until the artifact's on-disk trial tree is removed and inode reclaim is verified — leaving it is the top inode-leak source.
• Genuine FAILED (exit≠0, not a wall TIMEOUT) → diagnose + dated agent_logs/ entry. Recurring identical failures are not transient.

RL

┌─────────────────────────┬───────┬────────┬─────────────┬───────────┬─────────────────────────────────────────┐
│           Job           │ Step  │ Reward │ Policy Loss │ Grad Norm │                  Trend                  │
├─────────────────────────┼───────┼────────┼─────────────┼───────────┼─────────────────────────────────────────┤
│ <run> (shaped)          │ 15/80 │ 0.619  │ -0.0040     │ 0.006     │ Checkpoint saved. Slight dip from 0.652 │
│ <run> (base)            │ 26/80 │ 0.451  │ -0.0930     │ 0.021     │ Stable, gradients strong                │
└─────────────────────────┴───────┴────────┴─────────────┴───────────┴─────────────────────────────────────────┘

Box-drawing tables (┌─┬─┐), not markdown — hard user preference for RL. Columns: Job, Step (cur/max), Reward, Policy Loss, Grad Norm, Trend. Entropy + collapse signals are mandatory: include policy_entropy, TIS log_ratio, and grad_norm (in Trend or as extra columns) — without entropy you cannot apply the collapse rule. A metric not emitted yet → mark —. A fresh launch still in bring-up (gang/queue admission, mesh load, shared-memory broadcast waits, transient image-pull self-heal — all BENIGN) goes in the buffer-filling table with — until its first step lands.

Rewards from different shaping regimes are not comparable. Confirm the shaper state from integrality of reward × rollouts_per_step (fractional ⇒ shaping active) before putting two arms in the same column and drawing a conclusion.

New/untested RL run? → deep-probe it, don't trust the row. A row can read "healthy" on a silently dead run (weight-sync garbage, engine starvation, zero trials completing). For any RL job in a new setting — new config/geometry/model/image, a smoke test, or the first launch after a code or config change — dispatch a subagent with rl-job-health-deep-dive; it reads the literal rollouts and returns a KILL/NO-KILL recommendation.

Standard (non-agentic) RL has no Harbor trial artifacts. Its gates cannot be scored from trial evidence and must not be marked ERROR for lacking it. Substitute reward/avg_raw_reward, banked-step cadence plus durable checkpoints, timing/*, and generate/avg_num_tokens.

Banked steps come from durable evidence, not a progress line. Take the max global_step_N under the run's checkpoint prefix, and search every location the launcher may have written to — a run that resumed and a run that started fresh can bank to different paths. Corroborate with a purity-checked log parse. An exports/global_step_N signals completion only on a finished run; a running job also writes periodic saves there.

Metrics to track per RL run (priority order)

Core 5 (always): reward/avg_raw_reward (primary), reward/avg_pass_at_N (less noisy), policy/policy_loss, policy/policy_entropy (direction and magnitude both matter — pre-collapse), policy/raw_grad_norm (most predictive; healthy < 1.0; > 1.0 for ≥2 steps has predicted collapse 2–5 steps early). Under seqnorm global-denom, grad/policy_loss/log_ratio are genuinely ~1e-5 or smaller — that is the regime, NOT vanishing gradient. Clip ratio (if tracked): policy/ppo_clip_ratio ≈0 normally; >1 % indicates an LR↔eps_clip mismatch. Also policy/z_clip/triggered for clip-variant ablations. TIS: tis/imp_ratio_mean (~0.84–1.56 healthy), tis/imp_ratio_capped_fraction (~0 healthy). Per-token log-ratio diagnostics, where the trainer emits them: log_ratio_abs_{mean,p99,max}, n_tokens_dp_gt_{1,10,50}pct, positional buckets. Healthy: mean ~0.005–0.02, max < 0.5, gt_50pct ≈ 0, position buckets even.

NOT a collapse signal — rollout_train_prob_diff_mean

policy/rollout_train_prob_diff_mean = exp(rollout_lp − train_recompute_lp).abs().mean() — the mean per-token importance ratio, dominated by outlier tokens (a single ~20-nat disagreement gives exp(20)≈5e8). Millions or billions are NORMAL on healthy dense arms. Reward is verifier-computed and independent of logprobs, so this can never "hit the reward". For a per-token divergence read use the capped tis/imp_ratio_mean / imp_ratio_capped_fraction, the median, or log_ratio_abs_* — not this mean.

NOT a failure/hang cause — context-overflow + passthrough-exception lines

Engine ... N input tokens > M max, ContextLengthExceededError, and AgentTimeoutError are benign and expected in agentic rollouts — they are harbor passthrough_exceptions, the verifier still scores, the rollout completes, and they appear in successful runs. Never the reason a job hangs or fails. Find the real terminal signal: a Traceback, OOM / raylet death / SIGKILL, an RPC or sampling timeout, a RuntimeError, or a hung actor/trial that never returns.

Collapse rule (≥2 fire same step → cancel + salvage)

raw_grad_norm > 1.0 (or > 2× its window); policy_entropy off its 10-step trend by > 30 %; log_ratio_abs_mean > 2× its window while max stays bounded; trial pass-rate < 10 % over the last 100. Exception: spike-mitigation ablations are NEVER auto-cancelled on this rule — observing the recovery IS the experiment.

Where a no-kill instruction is in force, this rule gates a RECOMMENDATION, not an action. Capture the evidence that disappears at termination, record it, report it, and leave the job running.