Status

/rl:status — is the live run healthy?

Read-only diagnosis of a run in flight. Answers three things in order: which run · how far · is it sick. Never kills, restarts, cleans or edits anything — a wrong intervention here costs more than a slow answer.

Step 1 — Which run?

bash scripts/lib/live_probe.sh train 2>&1 | grep -E '^(WARN|OK) +(job|gpu):'
ls -t logs/*.log | head -5

Identify the live run from the job:trainer / job:runner lines, then map it to its log.

Do not assume the log is under logs/. scripts/templates/verl/common.env derives TRAIN_LOG=${HARBOR_LOG_DIR}/${TRAINER_EXPERIMENT_NAME}.log, and a real config overrides HARBOR_LOG_DIR to a per-experiment directory under the shared trials root; only the template default lands in <repo>/logs. Get the real path from the runner itself, in this order:

# 1. the runner printed it at startup (works even for a run launched by hand)
grep -hoE 'train log: +\S+' logs/launch_*.log *.out 2>/dev/null | tail -3
# 2. or resolve it from the config without launching anything
bash scripts/train/train.sh --dry-run <config> 2>&1 | grep -E 'train log|vLLM log|trials'
# 3. or find what is actually being written right now
find "$(dirname "$HARBOR_TRIALS_DIR")" -mindepth 3 -maxdepth 3 -type d -name logs \
     -mmin -30 2>/dev/null | head   # or point it at your trials root

A run launched by hand as nohup bash scripts/train/train.sh <config> > foo.out leaves foo.out wherever the launcher's cwd was — usually the repo root, not logs/. It holds the launch summary plus the same teed stream, so it is a superset of TRAIN_LOG and equally good to read; the train log: line near its top is the fastest way to recover the canonical path. What it is not is a file the dashboard can see, since it is outside any served log dir.

Because the repo lives on shared storage, that .out is visible from every box while the process is not: on a multi-node run only the ray-head node has the train.sh / tee / trainer processes. Seeing a growing log with no matching pid here means you are on the wrong node — not that the run died. Check stat -c %Y on the log before concluding anything from an empty pgrep.

Multi-node: each node evaluates EXP_NAME=…$(date …) separately, so one launch produces NNODES exp dirs whose timestamps differ by seconds. Only the ray-head dir holds the trainer log; the others hold just *_train_gpu_wandb.log. Diagnose from the head's, but remember trial counts must be summed across all sibling dirs.

If nothing is alive, say so and offer the last finished run instead; make it explicit in the report which of the two you are describing. If several runs are alive, list them and ask which one — do not merge metrics from two runs.

Step 2 — How far has it got?

LOG=<TRAIN_LOG resolved in Step 1>                # NOT assumed to be logs/<exp>.log
grep -oE 'step:[0-9]+ ' "$LOG" | tail -1          # latest step
grep -cE ' step:[0-9]+ - training/global_step' "$LOG"
ls -t harbor_trials/<project>/<exp_name> 2>/dev/null | head -3
tail -40 "$LOG"

Report: latest step, wall-clock since launch (ps -p <pid> -o etime=), average minutes/step, and whether the tail is still moving (compare stat -c %Y "$LOG" against now). A log that has not been written to in >30 min while the process is alive is itself the finding — that is the deadlock shape, not a slow step.

Step 3 — Are the numbers healthy?

Metrics live on the step lines as key:value pairs. Pull the latest step line and read the keys below (these names are exact — they come from the real logs):

grep -E ' step:[0-9]+ - training/global_step' "$LOG" | tail -1 \
  | grep -oE '(training/rollout_actor_probs_pearson_corr|actor/(lr|grad_norm|kl_coef|pg_clipfrac)|actor/rollout_corr/(kl|rollout_is_eff_sample_size|rollout_is_ratio_fraction_low)|critic/(rewards/mean|advantages/mean|vf_explained_var|vf_loss|grad_norm|lr)|num_turns/mean|trajectory_filter/[a-z_/]+|response_length/(mean|clip_ratio)):[0-9.e+-]+'

Tell a SAO / critic run apart first: its config block says gae with a critic line, and the log carries critic/vf_explained_var. On such a run training/rollout_actor_probs_pearson_corr and actor/entropy are absent by design (bypass mode: old_log_probs == rollout_log_probs, so pearson would be 1.0 by construction) — their absence is not the R3 signature. Read the actor/rollout_corr/* keys instead.

Also worth a line each when present: fully_async/processing_time/tp99 (long tail), fully_async/count/dropped_stale_samples (staleness pressure), rollout_corr/kl.