Run

/rl:run — launch one Lego-RL run

Preflight, review the parameters, confirm, then launch in the background. Refuses if any check blocks, or if a run is already live on this box.

Step 0 — Orient

Same as /rl:check Step 0: find the repo root, resolve the config (path / bare name / ask when absent or ambiguous), infer the kind (train | eval | infer). Never guess which config the user meant — a wrong guess here burns a cluster for hours.

Step 1 — Refuse if a run is already in flight

bash scripts/lib/live_probe.sh <kind> 2>&1 | grep -E '^WARN +job:'

If any job:trainer or job:runner line is alive, abort:

A run is already in flight; refusing to start another:
  pids:  <pid + etime, from ps -p <pid> -o pid=,comm=,etime=>
  log:   <most recent *.log under logs/>
Let it finish, or stop it yourself (kill -INT <pid>), then re-run /rl:run.

A bare job:vllm with no runner above it is a foreign serving job, not ours — that is Step 2's business (it blocks on GPU ownership), not an abort message about "our" run.

Step 2 — Preflight via /rl:check

Run the check skill's logic on this config (invoke that skill, or inline its Steps 1–3). If the verdict is SAFE TO RUN: ❌ NO, abort with the same consolidated report, prefixed:

Preflight failed — fix the items below before /rl:run.

Do not invent values, skip a check, or pass a force flag; there is no force flag. If the user says "just launch it anyway", explain exactly which check blocks and what it costs to ignore it (each ✗ FATAL maps to a real past incident, written up in the Troubleshooting section of the docs site), then ask them to fix the config. Configs are user-owned; this skill is a launcher, not an editor.

Step 3 — Show the parameters and confirm (mandatory, never skip)

PREFLIGHT_ONLY=1 already printed the run configuration (<kind>) block in Step 2. Show that block — verbatim — and then draw attention to the handful of fields that decide whether the run is worth its hours. Keep this short; the block above already has the detail.

Worth double-checking before this run:
  • data     <train/val index, or dataset, or shard>   ← is this the one you meant?
  • scale    <N nodes × 8 gpu>, <trials/step>, epochs=<N>  ← roughly <estimate> hours/step
  • resume   <RESUME_FROM, or "auto: picks up the newest ckpt in the exp dir">
  • naming   project=<...>  exp=<...>
  • logs     <the real TRAIN_LOG path>
             dashboard <visible / not visible: it is serving <log-dir>>
  • first step  <R3 on → pearson ≈ 0.999; otherwise routing misalignment silently
                destroys the gradients>

Where the log lands, and whether the board will see it — this line is not cosmetic. scripts/templates/verl/common.env derives TRAIN_LOG=${HARBOR_LOG_DIR}/${TRAINER_EXPERIMENT_NAME}.log, and a real config overrides HARBOR_LOG_DIR to a per-experiment directory under the shared trials root. Only the template default is <repo>/logs. The dashboard globs its --log-dir one level, no recursion (webui/server.py) — so a config that overrides HARBOR_LOG_DIR produces a run that trains fine and is invisible on the board for its whole life unless something links it in. Resolve both sides before asking for the yes:

bash scripts/train/train.sh --dry-run <config> 2>&1 | grep -E 'train log|vLLM log'
pgrep -af 'server\.py.*--log-dir'      # which dirs a board actually serves

If dirname(TRAIN_LOG) is not one of the served dirs — and "no board running at all" counts as not served — say so and ask. Do not launch silently, and do not pick for them:

This config writes its training log to
  <TRAIN_LOG>
but the dashboard (pid=<P>, port=<P>) is serving
  <served log-dir>
One-level glob, no match → this run will never appear on the board
(training itself is unaffected).

  [1] symlink it into <served-dir>/<name>.log right after launch
      (no dashboard restart — recommended)
  [2] leave it, watch wandb only
  [3] stop and change HARBOR_LOG_DIR in the config first
Which one?

Option 1 is one ln -s in Step 6, after the real exp_name is known. It is the only choice needing neither a restart nor a config edit, and the symlink name becomes the run's id in the UI — the analysis panels get their exp-dir mapping from the log contents, so any readable name works. Never move the log (the runner holds it open through tee), and never leave a dangling symlink under a log dir — that breaks /api/runs for every run.

Then ask, in one line:

Launch with this configuration? [Y]/[N]  (N = edit the config first)

Never launch without an explicit yes. If the user says no, stop cleanly and tell them to edit the config, then re-run /rl:run.

exp_name gotcha — unless the config pins EXP_NAME, train.sh derives it as harbor-<scaffold>-<EXP_TAG>-<timestamp> at launch time, so the name in the preflight output is not the name the run will have. Never report the preflight-time name as final; read the real one back in Step 6. If the user needs a stable name (resuming, comparing runs), suggest pinning EXP_NAME= in the config before launching.

Step 4 — Decide launch mode

Default = background. These runs last hours; foreground holds the session.

Launch mode? [B]ackground (default: nohup setsid, logs to logs/launch_<ts>.log)
             [F]oreground (holds this session until the run ends)

Accept B / F / <enter> (= background).

Step 5 — Multi-node: print the per-node command, do not SSH

Read topology (train) or vllm topo (infer) out of the parameter block.

• train, NNODES > 1 — ray_bringup.sh branches on local IP: the node whose IP equals MASTER_ADDR (default: the launching node's first IP) becomes the ray head and drives training; every other node joins and exits. So the same command must be run on each of the NNODES nodes.
• infer, VLLM_NNODES > 1 — same shape, keyed on VLLM_HEAD_HOST; rank-0 serves the API, the others run --headless DP shards.
• eval — always single-node.

Print the command block and tell the user to run it on the other nodes themselves. Never SSH to another node, and never assume the workers are up just because the head started.

Multi-node: run the command below once on every node (this machine is already covered
by the launch just performed)
  nodes <ip2>, <ip3>, ...:
    cd <repo_root> && MASTER_ADDR=<head_ip> nohup setsid bash scripts/train/train.sh <config> > logs/launch_<ts>_$(hostname -s).log 2>&1 &
The head node waits for all <NNODES> nodes to join before training starts; a worker that
never came up shows as the head spinning in its ray status poll.