Rl Standard Launch Leonardo

⚠ Do not add comments to YAMLs. Report your recommendations directly to the supervisor.

rl-standard-launch-leonardo

⚠ VERIFY checkpoint/export paths resolve to $WORK ($CHECKPOINTS_DIR), NOT $SF/$SCRATCH_FAST — scratch is 1 TB/over-quota; a ckpt write fails OSError [Errno 122] Disk quota exceeded mid-run (NOT an OOM). See .agents/ops/leonardo/ops.md "WRITE-PATH MANDATE".

Standard non-agentic SkyRL RL on Leonardo: GRPO on local math/reasoning parquet and on-policy distillation (OPD). Compute nodes are offline: no Harbor, Daytona, terminal_bench, or proxyserver.

Authoritative source docs (this skill distills them — read for full numbers):

• notes/RL/gsm8k_grid_leonardo/ — grid.md (throughput), accuracy_grid.md (pass@8 to convergence), grid_experiment_log.md (methodology), scripts/.
• notes/RL/opd_grid_leonardo/ — leonardo_opd_qwen3_plan.md, grid.md, throughput_grid.md.
• Leonardo access boilerplate (ssh/2FA, preamble, code/data paths, step-ca cert, login-node killer) → .agents/ops/leonardo/ops.md + CLAUDE.md.

Launch with an sbatch wrapper in hpc/skyrl_standard/leonardo/, not hpc.launch. It uses a writable sandbox directory and external uv venv, then calls the SkyRL entrypoint directly.

1. Cluster + env facts

A100-64GB, 4 GPUs/node, x86_64, SLURM. Account AIFAC_5C0_290, partition boost_usr_prod. QOS: boost_qos_dbg (≤30 min, ≤2 nodes) or normal (more nodes; 24h max).

• MarinSkyRL = marin-community/MarinSkyRL main @ 9bb6d5e at $WORK/code/MarinSkyRL ($WORK = /leonardo_work/AIFAC_5C0_290/bfeuer00). Container --pwd = MarinSkyRL/skyrl-train → SkyRL fixes go to MarinSkyRL main.
• Image = writable sandbox dir $SF/marinskyrl_sandbox ($SF = /leonardo_scratch/fast/AIFAC_5C0_290/bfeuer00), from docker://anyscale/ray:2.51.1-slim-py312-cu128. Binary /usr/bin/singularity (SingularityPRO 4.3.1; no apptainer on PATH).
• uv, not conda: venv $SF/marin_venv (uv sync --extra vllm → torch 2.8.0+cu128, vLLM 0.11.0, flash-attn 2.8.3). $VENV_PY=$VENV/bin/python.

Standing gotchas

1. No compute-node internet: HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1, WANDB_MODE=offline, HF_HOME/HF_HUB_CACHE=$WORK/data/hub. Pre-stage model + parquet on the LOGIN node first.
2. gcc for Triton JIT — the ray base image ships no compiler. Wrapper binds host miniforge $WORK/miniforge3/envs/otagent/bin onto PATH + exports CC/CXX (gcc 14.3.0). RAY_USAGE_STATS_ENABLED=0.
3. HOME is read-only in-container: set HOME=$SF/canary_home, ckpt_path/export_path at writable $SF. The /leonardo/home RO FileNotFoundError/Read-only file system/Traceback lines (tvm_ffi dlpack, vLLM telemetry) are benign engine-init noise — ignore them.

2. Pre-launch (login node, tmux)

Run the standard Leonardo preamble (ops.md), then pre-stage offline data:

ssh Leonardo                              # step-ca cert; 2FA once (ops.md)
cd /leonardo_work/AIFAC_5C0_290/bfeuer00/code/MarinSkyRL && GIT_TERMINAL_PROMPT=0 git pull
# Pre-stage on the LOGIN node (compute has no internet):
hf download Qwen/Qwen2.5-1.5B-Instruct    # → $WORK/data/hub
# gsm8k parquet → $WORK/data/gsm8k/{train,validation}.parquet  (MarinSkyRL examples/gsm8k/gsm8k_dataset.py)
# MATH:  hpc/skyrl_standard/leonardo/math_dataset.py → $WORK/data/math/

Edit code locally, commit/push, and git pull on Leonardo; never patch remote files.

3. Launch — single node

cd /leonardo_work/AIFAC_5C0_290/bfeuer00/code/OpenThoughts-Agent/hpc/skyrl_standard/leonardo
sbatch sbatch_gsm8k_canary.sh             # bare canary: 1 node × 4 A100, ≤30 min

The sbatch sets DATA_DIR/MODEL_PATH/NUM_GPUS=4/CKPT_DIR + offline env, then singularity exec --nv --no-home --bind /leonardo_work,/leonardo_scratch --pwd $MARIN $SANDBOX bash run_gsm8k_canary.sh, which calls $VENV_PY -m skyrl_train.entrypoints.main_base with the GRPO knobs.

Canary GRPO config (run_gsm8k_canary.sh, Qwen2.5-1.5B-Instruct): advantage_estimator=grpo, strategy=fsdp2, colocate_all=true, backend=vllm, run_engines_locally=true, weight_sync_backend=nccl, async_engine=true, 4 engines × TP1, use_kl_loss=false, lr=1e-6, n_samples_per_prompt=4, train_batch_size=32, max_prompt_length=512, max_generate_length=512, gpu_memory_utilization=0.70, env_class=gsm8k, epochs=1, logger=console (offline). Reference: job 44478923 COMPLETED, 233-step epoch, 9.58 s/step, reward 0.14→0.64, pass@4 0.78.

Grid-cell overrides

run_gsm8k_canary.sh ends in "$@" (trailing hydra overrides, last-wins), but sbatch_gsm8k_canary.sh does NOT forward "$@" — for grid cells use sbatch_gsm8k_grid.sh (passthrough + fresh per-cell CKPT_DIR, rm -rf'd before launch):

sbatch --job-name=grid_cudagraph sbatch_gsm8k_grid.sh generator.enforce_eager=false
sbatch --job-name=grid_tbs128    sbatch_gsm8k_grid.sh trainer.train_batch_size=128 trainer.policy_mini_batch_size=128

--job-name=grid_<cell> is load-bearing: the script derives CKPT_DIR=$SF/grid_ckpts/${SLURM_JOB_NAME#grid_} from it. Per-cell scripts in notes/RL/gsm8k_grid_leonardo/scripts/run_<cell>.sh; launchers launch_throughput_grid.sh/launch_accuracy_grid.sh + catalogs *_grid_cells.txt in hpc/skyrl_standard/leonardo/.

4. Grid structure (one-factor-at-a-time off the base)

• Throughput grid (grid.md, 18 cells, maximize sec/step / eff tok/s): varies train_batch_size (32→512), n_samples_per_prompt (4→16), gpu_memory_utilization (0.70→0.85), enforce_eager (CUDA graphs), engine layout (4×TP1 vs 2×TP2 vs 1×TP4), micro_*_batch_size_per_gpu, reshard_after_forward, colocate_all. Winners: enforce_eager=false = −29% sec/step (always on); 4×TP1 > 2×TP2 > 1×TP4; colocated > disaggregated at 4 GPU. Base width is gen-bound (cudagraph fixes it); past ~tbs128 it's policy_train compute-bound; never memory-bound (KV <11%).
• Accuracy grid (accuracy_grid.md, 20 cells, maximize pass@8 to convergence off the throughput winner combo_C): varies lr (dominant lever; GRPO knee 1e-5, 3e-7 undertrains, 3e-5 unstable), n_samples (n8 winner), max_generate_length (gen1024 winner), use_kl_loss/kl_loss_coef, rollout temp, eps_clip_high (DAPO), entropy bonus (use_entropy_loss=true, entropy_loss_coef=0.01 = anti-collapse winner), reward shaping. Best: combo_acc (lr1e-5 + n8 + gen1024) → pass@8 ~0.97 but entropy collapses; combo_acc_stab (+ entbonus) holds ~0.95–0.98 WITHOUT collapse.

gsm8k: short CoT (~245–268 tokens), exact-match ±1 reward, and lr knee 1e-5. MATH/aime needs max_generate_length=4096; 32B OOMs on a single 4×A100-64GB node, so use multi-node.

5. OPD — on-policy distillation (teacher→student)

Student (Qwen3-1.7B) generates; per-token reward = −KL(student‖teacher) over the student's tokens. Entrypoint examples.on_policy_distillation_logits.main_on_policy_distill_logits (NOT main_base, NOT the agentic main_tbench_opd_logits which needs Daytona). Knobs: advantage_estimator=no_op, policy_loss_type=importance_sampling, use_kl_in_reward=true, use_kl_loss=false; the FSDP ref worker is loaded with the teacher + a separate vLLM-served teacher supplies top-K logprobs (teacher.top_k_logprobs).

sbatch sbatch_opd_qwen3.sh                         # smoke defaults (2 nodes, ≤90 min)
sbatch --job-name=opd_q3_full --time=08:00:00 sbatch_opd_qwen3.sh \
  MAX_STEPS=60 EPOCHS=2 TRAIN_BATCH_SIZE=64 MINI_BATCH_SIZE=64 N_SAMPLES=8 MAX_GEN_LEN=1024 TOPK=128

Layout (2 nodes × 4 A100-64GB): student colocated (FSDP2 ↔ 4× vLLM TP1) on node-0; teacher Qwen3-32B TP2 (32B bf16 ≈ 64 GB > one 64 GB card) on its own Ray PACK PG on node-1; 2 GPUs spare. Shared tokenizer → retokenization is a no-op (Qwen3-1.7B is the nearest size to a nonexistent 1.5B).

OPD is teacher-score-bound (90–97% of each step). The speed lever is top_k; the lr knee is 3e-5. Recommended OPD: lr=3e-5, top_k=64, n_samples=8, gen=1024, teacher TP2, cudagraph off.