--- name: rl-agentic-job-cleanup description: >- Preserve + publish a finished RL (SkyRL/GRPO) training checkpoint after the job terminates (completed at max_steps OR early-stopped/scancelled) on an HPC cluster (Jupiter/Leonardo/Perlmutter). Covers: cancel pending retries, pick the BEST checkpoint by trailing-5 EMA of reward across the full restart chain, flatten weights to repo root, secret-scan, `hf upload` to laion/--, Supabase DB register (--training-type RL + cross-user FK safety pre-check), upload training traces to penfever/, parse metrics, and clean up. Use when an RL run needs its model uploaded + registered, or when asked to "run the RL cleanup checklist". Distinct from SFT cleanup (that's a different flow). --- > ⚠ **Do not add comments to YAMLs. Report your recommendations directly to the supervisor.** # rl-agentic-job-cleanup After an RL job terminates, publish `laion/--` (weights at repo root), trace dataset `penfever/`, and a Supabase `models` row (`training_type=RL`). ## Rules - **`hf upload`, NEVER `hf upload-large-folder`** (deprecated stub; deadlocks on HF LFS 429s). Wrap long uploads in **`tmux`**, not `nohup`. - **`--private` is a no-value flag** — do NOT pass `--private false`. Default is PUBLIC; omit it. - Run trace upload + `parse_skyrl_metrics.py` from the **`otagent` conda env** (the RL venv lacks `google.cloud.storage` + matplotlib). - On **Leonardo**, login-node `hf upload` is SIGKILLed at ~100s — use the sbatch+tunnel upload pattern. ## 0. Cancel pending retries ```bash squeue -u $USER --format='%.18i %.80j %.8T' | grep scancel ``` ## 1. Select the checkpoint — trailing-5 reward EMA ```bash # NOTE: there is an empty exports/ at the base level — ignore it. Real HF-exportable ckpts are nested: ls -lt $EXPERIMENTS_DIR///exports/ | head -10 ``` Use the **EMA of `reward/avg_raw_reward` over a trailing-5 window**. Rules: - **EMA across ALL chronological steps, regardless of chain restarts.** Collect every `.out` and sort by `trainer/global_step`; do not compute it per-chain link. - Standard 5-period EMA: `α = 2/(5+1) = 1/3`; `EMA_n = α·reward_n + (1−α)·EMA_{n−1}`, `EMA_1 = reward_1`. - **Never select the first saved checkpoint** (`global_step_5` with `hf_save_interval: 5`) — EMA not warmed up. Start from the second-saved step (typically 10). - Among saved, aligned checkpoints (multiples of `hf_save_interval`, excluding the first), upload the highest EMA. If scancelled before a save-aligned max-step, cap at the latest saved multiple. ```python import json, glob, re rewards = {} # step -> avg_raw_reward for fn in glob.glob(f"{EXP_DIR}/logs/*.out"): for line in open(fn): m = re.search(r'trainer/global_step":\s*(\d+).*avg_raw_reward":\s*([\d.eE+-]+)', line) if m: step, r = int(m.group(1)), float(m.group(2)) rewards.setdefault(step, r) # first-seen wins (chain links may overlap) steps = sorted(rewards) alpha = 1/3 ema = {}; prev = rewards[steps[0]] for s in steps: prev = alpha * rewards[s] + (1 - alpha) * prev ema[s] = prev SAVE_EVERY = 5 # match hf_save_interval aligned_eligible = [s for s in steps if s % SAVE_EVERY == 0 and s >= 2 * SAVE_EVERY] best = max(aligned_eligible, key=ema.get) print(f"best EMA={ema[best]:.4f} at step={best} (reward at that step={rewards[best]:.4f})") ``` Upload the checkpoint at `exports/global_step_/`. ## 2. Locate the W&B run (optional) From the job logs / `trainer_log.jsonl`: `https://wandb.ai/dogml/OpenThoughts-Agent/runs/`. (Jupiter has no W&B — omit.) ## 3. Flatten model files to the upload-dir root ```bash UPLOAD_DIR=/e/scratch/jureap59/feuer1/upload_staging/- mkdir -p $UPLOAD_DIR cp $EXPORT_DIR/policy/* $UPLOAD_DIR/ ls $UPLOAD_DIR/ # safetensors, config.json, tokenizer files all at root ``` ## 4. Copy the launch config for reproducibility ```bash cp hpc/skyrl_yaml/.yaml $UPLOAD_DIR/rl_config.yaml ``` ## 5. Scan for secrets before upload ```bash trufflehog filesystem $UPLOAD_DIR --no-update # if installed trufflehog filesystem $EXPERIMENTS_DIR// --no-update # logs/traces too # fallback: grep -rIE '(sk-[a-zA-Z0-9]{20,}|AKIA[0-9A-Z]{16}|ghp_[a-zA-Z0-9]{36}|hf_[a-zA-Z0-9]{34}|eyJ[a-zA-Z0-9._-]+)' $UPLOAD_DIR ``` Redact before proceeding (the wrapper emits a JSON finding record even when clean): ```bash python -m scripts.harbor.secret_redaction "$UPLOAD_DIR" ``` ## 6. Upload to HuggingFace — `laion/--` Include the global step and base-model size suffix (`-20-32B`, `-30-8B`). ```bash # tmux for long uploads. OMIT --private (no-value flag; default public). hf upload laion/-- $UPLOAD_DIR . --repo-type=model ``` > The SkyRL trainer auto-pushes `laion/` with weights under `checkpoints/step_N/`. Upload the > manually flattened export to `--` instead. ## 7. Register in the DB (`--training-type RL`) — with cross-user FK safety Delete the trainer's auto-registered duplicate **only if safe**, then push the correct row. If any **other-user** row in `sandbox_jobs`, `sandbox_trial_model_usage`, or elsewhere FKs the auto-row, stop; do not delete or mutate the FK'd rows. Restrict all writes to rows you own. ```python other_users_fk = (c.table("sandbox_jobs").select("id,username,model_id") .eq("model_id", auto_row_id).neq("username", os.environ.get("USER","")).execute()) if other_users_fk.data: print(f"SKIPPING auto-row delete — {len(other_users_fk.data)} other-user rows FK'd.") else: c.table("models").delete().eq("name", "laion/").execute() # optional, ONLY if pre-check passed: HfApi().delete_repo("laion/", repo_type="model") ``` Then register the `--` repo (`--training-type RL` is REQUIRED — the script defaults to SFT): ```bash python scripts/database/manual_db_push.py \ --hf-model-id laion/-- \ --base-model \ --dataset-name \ # comma-separated for multi-dataset → sets dataset_names --training-type RL # --wandb-run optional (defaults to now) ``` **Verify `--base-model`** is the exact HF repo trained from, not a default. Cross-check the job-name suffix, `trainer.policy.model.path` in the `.out` launch command, or `notes/ot-agent/rl_experiments.md`. ## 8. Upload RL traces → `penfever/` From the **otagent** env, always pass `--skip_register` for RL; register the model separately in step 7. ```bash python -m scripts.harbor.make_and_upload_trace_dataset \ --job_dir "$EXPERIMENTS_DIR//" \ --repo_id penfever/ --episodes last --skip_register ``` **Never subsample or cap**: upload the full trial set. The script reads the inner `/` `trace_jobs/`. For image-backed Jupiter runs, pass the same inner run root. The exporter detects `artifact_store.img`, refuses an unsafe mount while a writer lock is active, mounts it read-only after the link exits, and unmounts it when export finishes. Legacy bare `trace_jobs/` runs keep the existing path. > `make_and_upload_trace_dataset` buffers the full dataset; `chunk_size` does not bound peak RAM. Large login-node > uploads can OOM; do not respond by sampling. Then add a **"Training Traces"** section to `$UPLOAD_DIR/README.md` (append if a model card exists, don't overwrite) linking `penfever/`: ```markdown ## Training Traces Training-time Daytona/Harbor rollouts: **[penfever/](https://huggingface.co/datasets/penfever/)** (the `last` episode of each trial — the rollouts the policy trained on after rollback/truncation). ``` ## 9. Parse metrics and preserve training logs ```bash python scripts/analysis/parse_skyrl_metrics.py \ $EXPERIMENTS_DIR//logs $UPLOAD_DIR/training_logs \ --trace_jobs_dir $EXPERIMENTS_DIR///trace_jobs cp $EXPERIMENTS_DIR///trainer_log.jsonl $UPLOAD_DIR/training_logs/ 2>/dev/null cp $EXPERIMENTS_DIR//logs/_*.out $UPLOAD_DIR/training_logs/ hf upload laion/-- $UPLOAD_DIR . --repo-type=model # additive ``` Produces `metrics.csv`, `vllm_metrics.csv`, `trial_stats.csv`, `report.md`, `reward_plot.png`. The metrics reader also detects a missing `/trace_jobs` beside `artifact_store.img` and mounts the inactive image read-only for trial statistics. **WARNING:** never use `huggingface_hub.upload_folder()` without `delete_patterns=[]` — it deletes files absent locally and clobbers the weights. `hf upload` is additive (safe). ## 10. Verify the published model and write the completion record The model is complete only when its remote repo has weights, `README.md` with **Training Traces**, and redacted `training_logs/`. Record every artifact as **present**, **absent**, or **not applicable** in the cleanup handoff; an absent required artifact is not a completed cleanup. ```bash hf api repo-info laion/-- --repo-type model --expand siblings \ | python -c 'import json,sys; files=[x["rfilename"] for x in json.load(sys.stdin)["siblings"]]; print({"weights": any(x.endswith(".safetensors") for x in files), "readme": "README.md" in files, "training_logs": any(x.startswith("training_logs/") for x in files)})' ``` Fetch `README.md` and verify it contains `## Training Traces` and the exact `penfever/` URL. ## 11. Clean up the experiments dir After all prior steps succeed, `rm -rf` the local job dir. Detach a large GPFS removal with `nohup` or `tmux`; do not `du` or `find` it first.