--- name: sft-job-cleanup description: >- Publish + clean up a finished LLaMA-Factory SFT job on a no-internet HPC cluster (Jupiter/Leonardo): cancel pending retries, drop intermediate checkpoints, HF-upload the model to its configured --hub_model_id, register in Supabase via manual_db_push (--training-type SFT default), and free disk. Covers the 8B path (root safetensors, direct upload), the 32B/ZeRO-3 path (consolidate shards → safetensors first), the Qwen3.5 preprocessor_config copy, the don't-upload-partials policy, and the hf-upload gotchas (tmux not nohup, `hf upload` not `-large-folder`, Leonardo sbatch-tunnel not login node). Use when an SFT fine-tune finishes and needs uploading + registering, or "run the SFT cleanup checklist". Distinct from RL cleanup (rl-agentic-job-cleanup) and datagen cleanup (datagen-job-cleanup). --- # sft-job-cleanup After an SFT job completes on Jupiter or Leonardo, publish the model and clean up. ## Recognition heuristic — which path? (check the checkpoint root first) ```bash ls $CHECKPOINTS_DIR// | grep -E 'safetensors|global_step' ``` - Root `model-*.safetensors` → **8B path** (including Qwen3.5; no consolidation). - `global_stepN/` plus `zero_to_fp32.py`, without root safetensors → **32B path** (consolidate ZeRO-3 shards). ## Cross-cutting upload rules (apply to both paths) - **`hf upload`, NEVER `hf upload-large-folder`** (deprecated stub + deadlocks on HF LFS 429s). Wrap any non-trivial upload in **`tmux`**, not `nohup`/`disown`. - **`--private` is a no-value flag** — omit it (default public); `--private false` is a CLI parse error. - **Jupiter**: login node has direct internet → `hf upload` from the login node (in tmux) works. - **Leonardo**: the login node SIGKILLs long processes at ~100s → use the **sbatch compute-node + SSH-tunnel** upload (`.agents/ops/leonardo/ops.md` "Leonardo HF Upload — Use sbatch, NOT the Login Node" / `sft-launch`). - **Don't upload partials:** if training is below 100%, relaunch and auto-resume. Salvage-upload only with explicit approval, as `laion/--`. - **Tokenizer sanity check:** `tokenizer_config.json` `extra_special_tokens` must be a dict, not a list; replace a list with `{}` before upload. ```bash python -c "import json;d=json.load(open('/tokenizer_config.json'));assert isinstance(d.get('extra_special_tokens',{}),dict), 'LIST — coerce to {}'" ``` --- ## 8B SFT Job Cleanup Checklist **0. Cancel pending retries** (so stale restarts don't fire mid-upload): ```bash squeue -u $USER --format='%i %j %T' | grep | grep PENDING | awk '{print $1}' | xargs -r scancel ``` **1. Remove intermediate checkpoints** (don't upload cruft): ```bash rm -rf $CHECKPOINTS_DIR//checkpoint-* $CHECKPOINTS_DIR//.cache ``` **1b. Qwen3.5 only — copy `preprocessor_config.json` from the base model:** ```bash cp /path/to/Qwen3.5-9B/preprocessor_config.json $CHECKPOINTS_DIR// # or the -27B base ``` **2. Upload model weights to HuggingFace.** **Naming:** full final upload (training reached 100%) → the configured `--hub_model_id` from the launch command (`laion/`, NO step/size suffix — do NOT use the job name verbatim). (Partial salvage, only-if-OK'd → `laion/--`.) ```bash # Jupiter login node (direct internet). On LEONARDO use the §11 sbatch-tunnel — login-node hf upload dies at ~100s. source ~/secrets.env tmux new-session -d -s hf_upload_ \ "source ~/secrets.env && hf upload $CHECKPOINTS_DIR/ . \ --repo-type=model 2>&1 | tee $CHECKPOINTS_DIR//upload.log" # tmux attach -t hf_upload_ (Ctrl-b d to detach) ``` Wait for it to finish and verify the repo exists on HF Hub. **3. Register in the unified DB** (SFT is the DEFAULT `--training-type`, no flag needed): ```bash python scripts/database/manual_db_push.py \ --hf-model-id --base-model \ --dataset-name # comma-separated for multi-dataset → sets dataset_names ``` **SKIP for HF-only series** (e.g. Delphi #6279 — YAMLs set `enable_db_registration: false`; do not register, and do not pass an anchor as `--base-model` since that auto-creates a base-model row). **4. Clean up the experiments dir** — only after 1–3 succeed: ```bash rm -rf $EXPERIMENTS_DIR/ ``` --- ## 32B SFT Job Cleanup Checklist (DeepSpeed ZeRO-3 — consolidate first) For 32B ZeRO-3 SFT without `stage3_gather_16bit_weights_on_model_save: true`, consolidate shards before upload. **0. Cancel pending retries** (same as 8B). **1. Verify training reached 100%** — `trainer_log.jsonl` shows `current_steps == total_steps`. Default policy: don't salvage partials (relaunch + resume); only proceed if explicitly OK'd as a partial. **2. Consolidate** ZeRO-3 shards → fp32 state_dict → safetensors: ```bash python -m hpc.launch --job_type consolidate \ --consolidate_input $CHECKPOINTS_DIR/ \ --consolidate_output_repo \ --consolidate_workdir / \ --time_limit 02:00:00 --num_nodes 1 ``` Produces `//final_repo/` with root-level weights, tokenizer, and config. Do not rely on its final HF push; manually upload after `final_repo/` is complete. **3. Manually upload from `final_repo/`** (NOT the original checkpoint dir — it still holds ZeRO-3 shards). Naming same as 8B (full → `--consolidate_output_repo`/`--hub_model_id`, no suffix): ```bash # Jupiter login node. On LEONARDO use the §11 sbatch-tunnel (131GB → ~4 min). tmux; hf upload (not -large-folder). source ~/secrets.env tmux new-session -d -s hf_upload_ \ "source ~/secrets.env && hf upload //final_repo . \ --repo-type=model 2>&1 | tee //upload.log" ``` **4. Register in the unified DB** (same as 8B step 3; SFT is the default; skip for HF-only series). **5. Clean up** — only after 2–4 succeed, remove BOTH the sharded checkpoint dir AND the consolidate workdir (32B sharded ckpt ~700GB + workdir ~200GB): ```bash rm -rf $CHECKPOINTS_DIR/ / ``` > Launch-side details (preamble, configs, sbatch patching, the no-internet pre-download) live in the > **`sft-launch`** skill (per-cluster particulars in `ops//ops.md §SFT`); this skill is the post-run publish + cleanup. --- ## Operating notes - **Run full cleanup after a completed, 100% job:** cancel pending chain, drop checkpoints, upload, register, and clean the experiment directory. Flag obvious anomalies; cancellation of running jobs remains user-driven. - **Multi-dataset DB registration:** pass the full comma-separated list to `--dataset-name` so `dataset_names` is populated (not just one `dataset_id`). Known limitation: the script stores it as a single string and does NOT trigger the `multiple_datasets` path (`dataset_id` ends up null) — verify the right field after registering. Single-dataset `--dataset-name` works fine and populates `dataset_id`. - **Baseline model versioning (Sera/CoderForge):** flat monotonic `-v5`/`-v6`/`-v7` in HF repo names + README iteration tables, NOT nested `v4-v2`/`v4-v3`. In-flight runs keep their existing names; the NEXT retrain uses the new scheme (next Sera = v5, skipping v4 to avoid colliding with existing v4 artifacts).