Sft Launch

sft-launch

⚠ Local clone = ground truth (CLAUDE.md §Always). ALL code/config/sbatch edits go in the local Mac checkout (~/Documents/OpenThoughts-Agent) → commit → push → git pull on the cluster. NEVER hand-edit, git commit, or leave divergent/ untracked changes on a cluster; no patch-by-rsync. New/changed configs are authored locally + synced, never on the cluster. Bake this into every subagent you dispatch.

SFT runs through python -m hpc.launch --job_type sft on both backends and all clusters. Read .agents/ops/<cluster>/ops.md §SFT for the cluster preamble, paths, QOS/wall, and cleanup mechanics.

1. Pick backend, cluster, env — then launch

Universal launch shape (fill per cluster from ops/<cluster>/ops.md §SFT):

python -m hpc.launch --job_type sft [--sft_backend axolotl] \
  --train_config_path sft/<lf_configs|axolotl_configs>/<cfg>.yaml \
  --num_nodes N --gpus_per_node <4|1> --time_limit <cluster max> \
  --dataset <hf-dataset> --role_tag role --user_tag user --assistant_tag assistant --content_tag content \
  --hub_model_id laion/<name> [--conda_env <env>]

Always --dry_run the first cell and inspect model, template, epochs, LR, role tags, push_to_hub, and output_dir in <exp>/configs/*_train_config.yaml.

2. Backend: axolotl specifics

• aarch64 clusters (TACC Vista / Jupiter GH200): use SDPA. Set attn_implementation: sdpa and install torchao==0.17.0 without dependencies in sft-axolotl.
• The launcher rebuilds the dataset block from flags. Pass --dataset and schema flags at launch; hand-authored datasets: applies only to direct axolotl.cli.preprocess.
• On internet-node clusters, set WANDB_MODE=disabled — the launcher sets report_to=wandb; wandb 0.28.x crashes on the compute-node service socket (WANDB_MODE=disabled makes it a no-op; loss still logs to trainer_state.json).
• Precision: pure_bf16: true (fp32-master OOMs an 8B on 96 GiB).
• Validated on TACC Vista (Stage 3 smoke, Stage 4 footgun-through-launcher, delphi masking canary). Full backend gotcha list → .agents/projects/axolotl/axolotl.md.

3. Delphi model SFT (both backends)

Delphi checkpoints use the Llama-3 tokenizer with reasoning/tool tokens. Both backends require:

1. Prep the tokenizer FIRST (single-token delphi specials): python sft/delphi/prepare_delphi_tokenizer.py --model <ckpt> --output <dir> (reserved-slot rename + mean-init → <|start_think|>/<|end_think|>/ <|tool_call|>/<|tool_result|> become single tokens). Launch with --model_path <dir>.
2. Use a Llama-3-family template, NEVER qwen3. The delphi template = Llama-3 header/turn format (<|start_header_id|>…<|eot_id|>, EOS <|eot_id|>)
   • the reasoning/tool tokens. qwen3 (ChatML <|im_start|>) would shred every example. This template×tokenizer mismatch is the #1 silent ruin — --dry_run
   • eyeball the first rendered example of EACH source (an instruction turn AND a <think> warmup example) before launching.

• Datasets registered in sft/delphi/dataset_info.json (per-dataset schema tags — the instruction sets use heterogeneous ShareGPT schemas). Launch with --dataset_dir sft/delphi and the 90/10 mix: --dataset <instr>,delphi_warmup --mix_strategy interleave_under --interleave_probs 0.9,0.1.
• Axolotl delphi path = jinja-as-ground-truth (train == serve). Config: chat_template: delphi + tokenizer_save_jinja_files: false + the template_integrity plugin (embeds the chat_template into tokenizer_config.json, covering the per-checkpoint dirs the flag ignores). Validate the loss mask with axolotl.cli.preprocess <cfg> --debug (assistant + <|start_think|>…<|end_think|> trained, user/system masked, 0 Last turn is not trainable skips). Canary: sft/axolotl_configs/delphi_canary.yaml (validated, TACC job 802053).
• The Delphi RL-scaling-laws grid (#6279) is HF-upload ONLY — enable_db_registration: false; do NOT run manual_db_push.py. LR = shared conventional SFT LR (2e-5) across all cells for comparability.

4. Config maps (cluster-agnostic)

• Qwen3-8B (sft/lf_configs/qwen3/ + …/extra/): 32k_base.yaml (default 32k thinking), 32k_base_nothink.yaml, 131k_base.yaml; extra/32k_base_bs96.yaml (node-scaling, §4b), extra/32k_base_bs96_opt1k.yaml (small <1k-row: 7ep/lr4e-5), extra/32k_base_bs96_opt100k.yaml (large ≈11k+: 5ep/lr4e-5), plus other sizes/coder.
• Qwen3-32B (…/32k_base_32b*.yaml): DeepSpeed ZeRO-3, writes sharded global_stepN/, NOT root safetensors → launch WITHOUT --hub_model_id, then consolidate → upload (§6 + ops/<cluster>/ops.md §SFT).
• Qwen3.5 hybrid (9B/27B) (sft/lf_configs/qwen3_5/*.yaml): GDN+Attention arch not in transformers 4.x → needs the sft-qwen35 env (transformers ≥5.3) + DISABLE_VERSION_CHECK=1. 9B → root safetensors (SKIP consolidate, like 8B); 27B → 32B consolidate flow. Copy preprocessor_config.json from base into the ckpt before upload (LF doesn't emit it; vLLM needs it).
• axolotl (sft/axolotl_configs/): smoke.yaml, parity_llama3.yaml, delphi_canary.yaml, marin/delphi_all3.yaml (all-3-plugins). aarch64 → SDPA.

4b. Node-scaling — the bs96 configs

bs96 fixes global_batch_size: 96 and derives gradient_accumulation_steps = 96 / (num_nodes*gpus). Keep 96 % (num_nodes*gpus_per_node) == 0.

5. Dataset mixing & the parse-tags rule

• --dataset is repeatable. Concatenate: --dataset A --dataset B --mix_strategy concat. Interleave: --mix_strategy interleave_under|interleave_over --interleave_probs 0.7,0.3 (weights in dataset order).
• Role tags are mandatory for Harbor/DCAgent datasets: --role_tag role --user_tag user --assistant_tag assistant --content_tag content. Older ShareGPT uses from/human/gpt/value.
• Mixed-schema MIXES: a single global --role_tag silently yields 0 assistant turns on the mismatched source. Register per-dataset columns/tags in a dataset_info.json and launch with --dataset_dir <registry> (how the Delphi mix works — §3).