Agent skill

Sft Job Cleanup

by open-thoughts in open-thoughts/OpenThoughts-Agent

Publish + clean up a finished LLaMA-Factory SFT job on a no-internet HPC cluster (Jupiter/Leonardo): cancel pending retries, drop intermediate checkpoints, HF-upload the model to its configured…

Apache-2.0Auto-check passedAI & LLM Engineering

Install Sft Job Cleanup

skills CLI
$ npx skills add open-thoughts/OpenThoughts-Agent --skill sft-job-cleanup -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install open-thoughts/OpenThoughts-Agent sft-job-cleanup --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/open-thoughts/OpenThoughts-Agent.git skills-src && mkdir -p .claude/skills && cp -r skills-src/.agents/skills/sft-job-cleanup .claude/skills/sft-job-cleanup && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
sft-job-cleanup
GitHub stars
301
Token cost
~1.9k tokens
SKILL.md length
591 words
Files
1
Skills in repo
44
Repo updated
First seen
Licence
Apache-2.0

At a glance

Publish + clean up a finished LLaMA-Factory SFT job on a no-internet HPC cluster (Jupiter/Leonardo): cancel pending retries, drop intermediate checkpoints, HF-upload the model to its configured…

  • An SFT fine-tune finishes and needs uploading + registering
  • SKILL.md covers Recognition heuristic — which…, Cross-cutting upload rules…, 8B SFT Job Cleanup Checklist and 32B SFT Job Cleanup Checklist…, plus 1 more section
  • Calls hf and python
  • Run the SFT cleanup checklist

What it does

Sft Job Cleanup is an agent skill from open-thoughts/OpenThoughts-Agent. Publish + clean up a finished LLaMA-Factory SFT job on a no-internet HPC cluster (Jupiter/Leonardo): cancel pending retries, drop intermediate checkpoints, HF-upload the model to its configured --hubmodelid, register in Supabase via manualdbpush (--training-type SFT default), and free disk. Covers the 8B path (root safetensors, direct upload), the 32B/ZeRO-3 path (consolidate shards → safetensors first), the Qwen3.5 preprocessorconfig copy, the don't-upload-partials policy, and the hf-upload gotchas (tmux not…

Its SKILL.md is about 1.9k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.

It sits in AI & LLM Engineering. It works with Qwen, tmux and Supabase. The repository describes itself as: Data recipes and robust infrastructure for training AI agents. The licence is Apache-2.0.

When your agent uses it

  • An SFT fine-tune finishes and needs uploading + registering
  • Run the SFT cleanup checklist

Example prompts

  • “run the SFT cleanup checklist”
  • “/sft-job-cleanup”

Requirements

  • Python 3

What it can do on your machine

Read from SKILL.md and the folder at commit 3bd1917. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Shell commands in SKILL.md call:

    • hf
    • python

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    No URLs in SKILL.md.

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Sft Job Cleanup loads about 1.9k tokens when it runs. Until then it costs about 204 tokens; SKILL.md has 591 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~204
When it runs · the whole SKILL.md, loaded when a task matches
~1.9k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from open-thoughts/OpenThoughts-Agent at commit 3bd1917, republished under its Apache-2.0 licence (© open-thoughts). 591 words, ~1,858 tokens.

Download SKILL.mdSave it as .claude/skills/sft-job-cleanup/SKILL.md (or your agent's skills folder).
name
sft-job-cleanup
description
Publish + clean up a finished LLaMA-Factory SFT job on a no-internet HPC cluster (Jupiter/Leonardo): cancel pending retries, drop intermediate checkpoints, HF-upload the model to its configured --hub_model_id, register in Supabase via manual_db_push (--training-type SFT default), and free disk. Covers the 8B path (root safetensors, direct upload), the 32B/ZeRO-3 path (consolidate shards → safetensors first), the Qwen3.5 preprocessor_config copy, the don't-upload-partials policy, and the hf-upload gotchas (tmux not nohup, `hf upload` not `-large-folder`, Leonardo sbatch-tunnel not login node). Use when an SFT fine-tune finishes and needs uploading + registering, or "run the SFT cleanup checklist". Distinct from RL cleanup (rl-agentic-job-cleanup) and datagen cleanup (datagen-job-cleanup).

sft-job-cleanup

After an SFT job completes on Jupiter or Leonardo, publish the model and clean up.

Recognition heuristic — which path? (check the checkpoint root first)

bash
ls $CHECKPOINTS_DIR/<job_name>/ | grep -E 'safetensors|global_step'
  • Root model-*.safetensors → 8B path (including Qwen3.5; no consolidation).
  • global_stepN/ plus zero_to_fp32.py, without root safetensors → 32B path (consolidate ZeRO-3 shards).

Cross-cutting upload rules (apply to both paths)

  • hf upload, NEVER hf upload-large-folder (deprecated stub + deadlocks on HF LFS 429s). Wrap any non-trivial upload in tmux, not nohup/disown.
  • --private is a no-value flag — omit it (default public); --private false is a CLI parse error.
  • Jupiter: login node has direct internet → hf upload from the login node (in tmux) works.
  • Leonardo: the login node SIGKILLs long processes at ~100s → use the sbatch compute-node + SSH-tunnel upload (.agents/ops/leonardo/ops.md "Leonardo HF Upload — Use sbatch, NOT the Login Node" / sft-launch).
  • Don't upload partials: if training is below 100%, relaunch and auto-resume. Salvage-upload only with explicit approval, as laion/<job_name>-<step>-<size>.
  • Tokenizer sanity check: tokenizer_config.json extra_special_tokens must be a dict, not a list; replace a list with {} before upload.
    bash
    python -c "import json;d=json.load(open('<ckpt>/tokenizer_config.json'));assert isinstance(d.get('extra_special_tokens',{}),dict), 'LIST — coerce to {}'"

8B SFT Job Cleanup Checklist

0. Cancel pending retries (so stale restarts don't fire mid-upload):

bash
squeue -u $USER --format='%i %j %T' | grep <job_name> | grep PENDING | awk '{print $1}' | xargs -r scancel

1. Remove intermediate checkpoints (don't upload cruft):

bash
rm -rf $CHECKPOINTS_DIR/<job_name>/checkpoint-*  $CHECKPOINTS_DIR/<job_name>/.cache

1b. Qwen3.5 only — copy preprocessor_config.json from the base model:

bash
cp /path/to/Qwen3.5-9B/preprocessor_config.json  $CHECKPOINTS_DIR/<job_name>/   # or the -27B base

2. Upload model weights to HuggingFace. Naming: full final upload (training reached 100%) → the configured --hub_model_id from the launch command (laion/<descriptive_name>, NO step/size suffix — do NOT use the job name verbatim). (Partial salvage, only-if-OK'd → laion/<job_name>-<step>-<size>.)

bash
# Jupiter login node (direct internet). On LEONARDO use the §11 sbatch-tunnel — login-node hf upload dies at ~100s.
source ~/secrets.env
tmux new-session -d -s hf_upload_<short> \
    "source ~/secrets.env && hf upload <hub_model_id> $CHECKPOINTS_DIR/<job_name> . \
        --repo-type=model 2>&1 | tee $CHECKPOINTS_DIR/<job_name>/upload.log"
# tmux attach -t hf_upload_<short>  (Ctrl-b d to detach)

Wait for it to finish and verify the repo exists on HF Hub.

3. Register in the unified DB (SFT is the DEFAULT --training-type, no flag needed):

bash
python scripts/database/manual_db_push.py \
  --hf-model-id <hub_model_id> --base-model <base_model_hf> \
  --dataset-name <dataset_name>          # comma-separated for multi-dataset → sets dataset_names

SKIP for HF-only series (e.g. Delphi #6279 — YAMLs set enable_db_registration: false; do not register, and do not pass an anchor as --base-model since that auto-creates a base-model row).

4. Clean up the experiments dir — only after 1–3 succeed:

bash
rm -rf $EXPERIMENTS_DIR/<job_name>

Show full SKILL.md (296 more words)Show less

32B SFT Job Cleanup Checklist (DeepSpeed ZeRO-3 — consolidate first)

For 32B ZeRO-3 SFT without stage3_gather_16bit_weights_on_model_save: true, consolidate shards before upload.

0. Cancel pending retries (same as 8B).

1. Verify training reached 100% — trainer_log.jsonl shows current_steps == total_steps. Default policy: don't salvage partials (relaunch + resume); only proceed if explicitly OK'd as a partial.

2. Consolidate ZeRO-3 shards → fp32 state_dict → safetensors:

bash
python -m hpc.launch --job_type consolidate \
  --consolidate_input $CHECKPOINTS_DIR/<job_name> \
  --consolidate_output_repo <hub_model_id> \
  --consolidate_workdir <writable_workdir>/<job_name> \
  --time_limit 02:00:00 --num_nodes 1

Produces <workdir>/<job_name>/final_repo/ with root-level weights, tokenizer, and config. Do not rely on its final HF push; manually upload after final_repo/ is complete.

3. Manually upload from final_repo/ (NOT the original checkpoint dir — it still holds ZeRO-3 shards). Naming same as 8B (full → --consolidate_output_repo/--hub_model_id, no suffix):

bash
# Jupiter login node. On LEONARDO use the §11 sbatch-tunnel (131GB → ~4 min). tmux; hf upload (not -large-folder).
source ~/secrets.env
tmux new-session -d -s hf_upload_<short> \
    "source ~/secrets.env && hf upload <hub_model_id> <consolidate_workdir>/<job_name>/final_repo . \
        --repo-type=model 2>&1 | tee <consolidate_workdir>/<job_name>/upload.log"

4. Register in the unified DB (same as 8B step 3; SFT is the default; skip for HF-only series).

5. Clean up — only after 2–4 succeed, remove BOTH the sharded checkpoint dir AND the consolidate workdir (32B sharded ckpt ~700GB + workdir ~200GB):

bash
rm -rf $CHECKPOINTS_DIR/<job_name>  <consolidate_workdir>/<job_name>

Launch-side details (preamble, configs, sbatch patching, the no-internet pre-download) live in the sft-launch skill (per-cluster particulars in ops/<cluster>/ops.md §SFT); this skill is the post-run publish + cleanup.


Operating notes

  • Run full cleanup after a completed, 100% job: cancel pending chain, drop checkpoints, upload, register, and clean the experiment directory. Flag obvious anomalies; cancellation of running jobs remains user-driven.
  • Multi-dataset DB registration: pass the full comma-separated list to --dataset-name so dataset_names is populated (not just one dataset_id). Known limitation: the script stores it as a single string and does NOT trigger the multiple_datasets path (dataset_id ends up null) — verify the right field after registering. Single-dataset --dataset-name works fine and populates dataset_id.
  • Baseline model versioning (Sera/CoderForge): flat monotonic -v5/-v6/-v7 in HF repo names + README iteration tables, NOT nested v4-v2/v4-v3. In-flight runs keep their existing names; the NEXT retrain uses the new scheme (next Sera = v5, skipping v4 to avoid colliding with existing v4 artifacts).

© open-thoughts, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

Just SKILL.md in .agents/skills/sft-job-cleanup of open-thoughts/OpenThoughts-Agent.

Open the folder on GitHubat commit 3bd1917

Compare with similar skills

Sft Job Cleanup next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Sft Job Cleanup compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Sft Job Cleanup this skillopen-thoughts/OpenThoughts-Agent301—~1.9kAutomated safety check: PassApache-2.0
Agent Feature ReproductionQwenLM/qwen-code28k—~1.5kAutomated safety check: PassApache-2.0
tmux Real User TestingQwenLM/qwen-code28k—~2.3kAutomated safety check: PassApache-2.0
Fix Art IssuesOpenPipe/ART11k—~840Automated safety check: NotesApache-2.0
Diffusion Perf Optvllm-project/vllm-omni7.1k—~7.5kAutomated safety check: PassApache-2.0
Qwen Mtp GgufR6410418/Jackrong-llm-finetuning-guide1.7k—~1.7kAutomated safety check: PassMIT

Similar skills

  • Reproduces a feature from Codex or Claude Code in Qwen Code by running the reference agent under capture, reading the traces, then implementing matching behavior.

    28k GitHub stars~1.5k tokensUpdated today
    DevelopmentAuto-check passed
  • tmux Real User Testing

    QwenLM/qwen-code

    Drives Qwen Code in a real tmux session the way a user would and saves a readable step-by-step transcript of each screen for maintainers to review.

    28k GitHub stars~2.3k tokensUpdated today
    Testing & QAAuto-check passed
  • Fix Art Issues

    OpenPipe/ART

    Fix a GitHub issue on OpenPipe/ART and open a PR. An agent skill from OpenPipe/ART.

    11k GitHub stars~840 tokensUpdated today
    AI & LLM EngineeringAuto-check: notes
  • Diffusion Perf Opt

    vllm-project/vllm-omni

    Diagnose and optimize vLLM Omni diffusion workloads, especially Wan/Qwen/Flux-style image and video generation.

    7.1k GitHub stars~7.5k tokensUpdated today
    AI & LLM EngineeringAuto-check passed
  • Qwen Mtp Gguf

    R6410418/Jackrong-llm-finetuning-guide

    Complete agent-ready workflow for Qwen-family MTP or nextn GGUF conversion and release.

    1.7k GitHub stars~1.7k tokensUpdated 3 mo ago
    AI & LLM EngineeringAuto-check passed
  • Dy Note

    Rimagination/dy-note

    DyNote: systematically and efficiently extract raw Douyin/DY video data and analyze videos, comments, accounts, hashtags, and short-video scenes into evidence-graded learning notes, summaries…

    172 GitHub stars~4.5k tokensUpdated 3 mo ago
    AI & LLM EngineeringAuto-check passed

More from open-thoughts/OpenThoughts-Agent

All 44 skills in this repo
  • Analyze Dataset Token Length

    open-thoughts/OpenThoughts-Agent

    Analyze the token length of an OT-Agent conversation-format (ShareGPT-style) dataset — the per-trace distribution (median/p90/max) and/or counts under a token threshold + a metadata predicate (e.g.

    301 GitHub stars~1.5k tokensUpdated 12 days ago
    Auto-check passed
  • Analyze Id Eval Ranking

    open-thoughts/OpenThoughts-Agent

    Given a list of models (HF name stubs) that have valid agentic ID eval scores in Supabase, build a ranking table: raw per-benchmark accuracy on the 3 ID benchmarks (SWE-Bench-100…

    301 GitHub stars~3.1k tokensUpdated 12 days ago
    Auto-check passed
  • Analyze Job History Iris

    open-thoughts/OpenThoughts-Agent

    Run the Iris harbor job-history analyzer (scripts/iris/analyzeirisharborjob.py) on a datagen/eval job and read its JSON sidecar for trustworthy throughput / preemption / productive-trial stats.

    301 GitHub stars~2.9k tokensUpdated 12 days ago
    Auto-check passed
  • Analyze Rl Behavior

    open-thoughts/OpenThoughts-Agent

    Run the full RL behavioral-analysis pipeline (scripts/analysis/analyzerlbehavior.py) on a trained RL model to understand WHAT changed vs its pre-RL baseline, WHY, whether it PERSISTS, and its EVAL…

    301 GitHub stars~4.2k tokensUpdated 12 days ago
    Auto-check passed
  • Analyze Training Run Iris

    open-thoughts/OpenThoughts-Agent

    Detailed health check for a Levanter/executor TRAINING run on the marin Iris cluster (e.g.

    301 GitHub stars~2k tokensUpdated 12 days ago
    Auto-check passed
  • Code Create Staged Plan

    open-thoughts/OpenThoughts-Agent

    DESIGN a non-trivial codebase change (Harbor / MarinSkyRL / vLLM / OT-Agent / LLaMA-Factory) as a dependency-ordered STAGED PLAN before writing code — a feature port, a multi-step fix with parity…

    301 GitHub stars~1.5k tokensUpdated 12 days ago
    Auto-check passed

Questions about Sft Job Cleanup

What does Sft Job Cleanup do?

Publish + clean up a finished LLaMA-Factory SFT job on a no-internet HPC cluster (Jupiter/Leonardo): cancel pending retries, drop intermediate checkpoints, HF-upload the model to its configured…. Sft Job Cleanup is an agent skill from open-thoughts/OpenThoughts-Agent. Publish + clean up a finished LLaMA-Factory SFT job on a no-internet HPC cluster (Jupiter/Leonardo): cancel pending retries, drop intermediate checkpoints, HF-upload the model to its configured --hubmodelid, register in Supabase via manualdbpush (--training-type SFT default), and free disk.

When should I use Sft Job Cleanup?

Sft Job Cleanup fits situations like: an SFT fine-tune finishes and needs uploading + registering; run the SFT cleanup checklist.

How do I install Sft Job Cleanup in Claude Code?

Run `npx skills add open-thoughts/OpenThoughts-Agent --skill sft-job-cleanup -a claude-code`. Or copy the skill folder (.agents/skills/sft-job-cleanup in open-thoughts/OpenThoughts-Agent) into .claude/skills/sft-job-cleanup in your project. Claude Code loads it when a task matches its description.

How do I install Sft Job Cleanup in Codex?

Run `npx skills add open-thoughts/OpenThoughts-Agent --skill sft-job-cleanup -a codex`. Or copy the skill folder (.agents/skills/sft-job-cleanup in open-thoughts/OpenThoughts-Agent) into .agents/skills/sft-job-cleanup in your project. Codex loads it when a task matches its description.

Can I use Sft Job Cleanup in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add open-thoughts/OpenThoughts-Agent --skill sft-job-cleanup -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/sft-job-cleanup, .gemini/skills/sft-job-cleanup, .github/skills/sft-job-cleanup and .opencode/skills/sft-job-cleanup in your project.

What does Sft Job Cleanup need to run?

Going by SKILL.md and its folder, Sft Job Cleanup needs the command-line tools its instructions call (hf and python). Our summary lists: Python 3.

Does Sft Job Cleanup access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Sft Job Cleanup safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Sft Job Cleanup use?

Sft Job Cleanup is published under the Apache-2.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Sft Job Cleanup use?

About 1.9k tokens (SKILL.md is roughly 7.4k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Sft Job Cleanup?

Skills that share tags, products or a category with Sft Job Cleanup: Agent Feature Reproduction (QwenLM/qwen-code, 28k stars), tmux Real User Testing (QwenLM/qwen-code, 28k stars), Fix Art Issues (OpenPipe/ART, 11k stars) and Diffusion Perf Opt (vllm-project/vllm-omni, 7.1k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Sft Job Cleanup?

open-thoughts (a GitHub organization) maintains it in open-thoughts/OpenThoughts-Agent, which has 301 GitHub stars. The repository holds 44 skills in this directory. The repository was last updated on September 28, 2026.

Source: open-thoughts/OpenThoughts-Agent on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.