Huggingface LLM Trainer

TRL Training on Hugging Face Jobs

Overview

Train language models using TRL (Transformer Reinforcement Learning) on fully managed Hugging Face infrastructure. No local GPU setup required — models train on cloud GPUs and results are automatically saved to the Hugging Face Hub.

TRL provides multiple training methods:

• SFT (Supervised Fine-Tuning) — standard instruction tuning
• DPO (Direct Preference Optimization) — alignment from preference data
• GRPO (Group Relative Policy Optimization) — online RL training
• Reward Modeling — train reward models for RLHF

See references/training_methods.md for method overviews and selection guidance.

When to Use Unsloth

Use Unsloth (references/unsloth.md) instead of standard TRL when GPU memory is limited (~60% less VRAM), speed matters (~2x faster), training large models (>13B), or training Vision-Language Models (Unsloth has FastVisionModel support). See scripts/unsloth_sft_example.py for a production-ready training script.

Key Directives

1. Submit jobs via hf jobs uv run (CLI) or the hf_jobs() MCP tool if the Hugging Face MCP server is configured — pass the training script inline, don't save to a local file unless the user explicitly requests it. If the user asks to "train a model" or "fine-tune", create the training script AND submit the job immediately.
2. Always include Trackio for real-time monitoring — use scripts/ templates.
3. Provide job details after submission: job ID, monitoring URL, estimated time; note the user can request status checks later.
4. Use example scripts as templates: scripts/train_sft_example.py, scripts/train_dpo_example.py, etc.

Local Script Execution

Repository scripts use PEP 723 inline dependencies. Run them with uv run:

uv run scripts/estimate_cost.py --help
uv run scripts/dataset_inspector.py --help

Prerequisites Checklist

Account & Authentication:

• Hugging Face account with Pro/Team/Enterprise plan (Jobs require a paid plan); authenticated login.
• HF_TOKEN for Hub push is CRITICAL — the training environment is ephemeral, so results are lost unless pushed to the Hub. Token must have write permissions. Pass secrets={"HF_TOKEN": "$HF_TOKEN"} in the job config.

Dataset Requirements:

• Must exist on the Hub or be loadable via datasets.load_dataset().
• Format must match the training method (SFT: messages/text/prompt-completion; DPO: chosen/rejected; GRPO: prompt-only). Always validate unknown datasets first (see Dataset Validation below).
• Size appropriate for hardware (demo: 50-100 examples on t4-small; production: 1K-10K+ on a10g-large/a100-large).

Critical Settings:

• Timeout must exceed expected training time — default 30min is too short; minimum recommended 1-2 hours. The job fails and loses all progress if the timeout is exceeded.
• Hub push must be enabled: push_to_hub=True, hub_model_id="username/model-name", secrets={"HF_TOKEN": "$HF_TOKEN"}.

Asynchronous Jobs

Training jobs run in the background and can take hours. After submitting: report the job ID, monitoring URL, and estimated time; wait for the user to request status checks rather than polling. Initial logs can take 30-60 seconds to appear.

Quick Start

Sequence length: TRL config classes use max_length (not max_seq_length). Default is max_length=1024 (truncates from right) — override higher for longer context, lower under memory constraints, or None for vision models (to avoid cutting image tokens).

Approach 1: UV Scripts (default choice)

UV scripts use PEP 723 inline dependencies for clean, self-contained training:

hf_jobs("uv", {
    "script": """
# /// script
# dependencies = ["trl>=0.12.0", "peft>=0.7.0", "trackio"]
# ///
from datasets import load_dataset
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig
import trackio

dataset = load_dataset("trl-lib/Capybara", split="train")
dataset_split = dataset.train_test_split(test_size=0.1, seed=42)

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=dataset_split["train"],
    eval_dataset=dataset_split["test"],
    peft_config=LoraConfig(r=16, lora_alpha=32),
    args=SFTConfig(
        output_dir="my-model", push_to_hub=True, hub_model_id="username/my-model",
        num_train_epochs=3, eval_strategy="steps", eval_steps=50,
        report_to="trackio", project="my_project", run_name="my_run",
    ),
)
trainer.train()
trainer.push_to_hub()
""",
    "flavor": "a10g-large",
    "timeout": "2h",
    "secrets": {"HF_TOKEN": "$HF_TOKEN"},
})

The script parameter accepts inline code or a publicly-accessible/Hub/GitHub/Gist URL — local file paths do not work (jobs run in isolated containers with no access to the local filesystem). To use a local script, upload it to the Hub first (hf upload ...) and reference its resolved URL.

Approach 2: TRL Maintained Scripts

Run TRL's battle-tested example scripts directly from a URL, passing CLI-style script_args (--model_name_or_path, --dataset_name, --output_dir, --push_to_hub, --hub_model_id). Available at https://github.com/huggingface/trl/tree/main/examples/scripts.

Approach 3: HF Jobs CLI

When no hf_jobs-style tool is available, use the hf jobs CLI directly. Flags must come before the script URL, the subcommand order is hf jobs uv run (not run uv), and use --secrets (plural):

hf jobs uv run \
  --flavor a10g-large --timeout 2h --secrets HF_TOKEN \
  "https://huggingface.co/user/repo/resolve/main/train.py"

Check status: hf jobs ps, hf jobs logs <job-id>, hf jobs inspect <job-id>, hf jobs cancel <job-id>.

Approach 4: TRL Jobs Package

uvx trl-jobs sft --model_name Qwen/Qwen2.5-0.5B --dataset_name trl-lib/Capybara gives pre-configured defaults, automatic Trackio integration, and automatic Hub push — best for terminal-only, quick local experimentation. Repository: https://github.com/huggingface/trl-jobs.

Hardware Selection

Use LoRA/PEFT for models >7B; multi-GPU is handled automatically by TRL/Accelerate. See references/hardware_guide.md for full specs.

Saving Results to the Hub

The Jobs environment is ephemeral — everything is deleted when the job ends. Set push_to_hub=True and hub_model_id="username/model-name" in the training config, and pass secrets={"HF_TOKEN": "$HF_TOKEN"} in the job submission. See references/hub_saving.md for troubleshooting.

Timeout Management

Default is 30 minutes — too short for real training. Set explicitly ("timeout": "2h", formats: "90m", "2h", seconds as integer) with a 20-30% buffer for loading/checkpointing/Hub push. Guideline: quick demo 10-30min, development 1-2h, production (3-7B) 4-6h. On timeout the job is killed immediately and unsaved progress is lost.