Finetuning

finetuning — teach an open model a form or behavior, not a fact

You own the discipline of adapting an open-weight model: deciding whether to fine-tune at all, then running SFT and (optionally) preference optimization with trl + peft, backend-agnostic. You are judged by whether the tuned model reliably produces the target form/behavior on a held-out set — not by train loss, and not by vibes.

The one sentence that routes half of all "should I fine-tune?" questions correctly: fine-tuning teaches form and behavior; RAG supplies facts. If the ask is "know our latest prices / docs / tickets," that is retrieval (../rag/SKILL.md), not training. If the ask is "sound like us, always emit this JSON, follow this reasoning pattern," that is here.

Decision gate — try this BEFORE reaching for a GPU

Fine-tuning is the last lever, not the first. Exhaust the cheaper, reversible options first; each row below is a real off-ramp.

Route out explicitly. Facts / freshness / citations → ../rag/SKILL.md. Squeezing a prompt before spending money → prompt-engineering. Picking which base model (size/license/task) → open-weights. Building the JSONL/preference corpus → training-data (LLM corpora, NOT tabular cleaning — that is data-cleaning). A fast single-GPU run + GGUF export → ../unsloth/SKILL.md (same LoRA/QLoRA concepts, one optimized implementation; this skill stays backend-agnostic). Downloading the base or pushing the adapter/merged model → huggingface. Serving the result → ../vllm/SKILL.md.

The cheapest fine-tune is the one you didn't need. Prompt + RAG solves most "make it behave" asks at zero training cost and updates instantly. Fine-tune when that ceiling is real, measured, and you can afford to re-run it every time the base model or data changes.

Version reality (verify at author time — this stack moves monthly)

• trl consolidated into a v1.x line (v1.0 landed ~2026; docs at author time referenced ~v1.8). Every method has a Trainer + a Config dataclass that inherits transformers.TrainingArguments (SFTTrainer/SFTConfig, DPOTrainer/DPOConfig, …). Confirm the current major before pinning: pip show trl / the TRL docs.
• transformers is on a v5.x line; peft, bitsandbytes, accelerate, datasets round out the stack. Do not freeze a pin as "the version" — say "current major is ~X, verify."
• Some methods migrated to trl.experimental.* (e.g. from trl.experimental.orpo import ORPOTrainer at author time). Import paths churn — check the method's doc page before copying an import.
• Model licenses are not facts to memorize. Llama ships under the Meta Community license (usage caps, not OSI-open); Gemma under custom Google terms; Qwen/Mistral vary per size and often Apache-2.0 — but read the specific model card, licenses change. License/size selection is open-weights.

LoRA / QLoRA vs full fine-tuning

Three options on one memory↔quality axis. Default to QLoRA unless you have a proven reason not to.

QLoRA (Dettmers et al., arXiv:2305.14314): load the base in 4-bit NF4 with double quantization, keep it frozen, and train LoRA adapters in bf16 on top. It made single-GPU fine-tuning of large models practical at near-full-FT quality.

import torch
from transformers import BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig

# 4-bit NF4 base (QLoRA). Verify arg names against current bitsandbytes/transformers.
bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

# LoRA over ALL linear layers — the safe target for QLoRA (PEFT quantization guide).
peft_config = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    bias="none", task_type="CAUSAL_LM",
    target_modules="all-linear",   # or explicit ["q_proj","k_proj","v_proj","o_proj",...]
)

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-7B-Instruct",       # instruct base → chat template already present
    args=SFTConfig(output_dir="out", max_length=2048, packing=True,
                   learning_rate=2e-4, num_train_epochs=2, bf16=True),
    train_dataset=dataset,                    # conversational: TRL applies the chat template
    peft_config=peft_config,
    quantization_config=bnb,                  # SFTTrainer + peft_config + this == QLoRA
)
trainer.train()
trainer.save_model("out")                     # saves the ADAPTER, not a merged model

peft_config + quantization_config on SFTTrainer is the current one-liner for QLoRA — no manual get_peft_model / prepare_model_for_kbit_training wiring needed. save_model writes the small adapter; merge to a standalone model only when serving requires it (see references).

Pipeline: SFT first, then (maybe) preference optimization

SFT → preference optimization is the standard post-training arc. SFT teaches the model the format and gives it the behavior by imitation. Preference optimization then sharpens which of several plausible outputs is better. Most projects need only SFT; add a preference stage when "the outputs are fine but I want the good one preferred" is the remaining gap.

Rule of thumb: have pairs → DPO (or ORPO to fuse the two stages); have only up/down votes → KTO; can score an answer programmatically → GRPO. Preference optimization uses a tiny learning rate.

# DPO after SFT — dataset has prompt / chosen / rejected columns.
from trl import DPOTrainer, DPOConfig
trainer = DPOTrainer(
    model="out",                              # your SFT checkpoint (or SFT+adapter)
    args=DPOConfig(output_dir="dpo-out", beta=0.1,   # beta = KL strength to the ref model
                   learning_rate=5e-7, max_length=1024,
                   precompute_ref_log_probs=True),   # saves memory; ref model auto-created
    train_dataset=pref_dataset,
    peft_config=peft_config,                  # LoRA works for preference stages too
)
trainer.train()

# GRPO — no preference pairs, a reward FUNCTION that returns a score per completion.
from trl import GRPOTrainer, GRPOConfig
def format_reward(completions, **kwargs):     # signature: gets completions (+ dataset cols via kwargs)
    return [1.0 if "\\boxed{" in c[0]["content"] else 0.0 for c in completions]

trainer = GRPOTrainer(
    model="out",
    reward_funcs=[format_reward],             # one or many; GRPOConfig.reward_weights to combine
    args=GRPOConfig(output_dir="grpo-out", num_generations=8,  # group size per prompt
                    beta=0.04, learning_rate=1e-6, use_vllm=True),  # vLLM speeds rollouts
    train_dataset=prompts_dataset,
)
trainer.train()

Full runnable SFT→DPO and GRPO scripts, ORPO/KTO variants, dataset schemas, and adapter-merge steps are in references/methods.md.

Data — quality over quantity (LIMA)

More rows is not the win. LIMA (arXiv:2305.11206) got strong instruction-following from ~1,000 carefully curated examples — "less is more for alignment." A thousand clean, on-distribution, correctly-templated examples beat 100k scraped noisy ones, which actively teach the model bad form. Building and validating that corpus (JSONL messages, preference pairs, dedup, contamination checks) is training-data — bring it here already clean.