Training Data

training-data — the corpus a fine-tune actually eats

You own the training corpus: the JSONL of chat turns, instruction triples, or preference pairs that a trainer reads. The deliverable is a validated, deduplicated, decontaminated, license-clean file in the exact shape the trainer expects, rendered through the target model's chat template. You stop the moment that file loads cleanly and round-trips through apply_chat_template. You do not choose LoRA rank or launch the run — that is finetuning / unsloth.

Loud boundary. This is LLM training corpora — messages, instruction triples, preference pairs. It is not:

• Tabular row cleaning — nulls, dtypes, dedupe of CSV rows, category normalization → data-cleaning.
• A retrieval corpus — chunking documents and embedding them for search → embeddings-search.
• Actually training or serving — hyperparameters, the run, export → finetuning, unsloth, huggingface.

Version reality (verified July 2026 — re-verify, these move monthly). TRL is on the v1.x line (its dataset-formats doc was tagged v1.8.0 at author time); transformers is in the 4.57+ era (mixed text+vision data needs ≥4.57); datasets is 4.x (the Json() feature type needs ≥4.7). Pin whatever you install — do not trust these numbers as current.

1. The format the trainer expects (pick by trainer, not by taste)

The trainer dictates the columns. Get this wrong and TRL either errors or, worse, trains on a mangled string. Two axes: format (standard = plain strings vs conversational = messages lists) and type (the task). One JSON object per line = JSONL.

Tiny JSONL of each (conversational values are lists of {role, content}; label is a JSON boolean):

# Alpaca instruction (standard) — classic; NOT a native TRL type, see below
{"instruction": "Classify the sentiment.", "input": "The battery dies in an hour.", "output": "negative"}

# Conversational messages (SFT) — the default for chat fine-tunes
{"messages": [{"role": "system", "content": "You are a terse support agent."}, {"role": "user", "content": "My order never arrived."}, {"role": "assistant", "content": "Sorry about that — what is your order number?"}]}

# Preference pair (DPO) — chosen beats rejected for the same prompt
{"prompt": [{"role": "user", "content": "Define a hash map in one sentence."}], "chosen": [{"role": "assistant", "content": "A hash map stores key-value pairs and finds a value by hashing its key to a bucket, giving average O(1) lookup."}], "rejected": [{"role": "assistant", "content": "It's a fast dictionary thing."}]}

# KTO / unpaired preference — one completion + a good/bad boolean label
{"prompt": [{"role": "user", "content": "Define a hash map in one sentence."}], "completion": [{"role": "assistant", "content": "It's a fast dictionary thing."}], "label": false}

Alpaca is not a native TRL type. {instruction, input, output} is the Stanford-Alpaca convention, still common in Unsloth notebooks, but TRL trains on text/messages/prompt+ completion. You must either (a) map it into messages (instruction+input → user, output → assistant), or (b) render it into a single text string via a prompt template — and append the EOS token yourself, or the model never learns to stop (the #1 Unsloth-Alpaca bug). Prefer (a) messages for chat models. Full field matrix, tool-calling (tools column) and vision (images) extras, and every type→type conversion live in references/formats.md.

2. Chat templates — the silent run-wrecker

A chat template is a Jinja string stored in the tokenizer (in tokenizer_config.json under chat_template, or a standalone chat_template.jinja in newer tokenizers). It maps a messages list to the exact token string the model was trained on — special tokens (<|im_start|>, [INST], <|start_header_id|>, …) and all. You render it, you never hand-type it:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("<target-model>")   # the model you will fine-tune

# TRAINING: no trailing generation prompt — the assistant turn is already in the data
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=False)

# INFERENCE: add_generation_prompt=True appends the assistant turn-start so the model continues
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)

Three ways this silently destroys a run — no error, just a worse model:

• Hand-formatting the tokens. Writing <|im_start|>user\n… strings yourself and getting one token, one newline, or the BOS wrong. Train-time string ≠ inference-time string → the model learns a distribution it is never served. Always render via apply_chat_template.
• Using the wrong model's template. The template must be the one of the model you are fine-tuning. Copy Llama's template onto a Qwen fine-tune and every example is subtly malformed.
• A base model with no template at all. Base (non-instruct) checkpoints often ship chat_template = None. apply_chat_template then raises — you must choose and attach a template (e.g. ChatML) and use that same one at inference forever after.

Also decide loss masking: for chat SFT you usually train only on the assistant tokens (completion_only_loss / assistant-only masking in SFTTrainer, or a completion-only collator), so the model is not penalized for "predicting" the user's words. TRL applies the template for you when the dataset is conversational — let it, rather than pre-flattening to text.

3. Synthetic data & distillation

Not enough real examples? Generate them. Two workhorses: Self-Instruct (seed a few hand-written examples, prompt a strong model to produce more, filter) and Evol-Instruct (iteratively mutate prompts to be harder/deeper). Wrap them in a pipeline framework rather than ad-hoc loops (see §7).

Licensing trap — read before you distill. Generating your training data from another model's outputs ("distillation") is a terms-of-service question, not just a quality one. Some providers' terms restrict using their outputs to train competing models; some open-weight licenses carry naming/derivative obligations (e.g. Llama-derived data/models may inherit naming requirements). Never assert a model's license from memory — check the specific model card and provider ToS at author time (licenses change). If in doubt, distill from an openly-licensed-for-this-use model, and record the provenance per example.

4. Dedup + decontamination (skip these and your numbers lie)

• Dedup. Exact dedup is trivial (hash the text). Real corpora need near-dup removal: MinHash + LSH (Jaccard similarity over shingles) catches templated/boilerplate repeats that inflate a few patterns. Dupes waste compute and bias the model toward whatever is over-represented.
• Decontamination — the one people forget. Remove any training example that overlaps your eval / benchmark test sets (n-gram overlap, e.g. long-n-gram match against MMLU, GSM8K, your own held-out set). If test items leak into training, your eval score is inflated and meaningless — you measured memorization, not capability. Decontaminate against every metric you will report, including your private eval. Code for both in references/synthesis-dedup-quality.md.