Gemma Trainer

Gemma Training and Fine-Tuning Skill

1. Core Principles: Local Fine-Tuning Setup

When training locally, memory efficiency and execution speed are huge. Always guide the user to follow these best practices:

• Prioritize Unsloth: For local single-GPU training, always recommend Unsloth. It supports Gemma 4 natively, uses up to 70% less memory, and is up to 2x faster than standard Hugging Face PEFT training.
• Fall Back to TRL: For multi-GPU environments (using DDP/FSDP) or when Unsloth is unavailable, use Hugging Face TRL (SFTTrainer, DPOTrainer) coupled with PEFT and bitsandbytes (for QLoRA).
• Always use QLoRA (4-bit Quantization): Crucial for fitting Gemma models (like Gemma 4 12B/31B) into consumer VRAM.
• Manage Context Window & Max Length: Although Gemma 4 supports up to a 256K context window, recommend training with a context window of 2048 to 8192 tokens locally to prevent Out-Of-Memory (OOM) errors.

2. Choosing the Right Training Method

Help the user choose the correct workflow based on their goal:

• Supervised Fine-Tuning (SFT): Teaching new domains, specialized task instructions, or custom output structures.
  • Prerequisites: Raw text, instruction pairs, or chat logs.
  • Output: Adaptor trained on prompt/completion pairs.
• Direct Preference Optimization (DPO): Aligning model style, behavior, tone, or safety with human preferences.
  • Prerequisites: A previously SFT-trained Gemma model and preferred pairwise datasets.
  • Output: Aligning model weights directly without a separate reward head.
• Reward Modeling (RM): Training a scoring system to evaluate response quality.
  • Prerequisites: Binary preference pairwise datasets.
  • Output: A classification-style reward head on top of Gemma.

3. Dataset Preparation & Validation

Formatting issues are the #1 cause of poor training runs. Ensure you validate files using the utility script [assets/dataset_prep.py].

Gemma Chat Prompt Format

Ensure the dataset matches Gemma's official chat template:

<|turn>system
Your instruction here<turn|>
<|turn>user
Your query here<turn|>
<|turn>model
Your response here<turn|>

To avoid formatting drift, use the tokenizer's apply_chat_template during dataset tokenization.

Format Specifications

• SFT (Supervised Fine-Tuning): Format as a list of conversation turns.

  {
    "messages": [
      {"role": "user", "content": "Tell me a joke."},
      {"role": "model", "content": "Why did the computer go to the doctor? It had a virus!"}
    ]
  }

• DPO (Direct Preference Optimization): Requires pairwise samples containing a prompt, a chosen (better) response, and a rejected (worse) response.

  {
    "prompt": "Write a python function to compute factorial.",
    "chosen": "def factorial(n):\n    return 1 if n <= 1 else n * factorial(n - 1)",
    "rejected": "factorial is computed using recursion or loops. Just import math."
  }

• Reward Modeling: Format identically to DPO datasets. The RewardTrainer evaluates the pair and learns to output a higher logit score for chosen than for rejected.

Dataset Distillation & Synthesis (Teacher-Student)

Local knowledge distillation allows you to train small, lightweight student models (such as Gemma 4 E2B) using high-quality dataset outputs generated by larger, highly capable teacher models (such as Gemma 4 31B or Gemma 4 26B A4B).

Use the [assets/distill_dataset.py] utility script to generate fine-tuning datasets on your local machine.

4. Fine-Tuning Workflows

Supervised Fine-Tuning (SFT)

Use the [assets/sft_train.py] asset to launch a local QLoRA fine-tuning session.

• LoRA Hyperparameters:
  • Rank (r): 16 or 32 (Higher rank captures complex behaviors but consumes more memory).
  • Alpha (lora_alpha): 32 or 64 (Rule of thumb: lora_alpha = 2 * r).
  • Dropout (lora_dropout): 0.05 or 0.1 (Forcing the model to learn more robust features rather than relying on specific paths).
  • Target Modules: Use PEFT's Gemma 4 defaults scope to the LM layers.
  • Learning Rate: 2e-4 for QLoRA; 2e-5 for full fine-tuning.