SimPO Preference Training

What it does

When your agent uses it

• Aligning a model with preference pairs and no reference model
• Training an instruct model on chosen versus rejected responses
• Deciding between SimPO, DPO, PPO and GRPO for a project
• Fixing loss divergence or forgetting during preference training

Example prompts

• “Write a SimPO config to train Mistral 7B base on our chosen and rejected pairs.”
• “Adapt the Llama 3 8B Instruct SimPO config to our own preference dataset.”
• “Our SimPO loss is diverging; which learning rate and beta changes should I try?”
• “Compare SimPO and DPO for a team with a single node and limited compute.”

Requirements

• A conda environment with the SimPO training dependencies
• GPUs able to run accelerate with DeepSpeed ZeRO-3
• A dataset of chosen and rejected response pairs

What it can do on your machine

Read from SKILL.md and the folder at commit 773a529. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Shell commands in SKILL.md call:

  • conda
  • python
  • git

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  Hosts in commands or code, which the agent is likely to contact:

  • github.com
  • pytorch.org

  Also links to:

  • arxiv.org
  • huggingface.co

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

SimPO Preference Training loads about 1.5k tokens when it runs, and up to ~8.1k if it reads all its reference files. Until then it costs about 71 tokens; SKILL.md has 275 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~71

When it runs · the whole SKILL.md, loaded when a task matches

~1.5k

With references · SKILL.md plus every file in references/, read only if the agent opens them

~8.1k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from Orchestra-Research/AI-Research-SKILLs at commit 773a529, republished under its MIT licence (© Orchestra-Research). 275 words, ~1,479 tokens.

name

simpo-training

description

Simple Preference Optimization for LLM alignment. Reference-free alternative to DPO with better performance (+6.4 points on AlpacaEval 2.0). No reference model needed, more efficient than DPO. Use for preference alignment when want simpler, faster training than DPO/PPO.

version

1.0.0

author

Orchestra Research

license

MIT

tags

Post-Training, SimPO, Preference Optimization, Alignment, DPO Alternative, Reference-Free, LLM Alignment, Efficient Training

dependencies

torch, transformers, datasets, trl, accelerate

# Create environment
conda create -n simpo python=3.10 && conda activate simpo

# Install PyTorch 2.2.2
# Visit: https://pytorch.org/get-started/locally/

# Install alignment-handbook
git clone https://github.com/huggingface/alignment-handbook.git
cd alignment-handbook
python -m pip install .

# Install Flash Attention 2
python -m pip install flash-attn --no-build-isolation

ACCELERATE_LOG_LEVEL=info accelerate launch \
  --config_file accelerate_configs/deepspeed_zero3.yaml \
  scripts/run_simpo.py \
  training_configs/mistral-7b-base-simpo.yaml

# Model
model_name_or_path: mistralai/Mistral-7B-v0.1
torch_dtype: bfloat16

# Dataset
dataset_mixer:
  HuggingFaceH4/ultrafeedback_binarized: 1.0
dataset_splits:
  - train_prefs
  - test_prefs

# SimPO hyperparameters
beta: 2.0                  # Reward scaling (2.0-10.0)
gamma_beta_ratio: 0.5       # Target margin (0-1)
loss_type: sigmoid          # sigmoid or hinge
sft_weight: 0.0             # Optional SFT regularization

# Training
learning_rate: 5e-7         # Critical: 3e-7 to 1e-6
num_train_epochs: 1
per_device_train_batch_size: 1
gradient_accumulation_steps: 8

# Output
output_dir: ./outputs/mistral-7b-simpo

accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml \
  scripts/run_simpo.py training_configs/mistral-7b-base-simpo.yaml

model_name_or_path: meta-llama/Meta-Llama-3-8B-Instruct

dataset_mixer:
  argilla/ultrafeedback-binarized-preferences-cleaned: 1.0

beta: 2.5
gamma_beta_ratio: 0.5
learning_rate: 5e-7
sft_weight: 0.1             # Add SFT loss to preserve capabilities

num_train_epochs: 1
per_device_train_batch_size: 2
gradient_accumulation_steps: 4
output_dir: ./outputs/llama3-8b-simpo

accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml \
  scripts/run_simpo.py training_configs/llama3-8b-instruct-simpo.yaml

model_name_or_path: deepseek-ai/deepseek-math-7b-base

dataset_mixer:
  argilla/distilabel-math-preference-dpo: 1.0

beta: 5.0                   # Higher for stronger signal
gamma_beta_ratio: 0.7       # Larger margin
learning_rate: 3e-7         # Lower LR for reasoning
sft_weight: 0.0

num_train_epochs: 1
per_device_train_batch_size: 1
gradient_accumulation_steps: 16

learning_rate: 3e-7  # Reduce from 5e-7

beta: 1.0  # Reduce from 2.0

sft_weight: 0.1  # Add SFT loss component

beta: 5.0            # Increase from 2.0
gamma_beta_ratio: 0.8  # Increase from 0.5

per_device_train_batch_size: 1
gradient_accumulation_steps: 16  # Maintain effective batch

gradient_checkpointing: true

© Orchestra-Research, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 3 other files (references) in 06-post-training/simpo of Orchestra-Research/AI-Research-SKILLs.

• SKILL.md
• references/datasets.md
• references/hyperparameters.md
• references/loss-functions.md

Open the folder on GitHubat commit 773a529

Used in 4 other repositories

We found 4 copies of this SKILL.md (exact, near-identical or edited) in other folders, from 4 other GitHub owners. This page covers the copy in Orchestra-Research/AI-Research-SKILLs, which our catalogue first saw on October 7, 2026.

• Tommy-yw/RunbookHermesoptional-skills/mlops/simpo
• davila7/claude-code-templatescli-tool/components/skills/ai-research/post-training-simpo
• johnson7788/MultiUserClawhermes-agent/optional-skills/mlops/simpo
• OpenLAIR/dr-clawskills/post-training/simpo