simpo-training | SkillBundle

SimPO - Simple Preference Optimization

Quick start

SimPO is a reference-free preference optimization method that outperforms DPO without needing a reference model.

Installation:

# Create environment
conda create -n simpo python=3.10 && conda activate simpo
 
# Install PyTorch 2.2.2
# Visit: https://pytorch.org/get-started/locally/
 
# Install alignment-handbook
git clone https://github.com/huggingface/alignment-handbook.git
cd alignment-handbook

Training (Mistral 7B):

Common workflows

Workflow 1: Train from base model (Mistral 7B)

Config (mistral-7b-base-simpo.yaml):

Launch training:

Workflow 2: Fine-tune instruct model (Llama 3 8B)

Config (llama3-8b-instruct-simpo.yaml):

Launch:

Workflow 3: Reasoning-intensive tasks (lower LR)

For math/code tasks:

When to use vs alternatives

Use SimPO when:

Want simpler training than DPO (no reference model)
Have preference data (chosen/rejected pairs)
Need better performance than DPO
Limited compute resources
Single-node training sufficient

Algorithm selection:

SimPO: Simplest, best performance, no reference model
DPO: Need reference model baseline, more conservative
PPO: Maximum control, need reward model, complex setup
GRPO: Memory-efficient RL, no critic

Use alternatives instead:

OpenRLHF: Multi-node distributed training, PPO/GRPO
TRL: Need multiple methods in one framework
DPO: Established baseline comparison

Common issues

Issue: Loss divergence

Reduce learning rate:

Reduce beta:

Issue: Model forgets capabilities

Add SFT regularization:

Issue: Poor preference separation

Increase beta and margin:

Issue: OOM during training

Reduce batch size:

Enable gradient checkpointing:

Advanced topics

Loss functions: See references/loss-functions.md for sigmoid vs hinge loss, mathematical formulations, and when to use each.

Hyperparameter tuning: See references/hyperparameters.md for beta, gamma, learning rate selection guide, and model-size-specific recommendations.

Dataset preparation: See references/datasets.md for preference data formats, quality filtering, and custom dataset creation.

Hardware requirements

GPU: NVIDIA A100/H100 recommended
VRAM:
- 7B model: 1× A100 40GB (DeepSpeed ZeRO-3)
- 8B model: 2× A100 40GB
- 70B model: 8× A100 80GB
Single-node: DeepSpeed ZeRO-3 sufficient
Mixed precision: BF16 recommended

Memory optimization:

DeepSpeed ZeRO-3 (default config)
Gradient checkpointing
Flash Attention 2

Resources

Paper: https://arxiv.org/abs/2405.14734 (NeurIPS 2024)
GitHub: https://github.com/princeton-nlp/SimPO
Models: https://huggingface.co/princeton-nlp
Alignment Handbook: https://github.com/huggingface/alignment-handbook

SimPO - Simple Preference Optimization

Quick start

SimPO is a reference-free preference optimization method that outperforms DPO without needing a reference model.

Installation:

# Create environment
conda create -n simpo python=3.10 && conda activate simpo
 
# Install PyTorch 2.2.2
# Visit: https://pytorch.org/get-started/locally/
 
# Install alignment-handbook
git clone https://github.com/huggingface/alignment-handbook.git
cd alignment-handbook

Training (Mistral 7B):

Common workflows

Workflow 1: Train from base model (Mistral 7B)

Config (mistral-7b-base-simpo.yaml):

Launch training:

Workflow 2: Fine-tune instruct model (Llama 3 8B)

Config (llama3-8b-instruct-simpo.yaml):

Launch:

Workflow 3: Reasoning-intensive tasks (lower LR)

For math/code tasks:

When to use vs alternatives

Use SimPO when:

Want simpler training than DPO (no reference model)
Have preference data (chosen/rejected pairs)
Need better performance than DPO
Limited compute resources
Single-node training sufficient

Algorithm selection:

SimPO: Simplest, best performance, no reference model
DPO: Need reference model baseline, more conservative
PPO: Maximum control, need reward model, complex setup
GRPO: Memory-efficient RL, no critic

Use alternatives instead:

OpenRLHF: Multi-node distributed training, PPO/GRPO
TRL: Need multiple methods in one framework
DPO: Established baseline comparison

Common issues

Issue: Loss divergence

Reduce learning rate:

Reduce beta:

Issue: Model forgets capabilities

Add SFT regularization:

Issue: Poor preference separation

Increase beta and margin:

Issue: OOM during training

Reduce batch size:

Enable gradient checkpointing:

Advanced topics

Loss functions: See references/loss-functions.md for sigmoid vs hinge loss, mathematical formulations, and when to use each.

Hyperparameter tuning: See references/hyperparameters.md for beta, gamma, learning rate selection guide, and model-size-specific recommendations.

Dataset preparation: See references/datasets.md for preference data formats, quality filtering, and custom dataset creation.

Hardware requirements

GPU: NVIDIA A100/H100 recommended
VRAM:
- 7B model: 1× A100 40GB (DeepSpeed ZeRO-3)
- 8B model: 2× A100 40GB
- 70B model: 8× A100 80GB
Single-node: DeepSpeed ZeRO-3 sufficient
Mixed precision: BF16 recommended

Memory optimization:

DeepSpeed ZeRO-3 (default config)
Gradient checkpointing
Flash Attention 2

Resources

Paper: https://arxiv.org/abs/2405.14734 (NeurIPS 2024)
GitHub: https://github.com/princeton-nlp/SimPO
Models: https://huggingface.co/princeton-nlp
Alignment Handbook: https://github.com/huggingface/alignment-handbook