Direct Preference Optimization (DPO)

Help the user set up and run DPO training using the Tinker API.

Step 1: Understand the request

Ask the user (if not already specified):

• Model: Which model to train (e.g., meta-llama/Llama-3.2-1B, Qwen/Qwen3-8B)
• Dataset: Which preference dataset — built-in (HHH, HelpSteer3, UltraFeedback) or custom
• Starting checkpoint: Train from base model or from an SFT checkpoint

Step 2: Reference existing recipes

Read these files for patterns:

• tinker_cookbook/recipes/preference/dpo/train.py — DPO CLI with built-in datasets
• tinker_cookbook/preference/train_dpo.py — Core DPO training loop
• tinker_cookbook/preference/dpo_datasets.py — DPO dataset builders
• tinker_cookbook/recipes/preference/datasets.py — HHH, HelpSteer3, UltraFeedback builders
• docs/preferences/dpo-guide.mdx — DPO guide

Step 3: Configure the training run

Key Parameters

• dpo_beta: Controls how much the model deviates from reference. Start with 0.1 (recommended default).
  • Lower beta = more deviation from reference (more aggressive optimization)
  • Higher beta = stays closer to reference (more conservative)
• learning_rate: Typically 1e-5 for DPO (lower than SFT)
• lr_schedule: "linear" decay is standard
• batch_size: Number of tokens per batch (default: 256)
• max_length: Maximum sequence length (default: 8192)
• reference_model_name: Explicit reference model (defaults to the base model)

Preference Datasets

Built-in:

• "hhh" — Anthropic HHH (Helpful, Harmless, Honest) comparisons
• "helpsteer3" — NVIDIA HelpSteer3 preference data
• "ultrafeedback" — UltraFeedback preference data

Custom: Create a ComparisonBuilder that yields (chosen, rejected) conversation pairs. See recipes/preference/datasets.py for examples.

Dataset Construction

from tinker_cookbook.preference.dpo_datasets import DPODatasetBuilderFromComparisons
from tinker_cookbook.recipes.preference.datasets import HHHComparisonBuilder

common_config = ChatDatasetBuilderCommonConfig(
    model_name_for_tokenizer=model_name,
    renderer_name=renderer_name,
    max_length=8192,
    batch_size=256,
)
dataset = DPODatasetBuilderFromComparisons(
    common_config=common_config,
    comparison_builder=HHHComparisonBuilder(),
)

Step 4: Write the training script

Follow the pattern from recipes/preference/dpo/train.py:

import chz
from tinker_cookbook import checkpoint_utils, cli_utils
from tinker_cookbook.preference import train_dpo
from tinker_cookbook.preference.dpo_datasets import DPODatasetBuilderFromComparisons
from tinker_cookbook.supervised.types import ChatDatasetBuilderCommonConfig

config = train_dpo.Config(
    log_path="/tmp/tinker-examples/dpo/my_run",
    model_name="meta-llama/Llama-3.2-1B",
    renderer_name=renderer_name,
    dataset_builder=dataset,
    learning_rate=1e-5,
    lr_schedule="linear",
    dpo_beta=0.1,
    reference_model_name=None,  # Uses base model as reference
    load_checkpoint_path=None,  # Or path to SFT checkpoint
)

train_dpo.main(config)

Step 5: Run

# Basic DPO with HHH dataset
python -m tinker_cookbook.recipes.preference.dpo.train dataset=hhh

# With different model and dataset
python -m tinker_cookbook.recipes.preference.dpo.train \
    model_name=meta-llama/Llama-3.1-8B \
    dataset=ultrafeedback \
    dpo_beta=0.1 \
    learning_rate=1e-5

# From an SFT checkpoint
python -m tinker_cookbook.recipes.preference.dpo.train \
    load_checkpoint_path=/tmp/tinker-examples/sft/checkpoint_100

Step 6: Add tests

If you created a new DPO recipe, add a smoke test:

# tests/recipes/test_recipe_<name>.py
import pytest
from tests.helpers import run_recipe

@pytest.mark.integration
def test_<recipe_name>():
    run_recipe(
        "tinker_cookbook.recipes.<recipe_name>.train",
        ["behavior_if_log_dir_exists=delete"],
    )

run_recipe() automatically passes max_steps=2 so the recipe runs 2 training steps and exits. See tests/recipes/test_recipe_dpo.py for the existing example.

Step 7: Export weights (optional)

After DPO training, export weights using the tinker_cookbook.weights API:

from tinker_cookbook import weights

adapter_dir = weights.download(tinker_path="tinker://run-id/sampler_weights/final", output_dir="./adapter")
weights.build_hf_model(base_model="meta-llama/Llama-3.2-1B", adapter_path=adapter_dir, output_path="./model")
weights.publish_to_hf_hub(model_path="./model", repo_id="user/my-dpo-model")

Common pitfalls

• Start with dpo_beta=0.1 — this is well-tested. Tune from there.
• DPO LR should be lower than SFT (1e-5 vs 2e-4)
• DPO works best when starting from an SFT checkpoint, not a raw base model
• Reference model defaults to the base model — set reference_model_name explicitly if you want a different reference
• Preference data quality matters more than quantity — ensure chosen/rejected pairs have clear quality differences