Orchestra Research

Guide language model training with SimPO using pairs of preferred and rejected answers. Covers configuration, learning rate tuning, and memory troubleshooting for preference alignment, which adjusts model responses toward human preferences without a separate reference model.

• Training on preferred and rejected answer pairs
• Exploring an alternative to DPO without a reference model
• Tuning preference training for math and code tasks

Guide language model training with TRL, from learning to follow instructions to matching human preferences. Covers method selection, training data, reward models that score responses, evaluation, and adjustments for memory limits or unstable training.

• Train a model on instruction and response examples
• Align responses using preferred and rejected answer pairs
• Build a training pipeline using human feedback