Fine-Tuning LLMs

Fine-tuning adapts a pretrained model to your task, domain, or style. Done well, it's the highest-leverage way to get reliable specialized behavior; done poorly, it's an expensive way to degrade a good model. The data matters more than the hyperparameters.

Overview

The pipeline: curate high-quality demonstrations of desired behavior, choose the tuning method (full fine-tuning for maximum adaptation, LoRA/QLoRA for efficiency), train with careful hyperparameters, and evaluate against the base model on task-specific benchmarks. Most fine-tuning projects fail on data quality, not on training mechanics — a thousand excellent examples beat a million mediocre ones.

When to use

• The base model doesn't follow your domain's conventions, format, or style reliably.
• You need consistent structured outputs or specialized reasoning patterns.
• Prompting alone can't achieve the reliability you need.
• Adapting to proprietary knowledge or terminology (paired with retrieval for facts).

Core concepts

• SFT (supervised fine-tuning): training on input/output demonstrations of desired behavior. Teaches format, style, and task patterns — not new facts (that's what retrieval is for).
• Data quality: diverse, correct, well-formatted examples in the target distribution. Deduplicate, filter for quality, and match the length/style of real usage.
• LoRA/QLoRA: training low-rank adapters instead of all weights — 1% of parameters, near-full-tuning quality for most tasks, runnable on single GPUs. The default choice.
• Hyperparameters: learning rate (small — you're adjusting, not training from scratch), epochs (few — overfitting is the main risk), batch size, warmup. Start from known-good configs for your model size.
• Evaluation: task-specific benchmarks plus general capability checks (did tuning break general abilities?). Compare against the base model and against a prompt-only baseline.
• Catastrophic forgetting: aggressive tuning erases general capabilities. Mitigate with small learning rates, few epochs, and mixed-in general data.

Practical workflow

1. Define the target behavior precisely with examples; build an eval set before training anything.
2. Curate the dataset: quality over quantity, deduplicated, in the target distribution. Inspect samples manually.
3. Start with LoRA: known-good hyperparameters for your model size, 1–3 epochs.
4. Evaluate: task benchmark vs. base model vs. prompt-only baseline; check general capabilities didn't regress.
5. Iterate on data first (add failure-mode examples), hyperparameters second.
6. Version everything: dataset hash, config, base model, eval scores. Reproducibility is non-negotiable.

Fine-tune checklist:
[ ] Eval set built BEFORE training
[ ] Dataset inspected manually; deduplicated
[ ] Started with LoRA + known-good hyperparams
[ ] Compared: tuned vs base vs prompt-only
[ ] General capability regression checked
[ ] Data/config/model versioned

Common pitfalls

• Bad data, tuned confidently: noisy or misformatted training data teaches the model your mistakes. Curate ruthlessly.
• Too many epochs: overfitting to the training set's quirks. Few epochs, watch validation.
• No prompt-only baseline: fine-tuning when better prompting would've sufficed. Always compare.
• Forgetting check skipped: the tuned model aces your task and fails at everything else. Test general abilities.
• Teaching facts via SFT: fine-tuning memorizes poorly and hallucinates confidently. Use retrieval for knowledge, tuning for behavior.
• Unversioned runs: "the good model" with no record of data or config. Version everything or lose reproducibility.