Model Distillation

Distillation transfers capability from a large teacher model to a smaller student: train the student to mimic the teacher's outputs (or output distributions) on a representative dataset. The result runs cheaper and faster while retaining much of the teacher's behavior.

Overview

The core idea: the teacher's soft outputs (probability distributions, reasoning traces) contain more information than hard labels — "dark knowledge" about similarities between classes and plausible alternatives. The student trains on teacher-generated data, often with a loss combining teacher imitation and ground-truth labels. Modern practice extends this to distilling reasoning: training students on teachers' chain-of-thought traces to transfer reasoning ability, not just answers.

When to use

• Deploying big-model quality on limited hardware or budget.
• Reducing latency: smaller models respond faster.
• Creating specialized small models from a general large teacher.
• Offline/precompute scenarios where teacher cost is one-time and student cost is per-query.

Core concepts

• Teacher-student setup: the teacher generates training signal; the student learns from it. The teacher should be strong on the target task — distilling a weak teacher teaches weakness.
• Soft targets: training on the teacher's output distribution, not just its top answer. Temperature softens the distribution to expose more dark knowledge.
• Distillation data: representative of deployment inputs. Coverage matters more than volume — the student only learns what it sees the teacher do.
• Reasoning distillation: including the teacher's reasoning traces in training data transfers problem-solving patterns, not just final answers. The current best practice for reasoning tasks.
• Loss design: balancing imitation loss (match the teacher) with task loss (match ground truth). Pure imitation inherits teacher errors; pure task loss wastes the teacher.
• Capacity gap: students too small can't absorb the teacher. Match student size to task complexity; sometimes a medium student beats a tiny one at similar cost.

Practical workflow

1. Define the target behavior and build an eval set measuring it.
2. Choose the teacher (strong on the task) and student (fits the deployment budget) sizes.
3. Generate distillation data: diverse, representative inputs with teacher outputs (and reasoning traces for reasoning tasks).
4. Filter teacher outputs for quality — distilling teacher mistakes teaches mistakes. Keep ground-truth labels where available.
5. Train the student with combined imitation + task loss; tune temperature and loss weights.
6. Evaluate student vs. teacher vs. task baseline on the eval set; check where the student still lags and target those with more data.

Distillation checklist:
[ ] Eval set for the target behavior
[ ] Teacher strong on the task; student fits budget
[ ] Distillation data covers deployment distribution
[ ] Teacher outputs quality-filtered
[ ] Loss balances imitation + ground truth
[ ] Student evaluated vs teacher; gaps analyzed

Common pitfalls

• Distilling teacher errors: unfiltered teacher outputs bake mistakes into the student. Filter aggressively.
• Unrepresentative data: distilling on data unlike deployment. The student learns the wrong distribution.
• Answers without reasoning: distilling final answers only for reasoning tasks. Include traces — that's where the capability lives.
• Student too small: expecting a tiny model to absorb complex reasoning. Respect the capacity gap.
• No ground-truth anchoring: pure imitation with no task loss drifts from correctness. Anchor with labels where they exist.
• Skipping the eval: assuming the student retained "most" capability. Measure it — the gap is often task-specific.