Model Distillation Techniques: Compressing Large Models for Production

A 175-billion-parameter teacher and a 1.3-billion-parameter student can answer the same customer support ticket with indistinguishable quality, and the student costs roughly one hundredth as much to serve. That gap is the entire commercial argument for knowledge distillation, and it is why nearly every production language model shipped after 2023 is, in some sense, a distilled artifact rather than a from-scratch pretraining run.

The Hinton-Vinyals-Dean Formulation

The modern framing of distillation comes from Geoffrey Hinton, Oriol Vinyals, and Jeff Dean in their 2015 paper Distilling the Knowledge in a Neural Network (arXiv:1503.02531). Their insight was that the softmax outputs of a large model carry far more information than the argmax label. When a strong image classifier predicts 'cat' with probability 0.92, the remaining 0.08 mass distributed over 'lynx', 'fox', and 'dog' encodes the teacher's learned similarity structure. Training a student to match those soft targets, rather than just the hard label, transfers that structure.

The mechanism is temperature scaling. Both teacher and student logits are divided by a temperature T (commonly between 2 and 20) before the softmax. Higher temperatures flatten the distribution and expose the smaller probabilities that carry the dark knowledge. The student is trained with a weighted sum of two losses: a Kullback-Leibler divergence against the softened teacher distribution and a standard cross-entropy against the ground-truth label. A typical mixing coefficient is alpha around 0.5 to 0.9 in favor of the distillation term.

import torch.nn.functional as F

def distillation_loss(student_logits, teacher_logits, labels, T=4.0, alpha=0.7):
    soft_student = F.log_softmax(student_logits / T, dim=-1)
    soft_teacher = F.softmax(teacher_logits / T, dim=-1)
    kd = F.kl_div(soft_student, soft_teacher, reduction="batchmean") * (T * T)
    ce = F.cross_entropy(student_logits, labels)
    return alpha * kd + (1.0 - alpha) * ce

Three Families: Response, Feature, and Relation

Practitioners broadly group distillation into three families. Response-based distillation matches output logits or probabilities, which is what the original Hinton paper proposed. Feature-based distillation, popularized by FitNets in 2014, aligns intermediate hidden states between teacher and student, typically through a learned linear projection because the dimensions usually differ. Relation-based distillation, with methods like Relational Knowledge Distillation by Park et al. in 2019, matches pairwise or higher-order structure across a batch rather than per-example values.

The choice matters more than people admit. Response-based methods are simple and architecture-agnostic but tend to plateau quickly for hard tasks. Feature-based methods recover more of the teacher's behavior but couple the student to internal teacher geometry. Relation-based methods shine when absolute outputs are noisy but relative orderings are reliable, which is the case for retrieval embeddings and ranking models.

BERT Era: DistilBERT, TinyBERT, MobileBERT, MiniLM

The encoder era produced the cleanest case studies. DistilBERT, introduced by Victor Sanh and colleagues at Hugging Face in 2019 (arXiv:1910.01108), removed half the layers of BERT-base, initialized the student from alternating teacher layers, and trained with a triple loss combining masked language modeling, soft-target distillation, and a cosine similarity term on hidden states. It retained roughly 97 percent of GLUE score at 60 percent of the size and 60 percent of the latency.

TinyBERT from Huawei Noah's Ark Lab added attention-matrix distillation, training the student's attention heads to mimic the teacher's softmax attention distributions layer by layer. MobileBERT from Google reshaped the architecture with bottleneck layers so a deep but thin student could match a wide teacher head-for-head. MiniLM, from Microsoft Research, distilled only the last layer's self-attention via deep self-attention distillation, which proved that you do not need full layer-wise alignment to get strong students if you pick the right signal.

White-Box vs Black-Box LLM Distillation

Once teachers stopped being open-weight, the distillation literature bifurcated. White-box distillation assumes access to teacher logits, hidden states, and gradients. Black-box distillation assumes only sampled completions through an API. The Alpaca and Vicuna line in 2023 were canonical black-box runs: prompt GPT-3.5 or GPT-4 for 52,000 to 70,000 instruction-response pairs, then supervised fine-tune a LLaMA student on those strings. It works, but the student sees only one sample per prompt and inherits the teacher's stylistic tics rather than its underlying probability mass.

White-box distillation, when feasible, dominates on data efficiency. Sequence-level distillation by Kim and Rush in 2016 showed that matching token-level distributions across a generated sequence outperforms training on hard-decoded outputs alone. For chat models, on-policy distillation has become the new default: the student generates a response, the teacher scores or relabels it, and the student is updated against the teacher's distribution over the student's own samples. This closes the train-test distribution gap that haunts naive supervised distillation.

How Llama 3.2 Small and Phi-3 Were Built

Meta's Llama 3.2 1B and 3B models, released in September 2024, were explicitly described as pruned-and-distilled from Llama 3.1 8B and 70B. The pipeline applied structured pruning to identify redundant attention heads and MLP channels, then ran continued pretraining with logit-level distillation from the larger teacher. The technical report claims that distillation alone recovered most of the capability lost during pruning at a fraction of the token budget of a from-scratch run.

Microsoft's Phi-3 series took a different but compatible route. The Phi-3 technical report (arXiv:2404.14219) emphasized textbook-quality synthetic data generated and filtered by stronger models, which is distillation reframed as data curation. Phi-3-mini, at 3.8 billion parameters, was trained on 3.3 trillion tokens of this curated mix and reaches MMLU scores in the high 60s, competitive with models more than ten times its size. The lesson is that the teacher does not always appear in the loss; sometimes it appears in the dataset.

When You Should Distill, And When You Should Not

• Use response-based distillation with temperature 2-4 when teacher and student share a tokenizer and you need a quick win on classification, retrieval, or short-form generation.
• Prefer on-policy or sequence-level distillation for open-ended chat, because supervised distillation on teacher samples causes exposure bias and degrades multi-turn coherence.
• Stack distillation with INT8 or INT4 quantization for edge deployment; the order matters, distill first in FP16 or BF16, then quantize, because quantizing the teacher before distilling discards the soft-target nuance you wanted to transfer.
• Skip distillation entirely if your downstream metric is dominated by retrieval or tool use; a smaller model with a better RAG index almost always beats a distilled monolith.
• Watch for tokenizer mismatch in black-box distillation; if teacher and student tokenize differently, you cannot use logit-level signals at all and you should fall back to sample-based supervision.
• Budget at least 5-15 percent of the teacher's pretraining FLOPs for the distillation run if you want production-grade quality, not the often-quoted 1 percent figure that only applies to narrow benchmarks.
• Always evaluate on out-of-distribution prompts the teacher has never seen, because students that match teacher logits on the training distribution can collapse on novel inputs.

Evaluation Pitfalls and Dataset Distillation

The most common failure mode I see in deployed distilled models is benchmark overfitting. A student trained to match teacher outputs on Alpaca-Eval or MT-Bench will pass those benchmarks while regressing on the long tail of production traffic, because the teacher itself was sampled at low temperature and the student inherited the resulting mode collapse. The fix is to evaluate on a held-out sample of real production prompts with human raters or a separate, stronger judge model that did not provide the training signal.

Dataset distillation, introduced by Wang et al. in 2018 and recently scaled up by methods like MTT and DC, takes the orthogonal approach of compressing the training set rather than the model. You synthesize a small set of images or token sequences such that a model trained on that synthetic set matches one trained on the full dataset. For language, this is still nascent, but it shows promise for privacy-preserving fine-tuning where the synthetic data carries gradient information rather than raw user content.

Distillation is not compression of a model; it is compression of a probability distribution. Forget that, and your student will memorize your benchmarks while failing in the wild.

Two practical heuristics close this out. First, instrument the KL divergence between student and teacher on a fixed evaluation set during training, not just the loss, because loss can drop while KL on rare tokens explodes. Second, hold back at least one capability the student is allowed to lose, such as multilingual coverage or long-context reasoning, and measure it explicitly; without that, every distillation run silently trades capabilities you did not realize you cared about.