Knowledge Distillation in LLMs: Transferring Intelligence into Smaller Models

July 24, 2026 · 16 min read · LLMs · distillation · model compression · deep learning

Train a small model on a dataset and it does worse than a large model trained on the same data. Train the same small model on the large model’s predictions instead of the original labels, and it often does much better, even though it has seen no new information about the world.

The reason is that a trained model’s outputs carry more information than the labels it was trained on. A label says “the answer is cat”. A model’s output distribution says “probably cat, somewhat plausibly lynx, very unlikely truck”. That relative structure, which Hinton, Vinyals and Dean (2015) called dark knowledge, describes how the teacher generalizes. Distillation is the craft of transferring it.

For LLMs, distillation has moved from a compression trick to a central part of how models are built. Small open models in the 1B–10B range from several major labs are now trained partly or entirely on the outputs of larger ones. Understanding why that works, and where it breaks, matters for anyone choosing or building models for constrained deployment.

Why compress at all

The motivation is the economics of serving. Generating a token costs about $2N$ FLOPs for an $N$ -parameter model, and, more importantly, requires reading every weight from memory. A 70B model in 16-bit precision is about 140 GB of weights. It does not fit on one 80 GB GPU, and each decoded token needs 140 GB of memory reads. An 8B model is about 16 GB, fits comfortably, and decodes roughly an order of magnitude more cheaply.

Serving cost dominates a successful model’s lifetime cost, so there is strong pressure to get large-model quality from small-model compute. There are three main routes:

Train a small model better (distillation, better data).
Remove parts of a large model (pruning).
Store a model in fewer bits (quantization).

They are complementary rather than competing, and production pipelines often use all three.

The teacher–student setup

A teacher is a large, capable, already-trained model. It is frozen and used only for forward passes. A student is a smaller model being trained. The student’s objective includes, in some form, matching the teacher.

Distillation is defined by what the student matches:

Signal	What the student learns from	Requires
Output distribution (logits)	Teacher’s full next-token probabilities	White-box teacher, shared or mappable vocabulary
Intermediate features	Hidden states, attention maps	White-box teacher, a way to align layers and widths
Generated sequences	Text sampled from the teacher	Only API access to the teacher

Soft targets and temperature

Consider a classifier with logits $\mathbf{z}$ . The usual softmax gives probabilities, but a trained model is usually very confident. The correct class might get 0.999 and everything else shares 0.001. The informative relationships between wrong classes are crushed into tiny numbers that contribute almost nothing to a loss.

Hinton et al. introduced a temperature $T$ to soften the distribution:

p_i^{(T)} = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}.

At $T = 1$ this is the ordinary softmax. As $T$ grows, the distribution flattens and the differences between small probabilities become visible. A concrete example with three classes and teacher logits $(10, 5, 1)$ :

Temperature	p(cat)	p(lynx)	p(truck)
1	0.993	0.007	0.000
4	0.718	0.206	0.076

At $T = 1$ the teacher just says “cat”. At $T = 4$ it also says “lynx is a much more reasonable mistake than truck”, which is information about the shape of the problem that a one-hot label never contains.

Bar chart of teacher probabilities for cat, lynx and truck at temperatures 1, 2 and 4

Figure 1. Raising the temperature reveals the teacher’s ranking of the wrong answers: lynx is a far more reasonable mistake than truck.

Why should this help the student? Each soft target provides far more bits of supervision per example than a hard label, and the supervision has lower variance, because it is the teacher’s smoothed judgement rather than one noisy sample. The student learns not only what to predict but how the teacher’s uncertainty is structured, which acts as a strong regularizer.

The distillation loss

The student is trained on a weighted combination of two terms:

\mathcal{L} = \alpha \, T^2 \, \mathrm{KL}\!\left(p_{\text{teacher}}^{(T)} \,\Big\|\, p_{\text{student}}^{(T)}\right) + (1 - \alpha)\, \mathrm{CE}\!\left(y,\, p_{\text{student}}^{(1)}\right).

The first term matches the student’s softened distribution to the teacher’s. The second is the ordinary cross-entropy on the ground-truth label at $T = 1$ .

KL divergence versus cross-entropy. The KL term is

\mathrm{KL}(p \,\|\, q) = \sum_i p_i \log \frac{p_i}{q_i} = \underbrace{-\sum_i p_i \log q_i}_{\text{cross-entropy } H(p, q)} - \underbrace{\left(-\sum_i p_i \log p_i\right)}_{\text{entropy } H(p)}.

The teacher’s entropy $H(p)$ does not depend on the student, so minimizing the KL and minimizing the cross-entropy against soft targets give identical gradients. KL is usually reported because it is zero when the distributions match, which makes it easier to read.

Why the $T^2$ factor. The gradient of the soft term with respect to a student logit $z_i$ is

\frac{\partial\, \mathrm{KL}}{\partial z_i} = \frac{1}{T}\left(q_i^{(T)} - p_i^{(T)}\right),

where $q$ is the student and $p$ the teacher. Hinton et al. showed that at high temperature, assuming zero-mean logits, this approaches $\frac{1}{N T^2}(z_i - v_i)$ , with $v$ the teacher logits. The gradient magnitude shrinks as $1/T^2$ . Multiplying the loss by $T^2$ keeps the soft term’s contribution roughly constant as $T$ changes, so $\alpha$ and $T$ can be tuned independently. It also shows that at high temperature, distillation becomes logit matching, essentially a squared error between logits.

import torch.nn.functional as F

def distillation_loss(student_logits, teacher_logits, labels, T=2.0, alpha=0.5):
    """Logit distillation for classification or per-token LM training.
    logits: (N, vocab); labels: (N,) with -100 for positions to ignore."""
    soft_student = F.log_softmax(student_logits / T, dim=-1)
    soft_teacher = F.softmax(teacher_logits / T, dim=-1)
    kd = F.kl_div(soft_student, soft_teacher, reduction="batchmean") * (T * T)
    ce = F.cross_entropy(student_logits, labels, ignore_index=-100)
    return alpha * kd + (1 - alpha) * ce

For language models the same loss is applied at every token position: the student’s next-token distribution is matched to the teacher’s at each step of the training text. Typical temperatures for LLM distillation are lower than in the original image work, often between 1 and 2, because the vocabulary is huge and a high temperature spreads mass over tens of thousands of irrelevant tokens.

Distilling generative models: which KL?

For classifiers, matching the teacher is unambiguous. For generative LLMs, a subtle choice appears: the direction of the KL divergence.

Forward KL, $\mathrm{KL}(p_{\text{teacher}} \,\|\, q_{\text{student}})$ , is what the loss above uses. It heavily penalizes the student for assigning low probability anywhere the teacher assigns high probability. This is mode-covering: the student is pushed to spread its mass over everything the teacher considers plausible. If the student lacks the capacity to represent the teacher’s distribution precisely, it compromises by being too broad, and when sampled it puts probability on regions between the teacher’s modes, which in text means incoherent or low-quality outputs.

Reverse KL, $\mathrm{KL}(q_{\text{student}} \,\|\, p_{\text{teacher}})$ , penalizes the student for putting mass where the teacher has little. This is mode-seeking: a limited student commits to the teacher’s main modes and ignores the rest. For generation, where we sample from the student, that is often what we want. Gu et al. (MiniLLM, 2023) argued this case and trained students with reverse KL using policy-gradient methods, reporting more precise, higher-quality responses than forward-KL distillation.

A two-mode teacher distribution with a single Gaussian fitted by forward KL, which spans both modes, and by reverse KL, which sits on one mode

Figure 2. Fitting a single Gaussian to a two-mode teacher. Forward KL spreads the student across both modes and puts mass between them; reverse KL commits to one mode. Both fits are computed, not sketched.

The catch is that reverse KL is an expectation under the student’s distribution, so computing it requires sampling from the student. That leads to the next idea.

On-policy distillation

Standard distillation trains the student on fixed text: either real data with teacher labels at each position, or text generated by the teacher. In both cases, the student is supervised on prefixes it would not necessarily produce itself. At inference time it is run on its own outputs, and it has never been taught what to do after its own characteristic mistakes. This is the same exposure bias that affects teacher-forced pretraining.

On-policy distillation fixes the mismatch. The student generates sequences, and the teacher scores every token of the student’s sequence, providing its full distribution at each position. Agarwal et al. (Generalized Knowledge Distillation, GKD, 2023) framed it as a mix of fixed and student-generated data, with a flexible divergence (including a generalized Jensen–Shannon divergence that interpolates between forward and reverse KL).

def on_policy_distill_step(student, teacher, prompts, optimizer, T=1.0):
    with torch.no_grad():
        samples = student.generate(prompts, do_sample=True)     # the student's own text
        teacher_logits = teacher(samples).logits
    student_logits = student(samples).logits
    # per-token reverse KL: E_{y ~ student}[log q(y) - log p(y)], over the full vocab
    log_q = F.log_softmax(student_logits / T, dim=-1)
    log_p = F.log_softmax(teacher_logits / T, dim=-1)
    per_token = (log_q.exp() * (log_q - log_p)).sum(-1)          # (batch, seq)
    mask = completion_mask(samples, prompts)                      # 1 on generated tokens only
    loss = (per_token * mask).sum() / mask.sum()
    optimizer.zero_grad(); loss.backward(); optimizer.step()

Compared with reinforcement learning, which gives one scalar reward per sequence, on-policy distillation gives a dense signal: a full distribution at every token. That makes it far more sample-efficient, and it is increasingly used as a post-training stage for small models, for example to transfer a large model’s reasoning behaviour.

Sequence-level and black-box distillation

The teacher’s logits are not always available. A closed API returns text, not distributions. Even with an open teacher, storing a full 128K-dimensional distribution for every token of a trillion-token corpus is impossible. Two related approaches work from text alone.

Sequence-level distillation (Kim and Rush, 2016) trains the student with ordinary cross-entropy on sequences generated by the teacher, rather than matching per-token distributions. The intuition is that the teacher’s outputs are a “cleaner”, lower-entropy version of the data distribution: more consistent, less noisy, and easier for a small model to fit.

Synthetic-data distillation is the same idea at scale, and it is the dominant form of distillation in the LLM ecosystem. A strong model generates instruction–response pairs, explanations, or step-by-step reasoning traces, and a smaller model is fine-tuned on them. Stanford’s Alpaca (2023) fine-tuned LLaMA-7B on 52K instructions generated by an OpenAI model. Microsoft’s Orca (2023) argued that students learn much more from explanation traces, the teacher’s step-by-step reasoning, than from final answers alone. DeepSeek-R1 (2025) distilled its reasoning model into smaller Qwen and Llama models with plain supervised fine-tuning on roughly 800K curated samples, and reported that this produced better small reasoning models than applying large-scale reinforcement learning to the small models directly.

The weakness of the purely black-box form is that each training example carries one sampled answer, not the teacher’s uncertainty. Much of the dark knowledge is lost. When logits are available, a common compromise is to store only the top- $k$ logits per position (for example $k$ between 20 and 100) and renormalize, which keeps most of the probability mass at a fraction of the storage.

Feature-based distillation

Logits are only the final layer. A teacher’s intermediate representations carry information too, and matching them gives the student guidance about how to compute, not just what to output.

FitNets (Romero et al., 2015) introduced “hints”: a student layer is trained to predict a teacher layer’s activations through a learned projection $W_r$ , since the widths usually differ:

\mathcal{L}_{\text{hint}} = \left\lVert\, W_r\, \mathbf{h}^{\text{student}}_{\ell'} - \mathbf{h}^{\text{teacher}}_{\ell} \,\right\rVert_2^2.

For transformers, DistilBERT (Sanh et al., 2019) halved BERT’s depth, initialized the student from alternating teacher layers, and combined soft-target distillation with a cosine loss aligning hidden states. It was 40% smaller and 60% faster while retaining about 97% of BERT’s language-understanding performance. TinyBERT (Jiao et al., 2020) also matched attention matrices and embedding outputs layer by layer.

Feature distillation raises design questions that logit distillation avoids: which student layer should match which teacher layer when depths differ, how to project between widths, and how to weight many loss terms. It is most valuable when the student is initialized from or structurally close to the teacher, which is exactly the situation after pruning.

Distillation, pruning and quantization together

The three compression families attack different costs, and they compose.

Pruning removes parameters from a trained model. Unstructured pruning zeroes individual weights. It can remove a large fraction with little loss, but sparse matrices rarely run faster on GPUs without special hardware support. Structured pruning removes whole units, such as attention heads, FFN channels, entire layers, or embedding width, giving a genuinely smaller dense model that runs faster everywhere. Structured pruning damages the model, though, and the damage must be repaired by further training.

That repair training is where distillation comes in. The unpruned model is the natural teacher for its own pruned version. NVIDIA’s Minitron work (2024) pruned Nemotron-15B to 8B and 4B by width and depth, then retrained with distillation from the original model, and reported needing up to 40× fewer training tokens than training models of the same size from scratch. Meta described a similar recipe for Llama 3.2 1B and 3B: structured pruning from Llama 3.1 8B, then pretraining that used logits from the 8B and 70B models as token-level targets. Google trained Gemma 2’s 2B and 9B models with distillation from a larger teacher instead of plain next-token prediction.

Quantization stores weights (and sometimes activations or the KV cache) in fewer bits, for example 8-bit integers, FP8, or 4-bit formats. Post-training quantization needs no retraining and is very effective down to 8 bits and often to 4 bits for weights. At lower precision, or for sensitive models, quantization-aware training simulates quantization during fine-tuning so the model can adapt. Distillation helps here too: using the full-precision model as the teacher for its quantized self is a strong objective, because the teacher’s distribution is exactly what the quantized model should preserve.

A typical production pipeline therefore looks like: train or choose a large teacher, prune it to the target architecture, recover quality with distillation, then quantize for deployment, with each step evaluated against the teacher.

Applications

On-device and edge models. Phone, laptop and browser models in the 1B–4B range need distillation to be useful within tight memory and power budgets.
Cost-efficient serving. A distilled 8B model handling the bulk of routine traffic, with a large model reserved for hard cases, can cut serving costs dramatically.
Task-specific specialists. A large model labels or generates data for a narrow task, such as classification, extraction, routing or SQL generation, and a small model is distilled on it. For narrow tasks, small students frequently match the teacher.
Reasoning transfer. Long chain-of-thought traces from a reasoning model can be distilled into smaller models, which then reason far better than their size would suggest.
Speculative decoding. A small draft model proposes several tokens that the large model verifies in one forward pass. The speed-up depends on how often the draft agrees with the target, so drafts are often distilled from the target model to maximize agreement.
Retrieval components. Cross-encoder rerankers are routinely distilled into faster bi-encoders or smaller cross-encoders, a familiar technique in search systems.

Limitations and trade-offs

The capacity gap. A student cannot represent what its parameters cannot hold. Counter-intuitively, a much larger teacher is not always a better teacher. Cho and Hariharan (2019) and Mirzadeh et al. (2020) found that students distilled from very large teachers can do worse than students distilled from medium ones, because the larger teacher’s distribution is too complex to imitate. Mirzadeh et al. proposed an intermediate “teacher assistant” model to bridge the gap.

Imitation is not capability. Gudibande et al. (“The False Promise of Imitating Proprietary LLMs”, 2023) fine-tuned open models on outputs from a stronger chat model. Crowd raters judged the imitations close to the teacher, but targeted benchmarks showed the gap in factual knowledge and reasoning was barely reduced. The students learned the teacher’s style: its confident tone and formatting. Black-box distillation on a small amount of data transfers form much more easily than knowledge, which is mostly acquired in pretraining.

Students don’t fully match teachers, even when they could. Stanton et al. (“Does Knowledge Distillation Really Work?”, 2021) found that students often fail to match their teacher’s predictions closely even when they have enough capacity, largely because the optimization is hard. Distillation improves generalization, but “the student becomes a copy of the teacher” is not what usually happens.

Inherited errors and biases. The student learns the teacher’s mistakes, blind spots and biases along with its strengths. Repeated rounds of training on model-generated data can also narrow output diversity over time.

Vocabulary and architecture coupling. Logit distillation assumes the teacher and student share a tokenizer, so that position $i$ in both distributions is the same token. Across tokenizers, distributions must be aligned approximately (methods such as universal logit distillation try to do this), or the transfer falls back to text.

Teacher cost. Distillation makes inference cheaper, not training. Every training token requires a teacher forward pass, often of a model ten times larger, or a huge precomputed logit store. On-policy distillation adds student generation on top.

Licensing. Many commercial model licences and API terms restrict using outputs to train competing models. This is a practical constraint on synthetic-data distillation, separate from the technical questions.

When not to distill. If the task needs broad, up-to-date knowledge, a smaller model plus retrieval may beat a distilled model that tries to memorize. If the latency budget allows it, a quantized larger model sometimes outperforms a distilled smaller one at similar memory cost, since quantization keeps the teacher’s parameters (and knowledge) mostly intact. Comparing both against the same evaluation is worth the effort.

Clarifications

Training on a teacher’s generated answers is only the black-box special case of distillation, and it discards most of the signal. What defines distillation is learning from the teacher’s distribution.

A higher temperature is not always better. It trades signal about rare alternatives against noise from irrelevant tokens, and with LLM vocabularies moderate temperatures usually work best. Nor is the largest available teacher always the best one, for the capacity-gap reasons above.

Finally, a distilled model is not a compressed copy of its teacher. It is a separate model trained with a better objective than the raw data provides, with its own capacity limits and failure modes.

Open questions

How much of a large model’s capability can be transferred at all, and how much depends on scale, is still unclear. Current results suggest that style and narrow skills transfer easily, broad factual knowledge transfers poorly, and reasoning sits somewhere in between.