What Is Model Distillation?

Model distillation, also called knowledge distillation, trains a small “student” model to imitate a larger “teacher” model. The student learns from the teacher’s answers rather than only from the original labelled data, so it can reach much of the teacher’s quality on a specific task while being far cheaper and faster to run. Distillation is how many small language models get their skills, and how companies move a working prototype from an expensive model to an affordable one.

How Distillation Works

The idea was set out in the 2015 paper Distilling the Knowledge in a Neural Network by Hinton, Vinyals and Dean. Running a large model, or an ensemble of models, is expensive. So you train one small model to reproduce what the large one does.

There are two main styles:

  1. Soft-label distillation. The teacher outputs a full probability distribution for each prediction, rather than only its top answer. Those “soft” probabilities show which wrong answers are nearly right, which carries more information than a single label. The student is trained to match them. This needs access to the teacher’s internal outputs, so it is used mainly with models you host.
  2. Output distillation (sequence-level). The teacher generates complete answers to a large set of prompts, and the student is fine-tuned on those prompt-answer pairs. This works even when the teacher is behind an API, and it is the common approach for language models today.

Reasoning can be distilled too. The DeepSeek-R1 paper describes using the reasoning patterns of a large model trained with reinforcement learning to improve smaller models.

When Distillation Pays Off

  • A task is narrow and stable: classifying, extracting, routing, rewriting in a house style.
  • Call volume is high enough that per-token cost or latency hurts.
  • The model must run on your own hardware or on a device.
  • You already have a large model doing the job well and a log of its real inputs.

It is a poor fit when the task changes every month, when inputs are too varied for a small model to cover, or when you have no way to check that the student is as good as the teacher.

Common Failure Modes

  • Copying the teacher’s mistakes. The student learns whatever the teacher gets wrong, often more confidently. Filter or correct teacher outputs before training.
  • Narrow coverage. The student only learns the situations the training prompts covered. Rare but important cases need to be added on purpose.
  • Licence breach. Some model providers’ terms restrict using their outputs to build competing models, and some open-weight licences add conditions on derived models. Check the teacher’s terms before generating training data.
  • No regression test. Without an evaluation set, a distilled model can look fine in a demo and fail on edge cases in production.

Distillation vs Neighbouring Techniques

TechniqueWhat changesMain gain
DistillationA new, smaller model is trained to copy a larger oneSmaller, faster model with similar task quality
QuantizationThe same model stored in fewer bitsLess memory, faster inference
Fine-tuningThe same model trained further on your examplesBetter fit to your task or format
Model routingNo training; each request goes to the cheapest model that can handle itLower average cost

Distillation is often combined with the others: a large model generates synthetic data, a small language model is fine-tuned on it, and the result is quantized for serving.

In Practice

Distillation comes after the task works with a large model and real traffic has been logged. A held-out set of real cases, with answers checked by people, sets the bar the student model must reach, and the teacher’s outputs on the remaining logs become its training data, provided the teacher’s license allows that use. The student then runs in shadow mode next to the teacher, and traffic moves to it one task type at a time as the comparison holds.

We apply distillation in LLMOps services, machine learning development and edge AI development projects where cost or latency must come down.

Back to Glossary

Let's talk about your project

We'll help you harness AI and ML technologies to create innovative software solutions

File requirements: pdf, doc, docx, rtf, ppt, pptx — max 15 MB

No file selected