What Is Model Distillation and Why Smaller Models Keep Improving

Every few months, a small AI model shows up that performs surprisingly close to a model many times its size on common tasks. Distillation is usually a big part of why, and the concept behind it is simpler than the results suggest.

The teacher-student idea

Training a model from scratch on raw internet text is one way to build a capable system, it's how the largest "teacher" models are generally built. Distillation is a different, second step: taking a large, already-trained teacher model and using its outputs, and sometimes its internal probability distributions rather than just its final answers, to train a smaller "student" model. The student isn't just learning from raw data anymore, it's learning to imitate a much more capable model's behavior, which turns out to be a more efficient way to transfer capability than training small from scratch.

Why the student can end up so capable

The teacher model has effectively already done the hard work of figuring out which patterns in language and reasoning actually matter. Learning to imitate that filtered, higher-quality signal is an easier problem than learning everything from unstructured raw data with no guidance. A distilled model trained this way on a narrower set of tasks can sometimes match or beat a much larger general-purpose model specifically on that narrower slice, even though it would lose badly on the full breadth of what the larger model can do.

Why this matters practically

Running a smaller model costs less, responds faster, and can run on far more modest hardware, including in some cases directly on a phone or laptop. Distillation is one of the main reasons the practical gap between "what fits on your device" and "what's genuinely useful" has been shrinking steadily, rather than every meaningful capability requiring a trip to a massive cloud-hosted model.

The honest limitation

A distilled model is bounded by what its teacher actually knew and how well the distillation process captured that knowledge, it's not going to spontaneously exceed its teacher's ceiling on the tasks the teacher was actually good at. And distillation tends to work best on well-defined tasks, summarization, classification, specific kinds of question answering, rather than transferring the full open-ended generality of a frontier model into something a fraction of the size. The trade-off is real: narrower and cheaper in exchange for giving up some general-purpose breadth, which is exactly the right trade for a huge number of practical applications that never needed that breadth in the first place.

Why it keeps happening every generation

As teacher models get more capable, the ceiling for what a distilled student can learn rises along with them, so this isn't a technique that plateaus, it improves every time the underlying frontier models improve. That's the actual mechanism behind smaller models seeming to get better every few months without needing a fundamentally new training approach each time.