AI engineering · 30 of 42

Distillation

A small model taught by a big one

Scroll

A small model taught by a big one

Run a large model over a great many prompts, keep its answers, and train a small model on that as its training set. The student learns to imitate the teacher on the kind of work the prompts represent.

The result is a model that is much cheaper to run and close to the teacher within that range — which is where most of the small fast models you can actually afford come from, including the draft model in concept 4.

Close is not equal, and the gap widens outside the distribution it was trained on. The student also inherits the teacher's blind spots and confidently reproduces them, without the teacher's ability to recover.

Adaptation
A SMALL MODEL TAUGHT BY A BIG ONE teacher large, slow thousands of prompts and its answers to them trains on student small, fast the teacher's outputs become the training set close, not equal cheap to run, cheap to serve This is where the draft model in speculative decoding comes from, and most of the small fast models you can actually afford to run. The student inherits the teacher's blind spots along with its skills.
A large teacher answering many prompts, its answers becoming the training set for a small student.