AI engineering · 04 of 42

Speculative Decoding

Draft quickly, verify in batches

Scroll

Draft quickly, verify in batches

A small fast model guesses the next several tokens. The large model then checks all of those guesses in a single pass — and checking a batch is far cheaper than generating one at a time, because it parallelizes.

Where the guesses match what the large model would have produced they are kept. At the first mismatch everything after it is discarded and generation continues from the correction.

The output is what the large model would have produced anyway. What changes is how many sequential turns it took, and sequential turns are the thing you cannot buy your way out of with more hardware. When the draft model guesses badly the technique simply stops helping; it does not make the answer worse.

Inference
GUESS AHEAD, THEN CHECK IN ONE GO prompt draft model small, fast drafts t1 t2 t3 t4 proposed all at once main model verifies the batch accepted t1 t2 t3 t4 one mismatch: the rest is thrown away Same answer. Fewer sequential turns of the big model.
A small model proposing four tokens at once; the large model accepting three and rejecting the fourth.