AI engineering · 05 of 42

Continuous Batching

Keep the GPU busy

Scroll

Keep the GPU busy

GPUs are efficient when they work on many things at once, so requests are batched. The naive version waits for a whole batch to finish before starting the next one.

Continuous batching keeps the batch open. As soon as one request completes its slot is freed and a queued request drops straight into it, mid-flight.

Without this, a batch runs at the speed of its slowest member and the hardware idles while short requests wait for a long one. With it, requests of wildly different lengths can share the same run — which is most of the difference between a demo and a service.

Serving
THE GPU NEVER WAITS FOR A FULL BATCH request A request B request C live GPU batch slots update mid-flight A B C request D waits request D A finishes, its slot empties slot freed joins D no queue drain, no idle silicon The alternative wastes the whole batch on its slowest member. Requests of wildly different lengths can share one run.
Three requests batched together. When the first finishes, a waiting request takes its slot immediately.