AI engineering · 38 of 42

Rate Limiting & Backpressure

Say no early instead of failing late

Scroll

Say no early instead of failing late

A rate limiter caps how much work is accepted. Past the cap, callers get an immediate refusal with a hint about when to come back, instead of a slow request that eventually times out.

Backpressure is the same idea propagated: the limit travels upstream so queues do not grow without bound and the system degrades in a way somebody chose.

Without them, load arrives anyway. Every request gets slower, memory fills with queued work, and eventually everything fails together — the worst outcome, arrived at by trying to serve everyone. A fast honest refusal is a feature, and it pairs naturally with the circuit breaker in concept 16.

Reliability
SAY NO EARLY INSTEAD OF FAILING LATE more than you can serve limiter tokens/second allowed 429, retry after the caller waits, and knows how long backpressure the limit propagates upstream, so the queue never grows unbounded Without it, load arrives anyway and every request gets slower until they all time out — the worst outcome, reached by trying to serve everyone. A fast, honest refusal is a feature. Pair it with the breaker in concept 16.
Incoming load metered by a limiter: some allowed through, the rest refused immediately with a retry-after.