AI engineering · 27 of 42

Prompt Caching

Put the unchanging part first

Scroll

Put the unchanging part first

Providers can cache the processed form of a prompt prefix. Send the same opening bytes again and that portion is served from cache, at a fraction of the price and latency.

The match is on an exact prefix. Everything up to the first difference is reusable; everything after it is not.

Which makes ordering a design decision. Put the system prompt, tool schemas and stable documents first, and the volatile parts last. The classic own-goal is a timestamp or a session id near the top of the system prompt — it changes every call, so nothing is ever cached and you pay full price forever without any error to tell you.

Cost
PUT THE UNCHANGING PART FIRST stable prefix first system prompt + tools + documents this question cache hit on the first 300 reused variable part first this question system prompt + tools + documents cache miss: the bytes moved, so it is a different prefix reused Providers cache by exact prefix match, so a single changed byte near the front invalidates everything after it. Putting a timestamp at the top of your system prompt is the classic way to pay full price forever.
Two layouts: a stable prefix that gets reused, and a variable one that never does.