Almost every AI session begins with the same quiet waste. The system reads what it has read before, understands what it has understood before, and pays for both again. Nobody would repeat onboarding for an employee every morning. We do it to machines daily.
Reuse as a stance
Caching is treated as an optimisation you add at the end. It is actually a question of principle: may a system keep what it has understood? Answer yes and you build differently. Understanding then becomes a stock that grows instead of an expense that expires.
Understanding that is not kept is work you only rented.
What a prompt cache actually does
Suppliers now offer a mechanism for exactly this. At Anthropic, reading from the prompt cache costs a tenth of the normal input price, while writing costs 1.25 times or twice the base rate depending on how long the entry should live. That is the state of things in August 2026; other suppliers price differently and prices change. The saving is substantial, but it depends on a condition that many systems break by accident.
A hit requires an exactly identical prefix. The cache entry is formed over the cumulative beginning of the request, in the order tools, system prompt, messages. Change anything at or before the marked position and the hash differs, so the hit is gone. A timestamp in the system prompt is enough to do it.
The most common mistake
It looks harmless. The cache marker sits at the end of the prompt, on the incoming user message or on a block containing a timestamp. Every request then dutifully writes a new entry and never reads one, because no entry was ever written at that position. The bill actually rises, since writing costs more than ordinary input. If you switch caching on and see higher costs, you have most likely built precisely this.
The correction is unspectacular. The marker belongs on the last block that stays identical across requests, and the prompt should be built so that the stable material comes first and the changing material last. A detail question thereby turns into a design rule: the order of a prompt decides whether reuse is possible at all.
What a memory needs
Keeping is only useful if what is kept is true. A memory therefore needs three things. It needs invalidation, so that whatever changes is forgotten. It needs provenance, so that every entry knows its source and an error can be traced back. And it needs a budget, because memory competes for the window as well.
A word of caution: a cache that reuses whole answers rather than prefixes is a different animal. Serving an earlier answer because the new question sounds similar merely moves the confusion of similarity with relevance to another place in the system.
The honest benchmark for a system is therefore not how much it can read, but what it no longer has to read the second time.