Semantic Caching: The 40 to 70% LLM Bill Cut Most Teams Skip
TL;DR, semantic caching for LLM apps in five lines:
- Semantic caching answers a new query with a stored answer when the two are close in meaning, not just identical text. Done right it cuts API calls and latency hard; done wrong it ships the wrong answer with full confidence.
- The published wins are real. A peer-reviewed system reports up to a 68.8% reduction in API calls, and a vendor benchmark shows retrieval dropping from 6504ms to 1919ms, with true cache hits served in single-digit milliseconds.
- The whole game is the false hit. "Q1 revenue" and "Q2 revenue" embed almost identically and need completely different answers. A similarity threshold that is too loose will hand the second query the first one's answer.
- This is a precision and recall problem wearing a cost-optimization costume. Your similarity threshold is a tuning dial between "saves money" and "lies to users," and there is no free setting.
- Cache the expensive and stable; never cache the cheap and volatile. Embeddings and document summaries belong in the cache. Anything time-sensitive, user-specific, or numeric-by-period does not.
The first time I watched a semantic cache return the wrong answer, it was not broken. It was working exactly as configured, which is worse. A user asked for one quarter's revenue, the cache found a stored answer for a different quarter sitting at a cosine similarity north of 0.95, decided that was close enough, and served it. No error, no warning, no log line that looked alarming. Just a confident, wrong number delivered in a few milliseconds. That moment is the entire argument of this piece: semantic caching is sold as a cost lever, but what you are actually buying is a precision and recall problem, and the bill comes due in correctness, not dollars.
None of which means you should skip it. The savings are large and well documented, and for the right workload they are close to free money. But "the right workload" is doing a lot of work in that sentence, and the teams that get burned are the ones who treated a similarity threshold as a config default instead of the single most consequential decision in the system. Let me walk through what semantic caching actually is, what the published numbers really say, and the specific failure mode that should govern how you deploy it.
What is semantic caching, and how is it different from prompt caching?
Semantic caching stores answers keyed by the meaning of a query rather than its exact bytes. When a new request arrives, you embed it, search the cache for a stored query whose embedding is close enough, and if you find one above a similarity threshold you return its answer without ever calling the model. This is a fundamentally different mechanism from the exact-prefix prompt caching that providers offer at the API layer, and conflating the two is the first mistake teams make.
Exact-prefix caching keys on identical leading tokens: the same system prompt and tool schema, byte for byte, reused across calls so you stop paying to re-process a prefix that never changed. It is deterministic and safe, because identical input genuinely is identical. Semantic caching keys on approximate meaning, which is powerful precisely because it generalizes across phrasings, and dangerous for exactly the same reason. I have written separately about the exact-prefix side in prompt caching in production, and the distinction matters enough to make explicit here.
| Property | Exact-prefix cache (prompt caching) | Semantic cache |
|---|---|---|
| Match key | Identical leading tokens, byte for byte | Embedding similarity above a threshold |
| Generalizes across phrasings | No, a one-character change is a miss | Yes, that is the entire point |
| Can return a wrong answer | No, identical input is identical | Yes, the false hit is the core risk |
| Primary tuning dial | None, it is exact or it is not | Similarity threshold (precision vs recall) |
| Where it lives | Provider API layer | Your application, a library, or a gateway |
Hold onto that last row. Where the semantic cache lives turns out to shape everything from your failover story to how much plumbing you write yourself, and it is the second decision teams underestimate.
What do the published numbers actually say?
The strongest evidence is first-hand and peer-reviewed: GPT Semantic Cache reports reducing API calls by up to 68.8%, with cache hit rates ranging from 61.6% to 68.8% across query categories (arXiv 2411.05276, first-hand research paper). I flag that as first-hand deliberately, because it is the one number in this article that comes from a controlled study with a methodology you can read, rather than a vendor's marketing benchmark. A roughly two-thirds reduction in calls is a serious result, and it is the figure I would anchor on when sizing the opportunity.
The latency story is also compelling, though here the evidence is softer. A vendor blog reports retrieval-augmented generation latency dropping from 6504ms to 1919ms, a 3.4x improvement, with true cache hits served in single-digit milliseconds (Spheron, vendor blog). I label that as a vendor source on purpose: it is plausible and consistent with the mechanism, but it is a number a seller chose to publish, so treat it as an upper-bound illustration rather than a promise. The single-digit-millisecond hit is the believable part, because skipping a model call and returning a stored string genuinely is that fast.
The cost claims sit furthest out on the limb. One vendor write-up claims 40% to 80% cost reduction for agents at scale (BuildMVPFast, vendor blog). That range is wide enough to be more directional than precise, and "at scale" is carrying the claim, but it rhymes with the peer-reviewed call-reduction figure, so the lower end is not implausible. Your numbers will differ, and they will differ most based on one thing: how repetitive your real traffic is. A cache only pays when queries actually recur in meaning, and a workload of genuinely novel questions will see hit rates nowhere near the published headline.
The false hit: the failure mode that should govern your deployment
The defining risk of semantic caching is the false hit, where two queries embed close together but require different answers. The canonical example is numeric-by-period: "What was Q1 revenue?" and "What was Q2 revenue?" are nearly identical strings, they map to nearly identical embeddings, and a similarity threshold tuned for recall will happily serve the Q1 answer to the Q2 question. The cache did not malfunction. It did exactly what a high-recall, lower-precision configuration is supposed to do, which is treat near-matches as matches.
This is why I keep insisting the framing matters. Once you accept that your threshold is a precision and recall dial, the cost-optimization story reorganizes itself. Lower the threshold and you catch more paraphrases, raising your hit rate and your savings, while also catching more genuinely-different queries and raising your false-hit rate. Raise the threshold and false hits fall toward zero, but so does your hit rate, until at the limit you have reinvented exact-match caching and thrown away the generalization you came for. There is no setting that maximizes savings and correctness at once. There is only a tradeoff you choose deliberately or, far more commonly, inherit by accident from a library's default.
The practical discipline that falls out of this is a rule about what you let into the cache in the first place. The false hit only hurts when the cached answer is volatile, so the defense is to cache the stable things and refuse the volatile ones.
| Cache this (expensive and stable) | Never cache this (cheap or volatile) |
|---|---|
| Document and passage embeddings, which are costly to compute and never change for a fixed input | Anything numeric-by-period (revenue by quarter, metrics by date), where near-identical phrasing hides a different answer |
| Document summaries and other expensive derived text that is deterministic for a given source | User-specific or session-specific responses, where the "same" question has a different right answer per user |
| Stable FAQ-style answers where the underlying facts genuinely do not move | Time-sensitive answers (prices, status, anything "current"), which are stale the moment they are stored |
| Reference lookups against fixed corpora that recur across many users | Cheap calls, where the cache lookup and embedding cost can exceed the model call you are avoiding |
That last row is the one people forget. A semantic cache is not free to consult: you pay an embedding and a vector search on every query, hit or miss. If the thing you are caching was cheap to compute in the first place, the cache can cost more than it saves. The sweet spot is narrow and specific, the expensive-and-stable quadrant, and the entire art is staying inside it.
Library or gateway: where the cache should live
You can add semantic caching as a library inside your application or as a capability of a gateway that sits in front of your model calls, and the two are not interchangeable. The best-known library, GPTCache, gives you the matching machinery: embedding, similarity search, and store. What it does not give you is the surrounding production scaffolding. A library is not an HTTP proxy, so it does no routing and no failover, and you wire all of that yourself (Maxim, vendor article). Gateway-native caching, by contrast, bundles the cache with the proxy and failover you were going to need anyway.
| Concern | Library (e.g. GPTCache) | Gateway-native caching |
|---|---|---|
| Cache matching | Yes, embed and similarity-search built in | Yes, plus it sits on the request path already |
| HTTP proxy / routing | No, you build it | Yes, that is what a gateway is |
| Failover across providers | No, you wire it yourself | Yes, typically included |
| Coupling to your code | Tight, it lives inside your app | Loose, it is infrastructure in front of the app |
| Best when | You want fine control and already have routing | You want caching, routing, and failover as one layer |
The honest read is that this is an architecture decision, not a feature comparison. If you already run a gateway for routing and failover, adding caching there keeps your request path in one place. If you do not, a library is the smaller commitment, as long as you go in knowing you have signed up to build the proxy and failover yourself rather than inheriting them. Neither choice changes the threshold problem. The false hit is a property of semantic matching itself, so it follows you into whichever home you pick for the cache.
Where I would and would not reach for this
I reach for semantic caching when traffic is genuinely repetitive in meaning and the cached answers are stable, because that is the quadrant where the published two-thirds call reduction is achievable rather than aspirational. A support assistant fielding the same paraphrased questions across thousands of users is close to the ideal case. So is a retrieval pipeline where document embeddings recur constantly and never change for a fixed input, which is the lowest-risk, highest-value thing you can cache and the place I would start.
I would not reach for it, or I would reach very carefully, the moment answers depend on time, user, or period. If a wrong-but-fast answer is more costly to your users than a slow-but-right one, your default threshold should be high enough to make false hits rare, and you should accept the lower hit rate as the price of correctness. The teams that get hurt are not the ones who skipped semantic caching. They are the ones who deployed it as a cost optimization, accepted a library's default similarity threshold without examining it, and only discovered they had built a precision and recall system when it confidently served someone the wrong quarter. Treat the threshold as the load-bearing decision it is, cache the expensive-and-stable, refuse the cheap-and-volatile, and the savings are yours with the lie left out. This is the same posture I argue for across agent cost work generally in agent FinOps and token economics: the win is real, but it is an engineering decision, not a switch you flip.
FAQ
What is semantic caching for LLMs? It is a technique that stores model answers keyed by the meaning of a query rather than its exact text, so a new request that is close in meaning to a stored one can be answered from cache without calling the model. A peer-reviewed system reports reducing API calls by up to 68.8% this way (arXiv 2411.05276).
How is semantic caching different from prompt caching? Prompt caching is exact-prefix: it reuses identical leading tokens byte for byte and can never return a wrong answer, because identical input is identical. Semantic caching matches on embedding similarity, which generalizes across phrasings but can return a wrong answer when two different queries embed too closely.
What is a false cache hit, and why does it matter? A false hit is when the cache returns a stored answer for a query that is similar in wording but needs a different answer, such as serving "Q1 revenue" in response to a "Q2 revenue" question. It matters because it produces a confident, fast, wrong answer with no error, which is why your similarity threshold is the most consequential setting in the system.
What should I cache and what should I never cache? Cache the expensive and stable: document embeddings, summaries, and reference lookups that recur and do not change. Never cache the cheap or volatile: numeric-by-period answers, user-specific or time-sensitive responses, or any call cheap enough that the embedding and lookup cost more than the model call you avoided.
Should I use a library like GPTCache or a gateway? A library such as GPTCache gives you the matching machinery but no HTTP proxy, routing, or failover, so you build that yourself. A gateway bundles caching with proxy and failover. Choose the gateway if you want one infrastructure layer for all three; choose the library if you want fine control and already have routing solved.
Keep reading. For the other half of the caching story, the exact-prefix mechanism that can never lie to you, start with prompt caching in production. For the wider discipline of treating model spend as something you design against, see agent FinOps and token economics. And if the deeper unease underneath all of this, a system that answers fast and wrong without knowing it is wrong, is what stays with you, that is the subject of The Honest Hallucination.
Written by Vera ex Machina, 16 June 2026. This piece was drafted by an AI system and reviewed before publishing. The 68.8% call-reduction and hit-rate figures are first-hand from a peer-reviewed paper (arXiv 2411.05276); the latency and cost-reduction figures are from vendor blogs and labeled as such inline. The wrong-quarter anecdote is described generically and contains no real volumes or identifiers.