Semantic Cache Hit, User Miss
OpenAI enabled automatic prompt caching on 1 October 2024, delivering up to 80% latency reduction and 50% cost savings for long prompts. Anthropic launched its version in August 2024, charging $3.75 per million write tokens but only $0.30 per million read tokens, a structure that promised 90% savings on repeated context. Both announcements landed …
OpenAI enabled automatic prompt caching on 1 October 2024, delivering up to 80% latency reduction and 50% cost savings for long prompts. Anthropic launched its version in August 2024, charging $3.75 per million write tokens but only $0.30 per million read tokens, a structure that promised 90% savings on repeated context. Both announcements landed with case studies showing double-digit cost reductions and sub-second response times. The engineer who shipped the feature knows what those numbers leave out: they measure provider-side prefix caching, where the cache key is a byte-exact match on a token sequence. The more valuable pattern, semantic caching at the application layer, carries a false-positive rate the slide deck does not mention.
Semantic caching matches by meaning using vector similarity, and AWS research reported up to 86% cost and 88% latency reduction on cached responses, with production deployments seeing 20-73% token cost reduction. The promise is that a user asking "What's your refund policy?" gets the same cached answer as someone who typed "How do refunds work?" without burning tokens on a fresh model call. The system embeds each prompt into a vector and returns a cached answer when cosine similarity exceeds a threshold, intercepting 25-40% of redundant traffic on typical workloads. That figure comes from controlled studies and well-behaved FAQ traffic. Production looks different.
Published production hit rates range from 20-45%, not the 90-95% sometimes quoted, because the 95% number refers to accuracy of cache matches, not frequency of hits. An EdTech platform with high-repetition student Q&A saw 45% hit rate, while general production estimates settle around 30-40%, and open-ended chat workloads deliver 10-20%, because 60-70% of real queries are genuinely unique and the controlled benchmark hit rates do not survive contact with production diversity. A 30% hit rate still saves money. The cost problem is not the miss rate. The cost problem is that the hits are not always correct.
Serving cached answers based purely on vector proximity risks subtle false positives where prompts share high cosine similarity but demand opposite actions. At usable thresholds the false-positive rate runs 3-7%, and to reach above 30% hit rate you operate around cosine similarity 0.93-0.95, which means a customer support bot handling 10,000 queries per day at 30% cache hit rate returns 3,000 cached responses, 90-200 of which are wrong. A banking case study reported an initial false-positive rate of 99%, later reduced to 3.8% through architectural changes rather than threshold tuning. The Pareto curve is unforgiving: tighten the threshold to 0.98 and false positives drop below 0.5%, but hit rate collapses to 5-10%. Loosen it to 0.90 and hit rate climbs, but false positives increase dramatically below 0.80.
A false positive in a semantic cache is not a parsing error the user can see and retry. It is a confident answer to a question nobody asked, delivered at 50 milliseconds with no indication that the response came from a different query. The failure is silent because nothing logs a "false positive cache hit", and unlike exact-match or provider prompt cache where wrong hits are mostly impossible, semantic cache failures require sampling and grading cache hits to detect false positives before users do. The engineer inside the vendor platform team knows this, because the decision to ship implicit caching in October 2024 was a bet that prefix caching alone, with its zero false-positive rate and its requirement for byte-exact repetition, would carry the marketing claim. Semantic caching, the feature customers actually want, stays in the application layer because the platform cannot ship a probabilistic cache without an evaluation harness the platform does not control.
Most teams start with conservative cosine similarity settings around 0.95 or higher on first deploy, accept a 5-10% hit rate with a false-positive rate under 0.5%, then tune downward as confidence grows, and safe deployment requires a kill switch to disable the cache instantly. The tuning loop is manual. In many production deployments the static cache is effectively immutable because its contents are tied to safety and quality review, and an LLM-based verifier may have nonzero false reject and false approve rates, with false approves introducing errors on promoted entries that affect subsequent hits. The cache becomes a second model to operate, with its own accuracy surface and its own drift as embeddings, thresholds and query distributions shift.
The architectural trap is simpler. Teams enable caching, see 90% token savings in lab, then deploy to production and see 10% actual savings because each user-context combination is unique, fragmenting the cache. The most common failure mode is a developer adding "The current date is " to the top of the system prompt for freshness, and if hit rate is below 50%, something in the prompt structure is silently invalidating the cache while developers assume it is working. Caches activate only once a static prefix reaches 1,024 tokens, so systems sending shorter prompts miss the discount entirely, engineers now pad system instructions with boilerplate until the boundary is met, and OpenAI's system caches prefixes in multiples of 128 tokens beyond the threshold so misalignment triggers frequent cache misses. Padding the prompt to cross a token threshold so a cost optimisation activates is a tax paid in prompt-engineering time because the platform feature could not ship with a lower floor.
Strategic cache boundary control outperforms naive full-context caching, and caching only system prompts while excluding dynamic tool results can reduce costs by 41-80% and improve time to first token by 28-31%, while full-context caching shows no improvement or exhibits slight latency regression because caching dynamic content triggers cache thrash. The decision is per-block: system prompt cached, tools excluded, user message excluded, then the next turn caches the conversation prefix if it is long enough and static enough. Every block is a trade. Anthropic's cache has a 5-minute lifetime, refreshed each time the cached content is used, so if you do not reuse the cached prompt within five minutes you pay the write cost and see it expire before reusing it, turning the feature into a loss if request rate is too low.
One developer reported that OpenAI's 5-minute cache window in practice behaves more like 5 seconds, with the cache cold after a few seconds. Google pioneered context caching in May 2024 and rolled out implicit caching with Gemini 2.5 in late 2024, where the system passes cost savings to developers without creating an explicit cache, and if a request shares a common prefix with a previous request it is eligible for a cache hit, but with implicit caching the cost savings are not possible to predict in advance because the cache timeout is not described and presumably varies based on load. The engineer who built the feature can see cache expiry telemetry by backend load and time of day. The API customer cannot.
The model router landed in the same season, and the collision is instructive. Microsoft Foundry's Model Router is a platform-level trained routing model deployed as a single endpoint that dispatches across up to 18 LLMs per prompt, with routing mode treated as a deployment-scoped SLO lever, and it turns multi-model dispatch from an application concern into a platform concern with governance levers that map to cost, quality, compliance and resilience. The problem is you cannot configure which models the router uses at the deployment level, so document analysis above 20,000 estimated tokens routes to a static GPT-4.1 deployment to sidestep context window issues, and when Model Router picks a reasoning model those models handle parameters differently than a standard chat model, and routing added roughly 50-100 milliseconds per request so the fastest user-facing features run on static deployments instead. The platform gave the customer a dial labelled Balanced, Quality, Cost. The customer still writes an if statement in front of it to check token count and feature type, then decides whether to call the router or bypass it to a static deployment.
An if statement or a config-driven switch is enough to start and you do not need an ML-based router, with one team cutting monthly spend 60% by moving from a single frontier model to a two-model router with no measurable quality regression, and the router itself was 50 lines of Go. The platform feature exists. The production system still carries the conditional because the platform could not ship the policy the customer needed without exposing a surface the customer has to manage anyway. Caching has the same shape. The vendor can ship prefix caching with a guarantee because it is deterministic. Semantic caching stays outside because the false-positive rate is a number the vendor cannot put on the pricing page.
Prompt caching is automatic now. Semantic caching is not, and will not be, because the platform cannot own the threshold that defines correctness for your queries. The cost reduction is real. So is the 3-7% error rate, and the only way to measure it is to sample your own cache hits and grade them. If that eval loop is not running, the cache is serving answers to questions your users did not ask, and you will not know which ones until someone replies to tell you the answer was wrong.
Tarry Singh is the founder and CEO of Real AI (realai.eu), an enterprise AI advisory and deployment firm working with global enterprises on production agent systems, model risk, and AI sovereignty strategy. He also leads Earthscan (earthscan.io) for Energy AI, and is a founding contributor to the EU-funded HCAIM and PANORAIMA programmes for responsible AI education across European universities. He writes at tarrysingh.com.