- Essays··9 min read
The Similarity Threshold You Adjust on Tuesday and Regret by Friday
Semantic caching cuts LLM inference costs by 30–60% in benchmarks and causes a tenant data leak on Monday when the namespace is global and the threshold was tuned on last week's traffic. A single cosine similarity knob controls false-positive rate, hit rate, and customer-quality risk simultaneously — a coupling the vendor guides reduce to footnotes. The production stack requires cross-encoder reranking, per-tenant isolation, and threshold recalibration tied to upstream data changes.
semantic-cachingragllm-infrastructurevector-searchRead - Essays··8 min read
The Five-Minute Window
In March 2026, Anthropic silently reduced Claude's prompt cache TTL from one hour to five minutes — no blog post, no deprecation notice, no API version bump. For production workloads designed around the longer window, effective costs rose 30–60%. The change was not a bug; it was a vendor adjusting to its own cost curve after measuring what prefix caching at scale actually costs.
prompt-cachinganthropicapi-economicsllm-infrastructureRead