- Essays··8 min read
The Five-Minute Window
In March 2026, Anthropic silently reduced Claude's prompt cache TTL from one hour to five minutes — no blog post, no deprecation notice, no API version bump. For production workloads designed around the longer window, effective costs rose 30–60%. The change was not a bug; it was a vendor adjusting to its own cost curve after measuring what prefix caching at scale actually costs.
prompt-cachinganthropicapi-economicsllm-infrastructureRead - Essays··7 min read
Pulling Apart the Inference Stack
By mid-2026 every serious inference framework has accepted that the two halves of a forward pass want different hardware: prefill on compute-bound GPUs, decode on bandwidth-bound ones, the KV cache shipped between them over a fast fabric. It is the deepest reshaping of LLM serving since continuous batching, and it happened almost entirely without anyone outside the inference crowd noticing.
llm-inferenceprefill-decode-disaggregationvllmkv-cacheRead