AI Frontiers, part 19: KV-cache engineering — the memory wall of long context

Part 19from the AI Frontiers series · 65 parts in all

Every long-context announcement has the same shape. A model is released with a context window of some impressive number of tokens, the number goes up in the next release, and the commentary treats it as a property of intelligence. The number is real, but what it describes is a memory budget, and the budget is dominated by one data structure: the key-value cache. Understanding that structure is the difference between designing an application that uses long context well and one that quietly costs ten times what it should.

The cache exists for an obvious reason. Attention is autoregressive, so generating token n requires the keys and values for tokens 1 through n−1. Recomputing them every step would be quadratic work per token, so modern serving stacks store them. Those stored tensors are the cache, they grow linearly with sequence length, and they must live in the fastest memory available — which is also the scarcest.

The arithmetic, which everyone should do once

For a model with L layers, h key-value heads, head dimension d, and b bytes per element, the cache for a single sequence of n tokens is 2 × L × h × d × n × b bytes. Take a textbook 70B-class model: 80 layers, 64 query heads, head dimension 128, 16-bit cache. The cache costs about 327 kilobytes per token with full multi-head attention. At 128,000 tokens that is roughly 42 gigabytes for one request. The model weights in 16-bit need about 140 gigabytes. So a single long-context request doubles the memory footprint of the server, and a handful of concurrent ones exhausts it. That is the wall, and every architectural choice in the last several years is a response to it.

Three levers move the number.

Fewer key-value heads. Multi-query attention keeps all the query heads but shares one key-value head across them, cutting the cache by a factor of 64 in the example above (Shazeer). It costs some quality. Grouped-query attention (Ainslie et al.) interpolates: share keys and values within groups of query heads, recovering most of the quality while cutting the cache by the group factor — typically 4 to 8. GQA is why essentially every model released since 2023 can afford a long context at all. It is the single most consequential architectural change of the era that nobody outside the field has heard of.

A compressed representation. Multi-head latent attention, the mechanism DeepSeek introduced in V2 and carried into V3, projects keys and values into a low-dimensional latent vector and caches that, reconstructing on demand (DeepSeek-AI). Combined with RoPE positional encoding applied to a decoupled portion of the key, it cuts the cache by a large factor relative to equivalent-quality GQA. This is the mechanism behind the serving economics discussed in part 13, and it is the clearest example of a memory optimization that is also a capability optimization, because it is what makes a large effective context affordable.

Fewer bytes per element. Cache quantization to 8 bits is close to free; 2-bit schemes with per-channel and outlier handling are viable with care (Hooper et al.; Liu et al.). Which layers to protect and how aggressively to quantize depends on the architecture, so this is measurement work rather than configuration.

Paging, prefix reuse, and the invention that unlocked serving

Even with a small cache, naive serving wastes memory. Early implementations allocated a contiguous buffer for the maximum possible sequence length per request, so a request that might reach 8,000 tokens reserved 8,000 tokens of memory from the first moment, and a finished request's memory could not be reused until it was freed. Throughput suffered for a reason that had nothing to do with the model.

PagedAttention (Kwon et al.) applied the operating system's answer: split the cache into fixed-size blocks addressed by a table, allocate on demand, and let non-contiguous blocks serve as one logical sequence. Memory waste dropped from most of the reservation to a few percent, and — more importantly — shared prefixes became shareable. If a thousand requests arrive with the same system prompt, the blocks holding that prompt's cache are written once and referenced a thousand times.

Prefix caching is the optimization I would reach for first in any application with a fixed instruction preamble or a repeated document. In a retrieval-augmented system where a large reference document is prepended to every query, caching the document's cache can remove the dominant cost of the request. Radix-tree attention generalizes it to arbitrary shared prefixes (Zheng et al.), and every major provider now exposes some version of it — Anthropic's prompt caching being the visible example, where the economic difference between a cached and uncached prefix is exactly a memory-versus-compute trade the customer can see.

Prefill and decode are two different problems

The distinction matters more than the cache arithmetic. Prefill processes the prompt: it is a large matrix multiplication over the whole input, compute bound, and parallelizable. Decode generates tokens one at a time: it is memory-bandwidth bound and inherently sequential, as part 18 discussed. A long prompt is slow for a different reason than a long completion.

The consequence is scheduling. If a 100,000-token prefill occupies the accelerator, every other request stalls for its duration, which for a large prompt can be seconds. Chunked prefill splits the prompt into pieces and interleaves them with ongoing decode work so that latency stays bounded for everyone else (Agrawal et al.). This is the single feature that made long-context retrieval practical in a shared service, and it is invisible in every benchmark that measures one request at a time.

Eviction, and the uncomfortable question of what to forget

When the cache cannot fit, something has to go. A family of methods decides what by watching attention: heavy-hitter style policies keep the tokens that receive the most attention and drop the rest (Zhang et al.); attention-sink work showed that preserving the first few tokens matters disproportionately, which is what allows a sliding window over a stream to keep behaving (Xiao et al.); and later methods select a compressed set per attention head rather than uniformly (Li et al.).

These are the closest thing the field has to a principled theory of forgetting, and they all share the same limitation: the decision is made by the model's own attention pattern, which means it inherits the model's blind spots. The interaction that should worry anyone building on top of this is the positional one. The "lost in the middle" result (Liu et al.) showed that models retrieve information at the beginning and end of a long context far better than from the middle, and RULER (Hsieh et al.) showed that most models advertised at very long contexts degrade substantially before reaching their stated limit when tested on harder tasks. An eviction policy that keeps the head and tail and discards the middle is therefore consistent with how the model already behaves — which is fine until you need the middle.

The training-side cousin of the same wall

Inference is where the cache dominates, but the identical pressure shows up during training, and the solutions rhyme. Attention scores are an n×n matrix per head, so a single long sequence can exhaust accelerator memory before a single parameter update happens. FlashAttention (Dao et al.) addressed this by never materializing the full score matrix — it tiles the computation, keeps a running softmax normalization, and recomputes rather than stores, trading arithmetic for memory. Activation checkpointing does the same thing at the layer level: throw away intermediate activations and recompute them in the backward pass.

Both techniques are instances of a general principle that is worth internalizing because it recurs everywhere in this field: when memory is the binding constraint and compute is not, spend compute to save memory. The inference-side equivalents are reorganizing a cache rather than recomputing attention, and quantizing stored keys and values rather than keeping them in full precision. Every one of those choices is a bet that the arithmetic units are underused, which on modern accelerators is almost always true.

The operational reality nobody writes about

Two operational facts decide whether a long-context service works in practice, and neither appears in a paper.

The first is that cache behavior is a distribution, not a constant. Cache hit rate depends entirely on how much prefix the traffic shares, and that is a property of how the application is built rather than of the model. A product where every user starts with the same 3,000-token instruction block and the same retrieved document gets a hit rate that makes long context almost free. A product where every prompt is unique gets none of it, and the same feature costs several times as much. Instrumenting hit rate is the first thing to do when a long-context bill surprises you.

The second is isolation. A single request with an enormous context can consume the memory that a hundred normal requests needed, which makes it a denial-of-service vector as much as a cost problem. Every serious deployment ends up with separate pools — a tier for long-context work with its own limits, and a tier for interactive traffic — and refusing to run them on the same hardware. That is unglamorous capacity planning, and it is the difference between a feature that occasionally takes the site down and one that does not.

What to do about it as a builder

Four practical rules, in descending order of how much they change the bill.

Put the stable content first and the varying content last. Prefix caching rewards a stable prefix. A system prompt, a tool manifest, a schema and a document that rarely changes all belong at the front; the user's question belongs at the end. Reversing this ordering can cost an order of magnitude in prefix-cache hits for no benefit.

Retrieve less, and place it better. The temptation with a large window is to fill it. The evidence says a smaller, well-placed context outperforms a large, uniformly populated one, and the cost difference is linear in tokens. Retrieval quality beats context quantity, which is the theme of part 5 and part 20.

Measure prefill and decode separately. Time to first token and inter-token latency respond to completely different optimizations, and a single "average latency" number will hide which one you have a problem with.

Cap the cache, not the context. A request that grows unboundedly is a request that eventually evicts something another request needed. Explicit limits — maximum cached tokens per session, summarization past a threshold, separate sessions for separate documents — convert an unpredictable failure into a predictable one.

There is a fifth rule that is not about performance at all. Decide what your application retains, and for how long. A cache is a copy of your users' data sitting in an accelerator's memory, subject to whatever eviction policy the runtime chose, and a session that spans a long context is a session whose earlier content may still be resident in reviewable form long after the conversation ended. Most teams have established an opinion about what their embeddings store and almost none have one about cache lifetimes, which is an asymmetry that will eventually be discovered by an auditor rather than a load test.

It is also the layer where the honest vendors differentiate themselves, not by advertising a larger window but by publishing how their caching behaves: what hits, what costs more, what gets evicted and when. A context length is a specification and a cache policy is a contract. Only one of the two tells you what you will actually be charged, and it is not the one on the marketing page.

The reason this unglamorous layer deserves a part of its own is that it is where the industry's most impressive claims are actually decided. A million-token context window is not a model achievement; it is a claim about how efficiently someone can store and retrieve keys and values, and about whether the resulting service is priced at a level anyone will pay. The models get the headlines. The cache gets the business.

Works Cited

Agrawal, Amey, et al. "Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve." arXiv, 2024, arxiv.org/abs/2403.02310. Accessed 15 Jan. 2026.

Ainslie, Joshua, et al. "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints." arXiv, 2023, arxiv.org/abs/2305.13245. Accessed 15 Jan. 2026.

Anthropic. "Prompt Caching." Anthropic Documentation, docs.anthropic.com/en/docs/build-with-claude/prompt-caching. Accessed 15 Jan. 2026.

DeepSeek-AI. "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model." arXiv, 2024, arxiv.org/abs/2405.04434. Accessed 15 Jan. 2026.

Dao, Tri, et al. "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." arXiv, 2022, arxiv.org/abs/2205.14135. Accessed 15 Jan. 2026.

Hooper, Coleman, et al. "KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization." arXiv, 2024, arxiv.org/abs/2401.18079. Accessed 15 Jan. 2026.

Hsieh, Cheng-Ping, et al. "RULER: What's the Real Context Size of Your Long-Context Language Models?" arXiv, 2024, arxiv.org/abs/2404.06654. Accessed 15 Jan. 2026.

Kwon, Woosuk, et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention." arXiv, 2023, arxiv.org/abs/2309.06180. Accessed 15 Jan. 2026.

Li, Yuhong, et al. "SnapKV: LLM Knows What You Are Looking For Before Generation." arXiv, 2024, arxiv.org/abs/2404.14469. Accessed 15 Jan. 2026.

Liu, Zechun, et al. "KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache." arXiv, 2024, arxiv.org/abs/2402.02750. Accessed 15 Jan. 2026.

Liu, Nelson F., et al. "Lost in the Middle: How Language Models Use Long Contexts." arXiv, 2023, arxiv.org/abs/2307.03172. Accessed 15 Jan. 2026.

Pope, Reiner, et al. "Efficiently Scaling Transformer Inference." arXiv, 2022, arxiv.org/abs/2211.05102. Accessed 15 Jan. 2026.

Shazeer, Noam. "Fast Transformer Decoding: One Write-Head Is All You Need." arXiv, 2019, arxiv.org/abs/1911.02150. Accessed 15 Jan. 2026.

Su, Jianlin, et al. "RoFormer: Enhanced Transformer with Rotary Position Embedding." arXiv, 2021, arxiv.org/abs/2104.09864. Accessed 15 Jan. 2026.

Xiao, Guangxuan, et al. "Efficient Streaming Language Models with Attention Sinks." arXiv, 2023, arxiv.org/abs/2309.17453. Accessed 15 Jan. 2026.

Zhang, Zhenyu, et al. "H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models." arXiv, 2023, arxiv.org/abs/2306.14048. Accessed 15 Jan. 2026.

Zheng, Lianmin, et al. "SGLang: Efficient Execution of Structured Language Model Programs." arXiv, 2023, arxiv.org/abs/2312.07104. Accessed 15 Jan. 2026.