Why KV Cache Matters in LLM Inference: Attention Optimization, Memory Trade-offs, and Serving Efficiency

The key-value (KV) cache is the data structure that makes autoregressive generation affordable. In most production deployments it is also what limits how many users a GPU can serve, how long a context can be, and how fast tokens come out.

The cache exists because of a redundancy in how transformers generate. Removing that redundancy trades computation for memory, and the memory bill turns out to be large enough to shape model architecture (multi-query and grouped-query attention, latent attention), serving systems (PagedAttention, vLLM, prefix caching), and hardware choices. This article builds up the cache from the attention equation, does the memory arithmetic explicitly, and then works through the main techniques used to tame it.

The redundancy in autoregressive generation

A decoder-only LLM generates one token at a time. At step $t$ it has a sequence $x_1, \dots, x_t$ and computes a distribution over $x_{t+1}$ . Then it appends the sampled token and repeats.

Inside each layer, causal self-attention computes, for every position $i$ ,

\mathbf{q}_i = W_Q\mathbf{x}_i, \quad \mathbf{k}_i = W_K\mathbf{x}_i, \quad \mathbf{v}_i = W_V\mathbf{x}_i,

\mathbf{o}_i = \sum_{j \le i} \alpha_{ij}\, \mathbf{v}_j, \qquad \alpha_{ij} = \frac{\exp\!\left(\mathbf{q}_i^\top \mathbf{k}_j / \sqrt{d_h}\right)}{\sum_{j' \le i} \exp\!\left(\mathbf{q}_i^\top \mathbf{k}_{j'} / \sqrt{d_h}\right)}.

Two facts about this equation create the opportunity:

Only the newest position’s output is needed to predict the next token. We need $\mathbf{o}_t$ , not $\mathbf{o}_1, \dots, \mathbf{o}_{t-1}$ , which were already used in earlier steps.
The keys and values of earlier positions never change. Because of the causal mask, $\mathbf{x}_j$ at layer $\ell$ depends only on tokens $\le j$ . Appending a new token cannot change it, so $\mathbf{k}_j$ and $\mathbf{v}_j$ are fixed once computed.

A naive generation loop ignores both facts and re-runs the whole sequence every step, recomputing every key and value from scratch. Over $T$ generated tokens, the projections alone cost $\sum_t O(t\,d^2) = O(T^2 d^2)$ per layer, and full attention over all positions costs $\sum_t O(t^2 d) = O(T^3 d)$ .

The KV cache stores $\mathbf{k}_j$ and $\mathbf{v}_j$ for every layer and every previous position. Each new step then computes $\mathbf{q}_t, \mathbf{k}_t, \mathbf{v}_t$ for only the new token, appends $\mathbf{k}_t, \mathbf{v}_t$ to the cache, and attends from $\mathbf{q}_t$ over the cached keys and values:

\mathbf{o}_t = \text{softmax}\!\left(\frac{\mathbf{q}_t K_{\le t}^\top}{\sqrt{d_h}}\right) V_{\le t}.

Projections drop to $O(d^2)$ per step, so $O(T\,d^2)$ in total, and attention to $O(t\,d)$ per step, so $O(T^2 d)$ in total. The queries are not cached, because each query is used exactly once.

class Attention(torch.nn.Module):
    def forward(self, x, cache=None):           # x: (B, n_new, d)
        q, k, v = self.project(x)               # each (B, heads, n_new, d_h), RoPE applied
        if cache is not None:
            k = torch.cat([cache["k"], k], dim=2)   # append to cached keys
            v = torch.cat([cache["v"], v], dim=2)   # append to cached values
        new_cache = {"k": k, "v": v}
        # causal masking is only needed when n_new > 1 (prefill)
        out = scaled_dot_product_attention(q, k, v, is_causal=(x.shape[1] > 1))
        return self.out_proj(merge_heads(out)), new_cache

(Real implementations preallocate or page the cache instead of concatenating, for reasons covered below.)

Two phases: prefill and decode

With a cache, inference splits into two phases that behave very differently on hardware.

Prefill processes the whole prompt in one forward pass. All prompt positions are known, so attention for all of them is computed in parallel with a causal mask, exactly as in training, and the K and V of every prompt token are written into the cache. Prefill is a large batch of matrix multiplications and is compute-bound. Its duration largely determines time to first token (TTFT).

Decode then generates one token per step per sequence. Each step does very little arithmetic (a single position) but must read all of the model’s weights and the sequence’s entire KV cache from GPU memory. Decode is memory-bandwidth-bound. Its per-step duration is the time per output token (TPOT), or inter-token latency.

The difference shows up clearly in arithmetic intensity, the ratio of FLOPs performed to bytes moved. For the weight matrices during decode with batch size $B$ , each weight (2 bytes in FP16) is read once and used for $2B$ FLOPs, so the intensity is about $B$ FLOPs per byte. A modern datacenter GPU can do on the order of 300 FLOPs per byte of memory bandwidth (for example an H100: roughly 990 dense BF16 TFLOP/s against about 3.35 TB/s). Below a batch of a few hundred, decode is starved for memory bandwidth, not compute. This is why batching is the central lever in LLM serving: every additional sequence in the batch reuses the same weight reads.

The KV cache does not benefit from that amortization. Each sequence has its own cache, so attention during decode reads $O(t)$ bytes of K and V per sequence to do $O(t)$ FLOPs, an intensity of roughly 1 FLOP per byte with standard multi-head attention, regardless of batch size. As contexts get longer and batches get larger, attention’s cache reads start to dominate decode time. That makes cache size a latency problem, not just a capacity problem.

How big is the cache?

For each token, each layer stores one key vector and one value vector per KV head. The total size is

\text{KV bytes} = 2 \times L \times H_{kv} \times d_h \times s \times B \times b,

where the factor 2 is for K and V, $L$ is the number of layers, $H_{kv}$ the number of key/value heads, $d_h$ the head dimension, $s$ the sequence length, $B$ the batch size, and $b$ the bytes per element (2 for FP16/BF16).

def kv_cache_bytes(layers, kv_heads, head_dim, seq_len, batch, bytes_per_elem=2):
    return 2 * layers * kv_heads * head_dim * seq_len * batch * bytes_per_elem

GiB = 2**30
# Llama 2 7B: 32 layers, 32 KV heads (full MHA), head_dim 128
print(kv_cache_bytes(32, 32, 128, 1, 1) / 2**10, "KiB per token")       # 512.0
print(kv_cache_bytes(32, 32, 128, 4096, 32) / GiB, "GiB")              # 64.0
# Llama 3 8B: 32 layers, 8 KV heads (GQA), head_dim 128
print(kv_cache_bytes(32, 8, 128, 8192, 32) / GiB, "GiB")               # 32.0

Some concrete numbers in 16-bit precision:

Model	Layers	KV heads × head dim	KV per token	1 sequence, 4K tokens	32 sequences, 4K tokens
Llama 2 7B (MHA)	32	32 × 128	512 KiB	2 GiB	64 GiB
Llama 3 8B (GQA)	32	8 × 128	128 KiB	0.5 GiB	16 GiB
Llama 2 70B (GQA)	80	8 × 128	320 KiB	1.25 GiB	40 GiB
70B-class with full MHA (64 heads)	80	64 × 128	2.5 MiB	10 GiB	320 GiB

Llama 2 7B’s weights are about 13.5 GB. Its KV cache for 32 concurrent 4K-token sequences is 64 GiB (about 69 GB), roughly five times the weights. In serving, the cache rather than the model is usually what fills the GPU.

Line chart of KV cache size for one sequence against context length for four model configurations, with the 80 GB GPU memory line

Figure 1. KV cache for a single sequence, computed from the formula above. With full multi-head attention, one long-context sequence of a 70B-class model would not fit on an 80 GB GPU; grouped-query attention changes that.

The formula shows what each dimension does:

Sequence length and batch size scale the cache linearly, and they are the two quantities operators most want to increase. Doubling the context halves the number of sequences that fit.
Layers scale it linearly. Deeper models pay more per token.
KV heads × head dimension is the only term architecture can cut without changing the model’s depth, and it is where most of the innovation has gone.

Because memory not used for weights goes to the cache, the cache size per token directly sets maximum concurrency:

B_{\max} \approx \frac{M_{\text{GPU}} - M_{\text{weights}} - M_{\text{activations}}}{\text{KV bytes per token} \times s}.

On an 80 GB GPU serving a 7B model with 4K contexts, MHA allows roughly 30 concurrent sequences, and the same model with 8 KV heads would allow roughly 4× more. Since decode throughput scales with batch size, cache size per token is effectively a throughput number.

Shrinking the cache architecturally

Multi-query and grouped-query attention

In standard multi-head attention (MHA), each of the $H$ query heads has its own key and value heads. Shazeer (2019) proposed multi-query attention (MQA): keep $H$ query heads but share a single key head and value head across all of them. The cache shrinks by a factor of $H$ , often 32× or more, and decode gets much faster because far less K/V data is read per step. The cost is quality and training stability, since all heads must now look up information through the same keys and values.

Grouped-query attention (GQA) (Ainslie et al., 2023) interpolates between the two. Query heads are divided into $G$ groups, and each group shares one K/V head:

H_{kv} = G, \qquad 1 \le G \le H.

$G = H$ is MHA and $G = 1$ is MQA. With 8 KV heads serving 32 or 64 query heads, GQA gets most of MQA’s memory savings with quality close to MHA. Ainslie et al. also showed that an existing MHA checkpoint can be converted (“uptrained”) to GQA by mean-pooling its K/V heads within each group, then training briefly, with about 5% of the original pretraining compute. GQA is now standard: Llama 2 70B, the Llama 3 family, Mistral 7B and many others use it.

GQA also raises the arithmetic intensity of decode attention. Each loaded key and value is now used by $H/G$ query heads instead of one.

Multi-head latent attention

DeepSeek-V2 (2024) went further with multi-head latent attention (MLA). Instead of caching per-head keys and values, each token’s hidden state is compressed by a learned down-projection into a small latent vector, and only that latent is cached. Keys and values for all heads are reconstructed from it with up-projections, which can be folded into the query and output projections so they need not be materialized during decode. Rotary position information is carried in a separate small decoupled key component, because RoPE does not commute with the folding. DeepSeek reported a 93.3% reduction in KV cache compared with their earlier 67B dense model, and much higher generation throughput. The cost is extra complexity and some additional compute.

Other architectural options cap or share the cache:

Sliding-window attention limits each layer to the last $w$ tokens (Mistral 7B used $w = 4096$ ), so the cache stops growing at $w$ per layer. Information from further back can still flow indirectly through stacked layers. Several recent models interleave sliding-window layers with a few full-attention layers to keep long-range retrieval.
Cross-layer KV sharing lets adjacent layers reuse the same K/V, cutting the cache by the sharing factor at some cost in quality.

Managing the cache: PagedAttention and vLLM

Even with a small cache per token, how memory is allocated matters a great deal. Early serving systems preallocated a contiguous buffer per request sized for the maximum possible length, because output length is unknown in advance. That causes three kinds of waste:

Reservation waste: space reserved for tokens that are never generated.
Internal fragmentation: requests that finish early leave most of their buffer unused.
External fragmentation: free memory split into gaps too small for a new request’s contiguous buffer.

Kwon et al. (2023) measured that existing systems used only 20–40% of their KV cache memory for actual token states. Their system, vLLM, introduced PagedAttention, which borrows the idea of virtual memory paging from operating systems:

The cache is divided into fixed-size blocks, each holding K and V for a small number of tokens (for example 16).
Each sequence has a block table mapping its logical token positions to physical blocks, which can be anywhere in GPU memory.
Blocks are allocated on demand as the sequence grows, so waste is limited to the unfilled part of each sequence’s last block.
The attention kernel is rewritten to gather keys and values through the block table rather than from contiguous memory.

Diagram of PagedAttention: two sequences’ block tables mapping logical blocks to scattered physical blocks, with one block shared

Figure 2. PagedAttention. Each sequence’s block table maps its logical blocks to physical blocks anywhere in memory, and sequences with a common prefix point to the same block.

Paging also makes sharing cheap. When several sequences share a prefix, such as a long system prompt, several samples from one prompt, or beam search candidates, their block tables can point to the same physical blocks, with reference counting and copy-on-write when one of them diverges. The vLLM paper reported memory waste under 4% and 2–4× higher throughput than the previous state-of-the-art systems at the same latency.

Paging works together with continuous batching (also called in-flight batching, introduced by Orca in 2022). Instead of waiting for a whole batch to finish, the scheduler adds new requests and removes finished ones at every decode step, so the batch stays full. Fine-grained cache allocation is what makes this practical. When memory runs out, the scheduler preempts sequences and either swaps their blocks to CPU memory or discards them and recomputes them later.

Reusing the cache across requests: prefix caching

Many requests share long prefixes: the same system prompt, the same few-shot examples, the same retrieved document, or the earlier turns of a conversation. Their KV states are identical, so recomputing them for each request wastes prefill compute.

Prefix caching keeps KV blocks after a request finishes and reuses them when a later request begins with the same tokens. vLLM’s automatic prefix caching identifies blocks by a hash of their tokens plus everything before them. SGLang’s RadixAttention keeps cached prefixes in a radix tree and evicts least-recently-used branches. The effect on TTFT can be dramatic: a request whose 10K-token prefix is cached only has to prefill its new suffix.

This is also the mechanism behind the prompt caching features of commercial LLM APIs, which charge less for cached input tokens. It has a practical design consequence for applications: put stable content first. System prompts, tool definitions and long documents should come before per-request variable content, so that prefixes actually match.

Compressing the cache: quantization

Storing K and V in fewer bits cuts both memory and the bandwidth needed per decode step.

FP8 or INT8 KV caches halve the size relative to 16-bit, usually with negligible quality loss, and are supported by major serving engines.
Lower bit-widths need care, because keys and values have different statistics. KIVI (Liu et al., 2024) observed that keys have a few outlier channels with consistently large magnitudes, while values do not. It quantizes keys per channel and values per token, keeps the most recent tokens in full precision, and reaches 2 bits with small accuracy loss.

def quantize_per_channel(x, bits=4):            # x: (tokens, d_h); stats over tokens
    lo, hi = x.min(dim=0, keepdim=True).values, x.max(dim=0, keepdim=True).values
    scale = (hi - lo).clamp(min=1e-6) / (2**bits - 1)
    q = torch.round((x - lo) / scale).to(torch.uint8)
    return q, scale, lo                         # dequantize: q * scale + lo

The trade-offs are dequantization overhead inside the attention kernel, the need for kernel support, and accuracy degradation that tends to show up first on long-context retrieval tasks, where small errors in many keys can shift attention to the wrong place.

Discarding parts of the cache: eviction

If not every past token matters equally, some can be dropped.

StreamingLLM (Xiao et al., 2023) found that models dump large amounts of attention onto the first few tokens, which act as attention sinks, regardless of their content. Simply keeping a sliding window of recent tokens breaks the model once those initial tokens are evicted. Keeping the first few tokens plus a recent window allows stable generation over very long streams with a fixed-size cache.
H2O (Zhang et al., 2023) keeps “heavy hitter” tokens that have received the most accumulated attention, plus recent tokens, and evicts the rest.
Later methods compress per head or per layer, using different budgets where attention is more or less concentrated.

Eviction is lossy in a way the other methods are not. A token that seemed unimportant can become essential later, for example a name or a number that a question asks about many turns afterwards. Eviction works well for streaming and summarization-like workloads and worst for “needle in a haystack” retrieval from long context.

FlashAttention and FlashDecoding. FlashAttention makes prefill attention fast and memory-efficient by tiling. FlashDecoding parallelizes decode attention across the sequence dimension, which matters when batch size is small but the cache is long, since otherwise most GPU cores sit idle.
Chunked prefill. A long prompt’s prefill can stall every other sequence’s decode step. Splitting prefill into chunks and mixing them with decode steps in the same batch (as in Sarathi-Serve) smooths inter-token latency for everyone.
Prefill–decode disaggregation. Because the two phases have opposite hardware profiles, some systems (DistServe, Splitwise) run them on separate GPU pools and transfer the KV cache between them, tuning each pool separately for TTFT or TPOT.
Speculative decoding. A cheap draft model proposes several tokens, and the target model verifies them in a single forward pass, which writes several cache entries per step. It speeds up decode when the draft is often right, at the cost of extra cache management for rejected tokens.
Offloading. Cold parts of the cache can be moved to CPU memory or even disk, trading PCIe bandwidth for capacity. This is useful for very long contexts or many idle conversations.

The trade-off space

Technique	Memory	Latency	Throughput	Quality risk	Cost
GQA / MQA	↓ 4–32×	Faster decode	↑	Small (GQA) to moderate (MQA)	Must be in the architecture (or uptrained)
MLA	↓ large	Faster decode	↑	Small if trained in	Architectural complexity
PagedAttention	Near-zero waste	Neutral	↑ 2–4× vs contiguous	None	Custom kernels and scheduler
Prefix caching	Reuses memory	Much lower TTFT on hits	↑	None	Only helps when prefixes repeat
KV quantization (8-bit)	↓ 2×	Slightly faster decode	↑	Usually negligible	Kernel support
KV quantization (2–4 bit)	↓ 4–8×	Depends on kernels	↑	Measurable on long context	Careful calibration
Eviction / sliding window	Bounded	Constant per step	↑	Loses distant information	Workload-dependent
Offloading	Expands capacity	Slower on access	Depends	None	PCIe bandwidth

Quick reference

Question	Answer
Is the cache optional?	Not in practice. Without it, attention work over a long output grows cubically, so every inference engine uses one.
Does model size set serving memory?	Only partly. With long contexts and many users, the cache usually exceeds the weights.
Is a 128K context cheap if the model supports it?	No. One 128K-token sequence of a Llama-3-8B-like model needs about 16 GiB of cache in 16-bit precision.
Why aren’t queries cached?	Each query is used once, at the step where it is computed. Only keys and values are reused.
Why is decoding slow on a fast GPU?	It is limited by memory bandwidth, not arithmetic. It speeds up only with more bandwidth or fewer bytes per token.

In production

A modern serving stack combines most of these ideas. The model uses GQA or MLA, so the cache is small per token. The engine (vLLM, SGLang, TensorRT-LLM, TGI) manages the cache in pages, batches continuously, reuses shared prefixes, and often stores the cache in FP8. Long prompts are prefilled in chunks so they do not stall decoding, and very large deployments split prefill and decode onto different hardware. Capacity planning starts from the formula above: given the model’s KV bytes per token, the expected context lengths and the latency target, how many concurrent sequences fit on each GPU?