A model can fit on your GPU and still run out of memory halfway through a conversation. As the conversation grows, the model keeps more intermediate results around so it can reuse them while writing its reply.

Those results live in the key-value cache, or KV cache. Keeping them saves repeated work. It also takes memory.

What gets cached

An LLM writes one token at a time. Tokens are the pieces of text it works with: sometimes a word, sometimes part of one. To choose the next token, the model uses the text that came before it.

Consider this sentence:

The capital of France is Paris.

After producing “is”, the model needs the earlier words to help choose “Paris”. It has already processed those words. Repeating all that work would be wasteful.

At each attention layer, a stage that combines information from tokens, it stores two sets of numbers for each processed token:

• A key helps later tokens decide how much attention to pay to this position.
• A value carries information that attention can use from that position.

That is where the name comes from. Hugging Face’s caching documentation describes how each new token adds its keys and values to the stored ones.

This reuse works because the model processes each position using only that position and the ones before it. Adding “Paris” at the end does not change how “France” was processed earlier. Changing an earlier word would be a different matter.

The cache belongs to this input. It does not update the model’s weights or teach it anything permanently.

Watch it grow

There are two stages to generating a reply. Prefill processes the prompt and fills its cache. Decode processes the generated tokens, adding to the cache as the reply grows.

In the example below, prefill processes “The capital of France” and predicts “is”. The next step processes “is” to predict “Paris”. Click through and watch which boxes get reused.

Prefill: 4 prompt tokens

Process next token Reset

1 The K / V computed

2 capital K / V computed

3 of K / V computed

4 France K / V computed

5 is K / V pending

6 Paris K / V pending

7 . K / V pending

Compute all 4 prompt positions. The final position predicts "is".

4 new K/V pairs per layer. 0 reused.

After prefill, each step computes one new token’s keys and values. The earlier pairs are reused. A token gets its pair when it is fed back through the model, after being chosen as output.

But the model still has to read the stored information. For a model that attends to its full history, a longer conversation means more cached data to read at each step. Reusing that data is cheaper than rebuilding it, but reading it is still work. This memory traffic is the problem addressed in Fast Transformer Decoding.

Put a number on the memory

The amount of cache memory per token depends on the model and how the cache is stored. For one illustrative model, suppose each token needs 128 KiB.

At that size, 8,192 tokens need 1 GiB of cache. Double the conversation length and you need twice as much. Keep eight independent conversations of that length active at once and their caches need 8 GiB.

Try changing the conversation length and the number of conversations below.

Cache memory for one example model

Tokens per conversation

8,192

Conversations at once

1

1.00 GiB 128 KiB per token

These figures cover the cache alone. The model’s weights and the rest of the software need memory too. Both prompt tokens and processed reply tokens count, so asking for a short answer to a huge document can still require a large cache.

Some models keep only a limited history in parts of the cache, and some systems compress or share cached data. The cache strategies guide covers ways to reduce cache memory. The example here assumes each conversation keeps its full history separately.

Reusing work across requests

Suppose you send the same 20,000-token manual with every question. The server could process that manual from scratch each time. Or, if it supports prefix caching, it can keep the manual’s cached results and reuse them on the next request.

A prefix is the beginning of the prompt. Reuse requires that beginning to match, with the same model and relevant settings. Putting the same paragraph somewhere else in the prompt is insufficient: the words before it affect the results being cached. vLLM’s prefix caching documentation explains how it identifies matching prefixes.

For a document assistant, I would arrange the prompt like this:

Fixed system instructions
Reference manual
The user's question

Now the question can change while the expensive part stays the same. The server can reuse the manual’s processed state and generate a fresh answer.

A timestamp at the top of the prompt would spoil that arrangement. As soon as the timestamp changes, the matching prefix ends before the manual. Put changing information toward the end, and keep details the model does not need in your logs.

To check whether this helps, send two different questions about the same manual and inspect the server’s reported cached-token count. An identical prefix makes reuse possible; it does not guarantee the server still has it cached.

Compare how long each request takes to start answering. Prefix caching saves repeated prompt processing, so this is where you should look for an improvement. It does not remove the work of generating the answer token by token. vLLM documents that distinction: a reply can start sooner and still take just as long to write the remaining words.