When I first read about KV caching in AI Engineering by Chip Huyen, I remember wondering why I was reading this. I was never going to use it. This week I spent quite a lot of time looking at KV cache charts.
I’ve been learning deep learning for a while now, mostly through books and by building things. I built and trained a small LLM, then implemented multi-head latent attention from the DeepSeek-V2 paper. I also built a non-LLM model at work for aesthetic evaluation of photos.
More recently, I wanted to explore the other side: what happens when a model has been trained and there are requests waiting for it?
I’m currently reading Inference Engineering by Philip Kiely, alongside Build a Reasoning Model (From Scratch) by Sebastian Raschka. I like learning from books. But I was ready to get hands-on, and I chose vLLM as the inference engine to play with.
For the practical side, I’m using an LLM as a tutor. It put together a curriculum based on what I already knew and helps me turn the lessons into experiments and charts that I can question and explore. Sometimes that works very well; sometimes I have to challenge the explanation or the experiment. The theory from the books helps me spot when something doesn’t add up.
What followed was roughly a week of experiments on my desktop GPU. I watched requests overlap, forced the engine to repeat work it would normally reuse, squeezed its memory, and changed how it scheduled competing requests. The most exciting part was watching the observations start to connect with the theory, forming a mental model I can reason about.
This is my account of that process. I’m very new to vLLM, and some of my interpretations may be wrong. If you spot a mistake, please tell me. I’d like to learn.
All of the experiments shown here were run with vLLM 0.33.0rc1.dev997 at commit c06ddca81c, using Qwen2.5-1.5B-Instruct on an RTX 3090 with 24 GB of VRAM. These are small, synthetic, text-only experiments, not production benchmarks. Timings include the host-side work of driving and observing the engine, not just GPU execution.
Watching requests overlap
I started by increasing concurrency and plotting the requests. I wanted a picture of what it meant for one GPU to be serving several requests at once.
Every request in this experiment had 256 input tokens and generated 128 output tokens. I increased concurrency through powers of two, from one to 512. Each burst contained exactly that many requests. I recorded when each request was submitted, first scheduled, and produced its output tokens. The timelines show waiting, prefill, and decode.
Prefill: processing the input prompt before generating the response.
Decode: generating the output tokens one at a time.

This was where I started to understand concurrency in vLLM. It could work on several prompts in the same step, and it could combine that prefill work with decoding requests that were already underway. I began thinking of a step as one iteration of the engine, with a limited amount of token work to distribute across requests.
Here, the token budget (max_num_batched_tokens in vLLM) was 2,048 token positions per step. That counted input positions being processed as well as positions used to generate the next output tokens. It wasn’t a limit of 2,048 output tokens. Once the budget was used, the remaining work had to wait for another step.
The groups near the start of the larger bursts made sense once I connected that budget to the prompt length: 2,048 divided by 256 is eight. The first step could process eight complete prompts. Eight wasn’t a fixed limit on how many requests vLLM could prefill; it came from this workload and configuration.
With chunked prefill, a prompt can be processed over several steps alongside requests that are decoding. In the next step, the eight requests already decoding each used one token position first. That left 2,040 positions for prefill: enough for seven complete prompts and 248 tokens of an eighth. That last prompt would need another step before producing its first output token. As more requests entered decode, their combined share left less room for incoming prompts.
The shape of the timelines was the part I found especially interesting. In the bursts up to 256 requests, later requests waited longer for their first token, but then spent less time decoding. The early requests had to generate through more of the steps that were also processing incoming prompts. Later requests had fewer of those longer, prefill-heavy steps left to share. All requests still generated 128 tokens, but their blue decode bars had different lengths in elapsed time. Starts spread out more than finishes, giving the right-hand edge of the timelines a steeper slope.
At 512 requests, I could see a second constraint: the concurrency limit of 256 running requests (max_num_seqs). Once those slots were full, no more prefills started, even though token budget was available. The trace showed steps processing just one decode position for each of the 256 running requests, with the rest of the 2,048-token budget unused. Waiting requests began entering as existing requests finished and freed slots. Their prefills added work to the shared steps again, so the earlier pattern of progressively shorter decode periods no longer held uniformly.
Two limits explained much of that shape: the concurrency limit and the token budget. These were finite bursts submitted together. I wanted to try staggered arrivals too, but first I had a question about memory.
KV caching felt much more fundamental than I expected
KV cache: stored attention keys (K) and values (V) for previously processed tokens, reused when generating subsequent tokens.
In my own small LLMs, KV caching had been an optimization I would add later. I expected to toggle it off in vLLM to measure its impact, but I couldn’t find an off switch for reuse within a request.
I had underestimated how much repeated work it avoided. For ordinary full attention, generating n tokens from a fixed-length prompt involves O(n²) total attention work with KV caching, compared with O(n³) if every step recomputes the entire growing history. The official Transformers caching explanation describes the underlying per-step difference.
To see the effect, I added a scheduler extension that discarded the request’s KV cache and reset the related bookkeeping after each output token except the last. The next pass had to process the prompt and all output tokens generated so far. Prefix caching was disabled throughout; I was comparing reuse between steps of a single response.
Prefix caching: reusing the KV cache from an earlier request for matching tokens at the start of a new request.
The engine still allocated and wrote the KV cache on each pass, and the timings included the reset overhead. I varied both input and output length to see how the cost grew:

At 1,024 input tokens and 1,024 output tokens, the median request duration was about 6.3 seconds with KV caching and 83.3 seconds with forced recomputation.
I also wanted to separate the effects of input and output length, so I measured inter-token latency. First, I held the output at 128 tokens and increased the input length.
Inter-token latency (ITL): the time between successive output tokens.

The first token still required processing the prompt in both modes. The difference opened up afterward, when one mode could reuse the previous work and the other kept doing it again.
Then I held the input at 128 tokens and increased the output length:

As the response grew, recomputing its history made each subsequent pass more expensive.
The KV caching curves looked approximately flat over this range. I don’t take that to mean decoding with KV caching has a constant cost for arbitrarily long contexts. They just look flat in comparison to how horrible the other ones are.
From here on, ordinary KV caching within each request stayed enabled in all my experiments.
Reusing work across requests
Next, I wanted to understand prefix caching. I’d seen how much work I could save by keeping the KV cache while generating one answer. Now I wanted to see what could be reused when a new request arrived with some of the same input.
I started each pair with a 1,024-token prompt, then tried four variations for the second request:
- Repeating it exactly.
- Appending 128 tokens.
- Changing token 769.
- Changing the first token.
I ran each case with prefix caching enabled and disabled. I wanted to see both how quickly the second response began and whether reusing prompt work changed the pace of the tokens that followed.

Time to first token (TTFT): the wait from submitting a request to receiving its first output token.
For an exact repeat of a 1,024-token prompt, median time to first token fell from about 49 ms to 9 ms. The subsequent token intervals were roughly unchanged, around 6 ms in both cases.
That was the distinction I wanted to see: less work before the first output token, while the answer still had to be generated.
There was a detail I hadn’t expected. The exact repeat reused 960 tokens, not all 1,024. In this experiment, each cache block held KV for 64 tokens, and prefix caching reused only complete blocks.
Cache blocks: fixed-size units of KV storage, each covering a group of token positions.
The 1,024-token prompt fits exactly into 16 blocks of 64. Even so, vLLM needed to recompute the final prompt token to obtain the logits, the scores used to choose the first output token. The KV cache held attention keys and values, not those logits. Since vLLM couldn’t reuse only the first 63 tokens of that last block, it recomputed the whole block.
Changing just the first token prevented reuse of the entire remaining prefix in this experiment. Having the same later words wasn’t enough. Their preceding context had changed.
A conversation that keeps growing
This became more intuitive when I looked at a conversation over several turns. Each request included the accumulated history, followed by another 128 input tokens, and produced 64 output tokens. I replayed the same controlled transcript in both conditions to keep the inputs comparable.

By turn 20, median time to first token was about 202 ms without prefix caching and 20 ms with it.
What surprised me was that peak KV occupancy was the same with and without prefix caching. In retrospect, both modes needed the same keys and values while processing a turn. Prefix caching kept reusable blocks between turns. The separate occupancy panels show that difference. They measure blocks holding active or retained KV data, not the GPU memory reserved for the cache pool. Freeing a block makes it available to vLLM again; it does not mean returning that memory to the GPU allocator.
Why the reuse happened in steps
I wanted to see how the 64-token blocks affected prefix caching as the shared prefix grew.
I used pairs of 320-token prompts with prefix caching enabled. I changed one token in the second prompt to vary the shared prefix from zero to 256 tokens. For example, changing token 65 left a shared prefix of 64 tokens. Later tokens still matched, but their preceding context differed.
I cleared the cache before each pair, then ran the two requests in order. For the second request, I plotted how many token positions reused cached keys and values, how many needed computation, and the time to first token.

The number of reused tokens increased in steps: a 63-token shared prefix allowed no blocks to be reused; 64 tokens allowed one. Another block became reusable at 128, 192, and 256 shared tokens.
The timing curve was less tidy. Reusing another block didn’t always save the same amount of time. I hadn’t profiled why, so I left that as an observation.
Running out of room
I had seen how KV caching saved work. Next, I wanted to see what happened when requests competed for limited KV cache capacity.
I scheduled 32 requests to arrive 500 ms apart. Each had 4,096 input tokens and generated 512 output tokens, with distinct prefixes. I kept the requests and arrival schedule fixed and varied the KV cache capacity.

I expected enough KV cache capacity to avoid preemptions, and less memory to cause more of them. What surprised me was that they only happened in the middle: at 1.5 GiB, but not at the smaller or larger capacities I tested.
Preemption: interrupting a running request to free resources for other requests.
At the smallest capacities, fewer requests could start, so the others waited while those already running finished. At 1.5 GiB, more requests could start, but their growing KV cache then exhausted the available memory. The scheduler interrupted requests to free blocks and later recomputed their discarded state. At the larger capacities, there was enough room for those requests to continue without interruption.
Some runs had waiting requests without any preemptions. A zero preemption count didn’t mean requests could start promptly.
These capacity thresholds depend on the workload and execution timing: KV grows as requests generate tokens, and completed requests free their blocks.
I also plotted how many requests were running or waiting, alongside their KV cache occupancy:

I then looked at how long individual requests waited for their first token and how long they paused between output tokens:

For request 13, increasing KV cache capacity from 1 GiB to 1.5 GiB reduced time to first token from 2.3 seconds to 258 ms (medians). But its longest pause between output tokens grew from 125 ms to 803 ms (medians). At 1.5 GiB, it also recomputed 1,024 token positions (median).
Request 13 started sooner with more memory, then got interrupted. Looking only at time to first token would have missed the longer pause during its response.
Sharing a prefix, leaving room to grow
On my team at Canva, we have production workloads where the same prompt text is repeated across requests. I’ve seen this at many other companies too. I wanted to understand what happened when those requests shared KV cache blocks and competed for memory.
I compared two batches of 32 requests: one with prompts about different topics, and one with a common 1,024-token context followed by a different 64-token question for each request. All prompts were 1,088 tokens long, and each request generated 256 output tokens.
I used prefix caching with 168 MiB of KV cache capacity and a concurrency limit of 32.

I expected sharing to let more requests run at once and finish the batch sooner. What I learned was what happened after they started: all 32 could begin generating, but there wasn’t enough KV cache capacity for all their responses to grow to completion. Preemptions came in waves as the scheduler made room for requests to continue.
More requests were preempted with the common prefix: half the responses were interrupted to make room for others to finish. The batch finished much sooner, but from the user’s perspective, an early first token could be followed by a long pause.
That made me wonder whether using the memory savings to admit so many requests made their responses more vulnerable to interruptions. Would I want extra KV cache capacity for their growth? I hadn’t tested how much memory each workload would need for a chosen number of concurrent requests. Instead, I tried giving the running requests more room by admitting fewer of them.
I then lowered the concurrency limit from 32 to 16 for the common-prefix workload. Of the 32 submitted requests, only 16 could run at once.

With the lower limit, the second group waited before starting and no requests were preempted. The batch took slightly longer to finish.
The shared prefix needed to be stored only once, but each response needed its own growing KV cache. There was room to start all the requests, but not to let them all finish together. Admitting fewer left enough space for the running requests to grow without interruption.
Here, lowering the concurrency limit moved the waiting before the first token and avoided interruptions during decode. The official vLLM tuning guidance also suggests reducing concurrency or increasing KV cache memory when preemptions are frequent; I could now connect that advice to the interruptions I was seeing.
A new prompt interrupts the rhythm
Next I wanted to see what happened when a new request needed prefill while another request was still decoding.
I started one request, then introduced a second with a different prompt after the first had generated eight tokens. The engine had to prefill the new prompt while continuing to decode the first response. Each request had 4,096 input tokens and generated 32 output tokens, with a step budget of 256 tokens.

The step markers helped me see why the first response slowed down. It kept decoding one token per step, but those steps also included prefill for the new request and took longer. The first request was still making progress, yet its user had to wait longer between tokens.
I then tried ten requests. I scheduled the first five 160 ms apart, giving each prefill time to finish before the next of those requests arrived. I scheduled the next five 60 ms apart, so new prompts arrived before the previous prefill had finished and prefills began to overlap. Each request had 1,536 input tokens and generated 32 output tokens, with a step budget of 256 tokens.

In the first half, I could see inter-token latency spike during each prefill, then fall once it finished. In the second half, prefill work built up: inter-token latency stayed high, and new requests also waited before their prefill could start. Those initial waits are the gray sections of the timelines.
This made me curious about disaggregated prefill: running prefill and decode in separate vLLM instances. Prefill felt like a batch job, while decode felt like an interactive job where every pause reached the user. Giving them separate capacity and settings might improve that experience, even with the cost of transferring the KV cache from the prefill nodes to the decode nodes. I’d like to experiment with that, but for now I only have one GPU in one computer.
My next question was how much prefill work to allow into each step.
The lesson I still don’t entirely trust
The token-budget lesson was where the tutoring process became most complicated.
The expected tradeoff sounded straightforward. Smaller prefill chunks could leave shorter gaps between output tokens for existing requests, while larger chunks could get the new prompt processed sooner. The official vLLM tuning guidance discusses that relationship between the step budget, first-token latency, and inter-token latency.
But our early experiments didn’t make the story that simple.
With a small finite workload, increasing the budget could get the prefill work out of the way sooner and improve average ITL, while causing a few much longer pauses. Even p95 could hide the interruptions I cared about if they made up less than 5% of the measurements.
p95 (95th percentile): a value at or below which roughly 95% of the measurements fall.
I challenged the lesson because the charts weren’t showing the principle as it had been explained. We changed the workloads and metrics several times. Eventually I wanted a simpler view of the actual interruptions, rather than another summary statistic.
To see the individual steps clearly, I used eight requests, each with 16 input tokens and 64 output tokens. Once each had generated eight tokens, a new request arrived with a 4,096-token prompt and a 16-token output limit. I compared step budgets of 128, 256, 512, and 1,024 tokens. The new request is the bottom timeline in each panel, starting later than the others. All eight existing requests kept decoding while its prompt was being prefilled.

This finally gave me a picture I could follow. With a smaller token budget, the existing requests had lower inter-token latency, but prefill of the new prompt took longer. A larger budget finished that prefill sooner, but increased inter-token latency for the requests already decoding.
I also compared time to first token with the longest pause between output tokens (maximum ITL) in a separate, longer-workload experiment. Here, eight requests each had 32 input tokens and 256 output tokens. After each had produced 64 tokens, I introduced a 16,384-token prompt with a 64-token output limit. The budgets ranged from 512 to 16,384 tokens per step.

The L-shaped curve made the tradeoff clearer to me. I could favor a shorter wait for the new request’s first token or shorter pauses for the requests already decoding. Around the bend, roughly 2,048 tokens per step, there seemed to be a sweet spot for this workload: most of the improvement in time to first token without the much longer pauses at higher budgets.
I’m still uneasy about how we arrived at the lesson, though.
I had encouraged the LLM to try experiments until it found conditions that made the principle visible. That can be useful for isolating a mechanism. It can also become a search for a chart that agrees with the explanation we started with.
Playing with vLLM’s priority field
The last group of experiments looked at priority scheduling. By then I had several different constraints in mind: which requests could run, how much work could fit in a step, and how much KV cache capacity was available.
Scheduling priority: a number used to decide which requests get served first.
In vLLM’s priority scheduling policy, smaller numbers mean higher priority; arrival time breaks ties.
I wanted to see which of those priority affected.
Choosing the next waiting request
The first experiment submitted eight equal-length requests, 500 ms apart, with room for two running requests. Requests 4 and 6 had priority 0 and the others priority 1 in the priority-scheduling run. The first-come, first-served comparison used equal priorities.

The waiting order changed. The currently running requests weren’t immediately interrupted when a higher-priority request arrived.
A higher priority didn’t buy a share of every step
The next experiment made the token budget the constraint, rather than the number of running requests.
One request began processing a 1,024-token prompt with a budget of 128 tokens per step. After two steps, a higher-priority request arrived with a 256-token prompt. There was room for a second running request, but the first request was still consuming the token budget.

The higher-priority request waited. Once the first request was decoding, it needed one position per step, leaving 127 positions for the other request’s prefill. Both policies produced the same sequence of allocations in this experiment.
There was room for another request, but no token budget left for its prefill. Raising its priority didn’t reserve a share of each step.
Memory pressure changed which request was interrupted
A separate two-request experiment reduced the KV cache capacity enough that both prompts fit initially, but their growing responses eventually exhausted the available blocks.

Here, priority did affect a running request: it changed which one became the preemption victim when an allocation failed. In the first-come, first-served run, the later request was interrupted. In the priority run, the older, lower-priority request was interrupted.
The event was triggered by memory pressure, not merely by the arrival of a higher-priority request. After the surviving request finished, the victim resumed, rebuilt its missing KV cache blocks, and continued generating. The output token sequences matched across the comparisons.
Waiting can keep getting longer
Finally, I gave one early request lower priority while a finite stream of higher-priority requests continued arriving.

All seven later requests overtook request 3. In the pair shown, its initial queue wait grew from about 1.3 seconds under first-come, first-served to about 8.7 seconds under priority scheduling.
There were no preemptions here. That long interval was waiting to start. The request eventually ran because the stream stopped; the experiment didn’t measure indefinite starvation.
In these experiments, priority changed the waiting order and the choice of which request to preempt when memory ran out. It didn’t guarantee an immediate start or a share of every step.
More questions, and a much better picture
I started because I wanted some hands-on experience. What I enjoyed most was feeling things connect: ideas from books, behavior in the experiments, and questions I hadn’t known to ask.
I’m still at the beginning, but I can already see how much there is to explore. I love the depth and complexity of these systems, and I’d like to spend more of my time working on them. If one GPU gives me this much to investigate, I can only imagine how interesting the problems become at scale.
I’m excited to keep going.

I still haven’t found a good