How LLM Inference Works: From Prompt to Generated Tokens

A useful way to reason about LLM inference is to separate the mathematical loop from the system that operates it. The model repeatedly scores the next token. The serving system decides which requests advance, where their state lives, and how to keep the accelerator busy without making interactive users wait behind a long prompt.

The request lifecycle

A request arrives as text plus generation parameters: maximum output length, sampling policy, stop conditions, and often a large shared system prompt. The tokenizer converts text into token IDs. Those IDs are embedded, passed through every transformer layer, and finally projected to vocabulary-sized scores called logits.

The model does not return an answer from that one operation. The sampler selects one token from the logits—greedy selection, temperature, top-p, top-k, or another policy—and the selected token is appended to the sequence. That new token becomes the next model input. The loop ends only when the engine reaches an end-of-sequence/end-of-turn token, a configured stop condition, or the output cap.

What a transformer retains between tokens

Within each decoder layer, self-attention projects a token’s hidden state into a query (Q), key (K), and value (V). The query at the current position compares with keys from positions it is allowed to see; the resulting weights combine the corresponding values. Causal masking prevents access to future positions.

For generation, previously computed keys and values are exactly the information later attention operations need. Recomputing them at every step would revisit the entire conversation for every new token. Instead, the engine persists per-layer K/V tensors in a KV cache. This is attention state, not a semantic memory or a compressed summary of the conversation.

Prefill: establish the context

Prefill is the initial forward pass for a request. It processes the complete input sequence, writes K/V entries for all prompt positions, and produces logits for the final prompt position. The first generated token is selected from those logits at the end of prefill.

This phase is the primary contributor to time to first token (TTFT), along with queueing, tokenization, and network overhead. It has abundant token-level work, so it can often use the accelerator efficiently. But very long prompts can still dominate a server’s schedule if they are admitted without controls.

Decode: advance one position

Once prefill samples the first completion token, decode begins. A standard decode step runs the most recently generated token through the model. Its query attends to cached K/V from the prompt and earlier completion tokens. The new token’s K/V entries are appended to the cache, logits are produced, and the sampler chooses the following token.

One decode step means one next-token decision per sequence. It should not be confused with one GPU kernel: a model forward pass includes many operations, and a serving engine normally advances many active sequences together. Decode is commonly constrained by repeatedly reading model weights and growing cache data, so its performance profile differs materially from prefill.

KV-cache access: the critical path

At every layer during decode, the newest query reads the historical K/V entries for that layer; the layer then writes one K/V entry for the newest token. The cache therefore grows with context length and is replicated across transformer layers. Its footprint is driven by the number of layers, KV heads, head dimension, token count, precision, and active sequences.

The cache eliminates redundant compute, but it turns memory capacity and bandwidth into first-class serving constraints. A GPU can be compute-capable yet unable to admit more work because its cache pool is full. Conversely, an oversized maximum context length can reserve or pressure capacity that the request never consumes. This is why cache allocation policy is a throughput decision, not an implementation footnote.

The model weights determine what can be generated. KV-cache management often determines how many users can be served at the latency they expect.

What vLLM changes at the serving layer

A production engine has to schedule a mix of requests with different prompt lengths, completion lengths, and arrival times. vLLM’s core contribution is to make that dynamic workload tractable rather than treating inference as a single fixed batch.

Continuous batching

Rather than wait for a batch to finish before starting another, the scheduler can remove completed sequences and admit new work at subsequent iterations. This minimizes stranded capacity when one request finishes far earlier than its peers.

PagedAttention and block-based allocation

KV cache grows dynamically, while conventional contiguous allocations invite fragmentation and over-reservation. PagedAttention organizes cache storage into fixed-size blocks and maps each sequence’s logical token positions to physical blocks. The engine allocates blocks as a sequence grows and reclaims them when it finishes. This makes heterogeneous request lengths much easier to pack into GPU memory.

Chunked prefill and prefix reuse

Chunked prefill breaks a long prompt into schedulable units so it can coexist with latency-sensitive decode work. Prefix caching can avoid recomputing compatible cache blocks when requests share exactly the same tokenized prefix—an important win for common system prompts, RAG scaffolding, or repeated document context. A cache hit is not based on similar text; it depends on matching tokens and compatible cache state.

Disaggregating prefill and decode

At larger scale, prefill and decode may run on distinct worker pools. This allows independent capacity planning for compute-heavy prompt ingestion and latency-sensitive token streaming. The trade-off is explicit: the system must transfer or make available the KV state between stages, then absorb the extra orchestration and failure modes. It is a tool for a measured bottleneck, not a default architecture.

What to measure in production

Token throughput alone can obscure user experience. Track TTFT separately from inter-token latency (ITL); a system can produce an excellent aggregate tokens-per-second number while still queueing interactive prompts too long. Also observe scheduler queue time, cache utilization and eviction/preemption behavior, prefix-cache hit rate, active sequence count, prompt and output length distributions, and tail latency by traffic class.

The practical mental model is compact: prefill builds the context and selects token one; decode reads old K/V, appends new K/V, and selects the next token; the serving system controls how efficiently that loop runs across many requests. Once those boundaries are explicit, performance discussions about batching, memory sizing, and cache reuse become engineering decisions instead of magic.