LLM Inference Serving: PagedAttention, Continuous Batching, Speculative Decoding, and Quantization

Deploying autoregressive Large Language Models (LLMs) in production environments presents extreme engineering challenges. Unlike traditional deep learning models where execution is compute-bound, autoregressive text generation operates in two distinct phases:

Prefill Phase (Prompt Evaluation): Compute-bound GEMM operations over all input tokens in parallel.
Decode Phase (Token-by-Token Generation): Strictly memory-bandwidth bound GEMV operations. Each generated token requires reading every previous token's Key and Value attention activations (KV Cache) from High Bandwidth Memory (HBM) into SRAM.

This guide details the state-of-the-art serving stack: PagedAttention virtual memory management, Continuous Batching (Orca), Speculative Decoding, and modern FP8 / AWQ Quantization methods.

1. Quick-Reference: Serving Engine Optimization Stack

+-----------------------------------------------------------------------------------------------------------------------+
|                                           INFERENCE SERVING OPTIMIZATIONS                                             |
+-----------------------------------------------------------------------------------------------------------------------+
| Technique              | Primary Bottleneck Solved              | Throughput / Latency Impact| Hardware Requirement   |
+------------------------+----------------------------------------+----------------------------+------------------------+
| PagedAttention         | KV-cache memory fragmentation          | 2x - 4x Concurrency Uplift | Any Modern GPU (CUDA)  |
| Continuous Batching    | Static batching GPU idle bubbles       | 5x - 10x Throughput Uplift | Dynamic scheduler (vLLM)|
| Speculative Decoding   | Memory-bandwidth latency bound         | 2x - 3x Time-To-First-Tok  | Draft Model / Spec Head|
| FP8 / INT4 Quantization| HBM capacity & memory bus saturation   | 2x Lower HBM footprint     | NVIDIA Ada / Hopper    |
| Chunked Prefill        | Prefill inter-token latency spikes     | Smooth P99 Decode Latency  | TensorRT-LLM / vLLM    |
+-----------------------------------------------------------------------------------------------------------------------+

2. KV-Cache Memory Virtualization: PagedAttention

In naive serving systems, GPU memory for the KV-cache is allocated contiguously based on the maximum possible sequence length (e.g., 8,192 tokens). Because actual request lengths vary wildly, $60\% - 80\%$ of GPU memory is wasted on internal and external fragmentation, capping server concurrency.

PagedAttention Architecture

Inspired by virtual memory paging in operating systems, PagedAttention (Kwon et al., 2023) breaks the KV cache into fixed-size physical blocks (e.g., 16 or 32 tokens):

Logical KV Cache (Sequence 1):
[ Block 0: Tok 0-15 ] -> [ Block 1: Tok 16-31 ] -> [ Block 2: Tok 32-47 ]
         |                        |                         |
         v                        v                         v
Physical GPU HBM Blocks:
[ Phys Frame #42 ]       [ Phys Frame #108 ]       [ Phys Frame #15 ]

Zero Memory Waste: Memory is allocated dynamically one page at a time during autoregressive generation.
Copy-on-Write for Parallel Sampling: When performing beam search or generating $N$ parallel completions for the same prompt, sequences share physical prompt pages until branching occurs, reducing prompt memory by up to $95\%$ .

3. Continuous (Iteration-Level) Batching

Traditional batching waits for an entire batch of requests to reach their termination token (<|endoftext|>). If Request A finishes in 20 tokens while Request B requires 800 tokens, the GPU spends 780 iterations computing padding for Request A.

Static Batching (Massive Waste):
Req A: [ Tok 1 ][ Tok 20 ][ PAD ][ PAD ][ PAD ][ PAD ][ PAD ] ... (Idle)
Req B: [ Tok 1 ][ Tok 20 ][ Tok 100 ][ Tok 400 ][ Tok 800 ] ... (Running)

Continuous Batching (Dynamic Iteration Scheduler):
Iteration t:   [ Req A (Tok 1)  ][ Req B (Tok 20) ][ Req C (Tok 5)  ]
Iteration t+1: [ Req D (New!)   ][ Req B (Tok 21) ][ Req C (Tok 6)  ]  <-- Req A evicts clean!

4. Speculative Decoding Acceleration

Autoregressive decoding generates exactly one token per memory-bandwidth pass over the model weights:

ext{Latency per token} pprox rac{ ext{Model Weights Size (GB)}}{ ext{GPU Memory Bandwidth (GB/s)}}

The Draft-Verify Protocol

Speculative decoding pairs a large Target Model ( $M_T$ , e.g., Llama-70B) with a tiny Draft Model ( $M_D$ , e.g., Llama-1B):

Speculation: $M_D$ rapidly generates $K = 5$ candidate tokens autoregressively with near-zero latency.
Parallel Verification: $M_T$ evaluates all $K$ candidate tokens in a single forward prefill pass.
Rejection Sampling: $M_T$ accepts the prefix of tokens matching its probability distribution. If tokens $1, 2, 3$ match, all 3 are committed simultaneously, yielding a $2.5 imes - 3 imes$ latency reduction with mathematically identical mathematical output distributions.

References

Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. ACM SOSP '23.
Yu, G. I., et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models. USENIX OSDI '22.
Leviathan, Y., Kalman, M., & Matias, Y. (2023). Fast Inference from Transformers via Speculative Decoding. ICML 2023.