The "Attention Bottleneck" is the primary constraint in modern Agentic AI. As context windows grow to 1M+ tokens, the signal-to-noise ratio drops, costs spike, and model performance on "needle-in-a-haystack" tasks degrades. Context Compression is the discipline of reducing raw token count while preserving semantic signal.
Selective context techniques use a smaller, faster "compressor" model to identify and remove low-information tokens from the prompt before it reaches the target LLM.
The Key-Value (KV) cache stores the attention states for all tokens in a conversation. In long-running agent interactions, this cache becomes the primary memory bottleneck.
Standard windowed attention fails because the model loses "attention sinks" (often the first few tokens in a prompt) that are critical for the softmax distribution. StreamingLLM preserves the first N tokens (the sink) and the most recent L tokens (the local context), allowing for infinite conversation length without the quadratic memory cost of full attention.
H2O prunes the KV cache by identifying and keeping only the "heavy hitter" tokens—those that consistently receive high attention weights. This reduces the cache footprint by up to 80% without significantly impacting long-range recall.
For tasks that exceed even large context windows (e.g., analyzing a library of 100 books), agents use a hierarchical approach:
A "Knowledge Distillation" approach where a long context is passed through a model, which then generates a Dense Knowledge State. This state—a list of specific facts, constraints, and observations—is used as the "Short-Term Memory" in subsequent turns, effectively "forgetting" the original verbose text while retaining its meaning.