While modern foundation models advertise context windows exceeding 128,000 to 1,000,000 tokens, treating the context window as an infinite scratchpad leads to severe performance degradation: exponential latency growth, skyrocketing inference costs, and the "Lost in the Middle" attention decay phenomenon.
Token Budgeting is the rigorous discipline of statically and dynamically allocating, monitoring, and evicting token allocations across the prompt envelope. This guide details formal token reservation models, priority-tiered context eviction, semantic compression, and prompt caching economics.
+-----------------------------------------------------------------------------------------------------------------------+
| TOKEN BUDGET ALLOCATION (128K Envelope) |
+-----------------------------------------------------------------------------------------------------------------------+
| Segment | Allocation Target | Priority Tier | Mutation Invariant | Eviction Strategy |
+-------------------------+-------------------+---------------+------------------------+--------------------------------+
| System Instructions | 1,500 - 3,000 tok | Tier 0 (High) | Static & Immutable | Never evict (Cached prefix) |
| Schema & Tool Specs | 2,000 - 5,000 tok | Tier 0 (High) | Static / Deterministic | Never evict (Cached prefix) |
| Retrieved RAG Knowledge | 15,000 - 30,000 tok| Tier 1 (Med) | Dynamic per turn | Re-rank & Top-K Truncate |
| Episodic Chat History | 10,000 - 20,000 tok| Tier 2 (Low) | Expanding FIFO | Summarize / Sliding Window |
| Scratchpad / Working Mem| 4,000 - 8,000 tok | Tier 1 (Med) | Dynamic state machine | AST pruner / Deduplication |
| Generation Reserve | 4,096 - 8,192 tok | Tier 0 (High) | Unoccupied Headroom | Hard boundary limit |
+-----------------------------------------------------------------------------------------------------------------------+
Let C_{ ext{max}} be the maximum context window supported by the target model, and G_{ ext{res}} be the tokens reserved for output generation. The total input tokens across all segments S_i must strictly satisfy:
If an expanding conversation history threatens this invariant, dynamic eviction must trigger before the API payload is dispatched.
+----------------------------------+
| Inflow: New Message / Tool Input |
+-----------------+----------------+
|
v
+----------------------------------+
| Check Invariant: |
| Total Tokens > Budget Limit? |
+-----------------+----------------+
|
+---------------+---------------+
| |
[ YES ] [ NO ]
| |
v v
+---------------------------+ +-------------------+
| Tier 2 Eviction: | | Dispatch Payload |
| Summarize Oldest History | +-------------------+
+-------------+-------------+
|
v
+---------------------------+
| Still Over Budget? |
+-------------+-------------+
|
+--------+--------+
| |
[ YES ] [ NO ]
| |
v v
+---------------------------+ +-------------------+
| Tier 1 Eviction: | | Dispatch Payload |
| Prune Low-Scoring RAG ctx | +-------------------+
+---------------------------+
Rather than dropping old conversation turns entirely (which causes agent amnesia), conversations use a dual-state buffer:
[Running Conversation Summary: ...]).When tools return large JSON API responses or source code files, naive text slicing breaks braces and syntactic trees. A semantic AST pruner strips non-essential fields:
created_at, etag, http_headers) are stripped.Modern serving platforms (Anthropic Prompt Caching, OpenAI Prefix Caching, DeepSeek) leverage KV-cache reuse across requests that share identical prefix token sequences.
Where P_{ ext{cached}} pprox 0.10 imes P_{ ext{uncached}} (a 90\% discount on input tokens).
To maximize prompt caching hit rates, prompt templates must be ordered strictly from most static to most dynamic:
[CACHE ANCHOR 1] System Prompt & Personas.[CACHE ANCHOR 2] OpenAPI Tool Specifications & Schemas.[CACHE ANCHOR 3] Gold standard Few-Shot demonstration pairs.[DYNAMIC] User query, current dynamic tool results, and real-time state.