Efficient Context Passing: Semantic Compression, AST Pruning, and Dynamic Memory Scratchpads

As autonomous LLM agents execute multi-step tool interactions, the token context window expands rapidly with bulky tool outputs: raw JSON API responses, full source code files, database schemas, and tabular search results.

Passing bloated, unpruned context directly into subsequent agent turns triggers two fatal bottlenecks:

1. Exponential Latency and Cost Growth: Every additional 10,000 tokens of context increases prefill TTFT and degrades serving throughput.
2. Context Dilution: The "Lost in the Middle" attention decay causes models to miss subtle instructions when buried under massive payload noise.

Efficient Context Passing implements algorithmic filtering, Abstract Syntax Tree (AST) pruning, and incremental semantic compression before injecting tool results into working memory.

---

1. Quick-Reference: Context Compression Strategies

+-----------------------------------------------------------------------------------------------------------------------+
|                                           CONTEXT COMPRESSION BENCHMARK                                               |
+-----------------------------------------------------------------------------------------------------------------------+
| Strategy               | Compression Ratio                      | Latency Overhead           | Information Loss Risk  |
+------------------------+----------------------------------------+----------------------------+------------------------+
| Semantic AST Pruning   | 60% - 80% Reduction                    | < 5ms (Local Python AST)   | Zero (Preserves API)   |
| JSON Projection Filter | 70% - 90% Reduction                    | < 2ms (JmesPath / JQ)      | Zero (Explicit fields) |
| LLM Extractive Summary | 80% - 95% Reduction                    | 200 - 500ms (Draft model)  | Minor detail loss      |
| AutoCompressor / Soft  | 90% Reduction                          | GPU forward pass           | Moderate hallucination |
+-----------------------------------------------------------------------------------------------------------------------+

---

2. AST Pruning for Code Intelligence

When an agent reads a 2,000-line source code file to understand how a single function operates, injecting the entire file consumes valuable context tokens.

The Semantic AST Skeleton Invariant

A local tree-sitter or AST parser strips private implementation bodies while preserving top-level interfaces, class signatures, and type annotations:

# Raw Input (800 Tokens):
class OrderProcessor:
    def __init__(self, db: DatabaseConnection):
        self.db = db
        # 50 lines of internal setup...
    def execute_settlement(self, order_id: str) -> SettlementResult:
        # 120 lines of complex transaction math and locking...

# AST Skeletonized Output (65 Tokens - 92% Reduction):
class OrderProcessor:
    def __init__(self, db: DatabaseConnection): ...
    def execute_settlement(self, order_id: str) -> SettlementResult: ...

---

References

1. Jiang, H., et al. (2023). LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. EMNLP 2023.
2. Chevalier, A., et al. (2023). Adapting Language Models to Compress Contexts. arXiv:2305.14788.
3. Anthropic. (2024). Building Effective Agents: Context Management and Tool Result Filtering.