Training models is expensive but bounded; inference is unbounded — cost scales with usage. For successful products, inference cost dominates total ML spend.
This page covers practical levers for reducing inference cost.
Inference cost ≈ (compute per request) × (requests) / (compute per dollar)
You can attack any term:
The cheapest optimization: use a smaller model.
For LLMs:
For traditional ML:
Always test: many tasks don't need the strongest model.
Reduce numerical precision of weights and activations.
Quality impact is usually minimal for FP16 and INT8. INT4 needs care.
Tools: bitsandbytes, GPTQ, AWQ, llama.cpp.
Train a smaller "student" model to mimic a larger "teacher."
Common approach:
Works well for many tasks. Requires the teacher and good training infrastructure.
Remove weights that contribute little. Unstructured pruning saves memory but rarely speed; structured pruning (entire heads, layers) speeds inference.
Process multiple requests together. Modern GPUs are heavily underutilized at batch size 1.
Static batching: collect N requests, run together. Adds latency.
Dynamic batching: form batches at the inference engine. Used in vLLM, TGI.
Continuous batching: especially for autoregressive models, allows joining/leaving batches mid-generation. Major throughput gain.
Identical request? Return cached response.
Hash the request (or relevant parts) as cache key.
Works best for deterministic outputs.
For LLMs: cache the prefix computation. New requests reusing the prefix skip recomputation.
Anthropic, OpenAI, and others now offer this directly.
Major savings for long system prompts or RAG with repeated context.
See PromptCaching.
Cache based on semantic similarity, not exact match. "What's the capital of France?" and "Capital of France?" share an answer.
Use embeddings + nearest neighbor lookup.
Risk: false-positive matches return wrong answers.
For LLMs: a small "draft" model proposes tokens; the large model verifies in parallel.
Net effect: same outputs, fewer large-model forward passes. 2-3x speedup typical.
Use multiple models tiered by capability:
Routing logic ranges from rules to learned classifiers.
A100, H100: high throughput, expensive A10, L4: mid-tier T4: budget, still capable
For LLMs, memory bandwidth often matters more than FLOPs.
For small models or non-latency-sensitive workloads, CPU inference is often cheaper.
Modern CPUs with AVX-512 / AMX can run quantized LLMs surprisingly well.
See CPUInference.
TPUs, Inferentia, Groq — specialized hardware can offer better cost/performance for some workloads.
For batch inference: 50-90% cheaper but can be interrupted.
If you don't need real-time:
Many use cases don't actually need real-time.
API providers offer:
Self-hosted:
The breakeven varies. Many teams underestimate self-hosting ops cost.
Without metrics, optimization is guesswork. Track:
Hand-tuning quantization for a model you'll replace next month wastes effort.
Caching often saves 50%+ with little engineering.
Complex routing systems can cost more in maintenance than they save in inference.
Quality regressions from quantization, distillation, or routing can be subtle.
The strongest model is rarely needed. Test smaller models.