Training a model is a project; serving it is a system. Production inference serving has its own discipline distinct from training.
This page covers the practical concerns.
A production inference service must:
PyTorch's official serving framework. Decent default for PyTorch models.
Mature, performant. Strong for TF models.
Multi-framework. Excellent batching and GPU utilization. Industry standard for GPU serving.
Python-native, flexible composition. Good for complex pipelines.
LLM-specific. Continuous batching and PagedAttention give major throughput gains.
Quick to build; loses out on optimizations like batching.
Choose based on:
One request → one inference. Wastes hardware.
Wait for N requests, then batch. Adds latency.
Form batches based on queue + max wait time. Tunable latency-throughput tradeoff.
Requests can join/leave the batch mid-generation. Major throughput improvement for autoregressive models.
vLLM and TGI implement this.
Define p50, p95, p99 latency targets.
Components:
Profile each. Common surprises:
Scale up to handle load; scale down to save money.
GPUs and large models load slowly. Cold start can be 30s+.
Mitigations:
For non-critical, interruption-tolerant workloads, spot instances cut cost dramatically.
Multiple models share resources. Saves money for low-traffic models.
Output of one model feeds another. Common for vision + classification, retrieval + reranking.
Triton's ensemble feature; Ray Serve composition.
Choose model per request based on input characteristics.
Two environments; switch traffic atomically.
Send small % to new version. Monitor metrics. Increase gradually.
Run new version in parallel without serving its responses. Compare quality.
Send different traffic to different versions. Measure business impact.
For ML models, output drift between versions is common. Shadow testing catches surprises.
p50, p95, p99 — track all three.
QPS over time. Detect traffic anomalies.
Inference errors, timeouts, OOM.
This is unique to ML:
Cost per request, by model. Surprising things happen.
Identical input → cached output. Effective for queries with repetition.
For pipelines with embeddings, cache by content hash.
Cache prefix computations. Major savings for system prompts and RAG.
Multi-tenant models can interfere:
Mitigations:
Model down? Fall back to:
If error rate spikes, stop sending traffic. Lets the system recover.
Retry transient failures. Avoid retry storms.
Per-request timeouts prevent slow requests from blocking workers.
GPU "utilization" metric is misleading. A GPU at 100% utilization may be memory-bandwidth-bound.
Better: tokens/second, requests/second.
Out-of-memory is the most common failure mode. Monitor headroom.
For small models, CPU may be cheaper. See CPUInference.
Tokenization, preprocessing, network — often the bottleneck.
Without metrics, you can't optimize.
p99 matters for user experience even when p50 looks fine.
Autoscaling that creates 30s of timeouts.
New model deploys and quality silently drops.
One slow request blocks an entire batch. Mitigate with timeouts and dynamic batching.
Stages:
Most teams under-invest in stages 3-4.
Hosted inference services (Replicate, Together, Modal, AWS SageMaker, Vertex AI) handle a lot of this.
For small teams, hosted is often the right call until cost becomes prohibitive.