Standard observability — traces, logs, metrics — covers whether your LLM system is running. AI observability adds whether it's behaving. The question shifts from "is the API up" (easy) to "are responses still accurate, faithful, calibrated, fast, and cheap" (harder, more important).
This page is what to add on top of standard observability for LLM systems specifically. For the standard stuff, see AgentObservability; for the eval methodology, AgentTesting.
| Signal | Question | Source |
|---|---|---|
| Cost & latency | Is each call efficient? | LLM call telemetry |
| Quality | Are responses still as good? | Eval-in-prod, sampled human labels |
| Drift | Has the input distribution changed? | Embedding drift + statistical tests |
| Safety | Are we generating things we shouldn't? | Output classifiers, guardrails |
Standard APM covers the first; the other three are LLM-specific and are usually neglected in v1.
The basics:
Dashboards:
Alerts:
These are easy to implement and almost always worth the effort. See LlmTokenEconomicsAndPricing for the cost side.
LLM output quality is hard to measure automatically. Three complementary approaches:
A small percentage (1-5%) of production outputs go to human reviewers who grade them on a fixed rubric. Slow, expensive, gold-standard.
Practical setup:
Most teams under-invest here because it's expensive. It's also the only ground-truth source of "is the system actually good." Budget for it.
A second LLM call grades each production output. Cheap (compared to human review), reasonably correlated with human judgement once calibrated.
Caveats from [LlmEvaluationMetrics]: judge bias, requires calibration against humans, drifts when the judge model updates.
Practical use: judge every Nth response (1-10%); aggregate scores by task type, time window, prompt version. The trend matters more than absolute values.
Your fixed eval set (AgentTesting) doesn't have to be eval-only. Run it nightly on production-deployed prompts. Catches regressions immediately rather than waiting for users to notice.
Set up: eval task fixtures stored as JSON; nightly cron triggers rollouts against the live system; results sent to the same dashboard as production telemetry; alert on drop > 5%.
Cost: $1-50 per nightly run depending on eval set size and model. Compare to the cost of a quality regression in production for a day before someone noticed.
Production input distribution drifts. Users ask new things; data sources change; competitors launch features that shift behaviour. Drift detection surfaces this before it becomes a quality regression.
Two forms:
Compute embeddings of incoming queries (and/or retrieved docs). Compare distribution to a baseline window.
Methods:
Implement as a daily batch job; alert when drift score exceeds threshold; investigate (new use case? broken upstream filter? attack pattern?).
For specific high-value features, track distributions of:
Tools like Arize Phoenix, Fiddler, and Whylogs (open source) handle this. For small teams, a custom batch job is fine.
Most production LLM systems have output guardrails — classifiers that flag inappropriate, off-policy, or dangerous outputs. Track:
Sudden spikes in trip rate often signal:
For safety-critical deployments, also track:
Build these and monitor them. Without them, problems hide.
Most teams have #1 and call it done. Adding 2-6 catches the issues #1 misses.
Pager-worthy:
Ticket-worthy (not pager):
| Tool | Strengths | When to pick |
|---|---|---|
| Langfuse | Open source, model-agnostic, traces + evals + drift | Default starting point |
| LangSmith | Polished UI, deep LangChain integration | LangChain users |
| Arize Phoenix | Strong on drift detection, eval workflow | Drift-heavy use cases |
| Fiddler | Commercial AI observability, ML-ops focus | Larger orgs, traditional ML alongside LLM |
| Helicone | Lightweight, easy onboarding | Smaller teams |
| OpenLLMetry + your existing stack | OpenTelemetry-native | Teams with mature OTel |
Many teams start with Langfuse self-hosted; graduate to commercial tools when the operational overhead exceeds the cost.
Storing every prompt and completion captures whatever users put in. Compliance implications:
See AiDataPrivacyAndCompliance.
Agents add structure that raw LLM traces don't capture:
The standard tooling above is catching up; expect agentic-specific instrumentation to be a first-class feature in observability tools by late 2026.