Traditional application observability was built around a simple mental model: Your code runs, metrics come out and when something breaks, the logs tell you why. Large language models (LLMs) break that model in ways that are not obvious until you have shipped one and watched it misbehave in production.
An LLM-powered application can be up, serving requests, returning HTTP 200 responses and still be failing catastrophically — producing hallucinated content, silently truncating outputs, drifting toward unsafe responses or degrading in quality because the model provider quietly updated the underlying checkpoint. Standard infrastructure monitoring tells you nothing about any of this.
Over the past two years, I have built and operated a production LLM application that processes tens of thousands of requests daily. The observability stack I run today is substantially different from what I started with, and most of the changes came from incidents I could not have anticipated without experience. This article shares the architecture and tooling that actually work — not the theoretical monitoring stack, but the one running right now.
Why LLM Observability is Different
Conventional APM tools track latency, error rates and throughput. These are necessary but not sufficient for LLM systems. The failure modes that matter most are semantic, not structural.
A conventional API returns a well-typed response or throws an exception. An LLM returns a string. That string might be exactly what you asked for, a plausible-sounding but incorrect answer, an output in the wrong format that breaks a downstream parser, a response that violates content policies or a truncated completion because the context window was exceeded silently. None of these show up as an error in standard monitoring.
Four distinct problem classes in LLM production require dedicated observability signals, and each needs a different instrumentation approach.
The Four Problem Classes
Problem Class 1: Quality Drift
The output quality of your application degrades over time, usually for one of three reasons: The provider updated the underlying model, your prompt encountered input distributions it was not tested against or a downstream change altered the context your LLM receives. Quality drift is invisible without a baseline to compare against and a measurement approach that is not purely human-reviewed.
Catching this requires automated evaluation: A set of representative inputs with expected outputs or rubrics, run on a schedule against production traffic samples, with scores tracked over time. The signal is not a single evaluation run but the trend line. A sudden drop in average score on the evaluation set is an early indicator of a problem before users report it.
Problem Class 2: Prompt Failures
Prompts are code. They can fail due to inputs that the prompt was not designed to handle: Unexpected languages, edge-case formatting, adversarial inputs or simply unusually long inputs that cause important context to be dropped. Prompt failures often look like partial successes: The model returns something, but it is wrong in a way that requires domain knowledge to recognize.
Catching these requires logging full prompt-response pairs (with privacy handling for PII), tagging the output with structured metadata about the result type and sampling enough traffic for human review to catch failure patterns. Pattern-based prompt failures often affect a narrow slice of inputs and will not show up in aggregate metrics.
Problem Class 3: Cost Anomalies
LLM API costs are token-based and can spike unexpectedly. A bug that causes your application to include a large context document in every request, a prompt template that grew too large or a change that triggers a multi-turn conversation where a single-turn was expected can multiply your token consumption by 10x overnight. By the time the billing statement arrives, the damage is done.
Cost observability requires token-level tracking per request type, not just total spend. You need to know the average token count for each workflow, see that number in real time and alert on anomalies before they accumulate into a large bill.
Problem Class 4: Latency Degradation
LLM API latency is variable in ways that server-side APIs are not. Time to first token and total generation time depend on server load at the provider, prompt length, output length and model family. Latency can degrade without any change on your side and without the provider posting a status update. Monitoring p50 latency is insufficient — LLM latency distributions are fat-tailed, and p95 and p99 are where user experience breaks down.
The Minimal Instrumentation Stack
You do not need a commercial observability platform to get meaningful LLM monitoring. The following instrumentation can be built into any application in a day, and it covers 80% of the failure modes that matter in practice.
Step 1: Log Every LLM Call as a Structured Event
{
“request_id”: “abc123”,
“timestamp”: “2026-06-19T14:00:00Z”,
“model”: “gpt-4o”,
“workflow”: “document_summarization”,
“prompt_tokens”: 1842,
“completion_tokens”: 312,
“latency_ms”: 2240,
“ttfb_ms”: 480,
“finish_reason”: “stop”,
“cost_usd”: 0.00318,
“output_quality_score”: null
}
This structured log record is the foundation of everything else. It gives you the data to compute per-workflow cost trends, latency distributions and finish reason breakdowns (how often does your model hit the token limit instead of reaching a natural stop?). Finish reason analysis alone will surface truncation issues that users notice but that look fine in error rate dashboards.
Step 2: Track Finish Reasons Explicitly
The finish_reason field is one of the most underused signals in LLM monitoring. A high rate of ‘length’ finish reasons means your model is being cut off before completing its output. This is almost always a problem — it means users are receiving partial results — but it registers as a successful API call in every standard monitoring system.
Alert when the length finish reason rate for any workflow exceeds 5% of requests. Investigate when it exceeds 2%. In most cases, the fix is adjusting max_tokens, reformulating the prompt to produce more concise output or implementing chunked generation.
Step 3: Per-Workflow Token Budget Alerting
BUDGET_ALERTS = {
“document_summarization”: {“prompt_tokens_p95”: 3000, “total_tokens_p95”: 3500},
“slide_generation”: {“prompt_tokens_p95”: 5000, “total_tokens_p95”: 5800},
“qa_response”: {“prompt_tokens_p95”: 800, “total_tokens_p95”: 1200},
}
Define expected token ranges per workflow based on your baseline measurements. Alert when p95 token count for a workflow exceeds the budget by more than 20%. This catches context bloat early and surfaces regressions from prompt changes that inflate token usage.
Automated Quality Evaluation
Logs and metrics tell you about the mechanics of your LLM calls. Automated evaluation tells you whether the outputs are actually good. Setting this up is the highest-leverage observability investment you can make for a production LLM application.
The practical approach for most production systems is not model-graded evaluation running on every request — that doubles your inference cost. It is a two-tier system: Lightweight rule-based checks on every request, and deeper model-graded evaluation on a sample.
Tier 1: Rule-Based Checks on Every Response
- Format Validation: Does the output match the expected structure (valid JSON, correct number of sections, presence of required fields)? This catches a surprisingly large proportion of prompt failures at zero additional LLM cost.
- Length Sanity: Is the output within the expected character or word range for this workflow? Outputs that are dramatically shorter than expected indicate truncation or refusal; outputs dramatically longer indicate prompt leakage or runaway generation.
- Content Safety Signals: Apply a lightweight classifier for harmful content categories. This is especially important if your application accepts arbitrary user input as part of the prompt.
Tier 2: Model-Graded Evaluation on a 5% Sample
Sample 5% of production traffic (or more if volume allows) and run a secondary evaluation prompt against the original request and output. The evaluation prompt asks a smaller, cheaper model to score the output on dimensions relevant to your application: Accuracy, completeness, format adherence, tone. Store the scores alongside the original log record.
The power of this approach is the trend line. A stable average score that suddenly drops is a high-confidence signal that something changed — a model update, a prompt regression or a shift in input distribution. Without the trend line, you are flying blind until users complain.
Distributed Tracing for Multi-Step Pipelines
Various production LLM applications are not single-model calls — they are pipelines: Retrieve context, summarize, generate, validate and sometimes retry. Standard metrics do not show you where time is spent in the pipeline or where quality degrades.
Applying distributed tracing to LLM pipelines — using OpenTelemetry (OTel) spans with LLM-specific attributes — gives you visibility into the full execution path. The key convention is to create a span for each LLM call with attributes that match the structured log format above. A trace for a multi-step pipeline will then show you the individual latency of the retrieval step, the summarization call and the generation call, along with the token counts and quality scores at each stage.
The OTel GenAI semantic conventions (available as of 2025) provide a standard schema for LLM spans. Using the standard schema means your traces are compatible with any OTel-native observability back end without custom attribute parsing.
Provider-Level Monitoring
If you use more than one LLM provider — or are considering it — provider-level observability becomes important. You need to see error rates, latency and cost not just in aggregate but broken down by provider, so you can compare performance and catch provider-level degradation before it affects users.
Key Provider-Level Metrics to Track Separately:
- Error rate by error type (rate limit, timeout, content filter, API error) per provider
- P50, P95 and P99 latency per provider per model family
- Time to first token (TTFB) per provider (this is what users feel in streaming applications)
- Cost per 1,000 requests per provider at your actual usage mix (not the listed price per token)
Provider health dashboards are most valuable during incidents. When a provider has a partial outage, the health dashboard tells you whether to fail over and how severe the degradation is — information that often arrives before the provider posts a status update.
Alerting That Doesn’t Cry Wolf
LLM observability generates a lot of data. The risk is alert fatigue: Too many low-signal alerts that engineers learn to ignore. The following alert tiers have worked well in practice.
Tier 1 — Immediate Page-Level Alerts:
- Error rate on any workflow exceeds 10% over a 5-minute window
- Provider latency P99 exceeds 30 seconds sustained for 3+ minutes
- Estimated hourly cost exceeds 3x the expected rate
Tier 2 — Slack/Email Alerts (Investigate Within an Hour):
- Length finish reason rate exceeds 5% for any workflow
- Average quality score drops more than 15% from the 7-day rolling average
- P95 prompt token count exceeds budget threshold for any workflow
Tier 3 — Daily Digest Only:
- Token usage and cost by workflow versus prior week
- Quality score trend by workflow
- Finish reason distribution summary
Tier 3 metrics inform product and engineering decisions but do not need immediate action. Keeping them in a daily digest rather than active alerts reduces noise and keeps the on-call engineer focused on signals that actually require immediate response.
Practical Starting Point for Teams New to LLM Observability
If you are instrumenting an LLM application for the first time, the order that maximizes early value with minimal implementation cost:
- Week 1: Add structured logging for every LLM call (model, workflow, tokens, latency, finish reason, estimated cost). Route to your existing logging infrastructure. This alone surfaces most structural problems.
- Week 2: Add format validation as a rule-based check in your LLM call wrapper. Alert on validation failure rate exceeding 3%. This catches prompt failures that look like successes.
- Week 3: Build a simple dashboard that shows error rate, P95 latency and finish reason distribution per workflow over the last seven days. This gives you the baseline you need to detect regressions.
- Month 2: Add 5% sample quality evaluation. Pick the most important workflow, define a rubric, implement a model-graded evaluator and start tracking the score trend. This is where the observability stack becomes genuinely predictive rather than reactive.
Each step can be shipped independently. You do not need a fully instrumented stack before you get meaningful signal. Start with the structured log and build from there.
Key Takeaways
- Standard infrastructure monitoring is necessary but not sufficient for LLM applications. Semantic failures — quality drift, output truncation, format errors — are invisible to conventional APM tools.
- The finish_reason field is one of the most actionable and most ignored observability signals in LLM production. A high length rate means users are getting partial outputs and your system shows green.
- Automated evaluation does not need to run on every request to be valuable. A 5% sample with a model-graded rubric and a trend-tracked score is enough to catch quality regressions before users report them.
- Token budget alerting per workflow is the most cost-effective way to catch prompt regressions and context bloat before they show up on the billing statement.
- OTel GenAI semantic conventions provide a standard schema for LLM spans. Use the standard rather than inventing your own attribute names — it pays off when you change observability back ends.