I work on observability tooling for LLM-backed applications and the thing that surprised me most wasn’t a technical problem; it was how confidently wrong these systems can be while every dashboard stays green.
Here’s a scenario that makes the point concrete.
The Call That ‘Succeeded’
Picture an AI phone agent handling insurance calls. A customer calls in asking how to buy a policy. The agent mishears or misparses the request and walks the caller through canceling one instead.
No exception. No error code. No health check failures. Latency is normal. From the perspective of every metric on a conventional dashboard, that call is a complete success.
That’s the thing that actually sank in for me: Uptime and answer quality are orthogonal. You can have a system that’s up 100% of the time and wrong a meaningful fraction of the time, and nothing in a conventional APM setup tells you the difference.
Why I Couldn’t Just Bolt a Metric Onto the Existing Dashboard
My first instinct — I think most people’s first instinct — was to treat this as ‘add a quality metric’. That didn’t survive contact with how these applications actually work.
A single user request into a RAG pipeline or a multi-agent system doesn’t map to one unit of work. It fans out into a retrieval step, a few model calls (sometimes sequential, sometimes parallel), a tool invocation or two and a synthesis step at the end. Each of these has its own latency, token cost and independent way of going wrong. If I collapsed all of that into one opaque ‘request’ span, the way a normal APM trace would, I would have thrown away exactly the information I needed to debug it later.
So the pipeline I ended up with instruments at the span level — treating ‘prompt’, ‘LLM call’, ‘tool call’ and ‘parser’ as genuinely distinct span types — following OpenTelemetry’s GenAI semantic conventions so this data travels the same OTLP path as everything else instead of becoming its own weird side channel. Redaction happens at collection time, not afterwards. Prompts and completions are full of customer data, and if you capture everything at every span before filtering it, you’ve built a PII pipeline whether that was the plan or not.
Then, the part that would have mattered for catching a problem like the one above: A scoring stage runs hallucination detection and answer-quality scoring on those same spans, in parallel with the normal metric extraction. Not instead of the performance signal, but alongside it.
The Failure I Couldn’t See Until I Stopped Looking at Single Requests
Here’s the second thing that took me longer to understand than it should have: The most common quality problem in conversational systems — the model getting worse as a session goes on — is completely invisible if you only ever look at one request at a time. You only see it when you compare turn 10 of a conversation against turn 1 of the same conversation.
So I added a second layer on top of the request-level tracing, aggregated by session ID instead of by individual trace: Total tokens and duration at the session level, a breakdown of every trace inside that session and — this part turned out to matter a lot for multi-agent setups — a round-by-round view of which agent in a collaboration was burning the time and the tokens. That last piece is what let me say, “The second agent in this chain is where both the latency and the spend are concentrated,” instead of just knowing that a three-hour session cost too much, with no idea why.
I’ll admit the token/cost angle was the one I underestimated going in. A single agentic query can burn an absurd number of tokens, and unlike a slow response, you don’t feel it until someone forwards you the invoice. Tagging token spend by model and by session — sitting in the same dashboard I already check daily instead of in a monthly total I’d only see after the fact — turned out to be one of the highest-value things I built, not because it was technically hard, but because it made a previously invisible cost visible to the people who could actually act on it.
What I Still Don’t Have a Good Answer For
I don’t want to write this up as a solved problem, because it isn’t.
The one that bugs me most: Reviewing an individual capability — a skill that queries a monitoring API, say — tells you almost nothing about what happens when an agent starts chaining it with other capabilities at runtime in a sequence nobody explicitly tested. Each piece can look completely safe in isolation and still combine into something nobody signed off on. Runtime filtering and keeping a human-confirmation step in the loop for anything consequential are the mitigations I’ve got, and they’re partial. I don’t think anyone in this space has a full answer yet, and I’d be skeptical of anyone who claims they do.
The other one is more human than technical: How do an on-call engineer and an autonomous diagnostic agent actually share an incident together? Who has the final call when the agent’s confidence is low? What does a ‘war room’ look like when one of the participants is a model instead of a person? I don’t think tooling gets to define that on its own. It’s still something teams are figuring out in practice, ours included.
The Takeaway, If There is One
If you’re building or evaluating observability for LLM-backed applications, the following two things I’d actually prioritize based on what surprised me:
- Build the quality-scoring pipeline as a first-class parallel signal to your performance metrics from day one, because retrofitting it after you’ve already been burned once is a much worse position to build from.
- Add session-level aggregation before you think you need it, because by the time you notice the symptom (a conversation that quietly went off the rails), the data you’d need to diagnose why is usually gone.