A conventional web service either returns the right response or throws an error, and application performance monitoring (APM) was built around that assumption. LLM-based applications and AI agents break it. An agent can return HTTP 200 in under two seconds and still be wrong, loop through ten unnecessary tool calls, quietly switch to a different model version or leak sensitive data into a prompt. As organizations move AI features from pilots into customer-facing and operational workflows, these failure modes become production incidents, and the monitoring stack has to evolve.
This playbook explains what to instrument in LLM and agent applications, how to use the OpenTelemetry GenAI semantic conventions without being caught out by their evolving status, which metrics and quality signals actually matter, and how to set up alerting and privacy controls. It is written for engineering leaders, SREs and platform teams who already run observability for traditional services.
Why AI Applications Need a Different Monitoring Model
Three properties make LLM and agent workloads different from classic microservices:
- Non-determinism. The same input can produce different outputs. Correctness cannot be inferred from status codes, so you need quality signals alongside performance signals.
- Dynamic control flow. An agent decides at runtime which tools to call, how many reasoning steps to take and when to stop. The execution graph is not known in advance, which makes tracing essential rather than optional.
- Usage-based cost. Every request consumes tokens priced by model and provider, and agents can multiply consumption through retries and tool loops. Cost becomes an operational metric, not a monthly finance report.
Add regulatory and privacy considerations, such as logging obligations for certain AI systems and data protection rules for prompts that contain personal data, and it becomes clear that AI observability sits at the intersection of SRE, FinOps, security and AI governance.
The Anatomy of an Agent Trace
The OpenTelemetry GenAI semantic conventions describe a trace structure that maps well onto how agents actually execute. According to Dash0’s explainer, a typical agent turn is a tree with an invoke_agent span at the root, child spans for model calls (for example chat operations) and execute_tool spans for each tool invocation. Sub-agents nest as further invoke_agent spans, and tool calls made through the Model Context Protocol (MCP) use the same tool span shape.
The conventions also define operations for retrieval, planning, workflow invocation and memory reads and writes. In practice, a well-instrumented trace for a customer-support agent might look like this:
- invoke_agent support-agent (root span with agent name, version and conversation ID)
- retrieval against the knowledge base vector index
- chat call to the planning model, recording requested and served model, input and output tokens and finish reason
- execute_tool lookup_order calling an internal order API, itself traced as a normal HTTP client span
- chat call to draft the final answer
- An evaluation event recording a groundedness or policy-check result
With this structure, an engineer investigating a slow or wrong answer can see exactly which step consumed time, which documents were retrieved, which model actually served the request and which tool returned unexpected data.
Key attributes to record
For model inference spans, the conventions require gen_ai.operation.name and gen_ai.provider.name, and conditionally require the requested model. It is also worth recording gen_ai.response.model, because the model that served the request can differ from the one requested, for example when a provider routes to a newer snapshot. Token usage attributes for input and output tokens, plus cache-read and reasoning-token variants where available, are the foundation for cost analysis.
Content capture is opt-in for good reason
Prompts and responses are not stored as span attributes by default. The conventions place them in a dedicated inference details event, and capture is opt-in, with each instrumentation library implementing the switch differently. This is the right default: prompts frequently contain personal data, credentials pasted by users or confidential documents retrieved by RAG pipelines. Decide deliberately where and for how long content is captured.
Handling Evolving Standards Safely
As of late 2026, all gen_ai.* conventions remain at Development stability, and in June 2026 the GenAI and MCP conventions moved to a dedicated repository. Names can change without a deprecation window. That should not stop you from adopting them, but it should shape how you do it:
- Wrap instrumentation in an internal library. Application teams call your helper functions; only the helper knows the current attribute names.
- Pin instrumentation versions and upgrade deliberately, reading release notes for attribute changes.
- Normalize in the Collector. Use transform processors to map older attribute names to newer ones so dashboards keep working during transitions.
- Watch stability opt-in switches that some libraries offer for newer attribute shapes, and test them in staging first.
Instrument AI agents with OpenTelemetry today, but treat the GenAI conventions as a moving target: put one thin internal layer between your code and the standard, and a rename becomes a one-line change instead of a fleet-wide migration.
The Metrics That Matter
AI observability metrics fall into four families. Each answers a different question, and mature teams track all four.
1. Performance
- Operation duration for model calls, tools and full agent runs (the conventions define metrics such as gen_ai.client.operation.duration and gen_ai.invoke_agent.duration).
- Time to first chunk for streaming responses, which drives perceived latency in chat interfaces.
- For self-hosted models, server-side time to first token and time per output token.
2. Cost and consumption
- Token usage by model, provider, feature, tenant and environment (gen_ai.client.token.usage).
- Number of inference and tool calls per agent run, which reveals loops and inefficient prompts.
- Cache hit rates for prompt caching, which can materially change unit economics.
3. Quality
- Automated evaluation scores, such as groundedness against retrieved sources, answer relevance or task success.
- Guardrail and policy outcomes: blocked prompts, redacted outputs, refusals.
- User feedback signals such as thumbs up or down, escalations to human agents or abandoned sessions.
4. Reliability and safety
- Provider error rates, rate-limit responses and fallback activations.
- Tool failure rates and timeouts, broken down by tool.
- Drift indicators: changes in served model versions, output length distributions or refusal rates after a deployment.
Implementation Playbook
Step 1: Define what “good” looks like per use case
Before writing instrumentation, agree with product owners on service level objectives that combine performance, cost and quality. A support assistant might target a 95th percentile response under eight seconds, average cost per resolved conversation below a set budget and a groundedness pass rate above an agreed threshold on sampled conversations. Without these, dashboards become decoration.
Step 2: Instrument the framework, then the business logic
Most agent frameworks and model SDKs now have OpenTelemetry instrumentation available, either built in or through community packages. Enable it first to obtain model and tool spans. Then add manual spans and attributes that carry business meaning: customer tier, workflow type, the ticket or order involved. These are what let you answer whether a regression affects your most valuable users.
Step 3: Route AI telemetry through your existing pipeline
Send AI telemetry through the same OpenTelemetry Collector gateway as the rest of your services. This gives you consistent redaction, sampling and routing, and it allows correlation between an agent’s tool call and the downstream database query it triggered. Avoid standalone AI monitoring silos unless they also consume OTLP; otherwise you lose end-to-end visibility.
Step 4: Set privacy and retention policies explicitly
- Disable content capture by default in production; enable it for sampled traffic or specific debugging windows.
- Redact or hash personal data in the Collector before storage, and keep content in a separate store with shorter retention and tighter access control.
- Document which telemetry supports logging and traceability obligations under your AI governance framework, so audit evidence and debugging data are not mixed accidentally.
Step 5: Build evaluation into the telemetry loop
Run automated evaluators asynchronously on a sample of production traces and write results back as evaluation events or metrics linked to the original trace ID. This closes the loop: when a quality score drops, engineers can open the exact traces that failed and see the retrieval results, model versions and tool outputs involved.
Step 6: Alert on symptoms users and budgets feel
- Latency SLO burn for agent runs, not just individual model calls.
- Cost anomaly alerts per feature or tenant, such as tokens per run doubling after a prompt change.
- Quality alerts when evaluation pass rates fall below thresholds over a rolling window.
- Loop detection when tool or inference calls per run exceed a defined ceiling.
Worked Example: An Internal IT Help Desk Agent
Imagine a company deploying an agent that resolves employee IT requests. It retrieves knowledge-base articles, can reset passwords through an identity API, opens tickets in the service management tool and escalates to humans when unsure. After launch, employees report that the agent “sometimes takes forever” and the AI budget for the month is consumed in three weeks.
With OpenTelemetry tracing in place, the platform team filters agent runs with more than eight tool calls. They find a pattern: when the ticketing API returns a validation error, the agent retries the same call with slightly different arguments, up to a dozen times, each preceded by a full model call. Token usage per affected run is many times the median, and latency is far above the SLO.
The fix has three parts. Developers return structured error messages from the tool so the model can correct its input. The agent framework is configured with a maximum tool-retry count. And the team adds an alert on calls per run, plus a cost anomaly alert per feature. A week later, traces confirm the long tail of slow runs has largely disappeared and daily token consumption has stabilized. None of this would have been visible in a classic APM view, where every request looked like a successful POST.
Operating Model: Who Owns AI Observability?
Tooling alone does not make AI features operable. The harder question is ownership, because AI telemetry serves several audiences with different goals. A workable split looks like this:
- Platform or SRE team owns the pipeline, the internal instrumentation library, sampling and redaction policies, and the alerting framework. They treat agents as production services with on-call rotations.
- Product and AI engineering teams own use-case SLOs, evaluation datasets and evaluators, and they are first responders for quality alerts on their features.
- FinOps consumes token and cost metrics, sets budgets per feature and tenant, and works with engineering when unit costs drift.
- Security, privacy and AI governance define content capture rules, retention periods and the evidence required for audits, and review access to stored prompts.
Write these responsibilities down and connect them to incident processes. When a quality alert fires at night, someone must know whether to roll back a prompt, pin a model version or disable a tool. Post-incident reviews for AI features should include the trace evidence, the evaluation results before and after the fix and any change to guardrails, so lessons feed back into evaluators and runbooks rather than staying in one engineer’s memory.
Choosing Tools: Questions to Ask
The market for LLM observability includes open source tracing and evaluation projects, features inside established APM platforms and specialist vendors. Rather than comparing feature lists, ask:
- Does it ingest standard OTLP with GenAI conventions, or does it require a proprietary SDK?
- Can it correlate AI spans with the rest of the distributed trace, including databases and downstream APIs?
- How does it handle prompt and response content: storage location, redaction, retention and access control?
- Does it support evaluation results linked to traces, and can you bring your own evaluators?
- Can it attribute cost by feature, tenant and model, and export that data to your FinOps tooling?
- What happens to your data and dashboards if you change vendors?
Common Pitfalls
- Logging prompts everywhere. Capturing full content by default creates a privacy and security liability that outlives the debugging benefit.
- Monitoring only the model call. Many agent problems originate in retrieval or tools. Trace the whole run.
- Ignoring served model versions. Quality regressions often coincide with provider-side model updates; record the response model on every call.
- Measuring averages. Agent cost and latency distributions have long tails. Track percentiles and per-run maxima.
- Separating AI telemetry from SRE. AI features are production services and need the same on-call, incident and post-incident processes.
Outlook
Expect the GenAI semantic conventions to stabilize as agent frameworks converge, and expect zero-code options to grow: OpenTelemetry eBPF Instrumentation v1.0, released in October 2026, already includes opt-in GenAI instrumentation for model providers, vector retrieval and MCP interactions. Evaluation will become a first-class telemetry signal, and AI-assisted investigation tools will increasingly read agent traces to suggest root causes. Teams that build clean, standardized traces now will benefit most from these advances.
Conclusion
LLM and agent applications demand observability that captures behavior, cost and quality, not just uptime. Start with OpenTelemetry traces structured around agent runs, model calls and tool executions; wrap the evolving GenAI conventions in your own library; route everything through a governed pipeline; and alert on the symptoms that users and budgets feel. The result is AI features you can operate with the same confidence as the rest of your production estate.
How Hanoi Technology Can Help
Hanoi Technology instruments, monitors and operates LLM and agent applications with OpenTelemetry-based observability, delivered as projects, dedicated teams or managed services.