# LLM observability for production apps: what to trace, measure, and fix

> A practical LLM observability guide for production apps: trace model calls, measure quality and cost, protect sensitive data, and debug failures.
> By Dave · 2026-08-31
> Source: https://otf-kit.dev/blog/llm-observability-guide

## LLM observability starts with the whole request

LLM observability is the practice of connecting a user request to the model calls, retrieval steps, tool calls, latency, cost, and final outcome that followed. In production, logging the generated text alone is not enough. When an answer is slow, expensive, unsafe, or simply wrong, builders need to see which step caused the failure.

The practical approach is to create one trace for the application request and attach a span to every meaningful operation: prompt construction, retrieval, model invocation, tool authorization, tool execution, validation, and response delivery. Record measurements that help you make a decision, but do not automatically store sensitive prompts or user data.

This guide focuses on a small observability contract that works across model providers and can be implemented before adopting a specialized vendor.

## Define the signals before the dashboard

Start with questions, not charts. A useful production system should answer questions such as:

- Which workflow is slow?
- Is latency coming from retrieval, the model, a tool, or a retry?
- Which model and configuration produced the response?
- How often are outputs rejected by validation?
- Which requests exceeded their cost or latency budget?
- Did a tool call fail because of authorization, invalid input, or an upstream outage?

Group the signals into four categories:

1. **Traces** show the path of one request through the system.
2. **Metrics** show rates and distributions across many requests.
3. **Logs** preserve structured events and error details.
4. **Evaluations** measure whether the output was useful, correct, safe, or complete.

OpenTelemetry provides a vendor-neutral framework for capturing traces and metrics, and its context-propagation model is designed to correlate signals across service boundaries. Its GenAI semantic conventions have moved to a dedicated repository, so check the current conventions before fixing your own attribute names. Sources: [OpenTelemetry](https://opentelemetry.io/) and [OpenTelemetry context propagation](https://opentelemetry.io/docs/concepts/context-propagation/).

## Make the trace structure boring

A consistent trace shape is more valuable than a clever one. For one user request, use a structure similar to this:

```text
request
├── prompt.build
├── retrieval
│   ├── search
│   └── rerank
├── model.call
├── tool.authorization
├── tool.execute
├── output.validate
└── response.write
```

Not every request uses every span. The important rule is that each span has a stable name, a start and end time, a status, and a correlation ID. Use the same names for the same operations across workflows so that a dashboard can compare them.

For a model span, record the provider, model identifier, request mode, input and output token counts when available, time to first token when relevant, total duration, retry count, and finish category. Keep the raw prompt and response out of general-purpose telemetry by default. If you need samples for debugging, store them in a controlled system with access restrictions and an explicit retention policy.

For retrieval spans, record the index or collection identifier, query type, number of candidates, selected document count, retrieval duration, and whether the context was empty. Avoid placing full document contents in spans. A document reference, content hash, or redacted excerpt is usually enough to reproduce the path without copying private data into every telemetry backend.

## Measure the failures users actually feel

Average latency hides the tail. Track p50, p95, and p99 latency separately for the whole request and for major spans. A workflow can have a reasonable average while a meaningful group of users waits through a slow retrieval call or repeated model retries.

Measure the following rates by workflow and model configuration:

- Request success and failure
- Timeout and cancellation
- Retry frequency
- Output-validation rejection
- Tool-authorization rejection
- Empty retrieval context
- Fallback-model usage
- Human escalation

For cost, track estimated input and output usage by workflow, tenant, and model. Treat estimates as estimates when provider billing data arrives later or pricing changes. A budget alert should be based on a documented calculation, not a dashboard label that looks precise.

For quality, connect an evaluation result to the same request or trace identifier. A response that completed in 800 milliseconds is not necessarily successful if it cited the wrong record or omitted a required field. The evaluation loop should be separate from the runtime trace, but linked to it.

## Use structured events for decisions

A log line such as `agent failed` is almost useless. Emit structured events with a stable event name and a small set of safe fields.

```json
{
  "event": "llm_request_finished",
  "trace_id": "trace-identifier",
  "workflow": "support-answer",
  "model": "provider-model-id",
  "status": "validated",
  "latency_ms": 842,
  "input_tokens": 1240,
  "output_tokens": 318,
  "retry_count": 0,
  "tool_calls": 1
}
```

The identifier in this example is a correlation value, not a secret. Never put API keys, authorization headers, access tokens, or unredacted personal data into logs. Apply the same discipline to exception messages: upstream errors often contain request payloads or URLs that were not meant for a general log destination.

Use explicit status values such as `completed`, `timed_out`, `blocked_by_policy`, `invalid_output`, and `upstream_error`. These categories let operators separate product failures from infrastructure failures and policy decisions.

## Debug by comparing traces

When an incident arrives, compare a successful trace with a failing trace from the same workflow. Look for the first meaningful divergence:

- Did retrieval return fewer or different documents?
- Did the prompt exceed the intended context budget?
- Did the model configuration change?
- Did a tool authorization check reject a valid action?
- Did a retry reuse a non-idempotent operation?
- Did validation reject a response that the model considered complete?

Do not begin with the final answer and guess backward. Follow the trace from the request boundary. This avoids blaming the model for a timeout caused by a slow database or blaming retrieval for a schema mismatch introduced after generation.

Add a trace link to operational errors, evaluation failures, and support tickets where appropriate. That creates a short path from “the answer was wrong” to the exact request path that produced it.

## Protect telemetry from becoming a new data leak

Observability data can be more sensitive than application logs because it may contain user prompts, retrieved documents, model outputs, and tool arguments together. Define a telemetry policy before enabling verbose capture.

Use field allowlists rather than trying to redact everything after collection. Hash or classify identifiers when operators do not need the original value. Restrict access by tenant and role. Set retention periods for raw samples, traces, metrics, and aggregate reports separately.

Sample routine successful traces more aggressively than failures, but keep enough metadata to understand traffic changes. Never sample away all blocked, timed-out, or validation-failed traces; those are the traces most useful during an incident.

## An implementation sequence for builders

A safe rollout can happen in four passes:

1. Add a request ID and trace context at the application boundary.
2. Instrument model, retrieval, tool, validation, and response spans.
3. Add a small metric set for latency, failures, retries, tokens, and policy blocks.
4. Link offline evaluations and support reports back to trace IDs.

Only after this foundation is stable should you add expensive payload capture or complex dashboards. Start with one production workflow and verify that a deliberately induced timeout, validation failure, and tool rejection are all visible and distinguishable.

For related production patterns, compare this guide with [background jobs for AI features](https://otf-kit.dev/blog/ai-production-background-jobs), the [LLM evaluation loop](https://otf-kit.dev/blog/llm-evaluation-loop), and [safe AI agent tool permissions](https://otf-kit.dev/blog/safe-ai-agent-tool-permissions). The broader lesson is the same: reliability comes from explicit boundaries, measurable outcomes, and recoverable operations. OTF’s [templates](https://otf-kit.dev/templates) can be a starting point when you want to turn those production conventions into an owned application structure.

## Sources

- [OpenTelemetry](https://opentelemetry.io/)
- [OpenTelemetry context propagation](https://opentelemetry.io/docs/concepts/context-propagation/)
- [OpenTelemetry GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai)