# Structured production logs with correlation IDs agents can triage

> Stable JSON log shape, correlation IDs, and agent-readable fields so production triage is filterable — not free-text guessing.
> By Dave · 2026-09-20
> Source: https://otf-kit.dev/blog/production-structured-logging-for-agents

Unstructured production logs trap humans and coding agents the same way: you paste free-text lines, guess which request failed, and hope the next retry is the same bug. Structured logging fixes that when every line is a stable JSON object with the same keys, a correlation ID that survives hops, and fields an agent can query without inventing a parser. The goal is triageable signal — not prettier dashboards.

This is not an observability product tour, not a crash-issue workflow, and not a source-map upload checklist. Those seams matter, but they do not replace a log contract your backend owns. If your AI-built app only emits `"something went wrong"` into ephemeral preview logs, neither you nor an agent can reconstruct what happened after the sandbox disappears. Own the shape, the IDs, and the retention.

## What this post is not (and what to read instead)

For LLM-specific traces, token costs, and prompt/response measurement, start with [LLM observability for production apps](https://otf-kit.dev/blog/llm-observability-guide). That post is about model calls. This one is about application and request logs that humans and agents both read.

For queue workers, retries, and idempotency keys on long-running AI jobs, use [background jobs for AI features](https://otf-kit.dev/blog/ai-production-background-jobs). Jobs still need structured logs; that post owns the job lifecycle.

For reading a Sentry issue and shipping a fix, see [crash triage that ships fixes](https://otf-kit.dev/blog/sentry-crash-triage-ships-fixes). Crash trackers need readable stacks; they do not invent your log schema.

For privacy-scoped session replay, see [session replay scoping for production](https://otf-kit.dev/blog/session-replay-scoping-production). Replay complements logs; it is not a substitute for filterable fields.

![Free-text log wall versus indexed structured fields agents can filter](https://cdn.otf-kit.dev/blog/production-structured-logging-for-agents/inbody1-20260920b.png)


## The triage problem for humans and agents

A human on-call can scroll. A coding agent asked to "find why checkout failed for user X" cannot. It needs:

1. **Predictable keys** — `level`, `msg`, `service`, `env`, `request_id` / `trace_id`, `span_id`, `route`, `status`, `error.type` — same names every emit.
2. **One correlation key per request** — so you filter, not greppy-match concatenated prose.
3. **Machine-readable values** — numbers as numbers, enums as short strings, errors as typed fields, not embedded essays.
4. **Enough context without secrets** — user IDs hashed or scoped, never raw tokens or card data in the body.

OpenTelemetry's logs data model exists so backends agree on what a log record *is*: timestamp, severity, body, resource, attributes, and optional `TraceId` / `SpanId` for correlation with traces ([OpenTelemetry Logs Data Model](https://opentelemetry.io/docs/specs/otel/logs/data-model/)). The broader logs vision is explicit: without shared context propagation, logs from different components stay disjoint and correlation stays fragile ([OpenTelemetry Logging](https://opentelemetry.io/docs/specs/otel/logs/)).

That is the bar for agent triage. If a line cannot map to those fields, an agent invents structure or asks for more noise.

## Stable log shape (the contract)

Pick one shape and enforce it in a small helper every handler calls. Prefer a short human `msg` (or OTel `Body` as a display string) plus structured attributes — not a concatenated sentence that buries IDs.

Example contract (illustrative TypeScript; adapt to your runtime):

```ts
type AppLog = {
  level: "debug" | "info" | "warn" | "error";
  msg: string;
  service: string;
  env: string;
  ts: string; // ISO-8601
  trace_id?: string;
  span_id?: string;
  request_id?: string;
  route?: string;
  method?: string;
  status?: number;
  duration_ms?: number;
  error?: { type: string; message: string };
  // domain fields — stable names only
  order_id?: string;
  user_id?: string; // opaque / hashed
};

function log(fields: AppLog) {
  // One JSON object per line — never string-concat IDs into msg
  console.log(JSON.stringify(fields));
}
```

Bad:

```ts
console.log("user_id: " + userId + " failed checkout after " + ms + "ms");
```

Good (platform indexing cares about objects vs strings — Cloudflare Workers Logs indexes object keys so you filter `user_id` instead of full-text scanning a message; see [Workers Logs structured JSON](https://developers.cloudflare.com/workers/observability/logs/workers-logs/)):

```ts
console.log({
  level: "error",
  msg: "checkout failed",
  service: "api",
  env: "production",
  ts: new Date().toISOString(),
  trace_id,
  span_id,
  route: "/checkout",
  method: "POST",
  status: 500,
  duration_ms: ms,
  user_id: userId,
  error: { type: "PaymentDeclined", message: "issuer declined" },
});
```

On Workers, a string embeds everything in `message`; an object extracts and indexes fields. That gap is what separates agent-queryable logs from prose.

Keep cardinality intentional. High-cardinality values (`user_id`, `order_id`) are filter fields you *will* query. Do not invent a new key per deploy. Freeze names in a tiny schema file humans and agents both read.

## Correlation IDs that survive hops

Correlation is the difference between "a 500 happened" and "these twelve lines are the same request." OpenTelemetry log records optionally carry `TraceId` and `SpanId` so logs join traces from the same execution context ([Logs Data Model](https://opentelemetry.io/docs/specs/otel/logs/data-model/)). The industry wire format for that context across HTTP is [W3C Trace Context](https://www.w3.org/TR/trace-context/): `traceparent` carries version, `trace-id`, `parent-id`, and flags so every service can continue or participate in the same forest.

Minimum practice for an owned backend:

1. **Accept or mint** a `trace-id` (and span) at the edge. Prefer parsing inbound `traceparent` when present; otherwise generate a compliant ID.
2. **Propagate** `traceparent` on outbound HTTP and queue messages.
3. **Stamp every log** with `trace_id` / `span_id` (and a gateway `request_id` if your edge already issues one — keep both if both exist; document which is primary for search).
4. **Return** the correlation ID to clients on errors (response header or safe error payload) so support and agents can paste one token into the log UI.

Without propagation, each service restarts the story. Agents then stitch by timestamp proximity — which fails under load.

```ts
// Pseudocode: stamp + propagate
const traceparent = request.headers.get("traceparent") ?? mintTraceparent();
const { traceId, spanId } = parseOrCreate(traceparent);

log({
  level: "info",
  msg: "request start",
  service: "api",
  env: process.env.APP_ENV ?? "production",
  ts: new Date().toISOString(),
  trace_id: traceId,
  span_id: spanId,
  route,
  method,
});

await fetch(url, {
  headers: { ...headers, traceparent: formatTraceparent(traceId, childSpanId) },
});
```

Do not put PII in `tracestate` or free-form baggage. W3C Trace Context is for correlation, not identity ([Trace Context](https://www.w3.org/TR/trace-context/)).

![Traceparent baton propagated from edge gateway to service logs for agent filter](https://cdn.otf-kit.dev/blog/production-structured-logging-for-agents/inbody2-20260920b.png)


## Agent-readable fields (what to optimize for)

Coding agents triage by tool use: filter, then summarize. Design fields so a single query returns a tight set.

**Always useful**

- `level` / severity (align with OTel severity ranges when you bridge to OTLP)
- `msg` — short, stable verb phrase (`checkout failed`, not a novel)
- `service`, `env`, `version` / release
- `trace_id`, `span_id`, `request_id`
- `route`, `method`, `status`, `duration_ms`
- `error.type`, `error.message` (and stack only when you retain it safely)

**Domain fields with frozen names**

- Resource IDs your product already uses (`order_id`, `job_id`, `workspace_id`)
- Outcome enums (`outcome: "success" | "retry" | "fail"`)

**Avoid**

- Unbounded free text as the only signal
- Secrets, session cookies, authorization headers
- Per-request unique key names
- Logging entire request bodies "just in case"

When you bridge existing libraries into OpenTelemetry, the logs approach favors enriching records with resource and trace context rather than inventing a second parallel API for every language ([OpenTelemetry Logging](https://opentelemetry.io/docs/specs/otel/logs/)). For greenfield emits, put queryable structure in attributes (or indexed object keys on your platform) and keep `Body` / `msg` as the human one-liner.

Give agents a one-page "how we log" note: schema, primary correlation field, retention, and example queries. That beats pasting thousands of stdout lines into chat.

## Platform notes (official only)

**Cloudflare Workers.** Enable observability in Wrangler, then prefer `console.log({ ...object })` so Workers Logs indexes fields for filters ([Workers Logs](https://developers.cloudflare.com/workers/observability/logs/workers-logs/)). Invocation logs already carry request/response metadata; custom logs should add domain keys and correlation IDs, not restate the invocation as prose.

**Any stdout JSON pipeline.** Emit one JSON object per line (NDJSON). Collectors parse lines; multi-line pretty-print breaks shipping. Cap size — Workers documents a 256 KB max log size.

**OTLP path.** If you export via OpenTelemetry Collector, map fields onto the Logs Data Model so backends share resource attributes across signals ([Logs Data Model](https://opentelemetry.io/docs/specs/otel/logs/data-model/)). Uniform resource attributes make "same service version" joins reliable.

## Ownership outside the sandbox

Sandbox AI app builders often show a live preview whose console noise vanishes when the preview dies. Production triage needs:

1. Logs leaving the process to durable storage you control (or a vendor account you own).
2. The same schema in staging and production.
3. Retention long enough to debug yesterday's incident (Workers Logs retention is days-scale on the platform tier — plan export if you need longer).
4. Access for humans *and* automation, not only a personal dashboard tab.

Open Template Forest (OTF) ships owned kit code and AI configs so agents extend a known project instead of a disposable preview. Free MIT SDK; paid full-stack kits. Structured logging is an owned backend seam — schema, correlation, and retention in *your* repo — not a sandbox leftover. Browse kits at [otf-kit.dev/templates](https://otf-kit.dev/templates).

## Checklist you can enforce in CI

- [ ] Shared `log()` helper; no ad-hoc string logs in handlers
- [ ] Schema lists allowed keys; PRs reject new keys without update
- [ ] Edge mints/propagates `traceparent`; every service stamps `trace_id`
- [ ] Error responses expose a safe correlation ID
- [ ] Staging and production share field names
- [ ] Secret scanners fail if auth headers appear in log fixtures
- [ ] Runbook: filter by `trace_id`, then read `error.type`

When an incident hits, win with one filter, not a novel. Structured logs with correlation IDs and agent-readable fields make that filter real — for on-call humans and the coding agent beside them.

## Sources

- [OpenTelemetry Logging](https://opentelemetry.io/docs/specs/otel/logs/) — logs vision, correlation dimensions (time, trace context, resource), Collector enrichment
- [OpenTelemetry Logs Data Model](https://opentelemetry.io/docs/specs/otel/logs/data-model/) — LogRecord fields including TraceId, SpanId, Body, Attributes, Severity
- [W3C Trace Context](https://www.w3.org/TR/trace-context/) — `traceparent` / `tracestate` propagation for distributed correlation
- [Cloudflare Workers Logs](https://developers.cloudflare.com/workers/observability/logs/workers-logs/) — structured JSON object logging and field indexing