PII scrub at log egress: deny-list fields before agents read production logs

When agents and support can read production logs, the dangerous moment is not the first stack trace — it is the first raw email, access token, or card fragment that left the box into a vendor index. Scrub PII at the logger sink with a deny-list before egress. If a field can identify a person or replay a session, strip or hash it before ship, not after someone greps the dashboard.

This post is about the egress boundary on infrastructure you own after sandbox exit: what must never leave the process, how to put a scrub sink in front of every ship path, and how to prove it with canaries before you widen agent or support access. It pairs with structured production logs for agents (correlation IDs and triageable shape — different problem) and honest status pages (customer-facing truth without dumping secrets). It is not a structured-logging format tutorial, not crash triage, and not the abandoned pii-redaction-production-logs story. The claim is narrow: deny-list at the sink so logs leave the box already scrubbed.

What must never leave the box raw

The OWASP Logging Cheat Sheet lists values that should usually be removed, masked, sanitized, hashed, or encrypted before they are recorded: authentication passwords, access tokens, session identifiers (hash if you must correlate), database connection strings, encryption keys, bank or payment card holder data, and sensitive personal data such as health or government identifiers. It also flags that names, emails, and phone numbers often need special handling — deletion, scrambling, or pseudonymization — when identity is not required for the event.

OWASP ASVS V7.1 is blunt: do not log credentials or payment details; session tokens belong in logs only in irreversible hashed form; and other sensitive data defined by privacy policy must stay out (V7 Error Handling and Logging). Once those values sit in a searchable index that agents and support can query, the log store itself becomes a high-value PII asset — retention, encryption, and disclosure obligations follow the data, not the intent of the original debug line.

Practical deny-list seed for an owned backend (extend from your threat model, not copy blindly):

export LOG_EGRESS_DENYLIST="${LOG_EGRESS_DENYLIST:?set comma-separated field keys}"
# Example seed keys — never invent hosts; keys only:
# password,passwd,secret,authorization,cookie,set-cookie,
# access_token,refresh_token,id_token,session_id,api_key,
# card_number,pan,cvv,ssn,national_id,email,phone,full_name

Prefer logging user_id (stable opaque id) over email. Prefer has_token=true and token_len=36 over the token. Prefer a payment intent id over any card material. Correlation stays; replayable secrets and direct identifiers do not.

Put the scrub at the logger boundary, not the dashboard

Dashboard-only redaction is theater. If the collector, queue, fallback file, or vendor exporter already stored the raw field, agents with broader read access still see it in older indices, backups, and debug exporters. OpenTelemetry’s guidance is explicit: you are responsible for what instrumentation emits, and the Collector’s processors exist to scrub before export — attribute delete/hash, filter, redaction (allow-list attributes + blocked value patterns), and transform (Handling sensitive data; Collector configuration best practices — Scrub sensitive data).

On an owned box after leaving Lovable or Bolt, treat every ship path as untrusted until it passes one sink:

1. Application logger (or SDK processor) omits or hashes deny-listed keys before the record leaves the process.
2. A local scrub sink ($PII_SCRUB_SINK) is the only writer that may call the ship client.
3. Collector / ship config applies a second allow/deny pass so third-party SDKs cannot sneak fields past step 1.
4. Agents and support read only the post-scrub store — never the process stdout dump, never the pre-scrub ring buffer.

Wire the sink as required env — fail closed if unset:

export PII_SCRUB_SINK="${PII_SCRUB_SINK:?path or module that must scrub before ship}"
export LOG_SHIP_ENDPOINT="${LOG_SHIP_ENDPOINT:?set ship endpoint from secrets — no hardcoded host in prompts}"
export LOG_EGRESS_ALLOWLIST="${LOG_EGRESS_ALLOWLIST:-level,msg,ts,service,correlation_id,user_id,route,status,latency_ms}"

Agents that can mutate logging config must not disable $PII_SCRUB_SINK. A runbook that “temporarily” pipes raw JSON to a personal bucket is an egress bypass, not a debug tip.

Deny-list and allow-list work together

A deny-list alone protects the fields someone remembered. An allow-list inverts the default: unknown keys stay on the box. OpenTelemetry’s redaction processor deletes attributes that are not on an allowed list, then masks allowed values that match blocked patterns (Collector scrub guidance). Use both:

• Allow-list — structural fields needed for triage (level, msg, ts, service, correlation_id, user_id, route, status, latency_ms).
• Deny-list — known toxic keys always stripped even if someone adds them to allow by mistake.
• Blocked patterns — last line of defense for free-text msg (card-shaped digit runs, Bearer , eyJ JWT prefixes). Patterns are defense in depth, not the primary control.

Shape for a sink config file loaded from $PII_SCRUB_SINK (illustrative YAML — keys from env):

# Loaded by $PII_SCRUB_SINK — no hostnames here
allow_all_keys: false
allowed_keys_from_env: LOG_EGRESS_ALLOWLIST
denied_keys_from_env: LOG_EGRESS_DENYLIST
blocked_value_patterns:
  - '(?i)bearer\\s+[a-z0-9._\\-]+'
  - '(?i)eyJ[a-zA-Z0-9_-]+\\.[a-zA-Z0-9_-]+'
  - '\\b4[0-9]{12}(?:[0-9]{3})?\\b'
replace_with: '[REDACTED]'
hash_keys:
  - session_id   # irreversible hash only if correlation requires it

Hashing is pseudonymization, not magic anonymity — OWASP and OpenTelemetry both warn that small input spaces (emails, numeric ids) can be guessed. Prefer opaque user_id from your auth store over hashing email “for convenience.”

How to wire the scrub sink today

Keep the ship path one function wide. Pseudocode at the boundary (language-agnostic):

// logger/egress.ts — only this module may call ship()
import { scrubRecord } from process.env.PII_SCRUB_SINK!;

const DENY = new Set(
  (process.env.LOG_EGRESS_DENYLIST ?? "").split(",").filter(Boolean)
);
const ALLOW = new Set(
  (process.env.LOG_EGRESS_ALLOWLIST ?? "").split(",").filter(Boolean)
);

export async function emit(event: Record<string, unknown>) {
  const scrubbed = scrubRecord(event, { deny: DENY, allow: ALLOW });
  // Fail closed: never ship if scrubber threw or returned raw deny keys
  for (const k of DENY) {
    if (k in scrubbed && scrubbed[k] !== "[REDACTED]") {
      throw new Error(`egress_blocked: deny key survived scrub: ${k}`);
    }
  }
  await ship(process.env.LOG_SHIP_ENDPOINT!, scrubbed);
}

Ban whole-object logging in review (log.info(req), console.log(user), dumping headers). Those patterns are how deny-lists get bypassed: the toxic field hides inside a nested blob the key list never sees. Lint for JSON.stringify of request/response bodies in logger call sites.

Pair with Postgres connection pooling so debug traffic under agent load does not melt the DB while you chase a leak, and with rolling deploys so a scrub-sink change rolls with health checks and a fast rollback if ship volume drops to zero.

Prove scrub with canaries before agents get access

Do not grant agent or broad support log read until you can prove a planted secret never appears post-egress.

export SCRUB_CANARY_TOKEN="${SCRUB_CANARY_TOKEN:?set random canary — rotate after each drill}"

# 1) Emit a synthetic event that includes the canary in a deny-listed key
# 2) Force a ship cycle through $PII_SCRUB_SINK
# 3) Query the post-scrub store for the canary string — must be zero hits
# 4) Confirm [REDACTED] or hashed stand-in exists for the test event id

Run the drill on every path that can leave the box: primary exporter, debug exporter, file fallback, sidecar, and any “temporary” curl that support still has in a wiki. If any path shows the canary, that path is not behind $PII_SCRUB_SINK. Fix before widening access.

When a real secret does slip through, rotate the credential — deleting one log line is not remediation. Indices, caches, and backups already copied it. Treat discovery like an incident: rotate, shrink retention on the contaminated index if policy allows, and add the missed key or pattern to $LOG_EGRESS_DENYLIST in the same change that rotates.

For who-did-what without putting PII in operational logs, keep append-only audit events separate — see audit trail events for SaaS ops. Operational logs answer “what failed”; audit trails answer “who changed what,” with their own retention and access rules.

Access after scrub is still access

Scrubbing does not mean “everyone may read everything.” ASVS expects security logs protected from unauthorized access and modification. Give agents scoped queries (service + time window + correlation id), not warehouse-wide dump rights. Short retention on operational logs is the cheapest residual-risk cut after scrub. Support escalation should pull by correlation_id and user_id, never by pasting an email into a global search box that encourages someone to log email again “so we can find them.”

If you are assembling production ownership after sandbox exit, start from owned templates and kits that you can review in a PR — including logging and ship config as code — at https://otf-kit.dev/templates. Kits are secondary here; the durable control is the sink.

Put the deny-list at $PII_SCRUB_SINK before the first agent token can read production logs. Shape and correlation ids make triage possible; scrub at egress makes that triage safe. Ship scrubbed records, plant canaries, and only then open the read path.

Sources

• OWASP Logging Cheat Sheet — Data to exclude
• OWASP ASVS V7 — Error Handling and Logging
• OpenTelemetry — Handling sensitive data
• OpenTelemetry — Collector configuration best practices (Scrub sensitive data)