AI-Powered Linux Incident Triage: A Safer DevOps Approach

A read-only AI triage loop is the most underrated idea in agentic DevOps

The piece from Saim Ausman on AI-assisted Linux incident triage with Bash and Claude Code lands on a distinction most "AI for DevOps" coverage misses: AI as a careful assistant, not an autonomous operator. The system built there is read-only end-to-end — Bash gathers evidence, Claude Code analyses, a human decides and acts, Bash verifies. AI never mutates state. That single constraint is the entire idea, and it is genuinely hard to design around, because every vendor pitch in this space leans the other way.

The point of the design is not "AI helps." The point is: AI recommends, humans execute, the system verifies. That sentence alone makes the loop safer than 90% of the agentic DevOps stacks shipping this year.

Why "AI reads, humans write" is the actual enable

Most agentic DevOps pitches fall into one of two traps. Either the AI is autonomous enough to be scary — it restarts your production database, it rotates secrets, it ships a config change at 3am — or it is autonomous enough to be useless, summarising logs nobody opens. The triage system picks a third path: AI gathers and analyses, humans act.

Concretely, the loop is:

Gather → Analyze → Human Act → Verify

Each phase has one job. The Gather phase runs Bash to collect Linux and Nginx health data. The Analyze phase sends that evidence to Claude Code and gets an incident brief back. The Human Act phase has an engineer read the brief and decide what to run. The Verify phase re-runs Bash against the new system state and compares to the baseline.

If the AI hallucinates a recovery command, the worst case is a bad sentence in a chat window — not a deleted database. For incident response, where every minute of downtime costs money and trust, that risk profile is the enable.

The four components, named

The architecture is intentionally small. No message bus, no orchestrator, no agent framework.

• Ubuntu Server — the host under triage. The same machine the failure is happening on.
• Bash script — the system's only writer. Collects evidence, runs verifications, never mutates anything.
• Claude Code — the system's only reader of collected evidence. Produces an incident brief.
• Human operator — the only bridge between recommendation and execution.

The separation matters more than the components. Each layer can fail without the others failing catastrophically. If Claude Code goes down, you can run the collector and triage by hand. If the Bash script breaks, the human still has the brief from the last good run. If the human is unavailable, the loop simply waits — which is the correct behaviour for a production incident.

1. Establish a healthy baseline before you write a single line

Before any automation runs, the server must be known-good. The original walks through confirming a baseline before letting the loop touch anything, and that ordering is not a stylistic choice. A baseline is the reference point the verification step compares against. Without one, "verify" is meaningless — you would be confirming the system still looks like it did when it was already broken.

A good baseline records, at minimum:

• uptime and cat /proc/loadavg
• free -h and df -h (excluding tmpfs)
• Nginx active state via systemctl is-active nginx
• The last 50 lines of /var/log/nginx/error.log
• Connection states via ss -tan

Save it as baseline.txt. Every subsequent verification run diffs against it. Skipping this step is how teams end up "verifying" a system that was already degraded and trusting a brief that was wrong from the start.

2. The Bash evidence collector

The collector is a single Bash script that emits structured, machine-readable output. Strict mode is non-negotiable — a silent failure on the evidence layer would feed garbage to the AI.

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

OUT="${1:-./triage_report.txt}"

collect_system() {
  {
    echo "=== UPTIME ==="
    uptime
    echo "=== LOAD ==="
    cat /proc/loadavg
    echo "=== MEMORY ==="
    free -h
    echo "=== DISK ==="
    df -h | grep -v tmpfs
    echo "=== NGINX STATUS ==="
    systemctl is-active nginx || true
    echo "=== NGINX ERRORS (last 50) ==="
    tail -n 50 /var/log/nginx/error.log 2>/dev/null || echo "(no log)"
    echo "=== NGINX CONNECTION STATES ==="
    ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c
    echo "=== TOP PROCESSES (cpu) ==="
    ps -eo pid,pcpu,pmem,comm --sort=-pcpu | head -n 10
  } > "$OUT"
}

collect_system
echo "wrote $OUT"

The output is plain text with section headers — === UPTIME ===, === LOAD ===, and so on. That structure is the API Claude Code consumes. Treat the headers as a contract: every change here is a versioned migration, because every prompt downstream depends on the section names existing.

3. Claude Code's role — and the prompt that keeps it honest

Claude Code reads the Bash output and produces an incident brief: what is wrong, why it is wrong, what to try first, what to try second, what to do only as a last resort. The prompt should pin the format hard, because the freedom to invent commands is exactly the failure mode this loop is built to avoid:

You are a Linux incident analyst. Read-only.
Input: structured evidence from a triage Bash script (sections marked
=== SECTION ===). Cite the section name when you reference data.
Output, in this exact order:
1. Top suspect (one sentence).
2. Supporting evidence (bullet list, cite section name).
3. Recommended next action (one command, for the human to run).
4. Verification step (the exact Bash snippet that proves the fix worked).
Constraints:
- Do not propose changes to files or services.
- Do not invent commands not present in the evidence.
- If evidence is insufficient, say so. Do not guess.

Two constraints do most of the work. "Cite the section name" makes hallucinated metrics cheap to catch — if the brief cites === DISK === but the evidence says memory is fine, you know where the analysis went wrong. "If evidence is insufficient, say so" is the read-only invariant expressed in prompt form. The model is being told, in plain language, that the right answer to a thin evidence file is "I don't know," not invented confidence.

4. The human checkpoint

The human reads the brief, runs the recommended command, then re-runs the collector. If the diff against baseline.txt is clean, the incident is closed. If not, the fresh evidence is fed back into Claude Code — a new analysis on new data, no accumulated state across runs.

This is where the loop earns its name. The brief is a recommendation, never an action. The verification is a comparison, never a "looks fine." The human checkpoint is the only component with the authority to do both.

5. Nginx as the observation surface, not the action surface

The original uses Nginx to serve the latest evidence and the latest brief over plain HTTP, locked down to an internal network or VPN. That makes the loop inspectable: anyone on the team can see what Claude Code saw and what it recommended, without granting the AI any write path.

Concretely, that means Nginx serves three things:

• /baseline.txt — the known-good reference.
• /triage_report.txt — the most recent evidence run.
• /brief.txt — the most recent Claude Code brief.

Nginx never runs the AI. Nginx never executes the brief. Nginx is a window. The window is the entire production surface of the model.

What this loop actually gets you

Three properties the autonomous-AI framing cannot deliver.

Auditability. Every recommendation is paired with the evidence that produced it. If the AI is wrong, you can see why, because the prompt forced citation.

Reversibility. Nothing Claude Code does is reversible, because Claude Code does nothing. The worst artefact is a bad sentence in a chat. Compare that to an AI that runs systemctl restart on the wrong unit, in the wrong environment, at the wrong moment.

Composability. The Bash script can be extended — more sections, deeper Nginx metrics, journalctl queries — without touching the AI. The prompt can be tightened without touching the script. The human can be swapped for an on-call rotation without touching anything else. Each layer is independently editable.

The part that survives the model churn

Here is the angle worth keeping after the post closes. The AI in this loop is replaceable. Today's Claude Code is tomorrow's smaller hosted model, a local Ollama run, or whatever ships next month. The parts that do not change are the Bash contract (section headers, output format), the read-only invariant, and the human checkpoint. Those are the durable layer — the interface the model churns against, not the interface the model is.

That is the same shape as any well-designed boundary: the structured contract outlives the implementation. Whether the components run on a single Ubuntu VM today or a fleet tomorrow, whether the model is Claude Sonnet or a local 7B, the contract is what survives.

If you build the loop right, swapping the AI is a config change. Swapping the script contract is a migration. Swapping the human checkpoint is a policy decision. The phases — Gather, Analyze, Human Act, Verify — do not change at all. Build the Bash contract first, wire the AI second, leave the human in the loop last. The rest is tooling.