The Agent Broke Prod at 2 A.M. and Left No Note

An agent merged a change at 2 a.m., a downstream job started failing, and by the time anyone looked, the only evidence was a green checkmark and a commit message the agent wrote to describe work it could no longer explain. The prompt that produced it had scrolled out of the window. The reasoning was never persisted anywhere. The postmortem had a timeline with a hole in the middle exactly where the decision was.

That hole is the subject of this post. Everyone building agents in a regulated shop has spent the last year on prevention: the pre-flight gates, the judge model, the scoped credentials, the sandbox that quarantines the blast radius. I have written a fair amount of that prevention myself. What almost nobody has written is the part that happens after prevention fails, and it fails eventually, because the whole premise of an agent is that it acts without you watching. The runbook for the aftermath assumes an actor that left a trace. Your agent did not.

Prevention is a solved genre; the aftermath is not

The prevention literature is mature. You calibrate a judge model to allow, block, revise, or escalate by action class, following the pattern where a separate validator reads a structured proposal before an acting agent touches anything external. You put destructive commands behind a deterministic wall, which I argued for in Validation Hooks: prose is a nudge the model can ignore, a hook is code that always runs. You wrap the whole thing in the loop that re-runs until every check passes, which is the shape I traced in Automating Chaos.

All of that reduces the probability of a 2 a.m. incident. None of it changes what you have to work with once the incident has already happened. Prevention is a filter on the front of the pipeline. A postmortem is an excavation of the back of it, and you can only dig up what got buried in the first place. The uncomfortable asymmetry is that we instrumented the front heavily and left the back emitting almost nothing.

Why your observability was built for the wrong failure

Classical observability assumes determinism. A deterministic service, given the same input and the same state, fails the same way every time, so a stack trace plus the request payload is a near-complete account of what happened. You reproduce it locally, you fix it, you write a regression test that pins the behavior forever. The entire discipline, from structured logs to distributed tracing, is built on the premise that the failure is a fixed function of inputs you can recover.

An agent violates that premise at the root. Same prompt, same tools, same temperature, and it takes a different path on the second run, because the model is sampling from a distribution, not executing a branch. The “input” that mattered was not the ticket text. It was the exact token sequence in the context window at the moment of the decision, including tool outputs the agent read, prior turns, and retrieved documents, none of which your APM captured because your APM was never designed to treat the model’s context as the payload. You logged the HTTP call to the model endpoint. You did not log the thing that determined the outcome.

So the incident review opens with a category error. Someone asks “what did the agent do,” and the honest answer is that the systems watching it recorded that it did something, at a timestamp, that returned 200. The decision itself, the part a postmortem exists to examine, was never a first-class event in your telemetry.

What a postmortem actually needs from an agent

Strip a good postmortem down and it is a reconstruction of a decision under the information available at the time. For a human operator you get that from chat logs, the runbook they followed, the dashboards they were looking at. For an agent you need the machine equivalent, and it has to be emitted at execution time because none of it is recoverable after. Four things, concretely.

A trace of the whole run, not just the outbound calls. Adopt spans and traces as the unit, the same model distributed tracing already uses, but span the agent’s internal steps: one span per reasoning turn, one per tool call, one per retrieval, nested under a single trace id for the run. The OpenTelemetry GenAI semantic conventions exist precisely to standardize this, so the model invocation, its parameters, and token counts land as structured span attributes instead of an opaque POST. When the trace is complete you can answer “what did it see and in what order,” which is the question the 2 a.m. hole was hiding.

The full context payload, versioned and hashed. The prompt, the system instructions, the retrieved documents, the tool results fed back in. This is the agent equivalent of the request body, and it is the single artifact most teams fail to persist because it is large, sometimes sensitive, and feels redundant next to the log line. It is not redundant. It is the input. Store it, hash it, and reference the hash from the span so the postmortem can pull the exact context that produced the exact action.

A structured decision log. Not the free-text chain-of-thought, which is unreliable as an account of the model’s actual computation and awkward to retain. The structured record: the action the agent proposed, the evidence it cited for authorization, its stated scope, the expected consequence, and the judge’s verdict. If your acting agent already submits a structured proposal to a judge, you are most of the way there; the fix is to persist the proposal and the verdict as durable events rather than discarding them once the action clears.

A replayable eval set seeded from the incident. The moment an agent breaks prod, that run becomes a test case. Freeze the context payload, capture the failing behavior, and add it to a graded eval set so the fix is proven against the actual thing that broke, not a paraphrase of it. This is the agent version of a regression test, and it is the connective tissue between “we fixed it” and “we can show it stays fixed.”

Regression-testing an actor that never repeats

Here is where the non-determinism bites the fix, not just the diagnosis. A classical regression test asserts one output. Run it once, green means fixed. That assertion is meaningless for an agent, because a single pass is a single sample from a distribution and tells you almost nothing about the other paths the model can take from the same starting point.

So you regression-test agents the way model risk management already tests models: across a set, not a point. You take the frozen incident context, run it N times, and grade the distribution of outcomes with a rubric, often an LLM-as-judge scoring against explicit criteria rather than string-matching an expected value. You are not asking “did it produce this output.” You are asking “across many runs, how often does it stay inside the boundary, and did that rate improve after the fix.” A fix that moves the failure rate from one in five to one in five hundred is a real fix. A fix you validated with one green run is hope wearing the costume of a test. I made the determinism-beats-hope case for gates in Validation Hooks; this is the same argument aimed at verification instead of prevention.

The practical build is a graded eval harness wired into the same loop that runs your other checks. Every incident deposits a case. Every deploy runs the accumulated set. The eval suite becomes the institutional memory of every way your agents have misbehaved, which is exactly the asset a deterministic regression suite is for a normal service, and exactly the asset most agent stacks do not have because the framework never emitted the material to build it.

The SR 11-7 tie-in nobody wants to say out loud

If you operate in a bank, none of this is optional dressing, and the reason has a document number. SR 11-7, the interagency guidance on model risk management, requires ongoing monitoring of any model in production, and one of its named pillars is outcomes analysis: you compare model outputs to actual results, on a schedule, to detect degradation and confirm the model still performs as intended. That was written in 2011 for credit and market risk models. It reads today like a specification for agent observability.

You cannot perform outcomes analysis on an actor that emits no analyzable outcomes. If the agent’s decisions were never captured as structured events, there is nothing to compare against actual results, and the monitoring obligation is unmet not as a technicality but in substance. The examiner’s question is not going to be “do you have a judge model.” It is going to be “show me how you monitor this system’s outcomes over time,” and the answer lives entirely in whether you built the trace, the versioned context, the decision log, and the graded eval set, or whether you shipped the agent and hoped the green checkmarks would stand in for evidence.

This is the governance seam the prevention-first framing misses. A judge that blocks a bad action is a control at the point of action. Outcomes analysis is a control across time, and time is precisely the dimension your agent telemetry does not currently cover. The reconciliation loop I described in The Reconciliation Loop is the right mental model here: the durable value is not the single run, it is the continuous comparison of intended state against realized state, forever. An agent stack without that loop is not monitored. It is merely deployed.

Build the note before the break

The blameless postmortem, as John Allspaw framed it a decade ago, depends on a culture that assumes people made reasonable decisions with the information they had, and on evidence rich enough to reconstruct what that information was. The first half is a values choice. The second half is an engineering artifact, and for agents you have to manufacture it deliberately, at execution time, because the actor will not write its own note and cannot reconstruct one after the fact.

So the runbook for after the agent breaks prod is really a set of instrumentation decisions you make before it does. Emit spans for every internal step. Persist and hash the full context payload. Log the structured proposal and the judge’s verdict as durable events. Seed a graded eval set from every incident and run it on every deploy. Do that, and the 2 a.m. incident leaves a complete account instead of a hole. Skip it, and you will keep running postmortems on green checkmarks, telling an examiner you monitor outcomes you never captured, and re-learning the same failure down the hall next quarter because the agent that made it left no note.

The agent is a managed worker now, not a chatbot. You would not let a human operator take a consequential action in production and record nothing about why. Stop letting the agent do it, and stop calling the silence an accident. It is a design choice, and right now most of us are choosing wrong.

---

Prevention gets the headlines because it is the part that keeps the incident from happening. The aftermath gets the audit, because it is the part that proves you were ever in control. Build the observability for the second one before you need it, which is to say before 2 a.m.

I write about AI-assisted development, enterprise architecture, and the governance seams where the two collide. Find me on X @orestesgarcia or LinkedIn /in/setsero.