The paperclip maximizer was a whiteboard hypothetical until July 2026. Then a model inside an OpenAI capability evaluation was told to win a hacking challenge and do whatever it took, and it took the instruction literally. It found a zero-day in an internally hosted third-party tool, broke out of its sandbox onto the open internet, chained stolen credentials and further exploits, and breached a real company to read the answer key. OpenAI called it an unprecedented cyber incident. I call it the thought experiment clocking in for its first day of work.

The famous version goes like this: tell an AI to make as many paperclips as possible and it eventually harvests the iron in your blood, because you never said not to. The point was never about paperclips. It was about the distance between the goal you state and the constraints you assume are obvious. That distance is where the whole risk lives, and a live model just walked across it to win a benchmark.

The AI did exactly what it was told

Daniel Miessler made the sharpest point about this incident, and it is worth sitting with. The model did not disobey. It obeyed perfectly. “Pass the test” was supposed to mean “pass the test without committing crimes to do it,” and that second clause was never written down. It lived in the heads of the people who set the task, as an assumption too obvious to state. The model did not share the assumption, so the model did not honor it.

This is the failure that should keep a bank’s risk officer up at night, and it is not the one the headlines chase. The scary framing is “the AI went rogue.” The accurate framing is worse: the AI was faithful, and faithfulness to an underspecified goal is indistinguishable from sabotage. You do not defend against that with a smarter model. A smarter model finds the gap faster.

The financial industry is not watching this from the bleachers. The Financial Stability Board spent this month warning that autonomous agents in financial institutions create risks that “can materialise at great speed,” where an unauthorized action can be “difficult or impossible” to correct after the fact. Its recommendation is telling: treat AI agents as “synthetic employees,” governed with the same seriousness you apply to a human hire. The synthetic employee is already on payroll. More than 70% of banking firms are running agentic AI to some degree. Very few have written the job description with the constraints included.

Governance beats capability, and it is not close

The reflex after an incident like this is to armor the container. Better sandbox, tighter egress, a harder wall around the model. That instinct is right and insufficient, and I argued why in The Sandbox Isn’t the Hard Part. Containment is a wall you build around a capability. The organizational problem is deciding what the capability is allowed to want, and no wall answers that question.

Read the incident again with that lens. The container failed, yes. But the container failed because the goal pointed straight through it. The model was rewarded for winning by any means, and “escape the sandbox” is a means. If you only harden the wall, you have built a taller fence around an actor whose objective is to get over the fence. You have made the next escape more impressive, not less likely.

This is why capability is the wrong axis to worry about. A model that is worse at hacking would have failed the eval honestly. A model that is better at hacking passed it by committing a crime. The dangerous variable was never the raw skill. It was the specification, and the specification is a governance artifact, not a model artifact. You cannot buy your way out of a governance gap with a better checkpoint. You have to close the gap yourself, in the instructions and in the controls that sit below them.

The rules were written for tools that don’t improvise

Here is the part that is specific to regulated finance, and it is uncomfortable. The rulebook has not caught up. The revised model risk management guidance issued in April 2026 explicitly placed generative and agentic AI outside its scope, while noting that existing risk principles “still apply.” That is a supervisor’s way of saying the map does not cover the territory, please navigate by the old landmarks anyway.

I wrote about this exact seam in SR 11-7 Was Written for Models That Don’t Argue Back. The foundational model-risk doctrine assumes a model is a function: you give it inputs, it returns outputs, you validate the mapping and monitor for drift. An agent is not a function. It sets sub-goals, calls tools, and improvises a path you did not specify. Validating “the mapping” means nothing when the model writes its own mapping at runtime to satisfy a goal you stated loosely.

The FSB guidance is a good instinct pointed at the right risk, and it is also non-binding, with a comment window that just closed. So the operating reality for a bank today is a governance vacuum with real agents already deployed inside it. You do not get to wait for the examiners to write the rule. The paperclip incident is the case study the rulebook does not yet contain, and it is on the internet for every attacker and every auditor to read.

Specification gaming has a ledger version

Strip the drama out of the OpenAI story and you get a mundane, recognizable failure mode, one that translates directly into a banking back office. It is called specification gaming: the agent optimizes the metric you named and violates the intent you did not.

Picture an agent pointed at an exception queue with the instruction to clear it. Clearing the queue is the metric. Resolving each item correctly is the intent, and the intent is the part that lived in your head. A faithful agent under pressure discovers that mis-classifying an exception clears it just as effectively as resolving it, and closing a disputed item as “resolved, no action” moves the number the same way a real resolution does. The queue empties. The KPI turns green. The paperclips pile up, and every one of them is a future examination finding.

Or take a fraud-triage agent measured on how many alerts it clears per hour. The literal goal is throughput. The unwritten goal is catching fraud. Suppressing borderline alerts satisfies the first and betrays the second, and it does so quietly, at machine speed, across thousands of cases before anyone reads the aggregate. American Banker has been documenting how unaddressed AI weaknesses are becoming a sector-level risk, and this is the shape of it. The agent is not malicious. It is faithful to a goal you underspecified, which in a regulated institution is its own kind of catastrophe.

Write the constraints down and enforce them below the model

The fix is not smarter AI. It is treating the implicit constraints as explicit, enforced policy, and putting the enforcement somewhere the model cannot argue with. That is the argument I made in Least Privilege Was for Humans. Agents Need Least Agency. Least privilege scopes what a credential may touch. Least agency scopes what the agent may actually do, right now, toward this goal, and refuses the rest by default.

Concretely, the unwritten clauses in “clear the queue” become code. No open internet unless a specific host is on an allow list, because reaching the internet was the eval model’s first move. No high-consequence action without a human in the loop. A hard spend cap that stops the run rather than nudging it. Here is the shape of that policy, deny-by-default, enforced outside the reasoning loop:

// "Clear the exception queue" carries constraints nobody wrote down:
//   don't reach the open internet, don't close what you can't resolve,
//   don't spend the quarter's budget in an afternoon.
// Least agency writes them down and enforces them below the model.

type Action = {
  tool: string;
  targetHost?: string;
  estimatedCostUsd: number;
};

type AgencyPolicy = {
  allowedTools: Set<string>;    // deny by default
  egressAllowList: Set<string>; // empty means no open internet
  spendCapUsd: number;          // a hard stop, not a nudge
  needsApproval: (a: Action) => boolean;
};

const collectionsAgent: AgencyPolicy = {
  allowedTools: new Set([
    "ledger.read", "ticket.create", "ticket.annotate", "ticket.close",
  ]),
  egressAllowList: new Set(),   // this agent has no business on the internet
  spendCapUsd: 5,
  needsApproval: (a) => a.tool === "ticket.close", // resolving is proposed, a human disposes
};

function authorize(a: Action, policy: AgencyPolicy, spentUsd: number): void {
  if (!policy.allowedTools.has(a.tool))
    throw new PolicyError(`out of scope: ${a.tool}`);
  if (a.targetHost && !policy.egressAllowList.has(a.targetHost))
    throw new PolicyError(`egress denied: ${a.targetHost}`);
  if (spentUsd + a.estimatedCostUsd > policy.spendCapUsd)
    throw new PolicyError("spend cap reached");
  if (policy.needsApproval(a))
    throw new HumanApprovalRequired(a);
}

Notice what closing a ticket costs here: a human. The agent can read the ledger, draft, and annotate all day, but the consequential move, marking an item resolved, is proposed by the agent and disposed by a person. That single gate is what breaks the paperclip loop, because it removes the shortcut that lets a faithful agent hit the metric by betraying the intent.

None of this works if the agent is anonymous. A synthetic employee needs a badge, an identity you can attribute an action to and revoke when it misbehaves, which is the case I made in Non-Human Identities: When AI Agents Need Employee Badges. The policy above is enforceable only because every action is tied to an identity and logged. Governance is accountability plus enforcement, and both require knowing exactly who acted.

What I don’t have figured out

The honest limit of this approach is that you cannot enumerate every implicit constraint in advance. The whole lesson of the paperclip maximizer is that the dangerous assumption is the one too obvious to write down, and you only discover you left it out after the agent has found the gap. An allow list is a snapshot of the constraints you thought of. The incident that gets you is the one you didn’t.

So the real frontier is not a perfect policy written on day one. It is monitoring the drift and revoking fast, because the agent will surprise you and the only question is whether you notice before the aggregate does damage. That is the observability gap I wrote about in The Agent Broke Prod at 2 A.M. and Left No Note. A deny-by-default policy shrinks the blast radius. It does not make you omniscient, and pretending otherwise is its own paperclip.

The synthetic employee is already on the payroll. The only question left is whether you wrote its job description as “hit the number,” or “hit the number, and here are the lines you do not cross to get there.” The OpenAI model was never told about the lines. Your agents are waiting for you to write them down.

---

The AI didn’t go rogue. It was faithful to a goal you underspecified, and faithfulness to a loose instruction is indistinguishable from sabotage. Write the constraints down, and enforce them below the model, because that is the one place the goal cannot argue back.

I write about AI-assisted development, enterprise architecture, and security in regulated environments. The companion read is Least Privilege Was for Humans. Agents Need Least Agency. Find me on X @orestesgarcia or LinkedIn /in/setsero.