SR 11-7 Was Written for Models That Don't Argue Back

Your model risk framework was written in 2011 for a thing that computes the same output every time you feed it the same input. You are now pointing it at a system that will give you a different answer to the identical prompt twice in a row, and rewrite its own instructions halfway through the task.

That is not a small mismatch you paper over with an addendum. SR 11-7 is not neutral about what a model is. Its every requirement assumes a fixed function you can pin down, and a generative agent violates the assumption at the definition line. The guidance still governs you. The examiner still expects the artifacts. The thing you are producing them for no longer fits the shape the artifacts were cut to hold.

What the guidance actually says

The Federal Reserve’s SR 11-7, issued jointly with the OCC as Bulletin 2011-12 on April 4, 2011, is the supervisory spine of model risk management in US banking. Start with its definition, because the whole edifice rests on it. A model is “a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates.” Input in, quantitative estimate out, by a specified method. A credit scorecard fits. A logistic regression fits. A Monte Carlo engine fits.

On top of that definition the guidance builds three validation pillars, and they are the three every validator in the country can recite.

Conceptual soundness. You evaluate the design and construction of the model, the quality of the theory behind it, and the developmental evidence that supports it. You document the assumptions. You show the method is sound for the purpose. This is where you prove the model is built right before you argue it works.

Ongoing monitoring. Once the model is in production you confirm it still performs, that it is implemented as designed, and that the world it was built for has not shifted underneath it. Process verification and benchmarking live here. This is the drift-catching pillar.

Outcomes analysis. You compare model outputs to actual results. Back-testing is the canonical form. You hold the estimate against reality and measure the gap.

Wrapping all three is the principle that gives the guidance its teeth: effective challenge. SR 11-7 defines it as “critical analysis by objective, informed parties who can identify model limitations and assumptions and produce appropriate changes,” backed by the right competence, influence, and incentives. And it names the two ways model risk enters: the model has fundamental errors and produces inaccurate outputs, or the model is used incorrectly or inappropriately. Both assume a model whose behavior you can characterize, because you cannot challenge what you cannot pin down.

Every pillar assumes a model that holds still

Read those three pillars again with a frontier LLM behind an API in mind, and each one quietly breaks at a different joint.

Conceptual soundness asks you to evaluate design, construction, and theory. You cannot. The model is a few hundred billion weights trained on a corpus you will never see, by a lab that will not show you the data, the objective, or the architecture beyond a model card. There is no developmental evidence to review because none of it is yours. The vendor moved the “theory” of the model behind a boundary you do not cross. You are being asked to certify the soundness of a construction you are structurally barred from inspecting.

Ongoing monitoring assumes the thing you validated is the thing running in production. With a hosted model that assumption expires on the vendor’s schedule, not yours. The endpoint you approved in March is silently a different set of weights in June, retuned, safety-patched, or deprecated, and the change log you get is a version string and a blog post. I made the ownership version of this argument in You’re Renting the Model, Own the Harness: you never controlled the weights, so you cannot monitor them for drift in any sense SR 11-7 would recognize. The model drifts because the vendor shipped, not because the world moved, and your monitoring plan has no event for that.

Outcomes analysis is the one pillar that survives, and it survives by accident. You can still hold the agent’s outputs against ground truth and measure a gap. But non-determinism poisons the back-test. A single run proves nothing when the same input yields different outputs across calls, so your outcomes analysis has to become a distribution, not a comparison, and SR 11-7 was written before “the model is a random variable at inference time” was a sentence anyone had to say to a validator.

Then there is the part with no pillar at all. The agent calls tools. It reads a file, queries a system, writes a record, triggers an action. The “model” is no longer producing an estimate you analyze after the fact. It is taking actions inside your environment mid-inference, and SR 11-7’s entire mental model is input-to-estimate, with a human deciding what to do with the estimate. Remove the human from between the estimate and the action and you have left the document behind entirely. There is no pillar for a model that acts.

Effective challenge against a black box you rent

The deepest break is effective challenge, because it is the principle the whole guidance leans on, and it is the one an API destroys most completely.

Effective challenge demands objective, informed parties with the competence and access to identify a model’s limitations and force changes. Against a scorecard your validators have all three. They read the code, they re-derive the coefficients, they find the segment where it fails and they make the developer fix it. Against a frontier LLM they have the competence and none of the access. They cannot change the model. They cannot see its limitations except by probing it from the outside like a natural phenomenon. When they find a failure mode, the “appropriate change” they are empowered to produce is a support ticket to a vendor who owes them nothing and a prompt they can edit at the margin. Effective challenge presumes you can reach inside and turn a screw. The API is the wall that guarantees you cannot.

This is the honest core of the mismatch. SR 11-7 assumes a model that does not argue back, in the specific sense that it holds still while you interrogate it, behaves identically under identical conditions, and exposes its internals to anyone with the mandate to look. A generative agent argues back in every one of those senses. It answers differently on the second ask. It changes its own working instructions as the context fills. Its internals are a trade secret on someone else’s infrastructure. You are running a 2011 validation playbook against a 2026 artifact that violates the playbook’s first premise, and the gap is not a documentation problem. It is a category problem.

The obligations arriving on top of it

If this were only a US supervisory question you could wait for the agencies to issue interpretive guidance and manage the ambiguity in the meantime. It is not only a US question, and the calendar is not patient.

The EU AI Act, Regulation (EU) 2024/1689, classifies AI systems used for creditworthiness assessment and credit scoring of natural persons as high-risk under Annex III. High-risk classification is not a warning label. It triggers concrete obligations: a risk management system maintained across the lifecycle (Article 9), data governance for training and validation data (Article 10), technical documentation (Article 11), automatic logging of events (Article 12), transparency to deployers (Article 13), human oversight designed into the system (Article 14), and accuracy, robustness, and cybersecurity guarantees (Article 15). The obligations for high-risk systems phase in through 2026, and the general-purpose model provider duties already began applying on August 2, 2025. Read Article 11’s technical-documentation requirement next to a model whose provider will not show you its training data, and you are looking at the SR 11-7 conceptual-soundness break again, this time as statutory text with penalties attached.

Then DORA, Regulation (EU) 2022/2554, in force since January 17, 2025, comes at the same artifact from the operational-resilience side. DORA treats your dependence on ICT third parties as a first-class risk to be governed, with contractual requirements, concentration-risk analysis, and an oversight regime for critical ICT third-party providers. A frontier LLM reached across an API is an ICT third-party service by any reading, and a small number of labs sitting under a large share of the industry is exactly the concentration DORA was written to surface. Your model risk function and your operational-resilience function are now looking at the same vendor from two regulatory doors, and neither door was cut for a component that is simultaneously a model and a critical third-party dependency you cannot substitute on short notice.

I have argued before that this regulatory weight is not a bug to be minimized but a cost to be priced honestly, in The Compliance Tax. The point holds harder here. The tax is not slower demos. It is that the instrument you are handed to govern the system predates the system by a full technological era.

What actually ports

The instinct in every governance shop is to widen the existing box: add a generative-AI annex to the model risk policy, extend the validation template with a few new fields, and carry on. That treats a category problem as a paperwork problem, and it fails in the review it was built to pass.

The move that survives contact is to stop pretending you can validate the weights and start governing the layer you actually control. You cannot certify the conceptual soundness of the model. You can certify the harness around it: the retrieval that decides what the model sees, the tool permissions that bound what it can do, the deterministic checks that run on every output before it becomes an action, the logs that reconstruct what happened. That is the argument I made structurally in the harness piece, and it is the only reading of SR 11-7 that stays coherent when the model is a rental. Effective challenge cannot reach the weights, so you point it at the scaffolding, where you can still read the code, find the failure segment, and force the change. Outcomes analysis stops being a back-test of a fixed function and becomes a monitored distribution over a random one, with thresholds and alerts, which is a discipline your operational-resilience side already runs.

None of this is comfortable, and I will not pretend the regulators have blessed it. The guidance has not been rewritten. No agency has told you that governing the harness discharges your obligation to validate the model, and an examiner working from the 2011 text can reasonably ask for conceptual-soundness evidence you cannot produce. That is the real exposure, and it is not one a clever policy annex resolves. It is a question about whether the framework itself gets revised for a class of model it was never shaped to hold, which is the deeper version of the tension I traced in Governance or Containment: whether the controls attach to a runtime you own or only to a service you rent.

The framework will get rewritten. Frameworks always lag the technology and then catch up in a rush, usually right after the first enforcement action makes the gap concrete. Until then you are validating a random, self-modifying, tool-calling process with a document that assumes a fixed function, and the most dangerous thing you can do is file the paperwork that says you managed it and believe your own filing.

---

SR 11-7 is a genuinely good piece of supervisory thinking. It is also aimed at a target that stopped standing still. Governing a generative agent with it is not wrong so much as it is a category error wearing the costume of compliance, and the examiner and the architect should both be uneasy about that until the guidance grows a box the model can actually fit inside.

I write about enterprise architecture, regulated-AI governance, and the widening gap between the frameworks we are required to use and the systems we are actually building. Find me on X @orestesgarcia or LinkedIn /in/setsero.