og.rsts.dev
← writing

Sprinkled AI Tops Out at 3x

Amazon gave its teams the same AI tools. The teams that bolted them onto old habits stalled near 3x, and the ones that rebuilt how they work hit a median of 4.5x and sometimes passed 10x. I went through the same dip building my own factory. Read from a regulated enterprise, each of the five habits that separated them turns out to be a control you already wanted. What nobody budgets for is the slowdown that comes first.

Sprinkled AI Tops Out at 3x
Contents

Same tools. Same models. Same year. One group of teams came in under 3x, and another passed 10x.

The difference was never the model. It was what each team was willing to change around it.

Forrest Knight put this story on my radar in a video called “I Looked Into How Devs Are Actually Using AI” (opens in new tab). He talks to developers at AI labs, startups, banks, defense and healthcare, and their results barely resemble each other. Strip out the red tape, give everyone the same harness, and some still top out around 2x. The studies he read ranged from a measured slowdown to gains well past 2x. His question was not how the outliers ship so much. It was why the gap between developers is so wide.

I have a stake in that question from both sides. I build an AI factory for my own work, and it runs most of my content pipeline without me. I also spend my days in the kind of enterprise where “just change how you work” lands on a change calendar, a control framework and an audit plan. So I want to take the answer Amazon found and run it through both of those lives.

The short version: the habits that separate a 3x team from a 10x team are not exotic. Read with an auditor’s eyes, every one of them is a control a careful institution already wanted. What blocks them is the slowdown that comes first, because nobody puts the dip in a budget.

Same tools, different teams

The public record comes mostly from Amazon itself, so read it as a vendor telling its own story. That doesn’t make it wrong. It means the numbers deserve the same scrutiny you would give any case study.

It started with one team. Amazon Bedrock needed a new inference engine. According to the Kiro case study (opens in new tab), the original estimate was 30 engineers for 12 to 18 months. Six senior engineers built it in 76 days, with about 95% of the code produced through Kiro, roughly 100,000 lines of new Rust. That team was not ordinary. Every one of the six was a senior engineer with deep context, and they were building something new rather than changing a legacy system. And the story has its critics: Gal Ratner’s account (opens in new tab), citing people close to the work, says the team was bigger, the timeline longer, and the scope narrower than the headline.

Hold that thought. The interesting part comes next.

Forrest describes a second round at Prime Video: a 10-day sprint after which a 90-week estimate dropped to 24 weeks. The detail I keep coming back to is the one almost nobody repeats. Before the sprint, a senior engineer spent three weeks writing small, well-scoped tasks for the team to churn through. The prep was the hidden multiplier.

Then the test that matters for the rest of us. Amazon Stores ran a pilot with ordinary teams on existing systems, a normal mix of junior and senior people, working their regular backlogs. Per AWS’s own write-up (opens in new tab) and the Kiro summary of the pilot (opens in new tab), the teams that restructured their work around the agent hit a median 4.5x gain, and the best passed 10x. In Forrest’s retelling, the teams that only plugged the tool into their old workflow came in under 3x.

Two details deserve a highlighter. First, the measurement. Forrest stresses that Amazon counted what reached production, not commits. That is the honest metric, and it matters because self-reports lie. In METR’s 2025 randomized trial (opens in new tab), experienced open-source developers believed AI made them about 20% faster. Measured, they were 19% slower. If you count keystrokes or vibes, you will see a gain whether or not one exists.

Second, the timing. All of this ran on the models of 2025, before the generation most of us use today. The ceiling has moved. The habits are why some teams reached it and others didn’t.

My own dip

Forrest admits his own side projects blow past 10x, and he is careful to say side projects don’t represent the industry. Same caveat from me. But my factory is the closest thing I have to a lab, and it went through exactly the curve Amazon describes.

When I started building it, I got slower. Much slower. A post I could have drafted by hand in an evening took me days, because I was not writing the post. I was writing the thing that writes posts. Skills for each station: research, verification, drafting, images, narration, video, publishing. Hooks that fire on every tool call. A format contract the assistant checks before it answers me. Validation gates that refuse to call something done until a tool proves it. For weeks, my output went down while my infrastructure went up. If someone had measured me in that window, the AI investment would have looked like a mistake.

Then the curve turned. This post went from a link to a pull request with me answering two questions and approving a plan. The videos for yesterday’s post were narrated, rendered, mastered, uploaded with captions and chapters, and embedded back into the post, and my hands touched the publish button and not much else. I wrote about the shape of that line in I’m Building the Dark Factory.

Three habits from Amazon’s list showed up in my factory before I had names for them.

The failure is a missing sentence. When the agent gets something wrong, I don’t fix the output anymore. I ask what I failed to tell it, and I write that down in the skill. My video skill has a list of gotchas, and every one of them is a scar. The first round of videos came out in neon purple that looked nothing like the blog. I said “too much purple” once, and now the style guide says to match the blog, not the art. A frame zero that showed a blank screen became a rule about cueing the opening headline early. Each incident cost me once.

Rules have a shelf life. Some of what I wrote down was a crutch for a weaker model. I run a periodic audit on my own instructions that asks one question of every rule: would a smarter model make this unnecessary? The ones that fail get deleted. Amazon’s frontier teams did the same thing through 2025, cutting rules as the models improved. Forrest goes further and argues most people need fewer rules than they think, because piling them on eats the context the agent needs for the actual work. I agree, with a caveat I’ll come back to.

I stopped answering every turn. Early on I approved everything, every 30 seconds, and I was the bottleneck. Now the factory runs end to end and stops only where a human has to own the decision: a plan, a publish, anything outward-facing. Forrest’s line is blunt: “You’re never going to see the four, five, six X improvement because you’re the bottleneck.” He is right, and it took me months to believe it.

Why sprinkling is the rational choice

Here is where my second life pushes back on my first.

In a large enterprise, “sprinkling” the tool is not laziness. It is the sensible default, and the reasons are good ones.

Tools get procured. Workflows don’t. A license is a line item with a vendor review, a security assessment and an owner. A new way of working has none of those. It touches job descriptions, review practices, the change process, the definition of done, the audit evidence, and three committees that each own a slice. Plugging an assistant into the existing workflow changes nothing that anyone signed off on, and that is exactly why it gets approved.

The control side has a serious point, too. Change is risk, and much of a bank’s operating model exists so that a single person’s enthusiasm cannot reach production. Amazon’s own year makes the case. In March 2026, Fortune reported (opens in new tab) on a run of retail-site outages and an internal briefing that listed AI-assisted changes among the factors. Amazon pushed back publicly (opens in new tab): only one incident involved AI tooling at all, and that one came from “an engineer following inaccurate advice that an AI tool inferred from an outdated internal wiki.” Whichever account you believe, the lesson for a cautious institution is the same. Speed with weak guardrails turns small errors into big ones.

So the conservative path is defensible. It also has a ceiling, and the ceiling is around 3x. The people who chose it are not wrong about the risk. They are wrong about where the safety comes from.

Five habits, five controls you already wanted

The more I looked at what the top teams changed, the more familiar it seemed. I have sat through enough control reviews to recognize these habits. They are the things a good internal auditor has been asking engineering for since long before agents.

Diagram: the five habits of the top teams on the left, each mapped to the control it already satisfies on the right, with the productivity dip drawn as a J-curve along the bottom

Write everything down, and you retire key-person risk

The top teams moved whatever lived in Slack threads, code review comments and people’s heads into steering files and skills. AWS describes “agent steering files that define conventions, coding standards, testing patterns, and codebase navigation.” The agent can only follow what it can read.

Now read that as a risk officer. Undocumented knowledge held by two engineers is key-person risk. It shows up in business continuity plans, in operational resilience reviews, in every post-incident report that ends with “only one person knew how this worked.” Institutions spend real money trying to make tribal knowledge explicit, and they mostly fail, because humans don’t write documentation nobody reads.

Agents change the incentive. The documentation now has a reader that fails visibly when the document is missing or wrong. A steering file is a runbook with a test attached.

That’s also the catch. Amazon’s own statement about the outage blames an AI that read “an outdated internal wiki.” A stale document handed to an agent is worse than no document, because the agent acts on it with confidence. So the habit has a second half: written knowledge needs an owner and a review date, the same as any other control. The friction in your organization will not be the writing. It will be that some people’s value is the knowledge only they hold, and writing it down feels like giving it away.

Get slower first, and you are building capital

Every team that did well got slower at the start. They spent the dip on plumbing: MCP servers so the agent could reach real tools, error messages written for a model to read, code restructured so an agent can find its way around. Some switched languages. The Bedrock team used Rust partly because, per the case study, compile-time errors help the model debug itself. Forrest’s version: pick a stricter language “so compilers yell at the agent instead of you.”

There is no hidden control here. This one is pure investment. And that is the problem, because enterprise planning has no slot for it. A quarterly roadmap measures features shipped. A team that spends six weeks making its codebase agent-legible ships fewer features that quarter, and the dip lands in a status report as a miss.

I’d stop calling the dip a productivity loss and start calling it what it is: capital spending on a platform. You would never ask a team to migrate a data center while holding feature velocity flat. Making a codebase legible to agents is the same kind of work, with the same kind of payback curve. Fund it like one.

Write the spec before the code, and you get traceability for free

The top teams argued over a plain-English description of the work before any code existed. Forrest’s framing is economic: arguing over a document is cheaper than arguing over diffs the agent already wrote. Amazon’s Kiro builds that habit into the tool, with specs and acceptance criteria up front.

To a control function, that sentence reads differently. A requirement that exists before the change, links to the change, and states how you will know it worked is traceability. It is the thread an examiner pulls when they ask why a system behaves the way it does. Most institutions reconstruct that thread after the fact, from tickets and memory. A spec-first team produces it as a byproduct.

The trap is to overdo it. Forrest argues that a giant spec for every request is overkill and makes results worse. Smaller prompts are fine. Vague ones are not. In an enterprise, the temptation will be to turn “spec first” into a 40-page template. Resist that. The point is intent made explicit and reviewable, which I argued for at the architecture level in The Architecture Belongs in the Repo.

Build the full pipeline, and you get evidence

The top teams gave the agent more to loop against than a type checker: unit tests, linters, integration, performance and security tests. Some mocked every service on their laptops so responses came back fast and deterministic. The Bedrock team describes high-fidelity mocks and canaries that gave them about 99% confidence before a push. The principle, in one line from an Amazon speaker Forrest quotes: “The more loops that agent can go through… the better the output.”

Here is the extrapolation that I think matters most for large organizations. An agent works at the speed of your feedback loop, and your feedback loop is probably a calendar. In many enterprises the realistic test environment is shared, booked in windows, and wired to downstream systems owned by other teams. A human developer tolerates that because they run a test a few times a day. An agent wants to run it a few hundred times an hour. Service virtualization, contract tests and local stand-ins for the systems you don’t own used to be engineering hygiene. With agents in the loop, they decide whether you get 3x or 10x.

And the evidence lands in the same place as your audit trail. Every loop that passes is a recorded check. The pipeline that teaches the agent is the same pipeline that proves to an examiner what was tested. I made the full case for that in Certify the Pipeline, Not the Change, published yesterday. This is its other half: the pipeline you certify is also the agent’s teacher.

Feed the agent, don’t babysit it, and supervision moves to the gate

The top teams kept a backlog of well-scoped tasks, ran agents for long stretches, let them check their own work, and reviewed the results when they were done. Frontier teams, Forrest says, wrote only 1 to 2% of their code by hand. The advice he gives for getting there is the one I followed without knowing it: build one agent you really trust first, and only then run several in parallel.

This habit scares control functions the most, and it shouldn’t. Segregation of duties never meant a second human watching every keystroke. It meant no one person can push an unreviewed change to production alone. That requirement survives agents untouched. What changes is where you enforce it: at the merge and the deploy, through protected branches, a pipeline identity, and a named human who owns the approval. Watching an agent type is supervision theater. Gating what it ships is supervision.

The Amazon story backs this up from the other direction. Amazon’s own diagnosis of its outage was that “our systems allowed an engineering team user error to have broader impact than it should have.” That is a gate problem. More babysitting would not have caught it. A smaller blast radius would have.

Who pays for the dip

If the habits are controls you already wanted, the governance case is mostly made. What remains is money and sequencing, and Amazon’s order of operations is the part worth copying.

Start with your strongest people, not a broad rollout. Amazon began with a team of senior engineers on one well-bounded system, then a second sprint, and only then 50 ordinary teams. Broad seat-license rollouts run that order backwards. They put the tool everywhere and the new habits nowhere, which is the recipe for a 3x plateau.

Protect the dip in writing. Give the pilot team a window, six to eight weeks is a reasonable start, where reduced feature output is the expected result and not a miss. Write down what they are building instead: steering files, tool access, local test harnesses, error messages an agent can act on.

Name the task-writer. The three weeks of senior task-writing before the Prime Video sprint is a job. It deserves a name, an owner and time on the calendar. In many organizations this is exactly what a principal engineer or architect should be doing, and it is a better use of them than reviewing diffs.

Measure what ships. Deployment frequency, lead time, change failure rate, recovery time and rework, taken from the pipeline, per service. If you count commits or survey sentiment, the METR result tells you what you’ll get.

Then spread the habits, not the tool. When the pilot turns its curve, the thing to scale is the playbook: the steering-file conventions, the harness patterns, the gate design. The license was never the scarce part.

A checklist for the pilot team, before anyone talks about speed:

  • The codebase has steering files for conventions, testing patterns and navigation, each with an owner and a review date.
  • Every recurring agent mistake produces a written rule within a day, and every rule gets an annual “is this still needed” review.
  • The agent can run the meaningful test suite locally, with stand-ins for systems the team doesn’t own, in minutes.
  • Errors and logs are readable by a model, not just a person who already knows the system.
  • Every change starts from a short written intent with acceptance criteria, linked to the change.
  • Production is reachable only through the pipeline, under a pipeline identity, with a named human approver.
  • Delivery metrics come from systems, per service, and the dip is visible on them and expected.
  • One trusted agent workflow exists before anyone runs agents in parallel.

What I can’t prove yet

My numbers are side-project numbers. A personal content factory has one user, no regulator and no legacy core system, so it proves the shape of the curve and not the size of the gain in a large organization. The Amazon figures come from Amazon and from a creator retelling Amazon, and at least one insider account disputes the headline case. I trust the direction more than the multipliers.

I also have to admit my own system is still rule-heavy. I’ve cut a lot, and I keep finding more I wrote for a model that no longer exists. Forrest is probably right that most of us carry more rules than we need, and I’m not done pruning.

Parallel agents are where I’m least proven. I run one trusted line well. Running several at once, on shared code, under real change control, is the step I haven’t earned yet, and I suspect it is where most enterprise pilots will stall.

The 10x engineers are already here

“Where are all these 10x devs that we were promised?” Forrest asks. His answer: “They’re here.” I think he’s right, and I think they look less like lone geniuses and more like teams that rebuilt the work around the agent: wrote down what they knew, took the dip on purpose, put intent before code, built a loop the agent could run all day, and moved supervision to the gate.

If you work in a careful institution, that list should look familiar. You already wanted those controls. The agent is just the first coworker that makes them pay off every hour. The open question for your 2027 plan is whether anyone will fund the six slow weeks it takes to get there.

Prefer to watch? The video version is on YouTube (opens in new tab).

If you want the other half of this argument, read Certify the Pipeline, Not the Change, on turning the delivery pipeline itself into the control that lets you say yes to speed.

Are your teams sprinkling or rebuilding? I’d like to hear where the dip is getting stuck. Find me on X at @orestesgarcia (opens in new tab) or on LinkedIn (opens in new tab).