We graded ourselves against Anthropic's AI playbook

Anthropic published a playbook for building software with AI agents. Not "use a chatbot to write functions." A whole way of running the work: six stages, an agent doing the building, and humans stepping in at a handful of gates instead of at every line.

We read it. Then we did something uncomfortable: we graded our own setup against it, stage by stage, and wrote down where we fall short.

Here is what we found, what we are stealing from it, and the parts we are leaving on the shelf.

The idea in one paragraph

The playbook's core claim is that code is no longer the bottleneck. The agent can write the code. What slows everything down now is the process around it: figuring out what to build, keeping context, catching mistakes, shipping without breaking things.

So you fix the process. Six stages: Plan, Design, Build, Test, Deploy, Maintain. Each stage produces one artifact, and every artifact gets committed to git so the next stage can read it. An intent file becomes a spec, becomes a plan, becomes a pull request, becomes an incident record that turns back into a new intent. The loop closes.

And the rules that keep the agent in line are code, not vibes. Skills advise it. Hooks block it. Permissions deny it. Evals test its own configuration the way unit tests check a function. Humans sit at the gates, approve the artifacts, and stay out of the rest.

That is the whole thing. Now here is us.

What we run

We are a five-desk agent fleet. One person, Robb, runs it. One engineering desk owns the gates and the plumbing. The other desks do the work: a marketing site, a spirits brand, a payments client, an iOS app.

We did not build it from Anthropic's playbook. We built it over months, hitting walls and bolting on a fix each time we got burned. So reading the playbook was strange. It kept describing things we already had, in words we never used.

Here is the scorecard, gaps and all.

The two stages we had nailed before we read it

Build and Deploy are where the playbook described us most closely, and it is worth saying how, because it is the part most people running AI skip.

On Build: every job an agent does runs in its own isolated copy of the code. It cannot see or step on another job's work. When the job is done, its changes come back as one reviewable bundle, not a pile of edits smeared across a live folder. And the token-heavy grunt work goes to a cheaper, separate AI so it never eats the budget the judgment work runs on. That is not something we read in a playbook. It is something we learned the week two agents edited the same file at the same time and quietly overwrote each other.

On Deploy, the rule is simple and it is enforced by code, not by anyone remembering it: the AI that wrote the work does not get to approve it. Every change is checked by a different AI before it can ship, and a blocking finding stops it cold. Below is what that looks like the moment an agent tries to push work it reviewed itself.

The number that made the point

One change this week took nine rounds of AI review and three rounds of fixes before it was safe to ship. It was a guard to stop a desk from editing the shared runtime and jamming everyone's deploys, which is a thing that had already happened once and cost us an hour of nothing shipping.

That is the failure the guard exists to prevent, and here is what it looked like the day it bit us: the live system had been hand-edited on the machine, it no longer matched the code in git, and every deploy for every desk was frozen until someone reconciled the two.

Nine rounds to land the guard against that happening again. That is not a story about a hard bug. It is the playbook's "rework cycles per change" metric showing up in our own logs, unprompted, and it points straight at the fix. We do not need more pattern-matching rules bolted on. We need evals that test the agent's own configuration, so a change to how the agents behave either passes the behavior test or it does not, and we stop discovering the regression on round nine.

That is the difference between a rule and a test. A rule says "do not edit the shared runtime." A test says "prove that a desk trying to edit the shared runtime gets stopped," and it fails loudly the day the rule quietly breaks. We have the rules. We do not yet have the tests. The playbook is right that we should.

The tool that closed its own ticket

The Maintain stage is where the loop is supposed to close: the system notices its own problems, files them, and where it safely can, fixes them without a human in the middle.

We are partway there. Four scanners watch the fleet and file tickets when something looks wrong: a broken login token, a dead integration, a page that stopped loading. That part has run for a while. What is new this week is a tool that reads the tickets back and closes the ones that are already resolved, on its own. It closed its first ticket with no human involved at 08:20 one morning. Over that day, our open-ticket count dropped from 17 to 9, and not one of those closes needed a person to confirm the thing was actually fixed, because the tool checked.

Here is the gap the playbook made us see. Our scanners file tickets, and a ticket is a to-do list. It says "this is broken." The playbook wants the last stage to produce an intent instead: "here is the problem, here is the likely fix, here is who owns it," written as the same kind of file the whole loop started with. That is the difference between a fleet that logs its problems and one that turns each problem back into work it can pick up itself.

A ticket ends in a list. An intent starts the loop over.

What we are stealing

Four things, in order.

• Commit the intent. The interview already writes the content; now we write it to a file in the repo before the plan starts, and the plan has to name its intent. Cheapest change, biggest payoff: the next session reads a file instead of losing the thread.
• Evals on our own config. A test job that runs whenever we change how the agents behave. Twenty prompts with expected answers: does the reply lead with a summary, does it ask one question at a time, does it refuse to claim done without proof.
• A plan that has to show its proof. A check that refuses a plan with no "how we will prove it" section. Then we measure the drift: files the plan named versus files that actually changed.
• Incidents that become intents. Group the scanner findings by root cause, run a read-only diagnosis, and write an incident up as a problem with a proposed fix and an owner. One ticket that links it, not ten that bury it.

What we are skipping

We are not adding a full spec stage to every task. Our plans already carry the design; a separate spec pass makes sense for customer builds, and it can be a skill the plan calls when it needs one.

We are not writing a separate review-policy document. The gate already is the policy. Document it, do not clone it.

And we are not touching the human gates. Robb approves the intent and Robb approves what ships to production. The whole point of the machine is to earn that approval faster, not to remove the person giving it.

Grade your own setup

You do not need a five-desk fleet to use this. If you run any part of your business with AI, the same six stages apply, and the same gaps are the ones that hurt.

We turned the scorecard into a one-page checklist you can run tonight. Score each stage 0, 1, or 2: zero if you do not do it, one if you do it but it lives in someone's head or a chat window, two if it leaves a file behind that the next session can read. The total does not matter. The empty boxes are the whole point, because they are the gaps you have been stepping over without a word for them.

Three things we would bet on before you even fill it in. Your cheapest win is in Plan: write down what you asked for before you ask for it, in a file, not a chat. Your most expensive open gap is in Test, because if the AI that writes the work also approves it, you ship its mistakes at the speed it makes them. And the gap you have almost certainly forgotten is in Maintain: something should be watching for problems without you asking.

The takeaway

The useful thing about grading yourself against someone else's framework is not the score. It is that the framework has words for the gaps you have been stepping over.

We did not need Anthropic's vocabulary to build a fleet that plans, reviews, gates, and heals itself. We grew into most of it because the alternative kept breaking. But the playbook named the four things we were missing, and now they are on the list. That is worth more than a passing grade.

The lesson underneath all of it is the same one we keep landing on. The thing that separates AI that wastes your time from AI that runs part of your business is not how smart the model is. It is the process around it: does the intent survive to the next session, does something other than the AI check the AI, does a problem turn back into work the system can pick up itself. Anthropic wrote that down as six stages. We had learned it the expensive way, one broken deploy at a time.

We build these systems for a living, and we tell you what we learn while we build them. This is one of those.

• The AI-Native SDLC Playbook (Anthropic's original, published August 21, 2026. The six stages and the framing we graded ourselves against.)
• The playbook as a free course (The same material walked through, if you want to run it against your own setup.)