← Problem log

Agent rule changes kept clashing

Last verified Nov 1
Exact text to search:
AI agents rule changes conflict review CLAUDE.md

1What happened

we run a large fleet of AI agents. Each one has a rules file that tells it how to work. Agents propose changes to these files all day.

Each change looked fine by itself. Together, they kept clashing. One agent's new rule was stricter than the owner's own words. Another cut a line that a second agent still relied on. Two changes merged a minute apart each fit a size limit, and together they broke the build for every agent.

In the week of Oct 26, 89 of 181 rule changes had a clash on their first review.

2What we did

  1. One agent does just one job: it reviews every change to a rules file before the change merges. It doesn't write rules for others, and it doesn't merge.

  2. Every review asks the same 7 questions: what problem does this fix, which agents does it touch, does it clash with an existing rule or the owner's words, what could it break, which old lines should go, how will we know it worked, and did the agent simply lack the information.

  3. The owner's exact words live in one file, each with its date and time. A rule that quotes the owner gets checked against that file, not against memory.

  4. The answer is PASS or the clash, plus a pick for the fix. It goes on the change itself. The owning agent decides.

  5. A PASS names the exact commit it was given for. A merge hold reads it. If the change gets one more commit, the hold comes back until there's a new review.

    For buildersthe verdict's first line is PASS at <40-character commit hash>, and the hold compares that hash to the head of the change.
  6. Tests run on the commit that was actually pushed, not on the local copy. We learned this when a fix passed every test locally but was pushed without its last edit.

3How we checked

the reviewer agent is checked by a second agent (our security lead) on its own changes.

225 changes reviewed in the week of Oct 26. Each clash was fixed before the merge, not found after it.

The hold caught changes that got a new commit after their PASS (for example, a rebase that grew a size count by 5 words and failed a test).

4Try it

when agents can edit their own rules, the rules drift. Put one reviewer in the path, tie each PASS to an exact commit, and keep the owner's words in one file to check against.

ai agentsprocesscode reviewPublished Nov 1
Robb Lejuwaan

Every practical lesson, the wins and the flops, goes into Robb on AI, a free email newsletter.

Robb Lejuwaan, writer of Robb on AI
Get Robb on AI, free

Every lesson, win and flop, by email. Opens Substack.