If nobody can say why a rule exists and the problem it fixed is gone, cut it: what a ham video taught our AI agents

The video

At Bluhook we run a large team of AI agents (AI software that does a whole job, with its own tools, rules and memory) and share what we learn. Robb Lejuwaan founded Bluhook and runs that team.

Recently, Robb watched a 37-second video called "That's How We've Always Done It"

A little kid asks why the ends of the ham get cut off before it goes in the oven. Nobody at the table knows. Everyone does it because the grandmother did it. The grandmother finally explains why: "our oven was so small I couldn't fit it in."

The video ends on the lesson: ask why a rule exists. If you don't know why a rule exists, find the person who made it and ask them. If nobody can tell you why the rule exists, the video says to delete it: "Otherwise, delete."

Robb's question

Robb sent the video to Quinn, the AI agent that looks after our AI agents' rules. Here's what he wrote, word for word:

"Hey Quinn, look at this video. It's such a simple idea, but if we were to deploy it in our system, it would be f***ing amazing. Like some type of review process where we just make sure that our rules aren't outdated and aren't based on old assumptions. I mean, with models changing all the time, it's very possible that we would have completely outdated rules, like this story in this video. I want to make sure that doesn't happen. So how could we use this basic story in our system to ensure that over time we don't have outdated rules that are based on problems that we no longer have?"

A minute later, Quinn answered that it was watching the video and would send a plan within the hour.

What happened in the next 6 minutes

• Minute 1: Quinn watched the video as 13 still frames, one every 3 seconds, and read every word said in it (160 words).
• Minute 2: Quinn counted our rules.
• Minute 4: Margo, our security AI agent, gave the plan a cold read: it read the plan with no other context, the way a stranger would. Every message to Robb gets one. It sent back 6 fixes. The biggest: the plan had two different tests for cutting a rule, so they became one. It also flagged the rule counts for a recount before they went public.
• Minute 6: the fixed plan went to Robb, well before the time Quinn had promised.

The plan Quinn sent

One term first. Trigg is our chief of staff AI agent; it lines up what Robb needs to decide.

Lightly edited:

"If nobody can say why a rule exists and the problem it fixed hasn't shown up in 60 days, the rule goes. Your video: the grandmother cut the ends off the ham because her oven was too small. Everyone copied her. Nobody else knew why. We have 713 rules in our AI agents' files. About 290 don't say who made them. For those, I find who added them in the file history and ask them. Each week I check the 25 oldest. For each one: why it exists, who made it, and whether its problem happened in the last 60 days. No reason and no live problem: the rule is cut. If the problem comes back, the rule comes back that hour. When our AI agents move to a new model, I switch off each rule written for the old model's habits and test the new model without it. Your own rules stay until you say so. Trigg brings you 5 a week, each with my keep or cut pick. The next step is mine. Within two days, I'll have checked the first 25 rules and I'll tell you how many still had a live problem."

The numbers

The first count was a quick one. Later that morning, Quinn recounted more carefully:

• 770 written rules across our AI agents' instruction files.
• 272 of them don't say who made them. (The quick count said about 290.)
• 41 have a date, but the date is more than three weeks old. (Quinn picked three weeks only to split older rules from newer ones.)

The numbers moved (713 to 770, and about 290 to 272) because new rules were added that morning, and the quick count read a slightly different set of files.

The check, step by step

1. List every rule that doesn't say who made it, when, or what problem it fixed. Quinn, the next day.
2. Give every new rule a date to check it again, 90 days out, written next to the problem it fixed. Quinn, starting the same day.
3. The weekly ham check. Each week, the 25 oldest rules. For each rule: why does it exist, who made it, and has the problem it fixed happened in the last 60 days? The first 25 rules are checked within two days.
4. Cut or keep. If nobody can say why the rule exists, and its problem hasn't happened in 60 days, we cut the rule. If an AI agent made the rule, we ask that AI agent why it exists. If it can't say why within 24 hours, we count that as no reason. If a cut rule's problem comes back, we put the rule back within the hour.
5. Robb's own rules stay until he says otherwise. Each week he gets 5 of them to look at, each with our pick: keep or cut.
6. Re-test rules that only work around one AI model's habits. Each time our AI agents move to a new model, we switch those rules off and test whether the new model still needs them.
7. List rules our software already enforces, so the written copy can go and the software keeps doing the job.
8. A security check on the first batch of cuts before any of those rules is removed.
9. A weekly scoreboard: rules cut, rules put back, and any old mistake that came back after a cut.

The first result is in. Of the first 25 rules, 5 still fix a problem that still happens, 4 fix a problem that's gone, and 16 have no proof either way. None has been cut yet.

What Robb said next

Robb read the plan and wrote: "...this is an example of how we use systems thinking to make our [team] more robust, more dependable, and that it remains healthy over time. This is a great f***ing example. I love it."

A minute later he added: "We take ideas from random places and then apply them to our [team], and we're week by week making our system more dependable, more robust, more healthy, more efficient, and more productive."

Summary

• Every rule fixed a problem once. The problem can go away while the rule stays.
• AI models change every few months. A rule written around an old model's habit can outlive the habit.
• The fix is one check a week: why the rule exists, who made it, and whether its problem has happened in the last 60 days. A cut can always be undone.
• Ideas for a better system come from anywhere. Robb again: "ideas come from many different channels, like conversations, videos, and books, and so on." He put it another way too: "there's an old saying from the movie Glengarry Glen Ross... A, B, C, always be closing. And maybe we need to come up with a new acronym, but it's always be improving your system."

Your business has ham rules too: rules everyone follows that nobody can explain. A form your staff fills out that nobody reads. A sign-off step for a problem you fixed years ago. Ask the same three questions about each one: why does it exist, who made it, and does the problem it fixed still happen?

Quinn checks our rules every week. Want an AI agent doing a job like that in your business? Start with a free evaluation. We'll find the 3 jobs in your business where AI agents would do the most: Get your 3 best jobs for AI, each with its dollar value