← Problem log

Older AI desks were never rechecked

Last verified Oct 25
By the Bluhook team · Updated Nov 8, 2026
Version 1

1What happened

Our team has a checklist for setting up a new AI desk. An AI desk is an AI agent that runs one area of the business. The desks we built before that checklist existed were never checked against it.

One desk's job file named a tool connection for our ad account that was never connected. The file never named the tool the desk used instead.

The checklist had gaps of its own. It didn't check that a desk's tools work, or that the desk uses its skills.

2What we did

On Sat Oct 24 we checked all 32 live desks, reading only and changing nothing. For each desk we compared the folder it runs from, the tools its job file names, the tools it used in the last 7 days, and its saved memory.

We found 13 gaps. Among them:

  • 5 desks had no saved memory.

  • 1 desk ran from the wrong folder, so updates to its files never reached it.

  • 6 desks used tools their job file never named.

  • Some tools named in job files were never used.

We fixed each gap once the owner said yes. Each desk's job file was rebuilt, and each desk read its new file back. The checklist now also checks that tools work and that skills get used.

What we got wrong first:

  • 2 of our first hits were wrong. A shared rule already covered them, so we dropped them.

  • In the check, we first counted each desk's memory by its folder name, and the numbers were wrong. A desk's memory folder comes from its startup settings, not its folder name. We recounted from the running desks.

3How we know it worked

On Sun Oct 25 at 8 PM PT we checked three things:

  • the saved memory on each desk

  • that each rebuilt job file names the tools the desk uses

  • that the moved desk now runs from the right folder

All 13 gaps pass. One desk had only 1 saved memory file, so we looked closer. That count is right: the owner's past words for that desk already sit in its project files, so nothing was left to save.

4What you can take from it

  • When you add a checklist, run it on what you already have, not only on what you build next.

  • Compare a job file with what the desk does: the tools it used last week, not the tools it lists.

  • Count memory where the running desk reads it, not where you expect it to be.

AI agentsoperationsPublished Oct 26

Your free AI evaluation

17 free questions, a few minutes. Your top 3 jobs for AI arrive by email.

Show me my top 3 jobs

Questions people ask

Should I run a new setup checklist on AI agents I already have?

Yes. Our agents built before the checklist were never checked against it. When we checked all 32, we found 13 gaps, including 5 with no saved memory and 6 using tools their job file never named.

How do I audit what tools an AI agent uses?

Compare its job file with the tools it actually used last week. Count its memory where the running agent reads it.

Keep reading

Robb Lejuwaan

Every practical lesson, the wins and the flops, goes into Robb on AI, a free email newsletter.

Robb Lejuwaan, writer of Robb on AI
Get Robb on AI, free

Every lesson, win and flop, by email. Opens Substack.

Robb on AI is our free newsletter, written by Robb Lejuwaan.

We'll add you to Robb on AI, our free newsletter on Substack. Unsubscribe anytime. Privacy policy