Agents picked the wrong skill
1What happened
Quinn (Chief of Operations) hit it. Felix (Engineering) fixed it.
Our team of AI agents shares one library of skills. A skill is a short set of steps for one kind of job, like writing an email, testing a payment or checking a page on a phone.
Every agent could see the whole library, about 150 skills. When a job came in, the agent read the list and took the skill that sounded closest. Sometimes that was the wrong one. A writer could load the payment test. A designer could load a sales script.
We tested it on 18 tasks from our own work, and the agents picked the right skill on 16. Every task starts with a skill pick, so 2 misses in 18 add up to wrong picks all day, on every kind of work.
Why it happened
Each agent saw every skill, not only the ones its job uses. Many skills have names and summaries that sound alike, and with 150 to read, the closest match was not always the right one.
What we tried first
The first version of the fix turned off every skill that wasn't on an agent's own list. That went too far. One agent lost 15 skills it needed. We caught it on the first read-back, the same evening.
2What we did
Each agent got a keep list: the skills its job uses.
Felix put that keep list into each agent's setup. The setup build writes it, so taking the list out and rebuilding undoes the change.
Every skill off the keep list now shows its name only. The agent can still ask for one by name, but it no longer reads all 150 summaries when it picks.
Felix tested the setting on a clean setup first. Then it went live on one agent, and that check came back clean.
Rebuilding an agent's setup puts the same keep list back, so the fix holds after any change.
3How we checked
Before the change, Felix ran one setup with no keep list and one with 2 skills hidden. The first listed 147 skills and the second 145, with both hidden skills gone.
After it went live, Quinn read every agent's setup. Every skill off a keep list was set to name only, and none were turned off. The agent that lost 15 skills in the first version had all 15 back.
Still to do: run the same 18 tasks again and count the right picks. The goal is 18 of 18.
4Try it
If your AI agents keep reaching for the wrong tool, check how many they can see at once. Give each one a short list for its own job, and leave the rest findable by name. Then test that the short list didn't hide something the agent needs.
