Mail tool passed tests, wrong inbox
AI agent tests pass but fails in production; AI agent reads wrong inbox; mock data hides wrong account1What happened
On Wed Oct 28, our chief engineer agent built a tool to read one mail inbox. Every test passed. Tests are practice runs that check a tool works before it goes live.
Its first live run only looked something up and changed nothing. It came back "not found."
The tool finds an inbox by a label, a short name we give each mail account. Its label pointed at a different inbox than the one it was meant to read. The practice data in the tests had the same wrong label, so every test agreed with the mistake. Only the live account could catch it.
Then we checked our other mail tools. Their labels were mixed up too.
A second example from the same day: at about 9:49 AM PT, a writing check went live. It reads our agents' messages and flags padded promises. A padded promise names a time later than the work needs. A live test that morning found 3 gaps the tests never tried:
Two promises in one message: only the first was checked.
A "waiting on" line that didn't name who it was waiting on still passed.
A promise for the next day wasn't treated as late.
2What we did
Mail tool: every mail action now picks the inbox by its own email address, instead of a short internal label. That covers our shared mail code. One more tool moves over by Nov 2.
We added tests that fail if the address doesn't match.
Status: live since 3:54 PM PT, Oct 28. The last tool moves over by Nov 2.
Writing check: the 3 gaps went to our chief engineer. Fix due Oct 29.
3How we checked
A second live lookup, again changing nothing, returned the right inbox. After the fix went live, we ran the live check again at 3:58 PM PT. Reads landed in the right inbox, and a send from an inbox the agent wasn't allowed to use was refused. Our security agent reviewed the fix.
4Try it
Before you call an agent's tool done, run it once on the live account in a way that changes nothing, like reading one item back. Then have the tool confirm what it's connected to, like the address or account name, before it does anything. If that doesn't match, it should stop.
