Two alarms that fired on things nobody could act on
1What happened
Miles (release) fixed the first alarm. Quinn (systems) fixed the second and checked Miles's fix.
An alarm that watches our code-check line raised 3 critical alerts and kept them open. They pointed at old runs that had stopped long ago and blocked nothing. Every scan said "critical", so the alarm stopped meaning anything.
A second alarm had the same kind of fault. It wakes a reviewer for changes nobody has read. It woke our reviewing AI agent for changes that agent wrote itself. Nobody reviews their own change, and someone else had already reviewed all of them, so each wake was wasted.
Why it happened
The first alarm only asked "is a run waiting?" It never asked whether anyone still needed that run.
The second alarm asked one question, "is it held with no verdict?", and never asked who made the change. It was built to catch other people's changes that sat unread, but it read every open change.
What we tried first
On the first alarm, our first fix skipped every waiting run with no work in it. That cleared the old runs, but it also hid an earlier jam, when dozens of runs sat waiting with every runner idle, and they had no work in them either. A fix that hides the actual problem is worse than the noise, so we took it back.
2What we did
The first alarm now skips a waiting run only when all three are true: it has no work, it is over a day old, and a newer run on the same change already passed. Anything else still counts, including a run whose state can't be read.
The second alarm now skips the reviewer's own changes before it checks anything else.
3How we checked
We wrote 6 tests first, including "a fresh empty run still counts". All 6 failed on the old alarm and pass on the new one, alongside 40 related tests. After it went live, the next critical scan showed no stuck alerts. Quinn read that scan herself.
For the second alarm, a dry run listed the other teams' open changes and none of the reviewer's own. It woke nobody, because nothing was waiting.
4Try it
An alarm should ask "can anyone act on this?" before it fires. And when you quiet a noisy alarm, test that the problem it was built for still trips it.
