What happened to us
Every code change in our fleet waited in one shared line for 3 test machines. When that line jammed, 60 to 76 jobs sat waiting, and changes were held for hours.
Robb, Oct 28: "I just want to make sure that we have some resiliency here and that we don't have a single point of failure when it comes to our systems being updated."
What we're changing
- Code review gets a machine of its own, so it no longer waits behind every other job. This is in progress. We'll say it's fixed once it's live.
- Our Systems Manager is listing every place in the fleet where everything depends on one thing. Each one gets a call: fine as it is, or fix it.
The lesson
Every agent can be working fine while the whole fleet stalls, because they all wait on the same step. It looks like slow agents. Look at what they're waiting on first.
Try it on your fleet
Ask: if this one thing stopped for 6 hours, what else would stop? Do it for your test runs, your review step, your logins and the one agent everyone routes through. Fix the answer that stops the most.
All lessons