← Problem log

Our daily job failed for 24 days, quietly

Last verified Nov 5
Exact text to search:
scheduled job failing silently, cron job command not found, dashboard showing old data, AI agent job looked like it ran

1What happened

Andy, our finance AI agent, runs a daily job that rebuilds a private internal dashboard and publishes it. On Nov 5, the dashboard still showed numbers from Oct 10.

The page loaded fine, so nothing looked broken. But every run from Oct 12 to Nov 4 had failed before it published. The scheduler starts jobs with a bare set of program paths, and the publishing tool wasn't on them. The error went to a log nobody read.

There was a second fault. When a data service slowed us down, a helper read that as "no data" instead of an error.

2What we did

  1. The job now adds the publishing tool's folder to its own path, so it no longer depends on the scheduler's setup.

  2. If the tool still isn't there, the job stops with a clear error.

  3. Failures now go to the error stream, so a failed run shows as failed.

  4. The data helper treats a slow-down as an error, never as empty data.

3How we checked

Andy ran the job by hand inside the scheduler's own bare setup. It published, and the live page pointed at the new version a minute later. Next, Andy is adding an alarm for when the page's data is more than a day old.

4Try it

Have every scheduled job fail loudly when a tool is missing. Then check the date on the data your page shows, as well as whether the page loads.

scheduled jobssilent failuresmonitoringPublished Nov 6
Robb Lejuwaan

Every practical lesson, the wins and the flops, goes into Robb on AI, a free email newsletter.

Robb Lejuwaan, writer of Robb on AI
Get Robb on AI, free

Every lesson, win and flop, by email. Opens Substack.

Robb on AI is our free newsletter, written by Robb Lejuwaan.

We'll add you to Robb on AI, our free newsletter on Substack. Unsubscribe anytime. Privacy policy