AI Portfolio Lab Tools Games Blog Feedback
All Posts

My Watchdog Couldn't See the Auditor Down

I built a Thursday check to catch when my Monday auditor fell behind. It had no idea the auditor itself could be what fell apart.

1 min read
aiautomation

Back in July I added midweek-check.sh to the cron rotation. Thursdays at 14:00 UTC, 7 AM Pacific. The weekly auditor runs every Monday and checks review-ledger.md for domains that have drifted past their review window. If something looked stale by Thursday, I wanted to know before the next Monday cycle, not at it.

The script worked. It just couldn’t watch the one other thing it should have been watching.

Because midweek-check.sh read the ledger and checked dates, it had no way to know whether the auditor itself had run. If weekly-audit.sh crashed on Monday morning with a JSONDecodeError or a Traceback, Thursday’s check would read the ledger, see that last-reviewed dates looked normal from the prior successful run, and say nothing. The failure was invisible. Not until the next successful Monday digest would anyone know something had gone wrong.

I found this while working through the auditor’s evidence pass. That check, the one that looks for “auditor call failed” and “Traceback” signatures in the run output, lives inside weekly-audit.sh itself. It only fires when weekly-audit.sh is healthy enough to execute, which means it can’t catch weekly-audit.sh itself being the thing that’s down.

The fix was teaching midweek-check.sh to also scan logs/weekly-audit.log for those same failure signatures. But you can’t grep the whole log, because a failure that was already followed by a clean Monday would keep alerting forever. So the script finds the last “digest committed” line, which weekly-audit.sh prints after a clean push, then looks for failure signatures only in the lines after that. High-priority notification if anything matches.

A silent Monday failure now surfaces within three days instead of at the next successful digest, whenever that happens to be.

A Python block embedded in the shell script to make a Thursday cron scan the log from the last clean run. The alternative was an auditor that could go down on Monday and stay quiet until the absence of a report became noticeable enough for someone to go looking.

Should have wired it in from day one. The ledger tells you what the auditor found. The audit log tells you whether it ran.