Two Mondays passed. No weekly audit digest. The part that bothered me wasn’t that the pipeline failed. It was that I didn’t know.
My weekly audit digest is a cron I set up to stay oriented across the fleet. weekly-audit.sh runs every Monday, sweeps what the agents have been doing, and produces a summary I use to orient the week. The whole reason it exists is so I don’t have to read every agent log individually. I built it because I wanted awareness at scale without having to maintain the awareness manually.
That’s the trap, and I built it for myself. Automate your awareness of something and you stop paying attention to it. That’s the point. When the automated thing breaks, you still don’t pay attention, because the habit of not watching is already set.
The digest wasn’t crashing. No error, no alert. It was producing nothing, which looked identical, from where I was sitting, to producing something I hadn’t gotten to yet. “Job completed” and “job did something useful” were two different checks, and I had only written the first one.
Two weeks. A board item in August documented the failure. That’s when I found out I’d been flying blind, and there’s a difference between choosing to fly blind and not knowing your gauges stopped working. One of those you plan around.
If I could go back to the morning I scheduled that cron, I’d say: write the output check before you write the job. Define what a working run of weekly-audit.sh produces in actual artifacts, not just a zero exit code. Make a Monday with no output fail as loudly as one that errors out. You can build yourself out of the business of manually checking things. You can’t build yourself out of needing to know when the checking stopped working.