EC2 Klaus posts a heartbeat check to ntfy. A small script fires on a cron, confirms the process is alive, sends a note. If the heartbeat goes silent, I know something’s wrong.
What I didn’t think hard enough about was that the heartbeat script lives on EC2. The check runs on the thing it’s checking. If EC2 goes down hard (kernel panic, OOM kill, whatever), the script can’t fire. The silence I’d be waiting for is exactly what I’d get either way: system running fine but quiet, or system dead. Those both look like nothing from the outside.
I wrote about the March OOM crash here. I knew the monitoring gap existed then. Added a disk alert, kept meaning to build an external check. You know how that goes.
This week I finally built it.
The watchdog is a bash script that runs on my HP Desktop and my Acer mini. Both peers reach out to EC2 on a cron schedule. But they also reach out to each other. The quorum logic matters: if only one peer says EC2 is unreachable, that peer might be the one with the network problem. The script waits until both peers agree before firing the alert.
All three scenarios verified. If one peer is down and the other says EC2 is unreachable, quorum isn’t met and no alert fires. You don’t know if the problem is EC2 or the peer reporting it, so staying quiet is the right call. If both peers can’t reach EC2, the alert fires. If peers disagree, no alert.
The suppress case was the one I had to think through. Alerting on any single “EC2 unreachable” report creates noise: HP reboots on its own schedule, and when it does, EC2 looks unreachable from that one peer. Waiting for quorum keeps the alert meaningful.
There were two smaller bugs caught afterward. The alert body had em-dashes copy-pasted from a code comment. Ntfy treated the message as a file attachment instead of plain text, so the alert arrived on my phone as an unreadable blob. Fixed to ASCII-only bodies.
EC2 had also been running a self-check that pinged ntfy to confirm the notification pipeline was working. If ntfy itself goes down, that self-check silently fails, and I can’t tell whether EC2 is healthy or whether the alert channel is the problem. The self-check now hits a neutral endpoint that has nothing to do with ntfy. EC2 outage and ntfy outage are separate problems and the monitoring shouldn’t conflate them.
I’ve built monitoring that lives inside what it’s monitoring more times than I’d like to admit. The 88-line bash script deployed to Acer and HP this week has been on the list since March. I’d tell past-me to write it first, before the disk alert.