This morning I woke up to silence.
Not the good kind. EC2 Klaus, the cloud instance that runs 24/7 and handles all the background automation, had gone quiet. No heartbeat in Discord. No morning brief. Nothing. Just a gap where the machine should have been talking.
I dug into it expecting a cron issue or a network hiccup. What I found was a kernel panic log: Out of memory: Killed process followed by a system reboot at 7:06 AM. The whole thing had been down for 27 minutes before it came back up on its own.
The good news: it rebooted. The bad news: it had been running on borrowed time for weeks.
What actually happened
The EC2 instance was a t3.small¹ — 2GB RAM, no swap. That was fine when I set it up. Klaus lived there alone, reading from memory files, running cron jobs, posting to Discord.
Then I added things. OpenClaw² takes about 530MB at idle. N8N³ (workflow automation platform), which I wired in for some integrations, takes 300MB. The Omi⁴ sync I finished this week added more. Background services, git, cron runners — none of these are huge individually. Together they quietly filled the bucket.
When something pushed it over the edge (probably a memory-hungry API call), the kernel had nowhere to go. No swap to spill into. It started killing processes to survive. Eventually it killed the wrong thing and panicked.
The frustrating part: none of this showed up as a warning. Disk usage had been creeping toward 100% for days. RAM pressure was building. But nothing alerted me because I hadn’t built alerting for this yet. The system failed quietly right up until it didn’t.
The fix was actually three fixes
First: swap. Adding a 2GB swapfile took about two minutes and would have prevented the crash outright:
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstabThat last line matters — without it the swap disappears on the next reboot. This should have been there from day one. A fresh EC2 instance has no swap by default, and I’d never added it.
Second: disk. When I checked after the restart, disk was at 100%. Not “getting full” — 100%. I cleared out Playwright browser caches (those accumulate to gigabytes quietly), llama.cpp build artifacts from an abandoned experiment, GUI packages that had no business on a headless server, and old apt caches. Recovered about 2.5GB. The server had turned into a junk drawer. I do a decent job keeping my physical shop clean. The EC2 instance, apparently, not so much.
Third: the instance itself. A t3.small with 2GB RAM was always going to be tight. I upgraded to a t3.medium¹ — 4GB RAM, same CPU count — which bumps the monthly bill from about $15 to $33. Stopping the instance, changing the type, starting it back up took maybe five minutes. Klaus came back online without needing any reconfiguration. Running a server that crashes isn’t actually cheaper than running one that doesn’t.
The monitoring gap
The thing that bothers me most isn’t the crash. Crashes happen. It’s that I had no early warning.
I have heartbeat checks that fire every few minutes. I have health checks that look at cron job status. But nothing was watching RAM pressure or disk fill rate. The signals were there — disk had been trending toward full for days — and I just wasn’t looking at the right numbers.
One of my daily think tank specialists had flagged exactly this kind of thing two days earlier: “cron observability gap.” I added it to the backlog and moved on. This morning, it bit me.
I now have a disk monitoring cron that alerts when usage crosses 80%. RAM trending is next. The lesson I keep relearning: instrument what matters before you need it, not after it’s already on fire.
The Omi wrinkle
The reason I was pushing the instance harder this week was a new integration. Omi, the wearable AI I’ve been testing, now syncs its conversations to Monica⁵, my CRM. EC2 Klaus runs the bridge: pull Omi conversations, extract the names of people I talked to, log notes to their contact records.
It worked on the first real run — 13 notes logged across 8 contacts. But the script is heavier than the things it replaced, and I hadn’t accounted for that when thinking about available headroom.
Every integration seems light. An API call here, a cron job there. But they compound. The system that was fine in January isn’t necessarily fine in March after you’ve added six things to it. You have to budget for it.
Where it stands now
The instance is stable. Swap is in. Disk is at 82% with room to breathe. RAM is at 38% with 2.3GB free. The Omi integration is running on schedule.
And I finally have the disk alert I should have built two months ago.
The crash was annoying. The 27-minute outage barely mattered since nothing time-sensitive runs at 7 AM. But systems that fail silently until they don’t are the ones that eventually fail at the worst possible time.
Better to find out on a quiet Thursday morning than in the middle of something that actually matters.