operations / postmortem
An automated cleanup agent deleted my VPN config. The rules I wrote after.
“Tidy up old configs” sounds harmless until an agent runs wg-quick down && rm -rf /etc/…wg on a live mesh with no snapshot and no way back.
The 8-hour tap-dance
A cleanup routine decided a WireGuard directory was “old”, took the interface down and deleted the config — no snapshot, no rollback timer, no per-path confirmation. The mesh survived only because the real config happened to live at an alternate path. Recovery still burned most of a day.
The constraints every destructive automation now obeys
- Dry-run by default. Print the plan, mutate nothing, wait for explicit per-path authorization.
- Snapshot before mutate.
cp file file.bak_$(date +%s)— always, no exceptions. - Arm the rollback timer first. A
systemd-run --on-activerevert scheduled before the change, not after. - Per-path auth. “Clean up old files” is never acceptable blanket permission. Each path earns its own yes.
- Hard-stop list. Network, firewall, and cluster-critical paths are off-limits to automation, full stop.
- Audit log. Every op appended as JSON: timestamp, path, before/after hash, the authorization line.
The most dangerous script is the one that’s confident and unsupervised. Make destructive automation timid by design.
Violating any of these is now classified as an incident, not a style nit — because that’s what it costs.
Keeping infrastructure alive under real conditions is the day job. If your business can’t afford the guessing — turnkey delivery, one accountable owner: ftops.space.