Notes from infrastructure that doesn’t apologize.
Real debugging sessions from production — VPN transport, eBPF meshes, DDoS-scrubbed hosts, Kubernetes that has to stay up. No listicles. What actually broke and what actually fixed it.
Xray REALITY: “handshake did not complete successfully” is almost never your keys
Correct keys, open port, right shortId — and it still fails. The setting everyone overlooks, and why openssl and tcpdump lie to you.
Your anti-DDoS provider is eating your VPN — and QUIC is hiding it
Two silent failure modes on scrubbed hosts: inbound UDP that never arrives, and Google apps that hang on QUIC. One line each.
A CNI cutover took production down for 25 minutes
The green canary lied about the rollout. The pre-armed helm rollback saved it.
Running K3s over a WireGuard mesh
node-ip on the mesh plane, MTU discipline, dial-out, and version pinning — the four that matter.
Never bring up a full-tunnel client on your hub
0.0.0.0/0 on the hub = self-isolation. Plus a watchdog that heals it in 2 minutes.
Image edits don’t land? A ConfigMap is shadowing them
Mounted files win over image layers. Patch the CM, not the image.
A cleanup agent deleted my VPN config
Dry-run, snapshot, rollback timer, per-path auth — the rules destructive automation must obey.
A DaemonSet with no limits pegged six nodes
A crash-loop without CPU limits is a heater. Always set limits on cluster-wide workloads.
The Traefik regex that silently keeps your old config
Double-quoted \. in YAML breaks the regex, Traefik rejects the whole config silently.
tcpdump lied to me for three hours
A single crafted packet proves nothing. Validate the negative on a known-good first.