Running K3s over a WireGuard mesh: the settings that actually matter

A Kubernetes cluster whose nodes live behind CGNAT, hostile ISPs and DDoS scrubbers, stitched together by an obfuscated WireGuard mesh. It works — if you get four things right.

1. The node IP must be the mesh IP

Every node has to register with its mesh address as INTERNAL-IP, and the CNI must be pinned to the tunnel interface. If a node accidentally registers with its public or LAN IP, kubelet, etcd and pod networking start crossing planes and you get intermittent, maddening failures. Force it explicitly:

k3s agent \
  --node-ip=10.x.x.x \            # mesh address
  --node-external-ip=<public> \
  --flannel-iface=<tunnel-iface>

2. MTU discipline or silent black-holes

The tunnel adds overhead; the overlay adds more. Get the MTU stack wrong and small packets pass while large ones vanish — TLS handshakes stall, kubectl logs hangs, pulls die halfway. Don’t diagnose this with DF-pings alone; push a real multi-hundred-KB transfer through and watch it complete.

3. Outbound-initiated is your friend

Nodes behind scrubbing providers can’t receive unsolicited inbound UDP, but they dial out to the hub with a keepalive — so the tunnel stays bidirectional over the established flow. That’s why a node can be a perfectly good cluster member even when nothing can connect to it directly. Design around dial-out.

4. Version pinning, or drift bites

Nodes can’t reach upstream release servers (CGNAT / blocking), so the naive installer fails at hash download — or worse, silently drifts to “latest”. Deliver a known-good binary out-of-band, verify the checksum, and keep every node on one pinned version. On single-node datastores, back up by copying the DB file while stopped — snapshot tooling won’t save you there.

A mesh cluster isn’t fragile. It’s just honest about the assumptions a datacenter usually hides from you.