A CNI cutover took production down for 25 minutes. The rollback that saved it.
Swapping the CNI on a live Kubernetes cluster is one of those changes that looks reversible on paper and isn’t, until you’ve pre-wired the way back.
The setup
Migrating a running cluster to a new CNI, node by node, with a canary phase that had soaked cleanly for days. The final cluster-wide cutover shipped a manifest that was subtly wrong — and within minutes the ingress path started returning 504 across services. ~25 minutes of partial outage before it was fully back.
What actually went wrong
The canary proved the data path, not the rollout mechanics. A defective cutover step reconfigured every node at once instead of draining in waves. The lesson isn’t “the CNI is dangerous” — it’s that a green canary tells you the destination is fine, never that the journey is.
A canary validates the new state. It says nothing about the transition into it.
The way back
What turned a disaster into an incident was a pre-armed rollback: the previous release pinned, a one-command helm rollback ready, and a monitor watching cross-node pod-to-pod plus ingress 5xx. When the graph went red, recovery was a decision, not an investigation.
Rules I now treat as non-negotiable for CNI / dataplane changes
- Cut over in waves with drain, never cluster-wide in one manifest apply.
- Pin the previous release and rehearse the exact rollback command before the window.
- Treat it as a windowed operation, not a zero-downtime one — announce it, watch it, staff it.
- Keep logical backups of anything the CNI touches (identities, policies) taken while stopped.
The bundle that caused it is now labelled “do not re-run as-is.” The rollback that saved it is now the template.