A DaemonSet with no CPU limits pegged six nodes at 100%

A tiny watchdog DaemonSet, crash-looping quietly, burned a full core on every node it ran. No alert fired — because the thing meant to watch was the thing on fire.

The shape of it

A helper DaemonSet mounts a kernel interface and crash-loops on an unexpected condition. With no resource limits, each restart spins hot, and containerd’s restart churn drives CPU to 100% on every node in the set — six of them — for hours. Everything else on those nodes slows down; nothing points at the culprit because it looks like generic load.

Two rules that would have prevented it

• Every DaemonSet ships with CPU and memory limits. A crash-loop with limits throttles; a crash-loop without limits is a heater.
• Back-off and alert on restart counts, not just on “not ready.” A pod can be “Running” and still be pathological.

resources:
  limits: { cpu: "100m", memory: "64Mi" }
  requests: { cpu: "20m", memory: "32Mi" }

Cluster-wide workloads deserve cluster-wide paranoia. A DaemonSet is your bug multiplied by your node count.