tcpdump lied to me for three hours

Half of a hard networking bug is the network. The other half is trusting a tool that gives you a confident, wrong answer — and building a whole theory on it.

The false negative that cost me hours

Chasing a tunnel that wouldn’t pass traffic, I fired a single nc datagram at the listen port and watched tcpdump on the far end. Nothing. Obvious conclusion: the provider drops the port. I built an entire relay architecture around that “fact.”

Then I ran the exact same test against a server I knew was serving hundreds of gigabytes to real clients. tcpdump showed nothing there either. The test was worthless — a lone garbage packet is not a valid handshake, and it gets dropped for reasons that have nothing to do with the path.

The loudest hypotheses are the ones your tools can’t actually confirm. A single crafted packet proves almost nothing.

Tests that don’t lie

• End-to-end with a real listener. A real HTTP 200 across the port beats any packet capture for “does this port work.”
• Read the daemon’s own log. The server told me the real reason at info level the whole time; I just hadn’t asked it.
• Byte counters, not vibes. A working tunnel has an incrementing transfer counter. Trust that over a screenshot of silence.
• Reproduce the negative on a known-good. If your “proof of broken” also fires on something that works, your test is the bug.

The eventual fix was a one-word config change. Everything before it was me believing a measurement instead of validating it. Discipline over cleverness, every time.