PMTU Black Hole (Path MTU Discovery Failure)
In plain English
Plain definition
It's when large packets with the 'don't fragment' flag get silently dropped along a network path because the ICMP error message that's supposed to tell the sender to shrink its packets is itself being blocked, so the connection just hangs with no error.
Path MTU Discovery (PMTUD, RFC 1191) relies on a feedback loop: a router that cannot forward an oversized DF-flagged packet is supposed to drop it and return an ICMP Type 3 Code 4 (Fragmentation Needed) message back to the sender, which then reduces its effective MTU and retransmits. A black hole occurs when that ICMP reply never arrives — blocked by a stateless ACL, a misconfigured security group, a NAT gateway, or a firewall rule that treats all ICMP as noise to be dropped. The sender has no negative signal, assumes the packet was lost to congestion, and retransmits the exact same oversized segment forever, or until the application-layer timeout fires.
This is one of the most deceptive failure modes in operations because it is asymmetric by packet size. ping and TCP three-way handshakes use small packets and succeed cleanly, giving the false impression that connectivity is healthy. It is only once a TLS ServerHello with a large certificate chain, an HTTP response body, or a gRPC frame exceeds the constrained path MTU that the connection stalls. Packet captures on the sending side show endless retransmissions of the same sequence number with no corresponding ICMP error — the classic tell that distinguishes this from ordinary packet loss or congestion.
Overlay networking makes this endemic rather than exceptional. Encapsulation protocols like VXLAN (50 bytes overhead), GRE, IPsec ESP, and Geneve all consume header space from the outer frame, shrinking the effective MTU available to the inner payload. A Kubernetes cluster spanning a VXLAN overlay on top of a 1500-byte underlay effectively caps inner packets at ~1450 bytes; if any CNI node, cloud VPC peering link, or transit gateway along the path enforces a stricter and inconsistent MTU, or filters ICMP between nodes, connections between specific pod pairs will hang while others work fine depending on payload size and route.
- MSS Clamping: rewrite the TCP MSS option in the SYN/SYN-ACK at the tunnel ingress/egress so peers negotiate a safe segment size without ever depending on ICMP.
- PLPMTUD (RFC 4821): Packetization Layer PMTUD probes path capacity using TCP-layer signals instead of trusting ICMP, enabled via
net.ipv4.tcp_mtu_probingon Linux. - Static MTU alignment: explicitly set CNI plugin MTU (Calico, Flannel, Cilium) to underlay MTU minus encapsulation overhead rather than inheriting the host default.
- Selective ICMP allowance: permit ICMP Type 3 Code 4 through security groups and firewalls even in otherwise ICMP-denying policies.
The architectural consequence is that MTU consistency becomes a first-class configuration invariant across every hop of a multi-cluster or multi-cloud mesh — underlay, overlay, VPN, and load balancer alike — and cannot be inferred from a passing health check. Teams that treat MTU as a set-and-forget constant discover this failure only under production load, when large payloads start timing out asymmetrically across specific node pairs, and root-causing it requires packet-level capture rather than application logs, since nothing in the TCP or application stack reports an explicit error.