VXLAN (Virtual Extensible LAN)

VXLAN encapsulates an original Ethernet frame inside a UDP packet carrying an 8-byte VXLAN header containing a 24-bit VNI (VXLAN Network Identifier). The outer IP/UDP header rides on the underlay fabric, so the inner tenant frame’s MAC addresses, VLAN tags, and payload are opaque to underlay routers and switches. This decoupling is precisely why cloud providers and CNI plugins (Flannel VXLAN backend, Calico VXLAN mode, Cilium’s legacy overlay) use it: the physical fabric only needs IP reachability between VTEPs (VXLAN Tunnel Endpoints), typically implemented in the hypervisor, host kernel, or a ToR switch’s ASIC.

VTEP-to-VTEP MAC learning historically relied on IP multicast flooding for BUM (Broadcast/Unknown-unicast/Multicast) traffic, which is operationally painful in cloud environments that block multicast. Modern deployments instead use a control plane — EVPN (Ethernet VPN) with BGP, or a Kubernetes-native equivalent like Flannel’s etcd-backed FDB — to distribute MAC-to-VTEP mappings out-of-band, converting VXLAN from a flood-and-learn overlay into a push-based, deterministic one. This distinction matters operationally: flood-and-learn VXLAN in a large Kubernetes cluster can generate significant ARP/BUM traffic amplification during pod churn, while EVPN-VXLAN scales to tens of thousands of endpoints with predictable control-plane load.

The encapsulation tax is non-trivial and frequently mis-diagnosed. VXLAN adds 50 bytes of overhead (14 Ethernet + 20 IP + 8 UDP + 8 VXLAN), which silently reduces effective MTU and triggers fragmentation or PMTU black holes if the underlay MTU isn’t raised (jumbo frames) or the CNI doesn’t correctly clamp TCP MSS. Because the outer UDP source port is typically derived from a hash of the inner flow (5-tuple), VXLAN also interacts with ECMP flow hashing on the underlay — this is a deliberate design choice to preserve per-flow load distribution across multiple underlay paths, but it means visibility tools that only inspect outer headers cannot distinguish tenant flows, complicating underlay-based troubleshooting and requiring VTEP-side or eBPF-based decapsulated observability.

Performance-sensitive deployments increasingly bypass VXLAN’s software encapsulation path via NIC offload (VXLAN TSO/RSS in hardware) or replace it entirely with native routing (Calico BGP mode, Cilium’s native/geneve-free routing) once the underlay supports pod-CIDR advertisement, trading multi-tenant L2 flexibility for lower CPU overhead and simpler debugging. The choice between VXLAN overlay and native L3 routing is one of the most consequential early decisions in a Kubernetes cluster’s network architecture, because migrating away from an overlay after workloads depend on its L2 semantics (e.g., multicast-dependent legacy apps, or IP mobility across subnets) is operationally disruptive at scale.