ECMP Flow Hashing

ECMP is a stateless forwarding decision: when the RIB/FIB installs N equal-cost routes to a destination, the forwarding ASIC computes a hash over a subset of packet fields — typically the classic 5-tuple (src IP, dst IP, src port, dst port, protocol), sometimes reduced to a 2-tuple or 3-tuple in tunneled/encapsulated traffic (VXLAN, GRE, MPLS) where inner headers are opaque to the hardware. The hash result is reduced modulo N (or via a hash table indexed by hash bucket) to select a next hop. Because the hash is a pure function of packet fields, all packets in the same flow take the same path, preserving in-order delivery for TCP without requiring per-flow state in the forwarding plane.

The critical failure mode is hash polarization: if every switch in a multi-tier fabric (e.g., leaf-spine, or spine-superspine) uses the same hash function and seed, flows that collide at one tier will collide identically at every subsequent tier, collapsing what should be N-way parallelism into a much smaller effective fan-out. Vendors mitigate this by seeding the hash per-device (RFC 2992 originally described the base algorithm; most silicon now XORs a device-specific salt into the hash input) so collisions decorrelate hop-to-hop.

The second, operationally more dangerous failure mode is rehashing on topology change. A naive modulo-N hash table means that removing or adding a single ECMP member changes N, which changes the modulo mapping for every existing flow, not just the ones that were using the failed/added link. This produces a full rehash storm: every active TCP connection through that ECMP group gets silently reassigned to a (possibly different) path, and if any downstream stateful device (firewall, NAT, conntrack) lacks session state on the new path, connections reset en masse. This is the reason consistent-hashing-based schemes exist at the software load-balancer layer — Google’s Maglev and Facebook’s Katran build a large, sparse lookup table (e.g., 65537 slots) so that a single backend removal only remaps the slots that pointed to it, leaving the rest of the table — and therefore the rest of the live flows — untouched. This same problem recurs wherever ECMP is combined with Anycast: BGP route flaps toward an anycast VIP change the ECMP next-hop set at every transit router simultaneously, and TCP sessions terminating at anycast endpoints have no way to resume mid-flow if the new nearest instance lacks the connection’s state.

Advanced fabrics address the load-imbalance side (not the churn side) with flowlet switching: bursts of packets within a flow separated by an inter-packet gap larger than the maximum path-delay skew are treated as independently hashable sub-flows, allowing finer-grained load spreading without reordering risk, since a large enough gap guarantees the previous burst has already drained the old path. Weighted ECMP (WCMP) extends the basic model to unequal-capacity paths by biasing the hash distribution proportionally to link weight, which is essential in fabrics with asymmetric link failures where naive ECMP would otherwise send equal traffic shares down a degraded link.

Understanding ECMP hashing is a prerequisite for reasoning correctly about capacity planning and failure blast radius in any leaf-spine or anycast-fronted architecture: the abstraction of “N equal paths” silently breaks down into hotspot links, connection storms, or asymmetric load whenever hash seeding, table size, or rehash granularity is not explicitly engineered, and these failures manifest as intermittent, hard-to-reproduce tail latency rather than clean outages.