The Multi-Cluster Networking Decision

Every platform team eventually inherits a network topology decision it never consciously made. A second cluster gets stood up for disaster recovery, a third appears for a regulated workload, and by the time anyone asks “how do these actually talk to each other?” the answer is usually a tangle of ad-hoc VPN tunnels, inconsistent CIDR allocations, and a service mesh

#The three shapes

Multi-cluster networking reduces to a graph problem. Clusters are nodes; the question is how the edges are drawn and who owns them.

Hub-and-spoke routes all inter-cluster traffic through a central hub — typically a dedicated networking cluster or a cloud-native transit construct. Spokes never talk to each other directly. This centralises policy enforcement and simplifies the routing table on every spoke to a single default route toward the hub, but it makes the hub a shared failure domain and a latency tax on every cross-cluster call.

Full mesh gives every cluster a direct, peered connection to every other cluster. Latency is minimised because traffic takes the shortest path, but the number of edges grows as n(n-1)/2. At five clusters that’s ten peering relationships to provision, secure, and monitor; at fifteen it’s a hundred and five. Full mesh does not fail gracefully — it fails expensively.

Transit gateway (AWS Transit Gateway, Azure Virtual WAN, or a self-managed equivalent using Cilium Cluster Mesh with a routing tier) sits between the two extremes. It provides hub-like centralised route propagation without the hub itself sitting on the data path for every packet — the gateway advertises routes and offloads actual packet forwarding to the underlying cloud fabric or a dedicated routing plane.

The mistake most teams make is picking a topology based on the number of clusters they have today rather than the failure domain they are willing to accept. Hub-and-spoke is defensible at two or three clusters where the hub can be run with genuine high availability and its own blast-radius controls. Beyond that, the operational cost of keeping the hub from becoming a single point of failure usually exceeds the cost of a transit layer.

#Overlay versus native routing

Once the cluster-to-cluster topology is fixed, the harder decision is how pods actually reach each other across that topology. This is a CNI-layer decision, and it has more effect on latency and troubleshooting time than almost anything else in the stack.

Overlay networking (VXLAN or Geneve encapsulation, as used by default in Flannel and in Calico’s IP-in-IP mode) wraps every pod packet in an outer header addressed to the destination node. It works regardless of what the underlying network fabric supports, which is why it is the default in managed Kubernetes offerings. The cost is real: encapsulation adds 50-100 bytes of overhead per packet, forces smaller effective MTUs unless jumbo frames are configured end to end, and makes packet capture genuinely unpleasant — every tcpdump session now needs to decode two IP headers to find the traffic you actually care about.

Native routing, where pod CIDRs are advertised directly into the underlying network via BGP, avoids encapsulation entirely. Cilium and Calico both support this mode. The trade-off is that it requires control over the underlying routing infrastructure — you need to either run BGP speakers on every node and peer them with top-of-rack switches, or use a cloud provider’s native VPC routing integration (GKE’s native CIDR mode, or Calico’s AWS-VPC-native ENI mode).

A representative Calico BGP peering configuration for native routing between two clusters over a transit layer looks like this:

1apiVersion: projectcalico.org/v3
2kind: BGPPeer
3metadata:
4  name: peer-transit-router
5spec:
6  peerIP: 10.0.0.1
7  asNumber: 64512
8  keepOriginalNextHop: false
9---
10apiVersion: projectcalico.org/v3
11kind: BGPConfiguration
12metadata:
13  name: default
14spec:
15  asNumber: 64512
16  serviceClusterIPs:
17    - cidr: 10.96.0.0/12
18  nodeToNodeMeshEnabled: false

Setting nodeToNodeMeshEnabled: false here is deliberate — at anything beyond a handful of nodes, a full node-to-node BGP mesh inside a single cluster becomes as unmanageable as a full mesh between clusters, for exactly the same reason. Route reflectors or a dedicated peering tier should replace it once node count grows past roughly 50-100, depending on churn rate.

#Comparing the patterns under failure

Topology choices only really reveal their cost when something breaks. The table below reflects failure behaviour observed across production hub-and-spoke, transit gateway, and full-mesh deployments rather than theoretical throughput numbers.

The blast-radius column is the one platform teams underweight most consistently. A hub that also enforces network policy is attractive because it centralises governance, but it means a misconfigured policy push or a control-plane upgrade can take every spoke offline simultaneously. Separating the routing hub from the policy enforcement point — even at the cost of an extra component — is usually worth it once the platform is carrying anything customer-facing.

#Service discovery across the boundary

Topology and routing solve reachability; they do not solve discovery. A pod in cluster A resolving a service in cluster B needs either a global service registry or a DNS-level bridge, and the two dominant approaches behave very differently under partition.

Cilium Cluster Mesh and Istio’s multi-cluster mode both maintain a synchronised view of remote endpoints and expose them as if they were local Endpoints objects, updated via a watch on the remote API server. This gives near-real-time failover but means each cluster’s control plane needs direct, authenticated access to every peer’s API server — which is itself a topology decision, since that access has to traverse the same links being designed above.

The alternative — external DNS with health-checked records (Route 53 latency or failover routing, or an equivalent) — is simpler to reason about and does not require cross-cluster API access at all, but resolution latency and TTL caching mean failover is measured in tens of seconds rather than milliseconds. For most internal platform traffic this is an acceptable trade; for anything in a synchronous request path with tight SLOs, it is not.

#MTU and asymmetric routing: the two failures that actually happen

In practice, two failure modes account for most multi-cluster networking incidents, and neither shows up in a design review.

The first is MTU mismatch. Overlay encapsulation reduces the effective MTU available to the pod, but if the underlying transit link (a VPN tunnel, a peering connection, a Direct Connect circuit) has its own MTU ceiling, packets that clear the overlay’s reduced MTU can still be silently dropped or fragmented on the transit hop. This manifests as connections that establish successfully (small SYN/ACK packets pass) but hang on any request with a real payload — a classic case of “the handshake works but the transfer doesn’t,” which sends most engineers looking at application code long before they check ip link show for the actual MTU on each hop.

The second is asymmetric routing introduced by transit gateways with multiple route tables or by BGP configurations where inbound and outbound paths are learned from different peers. Stateful firewalls and NAT gateways sitting on only one of the two paths will drop return traffic that arrives via a different route than the one they saw the request leave on. The fix is almost always to enforce path symmetry explicitly — either through consistent route preference (local preference and AS-path prepending in BGP) or by removing stateful middleboxes from the transit layer entirely and pushing that enforcement to the cluster edge instead.

#Choosing deliberately

There is no universally correct topology; there is only the topology that matches the failure domain a platform team is prepared to own and the cluster count it expects to be running in two years, not two months. Teams that pick hub-and-spoke for its simplicity and then grow past four or five clusters without revisiting the decision are the ones who end up debugging cascading outages in a component nobody originally designed to be critical. Decide the topology, decide the routing mode, and write both down before the third cluster gets provisioned — because by the fourth, the decision will have been made for you by whatever happened to work.