Cutting Cross-AZ Costs via Topology Aware Routing

A three-tier microservice architecture spread across three Availability Zones looks resilient on paper, but the AWS bill tells a different story: $0.01/GB in each direction for inter-AZ transfer, multiplied across every east-west call a service mesh

#The Bottleneck: Kubernetes Doesn’t Care Where Your Traffic Lands

By default, a Kubernetes Service of type ClusterIP load-balances across every ready endpoint in the cluster, irrespective of which zone the calling pod lives in. kube-proxy’s iptables or IPVS rules apply uniform random selection. If you run three replicas of a downstream service split evenly across eu-west-1a, eu-west-1b, and eu-west-1c, roughly two-thirds of your calls will cross an AZ boundary purely by chance. Multiply that by a service mesh

The naive fix — pinning workloads to a single AZ — destroys the fault tolerance you deployed multi-AZ for in the first place. The correct fix is topology aware routing: preferring same-zone endpoints when they exist and are healthy, and only falling back to cross-zone traffic when local capacity is exhausted or unavailable. This preserves failure-domain isolation while collapsing the common-case network path to intra-zone, which is both free and lower-latency (sub-millisecond versus 1-2ms typical for inter-AZ hops within a region).

#Architectural Breakdown: Where Topology Awareness Actually Lives

There are three distinct layers where topology aware routing can be implemented, and they are not mutually exclusive:

• kube-proxy / EndpointSlice layer — native Kubernetes Topology Aware Hints, which annotates EndpointSlices with zone hints consumed directly by kube-proxy at the iptables/IPVS level. No sidecar required.
• Service mesh data plane — Istio, Linkerd, or Cilium’s mesh mode implement locality-weighted load balancing, where the Envoy or eBPF
  The KBY Lexicon

  eBPF (Extended Berkeley Packet Filter)

  eBPF is a Linux kernel virtual machine that runs sandboxed, verified programs at kernel hook points for near-native-speed observability, networking, and security enforcement without kernel modules.

  Read Full Context
  -based proxy actively scores endpoints by zone proximity and applies weighted distribution with automatic failover.
• CNI-level enforcement — Cilium’s ClusterMesh and native routing can apply zone-aware policy directly at the eBPF datapath, bypassing iptables entirely and reducing per-packet overhead.

Each layer trades off granularity against operational complexity. The EndpointSlice hint mechanism is the lowest-friction option because it requires no additional control plane component, but it is coarse: it operates on proportional capacity per zone rather than real-time latency or load, and it silently disables itself if zone distribution becomes too skewed (more on that in the failure modes section). Service mesh locality LB gives you weighted failover and outlier detection, at the cost of running a full sidecar proxy fleet and its associated CPU/memory tax — typically 50-150m CPU and 64-128Mi memory per pod for Envoy alone.

#How Topology Aware Hints Actually Decide Placement

The EndpointSlice controller calculates the proportion of endpoints that should be allocated per zone based on the ratio of schedulable CPU in that zone versus total cluster CPU, then annotates each endpoint with a hints.forZones field. kube-proxy reads this field and restricts its selection pool to same-zone endpoints for a given client, unless the local pool has zero healthy backends, in which case it falls back to the full endpoint set. This means topology aware routing under this model is fundamentally a capacity allocation problem, not a live traffic-shaping one — it doesn’t react to instantaneous load, only to the standing endpoint topology.

#Implementation Logic

Rolling out topology aware routing safely follows a fixed sequence, regardless of which layer you’re targeting:

• Audit current zone distribution of both nodes and pod replicas per service using kubectl get pods -o wide cross-referenced against node labels.
• Ensure replica counts are high enough and evenly spread across zones — Topology Aware Hints degrade to disabled if any single zone holds fewer than roughly 20 endpoints relative to cluster size, or if the distribution ratio breaches internal safety thresholds.
• Enable the feature at the Service level via annotation, not cluster-wide, so you can validate on a low-risk service first.
• Instrument cross-AZ byte counters before rollout using CNI-level flow logs (Cilium Hubble or VPC Flow Logs) to establish a baseline.
• Roll out incrementally, then compare inter-AZ transfer metrics over a full traffic cycle (minimum 24 hours to capture diurnal patterns).
• If running a mesh, layer locality-weighted load balancing on top for graceful failover behaviour that EndpointSlice hints alone don’t provide.

#Code and Configuration

Enabling native Topology Aware Hints requires only a Service-level annotation on clusters running Kubernetes 1.23+ (GA since 1.27):

1apiVersion: v1
2kind: Service
3metadata:
4  name: pricing-api
5  annotations:
6    service.kubernetes.io/topology-mode: "Auto"
7spec:
8  selector:
9    app: pricing-api
10  ports:
11    - port: 8080
12      targetPort: 8080
13  type: ClusterIP

For a full Istio-based approach with active failover and outlier detection, define locality load balancing on the DestinationRule so that the mesh prefers same-zone endpoints and only spills over cross-AZ when local capacity is degraded:

1apiVersion: networking.istio.io/v1beta1
2kind: DestinationRule
3metadata:
4  name: pricing-api-locality
5spec:
6  host: pricing-api.prod.svc.cluster.local
7  trafficPolicy:
8    loadBalancer:
9      localityLbSetting:
10        enabled: true
11        distribute:
12          - from: "region1/zone1/*"
13            to:
14              "region1/zone1/*": 80
15              "region1/zone2/*": 10
16              "region1/zone3/*": 10
17    outlierDetection:
18      consecutive5xxErrors: 5
19      interval: 30s
20      baseEjectionTime: 60s

To validate the actual financial impact, query cross-zone byte counts directly from Cilium Hubble metrics rather than relying on cloud billing lag, which can trail by 24-48 hours:

1hubble observe --since 1h 
2  --protocol tcp 
3  -o json | jq -r '
4    select(.source.labels[] | contains("zone=eu-west-1a")) |
5    select(.destination.labels[] | contains("zone=eu-west-1b") or contains("zone=eu-west-1c")) |
6    .verdict' | sort | uniq -c

Running this before and after enabling topology aware routing gives an immediate, billing-independent signal of whether cross-zone flow volume actually dropped, which you should treat as the primary success metric rather than trusting the annotation to “just work”.

#Failure Modes and Edge Cases

Topology aware routing fails silently more often than it fails loudly, which is precisely why it needs direct flow-level validation rather than configuration-level trust. The documented behaviour in the upstream Kubernetes networking documentation explicitly notes that hints are dropped — reverting to cluster-wide random distribution — under several conditions:

• Insufficient endpoints per zone — if a zone’s proportional CPU share doesn’t map cleanly to at least one endpoint, hints for that zone are removed and traffic reverts to full random distribution for the whole Service.
• Uneven node CPU allocation — heterogeneous instance types across zones skew the allocation ratio, causing the controller to disable hints entirely rather than risk hot-spotting a small zone.
• Rolling deployments — during a rollout, EndpointSlice churn can temporarily disable hints for several minutes per Service, briefly reverting to cross-zone traffic and masking regressions in short observation windows.
• Autoscaling lag — HPA-driven scale events change the endpoint count faster than the hint recalculation loop, producing brief periods of imbalance immediately after scale-up or scale-down.

On the service mesh side, locality-weighted load balancing introduces a different class of failure: if outlierDetection thresholds are too aggressive, a transient local blip (e.g. a GC pause on the in-zone pod) triggers ejection and forces failover cross-zone precisely when you wanted resilience without cost penalty. Tune consecutive5xxErrors and baseEjectionTime against real p99 latency data, not defaults copied from vendor documentation, since default thresholds are calibrated for availability, not cost containment.

A subtler edge case appears in multi-cluster or architectural patterns involving service mesh federation: ClusterMesh-style topologies span zones by design, so topology aware routing must be scoped per-cluster, not globally, or you risk routing traffic to a technically “closer” endpoint in a peer cluster that actually incurs cross-region transit charges far higher than the intra-region cross-AZ cost you were trying to eliminate.

#Scaling and Security Trade-offs

Adopting topology aware routing is not a free lunch. It changes the blast radius calculus for zone failures and introduces new observability requirements that need to be budgeted into the rollout:

• Cost vs resilience — aggressive zone-locality settings reduce transfer costs by 40-70% in typical east-west heavy architectures, but reduce effective redundancy during a full zone outage, since traffic that would have naturally spread now concentrates recovery load onto surviving zones’ local capacity.
• Latency vs correctness — for stateful services (databases, caches with write quorum), forcing zone-local reads can serve stale data if replication lag exists between zones; topology aware routing should generally be scoped to stateless, idempotent-read services first.
• Sidecar overhead vs native hints — EndpointSlice-based hints add zero runtime resource cost but offer no active health-based failover; mesh-based locality LB adds 50-150m CPU per pod but provides outlier detection and weighted distribution.
• Security posture — concentrating traffic within a zone reduces the network attack surface exposed to inter-AZ transit and simplifies mTLS certificate scoping, but also means a compromised node has a higher probability of intercepting traffic destined for services it would previously never have seen under random distribution.
• Operational complexity — every additional locality rule is another piece of config that must be kept in sync with actual node topology; a botched cordon or zone-drain during maintenance can silently disable hints cluster-wide without alerting unless you’re monitoring the EndpointSlice hint annotations directly, not just Service health.

None of this argues against topology aware routing — the cost savings and latency improvements are real and measurable within a single billing cycle — but it does mean the rollout needs the same rigour as any other traffic-shaping change: staged deployment, flow-level validation, and explicit monitoring of the hint-disablement conditions rather than assuming the annotation guarantees behaviour.

Treat topology aware routing as a capacity-planning discipline rather than a one-off annotation, and revisit the zone distribution ratios every time node pools or replica counts change materially.