Building Site-to-Site VPN Failover Between Branch and AWS

#Operational requirement

#Pre-implementation access review

#Route table propagation verification

Before signing off, confirm route propagation is actually enabled on the target route table rather than assumed. A tunnel showing UP with BGP Established can still fail to insert routes if propagation was never switched on for that specific table, particularly when a VPC has several route tables and only one was updated during initial build.

1aws ec2 enable-vgw-route-propagation --route-table-id rtb-0a1b2c3d --gateway-id vgw-0bbb2222
2
3aws ec2 describe-route-tables --route-table-ids rtb-0a1b2c3d --query "RouteTables[].Routes"

Check every subnet-associated route table that branch traffic could traverse, not just the one used during the lab build — a workload sitting in a subnet associated with an unmodified table will silently lose reachability during failover while everything else appears healthy.

#

Evidence to retain

Capture the output of describe-route-tables immediately after enabling propagation and again after the failover test, showing the destination CIDR pointing at the correct VGW attachment ID in both states. Auditors reviewing the change ticket later will want to see the route source shown as propagated rather than static, confirming BGP — not a manually entered route — is driving the path.

#Health check dependency on the branch router

CloudWatch’s TunnelState metric reflects the IPsec tunnel state as seen from the AWS side; it does not confirm the branch router’s BGP process is healthy. A router that has exhausted its BGP table memory or is mid-reload can show a tunnel as cryptographically up while still failing to advertise routes. Pair the CloudWatch alarm with an independent reachability probe from a host inside the branch LAN, scheduled outside the router itself, so a router-level fault doesn’t also disable the thing meant to detect it.

#

Symptom: alarm silent, users still affected

If users report an outage but no CloudWatch alarm has fired, check whether the affected subnet’s route table was excluded from propagation, per the section above, before assuming the alarm threshold is misconfigured. This is a distinct fault class from tunnel failure and won’t appear in TunnelState at all.

#Change-control record additions

Alongside the evidence already listed, record the exact route table IDs modified, the propagation enable/disable timestamps, and the identity of the engineer who ran the command. Where a route table serves production and non-production subnets together, note explicitly which subnets were in scope for this change, since a shared table is a common source of unintended blast radius during rollback.

#

Rollback addition

If route propagation must be reverted, disable it on the specific route table before deleting the VPN connection object, and confirm static routes are restored where required, otherwise the affected subnets are left with no path at all rather than reverting to the prior known-good state.

1aws ec2 disable-vgw-route-propagation --route-table-id rtb-0a1b2c3d --gateway-id vgw-0bbb2222

Before touching production routing, confirm the engineer’s IAM role has been scoped rather than inherited from a broader administrator policy. A common oversight is running these commands under a role that also holds ec2:DeleteRoute or ec2:ReplaceRoute against unrelated route tables, which turns a VPN build into an accidental blast-radius incident. Request a time-bound elevation through your identity provider, log the request ticket number against the change record, and confirm the elevation expires automatically rather than relying on someone remembering to revoke it.

#

Segregation of duties

Whoever approves the change record should not be the same person executing the router configuration. On a two-person operations team this can feel bureaucratic, but the point is a second set of eyes checking the ASN allocation and PSK handling before anything touches the live VGW. Record the approver’s name, the ticket reference and the approval timestamp in the same document that holds the BGP summaries — auditors will ask for all three together, not separately.

#Configuration detail often missed

#

Dead peer detection alignment

AWS generates specific DPD timeout and retry values per tunnel in the downloaded configuration file. If the branch router’s IPsec profile uses a locally defined DPD interval instead of the AWS-generated one, the tunnel can appear UP in the console while the router considers the peer dead, producing a black-hole state that ping alone won’t reveal until the hold timer expires. Pull the exact values with:

1show crypto isakmp policy
2show crypto session detail

Compare the DPD interval and retry count against the AWS-supplied configuration file line by line before accepting the tunnel as correctly built.

#

MTU and fragmentation

IPsec overhead reduces the effective MTU across the tunnel. Where the branch router doesn’t clamp TCP MSS, large transfers can succeed on ping but stall on file copies once encapsulated packets exceed path MTU. Set an explicit MSS clamp on the tunnel interface:

1interface Tunnel1
2 ip tcp adjust-mss 1387

Confirm with a large-payload transfer test, not just ICMP, before declaring the tunnel production-ready.

#Monitoring and alerting

A tunnel that fails silently outside business hours is worse than one that fails loudly during a change window. Configure CloudWatch alarms against the TunnelState metric for both VPN connections, with a threshold that pages on-call after two consecutive missed data points rather than a single blip, which reduces noise from transient carrier-side maintenance.

#

Threshold guidance

Route the critical alarm through the same paging channel used for other production routing incidents so the on-call engineer doesn’t need to know this is a VPN-specific alert to respond correctly.

#Realistic failure symptoms during operation

#

Slow convergence under load

Convergence measured during a quiet test window can differ from convergence during a genuine failure at peak traffic. If the branch router’s CPU is already under load handling other BGP peers or NAT translations, BGP hold-timer expiry detection can be delayed, pushing real-world failover past the 90-second figure captured in testing. Re-run the shutdown test during a representative traffic period at least once before final sign-off, and note the discrepancy if one exists.

#

Route flapping after partial circuit degradation

A circuit that is degraded rather than fully down — intermittent packet loss instead of a clean outage — can cause the BGP session to flap repeatedly rather than fail over cleanly. This produces a worse user experience than a single clean cutover because traffic oscillates between tunnels. Watch for repeated Established/Idle transitions in show ip bgp summary output and treat more than three flaps within an hour as a circuit-quality issue requiring ISP escalation rather than a VPN configuration fault.

#Change-control record keeping

The change ticket should carry more than a pass/fail note. Attach the pre-change route table export, the post-change route table export, both BGP summary captures, the timestamped ping log, and the CloudWatch alarm configuration screenshot. Where the build deviates from the standard — for example a non-default ASN or a manually adjusted local-preference — record the reason inline in the ticket rather than only in router comments, since router comments are lost on a config reload.

#

Escalation path

If convergence exceeds 90 seconds after two consecutive re-tests with correctly matched timers, escalate to the network engineering lead before attempting further timer tuning — repeatedly lowering hold timers without understanding the root cause risks session instability under normal conditions. If PSK mismatches recur after a Customer Gateway rebuild, escalate to whoever holds the secrets vault administration rights rather than regenerating credentials repeatedly, since repeated regeneration without process review tends to mask a deeper handover gap between infrastructure and network teams.

A branch office loses its primary circuit at 08:14 on a Monday morning. If the site-to-site VPN to AWS is pinned to a single tunnel on a single Customer Gateway, that branch is now cut off from every AWS-hosted workload — file shares, line-of-business apps, RADIUS. Tickets pile up, the on-call engineer gets paged, and someone asks why redundancy wasn’t built in from day one.

Your task as the newly hired engineer: build a dual-tunnel VPN across two Customer Gateways on two separate ISP circuits, get BGP failover working, and produce proof it actually converges before change management will sign off. This is a routing change to production. Treat it as one.

#Prerequisites

#Step 1 — Provision two Customer Gateways and a BGP-based VPN connection

Create one Customer Gateway per circuit, then a VPN connection using dynamic routing, not static routes.

1aws ec2 create-customer-gateway --type ipsec.1 --public-ip 203.0.113.10 --bgp-asn 65001 --device-name "Branch-ISP-A"
2
3aws ec2 create-customer-gateway --type ipsec.1 --public-ip 198.51.100.20 --bgp-asn 65001 --device-name "Branch-ISP-B"
4
5aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id cgw-0aaa1111 --vpn-gateway-id vgw-0bbb2222 --options "{"StaticRoutesOnly":false}"

Capture the JSON output: VpnConnectionId, tunnel outside IPs, pre-shared keys. PSKs go into a secrets vault, never into a ticket comment. Do not proceed until aws ec2 describe-customer-gateways shows both gateways as available.

#Step 2 — Configure branch routers with IPsec and BGP

Match the IKE version, DPD timers and encryption suite exactly as AWS generated them. Then bring up BGP per tunnel.

1router bgp 65001
2 neighbor 169.254.100.1 remote-as 64512
3 neighbor 169.254.100.1 timers 10 30
4 address-family ipv4
5  network 10.50.0.0 mask 255.255.255.0
6  neighbor 169.254.100.1 activate

Prefer the primary circuit with a higher local-preference or lower MED; ISP-B stays passive backup

#Step 3 — Force failover and time the convergence

Shut the tunnel interface, not the physical WAN link, so the test stays reversible.

1interface Tunnel1
2 shutdown

Run continuous probes throughout:

1ping -t 10.60.0.15

Expect traffic to shift to Tunnel2 inside the BGP hold-timer window — typically under 90 seconds with the timers above. Capture the ping output showing exact dropped packets and recovery timestamp, plus show ip bgp summary confirming Tunnel2 as active with correct AS path length. This is the evidence change management will ask for.

#Verification

Bring Tunnel1 back up and confirm traffic reverts to the primary path once BGP re-establishes and preference is restored. Re-check both tunnel states in the AWS console, confirm route table propagation matches expectations, and archive all captured evidence — JSON output, BGP summaries, ping logs — against the change ticket.

1show ip bgp summary
2aws ec2 describe-vpn-connections --vpn-connection-ids vpn-0123456789abcdef0

#Failure Modes

• BGP sessions establish but routes never propagate — usually a missing network statement or route table not associated with the VGW.
• Failover takes far longer than expected — hold timers left at defaults (180 seconds) instead of the tuned 10/30 values.
• Both tunnels show UP but traffic still drops during the test — asymmetric routing because the branch LAN default route wasn’t updated to prefer the VPN path.
• PSK mismatch after a rebuild — someone regenerated the Customer Gateway without updating the router config, and the tunnel silently fails to rekey.

#Rollback

If failover testing causes unexpected outages, immediately bring Tunnel1 back up, verify BGP re-establishes, and revert any manual local-preference changes on the router. If the entire build is unstable, delete the VPN connection and Customer Gateway objects, restore the previous single-tunnel configuration from backup, and document why in the change ticket before reattempting.

1aws ec2 delete-vpn-connection --vpn-connection-id vpn-0123456789abcdef0
2aws ec2 delete-customer-gateway --customer-gateway-id cgw-0aaa1111

#Operational Summary

The diagram below shows the intended failover path.

A dual-tunnel, BGP-backed VPN turns a single-circuit outage from a full branch disconnection into a brief, self-healing reroute. The value isn’t the configuration itself — it’s the evidence trail: BGP summaries, route table screenshots and timed packet-loss logs that prove convergence works before production sign-off, and that give the next engineer on call something concrete to check when a circuit fails at 08:14 on a Monday.