AWS Site-to-Site VPN Tunnel Down After BGP Flap to Azure: Fix

#Current method: why the tunnel keeps dropping

An AWS Site-to-Site VPN to Azure joins two provider-managed gateways: an AWS virtual private gateway or transit gateway, and an Azure VPN gateway. Each side has its own defaults, its own loggingThe KBY LexiconLoggingLogging is the practice of recording timestamped system events to durable storage for later diagnosis, correlation and audit. Read Full Context and its own owner. A common current method is to restart or reset a tunnel when it drops and hope it stays up. That clears the symptom but leaves no evidence, so the same flap returns.

The delay comes from three layers failing in different ways. IKE phase 1 authenticates the peers. Phase 2 builds the IPsec security associations that carry traffic. BGP then runs inside the tunnel to exchange routes. A BGP flap can be a consequence of a lower layer dropping, or a cause of its own, for example through mismatched ASNs, inside addresses or timers. Treating them as one problem hides which layer broke first.

Assumptions in this playbook: you hold read access to both clouds and change rights only through your normal change process. Version-specific limits, supported algorithms and timer defaults differ by gateway SKU and service version, so confirm them in the vendor documentation listed in the sources. The AWS Well-Architected Reliability Pillar documents failure management and operational resilience practices, which is the framing used here.

#Improved workflow: layer by layer, one change at a time

The improved workflow works from the bottom up and records evidence at each layer before changing anything.

1. Establish the timeline. Note when the tunnel state changed on each side. This tells you whether BGP dropped first or the tunnel did.
2. Check phase 1. Confirm IKE version, pre-shared key, encryption, integrity, DH group and lifetime agree on both sides. Mismatch here means the tunnel never establishes.
3. Check phase 2. Confirm IPsec proposals, PFS settings and traffic selectors. Route-based and policy-based behaviour differ, and a selector mismatch can produce a tunnel that is up but passes nothing.
4. Check BGP. Confirm local and peer ASNs, inside tunnel addresses used as peer IPs, and the advertised prefixes. A session that resets repeatedly points to hold-timer expiry, a lower-layer drop or a configuration disagreement.
5. Change one setting, then observe. Changing several at once removes your ability to attribute the fix.

The trade-off is speed against attribution. Slower, single-variable changes cost minutes but leave a defensible record and a clean rollback.

#Implementation

Prerequisites: a non-production or maintenance-window context, read access to both gateways, a change record, and a saved copy of the current configuration of both tunnels.

1. Capture current state. Export tunnel status, IKE and IPsec parameters and BGP neighbour state from both consoles or your approved tooling. Expected evidence: a dated record of each layer’s state. Stop if you cannot read either side.
2. Build a comparison table. List each phase 1, phase 2 and BGP parameter with the AWS value and the Azure value side by side. Expected evidence: every row marked match or mismatch.
3. Resolve one mismatch. Pick the lowest layer with a mismatch and align it through your change process. Stop if the change would affect other tunnels or connections sharing the gateway.
4. Observe. Watch the tunnel for a period agreed in the change record. Expected evidence: phase 1 and phase 2 up, BGP established, expected prefixes received.
5. Repeat for the next mismatch only if the symptom persists.

If both sides match and the tunnel still flaps, collect logs from both gateways for the same time window and escalate to the cloud provider support channel rather than guessing at hidden settings. For access and permission context on the AWS side, see fixing AWS S3 AccessDenied for an EC2 instance role, and for reachability problems after the tunnel is healthy, see fixing an EC2 SSH connection timeout.

#Guardrails

• Never rotate or reveal a pre-shared key in tickets or chat; share it through your approved secrets process.
• Apply least privilege: read-only roles for diagnosis, change roles only for the approved window.
• Do not delete a VPN connection or gateway as a troubleshooting step; recreation can change addresses and break the peer configuration.
• Do not change both tunnels at once if they provide redundancy; alter one, verify, then the other.
• Stop if traffic that matters is running only over the tunnel you are about to modify.

#Validation

1. Phase 1 and phase 2 show as established on both sides.
2. The BGP session stays established through at least one full hold-timer period beyond the original flap interval, as defined in your change record.
3. Each side receives the prefixes the other is expected to advertise.
4. A test connection between a host in AWS and a host in Azure succeeds across the tunnel.
5. No further tunnel state changes appear in logs for the agreed observation period.

#Common mistakes

• Symptom: BGP resets but IKE is stable. Likely cause: ASN, peer address or timer disagreement. Diagnose with BGP neighbour state on both sides.
• Symptom: tunnel up, no traffic. Likely cause: traffic selector or route mismatch. Check phase 2 selectors and route tables.
• Symptom: periodic drops near rekey time. Likely cause: lifetime or rekey behaviour difference. Compare lifetimes and review logs around the drop.
• Resetting repeatedly without notes. This destroys the timeline you need.
• Fixing one side only from memory of defaults. Defaults vary by version; read the live configuration.

#Recovery

Before each change, keep the previous value recorded. If the tunnel gets worse after a change, revert that single setting to its recorded value, confirm phase 1, phase 2 and BGP return to their pre-change state, and record the outcome. If a pre-existing tunnel was healthy for production traffic, restore it first and continue diagnosis on a secondary tunnel. Escalate to the network owner and provider support if state does not return. Overly broad permissions are a related risk when granting change access; see overbroad IAM trust policy principals.

#Measurable outcome

Baseline first: count tunnel state changes and BGP session resets per week from existing logs, and record time to restore for recent incidents. The success signal is fewer state changes and a shorter time to identify the failing layer. Review weekly for a month, then monthly. A reasonable threshold is that any recurrence triggers a comparison-table review. No improvement figure is claimed here; your baseline sets the target.

#Checklist

1. Timeline of state changes captured for both sides.
2. Phase 1 parameters compared and matched.
3. Phase 2 proposals and selectors compared and matched.
4. BGP ASNs, peer IPs, timers and prefixes verified.
5. One change at a time, each with a recorded rollback value.
6. Observation period passed with no state changes.
7. Baseline metricsThe KBY LexiconMetricsMetrics are numeric time-series measurements used to detect, diagnose and validate recovery from operational problems. Read Full Context recorded and review date set.
8. Unresolved cases escalated with logs from both gateways.

#Prerequisites and permissions for evidence gathering

Agree the access model before the maintenance window opens. Diagnosis needs only read permissions, so avoid granting write access until a change is approved.

#

AWS side

The diagnostic role needs the ability to describe VPN connections and read CloudWatch metrics and logs. These calls are read-only:

1aws ec2 describe-vpn-connections --vpn-connection-ids vpn-EXAMPLE --output json
2aws cloudwatch get-metric-statistics --namespace AWS/VPN --metric-name TunnelState \
3  --dimensions Name=VpnId,Value=vpn-EXAMPLE --start-time 2024-01-01T00:00:00Z \
4  --end-time 2024-01-01T06:00:00Z --period 60 --statistics Minimum

Replace the identifier and timestamps with your own values. The first command shows per-tunnel status, outside addresses and any status message the service returns. Save the output unaltered in the change record.

#

Azure side

A reader role scoped to the resource group holding the VPN gateway and connection is sufficient. Use the Azure CLI to export the connection and gateway configuration:

1az network vpn-connection show --name <connection> --resource-group <rg> -o json
2az network vnet-gateway show --name <gateway> --resource-group <rg> -o json

Check the exported configuration for the BGP setting, the peer ASN and the custom IPsec policy, if one is applied. If no custom policy is present, record that fact, because defaults then depend on the gateway version.

#Implementation detail: building a usable timeline

Convert every timestamp to UTC before comparing. Clock offset between consoles is a frequent reason for wrongly concluding that BGP dropped first. Build a simple table with four columns: time, source, event and layer. Include IKE negotiation messages, tunnel state transitions and BGP neighbour changes. Where gateway diagnostic logs are enabled, export the same window from both clouds and sort the merged events chronologically.

If logging was not enabled before the incident, say so in the record and enable it under change control so the next occurrence is captured. Do not claim a root cause from absent evidence.

#

Expected evidence by layer

• Phase 1: a negotiation completing without a rejected proposal or authentication failure message on either side.
• Phase 2: an established child security association that remains after a rekey event.
• BGP: neighbour state established, with the received prefix count matching the agreed list.

#Monitoring after the fix

Alert on tunnel state rather than on user complaints. A CloudWatch alarm on the tunnel state metric for each tunnel separately lets you see redundancy loss before an outage. Set the evaluation period to more than one sample so a single missing datapoint does not page anyone.

1aws cloudwatch put-metric-alarm --alarm-name vpn-tunnel1-down \
2  --namespace AWS/VPN --metric-name TunnelState \
3  --dimensions Name=TunnelIpAddress,Value=<tunnel1-outside-ip> \
4  --statistic Maximum --period 60 --evaluation-periods 3 \
5  --threshold 1 --comparison-operator LessThanThreshold \
6  --treat-missing-data breaching --alarm-actions <topic-arn>

The VpnId dimension reports one value for the whole connection (a fraction between 0 and 1 when only one tunnel is up), while TunnelIpAddress, the tunnel’s AWS-side outside address, gives one series per tunnel, so create one alarm for each tunnel address. On the Azure side, configure an equivalent alert on tunnel connectivity and BGP peer status through Azure Monitor, and route both alerts to the same on-call rota.

#Realistic failure symptoms to record

• Tunnel state flips at regular intervals that match a configured lifetime, suggesting rekey disagreement.
• BGP drops while the tunnel metric stays steady, suggesting a control-plane problem such as hold-timer expiry or a changed prefix limit.
• Only one of two tunnels flaps, suggesting a per-tunnel setting such as a different pre-shared key or inside address.
• Routes disappear after recovery and return minutes later, suggesting slow reconvergence rather than a persistent fault.
• Both tunnels drop together, which points towards a shared component such as the Azure gateway, the AWS gateway or a shared route change.

#Change-control records

Each change entry should hold the following, so another engineer can reproduce or reverse it without asking you:

1. Change identifier, approver, window start and end in UTC.
2. Exact parameter, tunnel and side affected.
3. Previous value and new value (never the pre-shared key itself; reference the secrets entry instead).
4. Pre-change and post-change exports of status, stored with a checksum or ticket attachment.
5. Observation period agreed, and who owns the decision to close.

#Escalation thresholds

Define thresholds before the window, not during it. Suggested triggers, to be adjusted to your service levels:

• Escalate to the network owner if the first single-variable change shows no improvement, or if a second tunnel begins to flap.
• Escalate to provider support if both sides match in the comparison table and the flap recurs, supplying merged UTC logs, connection identifiers and the timeline.
• Declare an incident if production traffic loses its only working path, regardless of diagnostic progress.

#Safe rollback actions

Rollback is a single-setting reversal, not a rebuild. Re-apply the recorded previous value through the same mechanism used for the change, then compare the new export with the pre-change export. If they differ in any field other than a timestamp, treat the rollback as incomplete and pause. Never delete and recreate a connection to achieve a rollback, since outside addresses may change. If the tunnel being modified carried live traffic, confirm the second tunnel is established and receiving the expected prefixes before touching anything further.

Next safe decision: if all checks pass, close the change; if not, revert and escalate rather than widening the change.