Testing and Verifying AWS RDS Multi-AZ Failover in Production

#Operational requirement

Every production RDS instance that carries a Multi-AZ flag in the console is not automatically fault-tolerant. Multi-AZ only protects you if the failover mechanism has actually been exercised, the application reconnects to the endpoint rather than a cached IP, and someone has timed how long the outage actually lasts. I have seen a "highly available" customer database go dark for eleven minutes during an unplanned AZ event because nobody had tested failover since the instance was provisioned eighteen months earlier. The connection string was hardcoded to a resolved IP address from a legacy deployment script, so DNS failover meant nothing to that consumer.

Your assignment: produce a documented, repeatable failover test for a nominated production RDS instance, capture the actual recovery time, and confirm the application tier reconnects without manual intervention. This is not a one-off exercise. It becomes part of the pre-go-live checklist for every new production database, and it gets re-run after any instance class change, engine version upgrade, or parameter group modification that touches the DB instance. Treat this the same way you would treat a fire drill: scheduled, evidenced, and boring when it works.

#Prerequisites and required permissions

You need an IAM identity (user or role, ideally an SSO-federated role rather than a static IAM user) with the minimum actions below. Do not request AdministratorAccess for this task; it will get flagged in the next access review and you will be asked to justify it.

1{
2  "Version": "2012-10-17",
3  "Statement": [
4    {
5      "Sid": "RDSFailoverTest",
6      "Effect": "Allow",
7      "Action": [
8        "rds:DescribeDBInstances",
9        "rds:DescribeEvents",
10        "rds:DescribeEventSubscriptions",
11        "rds:CreateEventSubscription",
12        "rds:DeleteEventSubscription",
13        "rds:RebootDBInstance",
14        "rds:ModifyDBInstance"
15      ],
16      "Resource": "arn:aws:rds:eu-west-1:123456789012:db:prod-customer-db"
17    },
18    {
19      "Sid": "SNSForAlerts",
20      "Effect": "Allow",
21      "Action": ["sns:CreateTopic", "sns:Subscribe", "sns:SetTopicAttributes"],
22      "Resource": "arn:aws:sns:eu-west-1:123456789012:rds-failover-alerts"
23    },
24    {
25      "Sid": "CloudWatchReadOnly",
26      "Effect": "Allow",
27      "Action": ["cloudwatch:GetMetricData", "cloudwatch:GetMetricStatistics"],
28      "Resource": "*"
29    }
30  ]
31}

You also need an approved change record before you touch anything. Multi-AZ failover testing on a live production database is a change with customer-visible impact, however brief, so it must go through your standard change management gate (ServiceNow, Jira Service Management, or whatever your organisation uses) with a scheduled maintenance window, a named approver, and a rollback plan attached, even though, as you will see, there is very little to actually roll back. Run this out of hours or during a declared low-traffic window unless your application has already been proven to survive a short connection blip gracefully.

#Step 1: Confirm the Multi-AZ configuration and identify the standby

Action: run the following against the target instance to confirm Multi-AZ is actually enabled and to record which AZ currently hosts the primary and which hosts the standby.

1aws rds describe-db-instances 
2  --db-instance-identifier prod-customer-db 
3  --query 'DBInstances[0].[MultiAZ,AvailabilityZone,SecondaryAvailabilityZone,DBInstanceStatus,Engine,EngineVersion]' 
4  --output table

Expected result: MultiAZ shows True, and SecondaryAvailabilityZone is populated with a real AZ name different from the primary. If MultiAZ shows False, stop here. You cannot test failover on a single-AZ instance, and this needs raising as a separate finding, not silently fixed on the spot.

Evidence to capture: paste the CLI output into the change record as the pre-test baseline.

Checkpoint: do not proceed to Step 2 until Multi-AZ is confirmed enabled and you know both AZ names by heart.

#Step 2: Enable RDS event notifications for failover events

Action: create an event subscription so failover start and completion are pushed to an SNS topic you control, rather than relying on polling and guesswork.

1aws sns create-topic --name rds-failover-alerts
2
3aws rds create-event-subscription 
4  --subscription-name prod-db-failover-alerts 
5  --sns-topic-arn arn:aws:sns:eu-west-1:123456789012:rds-failover-alerts 
6  --source-type db-instance 
7  --event-categories failover 
8  --source-ids prod-customer-db 
9  --enabled

Expected result: Status returns creating, then transitions to active within a minute or two. Confirm with aws rds describe-event-subscriptions --subscription-name prod-db-failover-alerts.

Evidence to capture: the subscription ARN and its active status, plus proof of a confirmed SNS subscriber, whether email or a webhook into your alerting tool such as PagerDuty or Opsgenie.

Checkpoint: do not proceed until the subscription shows active and at least one endpoint has confirmed the SNS subscription. Unconfirmed subscriptions receive nothing, silently.

#Step 3: Baseline the application connection and record RTO expectations

Action: from the application tier, or a bastion host on the same network path, open a persistent connection to the RDS endpoint, the DNS name, never a resolved IP, and run a lightweight polling loop that logs timestamped success or failure.

1while true; do
2  psql -h prod-customer-db.abc123xyz.eu-west-1.rds.amazonaws.com 
3    -U app_readonly -d appdb -c "SELECT 1;" >> failover-test.log 2>&1
4  echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) exit=$?" >> failover-test.log
5  sleep 2
6done

Expected result: a continuous log of successful one-to-two-second-resolution checks against the endpoint before the failover event is triggered.

Evidence to capture: the running log file, and confirmation that the application configuration references the RDS DNS endpoint rather than a hardcoded IP. If you find a hardcoded IP anywhere in application config, stop and flag it immediately. Failover will not help that consumer regardless of how well RDS performs underneath.

Checkpoint: at least sixty seconds of clean, unbroken polling logged before you move to Step 4.

#Step 4: Trigger the controlled failover

Action: force a failover using the reboot API with the --force-failover flag. This does not reboot the instance in place; it promotes the standby and demotes the current primary.

1aws rds reboot-db-instance 
2  --db-instance-identifier prod-customer-db 
3  --force-failover

Expected result: DBInstanceStatus transitions from available to rebooting, briefly to failing-over, then back to available. Monitor with:

watch -n 5 "aws rds describe-db-instances --db-instance-identifier prod-customer-db --query 'DBInstances[0].DBInstanceStatus' --output text"

Evidence to capture: the exact timestamp the command was issued, and the exact timestamp status returned to available.

Checkpoint: never issue a second failover attempt while the first is in progress. RDS will reject it, and stacking failover requests can extend the outage instead of shortening it.

#Step 5: Verify failover completion and application reconnection

Action: pull the RDS event log for the instance and cross-reference it against your polling log from Step 3.

1aws rds describe-events 
2  --source-identifier prod-customer-db 
3  --source-type db-instance 
4  --duration 60

Expected result: an event entry describing failover starting, followed later by one describing failover completion, with a timestamp gap that represents your actual recovery time. In practice this is typically well under two minutes for most engine and instance-class combinations, but the only number that matters is the one you measured, never a published average from a vendor page.

Evidence to capture: the failed connection window from failover-test.log (the gap between successful exit=0 lines), the RDS event timestamps, and confirmation that the AZ has swapped by re-running the Step 1 describe command; AvailabilityZone should now show what was previously SecondaryAvailabilityZone.

Checkpoint: the application reconnected automatically without a restart, redeploy, or manual DNS flush. If it did not, that is a real finding, not a test failure to quietly bury.

#Verification

Your completed evidence pack for the change record should contain: the pre-test describe-db-instances output, the active event subscription confirmation, the full failover-test.log showing the outage window, the describe-events output showing failover start and completion timestamps, the post-test AZ swap confirmation, and a one-line summary of measured recovery time signed off against the ticket. Store this in your runbook repository, not only in the ticket, because the next engineer testing a different instance will want the template rather than starting from a blank page.

#Change control and monitoring

This is a scheduled change, never an ad hoc one. Book the maintenance window with the application owner, agree an acceptable outage ceiling before you start (commonly under two minutes for a well-configured instance), and set a CloudWatch alarm on DatabaseConnections or a custom application heartbeat metric so you have a second, independent view of the outage window beyond your own polling loop. After the test, leave the event subscription in place for ongoing operational awareness rather than deleting it purely because the test finished; unexpected failovers happen outside test windows too, and you want to be told immediately, not discover it from a customer ticket the next morning.

#Failure Modes

#Rollback

There is no data-level rollback for a failover test itself. RDS does not let you force the standby back to the original AZ, and doing so is not a genuine requirement anyway, because both AZs are equally valid production infrastructure. What you do need to reverse cleanly are the artefacts you created purely for the test.

1# Remove the event subscription if it was created solely for this test
2aws rds delete-event-subscription --subscription-name prod-db-failover-alerts
3
4# Delete the SNS topic if no other alerting depends on it
5aws sns delete-topic --topic-arn arn:aws:sns:eu-west-1:123456789012:rds-failover-alerts
6
7# If Multi-AZ was enabled temporarily purely to run this test on an
8# instance that should remain single-AZ for cost reasons, disable it explicitly:
9aws rds modify-db-instance 
10  --db-instance-identifier prod-customer-db 
11  --no-multi-az 
12  --apply-immediately

Close the change record with the measured recovery time and mark the maintenance window as complete. If the test revealed a hardcoded IP or a broken reconnection path, open a separate remediation ticket. Do not fold an application fix into the same change as an infrastructure verification test; keep the two audit trails distinct.

#Operational Summary

Multi-AZ is a checkbox until someone forces a failover and measures what actually happens. This runbook gives you a repeatable, evidenced way to prove it, using nothing beyond standard AWS CLI calls and IAM permissions scoped to exactly what the task needs, no more. Run it before go-live for every new production RDS instance, after every engine version or instance class change, and on a fixed recurring schedule, quarterly is a reasonable default for most estates, so nobody discovers a broken reconnection path during a real outage instead of a planned drill. Keep the evidence pack. Auditors and incident reviewers will ask for it eventually, and having it ready beats reconstructing it from memory under pressure.

For the underlying failover mechanics and event category reference, see the AWS documentation on Multi-AZ deployments and the AWS documentation on RDS event notification categories.