Incident Overview
At 02:14 UTC a scheduled minor-version upgrade of the primary PostgreSQL 15 cluster (RDS Multi-AZ, three read replicas across eu-west-1a/b/c) completed with no reported errors. By 03:40 UTC, replica lag on eu-west-1c climbed from a steady 80ms baseline to over 240 seconds, while the other two replicas remained under 200ms.
At 04:05 UTC application error rates on the checkout service rose to 3.2%, with connection pool exhaustion warnings from PgBouncer showing 'server login has been failing' intermittently against the lagging replica, which is still in the read-only routing pool. CloudWatch shows WAL generation on the primary at 2.4x normal throughput; pg_stat_replication on the primary reports the c replica's write_lag and flush_lag both climbing linearly, with no corresponding disk I/O saturation on that instance (EBS gp3 burst balance at 94%).
Meanwhile, autovacuum on a 900GB orders table has been running for six hours, holding a ShareUpdateExclusiveLock, and pg_stat_activity shows eleven backend connections in 'waiting' state against that table since 03:50 UTC. Slow query logs show a spike in sequential scans against orders_history correlating with a recently deployed reporting feature that bypasses the expected index due to a planner statistics mismatch post-upgrade (pg_stat_user_tables shows last_analyze timestamp is stale, predating the upgrade).
Two hypotheses are live: first, that the minor-version upgrade altered checkpoint or WAL compression behaviour causing replication lag to cascade under increased load; second, that the stale planner statistics combined with the long-running autovacuum are independently driving lock contention and query degradation, with the replica lag being a downstream symptom of primary-side write amplification rather than a network or replica-local issue. Runbooks for planned failover exist but require a 90-second connection-draining window that checkout cannot currently tolerate without breaching its SLA.
The DBA on rotation is unreachable for another 40 minutes.
Investigation Options
Review the available operational moves and select the best immediate action.
Immediately run ANALYZE on orders and orders_history to refresh planner statistics, and in parallel query pg_stat_progress_vacuum to assess remaining autovacuum duration before deciding whether to terminate it.
Remove eu-west-1c from the PgBouncer read-only pool immediately to stop further connection attempts against the lagging replica, redistributing load across the two healthy replicas while investigating WAL throughput on the primary.
Initiate a controlled failover to a healthy replica using the existing runbook, accepting the 90-second connection-draining window now on the basis that continued primary-side write amplification will breach SLA more severely if left unaddressed.
Cancel the long-running autovacuum process on the orders table to release the ShareUpdateExclusiveLock and unblock the eleven waiting backends, then manually schedule a lower-impact vacuum during the next maintenance window.