One incident. Four plausible moves. Instant technical reasoning.
Incident brief
At 02:14 UTC a scheduled minor-version upgrade of the primary PostgreSQL 15 cluster (RDS Multi-AZ, three read replicas across eu-west-1a/b/c) completed with no reported errors. By 03:40 UTC, replica lag on eu-west-1c climbed from a steady 80ms baseline to over 240 seconds, while the other two replicas remained under 200ms.
At 04:05 UTC application error rates on the checkout service rose to 3.2%, with connection pool exhaustion warnings from PgBouncer showing 'server login has been failing' intermittently against the lagging replica, which is still in the read-only routing pool. CloudWatch shows WAL generation on the primary at 2.4x normal throughput; pg_stat_replication on the primary reports the c replica's write_lag and flush_lag both climbing linearly, with no corresponding disk I/O saturation on that instance (EBS gp3 burst balance at 94%).
Meanwhile, autovacuum on a 900GB orders table has been running for six hours, holding a ShareUpdateExclusiveLock, and pg_stat_activity shows eleven backend connections in 'waiting' state against that table since 03:50 UTC. Slow query logs show a spike in sequential scans against orders_history correlating with a recently deployed reporting feature that bypasses the expected index due to a planner statistics mismatch post-upgrade (pg_stat_user_tables shows last_analyze timestamp is stale, predating the upgrade).
Two hypotheses are live: first, that the minor-version upgrade altered checkpoint or WAL compression behaviour causing replication lag to cascade under increased load; second, that the stale planner statistics combined with the long-running autovacuum are independently driving lock contention and query degradation, with the replica lag being a downstream symptom of primary-side write amplification rather than a network or replica-local issue. Runbooks for planned failover exist but require a 90-second connection-draining window that checkout cannot currently tolerate without breaching its SLA.
The DBA on rotation is unreachable for another 40 minutes.
Given the SLA constraints, the ambiguous causality between the upgrade and the statistics/autovacuum issue, and the ticking risk of full connection pool saturation across all read replicas, what should the on-call Staff Engineer do next?
Make your initial decision, then use the deep dive to investigate the underlying failure mode.
Open technical deep dive