Skip to main content
KBY Technologies Logo
Databases & Storage

Scheduled minor-version upgrade of the primary PostgreSQL 15 cluster

Operational scenarios covering relational and distributed databases, replication, consistency, backups, latency, and storage durability.

Daily engineering quiz

One incident. Four plausible moves. Instant technical reasoning.

About 2 minutes
01Read the signals
02Choose a move
03Reveal reasoning

Incident brief

At 02:14 UTC a scheduled minor-version upgrade of the primary PostgreSQL 15 cluster (RDS Multi-AZ, three read replicas across eu-west-1a/b/c) completed with no reported errors. By 03:40 UTC, replica lag on eu-west-1c climbed from a steady 80ms baseline to over 240 seconds, while the other two replicas remained under 200ms.

At 04:05 UTC application error rates on the checkout service rose to 3.2%, with connection pool exhaustion warnings from PgBouncer showing 'server login has been failing' intermittently against the lagging replica, which is still in the read-only routing pool. CloudWatch shows WAL generation on the primary at 2.4x normal throughput; pg_stat_replication on the primary reports the c replica's write_lag and flush_lag both climbing linearly, with no corresponding disk I/O saturation on that instance (EBS gp3 burst balance at 94%).

Meanwhile, autovacuum on a 900GB orders table has been running for six hours, holding a ShareUpdateExclusiveLock, and pg_stat_activity shows eleven backend connections in 'waiting' state against that table since 03:50 UTC. Slow query logs show a spike in sequential scans against orders_history correlating with a recently deployed reporting feature that bypasses the expected index due to a planner statistics mismatch post-upgrade (pg_stat_user_tables shows last_analyze timestamp is stale, predating the upgrade).

Two hypotheses are live: first, that the minor-version upgrade altered checkpoint or WAL compression behaviour causing replication lag to cascade under increased load; second, that the stale planner statistics combined with the long-running autovacuum are independently driving lock contention and query degradation, with the replica lag being a downstream symptom of primary-side write amplification rather than a network or replica-local issue. Runbooks for planned failover exist but require a 90-second connection-draining window that checkout cannot currently tolerate without breaching its SLA.

The DBA on rotation is unreachable for another 40 minutes.

Decision point

Given the SLA constraints, the ambiguous causality between the upgrade and the statistics/autovacuum issue, and the ticking risk of full connection pool saturation across all read replicas, what should the on-call Staff Engineer do next?

Choose your first move

Choose your first move

Question 1 of 1

Pick the intervention you would prioritise. Every option is plausible; the challenge is deciding what should happen first with the evidence available.

Make your initial decision, then use the deep dive to investigate the underlying failure mode.

Open technical deep dive