One incident. Four plausible moves. Instant technical reasoning.
Incident brief
A 12-node PostgreSQL 15 cluster feeds a Kafka Connect sink via logical replication for downstream analytics. At 02:14 UTC, disk utilisation on the primary's pg_wal volume began climbing from a steady 40% baseline.
By 03:40 UTC, it reached 91%, triggering a pager alert for imminent volume exhaustion (500GB provisioned, growing at ~9GB/10min). pg_stat_replication shows the logical slot 'analytics_sink' with sent_lsn stalled 40 minutes behind write_lsn, while physical streaming replicas remain current. pg_replication_slots reports wal_status='reserving' with restart_lsn frozen since 03:02 UTC, the same minute Kafka Connect logs show repeated 'org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata' against the sink connector's target broker. CPU and memory on the primary are nominal; iostat shows write latency on the WAL volume climbing to 45ms p99 from a 3ms baseline, consistent with volume nearing capacity rather than causing it.
No schema migrations, DDL, or connector redeploys are recorded in the last 24 hours, but a network ACL change was merged to the Kafka broker security group at 02:50 UTC as part of a routine firewall consolidation. Two competing hypotheses are in play: (1) the Kafka Connect consumer is unable to reach the broker due to the ACL change, causing the logical slot to accumulate unconsumed WAL; or (2) a long-running analytics query or idle transaction on a replica is holding back the slot's confirmed flush LSN independently of Kafka connectivity.
Runbooks prohibit dropping replication slots without confirming no data loss will occur downstream, and the team has a strict RPO of zero for the analytics pipeline. Standby promotion is available but would not resolve slot bloat since replicas share the same physical storage class and provisioning lead time for emergency volume expansion is roughly 20 minutes via the cloud provider's console.
WAL exhaustion will halt write transactions cluster-wide within an estimated 25-30 minutes at current growth rate.
What should the on-call Staff Engineer do next to prevent a full outage while preserving replication guarantees?
Make your initial decision, then use the deep dive to investigate the underlying failure mode.