Incident Overview
A 12-node PostgreSQL 15 cluster feeds a Kafka Connect sink via logical replication for downstream analytics. At 02:14 UTC, disk utilisation on the primary's pg_wal volume began climbing from a steady 40% baseline.
By 03:40 UTC, it reached 91%, triggering a pager alert for imminent volume exhaustion (500GB provisioned, growing at ~9GB/10min). pg_stat_replication shows the logical slot 'analytics_sink' with sent_lsn stalled 40 minutes behind write_lsn, while physical streaming replicas remain current. pg_replication_slots reports wal_status='reserving' with restart_lsn frozen since 03:02 UTC, the same minute Kafka Connect logs show repeated 'org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata' against the sink connector's target broker. CPU and memory on the primary are nominal; iostat shows write latency on the WAL volume climbing to 45ms p99 from a 3ms baseline, consistent with volume nearing capacity rather than causing it.
No schema migrations, DDL, or connector redeploys are recorded in the last 24 hours, but a network ACL change was merged to the Kafka broker security group at 02:50 UTC as part of a routine firewall consolidation. Two competing hypotheses are in play: (1) the Kafka Connect consumer is unable to reach the broker due to the ACL change, causing the logical slot to accumulate unconsumed WAL; or (2) a long-running analytics query or idle transaction on a replica is holding back the slot's confirmed flush LSN independently of Kafka connectivity.
Runbooks prohibit dropping replication slots without confirming no data loss will occur downstream, and the team has a strict RPO of zero for the analytics pipeline. Standby promotion is available but would not resolve slot bloat since replicas share the same physical storage class and provisioning lead time for emergency volume expansion is roughly 20 minutes via the cloud provider's console.
WAL exhaustion will halt write transactions cluster-wide within an estimated 25-30 minutes at current growth rate.
Investigation Options
Review the available operational moves and select the best immediate action.
Immediately drop the analytics_sink replication slot to halt WAL retention growth, accepting downstream data loss as an acceptable trade-off given the imminent risk of a full write outage across the cluster.
Provision additional WAL volume capacity via the cloud provider's emergency resize path to buy investigation time, while querying pg_stat_activity and pg_replication_slots to confirm whether the ACL change or a stalled consumer transaction is the actual blocker.
Revert the 02:50 UTC Kafka broker security group ACL change first, on the assumption that restoring connectivity will let Kafka Connect resume consuming and naturally advance the slot's restart_lsn without further diagnosis.
Restart the Kafka Connect sink connector process to force it to re-establish a broker connection and resume consumption, without first confirming whether network connectivity or a stalled transaction is the root blocker.