Incident Overview
Your fleet runs PostgreSQL 14.9 on self-managed EC2 instances (io2 EBS, 2TB gp3 volume for pg_wal) backing a multi-tenant billing platform. A downstream analytics team consumes change data via a Debezium logical replication slot named analytics_cdc feeding a Kafka Connect cluster.
At 13:58 UTC the platform team deployed a schema migration adding three columns and a partial index to the invoices table via a rolling ALTER TABLE executed through Liquibase. At 14:06 UTC the Kafka Connect worker pool began crash-looping with OutOfMemoryError while failing to deserialise the new column type; connector status showed RUNNING in the Connect REST API but consumer offsets stopped advancing.
By 14:45 UTC node_filesystem_avail_bytes for the pg_wal mount dropped from 68% free to 31% free, and by 15:10 UTC it hit 12% free, triggering a PagerDuty P1. pg_replication_slots shows analytics_cdc with active=false and restart_lsn frozen at 14:04 UTC, roughly 46GB behind the current LSN. Autovacuum logs show no unusual bloat on invoices; pg_stat_activity shows no long-running or idle-in-transaction sessions. archive_command logs show successful archiving to S3 every 60 seconds with zero failures.
Primary CPU and IOPS are nominal (40% and 3,200 IOPS). Standby replication lag is under 200ms.
Two competing theories are in play. First, the frozen inactive analytics_cdc slot is preventing WAL segment recycling regardless of successful archiving, and disk pressure resolves once the connector resumes consuming or the slot is removed.
Second, the migration's new partial index triggered unexpected WAL amplification during its initial build, independent of the slot, and the slot's timing is coincidental. Constraints: this is a PCI-scope billing database; removing the slot requires data-platform lead sign-off because downstream analytics would need a full resync from a fresh snapshot, estimated at six hours.
Disk cannot be resized without a maintenance window per storage-team policy, though an emergency exception process exists. The standby cannot be promoted mid-incident without breaching a separate synchronous replication SLA.
Investigation Options
Review the available operational moves and select the best immediate action.
Investigate and restart the Debezium/Kafka Connect worker to resume consumption from analytics_cdc, confirming recovery via pg_replication_slots.active flipping true and restart_lsn advancing; escalate to slot removal only if the connector cannot be recovered within a defined time-box.
Trigger the emergency disk-resize exception process to expand the pg_wal volume immediately, buying operational headroom while the connector and migration are diagnosed in parallel, without committing to any WAL-retention or slot decision yet.
Drop the analytics_cdc replication slot immediately to force WAL segment recycling and resolve the disk-pressure alert, accepting that downstream analytics will require a full resync from a fresh snapshot before continuing.
Restart the PostgreSQL primary to clear suspected internal locking or memory pressure believed to be blocking WAL cleanup, on the assumption that the migration's index build is silently starving background processes.