Incident Overview
Your primary PostgreSQL 14 cluster (db-prod-01, 8 vCPU/64GiB, gp3 storage) feeds a downstream analytics pipeline via a logical replication slot (analytics_slot) consumed by a Debezium Kafka Connect connector. At 14:02 UTC, a migration deployed through CI/CD added a NOT NULL column with a sequence-backed default to the orders table (42M rows) as part of a feature release.
The migration job returned success with no pipeline errors. By 14:20 disk utilisation on db-prod-01 climbed from 61% to 78%; by 14:45 it hit 92%, triggering a PagerDuty alert for imminent volume exhaustion. pg_replication_slots shows analytics_slot with active=false, restart_lsn frozen at the value held since 14:04, matching confirmed_flush_lsn. pg_stat_activity shows no query running longer than three minutes; the migration transaction itself committed at 14:04.
The WAL directory has grown from 4GiB to 41GiB in under an hour. Kafka Connect's REST API for the Debezium connector returns a 500 error referencing an unmapped column type.
Autovacuum on orders has not run since 13:58, and pg_stat_progress_vacuum is empty. CPU load sits at an unremarkable 40%.
Constraints: the analytics pipeline feeds a regulatory financial report due at 18:00 UTC with zero tolerance for data loss; the storage volume's autoscaling policy caps at 500GiB and is already at 480GiB; dropping and recreating the slot would force a full table resnapshot estimated at four hours given current throughput. Two theories are circulating: one engineer suspects the Debezium connector crashed on the new column type immediately post-migration, leaving nothing to consume and acknowledge WAL, pinning restart_lsn; another suspects a hidden long-running transaction or lock-wait from the migration is holding back the xmin horizon, blocking both vacuum and slot advancement, despite pg_stat_activity currently showing nothing running.
Disk exhaustion is projected within roughly 90 minutes at the current growth rate, which would halt primary write availability entirely.
Investigation Options
Review the available operational moves and select the best immediate action.
Drop the analytics_slot logical replication slot immediately to release retained WAL, then instruct the Debezium connector to perform a fresh table snapshot once schema mapping is fixed, accepting a multi-hour resnapshot window before the 18:00 reporting deadline.
Query the Kafka Connect REST API and connector logs to confirm the 500 error's root cause, patch or roll back the connector's column-type mapping for the new field, then restart the connector so it resumes consuming from the existing restart_lsn without a full resnapshot.
Run pg_terminate_backend against any backend holding an old snapshot or lock from the migration, then re-check pg_stat_activity and pg_replication_slots to see whether restart_lsn begins advancing, ruling out the long-running-transaction hypothesis before touching the slot.
Expand the gp3 volume beyond the current autoscaling cap via manual console intervention to add headroom, buying time against the 90-minute disk exhaustion projection while the connector and replication slot issue are investigated in parallel.