One incident. Four plausible moves. Instant technical reasoning.
Incident brief
Your primary PostgreSQL 14 cluster (db-prod-01, 8 vCPU/64GiB, gp3 storage) feeds a downstream analytics pipeline via a logical replication slot (analytics_slot) consumed by a Debezium Kafka Connect connector. At 14:02 UTC, a migration deployed through CI/CD added a NOT NULL column with a sequence-backed default to the orders table (42M rows) as part of a feature release.
The migration job returned success with no pipeline errors. By 14:20 disk utilisation on db-prod-01 climbed from 61% to 78%; by 14:45 it hit 92%, triggering a PagerDuty alert for imminent volume exhaustion. pg_replication_slots shows analytics_slot with active=false, restart_lsn frozen at the value held since 14:04, matching confirmed_flush_lsn. pg_stat_activity shows no query running longer than three minutes; the migration transaction itself committed at 14:04.
The WAL directory has grown from 4GiB to 41GiB in under an hour. Kafka Connect's REST API for the Debezium connector returns a 500 error referencing an unmapped column type.
Autovacuum on orders has not run since 13:58, and pg_stat_progress_vacuum is empty. CPU load sits at an unremarkable 40%.
Constraints: the analytics pipeline feeds a regulatory financial report due at 18:00 UTC with zero tolerance for data loss; the storage volume's autoscaling policy caps at 500GiB and is already at 480GiB; dropping and recreating the slot would force a full table resnapshot estimated at four hours given current throughput. Two theories are circulating: one engineer suspects the Debezium connector crashed on the new column type immediately post-migration, leaving nothing to consume and acknowledge WAL, pinning restart_lsn; another suspects a hidden long-running transaction or lock-wait from the migration is holding back the xmin horizon, blocking both vacuum and slot advancement, despite pg_stat_activity currently showing nothing running.
Disk exhaustion is projected within roughly 90 minutes at the current growth rate, which would halt primary write availability entirely.
Given the connector's 500 error, the frozen restart_lsn since 14:04, the imminent disk-exhaustion window, and the 18:00 reporting deadline, what should the on-call Staff Engineer do next?
Make your initial decision, then use the deep dive to investigate the underlying failure mode.