Incident Overview
A PostgreSQL 14 primary (db-primary-03, 2TB gp3 volume) feeds a downstream analytics consumer via a logical replication slot named analytics_sub. At 02:14 UTC the consumer's deployment pipeline pushed a schema migration that broke its connection pool; the consumer stopped acknowledging WAL but the subscription remained technically connected at the network layer.
By 04:50, disk usage climbed from a steady 41% to 91%, rising roughly 3% every 15 minutes, projecting exhaustion within 40 minutes. pg_replication_slots shows analytics_sub with restart_lsn frozen at 04:12 while pg_current_wal_lsn has advanced by 210GB since. pg_stat_activity reveals a six-hour-old query from an analytics service account: SELECT ... FOR UPDATE against a 40-million-row events table, with wait_event_type NULL (actively running, not blocked).
Autovacuum logs show repeated "oldest xmin is far in the past" warnings on three large tables, and the pg_wal directory has grown from 4GB baseline to 214GB. Two competing theories are circulating on the incident bridge.
The first: the stalled slot itself is the sole cause of WAL retention, and dropping it will immediately release accumulated segments. The second: the six-hour transaction is holding back the xmin horizon independently of the slot, meaning even a slot drop would not fully resolve WAL bloat because vacuum is separately stalled, and the query's row locks may also be blocking downstream writes.
Complicating matters, it is 04:50 UTC during a regional trading window where the primary cannot be restarted or failed over without a change-advisory-board exception, and the analytics team has an SLA requiring eventual data continuity rather than real-time freshness. There is no immediately available storage headroom beyond a single volume-extend operation, which the cloud provider throttles to once per six hours.
Monitoring shows connection counts nominal, replication lag on the physical standby unaffected, and no application-tier errors yet, but disk alerts have escalated to page-and-a-half severity.
Investigation Options
Review the available operational moves and select the best immediate action.
Query pg_stat_replication, pg_replication_slots, and pg_stat_activity together to confirm whether the six-hour transaction is holding the xmin horizon independently of the slot; if confirmed, terminate that backend via pg_terminate_backend and verify WAL recycling resumes before any decision on the slot itself.
Immediately drop the analytics_sub replication slot to release WAL retention and free disk space, then coordinate with the analytics team to perform a full logical resync of their subscriber once their consumer service is restored and reconnected.
Extend the primary's underlying storage volume immediately to add headroom against the projected 40-minute exhaustion window, accepting the six-hour provider throttle on further extensions as a acceptable trade-off while root cause investigation continues in parallel.
Initiate failover to the physical standby replica to reset replication state on a fresh primary, expecting that promoting a new node will clear the stalled logical slot and long-running transaction without further diagnosis.