Incident Overview
A self-managed PostgreSQL 14 primary (db-primary-03) feeds a Debezium logical replication slot (slot_cdc_billing) that streams change events into Kafka for the billing reconciliation pipeline. At 02:14 UTC the Kafka Connect worker running the Debezium task was OOMKilled by the Kubernetes scheduler after a memory-limit change was rolled out the previous evening; no restart occurred because the deployment's restartPolicy was misconfigured to OnFailure with a crash-loop backoff cap that had silently expired.
Disk utilisation on the primary's data volume climbed from 62% at midnight to 89% by 08:40 UTC, triggering a PagerDuty alert on the node_filesystem_avail_bytes threshold. pg_wal now contains 41,000+ segment files versus a typical steady-state of 800-1,200. Querying pg_stat_replication returns no rows for slot_cdc_billing, while pg_replication_slots shows active=f and restart_lsn frozen at a value roughly nine hours behind the current LSN.
CPU, memory, and query latency on the primary remain within normal bounds; downstream synchronous replicas report zero replication lag, since they use a separate physical slot. Separately, pg_stat_activity shows a reporting service holding an idle-in-transaction session (state='idle in transaction', xact_start 7h22m ago) against the same database, which also advances the backend xmin horizon and could independently block autovacuum from reclaiming dead tuples on the large invoices table.
Two competing hypotheses are on the table: first, that the orphaned Debezium slot is retaining WAL because nothing is consuming from restart_lsn, which would be resolved by dropping the slot once the dead connector is confirmed unrecoverable; second, that the idle-in-transaction reporting session is the primary driver of table bloat and WAL retention pressure, independent of the slot's state, and killing that session would be the more effective fix. It is currently 08:55 UTC on a quarter-end processing day, so the billing team has an active change freeze on anything touching the CDC pipeline, and a full primary restart or failover has not been tested against this freeze window.
Disk has roughly three hours of headroom at the current growth rate before write failures begin.
Investigation Options
Review the available operational moves and select the best immediate action.
Query pg_replication_slots for restart_lsn and active status alongside pg_stat_activity for long-running idle-in-transaction sessions to determine definitively whether the slot or the reporting transaction is retaining WAL, then act on whichever is confirmed as the actual blocker.
Run VACUUM FREEZE VERBOSE against the largest bloated tables such as invoices to reclaim space immediately while the investigation into the slot and transaction continues in parallel, buying operational headroom without committing to a root-cause fix yet.
Resize the underlying EBS volume to add capacity immediately so the three-hour runway becomes non-critical, deferring any diagnosis of whether the slot or the transaction is responsible for the WAL growth.
Restart the PostgreSQL primary instance to force closure of the idle transaction and reset replication state, assuming this will clear both the stalled slot and any lock contention in a single action.