Incident Overview
KBY Technologies runs a multi-tenant billing platform on PostgreSQL 15, with logical replication feeding a Debezium connector into an analytics Kafka Connect cluster. At 08:02 UTC the data platform team completed a migration to a new Kafka Connect cluster and decommissioned the old connector instances, but did not drop the associated replication slot debezium_analytics_v1 on the primary.
At 09:15 UTC a disk utilisation alert fired at 80% on the pg_wal volume; by 09:52 UTC, when you are paged, utilisation sits at 92% of a 512GB volume, growing roughly 2GB every 10 minutes. The platform's automated runbook forces the instance into read-only mode at 95% utilisation, projected to trigger in under 20 minutes. pg_stat_replication shows no active backend for debezium_analytics_v1, and pg_replication_slots reports its restart_lsn is over 200GB behind the current WAL position, active=false.
Simultaneously, pg_stat_activity shows PID 44213, owned by an etl_batch_user role, idle in transaction for 6 hours 42 minutes, holding an old xmin snapshot. Autovacuum logs repeatedly show "could not remove tuples: xmin horizon" entries on high-churn billing tables starting at the same time the idle transaction began.
Both anomalies overlap almost exactly in start time, and either could plausibly account for the disk pressure: the orphaned slot is retaining WAL segments the primary cannot recycle, while the idle transaction is blocking autovacuum and causing tuple bloat that also consumes disk. Other active subscribers depend on separate slots that appear healthy, and the standby replica streams from the same primary via physical replication, so it will inherit whatever WAL retention issue exists there too.
The analytics team has not yet confirmed in writing that the old connector is fully retired, and a primary restart during business hours is against policy without executive sign-off. You have roughly 15-18 minutes before the automatic read-only failover is expected to trigger, after which billing writes would be blocked platform-wide.
Investigation Options
Review the available operational moves and select the best immediate action.
Confirm the old Debezium connector is fully retired via the data platform team's deployment record, then execute pg_drop_replication_slot on debezium_analytics_v1 to immediately release the 200GB-plus of WAL segments it is retaining and halt disk growth.
Terminate the long-running idle-in-transaction session (PID 44213) held by the ETL batch role to release the stale xmin horizon, allowing autovacuum to resume reclaiming dead tuples on the affected billing tables.
Manually issue pg_switch_wal and schedule an emergency VACUUM FULL against the largest bloated billing tables to reclaim disk space quickly, without first determining whether the slot or the idle transaction is the dominant contributor.
Initiate failover to the standby replica as an immediate mitigation for the disk-pressure alert, treating it as the fastest way to avoid the read-only threshold on the primary.