Incident Overview
A self-managed PostgreSQL 14 primary feeds a logical replication slot, cdc_orders_slot, consumed by a Debezium connector inside a Kafka Connect cluster that drives the order-enrichment ETL pipeline. Yesterday at 14:00 UTC, platform teams deployed a new enrichment microservice that added a synchronous HTTP call inside the Kafka Connect sink task, increasing per-record processing time from 4ms to roughly 380ms under load.
By 18:00 UTC, pg_stat_replication showed replay_lag climbing steadily, and pg_replication_slots reported that restart_lsn for cdc_orders_slot had not advanced in over four hours. WAL retention on the primary's data volume grew from 40% to 92% utilisation between 14:00 and 20:00 UTC.
Autovacuum workers on the orders and order_items tables are now blocked because the frozen xmin horizon held by the stalled slot prevents dead tuple cleanup, and table bloat has increased query latency on the orders table from p95 12ms to p95 340ms. Disk-space-critical and replication-slot-lag alerts fired simultaneously at 20:05 UTC.
Cloud infrastructure has an active storage-quota freeze pending a billing review, so additional disk cannot be provisioned before tomorrow at the earliest. Business SLA requires ETL lag to stay under six hours, and it is currently at four hours fifty minutes and rising.
Two competing hypotheses are in play: first, that the enrichment service's added latency is causing Kafka Connect sink task backpressure, stalling consumer offset commits and therefore slot advancement; second, that a firewall or security-group change made during the same deployment window is silently blocking acknowledgements from Kafka Connect back to Postgres, so the connector believes it is behind but Postgres never receives confirmation. Netflow logs show no dropped packets, but connector task logs have not been checked yet.
Investigation Options
Review the available operational moves and select the best immediate action.
Immediately drop and recreate the cdc_orders_slot replication slot to release WAL retention pressure, accepting that Debezium will need to perform a full initial snapshot of the source tables to rebuild downstream state.
Pull Kafka Connect connector status and Debezium task logs to confirm whether sink backpressure or a network acknowledgement failure is stalling offset commits, then restart the connector task while continuously monitoring restart_lsn and disk utilisation.
Run VACUUM FULL on the orders and order_items tables immediately to reclaim disk space from bloat, without first addressing why the replication slot's xmin horizon remains frozen.
Fail over to the standby replica to gain temporary disk headroom, without first determining why the primary's replication slot is not advancing or whether the standby shares the same slot state.