Skip to main content
daily-triage/a-stale-replication-slot-masks-rising-postgresql-replica-lag.md
Daily Triage Briefing

A Stale Replication Slot Masks Rising PostgreSQL Replica Lag

Impact Summary

A fictional Databases & Storage triage exercise in which conflicting PostgreSQL signalsu2014autovacuum activity, WAL growth and replica lagu2014point toward the wrong root cause until replication slot evidence is checked directly.

SeveritySEV-2
StatusResolved simulation
PlatformDatabases & Storage
Incident TypeDatabases & Storage
Published17 Aug 2026
Technologies Involved
PostgreSQL

Simulation Label

Fictional simulation. This exercise is a bounded, invented training scenario for practising operational reasoning. No real organisation, incident, telemetry, credentials or production identifiers are involved or implied.

Operational Summary

Root Cause

An inactive logical replication slot (reporting_consumer) retained WAL segments after its consumer was decommissioned, causing pg_wal to grow and replica replay lag to climb; the concurrent autovacuum run on orders_history was a coincidental correlation, not the cause.

Detection Method

Evidence-led guided investigation

Current Status

Resolved simulation

Affected Services

PostgreSQL

Incident Overview

Nordwell Retail runs a three-node PostgreSQL 15 cluster (one primary, two streaming replicas) supporting its order-management service. At 02:14 local time, the on-call platform engineer, Priya, receives two alerts within four minutes of each other: replica lag has crossed 30 minutes on both replicas, and disk utilisation on the primary's data volume has crossed 78% and is still climbing.

Application dashboards show order processing latency is normal and error rates are flat, so the incident is not yet customer-visible, but the disk trend threatens to become one within a few hours if it continues unchecked.

Investigation Options

Review the available operational moves and select the best immediate action.

A

Query pg_replication_slots and pg_stat_replication to confirm which slot or replica is retaining WAL before changing anything.

B

Immediately drop the inactive logical replication slot identified on the monitoring dashboard.

C

Terminate the long-running autovacuum worker process on orders_history to reduce disk growth.

D

Temporarily raise max_wal_size and archive_timeout as an immediate mitigation while root-cause investigation continues.

Tags:PostgreSQL
A Stale Replication Slot Masks Rising PostgreSQL Replica Lag | Daily Triage | KBY Technologies