Skip to main content
daily-triage/kafka-consumer-lag-rises-while-broker-health-stays-green.md
Daily Triage Briefing

Kafka Consumer Lag Rises While Broker Health Stays Green

Impact Summary

A fictional Apache Kafka exercise tests how to diagnose rising consumer lag despite healthy broker signals, while keeping recovery bounded to an isolated environment and escalating before any production change.

SeveritySEV-3
StatusResolved simulation
PlatformDistributed Systems
Incident TypeDistributed Systems
Published5 Sept 2026
Technologies Involved
Apache Kafka

Simulation Label

Fictional simulation. This exercise is a bounded, invented training scenario for practising operational reasoning. No real organisation, incident, telemetry, credentials or production identifiers are involved or implied.

Operational Summary

Root Cause

In the fictional reveal, delayed monitoring data made consumer lag appear to keep rising after current lag had stabilised.

Detection Method

Evidence-led guided investigation

Current Status

Resolved simulation

Affected Services

Apache Kafka

Incident Overview

A team operates a fictional event-processing workflow in an isolated validation environment. Producers continue to submit records, the broker-health dashboard remains green and no broad application outage is reported.

However, the dashboard for one consumer group shows steadily rising lag, while its application dashboard reports a stable processing rate. The simulated user impact is delayed downstream updates rather than total service loss, so the fictional incident is classified as SEV-3.The immediate objective is not to force lag down.

It is to determine whether the conflicting signals represent a consumer bottleneck, stale monitoring, uneven partition work or another unverified condition. Permissions and the installed Apache Kafka version must be confirmed before any later change.

The exercise supplies no verified configuration, logs, partition counts or external research, so those details must not be assumed.

Investigation Options

Review the available operational moves and select the best immediate action.

A

Pause changes and reconcile metric timestamps, consumer-group identity and partition-level lag using read-only evidence.

B

Restart every consumer instance immediately to try to reduce lag.

C

Increase topic partitions immediately because rising lag proves insufficient parallelism.

D

Close the incident because the broker-health dashboard is green.

Tags:Apache Kafka