Consumer Lag Climbs While Kafka Broker Disk Usage Stays Flat

Simulation Label: Fictional Kafka Distributed Systems Exercise

This is a fictional training simulation. No real telemetry, credentials, customer data or live incident is described or implied. All identifiers, group names and hosts are invented for this exercise.

Scenario

A platform team operates an order-processing pipeline built on Apache Kafka. A dashboard alert fires: consumer lag on the orders-consumer-group has grown steadily over the last thirty minutes on the orders topic. On-call engineer Priya opens the broker monitoring panel first, expecting to see disk or CPU pressure, since that has caused lag spikes before.

Broker CPU utilisation is flat at 22%. Disk usage on all three brokers is unchanged from the previous hour. Network throughput between brokers looks nominal. Yet the lag graph keeps climbing, and a colleague mentions seeing repeated "rebalancing" messages scroll past in the consumer application logs.

Evidence

• Consumer lag for orders-consumer-group has grown from roughly 200 messages to over 40,000 messages in thirty minutes.
• Broker-side metrics (CPU, disk, network) show no meaningful change across the same window.
• Application logs show the consumer group entering and leaving the PreparingRebalance and CompletingRebalance states repeatedly, roughly every 45 seconds.
• The consumer application's business logic occasionally makes a slow downstream HTTP call (up to 8 seconds) as part of message processing, introduced in a deployment two days earlier.
• Topic partition count and replication factor have not changed recently.

Hypotheses and Actions

Three competing explanations fit at least some of the evidence:

• Hypothesis A – Broker capacity exhaustion. If true, broker CPU, disk or network metrics should show sustained pressure. The evidence shows flat broker metrics, which weakens this hypothesis.
• Hypothesis B – Consumer rebalance loop caused by slow polling. If the consumer's per-poll processing time (including the newly slow downstream call) exceeds the configured max.poll.interval.ms, the group coordinator will treat the consumer as dead and force a rebalance, repeatedly, even though the process is still alive and working.
• Hypothesis C – Network partition between consumers and brokers. Intermittent partitions could also trigger rebalancing, but this would typically also show connection errors or timeouts in broker logs, which are not reported here.

The recommended immediate actions are read-only checks that discriminate between these hypotheses before any configuration change is made:

• Describe the consumer group to confirm the pattern and frequency of membership changes and current lag distribution across partitions.
• Describe the topic to confirm partition, replica and in-sync-replica health, ruling out a topic-level configuration change.
• Review recent broker logs for connection resets, GC pauses or disk I/O warnings that would support Hypothesis A or C.

Reveal

The consumer group's max.poll.interval.ms was left at its low default value from an earlier, faster version of the message handler. The recent deployment added an occasional slow downstream HTTP call inside the poll loop. When that call runs long, the group coordinator judges the consumer to have stalled and triggers a rebalance, even though the consumer eventually finishes processing. Each rebalance pauses consumption briefly, which compounds the lag. Broker capacity was never the constraint; the bottleneck was a mismatch between consumer-side processing time and the poll interval configuration.

Learning Outcome

This exercise distinguishes a symptom (rising lag, a broker-shaped assumption) from the actual mechanism (a client-side timeout misconfiguration interacting with a slower downstream dependency). Before making any configuration change in a real environment, confirm the hypothesis with read-only evidence: consumer group describe output showing rebalance frequency, and topic describe output ruling out partition or replica issues. Any change to poll or session timeout values should be applied in a non-production environment first, observed for at least one full rebalance-free window, and reverted immediately if lag does not stabilise or if new errors appear in broker logs. The next safe decision is to validate the fix on a single consumer instance, confirm lag recovery through the same describe command used for diagnosis, and only then roll the configuration change out to the remaining consumers.