Consumer Group Rebalancing Storm Stalls a Kafka Order Pipeline

Simulation Label

Fictional simulation. This exercise describes an invented distributed-systems incident for training purposes only. No real production system, customer, telemetry, credential or identifier is involved or implied.

Scenario

A fictional retail platform, "Northfield Commerce", runs its order-processing pipeline on Apache Kafka. A single topic, orders.events, carries order-created and order-updated messages to a consumer group named order-processor-group, which writes fulfilment records to a downstream warehouse system.

At 09:42 on a Tuesday, the on-call engineer, fictionally named Dana, is paged for "processing stalled — orders backing up". Dashboards show fulfilment writes intermittently pausing for thirty to ninety seconds, several times per hour, without a full outage.

Evidence

• Consumer group lag for order-processor-group spikes from near-zero to several thousand messages, then drains within two minutes, repeating roughly every fifteen minutes.
• Broker logs show the group entering PreparingRebalance and CompletingRebalance states at the same times the lag spikes occur.
• The broker team flags a handful of transiently under-replicated partitions on an unrelated topic and initially suspects broker-side instability.
• Consumer application logs show no exceptions, downstream timeouts, or process restarts.
• JVM diagnostics on the consumer hosts show garbage-collection pauses of four to seven seconds at intervals similar to the rebalances.
• The consumer group's session.timeout.ms is configured at 6000 (six seconds), with default heartbeat settings.

The evidence is initially conflicting: broker-side replication anomalies suggest a broker problem, while the timing correlation with consumer garbage-collection pauses points elsewhere.

Hypotheses and Actions

Three hypotheses are worth testing before any configuration change is made:

• Hypothesis A — broker instability. The under-replicated partitions on an unrelated topic are disrupting the group coordinator. Test by checking whether the coordinator broker for order-processor-group is one of the affected brokers.
• Hypothesis B — network disruption between consumers and coordinator. Intermittent network loss could delay heartbeats. Test by checking consumer host network error counters for the same time windows.
• Hypothesis C — consumer-side garbage-collection pauses exceeding the heartbeat window. If a pause exceeds session.timeout.ms, the coordinator evicts the consumer and triggers a rebalance even though the process is otherwise healthy. Test by correlating pause timestamps with rebalance start times.

The following read-only commands gather evidence without changing any configuration:

kafka-consumer-groups.sh --describe --group order-processor-group --bootstrap-server broker:9092

kafka-topics.sh --describe --topic orders.events --bootstrap-server broker:9092

jstat -gcutil <consumer_pid> 1000 5

Reveal

Correlating the timestamps shows every rebalance event lines up with a garbage-collection pause on the consumer JVM, not with the under-replicated partitions, which resolve independently on a different topic. The consumer's session.timeout.ms of six seconds is shorter than the observed pauses of four to seven seconds. When a pause exceeds the session timeout, the group coordinator marks the consumer as failed and starts a rebalance, even though the process is still running and resumes normally once the pause ends.

The broker-side under-replicated partitions are a coincidental, unrelated symptom on a different topic and are not the cause of this incident.

Learning Outcome

• Kafka's consumer group protocol treats a missed heartbeat within session.timeout.ms as consumer failure, regardless of the underlying cause.
• JVM garbage-collection pauses on consumer hosts are a common, under-examined cause of unexplained rebalances and should be checked before assuming a broker-side or network fault.
• Correlating evidence across layers — broker logs, consumer application logs and JVM diagnostics — before acting prevents wasted effort chasing an unrelated broker symptom.
• Any follow-up configuration change, such as raising session.timeout.ms or tuning JVM heap and garbage-collector settings, should be validated in a non-production environment first, with lag and rebalance frequency monitored afterwards, and with a documented rollback to the previous configuration if the change does not help or slows failure detection.