Kafka Consumer Group Falls Behind on One Partition

Simulation Label

Fictional simulation only. This exercise describes an invented company, an invented Kafka deployment and invented command output. No real production system, incident, credential or customer data is referenced. Do not treat any output shown here as live telemetry.

Scenario

Northwind Mercantile (a fictional retailer used only for this drill) runs its order pipeline on an Apache Kafka cluster. Order events are published to a topic named orders with 12 partitions and consumed by a single consumer group, order-processing-group, that writes each order into a fulfilment database.

At 09:40 on a normal trading day, the on-call platform engineer notices that the fulfilment dashboard is showing order confirmations arriving several minutes late for some customers, while most orders still confirm within seconds. No alerts have fired for broker CPU, disk or network. The engineer opens a read-only evidence session against the cluster to work out what is actually happening before deciding whether any change is needed.

Evidence

All commands below are read-only evidence collection against the fictional cluster. No partition reassignment, broker restart or configuration change has been made yet.

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group order-processing-group

Simulated output: lag is near zero on 11 of 12 partitions. Partition 7 shows a LAG value in the tens of thousands and is growing.

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders --under-replicated-partitions

Simulated output: empty result. No partition of the orders topic is currently under-replicated.

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders

Simulated output: 12 partitions, replication factor 3, leaders spread evenly across the three fictional brokers. Partition 7's leader is broker 2, which also leads several other partitions that are not lagging.

Hypotheses and Actions

Before making any change, rank the candidate explanations against the evidence above.

• Hypothesis A – Broker or ISR failure. Ruled out: the under-replicated-partitions check returned no results, and broker 2 leads other partitions that are healthy.
• Hypothesis B – Consumer application slowdown. Partially ruled out: a genuine processing slowdown would normally show elevated lag across most or all partitions consumed by the same application instances, not one partition in isolation.
• Hypothesis C – Hot partition caused by skewed message-key distribution. Consistent with the evidence: a single partition carrying disproportionate volume, with no ISR or broker symptoms, matches a producer key (for example, a high-volume account ID) landing repeatedly on partition 7.

Safe next action: request the producer team's message-key distribution for a short recent window and compare the top-N key volumes against partition 7's throughput, without touching partition assignment or restarting any broker or consumer.

Reveal

In this fictional scenario, the key-distribution sample confirms Hypothesis C. A small number of high-volume corporate accounts share a key prefix that Kafka's default partitioner hashes to partition 7, concentrating a disproportionate share of order events on a single partition. Because Kafka preserves ordering only within a partition, redistributing that key evenly across more partitions (or introducing a compound key) removes the concentration without any need to touch broker configuration, replication factor or the consumer group's assignment logic.

Learning Outcome

This drill rehearses separating a hot-partition symptom from a broker-health symptom using only read-only evidence, before any state-changing action is proposed.

• Verified operational checks: consumer-group lag is per-partition, not cluster-wide; under-replicated-partitions is empty; leader distribution is even.
• Rollback boundary: no change has been applied in this drill. If a key-strategy change were made in a real environment, it would be reversible by reverting the producer's partitioning logic, since no data is deleted or reassigned.
• Next safe decision: escalate to the producer-owning team with the partition-7 evidence and propose a key-distribution change for validation in a non-production environment before any production rollout.