Incident Overview
At 02:14 UTC, the platform team completed a rolling patch of the Kafka broker fleet (v3.6.1 to v3.6.2) to remediate a CVE in the inter-broker protocol handler. Within 40 minutes, the payments-events consumer group (14 consumers, 96 partitions) began exhibiting continuous rebalances.
Grafana shows kafka_consumer_group_rebalance_rate climbing from near-zero to 3-4 rebalances per minute, and consumer lag on the largest topic, order-state-changes, has grown from under 500 messages to 1.2 million and is still increasing. Broker logs show repeated "Member ... has failed, removing it from the group" entries with session.timeout.ms set to 10000ms, alternating with "Attempt to heartbeat failed since group is rebalancing" from the same consumer instances seconds later.
CPU on consumer pods is stable at 35%, but GC pause metrics show occasional 400ms stop-the-world pauses on three of fourteen JVMs. Network telemetry between consumers and the newly patched brokers shows p99 round-trip latency increased from 8ms to 45ms since the patch, though average latency remains under 12ms.
The broker patch notes mention a change to the group coordinator's heartbeat processing thread pool sizing. Two hypotheses are circulating: first, that the coordinator change introduced processing delays causing consumers to be falsely evicted despite healthy heartbeats being sent; second, that GC pauses on a subset of consumer JVMs are causing missed heartbeats, with the coordinator behaving correctly.
Downstream, the payments reconciliation service depends on order-state-changes and its SLA requires lag under 60 seconds; it has now paged twice. Rolling back the broker patch requires a further 35-minute rolling restart window and cannot be done partially without risking mixed-protocol incompatibility.
Session timeout tuning can be applied via consumer config redeploy in under 5 minutes but requires a rolling restart of consumer pods.
Investigation Options
Review the available operational moves and select the best immediate action.
Immediately increase session.timeout.ms and max.poll.interval.ms on the consumer group configuration and redeploy consumer pods on a rolling basis, using the brief availability gap to correlate whether rebalance frequency drops in step with the change.
Initiate a full rolling rollback of all Kafka brokers to v3.6.1 immediately, treating the patch as the confirmed root cause without further correlation, accepting the 35-minute restart window and temporary unavailability during the transition.
Capture heap and GC logs from the three affected consumer JVMs alongside broker coordinator thread pool metrics for a five-minute window, correlating heartbeat send timestamps against coordinator processing latency before changing any configuration.
Restart only the three consumer pods showing 400ms GC pauses, assuming JVM tuning alone will resolve the rebalance storm, while leaving broker configuration and consumer timeout settings unchanged.