Incident Overview
KBY Technologies operates a Confluent Cloud-hosted Kafka cluster (12 brokers, KRaft mode) backing the order-processing microservice, which consumes from the orders.events topic (48 partitions) via a consumer group order-processor-grp running 24 pods on Kubernetes. At 14:02 UTC a routine dependency bump shipped kafka-clients from 2.8.1 to 3.6.1 to five canary pods (20% of the fleet) as part of a scheduled minor-version rollout; no consumer-side configuration changes were listed in the change record.
By 14:11 UTC, Grafana shows consumer lag on partitions owned by the canary pods climbing from a steady 500 messages to 1.2 million within nine minutes, while lag on partitions owned by non-canary pods stays flat. Broker metrics show under_replicated_partitions=0, ISR shrink/expand counts at zero, and network throughput unchanged from baseline.
Consumer logs for the canary pods repeat 'Attempt to heartbeat failed since group is rebalancing' every 85-95 seconds, with 'Member id ... is no longer a member of group order-processor-grp' entries each cycle, indicating repeated eviction and re-join rather than one stuck member. CPU and JVM GC pause times on affected pods are within normal bounds (P99 GC pause 40ms).
Downstream SLA requires order events processed within 5 seconds of production; end-to-end latency has degraded past 45 seconds and is climbing, and the customer-facing order confirmation queue depth is rising. Two competing hypotheses exist.
First, kafka-clients 3.6.1 changed default rebalance behaviour: without an explicit static group.instance.id, canary members may be issuing new member IDs on every heartbeat cycle under moderate processing latency, triggering continuous eager rebalances. Second, controller-level instability in the KRaft quorum, unrelated to the client upgrade, could be issuing spurious group coordinator reassignments, with the canary timing being coincidental given the deploy window.
Operational constraints: Confluent Cloud is fully managed, so broker or controller restarts are not available to the on-call engineer; only client-side configuration and deployment actions are within control. The canary cannot remain past the next scheduled deployment freeze at 15:00 UTC.
Rolling back the canary requires redeploying the previous container image, roughly six minutes per pod.
Investigation Options
Review the available operational moves and select the best immediate action.
Immediately redeploy the previous kafka-clients 2.8.1 image to the five canary pods, restoring stable group membership while lag continues to be monitored separately on unaffected pods, then investigate the 3.6.1 configuration delta offline before any re-attempt.
Push a cluster-wide client configuration change increasing session.timeout.ms and max.poll.interval.ms and setting group.instance.id for static membership across all 24 pods, then perform a rolling restart to apply it without reverting the library version.
Restart the KRaft controller quorum nodes on the managed Kafka cluster, suspecting broker-side coordinator instability is causing spurious rebalances unrelated to the canary deployment timing.
Leave the canary deployment running and horizontally scale the order-processor deployment to 40 pods to absorb the growing lag, deferring investigation of the rebalance loop until after the SLA breach subsides.