Incident Overview
KBY Technologies operates an order-fulfilment pipeline consuming from an Apache Kafka cluster (12 brokers, KRaft mode) into PostgreSQL via the order-events consumer service, group order-fulfilment-v3, running 40 consumer instances. session.timeout.ms is 45000, heartbeat.interval.ms 3000, max.poll.interval.ms 300000. At 02:00 UTC a canary deploy pushed 20% of the fleet (8 instances) from kafka-clients 2.8.1 to 3.6.0, switching partition.assignment.strategy from RangeAssignor to CooperativeStickyAssignor; the remaining 32 instances still run 2.8.1 with the eager RangeAssignor.
This was the only change window scheduled before a change freeze for the mid-July flash sale begins at 06:00 UTC. At 02:15 UTC, consumer_group_rebalance_rate spikes: 47 rebalances logged for group order-fulfilment-v3 in 20 minutes, versus a baseline of one every 4-6 hours.
Broker logs show repeated 'Failed to commit offsets... RebalanceInProgressException' and 'Member X sent a leave group request'. records-lag-max on topic orders.events climbs from 500 to 1.2 million messages by 02:40 UTC.
Grafana shows commit_latency_p99 rising from 80ms to 9.4s. Meanwhile, the reconciliation job flags 218 orders double-charged, a $12,340 discrepancy, traced to duplicate rows in order_events where the insert path bypasses the ON CONFLICT DO NOTHING upsert used elsewhere.
No alerts have fired on ZooKeeper/KRaft controller metrics, broker JVM GC pause time, or top-of-rack network interfaces, though the network team reports a scheduled BGP maintenance on an adjacent rack completed at 01:55 UTC, five minutes before the canary. Two hypotheses are circulating in the incident channel: one engineer suspects the mixed CooperativeStickyAssignor/RangeAssignor fleet is causing a rebalance protocol negotiation loop; another suspects the BGP maintenance caused transient packet loss that pushed session timeouts and triggered a broker controller election, with the canary timing being coincidental.
Rolling back the canary requires draining in-flight offsets on the 8 upgraded instances, and the change-freeze policy technically prohibits further deploys after 06:00 UTC except emergency fixes. The order-processing SLA requires end-to-end latency under 30 seconds, and current backlog (1.2 million messages) is already breaching it.
Billing reconciliation is manual and cannot be paused.
Investigation Options
Review the available operational moves and select the best immediate action.
Increase session.timeout.ms and max.poll.interval.ms across the fleet and monitor rebalance rate for 15 minutes before taking further action.
Immediately halt the canary rollout and revert the entire consumer fleet to the previous kafka-clients version and assignor configuration, forcing protocol homogeneity, then re-enable the cooperative-sticky migration later using KIP-429's documented two-phase compatible rollout.
Force a controller failover on the Kafka broker cluster and restart the affected brokers, suspecting ZooKeeper/KRaft election instability caused by the BGP maintenance as the root cause of the rebalance storm.
Deploy an emergency hotfix to the order-events consumer adding ON CONFLICT DO NOTHING idempotency handling to the vulnerable insert path, without first stopping the ongoing rebalance storm.