Kafka Rebalance Storm Triggers Duplicate Order Writes in Postgres

KBY Technologies operates an order-fulfilment pipeline consuming from an Apache Kafka cluster (12 brokers, KRaft mode) into PostgreSQL via the order-events consumer service, group order-fulfilment-v3, running 40 consumer instances. session.timeout.ms is 45000, heartbeat.interval.ms 3000, max.poll.interval.ms 300000. At 02:00 UTC a canary deploy pushed 20% of the fleet (8 instances) from kafka-clients 2.8.1 to 3.6.0, switching partition.assignment.strategy from RangeAssignor to CooperativeStickyAssignor; the remaining 32 instances still run 2.8.1 with the eager RangeAssignor.

This was the only change window scheduled before a change freeze for the mid-July flash sale begins at 06:00 UTC. At 02:15 UTC, consumer_group_rebalance_rate spikes: 47 rebalances logged for group order-fulfilment-v3 in 20 minutes, versus a baseline of one every 4-6 hours.

Broker logs show repeated 'Failed to commit offsets... RebalanceInProgressException' and 'Member X sent a leave group request'. records-lag-max on topic orders.events climbs from 500 to 1.2 million messages by 02:40 UTC.

Grafana shows commit_latency_p99 rising from 80ms to 9.4s. Meanwhile, the reconciliation job flags 218 orders double-charged, a $12,340 discrepancy, traced to duplicate rows in order_events where the insert path bypasses the ON CONFLICT DO NOTHING upsert used elsewhere.

No alerts have fired on ZooKeeper/KRaft controller metrics, broker JVM GC pause time, or top-of-rack network interfaces, though the network team reports a scheduled BGP maintenance on an adjacent rack completed at 01:55 UTC, five minutes before the canary. Two hypotheses are circulating in the incident channel: one engineer suspects the mixed CooperativeStickyAssignor/RangeAssignor fleet is causing a rebalance protocol negotiation loop; another suspects the BGP maintenance caused transient packet loss that pushed session timeouts and triggered a broker controller election, with the canary timing being coincidental.

Rolling back the canary requires draining in-flight offsets on the 8 upgraded instances, and the change-freeze policy technically prohibits further deploys after 06:00 UTC except emergency fixes. The order-processing SLA requires end-to-end latency under 30 seconds, and current backlog (1.2 million messages) is already breaching it.

Billing reconciliation is manual and cannot be paused.