ZAB (ZooKeeper Atomic Broadcast Protocol)

ZAB is not a general-purpose consensus algorithm like Raft or Multi-Paxos in the academic sense; it was purpose-built for ZooKeeper’s specific workload: a single, primary-backup style replicated log where a distinguished leader assigns monotonically increasing zxid (ZooKeeper Transaction ID) values to every write. A zxid is a 64-bit value split into a high 32-bit epoch and a low 32-bit counter. The epoch increments on every leader election, and the counter resets to zero, giving every proposal a globally comparable, monotonically increasing identifier without a physical or hybrid clock.

The protocol operates in two interleaved phases across three conceptual modes: Discovery (leader election and epoch establishment via FLE, the Fast Leader Election algorithm), Synchronization (the new leader brings a quorum of followers up to date with its transaction history before accepting new writes), and Broadcast (steady-state two-phase commit-like propagation: leader sends PROPOSAL, followers persist to their local WAL and reply ACK, leader commits once a quorum acknowledges and broadcasts COMMIT). Unlike Raft, ZAB guarantees that a follower never applies a transaction out of the order the leader proposed it, and it explicitly guarantees primary order: if a leader broadcasts transaction T1 before T2, every server processes T1 before T2, even across leader changes.

• Epoch-based recovery: On leader failure, FLE selects the follower with the highest zxid as leader candidate, ensuring no committed transaction is ever lost or reordered during failover.
• Idempotent replay: Because zxids are strictly ordered, followers can safely discard or replay in-flight proposals from a deposed leader without ambiguity.
• Read scalability tradeoff: Followers serve reads locally (possibly stale) unless a client issues a sync() call, which forces the read to be preceded by a quorum round-trip, trading latency for linearizable reads.
• Single-writer bottleneck: All writes funnel through one leader, capping write throughput to a single node’s disk fsync rate, which is why ZooKeeper is explicitly positioned as a coordination substrate, not a general data store.

The most common operational failure mode is misunderstanding ZAB’s read consistency model: engineers assume ZooKeeper reads are always linearizable, but by default they are only guaranteed FIFO client order and eventual consistency across followers, which can produce stale-read bugs in leader-election or config-watch logic unless sync() is used deliberately. Another subtle edge case is epoch exhaustion during pathological rapid leader churn, and the classic “zxid overflow” bug class where the 32-bit epoch or counter wraps under extreme election storms, historically requiring operators to bound election frequency and monitor zk_num_alive_connections and session expiry metrics closely.

Architecturally, ZAB’s design choices explain why systems like Kafka (pre-KRaft), Hadoop HDFS HA, and HBase historically depended on ZooKeeper purely for leader election and small, low-throughput metadata coordination rather than as a general consensus substrate for application data. Understanding ZAB’s primary-order guarantee and its distinction from majority-quorum consensus like Raft is essential when reasoning about failover latency, split-brain avoidance during network partitions, and why ZooKeeper ensembles are deliberately kept small (typically 3, 5, or 7 nodes) since every write must be durably persisted and acknowledged by a quorum before the client receives success.