Skip to main content
KBY Technologies Logo
cd ../lexicon
sys/docs/lexicon/witness-replica.md
Lexicon Entry

Witness Replica

A lightweight cluster member that votes in quorum decisions without storing the full replicated dataset, used to break ties or maintain quorum cheaply.

Practical example

A MongoDB replica set spanning two data centres runs a primary and secondary in each, plus an arbiter in a third availability zone. The arbiter votes in elections and log-append acknowledgements but never holds data and can never become primary, letting the cluster survive a single-DC outage without a full third-site replica.

In classic Raft or Paxos deployments, quorum size is a function of node count: N/2 + 1. Operators often want an odd number of voters to avoid tie votes, but replicating full state to a third or fifth node in every region is expensive, especially across WAN links where storage cost and replication lag are non-trivial. A Witness Replica solves this by participating in the RequestVote / Prepare phases and acknowledging log entries for quorum-counting purposes, while persisting only metadata: term numbers, log indices, and commit watermarks. It never applies the state machine and typically cannot serve reads.

  • Vote-only participation: The witness responds to leader election and log-append RPCs, incrementing quorum count, but stores a truncated or null payload for the log entry itself.
  • Topology use case: Common in stretched clusters spanning two primary data centers plus one witness site, avoiding the cost of a full third DC while still surviving a single-DC failure without manual intervention.
  • MongoDB and vSAN precedent: Both implement this pattern explicitly — MongoDB’s arbiter node and VMware vSAN’s witness appliance are production analogues, though neither can be promoted to primary/leader.

The critical edge case is data durability illusion. Because the witness contributes to quorum arithmetic, a write can be acknowledged as committed once it reaches the leader plus the witness, even if only one data-bearing replica has the payload. If that single data replica then fails permanently before replicating to the survivors, the system has a majority-acknowledged write that is unrecoverable — a silent violation of durability guarantees that the consensus protocol nominally promised. Engineers must therefore distinguish commit quorum (votes) from durability quorum (replicas holding actual bytes), and many production systems enforce a stricter write concern that requires acknowledgment from a minimum number of data-bearing nodes independent of witness votes.

Witnesses also change failure-detector economics: because they carry near-zero storage load, they can be placed in a third availability zone purely for tie-breaking at minimal cost, improving split-brain resistance in two-DC topologies without tripling storage spend. The trade-off surfaces during leader election — a witness can never itself become leader (it lacks state), so election algorithms must explicitly exclude it from the leader-eligible candidate set while still counting its vote, requiring a small but consequential fork in the standard Raft candidate logic.