Read Repair

Read Repair operates at the coordinator node during a quorum read. When a client requests a key with read consistency R, the coordinator queries N replicas (or at least R of them) and compares the returned values using version metadata — typically a Vector Clock, timestamp, or Lamport-style counter. If replicas disagree, the coordinator resolves the conflict (last-write-wins, causal merge, or CRDT-style merge depending on the data model) and asynchronously pushes the resolved value back to the stale replicas before or after returning the response to the client, depending on whether the system implements blocking (synchronous) or non-blocking (async, fire-and-forget) repair.

There are two operational modes worth distinguishing:

• Read-triggered repair (foreground): Repair only occurs on keys that are actually read. This means cold, rarely-accessed keys can drift indefinitely and never self-heal, which is why systems like Cassandra pair this with a scheduled nodetool repair (Merkle-tree-based anti-entropy) as a backstop for the entire keyspace.
• Digest-based optimization: To avoid transmitting full payloads from every replica on every read, coordinators often request full data from one replica and lightweight digests (hashes) from the others. If digests mismatch, a full-value re-read is triggered — this is the classic Cassandra read_repair_chance / digest-mismatch pattern, and it materially changes tail latency because a digest mismatch converts a fast quorum read into a multi-round-trip operation.

The critical engineering tradeoff is consistency probability vs. tail latency. Blocking read repair (waiting for the repair write to succeed before acknowledging the client) increases R-quorum consistency guarantees but couples read latency to the slowest straggler replica’s write ack — a direct vector for Head-of-Line Blocking-style latency amplification under partial degradation. Non-blocking repair keeps read latency flat but means the client can still observe stale data on a subsequent read if it happens to hit an unrepaired replica before the async write lands, which is a real consideration when reasoning about session consistency or read-your-writes guarantees on top of a leaderless store.

Read repair also interacts poorly with high-cardinality, low-read-frequency workloads and with deletes: if a delete is represented as a Tombstone and read repair propagates a stale non-tombstoned value back over a tombstone on a replica that hasn’t yet seen the delete, you can effectively resurrect deleted data — this is precisely why tombstone GC-grace-period tuning and full anti-entropy repair windows must be coordinated so that grace periods always exceed the maximum expected repair convergence time. Systems exposing tunable read repair (e.g., a probabilistic read_repair_chance parameter) let operators trade CPU/network overhead against staleness risk, but misconfiguring this value relative to write throughput is a common source of silent data resurrection incidents in production Cassandra/Riak clusters.