Hybrid Logical Clock (HLC)

A timestamp generation scheme that combines physical wall-clock time with a logical counter to produce monotonically increasing, causality-tracking timestamps across distributed nodes. It matters because it lets systems approximate global ordering of events without the latency cost of consensus-based clocks, while still bounding clock drift to a known epsilon.

An HLC timestamp is a tuple (pt, l, c) where pt is the node’s physical clock reading (typically NTP-synced), l is the maximum logical timestamp observed so far (either the node’s own pt or a value received from a peer), and c is a counter used to disambiguate events that share the same l. On every local event, a node computes l' = max(pt, l); if l' == l, the counter increments, otherwise it resets to zero. On message receipt, the node merges its own l with the sender’s l using the same max-and-increment rule, which is what propagates causality through the cluster — this is structurally identical to a Lamport clock, except the logical component is anchored to real time instead of drifting arbitrarily far from it.

The critical invariant HLCs provide is that l never diverges from true physical time by more than the maximum clock skew (ε) plus network delay, assuming NTP/PTP bounds are respected. This is what allows systems like CockroachDB and MongoDB to use HLC timestamps directly as MVCC version numbers: a read at HLC T is guaranteed to observe all writes with HLC < T that could have causally preceded it, without requiring a round-trip to a centralized sequencer (as TrueTime-style approaches need, albeit with tighter bounds via atomic clocks/GPS).

Clock skew bound violation: if NTP fails silently and a node’s physical clock drifts beyond the configured max_offset, causality guarantees silently break — writes can be misordered relative to reads on other nodes. Most implementations (e.g., CockroachDB) enforce a hard max_offset check and will refuse to serve requests or panic if skew exceeds it, trading availability for correctness.
Counter overflow / logical clock explosion: under high skew or bursty traffic, the counter c can grow large before pt catches up, since every merge with a peer whose l already exceeds local pt forces counter-only increments. This is bounded but can create a backlog that only physical time eventually resolves.
Not a substitute for consensus: HLC gives you a total order consistent with causality (happens-before), not linearizability across concurrent, causally-unrelated events — two nodes can generate HLC timestamps that are close but whose real-world order is ambiguous, which is fine for MVCC snapshot isolation but insufficient for strict serializability without additional protocol (e.g., commit-wait in Spanner).

Architecturally, HLC is attractive because it’s a local, stateless-per-message computation — no coordinator, no consensus round — yet it degrades gracefully to plain physical time when clocks are well-synchronized, keeping timestamps human-interpretable for debugging and TTL/GC logic (unlike pure Lamport clocks, which are causally correct but physically meaningless). The trade-off engineers must internalize: HLC correctness is entirely contingent on your NTP/PTP infrastructure’s SLA, making clock synchronization observability (drift metrics, leap-second handling) a first-class operational concern rather than an afterthought.