Backup

Engineers frequently conflate backup with replication, but they solve orthogonal failure classes. Replication (via Raft, chain replication, or synchronous mirroring) protects against physical node loss but faithfully propagates logical errors: a bad DELETE or a corrupted WAL record is replicated to every replica in milliseconds. A backup exists specifically to be causally disconnected from live write paths, usually via a copy-on-write snapshot, a WAL/binlog archive, or an immutable object store write, so it cannot inherit the fault it’s meant to recover from. The critical design property is the consistency marker: a snapshot LSN, a transaction ID watermark, or a fenced checkpoint that lets you prove the backup represents a valid, atomic cut of state rather than a torn write across multiple files or shards.

Backups decompose into two consistency classes with very different recovery semantics:

• Crash-consistent: equivalent to what you’d get from a hard power-off—block/volume-level snapshot with no coordination with the application. Safe only if the application’s own recovery logic (e.g., an ARIES-style WAL replay) can bring itself to a valid state from that image. Cheap, fast, but dangerous for systems with write-back caches or multi-file state (e.g., a DB whose data files and WAL live on separate volumes snapshotted at slightly different instants).
• Application-consistent: requires quiescing writes or issuing a FLUSH/fsync/checkpoint barrier before the snapshot is taken, so the captured state satisfies the application’s own invariants (e.g., pg_start_backup(), VSS writers, or a distributed barrier across shards using something functionally equivalent to a fencing token to stop new commits during the cut).

At scale, full backups are architecturally untenable—the dominant pattern is incremental-forever: one full base image followed by a chain of block- or WAL-level deltas (change-block tracking, binlog shipping, or continuous archiving of WAL segments to object storage). This shifts the operational risk from backup capture to backup restore: RTO becomes a function of chain length and replay throughput, not just the base image size. This is why serious systems periodically synthesize new full images (“forever incremental” compaction, analogous to LSM tree compaction) to bound restore-time chain depth, and why backup validity must be tested by actual restore-and-replay, not by checksum verification alone—corruption in an intermediate delta silently breaks every subsequent point-in-time recovery target downstream of it.

The two governing SLOs—RPO (Recovery Point Objective, the acceptable data-loss window) and RTO (Recovery Time Objective, acceptable downtime)—directly dictate mechanism choice: continuous WAL archiving buys near-zero RPO at the cost of replay-time complexity, while nightly volume snapshots buy trivial restore mechanics at the cost of a 24-hour RPO. In multi-region or multi-tenant architectures, backup storage itself must violate the blast radius of the primary: cross-account, cross-region, with retention immutability (object lock/WORM) to survive credential compromise, since an attacker with write access to production typically also has delete access to same-account backups—making the backup’s isolation boundary, not its existence, the actual security control.