Etcd Defragmentation: Taming Raft Log Bloat

A control plane that reports healthy Raft consensus while quietly approaching a NOSPACE alarm is one of the more insidious failure modes in Kubernetes operations. The symptom is deceptive: etcdctl endpoint status shows a stable leader, latency graphs look fine, and then, without warning, every write request across the cluster starts returning etcdserver: mvcc: database space exceeded. The root cause is almost always the same — nobody scheduled etcd defragmentation, and the on-disk boltdb file has ballooned far beyond the logical size of the keyspace it stores. This article walks through why that happens, how the defragmentation mechanism actually reclaims space, and how to run it safely against a live quorum without triggering an election storm.

#The Problem: Compaction Frees Revisions, Not Disk Pages

etcd’s storage engine is built on MVCC (multi-version concurrency control) layered on top of boltdb, a single-file B+tree key-value store. Every write to the cluster — a ConfigMap patch, a lease

etcdctl compact removes old MVCC

#Architectural Breakdown: What Defragmentation Actually Does

Internally, running defragmentation against an etcd member triggers a full rewrite of the boltdb backend file. The process opens a new temporary file, copies every live key-value pair (and their B+tree structure) across in page order, fsyncs the result, and then atomically swaps it in for the original file. This is conceptually similar to a VACUUM FULL in PostgreSQL — it is not an incremental operation, and it is not free.

Two properties of this design matter enormously for production operators:

• Defragmentation is a blocking operation on the target member. While the rewrite is in progress, that member cannot serve reads or process Raft messages efficiently — it effectively stalls. If that member happens to be the current Raft leader, client-perceived latency across the entire cluster spikes for the duration.
• Defragmentation is per-member, not cluster-wide. Each etcd node maintains its own boltdb file. Running etcdctl defrag against one endpoint does nothing for the others. This is precisely why a naive one-shot script against the whole endpoint list is dangerous — if you don’t serialise it, you can defragment two nodes concurrently and lose quorum mid-operation.

Understanding this distinction — compaction as a logical revision garbage collector, defragmentation as a physical storage reclaimer — is the same class of reasoning applied elsewhere in architectural patterns where logical deletion and physical reclamation are deliberately decoupled for consistency guarantees.

#

Why NOSPACE Alarms Cascade

etcd enforces a hard ceiling via --quota-backend-bytes (default 2GB, commonly raised to 8GB in production Kubernetes clusters). Once the boltdb file crosses that threshold, etcd raises a cluster-wide NOSPACE alarm and rejects all writes — including the compaction request itself in some edge cases, because compaction is technically a write. Operators who haven’t automated etcd defragmentation frequently discover this the hard way: the cluster is technically alive, Raft consensus is intact, but the API server cannot persist anything, and every kubectl apply hangs until the alarm is manually disarmed.

#Implementation Logic: A Quorum-Safe Rolling Procedure

The correct sequence for a three-or-five-node cluster is strictly serial, never parallel, and should never target the current leader first if avoidable.

1. Check current DB size and fragmentation ratio on every member.
2. Run a logical compaction to the current revision, retaining a safety window.
3. Defragment followers first, one at a time, waiting for health checks between each.
4. Transfer leadership away from the current leader if it is last in the queue.
5. Defragment the former leader last, once it has stepped down to follower.
6. Verify DB size reduction and clear any NOSPACE alarm if raised.

1# 1. Inspect current DB size per endpoint
2ETCDCTL_API=3 etcdctl --endpoints=https://10.0.1.10:2379,https://10.0.1.11:2379,https://10.0.1.12:2379 n  --cacert=/etc/etcd/ca.crt --cert=/etc/etcd/client.crt --key=/etc/etcd/client.key n  endpoint status --write-out=table
3
4# 2. Compact to the latest committed revision, retaining recent history
5REV=$(etcdctl endpoint status --write-out="json" | python3 -c 
6  "import sys,json; print(json.load(sys.stdin)[0]['Status']['header']['revision'])")
7etcdctl compact $REV
8
9# 3. Defragment a single follower endpoint explicitly (never the whole cluster string)
10etcdctl --endpoints=https://10.0.1.11:2379 defrag
11
12# 4. Confirm the alarm state is clear before moving to the next node
13etcdctl alarm list

For clusters where leadership transfer must be explicit rather than incidental, use the dedicated Raft API before touching the leader’s boltdb file:

1# Move leadership to a healthy follower before defragmenting the current leader
2etcdctl move-leader <target-member-id>
3etcdctl --endpoints=https://10.0.1.10:2379 defrag

To avoid relying on manual runs entirely, most fleets wrap this logic in a Kubernetes CronJob or systemd

1apiVersion: batch/v1
2kind: CronJob
3metadata:
4  name: etcd-defrag-check
5  namespace: kube-system
6spec:
7  schedule: "0 3 * * 0"   # weekly, low-traffic window
8  jobTemplate:
9    spec:
10      template:
11        spec:
12          containers:
13            - name: defrag-runner
14              image: bitnami/etcd:3.5
15              command:
16                - /bin/sh
17                - -c
18                - |
19                  for ep in $ETCD_ENDPOINTS; do
20                    etcdctl --endpoints=$ep defrag --command-timeout=30s
21                    sleep 15
22                  done
23          restartPolicy: OnFailure

#Rolling Defragmentation Sequence

The ordering constraint is the single most common mistake in home-grown automation. The sequence below shows the safe path through a three-node cluster where node A is the current leader.

#Failure Modes and Edge Cases

Several failure patterns recur across clusters that skip proper etcd defragmentation planning:

• Leader election storms. Defragmenting an active leader without transferring leadership first causes heartbeat timeouts to followers, triggering an unnecessary election and a brief write-availability gap.
• Concurrent defrag across members. Scripts that fan out defrag commands in parallel across all endpoints can drop below quorum if two members become simultaneously unresponsive during their respective rewrites.
• Defrag during snapshotting. Running defragmentation while a periodic snapshot backup is in flight competes for the same disk I/O and file locks, extending both operations well past their normal duration.
• Silent NOSPACE persistence. Disarming the alarm with etcdctl alarm disarm without first compacting and defragmenting simply re-triggers the same alarm on the next write burst.
• Client timeout cascades. API server watches configured with aggressive timeouts will report transient etcd unavailability during a defrag window, which downstream controllers may misinterpret as a cluster-wide outage rather than routine maintenance.

#

Diagnostic Comparison

#Scaling and Security Trade-offs

Beyond the mechanics of individual defrag runs, teams operating etcd at fleet scale need to weigh several structural trade-offs:

• Frequency vs disruption: Running etcd defragmentation too rarely allows fragmentation to accumulate toward the NOSPACE ceiling; running it too frequently introduces recurring latency windows on production clusters that may coincide with peak scheduling activity.
• Quota sizing vs blast radius: Raising --quota-backend-bytes beyond 8GB reduces alarm frequency but increases the duration of every future defrag rewrite, since the entire file must be copied regardless of how much is genuinely live data.
• Automation vs blind execution: Fully automated CronJob-based defragmentation removes human error from the sequencing logic but requires health-check gating between each member to avoid quorum loss during unattended runs.
• TLS client exposure: Maintenance tooling with defrag and alarm-disarm privileges needs mTLS client certificates scoped separately from application-facing etcd access, since a compromised defrag credential can be used to force repeated cluster-wide latency spikes as a denial-of-service vector.
• Snapshot coordination: Backup and defragmentation windows must be mutually exclusive in the scheduler; overlapping the two multiplies disk contention and can extend a routine maintenance task into an incident.

Refer to the official etcd maintenance documentation for the authoritative flag reference and version-specific defaults before rolling any of the above into a production runbook.

None of this is exotic engineering — it is routine storage housekeeping that happens to sit underneath every Kubernetes API call in the cluster. Treating etcd defragmentation as a scheduled, health-gated, leadership-aware procedure rather than a reactive fix for a NOSPACE alarm is what separates a control plane that degrades gracefully under write pressure from one that fails hard and cluster-wide the moment the boltdb file quietly outgrows its quota.