Taming RocksDB Compaction Storms at Scale

A stateful Kafka Streams or Flink application that has run cleanly for months will, without warning, start missing its checkpoint SLA, backpressure will propagate upstream, and consumer lag will climb into the millions. The culprit is rarely the JVM heap or the network. It is the embedded RocksDB instance backing every keyed state store, entering a compaction storm where background I/O threads saturate the underlying disk and starve the foreground write path. This is one of the most under-diagnosed bottlenecks in distributed stream processing because the symptoms present at the orchestration layer (task manager timeouts, rebalances) while the root cause sits three layers down in the LSM-tree engine.

This article breaks down why compaction storms occur, how to model the write amplification that causes them, and the exact configuration levers required to keep RocksDB compaction storms bounded under sustained high-cardinality write load.

#Problem Statement: Why LSM-Trees Stall Under Load

RocksDB, like every log-structured merge-tree (LSM) engine, defers the cost of sorted storage. Writes land in an in-memory memtable, get flushed to an immutable SSTable file on disk, and are periodically merged (compacted) into successive levels (L0, L1, L2…) to maintain read efficiency and reclaim space from tombstoned or overwritten keys. Under a steady, moderate write rate, this deferred cost is amortised smoothly across background threads.

The problem emerges when the flush rate from memtables into L0 exceeds the rate at which the compaction scheduler can merge L0 files into L1. RocksDB’s default trigger, level0_file_num_compaction_trigger, is typically set to 4. Once L0 accumulates beyond this threshold, RocksDB begins to aggressively throttle (or entirely stall) foreground writes via its internal write-stop mechanism to prevent unbounded read amplification. This is by design, not a bug, but under bursty ingestion patterns — replays, reprocessing jobs, or a Kafka partition rebalance dumping a backlog — it manifests as a full write stall, which propagates as backpressure through the entire stream topology.

#Architectural Breakdown

#Leveled vs Universal Compaction

RocksDB supports two primary compaction styles, and the choice materially changes the failure profile of compaction storms:

• Leveled Compaction: Each level is roughly 10x larger than the previous. Write amplification is high (typically 10-30x) but space amplification and read amplification are low. This is the default and is what Kafka Streams and Flink’s RocksDB state backend use out of the box.
• Universal (Tiered) Compaction: Optimised for write-heavy workloads. Files of similar size are merged together rather than across strict levels, reducing write amplification to near 2-4x at the cost of higher space amplification and periodic large compactions.

For high-throughput stateful stream processing where state stores see continuous upserts (aggregations, joins, session windows), universal compaction frequently produces fewer catastrophic stalls, because it avoids the sharp L0-to-L1 cliff that leveled compaction enforces.

#Write Amplification as the Root Metric

Write amplification (WA) is the ratio of bytes physically written to disk versus bytes logically written by the application. A WA of 20x means a 1KB record write to your state store results in 20KB of actual disk I/O once you account for flushes and every subsequent compaction pass across levels. When your underlying EBS volume or NVMe device has a sustained IOPS ceiling, WA is the multiplier that determines how quickly you hit it. This is the number to instrument first, before touching any other RocksDB tunable.

#Implementation Logic

The remediation sequence for compaction storms follows a consistent order of operations: isolate I/O, widen the L0 buffer, rate-limit background compaction, and only then consider switching compaction style.

#Step 1 — Isolate RocksDB I/O from the Checkpoint/WAL Path

Co-locating RocksDB’s SST files with Flink’s checkpoint directory or Kafka’s log segments on the same volume means compaction I/O directly competes with your durability guarantees. Mount a dedicated NVMe volume for state.backend.rocksdb.localdir and keep it separate from checkpoint storage.

#Step 2 — Raise the L0 Compaction Trigger and Memtable Budget

Increasing the number of memtables and the L0 trigger buys headroom before RocksDB engages write-stop, giving the background compaction threads more time to catch up during a burst.

1RocksDBOptions options = new RocksDBOptions();
2options.setMaxWriteBufferNumber(6);
3options.setWriteBufferSize(128 * 1024 * 1024); // 128MB per memtable
4options.setLevel0FileNumCompactionTrigger(8);
5options.setLevel0SlowdownWritesTrigger(20);
6options.setLevel0StopWritesTrigger(36);
7options.setMaxBackgroundJobs(6); // parallel flush + compaction threads

#Step 3 — Apply a Compaction Rate Limiter

Rather than letting compaction burst to consume all available disk bandwidth (which starves foreground flushes), cap it explicitly. This is the single highest-leverage change for preventing compaction storms from ever reaching write-stop thresholds.

1// Cap background compaction/flush I/O to 80MB/s
2RateLimiter rateLimiter = new RateLimiter(
3    80L * 1024 * 1024, // bytes per second
4    100_000,            // refill period (microseconds)
5    10                  // fairness
6);
7options.setRateLimiter(rateLimiter);

#Step 4 — Evaluate Universal Compaction for Write-Heavy Stores

For state stores dominated by high-frequency upserts (e.g. windowed aggregations with short TTLs), switch compaction style entirely:

1# flink-conf.yaml
2state.backend: rocksdb
3state.backend.rocksdb.compaction.style: universal
4state.backend.rocksdb.compaction.level.use-dynamic-size: true
5state.backend.rocksdb.thread.num: 4
6state.backend.rocksdb.writebuffer.count: 6
7state.backend.rocksdb.writebuffer.size: 134217728

These four steps, applied in order, address the majority of compaction storms observed in production Flink and Kafka Streams deployments. Teams designing new stateful services from scratch should bake these tunables into their architectural patterns for state store provisioning rather than retrofitting them after an incident.

#Monitoring: Detecting a Storm Before It Stalls Writes

RocksDB exposes granular statistics via its built-in ticker counters. Export these into Prometheus so compaction storms are visible before they cause a write-stop event, not after.

1# Example: scraping RocksDB stats exposed via JMX bridge in a Kafka Streams app
2curl -s localhost:9404/metrics | grep -E "rocksdb_(compaction_pending|num_running_compactions|stall_micros)"
3
4# Expected healthy baseline
5rocksdb_compaction_pending 0
6rocksdb_num_running_compactions 1
7rocksdb_stall_micros 0

A sustained non-zero compaction_pending value combined with rising stall_micros is the definitive leading indicator of an imminent write-stop. Alert on this before latency degrades at the application layer, because by the time end-to-end lag increases, the storm has already begun consuming your I/O budget.

#Failure Modes & Edge Cases

Several edge cases turn a manageable compaction backlog into a full-blown outage:

• Kafka partition rebalance during a storm: A rebalance that migrates a heavily loaded state store to a new task manager forces a full RocksDB restore from the changelog topic, which itself generates a fresh burst of writes — compounding an existing compaction backlog rather than relieving it.
• TTL-driven tombstone accumulation: State stores using RocksDB’s TTL compaction filter can silently accumulate tombstones
  The KBY Lexicon

  Tombstone (Distributed Systems)

  A tombstone is a marker record written to signify the deletion of a key or value, rather than physically removing the data at write time. It exists to ensure delete operations propagate correctly across replicas in eventually consistent systems, preventing deleted data from being resurrected by stale replicas during anti-entropy or read-repair.

  Read Full Context
  faster than periodic compaction reclaims them, inflating read amplification even when write throughput looks nominal.
• Snapshot/checkpoint contention: Incremental checkpointing in Flink relies on referencing existing SST files. If compaction rewrites files mid-checkpoint, the checkpoint coordinator must wait, extending checkpoint duration and increasing the window during which further writes accumulate in the memtable.
• Disk-level throttling on cloud volumes: On burstable EBS volume types (gp3 baseline, gp2 burst-bucket exhaustion), a compaction storm that appears tunable on local NVMe becomes unrecoverable once the cloud volume’s burst credits deplete, because the underlying IOPS ceiling drops beneath what even a rate-limited compactor requires.

#Scaling & Security Trade-offs

Tuning compaction storms is not a purely performance exercise — every lever carries a corresponding trade-off in resource cost, durability guarantees, or operational complexity.

• Leveled vs Universal Compaction: Leveled compaction gives predictable read latency and lower space amplification (better for cost-sensitive multi-tenant clusters); universal compaction reduces write amplification and compaction storm frequency but requires roughly 1.5-2x more disk headroom for transient file merges.
• Rate Limiting Compaction I/O: Protects foreground write latency but extends the time state stores spend in a degraded read-amplified condition, increasing point-lookup latency for downstream joins during sustained bursts.
• Larger Memtable Buffers: Absorbs burst writes without triggering write-stop, but increases recovery time on task manager failure, since a larger memtable means more unflushed data must be replayed from the changelog topic.
• Dedicated NVMe Volumes per State Store: Removes I/O contention entirely but increases infrastructure cost and blast radius for capacity planning across a large fleet of stateful tasks.
• Security posture: Local RocksDB directories on ephemeral compute (EKS spot nodes, Kubernetes emptyDir volumes) must have encryption-at-rest enforced independently, since RocksDB itself performs no transparent encryption — this is a frequent compliance gap in stateful streaming deployments that is easy to miss when the focus is purely on compaction throughput.

For a canonical reference on the full set of tunables and their interactions, consult the official RocksDB Tuning Guide before applying changes to a production state backend, since several parameters (notably max_background_jobs and thread pool sizing) have non-obvious interdependencies with the host’s available CPU core count.

Treat compaction storms as a capacity planning problem rather than an incident to firefight reactively. Instrumenting write amplification and compaction pending counters at deployment time, rather than after the first stall, is what separates state stores that degrade gracefully under load from those that cascade into full pipeline outages.