Log Compaction (Kafka)

A topic-cleanup strategy that retains only the most recent record for each message key within a partition, rather than deleting records purely by age or size. It transforms an append-only log into a compacted changelog suitable for rebuilding keyed state, such as KTables or database snapshots.

Practical example

A Kafka Streams application maintains a KTable of account balances backed by a compacted topic; on node restart, the local RocksDB store is rebuilt by replaying the compacted changelog from offset zero instead of querying an external database, but a misconfigured dedupe buffer causes partial compaction and inflates the changelog restore time from seconds to minutes.

Kafka partitions are physically stored as a sequence of immutable segments. Under cleanup.policy=compact, a background log cleaner thread periodically scans closed segments and rewrites them, discarding all but the latest record for each key. This is fundamentally different from cleanup.policy=delete, which drops entire segments based on retention.ms or retention.bytes irrespective of key uniqueness. Compaction guarantees that a full scan of the partition from offset 0 reconstructs the latest state for every key ever written, which is why it underpins Kafka Streams’ KTable semantics and Kafka’s own internal __consumer_offsets and Connect config topics.

The cleaner operates on a dirty ratio threshold (min.cleanable.dirty.ratio): it only recompacts a partition once the ratio of uncompacted (dirty) bytes to total bytes exceeds this value, trading CPU/I/O cost against staleness. min.compaction.lag.ms further delays compaction of very recent records so consumers reading near the tail still see intermediate versions, which matters for exactly-once consumers doing delta processing. Deletion of a key is expressed via a tombstone: a record with the target key and a null value. Tombstones are retained for delete.retention.ms (default 24h) so that slow or lagging consumers still observe the delete before the cleaner physically removes it — if a consumer’s offset lag exceeds this window, it can resurrect a logically deleted key by never seeing the tombstone.

Compaction interacts poorly with a few operational patterns. Large values combined with high key cardinality make the cleaner’s in-memory offset map (log.cleaner.dedupe.buffer.size) a hard scaling constraint — if the map can’t hold all unique keys in a segment set, cleaning is deferred or partial, silently increasing storage and stale-read risk. Mixed policy (compact,delete) is used for topics that need both keyed dedup and a hard TTL, but ordering of cleaner passes vs. delete-based segment eviction can create edge cases where a still-relevant compacted record disappears prematurely. Compaction also does not guarantee compaction happens instantly after a write — the active (uncompacted) segment is never touched by the cleaner, so readers can transiently see multiple versions of a key even in a ‘compacted’ topic.

Architecturally, compacted topics are the durable source-of-truth substrate for event-sourced state rebuilding: Kafka Streams state stores and ksqlDB materialized tables replay a compacted changelog topic on startup/rebalance to restore local RocksDB state without needing a separate database. This shifts the durability and backup problem from an external store onto Kafka’s own retention and replication guarantees, making topic configuration (replication factor, min.insync.replicas, cleaner tuning) a direct dependency for state-store recovery time objectives.