Skip to main content
Lexicon
Lexicon
Apache Kafka

Apache Kafka

Difficulty: Intermediate
5 min read
Updated 2026-11-17

In plain English

Plain definition

Kafka is a shared, ordered log that many systems write to and read from. A payment service records "Alice paid Bob" as an event; a fraud check and a ledger each read it at their own pace; the producer never waits.

Apache Kafka is an event streaming platform from the Apache Software Foundation: a cluster of servers that stores events durably in partitioned, replicated topics, so applications can write (publish) and read (subscribe to) them independently, in real time or later. Events are not deleted when they are read; each topic keeps them for a configured retention period. In operation, durability matters most: a write is only as safe as the topic’s replication factor, min.insync.replicas and the producer’s acks setting together make it.

Technical definition

Kafka is a distributed system of servers and clients that talk over a TCP protocol. Its parts:

  • Events: records with a key, value, timestamp and optional headers.
  • Topics and partitions: a topic is split into partitions across brokers. Events with the same key go to the same partition, read in the order written.
  • Brokers: the servers that form the storage layer.
  • Replication: each partition has one leader and zero or more followers, together the replication factor. Followers that keep up form the in-sync replica set (ISR); one that cannot catch up within replica.lag.time.max.ms is removed.
  • Controllers (KRaft): servers with process.roles=controller manage cluster metadata. A majority must be alive, so three controllers tolerate one failure and five tolerate two. Kafka 4.0 and later run only in KRaft mode; ZooKeeper mode has been removed.

Operational relevance

Three topic settings decide most incidents:

  • retention.ms defaults to 604800000 (7 days); after that, old segments are discarded under the default delete cleanup policy. Setting cleanup.policy=compact instead keeps the latest value for each key.
  • min.insync.replicas defaults to 1. With acks=all, if the ISR is smaller than this value the producer receives NotEnoughReplicas or NotEnoughReplicasAfterAppend.
  • Unclean leader election is off by default (unclean.leader.election.enable=false): a partition whose replicas have all failed stays unavailable rather than electing a replica that may lack committed messages.

Architecture relationship

Kafka Connect moves data to and from other systems, Kafka Streams processes events inside applications, and the Admin API manages topics and brokers. Consumers in a group share a topic’s partitions, each tracking one offset per partition, so a consumer can rewind and re-read. More KBY guides are on the Apache Kafka technology page.

Example

A team creates a topic with replication factor 3, min.insync.replicas 2 and producers using acks=all, the typical combination in the Kafka documentation. Check it, read-only:

bin/kafka-topics.sh --describe --topic payments --bootstrap-server localhost:9092

Each partition line shows Leader, Replicas and Isr. If two of a partition’s three replicas drop out of the ISR, writes with acks=all to that partition are refused with NotEnoughReplicas instead of being accepted on a single copy: the durability setting working, not a client bug.

Common misunderstanding

acks=all does not mean every assigned replica has the write, only every current in-sync replica. With two replicas and one failed, acks=all writes still succeed on the single remaining copy unless min.insync.replicas is raised. Nor is Kafka a queue that deletes messages once read: retention decides when data goes.

  • Consumer lag: the gap between a group’s current offset and the log-end offset for a partition.
  • ISR: the replicas that are caught up with the partition leader.
  • Log compaction: retention that keeps the latest value per key. Compaction in other storage engines has its own costs; see taming RocksDB compaction storms.
  • KRaft: Kafka’s built-in metadata quorum of controllers.

Further reading

  • Apache Kafka, “Introduction”: events, topics, partitions and the core APIs.
  • Apache Kafka 4.4, “Design”: replication, ISR and durability.
  • Apache Kafka 4.4, “Topic Configs” and “Basic Kafka Operations”.

Checking Kafka durability and consumer lag safely

  1. Read the topic: run kafka-topics.sh --describe and confirm every partition’s Isr lists as many brokers as Replicas.
  2. Read consumer progress: bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group shows CURRENT-OFFSET, LOG-END-OFFSET and LAG per partition. Steadily growing lag means consumers are falling behind.
  3. Before changing a setting: record the current values; that record is your rollback. Test on a non-production topic and restore the recorded value if producers start failing.

The next safe decision is usually whether a topic needs min.insync.replicas above 1, trading write availability during broker loss for never acknowledging an acks=all write held on one copy.

About this page

Credited to

Dates

Published

Corrections

No corrections are listed for this page. Some review sweeps corrected pages without naming each one; those are in the corrections log.

Was this useful?

Reports go to the site owner and are used to correct pages. Corrections are listed at /corrections.