Apache Kafka
In plain English
Plain definition
Kafka is a shared, ordered log that many systems write to and read from. A payment service records "Alice paid Bob" as an event; a fraud check and a ledger each read it at their own pace; the producer never waits.
Apache Kafka is an event streaming platform from the Apache Software Foundation: a cluster of servers that stores events durably in partitioned, replicated topics, so applications can write (publish) and read (subscribe to) them independently, in real time or later. Events are not deleted when they are read; each topic keeps them for a configured retention period. In operation, durability matters most: a write is only as safe as the topic’s replication factor, min.insync.replicas and the producer’s acks setting together make it.
Technical definition
Kafka is a distributed system of servers and clients that talk over a TCP protocol. Its parts:
- Events: records with a key, value, timestamp and optional headers.
- Topics and partitions: a topic is split into partitions across brokers. Events with the same key go to the same partition, read in the order written.
- Brokers: the servers that form the storage layer.
- Replication: each partition has one leader and zero or more followers, together the replication factor. Followers that keep up form the in-sync replica set (ISR); one that cannot catch up within
replica.lag.time.max.msis removed. - Controllers (KRaft): servers with
process.roles=controllermanage cluster metadata. A majority must be alive, so three controllers tolerate one failure and five tolerate two. Kafka 4.0 and later run only in KRaft mode; ZooKeeper mode has been removed.
Operational relevance
Three topic settings decide most incidents:
retention.msdefaults to 604800000 (7 days); after that, old segments are discarded under the defaultdeletecleanup policy. Settingcleanup.policy=compactinstead keeps the latest value for each key.min.insync.replicasdefaults to 1. Withacks=all, if the ISR is smaller than this value the producer receivesNotEnoughReplicasorNotEnoughReplicasAfterAppend.- Unclean leader election is off by default (
unclean.leader.election.enable=false): a partition whose replicas have all failed stays unavailable rather than electing a replica that may lack committed messages.
Architecture relationship
Kafka Connect moves data to and from other systems, Kafka Streams processes events inside applications, and the Admin API manages topics and brokers. Consumers in a group share a topic’s partitions, each tracking one offset per partition, so a consumer can rewind and re-read. More KBY guides are on the Apache Kafka technology page.
Example
A team creates a topic with replication factor 3, min.insync.replicas 2 and producers using acks=all, the typical combination in the Kafka documentation. Check it, read-only:
bin/kafka-topics.sh --describe --topic payments --bootstrap-server localhost:9092
Each partition line shows Leader, Replicas and Isr. If two of a partition’s three replicas drop out of the ISR, writes with acks=all to that partition are refused with NotEnoughReplicas instead of being accepted on a single copy: the durability setting working, not a client bug.
Common misunderstanding
acks=all does not mean every assigned replica has the write, only every current in-sync replica. With two replicas and one failed, acks=all writes still succeed on the single remaining copy unless min.insync.replicas is raised. Nor is Kafka a queue that deletes messages once read: retention decides when data goes.
Related terms
- Consumer lag: the gap between a group’s current offset and the log-end offset for a partition.
- ISR: the replicas that are caught up with the partition leader.
- Log compaction: retention that keeps the latest value per key. Compaction in other storage engines has its own costs; see taming RocksDB compaction storms.
- KRaft: Kafka’s built-in metadata quorum of controllers.
Further reading
- Apache Kafka, “Introduction”: events, topics, partitions and the core APIs.
- Apache Kafka 4.4, “Design”: replication, ISR and durability.
- Apache Kafka 4.4, “Topic Configs” and “Basic Kafka Operations”.
Checking Kafka durability and consumer lag safely
- Read the topic: run
kafka-topics.sh --describeand confirm every partition’sIsrlists as many brokers asReplicas. - Read consumer progress:
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-groupshowsCURRENT-OFFSET,LOG-END-OFFSETandLAGper partition. Steadily growing lag means consumers are falling behind. - Before changing a setting: record the current values; that record is your rollback. Test on a non-production topic and restore the recorded value if producers start failing.
The next safe decision is usually whether a topic needs min.insync.replicas above 1, trading write availability during broker loss for never acknowledging an acks=all write held on one copy.
Authoritative Sources
Technology documentation
Introduction
Design (Apache Kafka 4.4)
Topic Configs (Apache Kafka 4.4)
Basic Kafka Operations (Apache Kafka 4.4)
Quick Start (Apache Kafka 4.4)
KRaft (Apache Kafka 4.4)
Upgrading (Apache Kafka 4.4)
About this page
Credited to
Dates
Published
Corrections
No corrections are listed for this page. Some review sweeps corrected pages without naming each one; those are in the corrections log.
Was this useful?
Reports go to the site owner and are used to correct pages. Corrections are listed at /corrections.