Exactly-Once Semantics (EOS)

EOS is built from two distinct layers that are frequently conflated. The first is the idempotent producer: on initialization, the broker assigns a producer a unique PID (Producer ID) and each message batch carries a monotonically increasing sequence number per partition. The broker rejects duplicate sequence numbers, which neutralizes the classic “retry after ack-timeout” duplicate problem for a single partition, single producer session. This alone does not provide atomicity across partitions or topics.

The second layer is the transactional API, which layers a two-phase-commit-like protocol on top. A producer registers a transactional.id with a Transaction Coordinator (a designated broker holding the internal __transaction_state topic). Each transaction increments a producer epoch tied to the same transactional.id; if a producer process is killed and a new instance starts with the same transactional.id, the coordinator fences the old epoch, rejecting any in-flight requests from the zombie instance. On commit, the coordinator writes control markers (COMMIT/ABORT) to every partition involved, and consumer offset commits for the input topic can themselves be written as part of the same transaction — this is what makes read-process-write loops (the dominant Kafka Streams pattern) atomic end-to-end within the Kafka cluster.

Consumers must opt into isolation.level=read_committed to respect these markers; otherwise they will observe uncommitted or aborted transactional writes, defeating the guarantee. Several edge cases dominate operational pain: transaction timeouts (transaction.timeout.ms) force an abort if a producer stalls mid-transaction, which can cascade into consumer-side stalls waiting for the commit marker; consumer group rebalances mid-transaction can orphan partially written state; and any transaction spanning more partitions than the coordinator’s batch write capacity increases commit latency, directly impacting p99 write latency under load. Critically, EOS guarantees are scoped strictly to the Kafka log — any side effect outside Kafka (an HTTP call, an external DB write, a cache invalidation) triggered during processing is not covered and can still be duplicated on retry, so “exactly once” is more precisely “exactly once within Kafka, effectively-once with an idempotent external sink.”

Architecturally, EOS shifts deduplication cost from N downstream consumers to a single producer/broker negotiation, but it is not free: throughput drops due to added coordinator round-trips, and stateful stream topologies (Kafka Streams, ksqlDB) pay additional latency for state-store changelog commits inside the same transaction. Teams building financial ledgers, billing pipelines, or exactly-once materialized views should treat EOS as a correctness primitive to compose with idempotent sinks, not a substitute for idempotency design at the system boundary.