Kafka Exactly-Once Semantics

Kafka offers three message delivery guarantees. At-most-once: messages may be lost but never duplicated (fire and forget). At-least-once: messages are never lost but may be delivered more than once if the producer retries after a failure. Exactly-once: every message is delivered and processed exactly one time - no loss, no duplicates. Exactly-once is the hardest guarantee and requires specific producer and consumer configuration. It matters most in financial pipelines, payment processing, and data migration where duplicates cause incorrect balances or double-counted rows.

Kafka achieves exactly-once through two features. The idempotent producer (enable.idempotence=true) assigns each message a unique sequence number so the broker can detect and discard retried duplicates from the same producer session. Transactions (transactional.id) extend this further by wrapping multiple produce operations into an atomic transaction that either all succeed or all fail - enabling the read-process-write pattern where you consume from one topic, process, and produce to another topic atomically. Both features are transparent to consumers that use isolation.level=read_committed.

The below example shows enabling idempotent production and using Kafka transactions to atomically consume, transform, and produce without duplicates.

It gives the following output,

Payment sent - exactly once guaranteed

# Without idempotence (default):
# Network failure -> producer retries -> 2 identical records in topic
# ACC-B receives $200 instead of $100

# With idempotence enabled:
# Network failure -> producer retries -> broker deduplicates
# ACC-B receives $100 exactly once

It gives the following output,

Transaction committed - exactly once

# The payment message in "payments" topic AND the consumer offset
# for "orders" are committed atomically.
# If the app crashes after commitTransaction(): message is visible, offset committed
# If the app crashes before commitTransaction(): message is invisible (aborted),
#   offset not advanced, message reprocessed on restart -> no duplicates

# Consumer must set isolation.level=read_committed to see only committed messages
# props.put("isolation.level", "read_committed");

Exactly-once has a performance cost - transactions add latency because the broker must coordinate commit state. For most use cases, at-least-once with idempotent consumers (making your processing logic idempotent by checking for duplicates before acting) is a simpler and faster alternative. Use Kafka transactions only when the cost of duplicates is genuinely unacceptable and your throughput requirements can absorb the overhead - typically financial ledgers, payment processing, and strict audit trails.