Exactly-Once Semantics & Transactional Guarantees
Your payment service processes Kafka messages. A producer sends 1000 transactions. Network hiccup causes timeout. Producer retries. Messages appear twice in broker. How many times does your service charge the customer?
Without exactly-once semantics, your service charges twice. The producer retries a message that already succeeded (at-least-once delivery). Your service must implement idempotency. Solutions: 1) Enable idempotent producer (Kafka 0.11+): set enable.idempotence=true. Kafka uses producer ID + sequence number to deduplicate. Producer can retry safely—broker discards duplicates. 2) Add idempotency key to your data: transaction_id=uuid. Your payment service stores processed IDs in a DB. On duplicate message, it checks if already processed and returns cached result without charging again. 3) Use Kafka transactions (Kafka 0.11+): producer writes payment messages in atomic transaction. Either all succeed or all fail, no partial writes. Set transactional.id=payments-service and isolation.level=read_committed on consumer. 4) For exactly-once: combine idempotent producer + transactional writes + idempotency DB. Best practice: idempotent producer is enabled by default in Kafka 3.0+. Test: produce 1000 messages, restart producer mid-way, verify no duplicates on broker. Run kafka-console-consumer --bootstrap-server localhost:9092 --topic payments --from-beginning | sort | uniq -d to find duplicates.
Follow-up: If idempotent producer is enabled but you still see duplicates, what's the likely cause?
You set up a consumer that reads from Kafka, processes payment, writes result to DB. Consumer crashes after writing to DB but before committing offset. On restart, consumer reprocesses the message and charges again. Fix it.
This is the "read → process → write → commit" problem. If crash happens between write and commit, reprocessing is guaranteed. Solutions in order of preference: 1) Transactional write: store offset + result in same transaction. Use KafkaProducer.sendOffsetsToTransaction() to include offset commit in transaction. On crash, neither offset nor DB write are committed. On restart, consumer reprocesses and retries. Requires DB to support transactions. 2) Idempotent DB writes: use payment_id as primary key. On duplicate message, DB insert fails (constraint violation). You catch exception and continue—idempotency achieved. This is the most practical for payment systems. 3) Offset before processing: commit offset before writing to DB. Risk: crash after commit but before write loses the message. Use if payment system has retry logic. 4) Manual offset management: implement on_commit_callback() to verify DB write succeeded before committing offset. Implementation: consumer.commitAsync(offsets, callback) where callback checks DB. Code pattern: catch exception, don't commit, let rebalance assign partition again on next poll. Test: inject crash after DB write, verify no charge on restart. Use JMX metric kafka.consumer:type=consumer-coordinator-metrics,name=commit-latency-avg to track offset commit timing.
Follow-up: If you use transactions but the DB crashes before committing, are you back to square one?
You enable Kafka transactions and set isolation.level=read_committed on consumer. Consumer reads message from topic. Does it see uncommitted messages from other producers?
No. With read_committed, consumer only sees committed messages. If producer A sends 3 messages in transaction but crashes before commit, consumer never sees them. Producer B's committed messages appear after producer A's aborted transaction. This is ACID-like isolation. Important: read_committed has performance cost—broker must track transaction state and filter uncommitted messages. Throughput drops ~10-20%. Default isolation.level=read_uncommitted sees all messages (higher throughput, lower durability). Use read_committed for payments, orders. Use read_uncommitted for logs, analytics. Behavior: when consuming with read_committed, broker stops sending messages at the LSO (Last Stable Offset) < high watermark. This means your consumer lags behind the leader's log. Measure: kafka-consumer-groups --group payments --describe --bootstrap-server localhost:9092 shows LAG including unflushed messages. Under high transaction load, LAG can grow even with low producer rate. To monitor: query JMX kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=* to see replication lag. If lag > 100K under normal load, investigate producer transaction frequency.
Follow-up: If a producer transaction fails, how long before consumer sees committed messages from other producers?
Your microservice reads from Kafka, calls an external payment API, writes result back to Kafka. Process crashes after API call but before writing result. On restart, it retries API call. API is idempotent but double-charges if called twice in 60 seconds. How do you guarantee exactly-once?
This is the hard problem: exactly-once across multiple systems. Kafka transactions only cover Kafka—they can't protect external API calls. Solutions: 1) Dual-write (not recommended): write result to DB first, then to Kafka. On crash after DB write, re-read from DB on restart and confirm with Kafka before reprocessing. Risky because DB and Kafka can diverge. 2) Kafka transactions + idempotent external API: use Kafka transactions for result writes, implement idempotency key in API request. On retry, API returns cached result without charging twice. Code: beginTransaction(); callAPI(idempotencyKey=offset); writeResult(); commitTransaction();. If crash between API and commit, restart reprocesses and calls API again with same key—API returns cached result. 3) Transactional outbox pattern: write results + offset to local DB in single transaction. Async process reads DB changes, publishes to Kafka, marks as published. On crash, outbox replay ensures exactly-once. This is the most robust. 4) Event sourcing: never delete/update, only append. On crash, reprocess same event (idempotent operations only). Replay from changelog. Recommended for payments: use approach #2 (Kafka transactions + idempotent API). Requires API to support idempotency keys (UUID in request header). Test: crash at each step, verify exactly-once on restart.
Follow-up: If the external API doesn't support idempotency keys, can you still achieve exactly-once with Kafka?
You have producer A sending to topic1, consumer reading from topic1, processing, and writing to topic2 via producer B. Both use Kafka transactions. Consumer crashes mid-process. What's guaranteed when it restarts?
With Kafka transactions end-to-end: exactly-once processing guarantee (E2E). When consumer crashes, it hasn't committed offset (transaction uncommitted). On restart, it reads same message from topic1, reprocesses, writes to topic2 again (same message), commits transaction. Kafka deduplicates on producer B side (if producer B has idempotence enabled). End result: message appears once in topic2, despite consumer crash. Implementation: consumer.poll(); producer.beginTransaction(); producer.send(topic2_msg); consumer.commitSync(); producer.endTransaction();. Actually, this pattern uses sendOffsetsToTransaction(): producer.beginTransaction(); producer.send(topic2_msg); producer.sendOffsetsToTransaction(consumer_offsets, consumer_group); producer.commitTransaction();. This ensures offset commit is atomic with output writes. If crash after send but before commit, all writes (output + offset) are rolled back. Exactly-once achieved. Caveat: this is slow (2-phase commit overhead). Throughput: ~1000 msg/sec vs 100K msg/sec without transactions. Use for high-value data only (payments, orders). Measure: kafka.producer:type=producer-metrics,name=transaction-begin-latency-avg and *-abort-latency-avg.
Follow-up: If you have 3 consumers in a group reading from topic1, do all 3 need to use transactions?
You enable transactions on your producer. First request works. Second request times out during endTransaction(). You retry from scratch (new transaction). Does the first transaction eventually commit or abort?
The first transaction will eventually commit or abort based on transaction.state.log.replication.factor (default 3). Kafka writes transaction state to __transaction_state topic. If your endTransaction() times out, the broker is still processing the commit. Your producer has transactional.id which is unique. On retry, producer opens new transaction with same ID. Kafka's transaction manager detects duplicate ID: it aborts the new transaction and waits for previous transaction to complete. If you retry too many times, previous transaction times out. Set transaction.max.timeout.ms (default 15 min)—if producer doesn't hear back by then, transaction is rolled back. Best practice: don't retry on endTransaction() timeout. Instead, gracefully shut down producer and check __transaction_state topic to verify commit. Or increase producer timeout: set request.timeout.ms from 30s to 60s. Monitor: set alert on kafka.producer:type=producer-metrics,name=txn-abort-total. If abort rate > 1/min, something is failing. Diagnose broker health: check disk space, I/O, replication lag on __transaction_state topic. Run kafka-topics --describe --topic __transaction_state --bootstrap-server localhost:9092.
Follow-up: If __transaction_state partition is offline, what happens to producer transactions?
You use exactly-once semantics. Consumer processes 100 messages, then rebalance happens. Consumer must commit offset before rebalancing. If it crashes during commit, will messages be reprocessed?
Yes, they will be reprocessed. During rebalance, if consumer crashes before offset is committed, the last committed offset is retained. Next consumer picks up from that offset and reprocesses. This is expected behavior—rebalance shouldn't lose data. To minimize reprocessing: ensure on_partitions_revoked() callback commits offset synchronously. Code: consumer.poll(); process_messages(); on_revoke: commitSync(); on_assign: initialize_state();. Kafka consumer automatically calls revocation callback before giving partition to new consumer. If you don't commit there, messages are replayed. For exactly-once + rebalance: use transactions + static membership. Set group.instance.id=consumer-1. On rebalance, consumer keeps partitions assigned (no handoff). After processing, commit in transaction: producer.sendOffsetsToTransaction(offsets, group_id). Rebalance happens, consumer is still alive, commits succeed. If consumer dies, next instance with same instance_id rejoins, continues from last committed offset. This minimizes unnecessary reprocessing. Measure: track rebalance_duration and messages_replayed_count. Healthy:
Follow-up: If rebalance takes 2 minutes, should you increase session.timeout.ms to prevent losing the consumer?