Your payment service processes a charge request from a mobile app. Network timeout occurs—client doesn't get response but server processed the charge (money deducted). Client retries, charge is processed again. Customer sees $100 charge twice. Root cause: no idempotency tracking. Your SLA: zero duplicate charges. How do you fix this?
This is the idempotency key problem. Solution: store idempotency key in database, check before processing. Architecture: (1) Client generates idempotency key: unique, stable ID for this transaction. `idempotency_key = sha256(user_id + action_id + amount + timestamp_second)`. Client includes this in request header: `Idempotency-Key: abc123def456`. (2) Server receives request, checks if key exists in idempotency store (Redis + DynamoDB): `if idempotency_key in store then return cached_response; else continue`. (3) If key doesn't exist, process charge: deduct money, update ledger, generate receipt. Store response in idempotency store: `{idempotency_key: abc123def456, response: {transaction_id: 123, status: success}, timestamp: now}`. (4) Return response to client. (5) If client retries with same idempotency key, server returns cached response (no double charge). (6) Idempotency store TTL: 24 hours (or 30 days for financial audit, depends on regulation). Cost: Redis/DynamoDB for idempotency store = $300/month. For 1M payments/day at $100 avg, this is acceptable. Expected outcome: guaranteed exactly-once semantics. Network retries don't cause duplicates. Trade: slight latency increase (check cache before processing, ~10ms). Caveat: idempotency key collision (two unrelated transactions generate same key) is a problem. Mitigation: use strong hash (SHA-256) and include timestamp_second (collision probability negligible). Alternative (simpler but less safe): client-assigned UUID. Client generates UUID (random), includes in request. Server checks if UUID exists; if so, return cached response. This is simpler but requires client cooperation (clients must reuse UUID on retry).
Follow-up: Your idempotency store hit capacity: 1M payments × 30-day TTL = 30M keys in Redis (requires 16GB). Cost exceeded budget. How do you reduce storage while maintaining exactly-once guarantees?
Your order service processes orders asynchronously: client submits order, order goes to queue, async processor handles it. During processing, if the processor crashes before publishing OrderCreated event to Kafka, the order isn't created but the queue message is reprocessed. Next attempt creates duplicate order. How do you prevent this?
This is transactional outbox problem. Async processing without atomicity causes duplicates. Solution: use outbox pattern (transactional writes). Architecture: (1) Order processing: (a) Read order from queue (mark as processing). (b) Write order to database + outbox table in single transaction: `BEGIN; INSERT INTO orders (id, status) VALUES (...); INSERT INTO outbox (id, event_type, payload) VALUES (...); COMMIT;`. (c) If transaction succeeds, both order and outbox event are persisted. If crash, transaction rolls back. (d) If transaction fails (network issue), don't acknowledge queue message (message stays in queue, reprocessed). (2) Outbox reader: separate process reads outbox table, publishes events to Kafka. On success, deletes outbox row. Logic: `SELECT * FROM outbox LIMIT 100; PUBLISH to Kafka; DELETE from outbox;`. (3) Exactly-once guarantee: if outbox reader crashes after publishing to Kafka but before deleting outbox row, on recovery, same event is republished to Kafka. Consumers must handle duplicates (idempotent processing). (4) Monitoring: track outbox table size. If outbox has >10K rows, indicates slow publisher (network issue, Kafka down). Alert. (5) Cost: extra outbox table (~1GB for 100M orders), outbox reader process (~$200/month). Expected outcome: exactly-once event delivery (to first consumer). Consumers still need idempotency for exactly-once end-to-end. Trade: 2-phase commit (order + outbox in same transaction) is more reliable but slightly slower (<50ms overhead). Alternative (simpler, less safe): assume crash is rare, don't use outbox. Acceptable for non-critical flows (e.g., analytics events). For financial flows (orders, payments), outbox is mandatory.
Follow-up: Your Kafka cluster goes down for 2 hours. During this time, outbox table fills up with 100K unpublished events. When Kafka recovers, outbox reader tries to publish all 100K events at once. Kafka is overwhelmed and rejects requests. How do you handle publisher backpressure?
Your payment gateway integrates with an external payment processor (e.g., Stripe). You send charge request via API, get response. But if your service crashes after charging Stripe but before saving transaction record locally, you've lost the charge data. On restart, you retry the charge, and Stripe processes it twice (duplicate charge). How do you handle this?
This requires idempotency coordination between your service and external API. Solution: (1) External API idempotency: Most payment processors (Stripe, Square) support idempotency keys. Before calling Stripe, generate idempotency key: `idempotency_key = uuid4()`. Include in API request header: `Idempotency-Key: abc123`. Stripe stores this key, if you retry with same key, returns cached response (no duplicate charge). (2) Your service: store idempotency key locally before calling Stripe. Sequence: (a) Generate idempotency_key = uuid4(). (b) Save to DB: `INSERT INTO pending_charges (key, user_id, amount, status) VALUES (...)`. (c) Call Stripe API with Idempotency-Key header. (d) On success, update DB: `UPDATE pending_charges SET status = 'charged', transaction_id = stripe_txn_id`. (e) On failure/timeout, DB still has pending charge. On retry (manual or automatic), query DB for pending charge by key, get transaction_id. If already charged (status = 'charged'), return success (cached). If still pending, retry Stripe API with same Idempotency-Key. (3) Reconciliation: at end of day, reconcile pending charges with Stripe: query Stripe API for all charges by idempotency_key. If Stripe has charge but local DB doesn't, update local DB (recovery). If local DB has charge but Stripe doesn't, alert (fraud/error). (4) Monitoring: track pending charges. If any pending >24 hours, indicate issue. Expected outcome: exactly-once charging. Retries don't cause duplicate charges. Trade: requires external API to support idempotency (most modern APIs do). Caveat: idempotency key must be stable across retries (client must reuse same key, not generate new key per retry). This requires client-side coordination (tricky).
Follow-up: You charge $100 with idempotency_key=abc123. Stripe succeeds but returns encrypted response. You save to local DB but don't save the Stripe transaction_id (your service crashes before writing transaction_id). On retry with same key, Stripe returns cached response but you can't decrypt it. How do you verify charge succeeded without decryption?
Your distributed system uses message queues (RabbitMQ) for task distribution. Consumer processes message, updates database, publishes event. If consumer crashes after DB update but before publishing event, event is lost. Next consumer doesn't know about the update (state inconsistency). How do you ensure exactly-once semantics across queues and DB?
This is transactional consistency across multiple systems (queue + DB + event queue). Solution: use saga pattern with compensation + idempotency. Architecture: (1) Task processing: (a) Consume message from RabbitMQ. (b) Extract task_id, generate idempotency_key = hash(task_id). (c) Check if already processed: query DB: `SELECT * FROM processed_tasks WHERE idempotency_key = ...`. If exists, publish cached event, skip processing. (d) Process task: update DB with status = processing. (e) Perform work (call external API, compute, etc.). (f) Update DB with status = completed, store result. (g) Publish event to Kafka. (h) Acknowledge message to RabbitMQ (safe to remove). (2) Idempotency: store all state changes in DB, keyed by idempotency_key. On restart, check if already processed. (3) Order of operations (crucial): - DB update must be first (before publishing event, before acknowledging queue). - Event publishing can be async (outbox pattern). - Queue acknowledgment must be last (ensures message is removed only after DB is durable). (4) If crash at any point: (a) Crash after step (c) (before DB): on recovery, message still in queue, reprocessed. (b) Crash after step (f) (event not published): outbox reader picks up unpublished event, publishes it. (c) Crash after step (h): message acknowledged, event published. No reprocessing. (5) Monitoring: track processed_tasks table. Identify slow consumers (processing time >timeout), rebalance load. Expected outcome: exactly-once processing. At-most-once or at-least-once is not good enough for transactional systems. Trade: complexity (3 systems to coordinate). Requires outbox pattern, idempotency tracking, careful ordering. Cost: extra DB tables (processed_tasks), outbox infrastructure (~$300/month).
Follow-up: Your processed_tasks table stores {idempotency_key, result}. For long-running tasks (e.g., video encoding), result can be 1GB. Storing all results in processed_tasks blows up table size. How do you optimize storage?
Your micro-service architecture has Services A, B, C in a workflow: A emits event, B consumes and emits event, C consumes. For exactly-once end-to-end: A must process order once, B must process event once, C must send notification once. But network failures and retries can cause each service to process multiple times. How do you guarantee exactly-once across all 3 services?
Exactly-once end-to-end requires coordination across all services. This is non-trivial. Solution: idempotency key propagation. Architecture: (1) User submits order to Service A. Include request_id (client-generated UUID). Service A generates event: `{order_id, request_id, ...}`. (2) Service A publishes OrderCreated event with request_id embedded. Service B subscribes: receives event with request_id. (3) Service B: check if already processed this request_id: `SELECT FROM processed WHERE request_id = ...`. If yes, return cached result. If no, process, store result with request_id. (4) Service B publishes NotifyUser event with request_id (propagate same request_id). (5) Service C: check if already processed this request_id. If yes, return. If no, send notification, store with request_id. (6) Request_id flows through entire workflow: A → B → C. All services track which request_ids they've processed. On retry (network timeout, crash), services check request_id, deduplicate. (7) Cost: each service needs processed-request tracking (table or cache). For 1M orders/day, tracking ~30 days = 30M rows per service × 3 services = 90M rows. ~500MB storage. (8) Monitoring: cross-service monitoring. Track request_id through entire workflow (distributed tracing). Alert if request_id takes >5 min to complete (indicates stuck request). Expected outcome: exactly-once end-to-end. Order processed once, notification sent once. Trade: requires all services to cooperate (propagate request_id), adds operational complexity. Caveat: if Service B crashes before publishing NotifyUser event, Service C never receives notification (Service B failure is not transparent to C). Mitigation: Service B has delayed retry + circuit breaker. If Service B can't reach Kafka to publish event, it retries with exponential backoff. After max retries, alert ops team (manual intervention).
Follow-up: Your request_id tracking uses Redis (distributed cache). Redis cache is evicted after 24 hours. A customer retries a payment after 25 hours (due to bank processing delay). Redis has no record of request_id, service processes it again, charges twice. How do you handle long-tail retries?
Your payment service uses exactly-once semantics via idempotency keys. A customer's bank processes the charge, returns success, but network packet is lost on the way back to your service. Your service times out, retries, bank is charged again. Your service thinks first attempt failed, submits second charge with different idempotency key. Bank sees two charges. Your idempotency tracking is useless because you're using different keys. How do you prevent this?
The issue: your service generates new idempotency key per retry instead of reusing the same key. This breaks idempotency. The bank (or external API) would deduplicate with same key, but your service never sends the same key again. Root cause: client-side implementation bug. Fix: (1) Idempotency key must be deterministic and stable. Generate once based on immutable request data, reuse on retries. Bad approach: `idempotency_key = uuid4()` (random, new per retry). Good approach: `idempotency_key = hash(user_id, amount, timestamp_day)` (stable, same value computed on retry). (2) Client implementation: before making any attempt, compute idempotency_key once. Store in local variable. All retries use same key. Code example: ```javascript const request = {amount: $100, user_id: 123}; const idempotency_key = hash(request); // compute once let response; for (let attempt = 0; attempt < 3; attempt++) { try { response = chargeAPI(request, idempotency_key); // reuse same key break; } catch (e) { // retry with same key } } ```. (3) Server-side validation: when receiving request, validate idempotency_key is deterministic (matches expected hash). If client sends different keys for same request, reject (fraud detection). (4) External API coordination: if calling external API (Stripe, etc.), let them manage idempotency. Your service generates idempotency_key = hash(user_id, amount, timestamp_minute), sends to Stripe with same key on all retries. Stripe deduplicates. (5) Testing: add tests for retry scenarios. Ensure client retries use same idempotency_key. Expected outcome: exactly-once charging. Retries use same key, external API deduplicates, no double charges. Trade: requires client-side discipline (compute key once, reuse on retry). If client is buggy (generates new key per retry), idempotency breaks.
Follow-up: Your deterministic idempotency_key = hash(user_id, amount, timestamp_minute). At minute boundaries (e.g., 2:00:59 → 2:01:00), the hash changes even though it's the same request. What's the issue and how do you fix it?
You're designing a critical financial system: account transfers between banks. You need exactly-once semantics: transfer $100 from account A to account B exactly once, no duplicates, no losses. You have 3 payment services (PS-1, PS-2, PS-3) for redundancy. One service might crash mid-transfer. How do you coordinate this?
Financial transfers require strict exactly-once semantics with durability. This is a distributed transaction problem. Solution: use 2-phase commit (2PC) or saga pattern. Architecture with 2PC: (1) Transfer coordinator: (a) Initiate transfer: generate transfer_id (unique, idempotent key). Store in persistent log: `{transfer_id, from_account, to_account, amount, status: initiated}`. (b) Phase 1 (Prepare): Ask payment service: "Can you deduct $100 from account A?" PS-1 reserves $100 (lock), responds yes/no. If yes, store response. If no, abort. (c) Phase 2 (Commit): If all services say yes, send commit command: "Deduct $100 from A and credit $100 to B." Both services apply atomically. If any service says no, send abort command: "Release reserves." (2) Durability: all state changes are logged (coordinator log, service logs). On coordinator crash, recovery: check log, find incomplete transfers, retry phase 1 or 2. (3) Transfer atomicity: either both services succeed or both fail. No partial transfers (A loses $100 but B doesn't gain $100). (4) Timeout handling: if PS-1 doesn't respond to "prepare" within 10 seconds, assume failure, abort transfer. (5) Idempotency: if coordinator retries same transfer_id (due to network failure), both services recognize the ID and return cached response (idempotent). (6) Monitoring: track in-flight transfers. If any transfer stuck in "prepare" phase >10 min, alert ops (manual intervention, might need to rollback). Cost: 2PC implementation is complex. Better: use saga pattern (no strict 2PC consistency, but simpler). Saga: if transfer fails mid-way, compensate (reverse transfer). Cost: extra compensation logic. (7) Trade-off: 2PC guarantees consistency but is slower (2 round trips). Saga is faster (1 round trip) but eventual consistency. For financial systems, 2PC is worth the complexity. Expected outcome: transfer processes exactly once, atomically, durable. If coordinator crashes, transfer can be rolled back/retried safely.
Follow-up: Your 2PC coordinator sends commit command to PS-1 and PS-2. PS-1 acknowledges commit, PS-2 doesn't respond (network down). Coordinator can't decide: is PS-2 committed or not? How do you resolve this?
You operate a subscription billing system. Subscription renewal happens monthly: deduct payment, send receipt email, update billing status. If payment succeeds but email sending fails, should you retry email indefinitely? If billing status isn't updated, customer doesn't see their subscription renewed. You need to handle partial failures gracefully without charging customers multiple times. How do you design this workflow?
This is partial failure recovery in a multi-step workflow. Solution: choreography + compensation (saga pattern with explicit rollback). Workflow: (1) Payment → Email → Update Status. If any step fails, compensate (reverse earlier steps). (2) Step 1 (Payment): charge customer using idempotency_key. On success, store transaction_id in DynamoDB. On failure, skip workflow (customer not charged, no need to compensate). (3) Step 2 (Email): send receipt email. If send fails (3rd-party service down), retry with exponential backoff (1s, 2s, 4s, 8s, max 5 retries). If all retries fail after 1 minute, log and continue (don't block customer's subscription). Store email send attempt with timestamp. (4) Step 3 (Update Status): update billing status to "active". If DB write fails, retry infinitely (your DB is critical, must eventually succeed). (5) Partial failure handling: (a) If Email fails but Payment + Status succeed: customer is charged, subscription is active, but no receipt. On next login, display banner: "Your receipt is being sent." Async job retries email (eventual delivery). (b) If Status fails but Payment succeeds: customer is charged, email sent, but subscription shows as "pending." Retry status update every 10 seconds until success. If fails >10 times, alert ops (manual intervention). (c) If Payment fails: don't proceed to Email or Status. Return error to user: "Payment declined. Retry with different card." (6) Idempotency: every step must be idempotent. Payment: same idempotency_key = no double charge. Email: same email address = only one receipt (idempotent). Status: same subscription_id = only one active status. (7) Monitoring: track workflow state for each subscription. Alert if any subscription stuck in "pending" state >1 hour (indicates failure). Expected outcome: exactly-once billing (no double charges), eventually-consistent email delivery, high availability (partial failures don't crash system). Trade: complexity increases (need compensation logic, retry handling). Cost: extra monitoring infrastructure ($200/month).
Follow-up: Your Email retry job retries sending receipt for 7 days. Customer's email address is wrong. Email never sends. After 7 days, retry stops. Customer never receives receipt but is charged. They demand refund, claiming they were never notified. How do you handle this?