Compacted Topics, Retention, and Storage

Options to free up disk: 1) Reduce retention: lower retention from 7 days to 3 days. Run kafka-configs --bootstrap-server localhost:9092 --entity-type topics --entity-name orders --alter --add-config retention.ms=259200000 (3 days = 259200000ms). Old messages are deleted immediately. Space freed within hours (after log cleanup runs). Risk: consumers with lag > 3 days will miss messages. Verify lag: kafka-consumer-groups --describe --group * shows max lag. If max lag < 3 days, safe to reduce. 2) Enable log compaction: for topics with business state (user profile, account balance), use compaction instead of time-based retention. Set cleanup.policy=compact. Kafka keeps only latest value per key (deduplicates). Space reduced by 10-50x if topic is high-churn (many updates per key). 3) Compress existing data: re-compress log segments. Kafka stores compressed, but if topic has mixed compression, recompression helps. Run broker tool: kafka-log-dirs --bootstrap-server localhost:9092 --describe --topic orders to see log sizes. 4) Archive to S3: stream data to S3 (using Kafka Connect S3 sink). Delete old messages from Kafka, archive stays in S3 (cheaper, slower access). 5) Delete unnecessary topics: kafka-topics --delete --topic old-topic if no longer needed. 6) Add disk: expand broker storage (add SSD, increase volume size). Most reliable but slowest to implement. Recommended approach: 1) Identify high-volume topics (use kafka-topics --describe | sort by size). 2) Check if compaction is applicable (stateful data = compact, event streams = time-based). 3) Apply compaction to 50% of topics (saves 20-30% space). 4) Reduce retention for remaining topics. 5) Monitor: set alert if disk > 80%. Run cleanup proactively before hitting limit.

With log compaction, Kafka keeps only 1 version per key (the latest). Topic has 1M users, each updated weekly = 52M messages over 1 year. After compaction, only 1M messages remain (latest profile for each user). This is the power of compaction: space reduction from 52M to 1M (98% reduction). Implementation: 1) Set topic config: kafka-configs --entity-type topics --entity-name user-profile --alter --add-config cleanup.policy=compact,segment.ms=86400000,min.cleanable.dirty.ratio=0.5. 2) Log compaction happens asynchronously (default every 30s). Broker scans logs, identifies old versions of same key, marks for deletion. 3) New consumers see only latest version. Old consumers (lagging) might see old versions briefly before cleanup deletes them. Important: consumers must be able to tolerate missing intermediate updates. If your app needs all versions (audit trail), use time-based retention instead. Consumer behavior: consumer reads compacted topic from earliest. It sees only latest value for each key. If key was updated 10 times, consumer sees only version 10 (assuming compaction ran). If consumer needs all versions, switch to non-compacted topic and adjust retention. Compaction trade-offs: Pros: space savings (10-100x), great for state topics (profiles, balances). Cons: intermediate updates lost, CPU overhead on broker (cleaner thread runs continuously). For topics with frequent updates and many keys, compaction is ideal. For immutable event streams, use time-based retention.

Consumer will see only the latest version (or fewer if cleanup has progressed). Compaction removes old versions before consumer reads them. Timeline: T=0: User profile v1 written. T=1s: v2 written. ... T=100s: v100 written (100 versions total). T=1day: compaction runs, deletes v1-v99, keeps only v100. T=1day + offset cleanup: broker purges log segments with old versions. T=7days later: lagging consumer resumes reading. It reads the compacted log (only v100 remains). It misses v1-v99 entirely. If consumer needs all versions, it must keep up with topic (lag < compaction interval). For audit trails or reprocessing: use separate non-compacted audit topic that stores all changes. Example: write to user-profile (compacted, latest state) AND user-profile-audit (non-compacted, all versions). Compacted topic is for current state, audit is for history. Compaction parameters control when cleanup happens: 1) cleanup.policy=compact: enables compaction. 2) segment.ms=86400000 (1 day): log segments rotate daily. Only complete segments are compacted (current segment stays). 3) min.cleanable.dirty.ratio=0.5 (50%): if 50% of segment is old versions, trigger cleanup. 4) delete.retention.ms=86400000 (1 day): keep deleted messages for 1 day (tombstone retention). To verify compaction worked: kafka-topics --describe --topic user-profile --bootstrap-server localhost:9092 and compare log size before/after. Should drop 10-100x.

No. Consumer's last offset was deleted after 7 days. When consumer rejoins group, it looks for last committed offset. If offset is older than 7-day retention, broker has no data (deleted). Consumer gets error: OffsetOutOfRangeException. Behavior depends on consumer config: 1) auto.offset.reset=earliest: consumer resets to earliest available offset (data from 10 days ago is gone, starts from 7 days ago). Some messages missed. 2) auto.offset.reset=latest: consumer resets to latest offset (skips all messages while offline). 3) auto.offset.reset=none: consumer throws exception and stops (safest, requires manual intervention). Recommended: set auto.offset.reset=none for critical data (payments, orders). When consumer crashes, page on-call to investigate instead of silently skipping messages. Prevention: 1) Increase retention to cover max expected downtime. If consumer can be offline 10 days, set retention to 14+ days. Trade-off: more disk usage. 2) Use log compaction (if applicable). Compacted topics have no time limit—data is kept until explicitly deleted (tombstone). Consumer can catch up after any downtime. 3) Enable offset cleanup policy: set offsets.retention.minutes=10080 (7 days default). This keeps offset commit for 7 days, longer than message retention if needed. Example: if message retention is 3 days and consumer is offline 5 days, offset is deleted too. Increasing offset retention helps. 4) Monitor consumer lag: alert if lag > 90% of retention window. Example: if retention is 7 days, alert if lag > 6 days. Page on-call to fix consumer before data loss. To recover from OffsetOutOfRangeException: manually reset offset: kafka-consumer-groups --group my-group --bootstrap-server localhost:9092 --reset-offsets --to-earliest --execute.

Do both, but in order: 1) Compaction first (low risk). Identify topics with key-value pattern (state data): user profiles, account balances, inventory. These are ideal for compaction. Enable on 300 topics (saves 3TB). Consumers must tolerate losing intermediate updates. Test with 1 topic first. If compaction runs smoothly and space freed, roll out to 300 topics. No consumer-visible impact (they see latest state, which they need anyway). 2) Retention reduction second (after compaction). For remaining 700 topics (event streams, logs), reduce retention from 7 days to 5 days (saves 2TB). Verify consumer lag < 5 days before applying. If any consumer lags > 5 days, keep that topic at 7 days. Phased approach: reduce 1 day at a time (7→6→5 days over 3 weeks). Monitor consumer lag daily. If lag grows close to limit, revert. Monitoring: set alerts: 1) Disk usage > 80%: page on-call. 2) Compaction efficiency: if saved space < 10%, re-evaluate compaction config. 3) Retention violations: if consumer lag > 90% of retention window, page on-call to fix consumer. Estimate impact: 1000 topics, 300 compacted (3TB saved), 700 reduced retention (2TB saved) = 5TB freed (50% of original). This buys 3-4 months before hitting disk limit again. Long-term: 1) Archive old data to S3 (lifecycle policy: move to S3 after 30 days, delete after 1 year). 2) Scale to more brokers/disk (add 10TB). 3) Implement sampling: only keep 1% of high-volume topics (reduce noise).

Compaction requires extra space. During cleanup, broker: 1) Reads old log segments (5TB). 2) Creates cleaned log segments (removes old key versions) in temporary directory. 3) Atomic swap: replaces old segments with cleaned segments. Space needed: old segments (5TB) + cleaned segments (~3TB after compaction) = 8TB. If you have only 1TB free, cleanup will fail: broker runs out of disk mid-cleanup, crashes. Solutions: 1) Free up space first: delete unnecessary data, archive to S3, reduce other topic retention. Free at least 50% of largest topic size before compacting. 2) Compact incrementally: don't compact entire 5TB topic at once. Instead, compact 1 partition at a time. But Kafka compacts all partitions in parallel (configured by num.log.cleaners). Can't easily control per-partition. 3) Add more disk: expand storage before compacting. Best option. 4) Manual segment deletion: delete old segments manually (risky—could lose data). Not recommended. Prevention: before enabling compaction, measure cleanup requirements: 1) Estimate cleanup savings: run kafka-log-cleaner-benchmark on sample data. 2) Calculate space needed: if compaction saves 50%, allocate 1.5x topic size as free space. 3) Gradual rollout: enable compaction on small topics first (100GB). Monitor cleanup process, measure actual space used. Then scale to larger topics. Monitoring: JMX metric kafka.log:type=LogCleaner,name=max-dirty-percent shows how full log segments are before cleanup. If metric grows without cleanup happening, broker is full (cleanup stuck). Alert: if max-dirty-percent > 50% for > 1 hour, escalate.