Your Redis stores 10M hashes, each with ~100 fields (user profiles). MEMORY STATS shows total used_memory is 50GB. You calculate expected: 10M hashes * 100 fields * ~100 bytes per field = 100GB. Redis is using only half! Why is Redis encoding more efficiently than expected?
Redis uses encoding optimizations to reduce memory. For hashes: (1) ziplist encoding (Redis < 7.0) stores hash fields/values inline as a doubly-linked list, very compact for small hashes (<512 bytes). Uses ~10 bytes overhead per field (vs 50+ bytes for dict entry). (2) hashtable encoding (for large hashes) uses dict, more overhead but faster lookups. (3) compression: if hash values are repetitive (e.g., many users from same country), Redis might use compression internally. (4) string encoding: Redis stores strings efficiently—short strings (<44 bytes by default) use embstr encoding (value inline with key object), not separate allocation. To verify encoding: redis-cli OBJECT ENCODING
Follow-up: If you change hash-max-ziplist-entries to a lower value, would existing hashes in memory immediately change encoding?
You store timestamps in Redis as integer fields. Some timestamps are stored as UNIX time (seconds since 1970, ~10 digits). Others are milliseconds (~13 digits). Integer encoding efficiency varies: 10-digit integers are encoded more compactly than 13-digit. Your memory usage is inconsistent. Should you standardize to seconds or milliseconds?
Integer encoding depends on value range: Redis can store integers 0-2^63-1. Encoding overhead is same for small integers (0-9 bytes) and large integers (up to 8 bytes). So 10-digit (seconds) vs 13-digit (milliseconds) both fit in 64-bit, same storage. However: (1) Redis stores integers inline in certain data structures (ziplist, intset) if value is representable as int64. Above ~9 billion, it's still int64 but stored inline (no extra pointer). (2) in hashes with ziplist: field names "timestamp" are stored as strings, values can be int or string depending on ziplist config. If you store "1609459200" (seconds), Redis encodes as int. If you store "1609459200123" (milliseconds), also encodes as int. Same size. Trade-off: (1) seconds: lose sub-second precision, faster comparisons (smaller numbers), but not suitable for high-frequency events. (2) milliseconds: gain precision, slightly slower comparisons (bigger numbers), but usually negligible. Recommendation: use milliseconds (13 digits) consistently. Memory impact is zero (both fit in int64). For consistency: (1) verify encoding with OBJECT ENCODING on sample keys. (2) measure MEMORY USAGE on keys with seconds vs milliseconds—should be identical. (3) standardize in code: always use milliseconds. If you have mixed data, migrate old data: SCAN and HGET each hash, convert seconds to milliseconds, HSET back. (4) test with redis-benchmark to ensure no latency impact from using milliseconds (there shouldn't be).
Follow-up: If you store timestamps as strings instead of integers, how would memory usage change?
You use a Set to track active user IDs. Each user ID is a 128-bit UUID (hex string, 32 characters). You add 10M UUIDs. MEMORY STATS shows 2GB. You calculate: 10M UUIDs * 32 bytes (per UUID string) + set overhead = ~320MB + overhead. Why is it using 2GB (6x more)?
Set members are stored using intset (for small integers) or hashtable (for strings). For UUIDs (strings): (1) hashtable encoding: each UUID is a key in internal hash table. Hash table has collision buckets, pointers, load factor overhead. At 10M members, load factor ~1.5 means ~15M buckets allocated. Each bucket entry = 24 bytes (key ptr + value ptr + hash) * 15M = 360MB, plus UUID string storage = 320MB. Total ~680MB. But also: (2) Redis malloc fragmentation: actual memory used is higher due to allocator fragmentation. Over time, freed memory isn't compacted, causing 2-3x overhead. (3) Redis internal memory overhead: each string object has overhead (refcount, encoding, lru bits) ~32 bytes per object. 10M UUIDs * 32 bytes overhead = 320MB added. Total: 320MB (UUIDs) + 360MB (hash table) + 320MB (object overhead) + 400MB (fragmentation) = ~1.4GB (close to 2GB). To reduce: (1) use MEMORY PURGE to defragment: CONFIG SET activerehashing yes and MEMORY PURGE (Redis 4.0+). This rewrites the set, removing fragmentation. (2) compress UUIDs: instead of 32-char strings, use binary representation (16 bytes). Requires custom encoding/decoding. (3) use different data structure: if you need UUID membership only, use Bloom filter (Redis module) instead of set. Much more memory-efficient. (4) split into multiple sets: if loading all 10M at once, split into shards (set:1, set:2, ..., set:100). Each shard has less fragmentation overhead. Verify: run MEMORY DOCTOR before and after MEMORY PURGE to see savings. Test compression impact on latency (encoding/decoding overhead).
Follow-up: If you can't reduce memory by optimization, how would you detect when a set is approaching memory limit and trigger cleanup?
You store user preferences as a Sorted Set (score = preference strength, member = feature-name). Each user has ~50 features. After adding data for 1M users, you notice memory usage is unexpectedly high. You use OBJECT ENCODING on a sample zset and see "hashtable", not "skiplist" (which you'd expect). Why is it using hashtable?
Sorted sets can use two encodings: (1) ziplist (small zsets): stores entries inline as list, very compact. Used if zset size < CONFIG GET zset-max-ziplist-entries (default 128 entries) AND all values < CONFIG GET zset-max-ziplist-value (default 64 bytes). (2) skiplist (large zsets): uses skiplist + dict for fast lookups and range queries. Default if zset > 128 entries. However: (1) if you have 50 features per user and 1M users, your single zset would be 50M entries (one zset per user, or one giant zset with member="user:feature"). At 50 entries per user zset, each still uses ziplist (below 128 threshold). (2) if you're storing all 1M users in one giant zset, it's definitely skiplist. Encoding depends on data structure, not just size. (3) "hashtable" encoding suggests you might have a misconfiguration or the zset was encoded as hashtable for some reason (shouldn't normally happen in standard Redis). To verify: (1) ZCARD user-prefs:user-123 to get zset size. If <128, should be ziplist. If >128, should be skiplist. (2) run OBJECT ENCODING on multiple keys and log results. If many show "hashtable" unexpectedly, check Redis version (older versions had different encodings). (3) memory optimization: if all zsets are small (50 entries), they should be memory-efficient with ziplist. Measure: MEMORY USAGE user-prefs:user-123 on sample key, multiply by 1M. If close to expected (50 * 8 bytes score + member name), ziplist is working. If much higher, skiplist/fragmentation is the issue. To reduce: (1) lower zset-max-ziplist-entries to 64 (more zsets use ziplist, slower for large operations). (2) use MEMORY PURGE to defragment. (3) verify actual encoding: redis-cli --eval zset-encoding-check.lua 0 (script to batch-check encodings).
Follow-up: If you have 1M small zsets (50 entries each), what's the optimal memory layout to minimize overhead?
You use HyperLogLog to track unique visitors: PFADD visitors:2024-01-01
HyperLogLog is designed for memory efficiency—it's a probabilistic data structure that uses only ~14KB to estimate cardinality of 1B+ items with 0.81% error. Fixed memory regardless of cardinality. Each HLL in Redis: (1) uses 14KB of memory (16384 bytes for the HLL state array, in 64-bit integers). This is constant, whether tracking 100 or 1B items. (2) PFADD updates the HLL by hashing the input and updating relevant bits. No additional storage per item. (3) PFMERGE combines HLLs by taking bitwise OR of all internal states. Result is also 14KB (not sum). This explains why all 365 HLLs + merged HLL all use ~12KB. To verify: MEMORY USAGE visitors:2024-01-01 should return ~12KB per HLL. Also MEMORY USAGE all:visitors should be ~12KB. Trade-off: (1) HyperLogLog gives approximate cardinality, not exact count. Error rate ~0.81%. If you need exact unique count, use Set (which grows with cardinality). (2) can't enumerate items in HLL (only cardinality). If you need to also enumerate, use Set alongside HLL (Set for enumeration, HLL for cardinality estimate to save memory). Performance: (1) PFADD is O(1), very fast. (2) PFMERGE is O(N) where N = number of HLLs merged, but still very fast (just bitwise OR). For your use case: (1) 365 HLLs * 12KB = 4.38MB (tiny!). With Set, would be ~365MB (1M users * 365 days * size per user * overlap). HLL is 100x more efficient. (2) test accuracy: run PFCOUNT and manually verify against database for sample days.
Follow-up: If you need exact cardinality with HyperLogLog accuracy, how would you combine it with another data structure?
You store frequently-accessed strings (page HTML, JSON responses) in Redis. Each is 1-50KB. You have 100K such strings = ~2.5GB total. You're concerned about memory fragmentation causing actual usage to be 5-7GB. Can you reduce fragmentation without rewriting entire cache?
Memory fragmentation (used_memory_rss / used_memory > 1.5) occurs due to: (1) allocator fragmentation: when Redis deletes keys, freed memory isn't always reused for new keys (depends on allocator). Over time, gaps accumulate. (2) large key evictions: if you delete a 50KB key and insert a 1KB key, 49KB is wasted. (3) RDB snapshots: fork process creates copy-on-write (COW) pages, temporarily doubling memory. After BGSAVE, memory should normalize, but fragmentation might persist. To reduce without rewriting: (1) MEMORY PURGE: Redis 4.0+ command that triggers allocator optimization. CONFIG SET allkeys-lru and run MEMORY PURGE to reclaim fragmented memory. (2) incremental rewriting: instead of rewriting all 100K strings, gradually remove and re-add high-fragmentation keys. Use SCAN: for each key, check OBJECT REFCOUNT (if 1, it's unused). Re-add to trigger reallocation: DUMP key, DELETE key, RESTORE key 0
Follow-up: If fragmentation persists even after MEMORY PURGE, what's the next step?
You're tuning Redis for large value sizes (100KB+ per key). You notice that for keys around 64 bytes, they use embstr encoding (string embedded in key object). For keys >64 bytes, they use raw encoding (separate allocation). Should you configure different thresholds for your 100KB keys?
Embstr encoding (embedded string) is only efficient for small strings (<44 bytes in Redis 3.2+, <64 bytes in older versions). For strings >64 bytes, Redis automatically uses raw encoding (separate allocation). For your 100KB keys: (1) raw encoding is optimal. No way to embed 100KB in key object. (2) embstr/raw threshold CONFIG is set at compile time, not runtime. Changing it requires recompiling Redis. Not practical for tuning. (3) for 100KB keys, optimization should focus on: (a) network overhead: 100KB takes 100ms at 10Mbps network. Use pipelining to amortize overhead. (b) memory fragmentation: large keys cause fragmentation. Use MEMORY PURGE periodically. (c) compression: compress 100KB values before storing. Example: gzip(json) might compress 100KB to 10KB, reducing storage 10x. Trade-off: compression/decompression CPU cost vs memory savings. For your scenario: (1) measure actual encoding: OBJECT ENCODING mykey should return "raw" for 100KB keys. (2) measure performance: MEMORY USAGE mykey * (number of 100KB keys) = expected memory. (3) for compression: benchmark gzip impact. If 100KB takes 1ms to compress and saves 90KB storage, worth it if you read frequently. If read infrequently, skip compression. (4) monitor: run INFO memory and watch for fragmentation. If used_memory_rss > used_memory * 1.5, run MEMORY PURGE. Test: create 1000 100KB keys, measure used_memory_rss, delete half, measure again, run MEMORY PURGE, measure final. Expected: should approach initial memory usage after PURGE.
Follow-up: If you compress 100KB JSON to 10KB, how would you handle versioning if compression algorithm changes?