Your app sends 100K SET commands to Redis, one at a time, without pipelining. Each round-trip takes 1ms (network RTT). Total time: 100 seconds. You're using a simple redis-cli or standard client library. Adding pipelining could reduce this to 1 second. But you're concerned: if you pipeline all 100K commands and one fails, do you lose the others?
Pipelining doesn't batch atomically—all commands are queued, and if one fails, others still execute. This is actually fine for most use cases (e.g., SET operations are idempotent). If any SET fails due to error (e.g., wrong number of args), that individual command fails; others proceed. To implement: (1) use client-side pipelining: instead of send-receive-repeat, send all 100K SETs, then receive all 100K responses. Example in Python: pipe = redis_client.pipeline() for i in range(100000): pipe.set(f"key{i}", f"val{i}") pipe.execute() (2) in redis-cli: use --pipe flag to enable pipelining. (3) batch size: don't pipeline all 100K at once (may cause memory spike on server). Batch into 1000-command chunks: send 1000, receive, repeat. Performance improvement: (1) reduces round-trip overhead: 100K commands with 1ms RTT = 100 seconds without pipelining. With pipelining in batches of 1000: 100 round-trips * 1ms = 100ms + command processing time (~500ms for 100K SETs) = ~600ms total. (2) monitor with redis-cli --stat to verify throughput improvement. (3) verify correctness: sample-check keys after execution. For safety: (1) use pipeline.watch() if you need atomicity (though this defeats pipelining benefits). (2) implement retry logic: if pipeline fails, retry individual commands to identify which failed. (3) log responses: capture response for each command to detect failures.
Follow-up: If you pipeline 100K commands and one of them causes Redis to crash, what happens to the queued commands?
You're optimizing a batch processing job that reads 10K keys from Redis (GET operations). Currently, you send GET one at a time: 10 seconds total (1ms per RTT). You want to use pipelining, but you also need to process results incrementally (can't load all 10K into memory at once). How do you balance memory usage with throughput?
Full pipelining (all 10K at once) would load all results into memory. Streaming pipelining (batches) balances memory and throughput. Strategy: (1) batch size = balance between memory and RTT. If each GET returns 1KB, batching 100 at a time = 100KB in memory (acceptable). Batching 10K at a time = 10MB (too much). Choose batch size = 100. (2) implement: for batch in chunks(range(10000), 100): pipe = redis_client.pipeline(); for key in batch: pipe.get(key); results = pipe.execute(); process(results) (processes batch, releases memory). (3) streaming alternative: use redis-cli SCAN to iterate keys instead of knowing them beforehand. SCAN doesn't require loading all keys at once. (4) use MGET for bulk gets: redis_client.mget([keys_list]) which pipelined automatically. MGET internally batches and is optimized. Comparison: (1) one-at-a-time: 10000ms. (2) pipelined (100-batch): ~100ms (100 batches * 1ms RTT) + processing. (3) MGET: ~5-10ms (internal batching). For large datasets: (1) measure with redis-benchmark -t get -c 1 -q --keyspace 10000. (2) compare: GET vs MGET vs pipelined GET. (3) choose based on memory constraints. Implementation: use redis_client.pipeline(transaction=False) to reduce overhead (transaction=True adds extra round-trip for WATCH).
Follow-up: If you're pipelining and one command depends on the result of a previous command, can you still use pipelining?
Your application uses redis-benchmark to test throughput. You run: redis-benchmark -c 100 -n 100000 -t set -q. You get 500K QPS. But when you check redis-cli INFO stats > instantaneous_ops_per_sec during the benchmark, it shows 200K QPS. The numbers don't match. Why?
Discrepancy between redis-benchmark reported QPS (500K) and instantaneous_ops_per_sec (200K) suggests the benchmark is measuring differently. Causes: (1) redis-benchmark with -q reports average QPS over entire run. If run is 10 seconds: 100K commands / 0.2 seconds = 500K QPS. But instantaneous_ops_per_sec is sampled during the run and includes read operations (INFO calls), which slow down throughput. (2) pipelining overhead: redis-benchmark uses pipelining (batches commands), which inflates reported QPS. The actual per-command throughput might be lower. (3) network latency: redis-benchmark on same machine as Redis shows higher QPS than real-world (client on remote machine would show lower due to network RTT). To get accurate numbers: (1) measure only write operations: redis-benchmark -t set -q and note QPS. Then during same run, redis-cli INFO stats and look at instantaneous_ops_per_sec. Expect them to be close (within 10%). (2) account for pipelining: redis-benchmark pipelined batch size defaults to 50. Real throughput per-command is lower. (3) measure real-world: use production client library (ioredis, Jedis) with realistic batch sizes and measure 99th percentile latency (not just QPS). For benchmarking: (1) run with single connection: redis-benchmark -c 1 to isolate client overhead. (2) run with multiple connections: redis-benchmark -c 100 to test server under load. (3) compare: single connection should have lower QPS but lower latency. Multiple connections distribute load. Measure: redis-cli --latency during benchmark to see actual network RTT.
Follow-up: If instantaneous_ops_per_sec is consistently lower than redis-benchmark reports, what's the bottleneck?
Your batch job writes 100K records to Redis using MSET: mset(key1, val1, key2, val2, ..., key100000, val100000) in a single command. This is fast (1 RTT), but if the command fails (network error, Redis crash mid-command), you lose all 100K records—no rollback. You want batch safety: either all succeed or all fail. Is MSET atomic?
MSET is atomic from Redis perspective: either all keys are set or none are (if Redis crashes during MSET). However: (1) network-level failures: if network dies before MSET reaches Redis, none are set (safe). If Redis crashes during MSET, partial writes might exist on disk (if persistence is enabled), but recovery re-executes MSET (all or none). (2) Redis doesn't rollback MSET if one SET fails (shouldn't happen for MSET though). Use Lua script for true transactional semantics: EVAL 'for i=1, #ARGV/2 do redis.call("SET", ARGV[2*i-1], ARGV[2*i]) end' 0
Follow-up: If you need atomic all-or-nothing semantics for 100K SET operations and can't use Lua (too slow), what's an alternative architecture?
Your application maintains a queue using RPUSH (write) and LPOP (read). You have 100 writers and 10 readers. Writers batch 100 messages per RPUSH (using pipelined RPUSH for throughput). Readers batch 100 messages per LPOP to avoid RTT overhead. But you see inconsistent queue length (LLEN fluctuates erratically). Is there a race condition?
No race condition (Redis is single-threaded), but the observable queue length fluctuates because: (1) RPUSH and LPOP happen concurrently (at different times). LLEN snapshot at moment T1 shows X messages, but at T2 (microseconds later), more writes happened. (2) if you're pipelining writers (batching 100 RPUSHes), they're buffered on client-side, not in Redis yet. LLEN returns Redis's queue (before pipeline flushes). (3) writers and readers have different batch sizes (100 messages = different RTTs for each). This creates observational variance. However, the queue is internally consistent. To verify: (1) LLEN + LRANGE to see actual queue state: redis-cli LLEN myqueue LRANGE myqueue 0 -1. (2) monitor queue depth over time: run LLEN every second and plot. Expected: (roughly monotonic growth if writers > readers, or monotonic decrease if readers > writers). Erratic fluctuation might indicate: (a) readers are faster than writers (queue empties), then writers catch up (queue fills). (b) batch size mismatch. (3) if you need accurate metrics: use INFO stats > list_total_items which tracks approximate list size across all lists. For consistency: (1) use XREAD with consumer groups instead of LPOP (Streams are better for queues). (2) measure throughput: writers QPS * (batches completed), readers QPS * (batches drained). If writers_qps > readers_qps, queue grows. Verify: run for 1 minute, measure LLEN at start and end, compare with (writers_qps - readers_qps) * 60 seconds.
Follow-up: If you need guaranteed in-order processing from the queue, would pipelining affect this guarantee?
You're pipelining INCR commands in a batch of 1000. Before sending the pipeline, Redis crashes and restarts. The pipeline is lost. After restart, the counter resumes from before the crash. Have you lost counts?
Yes, you've lost 1000 increments. The pipeline was buffered on client-side, never reached Redis. After restart, Redis doesn't know about them. This is the cost of pipelining without persistence: temporary data loss. Prevention: (1) enable persistence: CONFIG SET appendonly yes (AOF) or BGSAVE (RDB). Before pipelining critical counters, ensure AOF is flushed. (2) use ACK pattern: after each pipelined batch, verify Redis ack'd all commands before sending next batch. Example: send INCR batch, then send PING (which blocks until all prior commands complete). If PING returns error, retry batch. (3) for critical counters: don't pipeline. Use individual INCRs with write confirmation (slower but safer). (4) use Streams instead: XADD has built-in durability guarantees. Each add is immediately persisted. (5) batch with timeout: pipeline but set short timeout. If timeout, retry. This ensures no huge batches are lost. Implementation: (1) with persistence: EVAL 'for i=1, 1000 do redis.call("INCR", KEYS[i]) end; redis.call("BGSAVE"); return 1' 1000 key1 key2 ... (forces BGSAVE after batch). (2) with ACK: pipe = redis_client.pipeline(); for i in range(1000): pipe.incr("counter"); pipe.ping(); results = pipe.execute(); if not results[-1] == "PONG": retry_batch() (3) measure latency: redis-cli --latency during pipelining to detect if Redis is responding.
Follow-up: If you must pipeline for performance but also need durability, how would you achieve both?
Your Redis client library (e.g., ioredis) automatically pipelined multiple commands if you call them without awaiting. This is performance-transparent pipelining. But you're concerned about predictability: will commands execute in order? If I LPUSH then LPOP immediately after (without awaiting), is order guaranteed?
Yes, order is guaranteed. Auto-pipelining queues commands in the order sent and executes them on Redis in that order. Redis is single-threaded, so even if pipelined as separate commands, they execute sequentially. Example: client.lpush("mylist", "a") client.lpop("mylist") (both pipelined) will execute as: (1) LPUSH mylist a, (2) LPOP mylist. Result: LPOP returns "a". Order is deterministic. However, if you do: client.lpush("mylist", "a").then(() => client.lpop("mylist")) (chained promises), the LPOP is NOT pipelined (waits for LPUSH response). This defeats pipelining but ensures order. Auto-pipelining trade-off: (1) faster (reduced RTT), but (2) less transparent (harder to reason about when commands execute). Prevention: (1) understand your client library's pipelining behavior. ioredis docs explain auto-pipelining. (2) if you need explicit ordering, use Lua scripts (atomic) or transactions (MULTI/EXEC, but slower). (3) for fire-and-forget, auto-pipelining is fine. For critical operations, await after each command to ensure order. Implementation: test with redis-cli MONITOR to observe actual command order during auto-pipelined client calls. Verify: client.lpush() followed by client.lpop() shows both commands in MONITOR output in order. For production: measure latency with and without auto-pipelining. If latency improves >50%, keep it. Otherwise, disable auto-pipelining for clarity (use explicit pipeline() API).
Follow-up: If you rely on auto-pipelining for performance and later add error handling (retry on failure), how would you preserve order?