Your infrastructure team reports that one Redis Cluster node is consuming 3x the memory of others. You run CLUSTER NODES and see uneven hash slot distribution. A recent resharding operation failed midway. How do you investigate what went wrong and recover?
This indicates incomplete resharding state. First, check cluster status with CLUSTER INFO to see if resharding is in progress: if importing/migrating slots show active migration, the operation is stalled. Use CLUSTER NODES to list slots by node and identify the imbalance. If migration is stuck, force completion with CLUSTER BUMPEPOCH on the source node or restart the target node. For immediate relief, manually migrate the heaviest slots using redis-cli --cluster reshard with --pipelined-fix to complete partial transfers. Always verify with MEMORY STATS and DBSIZE before and after. Prevention: use --timeout 300000 with --cluster-replica-validity-factor set properly in redis.conf to avoid mid-operation disconnections.
Follow-up: If resharding created orphaned slots (slots mapped to nodes that no longer hold data), how would you detect and fix this without data loss?
You deploy Redis Cluster with 6 nodes (3 primaries, 3 replicas). After a few weeks, your application logs show MOVED and ASK redirects increasing by 10% daily. CLUSTER SLOTS shows 4096 slots still unmapped. What's the likely root cause and how do you fix it?
Unmapped slots mean hash slot ranges are orphaned—likely from incomplete initial cluster creation or slot migration corruption. The increasing MOVED errors indicate clients are hitting slots with no owner. Run CLUSTER NODES and look for slots listed as "?" or verify total slots across all nodes sum to 16384. If gaps exist, use redis-cli --cluster check to diagnose. To fix: (1) ensure all 16384 slots are assigned with CLUSTER ADDSLOTS or (2) if corruption is severe, run CLUSTER RESET and reinitialize cluster topology. For immediate stability, update client connection pooling to cache slot maps and retry on MOVED/ASK. Production prevention: run CLUSTER INFO monthly to verify cluster_state:ok and all slots assigned.
Follow-up: How do you handle hash slot resharding during peak traffic without dropping requests?
Your Redis Cluster spans 3 availability zones. A network partition isolates Zone 1 (2 nodes with 3 hash slots each). Your application sees 50% of SET commands failing with CLUSTERDOWN or MOVED errors for those slots. What's the cluster's failover behavior and how do you prevent this?
With a network partition, isolated nodes enter fail state—CLUSTER INFO shows cluster_state:fail. If the isolated partition has fewer nodes than the majority, those slots stop accepting writes (safe but causes downtime). If both a primary and replica are isolated, Sentinel won't failover within the partition because they can't reach the majority. To resume: (1) fix the network partition (primary solution), or (2) run CLUSTER FORGET on partitioned nodes from healthy nodes to remove them, then CLUSTER MEET to re-add them. Prevent with: configure cluster-node-timeout 15000 (aggressive failure detection), deploy CLUSTER_NODE_MONITOR to track node health, and use zone-aware resharding to pair replicas in different zones. Also set cluster-require-full-coverage no in redis.conf to serve other slots even if some are down.
Follow-up: If you must serve requests during a partial partition, what consistency guarantees are lost?
You're scaling from 3 to 6 cluster nodes. After running resharding, client code breaks with a 10% spike in CacheMiss exceptions. SLOWLOG shows GET commands taking 100ms+ (normally <5ms). Cluster topology is correct. What's happening?
During resharding, hash slot migration is in flight. Clients hitting migrating slots receive ASK redirect (not MOVED)—they must retry the exact same command on the target node. If client libraries don't handle ASK correctly (treating it like MOVED), they'll either fail or incur the redirect overhead repeatedly. Check CLUSTER NODES and look for nodes with [importing] or [migrating] tags. To verify: run SLOWLOG GET 10 and look for ASKING commands—if absent, the client isn't handling ASK. Fix: ensure client uses a Redis client with proper ASK redirect handling (go-redis, ioredis, jedis all support this). Immediate mitigation: pause new resharding operations and let migration complete, or reduce pipeline-size in resharding to decrease lock contention. Verify with redis-cli --cluster check --verbose to see live migration progress.
Follow-up: Explain the difference between MOVED and ASK redirects and why clients must handle them differently.
Your cluster has a cold shard (slot range 10000-10923) that's 100GB while others are 10GB. Users report performance degradation tied to operations on that shard. You want to split the cold shard into 4 new nodes. How do you plan and execute this without downtime?
Use redis-cli --cluster reshard with granular control. First, add 4 new nodes and initialize them with redis-cli --cluster add-node. Then use CLUSTER NODES to identify the overloaded primary and target the problematic slot range. Create a reshard plan: divide the 924-slot range (~231 slots per new node). Use redis-cli --cluster reshard --from
Follow-up: How would you handle a situation where one slot migration consistently times out after 10 retries?
You misconfigured cluster-node-timeout to 30000ms (should be 15000ms). Nodes are now declaring each other failed during normal latency spikes (50ms GC pauses). This causes unnecessary slot migration and cluster instability. How do you fix this without taking down the cluster?
Use CONFIG GET/CONFIG SET to update the timeout on all nodes live: redis-cli CONFIG SET cluster-node-timeout 15000. Verify with CONFIG GET cluster-node-timeout on each node. Then force a cluster heal by running CLUSTER RESET SOFT on all replicas (not primaries) to reset their node timeout tracking. Monitor CLUSTER INFO to ensure cluster_state transitions back to ok. Watch for pfail count with INFO cluster: if pfail_count is 0 and cluster_state:ok, the fix worked. If failovers already happened (slots migrated), run CLUSTER NODES to identify which nodes changed roles, then verify data consistency by comparing INFO replication on the original primary vs new primary. Long term: use redis.conf to set cluster-node-timeout once, not dynamic CONFIG SET, to prevent config drift. Also set cluster-replica-validity-factor 10 to prevent premature failover during network hiccups.
Follow-up: If this misconfiguration caused actual data to be reshard during a failover, how would you verify no data was lost?
Your cluster has 8 nodes (4 primary, 4 replica). One primary becomes slow (1000ms+ per command) but CLUSTER NODES still marks it as active. SLOWLOG on that node shows no slow commands. The latency is in network, not processing. However, client ASK redirects are flooding logs. What's the issue?
Network delay on the slow primary is causing cluster state messages to arrive late, resulting in Sentinel seeing it as potentially failed. During this window, Sentinel may move slots via resharding, triggering ASK redirects. This is cluster gossip lag. Check INFO cluster for cluster_state and compare timestamps with time on other nodes (use redis-cli INFO replication on all nodes, look for uptime). If the slow node's time is out of sync, resync via NTP immediately. Meanwhile, run CLUSTER NODES on the slow node and on a fast node—they should have identical slot ownership within 100ms. If they diverge, the slow node is stale and not receiving cluster updates. Temporary fix: reduce client timeout from default 5s to 500ms and enable auto-retry on slow nodes. Permanent: investigate why the node is slow (disk I/O, CPU, memory pressure) using top, iostat, and redis-cli INFO stats. If slow is persistent and not a network blip, failover to replica immediately: run CLUSTER FAILOVER --TAKEOVER on the replica or rely on Sentinel to do so.
Follow-up: How would you distinguish between a node with slow network versus slow disk I/O, and what's the remediation for each?
During a cluster add-node operation, you accidentally specify the wrong --cluster-replica flag. Now the new node claims 1000 slots as a primary, but your application expects it to be a replica of an existing primary. Resharding is in progress and partially complete. How do you safely undo this without data loss?
This is dangerous because data ownership is now split. First, STOP all clients to prevent writes during recovery. Run CLUSTER NODES and document the current slot distribution—screenshot or export to JSON. Identify which slots were successfully migrated vs which are still in transition (look for [importing] and [migrating] tags). If migration is <50% complete, run CLUSTER RESET on the accidentally-added node to purge its slots, then remove it with redis-cli --cluster del-node
Follow-up: If the resharding was completely botched and you lost track of which slots belong where, how would you rebuild the cluster from scratch without losing data?