Redis (redis-py v8.0)

Redis - in-memory data structure store, used as database, cache, and message broker.

redis-py v8.0 changes:

• Python 3.10+ required
• RESP3 is the default protocol (was RESP2 in 7.x)
• Async API: redis.asyncio module (not .asyncio property)

Installation

pip install redis

Basic Usage

Synchronous Client

import redis

# Default protocol is now RESP3
r = redis.Redis(host='localhost', port=6379, db=0)

# Basic commands
r.set('foo', 'bar')
value = r.get('foo')  # b'bar'

# Close connection
r.close()

Async Client

import redis.asyncio as redis

async def example():
    # Default protocol is RESP3
    r = redis.Redis(host='localhost', port=6379, db=0)
    
    await r.set('foo', 'bar')
    value = await r.get('foo')  # b'bar'
    
    # Close connection
    await r.aclose()

Migration from v7.x:

# OLD (v7.x and earlier)
from redis import Redis
r = Redis()
client = r.asyncio  # Property

# NEW (v8.0+)
import redis.asyncio as redis
r = redis.Redis()

Connection Pool

# Create a connection pool
pool = redis.ConnectionPool(
    host='localhost',
    port=6379,
    db=0,
    max_connections=50,
    decode_responses=True,
    socket_connect_timeout=5,
    socket_timeout=5,
    retry_on_timeout=True,
)

# Use pool with client
r = redis.Redis(connection_pool=pool)

Connection Pool with Password

pool = redis.ConnectionPool(
    host='localhost',
    port=6379,
    password='your_password',
    max_connections=100,
)
r = redis.Redis(connection_pool=pool)

Core Commands

# Strings
r.set('key', 'value', ex=300)
r.setnx('key', 'value')
r.mset({'key1': 'value1'})
r.get('key')
r.mget(['key1', 'key2'])
r.delete('key')
r.exists('key')
r.ttl('key')
r.expire('key', 300)

# Hashes
r.hset('user:1', mapping={'name': 'Alex', 'email': 'a@b.com'})
r.hget('user:1', 'name')
r.hgetall('user:1')
r.hexists('user:1', 'name')
r.hkeys('user:1')
r.hdel('user:1', 'name')

# Lists
r.lpush('mylist', 'item1', 'item2')
r.rpush('mylist', 'item3')
r.lpop('mylist')
r.rpop('mylist')
r.lrange('mylist', 0, -1)
r.llen('mylist')

# Sets
r.sadd('myset', 'item1', 'item2')
r.sismember('myset', 'item1')
r.smembers('myset')
r.srem('myset', 'item1')
r.sunion('set1', 'set2')
r.sinter('set1', 'set2')

# Sorted Sets
r.zadd('scores', {'player1': 100, 'player2': 200})
r.zrange('scores', 0, -1, withscores=True)
r.zrevrange('scores', 0, 9)
r.zrem('scores', 'player1')
r.zscore('scores', 'player1')

RESP3 Protocol

RESP3 (Redis Serialization Protocol version 3) is the default in redis-py v8.0. It introduces push messages, better type support, and client-side caching.

Negotiating RESP3

# Explicitly request RESP3 (default in v8.0)
r = redis.Redis(protocol=3)

# Check negotiated protocol
info = r.info('server')
print(info['redis_version'])

# Verify protocol version
client_info = r.client_info()
# Look for 'proto' field in response

RESP3 Features

Push Messages:

# Enable push message handling
r = redis.Redis(protocol=3, push_response_callback=lambda msg: print(msg))

# Client-side caching with CLIENT TRACKING
r.execute_command('CLIENT TRACKING', 'ON', 'REDIRECT', r.connection_pool.get_connection('_').pid)

New Commands/Options:

# GETDEL - atomic get and delete (RESP3-friendly pattern)
value = r.execute_command('GETDEL', 'key')

# EXPIRE with NX/XX/GT/LT options
r.expire('key', 300, nx=True)  # Only set if not exists
r.expire('key', 300, xx=True)  # Only set if exists

# OBJECT introspection
memory_usage = r.object_encoding('key')
idletime = r.object_idletime('key')

# COMMAND introspection
commands = r.command()

Operational Caution

Proxy/Managed Service Downgrade: A proxy (Twemproxy, Codis) or managed service in front of Redis may downgrade to RESP2 and silently remove RESP3 capabilities. Always verify the negotiated protocol before relying on RESP3-only behavior.

Check before using RESP3 features:

def verify_resp3_support(client):
    """Verify RESP3 is actually available."""
    try:
        # Try a RESP3-specific command
        client.execute_command('HELLO', 3)
        return True
    except Exception:
        # Downgraded to RESP2
        return False

Source: RESP3 Protocol Specification

Redis Stack Modules

Redis Search

r.ft().create_index((TextField('name'), NumericField('age'), TagField('tags')))
results = r.ft().search('@name:alex @age:[20 30]')
r.ft().dropindex()

Redis JSON

from redis.commands.json.path import Path

r.json().set('/user:1', Path.root_path(), {'name': 'Alex', 'age': 30})
r.json().get('/user:1')
r.json().get('/user:1', Path('$.name'))
r.json().merge('/user:1', Path.root_path(), {'age': 31})
r.json().del_('/user:1', Path('$.age'))

Redis TimeSeries

r.ts().create('sensor:1')
r.ts().add('sensor:1', '*', 25.3)
r.ts().range('sensor:1', 0, '*')
aggregated = r.ts().range('sensor:1', 0, '*', agg_type='avg', bucket_size=60000)

Pipelines

# Basic pipeline
pipe = r.pipeline()
pipe.set('key1', 'value1')
pipe.set('key2', 'value2')
pipe.get('key1')
results = pipe.execute()  # [True, True, b'value1']

# Watch for transactions
pipe = r.pipeline(True)
pipe.watch('mykey')
val = pipe.get('mykey')
pipe.multi()
pipe.set('mykey', val + 1)
pipe.execute()

Rules That Prevent Incidents

These guidelines prevent specific failure modes in production.

TTL Discipline Prevents Unbounded Key Growth

Failure prevented: Memory exhaustion from accumulating keys with no expiration.

Always set TTL on cache keys. Redis is an in-memory store; keys without TTL persist forever and accumulate.

# BAD - key persists forever
r.set('user:session:123', session_data)

# GOOD - key expires after 1 hour
r.set('user:session:123', session_data, ex=3600)

Never Use KEYS/SCAN Patterns on Large Keyspaces

Failure prevented: Single-threaded Redis blocking for seconds during KEYS * on millions of keys.

# BAD - blocks Redis for seconds
keys = r.keys('user:*')

# GOOD - non-blocking scan
cursor = 0
while True:
    cursor, keys = r.scan(cursor, match='user:*', count=100)
    # Process keys
    if cursor == 0:
        break

Pipeline vs Transaction vs Lua for Atomicity

Failure prevented: Race conditions when multiple operations must be atomic.

• Pipeline: Batch commands for single round-trip, not atomic
• Transaction (WATCH/MULTI/EXEC): Optimistic locking, fails if key changed
• Lua script: True atomicity, server-side execution

# Pipeline - fast but NOT atomic
pipe = r.pipeline()
pipe.incr('counter')
pipe.set('flag', 'done')
pipe.execute()  # Another client can interleave

# Transaction - atomic if key unchanged
pipe = r.pipeline(True)
pipe.watch('counter')
pipe.multi()
pipe.incr('counter')
pipe.execute()  # Raises WatchError if counter changed

# Lua - truly atomic
script = """
local current = redis.call('INCR', KEYS[1])
redis.call('SET', KEYS[2], 'done')
return current
"""
r.eval(script, 2, 'counter', 'flag')

Cache-Aside vs Write-Through Failure Behavior

Failure prevented: Stale data after cache invalidation.

• Cache-aside: App loads cache, misses, reads DB, writes cache. On write, invalidate cache. Risk: stale window between invalidate and next read.
• Write-through: App writes cache, cache writes DB. Risk: write latency doubles, cache write failure may lose data.

# Cache-aside pattern
def get_user(user_id):
    key = f'user:{user_id}'
    user = r.get(key)
    if user is None:
        user = db.get_user(user_id)  # Slow path
        r.setex(key, 3600, user)  # Populate cache
    return user

# Invalidate on write
def update_user(user_id, data):
    db.update_user(user_id, data)
    r.delete(f'user:{user_id}')  # Invalidate, don't write

Cache Stampede Prevention

Failure prevented: Thundering herd when popular key expires—thousands of requests hit DB simultaneously.

Use distributed lock or jittered TTL.

# Lock-based refresh
import redis.lock

def get_user_safe(user_id):
    key = f'user:{user_id}'
    user = r.get(key)
    if user is None:
        lock = r.lock(f'lock:{key}', timeout=10)
        if lock.acquire(blocking=True):
            try:
                # Double-check after acquiring lock
                user = r.get(key)
                if user is None:
                    user = db.get_user(user_id)
                    r.setex(key, 3600, user)
            finally:
                lock.release()
        else:
            # Wait and retry
            time.sleep(0.1)
            user = r.get(key) or db.get_user(user_id)
    return user

# Jittered TTL approach
import random
ttl = 3600 + random.randint(-300, 300)  # ±5 min jitter
r.setex(key, ttl, user)

Never Cache Per-Request Volatile Data

Failure prevented: Cross-request data leakage, memory waste from unique keys.

# BAD - each request creates unique key
cache.set(f'token:{request_id}', token, timeout=300)

# GOOD - session-scoped data stays in memory/session store
request.session['token'] = token

Production Failure Modes

Understanding these failure modes helps diagnose production issues.

Connection Exhaustion Under Burst

Symptom: Connection refused or timeout errors during traffic spikes.

Redis uses a single thread for command processing. Too many concurrent connections exhaust file descriptors and cause connection failures.

Fix: Use a connection pooler or enforce bounded concurrency.

# Bounded connection pool
pool = redis.ConnectionPool(
    max_connections=50,  # Cap connections
    socket_connect_timeout=5,
    socket_timeout=5,
    retry_on_timeout=True,
)
r = redis.Redis(connection_pool=pool)

maxmemory Eviction Policy Silently Dropping Keys

Symptom: Cache returns missing keys despite recent writes; data appears to "disappear".

When Redis hits maxmemory, it evicts keys based on the configured policy. The wrong policy turns a cache into a source of data loss.

Check eviction policy:

redis-cli CONFIG GET maxmemory-policy

Common policies:

• noeviction: Returns error when memory full (default for persistence)
• allkeys-lru: Evict least recently used keys (good for cache)
• volatile-lru: Evict LRU keys with TTL set (good for mixed workloads)
• allkeys-lfu: Evict least frequently used keys (Redis 4.0+)

# Set appropriate eviction policy
r.config_set('maxmemory-policy', 'allkeys-lru')

Replication Lag Causing Stale Read After Write

Symptom: Write succeeds, immediate read from replica returns old value.

In replicated setups, writes go to primary, reads may hit replica. Replication lag causes stale reads.

Fix: Read from primary after writes, or use WAIT to confirm replication.

# Wait for replica acknowledgment
result = r.set('key', 'value')
replicas_acked = r.wait(1, timeout=1000)  # Wait for 1 replica
if replicas_acked == 0:
    log.warning("Replication lag detected")

Blocking Commands Freezing Single-Threaded Server

Symptom: All clients experience latency spike; Redis appears frozen.

Redis is single-threaded. Blocking commands like KEYS, SORT, HGETALL on large hashes freeze the server.

Check slowlog:

redis-cli SLOWLOG GET 10

# BAD - blocks server
all_keys = r.keys('*')

# GOOD - use SCAN
cursor = 0
while True:
    cursor, keys = r.scan(cursor, count=100)
    if cursor == 0:
        break

Slowlog as First Diagnostic for Latency

Symptom: Gradual latency increase; occasional spikes.

The slowlog captures commands exceeding a threshold. Inspect it first when diagnosing latency.

# Set slowlog threshold (microseconds)
r.config_set('slowlog-log-slower-than', 10000)  # 10ms

# Get slow queries
slow_queries = r.slowlog_get(10)
for query in slow_queries:
    print(f"Command: {query['command']}")
    print(f"Duration: {query['duration']}µs")
    print(f"Timestamp: {query['timestamp']}")

When Not to Use Redis

Not every workload benefits from Redis. Know when to choose alternatives.

Relational Database Is Enough

Many caching use cases don't need Redis. If your access patterns are simple and data volume fits in database memory, use the database cache.

Use database instead of Redis when:

• Query patterns are straightforward (primary key lookups, indexed searches)
• Data consistency is critical over speed
• You already have connection pooling and query caching in place

Job Queues Belong to a Dedicated Broker

Redis can power job queues, but dedicated brokers (RabbitMQ, SQS, Bull) provide:

• Message acknowledgment and retry
• Dead letter queues
• Priority queues
• Better visibility and monitoring

Use a dedicated broker when:

• Jobs must not be lost
• Complex routing or priority is needed
• You need visibility into queue depth and processing

Caching Pages Nobody Requests

Caching costs memory. If a page is rarely accessed, caching it wastes memory and adds complexity.

Cache only when:

• Hit rate is high (>80%)
• Computation/cache miss is expensive
• Data is read-heavy, write-light

Multi-Region Writes Need Different Consistency

Redis is primarily single-primary. Multi-region active-active setups require different consistency models.

Consider alternatives when:

• You need true multi-region writes with low latency
• Strong consistency across regions is required
• Conflict resolution is complex

Django Integration

Django-specific patterns (django-redis cache backend, invalidation on model save, per-user rate limiting, Celery broker, Channels) live in references/django-integration.md.

Deep Dives

Load these reference files on demand for specific use cases:

• references/django-integration.md — When working with Django projects (cache backend, invalidation, rate limiting, Celery, Channels)

CLI Operations

# Connect
redis-cli

# Check keys (use with caution)
KEYS *

# Delete by pattern (use with caution)
SCAN 0 MATCH user:* COUNT 1000 | xargs redis-cli DEL

# Clear current database
FLUSHDB

# Monitor commands
MONITOR

# Check slowlog
SLOWLOG GET 10

# Check eviction policy
CONFIG GET maxmemory-policy

References

• redis-py Official Documentation
• redis-py v8.0 Release Notes
• Redis Stack Documentation
• RESP3 Protocol Specification