Fastly KV Store Concurrent Write Race Condition

Problem

When multiple concurrent requests perform read-modify-write operations on the same Fastly KV Store key, writes can fail with intermittent 500 errors. This commonly occurs during batch uploads where each request updates a shared list or counter.

Context / Trigger Conditions

• Error message contains "Failed to store list:" or similar KV write failure
• ~10-20% failure rate during concurrent operations
• Multiple requests from the same user/session updating shared state
• Pattern: read key → modify data → write key without any locking

Example error:

{"error":"Failed to store list:"}

Solution

Add a retry loop that re-reads the data before each write attempt:

/// Add item to list with retry for concurrent writes
pub fn add_to_list(key: &str, item: &str) -> Result<()> {
    // Retry up to 5 times for concurrent write conflicts
    for attempt in 0..5 {
        // Re-read current state on each attempt
        let mut items = get_list(key)?;

        if items.contains(&item.to_string()) {
            return Ok(()); // Already exists, done
        }

        items.push(item.to_string());

        match put_list(key, &items) {
            Ok(()) => return Ok(()),
            Err(e) if attempt < 4 => {
                eprintln!("[KV] Retry {} for list update: {}", attempt + 1, e);
                // Re-read picks up concurrent writes
                continue;
            }
            Err(e) => return Err(e),
        }
    }

    Err(Error::new("Max retries exceeded for list update"))
}

Key points:

1. Re-read on each retry - The fresh read picks up changes from concurrent writes
2. Check for duplicates - Avoid adding the same item twice
3. Log retries - Helps debug if issues persist
4. Limit retries - 5 attempts is usually sufficient

Verification

• Batch operations that previously failed ~16% should now succeed ~100%
• Retry log messages ([KV] Retry N for...) indicate the mechanism is working
• If still failing after 5 retries, there may be a deeper issue

Example

Before (race condition):

pub fn add_to_user_list(pubkey: &str, hash: &str) -> Result<()> {
    let mut hashes = get_user_blobs(pubkey)?;  // Read
    if !hashes.contains(&hash) {
        hashes.push(hash.to_string());         // Modify
        put_user_list(pubkey, &hashes)?;       // Write - CONFLICT!
    }
    Ok(())
}

After (with retry):

pub fn add_to_user_list(pubkey: &str, hash: &str) -> Result<()> {
    for attempt in 0..5 {
        let mut hashes = get_user_blobs(pubkey)?;
        if hashes.contains(&hash) {
            return Ok(());
        }
        hashes.push(hash.to_string());
        match put_user_list(pubkey, &hashes) {
            Ok(()) => return Ok(()),
            Err(e) if attempt < 4 => continue,
            Err(e) => return Err(e),
        }
    }
    Err(Error::new("Max retries exceeded"))
}

Notes

• Fastly Compute doesn't have sleep(), so retries happen immediately
• The re-read is what provides the "delay" by picking up concurrent changes
• Consider using atomic operations if available (Fastly KV doesn't support CAS)
• For high-contention scenarios, consider sharding the key space
• This pattern also applies to remove operations (read-filter-write)

References

• Fastly KV Store Documentation