RFDB Write Lock Edge-Flush Gotcha (RFD-67)

Problem

RFDB server saturates during enricher write storms. A single addEdges call can hold the exclusive write lock (db.engine.write()) for seconds while flushing data to disk, blocking ALL reads and other writes during that time.

Context / Trigger Conditions

• Enrichers (behaviorEnricher, packageApiEnricher) call client.addEdges() repeatedly
• Server was previously used for grafema analyze (analysis nodes still in write buffer)
• System memory > 70% used (very common on dev machines)
• First addEdges call spikes CPU; subsequent calls time out at 60s

Root Cause (code trace)

client.addEdges(10 edges)
  → with_engine_write()               ← acquires EXCLUSIVE write lock
    → engine.add_edges()
      → store.upsert_edges()          ← fast, in-memory (~0.01ms)
      → self.maybe_auto_flush()       ← ← ← THE BUG
        Path 2: memory_pressure > 0.7 && total_write_buffer_nodes() >= 500
          → store.flush_all()         ← DISK I/O (seconds) under write lock!
  → lock released

maybe_auto_flush has two paths:

1. Buffer overflow: any_shard_needs_flush(node_limit, byte_limit) — fine
2. Memory pressure: memory_pressure > 0.7 && total_write_buffer_nodes() >= 500 — BUG

Path 2 fires based on nodes, not edges. After grafema analyze, the write buffer holds analysis nodes. The enricher's first addEdges call triggers a full flush of all those analysis nodes — the edge write causes a massive node flush.

Files involved:

• packages/rfdb-server/src/graph/engine_v2.rs — add_edges(), maybe_auto_flush()
• packages/rfdb-server/src/bin/rfdb_server.rs — with_engine_write() (line ~2289)
• packages/rfdb-server/src/database_manager.rs — Database.engine: RwLock<Box<dyn GraphStore>>

Fix (applied in RFD-67)

Replace maybe_auto_flush() call in add_edges with maybe_auto_flush_edges():

fn maybe_auto_flush_edges(&mut self) {
    // Byte-limit only — never fires on memory pressure / node count.
    // Prevents edge writes from triggering node flushes under write lock.
    if self.store.any_shard_needs_flush(usize::MAX, self.cached_profile.write_buffer_byte_limit) {
        if let Err(e) = self.store.flush_all(&mut self.manifest) {
            tracing::warn!("auto-flush (edges) failed: {}", e);
        }
    }
}

Key invariant: add_nodes keeps maybe_auto_flush() (nodes are 120 bytes, OOM risk is real). add_edges uses maybe_auto_flush_edges() (edges are 98 bytes, byte limit = 100MB default).

Verification

Test: test_edge_writes_do_not_block_reads in engine_v2.rs — wraps engine in RwLock, runs concurrent writer + reader threads, asserts reads complete in < 1s during edge storm.

Enricher Contract After Fix

Enrichers must call Flush explicitly after their write phase:

await enrichBehaviors(client, lookup, { flushBatchSize: 50 });
await client.flush();  // required — edges won't auto-flush during normal write storms

Recommended flushBatchSize: 50–200 edges per batch. Under 50: excessive socket round-trips. Over 200: diminishing returns, larger per-request latency.

Diagnosis Checklist

If RFDB saturates again:

1. RFDB_VERBOSE=1 — log every request; look for AddEdges/Flush taking > 1s
2. GetStats → shardDiagnostics.writeBufferNodes — high = unflushed analysis nodes
3. samply record on rfdb-server PID → look for flush_with_ids holding write lock
4. Check if enricher calls client.flush() after writes

Notes

The fundamental fix (moving disk I/O off the write-lock path entirely via background flush thread) is deferred. The current fix eliminates the specific bug: edge writes triggering node flushes. If enrichers begin emitting both nodes AND edges, the add_nodes memory-pressure path could trigger — at that point a background flush thread becomes necessary.

Safe write throughput: up to ~100K addEdges batches (at 10 edges each) before auto-flush. That's 1M edges before the byte limit triggers. No enricher should hit this in practice.