race-condition-test-author

Race conditions are the canonical "works on my machine, breaks under load" bug. Tests must drive shared-state access deterministically (barriers, latches) AND non-deterministically (sanitizers, stress) to expose them.

When to use

• Service handles concurrent requests on shared in-process state (caches, counters, connection pools).
• Race-condition incident retro: build a regression test before fixing.
• Pre-release smoke: TSan-instrumented binary catches data races that compile clean.

Step 1 - Identify shared mutable state

Code-review checklist:

• Static / global / module-level mutable variables
• Singleton instances with mutable fields
• Cache structures with read-and-modify ("get-then-set" without CAS)
• Per-request state stored on shared objects (mutable request context)
• Connection pools / object pools without internal lock
• Lazy-init patterns ("if not initialized, initialize")

For each: ask "what if two threads / goroutines / async tasks hit this concurrently?"

Step 2 - Deterministic interleaving via barriers

import threading

def test_lazy_init_thread_safe():
    target = LazyService()  # has private _instance + lazy_get()
    barrier = threading.Barrier(parties=2)
    results = [None, None]

    def worker(idx):
        barrier.wait()  # both threads stop here, then race
        results[idx] = target.lazy_get()

    t1 = threading.Thread(target=worker, args=(0,))
    t2 = threading.Thread(target=worker, args=(1,))
    t1.start(); t2.start()
    t1.join(); t2.join()

    assert results[0] is results[1], "Lazy init created two instances under race"

Barrier ensures both threads start the contended section at the same time - much higher probability of triggering the race than naive threading.Thread().

Step 3 - ThreadSanitizer for C/C++/Go

Per the ThreadSanitizer docs, TSan detects data races at runtime with ~5-15× overhead. For C/C++:

clang -fsanitize=thread -g -O1 program.c -o program
./program

For Go, the native data race detector:

go test -race ./...
go run -race main.go

The full Go workflow - GORACE options (log_path, halt_on_error, history_size), stress amplification via -count/-cpu, go vet loop-capture checks, goroutine-leak gating with goleak, and the CI matrix - is in references/go.md.

Output for a detected race:

WARNING: DATA RACE
Read at 0x... by goroutine 7:
  main.read+0x...
    main.go:42

Previous write at 0x... by goroutine 6:
  main.write+0x...
    main.go:38

Per the ThreadSanitizer docs, adaptive delay injection (TSAN_OPTIONS=enable_adaptive_delay=1) helps surface races at synchronization points.

Step 4 - jcstress for JVM

Per the jcstress docs, jcstress is "the experimental harness ... for the correctness of concurrency support in the JVM."

@JCStressTest
@Outcome(id = "0, 0", expect = ACCEPTABLE, desc = "Initial values")
@Outcome(id = "1, 1", expect = ACCEPTABLE, desc = "Both writes seen")
@Outcome(id = "0, 1", expect = ACCEPTABLE, desc = "Saw partial")
@Outcome(id = "1, 0", expect = FORBIDDEN, desc = "Reordered - bug")
@State
public class CounterTest {
    int x, y;

    @Actor
    public void writer() { x = 1; y = 1; }

    @Actor
    public void reader(II_Result r) {
        r.r1 = y;
        r.r2 = x;
    }
}

@Actor methods run concurrently on different threads; @Outcome classifies observed (r1, r2) pairs. FORBIDDEN outcomes indicate a memory-model violation (in this case, reordering allowed under JMM unless volatile or final).

Run:

java -jar jcstress.jar -m quick CounterTest

Per the jcstress docs: tests are probabilistic; longer runs find more reorderings.

Step 5 - Loom virtual-thread interleavings (Java 21+)

Java 21+ Project Loom enables cheap virtual threads. Use to test many-concurrent-task interleavings without OS thread cost:

@Test
void test_handles_10000_concurrent_orders() throws Exception {
    var orderService = new OrderService();
    try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
        var futures = IntStream.range(0, 10_000)
            .mapToObj(i -> executor.submit(() -> orderService.place(i)))
            .toList();
        for (var f : futures) f.get();
    }
    assertEquals(10_000, orderService.totalProcessed());
}

Virtual threads run M:N on OS threads - the JVM scheduler interleaves them aggressively, exposing schedule-dependent races.

Step 6 - Property-based + concurrency

Combine Hypothesis (Python) / fast-check (JS) with concurrency:

from hypothesis import given, strategies as st
import threading

@given(operations=st.lists(st.tuples(st.sampled_from(["read", "write"]), st.integers()), min_size=10, max_size=100))
def test_counter_property_under_concurrency(operations):
    counter = ThreadSafeCounter()
    threads = []
    for op, val in operations:
        if op == "write":
            threads.append(threading.Thread(target=lambda v=val: counter.set(v)))
        else:
            threads.append(threading.Thread(target=counter.get))

    for t in threads: t.start()
    for t in threads: t.join()

    # Property: final value is one of the written values
    assert counter.get() in [v for op, v in operations if op == "write"]

Cross-ref qa-property-based plugin for property-based test authoring patterns.

Step 7 - CI integration

# Go
- name: Run race detector
  run: go test -race ./...

# C/C++
- name: TSan build + test
  run: |
    cmake -B build -DCMAKE_C_FLAGS="-fsanitize=thread -g -O1"
    cmake --build build
    ./build/test_runner

# Java
- name: jcstress quick run
  run: java -jar jcstress.jar -m quick -r results/

Note the latency cost: go test -race ~3× slower; jcstress quick mode ~minutes per test class.

Anti-patterns

Anti-pattern	Why it fails	Fix
time.sleep(0.001) to "force" interleaving	Non-deterministic; flake	Use barriers (Step 2)
Run race tests once and assume green	Probabilistic; some races take hours	Multiple runs OR longer runs OR sanitizers
Skip TSan in CI for "release" builds	Race in release; CI passed without -race	TSan in CI for at least one matrix dimension (Step 7)
Depend on assertion in worker thread	Thread death silent; main thread sees pass	Use futures + assert from main
Test only the bug-causing race, not similar	Other shared state has same pattern; bugs ship	Code-review checklist (Step 1)

Limitations

• TSan finds only data races (unsynchronized concurrent access), not higher-level race conditions (correct-but-non-atomic multi-step operations).
• jcstress is JVM-only; no equivalent for non-JVM stacks.
• Go's -race overhead is real (~5-10× memory); CI matrix budget required.
• Loom virtual threads (Java 21+) require recent JDK; older versions need OS threads.

References

• ThreadSanitizer docs - clang -fsanitize=thread, runtime options, adaptive delay
• jcstress docs - @JCStressTest, @Actor, @Outcome outcomes
• ThreadSanitizer C++ Manual - github.com/google/sanitizers/wiki/ThreadSanitizerCppManual
• references/go.md - Go race detector + goleak workflow (GORACE, stress, CI matrix)
• deadlock-detection-harness - sister skill for deadlock-specific patterns
• async-ordering-tests - sister skill for async/await ordering