flake-pattern-reference

Terminology note: "flaky test" is a practitioner-emergent term popularized by the Google Testing Blog (google-causes, google-flaky); ISTQB does not maintain a canonical entry. This catalog reflects industry-engineering consensus, not ISTQB authority.

A flake is rarely random - it almost always falls into one of eight recurring patterns. Identifying the pattern early shrinks the bisect search space dramatically. This catalog is a reference, not a workflow; the matching workflow is in flaky-test-quarantine.

The Google Testing Blog observed a near-linear correlation between test size and flakiness rate across ~4.2M tests (google-causes) - larger tests touch more of the eight patterns at once.

Pattern 1: async / timing

The most common flake category in UI and integration tests.

Signal	What's happening
Fails ~5 - 20% of runs; passes when the machine is faster	Test waits for an arbitrary setTimeout(N) instead of a deterministic event.
Fails on CI but never locally	CI runners have different cold-start timings than dev laptops.
Fails after a dependency upgrade with no test code change	Library's internal timing changed (e.g. Playwright auto-wait window).

Remediation:

• Replace fixed sleeps with deterministic waits - await expect(loc).toBeVisible(), page.waitForLoadState('networkidle'), page.waitForFunction(...), etc.
• For animations, disable them in test setup (animations: 'disabled' in Playwright; Cypress.config('animationDistanceThreshold', 0) in Cypress).
• For absolute clock dependencies, freeze time with sinon.useFakeTimers() / vi.useFakeTimers() / Playwright's page.clock.install().

Pattern 2: test ordering

Tests pass alone, fail when run with siblings.

Signal	What's happening
npm test -- --testNamePattern='^X$' passes; full run fails	Test relies on state from a previously-run test.
Adding a new test breaks an unrelated existing one	Implicit ordering dependency exposed by the new test pushing the old test into a different position.
Random-order test runners (Jest randomize) flag the suite	Suite is order-dependent.

Remediation:

• Run the suite with explicit randomization in CI to surface ordering deps early (jest --randomize, pytest --random-order, mocha --sort reverse).
• Move ANY shared setup into beforeEach / afterEach, never rely on beforeAll for state that the test mutates.
• Database tests: roll back transactions after each test instead of truncating between describe blocks.

Pattern 3: shared parallel state

Tests pass sequentially, fail when run in parallel workers.

Signal	What's happening
Fails ~50% of runs in CI matrix; passes locally with -j 1	Two workers writing to the same DB row / file / port.
Fails more often as worker count goes up	Linear shared-state contention.
Error message mentions "duplicate key" / "address in use" / "file already exists"	Direct collision evidence.

Remediation:

• Find the shared state two workers collide on - DB rows, files, ports, or global module state.
• Per-worker isolation: per-worker DB schemas (PG_SCHEMA=test_${WORKER}), per-worker temp dirs (TMPDIR=/tmp/test-${WORKER}), per-worker port ranges.
• For unique IDs: use UUIDs or a per-worker namespace prefix, not auto-increment integers shared across workers.

Pattern 4: resource leaks

Tests pass on a fresh machine, fail after the test process has run for hours.

Signal	What's happening
Fails increasingly often as suite duration grows	Memory or file-descriptor leak in the test setup.
EMFILE / EADDRINUSE errors mid-suite	File-descriptor or port exhaustion.
Long-running processes (Playwright browsers, Cypress runners) crash mid-suite	Process accumulating zombies.

Remediation:

• Always await browser.close() / await server.close() in afterAll, with a try/finally so failed tests still clean up.
• Set per-test timeouts and ensure the framework kills the process, not just the test (--testTimeout, test.setTimeout()).
• Run lsof | wc -l and ps aux | wc -l before / after the suite in CI to detect leaks; alert when growth exceeds a threshold.

Pattern 5: network / external service

Tests pass when the upstream is healthy, fail otherwise.

Signal	What's happening
Fails on the same handful of tests that hit the same external URL	Real network call to a flaky third party.
Fails right after a deploy of a non-test service	Test is hitting prod / staging of a sibling service.
ETIMEDOUT / ECONNRESET in error logs	Network-layer error, not test-logic error.

Remediation:

• Mock at the boundary - never let test code reach a real network endpoint. Use Mock Service Worker (MSW), nock, WireMock, or Playwright's page.route().
• For tests that must hit a real service (smoke / contract tests), isolate them in a separate suite that doesn't gate the main CI.
• DNS-level: pin to specific resolvers in CI to avoid resolver variance.

Pattern 6: locator drift

UI tests pass when the page looks one way, fail when it shifts.

Signal	What's happening
Fails after an unrelated CSS change	Selector matched by position rather than identity.
selector matched 2 elements errors	Ambiguous selector now matches more than one node.
Fails only at certain viewports	Layout shifts cause mobile / desktop selectors to differ.

Remediation:

• Use role-based selectors first (page.getByRole('button', { name: 'Submit' })), then data-testid, only text= / CSS as a last resort.
• For Playwright: enable strict: true so any ambiguous selector fails immediately rather than silently picking the first match.
• For viewport-specific UIs: snapshot at every breakpoint via the breakpoint matrix in playwright-snapshots; visual signal exposes layout-shift flakes faster than text checks.

Pattern 7: environment variance

Tests pass on Linux CI, fail on macOS dev machines (or vice versa).

Signal	What's happening
Fails only on a specific CI runner / OS	OS-specific path separator, line ending, or filesystem case sensitivity.
Snapshot tests fail with sub-pixel diffs across OS	OS font / anti-aliasing differences (see playwright-snapshots).
Fails in tz configurations not set to UTC	Timezone-sensitive assertion.

Remediation:

• Pin CI to one OS / one timezone (TZ=UTC) for deterministic runs.
• Run snapshot updates only in CI, never from a developer laptop (per playwright-snapshots).
• For path-sensitive code, normalize with path.posix.join() / node:path.

Pattern 8: randomness

Tests use random data without a controlled seed.

Signal	What's happening
Failures don't reproduce on retry	Test data was randomized; the failing combination is gone.
Test asserts a property that holds "almost always"	Property-based test exposing a real edge case (this is good - fix the production bug).
Faker-generated data triggers a layout overflow	Random string longer than the assertion expected.

Remediation:

• Seed every random source: Math.random via seedrandom, faker via faker.seed(N), property-based testing via fc.assert(prop, { seed }).
• For property-based failures, don't mark them as flake - copy the failing seed into a regression test.
• Persist the seed used in each CI run as a build artifact so a flake can be replayed.

Triage decision tree

Test fails ~50% of runs?
├── Yes → likely "shared parallel state" or "test ordering"
└── No → fails ~5-20% of runs?
    ├── Yes → likely "async/timing" or "network"
    └── No → fails only on specific OS / runner?
        ├── Yes → "environment variance"
        └── No → fails after long suite duration?
            ├── Yes → "resource leaks"
            └── No → fails after unrelated UI change?
                ├── Yes → "locator drift"
                └── No → does the test use random data?
                    ├── Yes → "randomness"
                    └── No → run a structured bisect

For systematic bisection, run a structured bisect that varies one axis at a time per the patterns above.

Code-level fixes

Once the pattern is named - by inspection above or by experiment via flake-axis-bisection - apply the smallest change the pattern calls for (a targeted edit, not a rewrite), then re-measure at a real depth: re-run at an N chosen from the failure rate you are willing to ship, not the screening N; a clean 0/20 does not prove the flake is gone (flake-axis-bisection). Quarantine via flaky-test-quarantine if the flake blocks the trunk while the fix is in review.

The per-pattern code fixes, grounded in Playwright, Cypress, MSW, and Faker official docs:

• Patterns 1-4 (async/timing, test ordering, shared parallel state, resource leaks): references/timing-and-isolation-fixes.md.
• Patterns 5-6 (network mock-at-the-boundary with page.route() / MSW; role-based locators with data-testid fallback): references/network-and-locator-fixes.md.
• Patterns 7-8 (pin TZ=UTC, freeze the clock, normalize paths; seed every RNG and persist the seed): references/environment-and-randomness-fixes.md.

Worked example

A checkout test, tests/checkout.spec.ts:42, fails about 15% of runs in CI and always passes locally.

1. Classify. flake-axis-bisection implicates the network-latency axis, and reading the source shows the assertion is gated on a fixed page.waitForTimeout(2000), not on the response. That is Pattern 1 (async / timing): the sleep is shorter than the slowest CI response.
2. Apply the fix. Replace the fixed sleep with a web-first assertion that retries until the condition holds:

// Before - the 2s sleep races a variable-latency XHR
await page.getByRole('button', { name: 'Place order' }).click();
await page.waitForTimeout(2000);
expect(await page.getByText('Order confirmed').isVisible()).toBe(true);

// After - retries until the confirmation renders or the timeout expires
await page.getByRole('button', { name: 'Place order' }).click();
await expect(page.getByText('Order confirmed')).toBeVisible();

3. Re-measure and ship. The team's tolerance is 1%, so re-run at N=300 (flake-axis-bisection Step 2). A clean 0/300 bounds the rate at roughly 1%; a clean 0/20 would have proved nothing. No quarantine was needed - the fix landed inside the PR the flake was blocking.

Quick-reference: pattern to fix

Pattern	Key fix	Primary API
async / timing	Replace sleep with auto-wait assertion	await expect(loc).toBeVisible()
test ordering	Move setup to beforeEach; roll back DB per test	test.beforeEach / test.afterEach
shared parallel state	Per-worker schema / dir / port via workerIndex	testInfo.workerIndex
resource leaks	browser.close() in afterAll with try/finally	test.afterAll + try/finally
network	Mock at boundary; never reach real endpoints	page.route() / MSW
locator drift	Role-based locators; data-testid fallback	getByRole()
environment variance	Pin TZ=UTC; freeze clock; normalize paths	page.clock.install()
randomness	Seed every RNG; persist seed in CI log	faker.seed(N)

Per-fix citations live in the three reference files above.

References

• google-flaky - Google Testing Blog overview.
• google-causes - Google's correlation analysis on ~4.2M tests; "test size correlates with flakiness rate."
• flaky-test-quarantine - workflow that uses this catalog during triage.