flaky-test-quarantine

Overview

A "flaky test" is a test that produces inconsistent pass/fail results across runs without an underlying code change (google-flaky). Industry consensus from Google Testing Blog and similar practitioner-engineering sources is that flaky tests should be isolated from the gating path rather than left to mask real regressions or be silently ignored (google-flaky).

Terminology note: "flaky test" is a practitioner-emergent term popularized by the Google Testing Blog. ISTQB does not maintain a canonical entry for it. This skill cites industry-engineering sources, not ISTQB authority.

How to use

1. Confirm it is a flake, not a regression - a 1% to 50% failure rate, not 100% (see "When to use"). A 100%-failing test is a regression: bisect and fix, do not quarantine.
2. Mark the test with the framework's skip / fixme / retry annotation (Step 1).
3. Annotate the body with date, issue link, failure rate, bisect status, and a Re-evaluate by date (Step 2) - this is the load-bearing, machine-parseable part.
4. Set the TTL that produces the Re-evaluate by date (Step 3).
5. Wire the expired-quarantine report into a scheduled CI job (Step 4 and references/ci-quarantine-report.md).
6. Prune on expiry - fix and un-quarantine, renew once (never past two consecutive renewals), or delete (Step 5).

When to use

• A test is failing on the trunk between 1% and 50% of runs (above 50% it's not flaky, it's broken; under 1% the noise is acceptable for most projects).
• The team's incident process has been triggered more than once by the same test.
• A new feature merge is blocked by a known-flaky pre-existing test that's unrelated to the change.

If the test fails 100% of the time after a code change, it's a regression - bisect to the introducing commit and fix, do not quarantine.

Worked example

Test tests/checkout.spec.ts:42 fails about 12% of runs on the tablet-768 project in CI. An unrelated feature PR is blocked by it.

1. Confirm it is a flake. 12% sits between 1% and 50%, and the test predates the PR - quarantine it, do not treat it as a regression of the PR under review.
2. Mark and annotate with a single test.fixme() carrying the full record (Steps 1 and 2):

test('checkout flow flaky test', async ({ page }) => {
  test.fixme(
    true,
    'Quarantined 2026-07-20 (#1234) - fails ~12% of runs on tablet-768; bisect inconclusive. Re-evaluate by 2026-08-19. Owner: @web-platform.',
  );
  // ... test body, no longer runs
});

3. Set the TTL. The default 30-day TTL (Step 3) puts Re-evaluate by 2026-08-19 in the annotation. The PR merges the same day, unblocked.
4. Let CI track it. The Monday report job (Step 4) greps every test.fixme annotation. On the first Monday after 2026-08-19 it prints EXPIRED for this entry and opens a tracking issue.
5. Prune. A fix for the tablet-768 layout race has since landed, so remove test.fixme(), re-run the test at depth to confirm it is green, and close the issue. Had it still failed, renew once with an updated annotation - never past two consecutive renewals.

Step 1 - Mark the test

Playwright

test.fixme() is the canonical Playwright primitive for "this test is broken; do not run past this point" (pw-test):

test('checkout flow flaky test', async ({ page }) => {
  test.fixme(
    true,
    'Quarantined 2026-05-04 (#1234) - fails ~12% of runs on tablet-768; bisect inconclusive. Re-evaluate by 2026-06-04.',
  );
  // ... test body, no longer runs
});

test.fixme(condition, description) skips with the description visible in the report. Unlike test.skip(), fixme carries the explicit "this needs to be fixed" intent (pw-test).

If the goal is to allow retries before quarantining, use the retries config first (pw-retries):

// playwright.config.ts
export default defineConfig({
  retries: process.env.CI ? 2 : 0,
});

A test that passes on retry is reported with the flaky status (distinct from passed and failed); track these separately - flaky-but-passing tests are quarantine candidates, not yet quarantined (pw-retries).

Cypress

Cypress configures retries at the suite level via Cypress.config('retries', { runMode: 2, openMode: 0 }). For quarantining individual specs, use it.skip(...) or the cypress-grep plugin's tagging convention.

Jest / Vitest

test.skip(...) and test.todo(...) are the canonical primitives. For periodic auto-evaluation, use test.skip.if(condition) patterns or introduce a project-specific tagging convention parsed by your CI.

JUnit / TestNG (JVM)

JUnit 5: @Disabled("Quarantined 2026-05-04 (#1234) - ..."). TestNG: @Test(enabled = false, description = "..."). For per-method retries before quarantine, JUnit 5's @RetryingTest(N) extension and TestNG's @Test(retryAnalyzer = ...).

Step 2 - Annotate with failure rate + bisect link + expiry

The annotation body is the load-bearing part of the workflow. Every quarantine record carries:

Field	Required	Format
Date	yes	YYYY-MM-DD of the quarantine.
Issue link	yes	#1234 or full URL - links a tracked ticket.
Failure rate	yes	~12% of runs - measured, not guessed.
Bisect status	yes	bisect inconclusive / bisected to commit abc1234 / not yet bisected.
Re-evaluate by	yes	YYYY-MM-DD - the auto-expiry date.
Owner	optional	@team-handle for routing.

The format is parseable by the re-evaluation report (Step 4):

Quarantined 2026-05-04 (#1234) - fails ~12% of runs on tablet-768;
bisect inconclusive. Re-evaluate by 2026-06-04. Owner: @web-platform.

Step 3 - Auto-expiry

Default TTL: 30 days. Picked because:

• Long enough to fix non-trivial issues without creating churn.
• Short enough that quarantined tests don't become a permanent graveyard.

Adjust per project:

• 90 days for low-traffic suites where the underlying issue is known but un-prioritized.
• 14 days for high-traffic CI where flakiness is an actively-monitored metric.

Step 4 - Re-evaluation report

A scheduled CI job greps every quarantine annotation, extracts the Re-evaluate by date, and lists the entries that have expired - then opens a tracking issue (or posts to Slack) per expired entry. Because the Step 2 format is machine-parseable, the whole report is a short grep. The Bash report script and the scheduled GitHub Actions workflow are in references/ci-quarantine-report.md.

Step 5 - Pruning rules

When a re-evaluation expires, the team has three options:

Outcome	Action
Underlying issue fixed	Remove test.fixme() and re-run; close the issue.
Underlying issue still present	Renew the quarantine for one more TTL with updated annotation; never more than two consecutive renewals - at that point, delete the test or rewrite it.
The test is no longer relevant	Delete the test outright; close the issue.

The two-renewal cap is the lever that prevents quarantine from becoming a permanent dead-letter. Past two renewals, the team has either lost interest in the assertion or the test is fundamentally unfixable - both signal "delete."

References

• google-flaky - Google Testing Blog on flaky tests at scale; practitioner-emergent canonical reference for the term.
• pw-test - Playwright test.fixme() / test.skip() / test.fail() API.
• pw-retries - Playwright retries config + flaky status reporting.
• references/ci-quarantine-report.md - the Step 4 report script + scheduled CI workflow.
• flake-pattern-reference - catalog of flake patterns to consult during bisect.