flaky-test-quarantine
Overview
A "flaky test" is a test that produces inconsistent pass/fail results
across runs without an underlying code change (google-flaky).
Industry consensus from Google Testing Blog and similar
practitioner-engineering sources is that flaky tests should be
isolated from the gating path rather than left to mask real
regressions or be silently ignored (google-flaky).
Terminology note: "flaky test" is a practitioner-emergent term
popularized by the Google Testing Blog. ISTQB does not maintain a
canonical entry for it. This skill cites industry-engineering
sources, not ISTQB authority.
How to use
- Confirm it is a flake, not a regression - a 1% to 50% failure
rate, not 100% (see "When to use"). A 100%-failing test is a
regression: bisect and fix, do not quarantine.
- Mark the test with the framework's skip / fixme / retry
annotation (Step 1).
- Annotate the body with date, issue link, failure rate, bisect
status, and a
Re-evaluate by date (Step 2) - this is the
load-bearing, machine-parseable part.
- Set the TTL that produces the
Re-evaluate by date (Step 3).
- Wire the expired-quarantine report into a scheduled CI job (Step 4
and references/ci-quarantine-report.md).
- Prune on expiry - fix and un-quarantine, renew once (never past two
consecutive renewals), or delete (Step 5).
When to use
- A test is failing on the trunk between 1% and 50% of runs (above 50%
it's not flaky, it's broken; under 1% the noise is acceptable for
most projects).
- The team's incident process has been triggered more than once by the
same test.
- A new feature merge is blocked by a known-flaky pre-existing test
that's unrelated to the change.
If the test fails 100% of the time after a code change, it's a
regression - bisect to the introducing commit and fix, do not
quarantine.
Worked example
Test tests/checkout.spec.ts:42 fails about 12% of runs on the
tablet-768 project in CI. An unrelated feature PR is blocked by it.
- Confirm it is a flake. 12% sits between 1% and 50%, and the test
predates the PR - quarantine it, do not treat it as a regression of
the PR under review.
- Mark and annotate with a single
test.fixme() carrying the full
record (Steps 1 and 2):
test('checkout flow flaky test', async ({ page }) => {
test.fixme(
true,
'Quarantined 2026-07-20 (#1234) - fails ~12% of runs on tablet-768; bisect inconclusive. Re-evaluate by 2026-08-19. Owner: @web-platform.',
);
// ... test body, no longer runs
});
- Set the TTL. The default 30-day TTL (Step 3) puts
Re-evaluate by 2026-08-19 in the annotation. The PR merges the same
day, unblocked.
- Let CI track it. The Monday report job (Step 4) greps every
test.fixme annotation. On the first Monday after 2026-08-19 it
prints EXPIRED for this entry and opens a tracking issue.
- Prune. A fix for the
tablet-768 layout race has since landed, so
remove test.fixme(), re-run the test at depth to confirm it is
green, and close the issue. Had it still failed, renew once with an
updated annotation - never past two consecutive renewals.
Step 1 - Mark the test
Playwright
test.fixme() is the canonical Playwright primitive for "this test is
broken; do not run past this point" (pw-test):
test('checkout flow flaky test', async ({ page }) => {
test.fixme(
true,
'Quarantined 2026-05-04 (#1234) - fails ~12% of runs on tablet-768; bisect inconclusive. Re-evaluate by 2026-06-04.',
);
// ... test body, no longer runs
});
test.fixme(condition, description) skips with the description
visible in the report. Unlike test.skip(), fixme carries the
explicit "this needs to be fixed" intent (pw-test).
If the goal is to allow retries before quarantining, use the
retries config first (pw-retries):
// playwright.config.ts
export default defineConfig({
retries: process.env.CI ? 2 : 0,
});
A test that passes on retry is reported with the flaky status
(distinct from passed and failed); track these separately -
flaky-but-passing tests are quarantine candidates, not yet
quarantined (pw-retries).
Cypress
Cypress configures retries at the suite level via
Cypress.config('retries', { runMode: 2, openMode: 0 }). For
quarantining individual specs, use it.skip(...) or the
cypress-grep plugin's
tagging convention.
Jest / Vitest
test.skip(...) and test.todo(...) are the canonical primitives.
For periodic auto-evaluation, use test.skip.if(condition) patterns or
introduce a project-specific tagging convention parsed by your CI.
JUnit / TestNG (JVM)
JUnit 5: @Disabled("Quarantined 2026-05-04 (#1234) - ...").
TestNG: @Test(enabled = false, description = "...").
For per-method retries before quarantine, JUnit 5's @RetryingTest(N)
extension and TestNG's @Test(retryAnalyzer = ...).
Step 2 - Annotate with failure rate + bisect link + expiry
The annotation body is the load-bearing part of the workflow. Every
quarantine record carries:
| Field | Required | Format |
|---|
| Date | yes | YYYY-MM-DD of the quarantine. |
| Issue link | yes | #1234 or full URL - links a tracked ticket. |
| Failure rate | yes | ~12% of runs - measured, not guessed. |
| Bisect status | yes | bisect inconclusive / bisected to commit abc1234 / not yet bisected. |
| Re-evaluate by | yes | YYYY-MM-DD - the auto-expiry date. |
| Owner | optional | @team-handle for routing. |
The format is parseable by the re-evaluation report (Step 4):
Quarantined 2026-05-04 (#1234) - fails ~12% of runs on tablet-768;
bisect inconclusive. Re-evaluate by 2026-06-04. Owner: @web-platform.
Step 3 - Auto-expiry
Default TTL: 30 days. Picked because:
- Long enough to fix non-trivial issues without creating churn.
- Short enough that quarantined tests don't become a permanent
graveyard.
Adjust per project:
- 90 days for low-traffic suites where the underlying issue is known
but un-prioritized.
- 14 days for high-traffic CI where flakiness is an actively-monitored
metric.
Step 4 - Re-evaluation report
A scheduled CI job greps every quarantine annotation, extracts the
Re-evaluate by date, and lists the entries that have expired - then
opens a tracking issue (or posts to Slack) per expired entry. Because the
Step 2 format is machine-parseable, the whole report is a short grep.
The Bash report script and the scheduled GitHub Actions workflow are in
references/ci-quarantine-report.md.
Step 5 - Pruning rules
When a re-evaluation expires, the team has three options:
| Outcome | Action |
|---|
| Underlying issue fixed | Remove test.fixme() and re-run; close the issue. |
| Underlying issue still present | Renew the quarantine for one more TTL with updated annotation; never more than two consecutive renewals - at that point, delete the test or rewrite it. |
| The test is no longer relevant | Delete the test outright; close the issue. |
The two-renewal cap is the lever that prevents quarantine from
becoming a permanent dead-letter. Past two renewals, the team has
either lost interest in the assertion or the test is fundamentally
unfixable - both signal "delete."
References
- google-flaky - Google Testing Blog on flaky tests at scale;
practitioner-emergent canonical reference for the term.
- pw-test - Playwright
test.fixme() / test.skip() /
test.fail() API.
- pw-retries - Playwright retries config +
flaky
status reporting.
- references/ci-quarantine-report.md -
the Step 4 report script + scheduled CI workflow.
flake-pattern-reference -
catalog of flake patterns to consult during bisect.