feature-flag-test-matrix-reference

Overview

A codebase with N feature flags, each having M variants, and users in K segments, has N × M × K possible flag-state-segment combinations. At realistic numbers (50 flags, 2 variants each, 5 segments) that's 500 - and at 50 flags with 3 variants and 10 segments, it's 1500. Testing every combination is infeasible, so the matrix has to be sampled deliberately rather than enumerated.

This skill is both the reference (the combinatorics + strategies below) and the coverage-suite-building workflow (see Building the coverage suite); the per-SDK test mechanics live in launchdarkly-testing and openfeature-sdk-testing.

When to use

• Designing the test surface for a new flag-heavy product.
• Auditing existing flag-test coverage - are critical combinations covered?
• PR review of a new flag - does it create a coverage gap?
• Investigating an "only happens with flag X + flag Y on" incident.

How to use

1. Inventory the flags: list every flag, its variant count (M), and the user segments (K) it targets.
2. Size the problem: compute N × M × K to confirm full enumeration is infeasible and sampling is required.
3. Discover interactions: mark flags as inert (independent) or known-interacting (auth + permissions, billing + plan-tier), pulling from a risk register where one exists.
4. Pick a coverage strategy from the five below - default-only smoke, per-flag isolation, pairwise, full matrix, or risk-driven - matching the flag set's risk.
5. Add the special flag-state categories (kill-switch, percentage-rollout, sticky-assignment, default-on-error) as their own tests.
6. Layer the resulting cases across unit, integration, E2E, and production-smoke.
7. Hand the matrix to the platform SDK skill (launchdarkly-testing, or openfeature-sdk-testing for OpenFeature / Unleash / Flagsmith / GrowthBook) to implement each case.

The combinatorics

Variable	Typical scale
Total flags in codebase	20-500
Variants per flag	2 (most), 3-5 (experiments), 10+ (multivariate)
User segments	5-20 (free, paid, enterprise, internal, beta, etc.)
Combinatorial total	Quickly enters thousands

Insight: most flag combinations are inert (independent). Only a small subset interact - the test matrix should target interactions.

Five coverage strategies

1. Default-only smoke

Test only the default-value combination ("all flags off" or "all flags at default"). Fast but misses everything.

Use when: flag-heavy codebase where defaults change rarely.

2. Per-flag isolation

For each flag, test default + each variant in isolation. N × M tests; ignores interactions.

Use when: flags are mostly independent (UI tweaks, language strings, low-risk).

3. Pairwise interaction

Test every pair of flags (combinatorial 2-way coverage). Per NIST SP 800-142 on combinatorial testing, pairwise catches ~67% of real defects with O(N²) combinations.

Use when: flags are known-interacting (auth + permissions, billing + plan-tier).

Implementation: tools like pict (Microsoft) generate the pairwise matrix from a flag inventory.

4. Full matrix

Every combination. N^M tests for boolean flags.

Use when: small (≤10) flag count with strong interaction; financial / regulatory paths.

5. Risk-driven

Custom matrix targeting (flag, segment) cells with known risk (per a risk register per risk-matrix in the qa-process plugin).

Use when: any non-trivial codebase. Best in practice.

Building the coverage suite

The workflow that turns the strategies above into a committed matrix + test skeletons:

1. Inventory flags - grep for SDK calls and emit a flag inventory:

grep -rn 'isOn\|isEnabled\|variation\|getFeatureValue' --include='*.{ts,js,py,go,java}' .

flags:
  - name: show-new-ui
    platform: launchdarkly
    type: boolean
    found_at: [src/components/Header.tsx:42, src/pages/Dashboard.tsx:88]
  - name: checkout-experiment
    type: multi-variant
    variants: [control, treatment-a, treatment-b]

2. Classify each flag - kill-switch (naming: *-kill, disable-*, emergency-*), experiment (multi-variant + analytics), permission-gated, UI tweak, migration (use-new-*), plan/tier gating. The class picks the strategy: default-only smoke for UI tweaks, per-flag isolation for migrations, pairwise for permission/plan interactions, full matrix for kill-switches + regulatory paths, risk-driven for the rest.
3. Generate the matrix - for pairwise use PICT (pict pict.txt > matrix.tsv emits a pairwise-covering matrix, e.g. ≤12 tests instead of 24 for full); for risk-driven, combine with the risk register (risk-matrix in the qa-process plugin) - high-impact x high-likelihood cells become required tests.
4. Emit a per-cell test skeleton - one describe block per flag (or flag-pair), one test per cell, with the SDK pinned via the platform's test data source:

describe('auth flag matrix', () => {
  test('free user, new auth on → new flow', () => {
    td.update(td.flag('use-new-auth').booleanFlag().on(true));
    expect(authFlow({ plan: 'free' })).toBe('new');
  });
});

5. Add the special-category tests regardless of matrix coverage: kill-switch deactivation latency (see references/killswitch.md), default-on-error (SDK failure returns the call-site default), and sticky-assignment (same user, same variant across evaluations).
6. Commit the matrix + document the gaps - flag-coverage.yaml in the repo, plus a coverage doc listing covered cells and deliberate gaps with reasons, so drift is reviewable.

Special flag-state test categories

Category	Test
Kill-switch	Setting flag → off must halt the feature within N seconds (cache TTL); full four-category treatment in references/killswitch.md
Percentage rollout	Flag at 10% → ~10% of users in 'on' bucket; SDK assignment stable per user
Targeted rollout	Targeting region=EU → only EU users get treatment
Sticky assignment	Same user → same variant across sessions and re-launches
Override hierarchy	User-specific override > segment override > default
Default-on-error	SDK fails / network down → default value returned
Fast-deactivate	Toggle flag off → live users see new state on next evaluation

These are per-platform behaviours (LaunchDarkly, Unleash, Flagsmith, GrowthBook implement them differently); test per platform per the SDK skills.

Flag-test layering

Tests should run at multiple layers:

Layer	Coverage
Unit	Resolver / handler logic gated on flag value (mock SDK)
Integration	SDK + handler together (test SDK against local-eval / fixture)
E2E	Real flag toggle → real user sees the change
Production smoke	After flag change → assert expected behaviour live

Flag-experiment distinction

Per ab-test-validity-checklist (in this plugin):

Flag	Experiment
Toggles behaviour	Measures outcome
Boolean / multi-variant	Multi-arm with metrics
Test the behaviour change	Test the assignment + outcome correlation
Ship decision on engineering judgment	Ship decision on statistical result

A feature flag can power an experiment (the flag is the allocation mechanism), but tests are layered: flag tests verify correct behaviour per variant; experiment tests verify assignment + analytics.

Worked example

A billing service has 3 boolean flags - new_checkout, annual_discount, tax_engine_v2 - across 3 segments (free, paid, enterprise). The team knows annual_discount and tax_engine_v2 interact: a discount changes the taxable amount.

1. Inventory: 3 flags, 2 variants each, 3 segments; the full flag-state-by-segment cross-product is too large to enumerate.
2. Interaction discovery marks the annual_discount × tax_engine_v2 pair as known-interacting; new_checkout is a UI change treated as independent.
3. Strategy: risk-driven. Per-flag isolation covers new_checkout; a pairwise cell covers (annual_discount on, tax_engine_v2 on) on the paid and enterprise segments where money moves.
4. Special categories: a kill-switch test asserts toggling tax_engine_v2 off reverts to v1 within the cache TTL; a sticky-assignment test asserts the annual_discount percentage rollout keeps each user in the same bucket across sessions.
5. Layering: unit tests exercise the tax resolver per flag value; an E2E test toggles tax_engine_v2 and asserts an enterprise invoice total.

Result: a handful of targeted cases instead of the full cross-product, with the discount × tax interaction covered explicitly.

Anti-patterns

Anti-pattern	Why it fails	Fix
Test only default-value path	Misses every flag-on case	Per-flag isolation minimum
Mock the SDK to return constant	Misses targeting / rollout logic	Local-eval mode or fixture-based SDK
Same test for every flag combination	Slow; flaky; opaque failures	Per-combination assertion logs
No kill-switch test	Production incident has no rehearsed response	Test deactivation latency
Don't test percentage-rollout sticky-assignment	Rollout produces non-deterministic UX	Per ab-test-validity-checklist
Tests assume flag-on default	Real default-off behaviour untested in CI	Test both paths
No cleanup test for removed flags	Stale flags accumulate	Periodic audit via the stale-flag-detector agent
Pairwise without flag-interaction discovery	Some pairs spuriously interact	Couple with risk-register input

Limitations

• Pairwise misses 3-way+ interactions. Some real-world bugs need 3-way coverage.
• Real-world matrices have ordering effects. Flag A enabled THEN flag B may differ from B then A; test ordering needs separate coverage.
• Coverage tooling lags. PICT / ACTS exist but integration with flag platforms is bespoke.
• Stale flags pollute the matrix. Cleanup pairs with the stale-flag-detector agent's removal runbook.

References

• LaunchDarkly flag-testing strategy: launchdarkly.com/blog/.
• NIST SP 800-142 (combinatorial testing): csrc.nist.gov/publications/detail/sp/800-142/final.
• GrowthBook test docs: docs.growthbook.io/lib/node.
• Kill-switch test categories: references/killswitch.md.
• Consumed by: launchdarkly-testing, openfeature-sdk-testing.
• Removal side: stale-flag-detector agent.
• Cross-plugin: ab-test-validity-checklist, feature-flag-test-harness.