pairwise-test-case-generator

Overview

When a function takes multiple inputs whose interactions matter, the naive test set is the Cartesian product: 4 roles × 4 tiers × 5 features × 6 locales = 480 tests. Most teams give up before authoring all 480 and end up with happy-path-only coverage that misses interaction bugs.

The fix is all-pairs (pairwise) testing - pick a smaller test set that covers every pair of input values across the full matrix. Empirically, 2-wise coverage finds the majority of interaction bugs at a fraction of the test count. ISTQB catalogues this technique as pairwise testing, "a black-box test technique in which test conditions are pairs of parameter-value pairs" (commonly also called all-pairs testing).

This skill takes a multi-input spec and emits a reduced test set covering 1-wise / 2-wise / N-wise combinations.

When to use

• A function / endpoint takes 3 or more enumerable inputs (roles, tiers, statuses, feature flags, locales).
• Cartesian product would produce >100 tests.
• Pairs of inputs interact (e.g. tier=enterprise + feature=sso has a different code path than tier=free + feature=sso).
• Existing tests cover only a few hand-picked combinations - the team wants systematic coverage.

If only one input has multiple values, use boundary-value-generator instead - boundary analysis is the right tool for single-input testing.

How to use

1. Confirm the target takes 3+ interacting inputs whose Cartesian product exceeds ~100 tests.
2. Pick a coverage strength (default 2-wise; go 3-wise only after a triplet bug slips through).
3. Choose the reduction tool (default PICT; AllPairs for pytest-native, CATS for an in-process library).
4. Author the input spec in YAML: each input's name and values, plus domain constraints.
5. Run the tool to compute the reduced set and emit it in your runner's parametrize format.
6. Read the coverage report - accept constraint-driven gaps, or raise strength for real uncovered pairs.
7. Commit the spec to git as the source of truth; regenerate cases whenever a value or rule changes.

Step 1 - Pick the coverage strength

Strength	Coverage
1-wise	Every individual value of every input appears in at least one test.
2-wise (pairwise)	Every pair of values across two inputs appears together.
3-wise	Every triplet of values across three inputs appears together.
N-wise	Every N-tuple appears.

Strength	Test count for 4×4×5×6 input space (480 max)	Notes
1-wise	6	Just touches every value once.
2-wise	~30	Standard recommendation; finds most interaction bugs.
3-wise	~120	Use when triplet interactions matter (e.g. role × tier × feature).
Cartesian	480	Use only for small input spaces (≤4 inputs, ≤3 values each).

Default: 2-wise. Promote to 3-wise only when an incident postmortem shows a triplet bug slipped through 2-wise.

Step 2 - Pick the tool that computes the reduced set

Tool	Language	Notes
PICT	Standalone	Microsoft's canonical pairwise tool; CLI; deterministic output.
AllPairs	Python	Library; integrates with pytest.
CATS	Multi-lang	Open-source Combinatorial Test Generator.

Default: PICT - language-agnostic CLI, deterministic CSV output that any test runner can parametrize. Use AllPairs when the spec needs to live in Python alongside pytest fixtures; use CATS when the team needs an in-process Java/multi-language library API.

Step 3 - Author the input specification

A YAML format the skill consumes:

inputs:
  - name: role
    values: [admin, manager, standard, read_only]
  - name: tier
    values: [free, starter, pro, enterprise]
  - name: feature
    values: [sso, audit_log, api_access, custom_branding, support_priority]
  - name: locale
    values: [en-US, ja-JP, de-DE, ar-SA, fr-FR, pt-BR]

strength: 2
constraints:
  # Mutually exclusive combinations (suppress these from generation):
  - "tier=free AND feature=sso"          # SSO is paid-only
  - "tier=free AND feature=audit_log"    # Audit log is paid-only

constraints exclude combinations that don't make sense in the domain - including them would generate tests for impossible states.

Step 4 - Emit in the test-runner-native format

The reduced set is emitted in the runner's native parametrize format - PICT-style CSV, pytest @pytest.mark.parametrize, Jest / Vitest test.each, or xUnit [Theory] + [MemberData]. Copy-paste-ready snippets for each format: references/output-formats.md.

Step 5 - Verify coverage

After generation, the skill emits a coverage report:

## Coverage report - strength 2

**Inputs:** 4 (role × 4 values, tier × 4 values, feature × 5 values, locale × 6 values)
**Cartesian total:** 480 cases
**Generated cases:** 30
**Constraints suppressed:** 8 cases (tier=free × {sso,audit_log})

### Pair coverage matrix

| Pair                    | Required | Covered | Coverage |
|-------------------------|---------:|--------:|---------:|
| role × tier              |       16 |      14 |     88%  |
| role × feature           |       20 |      20 |    100%  |
| role × locale            |       24 |      24 |    100%  |
| tier × feature           |       20 |      18 |     90%  |
| tier × locale            |       24 |      24 |    100%  |
| feature × locale         |       30 |      30 |    100%  |

### Gaps (uncovered pairs)

- (role=read_only, tier=free) - suppressed by constraint
- (tier=starter, feature=audit_log) - increase strength or hand-add

The team reviews gaps; either accepts them (constraint-driven gap = OK) or escalates to 3-wise.

Worked example

A capability check takes four inputs: role (4 values), tier (4), feature (5), and locale (6). The Cartesian product is 4 × 4 × 5 × 6 = 480 tests - too many to author or run per commit.

Author the spec with the two paid-only constraints, set strength: 2, and run PICT. It returns ~30 cases covering every value pair, with the 8 impossible tier=free × {sso, audit_log} combinations suppressed. The coverage report confirms 100% pair coverage on five of the six input pairs and flags the two constraint-driven gaps (e.g. role=read_only × tier=free) for review. Result: interaction coverage from 30 tests instead of 480, checked into git as the parametrize table.

Anti-patterns

Anti-pattern	Why it fails	Fix
Cartesian product test set when input count > 3	Tests grow combinatorially; CI time explodes; flaky.	Use 2-wise; promote to 3-wise per incident.
Skipping constraints for impossible combinations	Tests fail for the wrong reason - testing tier=free + sso, which the system rejects.	Always declare domain constraints upfront.
Hard-coded test list maintained by hand	Combinatorial set is fragile; adding one input value 2x's the test count.	Generate from the spec; check the spec into git.
Failing to update the spec when business rules change	Test set drifts from production reality.	Spec is the source of truth; reviewers reject PRs that update tests without updating the spec.
Using 1-wise as the default	Misses every interaction bug; trivially covers single-input behavior.	2-wise minimum; 1-wise only for inputs known not to interact.

Limitations

• Pairwise misses N-tuple bugs. A bug that requires a specific triplet to manifest won't be caught by 2-wise alone. Promote to 3-wise on incident.
• Constraint logic must be encoded explicitly. The tool can't infer "free tier doesn't have SSO" - the spec must say so.
• Parameterized tests can hide failure attribution. When one case in 30 fails, the report should clearly identify which combination failed; many runners do this well, some don't. Verify your runner's parameterized output is informative.

References

• ISTQB pairwise testing - canonical definition of the technique this skill applies.
• PICT - https://github.com/microsoft/pict
• AllPairs (Python) - https://pypi.org/project/allpairspy/
• boundary-value-generator - sibling skill for single-input boundary cases.
• negative-test-generator - sibling skill for rejection-path coverage.