A/B Test Planner

When to Use

• Before running any product experiment or feature test
• When optimizing conversion funnels or UX flows
• When leadership requires statistical rigor for feature decisions
• When planning multi-variant tests or sequential experiments

Procedure

Step 1 — Hypothesis

Write a clear, falsifiable hypothesis:

• If [change we're making]
• Then [metric we expect to change]
• Because [reasoning / user insight]

Step 2 — Metrics

Type	Metric	Current Baseline
Primary	The metric that determines success	[value]
Secondary	Supporting metrics that provide context	[value]
Guardrail	Metrics that must NOT degrade	[value]

Step 3 — Sample Size & Duration

Calculate required sample size using:

• Baseline conversion rate (p₁)
• Minimum Detectable Effect (MDE) — smallest meaningful change
• Statistical significance level (α) — typically 0.05
• Statistical power (1−β) — typically 0.80

n = f(p₁, MDE, α, β) → use standard sample size calculator

Duration = n / (daily traffic × allocation %)

Step 4 — Randomization Plan

1. Randomization unit: user, session, device, or account
2. Allocation: 50/50, or asymmetric with justification
3. Stratification: any segments to balance (geography, plan, device)
4. Exclusion: users to exclude (employees, bots, existing tests)

Step 5 — Decision Rules

Outcome	Criteria	Action
Winner	Primary metric ↑ ≥ MDE, p < 0.05, guardrails stable	Ship to 100%
Neutral	No significant difference	Keep control, iterate hypothesis
Loser	Primary metric ↓ significantly	Revert, analyze why
Guardrail breach	Any guardrail metric degrades > threshold	Stop test immediately

Step 6 — Rollout Playbook

1. Ramp: 5% → 25% → 50% → 100% over [days]
2. Monitoring: check metrics daily during ramp
3. Rollback trigger: guardrail breach or unexpected anomaly

Inputs

Input	Required	Format
Change description	Yes	What is being tested
Baseline metric	Yes	Current value
MDE target	Yes	Percentage or absolute
Daily traffic	Yes	Number of users/events
Test duration budget	Recommended	Max days willing to run

Output

## A/B Test Plan — [Test Name]

### Hypothesis
If we simplify the checkout form from 5 fields to 3 fields,
then checkout completion rate will increase by ≥ 5%,
because user research shows 40% abandon at the address step.

### Metrics

| Type | Metric | Baseline | Target |
|------|--------|----------|--------|
| Primary | Checkout completion rate | 45% | ≥ 50% |
| Secondary | Average order value | $65 | Stable |
| Guardrail | Revenue per user | $12 | No decrease |
| Guardrail | Error rate | 0.5% | No increase |

### Sample Size

| Parameter | Value |
|-----------|-------|
| Baseline rate | 45% |
| MDE | 5% (absolute) |
| Significance (α) | 0.05 |
| Power (1−β) | 0.80 |
| Required n per variant | ~1,600 |
| Daily traffic | 800 users |
| Allocation | 50/50 |
| **Estimated duration** | **4 days** |

### Randomization
- Unit: User (cookie-based)
- Allocation: 50% control / 50% variant
- Exclusion: Internal users, users in other active tests

### Decision Rules
[As defined in procedure]

### Rollout Playbook
Day 1–2: 10% ramp → monitor → Day 3–4: 50% → Day 5: 100%

Definition of Done

• Hypothesis is clear and falsifiable
• Primary, secondary, and guardrail metrics defined
• Sample size calculated with stated parameters
• Duration estimated based on traffic
• Randomization plan documented
• Decision rules and rollout playbook included

Examples

Prompt

We want to test if a new onboarding flow increases activation rate.
Current activation: 32%. MDE: 3 percentage points. Daily new users: 500.
We have max 14 days for the test.
Design a complete A/B test plan with sample size and decision rules.

Quality Criteria

• Every finding is tied to a specific evidence source (log, test, metric)
• Pass/fail criteria are binary and measurable — no subjective judgments
• Severity levels are assigned with clear thresholds
• Remediation steps are provided for all critical and high findings

Verification (4C)

Check	Question
Correctness	Are all pass/fail criteria applied against the correct standard or rule?
Completeness	Were all required dimensions or checklist items evaluated?
Context-fit	Does the verification scope match the actual risk level of the deliverable?
Consequence	If this passed verification but had a hidden flaw, what is the worst-case impact?

Edge Cases

• Incomplete data for full assessment — Document which checks were limited and flag for re-verification when data becomes available.
• Ambiguous pass/fail criteria — Request clarification from the standard owner before scoring. Mark as 'Needs Review'.
• Multiple overlapping standards — Identify the governing standard and note where others diverge.

Changelog

• v1.0.0 — Initial release