ab-test-validity-checklist
Overview
This skill produces the pre-flight + post-flight validity
checklist for an A/B test. Each item is a gate; failing one
without explicit acknowledgment invalidates the experiment.
Per Kohavi et al. Trustworthy Online Controlled Experiments
(ISBN 978-1108724265): "More than 50% of experiments in practice
are invalidated by issues the checklist catches."
The output: a per-experiment markdown checklist + a sign-off
form for the experiment owner.
When to use
- Launching a new experiment.
- Auditing an experiment that produced surprising results.
- Building experimentation governance / a peer-review process.
- PR review of experiment configuration changes.
Step 1 - Pre-registration
Document before launch:
| Item | What |
|---|
| OEC | The single metric (or weighted combination) to improve |
| Power | Expected effect size, sample size, alpha, beta |
| Guardrails | Per experiment-results-interpreter references/guardrails.md - list each + threshold |
| Randomization unit | User / session / device / cookie / IP / tenant |
| Allocation | Percentages per arm; rules for ramp-up |
| Look schedule | Pre-declared days; per experiment-results-interpreter references/peeking.md |
| Sequential method | Fixed / Pocock / O'Brien-Fleming / always-valid |
| Stop-early rules | What signals stop (loss on OEC, blocking guardrail) |
Candidate guardrail set to pick from, with typical block
thresholds (per experiment-results-interpreter references/guardrails.md;
Kohavi et al. Trustworthy Online Controlled Experiments, ISBN
978-1108724265):
| Class | Guardrail | Direction | Typical block threshold |
|---|
| Quality (web) | TTFB, LCP, INP | not-increase | > 10% or > 50ms, whichever is greater |
| Quality (API) | p95 / p99 latency | not-increase | > 10% or > 50ms, whichever is greater |
| Quality (API) | error rate, 5xx rate | not-increase | +0.1pp absolute |
| Quality (mobile) | crash rate, ANR rate, app start time | not-increase | any statistically significant increase |
| Engagement | DAU, sessions / user, retention day 7 | not-decrease | -1% |
| Revenue | gross revenue, AOV, conversion | not-decrease | any statistically significant decrease |
| Trust | opt-out, complaint, refund rate | not-increase | any statistically significant increase |
Each guardrail carries two levels: alert (statistically
significant degradation - investigate before ship) and block
(past the pre-declared limit - ship-decision flips to "no"). Per
Kohavi et al., always include a quality guardrail (latency /
error); it is the most-missed category. Percentage-only thresholds
on fast endpoints are a trap - use max(%, absolute).
Commit this to the repo as experiments/<id>/proposal.yml.
Any post-launch change requires explicit team approval.
Step 2 - Sample Ratio Mismatch (SRM)
Per Microsoft Experimentation Platform research (KDD 2019 paper
"Diagnosing Sample Ratio Mismatch in Online Controlled
Experiments"): if the observed allocation (e.g., 50.3% A,
49.7% B) deviates significantly from intended (50% / 50%), the
experiment is invalid until root cause is found. SRM signals:
- Logging bugs (assignments not all logged)
- Bot filtering (different ratios filtered per arm)
- Redirects (one variant redirects more)
- Telemetry drops (one variant has heavier client → more drops)
- Randomisation bugs (hash collisions)
Chi-square test:
χ² = Σ ((observed_i - expected_i)² / expected_i)
For 2 arms at 50/50 with N=1e6 users:
- Expected: 500k each
- Observed: 503k / 497k
- χ² = (3000²/500000) + (3000²/500000) = 36
- p-value: < 0.0001
Threshold: p < 0.0001 is the canonical SRM-detection
boundary (the chi-square is super-sensitive at large N; this
threshold prevents false-positive SRM alarms).
If SRM is detected: stop ship discussion; root-cause first.
Step 3 - Assignment integrity
Tests for the assignment SDK / service:
| Test | Pattern |
|---|
| Determinism | Same (user, experiment) → same arm across calls |
| Sticky assignment | User reassigned only if experiment reconfigured |
| Cross-experiment independence | Assignment to expt A doesn't bias expt B |
| Bot exclusion consistent | If bots filtered, filter applies before assignment |
| Latency | Assignment SDK adds < 5ms to request path |
These tests live in the per-vendor harnesses of
experiment-sdk-testing.
Step 4 - Telemetry correctness
Verify the event firing matches the proposal:
- Conversion events fire exactly once per user per
conversion-eligible session.
- Exposure events fire for everyone who could see the variant
(not just those who actually saw it - that's a different
measure, "treatment effect on the treated").
- Guardrail metrics are queryable against the experiment
partition (variant ID joined to event stream).
Step 5 - Peeking discipline
Per experiment-results-interpreter references/peeking.md:
| Rule | Test |
|---|
| If sequential / always-valid: p-value valid at any look | Dashboard p-value uses the valid math |
| If fixed-horizon: no early-stop UI | "Ship" button disabled until N reached |
| If Pocock/OBF: look schedule pre-declared | Dashboards lock looks outside the schedule |
Step 6 - Novelty / primacy effects
Per Kohavi et al.: users react differently to novel UX. Novelty
inflates the early-period effect; primacy depresses it. Mitigation:
- Run for ≥ 2 weeks (typical mature-effect period).
- Segment results by "first exposure vs returning to treatment."
- For long-running tests, segment by week to spot trend
reversal.
Step 7 - Post-experiment validation
Before ship:
| Gate | Pass criterion |
|---|
| Pre-registration honoured | OEC / guardrails / unit / schedule unchanged since launch |
| SRM clean | p > 0.0001 on the chi-square (Step 2) |
| OEC significant under the declared method | Sequential / always-valid / fixed-horizon p-value |
| All guardrails within thresholds | Each Step 1 guardrail below its block threshold; per experiment-results-interpreter references/guardrails.md |
| Multiple-comparison corrected | Bonferroni / BH if many metrics |
| Novelty assessment | Effect persisting in week 2+ |
| Segment-stability | Effect direction consistent across major segments (no Simpson's paradox) |
| Trust metric stable | Opt-out / complaint rate not up |
Document each pass in experiments/<id>/result.md with the
specific numbers.
Step 8 - Emit the checklist
The output of this skill: a markdown checklist + sign-off form.
# Experiment <id> - Validity Checklist
## Pre-registration (signed by: <owner>, date: <YYYY-MM-DD>)
- [ ] OEC declared: <metric>
- [ ] Power calc: N=<X>, alpha=0.05, beta=0.20, MDE=<Y>%
- [ ] Guardrails declared: <list with thresholds>
- [ ] Randomization unit: <user_id / device_id>
- [ ] Allocation: <50/50>
- [ ] Look schedule: <Pocock 5 looks at days 2,4,7,10,14>
- [ ] Stop-early rules: <on OEC reaching alpha-threshold>
## During experiment
- [ ] SRM check: chi-square p > 0.0001 ([result: <p>])
- [ ] Assignment integrity tests passing
- [ ] Telemetry validated
## Post-experiment (signed by: <reviewer>, date: <YYYY-MM-DD>)
- [ ] Pre-registration honoured (no scope changes)
- [ ] SRM final check: p > 0.0001 ([result])
- [ ] OEC significant (p=<X>; method: <Pocock>)
- [ ] All guardrails within thresholds:
- api_p95_latency: +<X>% / +<Y>ms - <status>
- dau: <X>% - <status>
- [ ] Multiple-comparison adjusted (method: <Bonferroni / BH>)
- [ ] Novelty assessment: effect persists week 2+? <yes / no>
- [ ] Segment stability: direction consistent? <yes / no>
- [ ] Trust metric stable? <yes / no>
## Ship decision: <ship / no-ship / extend>
Reasoning: <one paragraph>
Sign-off: <name>, <date>
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|
| Post-hoc OEC change | "We found a better metric" = p-hacking | Pre-register |
| Skip SRM check | Invalidates results without detection | Always run chi-square pre-ship |
| Decision before checklist completion | Ship-then-validate is rejected by trusted-experiments framework | Block ship on incomplete checklist |
| Reviewer = experiment owner | Self-sign-off; no second pair of eyes | Different sign-off than owner |
| Skip novelty assessment | Effect disappears post-ship | Look at week-2+ subset |
| Skip segment stability | Simpson's paradox: total positive, per-segment negative | Audit by major segments |
| Treat the checklist as paperwork | Items checked without verification | Each item produces evidence (number, link, calc) |
Limitations
- Checklist is necessary, not sufficient. Quality of the
underlying telemetry + assignment logic matter.
- Multiple-comparison corrections are conservative. May reject
real wins.
- Novelty assessment needs ≥ 2 weeks. Pressure to ship fast
conflicts.
- Doesn't catch ecosystem effects. Cross-experiment
interaction, carry-over, etc. require global statistics.
References
- Kohavi, Tang, Xu. Trustworthy Online Controlled Experiments
(Cambridge Univ. Press, 2020). ISBN 978-1108724265.
- KDD 2019 paper "Diagnosing Sample Ratio Mismatch"
(Microsoft Research).
- Microsoft Experimentation Platform articles:
microsoft.com/en-us/research/group/experimentation-platform-exp/.
- Companion:
experiment-results-interpreter (result reading; peeking +
guardrail methodology in its references/).
- Operationalised:
experiment-sdk-testing (Statsig / Optimizely /
Split.io / Amplitude / VWO harnesses).