ab-test-validity-checklist

Overview

This skill produces the pre-flight + post-flight validity checklist for an A/B test. Each item is a gate; failing one without explicit acknowledgment invalidates the experiment.

Per Kohavi et al. Trustworthy Online Controlled Experiments (ISBN 978-1108724265): "More than 50% of experiments in practice are invalidated by issues the checklist catches."

The output: a per-experiment markdown checklist + a sign-off form for the experiment owner.

When to use

• Launching a new experiment.
• Auditing an experiment that produced surprising results.
• Building experimentation governance / a peer-review process.
• PR review of experiment configuration changes.

Step 1 - Pre-registration

Document before launch:

Item	What
OEC	The single metric (or weighted combination) to improve
Power	Expected effect size, sample size, alpha, beta
Guardrails	Per experiment-results-interpreter references/guardrails.md - list each + threshold
Randomization unit	User / session / device / cookie / IP / tenant
Allocation	Percentages per arm; rules for ramp-up
Look schedule	Pre-declared days; per experiment-results-interpreter references/peeking.md
Sequential method	Fixed / Pocock / O'Brien-Fleming / always-valid
Stop-early rules	What signals stop (loss on OEC, blocking guardrail)

Candidate guardrail set to pick from, with typical block thresholds (per experiment-results-interpreter references/guardrails.md; Kohavi et al. Trustworthy Online Controlled Experiments, ISBN 978-1108724265):

Class	Guardrail	Direction	Typical block threshold
Quality (web)	TTFB, LCP, INP	not-increase	> 10% or > 50ms, whichever is greater
Quality (API)	p95 / p99 latency	not-increase	> 10% or > 50ms, whichever is greater
Quality (API)	error rate, 5xx rate	not-increase	+0.1pp absolute
Quality (mobile)	crash rate, ANR rate, app start time	not-increase	any statistically significant increase
Engagement	DAU, sessions / user, retention day 7	not-decrease	-1%
Revenue	gross revenue, AOV, conversion	not-decrease	any statistically significant decrease
Trust	opt-out, complaint, refund rate	not-increase	any statistically significant increase

Each guardrail carries two levels: alert (statistically significant degradation - investigate before ship) and block (past the pre-declared limit - ship-decision flips to "no"). Per Kohavi et al., always include a quality guardrail (latency / error); it is the most-missed category. Percentage-only thresholds on fast endpoints are a trap - use max(%, absolute).

Commit this to the repo as experiments/<id>/proposal.yml. Any post-launch change requires explicit team approval.

Step 2 - Sample Ratio Mismatch (SRM)

Per Microsoft Experimentation Platform research (KDD 2019 paper "Diagnosing Sample Ratio Mismatch in Online Controlled Experiments"): if the observed allocation (e.g., 50.3% A, 49.7% B) deviates significantly from intended (50% / 50%), the experiment is invalid until root cause is found. SRM signals:

• Logging bugs (assignments not all logged)
• Bot filtering (different ratios filtered per arm)
• Redirects (one variant redirects more)
• Telemetry drops (one variant has heavier client → more drops)
• Randomisation bugs (hash collisions)

Chi-square test:

χ² = Σ ((observed_i - expected_i)² / expected_i)

For 2 arms at 50/50 with N=1e6 users:

• Expected: 500k each
• Observed: 503k / 497k
• χ² = (3000²/500000) + (3000²/500000) = 36
• p-value: < 0.0001

Threshold: p < 0.0001 is the canonical SRM-detection boundary (the chi-square is super-sensitive at large N; this threshold prevents false-positive SRM alarms).

If SRM is detected: stop ship discussion; root-cause first.

Step 3 - Assignment integrity

Tests for the assignment SDK / service:

Test	Pattern
Determinism	Same (user, experiment) → same arm across calls
Sticky assignment	User reassigned only if experiment reconfigured
Cross-experiment independence	Assignment to expt A doesn't bias expt B
Bot exclusion consistent	If bots filtered, filter applies before assignment
Latency	Assignment SDK adds < 5ms to request path

These tests live in the per-vendor harnesses of experiment-sdk-testing.

Step 4 - Telemetry correctness

Verify the event firing matches the proposal:

• Conversion events fire exactly once per user per conversion-eligible session.
• Exposure events fire for everyone who could see the variant (not just those who actually saw it - that's a different measure, "treatment effect on the treated").
• Guardrail metrics are queryable against the experiment partition (variant ID joined to event stream).

Step 5 - Peeking discipline

Per experiment-results-interpreter references/peeking.md:

Rule	Test
If sequential / always-valid: p-value valid at any look	Dashboard p-value uses the valid math
If fixed-horizon: no early-stop UI	"Ship" button disabled until N reached
If Pocock/OBF: look schedule pre-declared	Dashboards lock looks outside the schedule

Step 6 - Novelty / primacy effects

Per Kohavi et al.: users react differently to novel UX. Novelty inflates the early-period effect; primacy depresses it. Mitigation:

• Run for ≥ 2 weeks (typical mature-effect period).
• Segment results by "first exposure vs returning to treatment."
• For long-running tests, segment by week to spot trend reversal.

Step 7 - Post-experiment validation

Before ship:

Gate	Pass criterion
Pre-registration honoured	OEC / guardrails / unit / schedule unchanged since launch
SRM clean	p > 0.0001 on the chi-square (Step 2)
OEC significant under the declared method	Sequential / always-valid / fixed-horizon p-value
All guardrails within thresholds	Each Step 1 guardrail below its block threshold; per experiment-results-interpreter references/guardrails.md
Multiple-comparison corrected	Bonferroni / BH if many metrics
Novelty assessment	Effect persisting in week 2+
Segment-stability	Effect direction consistent across major segments (no Simpson's paradox)
Trust metric stable	Opt-out / complaint rate not up

Document each pass in experiments/<id>/result.md with the specific numbers.

Step 8 - Emit the checklist

The output of this skill: a markdown checklist + sign-off form.

# Experiment <id> - Validity Checklist

## Pre-registration (signed by: <owner>, date: <YYYY-MM-DD>)

- [ ] OEC declared: <metric>
- [ ] Power calc: N=<X>, alpha=0.05, beta=0.20, MDE=<Y>%
- [ ] Guardrails declared: <list with thresholds>
- [ ] Randomization unit: <user_id / device_id>
- [ ] Allocation: <50/50>
- [ ] Look schedule: <Pocock 5 looks at days 2,4,7,10,14>
- [ ] Stop-early rules: <on OEC reaching alpha-threshold>

## During experiment

- [ ] SRM check: chi-square p > 0.0001 ([result: <p>])
- [ ] Assignment integrity tests passing
- [ ] Telemetry validated

## Post-experiment (signed by: <reviewer>, date: <YYYY-MM-DD>)

- [ ] Pre-registration honoured (no scope changes)
- [ ] SRM final check: p > 0.0001 ([result])
- [ ] OEC significant (p=<X>; method: <Pocock>)
- [ ] All guardrails within thresholds:
   - api_p95_latency: +<X>% / +<Y>ms - <status>
   - dau: <X>% - <status>
- [ ] Multiple-comparison adjusted (method: <Bonferroni / BH>)
- [ ] Novelty assessment: effect persists week 2+? <yes / no>
- [ ] Segment stability: direction consistent? <yes / no>
- [ ] Trust metric stable? <yes / no>

## Ship decision: <ship / no-ship / extend>

Reasoning: <one paragraph>

Sign-off: <name>, <date>

Anti-patterns

Anti-pattern	Why it fails	Fix
Post-hoc OEC change	"We found a better metric" = p-hacking	Pre-register
Skip SRM check	Invalidates results without detection	Always run chi-square pre-ship
Decision before checklist completion	Ship-then-validate is rejected by trusted-experiments framework	Block ship on incomplete checklist
Reviewer = experiment owner	Self-sign-off; no second pair of eyes	Different sign-off than owner
Skip novelty assessment	Effect disappears post-ship	Look at week-2+ subset
Skip segment stability	Simpson's paradox: total positive, per-segment negative	Audit by major segments
Treat the checklist as paperwork	Items checked without verification	Each item produces evidence (number, link, calc)

Limitations

• Checklist is necessary, not sufficient. Quality of the underlying telemetry + assignment logic matter.
• Multiple-comparison corrections are conservative. May reject real wins.
• Novelty assessment needs ≥ 2 weeks. Pressure to ship fast conflicts.
• Doesn't catch ecosystem effects. Cross-experiment interaction, carry-over, etc. require global statistics.

References

• Kohavi, Tang, Xu. Trustworthy Online Controlled Experiments (Cambridge Univ. Press, 2020). ISBN 978-1108724265.
• KDD 2019 paper "Diagnosing Sample Ratio Mismatch" (Microsoft Research).
• Microsoft Experimentation Platform articles: microsoft.com/en-us/research/group/experimentation-platform-exp/.
• Companion: experiment-results-interpreter (result reading; peeking + guardrail methodology in its references/).
• Operationalised: experiment-sdk-testing (Statsig / Optimizely / Split.io / Amplitude / VWO harnesses).