experiment-results-interpreter
Overview
The ab-test-validity-checklist skill confirms that an experiment
was run correctly - clean SRM, honest peeking discipline, pre-declared
OEC. This skill covers the next question: given a valid experiment,
what does the result actually mean, and is it safe to ship?
The two most common failure modes at this stage, per Kohavi, Tang, Xu
Trustworthy Online Controlled Experiments (Cambridge Univ. Press,
2020, ISBN 9781108724265), are:
- Shipping a result that is statistically significant but not
practically meaningful.
- Shipping a result that will not persist because it reflects
novelty, primacy, or interaction artefacts rather than genuine
long-term user value.
This skill is a pure reference for data scientists and PMs reading
final experiment dashboards.
When to use
- Reading the results of an experiment that has reached its pre-declared
sample size or end date.
- Preparing the ship/no-ship decision document.
- Auditing a past ship decision that produced unexpected post-launch
outcomes.
- Coaching a PM or analyst who is over-indexing on p-values or
under-checking guardrails.
How to use
Work through the six interpretation steps in order. Each step has a
hard stop: if a step blocks, do not proceed to the next.
Step 1 - Practical significance before statistical significance
A statistically significant effect can still be too small to matter:
results "may be statistically significant but not practically
significant" (Nielsen Norman Group, sources).
The minimum practically significant effect (MPSE) must be declared
in the pre-registration (proposal.yml). At read-time:
| Question | How to answer |
|---|
| Is the point estimate above the MPSE? | Compare OEC lift to the pre-declared threshold |
| Is the confidence interval entirely above the MPSE? | If the lower bound falls below the MPSE, treat as inconclusive |
| Would a 0.1% conversion lift justify the maintenance cost? | Engineering and product judgement, not statistics |
A statistically significant result with a point estimate well below the
MPSE is a no-ship unless the maintenance cost is zero and the
direction is consistent with strategy.
Step 2 - Confidence intervals, not just p-values
A 95% confidence interval gives the plausible range of the true effect -
"both directionality and magnitude of effects simultaneously" (Statsig,
sources) - which a binary p-value cannot.
Reading a result:
| CI position | Interpretation |
|---|
| Entirely above zero and above MPSE | Strong positive - candidate for ship |
| Entirely above zero, partially below MPSE | Positive but magnitude uncertain - extend or accept lower bound as the working estimate |
| Crosses zero | Inconclusive; do not ship on this signal |
| Entirely below zero | Negative treatment effect - do not ship |
Width matters. A narrow CI means high power and a precise estimate;
a wide CI means the experiment was underpowered, so extending runtime or
pooling more traffic will narrow it. Variance-reduction techniques (CUPED)
narrow the CI without inflating the false-positive rate (Microsoft ExP,
sources) - prefer platforms that apply it by
default.
Do not convert CI edges back to p-values to decide - the CI is the
complete picture.
Step 3 - Novelty and primacy effects
A statistically and practically significant result in week 1 may not
persist. Two opposing artefacts corrupt early-period estimates:
Novelty effect: users react to the mere newness of a change;
engagement (clicks, session length) inflates above the true long-run
level, then decays.
Primacy effect (resistance to change): new UI or workflows initially
hurt task-completion because users must relearn existing habits; the
treatment appears worse early, then improves as users adapt. Kohavi et
al. note novelty and primacy are "significant causes of treatment effects
changing over time but are not the sole causes"
(sources).
Detection and mitigation:
| Signal | Method |
|---|
| Week 1 lift much larger than week 2+ | Segment metric by experiment week; compute week-over-week trend |
| Effect reversal after ship | Look for Kendall's tau trending toward zero over 14+ day window |
| New-user cohort differs from returning-user cohort | Segment by first_exposure_date - new users see no novelty decay |
Microsoft ExP found "14-day surprises" - the second week's estimate
falling outside the first week's 3-sigma CI - at roughly 4% of
experiments, far more than the theoretical rate
(sources). Minimum run time: two full weeks
before drawing ship conclusions from experiments that change UI patterns;
feature launches with no UX learning curve may need only one week.
Step 4 - Interaction effects
An experiment running concurrently with other experiments may have its
treatment effect inflated, deflated, or reversed by interference.
Two types:
Between-experiment interaction: variant A of experiment X and
variant B of experiment Y are assigned to overlapping user populations.
If the two treatments interact (positively or negatively), the OEC
measured for X is partly caused by Y's presence (Microsoft ExP, "A/B
Interactions: A Call to Relax", sources).
Treatment spillover: a social or marketplace product where treating
some users changes outcomes for untreated users in the same experiment
(network effects). The control group is contaminated; the measured
effect is attenuated. Kohavi et al. (ISBN 9781108724265) categorise
this as a stable unit treatment value assumption (SUTVA) violation.
Detection checklist:
| Check | Pass criterion |
|---|
| Concurrent experiment audit | List all experiments running in the same user population during the experiment window |
| Mutual-exclusion / interaction check | For each concurrent experiment: did assignment overlap create a joint condition that was never intended? |
| SUTVA plausibility | Is the metric a per-user metric (e.g., clicks) or a network metric (e.g., messages sent to others)? Network metrics need holdout or cluster-level randomisation |
If a significant interaction is identified, the measured effect is
confounded. Options: (a) isolate with mutual exclusion and re-run,
(b) include the interaction term in a factorial model, (c) block ship
pending analysis.
Step 5 - Simpson's paradox
The aggregate OEC lift may be positive while every segment shows
a negative or neutral lift - or vice versa. This is Simpson's paradox.
Simpson's paradox is when "a trend appears in several groups of data but
disappears or reverses when the groups are combined" (Wikipedia,
sources).
In A/B testing, it surfaces when:
- Traffic allocation differs across segments (e.g., mobile gets 60% of
control but 40% of treatment due to a holdout policy or ramp-up).
- The OEC baseline differs strongly across segments (e.g., power users
convert at 12%, new users at 2%).
- The treatment effect differs in sign across segments.
Detection:
For each major segment (device, country, user-cohort, new vs returning):
1. Compute per-segment OEC lift and CI.
2. Verify direction is consistent with the aggregate result.
3. Check that per-segment traffic allocation matches the overall ratio.
If direction flips in a large segment: the aggregate result is
misleading. Segment-level results are the truth; the aggregate is an
artefact of unequal allocation. Do not ship on a positive aggregate
with a negative segment that represents > 20% of users.
The ab-test-validity-checklist Step 7 includes a "segment-stability"
gate for this reason - this skill provides the interpretive depth behind
that gate.
Step 6 - Guardrail check before ship
Per references/guardrails.md: no ship decision
is valid without confirming that no guardrail metric has breached its block
threshold. (Guardrail declaration, thresholds, and multiple-comparison
correction live there and in
references/thresholds-and-corrections.md;
the peeking discipline behind the "reached its pre-declared end" premise is
references/peeking.md.)
Ordered check:
1. Load the guardrail dashboard for the experiment.
2. For each declared guardrail metric:
a. Is the observed change within the alert threshold? (investigate, but not blocked)
b. Does the observed change breach the block threshold? (STOP - no-ship)
3. If any guardrail is on alert: document the finding and make an
explicit call (accepted risk + rationale OR extend experiment).
4. If all guardrails are within alert thresholds: proceed to ship.
Common guardrail check failures before ship:
| Anti-pattern | Consequence |
|---|
| OEC positive, latency guardrail in alert band, ship anyway | Regression ships; support tickets spike |
| Checking guardrails at 80% of sample (early) | Underpowered - guardrail CIs wide; false safe signal |
| Ignoring guardrails with wide CIs because "p > 0.05" | Wide CI is not clearance; it means underpowered, not unaffected |
| Trust guardrail omitted (opt-out rate) | Long-term retention damage, not captured by OEC |
Measuring only the OEC can "disregard important information" (Nielsen
Norman Group, sources) - always check guardrails
alongside it.
Example
Scenario: Redesigned onboarding flow experiment. Declared OEC:
7-day activation rate. MPSE: +0.5pp absolute. Alpha 0.05, 80% power.
Ran 14 days. Result reads: lift = +1.2pp (95% CI: +0.3pp, +2.1pp).
Step 1 - Practical significance: Point estimate +1.2pp > MPSE
+0.5pp. Lower CI bound +0.3pp is below MPSE. Minimum realistic effect
is 0.3pp - marginal. Discuss with product whether 0.3pp justifies the
complexity.
Step 2 - CI read: CI entirely above zero; statistically significant.
Width (1.8pp) is moderate. Acceptable - not underpowered.
Step 3 - Novelty check: Week 1 lift was +2.1pp; week 2 lift was
+0.9pp. Declining trend. Novelty effect likely inflating week 1. Use
week 2 estimate (+0.9pp) as the stable-state estimate - still above
MPSE.
Step 4 - Interaction: No other experiment running in onboarding
funnel. SUTVA: metric is per-user (not network). Clear.
Step 5 - Simpson's: Mobile segment (45% of traffic): lift +0.7pp.
Desktop segment (55%): lift +1.6pp. Directions consistent; no paradox.
Step 6 - Guardrails: API p95 latency +3% (alert threshold +5% or
+50ms block). Within alert, no breach. DAU stable. Opt-out rate flat.
All green.
Ship decision: Ship, citing week-2 stable estimate +0.9pp and
clean guardrails. Document novelty decay in the ship note.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|
| Ship on week-1 lift alone | Novelty effect inflates early results; may revert | Run at least 2 weeks; compare week 1 vs week 2 |
| Treat p-value < 0.05 as "the result" | Binary; ignores magnitude, direction, and CI width | Read the CI; compare CI to MPSE |
| Skip segment analysis | Simpson's paradox hidden in aggregate | Always segment by device, new vs returning, country |
| Ignore guardrail alerts as "not significant" | Wide CI is not clearance | Investigate every alert before ship |
| Ship on practical but not statistical significance | Effect may be noise at that magnitude | Wait for power target |
| Treat post-ship metric as experiment validation | Observational data after ship mixes causation | Experiment result is causal; post-ship is not |
| Combine result across concurrent experiments without interaction check | Confounded OEC | Audit the concurrent experiment list |
| Ship "because the direction is right" on a CI that crosses zero | Inconclusive result | Extend runtime or accept null |
Limitations
- This skill does not validate the experiment harness. SRM,
telemetry, and peeking discipline belong in
ab-test-validity-checklist.
This skill assumes the harness is valid.
- Novelty / primacy detection requires two or more weeks of data.
Products under launch pressure may not have it.
- Interaction effects are hard to detect without factorial design.
Audit-based detection (Step 4) is heuristic, not conclusive.
- Simpson's paradox analysis surfaces the descriptive pattern,
not the causal explanation. Requires domain knowledge to resolve.
- Guardrail thresholds are pre-declared. Post-hoc threshold
adjustment to clear a guardrail is p-hacking; this skill cannot
prevent that without governance enforcement.
References
Full source attributions and verbatim quotes:
references/sources.md. Primary source is Kohavi,
Tang, Xu, Trustworthy Online Controlled Experiments (Cambridge Univ.
Press, 2020, ISBN 9781108724265). Deep methodology:
references/peeking.md,
references/guardrails.md,
references/thresholds-and-corrections.md;
companion catalog ab-test-validity-checklist.