Overview
Based on "The Art of Statistics" by David Spiegelhalter. The core principle: statistical significance is not the same as practical significance, and a poorly designed experiment produces confidently wrong answers. Rigorous experiment design means defining the question, estimating the required sample size, and setting success criteria before any data is collected - then interpreting results within the limits of what the experiment can actually prove.
Workflow
Step 1: State the hypothesis in falsifiable form
A hypothesis must specify: the intervention, the population, the outcome metric, and the expected direction.
Format: "Applying [intervention] to [population] will [increase/decrease] [metric] compared to the control condition."
Example: "Showing personalized recommendations to new users in the first session will increase 7-day retention compared to showing trending content."
Also state the null hypothesis explicitly: "There is no difference in 7-day retention between personalized and trending content groups."
Spiegelhalter's warning: if you cannot state what result would cause you to reject your hypothesis, you do not have a hypothesis - you have a belief.
Step 2: Identify the primary metric and guard metrics
Primary metric: The single number that determines whether the experiment succeeded. Choose one. Multiple primary metrics make interpretation ambiguous.
Guard metrics: Metrics that must not significantly worsen. If the primary metric improves but a guard metric degrades, the experiment fails.
Example:
- Primary: 7-day retention rate
- Guards: session length (must not drop > 5%), support ticket volume (must not rise > 10%)
Step 3: Calculate required sample size before running
Use a power calculation to determine how many observations you need. Running underpowered experiments wastes time and produces false negatives.
Inputs needed:
- Baseline rate: current value of the primary metric (e.g., 22% 7-day retention)
- Minimum detectable effect (MDE): smallest change that would be meaningful (e.g., 2 percentage points)
- Statistical significance level: alpha = 0.05 (5% false positive rate)
- Statistical power: 1 - beta = 0.80 (80% probability of detecting a true effect)
from statsmodels.stats.power import NormalIndPower
analysis = NormalIndPower()
n = analysis.solve_power(
effect_size=0.02 / (0.22 * (1 - 0.22)) ** 0.5,
alpha=0.05,
power=0.80,
alternative="two-sided"
)
print(f"Required n per group: {n:.0f}")
If the required sample size exceeds your available traffic within a reasonable window, increase the MDE or extend the timeline. Do not run an underpowered test.
Step 4: Design the randomization and control
Randomization unit must match the analysis unit.
- If analyzing per-user metrics: randomize by user ID
- If analyzing per-session metrics: randomize by session ID
- If analyzing geographic effects: randomize by region (cluster randomization)
Document:
- How users are assigned (hash of user ID mod 2, for example)
- Whether assignment is sticky (user always sees the same variant)
- Exclusion criteria (new users only? logged-in only? specific device types?)
Run a sanity check (A/A test) before the real experiment if possible - confirm the two groups are balanced on baseline characteristics.
Step 5: Set the stopping rule before launch
Never stop an experiment early because the results look good. Peeking inflates false positive rates.
Define in advance:
- Minimum runtime: run for at least 1 full business cycle (usually 1-2 weeks) to account for day-of-week effects
- Maximum runtime: set a hard end date
- Sample size target: stop when you hit the calculated N, not when the p-value crosses 0.05
If early stopping is operationally required, use a sequential testing method (e.g., always-valid p-values, group sequential design) instead of standard frequentist stopping rules.
Step 6: Analyze and report results with effect size, not just p-values
Spiegelhalter's emphasis: p-values tell you whether an effect exists; effect sizes tell you whether it matters.
Report:
- Observed difference (absolute and relative)
- 95% confidence interval for the difference
- p-value (as supporting evidence, not the conclusion)
- Practical significance: is the effect size large enough to act on?
from scipy import stats
import numpy as np
control = df[df["group"] == "control"]["converted"]
treatment = df[df["group"] == "treatment"]["converted"]
t_stat, p_value = stats.ttest_ind(control, treatment)
effect_size = treatment.mean() - control.mean()
ci = stats.t.interval(0.95, len(df)-2, loc=effect_size, scale=stats.sem(np.concatenate([control, treatment])))
print(f"Effect: {effect_size:.4f} ({effect_size/control.mean()*100:.1f}% relative lift)")
print(f"95% CI: ({ci[0]:.4f}, {ci[1]:.4f})")
print(f"p-value: {p_value:.4f}")
State the conclusion as: "The data [supports / does not support] rejecting the null hypothesis. The observed effect was [X], with a 95% CI of [Y to Z]. This [is / is not] practically significant because [reason]."
Anti-Patterns
1. Peeking and stopping early
Bad: Checking results daily and stopping as soon as p < 0.05.
Good: Pre-commit to a sample size and stop date. Do not look at significance until the stopping criterion is met.
2. Multiple primary metrics
Bad: "We'll call it a win if conversion OR retention OR revenue improves."
Good: One primary metric. Guard metrics are binary pass/fail, not additional wins.
3. Reporting only p-values
Bad: "The result was statistically significant (p = 0.03), so we should ship."
Good: "The result was significant (p = 0.03). The absolute lift was 0.4pp (95% CI: 0.1 to 0.7). This is below our 1pp MDE threshold, so we will not ship."
4. Ignoring novelty effects
Bad: Running an experiment for 3 days and seeing a spike from users trying something new.
Good: Running for at least 1-2 weeks and examining whether the effect size stabilizes over time.
Quality Checklist