bayesian-statistics

Overview

Bayesian statistics treats parameters as random variables and updates beliefs with data: posterior ∝ likelihood × prior. This skill covers choosing priors that regularize without dictating, building likelihoods that respect the data-generating process, computing posteriors (MCMC, variational), checking models (posterior predictive checks), and comparing models without fooling yourself.

When to use

• Estimating parameters with proper uncertainty quantification
• Incorporating prior knowledge or handling small datasets where priors matter
• Hierarchical/multilevel modeling: partial pooling across groups
• Comparing models: Bayes factors, LOO-CV, WAIC
• Reporting Bayesian results so others can interpret and reproduce them

Core concepts

• Bayes' theorem: p(θ|y) ∝ p(y|θ)·p(θ) — the posterior balances what the data say (likelihood) against what was plausible before (prior); with enough data the likelihood dominates.

• Priors: weakly informative (regularize, keep computation stable — the modern default), informative (real prior knowledge — justify and sensitivity-test), and "noninformative" (often informative in disguise — check what they imply on the observable scale).

• Hierarchical models: parameters drawn from group-level distributions — partial pooling: groups borrow strength, extreme small-sample estimates shrink toward the mean. The single most useful Bayesian modeling pattern.

• Posterior predictive checks: simulate data from the fitted model and compare with observed data — systematic misfit means the model is wrong, however well it converged.

• Model comparison: LOO-CV/WAIC estimate out-of-sample predictive accuracy; Bayes factors quantify relative evidence but are exquisitely prior-sensitive — report with the priors that produced them.

• Decision framing: point estimates (posterior mean/median), intervals (credible intervals — and HDI vs equal-tailed), and loss functions — match the summary to the decision.

• Prior predictive checks: simulate data from the prior alone — if the prior generates absurd data, it's an absurd prior regardless of its mathematical convenience; tune until prior predictions are plausible-but-broad.

• The folk theorem of statistical computing: computational problems often signal model problems — persistent divergences or poor mixing usually mean the model needs reparameterization or better priors, not more iterations.

• Bayesian workflow (Gelman): build → fit → check (posterior predictive) → compare → expand — iteration is the method, not a sign of failure; document the iterations.

Practical workflow

1. Build the model generatively

1. Write the data-generating story first: how would you simulate this data? That story is the model.
2. Choose the likelihood for the measurement process: counts → Poisson/negative-binomial (overdispersion is the norm), proportions → binomial/beta-binomial, positive continuous → lognormal/Gamma, not Normal-by-default.
3. Set weakly informative priors on a standardized scale; prior predictive checks (simulate from priors alone) should produce plausible-but-broad data — absurd prior predictions mean absurd priors.

2. Fit with MCMC

# PyMC sketch: hierarchical model with non-centered parameterization
# mu_j = mu + tau * z_j ; z_j ~ Normal(0,1)   # non-centered: kills funnels

1. Use non-centered parameterizations for hierarchical models — the centered form creates funnels that HMC can't explore.
2. Run 4+ chains; require R̂ < 1.01, good ESS, zero divergences (see monte-carlo-methods).
3. If divergences persist: reparameterize first, strengthen priors second, raise adapt_delta last.

3. Check the fit

1. Posterior predictive checks: overlay replicated datasets on the observed — check means, variances, extremes, and any statistic the science cares about.
2. Residual structure: patterns in residuals vs predictors or time mean missing structure — add it or acknowledge it.
3. Sensitivity: refit with wider/narrower priors on key parameters — conclusions that flip are prior-driven, say so.

4. Compare and report

1. Compare models with LOO-CV (arviz.compare) — report elpd differences with standard errors, not just rankings.
2. Report posteriors as distributions (plots + quantiles), not just point estimates; state priors explicitly — a Bayesian result without priors is irreproducible.
3. Distinguish credible intervals ("95% probability the parameter is here, given model+prior") from confidence intervals — and from prediction intervals for new data.

5. Build a hierarchical model correctly

1. Write the generative story: population distribution → group parameters → observations — partial pooling is the payoff: noisy small groups borrow strength from the population.
2. Use the non-centered parameterization (parameters as population mean + scale × standard normal offsets) — the centered form creates funnels that defeat HMC.
3. Check shrinkage: group estimates should pull toward the population mean proportionally to their uncertainty — no shrinkage means the hierarchy isn't working (or the groups are truly distinct).

6. Quick-reference checklist

• Generative story written before the model
• Likelihood matched to the data type (counts, proportions, positive continuous)
• Priors weakly informative with prior predictive checks done
• Non-centered parameterization for hierarchical models
• ≥4 chains, R̂ < 1.01, good ESS, zero divergences
• Posterior predictive checks performed and reported
• Prior sensitivity tested on key conclusions
• Priors stated explicitly in every report

Common pitfalls

• Priors chosen by convenience: default flat priors on transformed scales can be strongly informative — always inspect prior implications.
• Likelihood misspecification: Normal likelihood on skewed/count data — the posterior is precisely wrong; match the likelihood to the data type.
• Ignoring hierarchical structure: complete pooling (ignores group differences) vs no pooling (noisy group estimates) — partial pooling dominates both.
• Bayes factors without prior sensitivity: a BF of 100 that becomes 2 under a reasonable prior change is not evidence — it's prior artifact.
• Convergence theater: R̂ < 1.01 on a misspecified model — diagnostics check computation, not correctness; predictive checks check correctness.
• Posterior hacking: trying models until the credible interval excludes zero — preregister the model or report the multiverse.
• Centered parameterization of hierarchical models: the funnel geometry stalls HMC and produces divergences — non-center by default, center only with strong group-level data.
• Posterior predictive checks skipped: a well-converged chain on a misspecified model is precisely wrong — predictive checks are part of fitting, not optional extras.