Pre-Analysis Plan Writer

Standards anchor. This skill operationalizes the pre-data-collection component of DA-RT (Data Access and Research Transparency; see Druckman-Green 2021 ch. 18 §18.1.3 for the canonical political-science framing and the APSA Guide to Professional Ethics). DA-RT obligates researchers to facilitate evaluation of evidence-based claims through data access, production transparency, and analytic transparency; a PAP is how analytic transparency is established before the data arrive. Reporting downstream of the PAP is the domain of the methods-reporting skill (APSA Experimental Section guidelines via Gerber et al. 2014; JARS).

Instructions

1. Registry Selection

• EGAP is closed. The EGAP registry stopped accepting new registrations on October 15, 2023; existing EGAP registrations remain searchable through OSF. Send political-science PAPs to OSF Registries (which offers an "EGAP Registration" form template mirroring the old fields) or, for randomized experiments, to the AEA RCT Registry.
• OSF Registries: Free-form documents, file attachments (analysis code, stimuli), versioning, and embargo; timestamped and immutable once confirmed. Best for complex designs with supplementary materials.
• AEA RCT Registry: For randomized controlled trials, particularly in economics, development, and governance. The form is structured around RCT fields (intervention, outcomes, randomization unit, power); entries are public on submission.
• AsPredicted: For simple designs with few analytical degrees of freedom. A structured 9-question format; private until the authors make it public; requires an academic email.
• Registered Reports: The journal peer-reviews the introduction and methods before data collection; in-principle acceptance depends on design quality, and final acceptance on following the registered methods. Suggest this format when the question is important but results are uncertain or likely null, since it removes publication bias by design. It needs substantially more lead time than standard pre-registration.

2. PAP Document Structure

• Recommended Section Order: (1) study information (title, authors, timeline, registry), (2) theoretical motivation and hypotheses, (3) research design (experimental conditions, randomization, sample), (4) sampling and recruitment, (5) variable definitions and measurement, (6) analysis plan (models, tests, decision rules), and (7) contingency plans.
• Cross-Reference, Don't Repeat: For hypothesis specification, reference the three-level specification framework (conceptual, operationalized, statistical) from the hypothesis-building skill. For reporting elements, reference the JARS six elements from the methods-reporting skill. For experiment-type-specific design fields, reference the conjoint-design or survey-design skills (e.g., attribute architecture, wording protocols, attention checks). The PAP should implement these frameworks, not redefine them.
• Write for a Reader: Every analytical decision should be decidable from the PAP alone, in formal notation or code where possible, by a reader who does not know the authors' local practices. "The usual approach" is not a specification.

3. Specifying the Analytical Strategy

• Three-Tier Classification: Classify every planned analysis as locked (primary hypothesis tests -- cannot be changed), conditional (executed only if a pre-specified condition is met, e.g., "if the manipulation check passes, estimate the interaction model"), or exploratory (clearly labeled hypothesis-generating analyses with uncontrolled error rates). This mirrors the JARS primary/secondary/exploratory distinction (Lakens 2025 §13.4) and the confirmatory-exploratory continuum in Waldron & Allen (2022); it generalizes the conjoint-specific version from conjoint-design to all experimental designs.
• Decision Rules: For each confirmatory hypothesis, state in advance what constitutes support, falsification, or an inconclusive result. Specify: the test statistic, the alpha level, the SESOI, and the decision mapping (e.g., "If p < 0.05 and the coefficient exceeds 3 percentage points in the predicted direction, the hypothesis is supported; if the equivalence test rejects effects larger than 3 percentage points, the hypothesis is falsified; otherwise, the result is inconclusive"). Illustrative thresholds (e.g., the 3-percentage-point SESOI used above) must be justified from the user's own design, prior literature, and decision context -- they are not defaults.
• Exact Model Specifications: Write out every primary model in formal notation or code. For regression models, specify: the dependent variable, all independent variables, interaction terms, fixed effects, clustering structure, and the estimator. Ambiguous prose descriptions ("we will control for demographics") are insufficient -- name every variable. This is the remedy for the "garden of forking paths" problem (Gelman & Loken 2014): implicit analytical choices, even when made in good faith and before seeing results, inflate the false-positive rate unless pinned down in advance.
• Multiple Testing Corrections: Pre-specify the correction procedure and define which tests belong to the same family. For families of related tests (e.g., AMCEs across attributes within a single hypothesis), specify Benjamini-Hochberg (FDR control) or Bonferroni. Document the family groupings and the rationale for each.

4. Analysis Code Pre-Registration

• Simulated Data Approach: Generate a mock dataset that matches the expected data structure (variable names, types, distributions, sample size, missingness patterns). Write all analysis code -- data cleaning, primary models, robustness checks, planned figures -- to run on this simulated dataset. Register the code alongside the PAP. Lakens (2025 §13.6) calls this "the gold standard for a preregistration."
• Tooling for Simulated Data. For formal declare-design-diagnose workflows, use DeclareDesign (Blair, Cooper, Coppock, & Humphreys 2019, cited in Druckman-Green 2021 Table 18.1), which lets researchers specify the data-generating model, the inquiry, the data strategy, and the answer strategy and then diagnose the design before running it. Simpler simulations can use faux or simstudy in R or numpy + pandas in Python. The point is to produce a runnable pipeline, not a polished simulation.
• Why code, not just prose: Code settles decisions prose leaves open (how covariates are centered, what happens to observations missing one covariate but not others) and surfaces specification errors before data collection.
• What to Include: The registered code should cover: (1) data import and cleaning pipeline, (2) variable construction (indices, recodes, exclusion criteria), (3) primary confirmatory models, (4) conditional models with the triggering conditions, (5) planned figures with axis labels and titles, and (6) robustness checks. Exploratory code need not be pre-registered but should be documented post hoc.

5. Contingency Planning

Contingencies enumerate data-generating-process (DGP) failure modes -- manipulation failure, attrition, imbalance, under-recruitment -- and bind the response to each before the data are seen. All illustrative thresholds below (15% differential attrition, 0.10 standardized difference, etc.) are placeholders: justify thresholds from the user's own design, prior literature, and decision context.

• Manipulation Check Failure: If the manipulation check indicates that the treatment did not shift the intended construct, pre-specify the response: (a) report the ITT estimate regardless (it answers the policy-relevant question of what happens when the treatment is delivered), (b) investigate moderators of treatment uptake, and (c) do not selectively exclude non-compliers without a pre-specified rule.
• Excessive Attrition: Define an attrition threshold (e.g., >15% differential attrition across conditions) and the response if exceeded: (a) conduct attrition analysis (are attriters different from completers on observables?), (b) report Lee bounds on the treatment effect, (c) consider sensitivity analysis under alternative missing-data assumptions (MNAR models).
• Covariate Imbalance: Despite randomization, baseline imbalance may occur. Pre-specify: (a) which baseline variables will be checked, (b) the threshold for concern (e.g., a standardized difference > 0.10), and (c) the response (include the imbalanced covariate in the primary model as a robustness check, report both adjusted and unadjusted estimates).
• Sample Size Shortfall: If recruitment falls short of the target N, pre-specify: (a) the minimum N below which the study will not be analyzed (the threshold at which power drops below a defensible level), (b) a sensitivity analysis showing the MDE for the achieved N, and (c) whether the study will be reframed as exploratory if underpowered.
• Standard Operating Procedures (SOPs). Where the user or their lab has a written SOP (e.g., default attrition-handling or noncompliance rules in the Lin-Green tradition, discussed in Druckman-Green 2021 Table 18.1), cite the SOP version by filename and date and let it carry the default practice rather than restating rules inside every PAP. This lowers per-PAP cost and produces consistency across studies in the same program.
• Decision Trees: For complex contingencies, use explicit if-then decision trees: "If condition X holds, analyze using Model A; if condition X does not hold, analyze using Model B and report the deviation." This prevents post hoc rationalization of analytical choices.

6. Deviation Documentation

• Four-Part Deviation Record: Document every deviation from the registered plan with: (a) what changed (the specific analysis, variable, or procedure that differs from the PAP), (b) why it changed (the substantive or methodological reason), (c) the impact on severity -- a property of the test, does the deviation make the test more or less capable of falsifying the hypothesis? -- and (d) the impact on validity -- a property of the inference, does the deviation improve or degrade construct, internal, or external validity? The severity/validity distinction traces to the error-statistical (severe-testing) tradition; the concrete four-part protocol is Lakens (2025 §13.5).
• Principled vs. Convenience Deviations: Distinguish between principled deviations (fixing a validity problem discovered after registration, e.g., a measure that does not function as intended) and convenience deviations (changing the analysis because the pre-registered approach yields unfavorable results). Principled deviations can strengthen a study; convenience deviations undermine credibility.
• Side-by-Side Reporting: When a deviation occurs, report the pre-registered analysis alongside the deviated one, so readers can judge whether the deviation changes the conclusions. The deviated analysis never replaces the registered one.

7. Timeline and Logistics

• When to Register: Register after the design is finalized and the analysis code runs on simulated data, but before any data collection begins. For registered reports, submit for journal review before data collection. For standard pre-registration, the timestamp must precede the first survey response.
• Embargo Options: OSF Registries permits embargoes of up to 4 years, after which registrations become public (Lakens 2025 §13.8); use one when premature disclosure could enable scooping. AEA RCT Registry entries are public on submission; AsPredicted registrations stay private until the authors share them.
• Updating Registrations: If the design changes after registration (e.g., after pilot results), create a new version or a new registration that references the original, documenting what changed and why. The original stays intact as the timestamped audit trail.
• IRB Coordination: In many institutions, IRB approval is required before data collection but not before pre-registration. Register early, then amend the PAP if IRB review requires design changes. Include the IRB protocol number in the PAP once approved. Budget for the possibility that IRB review will take several weeks (4--8 weeks is an institutional rule-of-thumb, not a methodological derivation).
• PAP-to-Paper Mapping: After data collection and analysis, include a PAP concordance table in the paper's appendix -- a table mapping each PAP element to the corresponding section of the paper, with deviations flagged. This makes compliance auditable and signals transparency. The concordance is the author-side complement to the audit checklists in the methods-reporting skill (Wicherts et al. 2016; Gerber et al. 2014).

Quality Checks

• DA-RT Framing: Does the PAP explicitly situate itself within DA-RT / APSA transparency obligations (data access, production transparency, analytic transparency)?
• Registry Selected: Is the registry platform chosen and justified (OSF Registries, AEA RCT Registry, AsPredicted, or registered report)?
• Readable Structure: Does the PAP follow a logical section order that an independent researcher could navigate without contacting the authors?
• Three-Tier Classification: Is every planned analysis classified as locked, conditional, or exploratory (Lakens 2025 §13.4; Waldron & Allen 2022)?
• Decision Rules: Are support, falsification, and inconclusive criteria specified for each confirmatory hypothesis, with each threshold (alpha, SESOI, effect-size floor) justified rather than defaulted?
• Exact Models: Are all primary models written in formal notation or code, with every variable named -- closing off the forking-paths problem (Gelman & Loken 2014)?
• Multiple Testing: Is the correction procedure and family grouping pre-specified?
• Code Registered: Does analysis code run on a simulated dataset that matches the expected data structure (DeclareDesign or equivalent)?
• Contingencies Specified: Are contingency plans written for manipulation check failure, attrition, imbalance, and sample size shortfall, with reference to any governing SOPs, and complex contingencies expressed as if-then decision trees?
• Deviation Protocol: Is the four-part deviation documentation framework (what, why, severity, validity) committed to?
• Side-by-Side Commitment: Does the PAP commit to reporting both pre-registered and deviated analyses when deviations occur?
• Timeline: Is the registration timestamped before data collection, with IRB coordination planned?
• PAP Concordance: Does the plan include a commitment to a PAP-to-paper concordance table?
• Researcher-DF Audit: Has the PAP been checked against the Wicherts et al. (2016) 34-item researcher-degrees-of-freedom checklist (cross-referenced in the methods-reporting skill)?