Conjoint Design Expert

Worked example (attribute table → power calculation → PAP tier assignment): see reference/example.md.

1. Is a Conjoint the Right Design?

• Identify the DGP Before Attributes: Articulate the data-generating process the theory implies: which component of the compound profile bears the causal effect of interest, and what respondents are assumed to do when bundling or separating information. This is the "Why → If-Then" step of hypothesis-building; the attribute table follows from it.
• When a Conjoint Is the Wrong Tool: Prefer another design when (a) the question concerns a single treatment dimension (a simple vignette experiment suffices), (b) the attributes cannot be independently randomized without producing incoherent profiles, or (c) the mechanism operates through holistic impression formation rather than attribute-by-attribute evaluation (Sniderman 2018).
• Between-Subjects to Conjoint Conversion: When a between-subjects vignette experiment suffers from confounds (outcomes not held constant across conditions), a conjoint (a) holds other features constant through independent randomization, (b) forces concrete content rather than vague descriptions, (c) gains power from repeated within-respondent measurements, and (d) enables estimation of interaction effects.

2. Attribute Architecture

• Independence: Randomize every attribute independently of every other so each component's causal effect is identified.
• Randomization of Order: Randomize attribute order at the respondent level, not the task level, to prevent primacy and recency effects without making respondents search for information in a new order on every task (Stantcheva 2023). A theoretically required logical flow may override this.
• D-Optimal Designs: Consider D-optimal or constrained randomization rather than pure randomization when the number of possible attribute combinations is large relative to the sample; these choose the administered conditions that maximize power (Auspurg & Hinz 2015; Stantcheva 2023).
• Attribute and Task Density: Bansak et al. (2018) find response quality does not degrade up to 30 tasks on MTurk; Bansak et al. (2021, "Beyond the Breaking Point") extend this to the attribute dimension; Stefanelli and Lukac (2020) cite stability up to 10 attributes. Still judge whether the complexity of the levels, not just their count, raises cognitive load.
• Nested/Constrained Randomization: Attributes need not all be fully crossed. Linking or nesting levels (e.g., origin countries nested within policy domain) is acceptable when (a) the nesting is theoretically justified, (b) the primary attributes of interest remain fully independently randomized, and (c) the analyst acknowledges that nested attributes cannot be cleanly separated from their parent (Auspurg & Hinz 2015).
• Attribute-Level Restrictions: Excluding implausible combinations is a judgment call, not a mandate. Eye-tracking evidence shows odd (incongruent or nonsensical) combinations have minimal, inconsistent effects on attention, search, and choice (Bansak & Jenke 2025), so decide on statistical, substantive, and theoretical grounds — e.g., whether the randomization distribution should reflect a real-world target distribution (De la Cuesta, Egami & Imai 2022) — rather than assumed cognitive burden. Document all restrictions in the PAP.
• Medium-Level Specificity: Describe levels "fully described but not overly described" (Sniderman 2018): concrete enough to be meaningful, not so specific that they introduce confounds. Neither "a policy that helps the economy" nor "a $2.3B infrastructure bill for Route 95 in Pennsylvania."

3. Statistical Power and Error Logic

• Effective N: N_eff = Respondents × Tasks × Profiles (the number of profile evaluations). Respondents and tasks are not interchangeable: within-respondent correlation means adding respondents improves precision more than adding tasks, so prioritize respondents (Stefanelli and Lukac 2020).
• Closed-Form Formula: SE(AMCE) ≈ $\sqrt{\text{Var}(Y) \times L / N_{\text{eff}}}$, where $L$ is the attribute's number of levels (Schuessler and Freitag 2020). Use it as a quick precision diagnostic.
• Interaction Power: In the balanced two-level-by-two-level case the interaction SE is about $\sqrt{2}$ times the main-effect SE, so interactions need roughly twice the sample of main AMCEs; the multiplier grows with the number of levels on each interacting attribute (Schuessler and Freitag 2020). Budget for this when interaction hypotheses are confirmatory.
• Empirical AMCE Benchmarks: Published AMCEs typically range from 0.02 to 0.10 (percentage-point changes in selection probability), median around 0.05; AMCEs above 0.15 are rare (Stefanelli and Lukac 2020). Use these to set the SESOI when no prior data exist.
• Cohen's d Does Not Apply: AMCEs are in percentage points of choice probability, not standard-deviation units; translating requires Var(Y), which depends on the task structure. Do not calibrate conjoint power with small/medium/large d benchmarks.
• Minimum Detectable Effect: Power the hardest contrast — the attribute with the most levels and the smallest expected AMCE — and report whether the MDE falls within the range of plausible AMCEs from prior literature.
• Levels-Power Tradeoff: Each added level reduces per-level N (4 → 5 levels cuts it by about 20%) with a corresponding precision loss (Schuessler and Freitag 2020). Add a level only when it is theoretically necessary.
• Low-N_eff Danger Zone (rule of thumb): Below ~3,000 effective profile evaluations, designs are at high risk of being underpowered for typical AMCEs (0.02–0.05). This threshold comes from plugging median benchmarks into the Schuessler and Freitag (2020) formula, not from a research finding; adjust for expected magnitude, levels, and design. Below it, report a sensitivity analysis of what effects can be detected.
• Type S and Type M Errors: At low power, significant estimates risk the wrong sign (Type S) and exaggerated magnitude (Type M), and both worsen steeply as power falls (Gelman and Carlin 2014). With a two-sided test at α = .05, a significant estimate overstates the true effect by about 1.4 times on average at 50% power, with a negligible sign risk; at 17% power the exaggeration is about 2.5 times and about 1 in 110 significant estimates has the wrong sign; at 8% power, about 4.8 times and 1 in 11. Base the design analysis on a plausible effect size from prior conjoints or theory, not on a pilot's noisy estimate.
• Multiple Testing: A conjoint estimates many AMCEs at once. Under the sharp null, a typical pipeline yields at least one significant AMCE in more than 90% of trials (Liu and Shiraito 2023) — the forking-paths and undisclosed-flexibility problem (Gelman and Loken 2014; Simmons, Nelson, and Simonsohn 2011). Pre-specify a correction across the full AMCE family: Bonferroni (most conservative), Benjamini-Hochberg (controls FDR, most lenient), or adaptive shrinkage (balanced; preferred in Liu and Shiraito's simulations). Report corrected and uncorrected results for confirmatory hypotheses.
• Compromise Power Analysis: When the respondent pool is fixed (hard-to-reach or elite samples), balance Type I and Type II error; an alpha above 0.05 can be defensible when it minimizes the combined error rate (Lakens 2025).
• Tools: cjpowR (Freitag 2021) or its Shiny app for simulation-based power curves for main effects and interactions. For a declare-diagnose-redesign workflow across estimands, diagnosands, and assignment schemes, use DeclareDesign (Blair, Cooper, Coppock, and Humphreys 2019).

4. Treatment Validation and Realism

• Experimental vs. Mundane Realism: Mundane realism (resemblance to real decisions) is neither necessary nor sufficient; what matters is experimental realism — whether the task creates the intended psychological state (Druckman 2022, citing Aronson and Carlsmith 1968). A tabular display can lack mundane realism yet succeed if respondents attend to and process the information.
• Attention and Salience: Even with internal validity secured, survey effects can amplify or reverse real-world effects because the survey compresses the consideration set (inflating AMCE magnitudes) and distorts relative attribute salience (which can flip signs) (Fu and Li 2024). Ask whether the table forces attention to attributes respondents would ignore in the real-world analog, and whether relative salience mirrors the target environment.
• Access, Attention, Processing: Validate that respondents (1) can see the attribute information, (2) read it, and (3) interpret it as intended. Attention checks and comprehension probes address 1–2; pilots and cognitive interviews address 3 (Druckman 2022).
• Names as Cues: Proper names, cultural referents, and country names can carry associations beyond the intended dimension. Pilot-test what respondents read into them.
• Practice Vignette: A non-experimental practice task before the conjoint block reduces learning effects in early tasks.
• Repeated Task for IRR: Plan to repeat the first task at the end of the block with left/right order reversed. This estimates intra-respondent reliability (about 77% agreement on average across Clayton et al.'s (2023) eight replications), respondents generally do not detect the repetition, and it enables bias-corrected AMCEs and marginal means via projoint. It costs one task and cannot be retrofitted after fielding.

5. Estimands and Estimation

• Estimand First: Define the unit-specific quantity, target population, and aggregation before selecting an estimator (Lundberg, Johnson, and Stewart 2021). State the reference level for every attribute.
• SESOI in AMCE Terms: For every confirmatory hypothesis, state the smallest percentage-point change in choice probability that would matter theoretically or practically, justified from prior conjoints, theory, or policy relevance (Lakens 2025).
• AMCE: The average change in selection probability when an attribute moves from the reference level to the level of interest (Hainmueller, Hopkins, and Yamamoto 2014). Baseline model: Y_itj = $\alpha_i$ + $\beta$(Attributes_tj) + $\varepsilon_{itj}$, respondent fixed effects, SEs clustered by respondent. The AMCE "critically relies upon the distribution of the other attributes" (De la Cuesta, Egami, and Imai 2022) — usually the uniform randomization distribution, which may not match the real world. When the target is a specific real-world or counterfactual distribution, use design-based weighting (e.g., factorEx).
• No Majoritarian Claims from AMCEs: The AMCE aggregates direction and intensity of preferences, so it does not identify the share of respondents who prefer a feature; "voters prefer X" is not supported by a significant AMCE without strong additional assumptions such as uncorrelated direction and intensity (Abramson, Koçak, and Magazinnik 2022). It is an effect on choice probability, not a parameter of the preference distribution (Ganter 2023). Valid readings include a change in expected vote share over the experimenter-defined contest distribution, a Borda-rule winner, or an average of individual ideal points. For majoritarian or electoral targets, use the bounding and structural approaches in Abramson et al. (2022) and Ganter (2023), and the outcome/estimand contingency analyses in Treger (2025).
• Marginal Means: Also report MMs — the predicted probability a profile is selected when a level is shown, averaged over other attributes. They give absolute support rather than relative differences and are the right basis for subgroup comparisons (Leeper, Hobolt, and Tilley 2020).
• Subgroup and Cross-Group Comparisons: Compare groups (e.g., countries) with MMs and differences in MMs; differences in conditional AMCEs depend on the reference category and can flip sign (Leeper, Hobolt, and Tilley 2020). A pooled model with group × attribute interactions supplies the formal heterogeneity test.
• Interactions via AMIE: Product terms in a dummy-coded AMCE regression are baseline-dependent artifacts that change with the reference category (Egami and Imai 2019). Use the Average Marginal Interaction Effect, AMIE(a, b) = ACE(a, b) − AME(a) − AME(b), which is invariant to the baseline. Estimate with penalized ANOVA in FindIt: CausalANOVA() with nway=2 (or 3), cv.CausalANOVA() to choose the regularization cost (1-SE rule), and AMIE() to decompose specific combinations; present AMIE matrices. Regularization handles the high dimensionality (5 attributes × 4 levels already yields 100+ interaction parameters) by shrinking weak interactions and collapsing similar levels. AMIEs apply to interactions among randomized attributes, not to attribute × respondent-characteristic moderation.
• Respondent Heterogeneity: To detect heterogeneity without pre-specified moderators:
  • BART (cjbart): Robinson and Duch (2024) fit a probit BART model (cjbart(), then IMCE()) to estimate Individual-level Marginal Component Effects (IMCEs) within an OMCE → IMCE → AMCE hierarchy. The IMCE distribution is the diagnostic: tight and centered on the AMCE suggests homogeneity; multimodal, skewed, or spanning zero indicates substantive heterogeneity. Report the AMCE with the IMCE standard deviation, and give individual IMCEs with their credible intervals (the standard deviation describes dispersion across respondents, not uncertainty about any one respondent's effect); het_vimp() ranks moderating covariates. Works reliably with 500+ respondents.
  • Bayesian mixture of regularized regressions (Goplerud, Imai, and Pashley 2025): groups respondents with similar effect patterns and models cluster membership with covariates. Prefer it when interpretable clusters and their moderators are the scientific target, or when hierarchical interaction structure should be enforced.
  • Either is exploratory unless moderators and directions were pre-registered (Robinson and Duch 2024).
• Profile-Context Heterogeneity (GML estimand): For configural claims ("the effect of X depends on the full bundle of who the candidate is"), designate one focal attribute and treat the rest as a context vector $\vec{x}$. The estimand $\theta(\vec{x})$ is the focal effect at context $\vec{x}$, and $\theta(\vec{x}{Max})$, $\theta(\vec{x}{Min})$ bound its range across all context combinations (Gosciak, Molitor, and Lundberg 2026). In non-adaptive data, $\theta(\vec{x}) = \Pr(\text{supported} \mid \text{focal} = A, \vec{x}) - \Pr(\text{supported} \mid \text{focal} = B, \vec{x})$ from cell means with respondent-clustered SEs, at the cost of thin cells in rare contexts; in an adaptive design (§6) it is the choice probability $\Pr(Y(\vec{x}, S, A) = A)$, and validation-phase estimates are unweighted means and asymptotically normal. AMIEs give the shape of low-order interactions and the CRT says whether any heterogeneity exists; only this estimand localizes it. Pre-register the focal attribute and any coarsening of context attributes if the analysis is confirmatory.
• Assumption-Free Factor Tests (CRT): The conditional randomization test in CRTConjoint (Ham, Imai, and Janson 2024) tests whether a factor matters in any way, including through interactions the AMCE averages over, and tests the profile-order, carryover, and fatigue assumptions behind standard AMCE estimation. It matters most when a narrow AMCE interval contains zero: that implies a weak marginal effect, not necessarily a weak total effect.
• Lexicographic Preferences: If the theory predicts categorical vetoes (always rejecting profiles with a given level), AMCEs and MMs can mislead because co-occurrence across task pairs inflates or deflates them. Use nested marginal means (Dill, Howlett, and Müller-Crepon 2024; cjRank): rank the most predictive attribute, compute MMs for the rest in tasks where it does not vary, and iterate. This shows whether apparently unimportant attributes are dominated by a veto attribute.
• Null Predictions: When an attribute is predicted to have no meaningful effect, use TOST equivalence testing with a pre-specified range in raw AMCE units rather than a non-significant p-value (Lakens 2025).
• Sensitivity Checks: Replicate all models on the rating outcome; consider specification curves across reference categories, models with and without respondent fixed effects, estimates by task number (fatigue), and a "stress test" with escalated levels when ceiling or floor effects appear (Dill, Howlett, and Müller-Crepon 2024). Report how estimates shift across outcome variables and estimands (AMCE, MM, rating-based) to characterize estimand contingency (Treger 2025).

6. Design Variants

• Forced Choice vs. Rating: Forced choice yields a binary DV for LPM estimation of AMCEs; ratings capture intensity. Default to forced choice as primary with ratings as a robustness check: in Hainmueller et al.'s (2015) validation against a Swiss citizenship referendum, paired-choice designs predicted real voting better than rating designs (Stantcheva 2023).
• Forced-Choice Caveat in Electoral Conjoints: Forced choice implicitly assumes no non-voters and no blank or protest votes, producing misclassification and external-validity bias even with unbiased estimators (Visconti and Yang 2024). When respondents would plausibly abstain in the real-world analog, consider an unforced design with "abstain" and "blank/null" options (see also Miller and Ziegler 2024 on preferential abstention), or at minimum report rating-based effects alongside forced-choice AMCEs.
• Factorial Vignettes: Attributes assembled into short paragraphs rather than tables enhance realism and engagement while preserving control (Auspurg & Hinz 2015), and can reduce social desirability bias because "characteristics can be smoothly hidden in stories" whereas tables make attributes salient (Stantcheva 2023). Suited to complex scenarios such as policy decisions. The claim that standard conjoints attenuate SDB is supported but bounded: Horiuchi, Markovich, and Yamamoto (2022) find reductions on some sensitive attributes but not others.
• Adaptive Randomization (focal/context conjoint): Gosciak, Molitor, and Lundberg (2026) port adaptive randomization (Thompson 1933; Offer-Westort, Coppock, and Green 2021; Kasy and Sautmann 2021) to conjoints whose argument is about one attribute. Pick the focal attribute ex ante; within a pair, hold the context $\vec{x}$ identical so profiles differ only on the focal attribute. Three phases: (1) warm-up with uniform allocation across contexts; (2) adaptive Thompson sampling that concentrates sample on candidate $\vec{x}{Max}$ and $\vec{x}{Min}$ (sequential searches are more efficient than parallel); (3) validation with fixed allocation at the two selected contexts, which defeats the winner's curse and sidesteps most adaptive-inference machinery (Hadad et al. 2021). Constraints:
  • One pair per respondent. Multiple tasks would expose the design, so there is no within-respondent leverage; GML's immigration replication needed 10,000 respondents for what a standard conjoint recovers with about 1,000. This is the central cost, and it cuts against Bansak et al. (2018) on task counts.
  • Two signals per context value (e.g., "Germany" and "Poland" for "Europe"), randomly permuted within pairs so the profiles are not literally identical. Choose substantively similar signals without differential social-desirability load; $\theta(\vec{x})$ stays identified even if signals have effects.
  • Low-dimensional context space (8–32 enumerable cells). A 7-attribute, 4-level conjoint must be coarsened hard before fielding; document the theoretical basis, since reviewers will challenge it. Use it when the claim is configural about one focal attribute, the context space can be honestly coarsened, the budget supports ~5,000+ respondents, and there is one country and platform. Otherwise keep the standard design and recover the GML estimand from cell means (§5). Qualtrics does not support continuous Thompson-sampling updates; deploy with github.com/dmolitor/adaptive-infra, with replication code at github.com/jennahgosciak/adaptive_conjoint.

7. PAP Tiers and Reporting

• Tiered PAP: Conjoints invite many analytic choices (which attributes to interact, how to subset MMs, which moderators to test). Separate (a) locked specifications (primary tests), (b) conditional specifications run only if stated conditions hold (e.g., "if the main effect is significant, test these interactions"), and (c) exploratory analyses labeled as hypothesis-generating. pre-registration-writing supplies registry selection, tier templates, and contingency trees.
• Cross-Attribute Comparisons: Comparing AMCEs of qualitatively different attributes (gender vs. education) is meaningful only if their levels span comparable ranges of the underlying dimension; AMCEs change with the levels chosen.
• Reporting Standards: Conjoint PAPs and methods sections follow APSA Experimental Section, JARS, and DA-RT. Audit with methods-reporting (attribute list, randomization scheme, restrictions, sample flow, estimator, SEs, weights, replication materials); the conjoint items here slot into that checklist rather than replacing it.

Quality Checks

• DGP and Fit: Is the DGP stated, and is a conjoint (rather than a single-dimension vignette) justified?
• Independence and Order: Are levels randomized independently, and attribute order randomized at the respondent level?
• Restrictions and Nesting: Are restrictions, nesting, and the handling of odd combinations documented and justified on statistical, theoretical, or target-distribution grounds rather than assumed cognitive burden (Bansak & Jenke 2025; De la Cuesta, Egami & Imai 2022)?
• Power: Is power computed for the hardest contrast, with respondents prioritized over tasks, the MDE checked against empirical AMCE benchmarks, and confirmatory interactions budgeted (~2× in the 2×2 case)?
• SESOI and Baselines: Is the smallest meaningful AMCE justified for each confirmatory hypothesis, and is every reference category stated?
• Treatment Validation: Were levels pilot-tested for access, attention, and intended interpretation?
• IRR: Is a repeated, order-reversed task included for measurement-error correction (Clayton et al. 2023)?
• Estimands: Is the AMCE marginalizing distribution documented and compared with real-world distributions, with MMs reported and subgroup contrasts made on MMs?
• No Majoritarian Claims: Are AMCEs framed without "voters prefer X" claims, or are the Abramson, Koçak, and Magazinnik (2022) bounds applied if such claims are central?
• Interactions: Are theoretically motivated interactions pre-specified and estimated as AMIEs (Egami and Imai 2019)?
• Heterogeneity: Is the detection method (cjbart, mixture model, or pre-specified subgroups) stated, with results labeled confirmatory or exploratory?
• Configural Claims: If the central claim is configural, are $\theta(\vec{x}{Max})$ and $\theta(\vec{x}{Min})$ reported or an adaptive design considered; if adaptive, are the one-pair, two-signal, and coarsening choices justified (Gosciak, Molitor, and Lundberg 2026)?
• Lexicographic Check: If veto preferences are predicted, are nested marginal means used (Dill, Howlett, and Müller-Crepon 2024)?
• Assumption Tests: Are CRT tests of profile-order, carryover, and fatigue planned or reported (Ham, Imai, and Janson 2024)?
• Multiple Testing: Is a correction pre-specified for the AMCE family (Liu and Shiraito 2023)?
• DV and Format: Is the primary DV identified with secondary DVs as robustness checks; is forced vs. unforced choice considered for electoral designs (Visconti and Yang 2024); is the vignette-vs-table SDB trade-off justified; and do assembled vignettes read coherently for all combinations?
• PAP Tiers: Does the PAP separate locked, conditional, and exploratory specifications?