Experiment Design

Experiment Design

Interview-driven design workflow producing design documents, power analysis scripts, and pre-analysis plans.

Modes

Default: Design. If user provides a .qsf file, auto-select Survey mode.

When to Use

• Designing a new experiment or survey
• Calculating required sample sizes
• Writing or auditing a pre-analysis plan
• Parsing a Qualtrics .qsf file to understand its structure
• Building a survey specification from a natural language description

When NOT to Use

• Running the analysis → data-analysis
• Auditing identification strategy for observational studies → causal-design
• Generating synthetic test data → synthetic-data

Shared References

• Method probing questions: shared/method-probing-questions.md — ask before designing (Experiments/RCTs, Survey sections)
• Validation tiers: shared/validation-tiers.md — tier determines required power and pre-registration
• Escalation protocol: shared/escalation-protocol.md — escalate when design has validity threats
• Engagement-stratified sampling: shared/engagement-stratified-sampling.md — stratify social media samples by engagement
• Inter-coder reliability: shared/intercoder-reliability.md — reliability planning for content analysis designs

---

Mode: Power

Read references/power-analysis-recipes.md for language-specific code patterns.

Workflow

1. Interview — ask for:
   • Primary outcome variable and expected effect size (or domain norms)
   • Design type (between-subjects, within-subjects, factorial, cluster-randomized)
   • Number of conditions/groups
   • Significance level (default: 0.05) and desired power (default: 0.80)
   • Any clustering or stratification
2. Generate script — R (DeclareDesign/pwr) or Python (statsmodels.stats.power)
3. Execute and report — produce a sample size table showing N for power = {0.80, 0.90, 0.95}
4. Write to project — save script to code/power_analysis.R (or .py), results to output/power_analysis_results.md

HPC escalation: If the power analysis uses Monte Carlo simulation (e.g., DeclareDesign with >10k replications, or a multi-design sweep), move execution to [HPC cluster] — drop the simulation script into hpc/ with templates/slurm/array.sbatch (array over seeds/designs). The SHA-logging snippet in the template pins results to the DGP version. See docs/guides/hpc.md.

Effect Size Guidance

If the user doesn't know the expected effect size, guide them:

Never default to Cohen's benchmarks without acknowledging they are arbitrary.

---

Mode: Design

Workflow

1. Research question interview — structured questions:
   • What is the causal question?
   • What is the treatment / intervention?
   • What is the primary outcome? Secondary outcomes?
   • What is the target population?
   • What is the assignment mechanism? (random, stratified, clustered, matched)
2. Design specification — produce a structured document covering:
   • Hypotheses (directional, with expected signs)
   • Conditions (treatment arms, control)
   • Randomization procedure
   • Outcome measures and scales
   • Sample and recruitment strategy
   • Timeline
3. Identification check — state the estimand, identifying assumptions, and potential threats
4. Write design document — save to docs/experiment-design.md or project-appropriate location
5. Lock the design — record in MEMORY.md Estimand Registry (if project has one) or flag for the user to lock before analysis

This design document is what data-analysis Phase 3 checks for before allowing estimation.

---

Mode: PAP

Read references/pap-template.md for the full template structure.

Workflow

1. Check for existing design — look for design document in docs/, log/plans/, or .context/
2. If no design exists — run Design mode first, then continue
3. Generate PAP — structured pre-analysis plan following AEA/OSF/EGAP conventions:
   • Study information (title, authors, IRB, registration)
   • Design overview (hypotheses, conditions, randomization, sample)
   • Data collection (instruments, timing, blinding)
   • Analysis plan (main specification, inference, multiple testing corrections)
   • Robustness and sensitivity analyses
   • Power analysis (reference or embed results from Power mode)
4. Output — save to docs/pre-analysis-plan.md (or .tex if user prefers LaTeX)
5. Registration prompt — remind user to register at AEA RCT Registry, OSF, or EGAP

---