Cogsci Power Analysis

Cognitive Science Power Analysis

Purpose

This skill encodes domain-specific knowledge for planning adequately powered studies in cognitive science and neuroscience. It provides:

• Effect size priors calibrated to specific paradigms and modalities (behavioral, EEG/ERP, fMRI, clinical/developmental)
• Sample size recommendations grounded in empirical meta-analyses rather than arbitrary conventions
• Power analysis workflow guidance tailored to the design complexities of cognitive neuroscience (repeated measures, multilevel, neuroimaging-specific tools)

An AI agent needs this because generic power analysis advice (e.g., "use G*Power with d = 0.5") fails to capture the enormous variability in effect sizes across cognitive science paradigms, and because neuroimaging modalities have unique statistical considerations.

When to Use This Skill

• A researcher is designing a new behavioral, EEG, or fMRI experiment and needs sample size justification
• A grant proposal requires a power analysis section
• A preregistration document needs effect size justification and sample size rationale
• Someone asks "how many participants do I need?" for a cognitive/neuroscience study
• Reviewing whether a published study was adequately powered

Research Planning Protocol

Before executing the domain-specific steps below, you MUST:

1. State the research question — What study is being planned and what effect is being powered for?
2. Justify the method choice — Why this design and analysis approach? What alternatives were considered?
3. Declare expected outcomes — What is the smallest effect size of interest (SESOI)?
4. Note assumptions and limitations — What assumptions does this power analysis make? Where could it mislead?
5. Present the plan to the user and WAIT for confirmation before proceeding.

For detailed methodology guidance, see the research-literacy skill.

⚠️ Verification Notice

This skill was generated by AI from academic literature. All parameters, thresholds, and citations require independent verification before use in research. If you find errors, please open an issue.

Core Workflow

Step 1: Identify the Research Modality and Design

Determine which modality and design type apply:

Step 2: Obtain an Effect Size Prior

Do not use generic benchmarks (Cohen's "small/medium/large"). Instead:

1. Best option: Use a meta-analytic estimate for the specific paradigm. See references/effect-sizes.md for a curated library organized by modality.
2. Second option: Use the smallest effect size of interest (SESOI) — the minimum effect that would be theoretically or practically meaningful (Lakens, 2022).
3. Third option: Use pilot data, but apply shrinkage correction — pilot studies systematically overestimate effect sizes (Albers & Lakens, 2018).
4. Last resort: Use the modality-specific median effect sizes from large-scale meta-analyses (see below).

Modality-level median effect sizes (use only when paradigm-specific estimates are unavailable):

Critical warning: The median statistical power in neuroscience has been estimated at only 21% (Button et al., 2013, Nature Reviews Neuroscience). Many published effect sizes are inflated by publication bias. Always apply skepticism to effect sizes from underpowered, unreplicated studies.

Step 3: Conduct the Power Analysis

Choose method based on design complexity:

Simple Designs (t-test, one-way ANOVA, correlation)

Use analytic solutions via G*Power or pwr (R):

Target: 80% power (minimum) or 90% power (recommended)
Alpha: 0.05 (two-tailed unless directional hypothesis is justified)

• Two-sample t-test: pwr.t.test(d = effect_size, power = 0.80, sig.level = 0.05, type = "two.sample")
• Within-subjects t-test: pwr.t.test(d = effect_size_dz, power = 0.80, sig.level = 0.05, type = "paired")
• Correlation: pwr.r.test(r = effect_size, power = 0.80, sig.level = 0.05)

Complex Designs (mixed ANOVA, multilevel, mediation)

Use simulation-based power analysis:

• simr (R package): For linear mixed-effects models (Green & MacLeod, 2016)
• Superpower (R/Shiny): For factorial ANOVA designs (Lakens & Caldwell, 2021)
• Monte Carlo simulation: For non-standard designs — simulate data under the expected effect, run analysis, repeat 10,000+ times

Neuroimaging-Specific

• fMRIpower: Power for fMRI group analyses (Mumford & Nichols, 2008)
• NeuroPowerTools: Web-based fMRI power calculator (Durnez et al., 2016)
• For EEG/ERP: No standard tool; use simulation with expected component amplitudes and noise levels. See references/sample-size-guide.md for worked examples.

Step 4: Apply Modality-Specific Rules of Thumb

Use these as sanity checks, not replacements for formal power analysis:

Step 5: Document and Report

For preregistration and manuscripts, the power analysis section must include:

1. Effect size used and its source (meta-analysis, pilot, SESOI)
2. Power analysis method (analytic, simulation-based, tool used)
3. Target power level (80% or 90%) and alpha level
4. Resulting sample size and any adjustments (attrition, exclusion rate)
5. Sensitivity analysis: What is the minimum detectable effect at the planned N?

Template language:

"Based on the meta-analytic effect size of d = [X] reported by [Author, Year], a power analysis using [tool] indicated that N = [X] participants per group would be needed to detect this effect with [80/90]% power at alpha = .05 (two-tailed). Anticipating a [X]% attrition/exclusion rate, we plan to recruit N = [adjusted X]."