Cogsci Statistics

Cognitive Science Statistical Analysis

Purpose

This skill encodes domain-specific statistical knowledge for cognitive science and neuroscience research. It addresses the modeling decisions, correction strategies, and reporting conventions that a general-purpose statistician or programmer would get wrong without training in the field. For concrete analysis recipes with code, see references/common-analyses.md.

When to Use This Skill

• Choosing between repeated-measures ANOVA and mixed-effects models for a cognitive experiment
• Specifying random effects structure for designs with subjects and items
• Deciding how to handle reaction time (RT) data distributions
• Selecting the appropriate multiple comparison correction
• Deciding whether to use frequentist or Bayesian analysis
• Reporting effect sizes and statistical results for journal submission

Research Planning Protocol

Before executing the domain-specific steps below, you MUST:

1. State the research question — What specific hypothesis is this statistical analysis testing?
2. Justify the method choice — Why this statistical model? What alternatives were considered?
3. Declare expected outcomes — What pattern of results would support vs. refute the hypothesis?
4. Note assumptions and limitations — What does this method assume? Where could it mislead?
5. Present the plan to the user and WAIT for confirmation before proceeding.

For detailed methodology guidance, see the research-literacy skill.

⚠️ Verification Notice

This skill was generated by AI from academic literature. All parameters, thresholds, and citations require independent verification before use in research. If you find errors, please open an issue.

Repeated-Measures ANOVA vs. Mixed-Effects Models

When to Use Repeated-Measures ANOVA

• Fully balanced design (no missing data, equal cell sizes)
• Only subjects as a random factor (no item variability)
• Simple factorial structure (2-3 factors, no continuous predictors)
• Sphericity is met or correctable (Greenhouse-Geisser / Huynh-Feldt)

When to Use Mixed-Effects Models (LMM/GLMM)

• Crossed random effects: Both subjects and items sampled from populations (Baayen, Clark, & Lucy, 2008; Clark, 1973). This is the norm in psycholinguistics, memory research, and any paradigm with stimulus variability.
• Unbalanced data or missing observations
• Continuous predictors (e.g., word frequency, stimulus duration)
• Non-normal response distributions (RT, accuracy)
• Need to generalize over both subjects AND items simultaneously

Critical domain knowledge: Clark (1973) demonstrated that failing to treat items as random effects inflates Type I error. This remains one of the most common statistical errors in cognitive science. If your stimuli are sampled from a larger population (e.g., words, faces, scenes), you must account for item variability.

Decision Logic

Are your stimuli sampled from a larger population?
 |
 +-- YES --> Mixed-effects model with crossed random effects
 | (subjects and items)
 |
 +-- NO (e.g., fixed set of 4 task conditions) -->
 |
 +-- Any missing data, unbalanced cells, or continuous predictors?
 | |
 | +-- YES --> Mixed-effects model (subjects as random effect)
 | |
 | +-- NO --> Repeated-measures ANOVA is acceptable
 |
 +-- Need trial-level analysis (e.g., RT distributions)?
 |
 +-- YES --> Mixed-effects model (operates on individual trials)
 +-- NO --> Repeated-measures ANOVA on condition means

Random Effects Structure

The Maximal Random Effects Principle

Barr et al. (2013) recommend fitting the maximal random effects structure justified by the design to minimize Type I error. This means including random intercepts and slopes for all within-unit factors.

For a typical 2x2 design with factors A (within-subjects, within-items) and B (within-subjects, between-items):

# Maximal structure (Barr et al., 2013)
lmer(RT ~ A * B + (1 + A * B | Subject) + (1 + A | Item), data = d)

When Maximal Models Fail to Converge

Convergence failures are common with complex random effects. Use this hierarchy (Barr et al., 2013; Matuschek et al., 2017):

1. First: Try a different optimizer (bobyqa, nlminb) with increased iterations (20000 iterations; lme4 default recommendation)
2. Second: Remove correlations between random effects (use || in lme4)
3. Third: Remove the highest-order random slopes first (interaction before main effects)
4. Fourth: Use a parsimonious approach guided by likelihood ratio tests (Matuschek et al., 2017)

Do NOT simply drop all random slopes to achieve convergence. This inflates Type I error and undermines the purpose of mixed-effects modeling (Barr et al., 2013).

Common Cognitive Science Designs and Their Random Effects

Handling Reaction Time Data

RT data in cognitive experiments are positively skewed, bounded below by physiological limits, and often contaminated by outliers. The approach matters.

RT Outlier Exclusion

Apply these criteria before modeling (Ratcliff, 1993; Luce, 1986):

Task-specific note: For simple RT tasks (e.g., detection), use 100 ms as the fast cutoff (Whelan, 2008). For choice RT tasks (e.g., lexical decision), use 200 ms (Ratcliff, 1993). Always report exclusion rates.

RT Transformation and Modeling Strategy

Is your primary interest in RT distributions (not just means)?
 |
 +-- YES --> Drift Diffusion Model or ex-Gaussian fitting
 |
 +-- NO --> Choose a modeling approach:
 |
 +-- Option 1: Log-transform RT, then fit LMM (Gaussian)
 | - Pro: Simple, widely understood
 | - Con: Back-transformation of means is biased;
 | changes the hypothesis being tested
 | (Lo & Andrews, 2015)
 |
 +-- Option 2: Inverse-transform RT (1/RT = speed), then LMM
 | - Pro: Often achieves better normality than log
 | - Con: Same back-transformation issues as log
 | (Ratcliff, 1993)
 |
 +-- Option 3 (Recommended): Generalized LMM with
 Gamma family + identity link
 - Pro: Models RT in original units; handles skew
 directly; avoids transformation issues
 (Lo & Andrews, 2015)
 - Con: Computationally slower; may have convergence
 issues with complex random effects

Recommended default: Gamma GLMM with identity link (Lo & Andrews, 2015). Report results on the original millisecond scale.

# Recommended RT model (Lo & Andrews, 2015)
glmer(RT ~ condition * group + (1 + condition | subj) + (1 | item),
 family = Gamma(link = "identity"), data = d)

Multiple Comparison Correction

Decision Guide for Cognitive Science

When NOT to Correct

• Single planned contrast testing a specific a priori hypothesis (Rubin, 2021)
• Sequential Bayesian testing with BF stopping rules (evidence accumulation replaces correction; Schoenbrodt et al., 2017)

Bayesian Alternatives

When to Use Bayesian Analysis

• Quantifying evidence for the null hypothesis: Frequentist tests cannot support H0; Bayes factors can (Wagenmakers, 2007)
• Small sample sizes: Bayesian methods with informative priors can be more efficient (Kruschke, 2015)
• Sequential testing: Bayes factors allow continuous monitoring without alpha inflation (Schoenbrodt et al., 2017)
• Complex models where p-values are unreliable: Mixed models with small cluster sizes, or when asymptotic assumptions are questionable

Bayes Factor Interpretation

Recommended Tools

Reporting Bayes Factors

Report the exact BF, not just the category (Wagenmakers et al., 2018):

"A Bayesian paired-samples t-test indicated moderate evidence for a difference between conditions, BF10 = 5.3 (default Cauchy prior, r = 0.707)."

Always specify:

1. The prior used (e.g., default Cauchy with scale r = 0.707 for BayesFactor t-test; Rouder et al., 2009)
2. Direction (BF10 = evidence for H1 over H0)
3. Robustness check: report BF across a range of prior widths

Effect Size Reporting

APA 7th Edition Requirements

APA 7th edition (2020, Section 6.6) requires reporting effect sizes for all primary analyses. The specific measure depends on the test:

Domain note: Always report confidence intervals around effect sizes (APA 7th, 2020). Use effectsize (R) or statsmodels (Python) for computation. The benchmarks above are Cohen's generic guidelines; paradigm-specific benchmarks are more informative (see ../cogsci-power-analysis/references/effect-sizes.md).

For Mixed-Effects Models

Traditional effect sizes are not straightforward for mixed models. Options:

1. Semi-partial R-squared via the r2glmm or effectsize package (Rights & Sterba, 2019)
2. Standardized regression coefficients: Standardize predictors before fitting
3. Conditional and marginal R-squared: R2m (fixed effects only) and R2c (fixed + random) via MuMIn::r.squaredGLMM() (Nakagawa & Schielzeth, 2013)

Common Statistical Mistakes in Cognitive Science

1. Treating Items as Fixed Effects

Problem: Analyzing condition means averaged over items, ignoring item variability, fails to generalize beyond the specific stimuli used (Clark, 1973).

Fix: Use mixed-effects models with crossed random effects for subjects and items.

2. Circular Analysis ("Double-Dipping") in Neuroimaging

Problem: Selecting voxels/channels/time-windows based on the effect of interest, then testing that same effect (Kriegeskorte et al., 2009). Inflates effect sizes by 2x or more (Vul et al., 2009).

Fix: Use independent localizer, leave-one-out cross-validation, or whole-brain corrected analysis.