Bayesian Cognitive Model Builder

Bayesian Cognitive Model Builder

Purpose

This skill encodes expert knowledge for building hierarchical Bayesian cognitive models using probabilistic programming languages (Stan, PyMC). It addresses the modeling decisions that require domain expertise beyond knowing Stan/PyMC syntax: how to choose priors that respect cognitive constraints, when to use hierarchical structure, how to diagnose MCMC pathologies, and how to evaluate model adequacy through posterior predictive checks.

A competent programmer without cognitive modeling training would get wrong: which prior families are appropriate for cognitive parameters (e.g., RT must be positive, learning rates bounded in [0,1]), when partial pooling outperforms complete pooling or no pooling, how to detect non-identifiability in cognitive models, and what constitutes adequate MCMC convergence for publishable results.

When to Use This Skill

• Building a generative model of a cognitive process (decision-making, learning, memory, perception) where parameters have psychological interpretations
• Estimating individual differences in cognitive parameters while borrowing strength across participants (hierarchical/multilevel models)
• Working with small samples or sparse data per participant where regularization through priors prevents overfitting
• Parameter uncertainty matters for your scientific conclusions (credible intervals, not just point estimates)
• Comparing competing cognitive models via information criteria (LOO-CV, WAIC) or Bayes factors
• Fitting established cognitive models (DDM, signal detection, reinforcement learning, multinomial processing trees) in a Bayesian framework

When NOT to Use This Skill

• If your model has a closed-form MLE and you have large, balanced samples, frequentist estimation may be simpler and adequate
• For purely predictive models where parameter interpretability is irrelevant (consider machine learning approaches)
• If you need a general-purpose Bayesian regression model without cognitive process parameters (see cogsci-statistics skill)
• For EEG/fMRI analysis pipelines without explicit cognitive models (see erp-analysis or fmri-glm-analysis-guide skills)

Research Planning Protocol

Before executing the domain-specific steps below, you MUST:

1. State the research question -- What cognitive mechanism is this model capturing?
2. Justify the method choice -- Why Bayesian (not MLE, not frequentist)? What alternatives were considered?
3. Declare expected outcomes -- What parameter patterns would support vs. refute the hypothesis?
4. Note assumptions and limitations -- What does this model assume about the cognitive process?
5. Present the plan to the user and WAIT for confirmation before proceeding.

For detailed methodology guidance, see the research-literacy skill.

⚠️ Verification Notice

This skill was generated by AI from academic literature. All parameters, thresholds, and citations require independent verification before use in research. If you find errors, please open an issue.

Model Structure Decision Tree

Choosing the right level of pooling is a fundamental modeling decision that a non-specialist routinely gets wrong.

Step 1: Do You Have Grouped Data?

If your data has a natural grouping structure (e.g., multiple trials per participant, participants within conditions), you need to decide on a pooling strategy. If not, fit a single model.

Step 2: Choose the Pooling Level

Critical domain knowledge: Hierarchical (partial pooling) models should be the default in cognitive science. They automatically regularize extreme individual estimates toward the group mean -- a property called "shrinkage" -- which is especially valuable with typical cognitive science sample sizes of 20-40 participants with 50-200 trials each (Lee & Wagenmakers, 2014, Ch. 8).

Step 3: Centered vs. Non-Centered Parameterization

For hierarchical models, the parameterization choice affects MCMC efficiency:

• Centered parameterization: theta_j ~ Normal(mu, sigma). Use when there are many observations per group (>100 trials per participant) and the data are informative relative to the prior (Betancourt & Girolami, 2015).
• Non-centered parameterization: theta_j = mu + sigma * eta_j where eta_j ~ Normal(0, 1). Use when there are few observations per group, the group-level variance is small, or you encounter divergent transitions with centered parameterization (Betancourt & Girolami, 2015; Stan User's Guide, Section 1.13).

When in doubt, use non-centered parameterization. It is more robust across a wider range of data configurations and is the Stan Development Team's default recommendation.

Prior Selection Principles

General Philosophy

Use weakly informative priors that encode known constraints without dominating the likelihood. The goal is to rule out impossible or implausible parameter values while remaining agnostic about the precise value (Gelman et al., 2008; Gelman et al., 2013, Ch. 2).

Domain-critical principle: Cognitive parameters have natural constraints that generic "flat" or "diffuse" priors violate. Reaction times cannot be negative. Probabilities must lie in [0,1]. Learning rates are bounded. Firing rates are non-negative. Encoding these constraints in the prior is not "being subjective" -- it is encoding physical and psychological reality (Lee & Wagenmakers, 2014, Ch. 4).

Prior Families for Common Cognitive Parameter Types

For detailed cognitive-domain-specific prior tables, see references/prior-selection-guide.md.

Prior Predictive Checking

Always run a prior predictive check before fitting to data (Schad et al., 2021; Gabry et al., 2019):

1. Sample parameters from your priors (no data)
2. Simulate data from the model using those parameters
3. Check: Does the simulated data look plausible for the domain?

• If the prior predicts impossible RTs (e.g., negative, or > 60 seconds), the prior is too diffuse
• If the prior predicts accuracy always near 50% or always near 100%, reconsider

4. Iterate on priors until prior predictive distributions cover plausible data ranges without including absurd values

Common Cognitive Models in Bayesian Framework

Drift-Diffusion Model (DDM / HDDM)

• Key parameters: drift rate (v), boundary separation (a), non-decision time (t), starting point bias (z)
• Bayesian implementation: HDDM package (Wiecki et al., 2013) uses informative priors from empirical meta-analysis (Matzke & Wagenmakers, 2009)
• Typical priors: See references/prior-selection-guide.md for parameter-specific recommendations
• Critical note: Within-trial noise (s) is a scaling parameter fixed by convention at 0.1 (Ratcliff, 1978) or 1.0 (Navarro & Fuss, 2009). All other parameter ranges depend on this choice.
• Also see the drift-diffusion-model skill for detailed DDM guidance

Signal Detection Theory (SDT)

• Key parameters: sensitivity (d'), criterion (c)
• Priors for d': Normal(0, 2) is weakly informative; typical empirical values range 0 to 4 (Macmillan & Creelman, 2005)
• Priors for c: Normal(0, 1.5) centered at no bias; typical range -2 to 2 (Macmillan & Creelman, 2005)
• Hierarchical structure: Individual d' and c drawn from group distributions (Rouder & Lu, 2005)
• Also see the signal-detection-analysis skill

Multinomial Processing Trees (MPT)

• Key parameters: Processing probabilities (all in [0,1])
• Priors: Beta(1,1) for non-informative or Beta(a,b) with shape informed by prior studies (Klauer, 2010)
• Hierarchical extension: Latent-trait MPT with probit-transformed parameters drawn from multivariate normal (Klauer, 2010)

Item Response Theory (IRT)

• Key parameters: Ability (theta), difficulty (b), discrimination (a)
• Priors: theta ~ Normal(0,1) by convention; b ~ Normal(0, 2); a ~ Lognormal(0, 0.5) to enforce positivity (de Boeck & Wilson, 2004)
• Cognitive application: Modeling learning, cognitive ability, or item difficulty in memory/attention tasks

Reinforcement Learning (RL)

• Key parameters: Learning rate (alpha in [0,1]), inverse temperature (beta > 0), decay, perseveration
• Priors for alpha: Beta(1.1, 1.1) weakly informative on [0,1] (Daw, 2011; Gershman, 2016)
• Priors for beta (inverse temperature): Gamma(2, 1) or Lognormal(0, 1) constraining to positive values; typical range 0.5 to 20 (Daw, 2011)
• Critical note: Learning rate and inverse temperature are often poorly identifiable in standard Q-learning; consider reparameterization or strong priors (Daw, 2011; Wilson & Collins, 2019)

MCMC Diagnostics

Every Bayesian analysis requires thorough convergence diagnostics. Never report posterior summaries without first verifying convergence. See references/diagnostics-checklist.md for the full step-by-step protocol.

Minimum Convergence Criteria

Critical domain knowledge: The older threshold of R-hat < 1.1 is outdated. Vehtari et al. (2021) demonstrated that the traditional R-hat can miss convergence failures. Use the rank-normalized split R-hat with a threshold of 1.01 and always report both bulk-ESS and tail-ESS.

When Diagnostics Fail

See references/diagnostics-checklist.md for remediation steps. The most common fixes in cognitive modeling:

1. Divergent transitions --> Switch to non-centered parameterization; increase adapt_delta to 0.95-0.99
2. Low ESS --> Run longer chains; check for multimodality; reparameterize
3. High R-hat --> Run more iterations; check for label switching in mixture models
4. E-BFMI warning --> Reparameterize; consider reducing model complexity

Model Comparison