Parameter Recovery Checker

Parameter Recovery Checker

Purpose

This skill encodes expert methodological knowledge for conducting parameter recovery studies -- a critical validation step before interpreting fitted model parameters. Parameter recovery determines whether a model's parameters are identifiable given the experimental design and sample size. A general-purpose programmer unfamiliar with computational modeling would not know that fitting a model is insufficient validation, or how to diagnose parameter tradeoffs and non-identifiability.

When to Use This Skill

• Before trusting fitted parameter values from any computational cognitive model
• When developing a new model and assessing whether parameters can be distinguished from data
• When planning an experiment and determining the minimum trial count for reliable parameter estimation
• When a reviewer asks for evidence of model identifiability
• When comparing models and needing to ensure each model can be distinguished (model recovery)
• When fitted parameters produce suspiciously extreme values or hit bounds

Research Planning Protocol

Before executing the domain-specific steps below, you MUST:

1. State the research question -- What specific question is this analysis/paradigm addressing?
2. Justify the method choice -- Why is this approach appropriate? What alternatives were considered?
3. Declare expected outcomes -- What results would support vs. refute the hypothesis?
4. Note assumptions and limitations -- What does this method assume? Where could it mislead?
5. Present the plan to the user and WAIT for confirmation before proceeding.

For detailed methodology guidance, see the research-literacy skill.

⚠️ Verification Notice

This skill was generated by AI from academic literature. All parameters, thresholds, and citations require independent verification before use in research. If you find errors, please open an issue.

Why Parameter Recovery Matters

Fitting a model to data and obtaining parameter estimates does NOT guarantee those estimates are meaningful (Wilson & Collins, 2019; Navarro, 2019). Common failure modes:

1. Non-identifiability: Multiple parameter combinations produce identical model predictions (e.g., drift rate and boundary in DDM trade off; Ratcliff & Tuerlinckx, 2002)
2. Insufficient data: Too few trials for the fitting procedure to recover true values
3. Local minima: Optimization converges to wrong parameter values
4. Model misspecification: The fitting procedure recovers parameters that do not reflect the assumed cognitive process

Parameter recovery is the standard diagnostic for these problems (Heathcote et al., 2015; Wilson & Collins, 2019).

Step-by-Step Recovery Procedure

Step 1: Define the Parameter Space

Choose ground-truth parameter values that span the plausible range for each parameter.

How many parameter sets to simulate?
 |
 +-- Minimum: 100 parameter sets (Wilson & Collins, 2019)
 |
 +-- Recommended: 500-1000 parameter sets for smooth recovery landscapes
 |
 +-- For publication: 1000+ parameter sets (Heathcote et al., 2015)

Sampling strategy:

Range selection: Use ranges from published parameter estimates in the domain. For example:

• DDM drift rate v: 0.5 -- 4.0 (Ratcliff & McKoon, 2008)
• DDM boundary a: 0.5 -- 2.5 (Ratcliff & McKoon, 2008)
• DDM non-decision time Ter: 0.1 -- 0.5 s (Ratcliff & McKoon, 2008)
• ACT-R activation noise s: 0.1 -- 0.8 (Anderson, 2007)

Step 2: Simulate Data

For each ground-truth parameter set:

1. Match the experimental design exactly -- Same number of trials, conditions, and structure as the real experiment
2. Use the same model -- The generative model must be identical to the model you will fit
3. Include realistic noise -- Use the model's noise mechanism (do not add external noise)
4. Store the ground-truth parameters for later comparison

Critical: The number of simulated trials per participant must match the actual experiment. Recovery with 10,000 trials tells you nothing about recovery with 100 trials (Wilson & Collins, 2019).

Step 3: Fit the Model to Simulated Data

Apply the exact same fitting procedure you use for real data:

• Same optimization algorithm (e.g., MLE, Bayesian, chi-square minimization)
• Same parameter bounds and constraints
• Same starting values or initialization strategy
• Same convergence criteria

Multiple starting points: Run the optimizer from at least 5-10 random starting points per simulated dataset to avoid local minima (Heathcote et al., 2015).

Step 4: Evaluate Recovery Quality

Compare recovered parameters to true (ground-truth) parameters using multiple metrics.

Primary Metrics

Visualization (essential)

1. Scatter plot: Recovered vs. true for each parameter (identity line = perfect recovery)
2. Bland-Altman plot: Difference vs. mean (detect range-dependent bias)
3. Parameter correlation matrix: Off-diagonal correlations reveal tradeoffs

See references/recovery-diagnostics.md for visualization templates.

Step 5: Check Parameter Tradeoffs

Correlation between recovered parameters:

Are any pairs of recovered parameters correlated |r| > 0.5?
 |
 +-- YES --> These parameters trade off. Consider:
 | - Fixing one to a theoretically motivated value
 | - Reparameterizing the model
 | - Collecting more data to improve identifiability
 | - Reporting the tradeoff and interpreting cautiously
 |
 +-- NO --> Parameters are identifiable given this design

Common parameter tradeoffs in cognitive models:

Model Recovery (Confusion Matrix)

Model recovery extends parameter recovery to test whether the correct model can be identified from data (Wagenmakers et al., 2004).

Procedure

1. For each candidate model M_k (k = 1, ..., K): a. Simulate data from M_k with representative parameters b. Fit ALL candidate models to the simulated data c. Select the best-fitting model using your comparison metric (AIC, BIC, Bayes factor)
2. Construct a K x K confusion matrix: rows = generating model, columns = selected model
3. Diagonal entries should dominate (correct model selected)

Quality Criteria

Warning: If model A is selected when data are generated from model B more than 20% of the time, those models are not distinguishable with your experimental design (Wilson & Collins, 2019).

Sample Size Effects

How Trial Count Affects Recovery

Recovery quality improves with more trials per participant. Test recovery at multiple trial counts:

Source: Wilson & Collins (2019); Ratcliff & Tuerlinckx (2002) for DDM-specific guidance.