Research Literacy

Research Literacy

Purpose

AI agents tend to execute analysis steps immediately without planning or justification. In research, every analysis decision needs a rationale grounded in theory, design, and data characteristics. This skill encodes the basic scientific thinking that should precede any domain-specific action.

A competent programmer without research training will typically: (a) pick a familiar method rather than the appropriate one, (b) skip assumption checks, (c) interpret results without considering alternative explanations, and (d) make undisclosed analytic choices that inflate false positive rates. This skill exists to prevent all four failure modes.

When to Use

• Before or alongside any domain-specific skill from this project (e.g., before running an ERP analysis, first formulate the research question and justify the method).
• Standalone when planning a study, reviewing an analysis pipeline, or interpreting results.
• Whenever an analysis involves researcher degrees of freedom — choices that could have been made differently and would affect the outcome.

---

⚠️ Verification Notice

This skill was generated by AI from academic literature. All parameters, thresholds, and citations require independent verification before use in research. If you find errors, please open an issue.

Research Question Formulation

From Vague Idea to Testable Hypothesis

A research question must be specific, falsifiable, and operationalized before any data analysis begins.

1. Start with the phenomenon: What behavior, neural signal, or cognitive process are you interested in?
2. Identify the gap: What is unknown or contested in the existing literature?
3. Formulate as a directional or non-directional prediction: Specify the expected relationship between variables.
4. Operationalize: Define how each construct is measured and what constitutes evidence for or against the hypothesis.

The PICOS Framework for Cognitive Science

Adapted from evidence-based medicine, PICOS structures research questions systematically:

Exploratory vs. Confirmatory Research

This distinction is critical for valid inference (Wagenmakers et al., 2012):

• Confirmatory research tests a pre-specified hypothesis. Statistical tests (p-values, confidence intervals) are only valid in this context. Requires preregistration of hypotheses and analysis plan.
• Exploratory research generates hypotheses from data. Results are descriptive and hypothesis-generating, not hypothesis-testing. Statistical tests in exploratory work should be interpreted as descriptive, not inferential.
• Mixing the two without disclosure is a primary driver of the replication crisis (Nosek et al., 2018). If you discover a pattern in the data and then test it in the same dataset, the resulting p-value is not valid.

Rule: Always declare whether an analysis is confirmatory or exploratory before executing it. If the analysis plan changed after seeing the data, label it exploratory.

---

Method Selection Justification

Match Question Type to Analysis Family

Decision Criteria for Method Selection

When choosing a method, consider and document the following:

1. Data type: Continuous, ordinal, categorical, count? This constrains the model family.
2. Design structure: Between-subjects, within-subjects, mixed? Nested or crossed random effects? This determines the error structure.
3. Sample size: Is N sufficient for the chosen method? Underpowered studies waste resources and inflate effect size estimates (Button et al., 2013). See references/common-assumptions.md for method-specific guidance.
4. Assumption profile: Does the data meet the method's assumptions? See references/common-assumptions.md.
5. Multiple comparisons: How many tests will be performed? What correction is appropriate? (Benjamini & Hochberg, 1995, for FDR; Bonferroni for strict family-wise control; cluster-based permutation for neuroimaging, Maris & Oostenveld, 2007).

The "Method Hammer" Anti-Pattern

"If all you have is a hammer, everything looks like a nail."

This anti-pattern occurs when a researcher applies the method they are most comfortable with, regardless of whether it is appropriate. Examples:

• Using a t-test when the design has multiple crossed factors (requires ANOVA or mixed model)
• Applying parametric tests to ordinal Likert data without justification
• Using mass-univariate analysis when the research question is about distributed patterns (requires MVPA)
• Defaulting to frequentist tests when the question is about evidence for the null (requires Bayesian analysis or equivalence testing)

Rule: Always articulate why THIS method and not alternatives. Document the alternatives considered and why they were rejected.

---

Expected Outcomes Declaration

Before running any analysis, declare what each possible outcome means:

The Three-Outcome Framework

1. If H1 is supported: What specific pattern of results would you expect? (e.g., "a significant interaction between condition and group, with a larger N400 for incongruent trials in the control group but not the patient group")
2. If H0 is supported: What would the data look like? (e.g., "no significant effects, Bayes factor favoring H0 > 3")
3. If results are ambiguous: What would be inconclusive? (e.g., "a trend-level effect, p = .05-.10, with a small effect size below the smallest effect of interest")

Why This Matters

Declaring expected outcomes in advance prevents:

• HARKing (Hypothesizing After Results are Known): presenting post-hoc hypotheses as if they were a priori predictions (Kerr, 1998). A survey of researchers found that 43% self-reported HARKing at least once (Fiedler & Schwarz, 2016).
• Post-hoc rationalization: finding a plausible story for any result after the fact.
• Outcome switching: changing the primary outcome measure after seeing which one yields significant results.

---

Assumptions and Limitations Awareness

Every Method Has Assumptions

No statistical method is assumption-free. Before applying any method, identify its key assumptions and check them. The full reference table is in references/common-assumptions.md.

Common Assumption Categories

1. Independence: Observations are not systematically related to each other. Violated by: repeated measures, clustered data, spatial/temporal autocorrelation in neural data.
2. Normality: The sampling distribution of the test statistic is normal. Often confused with normality of raw data. Relevant for small samples; large samples benefit from the central limit theorem.
3. Homogeneity of variance: Variance is equal across groups or conditions. Violated when group sizes are unequal and variances differ. Use Welch's correction or robust methods.
4. Stationarity: Statistical properties do not change over time. Relevant for EEG, fMRI time series. Violated by habituation, fatigue, scanner drift.
5. Measurement validity: The measure actually captures the construct of interest. No statistical test can fix a bad measure. Construct validity must be argued on theoretical grounds.
6. Correct model specification: The statistical model matches the data-generating process. Omitted variables, wrong functional form, and incorrect random effects structure all threaten validity (Barr et al., 2013).

Limitations Are Not Optional

Every study has limitations. Common categories:

• Internal validity threats: confounds, demand characteristics, order effects
• External validity threats: limited sample demographics, artificial lab conditions
• Statistical conclusion validity: low power, violated assumptions, multiple comparisons
• Construct validity threats: impure measures, task impurity in neuropsychology

Rule: List limitations upfront, not as an afterthought. This is not a weakness; it is scientific rigor.

---

Human-in-the-Loop Principles

Why AI Agents Must Pause

Research involves judgment calls where reasonable experts disagree. These "researcher degrees of freedom" (Simmons et al., 2011) can inflate false positive rates from a nominal 5% to as high as 60% when left unchecked (Simmons et al., 2011). AI agents must not make these decisions silently.

Mandatory Pause Points

ALWAYS present the analysis plan and WAIT for user confirmation before proceeding at these decision points:

1. Participant or trial exclusion: "I propose excluding 3 participants based on [criterion]. Here is the exclusion rationale and the impact on sample size."
2. Outlier treatment: "These data points are [N] SDs from the mean. Options: (a) winsorize, (b) trim, (c) transform, (d) use robust methods, (e) retain. Each has different implications."
3. Multiple comparisons correction: "With [N] comparisons, I recommend [method]. Alternatives are [list]. The choice affects sensitivity and specificity as follows..."
4. Model specification: "I am fitting [model]. Key choices include [random effects structure, covariates, link function]. Here is why, and here are alternatives."
5. Data transformation: "The data violate [assumption]. I propose [transformation/alternative method]. This changes the interpretation as follows..."
6. Unexpected results: "The results do not match the predicted pattern. Before interpreting, consider: (a) the analysis may be wrong, (b) the hypothesis may be wrong, (c) there may be a confound."