Sentence Stimulus Norming

Sentence Stimulus Norming

Purpose

This skill encodes expert methodological knowledge for norming linguistic stimuli before running psycholinguistic experiments. A competent programmer without linguistics training would likely construct stimuli based on intuition, failing to control for critical lexical variables (word frequency, length, neighborhood density), skipping cloze norming, using inappropriate rating scales, or under-powering the norming study. Poor stimulus norming is the single most common methodological weakness in psycholinguistic research, because confounds in the materials propagate to every analysis.

When to Use

Use this skill when:

• Creating sentence stimuli for reading experiments (self-paced reading, eye-tracking, ERP)
• Norming the predictability (cloze probability) of critical words in sentence contexts
• Collecting plausibility, naturalness, or acceptability ratings for sentence materials
• Controlling lexical properties of critical words across experimental conditions
• Designing Latin square counterbalancing for within-item designs
• Planning filler items and practice trials

Do not use this skill when:

• Working with single-word stimuli without sentence context (use lexical database tools directly)
• Designing non-linguistic stimuli (visual search arrays, tones)
• Analyzing existing normed materials without creating new ones

Research Planning Protocol

Before executing the domain-specific steps below, you MUST:

1. State the research question -- What specific question is this analysis/paradigm addressing?
2. Justify the method choice -- Why is this approach appropriate? What alternatives were considered?
3. Declare expected outcomes -- What results would support vs. refute the hypothesis?
4. Note assumptions and limitations -- What does this method assume? Where could it mislead?
5. Present the plan to the user and WAIT for confirmation before proceeding.

For detailed methodology guidance, see the research-literacy skill.

⚠️ Verification Notice

This skill was generated by AI from academic literature. All parameters, thresholds, and citations require independent verification before use in research. If you find errors, please open an issue.

Cloze Probability Norming

What Is Cloze Probability?

Cloze probability is the proportion of people who complete a sentence fragment with a particular word (Taylor, 1953). It is the standard measure of a word's predictability in context and is a critical control variable in nearly all sentence processing research.

Procedure

1. Create sentence fragments: Truncate each sentence immediately before the critical word
2. Present fragments one at a time to participants
3. Instruct: "Please complete each sentence with the first word that comes to mind. Write only one word."
4. Score: For each item, cloze probability = (number of completions matching the target word) / (total number of respondents)

Design Parameters

Scoring Conventions

• Exact match: Only the target word counts (standard)
• Morphological variants: Decide a priori whether "run" and "running" count as the same completion. Standard practice: count only the exact form (Staub et al., 2015)
• Spelling errors: Accept obvious misspellings of the target
• Blank/nonsense responses: Exclude from the denominator (participant did not engage)

Cloze Probability Benchmarks

Online vs. Lab Norming

Recommendation for online norming: Include 10-15% catch trials (sentences with obvious completions, e.g., "The dog chased the ___") and exclude participants who fail > 20% of catch trials.

Plausibility and Naturalness Ratings

When to Collect

• When cloze probability alone is insufficient (e.g., both conditions have low cloze but differ in plausibility)
• When manipulating semantic fit or thematic role plausibility
• When verifying that "anomalous" conditions are genuinely perceived as odd

Rating Scale Design

Instructions Template

"You will read a series of sentences. For each sentence, please rate how natural or plausible it sounds on a scale from 1 to 7, where 1 means 'very unnatural / makes no sense' and 7 means 'perfectly natural / makes complete sense.' There are no right or wrong answers; we are interested in your intuition."

Critical Design Considerations

• Within-list design: Each rater sees only one version of each item (Latin square). Raters should never see multiple conditions of the same item, or they will rate contrastively rather than absolutely.
• Filler items: Include filler sentences spanning the full rating range. This prevents range restriction.
• Order effects: Randomize item order per participant.

Acceptability Judgments

When to Collect

• When manipulating syntactic structure (grammaticality, island constraints, movement dependencies)
• When testing formal linguistic predictions about sentence well-formedness
• For factorial designs crossing syntactic factors (e.g., 2x2 designs testing island effects; Sprouse et al., 2012)

Rating Methods

Recommendation: Use 7-point Likert as the default. It provides sufficient sensitivity for most research questions and has been shown to replicate formal linguistic judgments as reliably as magnitude estimation (Sprouse & Almeida, 2012; Sprouse, 2011).

Sample Size for Acceptability

Lexical Controls

Variables That Must Be Controlled Across Conditions

Every critical word manipulation must control for confounding lexical variables. The target word and its condition-matched alternatives should be equated on the following:

Frequency Database Selection

Key recommendation: Use SUBTLEX log frequency values. They explain more variance in lexical decision and naming times than older norms (Brysbaert & New, 2009).

How to Match Across Conditions

1. Select critical words for each condition
2. Retrieve lexical metrics from SUBTLEX-US and norming databases
3. Compute condition means for each metric
4. Test for differences: Run t-tests or ANOVAs across conditions on each lexical variable
5. Criterion: No significant differences (p > 0.20 is a reasonable threshold; some use p > 0.30) on any controlled variable
6. If matching fails: replace items or add the unmatched variable as a covariate in the analysis

Latin Square Counterbalancing

Purpose

In a within-item design, each item appears in all conditions, but each participant sees each item in only one condition. A Latin square assigns items to conditions across participant lists.

Construction

For a design with k conditions and n items (where n is divisible by k):

1. Divide items into k groups of n/k items each
2. Create k lists; in each list, assign each item group to a different condition
3. Each participant receives one list
4. Result: every item appears in every condition across participants; each participant sees an equal number of items per condition

Example: 2-Condition Design

With 40 items and 2 conditions (A, B):

Requirements

Filler Items

Purpose

Fillers prevent participants from noticing the experimental manipulation and adopting strategies.

Design Parameters