Divergent Thinking Scoring

Divergent Thinking Scoring

Purpose

This skill encodes expert methodological knowledge for scoring responses from divergent thinking tasks (Alternative Uses Task, Unusual Uses Task, instances tasks, etc.). It covers the four standard scoring dimensions — fluency, flexibility, originality, and elaboration — plus modern automated scoring using semantic distance. A general-purpose programmer would typically count responses (fluency) but would not know the domain-specific decisions around flexibility category systems, originality thresholds, inter-rater reliability requirements, or how to compute semantic distance as a creativity metric.

When to Use This Skill

• Scoring responses from an AUT, Unusual Uses Task, or similar divergent thinking task
• Choosing between subjective (human-rated) and objective (automated) scoring approaches
• Computing semantic distance as an automated creativity metric
• Establishing inter-rater reliability for creativity coding
• Deciding how to handle the fluency-originality confound

Research Planning Protocol

Before executing the domain-specific steps below, you MUST:

1. State the research question — What specific aspect of divergent thinking is being measured?
2. Justify the method choice — Why these scoring dimensions? What alternatives were considered?
3. Declare expected outcomes — Which dimensions are expected to show effects?
4. Note assumptions and limitations — What does each scoring method assume? Where could it mislead?
5. Present the plan to the user and WAIT for confirmation before proceeding.

For detailed methodology guidance, see the research-literacy skill.

⚠️ Verification Notice

This skill was generated by AI from academic literature. All parameters, thresholds, and citations require independent verification before use in research. If you find errors, please open an issue.

Scoring Dimensions Overview

Fluency Scoring

Definition

Fluency = the total number of valid, non-redundant responses a participant generates.

Scoring Rules

1. Count each distinct response as one unit
2. Exclude:

• Exact duplicates
• Conventional/typical uses of the object (debated — some protocols include them; Reiter-Palmon et al., 2019)
• Gibberish or clearly irrelevant responses
• Responses that are minor variations of each other (e.g., "use as a hammer" and "use to pound nails" = 1 response)

3. When in doubt, count as separate responses and let originality scoring handle quality

Domain insight: Fluency is the most reliable but least interesting creativity measure. It correlates with personality traits (openness) and general cognitive ability, but does not distinguish truly creative responses from merely numerous ones (Silvia et al., 2008).

Flexibility Scoring

Definition

Flexibility = the number of distinct conceptual categories across a participant's responses.

Category Systems

COWA (Category of Words from AUT) system (Reiter-Palmon et al., 2019):

Provides a standardized taxonomy of response categories for common AUT objects. Example categories for "brick":

Scoring Procedure

1. Train coders on the category system (minimum 2 coders)
2. Assign each response to its most appropriate category
3. Count unique categories per participant = flexibility score
4. Compute inter-rater reliability: ICC ≥ 0.70 acceptable, ≥ 0.80 good (Lee & Chung, 2024; Shrout & Fleiss, 1979)

Originality Scoring

Method 1: Statistical Rarity (Objective)

A response is "original" if it is given by fewer than 5% of the sample (Wallach & Kogan, 1965; Lee & Chung, 2024).

Procedure:

1. Pool all responses across all participants
2. Normalize spelling and phrasing (e.g., "door stop" = "doorstop")
3. Compute the frequency of each unique response
4. Mark responses given by <5% of participants as original (score = 1; else = 0)
5. Originality score per participant = sum or proportion of original responses

Alternative thresholds: Some studies use <1% (very strict) or <10% (lenient). The 5% threshold is most common (Reiter-Palmon et al., 2019).

Method 2: Subjective Rating (Qualitative)

Human raters judge each response for creativity on a Likert scale (Silvia et al., 2008).

Procedure:

1. Scale: 1 (not at all creative) to 5 (highly creative) — or 1-7 for finer discrimination
2. Raters: Minimum 2 independent raters (Lee & Chung, 2024 used 3)
3. Training: Calibrate raters on anchor examples before scoring begins
4. Inter-rater reliability: ICC (two-way random, average measures) ≥ 0.70 (Lee & Chung, 2024 achieved ICC = 0.72-0.89)
5. Scoring: Average across raters for each response; then average or sum per participant

Method Selection Decision Logic

Is your sample size large (N > 100)?
|
+-- YES --> Do you need fine-grained creativity distinctions?
| |
| +-- YES --> Use subjective rating (richer information)
| |
| +-- NO --> Use statistical rarity (objective, faster)
|
+-- NO --> Statistical rarity is unreliable with small N
 (rare responses may be rare by chance)
 --> Use subjective rating

Semantic Distance (Automated Scoring)

Overview

Semantic distance measures how conceptually far a response is from the prompt word in a vector space model. More distant = more creative (Beaty & Johnson, 2021).

Method (Beaty & Johnson, 2021; Organisciak et al., 2023)

1. Embedding model: GloVe (Global Vectors for Word Representation; Pennington et al., 2014) trained on Common Crawl — 300-dimensional vectors
2. Computation:

• Represent the prompt word (e.g., "brick") as a vector
• Represent each response as a vector (average word vectors for multi-word responses)
• Compute cosine distance = 1 - cosine_similarity(prompt, response)

3. Per-participant score: Average semantic distance across all responses
4. Platform: SemDis (https://semdis.wlu.psu.edu/) — web-based tool by Beaty & Johnson (2021)

Interpretation

Validation: Semantic distance correlates with subjective originality ratings at r ≈ 0.40-0.60 and predicts real-world creative achievement (Beaty & Johnson, 2021; Organisciak et al., 2023).

Advantages and Limitations