Fact Check

Fact-Check Skill

Systematic verification of claims in generated content. Designed to catch hallucinations, confabulations, and unsupported assertions.

Why Separate Passes Matter

The Fundamental Problem: LLMs generate plausible-sounding content by predicting what should come next. This same mechanism produces hallucinations—confident statements that feel true but aren't. An LLM in generation mode cannot reliably catch its own hallucinations because:

1. Attention is on generation, not verification
2. Coherence pressure makes false claims feel correct in context
3. Same weights that produced the error will confirm it
4. No external grounding to contradict the confabulation

The Solution: Verification must be a separate cognitive pass with:

• Fresh attention focused solely on each claim
• Explicit source checking (not memory/training data)
• Adversarial stance toward the content
• External grounding where possible

Diagnostic States

F1: No Verification Pass

Symptoms: Content generated and delivered without any fact-checking. Risk: Hallucinations pass through undetected. Intervention: Run verification pass before delivery. Extract claims, check each against sources.

F2: Self-Verification (Invalid)

Symptoms: Same pass asked to "check your facts" while generating. Risk: False confidence—errors confirmed by same process that created them. Intervention: Complete generation first, then run separate verification pass with explicit source requirements.

F3: Memory-Based Verification (Unreliable)

Symptoms: Claims checked against "what I know" without external sources. Risk: Hallucinations verified by hallucinated knowledge. Intervention: Require explicit source citation for each verified claim. If no source available, mark as unverified.

F4: Selective Verification

Symptoms: Only some claims checked; others assumed correct. Risk: Unchecked claims may contain errors. Intervention: Systematic extraction of ALL verifiable claims. Check each, or explicitly mark unchecked items.

F5: Verification Complete

Symptoms: All claims extracted, each checked against sources, confidence levels assigned. Indicators: Source citations present, unverified claims marked, confidence explicit.

The Verification Process

Phase 1: Claim Extraction

Extract every verifiable statement from the content.

Claim types to extract:

• Factual assertions ("X is Y", "X causes Y")
• Statistics and numbers ("40% of...", "in 2023...")
• Attributions ("According to X...", "Research shows...")
• Definitions ("X means...", "X is defined as...")
• Historical claims ("X happened in...", "X was founded by...")
• Causal claims ("X leads to Y", "X prevents Y")
• Comparative claims ("X is better than Y", "X is the largest...")

What to skip:

• Opinions clearly marked as such
• Hypotheticals and speculation (if labeled)
• Logical deductions from stated premises
• Direct quotes (verify attribution, not content)

Phase 2: Claim Categorization

Categorize each claim by verifiability:

Phase 3: Source Verification

For each claim, attempt verification:

## Claim Verification Log

### Claim 1: "[exact claim text]"
- **Category:** [Verifiable-Hard/Soft/Attribution/Inference]
- **Source checked:** [specific source]
- **Finding:** [Confirmed/Partially supported/Not found/Contradicted]
- **Confidence:** [High/Medium/Low]
- **Notes:** [discrepancies, qualifications needed]

### Claim 2: ...

Verification outcomes:

Phase 4: Confidence Assignment

Assign overall confidence to the content:

Hallucination Patterns

Common hallucination types to watch for:

1. Plausible Fabrication

Pattern: Specific details that sound right but don't exist. Examples: Fake paper citations, non-existent statistics, invented quotes. Detection: Verify specific claims against primary sources.

2. Confident Extrapolation

Pattern: Reasonable inference stated as established fact. Examples: "Studies show..." (no specific study), "Experts agree..." (no citation). Detection: Require specific source for any claim of external support.

3. Temporal Confusion

Pattern: Mixing information from different time periods. Examples: Old statistics presented as current, defunct organizations described as active. Detection: Check dates on sources, verify current status.

4. Attribution Drift

Pattern: Correct information attributed to wrong source. Examples: Quote assigned to wrong person, finding attributed to wrong study. Detection: Verify attribution specifically, not just content.

5. Amalgamation

Pattern: Combining details from multiple sources into one fictional source. Examples: Invented study that combines real findings from separate papers. Detection: Verify the specific source exists and contains all attributed claims.