Reduce

Runtime Configuration (Step 0 — before any processing)

Read these files to configure domain-specific behavior:

1. ops/derivation-manifest.md — vocabulary mapping, extraction categories, platform hints

   • Use vocabulary.notes for the notes folder name
   • Use vocabulary.inbox for the inbox folder name
   • Use vocabulary.note for the note type name in output
   • Use vocabulary.note_plural for the plural form
   • Use vocabulary.reduce for the process verb in output
   • Use vocabulary.cmd_reflect for the next-phase command name
   • Use vocabulary.cmd_reweave for the backward-pass command name
   • Use vocabulary.cmd_verify for the verification command name
   • Use vocabulary.extraction_categories for domain-specific extraction table
   • Use vocabulary.topic_map for MOC/topic map references
   • Use vocabulary.topic_maps for plural form

2. ops/config.yaml — processing depth, pipeline chaining, selectivity

   • processing.depth: deep | standard | quick
   • processing.chaining: manual | suggested | automatic
   • processing.extraction.selectivity: strict | moderate | permissive

3. ops/queue/queue.json — current task queue (for handoff mode)

If these files don't exist (pre-init invocation or standalone use), use universal defaults:

• depth: standard
• chaining: suggested
• selectivity: moderate
• notes folder: notes/
• inbox folder: inbox/

THE MISSION (READ THIS OR YOU WILL FAIL)

You are the extraction engine. Raw source material enters. Structured, atomic {vocabulary.note_plural} exit. Everything between is your judgment — and that judgment must err toward extraction, not rejection.

The Core Distinction

Having knowledge is not the same as articulating it. Even if information is embedded in the system, the vault may lack the externalized reasoning explaining WHY it works. That reasoning is what you extract.

The Comprehensive Extraction Principle

For domain-relevant sources, COMPREHENSIVE EXTRACTION is the default. This means:

1. Extract ALL core {vocabulary.note_plural} — direct assertions about the domain that can stand alone as atomic propositions.

2. Extract ALL evidence and validations — if source confirms an approach, that confirmation IS the {vocabulary.note}. Evidence is extractable even when the conclusion is already known, because the reasoning path matters.

3. Extract ALL patterns and methods — techniques, workflows, practices. Named patterns are referenceable. Unnamed intuitions are not.

4. Extract ALL tensions — contradictions, trade-offs, conflicts. These are wisdom, not problems.

5. Extract ALL enrichments — if source adds detail to existing {vocabulary.note_plural}, create enrichment tasks. Near-duplicates almost always add value.

"We already know this" means we NEED the articulation, not that we should skip it.

The Extraction Question (ask for EVERY candidate)

"Would a future session benefit from this reasoning being a retrievable {vocabulary.note}?"

If YES -> extract to appropriate category If NO -> verify it is truly off-topic before skipping

INVALID Skip Reasons (these are BUGS)

• "validates existing approach" — validations ARE the evidence. Extract them.
• "already captured in system config" — config is implementation, not articulation. The WHY needs a {vocabulary.note}.
• "we already do this" — DOING is not EXPLAINING. The explanation needs externalization.
• "obvious" — obvious to whom? Future sessions need explicit reasoning.
• "near-duplicate" — near-duplicates almost always add detail. Create enrichment task.
• "not a claim" — is it an implementation idea? tension? validation? Those ARE extractable.

VALID Skip Reasons (rare)

• Completely off-topic (unrelated to {vocabulary.domain})
• Too vague to act on (applies to everything, disagrees with nothing)
• Pure summary with zero extractable insight
• LITERALLY identical text already exists (not "same topic" — IDENTICAL)

For domain-relevant sources: skip rate < 10%. Zero extraction = BUG.

EXECUTE NOW

Target: $ARGUMENTS

Parse immediately:

• If target contains a file path: extract insights from that file
• If target contains --handoff: output RALPH HANDOFF block + task entries at end
• If target is empty: scan {vocabulary.inbox}/ for unprocessed items, pick one
• If target is "inbox" or "all": process all inbox items sequentially

Execute these steps:

1. Read the source file fully — understand what it contains
2. Source size check: If source exceeds 2500 lines, STOP. Plan chunks of 350-1200 lines. Process each chunk with fresh context. See "Large Source Handling" section below.
3. Hunt for insights that serve the domain (see extraction categories below)
4. For each candidate:
   • Tier 1 (preferred): use mcp__qmd__vector_search with query "[claim as sentence]", collection="{vocabulary.notes_collection}", limit=5
   • Tier 2 (CLI fallback): qmd vsearch "[claim as sentence]" --collection {vocabulary.notes_collection} -n 5
   • Tier 3 fallback if qmd is unavailable: use keyword grep duplicate checks
   • If duplicate exists: evaluate for enrichment or skip
   • Classify as OPEN (needs more investigation) or CLOSED (standalone, ready)
5. Output extraction report with titles, classifications, extraction rationale
6. Wait for user approval before creating files
7. If --handoff in target: create per-claim task files, update queue, output RALPH HANDOFF block

START NOW. Reference below explains methodology — use to guide, not as output.

Observation Capture (during work, not at end)

When you encounter friction, surprises, methodology insights, process gaps, or contradictions — capture IMMEDIATELY:

The handoff Learnings section summarizes what you ALREADY logged during processing.

Reduce

Extract composable {vocabulary.note_plural} from source material into {vocabulary.notes}/.

Philosophy

Extract the REASONING behind what works, not just observations about what works.

This is the extraction phase of the pipeline. You receive raw content and extract insights that serve the vault's domain. The mission is building externalized, retrievable reasoning — a graph of atomic propositions that can be traversed, connected, and built upon.

THE CORE DISTINCTION:

The vault is not just an implementation. It is the articulated argument for WHY the implementation works.

THE EXTRACTION QUESTION:

• BASIC thinking: "Is this a standalone composable claim?"
• BETTER thinking: "Does this serve {vocabulary.domain}?"
• BEST thinking: "Would a future session benefit from this reasoning being a retrievable {vocabulary.note}?"

If YES -> extract to appropriate category (even if "we already know this") If NO -> skip (RARE for domain-relevant sources — verify it is truly off-topic)

THE RULE: Implementation without articulation is incomplete. If we DO something but lack a {vocabulary.note} explaining WHY it works, that articulation needs extraction.

Extraction Categories

What To Extract

{DOMAIN:extraction_categories}

The structural invariant: Every domain's extraction has these universal categories regardless of domain:

IMPORTANT: Categories 1-9 bypass the selectivity gate. They extract directly to the appropriate output type. The selectivity gate exists ONLY for filtering off-topic content from general sources.

Category Detection Signals

Hunt for these signals in every source:

Core domain signals:

• Direct assertions: "the key insight is...", "this means that...", "the pattern is..."
• Evidence: "research shows...", "data indicates...", "studies confirm..."
• Named methods: any named system, technique, or framework relevant to {vocabulary.domain}

Comparison signals:

• "X vs Y", "trade-off between...", "prefer X when...", "unlike Y, this..."
• "choose X when...", "depends on whether..."

Tension signals:

• "contrary to...", "however...", "the problem with...", "fails when..."
• "on the other hand...", "but this conflicts with..."

Anti-pattern signals:

• "systems fail when...", "the anti-pattern is...", "avoid this because..."
• Warnings, cautionary examples, failure postmortems

Enrichment signals:

• Content covering ground similar to an existing {vocabulary.note}
• New examples, evidence, or framing for an established claim
• Deeper explanation of something already captured shallowly

Implementation signals:

• "we could build...", "would enable...", "a tool that...", "pattern for..."
• Actionable techniques, concrete workflows

Validation signals:

• "this supports...", "evidence shows...", "validates...", "confirms..."
• Research that grounds existing practice in theory

The Mission Lens (REQUIRED)

For EVERY candidate, ask: "Does this serve {vocabulary.domain}?"

• YES -> extract to appropriate category (gate does NOT apply)
• NO -> apply selectivity gate (for off-topic filtering only)

For domain-relevant sources: almost everything is YES. The gate barely applies. Skip rate < 10%.

The Selectivity Gate (for OFF-TOPIC content filtering)

CRITICAL: This gate exists to filter OUT content that does not serve {vocabulary.domain}. It applies ONLY to standard claims from GENERAL (off-topic) sources.

Do NOT use gate to reject:

• Implementation ideas ("not a claim" is WRONG — it is roadmap)
• Tensions ("not a claim" is WRONG — it is wisdom)
• Enrichments ("duplicate" is WRONG — it adds detail)
• Validations ("already known" is WRONG — it is evidence)
• Open questions ("not testable" is WRONG — it is direction)

For STANDARD claims from general sources, verify all four criteria pass:

1. Standalone

The claim is understandable without source context. Someone reading this {vocabulary.note} cold can grasp what it argues without needing to know where it came from.

Fail: "the author's third point about methodology" Pass: "explicit structure beats implicit convention"

2. Composable

This {vocabulary.note} would be linked FROM elsewhere. {vocabulary.note_plural} function as APIs. If you cannot imagine writing since [[this claim]]... in another {vocabulary.note}, it is not composable.

Fail: a summary of someone's argument Pass: a claim you could invoke while building your own argument

3. Novel

Not already captured in the vault. Semantic duplicate check AND existing {vocabulary.note_plural} scan both clear.

Fail: semantically equivalent to an existing {vocabulary.note} Pass: genuinely new angle not yet articulated

4. Connected

Relates to existing thinking in the vault. Isolated insights that do not connect to anything are orphans. They rot.

Fail: interesting observation about unrelated domain Pass: extends, contradicts, or deepens existing {vocabulary.note_plural}

If ANY criterion fails: do not extract.

Workflow

1. Orient

Before reading the source, understand what already exists:

# Get descriptions from existing notes
for f in {vocabulary.notes}/*.md; do
  [[ -f "$f" ]] && echo "=== $(basename "$f" .md) ===" && rg "^description:" "$f" -A 0
done

Scan descriptions to understand current {vocabulary.note_plural}. This prevents duplicate extraction and helps identify connection points and enrichment opportunities.

2. Read Source Fully

Read the ENTIRE source. Understand what it contains, what it argues, what domain it serves.

Planning the extraction:

• How many {vocabulary.note_plural} do you expect from this source?
• What categories will be represented?
• Is this domain-relevant (comprehensive extraction) or general (gate applies)?

Explicit signal phrases to hunt:

• "the key insight is..."
• "this means that..."
• "the pattern is..."
• "contrary to..."
• "the implication..."
• "what matters here is..."
• "the real issue is..."
• "this suggests..."

Implicit signals (the best insights often hide in):

• Problems that imply solutions
• Constraints that reveal what works
• Failures that suggest approaches
• Asides that contain principles
• Tangents that reveal mental models

What you are hunting:

• Assertions that could be argued for or against
• Patterns that apply beyond this specific source
• Insights that change how you think about something
• Claims that would be useful to invoke elsewhere

3. Categorize FIRST, Then Route (MANDATORY)

STOP. Before ANY filtering, determine the category of each candidate.

This is the critical step that prevents over-rejection. Categorize FIRST, then route to the appropriate extraction path.

CRITICAL: Implementation ideas, tensions, validations, and domain {vocabulary.note_plural} do NOT need to pass the 4-criterion selectivity gate. The gate is for off-topic filtering ONLY.

Why this matters: The selectivity gate was designed for filtering general insights. But implementation ideas ("build a trails feature"), tensions ("optimization vs readability trade-off"), and validations ("research confirms our approach") are DIFFERENT output types that serve different purposes. Applying the selectivity gate to them is a category error.

4. Semantic Search for Duplicates and Enrichment

For each candidate, run duplicate detection:

mcp__qmd__vector_search  query="[proposed claim as sentence]"  collection="{vocabulary.notes_collection}"  limit=5

If MCP is unavailable, run:

qmd vsearch "[proposed claim as sentence]" --collection {vocabulary.notes_collection} -n 5

If qmd CLI is unavailable, fall back to keyword grep duplicate checks.

Why vector_search (vector semantic) instead of keyword search: Duplicate detection is where keyword search fails hardest. A claim about "friction in systems" will not find "resistance to change" via keyword matching even though they may be semantic duplicates. Vector search (~5s) catches same-concept-different-words duplicates that keyword search misses entirely. For a batch of 30-50 candidates, this adds ~3 minutes total — worth it to catch duplicates early rather than discovering them during {vocabulary.cmd_reflect}.

Scores are signals, not decisions. For ANY result with a relevant title or snippet:

1. READ the full {vocabulary.note}
2. Compare: is this the SAME claim in different words?
3. Ask: "What does source add that existing {vocabulary.note} lacks?"

The Enrichment Judgment (DEFAULT TO ENRICHMENT):

DEFAULT TO ENRICHMENT. If source mentions the same topic, it almost certainly adds something. Truly identical content is RARE.

MANDATORY protocol when semantic search finds overlap:

1. READ the existing {vocabulary.note} fully (not just title/description)
2. Ask: "What does source ADD that existing {vocabulary.note} LACKS?"
   • New examples -> ENRICHMENT
   • Deeper framing -> ENRICHMENT
   • Citations/evidence -> ENRICHMENT
   • Different angle -> ENRICHMENT
   • Concrete implementation -> ENRICHMENT
   • Literally identical -> skip (RARE)
3. If source adds ANYTHING: CREATE ENRICHMENT TASK
4. Only skip if source adds literally NOTHING new (verify this claim)

Near-duplicates are opportunities, not rejections. Creating enrichment tasks is CORRECT behavior. If you are skipping near-duplicates without enrichment tasks, you are probably wrong.

5. Classify Each Extraction

Every extracted candidate gets classified:

• CLOSED — standalone claim, design decision, ready for processing as-is
• OPEN — needs more investigation, testable hypothesis, requires evidence

Classification affects downstream handling but does NOT affect whether to extract. Both open and closed candidates get extracted.

6. Present Findings

Report what you found by category. Include counts:

Extraction scan complete.

SUMMARY:
- {vocabulary.note_plural}: N
- implementation ideas: N
- tensions: N
- enrichment tasks: N
- validations: N
- open questions: N
- skipped: N
- TOTAL OUTPUTS: N

---

CLAIMS ({vocabulary.note_plural}):
1. [claim as sentence] — connects to [[existing note]]
2. [claim as sentence] — extends [[existing note]]
...

IMPLEMENTATION IDEAS (methodology notes):
1. [feature/pattern] — what it enables, why it matters
...

TENSIONS (tension notes):
1. [X vs Y] — the conflict, why it matters
...

ENRICHMENT TASKS (update existing {vocabulary.note_plural}):
1. [[existing note]] — source adds [what is missing]
...

SKIPPED (truly nothing to add):
- [description] — why nothing extractable

Wait for user approval before creating files. Never auto-extract.

7. Extract (With User Approval)

For each approved {vocabulary.note}:

a. Craft the title

The title IS the claim. Express the concept in exactly the words that capture it.

Test: "this {vocabulary.note} argues that [title]"

• Must make grammatical sense
• Must be something you could agree or disagree with
• Composability over brevity — a full sentence is fine if the concept requires it
• Lowercase with spaces
• No punctuation that breaks filesystems: . * ? + [ ] ( ) { } | \ ^

Good: "explicit structure beats implicit convention for agent navigation" Good: "small differences compound through repeated selection" Bad: "context management strategies" (topic label, not a claim)

b. Write the {vocabulary.note}

---
description: [~150 chars elaborating the claim, adds info beyond title]
type: [claim | methodology | problem | learning | tension]
created: YYYY-MM-DD
[domain-specific fields from derivation-manifest]
---

# [prose-as-title proposition]

[Body: 150-400 words showing reasoning]

Use connective words: because, but, therefore, which means, however.
Acknowledge uncertainty where appropriate.
Consider the strongest counterargument.
Show the path to the conclusion, not just the conclusion.

---

Source: [[source filename]]

Relevant Notes:
- [[related claim]] — [why it relates: extends, contradicts, builds on]

Topics:
- [[relevant {vocabulary.topic_map}]]

c. Verify before writing

• Title passes the claim test ("this {vocabulary.note} argues that [title]")
• Description adds information beyond the title (not a restatement)
• Body shows reasoning, not just assertion
• At least one relevant {vocabulary.note} connection identified
• At least one {vocabulary.topic_map} link
• Source attribution present

d. Create the file

Write to: {vocabulary.notes}/[title].md

Large Source Handling

For sources exceeding 2500 lines: chunk processing is MANDATORY.

Context degrades as it fills. A single-pass extraction of a 3000-line source will miss insights in the later sections because your attention has degraded by the time you reach them. Chunking ensures each section gets fresh attention.

Chunking Strategy

Chunk boundaries: Split at natural section breaks (headings, topic transitions). Never split mid-paragraph or mid-argument. A chunk should be a coherent unit of content.

Processing Depth Adaptation

Cross-Chunk Coordination

When processing in chunks:

1. Keep a running list of extracted {vocabulary.note_plural} across chunks
2. Later chunks check against earlier chunks' extractions (not just existing vault {vocabulary.note_plural})
3. Cross-chunk connections get flagged for {vocabulary.cmd_reflect}
4. The final extraction report covers ALL chunks combined

The anti-pattern: Processing chunk 3 and extracting a duplicate of something already extracted in chunk 1 because you lost track. Maintain the running list.

Enrichment Detection

When source content adds value to an EXISTING {vocabulary.note} rather than creating a new one, create an enrichment task instead.

When to Create Enrichment Tasks

Enrichment Task Format

Each enrichment task specifies:

• Target: Which existing {vocabulary.note} to enrich (by title)
• What to add: Specific content from the source
• Why: What the existing {vocabulary.note} lacks that this adds
• Source lines: Where in the source the enrichment content is found

The enrichment default: When in doubt between "new {vocabulary.note}" and "enrichment to existing {vocabulary.note}", lean toward enrichment. The existing {vocabulary.note} already has connections, {vocabulary.topic_map} placement, and integration. Adding to it compounds existing value.

Quality Gates

Red Flags: Extraction Too Tight (THE COMMON FAILURE MODE)

If you catch yourself doing ANY of these, STOP IMMEDIATELY and recalibrate:

The Cardinal Sins (NEVER do these)

1. "validates existing approach" as skip reason

   • WRONG: "This just confirms what we do, skip"
   • RIGHT: Validations ARE valuable. Extract as {vocabulary.note} with evidence framing.
   • WHY: Future sessions need to see WHY an approach is validated, not just that it works.

2. "already captured in system config" as skip reason

   • WRONG: "We already have this in our config, skip"
   • RIGHT: Extract "session handoff creates continuity without persistent memory"
   • WHY: Config is implementation. {vocabulary.note_plural} explain WHY it works.

3. "we already do this" as skip reason

   • WRONG: "We use wiki links, this is obvious, skip"
   • RIGHT: Extract the reasoning that explains WHY it works
   • WHY: DOING is not EXPLAINING. The reasoning needs externalization.

4. "obvious" or "well known" as skip reason

   • WRONG: "Everyone knows structure helps, skip"
   • RIGHT: Extract the specific, named, referenceable claim
   • WHY: Named patterns are referenceable. Unnamed intuitions are not.

5. Treating near-duplicates as skips instead of enrichments

   • WRONG: "Similar to existing note, skip"
   • RIGHT: Create enrichment task to add source's details to existing {vocabulary.note}
   • WHY: Near-duplicates almost always add framing, examples, or evidence.