Ingest Source

Ingest Source

Fully autonomous pipeline that takes one source from raw file to integrated, connected knowledge. Each invocation handles a single source; drive a whole corpus by invoking once per source (e.g. via /loop or a shell loop over a file list), then run the corpus-level finalize steps once at the end.

ultrathink

Purpose

The unit of work for building a knowledge base from a book corpus. One source in → deduplicated, epistemically-classified notes out, embedded in the graph and linked to existing knowledge, with every auto-action logged for post-hoc review.

The design principle (established in the architecture discussion): insights are extracted against the current KB, not in a vacuum — retrieval-augmented extraction. The index is the KB's queryable state, so it must be refreshed before connection discovery, and refreshed again on the next source so each source builds on the last.

Configuration (safety rails that replace human gates)

These are the autonomous substitute for the two approval gates a gated version would have. See Architecture Note for why.

State Dependencies

Prerequisites

• Local Brain Search installed (resources/local-brain-search/run_index.sh, run_connections.sh)
• Brain Dependency Graph engine (resources/brain-graph/run_brain_graph.sh)
• Extractor agents available: document-insight-extractor
• Source-prep skills available: epub-chapter-extractor, multi-format-book-extractor, get-youtube-transcript

Process

Step 0: Read inputs and checkpoint

Parse $ARGUMENTS: first token = source path/URL, optional remainder = session name. Derive a <slug> and, if no session name given, derive one from the source title (e.g. "<Book Title>").

date '+%Y-%m-%d'
test -f resources/ingest-workspace/<slug>/.checkpoint && cat resources/ingest-workspace/<slug>/.checkpoint || echo "phase=start"

If a checkpoint exists, resume from the next phase rather than restarting (re-running earlier phases is safe — they are idempotent — but wasteful for large books).

Scope routing (book vs. non-book). Decide the destination scope now:

A book gets its own scope (Books/<slug>): non-core (never pollutes the core fingerprint or trains q-values), cognitive (its gems can graduate to core via the human /graduate-insights act), and pluggable (mount with BRAIN_READ_SCOPE=core,Books/<slug> or the whole shelf with core,Books). Set TARGET_DIR and BOOK_SLUG accordingly and use them in Steps 2/4/6 below.

Step 1: Prepare source → markdown

Detect format by extension/URL and convert to markdown:

Write prepared markdown into resources/ingest-workspace/<slug>/. Checkpoint: phase=prepared.

Step 2: Extract insights (against the live index)

Spawn the extractor directly (autonomous — skip the interactive insight-interview suggestion).

For a book (Step-0 routing → Brain/Books/<book-slug>/), invoke in Book Mode:

Task(
  subagent_type="document-insight-extractor",
  prompt="BOOK MODE. Extract insights from <prepared markdown path(s)> into the book scope
          Brain/Books/<book-slug>/ (NOT Document Insights). Follow your Book Mode doctrine:
          mine like a scientist THROUGH THE KB LENS (contextualize against the CURRENT index
          as a GATE); capture ONLY genuine frameworks / mental models / methods / evidenced
          contrarian claims; REJECT narrative, anecdote, biography, he-said-she-said, and
          restatements of the obvious or of concepts the KB already holds; MECE + atomic
          (one idea per note, no overlap within the book scope); expected shape ~10 core
          concepts + up to ~20 supporting nodes as REFERENCE POINTS not caps; prefer
          statistically-meaningful research (N, effect size, replication, year) over one-off
          examples and record dates. Apply Gate 1 source-tiering and provenance: encountered;
          SEARCH FOR DUPLICATES (mount BRAIN_READ_SCOPE=core,Books,document-insights) and
          merge/link rather than re-create; create the _book.md literature hub; update the
          changelog. Return: list of new note file paths, total count, epistemic split,
          assigned source-tier, and the book scope path."
)

For non-books (paper / article / web → Brain/Document Insights/<session>/):

Task(
  subagent_type="document-insight-extractor",
  prompt="Extract insights from <prepared markdown path(s)> into session '<session name>'.
          Follow your mandatory workflow: contextualize against the CURRENT index,
          apply Gate 1 source-tiering and Gate 2 provenance (provenance: encountered),
          epistemically classify each note, SEARCH FOR DUPLICATES and merge/link rather
          than re-create, write notes to Brain/Document Insights/<session>/, update the
          session changelog. Return: list of new note file paths, total count,
          epistemic split, and the assigned source-tier."
)

Safety rail (replaces Gate 1): if the returned source-tier is REJECT_ON_TIER, abort, log the rejection to the changelog, and notify. Low-confidence/speculative notes are kept but flagged in the changelog, never silently dropped.

Capture the new note paths — only these notes are processed downstream (this keeps connection discovery linear, not all-vs-all). Checkpoint: phase=extracted (persist the note list).

Step 3: Refresh the index

The FAISS index does not auto-update, and connection discovery looks notes up by title in the index — so new notes are invisible until reindexed. Refresh now, before discovery:

resources/local-brain-search/run_index.sh
resources/brain-graph/run_brain_graph.sh bootstrap --force
resources/local-brain-search/run_connections.sh --stats --json   # verify count grew

(This is exactly /refresh-index; one full rebuild per source also sweeps in the prior source's auto-links.) Checkpoint: phase=indexed.

Step 4: Discover connections (bounded, new → existing)

For each new note only, query its neighbors. Use a wide read-scope so a new note resolves and links against existing non-core neighbors (core-only would miss them once scope enforcement is on). Mount the destination scope: for a book use core,Books (so it links to core and across the whole shelf); for a non-book use core,document-insights:

# book:
BRAIN_READ_SCOPE=core,Books resources/local-brain-search/run_connections.sh "<New Note Title>" --json
# non-book:
BRAIN_READ_SCOPE=core,document-insights resources/local-brain-search/run_connections.sh "<New Note Title>" --json
resources/brain-graph/run_brain_graph.sh inspect "<New Note Title>" --json   # edge type + lifecycle

Collect top-LINKS_PER_NOTE connections per note with similarity scores and BDG edge type. Do not run all-pairs or hub/bridge sweeps here — that is the combinatorial explosion this step is designed to avoid. New→existing is linear in the source's note count. Checkpoint: phase=connections-discovered (persist the connection report).

Step 5: Auto-write links (the gated step, made safe)

For each new note, write wiki-links to its qualifying connections:

• Threshold: only connections with similarity ≥ AUTO_LINK_THRESHOLD.
• Cap: at most LINKS_PER_NOTE per note.
• Idempotent: skip if the [[link]] already exists in the note.
• Labeled + reversible: append under a clearly marked section in the NEW note only:

  ## Related (auto-linked by /ingest-source)
  <!-- AI-discovered via semantic similarity; review critically. similarity ≠ conceptual validity -->
  - [[Target Note]] — 0.82, derives-from

• Never mutate existing notes when MUTATE_EXISTING_NOTES=false — links go FROM the fresh notes outward, so hub notes are never polluted and every auto-edit is confined to notes created this run (trivially reversible by deleting the session folder).
• Below-threshold connections → recorded in the changelog as "candidates for human review," not written.
• Tension preservation: if a high-similarity connection is to a note with an opposing claim (BDG edge type tension, or the extractor flagged a contradiction), DO NOT collapse or dedupe it away. Record it in the changelog under "Tension candidates" for the manual /detect-tensions pass. Never auto-resolve a contradiction.

Use Edit on the new note files (their paths are known from Step 2). Checkpoint: phase=linked.