Memory Corpus Ingest

Corpus Ingest

Bring a large dataset into the assistant's working knowledge without stuffing it into memory. The model is a library: the workspace holds the stacks (the raw files, cold and complete), memory holds the card catalog (a small set of map pages that say what exists, when it is from, and where to look), and a purpose-built retrieval skill is the librarian that walks to the right shelf on demand.

Two invariants drive everything below:

1. Raw data never enters the memory corpus. Nothing from the dataset is written into memory/concepts/ except the map pages, and nothing is ever appended to memory/buffer.md (bulk buffer appends trip the consolidation burst guard and per-run caps; the map bypasses the buffer entirely via assistant memory ingest).
2. The map stays small. Roughly 10 to 50 pages regardless of corpus size. If the corpus doubles, the pages get denser or the slices get coarser; the page count does not double.

Procedure

Step 1: Scope and confirm

Identify the source and its size before committing:

du -sh /path/to/raw-corpus
find /path/to/raw-corpus -type f | wc -l

Tell the user what will happen: the raw files move into the workspace, a bounded number of summarization passes read them once to build the map, the map is ingested into memory, and a lookup skill is authored for drill-in. Skimming a large corpus is real LLM work that costs time and money; confirm before starting. For Fathom recording exports, read references/fathom.md first for format discovery and slicing guidance.

Step 2: Cold-store the raw corpus

Land the raw files under an imports directory in the workspace, one directory per source:

Screen for credentials BEFORE copying: an arbitrary corpus can carry secret material, and anything landed under imports/ becomes reachable by workspace tools, backups, and retrieval flows.

cd "$VELLUM_WORKSPACE_DIR"
# 1a. Screen for secret-bearing FILE NAMES; review every hit with the user.
find /path/to/raw-corpus \( -name '.env*' -o -name '*.key' -o -name '*.pem' \
  -o -name '*credential*' -o -name '*secret*' -o -name 'cookies*' \
  -o -path '*tokens*' -o -path '*oauth*' \) -print

# 1b. Screen file CONTENTS for credential shapes. --hidden and --no-ignore
#     matter: rg skips dotfiles and gitignored paths by default, which is
#     exactly where credentials live. Capture the FULL list (no truncation):
#     every file named here must be excluded below or cleaned with the user
#     before it lands.
rg -l -i --hidden --no-ignore \
  "api[_-]?key|access[_-]?token|refresh[_-]?token|client[_-]?secret|password\s*[=:]|passwd|bearer |AKIA[0-9A-Z]{16}|BEGIN [A-Z ]*PRIVATE KEY" \
  /path/to/raw-corpus > /tmp/corpus-secret-hits.txt
cat /tmp/corpus-secret-hits.txt

# 2. Build rsync exclusions from the content hits (paths relative to the
#    corpus root), then copy with ALL flagged paths excluded.
sed 's|^/path/to/raw-corpus/||' /tmp/corpus-secret-hits.txt > /tmp/corpus-secret-exclusions.txt
mkdir -p imports/<source>
rsync -a --exclude='.env*' --exclude='*.key' --exclude='*.pem' \
  --exclude='tokens/' --exclude='oauth/' --exclude='cookies*' \
  --exclude-from=/tmp/corpus-secret-exclusions.txt \
  /path/to/raw-corpus/ imports/<source>/

Rules:

• Never place raw corpus files under memory/. The cold store is imports/<source>/; the map is the only thing that enters memory.
• Never land credentials in the cold store. Exclude secret-bearing files during the copy; if the content screen finds embedded live tokens inside otherwise-wanted files, pause and resolve them with the user before landing those files.
• Treat the cold store as read-only once landed. The map pages and the drill-in skill both point at these paths; moving files later breaks every pointer.
• If the corpus is huge, check free disk space first and copy in batches. Prefer copy over move until the user confirms the original can be released.

Step 3: Inventory and slice plan

Census the corpus and produce a machine-readable slice plan:

mkdir -p "$VELLUM_WORKSPACE_DIR/imports/<source>/.staging"
bun run {baseDir}/scripts/inventory.ts "$VELLUM_WORKSPACE_DIR/imports/<source>" \
  > "$VELLUM_WORKSPACE_DIR/imports/<source>/.staging/plan.json"

The script prints a human census to stderr (file count, total size, extension mix, date range) and a JSON plan to stdout: { files, totalBytes, byExtension, dateRange, suggestedSlices }. Each suggested slice is a date-windowed group of files sized for one skim pass. Review the plan before skimming:

• If the slice count is outside roughly 10 to 50, adjust: merge sparse adjacent slices or split dense ones. The plan is a suggestion, not a contract.
• Files with no recognizable date cluster on file mtime; if mtimes are all import-day (a fresh copy), pick slices by directory or topic instead and say so in the map.

Step 4: Skim each slice into a staged map page

For each slice in the plan, run one summarization pass that reads the slice's files and writes one staged map page:

• Output goes to the staging directory as <slug>.md (for example imports/fathom/.staging/fathom-recordings-2025-q1.md). The slug is the filename minus .md.
• Every page follows references/map-page-template.md exactly: lead that stands alone as the retrieval card, a Raw data: pointer line in the lead, ## sections per topic or time slice, and source: / origin_date: / ref_files: / links: frontmatter.
• Also write one corpus index page (kind: index) whose links: enumerate all slice pages.

Bound the fan-out. The number of skim passes equals the number of slices in the plan, full stop. Run them a few at a time (parallelism of 3 or 4 is plenty). Never spawn one pass per file, never let a pass recursively spawn more passes, and never re-skim slices that already have a staged page unless their content changed. An unbounded fan-out over a large corpus is the expensive failure mode of this skill.

A skim pass extracts what a future search needs to route: decisions, recurring topics, named people and projects, date spans, open threads. It does not transcribe. Verbatim content stays cold; the map records that it exists and where.