Vellum Memory V3 Migration

Memory v3 Migration

Reform an existing memory-v2 corpus into the memory-v3 wiki: a cross-linked set of topical articles where each article's lead is its retrieval card and each ## section is an independently retrievable unit. The assistant is the sole reader and editor of this knowledge base; the goal is a clean, well-organized wiki optimized for retrieval.

This skill is the successor to vellum-memory-v2-migration. That skill backfills an empty corpus from scratch; this one reorganizes a populated one. If concepts/ is empty, stop and run that skill instead.

What actually changes, and why it's two jobs

The mechanical cutover is cheap and mostly automatic: v3 reads the same memory/concepts/*.md tree, the schema is shared, the DB tables already exist, and section embeddings backfill on demand. The work is the reform. v3 retrieval is section-grain: the selector sees a compact card per candidate (the article's lead + its section names), and what reaches context is the best-matching ## section of each selected article in full (the lead when the article was selected without a section match). A section longer than the section index's chunk window (6000 characters, SECTION_CHUNK_CHARS) is indexed, keyed, and injected as independent chunks (Title, Title~1, ...), and the selector may pick any one of them, so a section that should travel as one unit stays under the window. Each injected section is frozen into the conversation once, re-selections point back at it with a one-line pointer, and every resident section is pruned by recency under one byte cap, with no exemptions for any lane. A flat v2 page (bullets, no ## headings, a summary: field v3 ignores) collapses under v3 into one giant lead with no sections: every selection injects the whole page, nothing narrower can match, and that one oversized block crowds the resident budget. So this skill does two things at once:

1. Reshape each surviving page into the v3 article skeleton (lead + sections, flat slug, links: not edges:, optional current:).
2. Re-organize the corpus — merge over-fragmented pages into topical articles under hubs, so the wiki is navigable, not a pile of stubs.

⚠️ Prime directive: loss-proof. You are rewriting memory you cannot regenerate. Every stage runs against a read-only snapshot, writes to a separate staging tree, and carries provenance. The live corpus is never edited until a verified cutover. "Loss-proof" is a property you verify (Step 7), not one you intend.

Procedure

Step 0 — Read the principles

Read references/v3-wiki-principles.md end-to-end first. It defines the v3 article skeleton, the lead-is-the-card rule, the event-vs-topic distinction, hubs (kind: index), section discipline, the injection budget, and the banned content shapes. It owns what a good v3 article is; this SKILL.md owns what order to do things in.

Then read references/loss-proofing.md — the snapshot/staging/provenance/verify contract that every later step depends on.

Step 0.5 — Preflight

Resolve each before touching anything; if a check fails, stop and fix it.

(1) CLI surface. Confirm the v3 subcommands exist:

assistant --version
assistant memory v3 --help

Expect at least backfill-sections and rebuild-index (used at cutover, Step 8). If absent, the binary predates v3 — upgrade before proceeding.

(2) Cutover switch. Memory-v3 is gated by the workspace config memory.v3.live (a boolean the assistant sets directly — no feature flag, no operator hand-off). Confirm you can read it now, without changing it:

assistant config get memory.v3.live    # expect: false — you flip it at cutover (Step 9)

If assistant config get/set memory.v3.live errors, the binary predates the config gate — upgrade before proceeding. This is also the assistant config set path you use to pause consolidation in Step 1.

(3) Workflow engine. The heavy authoring + auditing fans out through the native workflow engine. Confirm it's reachable: the run_workflow tool should be available (load the workflows skill if needed). If workflows are unavailable, you can still run this skill inline for a small corpus (Step 5 branches on size) — but a large corpus without workflows will be slow and is not recommended.

(4) Pin a high-quality model. Reform is judgment-heavy. If the CLI exposes inference sessions, pin one for the duration:

assistant inference session open quality-optimized --ttl 4h

If quality-optimized isn't a profile, assistant config get llm.profiles and open against the highest-quality one. If the command doesn't exist, proceed unpinned (and skip the close in Step 10).

(5) Size the job + confirm. Count the corpus and tell the user what they're signing up for:

find "$VELLUM_WORKSPACE_DIR/memory/concepts" -name '*.md' | wc -l
du -sh "$VELLUM_WORKSPACE_DIR/memory/concepts"

Reform of a multi-thousand-page corpus is many millions of tokens and real money. Use the CLI gate:

assistant ui confirm --message "I'll snapshot the memory wiki, rewrite every page into the v3 shape, re-organize it into topical articles, verify nothing was lost, prove it retrieves at least as well as today, and only then cut over. This is judgment-heavy LLM work and scales with corpus size — it can take a while and cost real money. The live corpus stays untouched until the final step. Proceed?"

If the user declines, stop — no snapshot, no work. If assistant ui confirm is unavailable, ask in conversation.

Step 1 — Freeze consolidation + snapshot (loss-proof staging)

Consolidation is the only process that writes memory/concepts/. If it fires mid-migration it adds pages absent from your snapshot (never reformed) that the cutover then clobbers. Freeze it in two parts — disable the triggers, then wait out any run already in flight — before you snapshot. This leaves v2 retrieval live (unlike disabling memory.v2.enabled wholesale).

(a) Record, then disable, both triggers. Consolidation fires on a size trigger (consolidation_max_buffer_lines) and a time trigger (consolidation_interval_hours) — those are the only two paths. Each lives in two namespaces: memory.substrate.X overrides memory.v2.X, and the substrate key wins whenever it is present. Freeze in the winning namespace, and record the whole substrate override object first — that object, verbatim, is what Step 9 restores:

cd "$VELLUM_WORKSPACE_DIR"   # the workspace root — NOT /workspace on local installs
assistant config get memory.substrate                           # record VERBATIM — may print "(not set)"
assistant config get memory.v2.consolidation_max_buffer_lines   # record (shipped default 100)
assistant config get memory.v2.consolidation_interval_hours     # record (shipped default 8)
assistant config set memory.substrate.consolidation_max_buffer_lines null     # size trigger off
assistant config set memory.substrate.consolidation_interval_hours 876000     # time trigger off (~100y)

Freezing via memory.v2.* is not enough: on a workspace whose substrate twins are already set, a memory.v2 write persists but changes no behavior. config set prints a Warning: naming the winning key when that happens, and config get on a shadowed memory.v2 key prints a Shadowed: line with the value actually in effect — treat either as a signal that you are editing the wrong namespace.

(b) Wait out any in-flight run. A run holds memory/.v2-state/consolidation.lock (<pid> <ms>) while working (hard-capped at 15 min) and removes it when done. The triggers are off now, so no new run can start — just wait for the lock to clear:

cd "$VELLUM_WORKSPACE_DIR"
LOCK=memory/.v2-state/consolidation.lock
for _ in $(seq 1 80); do
  [ -f "$LOCK" ] || { echo "consolidation idle"; break; }
  PID=$(awk '{print $1}' "$LOCK" 2>/dev/null)
  kill -0 "$PID" 2>/dev/null || { echo "stale lock (pid $PID dead) — proceeding"; break; }
  echo "consolidation in progress (pid $PID) — waiting…"; sleep 15
done

(c) Snapshot — paper trail + read-only baseline:

cd "$VELLUM_WORKSPACE_DIR"
git add -A && git commit -m "memory-v3-migration: start" --allow-empty
mkdir -p .mv3/snapshot .mv3/staging .mv3/provenance .mv3/audit .mv3/eval
cp -R memory/concepts .mv3/snapshot/concepts          # read-only baseline — NEVER edit
[ -f memory/.v2-state/consolidation.lock ] && echo "WARN: a run started during the copy — re-snapshot"

.mv3/ is your scratch tree, separate from live memory/. All authoring writes to .mv3/staging/. You commit again at two milestones (mid — after authoring + audit, Step 7; complete — after cutover, Step 9). All sentinels use the exact prefix memory-v3-migration: so git log --grep finds them as one set. See references/loss-proofing.md for the full contract.

Step 2 — Inventory + analyze

Classify the snapshot mechanically before designing anything. For each page note: class (from its folder/edges: — episodic / operational / phrase / person / hub), in-degree (how many pages edge to it), and rough size. The goal is to find the over-fragmentation factor: how many tiny pages should collapse into one topical article. Read the densest and most-linked pages end-to-end — they anchor the taxonomy.

For a large corpus, this analysis pass itself can be a workflow (fan out one reader per chunk → structured class/size/in-degree rows). For a small one, do it inline.

Step 3 — Design the topical taxonomy (the assistant's editorial call)

This is the judgment the reform turns on — make it deliberately, not via a cold agent. Decide:

• Hubs (kind: index) — the 5–15 topic clusters everything files under (the user, the assistant itself, major projects, domains, key people, systems). A hub routes; it does not hold body content.
• Cluster membership — which snapshot pages merge into which article under which hub. Over-fragmented siblings merge; genuinely distinct topics stay separate.
• Event vs topic split per cluster — what HAPPENED (recorded in full) vs what IS (queryable, terse).

Bottom-up auto-clustering produces a re-shaped pile, not a wiki — design the hubs top-down from what you know matters, then route pages into them. Write the taxonomy to .mv3/taxonomy.md (hubs + cluster→source-pages map). This is the routing manifest the authoring step consumes.

Step 4 — Route fragments → clusters (provenance)

Turn .mv3/taxonomy.md into a machine-readable routing manifest: .mv3/provenance/routing.json = [{ cluster, hubSlug, sourcePaths: [snapshot paths] }]. Every snapshot page must appear in exactly one cluster's sourcePaths (a page may be cross-linked later, but it has one canonical home). A page that fits nowhere goes to an explicit unrouted bucket — never dropped. The loss-audit (Step 7) checks every snapshot path against this manifest.

Step 5 — Author the wiki (the fan-out)

Author one cluster at a time, at cluster grain — a single agent reads all of a cluster's source pages and writes that cluster's whole article set (hub + leaves) into .mv3/staging/. Cluster-grain (not page-grain) keeps the agent count low and the topical judgment coherent.

Branch on size:

• Small corpus / few clusters (≲ ~30 clusters): author inline, cluster by cluster, yourself.
• Large corpus: fan out via the workflow engine. Adapt the author-cluster template in references/workflows.md (§1) and launch it with run_workflow, passing the routing manifest as args. Author leaves read their cluster's source pages, and the run declares file_write scoped to .mv3/staging/. One leaf per cluster keeps ~dozens of leaves — well under the engine's 500-agent cap. If clusters exceed a few hundred, shard across multiple run_workflow launches (≤3 concurrent); the engine's resume replays completed clusters if a run is interrupted, so a deploy mid-run loses nothing.

Each authored cluster emits .mv3/provenance/<cluster>.json mapping every staged article slug → the snapshot source paths it consumed. No article is marked final by a fan-out leaf — leave a status: draft marker; you review in Step 7.

Step 6 — Link repair

Resolve cross-references. Walk every staged article's links: and inline [[wikilinks]]; repair targets against the new flat slugs using the provenance map (old path → new slug). Classify anything still dangling (missing target / intended-but-unwritten leaf / external ref) into .mv3/audit/dangling-links.md for review. Aim to resolve the large majority; a known-dangling link is acceptable only if recorded.