Agent Wiki Ingest

Agent Wiki — Ingest (end-to-end orchestrator)

Overview

This is the one-pass entry point for turning a batch of raw trajectories into a fully-built wiki. It orchestrates the rest of the agent-wiki family in the right order so no pass is skipped — in particular the cross-trajectory consolidation pass, which is easy to forget when each skill is invoked by hand.

You — the driving agent — run this by spawning one subagent per (trace × pass), not by doing the work inline. That keeps your own context small (you never load every trace's full JSON) and lets independent passes run in parallel. Each subagent acts as the corresponding single-purpose skill (agent-wiki-summarize, -extract-guidelines, -synthesize-skill, -compare-outcomes, -consolidate-guidelines); this skill only sequences them and passes the per-trace adapter notes.

The pipeline:

0.  Convert    raw bob / claude traces → normalized analysis JSON   (skip if already normalized)
1.  Bootstrap  create wiki scaffold + seed catalog                  (skip if wiki exists)
1.5 Skip       drop traces whose summaries/<sid>.md already exists   [pre-flight — idempotency]
2.  Summarize  1 subagent / new-trace → summaries/<sid>.md          [PARALLEL]
3.  Extract    1 subagent / new-trace → guidelines/*.md (+tags)     [SEQUENTIAL]
4.  Synthesize 1 subagent / new-trace → skills/<slug>/ --archive-covered  [SEQUENTIAL]
4.5 Compare    success/failure contrasts → contrastive guidelines   [CONDITIONAL]
5.  Consolidate 1 subagent over the whole corpus → cluster pages    [SINGLE — MANDATORY]
6.  Catalog    final bookkeeping → indexes, used-by, priority       [you run this directly]

Idempotent by default. Re-running on the same source dir reprocesses nothing: Step 1.5 filters out every trace that already has a summary page, so Steps 2–4 only touch genuinely new traces. The consolidate + catalog tail always runs (it's cheap and self-idempotent). To force a redo of an already- ingested trace, keep it in the list and pass --rewrite to its render-* calls.

Why this order. synthesize-skill runs before consolidate-guidelines so skills claim recipe-level territory first (and archive the atomics they cover via --archive-covered); consolidation then clusters only the surviving atomics. This matches the consolidate skill's own rule — "don't propose clusters that overlap a skill's territory."

Why parallel vs sequential. Summarize writes one independent file per trace (summaries/<sid>.md) → safe to parallelize. Extract and synthesize both mutate shared state (guidelines/_id_index.json, skills/_id_index.json, _config.yaml, and the _archived/ moves) → run them one trace at a time to avoid lost-update races.

Input

One of:

• a list of trace file paths
• a directory of traces (the skill globs it)
• already-normalized analysis JSON files

…plus a target --wiki-root (e.g. wiki-twobatch-skills).

Detecting trace shape (Step 0 dispatch)

Read the top-level JSON keys of each input to classify it:

Step 0 — Convert

Write converted output under a stable corpus dir: trajectories/normalized/<label>/items/.

bob session JSON:

NODE_OPTIONS='' node ~/.claude/skills/bob-trace-converter/scripts/convert_bob_trace.mjs \
  <trace.json> --out-dir trajectories/normalized/<label>/items --format both

The NODE_OPTIONS='' prefix is required — some shells inject a --require preload that breaks a bare node invocation. Strip it for this call.

The converter writes three files per trace; the ingest pipeline consumes the *-openai-chat-completions.analysis.json one.

claude stream-json:

uv run python explorations/agent-wiki/experiments/harness/normalize_stream_json_transcripts.py \
  --in <transcripts-dir> --out trajectories/normalized \
  --label <label> --user-prompt "<the task prompt>"

Already normalized: skip — use the path as-is.

Collect the resulting list of analysis-JSON paths; this is the trace set the rest of the pipeline iterates.

Step 1 — Bootstrap the wiki

If <wiki-root>/_index.jsonl does not exist:

mkdir -p <wiki-root>/{summaries,guidelines,tasks,skills}
uv run python explorations/agent-wiki/skills/scripts/build_agent_wiki.py \
  --wiki-root <wiki-root> catalog

The first catalog seeds AGENTS.md and _config.yaml from the bundled defaults and writes empty indexes. Skip this whole step if the wiki already exists — you're appending to it.

Piping JSON to the helper — avoid echo

Every render-* subcommand reads JSON on stdin. The echo '<json>' | … form in the per-pass skills breaks when the payload has multi-line content/narrative fields (literal newlines become invalid control characters in the shell-quoted string). Tell every subagent to write its payload to a temp file and cat it instead:

cat /tmp/ingest-payload.json | uv run python explorations/agent-wiki/skills/scripts/build_agent_wiki.py --wiki-root <wiki-root> render-guidelines

Step 1.5 — Skip already-processed traces (pre-flight)

This is what makes re-running the skill on the same source dir cheap. The helper's render-* subcommands skip-if-exists, but only after a subagent has already read the trace and synthesized its output — so the LLM cost is already spent. Filter before spawning any subagent.

For each normalized trace, read its session_id — it lives at metadata.id (bob-converted analysis JSON) or top-level session_id (claude-normalized). If <wiki-root>/summaries/<sid>.md already exists, the trace was ingested on a prior run → drop it from the work-list. The surviving new-trace list is what Steps 2–4 iterate.

Compute the new-trace list and log what was skipped (never let a silent no-op masquerade as success):

for f in <trace-glob>; do
  sid=$(uv run python -c "import json,sys; d=json.load(open(sys.argv[1])); print(d.get('session_id') or d.get('metadata',{}).get('id',''))" "$f")
  if [ -n "$sid" ] && [ -f "<wiki-root>/summaries/$sid.md" ]; then
    echo "skip (already ingested): $sid  $f"
  else
    echo "NEW: $sid  $f"
  fi
done

The NEW: lines are the work-list for Steps 2/3/4. If every trace is skipped, that's fine — jump straight to Steps 5–6 (the tail always runs).

Override. To force reprocessing of an already-ingested trace, keep it in the work-list and pass --rewrite to its render-* calls (the helper overwrites instead of skipping).

Step 2 — Summarize (parallel subagents)

Spawn one subagent per new-trace (from Step 1.5's work-list), all in parallel. Each acts as agent-wiki-summarize (point it at that skill's SKILL.md). In each subagent prompt include:

• the analysis-JSON path and the --wiki-root
• the trace's agent (bob, claude-code, …) — it must set agent: accordingly, not hardcode claude-code
• the bob field-mapping adapter notes (only if the trace came from bob): session_id ← metadata.id; model ← top-level model; tool calls live in messages[i].content[j] blocks with type: "tool_use"; transcript_path ← metadata.source_file; recalled_guidelines is empty for a freshly-built wiki
• do NOT run catalog — the orchestrator runs it once at the end

Each subagent pipes its summary JSON to:

echo '<json>' | uv run python explorations/agent-wiki/skills/scripts/build_agent_wiki.py --wiki-root <wiki-root> render-summary

Step 3 — Extract guidelines (sequential subagents)

Spawn one subagent per new-trace, one at a time (wait for each before starting the next — they share guidelines/_id_index.json and _config.yaml). Each acts as agent-wiki-extract-guidelines. In each prompt:

• the analysis-JSON path, --wiki-root, agent, and bob adapter notes
• the list of existing guideline slugs (from prior traces this run) so it suppresses near-duplicates
• instruct it to attach a tags: array to every entity (these now propagate to both the .md frontmatter and _config.yaml — see commit that fixed render-guidelines)
• instruct it to set "agent": "<source>" on every entity. The extract-guidelines entity schema does not list agent as a field, so the subagent must add it explicitly; otherwise the page defaults to agent: claude-code even for bob traces.
• skip arc: for single-summary sessions
• do NOT run catalog

Pipe via:

echo '<json>' | uv run python explorations/agent-wiki/skills/scripts/build_agent_wiki.py --wiki-root <wiki-root> render-guidelines

Step 4 — Synthesize skills (sequential subagents)

Spawn one subagent per new-trace, one at a time (shared skills/_id_index.json plus _archived/ moves). Each acts as agent-wiki-synthesize-skill. In each prompt:

• the analysis-JSON path, --wiki-root, agent, bob adapter notes
• the list of existing skill slugs so it doesn't re-author one
• tell it to decide promote-vs-skip per that skill's "When To Use" rubric (trivial single-command recipes → skip and emit nothing)
• when promoting, pipe with --archive-covered so the atomics the skill subsumes are soft-archived:

  cat /tmp/skill-payload.json | uv run python explorations/agent-wiki/skills/scripts/build_agent_wiki.py --wiki-root <wiki-root> render-skill --archive-covered

  --archive-covered is safe to run blind: the matcher only archives an atomic from another trajectory when the skill's tags are a true superset (≥2 non-generic shared tags). The weak lexical heuristics (a slug word or format token appearing in the atomic's title) fire only for atomics from the same trajectory this skill was synthesized from, so a skill can no longer reach across into an unrelated trace's atomic on a coincidental word like "python" or "csv".
• do NOT run catalog