Agent Wiki Tasks

Agent Wiki — Task Comparisons

Overview

Two cognitive moves in one pass:

1. Discover — read across all summaries and identify task families (groups of sessions that attempted the same thing across trials and conditions).
2. Compare — for each family, write a tasks/<slug>__task.md page with a per-trial table and a findings narrative that calls out the experimental signal.

This is the cross-trajectory analysis pass of the agent-wiki family.

When to run

• After enough summaries exist that a comparative pattern is visible (typically ≥3 sessions per family).
• When the experiment design (e.g. trial × condition matrices) explicitly cries out for a comparison page.

Workflow

Step 1: Read the corpus

uv run python explorations/agent-wiki/skills/scripts/build_agent_wiki.py dump-summaries > /tmp/summaries.json

Output is a JSON array of one row per summary: {session_id, goal, family, trial, condition, tool_calls, errors, recall_used, summary_filename}. family, trial, condition come from existing classification rules — they may be null if no rule has matched yet.

Read the file:

Read /tmp/summaries.json

Step 2: Decide task families

For each candidate task family:

• Slug: kebab-case identifier (e.g. extract-focal-length).
• Family: short label used to group sessions (often equals slug, but can be looser e.g. focal-length for a slug extract-focal-length).
• Family-match rules: how a future session gets classified. Currently supported: goal_substring: [list of substrings]. A session matches the family if its goal contains any substring (case-insensitive).
• Tags: a few short tags.
• Intro: 1–2 sentences setting up the question.
• Findings: 2–5 bullets summarizing what the data shows. This is the actual product — a comparison page without findings is just a table.

Rules:

1. A family needs ≥3 sessions. Smaller groups should not get their own page.
2. Findings must be evidence-grounded. Cite tool-call counts, error counts, recall-used Y/N from the dump.
3. Don't repeat what's in the table. Findings should explain why the metrics differ, not restate them.
4. Use overrides for sessions whose goal doesn't auto-match. The override key in _config.yaml/session_family_overrides is the session id.

Step 3: For each family, output JSON

{
  "slug": "extract-focal-length",
  "title": "Extract focal length from JPEG EXIF",
  "family": "focal-length",
  "family_match": {
    "goal_substring": ["focal length"]
  },
  "intro": "Question template: *what focal length was used to take @sample.jpg?* FocalLength (tag 0x920A) and FocalLengthIn35mmFilm (tag 0xA405) live in the Exif sub-IFD.",
  "findings": "**Net signal:** the gap between IFD0/GPS-only scripts and the Exif sub-IFD is the dominant cost. Sessions whose recall pointed at a script that already covered the sub-IFD finished in 2-3 tool calls; sessions that had to write an inline parser took 5+.",
  "tags": ["exif", "focal-length", "comparison"]
}

Pipe to:

echo '<json>' | uv run python explorations/agent-wiki/skills/scripts/build_agent_wiki.py render-task

The helper:

• Updates _config.yaml/tasks.<slug> entry.
• Reads classified sessions; selects those matching family.
• Writes tasks/<slug>__task.md with the per-trial table + findings.

Step 4: Add overrides if needed

If a session that should be in a family didn't classify automatically, patch _config.yaml:

echo '{"session_family_overrides": {"<session-id>": {"family": "image-dims", "trial": 0, "condition": "claude_md_strong"}}}' \
  | uv run python explorations/agent-wiki/skills/scripts/build_agent_wiki.py update-config

Step 5: Subtask pass — mandatory before refresh

Before refreshing indexes, scan the corpus for subtask candidates. The default reflex of "the dataset is uniform, no subtasks needed" is wrong for almost every dataset; even a 30-session benchmark of short workflows typically has 4-6 subtask-worthy sessions. See "## Subtasks" below for the heuristics + JSON contract + a worked example.

The minimum viable subtask layer for a condition × trial dataset: one subtask per condition, anchored in the session that best demonstrates that condition's distinctive behavior. Don't write 5 redundant subtasks when 1 representative captures the pattern.

Step 6: Refresh indexes

uv run python explorations/agent-wiki/skills/scripts/build_agent_wiki.py catalog

This re-reads _config.yaml, re-classifies every summary, regenerates each tasks/<slug>__task.md, scans tasks/<slug>__subtask.md files, and regenerates tasks/index.md and the root index.md.

Subtasks: per-session workstream pages

The tasks/ directory holds two kinds of pages distinguished by filename suffix:

• <slug>__task.md — cross-session task-comparisons (the workflow above).
• <slug>__subtask.md — narrative slices of a single session.

After Step 5 above, run a second pass to scan for subtask candidates. Don't skip this just because the dataset is uniform — a 30-session benchmark of short workflows still has 4-6 subtask-worthy sessions. The default "there are no subtasks worth writing" reflex is wrong for almost every dataset.