Compare Runs

Compare Runs

Overview

Retort is a Design of Experiments engine — every run sits at a point in the factor space, and replicates exist to measure noise. Unlike pourpoise's compare-attempts (which maintains a ranked leaderboard of ad-hoc submissions), comparing retort runs is about factor effects and qualitative divergence, not "who won".

This skill is the qualitative counterpart to retort analyze / retort report effects. The ANOVA tells you whether a factor is significant. This skill tells you what changed in the generated code as you varied it.

Parameters

• experiment_dir (required): e.g. experiment-1/. Must contain retort.db and a runs/ directory of per-run archives, each already evaluated by evaluate-run (i.e. each rep<N>/ has an evaluation.md and findings.jsonl).
• output_file (optional, default: {experiment_dir}/reports/comparison.md): Where to write the comparison report.
• group_by (optional, default: all factors): Comma-separated factors whose effect you want to isolate. The other factors are aggregated over.
• include_failed (optional, default: true): Whether to include failed runs in the comparison (they're still informative — failure itself is a finding).

Steps

1. Discover evaluated runs

find {experiment_dir}/runs -name evaluation.md -not -path "*/salvaged-*/*" | sort

Parse each path to extract the cell (from the parent directory name, e.g. language=rust_model=opus_tooling=beads) and replicate (from the rep<N> or rep<N>-failed segment).

Constraints:

• You MUST skip runs under salvaged-*/ — those are manually-archived artifacts from before auto-archival existed, and their layout is not regular.
• You MUST handle both rep<N>/ (success) and rep<N>-failed/ (failure) directory names.
• You SHOULD warn but not fail if some cells have no evaluation (evaluator hadn't run yet).

2. Load findings and metrics

For each evaluated run, load:

• Factors (from the cell directory name or the run's stack.json)
• Summary metrics (from evaluation.md header: requirements pass count, test counts, build status)
• Findings (from findings.jsonl) — categorize by kind and severity
• Architecture signal (from summary/index.md: the "Shape" one-liner)

Constraints:

• You MUST NOT re-evaluate. If evaluation.md is stale relative to the source, add a stale tag but use what's there.
• You MUST preserve replicates separately — do not average them into the cell yet.

3. Aggregate per cell

Group runs by cell (same factor combination across replicates):

Constraints:

• You MUST show mean ± sample standard deviation, not just mean. Variance across replicates is a first-class signal.
• "Distinct shapes" counts unique architecture summaries across replicates — a cell where the agent chose a different framework each time is a signal worth surfacing.
• Cells with only one replicate: drop the ±sd and mark as n=1.

4. Surface factor effects

For each factor in group_by, hold the others roughly fixed and report the effect:

## Effect of `tooling` (none vs beads)

Aggregating over language and model:

| Tooling | Mean req pass | Mean token cost | Mean build time | Build fail rate |
|---------|---------------|-----------------|-----------------|-----------------|
| none    | 8.1 / 10      | 289K tokens     | 124s            | 1/18 (6%)       |
| beads   | 7.2 / 10      | 441K tokens     | 156s            | 2/18 (11%)      |

Direction: beads costs ~52% more tokens and scored 11% lower on requirements.
This matches the p < 0.10 effect reported by `retort analyze`.

Qualitative: with `tooling: beads`, agents spent ~30% of their turns
on bd bookkeeping (visible in run transcripts). This accounts for the
token overhead.

Constraints:

• You MUST NOT claim statistical significance yourself — cite retort analyze or note "directional only".
• You MUST cross-reference the structured effect size reported by retort report effects when available at {experiment_dir}/reports/analysis/*.md.
• Qualitative claims MUST cite specific run IDs or findings (see rep3 of lang=python,model=opus,tooling=beads).