Data Analysis

Data Analysis Loop

A hypothesis → verify reflection loop over a dataset. The artifact is a findings report; the feedback signal is verification — a finding only counts if re-running the computation confirms it at a meaningful effect size. The discipline this enforces: no insight without a number behind it. A plausible claim the data does not support is discarded, not softened; every line in the report can be reproduced from the dataset.

When to use

Use this for open-ended, self-checking exploration of a bound dataset where each finding must survive an independent re-computation. Default to broad exploration across the columns; if the user gives a focus question, let it steer the hypotheses. Not for diagnosing one known anomaly or for checking an external claim against the literature.

Setup

Resolve bindings interactively. If loop.run.yaml exists in the working dir, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value for each binding and present it as the recommended option; on other hosts ask each as a quoted plain-text prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm the values before creating any other files.

Analysis snippets run in the user's environment via <analysis_cmd>, so they may use whatever the user has installed. Keep helper code stdlib-first (csv, statistics): if a snippet needs pandas/numpy, probe with try/except ImportError and degrade to a stdlib path, or offer a consented uv pip install "pandas==<ver>" — never assume the package is installed.

The loop

Copy this checklist and tick items off:

• Iteration 0 — profile <dataset> (shape, types, ranges, missingness); record nothing as a finding.
• Propose one specific, checkable hypothesis (steered by <question>; not already settled).
• Compute it: write <sandbox_root>/iter<N>/analysis.py, run with <analysis_cmd>, redirect to out.txt.
• Verify: re-derive the key number a second way; judge against a stated effect-size bar.
• Supported → append finding to <report> (verified); else log refuted, do not add it.
• Append a ledger row; stop on plateau (<patience>) or <budget>.

Iteration 0 — profile. Write and run a snippet that reports the shape of <dataset>: columns, inferred types, row count, and a quick summary (ranges, category counts, missingness). This grounds the hypotheses; record nothing as a finding yet.

Then, until stop (dry or budget):

1. Propose one hypothesis. A single, specific, checkable claim — e.g. "enterprise orders average higher value than consumer", "mobile has a higher return rate than other channels", "order value rises with signup tenure". Let <question> steer it; do not repeat a hypothesis already settled.
2. Compute it. Write <sandbox_root>/iter<N>/analysis.py that loads <dataset> and computes the relevant statistic plus an effect size (a group-mean difference, a rate gap, a correlation — not just a yes/no). Run it with <analysis_cmd>, redirecting output to <sandbox_root>/iter<N>/out.txt (never flood your context).
3. Verify — the gate. Re-derive the key number a second, independent way (a different grouping, a recount, or a sanity cross-check) and confirm the two agree. Then judge honestly: does the result support the hypothesis at a meaningful effect size, or is it negligible / within noise? Decide "meaningful" against a bar you state up front and apply consistently — a minimum effect size scaled to the group sizes and noise (e.g. roughly |Cohen's d| ≳ 0.2, risk ratio ≳ 1.5, or |r| ≳ 0.1, tightened when groups are small) — so the keep/refute threshold does not drift between iterations.
   • Supported → append a finding to <report>: the claim, the exact numbers, the effect size, and the method (so it is reproducible). Mark it verified.
   • Not supported / negligible → record it as refuted in the ledger and do not add it to the report. A null result is a real outcome, not a failure to hide.
4. Log one ledger row and continue.