Power Analysis

The loop

Copy this checklist and tick items off:

• Iteration 0 — draft the design to <design_doc>; record nothing as final.
• Simulate power: run <power_cmd> at the current n and the assumed effect.
• Solve N: if power < <target_power>, re-run at larger n (step up, then bisect) until it clears.
• Audit validity: list every flaw from the checklist below.
• Revise: fix the highest-priority flaw, set n to the power-adequate value, update <design_doc> (+ Preregistration section).
• Append a ledger row; stop when power clears the target and no flaws remain, or at <budget>.

Iteration 0 — draft. Write a first design to <design_doc>: the arms/conditions, the unit of analysis and how units are assigned, the primary outcome and the exact planned test, the assumed effect size (from <outcome>), and a first sample-size guess. Record nothing as final yet.

Then, until stop (power met + no flaws, or budget):

1. Simulate power. Run <power_cmd> at the current per-group n and the assumed effect, with the --design matching the planned test. Record the achieved power.
2. Solve N. If power < <target_power>, re-run the simulation at larger n — step up (e.g. double), then bisect — until power clears the target, and adopt that n.
3. Audit validity. Check the design against the checklist and list every flaw found:
   • Confounding / no control — is there a concurrent control group, or is the comparison against a historical/other-source baseline that differs in other ways?
   • Randomization — are units randomly assigned? If not, selection bias threatens any effect.
   • Selection / sampling — is the sample representative of the population the claim is about?
   • Multiple comparisons — more than one outcome/subgroup tested without correction?
   • Optional stopping / peeking — is there a pre-specified stopping rule, or will analysis run repeatedly until significant?
   • Outcome & analysis pre-specification — are the primary outcome and its single planned test fixed in advance (not chosen after seeing data)?
   • Measurement — is the outcome measured reliably and blind to condition where possible?
4. Revise. Fix the highest-priority flaw (or a tightly-coupled pair that cannot be fixed independently, such as adding a concurrent control and randomizing assignment to it) and set n to the power-adequate value. Update <design_doc>, including a Preregistration section: hypothesis, primary outcome, the one planned analysis, sample size + how it was derived, randomization scheme, and the stopping rule.
5. Log one ledger row and continue.

Stop when power ≥ <target_power> and the flaw list is empty, or at <budget>. Report the final design + preregistration, the achieved power and required n, and — if stopping on budget — the flaws still outstanding.

Ledger

<sandbox_root>/ledger.tsv, tab-separated, never commas in the text. Header:

iter	n_per_group	power	open_flaws	change

Example:

iter	n_per_group	power	open_flaws	change
0	50	0.50	2	draft: volunteers vs last-year cohort, n=50
1	100	0.80	1	solved n for 80% power at d=0.4
2	100	0.80	0	randomized concurrent control; pre-specified single primary outcome + stopping rule

Report the best iteration: the final design, the achieved power and required n, and any flaws still open if stopping on budget.

Constraints

• Power is computed at the minimal effect of interest, not an optimistic one, because a design powered for an effect bigger than reality detects nothing real — and the --design in the simulation must match the test named in the design. Do not edit tools/power_sim.py.
• A design does not pass on power alone — an adequately powered but confounded or non-randomized design still fails; both gates (power and flaws) must clear.
• Preregister before data, so the eventual test is confirmatory rather than chosen after seeing results: the analysis, outcome, sample size, and stopping rule are fixed in advance.
• One primary outcome and one planned test drive the power and the verdict; secondary analyses are labeled exploratory.
• The sandbox is self-contained — no ../ escapes. Do not pause the loop to ask whether to continue.