Tabular Cleanup

The loop

Let <best> be the current accepted version (starts at v0, an exact copy of <artifact>). Copy this checklist and tick items off, iterating on <best> until a stop condition fires:

• Profile <best> — run deterministic profiling code → profiles/profile-vN.json (schema below): per-column stats + every check's violation count + state + retention + headline checks_passing%; write a 4–8 line human summary.
• Pick the target & propose ONE transform — among open checks, pick by priority (severity high, then most violations). State the check id, its violation count, the repair strategy, and the expected effect. Never propose a transform you can't tie to a specific open check.
• Write the transform as a pure transform(df) -> df pandas function, deterministic, touching only what the target check requires (it must run standalone in pipeline.py later).
• Apply → candidate — load <best>, apply, write versions/vN+1.<ext>. On error, fix once; if still broken, log status=crash and discard (don't advance <best>).
• Re-profile the candidate — re-run the checklist on vN+1 for new violation counts + retention.
• Keep or revert (pure checklist logic, no epsilon) — accept iff the target check's violations strictly decreased AND no other check's violations increased AND no guardrail tripped. Else revert. If that was the last guardrail-safe strategy for the target, mark it residual.
• Log + persist — append one ledger.tsv row. If kept: copy the function to transforms/tNN_<slug>.py and append its call to pipeline.py in order.
• Check stops — if none fired, go to the next iteration on the new <best>.

Least-destructive principle (the loop's bias): prefer repair over removal. Try, and take the first that is guardrail-safe and regression-free: (a) repair (parse/standardize/ canonicalize the value), (b) impute (within <impute_cap>, flagged synthetic — nulling an unrecoverable value in a nullable:true column is repair, not imputation, and is uncapped), (c) remove (drop rows/cols, within retention + protected floors). Removal is a last resort. A check becomes residual only when all three genuinely fail — e.g. a malformed value in a nullable:false column where repair can't recover it, dropping breaches retention, and nulling would regress that column's .required check. An honest residual is correct; never null a required field or invent a value just to clear a check.

Stops

Because checks are deterministic counts, "done" is exact — no epsilon, no plateau heuristic. This is the key difference from the autoresearch loops: it is designed to terminate. Stop on any:

1. All green — every check is pass. The clean, successful exit.
2. All-residual — zero open checks remain (every check is pass or residual); the remaining violations are provably unfixable within the guardrails. The natural exit.
3. Budget — <gate>/<budget> reached. Backstop only.

Before marking the last open check residual (which triggers stop #2), confirm you actually tried all three strategies (repair → impute → remove) — don't declare it unfixable just because the first regressed another check or hit a floor.

On stop: set cleaned.<ext> to <best>, write report.md, and print the final summary (headline checks_passing%, pass/residual counts, which stop fired).

Outputs & formats

The run produces three deliverables:

1. cleaned.<ext> — the best version (copy of <best>).
2. pipeline.py — a standalone, replayable script: read raw <artifact> → apply each accepted transform in order → write cleaned. Deterministic and idempotent; re-running it on the raw dump reproduces cleaned.<ext> exactly. The audit-grade artifact.
3. ledger.tsv + report.md — the full audit trail and a before/after summary.

<sandbox_root>/tcl/ layout: schema.yaml (bindings + contract + compiled checklist), versions/ (v0.<ext> is the READ-ONLY copy of the raw artifact; new vN written per candidate), transforms/ (one file per accepted transform), profiles/, ledger.tsv, pipeline.py, report.md.

ledger.tsv — tab-separated, append-only, one row per attempt, never commas in free text. One row per schemas/ledger.schema.json; status ∈ {keep,revert,residual,crash}:

iter	transform_id	target_check	dimension	viol_before	viol_after	regressions	retention	status	rows_affected	cells_affected	summary
1	t01_drop_dupes	rows.unique	uniqueness	7	0	0	1.000	keep	7	0	remove 7 exact duplicate rows
2	t02_canon_status	status.categories	consistency	142	0	0	1.000	keep	0	142	rewrite variants to canonical set
3	-	contact.regex	validity	3	3	-	0.94	residual	-	-	repair impossible; drop breaches retention; null regresses contact.required

profiles/profile-vN.json — per schemas/profile.schema.json (per-column stats + every check's count & state + retention + headline). Compact instance:

{"version":"v2","rows":980,"cols":7,"checks_total":11,"checks_passing":9,
 "checks_passing_pct":0.82,"total_violations":3,"retention":0.98,
 "checks":[{"id":"status.categories","dimension":"consistency","scope":"status",
            "severity":"normal","violations":0,"state":"pass"}],
 "columns":[{"name":"status","dtype":"object","contract_dtype":"category",
             "pct_null":0.0,"n_unique":5}],
 "summary":"status canonicalized; one regex check residual on contact."}

report.md — v0-vs-final headline checks_passing%, the full checklist with start/end violation counts, the confirmed contract, the ordered accepted transforms (the pipeline), the residual set (with why), what was dropped/imputed, and which stop fired.

Constraints

• The raw <artifact> is never modified — all work is in <sandbox_root>/tcl/; v0 is a copy, and the loop only writes new vN versions, because the raw dump is the replay ground truth.
• Never add rows, fabricate a key/index, or invent values silently — imputation is allowed but flagged synthetic, logged, and capped at <impute_cap> per column.
• Respect the guardrails — a candidate breaching the retention floor, dropping a protected column, or exceeding the impute cap is reverted no matter how many checks it cleared, because a checklist is otherwise gameable by deletion.
• Every check is a deterministic count, not LLM judgement; the confirmed contract is the sole authority for "correct" — do not silently change the contract or compiled checklist mid-run.
• Keep/revert is pure checklist logic — keep iff the target check improves with no regression and no guardrail breach. No epsilon, no weighted score driving the decision.
• One transform per iteration, a pure pandas function tied to one open check, so each violation delta is attributable; no blind multi-step rewrites.
• A check unfixable within the guardrails becomes residual and is reported — never forced shut by fabricating or nulling required data.
• Keep pipeline.py faithful — exactly the accepted transforms in order; re-running it on the raw dump must yield cleaned.<ext>.
• Do not pause to ask whether to continue while open checks remain — setup is the only human checkpoint; the loop runs to a stop condition. Do not install packages the env lacks beyond a consented pinned install, and do not commit tcl/ to git.