ML Pipeline

ML Pipeline Discipline

Hard rule: no model is trained until every earlier step in the pipeline is done and the user has explicitly approved the phase gates before it. "Train a model on this data" is a request to start the pipeline at step 1, not at step 10.

The pipeline (strict order — never reorder, never skip silently)

RAW DATA
  → 1. Data inspection
  → 2. Exploratory data analysis (EDA)
  → 3. Define the prediction problem
  ──────────── GATE A: user approval ────────────
  → 4. Data cleaning
  → 5. Data engineering
  → 6. Train / validation / test split
  → 7. Feature engineering
  → 8. Preprocessing
  ──────────── GATE B: user approval ────────────
  → 9. Baseline model
  → 10. Model training
  → 11. Hyperparameter tuning
  → 12. Model evaluation
  ──────────── GATE C: user approval ────────────
  → 13. Error analysis
  → 14. Final test (test set touched ONCE)
  → 15. Deployment (only if user asks)
  → 16. Monitoring + retraining plan
  ──────────── GATE D: wrap-up report ───────────

Phase gates — explicit permission, every time

At each gate, STOP and give the user, in plain non-jargon language:

1. What was done — the steps completed, 1–2 sentences each.
2. What was found — key findings, with the visuals that show them.
3. Decisions made and why — e.g. "dropped 312 duplicate rows", "chose time-based split because the data has dates".
4. What comes next — the next phase's steps, in one short list. At Gate B this section includes the Model rationale: block (format under Progress tracking).
5. The question — ask for explicit permission to continue. Wait for a clear yes. Silence, ambiguity, or "hmm" is not a yes. If the user redirects, incorporate it.

If the user says "skip ahead" or "just train it": explain in 2–3 sentences which steps are missing and the concrete risk (usually leakage or garbage-in), then ask once for explicit override confirmation. If they confirm, proceed and record the override in PIPELINE.md.

Progress tracking (survives across sessions)

On first use in a project, create ml_pipeline/PIPELINE.md — a checklist of the 16 steps with status (todo / in progress / done / approved-gate / overridden), one line of results per finished step, and dated gate approvals. Update it after every step. On any new session, read it first and resume from the first unfinished step — never restart, never skip ahead of it.

Record each gate approval on its own line in exactly this shape — the enforcement hook parses it:

- Gate A: approved 2026-09-16
- Gate B: approved 2026-09-17
- Override: Gate B - user approved skipping to training 2026-09-17 - reason: <why>

A line that says a gate is todo, pending, or not yet approved never counts as approval.

After recording a gate approval in a git repository, checkpoint it: git add -A && git commit -m "ml-pipeline: Gate X approved" && git tag -f gate-X. Every approved phase is then a reproducible point to return to.

Before recording Gate B: approved, PIPELINE.md must contain a model rationale in exactly this shape (the hook checks the five bullets exist; the user judges their content):

Model rationale:
- traits: binary, 1,428 rows (small), minority 11.4%, has_datetime, no groups
- baseline: majority-class dummy + logistic regression (class_weight=balanced)
- candidates: gradient boosting with small trees; regularized logistic
- ruled out: neural nets (small data); k-NN (mixed scales, weak on tabular); random split (temporal)
- metric: PR-AUC primary, recall at fixed precision secondary

Tools: marimo notebooks + matplotlib visuals

• The workbench is a marimo notebook, not loose scripts. Keep notebooks in ml_pipeline/: 01_eda.py (steps 1–3), 02_prep.py (steps 4–8), 03_model.py (steps 9–12), 04_eval.py (steps 13–16). In Claude Code, drive them live with the marimo-pair skill so the user watches the work happen. In other harnesses, write the notebook files and tell the user to open them with marimo edit <file>.
• Every step that looks at data produces matplotlib figures (seaborn on top is fine). Also save each figure to ml_pipeline/figures/<step>_<name>.png so gates can reference them even without a live notebook.
• Explain every figure in 1–2 plain sentences: what it shows and why it matters for the next decision. A figure without an explanation is not done.

What each step must produce

1. Data inspection — copy the plugin's guard library to ml_pipeline/guard.py (its path is given at session start; in Codex/Kimi it is skills/ml-pipeline/lib/mlpipeline_guard.py next to this file), load the raw data read-only, and run guard.profile(df, target=..., time_col=..., group_col=...). Read ml_pipeline/data_profile.json before anything else: its leakage_suspects and traits decide the split, the metric, and the model family. Report rows × columns, column kinds, missing values, duplicates, and every leakage suspect. For images or text, build a manifest table first (path, label, size or length) and profile that. No modification yet.
2. EDA — guard.eda_figures(df, target=..., time_col=...) writes the required figures (01_missingness, 02_target_balance, 02_distributions, 02_correlations, 02_temporal_coverage when a date column exists) with explanations computed from the data; add any others with guard.fig(step, name, figure, explanation). If matplotlib is missing, install it — text descriptions are not figures and the gate will not accept them. Output: the figures, your interpretation of each, and a short list of hypotheses and problems spotted.
3. Define the prediction problem — write a short contract: target (exact definition, units), prediction unit and population, prediction time/horizon, information actually available at prediction time, objective, evaluation metric, constraints. The user must approve this contract at Gate A — it controls everything after. Consult references/model-selection.md with the profile's traits when choosing the metric and the candidate model families.
4. Data cleaning — missing values, duplicates, invalid/impossible values, inconsistent categories, unit/format issues. Report before/after counts for every rule. Document every rule in PIPELINE.md. Prefer preserving data over deleting. Never use information from the future or from the test rows to decide a cleaning rule.
5. Data engineering — joins/integration, aggregation, time alignment to an index date, business rules, one-row-per-prediction-unit feature table, data-quality checks (row counts, uniqueness, ranges).
6. Split before any fitting — copy the plugin's guard library to ml_pipeline/guard.py (its path is given at session start; in Codex/Kimi it is skills/ml-pipeline/lib/mlpipeline_guard.py next to this file) and split with it: train, val, test = guard.split(df, target=..., time_col=<col> if the data is temporal, group_col=<col> if the same entity appears in multiple rows). It drops exact duplicates, refuses a random split when a datetime column exists, keeps groups together, checks that no row lands in two splits, and freezes a fingerprint of the test set. The test set is touched exactly once, at step 14, through guard.final_test().
7. Feature engineering — design features on the training set's statistics only, then apply the same transformations to validation/test.
8. Preprocessing — scalers, encoders, imputers fit on train only, wrapped in a pipeline object so train and inference can never diverge.
9. Baseline first — a dummy predictor (majority class / mean) AND one simple model (logistic/linear regression or small tree). Record their metrics. Every later model is judged against this line; a complex model that can't beat it gets reported as such.
10. Model training — train candidate models on train, compare on validation. Log every run's config and score.
11. Hyperparameter tuning — on validation/cross-validation only. The test set is never part of tuning.
12. Model evaluation — the contract's metric plus supporting views: confusion matrix and ROC/PR curves for classification, residual plots for regression, always compared to the baseline. Figures required.
13. Error analysis — worst predictions, performance by slice/subgroup, calibration, where the model fails and a hypothesis for why.
14. Final test — score = guard.final_test(model.predict, test, target=..., metric_fn=...). It verifies the frame is the frozen test set, refuses a second call, and logs the result to PIPELINE.md. Report the number honestly, even if it is worse than validation. No going back to tune on it — if the result forces changes, agree a new test strategy with the user and record - Override: final test - user approved a second evaluation YYYY-MM-DD - reason: <why>.
15. Deployment — only when the user asks. Save the full pipeline artifact (preprocessing + model together), verify a reloaded artifact reproduces predictions, document the inference input contract.
16. Monitoring + retraining — write down: what drift to watch (input and prediction distributions), what metric threshold triggers retraining, and how retraining reuses this same pipeline from step 1.