Tao Run Automl Deft Pipeline

AutoML + DEFT Pipeline

A workflow-bridge skill that runs three phases in sequence by delegating to two existing skills — tao-run-automl for HPO and a DEFT application skill (default tao-run-deft-aoi for AOI; other skills/applications/deft-* skills for non-AOI cases) for the iterative data-improvement loop.

This skill does not re-implement AutoML or DEFT. It owns only the connective tissue: HPO spec inputs, the spec-handoff between AutoML and DEFT, and the post-DEFT AutoML re-run on the augmented dataset.

Routing policy

• User asks to "run the AOI workflow" or "improve my AOI ChangeNet model" — default to this skill, not tao-run-deft-aoi directly. The bare DEFT loop is the inner stage of this pipeline.
• User wants AutoML and DEFT chained on the same model/dataset
• User says "AutoML at both ends", "tune HPs then DEFT", "warm-start DEFT", "AutoML before and after DEFT"
• User has an AutoML-tuned spec and asks how to feed it into DEFT

When this skill does NOT apply

• User explicitly asks for the DEFT loop only ("run JUST the DEFT loop", "skip AutoML") → use tao-run-deft-aoi directly
• User wants only AutoML with no follow-on DEFT → use tao-run-automl directly
• User is doing zero-shot eval, RAG, or non-training workflows

The mental model

Phase 1 (AutoML baseline)        Phase 2 (DEFT loop, plain train)        Phase 3 (AutoML refinement)
─────────────────────────        ────────────────────────────────        ───────────────────────────
specs/baseline_spec.yaml         (Phase 1 winner pre-seeds baseline      ${RESULTS_DIR}/iter${N}/dataset/
train/base/training_set.csv       — DEFT skips its baseline train)       train_combined_iter${N}.csv
        │                                       │                                       │
        ▼                                       ▼                                       ▼
[ AutoML HPO sweep ]               [ DEFT: baseline-inference → RCA       [ AutoML HPO sweep ]
   N recommendations                 → iter 1..N (plain retrain) ]        re-tunes HPs against the
   pick best by val_loss / FAR      RCA / route / SDG / mining             DEFT-augmented dataset
        │                                       │                                       │
        ▼                                       ▼                                       ▼
best HPs spec + ckpt ─────►      DEFT-augmented CSV ───────────►        final best checkpoint
                                 + iter winner checkpoint               (the deliverable; no
                                 (Phase 3 warm-starts from it)           further retrain)

The two handoffs are:

• Phase 1 → Phase 2: a spec file AND the winning checkpoint — the bridge deep-merges Phase 1's HPs onto specs/baseline_spec.yaml, copies the checkpoint into ${RESULTS_DIR}/baseline/train/, and pre-populates deft_state.json so DEFT skips its baseline train and resumes at baseline inference → evaluate → RCA → iter 1. DEFT stays plain-train (automl_policy: off preserved).
• Phase 2 → Phase 3: a training CSV (train_combined_iter${N_final}.csv) AND the iter winner's checkpoint — the checkpoint is wired into each rec's train.pretrained_model_path so Phase 3 fine-tunes from Phase 2's winner. Phase 3's winning checkpoint is the deliverable; no separate retrain after Phase 3.

See references/phase-handoffs.md for the exact steps, code, and DEFT-honors-this-handoff details of both handoffs.

Why three phases instead of two

• Phase 1 alone finds good HPs on the original training distribution, but the model still has the distributional gaps DEFT is designed to fill.
• Phase 2 alone (just DEFT) fills the gaps but uses whatever HPs specs/baseline_spec.yaml was hand-authored with — usually not optimal.
• Phase 3 alone would run AutoML against the augmented dataset, but without a tuned baseline the DEFT loop's iteration cost is higher (slower convergence, more iterations to hit the KPI).

Running all three: AutoML cheap-tunes once on the original data, DEFT does the heavy data work with reasonable HPs, then AutoML tunes again on the now-richer dataset. Phase 3 is the most important of the three for the final deployed FAR/recall.

Cost up-front

The pipeline is sequential. Total wall-clock ≈ Phase 1 (N_automl × per-rec train) + Phase 2 (M iterations × per-iter cost) + Phase 3 (N_automl × per-rec train).

Note that Phase 2 has no separate baseline train — Phase 1's winning checkpoint is reused as DEFT's baseline, so the baseline cost lands inside Phase 1's N_automl trainings rather than as an extra retrain. Surface this to the user before kickoff. Typically Phase 2's iterations still dominate (each includes SDG + retrain), but Phase 1 and Phase 3 each add several hours on a single-GPU box. Use the per-job estimate from the user's setup (if they have one) rather than guessing minutes. See references/pitfalls-and-quality-checks.md (Compute budget) for the per-phase term breakdown.

Consolidated Pre-Flight — one gate, all three phases

The pipeline has exactly one user gate. Before any side-effecting action (docker pull, docker login, any job-launch call delegated to a downstream skill, file mutations under ${RESULTS_DIR}/), the agent must produce a single consolidated Pre-Flight Summary that subsumes every downstream skill's preflight. Once the user approves, the run is autonomous through all three phases — no further interactive pauses.

The user explicitly does not want to be paged between phases. The DEFT loop's own inline ## Pre-Flight Summary gate becomes a zero-question display step (every value pre-supplied from this consolidated gate), as does tao-run-automl's shared launch preflight in Phase 1 and Phase 3.

Before printing the summary, the agent must open and read every downstream skill's preflight section in full, run every read-only check those sections prescribe, and surface the outcome of each check. The summary has nine mandatory sections (workspace/host/platform/network; credentials status; container images; dataset table; Phase 1 config; Phase 2 config; Phase 3 config; compute estimate; confirmation line). After the gate, every downstream interactive gate is suppressed by passing through the collected values. The only allowed post-gate pauses are mid-run hard-stop safety gates the downstream skill cannot bypass.

See references/consolidated-preflight.md for: the full list of preflight sections to read, the required DEFT ## Pre-Flight run, the exact nine-section summary contents, the value pass-through for gate suppression, and the procedure when the skill bank version doesn't yet support gate suppression.

Phase 1 — AutoML baseline

Invoke tao-skill-bank:tao-run-automl with:

After the sweep finishes, AutoML's result["best"]["specs"] is the winning hyperparameter dict.

Handoff to Phase 2

Phase 1 hands over two artifacts: the winning spec and the winning checkpoint. Retraining the same HPs in DEFT's baseline step is wasted compute — instead, pre-seed DEFT's baseline state from Phase 1's outputs so DEFT starts at baseline inference → evaluate → RCA → iter 1. This is a four-step bridge (write merged spec → pre-seed baseline/train/ → initialise deft_state.json with baseline already done → invoke DEFT), followed by a quality check of the winning checkpoint (per-class prediction counts; compare to zero-shot ChangeNet).

See references/phase-handoffs.md for the verbatim Steps 1–4 (including the cp command and the deft_state.json pre-seed code) and the quality-check checklist.

Phase 2 — DEFT loop (plain training, baseline pre-seeded from Phase 1)

Invoke tao-skill-bank:tao-run-deft-aoi (read its SKILL.md for the full interface). For non-AOI applications, invoke the matching DEFT skill; the handoff shape is the same.

The DEFT loop's baseline-train sub-step is skipped. Phase 1 already produced a checkpoint trained at the winning HPs, and Phase 1's handoff (see references/phase-handoffs.md) pre-populated ${RESULTS_DIR}/baseline/train/ and ${RESULTS_DIR}/deft_state.json so DEFT resumes at baseline inference → evaluate → RCA → iter 1. The rest of the DEFT loop runs unchanged. Do not modify its automl_policy: off invariant.

The DEFT loop owns: its Pre-Flight Summary display step (not a fresh user gate — the Consolidated Pre-Flight above is the single gate; the DEFT summary still records the pre-seeded baseline/train/ source and must not re-prompt); baseline inference → evaluate → RCA on the pre-seeded checkpoint; the full per-iteration RCA → routing → SDG → mining → assemble → train cycle; KPI gating and stop conditions; and the ${RESULTS_DIR}/ layout (deft_state.json, DEFT_Loop_Report.html).

After the loop exits (KPI met or max_iterations reached), capture two values from deft_state.json: iterations.<best>.best_ckpt_path (the loop's best plain-train checkpoint) and the final iteration label N_final (used to locate the augmented training CSV).

If the DEFT loop hard-stops on an unrecoverable gate, skip Phase 3. There is no validated augmented CSV to feed AutoML.