Tao Validate Recipe Transfer

CV Recipe Transfer

Port published CV work onto customer data without inheriting the silent bugs that make the resulting numbers meaningless.

The core idea

This work splits into two phases with fundamentally different epistemic status:

• Phase A (Port) — get the official code running faithfully. There is external ground truth here: the authors' released checkpoint and reported number.
• Phase B (Transfer) — move the recipe onto customer data. There is no external ground truth. The paper's number stops being meaningful the moment the dataset changes.

Everything verifiable must be verified in Phase A, because the safety net disappears at the boundary. Once you are in Phase B, an implementation bug and a genuine domain mismatch produce identical symptoms: a mediocre number with a clean-looking loss curve.

Two consequences that drive everything below:

1. Never skip the Phase A gate to "save time." It costs hours and it is the only thing that licenses trust in every number that follows.
2. Stop treating the paper's number as the target once the dataset changes. The target is beat the strongest cheap baseline on the customer's own test set, measured at a deliberately chosen operating point.

Mode selection

Read the request and pick a mode. Say which one you picked and why.

Post-mortem mode — a run already happened and the result was wrong, disappointing, or suspiciously good. Go to references/postmortem.md. Do not start by rerunning anything. Walk the checks in likelihood order and find the gate that was skipped.

Forward mode — no run yet, or starting over. Work Phase 0 → A → G → T → R → E below.

Audit mode — a pipeline exists and works, someone wants it checked before it goes to a customer. Run the Phase A gate and the Phase G report against the existing artifacts, then the parity and leakage scripts. Skip the porting work.

If the request is ambiguous, ask which of the three it is before doing anything expensive.

Phase 0 — Feasibility gate

Cheap, and it prevents the most expensive class of mistake. Do this before reading any code.

Ask and answer, in writing:

1. Does this need training at all? Zero-shot and promptable models cover a lot of what used to require a fine-tune — Grounding DINO, SAM-family, OWL-ViT, CLIP, and plain pretrained COCO detectors. If the customer's classes overlap COCO/LVIS vocabulary, or the task is "find the obvious object," measure a pretrained baseline first. A surprising fraction of three-week fine-tuning projects are answered by an afternoon of prompting.
2. Is this the right paper? Papers get picked by benchmark rank, but rank on COCO predicts very little about performance on 800 images of factory parts. Check that the paper's benchmark resembles the customer's data on the axes in Phase G.
3. License. Research-only, non-commercial, and AGPL clauses are common in CV repos and this is customer-facing work. Check the repo license, the weights license (often different from the code), and the license of any pretrained backbone. This is a hard gate — flag it now, not after three weeks of work.
4. Is there enough labeled data, and does a test set exist that reflects production? If the only data is a random export, the test set will be optimistic no matter what you do.

Record the answers. If the feasibility gate fails, say so plainly and stop — that is a successful outcome of this skill, not a failure.

Phase A — Port

Goal: prove the code, the data pipeline, the preprocessing, and the metric implementation are all faithful. Read references/port-gate.md for the detailed procedure.

The gate, in short:

1. Pin the commit. Repos drift and released code routinely differs from the paper. Pin the hash. Then read the repo's issue threads — undocumented hyperparameters, known discrepancies with the paper, and "why can't I reproduce Table 2" live there. Assume the code and the paper disagree somewhere until proven otherwise.
2. Build the environment. Custom CUDA ops (deformable attention, DCN, custom NMS) in older research repos are the single biggest time sink specific to CV. Detect them early and containerize. Do not fight the host toolchain.
3. THE GATE: run their checkpoint through their eval on their benchmark and match the reported number. This single check validates environment, data pipeline, preprocessing, and metric implementation without training anything. Hours, not days.
   • Match within ~0.2 points: pass.
   • Off by 0.5–2 points: something in preprocessing or eval config is wrong. Do not proceed.
   • Off by a lot: wrong checkpoint, wrong class mapping, or a silently partial state-dict load. See the failure atlas.
4. Validate the training path with a short run on a benchmark subset. You are checking loss-curve shape and that it descends as expected, not the final number.
5. Scan for hardcoded benchmark assumptions. Official code is written for exactly one dataset. Class counts, normalization constants, anchor priors, path structure, and sometimes image size are baked in — occasionally inside the model definition rather than the config. Finding all of them is most of the porting work. references/port-gate.md has the grep patterns.

Do not enter Phase B until step 3 passes. If it cannot be made to pass, say so and report the gap — "this repo does not reproduce its own paper" is a legitimate and valuable finding.

Phase G — Domain gap report

Make the gap measurable rather than assumed. Run:

python scripts/domain_gap_report.py --source <source_annotations> --target <target_annotations> --out gap_report.md

It compares the source benchmark and the customer set on image count, resolution and aspect distribution, objects per image, class balance, and — most importantly — object size as a fraction of image area.

That last axis drives more recipe decisions than any other. COCO objects average a few percent of image area. If the customer's aerial, microscopy, or inspection imagery puts them at 0.1%, that immediately implicates input resolution, FPN level assignment, anchor scales, and rules out mosaic-style augmentation. The script emits these as explicit flags with reasons, not generic warnings.

The magnitude of the gap on each axis tells you which recipe fields in Phase T are at risk. Carry the flags forward — they are the justification for every deviation from the paper.

Also run split hygiene here, before any training:

python scripts/split_leakage_check.py --splits train=<dir> val=<dir> test=<dir> --report leakage.md

Near-duplicate images across splits are the single biggest source of inflated CV numbers. Video frames sampled at 5fps, the same scene from a fixed camera, the same patient, the same production lot — a random split gives 0.95 mAP and a model that collapses on deployment. Detectable via perceptual hashing and group metadata; almost nobody checks. If this skill does one thing well, make it this.

Phase T — Transfer the recipe

The intellectual core. Classify every field in the paper's recipe into one of three buckets before writing a config. references/recipe-fields.md has the full field-by-field table; the buckets are:

• Transfer-invariant — architecture, loss formulation, optimizer choice, most augmentation types. Carry over unchanged.
• Scale-dependent — schedule length, LR (via batch size and linear scaling), augmentation strength, weight decay, EMA decay. Must be adjusted for dataset size, and the direction of adjustment is usually predictable.
• Dataset-derived — anchor and box size priors, input resolution, class weighting, FPN level assignment, NMS/score thresholds. Must be recomputed from the customer's data, never copied.

The trap that catches almost everyone

If you initialize from the authors' released checkpoint — which you almost always should on a small customer dataset — their schedule is wrong for you by a large factor. Their recipe assumes ImageNet init or from-scratch training on 118k images. You are fine-tuning on 2,000. Copying the paper's 300-epoch schedule is a common, expensive, and entirely predictable mistake. Expect: much shorter schedule, lower LR (often 10x), shorter warmup, possibly a frozen stem, and weaker augmentation.

State the initialization explicitly in the config and derive the schedule from it. If you cannot say in one sentence whether you are fine-tuning or training from scratch, stop and resolve that first.