Tao Generate Anomalies

Phase 1 — fine-tune (skip when MODE=inference_only)

Read references/finetune.md §Phase 1 for dataset structure, config template details, and best-checkpoint selection. Four steps: (1) validate dataset / derive anomaly types, (2) generate the validation JSONL (skip if user supplied VALIDATION_JSONL), (3) generate the training config — show it to the user and confirm before writing — (4) launch training in the background. Then derive CKPT (path encodes upper-case MODEL_SIZE) and STEP (highest nn_score step from validation logs). If MODE=finetune_only, stop after training. See references/finetune-commands.md for the exact Step 1–4 commands and the CKPT/STEP derivation snippet.

Phase 2 — prep-testcase (skip when MODE=finetune_only)

Read references/inference.md §Phase 2 for AMP routing detail and n_seeds sizing. Do NOT pass --seeds — it is auto-computed and is not a recognized flag. prep_testcase.sh defaults to --mode inference (uniform allocation across defect types, no KPI floor), which Phase 2 always uses.

${ANOMALYGEN_SCRIPTS}/prep_testcase.sh \
    --name ${NAME} --num-sdg ${NUM_SDG} \
    --dataset-dir ${DATASET_DIR} \
    --clean-dir ${CLEAN_DIR} \
    --defect-spec ${DEFECT_DESC} \
    --amp-output-dir ag_inference/${NAME}/amp \
    --output-jsonl ${JSONL}

Custom per-defect counts: when the user specifies counts per defect type, translate to --num-sdg plus a --per-defect-counts JSON dict (types absent from the dict get 0; sum should equal --num-sdg, else the script warns on stderr and uses the override sum). Confirm the allocation when intent is ambiguous. See references/guard-and-custom-counts.md for the full --per-defect-counts command example and the ambiguity-handling detail.

Phase 3 — SDG → original/

Read references/inference.md §Phase 3 for JSONL validation against the checkpoint, multi-GPU caveats, and output verification.

python3 -m scripts.utilities.validate_checkpoint ${CKPT} --step ${STEP}
python3 -m scripts.utilities.validate_jsonl ${CKPT} ${JSONL}

${ANOMALYGEN_SCRIPTS}/run_sdg.sh \
    --checkpoint_dir ${CKPT} --step ${STEP} \
    --input_jsonl ${JSONL} --output_dir ${ORIGINAL} \
    --model_size ${MODEL_SIZE} --num_gpus ${NUM_GPUS}

${ANOMALYGEN_SCRIPTS}/verify_output.sh ${JSONL} ${ORIGINAL}

Phase 4 — eval original/

Read references/inference.md §Eval for score interpretation and feature-count explanation. run_eval.sh writes per_sample.csv and eval.log inside original/ and merges nn_score into SDG_result.csv.

${ANOMALYGEN_SCRIPTS}/run_eval.sh \
    --real-path ${DATASET_DIR} --generated-path ${ORIGINAL} \
    --anomaly-types ${DEFECTS[@]}

Phase 5 — per-sample search rounds

Read references/inference.md §Phase 5 for draw strategy, ranges, and re-AMP guidance. For r in 1..NUM_SEARCH_RUN:

1. Read prior round's per_sample.csv (or ${ORIGINAL}/per_sample.csv for r=1).
2. Write ${ROUNDS}/round_${r}/draws.json with selected (guidance, crop_ratio) per sample.
3. Run round via ${ANOMALYGEN_SCRIPTS}/run_round.sh (SDG + eval; the round dir gets its own sdg/{SDG_result.csv, per_sample.csv, eval.log}). See references/inference-commands.md §Phase 5 for the full command and flags.

NUM_SEARCH_RUN=0 is valid — skip this phase entirely and let Phase 6 clone original/ into searched/.

Phase 6 — assemble searched/ (stitch only)

Always run assemble (works with 0 rounds — searched/ clones original/, so downstream always reads searched/ regardless of num_search_run). Stitch-only: copies winning images per sample-index into searched/ and carries over per-sample nn_score / mnn_score from each pick's source-round per_sample.csv. No eval — Phase 7 emits the canonical searched/eval.log.

mkdir -p ${ROUNDS}
python3 -m scripts.utilities.assemble_searched \
    --original-dir ${ORIGINAL} --original-csv ${ORIGINAL}/per_sample.csv \
    --rounds-dir ${ROUNDS} --searched-dir ${SEARCHED}

Phase 7 — filter + regen + eval (default nn_threshold=0.4)

Phase 7 runs by default (nn_threshold=0.4) on every mode=full and mode=inference_only invocation; pass nn_threshold=0 to skip it. It filters searched/ by nn_threshold, regenerates dropped samples via re-AMP (fresh (clean, submask) pairing in the same defect type) for up to 5 attempts, then falls back to best-scoring non-passing regens and finally to dropped originals, so the final bucket always equals num_SDG.

Run python3 -m scripts.utilities.filter_with_regen. Pass --allocation ag_inference/${NAME}/allocation.json so regen targets the intended per-defect counts — without it a bucket left short (e.g. by an interrupted SDG) cannot be topped back up to num_SDG. It runs the final run_eval.sh internally — the only eval against searched/. Read references/inference.md §Phase 7 for regen mechanics, source-column tracing, and the regens/regen_summary.csv schema; see references/inference-commands.md §Phase 7 for the full command and flags.

Output layout

Every bucket that gets eval'd carries the same triad of files: SDG_result.csv (generation params + nn_score + guardrail_pass), per_sample.csv (per-sample nn + mnn), and eval.log (aggregate FID / per-defect avg). Buckets live under results/<name>/ as original/ (Phase 3+4), searched/ (Phase 6 stitch + Phase 7 filter+regen+eval), rounds/round_NN/ (Phase 5, plus search_summary.csv), and regens/regen_NN/ (Phase 7, plus regen_summary.csv).

Image content guardrail. A SigLIP content-safety check runs on every generated image and records its verdict in SDG_result.csv.guardrail_pass (1 safe / 0 blocked). A blocked image is replaced with an all-black image that still occupies its slot on disk, so counting files cannot detect it — never hand a guardrail_pass=0 sample downstream. Blacked-out samples score near-zero nn_score, so Phase 7 regenerates them like any other sub-threshold sample. Disable with ANOMALYGEN_IMAGE_GUARDRAIL=0.

See references/output-layout.md for the full directory tree with per-file annotations, the guardrail semantics, and the post-run Verification checklist (image counts per bucket, search_summary.csv / regen_summary.csv row checks, the per-type nn_score / mnn_score / fid fields in each eval.log, and the guardrail_pass sweep).

Error handling

Common pipeline failure modes (missing mask dirs, short/empty AMP output and the 0 entries written halt, mid-round SDG failure resume, off-boundary step, validate_dataset.py's non-zero exit on any pairing issue, the 2B-only check.sh default, and guardrail-blocked black images) are covered in references/error-handling.md; see also references/finetune.md and references/inference.md for phase-specific error handling.