Defect Image Generation with Cosmos AnomalyGen

Physical AI Defect Image Generation

Table of Contents

• Supported Flows
• Disambiguation (full table in references/disambiguation.md)
• Step 0: Select Flow, Cookbook, and Gather Inputs
• Common Preconditions (long-form in references/preconditions.md)
• Flow walkthroughs (one entry per flow; details in references/flows/)
• OSMO Monitoring
• Supporting files

End-to-end orchestration of defect image generation, augmentation, and labeling pipelines for AOI (Automated Optical Inspection) datasets. AnomalyGen = Cosmos-Predict2-2B finetuned per use case (Cosmos-AnomalyGen-PCB-2B, -Metal-2B, -Glass-2B). Every flow has a canonical OSMO workflow YAML in assets/configs/ that chains all steps non-interactively. Use-case cookbooks in assets/cookbooks/ provide PCBA usd2roi/image-edit configs and AnomalyGen training configs for PCBA, metal surface, and glass inspection. This skill governs flow selection, data handoffs, and submit commands; component internals live in each component's SKILL.md.

Supported Flows

All flows run on OSMO. Day 0 flows require image_edit_endpoint (Qwen Image-Edit OVSL2SL — existing URL or local deploy from references/nim/); Finetune Only has no external endpoints.

Pick the right workflow for the user's defect class

ChangeNet golden/defect pairs: submit good_image_generation.yaml + structural_defect_generation.yaml with the same --set name= (two-submission pairing convention).

Day 0 and Day 1 share the same downstream shape: a Jinja-gated finetune-job (omitted when use_pretrained_checkpoint=true) feeding anomaly-infer. Day 0 prepends usd2roi-render + augment-image-edit; Day 1 starts from <dig_url_root>/datasets/<usecase>/raw. Per-stage detail: each flow's walkthrough.

User intent → knob mapping

Every OV flow is two-stage: crop_max_emit=N caps the final per-cell crops (stage 2); render_patches=N caps raw scan-grid patches (stage 1, each yielding multiple crops). DO NOT auto-map "generate N images" → render_patches=N (wrong stage). crop_max_emit does not exist on structural_defect_generation.yaml (one crop per component — use render_patches) or texture_defect_generation_day1_real_alignment.yaml (narrow via the cookbook's crop.classes whitelist). Full knob table, smoke-test recipes, defaults, caveats: references/knob_mapping.md.

Structural-defect sizing (no crop_max_emit knob exists)

Structural output is non-linear in render_patches — doubling frames adds ~1.6–1.7× crops, not 2×. Don't use crop_max_emit (no effect) or render_patches=0 (fails). Validated yield table + target-size formula: references/flows/structural_defect_generation.md §"Sizing the output". For ambiguous "generate N images", surface the calibration table via AskUserQuestion.

---

Disambiguation: handle vague requests before committing

Underspecified prompts ("generate me some images", "run the PCBA flow", "give me defects") must not be resolved by silently assuming a flow / usecase / knob mapping. When intent is ambiguous, pause and present candidate interpretations via AskUserQuestion (2–4 mutually exclusive options) before submitting. Disambiguate the load-bearing choices: which flow, which use case, what stage a count refers to, finetune vs. passthrough.

Settled defaults you should NOT disambiguate: PCBA Day 1 → real-alignment; board → 0603_H100; image-edit endpoint → local cluster service (references/nim/); use_pretrained_checkpoint=true; Day 1 real-alignment default_spatial_dependency=cad (fall back to free only when CAD masks are unavailable, see references/flows/texture_defect_generation_day1_real_alignment.md).

dig_url_root is the one exception — NO silent default. First-time (no memory entry), MUST elicit via AskUserQuestion before any submit / osmo data upload / preflight_urls.sh. s3://osmo-workflows/dig is a suggestion to confirm, never auto-picked (~80 GB+ lands there). Later runs may reuse the remembered value silently. See Step 0 + memory rules (§4).

Full trigger table, prompt construction, and when-NOT-to-ask exceptions: references/disambiguation.md — load before assembling AskUserQuestion options for any vague request.

---

Step 0: Select Flow, Cookbook, and Gather Inputs

Before this step, if the request is vague (e.g. "generate me images", "run the PCBA flow", "give me defects"), pause and run the disambiguation cheat sheet above — present candidate interpretations via AskUserQuestion and let the user pick. Don't auto-pick a load-bearing default the user didn't actually choose.

First-time gate

If memory has no entries for this user, ASK the up-front preference questions in ONE AskUserQuestion call BEFORE any preflight / osmo / kubectl / osmo data upload, save to memory (§4), then proceed. Bundle:

• dig_url_root — MUST be elicited, not auto-picked. Offer s3://osmo-workflows/dig as a confirmable suggestion; else user provides their own OSMO-supported storage prefix. ~80 GB+ lands here. No escape hatch other than memory-recall of a previously confirmed value.
• Default OSMO --pool — candidates from osmo profile list → pool.accessible.
• Pod-template confirmation — only when osmo config show POD_TEMPLATE returns 403 (§2 has the exact question).
• Image-edit endpoint — Day 0 only: Option A (existing URL) vs Option B (deploy local NIM).

Subsequent conversations read these silently from memory. Per-flow choices (use case, checkpoint vs finetune, board, knobs) are asked each time — see below.

Preflight ordering (after the first-time gate)

Run §1 preflight_credentials.sh → §2 preflight_pod_template.sh → §3 preflight_urls.sh <flow> <usecase> → §4 generate the run stamp. Cadence: §1 and §2 are once-per-conversation gates with cross-conversation memory caching (see §4a in references/preconditions.md) — skip when memory records them as already verified / user-confirmed. §3 runs before every submit (varies by flow). §4 is the agent's job — fresh $STAMP per submit.

Pod-template enforcement is two layers: the pre-submit preflight_pod_template.sh gate (§2) plus an in-pod runtime preflight on every OV + training task (fails fast on missing /usr/share/nvidia/nvoptix.bin or /dev/shm < 16 GiB). Runtime failure despite §2 passing → template was patched out → route to physical-ai-infrastructure-setup-and-resilient-scaling. Missing creds / URL artifacts → offer to submit setup/setup_<case>.yaml + setup/setup_pretrained.yaml first.

Then ask the user in one message — per-flow choices only (the first-time gate above already covered dig_url_root, pool, pod-template, and endpoint preferences; pull those from memory):

1. Use case — PCBA (use Day 0 + pcb cookbook), metal surface (Day 1 + metal_surface cookbook), glass (Day 1 + glass cookbook), or custom?
2. Checkpoint available? — If yes (use_pretrained_checkpoint=true), use <dig_url_root>/models/<usecase> and provide checkpoint_step. If no, finetune from <dig_url_root>/datasets/<usecase>/raw.
3. Local-NIM pool capacity check (Day 0 Option B only) — before kubectl apply, check Total Capacity via physical-ai-infrastructure-setup-and-resilient-scaling. Total Capacity < 2 cannot host NIM + DIG concurrently → ask user to add GPUs or switch to Option A. image_edit_model is always nvidia/Qwen-Image-Edit-NVPCB-OVSL2SL, never generic qwen-image-edit.
4. Save user preferences to memory — after the first-time gate (and after any submit diverging from a documented default), persist load-bearing choices (dig_url_root, OSMO pool, default board, image-edit endpoint, pod-template state, osmo-admin role). Never save image_edit_model (constant — saving invites drift) or ephemeral state (STAMP, one-off anomaly_types_json). Full table: references/preconditions.md §4a "Memory rules". Read relevant memories at the start of every new conversation and apply silently.

Review the relevant flow reference before asking — most values have sensible defaults. Day 1 routing: PCBA defaults to real_alignment; metal/glass have no USD flow so always manual_roi; don't ask the user "manual or real-alignment?" for PCBA unless they explicitly ask to skip alignment.

---