Tao Run Deft Aoi Cosmos3

Skill: tao-run-deft-aoi-cosmos3

Installation

Install this application as part of the full TAO skill-bank root, not as only the companion skill folders: TAO_SKILL_BANK_PATH must point at a directory containing versions.yaml, scripts/resolve_versions_key.py, and the Cosmos model resolver scripts/resolve_tao_image.py, plus the skills/{applications,models,data,platform,core}/... tree listed in eval.config. Run bundled validation with the skill Python so dependencies match runtime: PYTHON=$(bash scripts/deft_python.sh); "$PYTHON" -m unittest tests.test_cosmos3_bare. Resolve network mode first. Missing air-gap imports are a hard stop; network-enabled setup lives only in references/network-bootstrap.md.

Execution Contract

Treat a run as a disk-backed state machine.

1. Preserve every explicit user value and show the source of each effective value (user, spec, or default) in the Pre-Flight Summary.
2. Ask which installed platform to use. Do not select Docker, SLURM, Kubernetes, Brev, virtualenv, or an external platform by default.
3. Resolve network mode, then read exactly one of references/air-gap.md or references/network-bootstrap.md. Run the selected platform skill's Preflight and stop on a missing system/native-CLI prerequisite.
4. Before any mutation or launch, invoke tao-launch-workflow and show its launch review plus this skill's Pre-Flight Summary. Wait for one explicit approval.
5. After approval, set PYTHON=$(bash scripts/deft_python.sh) and initialize ${RESULTS_DIR}/deft_state.json once with "$PYTHON" scripts/init_deft_state.py. Pass the exact GPU model reported by the selected platform's Preflight through --gpu-model (include accelerator memory when available), plus the resolved network mode/source and selected absolute Python. Never reinitialize a resumed run or edit deft_state.json by hand.
6. Before every stage, after context compaction, and before a completion claim, run "$PYTHON" scripts/deft_context.py --state ... --stage .... Use its durable next_stage and the state file's status, current_iteration, iterations.*.status, stage_completed, and latest events entry to resume. Do not infer progress from assistant prose or from an artifact that is not recorded in state.
7. Run every command that can install, fetch, log in, or launch a local container through "$PYTHON" scripts/deft_exec.py --state ... -- <command>. In an air-gap it rejects egress/package operations and enforces no-pull. Remote platforms must apply the equivalent immutable no-pull/offline policy.
8. Submit each GPU stage through the chosen platform's four verbs: submit / status / logs / cancel. The submit verb must open the job-record before native launch; the returned id is the only launch handle. Poll the backend, not the job-record, and map state to PENDING RUNNING COMPLETE ERROR CANCELED UNKNOWN.
9. Commit every completed or failed DEFT stage with "$PYTHON" scripts/commit_stage.py. It verifies the stage inputs and atomically updates both the resume snapshot and ordered events array in state. Every executed-stage commit requires a positive, measured --duration-sec: use backend elapsed wall time for submitted jobs and a host wall-clock timer for inline stages. A documented --skip may record 0; negative durations are always rejected.
10. Claim completion only after "$PYTHON" scripts/finalize_run.py verifies final Benchmark evidence, successfully commits loop_stop, and a fresh read of deft_state.json shows status == "complete", iterations.baseline.status == "complete", and the final iteration's status == "complete".

Never place secrets in a spec, command, transcript, job-record, or chat. Check credential presence only, for example [ -n "$HF_TOKEN" ] && echo SET || echo UNSET. Credentials come from the user's exported shell environment or from a user-approved env file — ~/.tao/secrets.env, ~/.config/tao/.env, or a path the user points at, never one merely found in the workspace — loaded with set -a; source /path/to/.env; set +a. Never print the file's contents or any credential value.

Cosmos3 Model Contract

• Model skill: tao-finetune-cosmos-reason.
• Supported canonical base models:
  • nvidia/Cosmos3-Nano — default;
  • nvidia/Cosmos3-Edge — only when explicitly requested;
  • nvidia/Cosmos3-Super — only when explicitly requested.
• Normalize the user aliases nano, edge, and super to those canonical IDs. Preserve any variant selected in the prompt. When no variant is selected, use Nano.
• Give hardware recommendations for the selected variant and report when the available compute is insufficient. If the prompt asks for a variant recommendation based on hardware or workload, recommend one with the tradeoff, but require an explicit selection before state initialization. Never silently switch or fall back to another variant.
• Keep the selected canonical ID as source-model lineage, but do not pass the native online checkpoint directly to Cosmos-RL.
• The published Cosmos Reason 3 reasoners ship in Cosmos3's own native Omni format (model_type="cosmos3_omni"), which Cosmos-RL cannot load. After launch approval and before baseline evaluation, run "$PYTHON" <model_skill>/scripts/prepare_cosmos3_vlm_checkpoint.py to convert the selected reasoner into a Qwen3-VL safetensors PTM, or validate and reuse an existing prepared output.
• Use the prepared PTM consistently for zero-shot evaluation, Train policy.model_name_or_path, and LoRA model.base_model_path. The model being trained is still the selected Cosmos Reason 3 reasoner — keep its canonical ID as checkpoint lineage; the Qwen3-VL PTM is only the on-disk format Cosmos-RL consumes.
• Nano may use the helper's packaged Qwen3-VL default. Edge and Super require a variant-specific, validated VLM base; never reuse Nano's conversion arguments.
• Container image: resolve the cosmos-rl backend from tao-finetune-cosmos-reason/references/skill_info.yaml with "$PYTHON" "$TAO_SKILL_BANK_PATH/scripts/resolve_tao_image.py"; never copy a Cosmos image pin into this application skill.
• Train action: cosmos-rl --config <spec.toml> /opt/cosmos_rl/tao_sft_example.py.
• Before the first evaluate job, run "$PYTHON" scripts/patch_eval_image_cap.py to source-classify the selected image. Mount its output read-only into every evaluation container only when it reports patch_required; no mount is needed for already_sufficient or cap_absent. An unrecognized cap/vLLM shape is a hard stop; see references/cosmos-reason.md.
• Workflow override: automl_policy: off. DEFT owns iteration and checkpoint selection; this is a workflow argument, not a TOML key.
• Default adaptation: LoRA over the language-side projections ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], leaving the vision tower's pretrained weights untouched. The schema also accepts "all-linear", which additionally adapts the vision linear layers; use it only when the user explicitly requests it. Derive all other Train defaults from the model skill's current template.
• Every spec is a nested dictionary serialized to TOML. Never write literal flat dotted keys into a spec.
• Do not mount user data over /workspace; cosmos-rl is installed there.
• Run every Docker container with a writable host mount as the invoking UID:GID with USER, LOGNAME, HOME=/tmp, and the read-only host passwd/group databases; never fall back to a root repair container. This covers checkpoint preparation, Train, Proxy/Benchmark evaluate, AnomalyGen, and mining. See references/cosmos-reason.md and references/tao-mine-aoi-images.md.

Read skills/models/tao-finetune-cosmos-reason/SKILL.md and its references/skill_info.yaml before authoring a spec. Start from the model skill's current packaged template for the selected action and apply only the AOI workflow overrides in references/cosmos-reason.md. Replace every dataset/output path with the chosen platform's compute-frame path. Prove that the selected Cosmos-RL image can load the prepared PTM and train the requested variant; do not reuse Nano conversion, parallelism, or memory assumptions for Edge or Super.

Bare OK/NG Contract

This migration supports one annotation mode: bare_okng.

• Each record is ShareGPT JSON with exactly two images in [AOI, golden_reference] order.
• The first human/user turn contains the inspection prompt.
• The final assistant/gpt response is exactly OK or NG; reasoning, prefixes, explanations, and final-answer wrappers are invalid training labels.
• NG is the positive class. NG -> OK is a false accept; OK -> NG is a false reject.
• Evaluation may normalize a model response by its last standalone OK/NG token, but training labels remain exact.
• Rich, reasoning, BCQ/MCQ, and task fan-out modes are outside this migration. Stop instead of silently accepting them.

Run "$PYTHON" scripts/validate_sharegpt.py on Proxy, Benchmark, Mining, and each generated iteration training file. There is no input Train annotation. Run "$PYTHON" scripts/validate_split_contract.py to prove that Proxy, Benchmark, and Mining targets are disjoint and that the frozen Benchmark annotation hash has not changed. When a generated Train file is supplied, the same validator requires its targets to come from Mining, the immediate --previous-train seed, or the current iteration's --synthetic AnomalyGen output, and to remain disjoint from Proxy and Benchmark. For iteration N>1, --previous-train is required and the validator proves that every preceding Train record was retained.

KPI Isolation

• Proxy: annotations/proxy_kpi.json. It is the only error source for RCCA, routing, mining targets, and data-mixture decisions. It never stops the loop.
• Benchmark: annotations/benchmark_kpi.json. It is frozen, evaluated at baseline and every iteration, and is the only stop-gate source. Benchmark sample errors never feed routing or mining.
• Default gate: recall_ng >= 1.0. If the user asks for accuracy, use accuracy >= <target>.
• Unknown model responses block the gate through the unknown_predictions <= 0 metric constraint.

scripts/analyze_gaps.py writes Proxy RCCA artifacts or Benchmark aggregate metrics plus metric_result.json. scripts/record_metric_result.py binds the Benchmark metric evidence to the configured metric contract.