Physical AI Video Augmentation on OSMO

Common Preconditions (all flows)

1. Credential and control-plane preflight

   bash scripts/preflight_credentials.sh --workflow assets/configs/osmo/<mode>.yaml

   Restricted egress:

   bash scripts/preflight_credentials.sh --no-probe --workflow assets/configs/osmo/<mode>.yaml

   Preflight does not require a workload-local .env. Runtime interpolation is driven by submit-time values (dataset, run_id, gpu_platform, video, storage_url, skills_dir) supplied in one --set-string list.

   Passing --workflow validates pull access for the active workflow image refs (workflow.groups[].tasks[].image) using anonymous bearer access with credential fallback when provided. If replacement NGC/HF secrets are provided in env, preflight refreshes existing nvcr_io / hf_token automatically when present. Use --refresh to force overwrite even when no new env secrets were supplied:

   bash scripts/preflight_credentials.sh --workflow assets/configs/osmo/<mode>.yaml --refresh

   If output contains USER_INPUT_REQUIRED:, ask one concise unblock question and stop.

   On workflow image 401/403, report registry access failure after probe checks on the listed image refs; do not claim a key family (for example nvapi-*) is categorically unsupported.

2. Storage interpolation policy

   storage_url must be derived from the actual dataset/upload backend for the current run.

   dataset_url=azure://storiondevxah69/osmo-workflows/datasets/vda-demo
   storage_url=azure://storiondevxah69/osmo-workflows
   dataset=vda-demo

   Never silently default to stale s3:// values on non-S3 backends.

3. Inference policy (non-negotiable)

   • Reuse healthy in-cluster persistent NIM endpoints by default.
   • If missing/unhealthy, deploy automatically — this is a prerequisite, not a user decision. Do NOT pause to ask; run the install with the VDA allow-list:

   export NIM_SERVICES="qwen3-vl qwen25-14b"
   skills/physical-ai-infrastructure-setup-and-resilient-scaling/components/inference-nim-operator/scripts/install.sh

   • See references/nim/README.md for full endpoint docs and health checks.
   • External endpoints are opt-in only (explicit request or explicit URLs); only then skip the in-cluster deploy.
   • Never infer external mode from credential presence.
   • Never scale down/delete existing NIMs to free GPUs.

4. Readiness guard

   osmo pool list --mode free
   osmo config show POD_TEMPLATE
   python3 scripts/pre_submit_guard.py --workflow assets/configs/osmo/<mode>.yaml

5. Cache auto-remediation

   If pre_submit_guard.py reports cache failure, default action is to run:

   osmo workflow submit assets/configs/osmo/setup_model_cache.yaml \
     --set-string storage_url=<backend-prefix> path=data

   Then rerun pre_submit_guard.py and submit the target VDA flow only after it passes. Ask user only when backend/prefix is ambiguous or cache setup fails.

6. Scheduling policy

   VDA templates schedule setup and workers on gpu_platform (no system pool dependency for user workloads).

Submit (all flows)

Every flow uses the same submit shape; only the workflow YAML changes. Choose the YAML for the requested flow, then run the command below. Full per-flow walkthroughs (stage matrix and flow details) live in the linked references.

SKILLS_DIR="$(cd "$(git rev-parse --show-toplevel)/skills/physical-ai-video-data-augmentation" && pwd)"
STAMP=$(cat /proc/sys/kernel/random/uuid | cut -c1-8)
osmo workflow submit assets/configs/osmo/<flow>.yaml \
  --pool <pool> \
  --set-string \
    dataset=<dataset> \
    run_id=run-$STAMP \
    storage_url=<backend-prefix> \
    gpu_platform=<gpu-platform> \
    video=<video-stem> \
    cosmos_model_cache_url=<backend-prefix>/data/models/cosmos_transfer \
    auto_labeling_model_cache_url=<backend-prefix>/data/models/auto_labeling \
    skills_dir="$SKILLS_DIR"

Compatibility note:

• Use exactly one --set-string flag and pass all the key/value pairs after it.
• Do not repeat --set/--set-string flags in the same command; some OSMO builds only honor the last occurrence.
• Do not mix --set and --set-string in one submit command.
• Pass explicit *_model_cache_url values to avoid nested-template interpolation differences across OSMO environments.
• Do not brute-force permutations of flags. Use this shape directly.

Common optional overrides (append key/value pairs to the same --set-string list):

cookbook=<scene_profile> \
vlm_url=<openai_base_url> \
llm_url=<openai_base_url> \
cosmos_model_cache_url=<url> \
auto_labeling_model_cache_url=<url>

The auto-labeling-only flow has no augmentation stage, so it omits cosmos_model_cache_url at runtime; passing it is harmless and keeps one submit shape across flows.

OSMO Monitoring

# Workflow status + task states
osmo workflow query <workflow_id> --format-type json \
  | jq '{status, tasks: [.groups[].tasks[] | {name, status, exit_code}]}'

# Logs for a specific task
osmo workflow logs <workflow_id> --task <task_name> -n 200

# Output retrieval
osmo data list --no-pager <output_url>
osmo data download <output_url> <local_dir>/

For completion artifacts, always mirror the full run output into workspace:

ROOT="$(git rev-parse --show-toplevel)"
RUN_LOCAL_DIR="$ROOT/media/vda/runs/<run_id>"
mkdir -p "$RUN_LOCAL_DIR"
osmo data download "<storage_url>/datasets/<dataset>-outputs/<run_id>/" "$RUN_LOCAL_DIR/"

For runs expected to exceed two minutes, send heartbeat updates at least every two minutes. For media evidence, emit one standalone MEDIA:<absolute-path> line per message bubble.

Execution continuity requirement:

• Heartbeats must report progress while continuing work; they are status updates, not permission prompts.
• Do not stop between green stages waiting for approval.
• Pause only on blocking failures or explicit user stop/redirect.
• If submit fails on interpolation, rerun once with the same canonical single-flag shape and corrected values; do not loop through ad-hoc flag experiments.

MEDIA formatting is strict:

• Emit exactly one line: MEDIA:/absolute/path/to/file.mp4
• Keep MEDIA: contiguous on a single line (never split across lines).
• No extra text in the same bubble.
• No code fences, bullets, or quotes around the directive.
• If render fails: retry once from a stable workspace path, then emit PNG fallback.

Post-Run Comparison Evidence (required for augmented flows)

Applies to augmentation_and_al, e2e, and e2e_super_resolution after a successful run.

Required completion output (do not stop at raw output URLs):

1. Stage full outputs + input video into workspace-local path:

   bash scripts/stage_run_artifacts.sh \
     --storage-url <storage_url> --dataset <dataset> --run-id <run_id> --video <video>

2. Render side-by-side from that local run copy:

   bash scripts/render_side_by_side.sh \
     --run-local-dir "<repo>/media/vda/runs/<run_id>" --dataset <dataset> --video <video>

3. Emit MEDIA from the local run copy and include:

   • augmentation summary from <run_local_dir>/setup_b0/configs/manifest.yaml (sampled_vars for <video>_aug0)
   • auto-labeling summary from <run_local_dir>/outputs/pseudo_labeled_augmented/<video>_aug0
   • for e2e / e2e_super_resolution, original-label summary from <run_local_dir>/outputs/pseudo_labeled/<video>

If ffmpeg is unavailable, emit input and augmented MEDIA from the same local run copy and still provide augmentation + auto-labeling summaries.

For demo runs (no user video provided), explicitly state that input came from nvidia/video-data-augmentation-demo.

Supporting files

Use these canonical locations:

• Workflows: assets/configs/osmo/*.yaml
• Runtime scripts: scripts/*.sh, scripts/*.py
• Flow walkthroughs: references/flows/*.md
• Setup and triage: references/setup.md, references/troubleshooting.md
• Images and endpoint policy: references/container-images.md, references/nim/README.md
• Cookbook tuning: assets/cookbooks/TUNING_GUIDE.md