Physical AI Event Video Generation

Procedure

1. Determine the input source.

   • For local data, validate before upload:

     python scripts/upload_images.py --path /path/to/seed-images --validate-only

   • Then upload:

     python scripts/upload_images.py \
       --path /path/to/seed-images --destination-path event-video-generation/my-run

   • For an existing storage URL, use it unchanged after confirming it names either a single image or a flat directory of images (.jpg, .jpeg, .png, .bmp, .gif, .tiff, .webp). Unlike person-crop workflows, there is no subdirectory convention — every matching file directly under input_path is one input image. When input_path names a directory, the DAG sorts matching files and takes the first max_images of them.

2. Select service mode.

   • external requires explicit VLM, LLM, and image2video endpoint URLs.
   • internal lets the DAG's service lifecycle deploy all three services in-cluster.
   • Choose service mode independently from controller placement. A local controller may use external inference endpoints.
   • Keep nested service mode and output directory consistent with the top level.
   • On Kubernetes, VLM and LLM each claim one GPU from k8s_gpu_1, but image2video claims two GPUs per replica (gpu_count: 2, host_ipc: true) — four GPUs for the internally deployed endpoints alone. That's on top of, not instead of, the three GPUs detection_and_tracking/captioning/visual_qa always claim from k8s_gpu_task regardless of service mode (see the readiness-check GPU breakdown above): external mode needs a minimum of three allocatable GPUs, internal mode a minimum of seven — not zero and four.

3. Read payload-contract.md, then render a payload from the values collected above. Do not copy checked-in dev or CI payloads — they contain deployment- specific endpoint URLs and bucket paths that must not be inherited by user runs.

   External:

   python scripts/payload.py render \
     --input-path s3://bucket/input/seed-images/ \
     --output-directory s3://bucket/output/event-video-generation/ \
     --service-mode external \
     --vlm-url https://vlm.example/v1 \
     --llm-url https://llm.example/v1 \
     --image2video-url https://image2video.example/v1 \
     --max-images 10 --num-augmentation 3 \
     --variable-distribution assets/variable-distribution.json \
     --output /tmp/evg-payload.json

   Internal:

   python scripts/payload.py render \
     --input-path s3://bucket/input/seed-images/ \
     --output-directory s3://bucket/output/event-video-generation/ \
     --service-mode internal \
     --max-images 10 --num-augmentation 3 \
     --output /tmp/evg-payload.json

   Show the user the rendered payload (or its validated contents) and get explicit confirmation before proceeding. Only continue to preflight and triggering if they confirm; if they want changes, re-render and re-confirm.

4. Preflight the DAG through the Airflow API. Check that the DAG is loaded, required pools have slots, and controller pods are healthy — see airflow-direct-api.md#preflight-direct-path. Confirm presence only; never print credential values.

5. Submit exactly one DAG run. Pass the payload from step 3 as conf.payload — see airflow-direct-api.md#trigger-a-run for the full request shape. Record and return the dag_run_id, input path, output directory, and service mode.

6. Immediately after triggering — without waiting to be asked — monitor the run until it reaches a terminal state (success or failed). Poll the Airflow API every 60–120 seconds:

   # Poll run state
   RESPONSE=$(curl -s -H "Authorization: Bearer $TOKEN" \
     "$AIRFLOW_URL/api/v2/dags/$DAG_ID/dagRuns/$RUN_ID")
   RESPONSE="$RESPONSE" python3 -c "import json, os; print(json.loads(os.environ['RESPONSE'])['state'])"

   For a per-task breakdown when state is running or failed, see airflow-direct-api.md.

   Stop polling as soon as the run state is success or failed. Use the polling loop that fits your runtime — a shell while loop, a background process, or a tool-native scheduler. Do not block the user waiting for each poll; report state changes as they occur.

   Tell the user they can also watch progress live in the Airflow UI. make port-forward runs in the foreground and never exits, so start it as a background job — and prefer that the user runs it in their own terminal, since an agent-owned forward dies with the session. Resolve the host's real address rather than reporting a placeholder or localhost, which is meaningless from another machine:

   HOST_IP=$(hostname -I | awk '{print $1}')
   echo "Airflow UI: http://$HOST_IP:8080"

   Default credentials are admin/admin, defined in deploy/values.yaml under airflow.createUserJob.defaultUser (not webserver.defaultUser). Update them before production use.

   For a full per-task breakdown see airflow-direct-api.md.

   To stop an in-progress run: open the Airflow UI, find the active DagRun, locate the running task, and mark it Failed (task menu → Mark Failed). This triggers the DAG's shutdown path, cleaning up Deployments, Services, and GPU pods. Do not delete the DagRun or the DAG — that bypasses cleanup and leaves stale cluster resources.

7. After the run reaches success or failed, ask the user: "Would you like to download and analyze the results?" Do not download automatically — wait for confirmation.

   If the user confirms, use whatever AWS credentials are already available in the shell environment (standard AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_DEFAULT_REGION, an AWS profile, or instance role). Never ask the user to paste credentials into the prompt. Run artifacts live under <output_directory>/<run_id>/, where <output_directory> is the payload value and <run_id> is the dag_run_id from step 5. The final dataset is in anomaly_dataset/:

   aws s3 sync "<output_directory>/<run_id>/anomaly_dataset/" /tmp/evg-results/
   python scripts/summarize_results.py --results-dir /tmp/evg-results

   To inspect intermediate generated videos instead, sync <output_directory>/<run_id>/cosmos/ and read metadata.json from each <video_key>/<augmentation_index>/ folder.

   Read outputs.md before interpreting files.

Guardrails

• Never use default endpoint URLs, bucket paths, or input paths from the codebase or checked-in payloads. Always ask the user for every deployment-specific value before building a payload. If a required value is missing, stop and ask — do not substitute a guess.
• Preserve explicit user inputs and endpoint/model selections throughout the session.
• Do not submit if payload validation, local dataset validation, or Airflow preflight fails.
• Do not show AWS credentials, Airflow bearer tokens, or S3 signed URLs.
• Do not start multiple runs unless the user explicitly requests them.
• Ask for a dataset location if none was supplied; this workflow has no implicit demo dataset.
• Do not invent generation-only or labeling-only DAG IDs — only the DAG listed above exists.
• Only offer a platform whose manifest exists and whose DAG is loaded in Airflow.
• Do not route person-crop clothing/attribute augmentation requests here — that is image-attribute-augmentation-workflow.

References

• Read setup-and-preflight.md for environment, storage, and policy requirements.
• Read payload-contract.md when creating or changing a payload.
• Read outputs.md when retrieving or interpreting results.
• Read troubleshooting.md after validation, API, or runtime errors.
• Read airflow-direct-api.md for all Airflow interactions: preflight, triggering, monitoring, per-task breakdown, and log retrieval.