Vss Generate Video Calibration

When to Use This Skill

Run AutoMagicCalib end-to-end on local files, RTSP streams, or the bundled sample dataset and (when needed) deploy the AMC microservice.

Do not use for non-AMC camera calibration or runtime analytics.

Workflow

Follow the routing tables and step-by-step workflows below. Each section that ends in workflow, quick start, or flow is intended to be executed top-to-bottom. Detailed reference material lives in references/; load only the reference needed for the selected input mode.

Examples

Worked end-to-end examples are kept under evals/ (each *.json manifest contains a runnable scenario) and inline in the per-workflow curl blocks below. Run a Tier-3 evaluation with nv-base validate <this-skill-dir> --agent-eval to replay them.

Limitations

• Requires the matching VSS profile / microservice to be deployed and reachable from the caller.
• NGC-hosted models and NIMs may be subject to rate-limits, GPU memory requirements, and license restrictions.
• Concurrency, GPU memory, and storage limits depend on the host hardware and the profile's compose file.

Troubleshooting

• Error: REST call returns connection refused. Cause: target microservice not running. Solution: probe /docs or /health; redeploy via vss-build-vision-ai or the matching vss-deploy-* skill.
• Error: HTTP 401/403 from NGC pulls. Cause: missing/expired NGC_CLI_API_KEY. Solution: docker login nvcr.io and re-export the key before retrying.
• Error: container OOM or model fails to load. Cause: insufficient GPU memory for the selected profile. Solution: switch to a smaller variant or free GPUs via docker compose down.

VSS Generate Video Calibration

Run AutoMagicCalib over one of three input sources and drive the calibration through the microservice REST API. The input-resolution work differs per source; everything from verify_project onward is identical and lives in this file. Pick the right input-mode reference and pair it with the Shared Calibration Tail below.

Shared helper references are loaded only when needed:

• Read references/common-steps.md when a mode reference needs the shared create_project, video-upload, or handoff snippets.
• Read references/calibration-tail.md when you need the reusable Python implementation of the stage-linear-media → verify → VGGT/post-process → AMC/post-process → compare-results tail.

Input Routing

Match the user's request to a mode, then load that mode's reference for input collection, mode-specific API calls, and the full Python script.

Disambiguation rule: if the user is asking to launch / deploy / set up AMC (no calibration verb) → deploy. If they provide RTSP URLs → rtsp. If they mention local files / a videos directory → videos. If they ask to verify install or test the bundled sample → sample-dataset. Combined intents (e.g. "launch AMC and calibrate my videos") → walk deploy first, then the calibration mode. When ambiguous, ask via AskUserQuestion.

Prerequisites (shared across calibration modes)

• Platform preflight from references/deploy-auto-calibration-service.md Step 0 passes before any AMC deploy or calibration API work. The calibration host needs Ubuntu 24.04 on x86_64, NVIDIA Driver 590 or newer, NVIDIA GPU access, and NVENC hardware encoder support. If the preflight fails, stop immediately, tell the user which requirement was not met, and ask them to provide an existing calibration.json, run calibration on a supported x86_64 dGPU host, or transfer generated calibration artifacts. Do not continue AMC setup, VIOS probing, capture, upload, or calibration automatically. DGX Spark is aarch64, so use existing/generated artifacts for this flow.
• AMC microservice + UI running. If not, walk references/deploy-auto-calibration-service.md first.
• Microservice reachable at http://<HOST_IP>:${VSS_AUTO_CALIBRATION_HOST_PORT:-8010}/v1/ready → {"code":0,...}.
• Projects directory writable by the container user. If you didn't just deploy (so Step 5 of the deploy reference hasn't run), confirm the write test in references/deploy-auto-calibration-service.md § Step 5 — otherwise the first create_project returns [Errno 13] Permission denied.
• Python 3 with requests installed (each input-mode reference includes a self-healing venv fallback for direct runs).

Mode-specific prerequisites (VIOS for rtsp, sample zip for sample-dataset) live in the respective references. The platform preflight applies even when an AMC service is already running.

Shared Calibration Tail

The shared sequence is stage-linear-media → verify → VGGT (when ready) → post-process → AMC → post-process → results. After the mode-specific reference has uploaded videos / automatically ingested RTSP clips / uploaded the bundled sample, run this tail. Use references/calibration-tail.md for the shared Python snippet.

AMC UI sequence: Step 1 Project Setup, Step 2 Video Configuration, Step 3 Parameters, Step 4 Rectification, Step 5 Manual Alignment, Step 6 Execute, Step 7 Results.

Step A — Stage Linear Media

AMC v3.3.0 cannot calibrate raw media. After the mode-specific workflow has uploaded videos or completed RTSP ingest, explicitly choose one path before verification:

• Already-linear/pinhole media — call POST /v1/linear_media/<project_id> and require rectification_state == "COMPLETED".
• Distorted media — open AMC UI Step 4: Rectification; select Auto, Manual, or Videos Are Rectified; review the estimate; then click Generate Rectified Videos. Auto supports simple_divisional (default), simple_radial, and radial; Manual supports per-camera model, k1, and k2 for radial. READY_FOR_REVIEW is not complete: require rectification_state == "COMPLETED" before continuing. Re-rectification invalidates verification, calibration, and post-processing outputs.

For REST-only rectification, use the running AMC service contract exposed by <MS_URL>/docs (OpenAPI: <MS_URL>/openapi.yaml):

1. Auto — POST /v1/rectification/<project_id> starts frame-0 estimation. Poll GET /v1/rectification/<project_id> until READY_FOR_REVIEW; then GET /v1/rectification/<project_id>/cameras, review every auto_estimate, and commit all camera parameters with POST /v1/rectification/<project_id>/manual using {"cameras":{"cam_00":{"model":"...","k1":0.0,"k2":0.0},...}}. This explicit commit generates full rectified videos.
2. Manual — POST /v1/rectification/<project_id>/manual/start; optionally preview each adjustment through POST /v1/rectification/<project_id>/preview/<camera_id>; then commit a complete per-camera cameras map to POST /v1/rectification/<project_id>/manual.
3. Poll GET /v1/rectification/<project_id> until COMPLETED. Stop on ERROR; do not verify or calibrate from READY_FOR_REVIEW.

Rectification produces rectified.mp4 and rectified.jpg. External alignment files normally use coord_space=original; use rectified only for points created on AMC rectified media. Never call /v1/calibrate/<project_id> before the linear-media or rectification state is complete.

Step B — Verify Project

POST /v1/verify_project/<project_id>

Response: {"project_state": "READY"} — must be READY before calibrating. If not READY, re-check that videos + alignment + layout are present (either via API or via UI manual alignment).

Step C — Independent VGGT Calibration

After verification and before AMC, inspect vggt_state. Start VGGT by default from READY, resume and wait from RUNNING, and always post-process a COMPLETED multi-camera result, including one completed before the current invocation. MODEL_MISSING or ERROR is reported as an AMC-only fallback. Check amc_state, vggt_state, and postprocess_state independently.

POST /v1/vggt/calibrate/<project_id>
GET  /v1/get_project_info/<project_id>                    # poll vggt_state
POST /v1/postprocess/<project_id>                          # multi-camera only, after VGGT
GET  /v1/get_project_info/<project_id>                    # require postprocess_state == COMPLETED
GET  /v1/vggt_results/<project_id>/evaluation_statistics  # VGGT metrics when GT exists

Step D — Start AMC Calibration

Confirm the plan before calibrating. Whether the settings file and detector were auto-detected or asked, present a short summary and confirm via AskUserQuestion before the POST /calibrate. The resolved values are the defaults, so confirming is one click — but the user can switch the detector or skip an auto-detected settings file. Summarize:

• Detector — resnet or transformer (the value to be sent).
• Calibration settings — the file being applied (path), or default parameters (with the option to tune them in the UI first — see below).
• Optional overrides — ground-truth zip and focal lengths, if any.

The sample-dataset install-check run uses a fixed resnet and can proceed without this confirmation.

POST /v1/calibrate/<project_id>
Content-Type: application/json

{"detector_type": "resnet"}   # or "transformer"

detector_type is a separate /calibrate parameter — not consumed by /v1/config/<id>. If the user provided a calibration settings file, parse it for "detector" / "detector_type" and use that value. If the file doesn't specify one, the default (resnet) is the value shown in the confirmation above — the user can switch it there before calibrating. If there's no settings file at all, ask the user via AskUserQuestion:

• resnet — default, fast.
• transformer — slower, better under heavy occlusion.

UI Step 3 (Parameters) does NOT cover detector choice; never assume the user picked one in the UI.

Also when there's no settings file, ask whether to tune the calibration parameters first (AskUserQuestion):

• Proceed with the default parameters — well-suited to typical warehouse scenes; recommended unless the user has specific tuning in mind.
• Adjust parameters in the UI first — open the project, go to Step 3: Parameters, change values, and click Save; then continue.

In Step 3, set layout_px_per_m directly or measure a known two-point distance. Re-run post-processing after a scale or alignment change.

Wait for the user's choice — and, if they choose to tune, for them to confirm they've Saved — before calling /calibrate.