Vss Summarize Video

Instructions

Follow the routing tables and step-by-step workflows below. Each section that ends in workflow, quick start, or flow is intended to be executed top-to-bottom. Detailed reference material lives in references/.

Examples

Worked end-to-end examples are kept under evals/ (each *.json manifest contains a runnable scenario) and inline in the per-workflow curl blocks below. Run a Tier-3 evaluation with nv-base validate <this-skill-dir> --agent-eval to replay them.

Call the VLM NIM or the video summarization microservice directly. Always run curl commands yourself; never instruct the user to run them.

Primary video workflow query type: "Summarize this video." Direct video summarization API and service-ops requests are handled by the reference-routed sections below.

Purpose

Produce a single, polished narrative summary of one recorded video clip, with timestamped events when the LVS microservice path is reachable.

Do NOT use this skill for:

• Live RTSP captioning — use vss-deploy-dense-captioning.
• Report generation, including incident or alert-window reports — use vss-generate-video-report Mode B.
• Semantic search across the archive — use vss-search-archive.

Prerequisites

• VSS lvs profile running on $HOST_IP (port 38111) OR a reachable VLM/RT-VLM endpoint as a fallback. The vss-deploy-profile skill brings these up.
• Network reachability from the agent host to both endpoints; clip URLs from VIOS must be fetchable by the chosen backend.
• jq and curl available on the agent host.

Limitations

• Direct VLM fallback uses a single fixed prompt and cannot target scenario/events — output quality is lower than the LVS path.
• Remote VLM endpoints generally cannot reach localhost/private clip URLs.
• One backend call per request; no parallel hedging or multi-pass summaries.

Troubleshooting

See references/video-summarization-debugging.md for deeper diagnostics.

Reference Map

Use these references only when the user asks for the relevant detail, or when the core workflow below needs deeper video summarization information:

• video summarization API details: references/video-summarization-api.md for /v1/summarize, /summarize, /v1/generate_captions, /v1/stream_summarize, health probes, /models, /recommended_config, /metrics, request fields, response shapes, and API gotchas.
• video summarization service configuration and ops: references/video-summarization-deployment.md for the VSS lvs profile, ports, required env vars, logs, status, dry-runs, teardown, model/backend swaps, Elasticsearch/Neo4j/ArangoDB backend selection, and service-level troubleshooting.
• Extended video summarization ops references: references/video-summarization-environment-variables.md, references/video-summarization-debugging.md, and assets/video-summarization.env.example.

Load video-summarization-api.md only when you need a request field, response shape, or endpoint that is not already covered by the Step 2 LVS or fallback VLM example below, or when handling a direct video summarization API request. Load video-summarization-deployment.md only for deployment, configuration, or service operations.

Video Summarization API And Service Ops Requests

If the user asks to call or debug video summarization endpoints directly, answer from references/video-summarization-api.md instead of running the end-to-end video summarization workflow. Examples: list video summarization models, check readiness, get recommended chunking config, inspect metrics, explain a 422 response, or build a /v1/summarize request body.

If the user asks to configure, deploy, restart, tear down, or troubleshoot the video summarization service, prefer the vss-deploy-profile skill for full VSS profile deployment and use references/video-summarization-deployment.md for video summarization-specific service details.

Routing

Decide purely from video summarization service availability (probed in Setup → Availability checks below). Duration does not drive routing.

Fallback message when the LVS service is unreachable — copy verbatim above the summary:

⚠ Note: Input video <name> is <N>s long. The video summarization service is not deployed, so this summary was produced by the VLM alone with a generic default prompt. Deploy the lvs profile for higher-quality summaries with scenario/events targeting.

Deployment prerequisite

The VSS lvs profile on $HOST_IP is the primary backend. If the /v1/ready probe (see Setup → Availability checks) returns anything other than 200 after the warmup retries, ask the user:

"The VSS lvs profile isn't running on $HOST_IP. Shall I deploy it now using the /vss-deploy-profile skill with -p lvs? Reply no to summarize with the VLM-only fallback instead (lower quality, no scenario/events targeting)."

• Yes → hand off to /vss-deploy-profile, then re-probe and continue with Step 2 (LVS + HITL).
• No → go straight to Step 2 fallback (VLM with default prompt) and prepend the Routing fallback note. Do not ask again, and do not run scenario/events HITL.
• Pre-authorized to deploy autonomously (caller said so explicitly) → skip the confirmation and invoke /vss-deploy-profile directly.
• Pre-authorized to use VLM fallback ("skip lvs, just use the VLM") → go straight to Step 2 fallback without prompting.

Setup

Endpoints (defaults for a local VSS lvs deployment):

• VLM / RT-VLM: ${VLM_BASE_URL} — default ${RTVI_VLM_BASE_URL:-http://${HOST_IP:-localhost}:8018}
• LVS service: ${LVS_BACKEND_URL} — default http://${HOST_IP:-localhost}:38111
• VIOS: owned by vss-manage-video-io-storage; refer there.

Use env vars when set (strip trailing /v1 from the VLM base — the skill appends it). Otherwise use the defaults. If neither works, ask the user — do not scan ports or read config files to guess.

Model name: read ${VLM_NAME} (default nim_nvidia_cosmos3-nano-reasoner_bf16-final). It must match the id RT-VLM /v1/models advertises; do not substitute the friendly nvidia/cosmos3-nano-reasoner.

For endpoint schemas, optional fields, response envelopes, and error handling, see references/video-summarization-api.md.

Availability checks (run both before routing). Readiness is determined by the HTTP status code only — the LVS /v1/ready may legitimately return 200 with an empty body, so do not inspect the body.

VLM="${VLM_BASE_URL:-${RTVI_VLM_BASE_URL:-http://${HOST_IP:-localhost}:8018}}"
VLM="${VLM%/v1}"

# VLM / RT-VLM: 200 on /v1/models
vlm_code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 3 --max-time 10 \
  "$VLM/v1/models")
[ "$vlm_code" = "200" ] && echo "VLM OK" || echo "VLM not reachable (HTTP $vlm_code)"

# Video summarization service: 200 on /v1/ready, with retry on 503 (warmup) for up to ~30s
VIDEO_SUMMARIZATION_URL=${LVS_BACKEND_URL:-http://${HOST_IP:-localhost}:38111}
video_sum_code=000
for i in $(seq 1 10); do
  video_sum_code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 3 --max-time 10 "$VIDEO_SUMMARIZATION_URL/v1/ready")
  case "$video_sum_code" in
    200) echo "video summarization OK"; break ;;
    503) sleep 3 ;;                 # warming up; keep polling
    *)   break ;;                   # any other code = not reachable, stop retrying
  esac
done
[ "$video_sum_code" = "200" ] || echo "video summarization service not reachable (HTTP $video_sum_code)"

How to interpret the results:

• video_sum_code = 200 → Step 2 (LVS + HITL) for every video.
• video_sum_code != 200, vlm_code = 200 → Step 2 fallback (VLM); prepend the Routing fallback note.
• vlm_code != 200 → fail; at least one backend must be reachable.
• A non-200 LVS code after the retry loop is the ONLY signal of unavailability. Empty stdout or missing JSON fields are NOT "unavailable."

Step 1 - Get the clip URL via vss-manage-video-io-storage (sub-task, NOT the final answer)

Use the vss-manage-video-io-storage skill for all VIOS interactions — it owns the canonical curl recipes, parameter defaults, and delete/upload flows. Do not fabricate URLs or hand-roll VIOS calls; they will drift.

This step is a sub-task — do NOT end your turn here; do NOT return the clip URL as the final answer. From VIOS collect three values:

1. streamId (via sensor/list → sensor/<id>/streams, or directly from an upload response).
2. Timeline - {startTime, endTime} (ISO 8601 UTC). endTime - startTime is the duration; needed only for the user-facing header (routing is driven solely by /v1/ready).
3. Temporary MP4 clip URL — the /storage/file/<streamId>/url variant with container=mp4. Response field: .videoUrl. Both backends need an HTTP(S) URL they can GET.

Everything else (auth, upload, disableAudio, expiry, etc.) lives in the vss-manage-video-io-storage skill — refer users there if VIOS fails.