Vss Summarize Video

VSS Summarize Video

Instructions

• Before readiness checks or video preparation, reject summarization of any stream (RTSP/RTSPS URL or registered camera), including recorded time windows: Live-stream summarization / report generation isn't supported. Then stop; do not ingest, extract clips, invoke inference/fallback, or deploy captioning. Reject an explicitly identified stream/camera even if its URL is absent. For a registered sensor name/id (not a local file path, uploaded file, or direct recorded-video URL), first check its current type with only vss vios list --sensor <name>; never reuse an earlier file classification. Reject a stream; if the type is unclear or the lookup fails, report that limitation and stop. Local/uploaded files follow the recorded-video workflow, including Stage 2 registration when absent; do not classify their paths as sensors.
• Execute the five workflow stages below in order.
• Run API commands yourself; do not tell the user to run them.
• Use the required references at their named decision points.

Examples

Runnable scenarios live under evals/. The command implementations are in references/end-to-end-example.md.

When to Use

Use when summarizing a recorded video through HITL-gated LVS, to produce one polished narrative summary with timestamped events when LVS is available.

Do not use this skill for:

• Live RTSP captioning: use vss-deploy-dense-captioning.
• Incident or alert-window reports: use vss-generate-video-report Mode B.
• Archive search: use vss-search-archive.

Required References

Load these files only as directed:

• references/end-to-end-example.md: load before executing the recorded-video workflow. It contains the exact readiness, VIOS preparation, single-run summarize, and VLM fallback commands.
• references/cli_usage.md: load before Stage 4. vss summarize run issues the summarize request and applies the operator's configured memory policy; this reference has its flags, exit codes, output shape, and read verbs.
• references/video-summarization-api.md: load before constructing a live LVS operation by hand — a direct API question. Follow its Runtime OpenAPI Discovery procedure: the LVS schema is /openapi.json under services.lvs.url; the origin's own /openapi.json is the Agent's. The ordered workflow does not build a summarize payload; the CLI owns that.
• references/hitl-prompts.md: load when collecting LVS scenario, events, and optional objects of interest.
• references/video-summarization-debugging.md: load only when diagnosing a failed or empty response.
• references/video-summarization-deployment.md: load only for deployment, configuration, logs, or service operations.
• references/video-summarization-environment-variables.md and assets/video-summarization.env.example: use when configuring the service environment.
• ../vss-build-vision-ai/references/deployment_resolution.md: Kubernetes VSS_PUBLIC_URL contract and the /lvs mount, for deployment questions. The workflow itself reads its service URLs from vss configure show.
• references/deploy-lvs-service.md: load when asked about LVS's own container image, GPU/CPU/storage sizing, or deployment contract as a peer service (heavier than video-summarization-deployment.md, which covers operating an already running deployment).
• references/integrate-lvs-service.md: load when another agent or skill needs to integrate with LVS as a peer service — required peers, integration interfaces, API schema, and network requirements.

Core Invariants

• Route by LVS readiness, never by video duration.
• HTTP 200 from /v1/ready selects LVS. Empty response bodies do not mean unavailable.
• Once LVS is selected, do not call a VLM /v1/chat/completions endpoint.
• Issue exactly one vss summarize run per recorded segment. One run is one POST /v1/summarize. Never retry, hedge, broaden events, or run a second backend automatically.
• Endpoints come from the deployment vss configure recorded. Never pass an endpoint, index, or model flag the caller did not name, and never replace a failed run with hand-rolled curl against /v1/summarize.
• Save the complete command and its stdout. Diagnose failures from those files, the run's own exit code, service logs, and non-mutating GET requests.
• Render video_summary and every returned event verbatim. Do not paraphrase, truncate descriptions, add fields, or fabricate id.
• When LVS is not ready, fall back to vss vlm run directly. Do not ask first, and do not offer to deploy LVS.

Prerequisites

• The lvs profile, reachable through the origin recorded by vss configure.
• curl for the readiness probe only, and jq for reading CLI JSON. Capture stdout before piping it, or use set -o pipefail. Exit codes and the common CLI rules live in the repository root AGENTS.md.
• Network reachability from the LVS service to the final VIOS clip URL (Docker: from vss-lvs; Kubernetes: deploy must mint a URL the LVS pod can fetch).
• The vss CLI on PATH. The OpenClaw and Hermes harness images ship it; anywhere else, install it from the same checkout as this skill so the CLI and the skill match: uv tool install <checkout>/libs/vss/cli.
• One recorded deployment origin:

vss summarize run --help >/dev/null || exit 1
vss configure show

vss configure show fails when nothing is recorded. Then the only setup is vss configure --base-url "${VSS_PUBLIC_URL}", with the ingress origin the operator gave you. If VSS_PUBLIC_URL is unset, stop and ask for that origin; do not substitute HOST_IP, localhost, or a port.

Configure against the ingress origin, never :38111 — that LVS container port exposes no Elasticsearch, so a deployment recorded from it cannot persist.

The vss-build-vision-ai skill can deploy the profile.

Limitations

• Direct VLM fallback cannot target LVS scenarios or events and is lower quality.
• Private VIOS URLs may be unreachable from remote VLM endpoints.
• One vss summarize run per recorded segment, with no automatic retry.
• Persistence needs a routed Elasticsearch. A deployment without one summarizes and reports the result unpersisted rather than failing the job.
• Both edges are configured to wait an hour, matching the CLI's own default, so a long summarization is not cut short by a 504 that would be recorded as a failed job. An Ingress the deployment overrides shorter still caps the wait.

Recorded services

Every URL the recorded-video workflow and a direct API question touch is one vss configure recorded. Read it from vss configure show; never assemble it from VSS_PUBLIC_URL, HOST_IP, or a port (the one exception is the Stage 2 rewrite of a loopback media_url host to the host's routable IP), and ignore leftover LVS_BACKEND_URL / VLM_BASE_URL / RTVI_VLM_BASE_URL. Do not use kubectl port-forward, Service DNS, NodePorts, docker exec, or docker inspect, and do not scan ports or configuration files for an endpoint.

The readiness probe is the only direct HTTP the workflow makes, and it exists because the CLI has no readiness verb: vss configure records LVS on liveness (/lvs/v1/live), while routing needs HTTP 200 from /v1/ready. It is not a fallback. When a vss command fails, report that failure; never repeat the work with curl against /v1/summarize or /v1/chat/completions.

Do not treat the origin's /openapi.json as the LVS schema; on stock Ingress that path is the Agent's.

Routing

Probe LVS /v1/ready using the loop in the end-to-end reference. Readiness is the HTTP status only: retry 503 warmup responses for about 30 seconds, and do not inspect the body. No recorded lvs service counts as not ready.

Recorded Video Workflow

Stage 1: Select the Backend

Load the end-to-end and CLI references. Run the LVS readiness probe before preparing the clip.

The summarization model needs no discovery: vss configure recorded the id LVS reports serving, and vss summarize run defaults to it on both Docker and Kubernetes. The VLM fallback needs none either: vss vlm run defaults to the model the deployment's RT-VLM reports. Pass --model only when the caller named one, and read the recorded value from vss configure show when it has to be reported.

A non-200 LVS readiness result after warmup is the only unavailability signal. An empty summary, empty events, missing optional fields, or empty readiness stdout must not trigger fallback.

Stage 2: Prepare the Video Through VIOS

Use the vss CLI for every step; no VIOS REST calls, and do not invoke a separate skill.

1. A named sensor goes straight to step 3. For a file, vss vios list --sensor <stem>; reuse the recording when present. VIOS names an uploaded sensor by its filename stem.
2. If absent and the exact local file is available, vss vios add <file>. It waits for the timeline; its default timestamp is 2025-01-01T00:00:00.000Z.
3. vss vios timeline --sensor <name>, then for each segment vss vios clip --sensor <name> --start-time <start> --end-time <end>. Always pass the segment's bounds: a window may not span a gap, and an RTSP sensor has no default window. Pass media_url to --url as returned, except that a localhost / 127.0.0.1 host becomes the host's routable IP: vss-lvs cannot fetch loopback and rejects it.
4. If warmed is false, stop and report it. warmed: true shows only that the CLI host fetched the URL, not that LVS can.

Require the exact recording, full timeline, and fresh clip URL before continuing. When the source file is available, compare VIOS timeline duration with source duration.

If preparation fails, stop and report the missing prerequisite. Do not choose an arbitrary /tmp video, alternate recording, local HTTP server, NvStreamer, or RTSP source unless the user explicitly requested that source.

Stage 3: Collect LVS Settings

When LVS is selected, load the HITL reference and collect scenario, events, and optional objects_of_interest before the summarize run.

When the caller explicitly says to run autonomously without prompting and asks for defaults or supplies no settings, use these values verbatim:

scenario="activity monitoring"
events=["notable activity"]

This is the only HITL bypass. Do not infer defaults from filenames or sensor names. Mention defaults in the final response and offer a separate rerun with specific settings.