Vss Troubleshoot

VSS Troubleshoot

Use this skill to diagnose a broken Video Search & Summarization deployment without guessing. The app is started with source setup.sh --summary, --search, --summary --search/--dual, or --summary-and-search/--unified; stopped with source setup.sh --down; user data reset with source setup.sh --clean-data.

Environment setup (run first)

This skill drives the Video Search & Summarization app through its real source files, so the VSS application must be present and you must run commands from its app root. Do this before anything else, and it works whether or not the VSS source is already in your workspace.

Run the bundled bootstrap. It first tries to find an existing VSS checkout - walking up from the current directory and inspecting the enclosing git repo - and reuses it without ever re-cloning. Only when no checkout is found does it do a shallow, single-branch, sparse checkout of just sample-applications/video-search-and-summarization from main. It prints the resolved app root on stdout:

# SKILL_DIR is THIS skill's own directory (shown to you when the skill loads);
# in-repo it is .github/skills/vss-troubleshoot. Works the same if the skill is installed standalone.
SKILL_DIR=".github/skills/vss-troubleshoot"
APP_ROOT="$(bash "$SKILL_DIR/scripts/vss-bootstrap.sh")"
cd "$APP_ROOT"

Every command below assumes the working directory is this APP_ROOT. To pull from a fork/branch or reuse a specific checkout dir, override VSS_REPO_URL, VSS_REPO_BRANCH, or VSS_CLONE_DIR before running it.

First collect status

Run the read-only collector:

"$SKILL_DIR/scripts/triage.sh"

It prints Docker Compose/container status, tails recent logs, curls key health endpoints, checks documented host ports, and reports GPU/NPU device visibility. This matters because most failures are dependency chains: pipeline-manager depends on storage/database/search/summary services, and UI symptoms often originate in OVMS, vLLM, EVAM, VDMS, MinIO, RabbitMQ, or Postgres.

If Docker Compose cannot resolve services, run from the app root and compare with setup's own generated config:

source setup.sh --summary config     # or --search config / --summary-and-search config

Quick health & mode check

Before diving into the decision tree, confirm whether the backend is even up and which mode is live. Set HOST=http://${HOST_IP:-localhost}:${APP_HOST_PORT:-12345} and run each command yourself, then relay the result. If nothing is deployed, hand off to the vss-deploy skill at .github/skills/vss-deploy/SKILL.md.

# 1. Is the Pipeline Manager reachable?
curl -sf --max-time 5 "$HOST/manager/health" && echo "  ← Pipeline Manager healthy" \
  || echo "UNREACHABLE - backend down or wrong HOST_IP/APP_HOST_PORT"

# 2. Which capabilities/mode are live, and the resolved config
curl -s "$HOST/manager/app/features" | jq .   # search/summary flags
curl -s "$HOST/manager/app/config"   | jq .   # resolved system config

# 3. Subsystem probes
curl -s "$HOST/metrics-manager/health"          # optional live metrics service
curl -s "$HOST/manager/audio/models"   | jq .  # whisper models (summary modes)
curl -s "$HOST/manager/pipeline/evam"  | jq .  # EVAM pipeline status

app/features returns string flags, not booleans - {"summary":"FEATURE_ON","search":"FEATURE_OFF"} - so test against the string (e.g. jq -e '.search=="FEATURE_ON"'). Use it to decide which workflow applies: vss-search-index needs search==FEATURE_ON; vss-summarize-video needs summary==FEATURE_ON. A backend that 404s on /manager/health while the model servers (ovms-service, vllm-cpu-service, embedding server) are still loading is usually starting, not broken - wait and re-probe.

Decision tree

1. Containers are missing, stopped, unhealthy, or crash-looping

Check these exact services first: nginx, pipeline-manager, postgres-service, minio-service, ovms-service, vllm-cpu-service, vllm-xpu-service, video-ingestion, audio-analyzer, rabbitmq-service, video-search, vdms-vector-db, multimodal-dataprep, vector-retriever, multimodal-embedding-serving, and optional metrics-manager.

Why: Compose depends_on gates many services on health. For example, summary mode needs ovms-service or vllm-cpu-service, video-ingestion, rabbitmq-service, and audio-analyzer; search mode needs multimodal-dataprep, vector-retriever, and multimodal-embedding-serving healthy.

Actions:

• Read the first failing dependency's logs from triage.sh; later services often fail only because they waited for it.
• Verify required environment variables from setup.sh: MinIO, Postgres, RabbitMQ credentials; VLM_MODEL_NAME, ENABLED_WHISPER_MODELS, OD_MODEL_NAME for summary; MULTIMODAL_EMBEDDING_MODEL for search; TEXT_EMBEDDING_MODEL for unified mode.
• If containers start but app state is corrupt, only then consider source setup.sh --clean-data (this deletes Docker volumes listed by setup, including MinIO/Postgres/VDMS/data-prep data).

2. Port conflict or UI unreachable

Default host ports from setup.sh/Compose:

• UI/nginx 12345; pipeline-manager 3001; search-ms 7890
• OVMS REST/gRPC 8300/9300; vLLM 8200; EVAM 8090; audio 8999
• RabbitMQ AMQP/management/MQTT 5672/15672/1883
• MinIO API/console 4001/4002; Postgres 5432; VDMS 55555; multimodal-dataprep 6016; vector-retriever 6008; embedding service 9777; telemetry 9273
• Model-download REST 8640 is loopback-only and transient while setup.sh downloads missing summary-path models; it should not remain running afterward.

Why: Compose publishes these host ports. If another process owns one, the container may fail to bind or the UI may talk to the wrong service.

Actions:

• Use the port section in triage.sh to identify listeners.
• Stop the conflicting process or override the corresponding environment variable before rerunning source setup.sh ....
• Curl http://localhost:3001/health for pipeline-manager and http://localhost:7890/health for video-search when applicable.

3. OVMS will not start or final summary is stuck

Inspect ovms-service logs and ov_models/ovms/config.json. Converted models live under ov_models/ovms/openvino_models/<device>/<precision>/<source-model>; setup registers storage-aware names such as Qwen_Qwen2.5-VL-3B-Instruct_CPU_int8.

Why: pipeline-manager sends VLM/LLM requests to http://ovms-service/v3 when ENABLE_VLLM is false. If OVMS is unhealthy, summary jobs can remain Ready or In Progress.

Likely fixes:

• Incomplete or incompatible host model cache: identify the affected model entry and directory first, stop VSS, then remove only that model directory and rerun setup so model-download recreates it. Do not delete all ov_models/ content unless the user accepts re-downloading every model.
• Token limit error like prompt tokens + max tokens exceed model length: lower PM_SUMMARIZATION_MAX_COMPLETION_TOKENS below the default 4000, or use a model with a larger context window.
• CL_OUT_OF_RESOURCES or cache at 100%: split VLM/LLM across CPU/GPU, use smaller/quantized models, or tune OVMS_CACHE_SIZE_GB cautiously.
• NPU errors: verify the model supports NPU; otherwise set VLM_TARGET_DEVICE=CPU or another supported device.

4. Setup fails during model download

The model-download container runs before Compose only when an OD artifact or an OVMS VLM/split LLM artifact is missing. Inspect the error's ov_models/model-download-*.log, or docker logs vss-model-download while the job is still running. Verify MODEL_DOWNLOAD_IMAGE, proxy settings, optional Hugging Face token, selected model/device/precision, free disk space, and that MODEL_DOWNLOAD_HOST_PORT (default 8640) is available. Increase MODEL_DOWNLOAD_JOB_TIMEOUT from its 5400 second default only when a valid download/conversion legitimately needs longer.