Benchmark Video Search

Step 1 — Configure the CLI

The CLI reads ~/.vss/config.json, which is separate state from --endpoint and points at a different port: it discovers services by path prefix on the unified origin, which the agent port does not route. Getting this wrong makes every query exit 4.

vss configure --base-url http://HOST:7777
vss configure show

localhost is valid here when the runner runs on the deployment host. This URL controls where the CLI reaches the services; it does not determine the clip URL sent to RT-VLM. If verification fails, inspect the VST-returned videoUrl and test that exact address from the RT-VLM container.

Step 2 — Dry run

Reports what would happen and contacts nothing.

uv run --with requests python3 scripts/run_eval_flows.py --endpoint http://HOST:8000 \
    --data-dir /path/to/datasets --dataset DATASET --dry-run

Step 3 — Verify the deployment is actually indexed

Registration with VST is not ingestion. A deployment can list every sensor and still have an empty Elasticsearch, in which case every query returns zero hits and every metric reads 0.0000 — which looks like catastrophic retrieval quality and is not.

curl -s --connect-timeout 5 --max-time 10 "http://HOST:7777/elasticsearch/_cat/indices?h=index,docs.count"

Expect mdx-embed-filtered-*, mdx-behavior-* and mdx-raw-* with non-zero counts. mdx-behavior-* is what the attribute and fusion paths read; if it is missing, only embed can score. If the indices are absent, go to Step 4.

Step 4 — Ingest

The default vst-direct flow uploads to VIOS and lets its webhooks trigger perception. It does not call the agent's /complete endpoint or receive a chunks_processed count. The runner waits for VST registration, then probes the search index before scoring. Use --ingest-flow agent-3step only when the deployment cannot run the webhook flow; see references/troubleshooting.md.

uv run --with requests python3 scripts/run_eval_flows.py --endpoint http://HOST:8000 \
    --data-dir /path/to/datasets --dataset DATASET --skip-download --skip-existing

Expect this to be slow while the webhook pipeline indexes the videos:

• A registered source is not necessarily indexed. The post-ingest index probe must pass before treating zero hits as a retrieval result.
• If webhooks are disabled, select --ingest-flow agent-3step explicitly. That flow calls /complete, which can return a transient 502 and is retried.
• --skip-existing is the default: ingest what is missing and delete nothing.
• --only-dataset deletes foreign sources only when explicitly requested with --confirm-delete; report its candidate inventory before it runs.
• --clear --confirm-delete deletes every source including other people's. Use it only on explicit request.
• --clear and --only-dataset list sources via --vst-url but delete through the agent endpoint. If those URLs have different hosts, verify they belong to the same deployment before deletion; otherwise source IDs from one stack could be sent to another.

Then re-run Step 3. Indices are lazy; they appear after webhook processing, not immediately after upload. See references/flags.md for ingest options.

Step 5 — Run the benchmark

uv run --with requests python3 scripts/run_eval_flows.py --endpoint http://HOST:8000 \
    --data-dir /path/to/datasets --dataset DATASET --subset SUBSET \
    --skip-download --skip-ingest --name RUN_NAME

--skip-ingest reuses an already-indexed dataset; it cannot be combined with --clear or --only-dataset. See references/flags.md before changing flags that affect run meaning.

Decomposition needs no flag. An LLM turns the sentence into {query, attributes, has_action, ...} and that decides which path runs — the same call the deployed agent makes. The script derives the NIM origin from --endpoint; if it cannot reach it the run falls back rather than stopping, and says which fallback it took.

fusion cannot be a fallback. It needs an attribute per query and there is nothing to supply one without decomposition — and feeding it the whole query as its attribute is the failure this eval already measured: mAP −25%, HIT@1 halved, latency +59%.

Leave --concurrency alone unless asked. It defaults to 1 because concurrent queries contend for the same VLM and embedding services, which inflates every per-stage latency in the report.

--name makes the result file findable; without it the name is <dataset>_<subset>_<timestamp>.

Step 6 — Report

Results land in scripts/cli_eval_result/<name>.json.

Report Recall and HIT@k as the quality signal. Precision and mAP are dominated by --top-k — five results against roughly one relevant segment keeps them low however good retrieval is — so they compare runs; they do not grade a deployment.

State these alongside any number, because each one changes what it means:

• Which paths ran. Search paths: in the summary. A run that was all embed says nothing about attribute.
• Whether decomposition was live. If so, give its share of query time and what it Routed to: — a routing shift changes what was measured.
• Whether the critic ran. Critic-filtered metrics print NA when the response carried no verification block. NA is not 0.

Read references/reading-results.md when interpreting stage latencies or metrics. Read references/flags.md when a flag might change comparability. Read references/troubleshooting.md when ingest fails or the CLI exits nonzero. Read references/dataset-format.md when validating a dataset's layout.

Error handling

A non-zero CLI exit aborts the run. A missing result envelope marks that query unanswered and aborts by default; --tolerate-unanswered permits an explicit partial run that excludes those queries from scoring. Never count them as misses. Diagnose an all-zero run with Step 3 before reporting quality. Critic-filtered NA means verification was not available, not zero quality. Use references/troubleshooting.md for the specific failure and recovery steps.

Conventions

• Never report 0.0000 as a quality result without checking Step 3 first. An unindexed deployment and a broken retriever look identical in the metrics and are not the same finding.
• A non-zero CLI exit aborts the run rather than scoring 0.0. A stale virtualenv exits 1 on every invocation, and scoring that as "no results" would report a broken environment as an accuracy regression.
• Queries do not merge adjacent windows by default. Upstream merging averages the scores of merged windows and changes the precision denominator, so a merged run is not comparable to an unmerged baseline.
• The critic is an LLM and is not deterministic. Raw metrics repeat exactly; critic-filtered ones move a few points between runs. Gate on raw.
• Live decomposition is not deterministic either. The same query can route differently between runs, so compare Routed to: before comparing scores.