Vss Pipeline Config

VSS pipeline configuration knobs

Use this skill when helping developers tune the Video Search & Summarization (VSS) sample application's summarization pipeline. Ground every answer in the current repository: if code changed, re-open the cited files before giving final guidance. For a full tabular reference, read references/parameter-reference.md.

Environment setup (run first)

This skill drives the Video Search & Summarization app through its real source files, so the VSS application must be present and you must run commands from its app root. Do this before anything else, and it works whether or not the VSS source is already in your workspace.

Run the bundled bootstrap. It first tries to find an existing VSS checkout - walking up from the current directory and inspecting the enclosing git repo - and reuses it without ever re-cloning. Only when no checkout is found does it do a shallow, single-branch, sparse checkout of just sample-applications/video-search-and-summarization from main. It prints the resolved app root on stdout:

# SKILL_DIR is THIS skill's own directory (shown to you when the skill loads);
# in-repo it is .github/skills/vss-pipeline-config. Works the same if the skill is installed standalone.
SKILL_DIR=".github/skills/vss-pipeline-config"
APP_ROOT="$(bash "$SKILL_DIR/scripts/vss-bootstrap.sh")"
cd "$APP_ROOT"

Every command below assumes the working directory is this APP_ROOT. To pull from a fork/branch or reuse a specific checkout dir, override VSS_REPO_URL, VSS_REPO_BRANCH, or VSS_CLONE_DIR before running it.

Mental model

The summarization path is:

1. UI builds SummaryPipelineDTO and POSTs it to POST /summary.
2. Pipeline manager stores reqBody.sampling as state.userInputs and validates/derives state.systemConfig.
3. EVAM/video ingestion receives parameters.frame = samplingFrame and parameters.chunk_duration = chunkDuration.
4. ChunkingService groups returned frames into VLM captioning calls using samplingFrame, frameOverlap, and multiFrame.
5. Optional audio transcription adds time-matched transcript snippets to chunk VLM prompts; optional full-transcript summary adds a condensed audio summary to the final LLM map-reduce prompt.

Key code paths:

• API DTO: pipeline-manager/src/summary/models/summary-pipeline.model.ts
• API validation/default merge: pipeline-manager/src/summary/controllers/summary.controller.ts
• Server defaults/env: pipeline-manager/src/config/configuration.ts, pipeline-manager/src/video-upload/services/app-config.service.ts
• EVAM request: pipeline-manager/src/evam/services/evam.service.ts
• VLM batching/audio transcript use: pipeline-manager/src/state-manager/queues/chunking.service.ts
• Audio/full transcript summary: pipeline-manager/src/state-manager/queues/audio-queue.service.ts, pipeline-manager/src/state-manager/queues/summary-queue.service.ts
• UI fields: ui/react/src/components/VideoActions/VideoSummarizeFlow.tsx and legacy drawer ui/react/src/components/Drawer/VideoUpload.tsx

Core knobs

Chunk duration

• User-facing name: chunk Duration (secs) in the UI.
• API field: sampling.chunkDuration in SummaryPipelineDTO.
• Where to set: UI NumberInput id='chunkDuration'; direct API request body; CLI YAML may use chunkDuration.
• Default: UI initializes to 8; API requires the field and does not add an API default.
• Range in code: UI minimum is 2; no explicit backend maximum. Use practical values based on content: short events ~5-15s, long lectures/presentations ~30-60s.
• What it does: Sent to EVAM as parameters.chunk_duration; also maps frame IDs back to transcript/search time windows.
• Trade-off: Shorter chunks improve temporal precision and reduce the amount of unrelated context per caption, but create more chunks and more downstream work. Longer chunks reduce orchestration overhead, but each caption covers a broader time span, so brief events and transcript alignment can be diluted.

Frames per chunk / frame sampling

• User-facing name: Frame per chunk.
• API field: sampling.samplingFrame.
• Where to set: UI NumberInput id='sampleFrame'; direct API request body; CLI YAML may use samplingFrame.
• Default: UI initializes to 8; API requires the field.
• Range in code: UI minimum is 2. Backend requires samplingFrame + frameOverlap == sampling.multiFrame and that the derived multi-frame value does not exceed the configured maximum batch size.
• What it does: Sent to EVAM as parameters.frame; used by ChunkingService as the number of frames per chunk when grouping frames and aligning audio transcript snippets.
• Trade-off: More sampled frames give the VLM more visual evidence and improve chances of capturing short actions. Cost grows roughly with the number of images captioned: more frames increase ingestion output, VLM payload size, queue pressure, and latency.

Frame overlap

• User-facing name: Frames Overlap in Advanced/Ingestion Settings.
• API field: sampling.frameOverlap.
• Where to set: UI NumberInput id='overrideMultiFrame' despite the misleading id; direct API request body.
• Default: 0 from UI state and server config.
• Range in code: UI min 0, max systemConfig.multiFrame; effective safe range is 0..(systemConfig.multiFrame - samplingFrame) because the backend rejects mismatches/oversized batches.
• What it does: ChunkingService uses it to slide VLM frame windows: windowLength = multiFrame - frameOverlap; overlapping frames are repeated across adjacent VLM captioning calls.
• Trade-off: Overlap reduces boundary misses when an event spans two frame groups. It also duplicates frames in VLM calls, so latency and token/image cost increase.