VideoDB Video and Audio Workflows

VideoDB Skill

Perception + memory + actions for video, live streams, and desktop sessions.

Use this skill when you need to:

1) Desktop Perception

• Start/stop a desktop session capturing screen, mic, and system audio
• Stream live context and store episodic session memory
• Run real-time alerts/triggers on what’s spoken and what's happening on screen
• Produce session summaries, a searchable timeline, and playable evidence links

2) Video ingest + stream

• Ingest a file or URL and return a playable web stream link
• Transcode/normalize: codec, bitrate, fps, resolution, aspect ratio

3) Understand + index + retrieve (timestamps + evidence)

• Understand: run analyzers over speech, scenes, objects, OCR, brands, activity
• Index: turn analyzer artifacts into semantic, filterable, and aggregatable indexes
• Retrieve: search moments, ask questions, filter exactly, count and group — with timestamps and playable evidence
• Auto-create clips from results

4) Timeline editing + generation

• Subtitles: generate, translate, burn-in
• Overlays: text/image/branding, motion captions
• Audio: background music, voiceover, dubbing
• Programmatic composition and exports via timeline operations

5) Live streams (RTSP) + monitoring

• Connect RTSP/live feeds
• Run real-time visual and spoken understanding and emit events/alerts for monitoring workflows

Common inputs

• Local file path, public URL, or RTSP URL
• Desktop capture request: start / stop / summarize session
• Desired operations: get context for understanding, transcode spec, index spec, search query, clip ranges, timeline edits, alert rules

Common outputs

• Stream URL — make it playable: https://console.videodb.io/player?url={STREAM_URL}
• Search results with timestamps and evidence links
• Generated assets: subtitles, audio, images, clips
• Event/alert payloads for live streams
• Desktop session summaries and memory entries

Canonical prompts (examples)

• “Start desktop capture and alert when a password field appears.”
• “Record my session and produce an actionable summary when it ends.”
• “Ingest this file and return a playable stream link.”
• “Index this folder and find every scene with people, return timestamps.”
• “Generate subtitles, burn them in, and add light background music.”
• “Connect this RTSP URL and alert when a person enters the zone.”

Running Python code

CRITICAL: Always cd to the user's project directory before running Python code. This ensures load_dotenv(".env") finds the correct .env file.

from dotenv import load_dotenv
load_dotenv(".env")

import videodb
conn = videodb.connect()

This reads VIDEO_DB_API_KEY from:

1. Environment (if already exported)
2. Project's .env file in current directory

If the key is missing, videodb.connect() raises AuthenticationError automatically.

Do NOT write a script file when a short inline command works.

When writing inline Python (python -c "..."), always use properly formatted code — use semicolons to separate statements and keep it readable. For anything longer than ~3 statements, use a heredoc instead:

python << 'EOF'
from dotenv import load_dotenv
load_dotenv(".env")

import videodb
conn = videodb.connect()
coll = conn.get_collection()
print(f"Videos: {len(coll.get_videos())}")
EOF

Setup

When the user asks to "setup videodb" or similar:

1. Install SDK

pip install "videodb[capture]>=0.5.0" python-dotenv

If videodb[capture] fails on Linux, install without the capture extra:

pip install "videodb>=0.5.0" python-dotenv

The >=0.5.0 pin matters — the understand/index/ask/aggregate APIs do not exist in earlier versions.

2. Configure API key

The user must set VIDEO_DB_API_KEY using either method:

• Export in terminal (recommended): export VIDEO_DB_API_KEY=your-key
• Project .env file: Save VIDEO_DB_API_KEY=your-key in the project's .env file

Get a free API key at https://console.videodb.io (50 free uploads, no credit card).

Do NOT read, write, or handle the API key yourself. Always let the user set it.

3. Authorize the hosted MCP server (Claude Code plugin only)

The plugin bundles the hosted MCP server at https://mcp.videodb.io/mcp, which is authorized separately from the SDK key above. Tell the user to run /mcp, select videodb, and complete browser authorization with their VideoDB account. Verify with "list my VideoDB collections".

The MCP tools need no local Python install and no API key. Skip this step when the skill was installed with npx skills add or is running outside Claude Code — there is no MCP server in that case, and the SDK path above is the only one available.

Quick Reference

Upload media

# URL
video = coll.upload(url="https://example.com/video.mp4")

# YouTube
video = coll.upload(url="https://www.youtube.com/watch?v=VIDEO_ID")

# Local file
video = coll.upload(file_path="/path/to/video.mp4")

Understand → index → retrieve (default path)

Three stages. Run analyzers to produce artifacts, index each artifact, then retrieve.

import time

# 1. Understand. Naming each analyzer keeps `analyzer.name` meaningful downstream.
understanding = video.understand(
    analyzers=[
        {"type": "spoken_words", "name": "transcript"},
        {"type": "vlm", "name": "scene",
         "config": {"prompt": "Describe the scene and any on-screen text."}},
    ],
    segmentation={"type": "shot", "threshold": 30},
)

# A run with a failed or skipped analyzer ends `partial`, which the SDK does not
# treat as terminal — wait_until_complete() would poll to TimeoutError. Poll the
# analyzers instead. The `analyzers and` guard is load-bearing: a refresh can
# transiently return an empty list, and all([]) is True, which would exit the
# loop while the run is still going.
deadline = time.time() + 3600
while time.time() < deadline:
    analyzers = understanding.refresh().list_analyzers()
    if analyzers and all(a.is_complete for a in analyzers):
        break
    time.sleep(15)

# 2. Index each artifact that succeeded.
for analyzer in understanding.list_analyzers():
    if analyzer.is_successful:
        video.index(source=analyzer, name=analyzer.name).wait_until_complete()

# 3. Retrieve.
response = video.search("discussion about pricing")
for shot in response.shots:
    print(f"[{shot.start:.1f}s - {shot.end:.1f}s] {shot.text}")
if response.response_type in ("shots", "deepsearch") and len(response):
    stream_url = response.compile()   # raises SearchError otherwise

Analyzer types: spoken_words (→ artifact transcript), vlm (→ scene), object_detection (→ objects), ocr, brand_detection (→ brands), activity_recognition (→ activity), location_detection (→ location), faces, audio_event_detection. They are plain strings — there is no SDK enum.

See reference/indexing.md for segmentation, sampling, field configuration, and cost tuning.

Retrieval

search(query) is the default — it plans the retrieval and picks the indexes itself. Reach past it when you need something specific:

# Target a specific index, with a relevance floor
video.semantic_search("a customer holding the product", index_names=["scene"], score_threshold=0.7)

# Exact filtering, no natural-language interpretation
video.query(index_name="objects",
            filter=[{"field": "frames.detections.label", "op": "contains", "value": "car"}])

# Counts and facets — returns the raw server payload, not a SearchResult
video.aggregate(index_name="objects", group_by="frames.detections.label", metric="count")

# A written answer plus the moments it came from
answer = video.ask("What did they say about pricing?", include_sources=True)

All five exist on Collection too, fanning out across every indexed video. See reference/search.md.

search() now returns SearchResponse, not SearchResult. get_shots(), compile(), play(), and iteration all work, but there is no .stream_url on it — use .compile().

Transcript + subtitle

# force=True skips the error if the video is already indexed
video.index_spoken_words(force=True)
text = video.get_transcript_text()
stream_url = video.add_subtitle()

index_spoken_words() is the correct call here even on 0.5.0 — add_subtitle() and CaptionAsset(src="auto") read the v1 spoken-word index. A v2 spoken_words artifact does not substitute for it. This is the one place v1 indexing is still the right answer.

Legacy indexing (existing codebases)

# v1 API — still supported in 0.5.0, not deprecated. New code should use the v2 path above.
from videodb import IndexType

video.index_spoken_words(force=True)
scene_index_id = video.index_scenes(prompt="Describe the visual content.")
results = video.legacy_search(
    "person writing on a whiteboard",
    index_type=IndexType.scene,
    scene_index_id=scene_index_id,
)

Recognise this pattern in existing repos and leave it alone unless asked to migrate — it still works. See reference/migration.md to port it, or reference/legacy/search.md to maintain it.

Timeline editing

Use the Editor API to compose videos, images, audio, and text. See reference/editor.md for full workflow.

from videodb.editor import Timeline, Track, Clip, VideoAsset, ImageAsset, AudioAsset, Fit

timeline = Timeline(conn)
timeline.resolution = "1280x720"

video_track = Track()
video_track.add_clip(0, Clip(asset=VideoAsset(id=video.id, start=10), duration=20))

audio_track = Track()
audio_track.add_clip(0, Clip(asset=AudioAsset(id=music.id, volume=0.2), duration=20))

timeline.add_track(video_track)
timeline.add_track(audio_track)
stream_url = timeline.generate_stream()

Transcode video (resolution / quality change)

from videodb import TranscodeMode, VideoConfig, AudioConfig

# Change resolution, quality, or aspect ratio server-side
job_id = conn.transcode(
    source="https://example.com/video.mp4",
    callback_url="https://example.com/webhook",
    mode=TranscodeMode.economy,
    video_config=VideoConfig(resolution=720, quality=23, aspect_ratio="16:9"),
    audio_config=AudioConfig(mute=False),
)

Reframe aspect ratio (for social platforms)

Warning: reframe() is a slow server-side operation. For long videos it can take several minutes and may time out. Best practices:

• Always limit to a short segment using start/end when possible
• For full-length videos, use callback_url for async processing
• Trim the video on a Timeline first, then reframe the shorter result

from videodb import ReframeMode

# Always prefer reframing a short segment:
reframed = video.reframe(start=0, end=60, target="vertical", mode=ReframeMode.smart)

# Async reframe for full-length videos (returns None, result via webhook):
video.reframe(target="vertical", callback_url="https://example.com/webhook")

# Presets: "vertical" (9:16), "square" (1:1), "landscape" (16:9)
reframed = video.reframe(start=0, end=60, target="square")

# Custom dimensions
reframed = video.reframe(start=0, end=60, target={"width": 1280, "height": 720})

Generative media

image = coll.generate_image(
    prompt="a sunset over mountains",
    aspect_ratio="16:9",
)

Sandbox Compute (self-hosted / open-weight models)

Run open-weight models (Gemma, Qwen, Whisper, OmniVoice, FLUX, RT-DETR) by creating a sandbox and passing sandbox_id to a supported job. Requires videodb>=0.5.1.

from videodb import SandboxTier, SandboxModel

# 1. Create a sandbox sized for the largest model, then wait until active.
sandbox = conn.create_sandbox(
    tier=SandboxTier.medium,
    models=[SandboxModel.GEMMA_4_31B.value],   # exact ID, NO -FP8 suffix
)
sandbox.wait_for_ready(timeout=300, interval=5)

# 2. Understanding: set config.model + config.sandbox_id on the analyzer.
understanding = video.understand(analyzers=[{
    "type": "vlm", "name": "scene",
    "config": {"model": "google/gemma-4-31B-it", "sandbox_id": sandbox.id,
               "prompt": "Describe the scene."},
}])

# 2b. Generation: pass model_name + sandbox_id (jobs return GenerationJob → .wait()).
response = coll.generate_text(prompt="Summarize this.", model_name="Qwen/Qwen3.5-9B",
                             sandbox_id=sandbox.id, max_tokens=300)
job = coll.generate_image(prompt="a city at sunset", model_name="black-forest-labs/FLUX.1-dev",
                          sandbox_id=sandbox.id)
image = job.wait(timeout=900, interval=5)

# 3. Stop when done — provisioning/active/alert all count toward the tier limit.
sandbox.stop(); sandbox.wait_for_stop()

Model IDs must match the catalog exactly (no -FP8 suffix) or create_sandbox raises Unsupported sandbox model. See reference/sandbox.md for the full model catalog, tiers, categories, pricing, and pitfalls.

Error handling

from videodb.exceptions import AuthenticationError, InvalidRequestError

try:
    conn = videodb.connect()
except AuthenticationError:
    print("Check your VIDEO_DB_API_KEY")

try:
    video = coll.upload(url="https://example.com/video.mp4")
except InvalidRequestError as e:
    print(f"Upload failed: {e}")