Media Processing

Ingest and track processing of media files (video, audio, images) through a configurable 3-phase pipeline.

End-to-End Workflow

The processing pipeline follows a sequential 3-phase flow:

1. Ingest (ingest_media) - Register a media file, detect MIME type, extract duration, deduplicate by content hash.
2. Preprocess (extract_keyframes) - Detect dead time, segment the video into windows, extract downscaled keyframes, build a subject registry, and write a pipeline manifest.
3. Map (analyze_keyframes) - Send each segment's frames to the configured Gemini vision model with assistant-provided extraction instructions and a JSON Schema for guaranteed structured output. Supports concurrency pooling, cost tracking, resumability, and automatic retries.
4. Reduce / Query (query_media) - Send all map output to Claude for intelligent analysis and Q&A. Supports arbitrary natural language queries about video content.
5. Clip (generate_clip) - Extract video clips around specific moments.

The processing pipeline service (services/processing-pipeline.ts) orchestrates phases 2-4 automatically with retries, resumability, and cancellation support.

Tools

ingest_media

Register a media file for processing. Accepts an absolute file path, validates the file exists, detects MIME type, extracts duration (for video/audio via ffprobe), and registers the asset with content-hash deduplication.

media_status

Query the processing status of a media asset. Returns the asset metadata along with per-stage progress details. Use this to monitor pipeline progress.

extract_keyframes

Preprocess a video asset: detect dead time via mpdecimate, segment the video into windows, extract downscaled keyframes at regular intervals, build a subject registry, and write a pipeline manifest.

Parameters:

• asset_id (required) - ID of the media asset.
• interval_seconds - Interval between keyframes (default: 1s). Use 0.5s for sports/action content where frame density matters.
• segment_duration - Duration of each segment window (default: 15s).
• dead_time_threshold - Sensitivity for dead-time detection (default: 0.02).
• section_config - Path to a JSON file with manual section boundaries.
• detect_dead_time - Whether to detect and skip dead time (default: false). Dead-time detection can be too aggressive for continuous action video like sports - it may incorrectly skip live play. Enable only for content with clear idle periods (e.g., lectures, surveillance footage).
• short_edge - Short edge resolution for downscaled frames in pixels (default: 480).
• include_audio - Whether to extract and transcribe audio for each segment (default: false). When enabled, each segment's audio is transcribed using the configured STT service and stored alongside visual frames.

analyze_keyframes

Map video segments through Gemini's structured output API. Supports two modes:

• keyframes (default) - Reads frames from the preprocess manifest, sends each segment's images to Gemini. Requires extract_keyframes to be run first. Best for longer videos (> 1 hour) or when you need fine-grained control over frame selection (interval, segment duration, dead-time skipping).
• direct_video - Uploads the video file directly to Gemini's Files API. Gemini sees actual motion and temporal context instead of static frames. Best for shorter videos (< 1 hour) where temporal context matters (detecting actions, transitions, motion patterns). Has a 2 GB file size limit. Does not require extract_keyframes preprocessing.

Both modes produce the same MapOutput format, so query_media works identically regardless of which mode was used.

Parameters:

• asset_id (required) - ID of the media asset.
• system_prompt (required) - Extraction instructions for Gemini.
• output_schema (required) - JSON Schema for structured output.
• mode - Analysis mode: 'keyframes' (default) or 'direct_video'.
• context - Additional context to include in the prompt.
• model - Gemini model to use (defaults to the Gemini provider's recommended vision model).
• concurrency - Maximum concurrent API requests (default: 10, keyframes mode only).
• max_retries - Retry attempts per segment on failure (default: 3).

query_media

Query video analysis data using natural language. Sends map output (from analyze_keyframes) to Claude for intelligent analysis and Q&A. Supports arbitrary questions about video content.

Parameters:

• asset_id (required) - ID of the media asset.
• query (required) - Natural language query about the video data.
• system_prompt - Optional system prompt for Claude.
• model - LLM model to use. Defaults to the assistant's active chat model.

generate_clip

Extract a video clip from a media asset using ffmpeg. Applies configurable pre/post-roll padding (clamped to file boundaries), outputs the clip as a temporary file.

Services

Processing Pipeline (services/processing-pipeline.ts)

Orchestrates the full processing pipeline with reliability features:

• Sequential execution: preprocess, map, reduce.
• Retries: Each stage is retried with exponential backoff and jitter (configurable max retries and base delay).
• Resumability: Checks processing_stages to find the last completed stage and resumes from there. Safe to restart after crashes.
• Cancellation: Cooperative cancellation via asset status. Set asset status to cancelled and the pipeline stops between stages.
• Idempotency: Re-ingesting the same file hash is a no-op. Re-running a fully completed pipeline is also a no-op.
• Graceful degradation: If a stage fails mid-batch (e.g., Gemini API errors), partial results are saved. The stage is marked as failed with the error details, and the pipeline stops without losing work.

Preprocess (services/preprocess.ts)

Handles dead-time detection, video segmentation, keyframe extraction, and subject registry building. Writes a pipeline manifest consumed by the Map phase.

Gemini Map (services/gemini-map.ts)

Sends video segments to the configured Gemini vision model with structured output schemas. Handles concurrency pooling, cost tracking, resumability, and retries.

Reduce (services/reduce.ts)

Sends Map output to Claude as text for analysis. Two modes:

• One-shot merge: assembles all Map results and sends to Claude with a system prompt.
• Interactive Q&A: loads existing map output + user query, sends to Claude.

Concurrency Pool (services/concurrency-pool.ts)

Limits concurrent API calls during the Map phase to avoid rate limiting.

Cost Tracker (services/cost-tracker.ts)

Tracks estimated API costs during pipeline execution.

Audio + Vision Multimodal Analysis

When include_audio is enabled on extract_keyframes, the pipeline transcribes each segment's audio track using the configured STT service and attaches the transcript to the segment data. During the Map phase (analyze_keyframes), Gemini receives both the visual frames and the audio transcript for each segment, enabling multimodal analysis that combines what is seen with what is said.

This is useful for:

• Lectures and presentations: Correlate slide content (visual) with speaker narration (audio).
• Sports broadcasts: Combine on-screen action with commentary for richer event detection.
• Meetings and interviews: Pair facial expressions and gestures with spoken dialogue.
• Tutorials and demos: Link on-screen actions with verbal instructions.

Audio transcription uses the STT service configured in assistant settings. If no STT service is configured or transcription fails for a segment (no audio track, service errors), the segment gracefully degrades to visual-only analysis.

Best Practices

Map Prompt Strategy: Go Broad, Not Targeted

The single most important insight: always use a broad, descriptive map prompt instead of a targeted one.

A targeted prompt like "find turnovers" locks you into one topic. If the user later wants to ask about defense, formations, or specific players, you'd need to reprocess the entire video. Instead, run a general-purpose descriptive prompt that captures everything visible, creating a rich, reusable dataset. Then all follow-up questions can be handled via query_media with no reprocessing.

One map run, many queries.

The map output will be larger (more tokens per segment), but Gemini Flash is cheap enough that this is a good tradeoff. Only use a targeted prompt if the user explicitly asks for something narrow.