Watch Video

/watch-video — Transcribe and analyze any video at the depth you choose

Replaces and broadens the prior youtube-transcript skill. YouTube is now one of many sources; depth is user-controlled.

Step 1 — Parse input

Accept:

• YouTube: full URL, youtu.be/<id>, youtube.com/shorts/<id>, raw 11-char ID
• Loom: loom.com/share/<id> or loom.com/embed/<id>
• Vimeo: vimeo.com/<id>
• Riverside: download URL or local file
• Zoom: local .mp4 from a downloaded recording
• X / IG / TikTok video: URL — defers to social-fetch for metadata, uses yt-dlp for the file
• Local file: any path to an .mp4 / .mov / .webm / .mkv

Detect source from URL pattern or file extension. If ambiguous, ask.

Step 2 — Parse depth mode

If the depth isn't specified and the video is >10 minutes, ask before defaulting (visual/multimodal cost real money on long videos).

Step 3 — Pull metadata

For URL sources, use yt-dlp:

yt-dlp --print "%(title)s|%(uploader)s|%(duration_string)s|%(upload_date>%Y-%m-%d)s|%(description)s" \
  --print "%(chapters)j" --skip-download "<url>"

Capture: title, uploader/channel, duration, upload date, description (first paragraph), chapters (JSON or null).

For local files, use ffprobe:

ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 "<file>"

Step 4 — Build workdir

~/Documents/videos/<source>-<slug>-<date>/

Where:

• source: youtube / loom / vimeo / riverside / zoom / local
• slug: kebab-case of title (first 4–6 words, max 50 chars)
• date: YYYY-MM-DD

Step 5 — Get the transcript

Backend selection (in order):

1. Platform-provided transcript if it exists and looks complete:

   • YouTube: yt-dlp --write-sub --write-auto-sub --skip-download --sub-lang en --sub-format vtt
   • Loom: fetch via https://www.loom.com/share/<id> page metadata or Loom API if $LOOM_API_KEY set
   • Riverside: built-in transcripts available on the recording's share page
   • If platform transcript exists and has timestamps, use it. Skip Whisper.

2. MLX-Whisper local (default on Apple Silicon Macs):

   # Install once: pip install mlx-whisper
   python3 -c "import mlx_whisper; mlx_whisper.transcribe('<file>', path_or_hf_repo='mlx-community/whisper-large-v3-turbo')" \
     > "<workdir>/transcript-raw.json"

   Or via the CLI: mlx_whisper <file> --model mlx-community/whisper-large-v3-turbo --output-dir <workdir>

3. Not on Apple Silicon (Linux, Intel Mac, Windows, cloud agents): faster-whisper (pip install faster-whisper, CPU or CUDA) or whisper.cpp. Same model size, same output handling.

Download the video file first if it's a URL (use yt-dlp; Loom/Vimeo/YT all supported):

yt-dlp -f "bv*[height<=720]+ba/b[height<=720]" -o "<workdir>/video.%(ext)s" "<url>"

720p is plenty for transcription and frame analysis (smaller download, faster processing).

Clean the transcript (only needed for YouTube auto-subs which have rolling captions; Whisper output is already clean):

# YouTube VTT cleanup — de-dup rolling captions, strip tags, paragraph-break on cue gaps >2s
awk '
  /^WEBVTT/ || /^Kind:/ || /^Language:/ || /^NOTE/ { next }
  /-->/ { in_cue = 1; last = ""; next }
  /^$/ { if (last) print last; in_cue = 0; last = ""; next }
  in_cue { gsub(/<[^>]+>/, "", $0); last = $0 }
  END { if (last) print last }
' "<workdir>/transcript.en.vtt" | awk '!seen[$0]++' > "<workdir>/transcript.txt"

Save final to <workdir>/transcript.txt.

Step 6 — If transcript mode: stop here

Output:

• transcript.txt
• metadata.json
• One-line summary in chat: title, source, duration, word count
• Path to workdir
• (Optional) Step 9 — offer to capture to second-brain

Step 7 — If visual mode: extract frames + vision pass

Frame extraction (ffmpeg)

Cadence by source heuristic:

mkdir -p "<workdir>/frames"
ffmpeg -i "<workdir>/video.mp4" -vf "fps=1/15" "<workdir>/frames/frame-%04d.png" -y

For scene-change detection (slide decks especially):

ffmpeg -i "<workdir>/video.mp4" -vf "select='gt(scene,0.3)',showinfo" -vsync vfr "<workdir>/frames/scene-%04d.png" 2> "<workdir>/scene-detection.log"

Vision pass

Pair each frame with the transcript chunk for the same timestamp window. Then batch-send to Claude vision for synthesis.

Per-frame batch prompt (up to ~10 frames per call):

Here are N frames from a video at timestamps T1..TN. For each frame, describe what's on screen in 1–2 sentences. Flag: (a) UI changes from previous frame, (b) text visible on screen, (c) any moment that looks like a decision, action, or notable event. Also note the transcript text spoken during this window.

Save the output as <workdir>/moments.md:

# Key moments — <title>

## 00:00:15 (frame-001.png)
**On screen**: Login form, email field focused
**Transcript**: "So you just open it up and..."
**Note**: Beginning of UI demo

## 00:00:45 (frame-002.png)
**On screen**: Dashboard with 4 cards
**Transcript**: "And here's where you see all your projects."
**Note**: Major view change — first time the dashboard appears

Generate summary

After moments are identified, synthesize the whole video into <workdir>/summary.md:

# Summary — <title>

**Source:** <source URL / file>
**Duration:** <hh:mm:ss>
**Watched at:** <date>
**Mode:** visual

## TL;DR
<2–4 sentences>

## Key moments
- 00:00:15 — <one-line>
- 00:00:45 — <one-line>

## Action items flagged
- <item> [timestamp]

## Decisions flagged
- <decision> [timestamp] — consider routing to /decide

## Quotes worth keeping
- "..." [timestamp]

## Open questions
- <question raised but not answered>

Step 8 — If multimodal mode

Backend selection

1. Gemini native if $GEMINI_API_KEY is set (much cheaper + faster than per-frame for long videos):

   Default model: gemini-3.5-flash (released May 2026, ~$1.50 input / $9 output per 1M tokens; ~$0.15/sec of video; beats 3.1 Pro on coding/agentic benchmarks at 4× the speed). Override to gemini-3.1-pro for brand audits / high-stakes analysis where details matter; gemini-2.5-flash-lite for bulk cheap processing.

   # Step 1: Upload video via Files API
   FILE_URI=$(curl -s -X POST "https://generativelanguage.googleapis.com/upload/v1beta/files?key=$GEMINI_API_KEY" \
     -H "X-Goog-Upload-Command: start, upload, finalize" \
     -H "Content-Type: video/mp4" \
     --data-binary "@<workdir>/video.mp4" | jq -r '.file.uri')
   
   # Wait until file is ACTIVE (Gemini processes the video first)
   while true; do
     STATE=$(curl -s "$FILE_URI?key=$GEMINI_API_KEY" | jq -r '.state')
     [ "$STATE" = "ACTIVE" ] && break
     sleep 3
   done
   
   # Step 2: Generate content with the file + multimodal-analysis prompt
   curl -s -X POST "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.5-flash:generateContent?key=$GEMINI_API_KEY" \
     -H "Content-Type: application/json" \
     -d "{
       \"contents\":[{
         \"parts\":[
           {\"file_data\":{\"mime_type\":\"video/mp4\",\"file_uri\":\"$FILE_URI\"}},
           {\"text\":\"<multimodal analysis prompt — see Step 7's summary template + use-case extensions>\"}
         ]
       }]
     }"

   Files persist in Gemini Files API for ~48 hours — useful for re-querying the same video with different prompts.

2. Dense Claude vision fallback if no Gemini key:

   • Frame cadence: 1 frame per 3s (much denser than visual mode)
   • Batch through Claude vision with the multimodal-analysis prompt
   • Slower and more expensive than Gemini for long videos — warn the user before running on >10min content

Multimodal output

Same summary.md template as Step 7 + an extended section:

## Multimodal observations
- **Body language / delivery**: <observations on talking-head video>
- **Pacing**: <fast/slow/uneven>
- **Visual style**: <brand audit, ad review, design observations>
- **Audio quality / atmosphere**: <music, silence, background>

Exact extra sections depend on the use case (brand audit, ad review, talk delivery review, client-call read). Use case is inferred from the source + the user's verbal framing when invoking.

Step 9 — Optional: capture to second-brain

After any mode completes, offer:

"Want to capture this to second-brain? I'll write a call-<slug>.md (or meeting- / note- / resource-) to ${SECOND_BRAIN_VAULT:-$HOME/Documents/SecondBrain}/raw/ with the summary, source URL, and transcript link."

Type prefix by source:

File body: 1-line source, the summary, link to full workdir.