Parsing Video

Parsing Video

Claude cannot play video, but it can read images. To interpret a video, sample frames evenly across its duration, tile them into timestamped contact sheets, and Read the sheets. A 4×4 sheet compresses ~16 moments into one image, preserving narrative flow — what changed, in what order, roughly when.

Requires ffmpeg/ffprobe (apt-get update && apt-get install -y ffmpeg if missing).

Workflow

1. Probe first

ffprobe -v quiet -print_format json -show_format -show_streams input.mp4

Note duration, resolution, and whether there's an audio stream. Duration drives how many sheets you need.

2. Generate contact sheet(s)

python3 scripts/contact_sheet.py input.mp4                          # 1 sheet, 4x4, whole video
python3 scripts/contact_sheet.py input.mp4 --sheets 3               # 48 frames across 3 sheets
python3 scripts/contact_sheet.py input.mp4 --start 120 --end 300    # zoom into 2:00–5:00
python3 scripts/contact_sheet.py input.mp4 --grid 3x3 --tile-width 500  # fewer, larger tiles

The script probes duration, samples frames at interval midpoints, stamps each tile with its source timestamp (H:MM:SS, bottom-left), and tiles them into <name>_sheet_NN.png. It prints each sheet's time range.

Sheet budget — more sheets = more Read calls; scale to duration and task:

3. Read and interpret

Read each sheet image. Tiles run left-to-right, top-to-bottom in time order; use the stamped timestamps to anchor observations ("the scene changes around 1:42"). Cross-sheet continuity: the last tile of sheet N immediately precedes the first tile of sheet N+1.

4. Zoom when needed

Contact sheets trade resolution for coverage. When something needs a closer look:

# Re-sheet a narrower window at higher tile resolution
python3 scripts/contact_sheet.py input.mp4 --start 95 --end 125 --grid 3x3 --tile-width 500

# Or extract a single full-resolution frame at the moment of interest
ffmpeg -ss 00:01:42 -i input.mp4 -frames:v 1 detail.png

Content-aware sampling

Uniform sampling guarantees temporal coverage but ignores structure: it can straddle a cut mid-interval, waste tiles on a static shot, or land mid-pan on a motion-blurred frame. Two refinements, both feeding --at. Choose by footage type: edited content (films, trailers, TV) → shot boundaries; continuously shot footage (handheld, drone, screen recordings, dashcam) → dwell points; unknown → uniform first, refine after.

Shot boundaries (cuts)

Detect cuts and align tiles to them:

# ffmpeg scene score: frames whose difference from the previous frame exceeds 0.3
TS=$(ffmpeg -i input.mp4 -vf "select='gt(scene,0.3)',metadata=print:file=-" -f null - 2>/dev/null \
     | grep -oP 'pts_time:\K[0-9.]+' | paste -sd,)
python3 scripts/contact_sheet.py input.mp4 --at "$TS"

Each selected frame is the first frame of the new shot (the score compares against the previous frame). Threshold 0.3–0.4 suits most content; lower it for subtle cuts, raise it for noisy footage.

Know what scene detection misses. The scene score is a frame-pair difference metric:

• Gradual transitions (dissolves, fades, wipes) spread the change across many frames, each below threshold — they often go undetected.
• Within-shot content changes: between two cuts, objects can enter, leave, or change substantially with no boundary ever firing. A long static-camera shot with lots of action yields one tile under pure scene-aligned sampling.
• Camera motion (pans, handheld shake) can fire false positives.

So treat scene-aligned sampling as a refinement, not a replacement: run uniform sheets first for guaranteed temporal coverage, then a scene-aligned sheet (or a union of both timestamp sets via --at) when shot structure matters. If cut detection quality itself matters, the purpose-built tool is PySceneDetect (uv pip install --system scenedetect[opencv-headless], then scenedetect -i input.mp4 list-scenes) — its content/adaptive detectors are more robust to motion and noise than the raw ffmpeg score, but they are still cut-oriented and share the within-shot blindness above.

Dwell points (where the camera settles)

In continuously shot footage cuts are rare or absent — the structure lives in camera moves. In the frame-difference signal, held compositions are the valleys and pans/zooms are the peaks between them. The valley midpoints are the frames worth sampling: sharp, deliberately framed, one per composition — where uniform sampling would land mid-pan on smeared pixels.

python3 scripts/contact_sheet.py input.mp4 --at "$(python3 scripts/dwell_points.py input.mp4 --max 16)"

dwell_points.py computes the motion signal cheaply (4 fps at 160 px via ffmpeg's scene metric), smooths it over ~1 s, and takes spans below a relative threshold (default p40 of the signal) lasting at least --min-dwell (1 s). It keeps the --max longest holds and prints their midpoints; stderr lists each hold span with its duration so you can see the video's rhythm before reading a single frame.

Caveats:

• The threshold is relative, so on constant-motion footage (one unbroken pan) it degrades to picking the least-motion moments — harmless but arbitrary; prefer uniform sampling there. It exits non-zero when no hold lasts --min-dwell.
• Subject motion during a held shot (a person gesturing in a static frame) raises the valley floor but rarely fills it; if a busy-but-held shot is missed, lower --percentile.
• Dwells say where the camera settled, not what changed — combine with uniform coverage for anything time-critical.