Clipify

Clipify

Find compelling moments in a video, cut them as standalone clips, optionally reformat 16:9 → 9:16 (face-pan or split-screen), time-warp pans for tighter reveals, and burn opus-style word-by-word captions.

Modes

Pick one before Step 1. The choice changes how you find moments and whether captions apply.

• Dialogue mode (default for podcasts, interviews, talking-heads): Whisper transcribes, you scan for punchlines / reactions / awkward pauses. → Step 1A. Captions in Step 5.
• Action mode (woodchopping, sports, percussion, drumming, anything with repeated impact sounds): skip Whisper entirely, run detect_transients.py to find each strike, cut on each one. → Step 1B. Skip Step 5; the strike sounds are the rhythm.
• Hybrid: dialogue mode for talking sections, action mode for the rest, intercut. Use both 1A and 1B.

If the user says "I'm not talking in this," you're in action mode. If they say "cut on each X" (chop, hit, punch, drumbeat, swing, footstep), you're in action mode. Default to dialogue mode otherwise.

Inputs

• A video file path (the user will provide it; otherwise ask)
• Optional: requested format (9:16, 16:9, 1:1) — if not given, ask after candidates are picked
• Optional: subtitle style preference — if not given, ask before captioning

Tooling (use only the fastest path)

• Whisper: whisper --model tiny.en --word_timestamps True --output_format json (≈10× faster than small.en; quality fine for English). For non-English: --model base (drop --language).
• ffmpeg: add -hwaccel videotoolbox for decode and -preset ultrafast for renders. Use -c:v libx264 -crf 20 for the final master.
• Numpy for audio alignment (FFT cross-correlation). No scipy/cv2 needed.
• Scripts: <skill-dir>/scripts/ (where <skill-dir> is the directory containing this SKILL.md — typically ~/.claude/skills/clipify/)
  • analyze.py — speaker timeline from two ROI motion files
  • build_pan.py — ffmpeg crop x-expression with hard cuts
  • build_ass.py — opus-style ASS captions from whisper JSON
  • audio_align.py — find offset of a sub-clip in a longer source
  • detect_transients.py — find sharp impact sounds (action mode); see --help

Working dir: /tmp/clipify/ (mkdir at start, leave artifacts for debugging).

Workflow

Step 1A — Find dialogue moments (dialogue mode)

mkdir -p /tmp/clipify
ffmpeg -y -hwaccel videotoolbox -i "$VIDEO" -vn -ac 1 -ar 16000 /tmp/clipify/audio.wav
whisper /tmp/clipify/audio.wav --model tiny.en --word_timestamps True --output_format json --output_dir /tmp/clipify --language en

Read the resulting JSON (or .txt) and pick 3–5 candidate clips. Funny signals to scan for:

• Punchlines and reactions: words like "what", "wait", "no way", laughter, "haha", swearing
• Reversal moments: setup question → unexpected answer
• Awkward pauses: Whisper segment with long gap, or filler ("uh", "um")
• Self-roast / quotable one-liners: short declarative sentences that stand alone
• Audio peaks: detect via ffmpeg -af volumedetect or look for rapid back-and-forth (alternating short Whisper segments)

For each candidate, propose: [start, end, why-it's-funny, suggested title]. Aim for 10–25s clips. Show the list and let the user confirm/pick.

Step 1B — Find each strike (action mode)

Replace Whisper with audio-transient detection. The detector finds sharp impacts (axe-on-wood, fist-on-pad, drumhead, ball-on-bat, wood landing on a pile) by computing spectral flux on the 1–6 kHz band — the broadband impulse an impact creates that ambient/wind doesn't.

ffmpeg -y -hwaccel videotoolbox -i "$VIDEO" -vn -ac 1 -ar 16000 /tmp/clipify/audio.wav
python3 <skill-dir>/scripts/detect_transients.py /tmp/clipify/audio.wav \
  --min-flux 30 --min-gap 0.6 > /tmp/clipify/strikes.json

Tuning:

• --min-flux 30 is the sane default. Real impacts register 50–300; ambient/wind sits at 5–15. Too many false positives → raise to 40–50. Too few hits → drop to 20.
• --min-gap 0.6 (seconds) is the closest two strikes can be. Fast drumming may need 0.2; chopping is fine at 0.6–1.0.
• --band 1000:6000 covers wood/metal/glass impacts. Heavy thuds (kick drum, body shots) live lower (--band 200:2000); whistles/clinks/snare cracks higher (--band 4000:10000).
• For multi-clip sources (a folder of phone clips), run the detector on each clip and pick a varied set across angles. Aim for 50–65 strikes for a 45–60s montage at ~0.9s per shot.

Each "shot" should start ~0.4s before the strike (showing wind-up) and end ~0.5s after (strike + brief follow-through). That's ~0.9s per beat.

Step 1.5 — Verify each candidate is in frame

Always do this before rendering, in either mode. Audio detection finds the sound of an event; the camera might not have captured it. Skipping this is the #1 way to get user feedback like "you cut on chops where I'm not even visible."

# Sample one frame at each candidate strike time T:
for T in $STRIKE_TIMES; do
  ffmpeg -y -ss "$T" -i "$VIDEO" -frames:v 1 -vf scale=320:-1 \
    /tmp/clipify/verify_${T}.jpg
done

Read each verify image and drop the candidate if:

• The subject is bent down, off-frame, or behind an obstacle
• A thumb is on the lens (yes, this happens — and you only catch it visually)
• It's clearly a different sound source (door slam, dropped tool) that triggered the detector

For ~30 candidates this is one fast ffmpeg call each plus a single batch image read. Cheap insurance.

Step 2 — Trim each chosen clip

ffmpeg -y -ss "$START" -t "$DURATION" -i "$VIDEO" -c copy /tmp/clipify/clip_$N.mp4

(Use -c copy for instant trim. Re-encode only if cuts must be frame-accurate.)

Step 3 — Decide the output format

Ask the user (skip if they already specified): "9:16 (TikTok / Reels), 16:9 (YouTube), or 1:1 (Insta feed)?"

Step 4 — If 16:9 → 9:16: pan-between-faces vs split-screen

Detect source aspect with ffprobe. If source is 16:9 and target is 9:16, ask:

"Two options: (a) hard-cut pan that follows whoever is speaking (single face on screen at a time), or (b) split-screen stack with both faces visible. Which do you want?"

Skip the question if there's only one face (single-talker clip). For single-talker, just center-crop.

Step 4a — Pan-between-faces (recommended for fast-cut talking-head dialogue)

1. Locate the two face ROIs. Sample one frame: ffmpeg -ss <middle> -i <clip> -frames:v 1 /tmp/clipify/probe.jpg. Read it. Eyeball each face's mouth+chin area as x,y,w,h in the source's pixel space. (No cv2 needed — camera is static within a clip; one frame is enough.) Verify by drawing boxes:

   ffmpeg -i probe.jpg -vf "drawbox=x=$LX:y=$LY:w=$LW:h=$LH:color=cyan@0.9:t=4,drawbox=x=$RX:y=$RY:w=$RW:h=$RH:color=magenta@0.9:t=4" verify.jpg

   Iterate at most twice. Boxes should cover mouth + chin and avoid hands/mics. Don't over-tune — frame differencing is forgiving.

2. Extract per-frame motion energy in each ROI:

   ffmpeg -y -i clip.mp4 -filter_complex "
   [0:v]split=2[a][b];
   [a]crop=$LW:$LH:$LX:$LY,format=gray,tblend=all_mode=difference,signalstats,metadata=mode=print:key=lavfi.signalstats.YAVG:file=/tmp/clipify/L.txt[la];
   [b]crop=$RW:$RH:$RX:$RY,format=gray,tblend=all_mode=difference,signalstats,metadata=mode=print:key=lavfi.signalstats.YAVG:file=/tmp/clipify/R.txt[ra]
   " -map "[la]" -f null - -map "[ra]" -f null -

3. Build speaker timeline (min dwell 1.0s — short interjections merge into the prior speaker):

   python3 <skill-dir>/scripts/analyze.py /tmp/clipify/L.txt /tmp/clipify/R.txt 1.0 > /tmp/clipify/segments.json

4. Pick pan x-coordinates for a 9:16 vertical strip from the source. With source W=1920 and target W=1080, crop strip width = 608.

   • LEFT_X = face_left_center_x - 304 (clamp ≥ 0)
   • RIGHT_X = face_right_center_x - 304 (clamp ≤ source_W - 608)

5. Generate the hard-cut x expression and render:

   EXPR=$(python3 <skill-dir>/scripts/build_pan.py /tmp/clipify/segments.json $LEFT_X $RIGHT_X)
   ffmpeg -y -hwaccel videotoolbox -i clip.mp4 -filter_complex \
     "[0:v]crop=608:1080:x='$EXPR':y=0,scale=1080:1920:flags=lanczos[v]" \
     -map "[v]" -map 0:a -c:v libx264 -preset fast -crf 20 -pix_fmt yuv420p \
     -c:a aac -b:a 192k /tmp/clipify/clip_panned.mp4

   Source 1920×1080 assumed; for 4K source either downscale first or double all coordinates.