Cut Video

Cut Video

Tighten a long-form recording aggressively: remove silences, fillers, hedges, weak transitions, false starts, and repetitions. Preserves laughs and comedic pauses. Outputs a cleaned MP4 ready to drop into CapCut for layouts/zooms/memes.

This skill is more accurate than cutting from Whisper alone. Cuts are driven by Montreal Forced Aligner (MFA) word boundaries (~10–20ms precision, with true inter-word silences as explicit intervals), not Whisper timestamps (±100–300ms, with pauses embedded inside word durations). Whisper is used only to produce the transcript text that MFA aligns to the audio. The result: tighter cuts, no clipped word onsets/tails, and reliable silence detection.

⚠️ But MFA times are a DRAFT, not ground truth (burned 2026-07-02, AI is bad at jokes DJI run). MFA can only align the transcript whisper gave it. On retake-heavy footage whisper COLLAPSES repeated lines, so MFA smears one transcribed instance across several spoken takes — measured drift on that run was 1–6 seconds in the back half ("the average of all working code": MFA said 217.2s, real onset 225.4s; the final "what this means" take: MFA said 114.7s, real 130.4s). Every keep boundary must be ground-truthed with isolated-window re-transcription (Step 2.5) before rendering. Skipping that step on that video would have produced garbage cuts for the entire second half.

Style target (calibrated from AI mogging my dad.mp4):

• ~1 cut every 1.3–1.5 seconds (median cut duration ~1.1s)
• ~40–60% total runtime reduction from raw
• Cuts happen mid-sentence, not just between sentences
• Word-by-word tightening — fillers, hedges, and weak openers get sliced

Inputs

• Video file path (the user will provide; otherwise ask)
• Optional: tone preference (aggressive / balanced / sentimental / documentary) — default: aggressive

Working directory

/tmp/cut-video/<basename>/ — mkdir at start, leave artifacts for debugging.

Step 0 — Make a proxy (the single biggest speed move)

If the source is HEVC, >500MB, or 4K+, transcode to a 1080p H.264 working copy FIRST. Every subsequent step runs against the proxy, not the source.

Check orientation first (ffprobe … width,height) — DJI/phone footage is often vertical 9:16, and a hard-coded scale=1920:1080 would squash it. Use scale=1920:1080 only for landscape; use scale=1080:1920 for portrait (or the orientation-proof scale=-2:1080 / scale=1080:-2).

ffmpeg -y -hwaccel videotoolbox -i "$SRC" \
  -vf scale=1920:1080 \        # portrait sources: scale=1080:1920
  -c:v libx264 -preset fast -crf 20 -pix_fmt yuv420p \
  -c:a aac -b:a 192k \
  /tmp/cut-video/$NAME/proxy.mp4

Critical flags:

• -hwaccel videotoolbox on the INPUT — hardware-decodes HEVC, ~5–10× faster on Apple Silicon. Skip this and you'll wait minutes instead of seconds.
• -preset fast — the libx264 default is medium, which is ~3× slower for no useful quality gain on a working copy.

Hardware encode alternative (even faster on M-series, slightly larger file):

-c:v h264_videotoolbox -b:v 8M

Step 1 — Get the transcript TEXT (whisper, as input for MFA)

MFA is a forced aligner, not a transcriber — it needs a transcript to align. Whisper's only job here is to produce that text; whisper word timestamps are NOT used for cutting (they're ±100–300ms off and turbo embeds pauses inside word durations). MFA (Step 1.5) supplies all timing.

ffmpeg -y -hwaccel videotoolbox -i /tmp/cut-video/$NAME/proxy.mp4 \
  -vn -ac 1 -ar 16000 /tmp/cut-video/$NAME/audio.wav
whisper /tmp/cut-video/$NAME/audio.wav \
  --model tiny.en --word_timestamps True --output_format json \
  --output_dir /tmp/cut-video/$NAME --language en

Model choice (revised 2026-07-02 — "we only need the words" was WRONG): the transcript text IS the alignment input, so transcript errors become timing errors. On the AI is bad at jokes run, tiny.en dropped exactly the words the old warning predicted ("bland", "slop"), misheard "Compare this to" as "comparative is", and collapsed full-line retakes — and MFA, aligning that wrong text, drifted 1–6s across the back half. Use tiny.en only for a first fast pass to see the take structure; if the transcript shows retakes/repetitions, or the footage is noisy/outdoor, the per-region ground-truth pass (Step 2.5) with small.en supplies the real cut times anyway. (A full-pass small.en transcript is a reasonable upgrade for the MFA input too — ~1–2 min on a 5-min video — but it ALSO collapses retakes, so it does not remove the need for Step 2.5.) For non-English use --model base and drop --language. We keep --word_timestamps True only so whisper times survive as a cross-check/fallback (Step 1.5 caveats, 2a-legacy) — never as the primary cut source.

Step 1.5 — Align the transcript to the audio with MFA (Montreal Forced Aligner) — the timing engine

This is the spine of the skill. MFA forced-aligns the whisper transcript text to the audio and returns word boundaries at ~10–20ms precision, with true inter-word silences as explicit empty intervals. All cut decisions (gaps, fillers, false starts, retakes) use MFA word times.

Auto-install MFA (the skill does this itself — don't make the user do it). Run this idempotent bootstrap at the start of every run; it's a no-op once everything's present (a few seconds), and a one-time ~2–3 min install on a fresh machine. Tell the user "installing MFA (one-time)…" only when it actually installs something.

# 1. Ensure conda exists (MFA's only supported install path). Install miniforge via brew if missing.
if ! command -v conda >/dev/null 2>&1; then
  echo "conda not found — installing miniforge (one-time)…"
  brew install --cask miniforge || brew install miniforge
  # make conda available in this shell
  eval "$("$(brew --prefix)/bin/conda" shell.bash hook 2>/dev/null || conda shell.bash hook)"
fi
source "$(conda info --base)/etc/profile.d/conda.sh"

# 2. Ensure the mfa env exists with MFA installed
if ! conda env list | grep -q '^mfa\b'; then
  echo "creating mfa conda env (one-time)…"
  conda create -n mfa -c conda-forge montreal-forced-aligner -y
fi

# 3. Ensure the English acoustic model + dictionary are downloaded (idempotent; skips if present)
conda run -n mfa mfa model download acoustic english_mfa  2>/dev/null || true
conda run -n mfa mfa model download dictionary english_mfa 2>/dev/null || true

If brew itself is missing, MFA can't be auto-installed — fall back to the whisper+silencedetect path (see the fallback bullet below) and tell the user, with the one-line brew install miniforge they'd need to unlock MFA precision.

Per run:

# corpus = a dir with audio.wav + audio.txt (the whisper transcript as PLAIN TEXT, no timestamps)
mkdir -p /tmp/cut-video/$NAME/corpus
cp /tmp/cut-video/$NAME/audio.wav /tmp/cut-video/$NAME/corpus/
python3 -c "import json;d=json.load(open('/tmp/cut-video/$NAME/audio.json'));open('/tmp/cut-video/$NAME/corpus/audio.txt','w').write(d['text'])"
conda run -n mfa mfa align --clean /tmp/cut-video/$NAME/corpus english_mfa english_mfa /tmp/cut-video/$NAME/aligned
# → aligned/audio.TextGrid — "words" tier: (start, end, word); EMPTY-label intervals = true silences

Parse the TextGrid words tier (pip praatio, or a 20-line parser — interval tiers are plain text). What MFA buys:

• True word boundaries → cut pads shrink to ~0.02s, no clipped word onsets/tails.
• True silences are explicit empty intervals → the 2a gap table works directly off the TextGrid; the whisper embedded-pause problem disappears.
• Caveat: MFA aligns the transcript it's GIVEN. If whisper collapsed a spoken stutter ("growth… growth marketer" → "growth marketer"), alignment drifts locally around it — keep the silencedetect pass (2a) as a CROSS-CHECK, and treat disagreements > 0.3s as suspect regions to re-inspect. On retake-heavy self-shot footage the drift is not local — it's cumulative and can reach seconds (see the warning at the top and Step 2.5). And when the noise floor kills silencedetect too (low-dynamic-range guard), you have NO automatic cross-check — Step 2.5 is then mandatory, not optional.
• Drift smoke alarm — stretched words: any short word whose MFA or whisper duration is ≳1s ("like" 1.16s, "there's" 1.5s, "of" 2.4s, "the" 4.9s on the 2026-07-02 run) is hiding a pause, a flub, or an ENTIRE collapsed retake inside it. Collect every stretched word up front; each one marks a region whose true content is unknown until re-windowed (Step 2.5). One stretched "the" turned out to contain a complete clean take of the whole sentence — the take that got kept.
• Fallback: if MFA/conda is unavailable and the user doesn't want the install, fall back to whisper words + audio silencedetect (2a) — and say so in the plan summary. This is the ONLY path where whisper timestamps drive cuts.

Step 2 — Build the cut list (AGGRESSIVE by default)

Compute a list of (start, end) intervals to KEEP from the MFA word intervals (whisper JSON only as fallback). Aggressive cutting removes content in four categories:

2a. Silence gaps — MFA empty intervals first, audio silencedetect as cross-check (updated 2026-06-10)

With MFA (Step 1.5), true silences are the TextGrid's empty-label intervals — apply the gap table below to those directly. Cross-check with audio silence detection — but CALIBRATE the threshold to the recording's own noise floor; never hard-code -30dB (added 2026-06-16, burned on a DJI-mic to-camera take whose floor sat at ~-24dB: a fixed -30dB saw the ENTIRE take as non-silent, so every "silent" gap measured "above threshold = content, keep" and ~8s dead-air pauses survived the cut):

# 1. Measure the noise floor F and speech level S from per-0.5s-window RMS
#    (slice the wav into 0.5s windows, run volumedetect on each, collect mean_volume;
#     F ≈ 10th-percentile window RMS  (room tone),  S ≈ 90th-percentile  (speech))
# 2. Set the silence threshold RELATIVE to the floor, not absolute:
THRESH=$(python3 -c "print(f'{F + 6:.0f}')")        # 6 dB above the measured floor
ffmpeg -i proxy.mp4 -af silencedetect=noise=${THRESH}dB:d=0.14 -f null - 2>&1 | grep silence_

⚠️ Low-dynamic-range guard (the real lesson, 2026-06-16): if S − F < ~8dB (noisy mic — outdoor/DJI/lav with AC hum, where the per-second RMS reads the SAME during speech and silence), NO energy threshold can separate speech from silence. Do not trust silencedetect/volumedetect at all in that case — drive every cut from MFA word boundaries, or in the no-MFA fallback from small.en word ONSETS + stretched-word pause detection (a whisper word whose duration ≫ a normal word IS a hidden pause: keep ~0.10 + 0.105·len(word)s of onset, then cut to the next word's start). Say which path you used in the plan summary.

Without MFA, the silencedetect pass is the trustworthy gap source ONLY when the dynamic-range guard passes: mlx_whisper/whisper-turbo (and tiny.en) EMBED pauses inside word durations — a 3s "word" is really "word [long pause]", inter-word gaps read ~0.00, and gap-based trimming does NOTHING. Use small.en (not tiny.en) for the fallback path — tiny stretches words across silence even worse and silently drops words ("bland", "slop") and whole retakes. Parse silence_start/silence_end, remove those intervals (keep ~0.04s pad; MFA-precision boundaries allow ~0.02s). For "almost no pauses" → d=0.12, pad 0.03. Combine with the dedup (2c/2d) by removing silences ∪ dropped-word-ranges. This is how you hit median ~1.1–1.3s.

• Stutters whisper hides: turbo collapses spoken repetitions ("growth… growth marketer" → "growth marketer") and varies run-to-run (drops/re-adds words). These are invisible to transcript-based cutting — MFA alignment drift + silencedetect disagreement marks the region; re-inspect it, or ask the user for the timecode and cut a precise window.

2a-legacy. Silence gaps between words (TIGHT thresholds)

The aggressive thresholds match the reference video's pacing (median cut 1.13s).

2b. Filler words — cut with 50ms padding

Always cut: um, uh, umm, uhh, er, erm, ah, mhm, hmm (standalone)

2c. Hedges & weak transitions — cut aggressively in default mode

Cut these when they're delivered as filler (not as substantive content). Mark for cut, then verify against context in the review step:

• Hedges: like (as filler, not as comparison), you know, I mean, I guess, I think (when hedging, not asserting), kind of, sort of, basically, literally, actually (as filler)
• Weak transitions: so (sentence-opener), and so, and then, but um, but uh, okay so, right so, well
• Redundant qualifiers: kind of like, sort of like, or whatever, or something
• Re-orientation phrases: anyway, so anyway, the thing is, what I'm saying is, to be honest, honestly

In balanced / sentimental / documentary modes, only cut hedges that precede a re-statement (false start, see 2d). Keep them in conversational moments.

2d. False starts and repetitions — cut aggressively

Detect by scanning for:

• Self-corrections: speaker re-states the same opening within ~3 seconds. Pattern: "I want — I wanted to...", "It's a — it's an interesting...". Drop the earlier attempt + the gap.
• Restart preambles: "so basically what I'm saying is...", "the point is...", "let me start over...". Drop the preamble, keep the actual point.
• Verbatim repetitions: speaker says same 3+ words twice in a row. Keep the cleaner take.

Implementation hint: build a sliding-window fuzzy-match on consecutive word n-grams. When 3-gram similarity > 0.8 within a 3-second window, flag the earlier instance for removal.

2d-bis. FULL-LINE RETAKES (self-shot to-camera recordings, added 2026-06-10)

Self-shot intros/promos contain whole-sentence retakes separated by 5–60s ("The next person is Carmen who works. … The next person, the next guest I invited is Carmen who's social and content lead at Slate") — far outside 2d's 3-second window. Detect and resolve them:

• Detection: fuzzy n-gram match (4-grams, similarity > 0.7) across a 60-second window; also match sentence OPENERS (first 3–5 words) since retakes usually restart the line.
• Keep the LAST take by default — speakers re-record until satisfied. Exception: if the last take is incomplete/flubbed and an earlier one is clean, keep the clean one and note the override in the plan summary.
• A retake boundary is usually preceded by a long pause, a breath, a click, or a slate-phrase ("okay", "again", "take two") — the pause+opener-match combo is the strongest signal.
• Surface every retake group in the Step 3 plan (line → takes at [t1, t2, t3] → keeping t3) so the user can override which take survives.

2e. NEVER cut these — protect explicitly

• Laughs. Whisper transcribes laughter as silence. Before applying ANY silence cut, check audio amplitude:

  ffmpeg -i audio.wav -af "volumedetect" -f null - 2>&1 | grep mean_volume

  Compare the window's RMS to the measured noise floor F (2a), NOT to a fixed dB: if window RMS > F + ~10dB it contains audible content (laugh, breath, reaction) — keep it. A hard-coded -30dB/-25dB cutoff fails on noisy mics whose floor already sits above it — it then "protects" pure room tone as if it were a laugh, which is exactly how dead air survives. Use ffmpeg -i audio.wav -af "astats=metadata=1:reset=1,ametadata=print:key=lavfi.astats.Overall.RMS_level" for per-window RMS. (And remember the low-dynamic-range guard in 2a: when S − F < ~8dB, RMS can't tell a laugh from room tone either — fall back to MFA/word-onset timing and surface anything ambiguous in the plan for the user to judge.)
• Comedic beats. A deliberate pause after a punchline. In aggressive mode, still trim to 0.4–0.6s (don't kill it, just tighten).
• Reaction sounds. Sighs, "ooh", "wow", gasps — all content.
• Single-word punchlines. If a sentence ends with a one-word zinger after a beat, keep the beat.

2f. STRICTNESS PASS — zero tolerance for repetitions (Louise, 2026-06-10: "get rid of ANY repetitions")

After building the keep list, run a machine scan — do NOT trust your eyes on the transcript:

1. Scan the MFA words that fall INSIDE kept ranges for (a) consecutive duplicate words, (b) duplicate bigrams ACROSS segment boundaries (a flubbed restart can straddle a cut — "…meet her. And we're [flub] / And we're going to…" survived a snap once), (c) duplicate sentence-openers in adjacent sentences ("So… So…" → cut the second).
2. Cut hedge flubs even mid-stat: "I believe", opener "Like", "I guess". Keep rhetorical repetition (parallel structure) — that's intentional.
3. When excising a word, cut to the FULL MFA word end (+0.01s) — a 15ms vowel residue still reads as the word.
4. MFA OOV trap (burned 2026-06-10): acronyms/names missing from MFA's dictionary ("AI", brand names) silently VANISH from the alignment — the word gets absorbed into its neighbor, and snapping a keep-end to "the last MFA word" then CLIPS the real spoken word. Guard: if whisper's chunk end exceeds the last MFA word end by >0.2s, trust the LATER of (whisper end, next silencedetect onset) for that boundary — and verify that junction with an isolated-segment transcription.
5. Verification trap: whisper-of-the-output LIES twice. It (a) COLLAPSES surviving stutters (a real "and we're… and we're" transcribes once — looks clean, isn't), and (b) HALLUCINATES context words at cut junctions (a "girls"+"Enjoy" junction transcribes as "So enjoy" — looks dirty, isn't). So verify with BOTH: the kept-range MFA dup-scan (catches real stutters), and per-segment isolated transcription of any suspect junction (clears hallucinations). Only ship when the dup-scan returns NONE.
6. ⭐ CROSS-ASR VERIFICATION — the strongest stutter detector (discovered 2026-06-10). The MFA dup-scan only sees the words WHISPER transcribed — if whisper collapsed a phrase retake ("with the most recent video… with the most recent video" → once), MFA never knows it exists and the scan passes a dirty cut. The fix: run the cut through a SECOND, independent ASR and diff. Tella's transcript after upload is perfect for this (different model → different blind spots): it caught 5 repetitions in one pass that the whisper+MFA pipeline certified clean (a doubled phrase, a surviving "or, or", two mid-word fragments "pers-"/"per-", a doubled "into"). Workflow: upload the cut to Tella (or any second ASR) → scan ITS transcript for dup words/bigrams/phrases and mid-word fragments ("xyz-") → map hits back to source times → cut → re-verify. Disagreement between the two ASRs marks exactly the regions to fix or flag to the user.