Video Production

Video Production Skill

End-to-end pipeline for producing AI-narrated educational videos with slides. Distilled from producing 40+ real videos for an AI-run YouTube channel (蝦說 AI), including every mistake we only made once.

Who this is for: an AI agent (Claude, Codex, Gemini, or any coding agent) asked to produce a narrated slide video. A human can follow it too.

Quick Start

0. Read references/teaching-style.md + references/narration-style.md (content quality rules)
1. Create project directory and cd into it; copy config.json from references/config-example.json and fill it in
2. Write narration.json (the script — one string per slide)
3. Generate slides (Path A: gpt-image-2 hand-drawn / Path B: HTML + Playwright screenshot)
4. Visually inspect every slide PNG (wrong characters / clipping / unreadable fonts)
5. TTS narration → MP3 (with built-in ASR verification)   → node scripts/tts_with_asr.js
6. FFmpeg assemble → video.mp4                            → node scripts/assemble.js
7. Quality check (bitrate + frame extraction + visual)
8. Subtitles → SRT (+ optional burn-in)                   → node scripts/gen_subtitles.js
9. Cover image → thumbnail                                → python scripts/cover_gen.py
10. Upload wherever you publish; verify the thumbnail and visibility after upload

Requirements

• Node.js ≥ 18 (scripts use only built-ins + playwright for HTML screenshots)
• Python ≥ 3.9 (only for gpt-image-2 slide/cover generation and optional rescore)
• FFmpeg + FFprobe on PATH (or set explicit paths in config.json)
• API keys (environment variables, or a .env file in the project directory):
  • ELEVENLABS_API_KEY — TTS
  • OPENAI_API_KEY — Whisper ASR verification + gpt-image-2 slides/cover
• A TTS voice: set tts.voiceId in config.json (any voice from your ElevenLabs voice library — a premade voice works out of the box; a cloned voice makes it yours).

⚠️ Never hardcode API keys in scripts and never commit them to git. One of our keys was auto-revoked from a public repo push and 32 videos rendered silent. Environment variables only.

Configuration

Create config.json in your project directory (start from references/config-example.json):

{
  "tts": {
    "provider": "elevenlabs",
    "voiceId": "YOUR_ELEVENLABS_VOICE_ID",
    "model": "eleven_multilingual_v2",
    "maxRetries": 5
  },
  "asr": {
    "provider": "openai",
    "model": "whisper-1",
    "language": "zh",
    "passThreshold": 0.85
  },
  "video": {
    "width": 1920,
    "height": 1080,
    "audioBitrate": "192k",
    "slidePadding": 1.0
  },
  "branding": {
    "watermark": "🎬 Your Channel Name"
  },
  "ffmpeg": "ffmpeg",
  "ffprobe": "ffprobe"
}

Checklist (mandatory, do not skip steps)

□ 0. Read references/teaching-style.md — content/teaching rules (pre-flight constitution)
□ 1. Write narration.json
□ 1.5 ⭐ ALIGNMENT CHECK: narration entries count == slide count. MUST be equal!
     If narration splits a topic across 2 entries, there must be 2 separate slides.
     Mismatch = audio/visual desync for every slide after the mismatch point.
□ 2. Generate slides (Path A or B below)
□ 3. ⭐ Visual check: open every slide PNG with an image tool
     (wrong/garbled characters? clipped edges? readable on a phone?)
□ 4. TTS synthesis + ⭐ ASR verification (similarity ≥ 0.85, never skip)
□ 5. FFmpeg assemble → video.mp4
□ 6. ⭐ Quality check: ffprobe audio bitrate + extract a frame + visual verify
□ 7. Subtitles → subtitles_aligned.srt (external track and/or burn-in)
□ 8. ⭐ Cover image → thumbnail (16:9; pad, don't crop)
□ 9. Upload (start unlisted/private, review, then publish)
□ 10. ⭐ After upload: verify the thumbnail actually shows YOUR cover, not an auto frame

The ⭐ steps are non-negotiable. Each one exists because skipping it once ruined an entire batch of videos.

🔴 The cover is part of the video, not an optional extra. We once shipped a video where every step was done except the cover — it went live with a default gray frame. "No cover = not shipped."

Step 1: Script (narration.json)

A JSON array of strings, one per slide (CRITICAL: array length MUST equal slide count):

[
  "開場白，吸引注意力...",
  "第一個重點，配合比喻...",
  "第二個重點...",
  "結尾，call to action..."
]

Writing guidelines

• 80–150 characters per slide (Chinese). Too short = rushed; too long = TTS breaks.
• Conversational tone: metaphors, rhetorical questions, humor.
• One concept per slide. Don't cram.
• Story > Information. Viewers remember stories, not bullet lists.
• Full teaching methodology (audience empathy, glossing jargon, honest attribution, first-person endings…): references/teaching-style.md. Voice & TTS-safe writing: references/narration-style.md.

TTS-friendly writing (important!)

TTS engines mispronounce things. Prevent it at the script stage:

Punctuation and pacing: our TTS script strips punctuation before synthesis (each punctuation mark becomes a pause in Chinese TTS — dense commas produce machine-gun narration). Write narration that flows in breath-groups of ~8–22 characters; put commas only at real breath points. Subtitles still use the original punctuated text.

Step 2: Slides

Two first-class paths. Pick ONE per video (Path A is our channel default; Path B has no image-API dependency).

Path A — gpt-image-2 hand-drawn teaching style (scripts/slides_gen.py)

Full-bleed AI-generated slides in a "professor's hand-drawn lecture notes" style: white background, bold black CJK title top-left with underline, thin black arrows, lots of whitespace, a small mascot in the corner, stick figures only (no real faces).

1. Write one Chinese prompt per slide into slides_prompts.json (array of strings). Wrap every string that must appear on the slide in 「」. Always end the shared style block with: 「所有中文字必須完全正確、清楚可讀、不可有亂碼或錯字。數字要正確。」
2. python scripts/slides_gen.py → generates slides_raw/slide_NN.png (1536×1024). Batch ≤4–5 concurrent lanes (~120s per image). Do not run Whisper at the same time — concurrent heavy API calls starve each other.
3. Visually inspect every image (garbled characters / wrong or invented numbers / typos → regenerate just that slide; HTTP 502 → just retry that slide). gpt-image-2 WILL invent numbers to fill tables unless your prompt explicitly says 「畫面只能出現 X 這幾個數字，其他留空」.
4. node scripts/pad_and_burn.js pad → 1536×1024 scaled to 1410×940, padded onto a 1920×1080 white canvas with a 140px bottom band reserved for subtitles.

Path B — HTML slides + Playwright screenshot (scripts/screenshot.js)

Create one HTML file per slide: slides/slide_01.html, slide_02.html, … (template: references/slide-template.html), then node scripts/screenshot.js.

Design rules (hard-won; mobile viewers are the majority):

• Resolution = video.width × video.height (default 1920×1080).
• Fill 80%+ of the canvas — padding: 60-80px, flexbox centering, no floating cards.
• Every element on the slide must be mentioned in the narration (and vice versa).
• 🔴 Never use <24px body text. Fewer points in bigger font > more points in tiny font.

Path C — Node Canvas fallback (scripts/generate_slides.js)

No browser, no image API: renders simple title+bullets slides from slides.json. Use only when Playwright and the image API are both unavailable.

Step 3: TTS + ASR Verification (scripts/tts_with_asr.js)

node scripts/tts_with_asr.js [project_dir]

Reads narration.json, synthesizes each entry via ElevenLabs, saves audio/slide_XX.mp3, then verifies every clip with Whisper ASR:

1. Transcribe the generated audio
2. Compute character-overlap similarity vs the original text
3. ≥ 0.85 = PASS (target ≥0.90 for important videos)
4. < 0.85 = FAIL → adjust wording and retry (up to tts.maxRetries)

Adjustment rules: swap synonyms / split long sentences / write numbers in Chinese — but never change the meaning, never drop information.

Don't chase 0.85 forever — verify the words, ship on redundancy. Whisper produces false alarms on Chinese: same-sound different-tone homophones, digits vs 中文數字, Simplified output vs Traditional script. If the ASR "errors" are homophones and every key word/number comes through, the audio is correct — keep the best attempt. The slide shows the number visually and the subtitle uses the original text, so the viewer has triple redundancy. For digit-heavy narration, run python scripts/rescore.py — it strips numerals and compares toneless-pinyin multisets (≥0.90 = pass), which kills most false failures. A real defect = ASR gets the SAME wrong word consistently across multiple synth attempts (that's the TTS mispronouncing, not Whisper mishearing → rewrite that word).