Video Pipeline

Video Pipeline - from request to MP4

Engine roles (never swap them)

How the two engines talk to each other: intermediate MP4 files in video-projects/<name>/renders/. HyperFrames does not know Remotion exists and vice versa - the only shared thing is the manifest.

Project lifecycle

1. Initialize

• Create video-projects/<kebab-case-name>/ with index.html, compositions/, assets/, renders/, hyperframes.json, meta.json.
• meta.json is the central manifest:

{
  "id": "tiktok-paper-gpt5",
  "name": "Mổ xẻ paper GPT-5",
  "width": 1080, "height": 1920, "fps": 30,
  "status": "draft",
  "scenes": [
    { "id": "s01-hook",   "src": "compositions/s01-hook.html",   "durationInFrames": 90,  "render": "renders/s01-hook.mp4" },
    { "id": "s02-talking","srcVideo": "assets/talking-head.mp4", "from": 2.5, "to": 14.0 }
  ],
  "audio": {
    "voice": "assets/voice.mp3",
    "sfx": [ { "file": "assets/sound-effects/whoosh-01.mp3", "atFrame": 88 } ]
  },
  "output": null
}

• scenes[] is the contract between the two engines: a scene with src is rendered by HyperFrames; a scene with srcVideo is footage used as-is. Remotion reads this file to assemble - never hardcode the scene list in the Remotion code.

2. Build the scenes (HyperFrames)

• Copy assets/brand/brand-tokens.css into the project, write compositions following the window.__timelines convention.
• Vietnamese videos: apply the verified fixes (gradient text losing diacritics -> render text as real elements + use background-clip correctly; check that every diacritic is present on the first/last frame of each reveal).
• Lint clean before rendering: npx hyperframes lint.
• Render a draft of each scene: npx hyperframes render --quality draft --output renders/<scene>.draft.mp4.

3. Frame verification (MANDATORY, before assembly)

ffmpeg -ss <hero-moment-seconds> -i renders/<scene>.draft.mp4 -frames:v 1 verify/<scene>.png

Inspect every image: does the Vietnamese text keep all its diacritics? Is any text overflowing the edges? Any unexpected white/black frames? Is a face cropped? If anything is wrong, fix the scene - do not move on.

4. Assembly (Remotion) - see the remotion-assemble skill for details

• The Remotion composition reads meta.json, builds <Sequence> entries from scenes[], and inserts sfx at atFrame.
• Render a full draft -> review it in the web UI -> approve.

5. Automated QC (MANDATORY, blocks final)

curl -s -X POST http://localhost:6869/api/projects/<id>/qc -H "content-type: application/json" -d "{}"

The server measures the latest draft with ffmpeg and returns { report: { status, checks[] } }:

• status: "fail" -> the server rejects the assemble-final job with 409 QC_REQUIRED / QC_FAILED. Fix the root cause, re-render the draft, run QC again. Do not use force: true to dodge it unless the user explicitly asks.
• status: "warn" -> consider fixing, not blocking.
• The final report must state the QC results and how each fail/warn check was handled.

6. Final

• Re-render the HyperFrames scenes at --quality standard.
• Remotion final render -> outputs/<project>-v<N>.mp4.
• Update meta.json: status: "done", output: "outputs/...". The web UI reads status from here.

7. Thumbnail + publish package

curl -s -X POST http://localhost:6869/api/projects/<id>/thumbnail -H "content-type: application/json" -d "{\"title\":\"...\",\"frameAt\":12}"
curl -s -X POST http://localhost:6869/api/projects/<id>/publish   -H "content-type: application/json" -d "{}"

publish generates .srt/.vtt from the transcript and has the AI write the title/description/hashtags for TikTok, YouTube and Facebook following the Style Design. Precondition: the transcript must be the FINAL one. If it was cut/remapped, write the final version to assets/transcript.final.json (the transcript loader prefers this file) - otherwise the subtitles will drift out of sync with the cut video.

Render queue rules (backend)

1. Every render goes through the queue - including the ones Claude runs by hand. Jobs are written to SQLite: id, projectId, type (scene-draft|scene-final|assemble-draft|assemble-final|image-gen), status, progress, log, startedAt, finishedAt.
2. Jobs run in parallel up to QUEUE_CONCURRENCY (default 2, configurable via env; set 1 on weak machines). Safety constraint: two jobs from the same project never run at the same time (to avoid trampling renders/meta) - parallelism only happens across different projects.
3. Progress: parse the CLI stdout (both engines print frame progress) -> update the DB -> push SSE to the web UI.
4. Failed jobs: keep the complete log in the DB, show it in the UI, and do not auto-retry more than once.
5. Draft always comes before final. The backend rejects a final job if the project has no successful draft for the current version of the scenes.

Sound effects

• Shared library: assets/sound-effects/, every file has an entry in assets/sound-effects/library.json (file, tags, durationMs, and a description written in Vietnamese).
• When using one in a video: copy it into video-projects/<name>/assets/sound-effects/ then declare it in meta.json - a project must contain all of its own assets (re-rendering must not depend on a library that may change).
• The Sound Effects page in the web UI reads library.json, plays previews inline, and allows uploading new files (the backend updates the json).