Open Video

open-video — autonomous director skill

v0.0.1 default for high-quality H3 clips is skill/h3-video
(Ollama for H3 + harness). Use this skill when the user wants multi-shot / long-film director behavior (plan → judge → stitch) beyond a single strong clip.

1. What open-video does

open-video is the autonomous director layer on top of open video models. It turns a concept into a finished film by running the loop no open engine ships natively:

plan → craft → validate → generate → judge → refine → stitch → deliver

• Model-agnostic core (core/): planner (coherence bible), crafter, validator, judge-loop, stitcher, selector, ref-pack builder.
• Pluggable backends (backends/<model>/): baseline = MiniMax H3 (#1 open model, Arena parity with closed, native stereo audio). Future: Wan2.2 (physics), LTX-2.3 (speed).
• Engine adapter (engines/comfyui/): drives ComfyUI via its HTTP API. open-video is the brain; ComfyUI is the hands.

It is NOT a video engine (ComfyUI is the engine) and NOT a model (H3/Wan/LTX are backends). It is the agent brain ComfyUI lacks: judge→refine + multi-shot stitch + coherence planning as they land.

v0.0.1 honesty: prefer skill/h3-video for reliable high-quality single clips. Multi-minute film is the flagship design. The judge is REAL when env-wired: set OPEN_VIDEO_VLM_URL + OPEN_VIDEO_VLM_MODEL (+ OPEN_VIDEO_VLM_KEY) to any OpenAI-compatible vision endpoint and every shot is scored + diagnosed automatically; with the env unset it is an honest PASS stub — then manual frame review is mandatory. Single open models still cap ~15s/shot; longer output needs multi-shot orchestration (partial).

2. Agentic procedure — run these steps in order

Step 1 — Understand the request. Classify the job: single shot vs multi-shot film; target duration; did the user supply reference image(s) / video / audio; desired aspect; quality bar. If the request is ambiguous and the target is a film >30s, ask ONE focused question (subject + mood + length). Do not generate a long film on guesswork.

Step 2 — Pick the mode. Mode is auto-derived from inputs (see backends/h3/backend.py and scripts/validate_prompt.py detect_mode):

• T2V / T2VA — text only, no image. Native audio + video.
• I2V / I2VA — one image (first frame). Instruction line: "For the target video, at 0.00 seconds into the target video, <Picture 1> (from [Shot 1]) is fully referenced."
• FL2VA — two images (first + last frame). Continuous interpolation. This is the multi-shot chain mode: previous shot's last frame → next shot's first frame for continuous handoff.
• L2VA — one image referenced at the final timestamp.
• R2V / Ref2VA — reference video/audio for identity / style / motion / voice. Tag refs <Picture 1>, <Video 1>, <Audio 1> and explicitly assign each a role.

Step 3 — Craft the 3-field prompt (per backends/h3/PROMPT_GRAMMAR.md). Exact structure:

[<instruction line>     # ONLY for I2V/FL2VA/L2VA — first line, then a blank line]

integrated_multimodal_description: [Shot 1] <style first>, <composition/subjects/scene>. <camera type + amplitude + speed>. [Shot 2] At 00:0X.XXX, the camera cuts to <next beat>.
overall_soundscape: <1–4 sentences: ambient / physical / non-verbal human sound>
non_diegetic_music: <1–3 sentences: instrumentation / tempo / rhythm / dynamics — NO mood words>

Hard rules: state style first in Shot 1 (Cinematic / live-action / 2D-animated / 3D CG / claymation / watercolor / vintage film); don't timestamp Shot 1; later shots use strictly increasing cut times within the duration; keep identity/wardrobe/color/objects/spatial relations consistent across shots; camera motion = natural prose combining type + amplitude + speed (Push In / Pull Out / Pan / Truck / Tilt / Arc / Tracking / Static / POV / Roll — omit amplitude/speed when medium/normal); dialogue as <d>[lang] verbatim words</d> with stable (S1)/(S2) speaker IDs (first appearance gives age/gender/on-screen/pitch/timbre/rate/accent); on-screen text in English double quotes, verbatim; every detail must be visible or audible — no abstract mood/emotion words; prefer camera motion over a cut for mere distance/angle changes.

Step 4 — Validate before generate (core/validator.py, hard gate — never skip). Checks: all 3 required fields present; duration within 4–15s; mode matches the instruction line; image/ref counts match the mode; cut times strictly increasing and within duration; <d>[lang]…</d> well-formed. Fix every issue before spending GPU. A shot that fails validation will fail the judge.

Step 5 — Generate via ComfyUI (backends/h3/backend.py + engines/comfyui/adapter.py). Confirm the server is up first (§3). H3 defaults: 1344×768, 20 steps, res_multistep / simple scheduler, shift_video=12.0 / shift_audio=3.0, INT8 ConvRot quants, engine flags --lowvram --use-sage-attention. length snaps to the 17k+5 grid.

Step 6 — Judge the output (core/judge.py). Activate the real judge with env: OPEN_VIDEO_VLM_URL + OPEN_VIDEO_VLM_MODEL (+ OPEN_VIDEO_VLM_KEY) — the pipeline then extracts frames and assesses vs prompt intent + quality bar automatically (QualityJudge.from_env() is the entry point; explicit QualityJudge(vision_fn=…) also works). Verdict: PASS / REFINE / FAIL, with score + issues in the receipt (run --json exposes them). With the env unset the judge auto-PASSes — then you must manually review frames.

Step 7 — Refine if REFINE or FAIL. Diagnose the specific issue, apply a targeted fix (prompt tweak / +steps / different mode / ref-pack for identity lock / different seed), regenerate. Strategy is refine-primary, not best-of-N: H3 raw quality is already at parity — the loop fixes adherence, length, and consistency, which is where the gap actually lives. Best-of-N is an optional escape hatch, not the default.

Step 8 — Stitch multi-shot (core/pipeline.py LongFilmPipeline.make_film). Each subsequent shot's first_frame = the previous shot's last frame (ffmpeg-extracted at -sseof -0.1); a t2v shot is auto-upgraded to i2v when a handoff frame exists. Then ffmpeg concat (-f concat -c copy)

• cross-shot audio continuity (music theme / dialogue language / ambient crossfade). Optional 2K upscale via API as a final step.

Step 9 — Deliver. One coherent film + per-shot receipts (prompt, seed, settings, judge verdict, extracted frames). Persist receipts under artifacts/verify/.