Director

Director — Story-to-Video Production Pipeline

Overview

The Director skill runs a complete short film production from a text story. It breaks the story into scenes, generates start/end frames for each, creates video clips from frame pairs, and concatenates everything into a final video.

The pipeline runs Story Planning → Z-Image Hero + Character Refs → Qwen Edit Chain (all frames) → WAN 2.2 FLF Video Clips → ffmpeg Concatenation.

Key architectural decisions:

• 1 hero frame + edit chain for character consistency (NEVER independent Z-Image per scene)
• Inter-scene frame continuity. Scene N's end frame IS Scene N+1's start frame (same image file, no edit gap)
• Character reference images fed into Qwen Edit's extra image slots
• State file persists to disk for context compaction survival
• Each scene is independently retryable without affecting others
• clear_vram between every model family switch

CRITICAL: Inspect modes + verify every output

This pipeline drives the user's live canvas across many stages, so two habits are non-negotiable:

• Inspect node modes before each render. After loading any pack/template/subgraph and before panel_run, check each node's mode (panel_graph_outline marks [bypass]/[mute]; panel_query_graph detail rows carry it). A bypass node is skipped (passes input through); a mute node and everything downstream don't execute. If the path, branch, or switch you need is bypassed or muted, enable it with panel_set_node_mode (set the wanted node active, the unwanted one bypass/mute). Never assume a switch or route is already active.
• Verify the output matches before moving on. Every Phase-N render is a gate. Look at the produced frame or clip (view it) and confirm it matches the intent BEFORE advancing or reporting progress. If it's wrong, diagnose (wrong prompt path? a bypassed/muted builder or switch? wrong widget? wrong ref image?), fix, and rerun. Do NOT declare a phase done or report progress you haven't verified.
• Confirm VIDEO renders via the filesystem, not /history. For a VHS_VideoCombine / LTX / WAN clip, do NOT rely on get_history / queue (action:"status") to confirm it exists. VHS-style video nodes write the .mp4 but frequently do NOT register an output in ComfyUI's /history (prompt shows done, empty outputs, no error). Confirm the file with get_image (action:"list_outputs") (now lists videos too, tagged kind: "video") by filename_prefix + fresh mtime, then chain it forward with upload_image (action:"stage").
• Bypass completed stages before queuing the next one. If you build the multi-stage pipeline on ONE canvas (e.g. Krea2 → LTX → WAN) rather than running each phase in isolation, once a stage has run and its output is captured/staged, panel_set_node_mode(mode:"bypass") that stage's nodes BEFORE you panel_run the next stage. Otherwise panel_run re-executes the whole graph and you pay for and wait on already-finished work, a real and costly failure mode. Keep only the active stage live; feed the prior output forward with upload_image (action:"stage") (bypass the producer, feed its captured output to the consumer's loader).

CRITICAL: Character Consistency

Independent Z-Image generations per scene produce different-looking characters. This was the #1 problem discovered during testing. The solution:

1. Generate ONE hero frame with Z-Image. It establishes the main character, setting, and lighting
2. Generate character reference images: close-up portraits of each character, key props, and the background
3. Create ALL other scene frames via the Qwen Edit chain from the hero, with character refs in extra image slots
4. This keeps the same face, clothing, and environment across every frame

8-Phase Pipeline

Phase 1: Story Planning       → Break story into scenes (Claude reasoning, no ComfyUI)
Phase 2: Hero + Refs          → Z-Image: 1 hero frame + character ref portraits + background ref
Phase 3: Hero Review          → Visual verify hero and refs, user approves
Phase 4: Edit Chain           → Qwen Edit: chain ALL scene frames from hero (with char refs in slots 2-3)
Phase 5: Frame Review         → Visual verify all frames, approve/reject/retry
Phase 6: Video Clips          → WAN 2.2 FLF dual Hi-Lo (one clip per scene)
Phase 7: Video Review         → Preview each clip
Phase 8: Final Assembly       → ffmpeg concat all clips into one MP4

State File Format

Saved at ~/code/comfyui-mcp/workflows/director_state_{project_id}.json. Updated after every edit or phase completion.

{
  "project_id": "story_20260216_143022",
  "created": "2026-02-16T14:30:22Z",
  "story": "Original user story text",
  "current_phase": 4,
  "orientation": "portrait",
  "hero_frame": { "file": "director_hero_00001_.png", "seed": 428571, "approved": true },
  "character_refs": {
    "man": "director_ref_man.png",
    "cat": "director_ref_cat.png",
    "woman": "director_ref_woman.png",
    "background": "director_ref_bedroom.png"
  },
  "scenes": [
    {
      "id": 1,
      "description": "Brief scene description",
      "edit_prompt_start": "Qwen Edit instruction to create start frame from source",
      "edit_prompt_end": "Qwen Edit instruction to create end frame from source",
      "edit_source_start": "hero",
      "edit_source_end": "hero",
      "video_prompt": "WAN motion description",
      "start_frame": { "file": "director_s1_start_00001_.png", "seed": 12345, "approved": true },
      "end_frame": { "file": "director_hero_00001_.png", "seed": null, "approved": true },
      "video_clip": { "file": "director_s1_00001.mp4", "seed": 11111, "approved": false },
      "status": "video_pending"
    }
  ],
  "final_video": null,
  "settings": {
    "start_frame_resolution": [832, 1472],
    "video_resolution": [480, 720],
    "video_frames": 81,
    "video_fps": 16
  }
}

Models Used Per Phase

CRITICAL: clear_vram between every model family switch.

Phase 1: Story Planning

Break the story into 2 to 6 scenes. For each scene, identify:

• description: what happens (1 to 2 sentences)
• start frame: what the opening frame looks like
• end frame: what the closing frame looks like
• video_prompt: motion description for FLF transition

Identify a hero frame, the single most representative scene image that establishes the main character and setting. This hero will anchor all other frames via Qwen Edit.

Also identify which character reference images are needed (portraits of each character, key props, background).

CRITICAL: Inter-Scene Frame Continuity

The end frame of Scene N must be the EXACT same image file as the start frame of Scene N+1. Do NOT create separate Qwen-edited start frames for subsequent scenes. That causes visible jumps at scene boundaries when the videos are concatenated.

The frame chain for video generation:

Scene 1: S1_start (unique)        → hero (end)
Scene 2: hero (= S1 end)          → S2_end
Scene 3: S2_end (= S2 end)        → S3_end
Scene 4: S3_end (= S3 end)        → S4_end
Scene 5: S4_end (= S4 end)        → S5_end

Only Scene 1 needs a unique start frame. All other scenes inherit their start from the previous scene's end.

Edit Chain Planning

The edit chain produces only end frames (plus Scene 1's unique start frame). Map which end frame derives from which source:

• Some end frames edit directly from the hero
• Later end frames may chain from earlier end frames
• Keep chains shallow (max 4 to 5 deep) to minimize drift

Example chain:

Hero (man+cat on bed)
  ├─ S1 Start: edit hero → remove cat, man alone
  ├─ S2 End: edit hero → replace cat with woman
  │    └─ S3 End: edit S2End → both sit up, man startled
  │         └─ S4 End: edit S3End → sitting close, warm smiles
  │              └─ S5 End: edit S4End → warm embrace

Phase 2: Hero + Character References — Z-Image

Generate with Z-Image RedCraft DX1 (10 steps, CFG 1, euler/simple):

1. Hero frame: the establishing shot with main character + key elements
2. Character ref portraits: close-up of each character (man, woman, animal, etc.)
3. Background ref: the setting without characters

Add to negative prompts for character refs to exclude wrong subjects (e.g., "woman, female" when generating man portrait).

Hero Frame Workflow Template

{
  "1": { "class_type": "CheckpointLoaderSimple", "inputs": { "ckpt_name": "redcraftRedzimageUpdatedJAN30_redzibDX1.safetensors" }},
  "2": { "class_type": "CLIPTextEncode", "inputs": { "clip": ["1", 1], "text": "<hero_prompt>" }, "_meta": { "title": "Positive" }},
  "3": { "class_type": "CLIPTextEncode", "inputs": { "clip": ["1", 1], "text": "3D, ai generated, semi realistic, illustrated, drawing, comic, digital painting, 3D model, blender, video game screenshot, render, smooth textures, CGI, text, writing, subtitle, watermark, logo, blurry, low quality, jpeg artifacts, grainy" }, "_meta": { "title": "Negative" }},
  "4": { "class_type": "EmptyLatentImage", "inputs": { "width": 832, "height": 1472, "batch_size": 1 }},
  "5": { "class_type": "KSampler", "inputs": {
    "model": ["1", 0], "positive": ["2", 0], "negative": ["3", 0], "latent_image": ["4", 0],
    "seed": 42, "steps": 10, "cfg": 1, "sampler_name": "euler", "scheduler": "simple", "denoise": 1
  }},
  "6": { "class_type": "VAEDecode", "inputs": { "samples": ["5", 0], "vae": ["1", 2] }},
  "7": { "class_type": "SaveImage", "inputs": { "images": ["6", 0], "filename_prefix": "director_hero" }}
}

Queue hero + all refs while Z-Image checkpoint is loaded (same checkpoint, different prompts).

Phase 3: Hero Review

Show hero frame and all character refs. User approves or requests regeneration with new seed.

Phase 4: Edit Chain — Qwen Image Edit

CRITICAL: Consistency Rules for Edit Prompts

1. Always explicitly anchor clothing. "The man wears his grey t-shirt" in EVERY prompt
2. Always state what doesn't change. "Same bedroom, same warm lighting, same clothing"
3. Use strong emotion words. "extremely shocked and startled" >> "surprised"
4. Include proportionality. "Her head and body should be proportional and natural looking"
5. Prevent head enlargement. In embrace/close-up poses, Qwen Edit tends to enlarge heads. Add an explicit "do not enlarge her head, keep the same small natural size as in the original image"
6. Describe the transformation, not just the end state