Pneuma Clipcraft

ClipCraft

In script examples, <SKILL_DIR> means the actual directory containing this loaded SKILL.md. Substitute its full path and keep shell paths quoted. The runtime installs it under .claude/skills for Claude Code, .agents/skills for Codex, or .kimi-code/skills for Kimi; use the path given in your instructions.

ClipCraft is a video-production mode where the source of truth is a structured domain model, not a file. The in-memory model is an event-sourced craft store from @pneuma-craft: an Asset registry, a Composition with Tracks and Clips, and a Provenance DAG that tracks how each asset was generated (and from what). The file project.json at the workspace root is a projection of that store — when you edit it with Write/Edit, the viewer auto-re-hydrates. No reload, no refresh signal.

ClipCraft is built for AIGC workflows: assets are generated, not uploaded. You orchestrate image / video / TTS / BGM generation by running bundled scripts, then record the lineage in project.json.

Working with the viewer

The viewer is the exploded 3D timeline rendering of project.json. It is the user's source of truth for what's currently selected, what the playhead is on, and what they're pointing at. Four channels link the three actors (you, the user, the viewer):

Reading what the user sees

Every user message arrives wrapped in <viewer-context> and (if the user just clicked / dragged / seeked) <user-actions>. Read them before you act. Typical clipcraft payloads:

• <viewer-context> — selectedClipId, selectedAssetId, selectedTrackId, playheadTime (seconds), composition.duration, the active asset's metadata. Use this to disambiguate a vague request like "try another take" — there's almost always a clip selected that tells you which one.
• <preview-frames> (nested inside <viewer-context>) — summary of the planning layer: total="N" attribute + per-track <track id="..." name="..." count="..." /> lines for tracks that carry preview frames. When total="0" (or the tag is absent), the timeline has no planning layer yet — a fresh project. Use this to decide whether to start with sketch overlay (see references/storyboard-workflow.md) vs jumping straight to generation.
• <user-actions> — recent UI events: playhead:seek ({time}), clip:select ({clipId, trackId}), asset:select ({assetId}), clip:drag ({clipId, startTime, trackId}), track:toggle-mute, track:toggle-visible. Treat these as hints, not commands — the user usually expects you to read them and act, not echo them back.

If both are absent (cold start, command-button click), inspect project.json directly and ask if intent is ambiguous.

Locator cards

After creating or editing assets, clips, or moving the playhead, embed <viewer-locator> cards so the user can jump straight to the change. Emit one card per distinct thing you changed — a newly generated asset, a clip you just placed, a time beat you built around — not one per response. The user sees these as clickable chips in chat. Use short concrete labels — "新的 VO 开场", "panda clip on Main", "3.5s — punchline beat" — not generic ones like "see asset".

ViewerAddress — naming an object in the preview

A <viewer-locator> carries an address — a small JSON object that names one referent inside the viewer. clipcraft's address vocabulary is timeline-oriented; every key below is coarse (each one stands alone and drives navigate-then-flash — clipcraft has no element-level fine handle):

Supply exactly one key per address. clipId / previewFrameId / assetId / trackId are stable ids (survive move / re-pace), so locator references stay valid; time is a raw second.

<!-- assetId — scrolls the asset library to the asset and flashes it. -->
<viewer-locator address='{"assetId":"asset-vo-tagline"}'>新的 VO 开场</viewer-locator>

<!-- clipId — selects the clip on the timeline AND seeks the playhead
     to its startTime, so the user lands on the frame you mean. -->
<viewer-locator address='{"clipId":"clip-shot1-spark"}'>panda clip on Main</viewer-locator>

<!-- time — seeks the playhead only (no selection change). Use for
     pure "go look at this beat" pointers. -->
<viewer-locator address='{"time":3.5}'>3.5s — punchline beat</viewer-locator>

<!-- trackId — scrolls and flashes a track header. Use when the
     change is track-level (mute/solo, reordered, new track). -->
<viewer-locator address='{"trackId":"track-narration"}'>narration track</viewer-locator>

<!-- previewFrameId — selects the referenced image asset, seeks the
     playhead to the preview frame's anchor time, and flashes the
     strip thumbnail on the timeline. Use when pointing at a sketch
     or anchor on the planning layer (e.g. "I generated 3 sketches
     for the opening; here's panel 4"). The preview frame's id is
     stable across move/rebind, so locator references survive
     re-pacing. -->
<viewer-locator address='{"previewFrameId":"pf-04"}'>panel 4 — opening sketch</viewer-locator>

Viewer commands (user → agent)

The viewer toolbar exposes six command buttons. Clicks arrive as short natural-language messages in chat — they're hints about what the user wants next, usually with a clip or scene pre-selected, not rigid tool calls. Read <viewer-context> to figure out the target, then run the matching workflow. If intent is ambiguous (e.g. a vague "generate video"), confirm before spending money on veo3.1.

Agent → viewer actions (HTTP)

When you need to drive the viewer (not just respond), POST to $PNEUMA_API/api/viewer/action. Reach for this when the user asks "show me the part where..." and you want the playhead to land there before you explain, or when you've just registered an asset and want it pre-selected for the next take.

# Seek the playhead to a specific second.
curl -s -X POST "$PNEUMA_API/api/viewer/action" \
  -H 'content-type: application/json' \
  -d '{"action":"playhead:seek","payload":{"time":4.2}}'

# Select a clip on the timeline (also seeks to its start).
curl -s -X POST "$PNEUMA_API/api/viewer/action" \
  -H 'content-type: application/json' \
  -d '{"action":"clip:select","payload":{"clipId":"clip-shot1-spark"}}'

Prefer <viewer-locator> cards when the user benefits from a clickable hand-off. Use HTTP actions when you need the viewer state to change before the next step (e.g. taking a screenshot via /api/native/screenshot).

The 6-layer technique stack

ClipCraft is more than a thin wrapper over generation APIs. The mode encapsulates a body of techniques — a way of thinking about AIGC video production — organized in six layers. When a brief lands, scan top-down and figure out which layers it touches; drill into those references and skip the rest. A one-shot 4s clip is Layer 6 only. A music video exercises all six.

Decision tree by brief shape

When in doubt — Layer 1 first

The single biggest failure mode in multi-shot AIGC video work is drift across shots: the character's face changes, the location mutates, the palette wanders. Layer 1 is the antidote, and it's cheap (~$0.50–$2 in upstream image generations) compared to the cost of regenerating a Path A run because the protagonist looks different in panel 5. If a brief has any recurring character, location, or signature prop, build the bible before the storyboard.

Layers are not strictly sequential

Plan top-down (1 → 2 → 3 → 4) but execute bottom-up where appropriate: you might generate a quick test sketch (Layer 6) to calibrate the bible (Layer 1), then redesign. Provenance (Layer 5) is recorded continuously, not in a phase. Direction Notation (Layer 3) is consulted whenever you write a prompt at any other layer.

Domain vocabulary (2-minute version)

• Asset — an addressable piece of media. Has id, type, uri, name, metadata, status, createdAt.
• Track — a horizontal lane in the timeline. type is video / audio / subtitle.
• Clip — a span on a track that references an asset via assetId. Has startTime, duration, inPoint, outPoint (all in seconds). Subtitle clips also carry text directly.
• Scene — a logical chunk of the composition that groups clips across tracks. Purely a human organization aid.
• Provenance edge — { toAssetId, fromAssetId, operation }. Captures "how was this asset created". fromAssetId: null means generated from nothing; a real id means derived from another asset.
• Composition — the top-level container: settings (width/height/fps), tracks, transitions, duration.

Full schema in references/project-json.md. Id rules in references/asset-ids.md.

Making creative decisions

Most of the work in ClipCraft is judgment, not mechanics. Before writing a generation prompt, choosing shots, picking music, or deciding what to cut, read references/craft.md — a field guide to short-video craft. It's principles, not procedures. Reach for it when the brief is open ("make something about X"), when a generated clip feels close but wrong, or any time you're about to settle for a generic answer. The rest of this document tells you how to produce; craft.md tells you what is worth producing.

Generation scripts

Seven CLI scripts wrap the provider APIs. Call them via the Bash tool.

generate_image.mjs and edit_image.mjs share their output shape — a JSON object on stdout with files, urls, and description. They take --output-dir + --filename-prefix (NOT --output). The prompt is a positional argument, not a flag. The other four follow the older flag-based convention where --output <path> is required and stdout is just the output path.

All scripts read their API keys from process.env or from a .env file in the skill directory.

Why GPT Image 2.5 matters for video work

The shared image model is gpt-image-2.5-sunburst via OpenRouter; gpt-image-2.5-flare is selected automatically for editing and calls with reference images. Use it for:

• First / last frames. Seedance's from-image and first-last-frame video modes inherit the quality of their anchor images. GPT Image 2.5 holds a specific aesthetic, character, and composition across paired calls — so the two frames actually look like they belong to the same shot, and the interpolated video doesn't need to fight a stylistic mismatch at the seams.
• Complex single-frame compositions — foreground subject + environment + overlay text all in one image, rendered legibly. Title cards, end cards, lower thirds, memes with baked-in captions, data callouts over b-roll, diagrammed explainers — all viable as standalone assets rather than ffmpeg/post overlays.
• Text rendering that actually reads. "A sign that says X" or "a poster with the headline Y" comes back legible, not glyph soup. Use it for signage, lower-third strap text, brand marks, chyron-style overlays.
• Multi-reference stitching via --image-urls. Pass a character portrait plus an environment plate plus a style plate and GPT Image 2.5 composes them coherently — a stronger control surface for prompting first frames and character sheets than free-text alone.
• Reference-guided edits. Pass the source via --image-urls and describe what to change and preserve. edit_image.mjs --annotation accepts a visual location guide. OpenRouter does not expose a pixel-mask parameter.

Be ambitious with the creative brief: text-heavy frames, multi-layer compositions, and explicit character continuity are viable in a single generation. See references/craft.md for the principles that should drive those choices.