Comfyui Media Workflows

ComfyUI media workflow production

Use this skill to turn a creative or technical brief into a production-ready ComfyUI workflow package. Treat ComfyUI as a graph-based media production runtime, not as a prompt box. Your output should make the graph reproducible, reviewable, safe to run, and easy for another operator or system to execute.

This is provider-independent. Do not assume a specific checkpoint, hosted Comfy service, custom node set, or cloud vendor unless the user or the existing project explicitly names one. Re-check ComfyUI, custom-node, model, license, and cloud API facts at production time because they change frequently.

Operating principles

• Start from the deliverable and constraints, then choose graph topology, models, nodes, and runtime. Do not start by pasting a popular workflow.
• Separate documented facts from production heuristics. Documented facts come from current docs, model cards, licenses, API schemas, and direct tests. Heuristics are your production judgment.
• Keep two workflow artifacts when possible:
  • workflow.json: frontend/save-format graph with layout, grouping, notes, and operator readability.
  • workflow_api.json: API-format graph for programmatic execution.
• Keep enough metadata to recreate a run: workflow JSON, input files with hashes, model identifiers and hashes, custom-node repo revisions, ComfyUI version, parameters, seed policy, hardware/runtime, output paths, and QA notes.
• Prefer fewer custom nodes and fewer model dependencies unless the creative requirement genuinely needs them. Every custom node is code execution and supply-chain risk.
• Never treat model availability as rights clearance. A file being downloadable does not mean the project can use it commercially, use its outputs freely, train on outputs, or use a person's likeness.
• Decide intentionally whether workflow metadata should be embedded in outputs. Embedded graphs aid reproducibility, but may leak prompts, model names, private paths, client references, API node titles, or workflow IP.

Required first pass

Produce a short production brief before building or modifying a graph:

1. Deliverable: media type, count, dimensions/aspect ratio, duration/fps if video, file formats, naming, platform or downstream editor.
2. Creative intent: subject, style, constraints, references, negative constraints, must-preserve elements, brand rules.
3. Inputs: text prompts, images, masks, video, audio, depth/pose/control maps, brand assets, identity/likeness permissions, source ownership.
4. Runtime: local ComfyUI, Comfy Cloud, another hosted Comfy-compatible service, or handoff-only; available GPU/VRAM, concurrency, storage, network and privacy constraints.
5. Model inventory: intended checkpoint/diffusion model, VAE, text encoders, LoRAs, ControlNets, adapters, upscalers, video models, audio/3D models if any.
6. Custom-node inventory: every non-core node package, install source, version/commit, license, dependency impact, trust level, and replacement plan.
7. Rights and safety: commercial status, client/counsel approval needs, likeness/voice consent, minors, regulated domains, impersonation/deception, prohibited content, platform restrictions.
8. Reproducibility target: exact rerun, near-match, or exploratory batch. Exact rerun requires stricter seeds, versions, hashes, and deterministic caveats.
9. Handoff target: another artist, an API service, a cloud deployment, a batch render queue, or an archive.

If any of these are missing, make a bounded assumption only when it does not change the workflow design materially. Otherwise ask.

Workflow formats and API boundaries

ComfyUI workflows are JSON graphs. The frontend save format and API format are not equivalent:

• Save format is for editing in the UI. It includes layout, colors, groups, node sizes, and similar visual metadata.
• API format is for execution. It uses node IDs as keys with class_type, inputs, and optional _meta; it omits most UI layout data.
• For programmatic execution, export from the frontend with File -> Export Workflow (API) or otherwise use a conversion method verified against the target runtime.
• Before submitting an API workflow, validate that each class_type exists in the runtime's node registry and that each exposed input name matches the current node definition.

When you need a static preflight of an API-format workflow, use the bundled Python 3.11+ stdlib helper before runtime validation:

python scripts/inspect_workflow.py workflow_api.json --approved-classes approved_classes.txt --unknown-class warning --pretty

The helper inspects only local JSON. It validates the API-format shape, node IDs, class_type, inputs, optional _meta, link inputs shaped as [node_id, output_index], dangling links, nonnegative integer output indexes, graph cycles, and warnings for secret-like keys/values or absolute/private-looking path strings. It emits a stable normalized JSON inventory with node, edge, class, and finding counts. Exit codes are 0 for no validation errors, 2 for validation errors, and 3 for parse or operational failures. If you provide a newline-delimited approved class list, class types absent from that list can be reported as warnings, errors, or ignored with --unknown-class.

This helper is not a substitute for ComfyUI runtime validation. It never executes the graph, submits to /prompt, installs missing custom nodes, loads models, verifies input names against /object_info, or proves that a workflow will fit VRAM or run on a target server.

For local ComfyUI server execution, the core pattern is:

1. Upload needed inputs through the runtime's upload route or place them in the expected input directory.
2. Submit the API-format graph to the prompt queue.
3. Monitor status through WebSocket messages or polling.
4. Retrieve outputs from the run history or view/download route.
5. Archive run metadata and outputs outside transient runtime directories.

For cloud execution, do not assume local-route semantics. Confirm authentication, endpoint prefixes, concurrency limits, storage behavior, signed URL behavior, timeouts, and partner-node key handling in the current cloud documentation.

The production workflow card

For every non-trivial workflow, maintain a workflow card in the project notes or handoff message. It should be concise but complete:

workflow_name: product-hero-sdxl-controlnet-v003
purpose: "Generate 6 square product hero image candidates with locked silhouette and color palette."
formats:
  save_graph: workflow.json
  api_graph: workflow_api.json
runtime_verified:
  comfyui_version: "record exact version/commit"
  execution_target: "local RTX 4090 / Comfy Cloud / other"
  verified_at: "YYYY-MM-DD"
models:
  - role: checkpoint
    id: "repository-or-file-name"
    file: "model.safetensors"
    sha256: "..."
    source_url: "..."
    license: "..."
    license_verified_at: "YYYY-MM-DD"
    approval: "approved / needs counsel / noncommercial only / unknown"
custom_nodes:
  - package: "repo-or-registry-id"
    version_or_commit: "..."
    source_url: "..."
    license: "..."
    risk: "low / medium / high"
    reason: "why needed"
exposed_inputs:
  positive_prompt_node: "6.inputs.text"
  negative_prompt_node: "7.inputs.text"
  seed_node: "3.inputs.seed"
  width_height_node: "5.inputs.width|height"
seed_policy: "fixed seeds for rerun; randomized only in variants manifest"
input_custody:
  source_assets: "paths, hashes, consent notes"
output_custody:
  destination: "..."
  metadata_policy: "embed workflow / strip workflow / add C2PA externally"
qa_status: "not run / smoke passed / approved / blocked"
known_limits:
  - "..."

Do not invent hashes, versions, license status, or approvals. Mark unknowns plainly.

Model and license inventory

Inventory every model-like artifact, not only the main checkpoint:

• checkpoint or diffusion model
• VAE
• text encoder(s)
• LoRA, LyCORIS, embedding, textual inversion
• ControlNet, T2I adapter, IP/reference adapter, pose/depth/segmentation model
• upscaler, face restoration model, background remover
• video generation model, interpolation model, frame enhancer
• audio, 3D, captioning, or LLM/VLM model used by nodes

For each, record:

• exact file name and folder category expected by the loader node;
• source URL or repository ID;
• version, revision, or download date;
• file hash when available or computed locally;
• model card or license URL;
• license terms relevant to this job: commercial use, attribution, output use, derivatives/fine-tunes, redistribution, synthetic-data/training restrictions, content-use restrictions;
• known runtime requirements: dtype/quantization, minimum VRAM, required text encoders or VAE;
• who approved it and when, or what remains unresolved.

Use model-card metadata as a starting point, not as the full clearance. Hugging Face supports license metadata and model cards, but missing or ambiguous metadata is common enough that you must inspect the linked license or repository files for consequential work. If the model is a derivative or merge, verify upstream license compatibility when the project is commercial, public, client-facing, or redistributed.

Escalate to the client, rights owner, platform owner, or counsel when:

• the license is absent, contradictory, noncommercial, research-only, unclear about outputs, or incompatible with the intended use;
• the model is a merge or derivative with unknown upstreams;
• the workflow uses a person's likeness, a private reference, a living artist/style request, a trademarked character, a brand asset, or regulated-domain content;
• outputs will be sold, advertised, syndicated, used for political persuasion, used in a regulated domain, or used as training data;
• the workflow or custom-node package will be redistributed to others.

Do not provide legal advice. State the risk and the decision needed.

Custom-node risk handling

Custom nodes are extensions that execute code inside the ComfyUI environment. Treat them as supply-chain dependencies.

Risk-rank each custom node:

• Low: official/core node, current registry package from a known publisher, active maintenance, clear license, no network behavior beyond documented model/API calls, pinned version.
• Medium: popular community node with active repo but broad dependencies, unclear release pinning, limited docs, or non-critical network/file behavior.
• High: obscure repo, no license, no pinned release, unreviewed install script, binary wheels from unknown source, broad filesystem/network access, credential handling, code obfuscation, stale issues about breakage/security, or required only for a minor convenience.

Required handling:

• Prefer core nodes and official templates when they satisfy the brief.
• If a custom node is necessary, pin a release or commit and record the install source. Avoid "latest" for production.
• Read the node README and installation requirements; check whether it requires models, external APIs, credentials, ffmpeg, CUDA extensions, compiler toolchains, or system packages.
• Install dependencies only into the intended ComfyUI environment. Do not contaminate system Python or unrelated projects.
• Do not run untrusted install scripts or post-install commands without approval.
• Do not pass secrets through node parameters that can be saved into workflow JSON, output metadata, logs, or screenshots unless the node's docs explicitly require that pattern and the user accepts it.
• When using a workflow from an external source, inspect unknown nodes before executing. A missing-node error is safer than blindly installing everything.

If a required custom node is high-risk and no safe replacement exists, stop and ask for approval with options: accept risk in a sandbox, replace the creative approach, use a hosted provider, or pause.

Graph design patterns

Use these as production patterns, not rigid templates. Node names vary by ComfyUI version, model family, and custom-node package.

Text-to-image stills

Common structure:

1. Load checkpoint or model family-specific model and required text encoders/VAE.
2. Encode positive and negative prompts.
3. Create latent at target dimensions or model-native dimensions.
4. Sample with a documented sampler/scheduler/steps/CFG/seed.
5. Decode.
6. Optional: upscale, color pass, face/detail pass, metadata/provenance pass.
7. Save with deterministic filename prefix.

Production notes:

• Expose positive prompt, negative prompt, seed, width, height, steps, CFG/guidance, sampler/scheduler, denoise, batch size, and output prefix.
• Keep model-native aspect and resolution constraints visible. Do not silently upscale tiny generations to final delivery unless quality permits.
• Use fixed seeds for approval candidates. Use variant manifests for randomized exploration.

Image-to-image and stylization

Common structure:

1. Load source image.
2. Preprocess if needed: resize/crop/pad, color normalize, segment, depth, edges, pose, or reference embedding.
3. Encode image to latent or feed it to the relevant conditioning node.
4. Sample with denoise strength chosen for the preservation target.
5. Composite or restore details.
6. Save output and comparison contact sheet.

Heuristic denoise ranges:

• Low denoise for preservation and cleanup.
• Medium denoise for style transfer while retaining composition.
• High denoise for new image generation loosely inspired by the input.

Document the actual denoise value and expected preservation level. For client assets, preserve composition with explicit control inputs rather than hoping a prompt will retain it.

Inpaint and outpaint

Common structure:

1. Load source image and mask.
2. Verify mask polarity and edge softness.
3. Crop to masked region if using a detail/inpaint crop path, or run full-frame if global consistency matters.
4. Encode image/mask to the inpaint model or conditioning path.
5. Sample with a preservation-appropriate denoise.
6. Composite back into original image.
7. QA seam quality, geometry, shadows, texture, and brand/product fidelity.

Production notes:

• Store the mask as a source artifact with hash; do not rely on a UI-only mask that cannot be handed off.
• Include a "mask check" output or screenshot before long runs.
• If the target is a product, logo, face, or medical/legal/technical object, use stricter review and ask whether AI reconstruction is allowed.

Control and conditioning

Conditioning turns ambiguous prompt intent into controllable structure. Use it when layout, pose, depth, product silhouette, edge fidelity, camera angle, or identity-adjacent consistency matters.

Common controls:

• edge/lineart/canny for shape and hard contours;
• depth for spatial layout and camera perspective;
• pose/openpose-like skeletons for body positioning;
• segmentation for object regions;
• reference/image adapters for color, style, or identity-adjacent visual cues;
• LoRA or embeddings for repeatable style, subject, or product-family treatment.

Production notes:

• Record preprocessing model and parameters, not only the final control image.
• Prefer one strong control over many weak, conflicting controls.
• If using multiple controls, define priority: structure first, identity/reference second, style last, unless the brief says otherwise.
• QA control overfitting: outputs can look traced, rigid, distorted, or texture-poor when control strength is too high.

Upscale and finishing

Common structure:

1. Generate at model-appropriate base size.
2. Choose latent upscale or pixel upscale based on artifact type.
3. Apply detail pass only when it improves the image rather than hallucinating new detail.
4. Save original, upscaled, and final files with linked metadata.

Production notes:

• Upscaling is not a substitute for correct base composition.
• Avoid face/detail restoration on people without checking likeness and consent implications.
• For print, inspect at final physical size and resolution. For web/social, inspect compression and crop safe areas.

Video and motion workflows

Video workflows multiply every risk: VRAM, time, temporal artifacts, rights, and file custody.

Common structure:

1. Define motion spec: duration, fps, aspect ratio, camera/subject motion, source image or text prompt, loop/continuity needs.
2. Load video model and required encoders/VAE/motion modules.
3. Provide source image/video/control frames as needed.
4. Generate short clips before long clips.
5. Optional: interpolate, upscale, stabilize, restore, caption, or composite externally.
6. Save frames and final video with run metadata.

Production notes:

• Smoke-test at short duration, low resolution, and fixed seed before full-resolution runs.
• Lock fps, frame count, resolution, and seed policy. A "5 second" target is not enough; record exact frames and fps.
• Save intermediate frames when the downstream edit may need repairs or temporal QA.
• Watch for temporal identity drift, flicker, texture boiling, camera jumps, text/logo distortion, and motion that contradicts physics or brand requirements.
• For source footage, confirm rights to transform it and whether AI alteration must be disclosed.

Batch variants

Batching is for controlled exploration, not chaos.

Use a variants manifest:

base_workflow: workflow_api.json
locked:
  checkpoint: "..."
  dimensions: "1024x1024"
  negative_prompt: "..."
  control_image_sha256: "..."
variants:
  - id: A01
    seed: 101
    prompt_delta: "warmer backlight, softer shadows"
  - id: A02
    seed: 102
    prompt_delta: "cooler palette, sharper product edge"

Change one or two variables per batch unless the purpose is broad exploration. Name outputs with variant IDs and seed. Generate contact sheets or review grids for human selection.

Prompt, seed, and parameter control

Expose prompts as inputs, but do not leave them as the only control surface.

Prompt handling:

• Split prompt intent into subject, composition, medium/style, lighting, camera/lens, material/texture, constraints, and brand terms when the model responds well to such structure.
• Keep negative prompts focused on actual failure modes. Giant negative lists are hard to debug.
• Record prompt versions. A one-word change can alter approval.
• Remove private client names, unreleased product codenames, personal data, or secret URLs from prompts if outputs or workflow JSON will be shared.

Seed handling:

• For approval, use fixed seeds and record them.
• For exploration, define a seed range or list and record the generator policy.
• If exact reproducibility matters, state that seeds alone are insufficient; model files, node versions, sampler, scheduler, precision, hardware/backend, and preprocessing must also match.

Parameter handling:

• Record sampler/scheduler, steps, CFG/guidance, denoise, dimensions, batch size, model weights, LoRA strengths, control strengths, start/end percentages, upscale scale, fps/frame count, and filename prefix.
• Do not silently leave values at defaults when they materially affect the result. If using defaults, say so.