Evaluator

evaluator

Trigger refinements

ALSO FIRE when the user just dropped a path that ends in .mp4 from .ralphy/workspaces/<ws>/projects/<id>/render/ with no other instructions, or when an editor handed off and the user asks "and now?" (any language).

DO NOT FIRE for unrendered projects (handback to editor for ralphy render), for raw research downloads (those go through researcher's analyze-video flow, not eval), or for source media that hasn't been composed yet.

Hard invariants

• Every model call (vision pass) routes through cli/lib/providers/llm.ts → callLLM() via the CLI. No direct OpenAI / fal calls.
• Findings are deterministic outputs of cli/lib/eval/* — don't paraphrase them; pass through verbatim to the fixer agent.
• Keyframe slicing is a cheap diagnostic, NOT a ship gate (#411). A keyframe-only (or structure-only) report can NEVER mark a Unit ship-ready — report.gate.shipReady is hard-false on it. The final gate before forming/publishing a Unit is the native-video pass (full mp4 → model), or deep-style when a STYLE_LOCK/brief exists. Screenshot slicing misses temporal continuity, audio-picture alignment, pacing, and caption sync — exactly the failures that hallucinate when the model only sees stills.

Where this sits in the Unit lifecycle. Eval is phase 14 of the canonical Unit lifecycle. Its native-video gate is the one that flips polished to true in ralphy project status <id> --contract; the quality-gate-failed and native-gate-required stop conditions are derived from eval.json's gate / scoring.verdict. A block verdict feeds the repair loop (phase 15) and the optional polish council (phase 16).

You evaluate rendered UGC videos and produce a report that another agent (the fixer) can act on without reading the video itself. The contract is: the report is the handoff.

What this skill is not

• Not a researcher tool. For "analyze this TikTok/Reel from a creator I want to imitate", route to /researcher.
• Not a fixer. The findings list is meant to be read by a separate agent (or the editor / art-director / scenarist) that will execute the fixes. Don't try to fix issues from inside this skill — that's a different role and would skip the user's chance to triage.
• Not a publisher / scheduler. Verdict is informational, not a publish gate.

The single command

ralphy eval video <path-to-mp4>

Auto-detects the project ID when the mp4 lives at .ralphy/workspaces/<ws>/projects/<id>/render/... (the current layout — fixed in #411; the legacy workspace/projects/<id>/ shape still resolves as a fallback). If detected, the report incorporates scenario.json, captions.json, BRIEF.md, STYLE_LOCK.md, and the template name from the project — these unlock the declared-vs-actual findings (duration drift, hook-zone-thin-vo, intent-drift, etc.) that are otherwise unavailable.

Validation modes (--mode, #411)

Eval has four explicit modes, cheapest → most thorough. Choose by what you're doing: a quick smoke check vs. the final gate before a Unit ships.

ralphy eval video <mp4> --mode native-video      # the final gate before forming/publishing a Unit
ralphy eval video <mp4> --mode keyframe           # cheap diagnostic only — does NOT approve a polished Unit

Default (no --mode) = the final gate. When you omit --mode, eval runs the native-video gate automatically if a model provider is configured (OPENROUTER_API_KEY), and upgrades to deep-style when a project STYLE_LOCK.md / BRIEF.md is discoverable. With NO credentials it falls back to structure and explicitly marks the report not ship-ready (a eval.mode-downgrade info finding records why).

Why keyframe is not enough for a polished Unit: a still never reveals a continuity jump between cuts, a caption that lags the VO, a music hit on the wrong frame, or a draggy hold. Those are exactly the failures the native-video pass catches and the keyframe pass hallucinates around. Use keyframe to triage fast and free; use native-video (or deep-style) as the gate before ralphy unit / publish. The report's gate.shipReady boolean is the authoritative signal — it is hard-false on any non-native report.

Legacy flags (still work, mapped to modes):

• --no-vision ⇒ --mode structure.
• --no-deep-vision ⇒ caps at --mode keyframe (never escalates to the full-mp4 pass).
• --style-sheet / --brief ⇒ implies --mode deep-style.

Deep-style pass (project-specific, anti-generic findings)

When the user asks "validate against my niche / style / creator reference" — or the project carries a style-sheet (typically from ralphy research scrape-profile or ralphy project style-lock) — the deep-style mode scores the full mp4 against every rule in the style sheet's "Vibe & visual register" and "What this creator NEVER does" sections. Trigger it explicitly with --mode deep-style, or just pass --style-sheet (which implies it):

ralphy eval video <mp4> --style-sheet <style-sheet.md> [--brief <BRIEF.md>] [--reference-urls <url> <url> ...]

Both native-video and deep-style produce a structured JSON output at <out-dir>/eval-deep-vision.json (the repair loop, #409, consumes its what_to_redo). It carries:

• overall_verdict — holistic pass/warn/fail
• register_match — declared vs observed cinematographic register, with severity if mismatched
• rule_conformance[] — per-rule pass/warn/fail with verbatim style-sheet quotes and specific timestamp evidence from the rendered video
• brief_conformance[] — same shape, scoring against BRIEF.md intent
• uncanny_mechanism_check — whether the render delivers the style sheet's proprietary aesthetic mechanism or just mimics the surface
• pacing_and_timing — hook / body / closer evaluation
• ai_artifacts[] — concrete timestamp-tagged artifacts the model spotted
• what_works — be honest, what the render did right
• what_to_redo — prioritized 1-6 item fix list with target (start-frame / end-frame / i2v / audio / scene-prompt / model-swap / regen-entire)

Each rule violation also flows into the main findings[] array under style.register-mismatch, style.rule-violation, brief.intent-drift, style.aesthetic-mechanism-missing, or style.timing-* categories so the unified scoring + downstream fixer pipeline pick them up.

When to use deep-style over native-video:

1. The user said "validate against [creator]" / "evaluate against my style" / "is this on-brand for [niche]".
2. The user shows you a scrape-profile style-sheet path and then drops an mp4.
3. The project has a discoverable STYLE_LOCK.md / style-sheet.md (auto-discovered by walking up from the mp4 path). Eval auto-upgrades the default gate to deep-style in that case.

When native-video is the right gate (no style scoring): a generic Unit-readiness check with no creator-style reference — "is this ready to ship", "QA the final cut". This is the default. It still catches temporal/audio/pacing/caption/format failures; it just doesn't score against a specific creator's rules.

When NOT to run the full-mp4 pass at all:

• A fast triage pass mid-iteration — use --mode keyframe (cheap, free-ish) but remember it can't approve a Unit.
• The mp4 is over 40 MB — the model rejects on body size. Re-encode at lower bitrate first.