Mm3 Captioning

MiniMax-Music3 Captioning

MiniMax-Music3 is the second local generation backend (alongside the native ACE-Step LM→DiT→VAE pipeline) — see engine/src/minimax/mm3-*.h, engine/tools/convert-mm3.py, server/src/services/backends/types.ts, ui/src/stores/backendStore.ts. Unlike ACE-Step's caption field, MM3 was trained on a specific three-section Structured Caption format, and adherence to that format is the main lever for controlling the output (see "Empirical context" below).

This skill vendors MiniMax's own official caption-authoring skill (upstream/, fetched verbatim from MiniMax-AI/MiniMax-Music3) and summarizes it for our agents. Read upstream/SKILL.md in full before writing or reviewing any MM3 caption-assembly code — this file is a map and a set of house notes, not a replacement.

When to use this skill

• Formatting a caption/prompt string for an MM3 generation request (engine mm3-* backend, or the instructions field the reference HTTP API expects).
• Building or reviewing the MM3 prompt-assembly / request-translator increment — the code that turns Lyric Studio / UI form fields into the text MM3's tokenizer wraps in <|caption_start|>...<|caption_end|>.
• Wiring Lyric Studio (server/src/services/lireek/) output toward an MM3-targeted caption instead of (or alongside) the ACE-Step caption it already produces.
• Debugging MM3 output that drifts genre/mood across seeds, ignores section-local instructions, or otherwise seems to be "not listening" to the caption.

The Structured Caption contract (from upstream/SKILL.md)

MM3 captions have exactly three top-level sections, in this order, as plain text labels (not markdown headings — see "Skill vs. pipeline-template discrepancy" below):

1. Global Metadata — genre/subgenre, tempo (exact BPM only if explicit/strongly justified, else a range or qualitative tempo), key/scale (only if explicit or musically useful), global emotional progression, sonics/production profile.
2. Vocal Details — for vocal music: lead configuration, timbre, register, delivery, harmony/backing vocals, restrained vocal FX. For instrumental music: state it's instrumental and name the instrument/texture carrying the lead line. Never invent lyrical subject matter or reproduce lyrics here.
3. Arrangement — section-by-section timeline: what enters/exits/changes/ intensifies per section, instrument lifecycle (primary/secondary), groove development, transitions, embellishments/spatial FX. ~250–450 English words by default.

Other hard rules worth internalizing:

• 5,000-token limit on the tokenized text prompt (README limitation) — the caption competes with lyrics for that budget; don't let Arrangement balloon.
• Lyrics stay out of the caption. Bracketed section tags ([Verse], [Chorus], [Bridge], [Instrumental], …) in the lyric text act as musical directives for that section's local arrangement — they must be honored in the Arrangement section — but the lyric words themselves are never quoted, paraphrased, or summarized into the caption.
• Explicit instrumental requests must stay instrumental — never add vocals to fill in an unspecified case; be conservative instead.
• Never fabricate precision: no invented exact BPM/key/production technique when the user's brief only supports a broader description.
• Section-local directives (tag-level) can change that section's local arrangement but must never silently override a global exclusion or an explicit top-level constraint (vocal gender, instrumental requirement, tempo limit, required/prohibited instrument).

Navigating upstream/references/ (genre router → family index → template)

This is a progressive-disclosure retrieval system — do not scan all 1,000 templates. Three layers:

1. upstream/references/genre-router.md — entry point. Maps genre/mood/cultural cues (with a CN/EN alias table) to one of 18 style families, e.g. east-asian-modern, hip-hop-rap, metal-heavy-rock, cinematic-orchestral-epic, contemporary-folk-acoustic, general-pop-ballad (fallback when only mood/imagery is given). Read this first, pick at most one primary + one secondary family.
2. upstream/references/index-<family>.md (18 files) — compact style cards per family. Read only the 1–2 files the router pointed at.
3. upstream/templates/<slug>_NNNN.txt (1,000 files) — full example captions in the exact target format. Select up to three by distinct role — Foundation (overall identity/groove), Modifier (one specific requested dimension: secondary genre, vocal character, cultural color, production texture), Arrangement (section timeline/energy-contour logic only) — then synthesize a new caption; never copy a template's sentences, exact key/BPM, or full section order verbatim.

Example templates worth opening as calibration references (plain-text label format, not markdown):

• upstream/templates/acoustic-blues-folk_0001.txt — sparse solo-instrument arrangement, good minimal-instrumentation example.
• upstream/templates/index-east-asian-modern.md cards → templates therein, if MM3 output needs to match HOT-Step's existing East-Asian-heavy adapter/dataset mix.
• Any index-hip-hop-rap.md or index-metal-heavy-rock.md card, for genres where our current ACE-Step captions already lean on strong groove/production language — good starting point for a side-by-side format comparison.

Skill vs. pipeline-template discrepancy (the request-translator seam)

Cross-checked upstream/SKILL.md's Output Contract against D:\Ace-Step-Latest\mm3-weights\fixtures\tok_prompt_template.txt, the literal prompt-assembly template the reference pipeline builds:

<|im_start|><|caption_start|>Energetic synthwave with driving bass, retro drums, and soaring lead synths. 120 BPM, A minor.<|caption_end|><|lyrics_start|>[start]
[verse]
Neon lights across the bay
...
<|lyrics_end|><|im_end|><|audio_start|>

Findings:

• upstream/SKILL.md presents the three required sections as markdown ### headings in its own instructions (### Global Metadata, etc.), which could be misread as "the output caption text should literally contain ### markdown". The actual reference templates (e.g. templates/acoustic-blues-folk_0001.txt) use plain text section-name lines (Global Metadata / Vocal Details / Arrangement, no #, no blank-line separation requirement) — confirmed by reading a template directly. The request-translator should emit plain-text labels, not markdown, to match what MM3 was actually trained/templated on.
• tok_prompt_template.txt's <|caption_start|>...<|caption_end|> slot is a fully opaque string — the tokenizer doesn't parse or require internal structure, it just wraps whatever text is handed to it. The fixture's own example caption is a minimal one-liner ("Energetic synthwave... 120 BPM, A minor."), not a three-section Structured Caption — i.e. the reference pipeline was smoke-tested with minimal captions, not the skill's prescribed structured form. This is consistent with the skill's own README calling the plain description "used directly" as one valid mode, with the Structured Caption as the richer, optional upgrade path — both are legal inputs to the same <|caption_start|> slot.
• Net implication for the request-translator increment: build the caption as plain-text section labels (matching template convention) when emitting a Structured Caption, keep it under the shared 5,000-token budget alongside lyrics, and don't assume the pipeline needs or wants markdown syntax anywhere in the string it hands to <|caption_start|>.

Empirical context (measured 2026-08-13)

Minimal one-line captions (the fixture's own smoke-test style) produce high take-variance across seeds — genre drift for MM3: the same short prompt lands in noticeably different genre/mood territory seed to seed. Detailed Structured Captions (the three-section form this skill describes) are the adherence lever — more explicit Global Metadata / Vocal Details / Arrangement content reduces that drift. This directly motivates building the request-translator to always emit a full Structured Caption rather than passing a short user-typed description straight through to <|caption_start|>.

DIALECT beats format: the prose decides the genre, not the label (2026-08-21)

The format is necessary and not sufficient. A caption can pass validateMm3Caption cleanly, name a specific genre, and still render a different one — because MM3 reads the ~560 words of description, not the two words after "scale is minor.".

Worked case: a caption whose Basic Attributes line ended Hardcore Punk. rendered as southern rock on every seed. Grepping its distinctive vocabulary against the 1,000 templates says why:

And the corpus's own punk/hardcore templates say the opposite on every axis: "heavily distorted" not mild distortion; "wide soundstage, panned hard L/R, wall of sound" not "tight midrange focus"; "heavily compressed, modern rock radio" not "live-room honesty"; "clear youthful tenor with a nasal edge" not "chest-voice baritone"; "palm-muted chugging / power chords" not "ringing open-chord texture".

Two rules follow, both checkable:

1. Write in the target family's dialect — grep the templates for the words you are about to use and confirm they cluster in the right family. "Authentic / raw / garage / minimal polish / live-room" is roots-rock vocabulary in this corpus no matter what genre the label claims.
2. The genre clause is a SLASH PAIR. Hardcore Punk. appears in none of the 1,000 templates. Every punk/hardcore entry is paired — Pop Punk / Alternative Rock, Metalcore / Post-Hardcore, Alternative Rock / Post-Hardcore, J-Rock / Pop Punk — and the corpus-wide mode is two slash-joined genres. An unpaired, unattested term is a genre the model was never taught.

This supersedes nothing above; it is the layer under "THE GENRE MUST BE SPECIFIC" in MM3_CAPTION_SYSTEM_PROMPT. Specific and attested and consistent with the prose. The structural fix is retrieval — few-shot the caption writer with 2-3 real templates from the routed family — designed in docs/plans/2026-08-21-mm3-prompt-translator.md.

Confound to control first: the caption is consumed ONLY by the LM (the flow DiT's uncond branch is zeros_like(condition), not an empty prompt — engine/src/minimax/mm3-dit-graph.h:459). Never judge caption adherence on a quantised LM; q8_0/NVFP4/MXFP4 degrade exactly the stage being measured.

The format is MANDATORY, not preferred (ear-verified 2026-08-14)

A controlled A/B settled this for training data. One track (the album-A artist, albumA), 30 s, identical lyrics, 5 seeds per arm, f16/f16, no adapter:

• Arm A — a good ACE-style caption verbatim: 219 words of Gemini descriptive prose already covering groove, per-instrument detail, timbre, mix AND arrangement-over-time. Not tag prose; genuinely rich.
• Arm B — the SAME content restructured into the three sections, 503 words, Basic Attributes synthesised from the sidecar's bpm/key/signature.

Rob's verdict by ear: B better "by a massive margin"; most A takes were not even the right genre (1 of 5 was), while B was on-genre and sounded far better.

Consequences, both load-bearing:

1. Rich descriptive prose is NOT a substitute for the format. Arm A had most of the CONTENT and still produced wrong-genre output. So restructuring an existing caption corpus is required work before it can condition MM3 — for generation and for building a training conditioning cache alike.
2. Lead the Arrangement with whatever OPENS the track. B lost a piano intro that A reproduced in 4 of 5 takes. The piano was present in B, but as a subordinate clause two-thirds into a 503-word caption ("Primary: Distorted electric guitars carry the harmonic weight… A clean, bright piano opens the piece alone"). Put the opening instrument first in Instrument Lifecycle, and name it in Global Metadata.

Do not judge this class of change by spectral proxies. In the same run, flatness (0.080 → 0.200) and centroid (1847 → 3187 Hz) both rose sharply, which reads as "noisier/harsher" — and by ear that was simply the genre arriving (crash cymbals and wall-of-sound distortion ARE spectrally flat and bright). An intro_ratio (opening RMS ÷ body RMS) also favoured B, while B was the arm with no piano at all: it measures a dynamic envelope, not instrumentation. Cross-seed spread rose for B rather than falling, contradicting the "informative caption → tighter clustering" hypothesis. Every proxy either misled or measured something adjacent. Ears decided it; keep it that way.

Artifacts: M:\HOT-Step-CPP\_experiments\caption-ab-2026-08-14\ (10 WAVs, both captions, results.json, and the runner).

RESOLVED 2026-08-15: caption from the AUDIO with ace-caption --mode mm3

The two sections below describe a problem that is now solved. Read them for the traps, not for the recommendation.

MOSS-Music-8B, ported natively by the other agent, writes MM3 Structured Captions from the audio. ace-caption --models <gguf dir> --src-audio <file> --mode mm3 --ffmpeg <path>. Rob's ear test — same track, same lyrics, same 5 seeds, both arms declaring IDENTICAL Basic Attributes so the only variable was the prose:

Two of five seeds landed on the target artist with NO ADAPTER LOADED, from the caption alone. Caption quality is that dominant.

Two things to carry forward:

1. Override Basic Attributes with Essentia. MOSS is unreliable on the two facts the sidecar already knows exactly — on 03-burn it said ~102 BPM / C# minor where Essentia has 90 / E major. Tempo is a documented MOSS blind spot and key behaves the same. engine/tools/mm3-caption-hybrid.py keeps every section MOSS heard and rebuilds only that line from bpm/keyscale/timesignature.
2. The genre word must be SPECIFIC. A bug in the hybrid collapsed every track to the umbrella "Rock", which is precisely the failure Rob described in the losing arm. Prefer "Pop-Punk"/"Punk Rock"/"Emo" over "Rock", and search every source that observed the track (MOSS's body, the Gemini caption), most specific first — not just MOSS's own Basic Attributes line.

mm3-caption-restructure.py is superseded and carries a header saying so.