Add Tts Model

TTS Model Integration Workflow

Overview

HF Reference -> Stage Separation -> Online Serving -> Async Chunk -> CUDA Graph -> Pre-commit/DCO
   (Phase 1)      (Phase 2)          (Phase 3)        (Phase 4)     (Phase 5)      (Phase 6)

Three architecture patterns are supported:

• Two-stage pipeline (e.g. Qwen3-TTS, Fish Speech, CosyVoice3): AR code-predictor → audio decoder, connected via async_chunk for low-latency streaming. Use this for maximum performance.
• Single-stage AR via generator (e.g. MOSS-TTS-Nano): entire model runs inside one AR worker, streaming audio chunks through a per-request inference_stream() generator. Use this when the upstream model bundles AR
  • codec inseparably. See references/single-stage-ar.md.
• Single-stage, vLLM-native base LM + side computation (e.g. VoxCPM2): the base language model runs under vLLM's PagedAttention as a normal AR model; diffusion / VAE / side computations run outside vLLM and are attached via the runner post-processing hook. This is a distinct pattern from the generator approach above — do not confuse the two.

The single-stage variants skip Phase 4 (async_chunk) but Phase 5 (CUDA graph) is still encouraged for the inner AR loop.

Cross-Cutting Invariants

These rules apply to every TTS model regardless of architecture (AR vs AR+diffusion, single-stage vs two-stage, codec-based vs VAE-based). They surface repeatedly across PRs — check them at the end of every phase.

I1. Streaming output contract

Pick exactly one per-step semantics for forward() and document it in the docstring:

• Delta: yield only new audio samples produced this step. Preferred — linear cost, low memory.
• Cumulative: re-decode from step 0 every call. O(N²); only acceptable if the codec has no streaming decode path.

If you choose delta, verify the full emit→consolidate→consume chain:

1. forward() returns {"model_outputs": <new_chunk_only>, ...}
2. _consolidate_multimodal_tensors() in vllm_omni/engine/output_processor.py concatenates the audio key into one tensor at finish. If it skips the key (continue), offline consumers receive only the final chunk. See output_processor.py for the concrete list of handled modality keys.
3. Streaming consumers (SSE, Gradio) receive per-step deltas; offline consumers (engine.generate()) receive a single concatenated tensor.

Cumulative-vs-delta mismatch is the most common silent bug — offline RTF benchmarks pass, but users hear replays or truncation.

I2. Multimodal output consumer hygiene

outputs[0].outputs[0].multimodal_output[<key>] can be any of Tensor, list[Tensor] (pre-consolidation snapshot), np.ndarray, or scalar. When writing tests, examples, and benchmarks:

• Never use dict.get("a") or dict.get("b") on tensor values — Python evaluates the tensor's boolean, raising RuntimeError: Boolean value of Tensor with more than one value is ambiguous. Use explicit if x is None chains.
• Always defensively handle the list form: if isinstance(x, list): x = torch.cat([t.reshape(-1) for t in x], dim=0).
• Assert shape / dtype / duration explicitly; do not rely on truthiness for presence checks.

I3. Hot-loop GPU discipline

Inside any per-step model loop (AR decode, diffusion solver, CFM Euler, vocoder block loop):

• No tensor.item(), .cpu(), or .tolist() — each triggers a GPU→CPU sync; at 10 steps × 60 frames × 4 ops that is 2400 syncs per request.
• Prefer dst.copy_(src) over dst.fill_(src.item()) when writing a scalar tensor into a buffer.
• Prefer torch.compile(Model.forward, fullgraph=False) on the whole forward over per-submodule compile — fewer dispatch boundaries, larger fusion regions. Measure before choosing granularity.
• No Python-side control flow that depends on tensor values; use torch.where / masking instead.

Profile first, optimize second. See the profiling docs / project memory for the trace-analysis workflow.

I4. Validation pyramid

Offline RTF alone is necessary but not sufficient. Every new TTS model must pass all three:

Declaring a model "done" without all three has shipped regressions more than once.

I5. Per-request state is owned by the request, not the model

If the model caches anything across forward() calls (streaming generators, codec buffers, sliding-window pads, CUDA graph state), key it by request ID:

self._state: dict[str, YourState] = {}    # request_key → state
# fetch: request_key = str(info.get("_omni_req_id", "0"))
# free on finish: del self._state[request_key]

A shared buffer silently corrupts audio across concurrent requests — the symptom is crosstalk or truncation only under load.

Phase 1: HuggingFace Reference

Goal: Understand the reference implementation and verify it produces correct audio.

Steps

1. Run the reference model end-to-end using the official HuggingFace / GitHub code
2. Document the architecture:
   • What are the sub-models? (AR decoder, codec decoder, vocoder, etc.)
   • What is the token vocabulary? (semantic codes, RVQ codebooks, special tokens)
   • What is the output format? (sample rate, channels, codec type)
3. Capture reference outputs for comparison during integration
4. Identify the config structure: config.json fields, model_type, sub-model configs

Key Questions

• How many codebooks? What are the codebook sizes?
• What special tokens exist? (<|voice|>, <|audio_start|>, <|im_end|>, etc.)
• What is the token-to-ID mapping for codec codes?
• What is the hop length / frame rate of the codec?
• Does the model support voice cloning? How? (reference audio encoding, speaker embeddings, etc.)

Deliverables

• Working reference script that produces audio
• Architecture diagram / notes
• Token vocabulary mapping
• Reference audio samples for regression testing

Phase 2: Stage Separation (Offline Inference)

Goal: Split the model into vLLM-Omni stages and get offline inference working.

Steps

1. Register the model in vllm_omni/model_executor/models/registry.py
2. Create config classes (configuration_<model>.py) with model_type registration
3. Implement Stage 0 (AR model):
   • Subclass appropriate base (e.g., wrap Qwen3 decoder layers)
   • Implement forward() for autoregressive token generation
   • Handle special token logic (start/stop tokens, codec token mapping)
   • If dual-AR (like Fish Speech), implement Fast AR as a nested module
4. Implement Stage 1 (Decoder):
   • Load codec weights (may need lazy loading from separate checkpoint)
   • Implement forward(): codec codes -> audio waveform
   • Return OmniOutput with multimodal_outputs
5. Define and register the pipeline topology in vllm_omni/model_executor/models/<model>/pipeline.py
6. Create the deploy YAML under vllm_omni/deploy/ for placement, memory sizing, connectors, and runtime overrides
7. Create stage input processor for prompt building and inter-stage handoff
8. Write end2end.py test script

Critical Parameters to Get Right

Debugging Priority (from experience)

When audio output is wrong, check in this order:

1. RoPE / attention: Are position encodings correct? Is the attention mask right?
2. Normalization: RMSNorm epsilon, layer norm placement (pre vs post)
3. Hop length: Product of all upsample rates in the codec decoder
4. Token mapping: Are codec IDs correctly offset from the vocabulary base?
5. Sampling parameters: Temperature, top_k, top_p, repetition_penalty
6. Tensor layout: Codebook-major vs frame-major ordering
7. dtype: Float32 for codec decoders (autocast can corrupt audio)

Streaming Correctness Rules (single-stage and two-stage)

These bugs appear in almost every new TTS PR. Check all before the first push. See also the cross-cutting invariants I1 (output contract) and I5 (per-request state) above — the rules below are the Phase 2-specific instances of those invariants:

• Accumulate codes across AR steps — each forward() appends new codes; do not reset between steps or audio will be truncated (fish speech: fix: accumulate audio_codes across steps)
• Emit delta audio, not full waveform — in streaming mode yield only the new chunk per step, not the re-decoded full waveform from step 0 (fish speech: fix: emit delta audio not full waveform)
• All return paths must emit model_outputs — if any early-return branch skips setting model_outputs, the serving layer silently drops that step's audio (fish speech: fix: ensure ALL return paths emit model_outputs)
• Per-request state isolation — for batched concurrent requests, key all state by request ID; a shared buffer corrupts audio across requests (fish speech: fix: per-request vocode + delta emission)
• Codec tensor device — move codec codes to the codec decoder's device before calling decode; mismatches cause silent CPU fallback or crashes (fish speech: fix: use model device for CUDA stream)
• AR stage max_num_seqs — set to at least 4 in production deploy configs; for single-stage models this is the only stage. For two-stage models, Stage 0 (AR) needs max_num_seqs ≥ 4 to pipeline concurrent requests; Stage 1 (codec decoder) is model-specific and may intentionally use max_num_seqs: 1. Defaulting the AR stage to 1 causes audio gaps under concurrency because the codec window round-robins across requests (RFC #2568)

Optional Dependency Handling

Patch optional dependencies (torchaudio / torchcodec / soundfile) at the top of load_weights(), not at module import. Failures to do so cause cryptic errors only on environments missing the optional package — after the model is already deployed. See references/optional-deps.md for the full pattern, signature constraints, and MOSS-TTS-Nano reference.

Single-Stage AR Pattern (alternative to two-stage)

When the upstream model cannot be cleanly split into an AR stage and a separate decoder, run the full pipeline inside a single AR worker and stream audio through a per-request inference_stream() generator keyed by _omni_req_id. Define one StagePipelineConfig with execution_type=StageExecutionType.LLM_AR, engine_output_type="audio", final_output=True, and owns_tokenizer=True. Set async_chunk: false in the deploy YAML. Only extract params from additional_information that you actually forward, or pre-commit fails ruff F841.

Full walkthrough with the complete forward() / _create_stream_gen() skeleton plus pipeline/deploy definitions: references/single-stage-ar.md. For an in-tree reference, look for any single-stage AR model under vllm_omni/model_executor/models/, such as MOSS-TTS-Nano.

VoxCPM2 is a different pattern and should not reuse this skeleton — it runs the base LM under vLLM PagedAttention with external side-computation. See plan/voxcpm2_native_ar_design.md.

Deliverables

• Model files in vllm_omni/model_executor/models/<model_name>/
• Registered pipeline.py topology
• Deploy YAML under vllm_omni/deploy/
• Working end2end.py at examples/offline_inference/text_to_speech/<model>/end2end.py
• New section in examples/offline_inference/text_to_speech/README.md (table row + per-model section). Do not create a top-level examples/offline_inference/<model>/ dir or a per-model README.md inside text_to_speech/<model>/ — the hub README is the documented surface and the mkdocs generate_examples hook only descends one level into examples/<category>/.

Phase 3: Online Serving

Goal: Expose the model via /v1/audio/speech API endpoint.

Steps

1. Write one adapter under vllm_omni/entrypoints/openai/tts_adapters/. serving_speech.py should not need an edit — detection, stage discovery and dispatch are all derived from what the adapter declares.

   Create vllm_omni/entrypoints/openai/tts_adapters/your_model.py:

   @register_tts_adapter
   class YourModelAdapter(ARTTSAdapter):
       name = "your_model"                              # registry key + log label
       stage_keys = frozenset({"your_stage_key"})       # the deploy yaml's model_stage
   
       def validate(self, request) -> str | None:
           if not request.input or not request.input.strip():
               return "Input text cannot be empty"
           return None
   
       async def build(self, request, sampling_params_list, has_inline_ref_audio):
           params = {"text": [request.input]}
           if request.voice is not None:
               params["voice"] = [request.voice]
           return PreparedRequest(
               prompt={"prompt": request.input},
               tts_params=params,
               model_type=self.name,
           )

   Then add the module to the import block at the bottom of tts_adapters/__init__.py so it registers. That import line is the only shared file a new model touches — which is also why the old rebase-conflict hotspot is gone.

   Pure-diffusion TTS does not go through adapters yet. Under for_diffusion(), create_speech() routes straight to _create_diffusion_speech() and never calls validate()/build(). DiffusionTTSAdapter is scaffolding with no production subclass, so logic placed in one would silently never run. Diffusion-engine models follow the existing diffusion path; wiring it through adapters is open work (#4855).

   If a stage key alone cannot identify the model, declare model_archs = frozenset({"YourModelForConditionalGeneration"}); add arch_identifies_entry_stage = True when the model owns no stage key at all (Ming dense). For a rule that is not set membership, override matches() (see covo_audio.py). For a genuine overlap with another adapter, give one an explicit detect_priority — test_tts_detection.py fails on an unordered overlap. Models that only serve speech in some topologies override stage_serves_speech() (see audex.py).

   Reuse shared helpers via self.ctx.server rather than reimplementing them: _resolve_ref_audio, _apply_uploaded_speaker, _validate_ref_audio_format, _max_instructions_length. Read a comparable adapter first — fish_speech.py (voice cloning), higgs_audio_v3.py (parameter-heavy), moss_tts.py (family sharing a base class).

   Do not add self._tts_model_type == ... branches to serving_speech.py. Older models predate the adapter framework and still have them; they are being migrated out (RFC #4327, #4855). tools/pre_commit/check_tts_adapter.py is a ratchet on the remaining count and fails the commit if it grows. Behaviour that no adapter hook can express is a missing hook — propose it on the RFC.

   Unused variable rule: only extract fields in build() that are actually forwarded to the model. Unused extractions fail ruff F841. For voice-cloning fields (ref_audio -> prompt_audio_path, ref_text -> prompt_text), add them to the params and verify they reach the model call.

2. Handle model-specific parameters:

   • Voice cloning: ref_audio encoding and prompt injection
   • max_new_tokens override in sampling params
   • Model-specific default values

3. Create client scripts: speech_client.py, run_server.sh

4. Test all response formats: wav, mp3, flac, pcm

5. Add Gradio demo: Interactive web UI with streaming support

Voice Cloning Pattern

import base64
from pathlib import Path

def build_voice_clone_prompt(ref_audio_path: str, text: str, codec) -> list:
    """Build prompt with reference audio for voice cloning, called from the adapter."""
    audio_bytes = Path(ref_audio_path).read_bytes()
    codes = codec.encode(audio_bytes)  # Encode on CPU using model's codec (e.g., DAC)
    token_ids = [code + codec.vocab_offset for code in codes.flatten().tolist()]
    return [
        {"role": "system", "content": f"<|voice|>{''.join(chr(t) for t in token_ids)}"},
        {"role": "user", "content": text},
    ]

Test Case Writing (CI Levels)

Follow the vllm-omni-test skill for markers, file naming (test_{slug}.py / test_{slug}_expansion.py), Buildkite wiring, and copy-paste run commands. Also read test_system_overview.md and test_writing_guide.md.

Classify the model's CI priority first (high / medium / low). High-priority TTS models are typically those on the integration hot path or listed in tracking issues such as #1832; medium and low tiers cover the long tail. When unsure, ask the reviewer which tier applies.

Per-level deliverables (TTS / pytest.mark.tts):

L2 & L3 online — same file, dual marks on the baseline smoke: The first / simplest case in test_{slug}.py (default deploy, single non-streaming /v1/audio/speech or equivalent offline path) should carry both @pytest.mark.core_model and @pytest.mark.advanced_model on the same function so it runs in L2 (test-ready.yml, --run-level core_model, basic validation) and L3 (test-merge.yml, --run-level advanced_model, deeper validation) without duplicating the test. In-tree examples: test_voxcpm2_tts.py::test_text_to_audio_001, test_qwen3_tts_customvoice.py::test_text_to_audio_001.

Heavier scenarios in the same file use advanced_model only (streaming, extra languages, concurrency, async_chunk, batch). Example: test_voice_clone_en_streaming_001 → advanced_model only. When migrating L3 to nightly, move those heavier cases into test_{slug}_expansion.py with full_model and drop the dedicated merge job (see test_ming_tts_expansion.py, test_glm_tts_expansion.py).

@pytest.mark.core_model
@pytest.mark.advanced_model
@pytest.mark.tts
@hardware_test(res={"cuda": "L4"}, num_cards=1)
@pytest.mark.parametrize("omni_server", tts_server_params, indirect=True)
def test_voice_clone_en_non_streaming_001(omni_server, online_client) -> None:
    online_client.send_audio_speech_request({...})

L4 consolidation: Prefer parametrized OmniServerParams rows (default, async_chunk, feature flags) in one expansion module rather than many merge-only files (#1832).

L4 performance (high-priority models): Add latency / throughput / stress rows in tests/dfx/perf/tests/test_tts.json, or a dedicated tests/dfx/perf/tests/test_{slug}.json when the model must not join the shared nightly server matrix before integration lands (see VoxCPM2 / Coqui XTTS pattern). Register the model in benchmarks/tts/model_configs.yaml for local bench_tts.py. Wire a separate test-nightly.yml Perf Test step when the JSON is not merged into test_tts.json yet.

Keep model-specific code inside test modules — not tests/helpers/{slug}.py:

• Put MODEL, deploy path, vendored REF_AUDIO_URL, get_prompt(), and inline request_config dicts in each test_{slug}.py, test_{slug}_expansion.py, offline test_{slug}.py, and L1 test_{slug}_*.py as needed.
• Do not add tests/helpers/{slug}.py (or tests/helpers/{model_name}.py) to deduplicate constants or request builders across those files. A little duplication is intentional; follow in-tree references such as tests/e2e/online_serving/test_glm_tts.py and tests/e2e/online_serving/test_cosyvoice3_tts_expansion.py.
• tests/helpers/ is for repo-wide harness code only (mark.py, media.py, runtime.py, stage_config.py, assertions.py, fixtures/). Import those; do not extend the tree with per-model modules.

Runtime send helpers (tests/helpers/runtime.py) — online and offline e2e:

1. Grep runtime.py first — reuse send_omni_request, send_diffusion_request, send_audio_speech_request (online + offline Qwen-style TTS), send_single_stage_tts_request (Coqui XTTS / MOSS-TTS-Nano offline), etc.
2. No matching helper → add send_<feature>_request (or send_<route>_http_request for negative/dfx) in runtime.py with general assert_* bundled inside, then call it from the test.
3. Test file holds request_config dicts only — not omni.generate, not _collect_audio(), not raw HTTP/SDK.
4. Wire tests/helpers/runtime.py into Buildkite source_file_dependencies when you add helpers.

See vllm-omni-test skill § Runtime send helpers for full tables and exceptions.