Community Model Conversion

Community model conversion (HF fine-tune -> sherpa-onnx -> catalog entry)

This pipeline turns "user wants language X" into a one-URL import in the app. Every step below was learned from a real conversion (Arabic 2026-08, Swiss German 2026-08-24, GH #63). The gotchas are the point of this file: each one cost hours because the failure mode is silent (models load but transcribe garbage, or decode to empty strings).

0. Triage the user's repo FIRST (before any conversion work)

If the user points at onnx-community/* or any transformers.js-layout export (onnx/encoder_model*.onnx + tokenizer.json, inputs named input_features / last_hidden_state): STOP. It is structurally incompatible with sherpa-onnx (no n_mels/model_type metadata, wrong tensor names, merged decoder KV cache). Our app's rejection message is correct; relaxing validation would only turn it into a native exit 255. Verify cheaply with the eval harness: OfflineRecognizer.from_whisper(...) fails at InitEncoder: 'n_mels' does not exist in the metadata. The fix is a re-export from the fine-tuned PyTorch checkpoint, not the ONNX files.

Then find the PyTorch source of the fine-tune (check the HF model card's base_model, search ?search=<model name> on the HF API). Swiss German had Flurin17/whisper-large-v3-turbo-swiss-german behind the onnx-community mirror.

1. Environment (CPU only is fine, ~40 min end to end)

• Workspace on REAL disk (NOT /tmp: it is tmpfs, the fp32 graphs are ~3 GB and eat RAM twice), e.g. /var/tmp/chwork.
• venv: torch+cpu, transformers, onnx, onnxruntime, onnxscript, openai-whisper, safetensors, numpy<2. Python 3.14 + onnx may need pip install -U ml_dtypes or onnx<1.20 (float4 attribute error).
• sherpa-onnx sources at the pinned tag (git clone --branch v1.13.5 ...; the tag equals the .sherpa-version srclib commit) for scripts/whisper/export-onnx.py.
• Validate with the eval harness venv (eval/.venv, sherpa-onnx Python).

2. HF transformers checkpoint -> openai-whisper .pt

The export script consumes openai-whisper models. Write a key remapper (kept at the workspace; see memory checkpoint for the working one): encoder/decoder layer subkeys map (self_attn.q_proj->attn.query, layer_norm1->attn_ln, final_layer_norm->mlp_ln, bare fc1/fc2-> mlp.0/mlp.2, encoder_attn.*->cross_attn.*), drop lm_head (tied), drop k_proj.bias (softmax-invariant: adds q.b, constant across keys).

GOTCHA 1 (the big one): turbo's decoder positional embedding is LEARNED, not sinusoid. Download the official turbo.pt from the URL in whisper/__init__.py and check: decoder.positional_embedding has std ~0.0066 (near-zero learned table), while a sinusoid has std ~0.7. You MUST copy model.decoder.embed_positions.weight (and the encoder one) from the HF checkpoint; injecting a hand-computed sinusoid produces models that load, run, and emit total garbage. Symptom if you get this wrong: weights verified identical, architecture verified identical, output still gibberish.

Wrap the result as {'dims': {...}, 'model_state_dict': sd} with dims read off the shapes (turbo: n_mels 128, n_audio_* 1500/1280/20/32, n_text_* 448/1280/20/4, n_vocab from token_embedding). whisper.load_model infers from dims.

Validate BEFORE exporting: transcribe a short wav with openai-whisper from the converted .pt AND with the original HF pipeline; they must agree. If you see NaN logits, you probably read a 22 kHz int16 wav as float32 (use whisper.load_audio / ffmpeg), not a model bug.

3. sherpa export (export-onnx.py, patched for a local checkpoint)

Local patches needed (keep them in the working copy):

• add the model name to --model choices and to a load_model branch returning whisper.load_model("./<name>.pt");
• the n_mels branch (large/turbo -> 128) must include your name;
• the external-data save branches ("large" in filename) must include your name: the fp32 graphs exceed the 2 GB protobuf limit (FileExistsError on rerun: delete *.weights too, not just *.onnx*). NEVER delete these files while the export is still running: the .weights beside the .onnx is the canonical external data the proto points at, and deleting it mid-run (2026-08-28, "looked like a duplicate") crashes quantization and forces a full re-export. Clean up only after the process exits.

GOTCHA 2a (int8 truncation): per-tensor int8 quantization of a FINE-TUNED whisper decoder causes premature EOT after ~1 phrase per 30s chunk. The fine-tune shifts weight distributions; per-tensor crushes outlier channels and the greedy decoder stops early. Short audio looks correct, 30s chunks yield 1 phrase instead of 15. FIX: pass per_channel=True to quantize_dynamic() for the decoder (ORT docs recommend it when accuracy loss is large). Verified: per-channel int8 = identical output to fp32 at the same size. The encoder is unaffected. MODEL-DEPENDENT (2026-08-27, primeline German): a light fine-tune (3 epochs, lr 1e-6) moved distributions so little that per-tensor survived a 62s tiled test with no truncation (only a token-casing wobble); the Swiss fine-tune truncated at 30s. You cannot predict which kind you have: always quantize the decoder per-channel, and always test with a tiled ~30s audio file (60s is better) before publishing.

GOTCHA 2b: torch >= 2.9 defaults torch.onnx.export to the dynamo exporter, which bakes a wrong cross-KV reshape in the whisper decoder. Symptom: sherpa decodes to EMPTY text and logs Caught exception ... Reshape ... input_shape_size == requested_shape_size was false, Input shape {1,4,1280} requested {1280} (hidden behind the misleading "tail_paddings" message). Fix: pass dynamo=False for the decoder export. The encoder is unaffected.

Quantization (int8, MatMul-only) is produced by the script itself.

Validate: decode the same wav via OfflineRecognizer.from_whisper in eval/.venv; must match the HF ground truth. Use audio >= 10 s or sherpa's "Return an empty result ... input frames" fires (frames must exceed tail_paddings, 1000 default). Load the wav with ffmpeg resampling, not np.frombuffer.