Veomni New Model

The hard part of a new transformers-family model — the patchgen config, parallel plan, MoE weight conversion, __init__.py registration, codegen — lives in /veomni-patchgen-model. This skill is the wrapper around it: it decides what you are adding, then hands off, then does the config, trainer and data work that patchgen does not cover.

Before You Start: Create a Plan

Track the phases with whatever todo/plan tool the running agent provides:

Phase 1: Analyze HF model             -> in_progress
Phase 2: Modeling (/veomni-patchgen-model)  -> pending
Phase 3: Write training config         -> pending
Phase 4: Integrate with trainer        -> pending
Phase 5: Test and document             -> pending

Phase 1: Analyze HuggingFace Model

1. Identify the model on HuggingFace. Read its config.json, modeling_*.py, and any processor configs.

2. Determine model category:

   • Text-only LLM -> veomni/models/transformers/<model_name>/
   • Vision-Language -> veomni/models/transformers/<model_name>/ + veomni/data/multimodal/
   • MoE model -> additional veomni/distributed/moe/ integration
   • Diffusion model -> veomni/models/diffusers/<model_name>/

3. Check existing similar models: Find the closest existing model in veomni/models/transformers/ and use it as a reference. E.g., if adding a new Qwen variant, reference qwen3/ or qwen3_vl/.

4. Identify required patches: VeOmni uses a patchgen system (veomni/patchgen/) to generate model patches from the HuggingFace modeling. Check whether a sibling model already has a config you can extend via name_map — that is usually the difference between a 60-line config and a 1000-line one.

5. Compare checkpoint keys against the supported upstream version and any existing VeOmni model. Apply the decision rule below before resolving a mismatch.

Checkpoint key conflicts require a user decision

When upstream model, VeOmni model, or checkpoint parameter keys disagree, show the concrete old/new keys and explain the impact on weight loading, export, and optimizer/DCP resume. Ask the user how to resolve the conflict before implementing a rename, alias, or compatibility mapping. Do not silently retain an obsolete model hierarchy just to preserve checkpoint keys.

If the user has already chosen a resolution in the current task, apply it without asking again. When that choice is to follow current upstream keys, keep those keys in the model and handle approved legacy-key conversion in the checkpoint layer. Verify the chosen direction with strict loading and checkpoint round-trip tests; do not hide mismatches with strict=False.

Phase 2: Modeling — hand off to /veomni-patchgen-model

1. Create the model directory: veomni/models/transformers/<model_name>/.

2. Switch to /veomni-patchgen-model. It owns the whole modeling surface — the <model_name>_{gpu,npu}_patch_gen_config.py files, ExtraParallel parallel_plan.py, any required MoE checkpoint_tensor_converter.py, __init__.py registration, make patchgen, and the model-level test cases — with the working examples and the pitfalls that cost the most time. Do not re-derive it from this file.

   Note that parallel_plan.py is not an FSDP wrapping policy: FSDP2 wraps generically in build_parallelize_model(), and ParallelPlan (veomni/distributed/parallel_plan.py) only describes ExtraParallel sharding, such as expert parallelism or embedding sharding. Add a plan whenever the model uses ExtraParallel, including dense models that shard embeddings; a model without ExtraParallel does not need one.

3. Exception — non-transformers architectures. Diffusion models under veomni/models/diffusers/<model_name>/, and the flux / movqgan / wan directories, have no generated/ output and no patchgen config: they patch through device_patch.py or direct modeling. Copy the closest existing one and skip to Phase 3.

Come back here once the model loads and its registry / patch tests pass.

Phase 3: Write Training Config

1. Model config: Create configs/model_configs/<model_family>/<ModelName>.json matching HuggingFace format.

2. Training config: Create YAML in the appropriate directory:

   • Text: configs/text/<model_name>.yaml
   • Multimodal: configs/multimodal/<model_name>/<model_name>.yaml
   • DiT: configs/dit/<model_name>.yaml

3. Config must include: model path, data config, optimizer settings, parallelism config, checkpoint settings.

4. Verify against existing configs — match the structure of similar model configs.