Vllm Metax Model Trim

vLLM-MetaX Model Trimming

Create a reduced model directory that preserves the execution paths under test. Select the output mode from the user's goal:

• Dummy: portable configuration and assets for --load-format dummy smoke tests. Dummy loading skips checkpoint reads but still allocates parameters, runs applicable quantization postprocessing, and consumes GPU memory.
• Real weights: a self-contained checkpoint containing the retained tensors, with their names remapped to the reduced model and a validated non-dummy load. Generate and execute a reproducible extraction script for the specific model and checkpoint format; keep the script with the deliverable. Read the real-weight guide before changing the configuration or copying weights. Do not treat a configuration-only result as a loadable checkpoint.

Neither mode establishes original-model accuracy, routing distributions, or performance.

Establish inputs and coverage

• Determine the source and output directories, output mode, available GPUs/free memory, TP/PP/EP, quantization, context length, and target features from the conversation and existing configurations. For real weights, locate the complete source checkpoint and its format, and verify that it is readable before producing an output. Ask only for missing inputs that affect trimming. Without explicit feature targets, preserve distinct backbone layer types and list MTP/DSpark separately as optional coverage.
• For configuration-only requests, do not allocate GPUs. When functional validation is requested, proceed through service and request tests. Check occupancy if available GPUs are unspecified; do not terminate other workloads to make room.
• Record the actual Python, vLLM, vllm_metax, and Transformers versions and import origins. Inspect model registration, configuration classes, constructors, and dispatch conditions. Workspace source may differ from runtime source. This skill does not require an environment upgrade or automatically invoke the upgrade skills' adaptation audits.
• Existing examples are under tools/batched_test/models/, not a root-level batched_test/. Read a relevant example's config.json, documentation, and launch script first. Avoid reading large tokenizer JSON files to understand architecture. Recheck example paths, environment variables, and historical success records against the current environment.

Design the reduction

Read the relevant sections of the structure guide. For unfamiliar architectures, derive field semantics from their actual consumers rather than applying another model family's field conventions.

1. Inventory each submodel's depth, layer types, attention/cache types, Dense/MoE boundaries, quantization, sharing/references between layers, and optional prediction heads. Create a small table mapping each target path to retained layers/dependencies and a triggering request.
2. Reduce depth first; preserve width, expert counts, head/LoRA dimensions, and quantization where possible. Select the fewest layers that cover the targets and satisfy dependencies. A homogeneous model can start with 1–2 layers. For heterogeneous models, do not blindly keep the first N layers or set every num_hidden_layers field to the same value. Removing the only retained instance of a distinct path, such as Engram, reduces feature coverage; it is not equivalent to removing repeated homogeneous layers. A model that can still generate text does not necessarily preserve all text-model paths. See the structure guide's Engram guidance.
3. Record the original-to-new layer mapping. Update layer arrays, periodic/offset rules, sharing relationships, quantization module paths, MTP indices, and cross-layer references together. Determine which fields the implementation reads; descriptive JSON fields may not control execution. In real-weight mode, apply the same mapping to checkpoint tensor names and verify that retained tensor shapes match the reduced model. A structural mapping alone does not transfer weights.
4. If memory is still insufficient, reduce runtime context, concurrency, and cache budgets before considering fewer experts or narrower dimensions. Check sharding, quantization groups, and kernel constraints before changing shapes. Record the original shapes and paths no longer covered. gpu_memory_utilization sets a budget; it does not shrink parameters, and an overly small budget can prevent startup.

Keep depth and TP decisions separate. Depth-only trimming normally leaves the retained tensors' TP partition constraints unchanged; fewer layers do not require fewer TP ranks, and layer count need not be divisible by TP. Check actual dimensions, quantization, kernels, and loader support for the requested TP. PP layer placement is a separate concern. Distinguish storage shards from rank partitions using the checkpoint parallelism contract. Explain changes to pre-partitioned files as re-sharding for a new TP, not as a consequence of having fewer layers.

For real weights, budget output disk space separately from GPU memory, including large lookup tables and any additional checkpoint formats. If capacity would require dropping a distinct target path, give the concrete size and coverage tradeoff and resolve that scope choice before omitting it, unless already authorized. Do not silently reinterpret “preserve text and image” as permission to remove an expensive text module; consider a larger destination or fewer duplicate output formats first.