Megatron GPT to Hybrid Migration

What it does

When your agent uses it

• Converting a GPTModel checkpoint or provider to HybridModel
• Editing a pretrain_gpt.py launch script into its hybrid equivalent
• Mapping GPT layers to a hybrid layer pattern for dense or MoE blocks
• Debugging a migrated training command that behaves unexpectedly

Example prompts

• “Migrate my pretrain_gpt.py launch script for a 32-layer GPT model to the hybrid entrypoint.”
• “Build the hybrid layer pattern for a 32-layer dense model split into 4 pipeline segments.”
• “My converted run starts fine, but I suspect a stale --num-layers value is being overridden, so check the launch command.”

Requirements

• A Megatron-LM checkout that includes docs/user-guide/hybrid-model-migration.md

Workflow steps

5 steps, taken from the first numbered list in SKILL.md.

1. 1Pull the task artifact first: checkpoint metadata, model provider or config,
2. 2Read the canonical migration document completely.
3. 3Follow only the relevant document sections. Do not invent an unsupported
4. 4Validate the result proportionately, invoking the relevant repository build
5. 5Report the outcome and link the canonical document for human readers.

What it can do on your machine

Read from SKILL.md and the folder at commit a3e1f82. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  No scripts in the folder and no shell commands in SKILL.md (its code samples are bash).

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  No URLs in SKILL.md.

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Megatron GPT to Hybrid Migration loads about 1.6k tokens when it runs. Until then it costs about 173 tokens; SKILL.md has 583 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~173

When it runs · the whole SKILL.md, loaded when a task matches

~1.6k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from NVIDIA/Megatron-LM at commit a3e1f82, republished under its Apache-2.0 licence (© NVIDIA). 583 words, ~1,636 tokens.

name

mcore-migrate-gpt-to-hybrid

description

Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel, including the mechanical steps for transferring an existing pretrain_gpt.py launch script. Use when migrating or reviewing a GPTModel checkpoint or training workflow for HybridModel; transferring an existing pretrain_gpt.py script, sbatch, or launcher to pretrain_hybrid.py; choosing or reviewing a hybrid layer pattern; running gpt_hybrid_conversion.py; loading a converted checkpoint; diagnosing GPT-to-Hybrid migration issues; 'migrate GPTModel to HybridModel', 'convert GPT checkpoint to HybridModel', 'hybrid layer pattern'.

license

Apache-2.0

metadata.author

Philip Petrakian <ppetrakian@nvidia.com>

n=32                                   # source GPT layer count
blk='*-'                               # '*-' dense, '*E' every-layer MoE
pat=$(printf "$blk%.0s" $(seq $n))

n=32; seg=4; per=$((n/seg))
b=$(printf "$blk%.0s" $(seq $per)); pat=$b
for ((i=1;i<seg;i++)); do pat="$pat|$b"; done

torchrun ${DISTRIBUTED_ARGS[@]} pretrain_gpt.py ${MODEL_ARGS[@]}

touch 'a-b-'; ARGS=(--hybrid-layer-pattern '*-*-')
printf '[%s]\n' ${ARGS[@]}     # -> [--hybrid-layer-pattern] [a-b-]   silently corrupted
printf '[%s]\n' "${ARGS[@]}"   # -> [--hybrid-layer-pattern] [*-*-]   correct

HYBRID_PATTERN=$(printf '*-%.0s' $(seq $NUM_LAYERS))
MODEL_ARGS=( ... --hybrid-layer-pattern "$HYBRID_PATTERN" ... )
torchrun "${DISTRIBUTED_ARGS[@]}" pretrain_hybrid.py "${MODEL_ARGS[@]}" ...

# 1. No rejected or stale arguments survived -- must print nothing.
grep -nE -- '--(num-layers|num-layers-per-virtual-pipeline-stage|num-virtual-stages-per-pipeline-rank|pipeline-model-parallel-layout|account-for-embedding-in-pipeline-split|account-for-loss-in-pipeline-split|hybrid-override-pattern|fim-data)\b' train_hybrid.sh

# 2. Pattern shape -- attn and mlp must each equal the source GPT layer count.
p='*-*-|*-*-'
main=${p%%/*}; main=${main//|/}
attn=${main//[^\*]/}; mlp=${main//[^-E]/}; segs=${p%%/*}; segs=${segs//[^|]/}
echo "layers=${#main} attn=${#attn} mlp=${#mlp} segments=$(( ${#segs} + 1 ))"

© NVIDIA, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

Just SKILL.md in skills/mcore-migrate-gpt-to-hybrid of NVIDIA/Megatron-LM.

Open the folder on GitHubat commit a3e1f82