New: 98,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Official agent skill

Nemo Mbridge Perf Parallelism Strategies

by NVIDIA in NVIDIA/skills

Operational guide for choosing and combining parallelism strategies in Megatron Bridge, including sizing rules, hardware topology mapping, and combined parallelism configuration.

OfficialApache-2.0Auto-check passed

Install Nemo Mbridge Perf Parallelism Strategies

skills CLI
$ npx skills add NVIDIA/skills --skill nemo-mbridge-perf-parallelism-strategies -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install NVIDIA/skills nemo-mbridge-perf-parallelism-strategies --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/NVIDIA/skills.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/nemo-mbridge-perf-parallelism-strategies .claude/skills/nemo-mbridge-perf-parallelism-strategies && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
nemo-mbridge-perf-parallelism-strategies
GitHub stars
3.6k
Token cost
~2.4k tokens
SKILL.md length
901 words
Files
6
Skills in repo
390
Repo updated
First seen
Licence
Apache-2.0

At a glance

Operational guide for choosing and combining parallelism strategies in Megatron Bridge, including sizing rules, hardware topology mapping, and combined parallelism configuration.

  • Works in 8 steps: TP across nodes destroys throughput.… → PP without interleaving has large… → SP requires tensor_model_parallel_size >… → …
  • SKILL.md covers Decision by Model Size, Decision by Hardware Topology, Decision by Sequence Length and Combined Parallelism Enablement, plus 5 more sections
  • Calls uv

What it does

Nemo Mbridge Perf Parallelism Strategies is an agent skill from NVIDIA/skills, published by the product's own GitHub organization. Operational guide for choosing and combining parallelism strategies in Megatron Bridge, including sizing rules, hardware topology mapping, and combined parallelism configuration.

Its SKILL.md is about 2.4k tokens, which your agent loads only when the skill is triggered. The skill folder holds 6 other files (for example `BENCHMARK.md`, `card.yaml` and `evals/evals.json`).

It works with NVIDIA AI Platform. The repository describes itself as: Agent Skills for NVIDIA products — install into Claude Code, Codex, and other coding agents to run Physical AI, robotics, simulation, CUDA, and RAG workflows end to end. The licence is Apache-2.0.

Example prompts

  • “/nemo-mbridge-perf-parallelism-strategies”

Requirements

  • Python 3

Workflow steps

8 steps, taken from the first numbered list in SKILL.md.

  1. TP across nodes destroys throughput. Always keep TP within a single
  2. PP without interleaving has large pipeline bubbles. Use
  3. SP requires tensor_model_parallel_size > 1. Enabling SP alone
  4. CP requires seq_length % (2 * context_parallel_size) == 0.
  5. EP is only for MoE models. Setting expert_model_parallel_size on a
  6. The model-size-to-parallelism table above is a starting heuristic.
  7. CUDA_DEVICE_MAX_CONNECTIONS and related env vars interact with
  8. The minimum GPU count for an MoE config is PP * max(TP*CP, EP*ETP),

What it can do on your machine

Read from SKILL.md and the folder at commit 14a98ae. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Shell commands in SKILL.md call:

    • uv

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    Links to these hosts (documentation or services it may open):

    • arxiv.org

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Nemo Mbridge Perf Parallelism Strategies loads about 2.4k tokens when it runs. Until then it costs about 55 tokens; SKILL.md has 901 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~55
When it runs · the whole SKILL.md, loaded when a task matches
~2.4k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from NVIDIA/skills at commit 14a98ae, republished under its Apache-2.0 licence (© NVIDIA). 901 words, ~2,390 tokens.

Download SKILL.mdSave it as .claude/skills/nemo-mbridge-perf-parallelism-strategies/SKILL.md (or your agent's skills folder). This skill also uses 5 other files; get the full folder from GitHub.
name
nemo-mbridge-perf-parallelism-strategies
description
Operational guide for choosing and combining parallelism strategies in Megatron Bridge, including sizing rules, hardware topology mapping, and combined parallelism configuration.
license
Apache-2.0
when_to_use
Choosing or sizing TP/DP/PP/CP/EP degrees, or tracing an OOM or regression to a parallelism config change; 'how to parallelize', 'tensor parallel', 'pipeline…

Parallelism Strategy Selection Skill

For stable background on each parallelism type, see:

  • @docs/parallelisms.md
  • @skills/nemo-mbridge-perf-parallelism-strategies/card.yaml

Decision by Model Size

Dense models
Model sizeGPUsRecommended starting point
< 1B1-8DP only
1-10B8-16TP=2-4 + DP
10-70B16-64TP=4-8 + PP=2-4 + DP
70-175B64-256TP=8 + PP=4-8 + DP
175-500B256-1024TP=8 + PP=8-16 + CP=2 + DP
MoE models

MoE parallelism differs from dense models. Because only a fraction of parameters are active per token, TP can often stay at 1 or 2 — the active parameter shard already fits on a single GPU. EP is the primary scaling dimension, with PP handling cross-node layer distribution.

Model (total / active)TPPPEPNotes
OLMoE 7B / 1B118EP only, fits single node
Moonlight 16B / 3B218small TP for shared layers
DeepSeek-V2 236B / 21B1432no TP at all
GLM-4.5 Air 106B / 12B148no TP at all
Qwen3 30B-A3B424
GLM-4.5 355B / 32B2816
Qwen3 235B-A22B4168CP=2 for pretrain
DeepSeek-V3 671B / 37B21664TP=2, not 8
Kimi-K2 1T21632

Key patterns:

  • TP is sized by active params, not total params. A 671B MoE with 37B active needs far less TP than a 70B dense model.
  • EP scales with expert count. Common: EP = num_experts or num_experts / experts_per_gpu.
  • PP handles depth. Large MoE models use PP=8-16 across nodes.
  • ETP (expert tensor parallelism) is rarely used. Llama 4 is an exception (ETP=4).

These are starting points, not hard rules. Always profile the first iteration to verify memory and communication.

Decision by Hardware Topology

Single node with NVLink:

python
cfg.model.tensor_model_parallel_size = 8

Multiple nodes with InfiniBand:

python
cfg.model.tensor_model_parallel_size = 8
cfg.model.pipeline_model_parallel_size = N

Limited network (Ethernet):

python
cfg.model.tensor_model_parallel_size = 4
cfg.model.pipeline_model_parallel_size = M

The stable rule is: keep TP within a single NVLink domain. Use PP or DP for cross-node scaling. TP across nodes is almost always a performance loss.

Decision by Sequence Length

Sequence lengthRecommendation
< 2Kstandard TP + PP + DP
2K-8Kadd SP (sequence_parallel=True)
8K-32Kadd CP=2
32K+add CP=4-8, consider a2a+p2p for large CP

Combined Parallelism Enablement

3D parallelism (TP + PP + DP):

python
cfg.model.tensor_model_parallel_size = 4
cfg.model.pipeline_model_parallel_size = 4
cfg.model.sequence_parallel = True

4D parallelism (TP + PP + CP + DP):

python
cfg.model.tensor_model_parallel_size = 8
cfg.model.pipeline_model_parallel_size = 8
cfg.model.context_parallel_size = 2
cfg.model.sequence_parallel = True

MoE with EP + PP (e.g. DeepSeek-V2 236B on 128 GPUs):

python
cfg.model.tensor_model_parallel_size = 1
cfg.model.pipeline_model_parallel_size = 4
cfg.model.expert_model_parallel_size = 32
cfg.model.sequence_parallel = False

MoE with small TP + PP + EP (e.g. DeepSeek-V3 671B on 256 GPUs):

python
cfg.model.tensor_model_parallel_size = 2
cfg.model.pipeline_model_parallel_size = 16
cfg.model.expert_model_parallel_size = 64
cfg.model.sequence_parallel = True

DP size is always implicit:

data_parallel_size = world_size / (TP * PP * CP)        # dense path
expert_data_parallel_size = world_size / (PP * EP * ETP) # MoE path

Minimum GPU Count

The minimum GPUs needed to run a config (i.e. with DP=1, EDP=1) is not the product of all parallelism dimensions. The dense path uses a TP*CP-mesh and the MoE path uses an EP*ETP-mesh, and within each PP stage these two meshes share the same set of GPUs — they overlap, they don't multiply. Only PP stages multiply (they're disjoint slices of the model). So:

min_gpus = PP * max(TP * CP, EP * ETP)

Common simplification (WRONG): PP * TP * CP * EP * ETP. This over-allocates GPUs and shows up in many READMEs and slurm sizing tables. Don't propagate it.

The decoupling of attention and MoE parallelism (different mesh shapes for the dense and expert paths sharing the same PP-stage GPUs) is detailed in Pangu Ultra MoE (arXiv:2504.14960).

Examples
ConfigWrong (PP·TP·CP·EP·ETP)Correct (PP·max(TP·CP, EP·ETP))
PP=1, TP=2, CP=1, EP=8, ETP=1168 (1 node)
PP=1, TP=4, CP=1, EP=8, ETP=1328 (max(4, 8))
PP=1, TP=2, CP=2, EP=8, ETP=1328 (max(4, 8))
PP=1, TP=2, CP=4, EP=8, ETP=1648 (max(8, 8))
PP=2, TP=2, CP=1, EP=8, ETP=13216 (2 · max(2, 8))
PP=1, TP=2, CP=1, EP=4, ETP=2168 (max(2, 8))
Show full SKILL.md (353 more words)Show less
Scaling above the minimum

Adding GPUs scales DP and/or EDP (the world_size must satisfy both equations simultaneously). At min_gpus the larger-mesh side has DP (or EDP) = 1 and the smaller side absorbs the slack.

Example — TP=2, CP=1, EP=8, ETP=1, PP=1:

  • 8 GPUs (min_gpus): dense DP = 8/2 = 4, MoE EDP = 8/8 = 1
  • 16 GPUs: dense DP = 8, MoE EDP = 2 → 2× global batch
  • 32 GPUs: dense DP = 16, MoE EDP = 4 → 4× global batch

When sizing slurm scripts, compute --nodes from min_gpus (or a multiple of it for higher throughput via DP/EDP).

When answering MoE sizing prompts, include this checklist:

  • compute min_gpus = PP * max(TP * CP, EP * ETP) with the requested values
  • explicitly reject the wrong PP * TP * CP * EP * ETP full product
  • give both DP formulas: dense world_size / (TP * PP * CP) and MoE world_size / (PP * EP * ETP)
  • mention TP topology, SP, CP divisibility, and long-sequence CP guidance

Memory Estimation

Without parallelism (70B model, FP16):

parameters:       140 GB
gradients:        140 GB
optimizer states: 280 GB (Adam)
activations:       48 GB (batch=1, seq=4K)
total:            608 GB

With TP=4, PP=4, DP=4 (64 GPUs):

parameters:        8.75 GB per GPU
gradients:         8.75 GB per GPU
optimizer states: 17.50 GB per GPU
activations:       3.00 GB per GPU
total:           ~38    GB per GPU

Code Anchors

Parallelism dimensions set in model provider:

6681docsparallelisms
model_config = GPTModelProvider(
    tensor_model_parallel_size=2,
    # ... other model parameters
)

DP size calculation:

424436docsparallelis
data_parallel_size = world_size / (tensor_model_parallel_size × pipeline_model_parallel_size × context_parallel_size)

Bridge initialization wires parallelism into process groups:

618628srcmegatronbri
parallel_state.initialize_model_parallel(
    tensor_model_parallel_size=model_config.tensor_model_parallel_size,
    pipeline_model_parallel_size=model_config.pipeline_model_parallel_size,
    ...
    context_parallel_size=model_config.context_parallel_size,
    hierarchical_context_parallel_sizes=model_config.hierarchical_context_parallel_sizes,
    expert_model_parallel_size=model_config.expert_model_parallel_size,
    ...
)

Pitfalls

  1. TP across nodes destroys throughput. Always keep TP within a single NVLink domain.

  2. PP without interleaving has large pipeline bubbles. Use virtual_pipeline_model_parallel_size when possible.

  3. SP requires tensor_model_parallel_size > 1. Enabling SP alone without TP is a config error.

  4. CP requires seq_length % (2 * context_parallel_size) == 0.

  5. EP is only for MoE models. Setting expert_model_parallel_size on a dense model is a no-op or error.

  6. The model-size-to-parallelism table above is a starting heuristic. Always profile the first iteration to check memory and communication.

  7. CUDA_DEVICE_MAX_CONNECTIONS and related env vars interact with overlap settings. See @skills/nemo-mbridge-perf-tp-dp-comm-overlap/SKILL.md.

  8. The minimum GPU count for an MoE config is PP * max(TP*CP, EP*ETP), not the product of all dimensions. The dense TP*CP-mesh and MoE EP*ETP-mesh share the same GPUs in each PP stage. See "Minimum GPU Count" section above.

Verification

Quick sanity check that combined parallelism initializes correctly using the smallest available recipe with overridden parallelism:

bash
CUDA_VISIBLE_DEVICES=0,1,2,3 uv run python -m torch.distributed.run --nproc_per_node=4 \
  scripts/training/run_recipe.py \
  --recipe llama32_1b_pretrain_config \
  model.tensor_model_parallel_size=2 \
  model.pipeline_model_parallel_size=2 \
  model.sequence_parallel=True \
  train.train_iters=3 train.global_batch_size=8 train.micro_batch_size=1 \
  scheduler.lr_warmup_iters=0 \
  validation.eval_iters=0 validation.eval_interval=0 \
  checkpoint.save_interval=0 \
  logger.log_interval=1

Success criteria:

  • exit code 0
  • finite loss at iteration 3 (e.g. lm loss: 1.003808E+01)
  • log shows TP=2 PP=2 DP=1 layout with 4 ranks

© NVIDIA, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 5 other files in skills/nemo-mbridge-perf-parallelism-strategies of NVIDIA/skills.

  • SKILL.md
  • BENCHMARK.md
  • card.yaml
  • evals/evals.json
  • skill-card.md
  • skill.oms.sig

Open the folder on GitHubat commit 14a98ae

Compare with similar skills

Nemo Mbridge Perf Parallelism Strategies next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Nemo Mbridge Perf Parallelism Strategies compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Nemo Mbridge Perf Parallelism Strategies this skillNVIDIA/skills3.6k—~2.4kAutomated safety check: PassApache-2.0
LLM Torch Profiler Analysissgl-project/sglang37k2 repos~6.4kAutomated safety check: PassApache-2.0
Skill InspectorNVIDIA/SkillSpector20k—~1.8kAutomated safety check: PassApache-2.0
Megatron-LM Container and Dependency SetupNVIDIA/Megatron-LM18k—~2.6kAutomated safety check: PassApache-2.0
Embeddings via 9Routerdecolua/9router31k—~604Automated safety check: PassMIT
Megatron-LM Base Image BumpNVIDIA/Megatron-LM18k—~2.8kAutomated safety check: PassApache-2.0

Similar skills

  • LLM Torch Profiler Analysis

    sgl-project/sglang

    Unified LLM torch-profiler triage skill for sglang, vllm, TensorRT-LLM, and TokenSpeed.

    37k GitHub starsUsed in 2 repos~6.4k tokens
    DevelopmentAuto-check passed

  • Skill Inspector

    NVIDIA/SkillSpector

    Official

    Decides whether an agent skill is safe to install by combining a SkillSpector static scan with the agent's own source review, ending in APPROVE, CAUTION or REJECT.

    20k GitHub stars~1.8k tokensUpdated today
    SecurityAuto-check passed

  • Megatron-LM Container and Dependency Setup

    NVIDIA/Megatron-LM

    Official

    Walks an agent through working inside the Megatron-LM CI container and changing dependencies with uv, so lock files resolve the same locally and in CI.

    18k GitHub stars~2.6k tokensUpdated today
    DevelopmentAuto-check passed

  • Embeddings via 9Router

    decolua/9router

    Generates vector embeddings through the 9Router /v1/embeddings endpoint, using models from providers such as OpenAI, Gemini, Mistral and Voyage for RAG and semantic search.

    31k GitHub stars~604 tokensUpdated 2 days ago
    AI & LLM EngineeringAuto-check passed

  • Megatron-LM Base Image Bump

    NVIDIA/Megatron-LM

    Official

    Moves Megatron-LM CI to a newer NVIDIA PyTorch base image, updating both the GitHub and GitLab pins together and handling the CI follow-up.

    18k GitHub stars~2.8k tokensUpdated today
    DevOps & CloudAuto-check passed

  • Nemoclaw Maintainer Normalize Title Tags

    NVIDIA/NemoClaw

    Official

    Remove bracketed NemoClaw tags from GitHub issue and PR titles.

    23k GitHub stars~693 tokensUpdated today
    Marketing & SEOAuto-check passed

More from NVIDIA/skills

All 390 skills in this repo

  • Vss Deploy Detection Tracking 2D

    NVIDIA/skills

    Official

    A skill your agent uses when the user wants to deploy, run, debug, tear down, or call the REST API of the RTVI-CV 2D detection / tracking microservice.

    3.6k GitHub starsUsed in 1 repo~4.5k tokens
    Auto-check passed

  • HSB IP Def File Tool

    NVIDIA/skills

    Official

    Generates, validates, compares and explains HOLOLINK_def.svh macro files for the HSB IP, using bundled Python scripts and asking before it writes anything.

    3.6k GitHub stars~2.9k tokensUpdated yesterday
    Auto-check passed

  • Isaac Sim Mission Control Showcase

    NVIDIA/skills

    Official

    Runs and validates an end-to-end Mission Control demo in a locally installed Isaac Sim, with a Nova Carter robot driven through a Python server.

    3.6k GitHub stars~4.8k tokensUpdated yesterday
    Auto-check passed

  • Defect Image Generation with Cosmos AnomalyGen

    NVIDIA/skills

    Official

    Orchestrates defect image generation for PCBA, metal surface and glass inspection with NVIDIA Cosmos AnomalyGen on OSMO, from cold-start Day 0 to real-photo Day 1 labeling.

    3.6k GitHub stars~5k tokensUpdated yesterday
    Auto-check: notes

  • Physical AI Video Augmentation on OSMO

    NVIDIA/skills

    Official

    Orchestrates video data augmentation and auto-labeling workflows on OSMO, from flow selection and preflight checks to submission, monitoring and output download.

    3.6k GitHub stars~4.7k tokensUpdated yesterday
    Auto-check: notes

  • TAO Detection KPI Analysis

    NVIDIA/skills

    Official

    Runs NVIDIA TAO Data Services KPI analysis on object detection results, comparing predictions to ground truth and writing per-class precision, recall and AP to a CSV.

    3.6k GitHub stars~2.7k tokensUpdated yesterday
    Auto-check: notes

Works with

Questions about Nemo Mbridge Perf Parallelism Strategies

What does Nemo Mbridge Perf Parallelism Strategies do?

Operational guide for choosing and combining parallelism strategies in Megatron Bridge, including sizing rules, hardware topology mapping, and combined parallelism configuration. Nemo Mbridge Perf Parallelism Strategies is an agent skill from NVIDIA/skills, published by the product's own GitHub organization. Operational guide for choosing and combining parallelism strategies in Megatron Bridge, including sizing rules, hardware topology mapping, and combined parallelism configuration.

How do I install Nemo Mbridge Perf Parallelism Strategies in Claude Code?

Run `npx skills add NVIDIA/skills --skill nemo-mbridge-perf-parallelism-strategies -a claude-code`. Or copy the skill folder (skills/nemo-mbridge-perf-parallelism-strategies in NVIDIA/skills) into .claude/skills/nemo-mbridge-perf-parallelism-strategies in your project. Claude Code loads it when a task matches its description.

How do I install Nemo Mbridge Perf Parallelism Strategies in Codex?

Run `npx skills add NVIDIA/skills --skill nemo-mbridge-perf-parallelism-strategies -a codex`. Or copy the skill folder (skills/nemo-mbridge-perf-parallelism-strategies in NVIDIA/skills) into .agents/skills/nemo-mbridge-perf-parallelism-strategies in your project. Codex loads it when a task matches its description.

Can I use Nemo Mbridge Perf Parallelism Strategies in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add NVIDIA/skills --skill nemo-mbridge-perf-parallelism-strategies -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/nemo-mbridge-perf-parallelism-strategies, .gemini/skills/nemo-mbridge-perf-parallelism-strategies, .github/skills/nemo-mbridge-perf-parallelism-strategies and .opencode/skills/nemo-mbridge-perf-parallelism-strategies in your project.

What does Nemo Mbridge Perf Parallelism Strategies need to run?

Going by SKILL.md and its folder, Nemo Mbridge Perf Parallelism Strategies needs the command-line tools its instructions call (uv). Our summary lists: Python 3.

Does Nemo Mbridge Perf Parallelism Strategies access the network?

SKILL.md names 1 domain. As links in the text: arxiv.org. This is read from the text; nothing was executed.

Is Nemo Mbridge Perf Parallelism Strategies safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Nemo Mbridge Perf Parallelism Strategies use?

Nemo Mbridge Perf Parallelism Strategies is published under the Apache-2.0 licence (declared in SKILL.md). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Nemo Mbridge Perf Parallelism Strategies use?

About 2.4k tokens (SKILL.md is roughly 9.6k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Nemo Mbridge Perf Parallelism Strategies?

Skills that share tags, products or a category with Nemo Mbridge Perf Parallelism Strategies: LLM Torch Profiler Analysis (sgl-project/sglang, 37k stars), Skill Inspector (NVIDIA/SkillSpector, 20k stars), Megatron-LM Container and Dependency Setup (NVIDIA/Megatron-LM, 18k stars) and Embeddings via 9Router (decolua/9router, 31k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Nemo Mbridge Perf Parallelism Strategies?

NVIDIA (a GitHub organization, an official publisher) maintains it in NVIDIA/skills, which has 3,555 GitHub stars. The repository holds 390 skills in this directory. The repository was last updated on October 9, 2026.

Source: NVIDIA/skills on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.