Megatron-LM on SLURM

What it does

When your agent uses it

• Writing an sbatch script for multi-node Megatron-LM training
• Choosing the right CUDA_DEVICE_MAX_CONNECTIONS value for a GPU generation
• Setting MASTER_ADDR and WORLD_SIZE from a SLURM allocation
• Diagnosing a failing rank in a distributed run

Example prompts

• “Write an sbatch script that runs Megatron-LM across four nodes with eight GPUs each.”
• “Tell me the right CUDA_DEVICE_MAX_CONNECTIONS for Hopper with tensor parallelism and no FSDP.”
• “Our multi-node job hangs on startup, so check the torch.distributed.run environment variables.”

Requirements

• A SLURM cluster login with access to a GPU partition
• Megatron-LM checked out on a filesystem shared by all nodes
• uv installed, with uv sync run before submission

What it can do on your machine

Read from SKILL.md and the folder at commit eb50202. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Shell commands in SKILL.md call:

  • uv

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  No URLs in SKILL.md. Its commands use uv, which can reach the network depending on how they are called.

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Megatron-LM on SLURM loads about 1.8k tokens when it runs. Until then it costs about 145 tokens; SKILL.md has 714 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~145

When it runs · the whole SKILL.md, loaded when a task matches

~1.8k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from NVIDIA/Megatron-LM at commit eb50202, republished under its Apache-2.0 licence (© NVIDIA). 714 words, ~1,804 tokens.

name

mcore-run-on-slurm

description

How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis. Use when submitting a SLURM job; writing or debugging an sbatch script; configuring multi-node distributed training; setting MASTER_ADDR / MASTER_PORT / WORLD_SIZE; diagnosing a SLURM job failure; 'how do I run on the cluster', 'sbatch', 'multi-node training'.

license

Apache-2.0

metadata.author

Oliver Koenig <okoenig@nvidia.com>

#!/bin/bash
#SBATCH --job-name=megatron
#SBATCH --account=<SLURM_ACCOUNT>
#SBATCH --partition=<SLURM_PARTITION>
#SBATCH --nodes=<NODES>
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=<GPUS_PER_NODE>
#SBATCH --time=<HH:MM:SS>
#SBATCH --output=logs/%x-%j.out
#SBATCH --error=logs/%x-%j.err

set -euo pipefail
cd <MEGATRON_WORKTREE>

export MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n1)
export MASTER_PORT=${MASTER_PORT:-29500}
export NNODES=${SLURM_NNODES}
export GPUS_PER_NODE=<GPUS_PER_NODE>
export WORLD_SIZE=$((NNODES * GPUS_PER_NODE))

# Set CUDA_DEVICE_MAX_CONNECTIONS only when your configuration requires it
# (see the section below). Example for pre-Blackwell with TP>1 or CP>1
# (non-FSDP):
#   export CUDA_DEVICE_MAX_CONNECTIONS=1

srun --ntasks=${NNODES} --ntasks-per-node=1 bash -c '
  # NODE_RANK comes from SLURM_NODEID with one task per node.
  NODE_RANK=${SLURM_NODEID}
  uv run python -m torch.distributed.run \
    --nnodes='"${NNODES}"' \
    --nproc-per-node='"${GPUS_PER_NODE}"' \
    --node-rank=${NODE_RANK} \
    --master-addr='"${MASTER_ADDR}"' \
    --master-port='"${MASTER_PORT}"' \
    pretrain_gpt.py \
      <MEGATRON_ARGS>
'

mkdir -p logs && JOB_ID=$(sbatch --parsable run_megatron.slurm)
echo "Submitted ${JOB_ID}"

squeue -j "$JOB_ID" -o "%.10i %.8T %.10M %.6D %R"
sacct -j "$JOB_ID" --format=JobID,State,ExitCode,Elapsed
scancel "$JOB_ID"

© NVIDIA, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 4 other files in skills/mcore-run-on-slurm of NVIDIA/Megatron-LM.

• SKILL.md
• BENCHMARK.md
• evals/evals.json
• skill-card.md
• skill.oms.sig

Open the folder on GitHubat commit eb50202

Used in 1 other repository

We found 1 copy of this SKILL.md (exact, near-identical or edited) in other folders. This page covers the copy in NVIDIA/Megatron-LM, which our catalogue first saw on October 7, 2026.

• NVIDIA/skillsskills/mcore-run-on-slurm