Megatron-LM CI/CD Guide

What it does

When your agent uses it

• Choosing PR labels to set the CI test scope in Megatron-LM
• Triggering the internal GitLab CI safely for a branch
• Investigating why a CI run failed

Example prompts

• “Which labels do I need so my PR also runs the MBridge tests?”
• “Do a dry run of triggering the internal CI for my branch and show me the destination ref.”
• “Why did the cicd-main workflow fail on my pull request?”

Requirements

• A Megatron-LM checkout
• Access to the internal GitLab remote for triggering CI

Workflow steps

5 steps, taken from the first numbered list in SKILL.md.

1. 1Linting failure — re-run tools/autoformat.sh locally; the diff shows exactly what needs to change.
2. 2Container build failure — inspect the cicd-container-build job log.
3. 3Unit test failure — the failing bucket is in the cicd-unit-tests-latest job matrix.
4. 4Functional test failure — look at the cicd-integration-tests-* job. Start with stdout.log for rank 0.
5. 5Flaky test — the runner retries automatically up to 3 times. If all retries exhausted and the pattern matches a known transient (NCCL…

What it can do on your machine

Read from SKILL.md and the folder at commit 486a126. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Shell commands in SKILL.md call:

  • gh
  • python
  • git

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  No URLs in SKILL.md. Its commands use gh and git, which can reach the network depending on how they are called.

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names these keys or tokens, usually read from environment variables:

  • GITLAB_TOKEN

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Megatron-LM CI/CD Guide loads about 1.8k tokens when it runs. Until then it costs about 80 tokens; SKILL.md has 545 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~80

When it runs · the whole SKILL.md, loaded when a task matches

~1.8k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from NVIDIA/Megatron-LM at commit 486a126, republished under its Apache-2.0 licence (© NVIDIA). 545 words, ~1,754 tokens.

name

mcore-cicd

description

CI/CD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request/BRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.

license

Apache-2.0

when_to_use

Investigating a CI failure; understanding the pipeline structure; which CI label to attach; triggering internal GitLab CI; 'CI is red', 'how do I trigger CI'…

metadata.author

Oliver Koenig <okoenig@nvidia.com>

is-not-external-contributor
  └─ pre-flight
       └─ configure          # determines scope, container tag, n_repeat
            ├─ linting
            ├─ cicd-container-build
            │    ├─ cicd-parse-unit-tests → cicd-unit-tests-latest
            │    ├─ cicd-parse-integration-tests-h100 → cicd-integration-tests-latest-h100
            │    └─ cicd-parse-integration-tests-gb200 → cicd-integration-tests-latest-gb200 (maintainers only)
            └─ Nemo_CICD_Test  # final pass/fail gate

python tools/trigger_internal_ci.py --gitlab-origin gitlab --dry-run

# Extract PR number from the current branch
PR_NUMBER=$(git rev-parse --abbrev-ref HEAD | grep -oP '(?<=pull-request/)\d+')

# Fetch the PR metadata (title, labels, author, base branch)
gh pr view "$PR_NUMBER" --repo NVIDIA/Megatron-LM

# Show the changeset for that PR
gh pr diff "$PR_NUMBER" --repo NVIDIA/Megatron-LM

# List recent workflow runs for the PR
gh run list --repo NVIDIA/Megatron-LM --branch "pull-request/$PR_NUMBER"

# Stream failing job output
gh run view <run-id> --repo NVIDIA/Megatron-LM --log-failed

# 1. Find artifact name
gh run view <run-id> --repo NVIDIA/Megatron-LM --json artifacts \
  --jq '.artifacts[].name'

# 2. Download the artifact zip
gh run download <run-id> --repo NVIDIA/Megatron-LM \
  --name "logs-<artifact-name>" -D ./ci-logs

# 3. Locate which rank logs contain errors
grep -r -l "ERROR\|Traceback\|FAILED\|fatal" ./ci-logs/

# 4. Log files can exceed 10 000 lines — never read a full log at once.
wc -l ./ci-logs/<test>/<attempt>/attempt_0/<rank>/stderr.log
sed -n '1,200p' ./ci-logs/.../stderr.log   # read in chunks

# Find unit tests that cover a changed source file
grep -r "from megatron.core.transformer.attention" tests/unit_tests/ -l

# Check CODEOWNERS for reviewer assignment
cat .github/CODEOWNERS | grep "<changed-path>"

© NVIDIA, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 4 other files in skills/mcore-cicd of NVIDIA/Megatron-LM.

• SKILL.md
• BENCHMARK.md
• evals/evals.json
• skill-card.md
• skill.oms.sig

Open the folder on GitHubat commit 486a126