Analyze Nsys Profile

Analyze Nsys Profile

A passive query toolkit for PithTrain nsys traces. The agent asks a specific question; the skill provides primitives that answer it fast and correctly. The skill does not produce an unsolicited full report. It expects the agent to compose the right query for the question being asked.

Prerequisites

• A captured .nsys-rep exists (default location: workspace/capture-nsys-profile/pithtrain_node*.nsys-rep).
• The repo venv is active: source .venv/bin/activate.
• nsys CLI on PATH (for the one-time SQLite export).

Step 1 — Export the trace to SQLite

nsys export --type=sqlite --force-overwrite=true --output=workspace/capture-nsys-profile/pithtrain_node0.sqlite workspace/capture-nsys-profile/pithtrain_node0.nsys-rep

All subsequent queries hit the SQLite, not the raw .nsys-rep.

Step 2 — Shared preparation (always run first)

Three primitives establish who, when, and what — every downstream analysis depends on the data they surface. Run (or at least understand the output of) all three before reaching for the analysis scripts below.

Pipeline: show_setup → find_window → classify_streams. show_setup gives you the mapping pid ↔ rank ↔ mesh coordinates; find_window picks the median DualPipeV chunk per rank (deterministic across re-runs, so before/after comparisons are valid); classify_streams identifies which CUDA streams in that window are compute vs comm, and labels the comm streams' purpose (ep_a2a, cp_ring, pp_p2p).

Step 3 — Measure overlap per DualPipeV stage

python .agents/skills/analyze-nsys-profile/scripts/compute_overlap.py workspace/capture-nsys-profile/pithtrain_node0.sqlite

Emits one row per (rank, stage) with columns: pid | stage | exposed_ns | overlap | overlap_min | overlap_max. The overlap column is the time-weighted hidden fraction across the stage's comm kernels; overlap_min / overlap_max are the extremes of the per-kernel overlap percentage and surface whether the stage is uniformly bad or bimodal.

See references/examples.md for recipes that compose this with the Step 2 primitives.

Critical conventions

Before composing a custom SQL query, read references/conventions.md. Highlights:

• pid (Linux PID) is the per-rank join key, extracted exactly as the nsys docs prescribe: globalPid / 0x1000000 % 0x1000000 (kernel rows) == globalTid / 0x1000000 % 0x1000000 (NVTX rows). Single SQLite per node → PIDs unique within a trace.
• Always filter start >= 0 — pre-cudaProfilerStart NCCL init ranges have negative timestamps.
• Per-rank setup label is the first in-window NVTX event per rank: rank=N; pp=R/S dp=R/S cp=R/S ep=R/S; mbs=M seq=Q.
• Chunk anchor for steady state: the median-indexed forward chunk X (phaseY) backward chunk Z (phaseW) NVTX range emitted by DualPipeV. Match with LIKE 'forward chunk%backward chunk%' to disambiguate from per-stage forward markers.
• Compute-vs-comm classification: a kernel is communication if its short name starts with nccl, otherwise compute. A stream is a comm stream iff every one of its kernels is NCCL.
• Comm-stream purpose is discovered from the PithTrain stage-NVTX (layer*.stageN_*) enclosing each kernel at its CPU-side launch time — every kernel must agree on the label (unanimity), otherwise mixed.

Non-fragile classification rules

Avoid these heuristics — they break across configs:

• "Stream with > N kernels of type X is comm" (N depends on layer count, chunks, seq length).
• "Kernel duration > T µs means data movement" (long duration can also be a straggler wait).
• "Stream with avg µs < threshold is EP" (depends on token volume per rank).

Use these instead:

• One-sided purity check for compute-vs-comm streams.
• NVTX-context labeling for stream purpose: look up the innermost PithTrain stage range enclosing each kernel and require unanimous agreement. Implemented in classify_streams.py.

Worked examples

See references/examples.md for recipe-style answers to:

• How well is each EP phase overlapped with compute?
• Which EP phase has the worst overlap?
• Are the PP stages balanced?
• Which (rank, stage) carries the most exposed comm?