LLM Torch Profiler Analysis

Unified LLM Torch Profiler Analysis

Overview

Use this skill for torch.profiler analysis across:

• sglang
• vllm
• TensorRT-LLM
• TokenSpeed

There is only one public workflow:

• triage

Preferred unified entrypoint:

• scripts/analyze_llm_torch_profile.py

Backwards-compatibility shim (kept so older docker exec ... analyze_sglang_torch_profile.py ... calls keep working; it just forwards to the unified entrypoint):

• scripts/analyze_sglang_torch_profile.py

Markdown bundling helper:

• scripts/render_triage_markdown_bundle.py

triage always prints the same three tables:

• kernel table
• overlap-opportunity table
• fuse-pattern table

By default, all three tables only render rows at or above 1.0% cumulative GPU-time share. Rows below that are hidden by default unless the user asks for a lower cutoff.

Keep the fuse-pattern table source-backed and deterministic. Do not turn it into a fuzzy matcher.

If exact source-backed matching is weak but a kernel cluster is still close to a known family, add one short note after the tables with exactly one of:

• high
• medium
• low

Capability Matrix

For TensorRT-LLM, live capture only works when the server exposes /start_profile and /stop_profile, and when the deployment already provides a shared trace path plus the required env vars.

For TokenSpeed, this skill supports both existing trace triage and live capture against current servers that expose /start_profile and /stop_profile. The live helper sends output_dir, activities, with_stack, record_shapes, and profile_id in the start payload. TokenSpeed also has its own native profile_by_stage field for manual capture, but the unified helper uses workload-separated prefill/ and decode/ directories by default so the tables stay comparable across frameworks.

Real H100 Validation

The current reference run is the 4x H100 matrix captured on 2026-04-23 on h100_sglang under:

• /data/bbuf/validate/unified_llm_profiler_skill/runs/20260423_h100_large_model_matrix_v3

Rendered markdown bundle:

• /data/bbuf/validate/unified_llm_profiler_skill/runs/20260423_h100_large_model_matrix_v3/h100_large_model_matrix_v3_bundle.md

Validated model directories:

• mixtral_8x7b_instruct
• qwen2_5_32b_instruct
• qwen3_32b

Each model directory contains:

• analysis_sglang.txt
• analysis_vllm.txt
• analysis_trtllm.txt
• framework-specific trace roots and probe artifacts

Validated matrix:

Use this run as the main H100 reference. The older 2026-04-22 single-card Qwen3 matrix is still useful for bring-up, but it is not the default reference anymore. TokenSpeed support was added later and is covered by existing-trace triage and HTTP profiler-control support, but it is not part of this older H100 validation matrix yet.

Stage-separated workload validation captured on 2026-05-01 on h100_sglang:

• /data/bbuf/validate/unified_llm_profiler_skill/runs/20260501_stage_split_validation
• /data/bbuf/validate/unified_llm_profiler_skill/runs/20260501_stage_split_validation_large

Validated models:

This validation also covers the compatibility fix for older SGLang profiler state machines: workload-separated live capture labels stages by output directory and avoids nesting SGLang's internal profile_by_stage state machine inside each workload. The helper adds one internal scheduler guard step because SGLang increments forward_ct before checking whether the profiler should stop; without that guard, a num_steps=1 prefill capture can stop just before the actual prefill forward. The 2026-05-01 two-card validation artifacts for the additional models are:

• /data/bbuf/validate/core_skill_validation_20260501/qwen3_8b/profiler
• /data/bbuf/validate/core_skill_validation_20260501/mistral_7b_instruct_v03/profiler

To render a validated run into one markdown document:

python3 scripts/render_triage_markdown_bundle.py \
  --analysis-root /data/bbuf/validate/unified_llm_profiler_skill/runs/20260423_h100_large_model_matrix_v3 \
  --output /data/bbuf/validate/unified_llm_profiler_skill/runs/20260423_h100_large_model_matrix_v3/h100_large_model_matrix_v3_bundle.md

The bundle groups by model and keeps the three tables for each framework.

H100 notes:

• all three frameworks now render kernel, overlap, and fuse tables with separate extend/prefill and decode sections when the trace contains a clean stage split
• SGLang live capture is validated and calls the server profiler API directly instead of shelling out to sglang.profiler
• SGLang trace flush can lag well beyond a few seconds, so the runner waits longer for artifacts than the earlier implementation
• SGLang kernel-site reconstruction keeps sampling disabled in the mapping path so the optimized parser does not perturb SGLang table output; equality rechecks matched for Mixtral-8x7B-Instruct-v0.1, Qwen3-32B, and nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8
• vLLM live capture requires --output-dir to match the server torch_profiler_dir; the validated H100 flow uses --profiler-config {"profiler":"torch","torch_profiler_dir":"..."} and then drives /start_profile and /stop_profile
• TensorRT-LLM validation stays on --backend pytorch; the H100 flow writes the trace with TLLM_TORCH_PROFILE_TRACE and then analyzes the saved trace
• TensorRT-LLM current mainline was rechecked at 0722c5f47d2cae69ac1a237da51e550dd214532c on 2026-06-26; the latest delta affects KV eviction / block-offset staging rather than profiler trace controls, so the b9e1945 profiler evidence still applies: PyTorch profiling uses record_shapes=True and with_modules=True, but not with_stack=True; keep the override path for table-quality Python locations unless the target image proves otherwise
• TokenSpeed trace analysis has first-class registry rows for native TokenSpeed CuTe DSL MLA, MLA KV pack + FP8 quantize, fused top-k/top-p sampling, persistent lm_head GEMM, and NVFP4 GEMM + SwiGLU + quant; live capture still requires an existing torch-profiler trace until the target TokenSpeed image exposes a supported profiler API
• on this host, keep all trace roots under /data/..., not /home/...

When To Use It

• inspect a torch.profiler trace or profile directory from sglang, vllm, TensorRT-LLM, or TokenSpeed
• profile a live serving endpoint and analyze the result
• summarize which kernel families dominate prefill or decode
• map kernels back to Python code paths
• judge whether a code path still leaves overlap opportunity
• check whether an already-known fusion or overlap path should have applied

Diffusion Backend Gate

For diffusion benchmark or profiling work, only analyze traces produced by the native SGLang diffusion backend.

If the run that generated the trace logs any of:

• Falling back to diffusers backend
• Using diffusers backend
• Loaded diffusers pipeline

stop the workflow instead of analyzing the trace. Handle it as a backend-selection issue, not as native-kernel profiler evidence.

Main Flows

Stage-Separated Live Capture Contract

Live capture must not use one mixed prompt as the default. By default, analyze_llm_torch_profile.py --url ... captures two labeled workloads and then renders the same three tables with separate stage sections:

• prefill: synthetic input length 4090, output length 1
• decode: synthetic input length 1, output length 2048

Every live profiler path warms up 10 steps before arming the profiler and then captures 5 active steps by default. Keep this warmup/active split aligned across SGLang, vLLM, and TensorRT-LLM before comparing kernel tables.

Use these options to override the contract when the benchmark workload is known:

--profile-workload both \
--warmup-steps 10 --num-steps 5 \
--prefill-input-len 4090 --prefill-output-len 1 \
--decode-input-len 1 --decode-output-len 2048

Allowed --profile-workload values:

• both: default; capture prefill and decode separately
• prefill: capture only the long-input / one-token workload
• decode: capture only the one-input / long-output workload
• legacy: keep the old --probe-prompt / --probe-max-new-tokens behavior

For sglang-sota-humanize-loop, do not use the defaults if the slow SGLang benchmark scenario has a known input/output distribution. Set the profiler lengths from that slow scenario instead: prefill uses the slow input length with output 1, and decode uses input 1 with the slow output length. For a mixed dataset, profile the slowest representative bucket such as the p50 or p95 input/output pair used in the benchmark report, and record the bucket in the artifact notes.

1. Single-trace triage from an existing profile dir or trace

python3 scripts/analyze_llm_torch_profile.py \
  --input /path/to/profile_dir_or_trace.json.gz

Use this when one trace is enough. The overlap table stays conservative in single-trace mode and will tell you when a mapping/formal pair is needed.

2. Single-trace live capture from SGLang

python3 scripts/analyze_llm_torch_profile.py \
  --framework sglang \
  --url http://127.0.0.1:30000 \
  --output-dir /data/bbuf/validate/unified_llm_profiler_skill/runs/example/sglang_profile_live \
  --num-steps 5 \
  --warmup-steps 10 \
  --profile-by-stage \
  --profile-workload both

The script sends POST /start_profile to the SGLang server directly. Keep --output-dir under /data/... so later analysis and docs can see the trace. The script writes server_args.json, warms up with the same workload shape, sends the active probe requests after profiling is armed, captures separate prefill/ and decode/ profile roots by default, and waits longer for trace flush than the earlier implementation. For the default workload-separated capture, the directory name labels the stage and the SGLang internal profile_by_stage mode is not used inside each workload. This avoids mixing a one-token prefill probe with a separate decode profile. The helper still adds one internal guard step because older SGLang profilers check the target counter before running the next forward.

3. Single-trace live capture from vLLM

Launch vLLM with torch profiler enabled, for example:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --profiler-config '{"profiler":"torch","torch_profiler_dir":"/data/bbuf/validate/unified_llm_profiler_skill/runs/example/vllm_profile"}'

Then run:

python3 scripts/analyze_llm_torch_profile.py \
  --framework vllm \
  --url http://127.0.0.1:8000 \
  --output-dir /data/bbuf/validate/unified_llm_profiler_skill/runs/example/vllm_profile \
  --num-steps 5 \
  --warmup-steps 10 \
  --no-profile-by-stage \
  --profile-workload both

For vLLM, --output-dir must point to the same torch_profiler_dir the server uses. The current vLLM profiler config already defaults torch_profiler_with_stack=true, so the runner only needs to set torch_profiler_dir. On h100_sglang, external vLLM containers should mount both:

• /data/.cache/huggingface:/root/.cache/huggingface
• /data/bbuf/validate/unified_llm_profiler_skill:/data/bbuf/validate/unified_llm_profiler_skill