Debug Distributed Hang
sgl-project/sglang
Debug hanging issues in SGLang distributed inference (TP/PP/DP/EP).
HIP and ROCm skill for AMD GPU programming. An agent skill from mohitmishra786/low-level-dev-skills.
$ npx skills add mohitmishra786/low-level-dev-skills --skill hip-rocm -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install mohitmishra786/low-level-dev-skills hip-rocm --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/mohitmishra786/low-level-dev-skills.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/gpu/hip-rocm .claude/skills/hip-rocm && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "hip-rocm" agent skill from https://github.com/mohitmishra786/low-level-dev-skills/tree/main/skills/gpu/hip-rocm into .claude/skills/hip-rocm/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "hip-rocm", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/mohitmishra786/low-level-dev-skills/tree/main/skills/gpu/hip-rocmType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add mohitmishra786/low-level-dev-skills --skill hip-rocm -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install mohitmishra786/low-level-dev-skills hip-rocm --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/mohitmishra786/low-level-dev-skills.git skills-src && mkdir -p .agents/skills && cp -r skills-src/skills/gpu/hip-rocm .agents/skills/hip-rocm && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "hip-rocm" agent skill from https://github.com/mohitmishra786/low-level-dev-skills/tree/main/skills/gpu/hip-rocm into .agents/skills/hip-rocm/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "hip-rocm", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add mohitmishra786/low-level-dev-skills --skill hip-rocm -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install mohitmishra786/low-level-dev-skills hip-rocm --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/mohitmishra786/low-level-dev-skills.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/skills/gpu/hip-rocm .cursor/skills/hip-rocm && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "hip-rocm" agent skill from https://github.com/mohitmishra786/low-level-dev-skills/tree/main/skills/gpu/hip-rocm into .cursor/skills/hip-rocm/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "hip-rocm", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/mohitmishra786/low-level-dev-skills.git --path skills/gpu/hip-rocm--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add mohitmishra786/low-level-dev-skills --skill hip-rocm -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install mohitmishra786/low-level-dev-skills hip-rocm --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/mohitmishra786/low-level-dev-skills.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/skills/gpu/hip-rocm .gemini/skills/hip-rocm && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "hip-rocm" agent skill from https://github.com/mohitmishra786/low-level-dev-skills/tree/main/skills/gpu/hip-rocm into .gemini/skills/hip-rocm/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "hip-rocm", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install mohitmishra786/low-level-dev-skills hip-rocmInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add mohitmishra786/low-level-dev-skills --skill hip-rocm -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/mohitmishra786/low-level-dev-skills.git skills-src && mkdir -p .github/skills && cp -r skills-src/skills/gpu/hip-rocm .github/skills/hip-rocm && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "hip-rocm" agent skill from https://github.com/mohitmishra786/low-level-dev-skills/tree/main/skills/gpu/hip-rocm into .github/skills/hip-rocm/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "hip-rocm", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add mohitmishra786/low-level-dev-skills --skill hip-rocm -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install mohitmishra786/low-level-dev-skills hip-rocm --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/mohitmishra786/low-level-dev-skills.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/skills/gpu/hip-rocm .opencode/skills/hip-rocm && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "hip-rocm" agent skill from https://github.com/mohitmishra786/low-level-dev-skills/tree/main/skills/gpu/hip-rocm into .opencode/skills/hip-rocm/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "hip-rocm", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
hip-rocmHIP and ROCm skill for AMD GPU programming. An agent skill from mohitmishra786/low-level-dev-skills.
Hip Rocm is an agent skill from mohitmishra786/low-level-dev-skills. HIP and ROCm skill for AMD GPU programming. Use when writing HIP kernels with hipcc, porting CUDA code via HIPIFY, profiling with rocprof, debugging with rocgdb, or optimizing for MI300X. Activates on queries about HIP, ROCm, hipify, hipcc, rocprof, or CUDA to AMD porting.
Its SKILL.md is about 1.6k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.
It sits in Development, covering Debugging and GPU and accelerator computing. It works with CUDA. The repository describes itself as: A curated suite of AI agent skills for systems and low-level programming with C/C++, Rust, and Zig toolchains, covering compilers, debuggers, profilers, build systems…. The licence is MIT.
8 steps, taken from the step headings in SKILL.md.
Read from SKILL.md and the folder at commit bdc5847. It shows what the files ask for, not the result of running them.
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Shell commands in SKILL.md call:
aptFrom the folder's file list and the shell code blocks in SKILL.md.
No URLs in SKILL.md.
From URLs in SKILL.md, links to its own repository left out.
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Hip Rocm loads about 1.6k tokens when it runs. Until then it costs about 71 tokens; SKILL.md has 361 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
The automated check noted patterns worth knowing about, such as sudo or a known installer.
sudo apt install rocm-dev rocm-libs hip-devAutomated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
The full file from mohitmishra786/low-level-dev-skills at commit bdc5847, republished under its MIT licence (© mohitmishra786). 361 words, ~1,581 tokens.
.claude/skills/hip-rocm/SKILL.md (or your agent's skills folder).Guide agents through AMD GPU programming with HIP: the HIP runtime API, hipcc compilation, porting CUDA code with HIPIFY (hipify-perl, hipify-clang), ROCm toolchain setup, profiling with rocprof, debugging with rocgdb, HIP-vs-CUDA API mapping, and MI300X-specific optimizations.
# Ubuntu/Debian (check ROCm docs for your distro version)
sudo apt install rocm-dev rocm-libs hip-dev
# Verify
rocminfo | head -30
hipconfig --version
hipcc --version
# List devices
rocm-smiSet GPU target for compilation:
export AMDGPU_TARGETS=gfx942 # MI300X
export HIP_PLATFORM=amd// vector_add.hip
#include <hip/hip_runtime.h>
#include <stdio.h>
__global__ void vector_add(const float *a, const float *b, float *c, int n) {
int i = blockIdx.x * blockDim.x + threadIdx.x;
if (i < n)
c[i] = a[i] + b[i];
}
int main(void) {
const int n = 1 << 20;
size_t bytes = n * sizeof(float);
float *d_a, *d_b, *d_c;
hipMalloc(&d_a, bytes);
hipMalloc(&d_b, bytes);
hipMalloc(&d_c, bytes);
int threads = 256;
int blocks = (n + threads - 1) / threads;
hipLaunchKernelGGL(vector_add, dim3(blocks), dim3(threads), 0, 0,
d_a, d_b, d_c, n);
hipDeviceSynchronize();
hipFree(d_a); hipFree(d_b); hipFree(d_c);
return 0;
}hipcc -O3 --offload-arch=gfx942 -o vector_add vector_add.hip
./vector_add# Perl-based batch converter (quick port)
hipify-perl cuda_kernel.cu > cuda_kernel.hip
# Clang-based (more accurate, preserves structure)
hipify-clang cuda_project/ -o hip_project/ --cuda-path=/usr/local/cuda
# Convert single file in place
hipify-clang -inplace --cuda-path=/usr/local/cuda main.cuCommon API mappings:
| CUDA | HIP |
|---|---|
cudaMalloc | hipMalloc |
cudaMemcpy | hipMemcpy |
cudaMemcpyAsync | hipMemcpyAsync |
cudaStream_t | hipStream_t |
<<<grid, block>>> | hipLaunchKernelGGL or <<<>>> (HIP supports CUDA syntax) |
__syncthreads() | __syncthreads() (same) |
threadIdx / blockIdx | Same builtins |
Portability header for dual compilation:
#ifdef __HIP_PLATFORM_AMD__
#include <hip/hip_runtime.h>
#else
#include <cuda_runtime.h>
#define hipMalloc cudaMalloc
#define hipMemcpy cudaMemcpy
// ... more macros
#endif# Target specific GPU architecture
hipcc --offload-arch=gfx942 -O3 -o app main.hip
# Multiple architectures
hipcc --offload-arch=gfx90a --offload-arch=gfx942 -o app main.hip
# Debug
hipcc -g -O0 --offload-arch=gfx942 -o app_debug main.hip
# Link with rocBLAS
hipcc -lrocblas -o app main.hip# Basic kernel trace
rocprof --stats ./app
# CSV metrics output
rocprof -i input.csv -o output.csv ./app
# input.csv example:
# pmc: SQ_INSTS_VALU_ADD_F32,SQ_INSTS_VALU_MUL_F32,GRBM_COUNT# ROCm 6.x rocprofiler-sdk (preferred for new projects)
rocprofv3 --kernel-trace -- ./appKey metrics (AMD terminology):
# Build with debug symbols
hipcc -g -O0 --offload-arch=gfx942 -o app_debug main.hip
rocgdb ./app_debug(rocgdb) break vector_add
(rocgdb) run
(rocgdb) info rocm kernels
(rocgdb) rocm thread 0 0 0
(rocgdb) print iAMD also supports compute-sanitizer equivalents via ROCm's roc-obj-extract and memory checking tools where available.
# Enable MFMA (matrix fused multiply-add) instructions
hipcc --offload-arch=gfx942 -munsafe-fp-atomics -O3 -o app main.hip| Optimization | MI300X note |
|---|---|
| Matrix ops | Use rocBLAS/hipBLASLt for GEMM; MFMA intrinsics for custom |
| HBM bandwidth | ~5.3 TB/s peak (MI300X) — maximize memory coalescing to approach it |
| Wavefront size | 64 threads (vs CUDA warp 32) — adjust reduction patterns |
| LDS (shared mem) | 64 KB per CU; watch bank conflicts |
Wavefront-aware reduction:
__device__ float warp_reduce_sum(float val) {
// AMD wavefront = 64 lanes
for (int offset = 32; offset > 0; offset >>= 1)
val += __shfl_down(val, offset);
return val;
}| NVIDIA | AMD ROCm |
|---|---|
| cuBLAS | rocBLAS / hipBLAS |
| cuDNN | MIOpen |
| NCCL | rccl |
| Thrust | hipCUB (portable) |
| cuFFT | rocFFT |
hipcc -lrocblas -o gemm_test gemm.hip| Symptom | Cause | Fix |
|---|---|---|
hipErrorNoDevice | ROCm driver not loaded | Check rocm-smi; add user to render group |
| Wrong architecture binary | Mismatched gfx* target | rocminfo → set --offload-arch |
| hipify incomplete port | CUDA-specific APIs | Manual fix: cooperative groups, texture refs |
| Slower than CUDA reference | Wavefront 64 vs warp 32 | Tune block size to multiples of 64 |
HSA_STATUS_ERROR | GPU busy or OOM | rocm-smi --showmeminfo; reduce allocation |
| rocprof empty output | No kernels launched | Verify hipGetLastError() after launch |
skills/gpu/cuda — source CUDA patterns being portedskills/gpu/cuda-profiling — Nsight concepts map to rocprofskills/gpu/gpu-memory-model — wavefront vs warp, coalescing rulesskills/gpu/triton-lang — Triton supports AMD via ROCm backendskills/compilers/llvm — HIP uses Clang/LLVM toolchain© mohitmishra786, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Just SKILL.md in skills/gpu/hip-rocm of mohitmishra786/low-level-dev-skills.
Open the folder on GitHubat commit bdc5847
Hip Rocm next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Hip Rocm this skillmohitmishra786/low-level-dev-skills | 252 | — | ~1.6k | Automated safety check: Notes | MIT | |
| Debug Distributed Hangsgl-project/sglang | 37k | 2 repos | ~2.4k | Automated safety check: Pass | Apache-2.0 | |
| CUTLASS FMHA Incremental Rebuildmicrosoft/onnxruntime | 22k | — | ~1.3k | Automated safety check: Pass | MIT | |
| Cudatechnillogue/ptx-isa-markdown | 229 | — | ~2.5k | Automated safety check: Pass | None | |
| Cuda Cpp Kernelvipshop/cache-dit | 1.3k | — | ~2.3k | Automated safety check: Pass | Apache-2.0 | |
| Aoti Debugpytorch/pytorch | 104k | 1 repos | ~1.7k | Automated safety check: Pass | Custom licence |
sgl-project/sglang
Debug hanging issues in SGLang distributed inference (TP/PP/DP/EP).
microsoft/onnxruntime
Explains why editing CUTLASS fused-MHA headers in ONNX Runtime can leave stale CUDA kernels after an incremental build, and how to force and verify a real rebuild.
technillogue/ptx-isa-markdown
CUDA kernel development, debugging, and performance optimization for Claude Code.
vipshop/cache-dit
A skill your agent uses when writing, debugging, porting, reviewing, or optimizing CUDA C++ or PTX kernels; investigating CUDA Runtime or Driver API behavior; profiling kernels with Nsight Systems…
pytorch/pytorch
Debug AOTInductor (AOTI) errors and crashes. An agent skill from pytorch/pytorch.
stas00/the-art-of-debugging
Condensed debugging method and tool recipes for Unix, Python and PyTorch programs: crashes, hangs, segfaults, wrong output, CUDA OOM, NaN values and slowness.
mohitmishra786/low-level-dev-skills
Guides reading and writing AArch64 and ARM Thumb assembly: compiler output, inline asm, registers, the AAPCS calling convention and NEON or SVE basics.
mohitmishra786/low-level-dev-skills
Reference for RISC-V assembly on RV32 and RV64: register names and calling convention, extension naming, GCC and Clang inline asm, and QEMU with GDB debugging.
mohitmishra786/low-level-dev-skills
Explains x86-64 registers, the System V AMD64 calling convention, and how to read compiler-generated or inline assembly.
mohitmishra786/low-level-dev-skills
Guides your agent through Bazel for C/C++ projects: BUILD files, Bzlmod dependencies, toolchain registration, remote execution, dependency queries and sandbox debugging.
mohitmishra786/low-level-dev-skills
Binary hardening skill for security-hardened C/C++ builds. An agent skill from mohitmishra786/low-level-dev-skills.
mohitmishra786/low-level-dev-skills
GNU binutils skill for binary manipulation and analysis. An agent skill from mohitmishra786/low-level-dev-skills.
Works with
Categories
HIP and ROCm skill for AMD GPU programming. An agent skill from mohitmishra786/low-level-dev-skills. Hip Rocm is an agent skill from mohitmishra786/low-level-dev-skills. HIP and ROCm skill for AMD GPU programming.
Hip Rocm fits situations like: writing HIP kernels with hipcc; porting CUDA code via HIPIFY; profiling with rocprof; debugging with rocgdb.
Run `npx skills add mohitmishra786/low-level-dev-skills --skill hip-rocm -a claude-code`. Or copy the skill folder (skills/gpu/hip-rocm in mohitmishra786/low-level-dev-skills) into .claude/skills/hip-rocm in your project. Claude Code loads it when a task matches its description.
Run `npx skills add mohitmishra786/low-level-dev-skills --skill hip-rocm -a codex`. Or copy the skill folder (skills/gpu/hip-rocm in mohitmishra786/low-level-dev-skills) into .agents/skills/hip-rocm in your project. Codex loads it when a task matches its description.
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add mohitmishra786/low-level-dev-skills --skill hip-rocm -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/hip-rocm, .gemini/skills/hip-rocm, .github/skills/hip-rocm and .opencode/skills/hip-rocm in your project.
Going by SKILL.md and its folder, Hip Rocm needs the command-line tools its instructions call (apt).
SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.
Our automated static check of SKILL.md found notes only (runs commands with sudo), nothing it rates as a warning. It is not a guarantee. Review the folder before installing.
Hip Rocm is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
About 1.6k tokens (SKILL.md is roughly 6.3k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
Skills that share tags, products or a category with Hip Rocm: Debug Distributed Hang (sgl-project/sglang, 37k stars), CUTLASS FMHA Incremental Rebuild (microsoft/onnxruntime, 22k stars), Cuda (technillogue/ptx-isa-markdown, 229 stars) and Cuda Cpp Kernel (vipshop/cache-dit, 1.3k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
mohitmishra786 (a GitHub user) maintains it in mohitmishra786/low-level-dev-skills, which has 252 GitHub stars. The repository holds 138 skills in this directory. The repository was last updated on June 27, 2026.
Source: mohitmishra786/low-level-dev-skills on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.