The Art of Debugging
stas00/the-art-of-debugging
Condensed debugging method and tool recipes for Unix, Python and PyTorch programs: crashes, hangs, segfaults, wrong output, CUDA OOM, NaN values and slowness.
A skill your agent uses for performance profiling and optimization.
$ npx skills add ByteDance-Seed/VeOmni --skill veomni-profile -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install ByteDance-Seed/VeOmni veomni-profile --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/ByteDance-Seed/VeOmni.git skills-src && mkdir -p .claude/skills && cp -r skills-src/.agents/skills/veomni-profile .claude/skills/veomni-profile && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "veomni-profile" agent skill from https://github.com/ByteDance-Seed/VeOmni/tree/main/.agents/skills/veomni-profile into .claude/skills/veomni-profile/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "veomni-profile", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/ByteDance-Seed/VeOmni/tree/main/.agents/skills/veomni-profileType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add ByteDance-Seed/VeOmni --skill veomni-profile -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install ByteDance-Seed/VeOmni veomni-profile --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/ByteDance-Seed/VeOmni.git skills-src && mkdir -p .agents/skills && cp -r skills-src/.agents/skills/veomni-profile .agents/skills/veomni-profile && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "veomni-profile" agent skill from https://github.com/ByteDance-Seed/VeOmni/tree/main/.agents/skills/veomni-profile into .agents/skills/veomni-profile/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "veomni-profile", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add ByteDance-Seed/VeOmni --skill veomni-profile -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install ByteDance-Seed/VeOmni veomni-profile --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/ByteDance-Seed/VeOmni.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/.agents/skills/veomni-profile .cursor/skills/veomni-profile && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "veomni-profile" agent skill from https://github.com/ByteDance-Seed/VeOmni/tree/main/.agents/skills/veomni-profile into .cursor/skills/veomni-profile/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "veomni-profile", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/ByteDance-Seed/VeOmni.git --path .agents/skills/veomni-profile--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add ByteDance-Seed/VeOmni --skill veomni-profile -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install ByteDance-Seed/VeOmni veomni-profile --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/ByteDance-Seed/VeOmni.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/.agents/skills/veomni-profile .gemini/skills/veomni-profile && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "veomni-profile" agent skill from https://github.com/ByteDance-Seed/VeOmni/tree/main/.agents/skills/veomni-profile into .gemini/skills/veomni-profile/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "veomni-profile", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install ByteDance-Seed/VeOmni veomni-profileInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add ByteDance-Seed/VeOmni --skill veomni-profile -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/ByteDance-Seed/VeOmni.git skills-src && mkdir -p .github/skills && cp -r skills-src/.agents/skills/veomni-profile .github/skills/veomni-profile && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "veomni-profile" agent skill from https://github.com/ByteDance-Seed/VeOmni/tree/main/.agents/skills/veomni-profile into .github/skills/veomni-profile/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "veomni-profile", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add ByteDance-Seed/VeOmni --skill veomni-profile -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install ByteDance-Seed/VeOmni veomni-profile --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/ByteDance-Seed/VeOmni.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/.agents/skills/veomni-profile .opencode/skills/veomni-profile && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "veomni-profile" agent skill from https://github.com/ByteDance-Seed/VeOmni/tree/main/.agents/skills/veomni-profile into .opencode/skills/veomni-profile/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "veomni-profile", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
veomni-profileA skill your agent uses for performance profiling and optimization.
Veomni Profile is an agent skill from ByteDance-Seed/VeOmni. Use this skill for performance profiling and optimization. Two modes: (1) Analyze existing profile files (Chrome traces, memory snapshots) — write scripts to parse and summarize metrics per user requirements. (2) Generate profiles during development — configure ProfileConfig, run training, collect traces, analyze bottlenecks, and suggest optimizations. Trigger: 'profile', 'performance', 'slow', 'MFU', 'throughput', 'bottleneck', 'memory usage', 'trace', 'optimize training speed'.
Its SKILL.md is about 1.7k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.
It sits in Development, covering Performance optimization. It works with CUDA. The repository describes itself as: VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo. The licence is Apache-2.0.
5 steps, taken from the step headings in SKILL.md.
Read from SKILL.md and the folder at commit 8791a71. It shows what the files ask for, not the result of running them.
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Shell commands in SKILL.md call:
pythonFrom the folder's file list and the shell code blocks in SKILL.md.
No URLs in SKILL.md.
From URLs in SKILL.md, links to its own repository left out.
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Veomni Profile loads about 1.7k tokens when it runs. Until then it costs about 125 tokens; SKILL.md has 567 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
The full file from ByteDance-Seed/VeOmni at commit 8791a71, republished under its Apache-2.0 licence (© ByteDance-Seed). 567 words, ~1,743 tokens.
.claude/skills/veomni-profile/SKILL.md (or your agent's skills folder).Key components:
| Component | Location | Purpose |
|---|---|---|
ProfileConfig | veomni/arguments/arguments_types.py | Config fields: enable, start_step, end_step, trace_dir, profile_memory, with_stack, etc. |
create_profiler() | veomni/utils/helper.py | Builds torch.profiler.profile (CUDA) or torch_npu.profiler (NPU) with schedule |
ProfileTraceCallback | veomni/trainer/callbacks/trace_callback.py | Integrates profiler into the training loop via BaseTrainer |
VeomniFlopsCounter | veomni/utils/count_flops.py | Analytical FLOPs/MFU computation per model family |
EnvironMeter | veomni/utils/helper.py | Step-level throughput metrics (tokens/s, FLOPs, MFU) |
merge_chrome_trace.py | scripts/profile/merge_chrome_trace.py | Merge multi-rank Chrome traces for unified viewing |
Output formats:
veomni_rank{R}_{timestamp}.pt.trace.json.gz — viewable in chrome://tracing or Perfetto.pkl file via torch.cuda.memory._dump_snapshot — viewable with PyTorch Memory VizUser provides one or more profile files (Chrome traces, memory snapshots, logs). Write scripts to parse and analyze them.
Identify file types: .json.gz / .json (Chrome trace), .pkl (memory snapshot), .log / .txt (training logs with throughput metrics).
Understand the analysis goal — ask the user what they want to know:
Write an analysis script using torch.profiler APIs or raw JSON parsing:
import json, gzip
from collections import defaultdict
def load_chrome_trace(path):
opener = gzip.open if path.endswith('.gz') else open
with opener(path, 'rt') as f:
return json.load(f)
def analyze_kernel_time(trace):
"""Group events by kernel name, sum durations."""
kernel_times = defaultdict(float)
for event in trace.get('traceEvents', []):
if event.get('cat') == 'kernel':
kernel_times[event['name']] += event.get('dur', 0)
return sorted(kernel_times.items(), key=lambda x: -x[1])Adapt the script to the user's specific analysis goal. Output tables, summaries, or CSV for further processing.
For multi-rank traces: use scripts/profile/merge_chrome_trace.py to merge before analysis, or analyze per-rank and compare.
For memory snapshots: load with pickle, analyze allocation records, identify peak usage and largest tensors.
Present findings: summarize top bottlenecks, compute/comm ratio, and actionable optimization suggestions.
Actively profile a training run to identify performance bottlenecks or validate optimizations.
Add or modify the profile section in the training YAML config:
train:
profile:
enable: true
start_step: 5 # skip warmup steps
end_step: 10 # capture 5 steps
trace_dir: ./profile_output
record_shapes: true
profile_memory: true # enable memory snapshot (CUDA only)
with_stack: true # capture Python call stacks
with_modules: true # annotate with nn.Module names
rank0_only: true # profile only rank 0 to reduce overheadOr pass via CLI overrides: --train.profile.enable=true --train.profile.start_step=5 ...
source .venv/bin/activate
# Single GPU
python tasks/train_text.py --config configs/text/<model>.yaml
# Multi-GPU (profile will capture per-rank traces)
torchrun --nproc_per_node=8 tasks/train_text.py --config configs/text/<model>.yamlLocate outputs in trace_dir:
veomni_rank*_.pt.trace.json.gz — Chrome traceveomni_rank*_.pkl — memory snapshot (if profile_memory: true)Write analysis scripts as in Mode 1 to extract the metrics the user needs.
Quick analysis shortcuts:
cat == 'kernel'nccl (e.g. ncclAllReduceRingLLKernel)with_modules trace annotations to separate phases.pkl snapshot, find max allocated_bytesEnvironMeter already logs flops_achieved and flops_promised — grep training logsFor multi-rank comparison: merge traces with scripts/profile/merge_chrome_trace.py or analyze per-rank to find stragglers.
Based on findings, suggest and implement optimizations:
| Bottleneck | Typical solutions |
|---|---|
| Attention kernels dominate | Switch to FlashAttention 3/4 (veomni/ops/kernels/attention/), check FA is actually active |
| NCCL communication > 30% | Increase compute/comm overlap, adjust FSDP reshard policy, try async SP |
| Memory OOM / high peak | Enable activation checkpointing, reduce micro-batch size, check for memory leaks |
| Data loading stalls | Increase num_workers, enable prefetch, check I/O throughput |
| Low MFU (< 40%) | Check dtype (bf16 vs fp32), verify tensor cores are used, check for host-device syncs |
| Uneven per-rank time | Check MoE load balancing, verify data distribution across ranks |
After optimization:
On NPU, create_profiler() uses torch_npu.profiler instead of torch.profiler. Key differences:
is_torch_npu_available().© ByteDance-Seed, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Just SKILL.md in .agents/skills/veomni-profile of ByteDance-Seed/VeOmni.
Open the folder on GitHubat commit 8791a71
Veomni Profile next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Veomni Profile this skillByteDance-Seed/VeOmni | 2.2k | — | ~1.7k | Automated safety check: Pass | Apache-2.0 | |
| The Art of Debuggingstas00/the-art-of-debugging | 1.7k | — | ~6.1k | Automated safety check: Notes | CC-BY-SA-4.0 | |
| Cudatechnillogue/ptx-isa-markdown | 229 | — | ~2.5k | Automated safety check: Pass | None | |
| Torch Profiler Layer TrackBBuf/AI-Infra-Auto-Driven-SKILLS | 938 | — | ~2k | Automated safety check: Pass | None | |
| Cppcrazyguitar/cppcheatsheet | 290 | — | ~1.8k | Automated safety check: Pass | MIT | |
| LLM Torch Profiler Trace AnalysisBBuf/AI-Infra-Auto-Driven-SKILLS | 938 | — | ~2.8k | Automated safety check: Pass | None |
stas00/the-art-of-debugging
Condensed debugging method and tool recipes for Unix, Python and PyTorch programs: crashes, hangs, segfaults, wrong output, CUDA OOM, NaN values and slowness.
technillogue/ptx-isa-markdown
CUDA kernel development, debugging, and performance optimization for Claude Code.
BBuf/AI-Infra-Auto-Driven-SKILLS
Adds verified layer guides such as L0 and L1 and compact GPU lanes to an existing Torch Profiler Chrome trace, changing how it looks but not how it ran.
crazyguitar/cppcheatsheet
Comprehensive C/C++ programming reference covering everything from C11-C23 and C++11-C++23, system programming, CUDA GPU computing, debugging tools, Rust interop, and advanced topics.
BBuf/AI-Infra-Auto-Driven-SKILLS
Analyzes Torch Profiler traces from SGLang, vLLM and TensorRT-LLM servers into kernel attribution, overlap and fusion tables.
slowlyC/agent-gpu-skills
Write, debug, and optimize TileLang kernels from local upstream language, JIT, autotuning, profiling, compiler, test, and example source.
ByteDance-Seed/VeOmni
Create a pull request for the current branch. An agent skill from ByteDance-Seed/VeOmni.
ByteDance-Seed/VeOmni
A skill your agent uses for ANY bug, error, crash, wrong output, loss divergence, gradient explosion, test failure, CUDA error, distributed training hang, checkpoint load failure, or unexpected…
ByteDance-Seed/VeOmni
A skill your agent uses when adding support for a new model to VeOmni.
ByteDance-Seed/VeOmni
A skill your agent uses when adding a new optimized kernel or operator to veomni/ops/.
ByteDance-Seed/VeOmni
Author or refresh a VeOmni model's patchgen-generated modeling under generated/ — GPU and/or NPU config, dense or MoE, text / VLM / Omni.
ByteDance-Seed/VeOmni
Pre-PR code review gate. An agent skill from ByteDance-Seed/VeOmni.
Works with
Categories
A skill your agent uses for performance profiling and optimization. Veomni Profile is an agent skill from ByteDance-Seed/VeOmni. Use this skill for performance profiling and optimization.
Veomni Profile fits situations like: performance profiling and optimization; tasks that involve Performance optimization.
Run `npx skills add ByteDance-Seed/VeOmni --skill veomni-profile -a claude-code`. Or copy the skill folder (.agents/skills/veomni-profile in ByteDance-Seed/VeOmni) into .claude/skills/veomni-profile in your project. Claude Code loads it when a task matches its description.
Run `npx skills add ByteDance-Seed/VeOmni --skill veomni-profile -a codex`. Or copy the skill folder (.agents/skills/veomni-profile in ByteDance-Seed/VeOmni) into .agents/skills/veomni-profile in your project. Codex loads it when a task matches its description.
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add ByteDance-Seed/VeOmni --skill veomni-profile -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/veomni-profile, .gemini/skills/veomni-profile, .github/skills/veomni-profile and .opencode/skills/veomni-profile in your project.
Going by SKILL.md and its folder, Veomni Profile needs the command-line tools its instructions call (python). Our summary lists: Python 3.
SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
Veomni Profile is published under the Apache-2.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
About 1.7k tokens (SKILL.md is roughly 7k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
Skills that share tags, products or a category with Veomni Profile: The Art of Debugging (stas00/the-art-of-debugging, 1.7k stars), Cuda (technillogue/ptx-isa-markdown, 229 stars), Torch Profiler Layer Track (BBuf/AI-Infra-Auto-Driven-SKILLS, 938 stars) and Cpp (crazyguitar/cppcheatsheet, 290 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
ByteDance-Seed (a GitHub organization) maintains it in ByteDance-Seed/VeOmni, which has 2,235 GitHub stars. The repository holds 10 skills in this directory. The repository was last updated on October 10, 2026.
Source: ByteDance-Seed/VeOmni on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.