Debug Failing GPU

What it does

When your agent uses it

• A command (pytest
• A TLX/Triton kernel run
• A benchmark) fails with errors indicating the GPU is busy
• Unavailable — e.g

Example prompts

• “CUDA error: out of memory”
• “all CUDA-capable devices are busy or unavailable”
• “CUDA-capable device(s) is/are busy or unavailable”
• “/debug-failing-gpu”

Requirements

• Python 3

Workflow steps

4 steps, taken from the first numbered list in SKILL.md.

1. 1Run the scanner
2. 2Read the final line, WORKING_GPUS=... (e.g. WORKING_GPUS=0,2,3). These are
3. 3Pick the first working index. If several are free, mention them so the user can
4. 4Re-issue the original failing command with CUDA_VISIBLE_DEVICES=

What it can do on your machine

Read from SKILL.md and the folder at commit 6f3dd70. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Shell commands in SKILL.md call:

  • bash
  • pytest

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  No URLs in SKILL.md.

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Debug Failing GPU loads about 709 tokens when it runs. Until then it costs about 144 tokens; SKILL.md has 247 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~144

When it runs · the whole SKILL.md, loaded when a task matches

~709

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from facebookexperimental/triton at commit 6f3dd70, republished under its MIT licence (© facebookexperimental). 247 words, ~709 tokens.

name

debug-failing-gpu

description

Recover from GPU-busy / GPU-unavailable failures. Use when a command (pytest, python, a TLX/Triton kernel run, a benchmark) fails with errors indicating the GPU is busy, out of memory, or unavailable — e.g. "CUDA error: out of memory", "all CUDA-capable devices are busy or unavailable", "CUDA-capable device(s) is/are busy or unavailable", "RuntimeError: No CUDA GPUs are available", "device-side assert", or a hang on the first CUDA call. Runs find_working_gpu.sh to locate a healthy GPU and re-runs the failed command pinned to it via CUDA_VISIBLE_DEVICES.

# No device set -> prepend
pytest third_party/tlx/tutorials/testing/test_correctness.py
# becomes
CUDA_VISIBLE_DEVICES=2 pytest third_party/tlx/tutorials/testing/test_correctness.py

# Device already set -> replace, don't stack
CUDA_VISIBLE_DEVICES=4 third_party/tlx/denoise.sh python bench.py
# becomes
CUDA_VISIBLE_DEVICES=2 third_party/tlx/denoise.sh python bench.py

© facebookexperimental, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

Just SKILL.md in .claude/skills/debug-failing-gpu of facebookexperimental/triton.

Open the folder on GitHubat commit 6f3dd70