The Art of Debugging

The Art of Debugging

Distilled from The Art of Debugging Open Book by Stas Bekman - source: https://github.com/stas00/the-art-of-debugging (CC BY-SA 4.0). This skill is a condensed index; each section links back to the full chapter for depth.

Actionable methodology + copy-paste recipes for debugging Unix / Python / PyTorch programs. Apply the general loop first; then jump to the domain cheatsheet for the failure at hand. For scaling this up to large-model training/inference on real clusters (compute/storage/network, SLURM, throughput/memory, instabilities, fault tolerance, inference), pair this with Machine Learning Engineering.

The debugging loop

The single most important idea: most of the effort is in locating the cause; once you truly understand it, the fix is usually easy. Optimize everything for reaching understanding faster.

1. Reproduce reliably. Get one command that triggers the bug every time. If it's flaky, pin the nondeterminism (seeds, ordering, timing, network, uninitialized memory) first - you can't debug what you can't repeat.
2. Shrink the payload. Make the repro fast: fewer layers, tiny model/data, one process, one CPU/GPU, one node. A 2-second repro beats a 2-minute one - you'll run it hundreds of times. See methodology.
3. Localize. Confirm you're editing the code that actually runs (the die trick), then bisect the search space: which commit, which file/function, which line, which input, which rank.
4. Get a usable signal. Turn a cryptic failure into a precise one: a real traceback (sync mode), a stack dump (py-spy/gdb), a syscall trace (strace), a printed value at the boundary, or a min/max/NaN check on a tensor.
5. Change one thing, re-run, verify. Fix on the fast repro, confirm, then re-widen to the full payload. Revert anything that didn't help.

Make the loop fast and reliable

• Atomic debug cycles. Make each iteration a single self-contained, repeatable command (setup + run in one shot) so you never hand-redo multi-step state. See atomic debug cycles.
• Automate diagnostics, minimize typing. Alias the repro and your most-used commands; one keystroke to re-run. See alias frequently used commands and automate diagnostics.
• One-liner programs. perl/awk/python -c to slice logs, extract fields, transform data on the fly instead of writing throwaway scripts. See the power of one-liner programs.
• Juggle configs cleanly when running many debug experiments so results don't get confused. See juggling multiple sets of configs.

Localization techniques

• Am I editing the right file/class? Insert a guaranteed break where you think execution goes; if the program doesn't die, you're in the wrong file/class/env:

  def suspect():
      die   # NameError -> proves this code runs; the traceback also names the caller

  traceback.print_stack() shows callers without stopping (useful when the same function is reached via many paths). See am I editing the right file and the right class?.
• Bisect a regression. git bisect start / bad / good <rev> walks commits automatically to the one that broke things - script the test for git bisect run. See finding a breaking commit by bisecting.
• Small/synthetic payload first. Use tiny or synthetic inputs; switch to real data only when the bug is data-dependent. See real vs random vs synthetic data.
• Race conditions. Reordering/timing bugs hide under async; forcing synchronous execution can expose (or mask) them - note which. See avoiding race conditions and async vs sync mode.

Reproducing resource & environment issues

• Cap resources on purpose to test failure paths: emulate a nearly-full disk, limited CPU RAM, or limited GPU memory. See running out of resources.
• Watch resources live. watch -n1 nvidia-smi / free -h / df -h in a second visible terminal to correlate a hang/OOM with what the machine is doing. See watching and reproducing resource issues.
• Inject sleep to freeze a program at the interesting moment so you can attach a debugger or snapshot state. See uses for sleep.
• HPC/SLURM: keep the allocation and re-run with srun instead of re-sbatch-ing to cut per-iteration overhead. See SLURM salloc and srun fast debug combo.

Unix / shell

Full chapter: Unix Tools for Debugging.

• Make shell scripts fail loudly and traceably:

  set -e          # abort on first error
  set -o pipefail # a failing command anywhere in a pipe fails the whole pipe
  set -u          # abort on undefined variables (catches typos)
  set -x          # trace: print each command with expanded values as it runs
  set +x          # turn tracing back off around a noisy region

  Combine as set -euo pipefail. See controlling script execution.
• strace - trace system calls to see what a program actually does (files, network, why it's stuck):

  strace python -c "print('hi')"                 # trace from the start
  strace --pid PID                               # attach to a running/stuck process
  strace -o log.txt -f torchrun ...  # -f follows forked children
  strace -e trace=open,openat,read python prog.py           # filter to specific syscalls
  strace -e trace=network -p PID                            # is it stuck on a socket?

  Classic use: a process at 100% CPU with no output, or hung on I/O/network. See strace.
• nohup - survive logout/disconnect (don't lose a long run to a dropped SSH):

  nohup ./long-running-command > log.txt &

  See nohup.
• make - after editing compiled sources, rebuild before re-testing, or you'll debug a stale binary. See make.
• Terminal ergonomics: search long scrollback and copy multi-line commands cleanly; keep an informative prompt (host, path, git branch, last exit code) so you always know where/what ran. See shell environment.

Compiled programs (C/C++, extensions, shared libraries)

Full chapter: Debugging Compiled Programs. Compile with -g for debug symbols.

• Segfault -> backtrace from a core file:

  ulimit -c unlimited                                       # allow core dumps in this shell
  sudo sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%h.%t  # control where cores go
  ./program                                                 # crash -> core file written
  gdb ./program /tmp/core-...                               # or: gdb -c core ./program

  At the (gdb) prompt:

  bt                    # backtrace (read bottom-up: outermost caller -> crash site)
  bt full               # + local variable values at each frame
  thread apply all bt   # backtrace for every thread (essential for multithreaded crashes)

  See segmentation fault, core files and gdb.
• No core? Run it under gdb and step to the crash:

  gdb ./program
  (gdb) run            # then: bt / break FILE:LINE / next / step / print VAR / continue

  See run the program under gdb.
• Inspect / snapshot a running process:

  sudo gdb --pid=PID    # attach; then: thread apply all bt
  gcore PID             # force a core dump without killing (or: kill -ABRT PID)

  See get the backtrace from the still running process.
• "symbol not found" / wrong library loaded:

  ldd ./program                             # which shared libs resolve, and to what paths
  LD_LIBRARY_PATH=/path/to/libs ./program   # prepend a search dir
  nm -D libfoo.so | grep symbol             # is the symbol actually exported? (T=defined, U=undefined)
  LD_PRELOAD=/path/to/shim.so ./program     # force-load / override a library

  See debugging shared libraries and symbol resolution (ldd, nm).

Python

Full chapter: Debugging Python Programs.

• Print effectively instead of scattering bare print:
  • auto-print name+value so you never mislabel output. See auto-print what's being observed.
  • dump all attributes of an object to see its real state. See printing object variables.
  • trace calls/returns with q (writes to /tmp/q, doesn't pollute stdout). See q.
• Run the code you think you're running. Edits not taking effect? Wrong copy is imported:

  pip install -e .               # run from the source tree, not a copied install
  PYTHONPATH=src python prog.py  # or point Python straight at the source
  python -c "import pkg; print(pkg.__file__)"   # confirm which file is actually loaded

  See ensuring the Python package you edit is the one that is run and make tests use the git repo's packages.
• Who called this? traceback.print_stack() or the die trick to reveal the caller in complex codebases. See who is calling?.
• Diagnose a hang (process alive but stuck) with py-spy - no code changes, attaches live:

  pip install py-spy
  py-spy dump -n -p PID          # -n also shows native (C/C++ extension) frames
  # all Python subprocesses at once (skip the launcher):
  pgrep -P $(pgrep -o python) | xargs -I {} py-spy dump --pid {}

  No sudo? echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope. The first line of each dump is where it's stuck. See py-spy.
• Slow code -> profile before optimizing (measure, don't guess):

  python -m cProfile -s cumtime prog.py     # what dominates cumulative time
  kernprof -l -v prog.py                     # line_profiler: per-line timing of @profile funcs

  For sub-ms functions, bump pstats precision (e.g. pstats.f8 = lambda x: f"{x:6.3f}") so timings aren't all 0.000. See profilers and cProfile.