GPU Optimization

GPU Optimization Pipeline

Workflow for profiling and optimizing CUDA kernels in libuipc using uipc.profile + uipc.profile.nsight + Nsight Compute CLI (ncu).

For building and installing, see cmake-workflow.

Build type: Always use Release for benchmarking and profiling. Debug is too slow; RelWithDebInfo adds debug info overhead that skews results.

Agent Rules

• Do NOT read .ncu-rep files — they are binary, for Nsight Compute GUI only.
• Do NOT benchmark or profile with Debug builds — results are meaningless.
• Do NOT optimize kernels in stages that take <5% of total frame time.
• Always read timer_frames.json for the actual timer hierarchy — it is dynamic and varies per scene/run.

CLI Commands

# List available benchmark scenes
python -m uipc.cli.benchmark list

# Run a benchmark (headless, timer-based); --scene accepts multiple names
python -m uipc.cli.benchmark run --scene <scene> [<scene2> ...] --frames <N> --output <dir>

# Profile with Nsight Compute (kernel-level GPU metrics)
python -m uipc.cli.benchmark profile --scene <scene> --frames <N> --output <dir> --count <K> --skip <S> --ncu-set <set>

# Analyze bottlenecks from a benchmark result directory
python -m uipc.cli.benchmark analyze <result_dir> --ncu-csv <csv_path>

# Compare two benchmark result directories
python -m uipc.cli.benchmark compare <before_dir> <after_dir> --output <out_dir>

profile flags:

• --ncu-set default — basic metrics (SM%, occupancy, registers). Use full for duration/memory but requires admin/root.
• --count <K> — profile first K kernel launches.
• --skip <S> — skip initial warmup/init kernels.
• --ncu-path <path> — explicit path to ncu executable. Auto-detected via NCU_PATH env var, PATH, or default NVIDIA install locations.

Note: The CLI profile defaults to --ncu-set default. The Python API nsight.run() defaults to ncu_set='full'. Be explicit to avoid confusion.

Agent Workflow

Step 1: Run Benchmark + Profile

Run both run (for SimulationStats timer data) and profile (for Nsight Compute kernel metrics). Both are needed to make good optimization decisions.

Step 2: Read Both Reports

Benchmark report (report/report.md from run):

• Shows wall-clock time per simulation stage as a hierarchical tree (e.g., Newton Iteration: 45%, Line Search: 30%)
• Tells you which stages dominate total frame time — this is where optimization has the most impact

Nsight Compute report (<scene>_report.md / <scene>_report.json from profile):

• Shows per-kernel GPU metrics (duration, SM%, occupancy, registers)
• Tells you which kernels are inefficient and why (high register usage, low occupancy, memory-bound, etc.)

Step 3: Cross-Reference to Prioritize

A kernel is worth optimizing only if it's both inefficient AND in a hot stage.

1. Read timer_frames.json or report/report.md to find which stages take the most wall-clock time (e.g., Detect DCD Candidates: 40%).
2. Read the ncu report to find which kernels in those stages have bad metrics.
3. Match kernel names to stages using the class/function names (e.g., StacklessBVH::* -> collision detection stage). Use the source_hint field in the JSON report.
4. Ignore kernels with bad metrics in stages that take <5% of frame time — not worth optimizing.
5. Prioritize kernels that are both in a hot stage AND have clear inefficiencies (high registers, low occupancy).

Step 4: Optimize

1. Follow source_hint in the JSON report to find the CUDA source.
2. Search for the kernel's class/method name in that directory.
3. Read the .cu file and optimize the kernel.
4. Rebuild with Release (see cmake-workflow), re-benchmark, and compare.

Adding Finer-Grained Timers in C++

If the benchmark report shows a hot stage but you need more detail to locate the bottleneck within it, add Timer scopes in the C++ source:

#include <uipc/common/timer.h>

void MySystem::do_something()
{
    {
        Timer timer{"MySystem::phase_A"};
        // ... GPU work ...
    }
    {
        Timer timer{"MySystem::phase_B"};
        // ... GPU work ...
    }
}

The Timer is scoped — it starts on construction and stops on destruction. Nested timers form a tree. The names appear in timer_frames.json and report/report.md, letting you drill down into which sub-phase of a hot stage is the actual bottleneck before running the more expensive ncu profiling.

Simulation Timer Hierarchy

The timer tree is dynamic — it varies per run depending on which simulation features are active (contact, friction, animation, etc.). The actual tree for any run is stored in timer_frames.json (written by run). Always read that file for the real hierarchy.

Below is a representative example with all features enabled, showing the typical nesting from sim_engine_do_advance.cu, global_linear_system.cu, and linear_pcg.cu:

Pipeline                                    engine/
├── Rebuild Scene                           engine/
│   └── Update Diff Parm                    diff_sim/               (conditional)
└── Simulation                              engine/
    ├── Clear External Forces               external_force/         (conditional)
    ├── Step Animation                      animator/               (conditional)
    ├── Compute External Force Accel.       external_force/         (conditional)
    ├── Detect DCD Candidates               collision_detection/    (conditional)
    ├── Newton Iteration                    engine/                 (LOOP)
    │   ├── Detect DCD Candidates           collision_detection/    (iter > 0)
    │   ├── Compute DyTopo Effect           dytopo_effect_system/   (conditional)
    │   │   ├── Assemble Dytopo Effect
    │   │   ├── Convert Dytopo Matrix
    │   │   └── Distribute Dytopo Effect
    │   ├── Solve Global Linear System      linear_system/
    │   │   ├── Build Linear System
    │   │   └── Solve Linear System
    │   │       └── PCG
    │   │           ├── Apply Preconditioner
    │   │           └── SpMV               (per PCG iteration)
    │   └── Line Search                     line_search/
    │       ├── Detect Trajectory Cand.     collision_detection/
    │       ├── Compute Energy              line_search/            (initial E0)
    │       ├── Filter CCD TOI              collision_detection/
    │       ├── Compute CFL Condition       contact_system/
    │       └── Line Search Iteration       line_search/            (LOOP)
    │           ├── Filter Contact Cand.    collision_detection/
    │           └── Compute Energy          line_search/
    └── Update Velocity                     time_integrator/

Parent timer durations include their children. E.g., if Newton Iteration is 80% of frame time, PCG and Line Search durations are already counted inside that 80%.