Warp Compile Time Optimizer

Warp cold-start compile time

Start with a runnable command

The probe runs the target in a subprocess, writes only to temporary directories, and needs no network, external tool servers, or Warp checkout.

Use run_script("scripts/warp_compile_probe.py", args=[...]) when supported; otherwise use the Python command below. The target must run to completion.

Compilation model

Warp compiles modules, not individual kernels. A module's identity is:

(live kernel & function set) x (module options) x (CUDA block_dim) x (generic instances)

Each identity requires code generation and native compilation for the full module.

Cold-start cost is roughly:

number of distinct module identities you touch  x  size of each module

Reduce it in two ways:

1. Stop identity churn. A module that changes after loading compiles again.
2. Stop source duplication. A module used at three block dimensions compiles every kernel three times.

Deleting one kernel from a module that still builds saves only part of one build. Removing an unnecessary module identity saves the full build.

When neither applies, overlap independent CUDA builds (CS-13). This changes when work happens, not how much is compiled, so judge it on elapsed time.

When a module's options are fixed

Options have two deadlines:

1. Module creation (Python import). The module copies enable_backward, max_unroll, lineinfo, deterministic, deterministic_max_records, and compile_time_trace from warp.config. Setting a global later is silently ignored by that module. default_grid_stride is the exception.
2. First load. Before compilation, change the existing module with wp.set_module_options() or wp.get_module(name).options. Changing an option after load creates a new identity and rebuilds the module (CS-3).

After changing an option, confirm the hash moved for every target module. An unchanged hash means the option never arrived.

Preserve behavior

Change how Warp compiles the code, not the workload.

Do not delete or merge kernels to claim a gain. Apparently redundant stages may preserve ownership, aliasing, retained outputs, numerical boundaries, or API behavior. Fix duplication at the module level.

Preserve every launch and its order, dimensions, dtypes, devices, block dimensions, gradients, numerical modes, dynamic/plugin behavior, and public API signatures. Keep kernel names when moving definitions to module scope because logs, cache artifacts, and external tools expose them.

Instructions

1. Find the real cost, and confirm it is compilation

Ask what command the user actually waits on, then measure it cold:

python scripts/warp_compile_probe.py measure --samples 3 \
    --json baseline.json -- <the user's command>

The probe gives each sample private WARP_CACHE_PATH, WARP_CACHE_ROOT, and CUDA_CACHE_PATH directories, enables module timers, and records launches. It creates those directories under the system temporary directory and removes them itself, so isolating a sample never requires writing a cache into the project or deleting anything to re-measure. Isolate a hand-rolled sample the same way. Never clear a live cache with wp.clear_kernel_cache() or wp.clear_lto_cache(); clearing is not isolated and can disrupt other processes.

Read the probe output before source. If compilation is a small part of wall time, report the real bottleneck and stop. For libraries and tests, use the smallest command that compiles the workload's modules.

Modules that each compiled once, with no repeated hashes, block-dimension variants, or LTO, have no structural churn. This rules out redundant builds, not oversized builds; still check cache reuse (CS-2), backward codegen (CS-10), unrolling (CS-11), the precompiled header (CS-12), and overlap when several CUDA modules remain (CS-13).

Every sample also re-runs the command against the cache it just populated. If warm module work is not near zero, diagnose cache reuse (CS-2) before changing module structure.

2. Ask about runtime tradeoffs when needed

Apply ordering fixes, lifecycle grouping, and option hoists without asking. Ask before changing fast_math, max_unroll, or a MathDx/tile implementation:

Some of these knobs cut compile time but can make the compiled kernels slower or change numerics. Are you optimizing a fast edit-run loop (where slower kernels are usually fine), or production startup (where they usually are not)?

If the user is unavailable:

• Leave numerics-changing options and implementation swaps alone.
• Before removing a capability such as backward codegen, test whether it is used; report what is removed, the measured benefit, and how to revert.
• Set global wp.config.* options at application entry points, not in library code.

Record declined options and their measured benefits in the step 6 ledger.

Match the scope of the change to the scope of the evidence

A profile supports a change to the measured application, not every consumer of a shared library. Repository searches also miss out-of-tree and future callers. For example, a forward-only application does not justify disabling gradients inside a solver library that another application differentiates through.

Scope the option to the measured process, before importing the library:

import warp as wp

wp.config.enable_backward = False   # must precede the library import

import the_library

This also reaches every module the application loads. CS-10 covers the silent import-order trap. If only a library change works, send its maintainers the measurement and let them decide the contract.

3. Diagnose from the measurement, not from reading the source

The probe prints every compiled module identity with its name, hash, device, and block dimension, then names which modules built more than once. Match what you see:

references/mechanisms.md has one section per mechanism: how to confirm it, the fix, its limits, and its failure mode. Read only the sections selected by the measurement.