Rlt Perf Opt

Looped Transformer Performance Analysis and Optimization

The goal is to answer: how does the feature work, which code or execution stages account for performance changes, how do those changes arise quantitatively, and which further optimizations are worth implementing? Final conclusions must be verifiable from configurations, raw results, traces, and source locations.

This skill covers performance analysis and optimization after feature implementation. Code-style refactoring belongs to a separate workflow; do not mix unrelated refactoring into performance experiments. When the user requests analysis only, deliver measurements and recommendations; implementation changes must stay within the user's authorization.

1. Establish Requirements and Understand the Implementation

Before measuring, use the requirements, PR diff, and actual code to explain:

• What problem the feature solves, which metric it is expected to improve, and what costs are acceptable.
• The main request path from arrival to output, including key computations, scheduling, KV state transitions, and synchronization points.
• How loop depth, exit conditions, and the lifecycle of KV and hidden states at each depth participate in that path.
• Whether the implementation meets the requirements and whether the relevant feature combinations, backends, and hardware are actually supported.

A line-by-line walkthrough is unnecessary, but do not infer behavior from a feature name alone. For unsupported combinations, record the current code version, the source location checked or minimal reproduction results, and mark the combination as untested. Do not combine gains from two independent tests and present them as a result for the combined mode.

2. Fix a Reproducible Test Contract

First read the project's existing benchmarks, user guides, and profiling interfaces, and prefer reusing existing tools. Put any missing diagnostic scripts in a local experiment directory; do not add a new benchmark framework to the repository by default.

Save three types of entry points: service/engine startup commands, single-request reproduction commands, and batch benchmark commands. Associate each experiment with the following information:

• Repository commit and workspace diff; model version, tokenizer, dtype, and quantization method.
• Hardware count and model, interconnects, and device mapping; driver, runtime, framework, and attention backend versions.
• KV capacity, maximum request count and token budget, chunked prefill, prefix caching, asynchronous scheduling, and CUDA Graph configuration.
• Loop and exit configuration; speculative draft/target depths, draft length, and sampling configuration.
• Data source and version, prompt template, tokenized inputs, output-length policy, and random seeds.
• Concurrency, arrival model, total request count or duration, warmup rules, number of repetitions, timeouts, and failure handling.

State whether the experiment isolates the cost of a particular feature or compares the best tuned configurations of different modes. Do not mix these two types of results in the same performance-gain claim. Explain any configuration differences that cannot be held constant.

Resource fairness: compare aggregate throughput for a PD 1P1D deployment against a conventional inference deployment using the same two GPUs. For single requests, an additional comparison against single-GPU latency is acceptable, but state the resource cost. For speculative versus target-only inference, hold the model, target exit policy, and hardware budget fixed.

Choose Data and Workloads

Choose based on the mechanism being tested; no particular dataset is mandatory:

• Fixed-token-length benchmarks isolate the effects of input length, concurrency, and scheduling. Retain the actual inputs and token IDs; describe any truncation, repetition, or random synthesis.
• Real datasets validate natural length distributions, speculative acceptance rates, early-exit behavior, and task quality. Fix the sampling method; do not replace the test set because another sample batch yields better gains.
• Performance benchmarks are not accuracy tests. Experiments with random weights or synthetically enlarged models may only support explicitly labeled capacity, performance, and latency studies; they cannot represent real acceptance rates, exit distributions, or task quality.

Optional starting scenarios: 128 input tokens / 64 output tokens / one request; 1024 input tokens / 64 output tokens / 16 simultaneous arrivals; 4096 input tokens / 64 output tokens / fixed-rate arrivals. Adjust to the objective. Short bursts cannot establish steady-state throughput or saturation capacity. High-concurrency conclusions require a load sweep and a sufficiently long stable measurement window.

Record fixed-shape ignore-EOS experiments separately from natural generation. Also separate cold/warm cache conditions, graph compilation/capture, and steady-state execution.

3. Establish an Unprofiled Baseline First

Performance numbers must come from independent runs with the profiler disabled. Warmup must cover the actual shape, batch, and graph paths. By default, collect at least three valid repetitions and retain each run's results and variability. For noise-sensitive or small gains, check for drift using methods such as interleaved A/B runs.

Record at least:

When SLOs apply, also report goodput subject to TTFT/TPOT constraints. Do not silently exclude failed requests. Means, medians of per-run percentiles, and percentiles of pooled samples are different statistics; specify which is being reported.

4. Capture ops-only and full Profiles Separately

Independently capture both modes using the same representative request configuration, with fresh sessions and output directories:

Use ops-only to identify operators, launches, copies, synchronization, and device timelines. Use full to further locate shapes, allocations, and call stacks. Full does not mean enabling only shapes or only stacks. On non-CUDA platforms, use the device profiler actually supported by that platform and disclose missing capabilities; do not describe a CPU-only capture as a complete device capture.

Keep Captures Small and Adapt the Window

Start with the smallest window that covers the mechanism under investigation, usually one complete representative speculative round or decode iteration after warmup. An engine step is not necessarily a complete token iteration; ensure the window includes the relevant draft, verification, commit/rollback, or transfer stages. Increase the window only when the initial trace lacks necessary evidence. Benchmark repetition counts do not determine profiling step counts.

Inspect the first capture's file size, event count, export time, and parsing cost before launching a profiling matrix. If traces become too large, export or analysis takes too long, or capture/export fails, first reduce the number of active steps or narrow the capture to the relevant stage. Keep the request configuration representative, and use matching windows for the baseline and candidate and for ops-only/full. Split distinct stages into separate short captures when needed. Do not repeatedly rerun the same oversized window or broaden the matrix before validating a small capture.

Reduce the window, not the definition of full: keep record_shapes, profile_memory, and with_stack enabled. Preserve enough events to support the conclusion; an incomplete round cannot stand in for the whole round. If reducing the window does not resolve an export error, retain the error and then investigate profiler/runtime compatibility. Do not assume file size caused an error without evidence.

Inspect summaries and selected events before loading an entire large trace. Preserve existing valid captures and reuse them; do not recapture completed evidence merely to obtain smaller files. Report unprofiled A/B results as soon as they are available, with their validation status, rather than withholding all results while profiling or export issues are being resolved.

• Choose a short window containing the suspected bottleneck. Warm up the shapes used in that window first, so initialization, JIT compilation, and first graph capture do not contaminate steady-state conclusions.
• If cold startup is the subject of study, create a separate experiment that retains initialization costs.
• For PD, capture P and D separately. For speculation, cover drafting, verification, and commit/rollback. For combined modes, capture according to the actual execution roles.
• Verify request success, actual step counts, rank/session mappings, export completion, and readability of the full profile's shape, memory, and stack artifacts.
• Disclose missing trace events, incomplete windows, export errors, or missing ranks. Do not interpret missing activity as zero cost.
• Profiling changes CPU scheduling, batch composition, and execution time. Do not claim gains using latency or throughput measured with profiling enabled, and do not interpret full being slower than ops-only as an engine regression.

Retain traces, operator statistics, metadata, capture commands, and corresponding requests. Do not commit large traces directly to the code repository.