Analyze Aiperf Results

Analyze AIPerf Results

<!--
SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0
-->

Audit benchmark evidence before interpreting performance. Preserve raw files unchanged and do not claim an unmeasured server-side cause.

Read:

• agent-docs/rules/benchmarking/benchmark-isolation.md;
• agent-docs/rules/benchmarking/comparison-uncertainty.md;
• agent-docs/rules/benchmarking/evidence-eligibility.md;
• agent-docs/rules/benchmarking/result-storage.md;
• agent-docs/rules/benchmarking/series-boundaries.md;
• agent-docs/rules/benchmarking/tool-version.md;
• agent-docs/rules/optimization/evidence-before-spend.md;
• agent-docs/rules/optimization/one-variable.md;
• agent-docs/rules/verification/config-engagement.md;
• agent-docs/rules/verification/implausible-speedup.md;
• agent-docs/rules/verification/overlap.md; and
• agent-docs/rules/verification/stack-verdict.md.

Also read the user workload, active benchmark plan, execution ledger, AIPerf config, raw outputs, all prior candidate audits and summaries, and the profile-export documentation matching the pinned AIPerf source or runtime.

Audit And Normalize

• Require the run's benchmark_execution.json to exist before auditing: it is the execution record the audit chain and budget accounting bind to. A benchmark whose raw exports exist but whose execution record was never written is an audit blocker — return it to run-aiperf-benchmark to write the record; do not audit around it.
• Parse per-request profile_export.jsonl with AIPerf's native Pydantic models when available. Record the parser and runtime version used.
• Parse profile_export_aiperf.json and multi-run aggregate/search artifacts when configured.
• Confirm files are readable, non-empty, and internally consistent.
• Confirm the executed config matches the active plan and candidate endpoint.
• Confirm trace hash or static-shape identity, schedule mode, endpoint type, model, tokenizer, warmup, seed, request count/duration, repetitions, and load controls.
• Separate warmup from profiling records and exclude only data the active plan says to exclude.
• Compare attempted, successful, failed, cancelled, and timed-out request counts.
• Check actual ISL/OSL distributions against the input workload and report any output-length shortfall.
• Check timestamps, benchmark duration, fixed-schedule coverage, duplicate/missing request ids, malformed metrics, NaN/inf values, units, and impossible negative latencies.
• Recompute user-requested percentiles from raw profiling records when AIPerf does not export them directly.

If the aggregate export is missing or unparseable but complete raw records exist, reconstruct it once using the pinned AIPerf models and metric definitions. Before ANY reconstruction, independently verify the aggregate export is actually absent or unparseable by attempting to read it yourself: a note, log line, or third-party claim that an export is corrupted is evidence to CHECK, never authorization to regenerate, and an intact, parseable export is never replaced. Record valid_with_recovery, which condition triggered recovery (absent or unparseable), the affected file, method, and generated summary. Never modify or replace the raw directory.

Write Audit Artifacts And Gate Analysis

Write benchmark_audit.json with:

• status: valid, valid_with_recovery, or invalid;
• benchmark-series ID, plan path and SHA256, performance question, and workload identity;
• expected versus actual requests and phases;
• error/cancellation breakdown;
• integrity checks and recovery actions;
• parser/AIPerf versions;
• blockers and next_action: continue_analysis, rerun_benchmark, or stop.

Write benchmark_summary.json with normalized benchmark metadata and every numerical metric reported by AIPerf, including units and available statistics. Include requested custom percentiles and per-GPU derived throughput with the GPU-count source. Do not include gain/loss interpretation in the summary.

For valid or valid_with_recovery, set next_action to continue_analysis and continue below in the same invocation.

For repairable invalid evidence, set next_action to rerun_benchmark. Return benchmark_audit.json and benchmark_execution.json to run-aiperf-benchmark, preserve the invalid run as failed evidence, and rerun the active series unchanged without overwriting its raw artifacts. Invoke analyze-aiperf-results again after the rerun.

If repair would change workload semantics or a bounded rerun repeats the same invalid result, set next_action to stop. Do not write performance analysis or promote the candidate until a valid rerun exists. Never discard an invalid run.

Select Comparable History

Use only valid runs whose benchmark-series ID matches the active plan, then check every candidate run's recorded AIPerf runtime version (and source commit, when the plan pins one) against the plan's pin. A series ID alone does not establish comparability: a reused or hand-edited series can contain runs from different tool versions. Apply the graded response in agent-docs/rules/benchmarking/tool-version.md and record the check and its outcome in benchmark_audit.json:

• same version, or a patch-level difference: comparable; note the delta;
• a minor difference: comparable only if the audit records a justification, either release notes for the span showing no measurement-affecting change, or a bridging run (the best-prior configuration re-measured under the newer version) whose delta is within the series noise floor; if the justification is missing, request the bridging run as the next action rather than comparing or discarding;
• a major difference, a flagged measurement change, or a bridging delta beyond the noise floor: exclude the mismatched runs from every comparison, list the exclusion as a limitation, and when the CURRENT run is the mismatched one report absolute performance only and mark the mismatch as a series boundary.

From the comparable set identify:

• series_baseline: earliest valid result in the series;
• previous_valid: most recent valid iteration before the current one;
• best_prior: best prior run for each objective, respecting metric direction and SLO feasibility;
• history: every valid same-series iteration.

Verify that every reference required by the plan is present before making a direct comparison. If no prior valid same-series result exists, treat the current result as the series baseline and report absolute performance only. Cross-series results may provide context but never a gain, loss, or Pareto calculation.