Perform Adversarial Review

Perform Adversarial Review

<!--
SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0
-->

Try to falsify a proposed optimization experiment before it consumes GPU time. Review the proposal; do not generate a second one, edit its draft, deploy it, or run AIPerf.

Inputs

Require:

• EXP_ROOT and the current source iteration;
• exact EXP_ROOT/user_workload.yaml path and SHA256 supplied by the parent;
• exact active benchmark-plan path, SHA256, and series ID returned by perf-analyzer;
• current DEPLOY_ROOT/deployment_ledger.json;
• current DEPLOY_ROOT/applied_manifests/deploy.yaml;
• current DEPLOY_ROOT/benchmark/benchmark_audit.json;
• current DEPLOY_ROOT/benchmark/benchmark_summary.json;
• current DEPLOY_ROOT/benchmark/performance_analysis.json;
• current DEPLOY_ROOT/next-candidate/knowledge-consult.md;
• current DEPLOY_ROOT/next-candidate/deploy-draft.yaml (proposal reviews only; a stop-request carries no draft and its absence is not an objection);
• prior deployment and benchmark artifacts;
• EXP_ROOT/analysis/search-calibration.md (and, for a stop-request, the submitted ledger SHA256 cited in knowledge-consult.md); and
• EXP_ROOT/analysis/hypothesis-backlog.jsonl and EXP_ROOT/analysis/challenger-reviews.jsonl when present; and
• EXP_ROOT/manifest.yaml (session start time, for stop-request budget arithmetic); and
• for a stop-request: the three Finalize paths under EXP_ROOT/final/ (recommended_config.md, reproduced_commands.sh, known_limitations.md), derived from the submitted EXP_ROOT, whose on-disk existence the validation gate verifies.

Review only a consultation whose decision is proposed and whose draft materialization completed successfully. For no-proposal or blocked that carries no stop-request, return without writing a candidate verdict. For a stop-request (a no-proposal consultation whose delta cites the search-calibration ledger path and a submitted SHA256 — a blocked consultation never carries one), run the Stop-Request Validation below instead of returning.

Stop-Request Validation

1. Verify the SHA256 cited in knowledge-consult.md matches the on-disk EXP_ROOT/analysis/search-calibration.md. On mismatch, reject: the ledger moved after submission.
2. Validate completeness and evidence class against the ledger, not the consult file (which carries only the delta): every lever family carries a terminal disposition (tested, ruled-out, not-applicable, or deferred — an answered ask resolves its family into one of these; untested-promising and reopened-by-new-evidence are non-terminal); every ruled-out row cites a measurement, a sourced hard constraint, a confirmed incompatibility, or an explicit operator decision; every deferred row is terminal on a recorded ground - upside below the primary series' measured minimum detectable effect (read from series_noise_floor and minimum_detectable_effect in the current performance_analysis.json, the authoritative source per run-artifacts.md), or a cited cost-estimate-vs-remaining-budget arithmetic (reject when an above-MDE deferred family lacks that arithmetic, or when its estimate fits the remaining budget; return that family as the required follow-up); for a throughput-class objective the recommendation carries saturation evidence or a recorded budget/operator reason in known_limitations.md ("still rising at the top of the measured grid" is not terminal); and all three Finalize files EXIST ON DISK at EXP_ROOT/final/ — recommended_config.md (carrying its required Correctness status: line), reproduced_commands.sh, and known_limitations.md — verified by path, not by the submitter's claim (require those paths as inputs for stop-request validation; a recommendation that exists only in conversation is a blocking objection).
3. Verify the stop-request delta cites derived budget consumption (wall clock from manifest.yaml session start; failed-deploy count from the deployment ledgers; GPU-hours from GPU allocation time, per deployment ledger allocated_at-to-torn_down_at span — or to now, for a live deployment — times its gpus_requested, summed across deployments) and that the cited sources support the arithmetic, whenever any granted budget is non-null.
4. Append the verdict to EXP_ROOT/analysis/challenger-reviews.jsonl as for any review, binding it to the submitted ledger SHA256, and state in it that this is procedural validation, not independent adversarial assurance. A validated stop-request returns to the PARENT with state STOP_REQUESTED for operator grant; it is never a deployment handoff, and return_to is the parent, not recipe-deployer.

Read The Applicable Rules

Always read:

• agent-docs/rules/benchmarking/evidence-eligibility.md;
• agent-docs/rules/benchmarking/comparison-uncertainty.md;
• agent-docs/rules/benchmarking/series-boundaries.md;
• agent-docs/rules/optimization/evidence-before-spend.md;
• agent-docs/rules/optimization/one-variable.md;
• agent-docs/rules/verification/config-engagement.md;
• agent-docs/rules/verification/implausible-speedup.md;
• agent-docs/rules/verification/overlap.md;
• agent-docs/rules/verification/stack-verdict.md;
• agent-docs/rules/execution/user-workload.md (the resources.pinned and resources.gpu_ceiling semantics);
• agent-docs/guides/knob-tuning/tuning-hierarchy.md; and
• the sources and repository guides cited for the proposal's central mechanism.

Read the Dynamo catalog for a Dynamo-owned knob, only the active engine guide for an engine-owned knob, the model-sizing guides for a topology or memory-fit proposal, and the rate-matching guide for a disaggregated allocation proposal. Do not invoke consult-perf-knowledge again or reconstruct a new shortlist. When relevant to the attached evidence, also read the proxy-workload, concurrency-grid, or benchmark-isolation rule.

Establish Review Integrity

Before judging the idea:

1. Require benchmark_audit.json to report valid or valid_with_recovery.
2. Confirm the plan, audit, summary, and analysis identify the same active series and every direct comparison is same-series. Record material resource or placement differences as limitations.
3. Recompute the source manifest, consultation, and draft SHA256 hashes.
4. Require the source and draft hashes to match Materialization Handoff.
5. Parse both manifests and independently compute their semantic diff.
6. Require the consultation, materialized diff, user workload, and performance analysis to identify the same model, engine, deployment, objective, and source operating region.
7. Require a concrete performance question, target operating region, and expected measurable effect for the candidate.

Treat a missing, stale, contradictory, or non-comparable input as a blocking objection. Do not repair it inside the review.