Audit Model Validation

Audit Method Validation

Judge whether the alternatives considered and evidence gathered support the stated conclusion; do not choose a method or redesign the work.

Separate operational and empirical evidence

Classify every supplied check before judging the claim:

• Operational evidence covers importability, tensor shapes, finite outputs, parameter counts, deterministic evaluation, packaging, gradient flow, and memorization of synthetic or random data. It can establish that an artifact runs, but normally cannot establish task suitability or generalization.
• Empirical evidence measures decision-relevant behavior on representative real data with labels, splits, metrics, and conditions matched closely enough to the intended use. It supports performance and selection claims only within those observed conditions.

Do not let the volume, precision, or independence of operational checks substitute for missing empirical evidence. If the intended claim is that a method is suitable, effective, robust, or preferred and representative real-data evidence is absent, mark empirical adequacy unverified. This is a material finding requiring REVISE or BLOCK, even when every implementation and protocol-compliance check passes. A report may separately confirm the narrower claim that the artifact is operationally valid.

Checks

1. Identify the intended use and claim. Determine whether the evidence source, comparison unit, conditions, measures, and time horizon represent that use.
2. Inventory the evidence by source: representative real data, non-representative real data, simulated data, random tensors, literature, or implementation inspection. Verify that the strongest conclusion does not exceed the strongest applicable evidence class.
3. Verify that discovery covered a sufficiently broad set of credible, substantively different alternatives for the claim. Check established baselines and require justification for material omissions; a long list of minor variants is not breadth. The previous working baseline is an essential comparison when a proposed replacement claims improvement or suitability.
4. Distinguish operational correctness and feasibility from evidential adequacy. Synthetic, self-consistency, or convenience evidence establishes real-world suitability only when the task makes it representative.
5. Check that comparisons are fair and that screening, pruning, adaptation, and resource-driven reductions follow declared rules without discarding essential breadth merely for implementation convenience. Treat a missing reader, dependency, accelerator, or preprocessing artifact as a blocked evidence path, not as justification for replacing real-data validation with synthetic checks.
6. Check applicable baselines, chance levels, negative controls, grouped metrics, uncertainty, multiplicity, evidence reuse, and anomalously optimistic results. Treat nuisance, proxy, or setting-specific signals that can mimic the intended target as material risks.
7. Check for degenerate predictions and hidden training failure before accepting aggregate scores. Require class coverage and per-group diagnostics when class collapse is plausible.
8. Verify that implementation parameters preserve their scientific meaning. Recompute durations, frequencies, window sizes, and other physical quantities after resampling or unit conversion; copying sample counts across sampling rates is not protocol fidelity.
9. For each material iteration, verify that its representative real-data validation ran on the same or later code and configuration revision than the implementation diff under review. Preserve the valid iterative sequence in which one result motivates the next decision; final validation of an older revision cannot validate a newer candidate.
10. Bound every conclusion to the alternatives, evidence, conditions, and checks actually completed.