Quark Onnx Eval Runner

quark-onnx-eval-runner

Purpose

Walk a maintainer through manual verification of the ONNX skill family across the four contract categories: routing, planning, artifact, and recovery. Run this after modifying any quark-onnx-* skill, after a Quark ONNX upgrade, or before tagging a release.

Inputs

• The current ONNX skill files under .claude/skills-impl/{l1-atomic,l2-workflows,l3-recipes}/onnx/
• The entry stubs under .claude/skills/quark-onnx-*
• The contract schemas under .claude/skills-impl/shared/contracts/
• The example prompts under examples/agent_skills/prompts/ (add ONNX-specific cases as the catalog grows)

Outputs: validation_report.md

A markdown report recording per-category pass/fail and concrete evidence for each finding.

Schema: validation_report.schema.json

# ONNX Skill Verification Report

## Summary
| Category | Cases Run | Pass | Fail |
|----------|-----------|------|------|
| routing  | N         | N    | 0    |
| planning | N         | N    | 0    |
| artifact | N         | N    | 0    |
| recovery | N         | N    | 0    |

## Failures
### <category> / <case name>
- **Expected**: ...
- **Got**: ...
- **Impact**: ...
- **Fix**: ...

Manual Verification Protocol

For each category below, run at least one case and record the result in the report. As ONNX prompts are not yet enumerated in examples/agent_skills/prompts/, the cases below double as the seed catalog — add more as the ONNX skill set grows.

1. Routing

Goal: verify that quark-onnx-router (and Claude's auto-routing via the descriptions in .claude/skills/quark-onnx-*) maps natural-language ONNX goals to the correct downstream skill and never silently routes ONNX requests through a torch skill.

Manual procedure:

1. Pick a user-style prompt that names a .onnx artifact or ONNX-specific vocabulary.

2. In a fresh Claude Code session at the Quark repo root, paste the prompt.

3. Observe which skill Claude invokes first.

4. Compare against the expected target skill. Examples of expected mappings:

   • "Quantize my ./models/yolov8n.onnx to XINT8 for AMD NPU CNN" → quark-onnx-ptq (which loads quark-onnx-ptq-workflow)
   • "Run AutoSearchPro on this .onnx with the XINT8_SEARCH preset" → quark-onnx-autosearch-pro
   • "Analyze this .onnx — what opset is it, is it NPU-compatible, is it already QDQ?" → quark-onnx-model-intake
   • "Validate my quantized model.onnx — did QDQ insertion happen, are the non-quantized initializers byte-identical?" → quark-onnx-result-validator
   • "onnxruntime-gpu import fails, CUDAExecutionProvider not in providers list" → quark-onnx-install (or quark-onnx-debug if the user already attempted install)
   • "quantize_static failed with custom-op library load failure for BFPQuantizeDequantize" → quark-onnx-debug
   • "Is onnxruntime-rocm installed correctly? Show me the install matrix" → quark-onnx-install

5. Cross-backend guard: also run one negative prompt that mentions a .onnx path and confirm Claude does not route to quark-torch-* (e.g., "quantize ./models/foo.onnx with FP8" must not land on quark-torch-ptq).

Pass criteria: the first skill invoked matches the expected target, and no ONNX prompt is routed to a torch skill.

2. Planning

Goal: verify that quark-onnx-quant-plan produces internally consistent plans for typical ONNX inputs and that the deployment-target gates are respected.

Manual procedure:

1. Construct (or take from a prior session) a model_analysis.json produced by quark-onnx-model-intake for a representative model (e.g., YOLOv8n exported at opset 17, Conv-heavy, 6.2 MB inline).
2. Hand it to quark-onnx-quant-plan with a target preset (e.g., XINT8) and a deployment target (e.g., AMD NPU CNN).
3. Inspect the produced quant_plan.json for:
   • preset matches the requested preset
   • activation_spec and weight_spec are consistent with the preset (e.g., both XInt8Spec for XINT8)
   • EnableNPUCnn=True is set when the target is AMD NPU CNN
   • use_external_data_format is True iff the model is >2 GB
   • algo_config is a non-empty list when CLE or AdaRound was requested or recommended
   • exclude is a list (may be empty) and never contains an op the plan also quantizes
   • requires_confirmation is set when the plan deviates from preset defaults
   • Negative gate: the plan refuses incompatible combos (e.g., BFP16 + AMD NPU CNN) rather than silently downgrading

Pass criteria: the plan validates against quant_plan.schema.json, contains no internal contradictions, and explicitly rejects unsupported deployment-target / preset combinations.

3. Artifact

Goal: verify that ONNX workflow output artifacts conform to their JSON schemas and that the generated standalone script + manifest produced by quark-onnx-ptq-workflow agree with each other.

Manual procedure:

1. Take the quant_plan.json from the planning case.
2. Run quark-onnx-ptq-workflow to produce a run_manifest.yaml and the standalone <name>_ptq.py script in the user's working directory.
3. Validate the manifest against .claude/skills-impl/shared/contracts/run_manifest.schema.json (use any JSON-schema validator, e.g., the jsonschema Python package).
4. Spot-check that:
   • All required fields are present in the manifest
   • The manifest's command references the generated script path and uses python3
   • The manifest's resolved QConfig matches the plan's preset / algo_config / EnableNPUCnn / use_external_data_format
   • The generated script imports only from quark.onnx and the standard ORT calibration API (no editing of upstream examples/onnx/ or quark/onnx/ files)
   • Output paths reflect any .onnx_data sidecar when external data is enabled

Pass criteria: schema validation passes; the manifest's resolved config matches the plan; the generated script is self-contained in the user's working directory.