--- name: quark-onnx-result-validator description: > Validate Quark ONNX quantization output using four lightweight checks: auxiliary file copy alignment, expected non-quantized initializer MD5 byte-identity (inline `raw_data` + external-data byte ranges), model metadata equality after stripping quantization-only opset entries / Quark domains, and fuzzy node-pattern + op-type + dtype summaries with QDQ / `com.amd.quark` custom-op presence. Intended for post-quantization inspection of `model.onnx` (with or without `model.onnx_data`). Trigger for "validate ONNX quantization result", "check quantized .onnx output", "verify ONNX initializers", "did QDQ insertion happen", "are the non-quantized weights byte-identical". layer: l1-atomic primary_artifact: validation_report.md source_knowledge: - examples/onnx/yolo_quantization/quantize_yolo.py - examples/onnx/language_models/opt/quantize_model.py - examples/onnx/weights_only_quantization/int4_matmul_nbits/llama2/quantize_model.py - quark/onnx/quantization/api.py - quark/onnx/quantization/config/custom_config.py - quark/onnx/operators/custom_ops/__init__.py --- # quark-onnx-result-validator ## Purpose Run four lightweight checks on a completed Quark ONNX quantization output. Reads only ONNX graph headers (`onnx.load(..., load_external_data=False)`), initializer metadata, and small auxiliary files. Raw payload bytes are only touched for the bounded MD5 spot-check (and only for tensors matched by the user's exclude rules). Results feed a structured `validation_report.md`. ## Runtime Assumptions All scripts (`quant_validation_onnx.py`, `run_validation.py`) live in the same directory as this `SKILL.md`, under `.claude/skills-impl/l1-atomic/onnx/quark-onnx-result-validator/`. Resolve `SKILL_DIR` from the repo root before running any command: ```bash SKILL_DIR=.claude/skills-impl/l1-atomic/onnx/quark-onnx-result-validator ``` `run_validation.py` writes JSON to **stdout**; `quant_validation_onnx.py` diagnostics go to **stderr** with the prefix `[quant-validation-onnx][tag]`. Never treat stderr as structured output. Requires the `onnx` Python package in the supported range (`onnx>=1.21.0,<=1.22.0`, per `requirements.txt`). ## Contracts - Input: `session_context.json`, `quant_plan.json` (for exclude / op-type filters and model paths) - Output: `validation_report.md` - Schemas: `shared/contracts/validation_report.schema.json` ## Inputs | Field | Source | Required | |-------|--------|----------| | `source_model_path` | user or `session_context` | Step 2 + Step 3 | | `quantized_model_path` | user or `run_manifest` | All steps | | `source_model_dir` | parent dir of source model (or `session_context`) | Step 1 | | `quantized_model_dir` | parent dir of quantized model (or `run_manifest`) | Step 1 | | `quant_config` | `quant_plan.json` or user-supplied JSON | Step 2 only | `quant_config` for step 2 supports the following keys (all optional unless noted): | Key | Purpose | Default | |-----|---------|---------| | `exclude` | Glob list of initializer names expected to remain unchanged | — | | `exclude_initializers` | Alias of `exclude` | — | | `nodes_to_exclude` | Node names whose initializer inputs should remain unchanged | — | | `op_types_to_quantize` | When set, any initializer **not** wired into one of these op types becomes an implicit exclude | — | | `max_samples` | Random spot-check cap for large models | `200` | | `random_seed` / `seed` | Deterministic sampling seed | `None` | At least one of `exclude` / `exclude_initializers` / `nodes_to_exclude` / `op_types_to_quantize` **must** be provided; otherwise step 2 is marked skipped. ## Outputs `validation_report.md` with one section per executed step. Unexecuted steps are marked **skipped**. ## Interaction Flow 1. Confirm `SKILL_DIR`, `source_model_path` (if available), and `quantized_model_path` are resolvable. 2. Run the self-test to verify scripts are intact: `python3 "$SKILL_DIR/run_validation.py" self-test` 3. Execute steps in cheap-to-expensive order (**4 → 1 → 3 → 2**). 4. Collect JSON from stdout for each step; write `validation_report.md`. 5. Surface any `ok: false` steps with their `errors` / `mismatches`. ## Four Validation Steps | Order | Function | CLI subcommand | Purpose | |-------|----------|----------------|---------| | 1 | `check_auxiliary_files_copied` | `auxiliary` | Compare non-`.onnx`/non-`.onnx_data` auxiliary files between source and quantized directories | | 2 | `check_non_quantized_initializers_md5_unchanged` | `md5` | MD5 spot-check initializer payload bytes (inline `raw_data` or external-data byte ranges) for tensors expected to remain non-quantized | | 3 | `check_model_metadata_equal_except_quantization` | `metadata` | Compare IR version, producer, default-domain opset, and graph input/output signatures after stripping Quark-injected custom-op domains | | 4 | `get_fuzzy_node_op_summary` | `fuzzy` | Header-only summary: op-type histogram, canonical node-name patterns, initializer dtype counts per pattern, QDQ / `com.amd.quark` custom-op presence | Run in cost order: **4 → 1 → 3 → 2**. ## Agent Execution Contract ### Self-Test ```bash SKILL_DIR=.claude/skills-impl/l1-atomic/onnx/quark-onnx-result-validator python3 "$SKILL_DIR/run_validation.py" self-test ``` Exits `0` and prints `exported symbols self-test (__all__): ok` on success. ### CLI Commands ```bash SKILL_DIR=.claude/skills-impl/l1-atomic/onnx/quark-onnx-result-validator # 4. get_fuzzy_node_op_summary (cheapest — header only) python3 "$SKILL_DIR/run_validation.py" fuzzy \ --model-path ./quantized/model.onnx # 1. check_auxiliary_files_copied python3 "$SKILL_DIR/run_validation.py" auxiliary \ --source-model-dir ./source-dir --quantized-model-dir ./quantized-dir \ --ignore 'README*' # 3. check_model_metadata_equal_except_quantization python3 "$SKILL_DIR/run_validation.py" metadata \ --source-model-path ./source/model.onnx \ --quantized-model-path ./quantized/model.onnx # 2. check_non_quantized_initializers_md5_unchanged (most expensive) python3 "$SKILL_DIR/run_validation.py" md5 \ --source-model-path ./source/model.onnx \ --output-model-path ./quantized/model.onnx \ --quant-config '{"exclude":["*.bias","embeddings.*.weight"],"max_samples":50}' ``` For `md5`, `--quant-config` accepts a JSON string or a path to a JSON file. If `SKILL_DIR` or model paths cannot be resolved, mark the affected step **skipped**. ## Public API (for direct Python import) ```python from quant_validation_onnx import ( check_auxiliary_files_copied, check_non_quantized_initializers_md5_unchanged, check_model_metadata_equal_except_quantization, get_fuzzy_node_op_summary, ) ``` All four functions are in `__all__`. Other public-named helpers are internal utility surface. ## Recovery | Failure | Recovery | |---------|----------| | Self-test exits non-zero | Report script integrity failure; do not run further steps | | `onnx` import fails | Hand off to `quark-onnx-install`; do not run any step | | `source_model_path` missing | Mark steps 2, 3 as skipped; run steps 1, 4 if quantized path is available | | `quant_config` missing exclude rules | Mark step 2 as skipped | | External-data file missing alongside `.onnx` | Recorded under `external_data_missing`; affected tensors marked `read_error` in step 2; step 4 still runs against the graph proto | | Quantized model has zero QDQ / Quark-custom nodes | Step 4 emits a high-severity warning (`quantization_did_not_run`) | ## Report Template ```text ## Validation Report — quark-onnx-result-validator **Step 4 — fuzzy node / op summary**: ok / FAIL / skipped - op types: , QDQ nodes: , com.amd.quark nodes: - Notable: → - Quantization signal: present / **MISSING** / partial **Step 1 — auxiliary files**: ok / FAIL / skipped - missing: , mismatched: , extra: **Step 3 — model metadata**: ok / FAIL / skipped - ir_version: / - opset_import (default domain): / - input/output signature diffs: - Quark-injected opset domains: **Step 2 — MD5 spot-check (initializers)**: ok / FAIL / skipped - candidates: , checked: , sampled: true/false - mismatches: - external_data_missing: ``` ## Canonical Name Rules - Replace only **pure numeric path segments** with `*`: `Conv_12` → `Conv_*`, `model.layer.3.Conv` → `model.layer.*.Conv` - Do not alter digits embedded in non-numeric names: `Conv1`, `MatMul_w2` stay unchanged - `op_type_counts` and `dtype_counts` are aggregated per pattern; multiple op types / dtypes in one pattern signals partial or mixed-precision quantization ## Optional Dtype Hints Heuristics only — not mandatory pass/fail rules: - **INT8 QDQ**: initializers with `INT8` / `UINT8`; nodes `QuantizeLinear` / `DequantizeLinear` - **INT4 MatMulNBits**: initializers with `UINT8` packed-low-nibble shape; `MatMulNBits` op - **BFP16 / MX / MXFP**: `com.amd.quark` opset domain present; nodes `BFPQuantizeDequantize` / `MXQuantizeDequantize` / `ExtendedQuantizeLinear` - **FP16 / BF16 keep**: initializers with `FLOAT16` / `BFLOAT16` and no QDQ neighbors ## ONNX-vs-Torch Behavioural Notes - Step 2 walks **initializers** (the ONNX analog of safetensors tensors), not safetensors entries. Inline tensors are read via `tensor.raw_data`; external-data tensors are read by `(offset, length)` from the external-data file declared in `tensor.external_data`. - Step 3 compares model-level metadata (IR / producer / opset) and graph I/O signature. ONNX has no `config.json` equivalent — opset comparison strips Quark's custom domains before equality so the only allowed diff is the addition of `com.amd.quark` (or similar) on the quantized side. - Step 4 is structurally similar to the Torch fuzzy summary: it groups nodes / initializers by canonical name pattern and reports `op_type_counts` / `dtype_counts`. Additionally surfaces whether quantization actually ran (`QuantizeLinear` / `com.amd.quark` presence).