Quark Onnx Ptq Workflow

quark-onnx-ptq-workflow

📘 Quick-start example — read this first. A fully worked end-to-end walkthrough is available at example-xint8-yolov8n.md (XINT8 quantization of YOLOv8n for AMD NPU CNN deployment, calibrated on COCO val2017). It is the fastest way to see exactly what this workflow produces — open it alongside this SKILL.md before running anything.

Purpose

Chain the ONNX PTQ path — model intake → quantization planning → script + manifest generation → confirmed execution — while keeping the user informed at each checkpoint. This workflow orchestrates the atomic ONNX skills so the user does not have to manually chain them. Unlike the Torch flow there is no single shipped quantize_quark.py for ONNX; the workflow generates a small standalone Python script in the user's working directory that imports from quark.onnx, then runs that script after the user confirms.

📂 Worked example: see example-xint8-yolov8n.md for the full YOLOv8n + XINT8 + NPU-CNN walkthrough referenced throughout the steps below.

Inputs

• Input .onnx model path (with optional sibling .onnx_data external-weights file)
• Calibration data: folder of representative samples (or CalibrationDataReader Python class)
• Output .onnx path (and optional .onnx_data if use_external_data_format=True)
• User goal: scheme (XINT8 / A8W8 / A16W8 / BF16 / BFP16 / MX* / MXFP*), deployment target (CPU / CUDA / ROCm / AMD NPU CNN / AMD NPU Transformer), accuracy target
• session_context.json with constraints.backend = "onnx" for user goal and constraints
• env_context.json for hardware / execution-provider facts
• workspace_context.json for validated paths
• onnx_install_result.json and quark_install_result.json to confirm runtime is ready

Outputs: run_manifest.yaml

Records the generated calibration/quantization script path, the exact python3 invocation, and the resolved QConfig. Side artifacts: the generated script in the user's working directory, the quantized .onnx (and .onnx_data if external) in the user's output directory. The manifest is built in Step 3.

Schema: run_manifest.schema.json

Interaction Flow

1. Intake — call quark-onnx-model-intake to produce model_analysis.json
2. Plan — call quark-onnx-quant-plan to produce quant_plan.json
3. Manifest — generate a standalone quantization script in the user's working directory + run_manifest.yaml, then stop for user approval
4. Execute — run the confirmed script and report the artifact paths

CRITICAL RULES

1. NEVER call ModelQuantizer.quantize_model(...) directly from this workflow. Always generate a standalone script the user reviews first.
2. NEVER skip a step. Even if the user provides all details upfront, execute each step in order.
3. STOP at every checkpoint and wait for user confirmation before continuing.
4. Show concrete output at each step — tables, JSON, the full script body, the exact command — not just prose descriptions.
5. NEVER modify Quark's own source code, examples, or tutorials. The Quark repo (quark/, examples/, tutorials/, tools/, docs/, tests/) is read-only from this workflow's perspective. See Upstream Quark Code is Read-Only below.
6. NEVER silently fall back to CPU execution. If the user requested CUDA/ROCm and the provider is unavailable, stop and surface the gap to quark-onnx-install / quark-onnx-debug — do not quietly degrade.

Upstream Quark Code is Read-Only

The Quark repository is the upstream source of truth. This workflow may read it freely (config sources, example scripts, tutorial notebooks, custom-op headers) but must not write into it. That includes:

• No edits to quark/ package source.
• No edits to anything under examples/onnx/ — including examples/onnx/yolo_quantization/quantize_yolo.py. Even small "just to make the script accept my preprocessing" patches are forbidden, because they make the run irreproducible against a clean Quark install.
• No edits to anything under tutorials/onnx/ — the Ryzen AI tutorials for YOLOv8 / ResNet-50 / image classification are reference material only.
• No edits to tools/, docs/, tests/, or pyproject.toml / requirements.txt.

If a shipped example or tutorial as-is does not cover what the user needs (different model, different preprocessing, different exclude list, different calibration source), write a fresh standalone script in the user's working directory (or /tmp/) that imports from quark.onnx. Pattern:

# user_workspace/my_onnx_ptq.py — NOT inside the Quark repo
from quark.onnx import ModelQuantizer, QConfig, QLayerConfig, XInt8Spec, CLEConfig
from onnxruntime.quantization.calibrate import CalibrationDataReader
# ... user-specific data reader + QConfig the shipped example doesn't cover ...

Then reference that script in the run_manifest.yaml instead of a modified example. The shipped scripts stay untouched, the user's customization is local to their workspace, and the run remains reproducible against any Quark version.

If the user actually needs an upstream Quark change (a new preset, a new algorithm config), surface it as a Quark contribution — do not silently patch their local checkout.

Required Artifact Flow

Step 1 (Intake)   ──► model_analysis.json
Step 2 (Plan)     ──► quant_plan.json
Step 3 (Manifest) ──► run_manifest.yaml  +  user_workspace/<name>_ptq.py
Step 4 (Execute)  ──► quantized .onnx (+ .onnx_data if external)  (only after user says yes)

Step 1: Model Intake

Goal: Analyze the .onnx graph and produce model_analysis.json. Hand off to quark-onnx-model-intake.

Actions

1. Confirm the input path. It must end in .onnx. If a sibling .onnx_data exists, both must move together.
2. Run quark-onnx-model-intake. That skill produces:
   • opset / IR version
   • input / output names, shapes, dtypes
   • op-type histogram and quantizable-op count
   • whether QDQ nodes are already present (if yes — model is already quantized; warn and stop)
   • external-data status (model.onnx_data present? total bytes? >2 GB?)
   • deployment-target compatibility (CPU / CUDA / ROCm / NPU CNN / NPU Transformer)
3. Surface risks:
   • Already quantized → stop, do not re-quantize.
   • opset < 13 → recommend upgrading to opset 17+ for best support.
   • Dynamic input shapes → calibration data reader must produce matching shapes; warn if shapes vary.

   • 2 GB model → external data must be handled; set use_external_data_format=True later.

   • NPU CNN target → only XINT8 / A8W8 are gated as supported; warn early.

Output to Show User

Present a summary table:

Model Analysis:
  Model path:       ./models/yolov8n.onnx
  Opset / IR:       17 / 8
  Input:            images, [1, 3, 640, 640], float32
  Output:           output0, [1, 84, 8400], float32
  Total ops:        ~226 nodes (Conv, MatMul, Add, Mul, Sigmoid, Concat, …)
  Quantizable ops:  ~118 (Conv + MatMul + …)
  QDQ already?      No
  External data:    No (6.2 MB inline)
  Risks:            None
  NPU CNN target:   Compatible (Conv-heavy, no unsupported ops)

>>> CHECKPOINT 1: Confirm model analysis is correct before continuing

Step 2: Quantization Plan

Goal: Build quant_plan.json from the model analysis and user's stated preferences. Hand off to quark-onnx-quant-plan.

Actions

1. Determine the preset. If the user stated one (e.g., "XINT8"), use it. Otherwise, recommend based on deployment target + priority:

   Deployment	Priority	Recommended Preset	Algorithm
   AMD NPU CNN (Ryzen AI)	Best accuracy	XINT8 + EnableNPUCnn=True	CLE (optional AdaRound)
   AMD NPU Transformer	Best accuracy	BFP16	none
   CPU general	Smallest model	A8W8	CLE (optional)
   CPU general	Best accuracy	A16W8	MinMax
   CUDA / ROCm	Best accuracy	BF16	none
   CUDA / ROCm	High throughput	BFP16 / MXFP4	none
   Any	Recover lost accuracy	+ AdaRound / AdaQuant	as additional algorithm

2. Fill the decision table. Show ALL decisions with defaults:

   Decision	Value	Reason
   preset	XINT8	User requested XINT8 (NPU CNN)
   activation_spec	XInt8Spec()	Matches preset
   weight_spec	XInt8Spec()	Matches preset
   calibration_method	MinMax (default)	Standard for XINT8
   algo_config	[CLEConfig()]	Improves XINT8 accuracy on Conv networks
   EnableNPUCnn	True	XINT8 + NPU CNN deployment
   use_external_data_format	False	Model < 2 GB
   exclude_nodes / exclude_subgraphs	[]	None identified
   calibration_data_path	./calib_data/	User-provided folder of representative samples
   num_calib_data	100	Standard default for vision
   batch_size	1	Safe default
   evaluation_intent	smoke	Quick mAP / Prec@1 check after quantization

3. Map deployment-target gates explicitly. Note any preset+target combinations that are unsupported (e.g., BFP16 on NPU CNN). Surface them, do not silently change the user's choice.

4. Ask the user if they want to change anything.

>>> CHECKPOINT 2: User MUST confirm or adjust the plan before continuing

Wait for the user to say "ok", "confirm", "looks good", "continue", or similar. If they request changes (e.g., "add AdaRound", "raise calib data to 500"), update the table and re-present.

Step 3: Manifest Generation

Goal: Translate the confirmed plan into a runnable standalone script in the user's working directory and a run_manifest.yaml. Do not write into the Quark repo.