Quark Torch LLM Ptq Workflow

quark-torch-llm-ptq-workflow

Purpose

Chain the full PTQ path — model intake → quantization planning → manifest generation → confirmed execution — while keeping the user informed at each checkpoint. This workflow orchestrates the atomic skills so the user does not have to manually chain them. It stops at the quantized output; for a run that also validates and evaluates the result, use the quark-torch-llm-ptq-eval recipe.

Inputs

• Model path (HuggingFace ID or local directory)
• User goal (target precision, hardware, accuracy)
• Output directory for the quantized model
• session_context.json for user goal and constraints
• env_context.json for hardware facts
• workspace_context.json for validated paths
• pytorch_install_result.json and quark_install_result.json to confirm runtime is ready

Outputs: run_manifest.yaml

Records the executed command and config. Side artifacts: quantized model files written to the user's output directory. The manifest is built in Step 3 above.

Schema: run_manifest.schema.json

Interaction Flow

1. Intake → quark-torch-model-intake → model_analysis.json
2. Plan → quark-torch-quant-plan → quant_plan.json
3. Confirm → run_manifest.yaml (stop for user approval)
4. Execute → run the confirmed command, write quantized model

CRITICAL RULES

1. NEVER run quantize_quark.py directly. Always go through the 4 steps below.
2. NEVER skip a step. Even if the user provides all details upfront, execute each step in order.
3. STOP at every checkpoint and wait for user confirmation before continuing.
4. Show concrete output at each step — tables, JSON, commands — not just prose descriptions.
5. NEVER modify Quark's own source code or examples. The Quark repo (quark/, examples/, tools/, docs/, tests/) is read-only from this workflow's perspective. See Upstream Quark Code is Read-Only below.

Upstream Quark Code is Read-Only

The Quark repository (quark/, examples/, tools/, docs/, tests/, pyproject.toml, requirements.txt) may be read freely but never modified. This includes "just to make the script accept my flag" patches to examples/torch/language_modeling/llm_ptq/quantize_quark.py — they break reproducibility against a clean Quark install.

When the shipped example does not cover the user's needs, write a fresh standalone script in the user's working directory (or /tmp/) that imports from quark:

# user_workspace/my_custom_ptq.py — NOT inside the Quark repo
from quark.torch import ModelQuantizer
from quark.torch.quantization.config.config import Config
# ... user-specific logic ...

Reference that script in run_manifest.yaml. The shipped quantize_quark.py stays untouched; the run remains reproducible against any Quark version. If the user actually needs an upstream Quark change, surface it as a contribution — do not silently patch their local checkout.

Required Artifact Flow

Step 1 (Intake)   ──► model_analysis.json
Step 2 (Plan)     ──► quant_plan.json
Step 3 (Manifest) ──► run_manifest.yaml      (contains the exact command)
Step 4 (Execute)  ──► quantized model output (only after user says yes)

Step 1: Model Intake

Goal: Produce model_analysis.json.

Actions

1. Locate the Quark PTQ script (quantize_quark.py) under <Quark repo>/examples/torch/language_modeling/llm_ptq/ — find / -name quantize_quark.py -path "*/llm_ptq/*" 2>/dev/null | head -1 if the path is unknown. Record it for Step 3.

2. Call quark-torch-model-intake with the model path. It handles config parsing (no weight load), supported-template matching, and risk identification (MoE, >70B, transformers version constraints) and emits model_analysis.json.

Output to Show User

Render the summary from model_analysis.json:

Model Analysis:
  Model path:       Qwen/Qwen3-8B
  Model type:       qwen3
  Hidden layers:    36
  Linear layers:    ~224 (quantization targets)
  MoE:              No
  Exclude defaults: [lm_head]
  Risks:            <list>
  Compatibility:    OK

>>> CHECKPOINT 1: Confirm model analysis is correct before continuing

Step 2: Quantization Plan

Goal: Build quant_plan.json from the model analysis and user's stated preferences.

Actions

1. Determine the scheme. If the user stated a scheme (e.g., "FP8"), use it. Otherwise, recommend based on their priority:

   Priority	Recommended Scheme	Algorithm
   Best accuracy	fp8 or ptpc_fp8	smoothquant (optional)
   Smallest model	int4_wo_32	awq or gptq
   CPU deployment	int8	none
   AMD MI300X/MI355X	fp8 or amdfp4	none
   NVIDIA H100	fp8	none
   GGUF export	uint4_wo_32	awq

2. Fill the decision table. Show ALL decisions with defaults:

   Decision	Value	Reason
   global_scheme	fp8	User requested FP8
   kv_cache_scheme	fp8	Recommended for FP8 inference
   exclude_layers	["lm_head"]	Standard — lm_head stays full precision
   layer_quant_config	{}	No per-pattern overrides (or e.g. {"*self_attn*": "fp8"} if the user asked to quantize attention with a non-global scheme)
   algorithm	null	RTN baseline (fastest)
   calibration_dataset	pileval	Fast default
   num_calib_data	128	Standard default
   seq_len	512	Standard default
   evaluation_intent	smoke	Quick PPL check after quantization

3. Ask the user if they want to change anything.

>>> CHECKPOINT 2: User MUST confirm or adjust the plan before continuing

Wait for the user to say "ok", "confirm", "looks good", "continue", or similar. If they request changes (e.g., "use smoothquant", "increase calibration to 256"), update the table and re-present.

Step 3: Manifest Generation

Goal: Translate the confirmed plan into the exact quantize_quark.py command and produce run_manifest.yaml.