Quark Torch LLM Ptq Eval

quark-torch-llm-ptq-eval

Purpose

Run the complete PTQ lifecycle for a Torch LLM in one recipe: quantize → validate → evaluate. This recipe composes existing skills rather than re-implementing them — Phase 1 hands off to the quark-torch-ptq workflow (L2) for the PTQ steps, then Phases 2 and 3 chain the atomic validation and evaluation skills (L1). The end result is identical to running the PTQ workflow and then the validator and eval skills by hand; this recipe just drives the whole chain so the user does not have to invoke them separately.

Use this recipe when the user wants quantization plus a correctness and/or accuracy check in a single flow. If they only want the quantized model (no validation, no eval), route to quark-torch-ptq instead — do not run this recipe and then skip its later phases.

Inputs

• Model path (HuggingFace ID or local directory)
• User goal (target precision, hardware, accuracy)
• Output directory for the quantized model
• session_context.json for user goal and constraints
• env_context.json for hardware facts
• workspace_context.json for validated paths
• pytorch_install_result.json and quark_install_result.json to confirm runtime is ready

Outputs

• run_manifest.yaml — executed PTQ command and config (from Phase 1)
• quantized model files in the user's output directory (from Phase 1)
• validation_report.md — structural validation result (from Phase 2, mandatory)
• eval_report.md — accuracy evaluation result (from Phase 3, optional)

CRITICAL RULES

1. Compose, do not re-implement. Phase 1 is the quark-torch-ptq workflow run verbatim, with all its checkpoints. Do not inline or re-derive the intake/plan/manifest/execute steps here.
2. STOP at every checkpoint surfaced by the delegated workflow and at this recipe's eval checkpoint.
3. Validation (Phase 2) is mandatory and automatic. Evaluation (Phase 3) requires explicit user opt-in.
4. NEVER modify Quark's own source code or examples. The Quark repo (quark/, examples/, tools/, docs/, tests/) is read-only. See the same rule in the quark-torch-ptq workflow.

Required Artifact Flow

Phase 1 (PTQ via quark-torch-ptq)
  Step 1 (Intake)   ──► model_analysis.json
  Step 2 (Plan)     ──► quant_plan.json
  Step 3 (Manifest) ──► run_manifest.yaml      (contains the exact command)
  Step 4 (Execute)  ──► quantized model output (only after user says yes)
Phase 2 (Validate)  ──► validation_report.md   (auto, mandatory)
Phase 3 (Eval)      ──► eval_report.md         (optional, requires user opt-in + ROCm for Tier 3)

Phase 1: Quantize (delegate to quark-torch-ptq)

Goal: Produce the quantized model plus model_analysis.json, quant_plan.json, and run_manifest.yaml.

Run the quark-torch-ptq workflow end-to-end (its Steps 1–4, including CHECKPOINT 1/2/3 and the Step 3 "shall I run this?" confirmation). Carry forward, for the later phases:

• <MODEL_PATH> — source model, from intake
• <OUTPUT_DIR> — quantized model directory, from the manifest
• quant_plan.json — exclude rules and model paths, for the validator

Do not proceed to Phase 2 until Phase 1 has produced a quantized model in <OUTPUT_DIR>. If the PTQ workflow stops at a checkpoint or fails, stop here too — there is nothing to validate or evaluate.

Phase 2: Validate Output

Goal: Verify the quantized model is structurally sound. Runs automatically after Phase 1.

Actions

Call quark-torch-result-validator with:

• source_model_dir: from intake (<MODEL_PATH>)
• quantized_model_dir: from manifest (<OUTPUT_DIR>)
• quant_config: from quant_plan.json (exclude rules for the MD5 check)

The validator runs four checks (fuzzy header → aux files → config.json → MD5) and emits validation_report.md.

Result handling

• All four checks pass → one-line summary, proceed to Phase 3.
• fuzzy layout or MD5 fails → STOP. These are correctness checks; report mismatches and suggest fixes via quark-torch-debug. Do not proceed.
• only config.json or aux files fail → if the diff is benign metadata drift from a newer transformers re-serializing the model (renamed config keys, repackaged tokenizer files), mark it benign and continue. Otherwise treat as a real failure and STOP.
• a real failure with an identifiable cause → diagnose the root cause: is it a Quark issue (export/quantization bug) or an environment issue (transformers / accelerate / torch version, model loading, paths)? Report the cause briefly. If it is fixable, ask the user whether they want help resolving it, and on approval apply the fix and re-run validation. Fix the environment or regenerate the artifact — never edit upstream Quark code (Critical Rule 4). If the cause is unclear or not safely fixable, STOP and hand off to quark-torch-debug.

No checkpoint — validation is mandatory and automatic.

Phase 3: Optional Accuracy Evaluation

Goal: Measure post-quantization accuracy and judge whether quantization hurt the model. Opt-in only.

Actions

Ask the user which evaluation tier to run (or skip):