Quark Torch File2file Quantization

quark-file2file-quantization-runner

Purpose

Quantize very large safetensors checkpoints without loading the full model into memory, using ModelQuantizer.direct_quantize_checkpoint(). Covers the full adaptation path: checkpoint inspection, external LLMTemplate registration, optional naming normalization via a one-time conversion script, minimum-scale experiment gating, full file2file execution, and output validation via quark-torch-result-validator.

Default policy: solve naming or layer-selection mismatches with external adapters (LLMTemplate.register_template(), weight_converters) or a temporary conversion script. Do not modify Quark source unless the required capability is absent, the external path has been ruled out, and the user explicitly agrees.

Inputs

• pretrained_model_path — local directory of the safetensors checkpoint
• save_path — output directory
• quant_scheme — quantization scheme (e.g., w_fp8_a_fp8, w_int4_a_bf16)
• device — e.g., cuda:0, cpu
• User intent: direct quantization, checkpoint conversion first, HF-format export, or script generation only
• model_analysis.json from quark-torch-model-intake (optional but recommended)

Outputs: run_manifest.yaml

Records the adaptation path taken, scripts generated, experiment results, and validation status.

pretrained_model_path: /models/DeepSeek-V3
save_path: /output/DeepSeek-V3-fp8
quant_scheme: w_fp8_a_fp8
device: cuda:0
adaptation_path: external_template          # direct | external_template | weight_converters | conversion_script
conversion_script: null                     # path if generated
wrapper_script: /tmp/run_ds_v3_f2f.py
min_experiment:
  status: passed                            # passed | failed | skipped
  layers_covered: [layer_0_expert_0, ...]
  moe_covered: true
full_run_status: completed                  # pending | completed | failed
validation_status: passed                  # passed | failed | not_run

Interaction Flow

Step 1 — Inspect checkpoint

Read keys and config without loading weights:

python - <<'PY'
import json, os
from glob import glob
from safetensors.torch import safe_open

model_dir = "<pretrained_model_path>"
index_path = os.path.join(model_dir, "model.safetensors.index.json")
print("config:", os.path.exists(os.path.join(model_dir, "config.json")))
print("index:", os.path.exists(index_path))
files = sorted(glob(os.path.join(model_dir, "*.safetensors")))
print("safetensors:", len(files))
if files:
    with safe_open(files[0], framework="pt", device="cpu") as f:
        keys = list(f.keys())
    print("sample_keys (first 80):")
    for k in keys[:80]: print(" ", k)
if os.path.exists(index_path):
    with open(index_path) as f:
        wm = json.load(f).get("weight_map", {})
    print("index_keys:", len(wm))
PY

Verify: model_type, weight-name suffixes (*.weight, *_scale_inv, *.scale), shard count, MoE expert / shared-expert / gate naming, and whether scale tensors are co-located with weights.

Step 2 — Choose adaptation path

_apply_weight_converters limits: suited for post-recovery single-suffix rename or one-source split. Not suited for multi-source merge, cross-shard scale pairing, or FP4→FP8 dtype conversion.

Step 3 — Generate wrapper or conversion script

For external template registration, generate a wrapper script (do NOT modify quantize_quark.py):

from quark.torch import ModelQuantizer
from quark.torch.utils.llm import LLMTemplate

template = LLMTemplate(
    model_type="<model_type>",
    kv_layers_name=["<pattern>"],
    q_layer_name=["<pattern>"],
    exclude_layers_name=["embed", "head", "<other>"],
)
LLMTemplate.register_template(template)

quantizer = ModelQuantizer(config)
quantizer.direct_quantize_checkpoint(
    pretrained_model_path=pretrained_model_path,
    save_path=save_path,
    keep_excluded_layers_as_original_model_state=False,
    weight_converters=weight_converters,
    device=device,
)

Wrapper must print: registered model_type, input/output dirs, quant scheme, and exclude rules. Conversion scripts must stream safetensors (no full-model load), include explicit remap_name(), scale/weight pairing validation, shard output in HF style, index rebuild, and atomic output.

Step 4 — Minimum-scale experiment (mandatory gate)

This step is not optional. Full file2file must not run until the minimum experiment passes.

Construct the minimum input:

1. If num_hidden_layers is safely reducible, copy config.json with the smallest value that still covers at least one MoE layer (for MoE models, use first_moe_layer_id + 1).
2. If not, generate a subset checkpoint filtered by --key-regex covering at least one complete MoE expert + its scale tensor + shared expert/router/gate + adjacent non-quantized tensors.
3. Never set num_hidden_layers=1 for a MoE model if layer 0 is dense — verify from config or key patterns which layer is the first actual MoE layer.

Run the minimum experiment, then call quark-torch-result-validator with: inspect_safetensors, summarize_dtypes, check_index_consistency, check_scale_pairs, get_fuzzy_tensor_names, and auxiliary-file copy check (if source dir available).

If validation fails, fix template / naming / subset and re-run. Do not proceed to full file2file.