GPTQModel

GPT-QModel ⚡

An extensible platform for LLM quantization, validation, and deployment.
GPTQ, AWQ, ParoQuant, GGUF, FP8, EXL3, QQQ, and more—across NVIDIA CUDA, AMD ROCm, Huawei Ascend, Intel XPU, and CPU, with Transformers, vLLM, and SGLang.

Latest News 🗞️🚀

• 7.10.0:

  • Models: Added Ricursive Ripple (ricursive) quantization support for the 65B preview checkpoint, including load-time Transformers 5 generation/masking patches, per-head Q/K norm and gated-attention handling, and v_proj Hessian isolation. Fixed pre-lm-head norm resolution across multimodal model definitions.
  • Performance: Accelerated Ricursive Ripple decode by up to 10x per row at batch 16 with fused RMSNorm and native GQA attention, and added chunked MLP prefill to bound activation memory. The CUDA expandable-segments allocator is now installed before Torch initializes, so wide-layer multi-GPU runs actually use it.
  • Checkpointing: Added gptqmodel.reshard() for per-layer safetensors sharding with one global shard, byte-for-byte tensor copies, preserved metadata, and config/tokenizer copying. Added a one-time warning when quantization shards are not layer-wise.
  • Multi-GPU: Honored explicit device lists for quantization placement, routed cross-device moves per direction to avoid asymmetric ROCm SIGSEGVs, and documented cudaMallocAsync for wide-layer Hessian allocation.
  • ROCm: Made AMD device families explicit and enforced kernel support contracts; gated W4A FP8/NVFP4 and ExLlamaV2/VPTQ off unsupported HIP paths with fast failures; fixed QQQ kernel launches to use the tensor’s device context.
  • Quantization: Added method-specific integer bit-width validation across GPTQ, AWQ, QQQ, ParoQuant, FP8, bitsandbytes, EXL3, GGUF, and VPTQ; EXL3 now rejects fractional trellis widths; MoE routing-bypass batch sizes are validated.
  • Correctness: Fixed Ricursive Ripple cached decode, mask alignment, per-row positions, and labeled-loss handling, plus EXL3 tile-scratch bounds and sign-vector precision.

• 7.9.0:

  • Correctness: Fixed packing/inference overflow, underflow, FP32 reductions, BF16 bias cancellation, 8-bit BitBLAS zero points, and stale fused Torch metadata after checkpoint loads or tensor changes.
  • MoE: Fixed skipped routed experts in MiniMax M2, Granite MoE/Hybrid, Bailing Hybrid, and AfMoE; added warnings for unresolved module paths.
  • Memory: Removed retained FP8 weight expansion from the experimental NVFP4 path and dense-weight scratch from ExLlamaV2.
  • Kernels: Added measured RTX 4090 (Ada, SM 8.9) and RDNA4 (gfx1200/gfx1201/gfx1250) launch tables to every tuned Triton pipeline — GPTQ GEMV/GEMM/dequant, ParoQuant GEMM, AWQ GEMM/GEMV, GGUF Triton, and bitsandbytes 4-bit — enforced by a hardware-free parity contract; the fused bitsandbytes 4-bit GEMM measured 1.4x-6.3x faster than the vendor HIP kernel on RDNA4 for M >= 16, while Ada keeps the vendor kernels. Added opt-in AllSpark W8A16 for compatible per-channel 8-bit GPTQ/AWQ on CUDA and RDNA4 (gfx1200/gfx1201). EXL3 2-bit quantization now fits RDNA4’s 64 KiB shared memory.
  • MLX: Accelerated GPTQ/RTN quantization, GPTQ/AWQ group-16 decode and short prefill, and GGUF K-quant/ternary packing.
  • GSQ/RCO: Added teacher-KL optimization, per-projection bit allocation, and native MLX search for dense Llama/Qwen3, with token-based batching and device/disk teacher caches.
  • Models: Added LFM3, LFM3-VL, and LFM3-Omni text-decoder quantization, OLMo/Helium support, and Qwen2 quantization resume.
  • Compat: Added native Transformers/Optimum load hooks preserving dynamic settings, fused/MoE structure, rotation, and W4A policies. Replaced method-specific Marlin/Machete/Swordfish enums with BACKEND.MARLIN, BACKEND.MACHETE, and BACKEND.SWORDFISH; legacy string aliases remain supported by the hf_ APIs.

• 7.8.0:

  • GSQ: Added staged GSQ training with public method adapters, distributed staged training that reuses captured activations, and per-block bit allocation driven by an RCO search verified bit-identical against the IST-DASLab reference (measured results on Llama 3.2 1B). Added a paper-reproduction harness, best-epoch restore, HIPRTC-compiled fused scalar relaxation on ROCm, and segment-sum scale gradients.
  • ROCm: Added RDNA4 (gfx1200/gfx1201/gfx1250) kernel and quantization support with per-architecture kernel tuning. MI350/gfx950 behavior is unchanged.
  • GGUF: Added CPU coverage for every ggml tensor type (all K-quant and IQ variants plus explicit MXFP4/NVFP4) and a native ggml SIMD inference path with per-shape autotuning, measured 3–100x faster than the torch paths on AVX2. New gguf install extra.
  • Multi-GPU: Added automatic GPU peer-access enablement on CUDA and ROCm with measured-link validation, so multi-GPU quantization copies stay on the interconnect instead of the driver’s staging path.
  • Experimental W4A: Revalidated the NVFP4 packed layout at load time, removed the retained E4M3 weight expansion, and relaxed the Triton build requirement on the Gluon FP8 GEMM.
  • Performance: Package import now builds the kernel registry lazily and defers optional backend probes (previously ~9s and 6767 modules). Vectorized the GPTQ/QQQ MSE shrink-grid and AWQ clip-grid evaluation; grid workspace budgets moved from environment variables to a new MSE configuration option.
  • Models: Added RecursiveSI Flint and Raon-1 mini support, Gemma 3 multimodal checkpoint compatibility, Qwen2.5-Omni thinker lm_head quantization, and the Transformers 5.22 Qwen-Omni layout.
  • Compat: Kept the AVX512-BF16 GPTQ/AWQ CPU kernels trainable for PEFT LoRA; fixed OpenAI-server auth, stop handling, and ChatCompletion schema; surfaced compatibility-shim failures instead of failing silently; fixed the Marlin CPU-offload snapshot refresh; fixed the calibration minimum-length setting not reaching the dataset filter; re-checked for an emptied calibration set after min-length filtering; tolerated MoE early-stop in data-parallel forward; bounded remote image fetches; read accelerator memory telemetry from the active device; fixed config serialization mutating dynamic overrides; fixed GPTAQ loss-log capture and evaluation-metadata coercion for fast eval tasks.

• 7.7.0:

  • CPU inference: Added the shared AVX512-BF16 int4_qzero_group_scaled_cpu kernel for GPTQ/AWQ group sizes 16/32/64/128/256/512 and prioritized it ahead of ATen when supported. Added packed/low-precision CPU decode paths for QQQ, VPTQ, EXL3, FP8, NVFP4, and BitsAndBytes, plus oneDNN BF16x3 accumulation and shape-aware BitsAndBytes INT8 dispatch. GPTQ/AWQ measured up to 2.02x faster than ATen; BitsAndBytes INT8 reached 1.28x at M=64.
  • GGUF: Added fused CPU Q4_0/Q8_0/Q4_K/Q5_K/Q6_K forward with FP32 accumulation under oneDNN BF16x3, fused bias, per-shape autotuning, and no retained dense weights. K-quant fused paths measured up to 1.92x faster than dense at M=7.
  • Quantized storage: Removed persistent dense, dequantized, expanded, and INT4-to-INT8 inference caches across GPTQ/AWQ, QQQ, EXL3, VPTQ, FP8, FP4, GGUF, BitsAndBytes, and INT8. Only packed weights, scales, zero points, and metadata persist; temporary dequantization uses bounded tiles.
  • Quantization quality: Added token-normalized GPTQ loss reporting, on-device embedding diagonal flooring and VPTQ Hessian damping, and QQQ vLLM compatibility/CUDA saturation fixes.
  • Models and runtimes: Added LLaVA quantization support, fused VPTQ CUDA inference, and hardened remote LoRA adapter loading.
  • Docs and tooling: Added the GPTQ FSDP/PEFT LoRA example, multi-GPU MoE quantization guide, and independent Torch-oracle accuracy and CPU speed benchmarks.
  • Dependencies: Refreshed runtime and optional dependency floors to current stable releases, including PyTorch 2.14.1, Transformers 5.20.2, Datasets 5.1.0, and Optimum 2.3.0. vLLM/SGLang extras retain their upstream framework pins and should be installed in dedicated environments.

• 7.6.0:

  • Model support: Added RecursiveSI Flint, Jamba/Jamba2, SmolVLM, Granite MoE, InternVL, EXAONE 4.5, Qwen3-VL MoE, Qwen4-Exp text, OLMoE, ZDTaichu 5.0, Agnes 3.0 Flash, and Xing4.0 quantization support, plus MiMo V2.6 Flash-RL and Pro-RL checkpoint support.
  • Apple silicon: Added native MLX GPTQ/AWQ quantization and an EXL3 weight-quantization pipeline. Expanded Metal quantization and packed inference kernels for ParoQuant, GGUF, FP8, QQQ, and BitsAndBytes, with GPTAQ/FOEM/RTN acceleration, broader GPTQ/AWQ bit-width and group-size support, Marlin/BitBLAS checkpoint loading, and FP16/BF16 output preservation.
  • GPTQ performance: Added fused CUDA block updates for 2–8-bit quantization, faster Hessian collection and factor reuse, lower peak Hessian memory, and faster Cholesky damping recovery.
  • Experimental W4A: Added Llama decoder activation streams using FP8 on NVIDIA SM89+ or NVFP4 on GB10 (SM121), while preserving native packed GPTQ INT4 checkpoints. See the W4A guide for supported configurations and measured quality.
  • Source formats: Added automatic decoding of native FP8/NVFP4 and OCP MXFP4/MXFP8 source checkpoints for requantization.
  • Runtime and integration: Added stable hf_ kernel selection, packing, and repacking APIs, improved AWQ Triton scheduling, and added opt-in Marlin scratch-buffer reuse. Improved SGLang format validation, dtype handling, and generation parameter compatibility.
  • Reliability: Fixed model-tree isolation and dynamic-rule caching/serialization, offloaded tensor device/dtype restoration, fused DeepSeek expert loading, Qwen3.5-MoE multimodal calibration, Gemma4 calibration KV memory retention, ParoQuant positional replay, GLM NoPE configuration handling, and GPTQ damping recovery after inverse-factor failures.

Older releases and development notes: changelog archive.

A unified quantization platform

GPT-QModel provides a consistent API for calibration, quantization, quality evaluation, model conversion, and accelerated inference. It supports GPTQ, AWQ, ParoQuant, QQQ, GGUF, FP8, EXL3, GPTAQ, EoRA, GAR, FOEM, and opt-in GSQ refinement for compatible scalar methods across supported hardware and runtime integrations.

Its method, format, backend, and kernel layers are modular: method-specific controls remain available where needed, while implementations share the same model lifecycle. This architecture supports mixed and model-specific quantization workflows today and provides a clear integration path for additional methods, formats, kernels, and accelerators.

GPTQ Hessian capture and repeated finalization are described in the Hessian accumulation flow. The GPTQ mock quantization group flow describes group-boundary handling and metadata ordering. AWQ module overrides follow the dynamic group-size flow through search, staging, and packing.

TorchLinear never retains dense/dequantized weights between inference calls. The legacy GPTQ_TORCH_CACHE_WEIGHTS / GPTQ_TORCH_LOOKAHEAD flags and enable_weight_cache(True) / enable_lookahead(True) APIs now warn and leave these features disabled. CUDA streaming uses per-forward tiles, not persistent dequantized workspaces. Only zero-point and group-index metadata is cached; call clear_weight_cache() after manually mutating inference-mode tensors without version counters. Normal tensor mutations, checkpoint loads, and device/dtype moves invalidate this metadata automatically. The ParoQuant rotation identity lifecycle explains how rotation buffer changes, state loads, and module moves refresh its identity shortcut. The CUDA quantized linear epilogue documents wide streaming reductions, bias before low-precision output rounding, fallback for empty FP8 inputs, and CUDA gradient propagation for the portable backends. The GPTQ Torch CPU bias epilogue documents fused low-precision bias addition and FP32 checkpoint bias handling. The low-precision bias epilogues reference keeps FP4, VPTQ, FP8, GGUF, and NF12 bias widths and their failure paths in one place. The GGUF Triton packed metadata cache documents weight update invalidation and the explicit-clear contract for inference tensors. The GGUF Triton activation range documents per-row BF16/FP32 normalization before fused GGUF matmuls. The GGUF packed scale overflow cache documents FP16 operand-width selection for GGUF Torch inference. The GGUF dense fallback precision page explains BF16 weight reconstruction, FP32 bias reload, CPU autocast reduction, and output rounding across formats. The GGUF C++ bias epilogue explains how CPU and CUDA GGML outputs add FP32 bias before low-precision rounding.

BACKEND.GPTQ_TORCH_INT8 (including its TORCH_INT8 alias) accepts only 8-bit GPTQ sources. Converting 2/4-bit weights into a persistent 8-bit runtime representation is no longer supported. AWQ_TORCH_INT8 / TORCH_INT8_AWQ are retired and fail with migration guidance. Use BACKEND.AUTO to select a compatible packed kernel or BACKEND.TORCH for a portable fallback. Native 8-bit GPTQ still uses the existing per-channel INT8 requantizer; it is not a lossless repacking of arbitrary grouped/asymmetric GPTQ weights. See Torch INT8 checkpoint lifecycle for save/reload ownership and runtime cache invalidation.

Quantization Support 🛠️

Start with the quantization guide and end-to-end recipes to choose weight and activation precision, prepare calibration data, quantize, save, reload, and validate a model. It also maps advanced features to their guides.

For checkpoint reshaping, see reshard source selection. For checkpoint-backed module loading, see lazy checkpoint materialization. For the temporary model mutation during calibration input capture and how it is released, see input capture cleanup lifecycle.

Every quantization method shares a common lifecycle—calibrate, quantize, validate, save, load, and serve—while retaining method-specific controls where they matter. New methods and kernels plug into that lifecycle instead of creating another disconnected toolchain.

Module roles—attention projections, MLP projections, MoE experts and shared experts, and quantizable leaves—are resolved from each architecture’s declared module tree, not from runtime module names. See model tree resolution for the resolution flow, the declaration syntax, and the fail-closed contracts. Expert cardinality follows the same rule: a # placeholder expands only when the definition declares dynamic_expert_index and the owning config node resolves it, so expert cardinality resolution covers that step and why a surviving placeholder is a silent miss.

Use the module-plan guide for model.module_plan(), terminal/JSON inspection, model-definition tags, and the required new-model tests. The DeepSeek V4.1 Flash inventory shows its verified checkpoint structure and remaining integration requirements.

The fusion planner requires a model definition’s FusableConfig and uses the compiled module plan for opt-in post-quantization QKV and gate/up fusion. The model qualification matrix lists validated profiles for 101 model types and explicit qualification gaps for the remaining registry entries. Definitions reuse immutable role profiles with FusableConfig("qwen2").merge(LlamaFusableConfig()); concrete paths and parents resolve exclusively from the compiled tree, with no HF class-path allowlist; unqualified model types raise when fusion is requested. Fusion is disabled by default. Use plan = model.fuse() to apply it and receive the plan, plan.print() or plan.to_dict() to inspect eligibility and skip reasons, and model.fuse(False) to disable it and return None. Original projection weights become views into the fused packed storage, preserving checkpoint keys without retaining duplicate weights. AUTO retains native ATen/AVX512 on CPU, Marlin on NVIDIA, and eligible Torch/Triton paths on AMD. See CPU, CUDA and ROCm validation and the complete kernel inventory for tested native backends, dtypes, accuracy results and explicit exclusions.

Feature	GPT-QModel	Transformers	vLLM	SGLang	Lora Training
GPTQ	✅	✅	✅	✅	✅
AWQ	✅	✅	✅	✅	✅
ParoQuant	✅	x	x	x	✅
GGUF	✅	x	x	x	x
FP8	✅	x	x	x	x
Exllama V3 / EXL3	✅	x	x	x	x
EoRA	✅	✅	✅	✅	x
Group Aware Act Reordering	✅	✅	✅	✅	✅
QQQ	✅	x	x	x	x
VPTQ	✅	x	x	x	x
NF12-AD	✅	x	x	x	x
Rotation	✅	x	x	x	x
GPTAQ	✅	✅	✅	✅	✅
FOEM	✅	✅	✅	✅	✅
GSQ (GPTQ, AWQ, RTN, QQQ, FP8, ParoQuant)	✅	Existing format†	Existing format†	Existing format†	Existing format†

GGUF, FP8, EXL3, ParoQuant, VPTQ, and NF12 are currently native GPT-QModel quantization/runtime paths. SGLang loading is limited to METHOD.GPTQ with FORMAT.GPTQ, FORMAT.GPTQ_V2, or FORMAT.MARLIN, and METHOD.AWQ with FORMAT.GEMM or FORMAT.MARLIN.

† GSQ runs during quantization and saves an existing method format. Each external runtime or training integration supports only the formats it already accepts; GSQ does not add a new runtime format.

SGLang accepts the common engine aliases tensor_parallel_size → tp_size, gpu_memory_utilization → mem_fraction_static, max_model_len → context_length, seed → random_seed, and enforce_eager → disable_cuda_graph. An explicit legal dtype is preserved; deprecated torch_dtype is normalized to SGLang’s string dtype names. AWQ FORMAT.GEMV_FAST and FORMAT.LLM_AWQ still require torch.float16, but those formats are not in SGLang’s supported-format list.

Quant Method / Format / Backend Matrix 📋

Canonical backend names are shown below. Method-specific aliases are only accepted where explicitly implemented by that quant method.

Quant Method	Formats	Backends / Kernels
METHOD.GPTQ CPU	FORMAT.GPTQ, FORMAT.GPTQ_V2	BACKEND.GPTQ_TORCH_CPU_AVX512
METHOD.AWQ CPU	FORMAT.GEMM	BACKEND.AWQ_TORCH_CPU_AVX512
METHOD.GPTQ	FORMAT.GPTQ, FORMAT.GPTQ_V2, FORMAT.GPTQ_P, FORMAT.MARLIN, FORMAT.BITBLAS	FORMAT.GPTQ: BACKEND.GPTQ_TORCH_ATEN, BACKEND.MACHETE, BACKEND.MARLIN, BACKEND.GPTQ_EXLLAMA_V2, BACKEND.GPTQ_TORCH_FUSED, BACKEND.GPTQ_TRITON, BACKEND.GPTQ_BITBLAS, BACKEND.GPTQ_TORCH, BACKEND.GPTQ_TORCH_INT8 FORMAT.GPTQ_V2: BACKEND.GPTQ_TORCH_ATEN, BACKEND.GPTQ_EXLLAMA_V2, BACKEND.GPTQ_TORCH_FUSED, BACKEND.GPTQ_TRITON, BACKEND.GPTQ_BITBLAS, BACKEND.GPTQ_TORCH, BACKEND.GPTQ_TORCH_INT8 FORMAT.GPTQ_P: BACKEND.GPTQ_TORCH FORMAT.MARLIN: BACKEND.MARLIN FORMAT.BITBLAS: BACKEND.GPTQ_BITBLAS
METHOD.AWQ	FORMAT.GEMM, FORMAT.GEMV, FORMAT.GEMV_FAST, FORMAT.LLM_AWQ, FORMAT.MARLIN, FORMAT.BITBLAS	FORMAT.GEMM: BACKEND.AWQ_TORCH_ATEN, BACKEND.MACHETE, BACKEND.MARLIN, BACKEND.AWQ_EXLLAMA_V2, BACKEND.AWQ_GEMM, BACKEND.AWQ_GEMM_TRITON, BACKEND.AWQ_TORCH_FUSED, BACKEND.AWQ_TORCH, BACKEND.AWQ_TORCH_INT8, BACKEND.AWQ_BITBLAS FORMAT.GEMV: BACKEND.AWQ_GEMV FORMAT.GEMV_FAST: BACKEND.AWQ_GEMV_FAST FORMAT.LLM_AWQ: BACKEND.AWQ_GEMV_FAST FORMAT.MARLIN: BACKEND.MACHETE, BACKEND.MARLIN FORMAT.BITBLAS: BACKEND.AWQ_BITBLAS
METHOD.PARO	FORMAT.PAROQUANT	BACKEND.PAROQUANT_CUDA, BACKEND.PAROQUANT_TRITON
METHOD.QQQ	FORMAT.QQQ	BACKEND.QQQ, BACKEND.QQQ_TORCH
METHOD.VPTQ	FORMAT.VPTQ	BACKEND.VPTQ_CUDA, BACKEND.VPTQ_TORCH
METHOD.NF12_AD (nf12)	FORMAT.NF12_AD (nf12)	BACKEND.NF12_AD (nf12_ad)
METHOD.GGUF	FORMAT.GGUF	BACKEND.GGUF_TRITON, BACKEND.GGUF_CPP_CUDA, BACKEND.GGUF_CPP_CPU, BACKEND.GGUF_TORCH
METHOD.FP8	FORMAT.FP8	BACKEND.FP8_TORCH
METHOD.BITSANDBYTES	FORMAT.BITSANDBYTES	BACKEND.BITSANDBYTES
METHOD.EXL3	FORMAT.EXL3	BACKEND.EXL3_EXLLAMA_V3, BACKEND.EXL3_TORCH

Marlin, Machete, and Swordfish are each one backend (BACKEND.MARLIN, BACKEND.MACHETE, BACKEND.SWORDFISH). The quantization method and checkpoint format select the GPTQ or AWQ sub-kernel automatically, so gptq_marlin/ awq_marlin, gptq_machete/awq_machete, and gptq_swordfish/ awq_swordfish are no longer separate backend names. The stable hf_ packing and repacking APIs still accept those legacy strings and resolve them to the shared backend.

BACKEND.GPTQ_TORCH_CPU_AVX512 and BACKEND.AWQ_TORCH_CPU_AVX512 share the same AVX512-BF16 int4 GEMM on Linux x86-64 hosts with AVX512-BF16, AVX512-VBMI, and OS AVX512 state enabled. GPTQ and AWQ use separate fused checkpoint-to-kernel packers but no separate compute kernels. AUTO selects the AVX512 path ahead of the ATen int4pack fallback when the layer shape satisfies the kernel’s 32-row and group-size alignment requirements. Supported group sizes are 16, 32, 64, 128, 256, and 512.

BACKEND.GPTQ_TORCH_FUSED keeps INT4 weights in the packed int4pack layout and reorders activations only when the g_idx permutation is not the identity; BACKEND.AWQ_TORCH_FUSED uses the same packed layout without a g_idx activation reorder. See Torch Fused INT4 transformations for asymmetric GPTQ zero-point offsets, packing, and activation-reorder contracts.

BACKEND.QQQ_TORCH uses the same packed INT4 CPU layout with a QQQ-specific integer GEMM that preserves round-to-nearest-even and INT8 saturation for group size 128, plus the signed per-channel layout for group size -1. Small batches use AVX512 integer dot products; larger batches use oneDNN integer GEMM with temporary 128-by-32 weight tiles when available. Runtime caches retain only packed INT4 codes and scales. Unsupported hosts use the Torch fallback without retaining dequantized weights. Both FP16 and BF16 activations keep their output dtype.

BACKEND.EXL3_TORCH uses oneDNN BF16x3 FP32 matmul on AVX512-BF16 hosts for GEMMs large enough to amortize dispatch. Forward decodes at most 512 output columns at a time and never retains a dense dequantized weight; get_weight_tensor() materializes one on demand for export and tests. Set GPTQMODEL_EXL3_CPU_BF16X3=0 to disable the BF16x3 path.

BACKEND.VLLM, BACKEND.SGLANG, and BACKEND.MLX are external runtime backends and are not part of the native kernel matrix above.

Native backend availability, shape validation, wide activation inputs, and derived cache invalidation follow the kernel validation and cache lifecycle contract.

FP8 source weights must be finite before quantization; see the FP8 source weight validation contract.

GSQ is an opt-in scalar refinement for supported METHOD.GPTQ, METHOD.AWQ, METHOD.QQQ, METHOD.FP8, METHOD.PARO, and METHOD.RTN paths. Staged block training currently supports dense Llama GPTQ and AWQ export. Each path retains its method’s existing checkpoint format; method-specific GSQ constraints are described below.

AWQ FP16 boundary staging and packed decode follow the AWQ affine range contract.

Marlin uses GPTQMODEL_MARLIN_USE_FP32 (default: enabled) to control fp32 accumulation.

On ROCm, quantization runs with torch’s rocBLAS backend instead of hipBLASLt: hipBLASLt loads one Tensile library per (dtype, layout) pair on first use, and those one-time loads dominate a cold quantize while rocBLAS covers the same math. The override is scoped to the quantization phase and the previous backend is restored afterwards. Set GPTQMODEL_ROCM_QUANT_BLAS=hipblaslt to keep torch’s default backend. This applies to RDNA4 as well; the RDNA4 hipBLASLt Tensile libraries exist but are not tuned, so rocBLAS remains the default there too.

Features ✨

• ✨ Native integration with HF Transformers, Optimum, and Peft
• ✨ Stable hf_ packing/repacking API for layer-wise kernel feasibility, packing, and repacking used by external integrators such as AutoRound.
• 🚀 vLLM and SGLang inference integration for quantized models. SGLang supports GPTQ FORMAT.GPTQ/FORMAT.GPTQ_V2/FORMAT.MARLIN and AWQ FORMAT.GEMM/FORMAT.MARLIN.
• ✨ GPTQ, AWQ, ParoQuant, QQQ, GGUF, FP8, EXL3, GPTAQ, and FOEM quantization support, plus opt-in GSQ refinement for compatible GPTQ, AWQ, RTN, QQQ, FP8, and ParoQuant paths.
• ✨ GPTQ, AWQ, ParoQuant, QQQ, VPTQ, GGUF, FP8, EXL3, GPTAQ, and FOEM quantization support, plus opt-in GSQ refinement for compatible GPTQ, AWQ, RTN, QQQ, FP8, and ParoQuant paths.
• ✨ All current GGUF quantized tensor types are supported end to end for CPU quantization and inference: every Q*_0/Q*_1, Q2_K-Q8_K, IQ1_S-IQ4_XS, TQ1_0/TQ2_0, MXFP4, NVFP4, Q1_0, Q2_0, and the Q1_0_g128 sign-only variant. Install the bindings with pip install gptqmodel[gguf] (gguf>=0.19.0, llama-cpp-python>=0.3.36). CPU inference uses llama.cpp’s native SIMD kernels (BACKEND.GGUF_CPP_CPU) with a per-shape autotuned Torch fallback. Native ggml dot kernels also quantize activations, so outputs can differ from weight-only Torch inference; set GPTQMODEL_GGUF_CPP_NATIVE_MATMUL=0 to keep the Torch path. Q8_1/Q8_K always use Torch, including when native execution is forced, since upstream ggml exposes no matmul for them. CPU dequantization reconstructs weights in FP32 before casting to FP16/BF16. See the GGUF compatibility notes.
• 🚀 Routing-aware MoE quantization controls for extreme activation bias via Moe.Routing and/or FailSafe.
• 🚀 Data Parallelism for 80%+ quantization speed reduction with Multi-GPU.
• 🚀 Optimized for Python >= 3.13t (free threading) with lock-free threading.
• ✨ Linux, macOS, Windows platform support for CUDA (NVIDIA), NPU (Huawei Ascend), XPU (Intel), ROCm (AMD), MPS (Apple Silicon), CPU (Intel/AMD/Apple Silicon).
• ✨ Dynamic per-module mixed quantization control: each layer/module can have a unique quantization config or be excluded from quantization.
• 🚀 Intel Torch 2.8 fused kernel support for XPU [Arc + Datacenter Max] and CPU [avx, amx].
• 🚀 Python 3.13.3t (free-threading, GIL disabled) support for multi-GPU accelerated quantization for MoE models and multi-core CPU boost for packing.
• ✨ Asymmetric Sym=False support.
• ✨ lm_head module quant inference support for further VRAM reduction.
• 🚀 Microsoft/BITBLAS optimized tile based inference.
• 💯 CI unit-test coverage for all supported models and kernels including post-quantization quality regression.

Organizations and projects using GPT-QModel 🌐

Selected public references where teams or companies explicitly mention GPT-QModel in documentation, integration notes, or quantized model usage. This is not an exhaustive customer list.

• Hugging Face
• Intel
• NVIDIA
• Alibaba Cloud

Quality: GPTQ 4-bit evaluation 🏆

🤗 ModelCloud quantized Vortex models on HF

Model Support 🤖

The table mirrors every explicit registration in gptqmodel.models.auto.MODEL_MAP, including text-model and backward-compatibility aliases. Qwen 3.5 registrations require Transformers 5.2.0 or newer.

Model family	Registered Transformers model_type values
A.X-K2	axk2
AfMoE / Trinity	afmoe
Agnes 3.0 Flash	agnes
Apertus 1 / 1.5	apertus, apertus1p5, apertus1p5_text
Baichuan	baichuan
Bailing MoE / Hybrid (LING / RING)	bailing_moe, bailing_hybrid
Bloom	bloom
Brumby	brumby
ChatGLM	chatglm
CodeGen	codegen
Cohere 1 / 2 / 2 MoE / Compass (North Mini / Micro Vision)	cohere, cohere2, cohere2_moe, cohere_compass
DBRX / DBRX Converted	dbrx, dbrx_converted
DeciLM	deci
DeepSeek V2 / V3 / V3.2 / V4 / VL / VL2 / OCR2	deepseek_v2, deepseek_v3, deepseek_v32, deepseek_v4, deepseek_vl, deepseek_vl_v2, deepseek_ocr2
KRAFTON Raon-1 mini	raon1_mini
DiffusionGemma	diffusion_gemma
Dots1	dots1
Dream	dream
ERNIE 4.5 / MoE / VL MoE	ernie4_5, ernie4_5_moe, ernie4_5_moe_vl, ernie4_5_vl_moe
EXAONE 3 / 4 / 4.5	exaone, exaone4, exaone4_5
Falcon / Falcon H1 / Falcon Mamba	falcon, falcon_h1, falcon_mamba
Gemma 1-4 / 3n / Unified	gemma, gemma2, gemma3, gemma3_text, gemma3n, gemma3n_text, gemma4, gemma4_text, gemma4_unified, gemma4_unified_text
GLM / GLM4 / GLM4V / GLM5 / OCR / ASR	glm, glm4, glm4_moe, glm4_moe_lite, glm4v, glm4v_moe, glm4v_moe_text, glm5_next, glm_moe_dsa, glm_ocr, glmasr
GPT-2	gpt2
GPT BigCode	gpt_bigcode
GPT-Neo / GPT-NeoX	gpt_neo, gpt_neox
GPT-OSS	gpt_oss
GPT-J	gptj
Granite / Granite MoE / Granite MoE Hybrid	granite, granitemoe, granitemoehybrid
GRIN-MoE	grinmoe
Helium	helium
HRM	hrm_text
Hunyuan V1 / VL / OCR	hunyuan_v1_dense, hunyuan_v1_moe, hunyuan_vl
HY-V3	hy_v3
Hymba	hymba
Inkling	inkling_mm_model
Instella	instella
Intern S1 / S2 Preview	interns1, intern_s2_preview
InternLM 1 / 2 / 2.5	internlm, internlm2
InternVL 3 / InternVL Chat	internvl, internvl_chat
Jamba / Jamba2	jamba
K2-Horizon (Dense / MoVA)
Kimi K2 / K2.5	kimi_k2, kimi_k25
Klear	klear
Laguna	laguna
LFM2 / LFM2 MoE / LFM2-VL	lfm2, lfm2_moe, lfm2_vl
LFM3 / LFM3 Omni / LFM3-VL	lfm3, lfm3_omni, lfm3_vl
LLaDA2 MoE	llada2_moe
Llama 1-4 / TinyLlama / Nemotron Ultra	llama, llama4, llama4_text
Llama 3.2 VL	mllama, mllama_text_model
LLaVA-1.5	llava
FastVLM / LLaVA-Qwen2	llava_qwen2
LocateAnything	locateanything
LongCat Flash	longcat_flash
LongLLaMA	longllama
Mage-VL	mage_vl
Marin	marin
MiMo / MiMo V2	mimo, mimo_v2
MiniCPM / MiniCPM3 / MiniCPM-O / MiniCPM-V	minicpm, minicpm3, minicpmo, minicpmv, minicpmv4_6
MiniMax M2 / M3-VL	minimax, minimax_m2, minimax_m3_vl
Mistral / Mistral3 / Ministral3	mistral, mistral3, ministral3
Mixtral	mixtral
MobileLLM	mobilellm
MOSS	moss
MPT	mpt
Muse Glimmer	muse_glimmer
Nanbeige 4.2	nanbeige
Nemotron NAS / H / H Puzzle / Omni / Labs Diffusion	nemotron-nas, nemotron_h, nemotron_h_puzzle, nemotronh_nano_omni_reasoning_v3, nemotron_labs_diffusion
OLMo / OLMo 2 / 3 / OLMoE	olmo, olmo2, olmo3, olmoe
OpenVLA	openvla
OPT	opt
Ouro	ouro
Ovis 1.6 / 2 / 2.5 / 2.6 MoE / 2.6 Next	ovis, ovis2, ovis2_5, ovis2_6_moe, ovis2_6_next
PanGu-α	gpt_pangu
Phi 1-4 / Phi MoE	phi, phi3, phi4mm, phimoe
Qwen 1-4 / 3.5 / 3.6 / 3.8 / MoE / Next	qwen, qwen2, qwen2_moe, qwen3, qwen3_moe, qwen3_next, qwen3_5, qwen3_5_text, qwen3_5_moe, qwen3_5_moe_text, qwen4_exp, qwen4_exp_text
Qwen 2 / 2.5 / 3 VL / 3 VL MoE	qwen2_vl, qwen2_vl_text, qwen2_5_vl, qwen2_5_vl_text, qwen3_vl, qwen3_vl_moe
Qwen 2.5 / 3 Omni	qwen2_5_omni, qwen3_omni_moe
Qwen2-Audio	qwen2_audio
Qwen-Drive 1.0	qwen_drive
RefinedWeb	refinedWeb, refinedWebModel
Ricursive Ripple (cached generation)	ricursive
Seed-OSS	seed_oss
SmolLM3	smollm3
SmolVLM	smolvlm
Solar Open / Open 2	solar_open, solar_open2
Spark 2.5	spark2_5
StableLM	stablelm, stablelm_epoch
StarCoder2	starcoder2
TeleChat2	telechat
Unlimited-OCR	unlimited_ocr
Voxtral	voxtral
Xing4.0 (29B-A4B)	xing4_0
XVERSE	xverse
Yi	yi
Zamba / Zamba2	zamba, zamba2
ZDTaichu 5.0	zdtaichu5_0

Qwen2.5-Omni supports quantizing its text output head with lm_head=True. The head is thinker.lm_head; GPTQ defaults it to 8-bit, group size 32. To use a 4-bit head, configure an explicit dynamic override:

from gptqmodel import QuantizeConfig

quantize_config = QuantizeConfig(
    bits=4,
    group_size=128,
    lm_head=True,
    dynamic={r"thinker\.lm_head$": {"bits": 4, "group_size": 32, "desc_act": False}},
)

This option quantizes the thinker’s text head; the talker and audio decoder retain their original weights. Tied output-head weights must be untied first.

Platform and HW Support 🖥️

GPT-QModel is validated on Linux, macOS, and Windows 11:

Platform	Device		Optimized Arch	Kernels
🐧 Linux	NVIDIA GPU	✅	Turing+ (sm_75+), NF12-AD sm_80+	Machete, Marlin, Exllama V3 / EXL3, Exllama V2, AWQ GEMM/GEMV, ParoQuant CUDA/Triton, GGUF CUDA/Triton, QQQ, VPTQ, NF12-AD, BitBLAS, Triton, BitsAndBytes, Torch
🐧 Linux	AMD GPU	✅	7900XT+, ROCm 10.1+, gfx950 (CDNA4), gfx1200/1201/1250 (RDNA4)	Exllama V2, AWQ Triton GEMM/GEMV, ParoQuant Triton, GGUF Triton, BitsAndBytes 4-bit Triton, QQQ, FP8 Torch, Torch (fused GPTQ GEMV/GEMM on gfx950 and gfx120*/gfx125*)
🐧 Linux	Huawei Ascend NPU	✅	Ascend 910B, torch-npu / CANN	Native Torch kernels for GPTQ, AWQ, ParoQuant, GGUF, QQQ, and EXL3
🐧 Linux	Intel XPU	✅	Arc, Datacenter Max	TorchFused, TorchFusedAWQ, FP8 Torch, Torch
🐧 Linux	Intel/AMD CPU	✅	avx, amx	TorchFused, TorchFusedAWQ, TorchAten int4, TorchInt8, GGUF C++, BitsAndBytes, VPTQ, NF12-AD Torch, Torch
🍎 macOS	GPU (Metal) / CPU	✅	Apple Silicon, M1+	Torch, FP8 Torch, NF12-AD Torch, MLX via conversion
🪟 Windows	GPU (NVIDIA) / CPU	✅	NVIDIA	Torch

ROCm 10.1 or newer is required. GPT-QModel only supports and validates ROCm >= 10.1; older releases are unsupported and import gptqmodel or model loading logs a warning asking you to upgrade. Use the latest ROCm release and a matching ROCm PyTorch build. ROCm 10.1 ships HIP 7.16; torch.version.hip reports the HIP runtime version, not the ROCm release, so version checks use torch.version.rocm.

ROCm pip SDK installations that expose _rocm_sdk_core as ROCM_HOME need the matching rocm-sdk-devel package and rocm-sdk init before building JIT kernels. JIT builds on ROCm always use the HIP/torch headers only: NVIDIA nvidia wheel headers are never added to a hipcc command line, even when those packages are installed in the environment, because their vector_types.h conflicts with HIP’s vector type aliases.

On gfx950 (CDNA4) and the gfx120*/gfx125* RDNA4 family, the Torch GPTQ backend uses a fused Triton GEMV for single-token 2/4/8-bit inference and a fused small-M GEMM for M=2…16, and falls back to the existing path for other shapes or unregistered architectures. Launch constants are selected per architecture by gptqmodel/utils/arch.py. On gfx1201 (R9700) the fused small-M GEMM measured 1.1x-3.2x faster than dequantize+matmul for M=2…16 in FP16 up to K=17408, and its .amdgcn contains v_wmma_f32_16x16x16_f16. The fused small-M GEMM is FP16-only on RDNA4: BF16 measured 0.43x-1.42x of the split path there (median 0.85x), so BF16 falls back to dequantize+matmul (GEMV keeps both dtypes). Set GPTQ_TORCH_ROCM_GEMV=0 / GPTQ_TORCH_ROCM_GEMM=0 to disable the fused paths. See docs/rocm-rdna4.md for the per-backend RDNA4 matrix and known limitations.

bitsandbytes 4-bit (NF4/FP4) inference on ROCm routes M >= 16 through a fused Triton GEMM that dequantizes the packed codes inside the matmul, because the vendor HIP kernel switches to a much slower path at that row count. Measured on gfx1201 (R9700, FP16) it is 1.4x-6.3x faster than bnb.matmul_4bit; below M = 16 bitsandbytes’ own GEMV is faster and is kept. CUDA/Ada keeps the vendor kernels, which are faster there; set GPTQMODEL_BNB_TRITON=1 to force the Triton path for A/B runs.

Marlin and JIT CUDA kernels now support NVIDIA Turing+ (sm_75+) GPUs. Huawei Ascend NPU support uses native Torch kernels through torch-npu / CANN.

Native MLX quantization on Apple silicon

Install the optional MLX dependency and quantize a dense model directly to an MLX-LM checkpoint:

pip install 'gptqmodel[mlx]'

from gptqmodel import GPTQModel

GPTQModel.quantize_mlx(
    "Qwen/Qwen3-0.6B",
    "./qwen3-mlx-gptq",
    method="gptq",  # or "awq"
    bits=4,
    group_size=64,
    num_samples=128,
)

GPTQ uses MLX for calibration and weight updates, with one fused Metal dispatch per weight group. MLX currently performs the Cholesky factorization on its CPU stream. AWQ uses MLX-LM’s native scale and clip search. The output uses MLX-LM’s quantized model format and can be loaded with mlx_lm.load. Pass an MLX token array as calibration_data to use your own calibration set.

Install 💾

PIP/UV 💿

# vLLM and SGLang publish their own Torch/Transformers pins. Install one engine
# in a dedicated environment aligned to that engine's release.
# Example optional module: pip install -v "gptqmodel[bitblas]"
pip install -v gptqmodel
uv pip install -v gptqmodel

The package depends on ninja for first-use JIT kernel compilation.

AMD ROCm support requires ROCm >= 10.1 and a matching ROCm PyTorch build. Older ROCm releases are unsupported; GPT-QModel logs a warning at import/model load time and asks you to upgrade.

Install from source 🛠️

# clone repo
git clone https://github.com/ModelCloud/GPTQModel.git && cd GPTQModel

# python3-dev is required for some source installs
apt install python3-dev

# pip: install from source
# Example optional module: pip install -v ".[bitblas]"
pip install -v .

Inference 🔮

Three-line API to use GPT-QModel for GPTQ model inference:

from gptqmodel import GPTQModel

model = GPTQModel.load("ModelCloud/Llama-3.2-1B-Instruct-gptqmodel-4bit-vortex-v2.5")
result = model.generate("Uncovering deep insights begins with")[0] # tokens
print(model.tokenizer.decode(result)) # string output

On Apple Silicon, install the optional MLX runtime with pip install "gptqmodel[mlx]". GPTQModel.load(..., backend=BACKEND.AUTO) selects MLX Metal for supported 4-bit GPTQ and AWQ GEMM checkpoints when the model architecture is supported by MLX-LM. The converter transfers compatible packed weights directly to MLX’s quantized matmul layout. Supported affine group sizes are 32, 64, and 128. Explicit backend choices remain available.

from gptqmodel import GPTQModel

model = GPTQModel.load("ModelCloud/Qwen2.5-0.5B-Instruct-gptqmodel-w4a16")
print(model.generate(prompt="The capital of France is", max_tokens=16))

To use models from ModelScope instead of HuggingFace Hub, set an environment variable:

export GPTQMODEL_USE_MODELSCOPE=True

FP32 accumulation toggle 🔢

Some AWQ and ParoQuant CUDA/Triton kernels support an fp32 accumulation mode to reduce numerical drift during fused quantized matmul. This setting defaults to True because accuracy is prioritized over speed. Biased ParoQuant Triton calls always accumulate and add bias in FP32 before output rounding, including when this toggle is 0. See the ParoQuant Triton bias epilogue.

# default behavior: higher accuracy, slightly lower speed on some kernels
export GPTQMODEL_FP32_ACCUM=1

# optional speed-first mode for some kernels
export GPTQMODEL_FP32_ACCUM=0

JIT kernel cache and multi-process quantization ⚙️

JIT-compiled kernels are cached at ~/.cache/gptqmodel/torch_extensions by default. Multiple processes on one host may safely share the cache: builds are serialized with a cross-process file lock that the OS releases automatically if a process dies.

Machete’s pinned CUTLASS checkout and generated CUDA sources are kept in the versioned user cache (~/.cache/gptqmodel, or GPTQMODEL_CACHE_DIR; when set, XDG_CACHE_HOME/gptqmodel is used). Set GPTQMODEL_CUTLASS_DIR to use an already-installed, read-only CUTLASS 4.7.1 checkout. GPTQMODEL_OFFLINE=1 disables downloads and requires that checkout or a verified cache hit already exist. For environments where compilation is not allowed, point GPTQMODEL_MACHETE_PRECOMPILED_LIBRARY at a compatible Machete shared library; an invalid or missing explicit library is reported as an error and does not fall back to JIT. The generated source cache can be populated ahead of time by prewarming the extension:

python -c "from gptqmodel import extension; extension.load('machete')"

# optional: relocate the kernel cache (e.g. one cache per process)
export GPTQMODEL_TORCH_EXTENSIONS_DIR=/path/to/cache

# optional: max seconds to wait for another process's in-flight build before
# falling back to non-JIT paths. Default: 600 or 5x the kernel's compile
# baseline, whichever is larger.
export GPTQMODEL_TORCH_OPS_LOCK_TIMEOUT=600

Notes:

• This is a runtime toggle. It does not change model weights or saved checkpoints.
• It mainly affects some fused AWQ and ParoQuant CUDA/Triton kernels. Dense/dequantize fallback paths are mostly unaffected.
• 1 is recommended for regression testing and quality-sensitive evaluation. 0 may be useful when chasing a small latency win and the quality tradeoff is acceptable.

OpenAI API compatible endpoint 🌐

# load model using above inference guide first
model.serve(host="127.0.0.1", port=8000)  # the bind address is your choice

# Exposing the port beyond localhost requires an API key (or GPTQMODEL_OPENAI_API_KEY).
model.serve(host="0.0.0.0", port=8000, api_key="<secret>")

# Multiple keys are accepted; any of them authorizes a request.
model.serve(host="0.0.0.0", port=8000, api_key=["<k1>", "<k2>"])

When at least one key is configured, every path under /v1, /v2 and /inference requires Authorization: Bearer <key>; /health, / and CORS preflight stay open. An explicitly passed api_key replaces (does not merge with) GPTQMODEL_OPENAI_API_KEY, and blank keys are ignored. Starting on a non-loopback host without a key only emits a warning — the server still starts. /shutdown is limited to loopback callers unless a valid key is presented.

Quantization 🔧

Example: quantizing an LLM with GPT-QModel:

from datasets import load_dataset
from gptqmodel import GPTQConfig, GPTQModel

model_id = "meta-llama/Llama-3.2-1B-Instruct"
quant_path = "Llama-3.2-1B-Instruct-gptqmodel-4bit"

calibration_dataset = load_dataset(
    "allenai/c4",
    data_files="en/c4-train.00001-of-01024.json.gz",
    split="train"
  ).select(range(1024))["text"]

quant_config = GPTQConfig(bits=4, group_size=128)

model = GPTQModel.load(model_id, quant_config)

# increase `batch_size` to match GPU/VRAM specs to speed up quantization
model.quantize(calibration_dataset, batch_size=1)

model.save(quant_path)

Calibration grid workspaces

GPTQ/QQQ MSE shrink searches and AWQ clipping searches evaluate candidates in chunks. Their temporary workspace targets can be adjusted per QuantizeConfig via MSEConfig on the mse field (a bare float, e.g. mse=2.4, still works and only sets the search exponent norm):

from gptqmodel import GPTQConfig, MSEConfig

quant_config = GPTQConfig(
    bits=4,
    group_size=128,
    mse=MSEConfig(norm=2.4, grid_budget="64M", clip_grid_budget="64M"),
)

MSEConfig field	Search	CPU default	GPU default
norm	MSE objective exponent (0 disables the search)	—	—
grid_budget	GPTQ/QQQ MSE shrink grid	16M	256M
clip_grid_budget	AWQ clipping grid	16M	256M

Budget values are byte counts (ints) or strings with a K/M/G postfix (e.g. "256M", "1G").

Set either field to 0 to evaluate one candidate at a time; AWQ also uses one-token product tiles. These are chunk-sizing targets, not limits on total process memory: AWQ sizes its scratch buffers independently, and a single candidate, full-token error buffer, or minimum product tile can exceed the target. These settings affect calibration scratch only, not stored quantized weights.

Checkpoint and Resume Long Quantization Jobs 💾

Checkpointing commits completed transformer layers so an interrupted quantization job can reload the original model and continue without repeating those layers. Use an explicit checkpoint path that survives process restarts:

from gptqmodel import CheckpointConfig, CheckpointStopped, GPTQConfig, GPTQModel

quant_config = GPTQConfig(
    bits=4,
    group_size=128,
    offload_to_disk=True,
)
model = GPTQModel.load(model_id, quant_config)

try:
    model.quantize(
        calibration_dataset,
        batch_size=1,
        checkpoint=CheckpointConfig(
            path="checkpoints/Llama-3.2-1B-Instruct-gptqmodel-4bit",
            resume="auto",
            interval="layer:1",
            keep_last=2,
        ),
    )
except CheckpointStopped:
    # Run the script again with the same source, calibration, config, and path.
    raise SystemExit(75)

model.save(quant_path)

Ctrl+C and SIGTERM finish the current layer, publish a safe checkpoint, and raise CheckpointStopped. A hard kill resumes from the last published layer; the unfinished layer is repeated. Resume validation requires the same source weights, calibration data, quantization settings, software versions, and device topology. Current checkpoint support is limited to Llama and Qwen3 MoE model types, requires offload_to_disk=True, and does not yet support dynamic exclusions, rotation, or embedding/lm_head quantization.

See Quantization checkpointing and resume for resume policies, supported methods, safe-stop behavior, storage requirements, recovery procedures, and operational gotchas.

For writers that reference an existing safetensors payload instead of making a second offload copy, see Safetensors reference offload. For nested offload tensor names, see Offload index resolution.

Other Quantization Formats 📦

QuantizeConfig remains the broad factory. The concrete config classes are now GPTQConfig, AWQConfig, ParoConfig, QQQConfig, VPTQConfig, NF12ADConfig, RTNConfig, GGUFConfig, FP8Config, BitsAndBytesConfig, and EXL3Config.

GPTQ, AWQ, ParoQuant, VPTQ, and EXL3 are calibration-based. GGUF, FP8, BitsAndBytes, and NF12 are weight-only and should be quantized with calibration=None.

QQQ Example: Llama 3.2 1B Instruct

QQQ supports 4-bit symmetric weights with group_size=-1 or group_size=128. Use desc_act=False: the packed QQQ runtime stores contiguous group scales and does not support GPTQ-style activation reordering. MSE scale search (mse=2.4) is the recommended quality recipe.

import torch
from datasets import load_dataset
from gptqmodel import GPTQModel, QQQConfig

model_id = "meta-llama/Llama-3.2-1B-Instruct"
quant_path = "Llama-3.2-1B-Instruct-QQQ-4bit"

calibration_dataset = load_dataset(
    "allenai/c4",
    data_files="en/c4-train.00001-of-01024.json.gz",
    split="train",
).select(range(256))["text"]

quant_config = QQQConfig(
    bits=4,
    group_size=128,
    desc_act=False,
    mse=2.4,
)

model = GPTQModel.load(model_id, quant_config, dtype=torch.float16)
model.quantize(
    calibration_dataset,
    batch_size=1,
    calibration_concat_size=2048,
)
model.save(quant_path)

Experimental W4A streams on NVIDIA GB10 (SM121) keep GPTQ INT4 weights and quantize the activations: E4M3 FP8 with a per-token scale, or E2M1 NVFP4 with E4M3 block scales. Selected Llama decoder operators pass encoded activations and scales across module and layer boundaries instead of decoding to BF16. See the W4A guide for the policy reference, and the accuracy appendix for measured quality.

Preprocessors 🧹

preprocessors=[...] adds optional module-weight preparation steps before quantization or repacking. They are available on GPTQConfig, AWQConfig, ParoConfig, RTNConfig, GGUFConfig, FP8Config, and BitsAndBytesConfig.

• SmootherConfig: apply weight smoothing before quantization.
• AutoModuleDecoderConfig: decode FP8/FP4 source modules to a dense target_dtype before downstream quantization or repacking.
• TensorParallelPadderConfig: opt-in tensor-parallel padding metadata for TP-aligned packing.

import torch
from gptqmodel import GGUFConfig, GPTQConfig
from gptqmodel.quantization import (
    AutoModuleDecoderConfig,
    SmoothMAD,
    SmootherConfig,
    TensorParallelPadderConfig,
)

gptq_cfg = GPTQConfig(
    bits=4,
    group_size=128,
    preprocessors=[
        SmootherConfig(smooth=SmoothMAD(k=2.0)),
        AutoModuleDecoderConfig(target_dtype=torch.bfloat16),
        TensorParallelPadderConfig(),
    ],
)

gguf_cfg = GGUFConfig(
    bits=4,
    format="q_k_m",
    preprocessors=[
        AutoModuleDecoderConfig(target_dtype=torch.bfloat16),
        TensorParallelPadderConfig(),
    ],
)

GGUF Example: Llama 3.2 1B Instruct

from gptqmodel import BACKEND, GGUFConfig, GPTQModel

model_id = "meta-llama/Llama-3.2-1B-Instruct"
quant_path = "Llama-3.2-1B-Instruct-GGUF-Q4_K_M"

qcfg = GGUFConfig(
    bits=4,
    format="q_k_m",
)

model = GPTQModel.load(model_id, qcfg)
model.quantize(calibration=None, backend=BACKEND.GGUF_TORCH)
model.save(quant_path)

FP8 Example: Llama 3.2 1B Instruct

See FP8 inverse-scale checkpoint loading for how FP32 scales are preserved when loading a BF16 model. See packed FP8 weight lifecycle for checkpoint reload, module moves, and persistent storage.

from gptqmodel import BACKEND, GPTQModel
from gptqmodel.quantization import FP8Config

model_id = "meta-llama/Llama-3.2-1B-Instruct"
quant_path = "Llama-3.2-1B-Instruct-FP8-E4M3"

qcfg = FP8Config(
    format="float8_e4m3fn",  # or "float8_e5m2"
    bits=8,
    weight_scale_method="row",
)

model = GPTQModel.load(model_id, qcfg)
model.quantize(calibration=None, backend=BACKEND.GPTQ_TORCH)
model.save(quant_path)

Exllama V3 / EXL3 Example: Llama 3.2 1B Instruct

from datasets import load_dataset
from gptqmodel import BACKEND, GPTQModel
from gptqmodel.quantization import EXL3Config

model_id = "meta-llama/Llama-3.2-1B-Instruct"
quant_path = "Llama-3.2-1B-Instruct-EXL3"

calibration_dataset = load_dataset(
    "allenai/c4",
    data_files="en/c4-train.00001-of-01024.json.gz",
    split="train",
).select(range(1024))["text"]

qcfg = EXL3Config(
    bits=4.0,        # target average bits-per-weight
    head_bits=6.0,   # optional higher bitrate for attention heads / sensitive tensors
    codebook="mcg",  # one of: mcg, mul1, 3inst
)

model = GPTQModel.load(model_id, qcfg)
model.quantize(calibration_dataset, batch_size=1, backend=BACKEND.EXL3_EXLLAMA_V3)
model.save(quant_path)

See EXL3 inference buffer lifecycle for native-handle invalidation and checkpoint-stable Torch wide-input inference. See EXL3 wide activation dispatch for BF16/FP32 routing, the 32-row native boundary, and packed-weight reconstruction.

VPTQ Example: Llama 3.2 1B Instruct

VPTQ learns vector codebooks from calibration data with GPTQ-style error compensation; Hessians are accumulated online, so no pre-generated Hessian files are needed. See the VPTQ guide for the full configuration reference.

from datasets import load_dataset
from gptqmodel import BACKEND, GPTQModel
from gptqmodel.quantization import VPTQConfig

model_id = "meta-llama/Llama-3.2-1B-Instruct"
quant_path = "Llama-3.2-1B-Instruct-VPTQ"

calibration_dataset = load_dataset(
    "allenai/c4",
    data_files="en/c4-train.00001-of-01024.json.gz",
    split="train",
).select(range(1024))["text"]

qcfg = VPTQConfig(
    vector_lens=[-1, 8],         # 8-weight vectors, no outlier codebook
    num_centroids=[-1, 4096],    # 4096-entry main codebook
    num_res_centroids=[-1, 16],  # 16-entry residual codebook
)

model = GPTQModel.load(model_id, qcfg)
model.quantize(calibration_dataset, batch_size=1, backend=BACKEND.VPTQ_TORCH)
model.save(quant_path)

NF12 Example: lossless 12-bit BF16

NF12 stores BF16 weights in Together AI’s lossless 12-bit NF12 format, using AlpinDale’s tiled layout and CUDA kernels (dphnAI/nf12). It is weight-only, needs no calibration data, and decodes back to the exact BF16 matrix. The modules must satisfy in_features % 64 == 0 and out_features % 16 == 0. The native RDNA4 path was validated on ROCm 10.1.0 / HIP 7.16.26385 with Radeon AI PRO R9700 (gfx1201); see the NF12 HIP toolchain notes.

import torch
from gptqmodel import GPTQModel
from gptqmodel.quantization import NF12ADConfig

model_id = "meta-llama/Llama-3.2-1B-Instruct"
quant_path = "Llama-3.2-1B-Instruct-NF12"

model = GPTQModel.load(model_id, NF12ADConfig(), dtype=torch.bfloat16)
model.quantize(calibration=None)
model.save(quant_path)

The packed weight is saved as nf12_packed (uint8, N x K*3/2) with nf12_escapes and nf12_exp_base, so loaders treat it as a quantized checkpoint rather than a dense weight.

The NF12 validation backend and performance flow explains device selection, native encoder measurements, and Torch fallback. The NF12 fallback epilogue explains mixed-dtype bias handling and BF16 weight-range preservation before output rounding. The NF12 vendored runtime flow describes how GPT-QModel owns the NF12 Python API, native build, and inference dispatch. The NF12 HIP device and workspace flow describes matrix-core device selection, split-K scratch ownership, activation layout normalization, routing bounds, and exact-byte encoder scheduling.

W4A Example: Llama 3.2 1B Instruct (GB10)

W4A quantizes activations while leaving the saved GPTQ checkpoint in its native INT32-packed INT4 layout. It requires NVIDIA GB10 (SM121), a Llama decoder model, bits=4, group_size=128, sym=True, desc_act=False, pack_dtype="int32", lm_head=False, and Triton.

import torch
from gptqmodel import BACKEND, GPTQModel
from gptqmodel.quantization import QuantizeConfig

model_id = "unsloth/Llama-3.2-1B-Instruct"
quant_path = "Llama-3.2-1B-Instruct-W4A"

qcfg = QuantizeConfig(
    bits=4, group_size=128, sym=True, desc_act=False,
    pack_dtype="int32", rotation="hadamard", offload_to_disk=False,
    activation={
        "mode": "w4a_nvfp4",
        "recipe": "least_squares_grid",
        "attention": {"mode": "w4afp8"},
    },
)

model = GPTQModel.load(model_id, qcfg, device="cuda", dtype=torch.bfloat16)
model.quantize(calibration, backend=BACKEND.GPTQ_W4A_NVFP4)
model.save(quant_path)

loaded = GPTQModel.load(quant_path, backend=BACKEND.AUTO, device="cuda", dtype=torch.bfloat16)

activation accepts a bare mode string as shorthand for that stream (activation="w4afp8" or activation="w4a_nvfp4"). The dict form adds recipe, attention, mlp, and global_scales; see the activation policy reference for every field and its validation rules.

Two configurations are worth calling out. activation="w4afp8" is the highest accuracy option and matches W4A16 within noise. The balanced NVFP4-MLP / FP8-attention config above is the accepted W4A4 operating point; it costs about 2.3 percentage points against W4A16 on full GSM8K Platinum, which is the measured price of the 4-bit activation grid on this model.

MoE Quantization 🧩

Some MoE (mixture of experts) models have extremely uneven/biased routing (distribution of tokens) to the experts causing some expert modules to receive close-to-zero activated tokens, thus failing to complete calibration-based quantization (GPTQ/AWQ). To better quantize these heavily biased MoE routed modules, GPT-QModel exposes 3 controls:

• Moe.Routing = ExpertsRoutingOverride: Manually override the num_experts_per_tok used for model routing math, i.e., if a model only routes 4 experts per token out of 48 total experts, you can set this equal to 24 for 50% routing or 48 for 100% routing. ExpertsRoutingOverride requires the model exposes num_experts_per_tok or equivalent configuration control.
• Moe.Routing = ExpertsRoutingBypass: Brute-force and bypass all routing math so all experts receive all activated tokens. This is akin to ExpertsRoutingOverride.num_experts_per_tok set to total number of experts. ExpertsRoutingBypass is enabled/tested for some models and, due to the lifecycle complexity, it needs to be validated for every model.
• FailSafe: This is enabled by default and is a naive weight-only quantization technique using simple (naive) quantization methods such as nearest with optional smoothing. There are various FailSafeStrategy options, along with SmoothMethod options, to complement this feature. FailSafe does not require activations but has higher quantization error loss than normally activated GPTQ/AWQ. It is fast and applicable for all MoE models.

FailSafe can be combined with ExpertsRoutingOverride. There is no single best way to quantize MoE, and we recommend users to test all three methods.

Quantized Inference 🔍

# test post-quant inference
model = GPTQModel.load(quant_path)
result = model.generate("Uncovering deep insights begins with")[0] # tokens
print(model.tokenizer.decode(result)) # string output

EoRA Accuracy Recovery: Enhanced Post-Quant Error Recovery via Lora 🎯

GPT-QModel supports EoRA, a LoRA method developed by Nvidia that can further improve the accuracy of the quantized model.

# EoRa is currently only validated for GPTQ
# higher rank improves accuracy at the cost of VRAM usage
# suggestion: test rank 64 and 32 before 128 or 256 as latter may overfit while increasing memory usage
eora = Lora(
  # for eora generation, path is adapter save path; for load, it is loading path
  path=f"{quant_path}/eora_rank32", 
  rank=32,
)

# provide a previously GPTQ-quantized model path
GPTQModel.adapter.generate(
  adapter=eora,
  model_id_or_path=model_id,
  quantized_model_id_or_path=quant_path,
  calibration_dataset=calibration_dataset,
  calibration_dataset_concat_size=0,
)

# post-eora inference
model = GPTQModel.load(
  model_id_or_path=quant_path,
  adapter=eora
)

tokens = model.generate("Capital of France is")[0]
result = model.tokenizer.decode(tokens)

print(f"Result: {result}")
# For more details on EoRA, please see docs/eora/
# Please use the benchmark tools in later part of this README to evaluate EoRA effectiveness

See runtime LoRA output composition for how quantized CUDA outputs and adapter gradients are combined.

Remote LoRA Adapters 🔒

Adapter paths stored in quantize_config.json are confined to the checkpoint’s own storage (including same-repository Hugging Face cache blobs); metadata cannot authorize Hub downloads. Loading an adapter hosted on the Hub requires an explicit opt-in with an immutable commit pin and a SHA-256 digest of adapter_model.safetensors:

model = GPTQModel.load(
  model_id_or_path=quant_path,
  adapter=Lora(path="org/adapter-repo", rank=32),
  allow_remote_adapter=True,
  adapter_revision="<40-char Hub commit sha>",
  adapter_sha256="<sha256 of adapter_model.safetensors>",
)

Manifest adapter directories must be checkpoint-relative and cannot contain directory symlinks. Adapter-file symlinks are rejected except for the standard Hugging Face cache links into the same repository’s blobs directory. This confinement also applies when using QuantizeConfig.from_pretrained() and initializing its adapter directly. Explicit local adapter= arguments remain available for adapters stored elsewhere. For pinned Hugging Face blob URLs, the config and weights are loaded from the URL’s containing folder at the supplied revision.

PEFT LoRA Training and Adapter Loading 🤗

GPT-QModel GPTQ/AWQ checkpoints can be fine-tuned with PEFT LoRA and loaded through PeftModel. Use a current PEFT release with optimum>=2.3.0; older combinations may warn that the quantized layer type is unsupported. For training, select the method-specific trainable backend (gptq_torch for GPTQ, torch_awq for AWQ); default inference kernels do not propagate gradients.

PEFT cannot merge Lo

README 内容过长，已截取前 64 KB。完整内容请前往 GitHub 仓库查看。