GPTQModel

GPT-QModel ⚡

An extensible platform for LLM quantization, validation, and deployment.
GPTQ, AWQ, ParoQuant, GGUF, FP8, EXL3, QQQ, NF12, and more—across NVIDIA CUDA, AMD ROCm, Tenstorrent, Huawei Ascend, Intel XPU, Apple silicon, and CPU, with Transformers, vLLM, SGLang, and MLX.

Latest News 🗞️🚀

• 7.11.0:

  • Tenstorrent: Added packed GPTQ inference through Torch/XLA/PJRT and native TTNN, including multi-card execution and opt-in bf8_b weights. Added block-float GPTQ calibration and checkpoint export, plus device-resident Llama decode with paged KV caching and chunked prefill up to 64K context. bf4_b/bf8_b checkpoints also run on NVIDIA CUDA and AMD ROCm; bit-exact CPU kernels support bf8_b/bf4_b/bf2_b without Tenstorrent hardware.
  • NF12: Added lossless 12-bit BF16 weight compression with packed linear, embedding, and fused inference APIs, optimized NVIDIA CUDA and AMD RDNA4 HIP kernels, and a portable Torch fallback. Automatic NF12 source storage is now the default during calibration for eligible weights, restoring exact source values before quantization. Accelerated embedding lookups, short-batch inference, and FP32 decoding, with Ampere, Ada, and Blackwell optimizations.
  • Security: Added startup checks for known-vulnerable PyTorch runtimes, explicit trust requirements for checkpoint-provided code, checkpoint path and index validation, and bounded server requests and remote media downloads.
  • Quantization and recovery: Added resumable, distributed EXL3 quantization with mixed-tier allocation, projection-level recovery, and disk-backed calibration replay. Expanded RCO allocation to AWQ, EXL3, and GGUF under packed-weight budgets. Corrected AWQ scaling and packed-layout handling, FP8 scale consistency, and mixed-attention calibration masks and rotary embeddings.
  • Inference: Added opt-in QKV and gate/up projection fusion driven by model-defined module plans, preserving native backend selection and packed-weight storage. Accelerated quantized embedding lookups, AWQ and Triton small-batch inference, and packed zero-point conversion. Improved low-precision numerical accuracy across kernels, preserving wide reductions and checkpoint bias precision.
  • Models: Added OLMo Hybrid, Granite SWA, Granite MoE SWA, and Dots3 Note quantization support. Fixed routed-expert coverage in Jamba and ERNIE 4.5 VL MoE, and Gemma 3 checkpoint loading.
  • Multi-GPU: Added GIL-safe peer copies throughout quantization and bounded disk-backed activation staging for large-model calibration. Improved Hessian device placement and free-threaded execution safety.
  • Compat: Improved checkpoint save/reload fidelity, nested checkpoint loading, legacy GPTAQ compatibility, grouped INT8 scales, and LoRA/EoRA adapter scaling and matching. Fixed vLLM, SGLang, and OpenAI-compatible generation behavior. Deferred Torch compilation failures now fall back to eager inference; ROCm JIT toolkit discovery is more robust, and NumPy is optional for the core.

• 7.10.0:

  • Models: Added Ricursive Ripple (ricursive) quantization support for the 65B preview checkpoint, including load-time Transformers 5 generation/masking patches, per-head Q/K norm and gated-attention handling, and v_proj Hessian isolation. Fixed pre-lm-head norm resolution across multimodal model definitions.
  • Performance: Accelerated Ricursive Ripple decode by up to 10x per row at batch 16 with fused RMSNorm and native GQA attention, and added chunked MLP prefill to bound activation memory. The CUDA expandable-segments allocator is now installed before Torch initializes, so wide-layer multi-GPU runs actually use it.
  • Checkpointing: Added gptqmodel.reshard() for per-layer safetensors sharding with one global shard, byte-for-byte tensor copies, preserved metadata, and config/tokenizer copying. Added a one-time warning when quantization shards are not layer-wise.
  • Multi-GPU: Honored explicit device lists for quantization placement, routed cross-device moves per direction to avoid asymmetric ROCm SIGSEGVs, and documented cudaMallocAsync for wide-layer Hessian allocation.
  • ROCm: Made AMD device families explicit and enforced kernel support contracts; gated W4A FP8/NVFP4 and ExLlamaV2/VPTQ off unsupported HIP paths with fast failures; fixed QQQ kernel launches to use the tensor’s device context.
  • Quantization: Added method-specific integer bit-width validation across GPTQ, AWQ, QQQ, ParoQuant, FP8, bitsandbytes, EXL3, GGUF, and VPTQ; EXL3 now rejects fractional trellis widths; MoE routing-bypass batch sizes are validated.
  • Correctness: Fixed Ricursive Ripple cached decode, mask alignment, per-row positions, and labeled-loss handling, plus EXL3 tile-scratch bounds and sign-vector precision.

• 7.9.0:

  • Correctness: Fixed packing/inference overflow, underflow, FP32 reductions, BF16 bias cancellation, 8-bit BitBLAS zero points, and stale fused Torch metadata after checkpoint loads or tensor changes.
  • MoE: Fixed skipped routed experts in MiniMax M2, Granite MoE/Hybrid, Bailing Hybrid, and AfMoE; added warnings for unresolved module paths.
  • Memory: Removed retained FP8 weight expansion from the experimental NVFP4 path and dense-weight scratch from ExLlamaV2.
  • Kernels: Added measured RTX 4090 (Ada, SM 8.9) and RDNA4 (gfx1200/gfx1201/gfx1250) launch tables to every tuned Triton pipeline — GPTQ GEMV/GEMM/dequant, ParoQuant GEMM, AWQ GEMM/GEMV, GGUF Triton, and bitsandbytes 4-bit — enforced by a hardware-free parity contract; the fused bitsandbytes 4-bit GEMM measured 1.4x-6.3x faster than the vendor HIP kernel on RDNA4 for M >= 16, while Ada keeps the vendor kernels. Added opt-in AllSpark W8A16 for compatible per-channel 8-bit GPTQ/AWQ on CUDA and RDNA4 (gfx1200/gfx1201). EXL3 2-bit quantization now fits RDNA4’s 64 KiB shared memory.
  • MLX: Accelerated GPTQ/RTN quantization, GPTQ/AWQ group-16 decode and short prefill, and GGUF K-quant/ternary packing.
  • GSQ/RCO: Added teacher-KL optimization, per-projection bit allocation, and native MLX search for dense Llama/Qwen3, with token-based batching and device/disk teacher caches.
  • Models: Added LFM3, LFM3-VL, and LFM3-Omni text-decoder quantization, OLMo/Helium support, and Qwen2 quantization resume.
  • Compat: Added native Transformers/Optimum load hooks preserving dynamic settings, fused/MoE structure, rotation, and W4A policies. Replaced method-specific Marlin/Machete/Swordfish enums with BACKEND.MARLIN, BACKEND.MACHETE, and BACKEND.SWORDFISH; legacy string aliases remain supported by the hf_ APIs.

• 7.8.0:

  • GSQ: Added staged GSQ training with public method adapters, distributed staged training that reuses captured activations, and per-block bit allocation driven by an RCO search verified bit-identical against the IST-DASLab reference (measured results on Llama 3.2 1B). Added a paper-reproduction harness, best-epoch restore, HIPRTC-compiled fused scalar relaxation on ROCm, and segment-sum scale gradients.
  • ROCm: Added RDNA4 (gfx1200/gfx1201/gfx1250) kernel and quantization support with per-architecture kernel tuning. MI350/gfx950 behavior is unchanged.
  • GGUF: Added CPU coverage for every ggml tensor type (all K-quant and IQ variants plus explicit MXFP4/NVFP4) and a native ggml SIMD inference path with per-shape autotuning, measured 3–100x faster than the torch paths on AVX2. New gguf install extra.
  • Multi-GPU: Added automatic GPU peer-access enablement on CUDA and ROCm with measured-link validation, so multi-GPU quantization copies stay on the interconnect instead of the driver’s staging path.
  • Experimental W4A: Revalidated the NVFP4 packed layout at load time, removed the retained E4M3 weight expansion, and relaxed the Triton build requirement on the Gluon FP8 GEMM.
  • Performance: Package import now builds the kernel registry lazily and defers optional backend probes (previously ~9s and 6767 modules). Vectorized the GPTQ/QQQ MSE shrink-grid and AWQ clip-grid evaluation; grid workspace budgets moved from environment variables to a new MSE configuration option.
  • Models: Added RecursiveSI Flint and Raon-1 mini support, Gemma 3 multimodal checkpoint compatibility, Qwen2.5-Omni thinker lm_head quantization, and the Transformers 5.22 Qwen-Omni layout.
  • Compat: Kept the AVX512-BF16 GPTQ/AWQ CPU kernels trainable for PEFT LoRA; fixed OpenAI-server auth, stop handling, and ChatCompletion schema; surfaced compatibility-shim failures instead of failing silently; fixed the Marlin CPU-offload snapshot refresh; fixed the calibration minimum-length setting not reaching the dataset filter; re-checked for an emptied calibration set after min-length filtering; tolerated MoE early-stop in data-parallel forward; bounded remote image fetches; read accelerator memory telemetry from the active device; fixed config serialization mutating dynamic overrides; fixed GPTAQ loss-log capture and evaluation-metadata coercion for fast eval tasks.

• 7.7.0:

  • CPU inference: Added the shared AVX512-BF16 int4_qzero_group_scaled_cpu kernel for GPTQ/AWQ group sizes 16/32/64/128/256/512 and prioritized it ahead of ATen when supported. Added packed/low-precision CPU decode paths for QQQ, VPTQ, EXL3, FP8, NVFP4, and BitsAndBytes, plus oneDNN BF16x3 accumulation and shape-aware BitsAndBytes INT8 dispatch. GPTQ/AWQ measured up to 2.02x faster than ATen; BitsAndBytes INT8 reached 1.28x at M=64.
  • GGUF: Added fused CPU Q4_0/Q8_0/Q4_K/Q5_K/Q6_K forward with FP32 accumulation under oneDNN BF16x3, fused bias, per-shape autotuning, and no retained dense weights. K-quant fused paths measured up to 1.92x faster than dense at M=7.
  • Quantized storage: Removed persistent dense, dequantized, expanded, and INT4-to-INT8 inference caches across GPTQ/AWQ, QQQ, EXL3, VPTQ, FP8, FP4, GGUF, BitsAndBytes, and INT8. Only packed weights, scales, zero points, and metadata persist; temporary dequantization uses bounded tiles.
  • Quantization quality: Added token-normalized GPTQ loss reporting, on-device embedding diagonal flooring and VPTQ Hessian damping, and QQQ vLLM compatibility/CUDA saturation fixes.
  • Models and runtimes: Added LLaVA quantization support, fused VPTQ CUDA inference, and hardened remote LoRA adapter loading.
  • Docs and tooling: Added the GPTQ FSDP/PEFT LoRA example, multi-GPU MoE quantization guide, and independent Torch-oracle accuracy and CPU speed benchmarks.
  • Dependencies: Refreshed runtime and optional dependency floors to current stable releases, including PyTorch 2.14.1, Transformers 5.20.2, Datasets 5.1.0, and Optimum 2.3.0. vLLM/SGLang extras retain their upstream framework pins and should be installed in dedicated environments.

• 7.6.0:

  • Model support: Added RecursiveSI Flint, Jamba/Jamba2, SmolVLM, Granite MoE, InternVL, EXAONE 4.5, Qwen3-VL MoE, Qwen4-Exp text, OLMoE, ZDTaichu 5.0, Agnes 3.0 Flash, and Xing4.0 quantization support, plus MiMo V2.6 Flash-RL and Pro-RL checkpoint support.
  • Apple silicon: Added native MLX GPTQ/AWQ quantization and an EXL3 weight-quantization pipeline. Expanded Metal quantization and packed inference kernels for ParoQuant, GGUF, FP8, QQQ, and BitsAndBytes, with GPTAQ/FOEM/RTN acceleration, broader GPTQ/AWQ bit-width and group-size support, Marlin/BitBLAS checkpoint loading, and FP16/BF16 output preservation.
  • GPTQ performance: Added fused CUDA block updates for 2–8-bit quantization, faster Hessian collection and factor reuse, lower peak Hessian memory, and faster Cholesky damping recovery.
  • Experimental W4A: Added Llama decoder activation streams using FP8 on NVIDIA SM89+ or NVFP4 on GB10 (SM121), while preserving native packed GPTQ INT4 checkpoints. See the W4A guide for supported configurations and measured quality.
  • Source formats: Added automatic decoding of native FP8/NVFP4 and OCP MXFP4/MXFP8 source checkpoints for requantization.
  • Runtime and integration: Added stable hf_ kernel selection, packing, and repacking APIs, improved AWQ Triton scheduling, and added opt-in Marlin scratch-buffer reuse. Improved SGLang format validation, dtype handling, and generation parameter compatibility.
  • Reliability: Fixed model-tree isolation and dynamic-rule caching/serialization, offloaded tensor device/dtype restoration, fused DeepSeek expert loading, Qwen3.5-MoE multimodal calibration, Gemma4 calibration KV memory retention, ParoQuant positional replay, GLM NoPE configuration handling, and GPTQ damping recovery after inverse-factor failures.

Older releases and development notes: changelog archive.

A unified quantization platform

GPT-QModel provides a consistent API for calibration, quantization, quality evaluation, model conversion, and accelerated inference. It supports GPTQ, AWQ, ParoQuant, QQQ, GGUF, FP8, EXL3, NF12, GPTAQ, EoRA, GAR, FOEM, and opt-in GSQ refinement for compatible scalar methods across supported hardware and runtime integrations.

Its method, format, backend, and kernel layers are modular: method-specific controls remain available where needed, while implementations share the same model lifecycle. This architecture supports mixed and model-specific quantization workflows today and provides a clear integration path for additional methods, formats, kernels, and accelerators.

GPTQ Hessian capture and repeated finalization are described in the Hessian accumulation flow. Every supported quantization path ends with one run summary emitted after finalization. Disk-backed replay outputs use bounded staging and shard verification on every read. Replayed batches and AWQ captures release their device and host memory as soon as the consuming step ends, and sibling projections share one capture; see the quantization memory and tensor lifetimes contract. Quantized kernels receive packed rows and launch in the operand’s device; see the kernel input contiguity and launch contracts for activation packing, 64-bit offsets, empty batches, and per-token compiles. The GPTQ mock quantization group flow describes group-boundary handling and metadata ordering. Transformers 5.22 model-family fixes for module trees, Defuser GLM-4.x router tuples, native Kimi-Linear dispatch, and HRM recurrent calibration are described in the Transformers 5.22 compatibility flow. AWQ module overrides follow the dynamic group-size flow through search, staging, and packing. Torch AWQ inference keeps checkpoint-precision scales and bias; see the AWQ runtime dtype precision flow. Torch AWQ reconstruction is tiled over input rows and broadcasts one scale row per group; see the AWQ GEMM dequantization tiling flow. AWQ scale folding resolves offset gains from the norm’s live forward, picks the norm that actually feeds the quantized block, and gauges the scale to max(s) <= 1 so a BF16 1 + weight fold stays neutral; see the AWQ scale folding flow. AWQ selects fold sources from declared data flow and skips sandwich post-norms and unresolvable gains; see the AWQ normalization selection flow. MoE AWQ folds also scale the router and shared_expert_gate, and calibration padding is kept out of expert Hessians; see the MoE AWQ scale folding and calibration row ownership flow. Padding-only MoE experts stay on the RTN fallback, sibling AWQ captures share one host copy under inference_mode, and FP16 CPU fused kernels keep their bias epilogue; see the MoE padding ownership and calibration capture lifetimes flow. GPTAQ and FOEM disable the native-input delta for routed experts (their token subset changes after attention is quantized) and fall back to RTN when an expert receives no rows; GGUF q*_k_s/q*_k_m/q*_k_l names warn that they store a uniform K-quant, and Hessian staging never narrows a bf16 input to float16; see the quantizer MoE, GGUF alias, and Hessian staging flows. Every AWQ module packs bounded token rows across all variable-length calibration captures and fallback coverage is checked per layer; see the AWQ calibration aggregation flow. AWQ scaling groups are validated against the decoder layer’s forward data flow, and a layer with no usable group is quantized plainly; see the AWQ scaling groups contract. LoRA rank_pattern and alpha_pattern use PEFT’s dot-boundary rule and accept float alphas; see the adapter pattern matching flow. AWQ packed readers decode the 4-bit interleave and the natural 8-bit stream separately; see the AWQ packed layout flow. AWQ dequantization dispatches on the saved checkpoint layout and per-module dynamic widths; see the AWQ dequantization layouts flow. AWQ split-K prefill covers every K tile and the decode-batch bound trims on the tile count; see the AWQ split-K kernel boundaries contract. Legacy 5.4.x meta.gptaq booleans, untied lm_head quantization, and the metadata plain Transformers or vLLM need are described in the checkpoint compatibility contract. Third-party module filters, format-changing re-saves, save_pretrained() layout safety, load_state_dict() round-trips and partial loads, HF quant_method for AWQ exports and packed GGUF, torch.save(model) pickling, Marlin g_idx, is_marlin_format: false, dense export, and mixed-bit dequantization are described in the checkpoint interop contract. Transformers/Optimum GPTQ loading, legacy checkpoint key layouts, AWQ group-size retuning, and the saved-config auto_map contract are described in the HF interoperability flow. FP8 dequantization divides its FP32 buffer in place and AWQ GEMM switches to an FP32-output matmul from 640 rows; see the FP8 dequantization scaling and AWQ GEMM dispatch flow. The native FP8 forward wrapper and the W4A16 quant source resolve one weight_scale_semantics per checkpoint, so a DeepSeek multiplier and a native inverse scale decode identically; see the FP8 checkpoint scale semantics flow. GPTQ packers validate FP16 scale storage before writing a checkpoint; see the GPTQ scale storage range contract. Checkpoint resaves, GGUF BF16 bytes, packed biases, explicit dtype strings, and FP8 block scales preserve their declared precision and bit patterns; see checkpoint dtype boundary preservation for the selection and failure flows. Quantized saves leave packed payloads (weight codes owned by a module with QUANT_TYPE) to the disk-offload restore instead of aliasing them from the dense source checkpoint, and a capture-only AUTO_NF12 forward keeps the dense module in the tree; see the packed payload save ownership flow. GPTAQ and FOEM consume one native calibration entry per sample, including batched calibration; see the native calibration batch split flow. 3-bit zero-point packing requires 32-aligned out_features, and every packer fails closed instead of truncating the tail; see the 3-bit zero-point packing width contract. Remote calibration image downloads run in cancellable subprocesses under one wall-clock deadline; expired requests release their sockets and workers, and decoding checks the remaining budget; see the remote image fetch bounds contract. GPTQ v1/v2 zero-point conversions decode and repack on the checkpoint’s device, so a CUDA shard never round-trips through a host loop; see the qzeros device conversion contract. A v2 kernel keeps logical zero points in memory while the v1 checkpoint is written from a converted copy, so in-process evaluation and the saved file agree; see the qzeros runtime vs checkpoint layout contract.

TorchLinear never retains dense/dequantized weights between inference calls. Torch compilation is best-effort: a torch.compile failure at first use (for example a dead inductor compile-worker pool) logs once and continues eagerly, so inference is never aborted; see the torch compile fallback flow and the TorchLinear(compile=False) / QuantizeConfig(compile=False) / TORCHDYNAMO_DISABLE=1 opt-outs. The legacy GPTQ_TORCH_CACHE_WEIGHTS / GPTQ_TORCH_LOOKAHEAD flags and enable_weight_cache(True) / enable_lookahead(True) APIs now warn and leave these features disabled. CUDA streaming uses per-forward tiles, not persistent dequantized workspaces. Only zero-point and group-index metadata is cached; call clear_weight_cache() after manually mutating inference-mode tensors without version counters. Normal tensor mutations, checkpoint loads, and device/dtype moves invalidate this metadata automatically. The ParoQuant rotation identity lifecycle explains how rotation buffer changes, state loads, and module moves refresh its identity shortcut. QuaRot/SpinQuant Hadamard rotation resolves its backend per call from the tensor’s device, so a rotated checkpoint runs on CPU and a CPU-only host can apply the exact Hadamard; see the rotation Hadamard device dispatch flow. The quantized embedding row decode page explains how a token lookup reads only its packed rows, why small tables and large token counts still use whole-table decode, and what that means for packed-weight storage. The CUDA quantized linear epilogue documents wide streaming reductions, bias before low-precision output rounding, fallback for empty FP8 inputs, and CUDA gradient propagation for the portable backends. The GPTQ Torch CPU bias epilogue documents fused low-precision bias addition and FP32 checkpoint bias handling. The low-precision bias epilogues reference keeps FP4, VPTQ, FP8, GGUF, and NF12 bias widths and their failure paths in one place. The fused kernel dispatch and bias epilogues page covers live GPTQ dispatch metadata, BF16 fused dispatch, and the ExLlamaV2 FP32-output epilogue. The FP16 inference overflow and bias epilogues page covers BF16-through-FP16 activation quantization and every kernel that now adds the stored bias before the final cast. The AllSpark W8A16 bias epilogue documents how the CUDA and RDNA4 kernels add a checkpoint bias to the FP32 accumulator before output rounding. The GGUF Triton packed metadata cache documents weight update invalidation and the explicit-clear contract for inference tensors. The GGUF Triton activation range documents per-row BF16/FP32 normalization before fused GGUF matmuls. The GGUF packed scale overflow cache documents FP16 operand-width selection for GGUF Torch inference. The GGUF dense fallback precision page explains BF16 weight reconstruction, FP32 bias reload, CPU autocast reduction, and output rounding across formats. The GGUF C++ bias epilogue explains how CPU and CUDA GGML outputs add FP32 bias before low-precision rounding.

BACKEND.GPTQ_TORCH_INT8 (including its TORCH_INT8 alias) accepts only 8-bit GPTQ sources. Converting 2/4-bit weights into a persistent 8-bit runtime representation is no longer supported. AWQ_TORCH_INT8 / TORCH_INT8_AWQ are retired and fail with migration guidance. Use BACKEND.AUTO to select a compatible packed kernel or BACKEND.TORCH for a portable fallback. Native 8-bit GPTQ keeps the checkpoint’s own per-group scales and zero points, so the runtime layout is a lossless repack of grouped/asymmetric GPTQ weights rather than a second quantization: a group whose scale is far smaller than its sibling’s no longer collapses to zero. See Torch INT8 checkpoint lifecycle for save/reload ownership, the group-preserving layout, and runtime cache invalidation.

Quantization Support 🛠️

Start with the quantization guide and end-to-end recipes to choose weight and activation precision, prepare calibration data, quantize, save, reload, and validate a model. It also maps advanced features to their guides.

For checkpoint reshaping, see reshard source selection. For how tensor-derived layer directories and weight_map shard names are kept inside the output directory, see checkpoint path containment. For how checkpoint index tensors (g_idx, NF12 escapes, ParoQuant pairs, VPTQ outliers, EXL3 K) are validated before any native kernel indexes memory, see checkpoint index validation. For checkpoint-backed module loading, see lazy checkpoint materialization. For how Transformers 5.22 class-keyed conversion maps, legacy mixer patch guards, the Nemotron-H tree root, and non-persistent Mamba buffers are handled, see Transformers 5.22 model compatibility. For a checkpoint that lives in a repository subfolder, see quantized checkpoint subfolder resolution for how model-type detection, the quantize config, the tokenizer, the shards, and the adapter manifest all stay inside the requested directory and revision. For how from_pretrained hands an already-quantized checkpoint to from_quantized without dropping explicit load options, see quantized-checkpoint redirect. For the temporary model mutation during calibration input capture and how it is released, see input capture cleanup lifecycle. For how each decoder layer is replayed with the attention mask and rotary embeddings of its own config.layer_types entry, see calibration layer replay. For the outer-loop work multimodal models apply outside a decoder layer—DeepStack residuals, Gemma-3 image-block masks, padded batches, and the Qwen2.5-Omni lazy checkpoint aliases—see multimodal calibration replay. For the base-type mask, the super() contract for definition replay overrides, and AWQ’s typed rotary call, see layer-type replay masks. For how kernel dispatch caches torch.ops handles, ExLlamaV2 q-matrix handles, bitsandbytes validation, and the AWQ FP16 bias add, see kernel dispatch caching. For the CPU int4pack group-size gate, CPU-offload kernel selection, max_memory layer placement, the torch.compile paths that must stay lowerable, HIP JIT source staging, and the bitsandbytes/GGUF autograd gate, see CPU and offload dispatch.

Every quantization method shares a common lifecycle—calibrate, quantize, validate, save, load, and serve—while retaining method-specific controls where they matter. New methods and kernels plug into that lifecycle instead of creating another disconnected toolchain. The quantization lifecycle contracts show how QuantizationRunPlan, ProcessorLifecycleConfig, and ResolvedModuleConfig connect method selection, calibration policies, per-module settings, packing, and checkpoint support. For how VPTQ and QQQ fold per-device calibration partials into a reusable Hessian, see calibration Hessian materialization. For how QQQ and GPTQ capture convolution inputs—including padding="same" and non-zero padding_mode—see convolution capture padding. QQQ selects its in-memory kernel from the model dtype’s SUPPORTS_DTYPES declaration and defaults to GPTQ’s Hessian damping; see the QQQ kernel selection and QQQ default damping contracts. See quantization scale storage for near-dead group handling and checkpoint precision requirements.

Module roles—attention projections, MLP projections, MoE experts and shared experts, and quantizable leaves—are resolved from each architecture’s declared module tree, not from runtime module names. See model tree resolution for the resolution flow, the declaration syntax, and the fail-closed contracts. Expert cardinality follows the same rule: a # placeholder expands only when the definition declares dynamic_expert_index and the owning config node resolves it, so expert cardinality resolution covers that step and why a surviving placeholder is a silent miss.

Subset digits order the quantization blocks: a projection that consumes another projection’s output must live in a later subset. See module tree subset ordering for the producer-before-consumer contract and the rule that every called nn.Linear is either quantized or explicitly excluded.

Use the module-plan guide for model.module_plan(), terminal/JSON inspection, model-definition tags, and the required new-model tests. The DeepSeek V4.1 Flash inventory shows its verified checkpoint structure and remaining integration requirements.

The fusion planner requires a model definition’s FusableConfig and uses the compiled module plan for opt-in post-quantization QKV and gate/up fusion. The model qualification matrix lists validated profiles for 101 model types and explicit qualification gaps for the remaining registry entries. Definitions reuse immutable role profiles with FusableConfig("qwen2").merge(LlamaFusableConfig()); concrete paths and parents resolve exclusively from the compiled tree, with no HF class-path allowlist; unqualified model types raise when fusion is requested. Fusion is disabled by default. Use plan = model.fuse() to apply it and receive the plan, plan.print() or plan.to_dict() to inspect eligibility and skip reasons, and model.fuse(False) to disable it and return None. Original projection weights become views into the fused packed storage, preserving checkpoint keys without retaining duplicate weights. AUTO retains native ATen/AVX512 on CPU, Marlin on NVIDIA, and eligible Torch/Triton paths on AMD. See CPU, CUDA and ROCm validation, the complete kernel inventory, and the ExLlamaV2 descriptor lifecycle for native handle ownership and cache invalidation.

Feature	GPT-QModel	Transformers	vLLM	SGLang	Lora Training
GPTQ	✅	✅	✅	✅	✅
AWQ	✅	✅	✅	✅	✅
ParoQuant	✅	x	x	x	✅
GGUF	✅	x	x	x	x
FP8	✅	x	x	x	x
Exllama V3 / EXL3	✅	x	x	x	x
EoRA	✅	✅	✅	✅	x
Group Aware Act Reordering	✅	✅	✅	✅	✅
QQQ	✅	x	x	x	x
VPTQ	✅	x	x	x	x
NF12	✅	x	x	x	x
BitsAndBytes	✅	x	x	x	x
Rotation	✅	x	x	x	x
GPTAQ	✅	✅	✅	✅	✅
FOEM	✅	✅	✅	✅	✅
GSQ (GPTQ, AWQ, RTN, QQQ, FP8, ParoQuant)	✅	Existing format†	Existing format†	Existing format†	Existing format†

GGUF, FP8, EXL3, ParoQuant, VPTQ, and NF12 are currently native GPT-QModel quantization/runtime paths. SGLang loading is limited to METHOD.GPTQ with FORMAT.GPTQ, FORMAT.GPTQ_V2, or FORMAT.MARLIN, and METHOD.AWQ with FORMAT.GEMM or FORMAT.MARLIN.

BaseQuantizeConfig.auto_dtype_cast defaults to AUTO_NF12, which uses lossless NF12 storage for eligible BF16 source matrices during calibration forwards and restores the exact source dtype before quantization. See automatic NF12 source storage for the decision flow, FP16 eligibility rule, and native floatx precedence, and excluded-layer replay and AUTO_NF12 for the dynamic exclusion contract and the writer’s fail-closed check.

† GSQ runs during quantization and saves an existing method format. Each external runtime or training integration supports only the formats it already accepts; GSQ does not add a new runtime format.

SGLang accepts the common engine aliases tensor_parallel_size → tp_size, gpu_memory_utilization → mem_fraction_static, max_model_len → context_length, seed → random_seed, and enforce_eager → disable_cuda_graph. An explicit legal dtype is preserved; deprecated torch_dtype is normalized to SGLang’s string dtype names. AWQ FORMAT.GEMV_FAST and FORMAT.LLM_AWQ still require torch.float16, but those formats are not in SGLang’s supported-format list.

Prompts and attention masks are translated before they reach vLLM or SGLang. Only boolean or 0/1 masks that remove contiguous left or right padding are accepted; an interior hole, a non-binary value, or a mask supplied with a text prompt is rejected instead of silently changing the prompt. See runtime prompt input normalization for the contract and the vLLM/SGLang comparison.

Device requests are translated per runtime: SGLang’s cuda spelling aliases an explicit rocm request only on a HIP build, and an AMD request on an NVIDIA-only build is rejected rather than silently launched on NVIDIA hardware. See external runtime device mapping.

Quant Method / Format / Backend Matrix 📋

Canonical backend names are shown below. Method-specific aliases are only accepted where explicitly implemented by that quant method.

Quant Method	Formats	Backends / Kernels
METHOD.GPTQ CPU	FORMAT.GPTQ, FORMAT.GPTQ_V2	BACKEND.GPTQ_TORCH_CPU_AVX512
METHOD.AWQ CPU	FORMAT.GEMM	BACKEND.AWQ_TORCH_CPU_AVX512
METHOD.GPTQ Tenstorrent	FORMAT.GPTQ, FORMAT.GPTQ_V2	BACKEND.GPTQ_TT_XLA, BACKEND.GPTQ_TTNN, BACKEND.GPTQ_TT_BF8 (opt-in bf8_b conversion)
METHOD.GPTQ	FORMAT.GPTQ, FORMAT.GPTQ_V2, FORMAT.GPTQ_P, FORMAT.MARLIN, FORMAT.BITBLAS	FORMAT.GPTQ: BACKEND.GPTQ_TORCH_ATEN, BACKEND.MACHETE, BACKEND.MARLIN, BACKEND.GPTQ_EXLLAMA_V2, BACKEND.GPTQ_TORCH_FUSED, BACKEND.GPTQ_TRITON, BACKEND.GPTQ_BITBLAS, BACKEND.GPTQ_TORCH, BACKEND.GPTQ_TORCH_INT8 FORMAT.GPTQ_V2: BACKEND.GPTQ_TORCH_ATEN, BACKEND.GPTQ_EXLLAMA_V2, BACKEND.GPTQ_TORCH_FUSED, BACKEND.GPTQ_TRITON, BACKEND.GPTQ_BITBLAS, BACKEND.GPTQ_TORCH, BACKEND.GPTQ_TORCH_INT8 FORMAT.GPTQ_P: BACKEND.GPTQ_TORCH FORMAT.MARLIN: BACKEND.MARLIN FORMAT.BITBLAS: BACKEND.GPTQ_BITBLAS
METHOD.AWQ	FORMAT.GEMM, FORMAT.GEMV, FORMAT.GEMV_FAST, FORMAT.LLM_AWQ, FORMAT.MARLIN, FORMAT.BITBLAS	FORMAT.GEMM: BACKEND.AWQ_TORCH_ATEN, BACKEND.MACHETE, BACKEND.MARLIN, BACKEND.AWQ_EXLLAMA_V2, BACKEND.AWQ_GEMM, BACKEND.AWQ_GEMM_TRITON, BACKEND.AWQ_TORCH_FUSED, BACKEND.AWQ_TORCH, BACKEND.AWQ_BITBLAS FORMAT.GEMV: BACKEND.AWQ_GEMV, BACKEND.AWQ_GEMV_TRITON FORMAT.GEMV_FAST: BACKEND.AWQ_GEMV_FAST FORMAT.LLM_AWQ: BACKEND.AWQ_GEMV_FAST FORMAT.MARLIN: BACKEND.MACHETE, BACKEND.MARLIN FORMAT.BITBLAS: BACKEND.AWQ_BITBLAS
METHOD.GPTQ / METHOD.AWQ Swordfish	FORMAT.GPTQ / FORMAT.GEMM	BACKEND.SWORDFISH
METHOD.GPTQ / METHOD.AWQ AllSpark	FORMAT.GPTQ, FORMAT.GPTQ_V2 / FORMAT.GEMM	BACKEND.ALLSPARK (per-channel 8-bit)
METHOD.PARO	FORMAT.PAROQUANT	BACKEND.PAROQUANT_CUDA, BACKEND.PAROQUANT_TRITON
METHOD.QQQ	FORMAT.QQQ	BACKEND.QQQ, BACKEND.QQQ_TORCH
METHOD.VPTQ	FORMAT.VPTQ	BACKEND.VPTQ_CUDA, BACKEND.VPTQ_TORCH
METHOD.NF12 (nf12)	FORMAT.NF12 (nf12)	BACKEND.NF12 (nf12)
METHOD.GGUF	FORMAT.GGUF	BACKEND.GGUF_TRITON, BACKEND.GGUF_CPP_CUDA, BACKEND.GGUF_CPP_CPU, BACKEND.GGUF_TORCH
METHOD.FP8	FORMAT.FP8	BACKEND.FP8_TORCH
METHOD.BITSANDBYTES	FORMAT.BITSANDBYTES	BACKEND.BITSANDBYTES
METHOD.EXL3	FORMAT.EXL3	BACKEND.EXL3_EXLLAMA_V3, BACKEND.EXL3_TORCH

Marlin, Machete, and Swordfish are each one backend (BACKEND.MARLIN, BACKEND.MACHETE, BACKEND.SWORDFISH). The quantization method and checkpoint format select the GPTQ or AWQ sub-kernel automatically, so gptq_marlin/ awq_marlin, gptq_machete/awq_machete, and gptq_swordfish/ awq_swordfish are no longer separate backend names. The stable hf_ packing and repacking APIs still accept those legacy strings and resolve them to the shared backend.

BACKEND.GPTQ_TORCH_CPU_AVX512 and BACKEND.AWQ_TORCH_CPU_AVX512 share the same AVX512-BF16 int4 GEMM on Linux x86-64 hosts with AVX512-BF16, AVX512-VBMI, and OS AVX512 state enabled. GPTQ and AWQ use separate fused checkpoint-to-kernel packers but no separate compute kernels. AUTO selects the AVX512 path ahead of the ATen int4pack fallback when the layer shape satisfies the kernel’s 32-row and group-size alignment requirements. Supported group sizes are 16, 32, 64, 128, 256, and 512.

BACKEND.GPTQ_TORCH_FUSED keeps INT4 weights in the packed int4pack layout and reorders activations only when the g_idx permutation is not the identity; BACKEND.AWQ_TORCH_FUSED uses the same packed layout without a g_idx activation reorder. See Torch Fused INT4 transformations for asymmetric GPTQ zero-point offsets, packing, activation-reorder contracts, and the checkpoint-layout restore versus runtime state_dict() round-trip.

BACKEND.QQQ_TORCH uses the same packed INT4 CPU layout with a QQQ-specific integer GEMM that preserves round-to-nearest-even and INT8 saturation for group size 128, plus the signed per-channel layout for group size -1. Small batches use AVX512 integer dot products; larger batches use oneDNN integer GEMM with temporary 128-by-32 weight tiles when available. Runtime caches retain only packed INT4 codes and scales. Unsupported hosts use the Torch fallback without retaining dequantized weights. Both FP16 and BF16 activations keep their output dtype.

BACKEND.EXL3_TORCH uses oneDNN BF16x3 FP32 matmul on AVX512-BF16 hosts for GEMMs large enough to amortize dispatch. Forward decodes at most 512 output columns at a time and never retains a dense dequantized weight; get_weight_tensor() materializes one on demand for export and tests. Set GPTQMODEL_EXL3_CPU_BF16X3=0 to disable the BF16x3 path.

The EXL3 runtime weight lifecycle describes how deferred dense replay weights are refreshed before every layer forward, why the generic path zero-pads output columns to 128 and rejects unaligned input features before calibration, and why the layer inventory keeps NamedModule wrappers so the AUTO_NF12 path reaches EXL3’s deferred replay state.

BACKEND.VLLM, BACKEND.SGLANG, and BACKEND.MLX are external runtime backends and are not part of the native kernel matrix above.

Native backend availability, shape validation, wide activation inputs, and derived cache invalidation follow the kernel validation and cache lifecycle contract. Opt-in AllSpark W8A16 keeps its GPTQ/AWQ checkpoint tensors authoritative and its N32K16/WMMA kernel layout a derived cache; see AllSpark checkpoint layout.

Tenstorrent Blackhole uses the explicit DEVICE.XLA family. Packed 4-bit GPTQ runs through Torch/XLA/PJRT (BACKEND.GPTQ_TT_XLA, selected by AUTO) or native TTNN (BACKEND.GPTQ_TTNN, opt-in), with optional multi-card execution. BACKEND.GPTQ_TT_BF8 explicitly converts GPTQ weights to bf8_b for hardware-native unpacking. See the Tenstorrent execution guide for runtime pins, numerical results, and backend constraints, and the validated hardware inventory for Blackhole measurements. Out-axis block-float Llama checkpoints are served fully on device with a paged KV cache and chunked prefill; see Tenstorrent Llama serving. Install the runtime from requirements/tenstorrent-py312.txt; the PJRT plugin is not on the default PyPI index.

Packed bf4_b / bf8_b checkpoints also run without Tenstorrent cards using the JIT-loaded CUDA and ROCm HIP kernels. These preserve the checkpoint’s block-float values without re-quantizing the weights; GEMM arithmetic depends on the execution backend. For CPU format validation and fused GEMM, see Tenstorrent block floats on CPU. The kernel dispatch contract records when the native block-float kernels are used.

FP8 source weights must be finite before quantization; see the FP8 source weight validation contract.

GSQ is an opt-in scalar refinement for supported METHOD.GPTQ, METHOD.AWQ, METHOD.QQQ, METHOD.FP8, METHOD.PARO, and METHOD.RTN paths. Staged block training currently supports dense Llama GPTQ and AWQ export. Each path retains its method’s existing checkpoint format; method-specific GSQ constraints are described below.

AWQ FP16 boundary staging and packed decode follow the AWQ affine range contract.

Marlin uses GPTQMODEL_MARLIN_USE_FP32 (default: enabled) to control fp32 accumulation. The Marlin device-profile validator checks device ordinals before asking Torch or the extension for a profile.

On ROCm, quantization runs with torch’s rocBLAS backend instead of hipBLASLt: hipBLASLt loads one Tensile library per (dtype, layout) pair on first use, and those one-time loads dominate a cold quantize while rocBLAS covers the same math. The override is scoped to the quantization phase and the previous backend is restored afterwards. Set GPTQMODEL_ROCM_QUANT_BLAS=hipblaslt to keep torch’s default backend. This applies to RDNA4 as well; the RDNA4 hipBLASLt Tensile libraries exist but are not tuned, so rocBLAS remains the default there too.

Features ✨

• ✨ Native integration with HF Transformers, Optimum, and Peft
• ✨ Stable hf_ packing/repacking API for layer-wise kernel feasibility, packing, and repacking used by external integrators such as AutoRound.
• 🚀 vLLM and SGLang inference integration for quantized models. SGLang supports GPTQ FORMAT.GPTQ/FORMAT.GPTQ_V2/FORMAT.MARLIN and AWQ FORMAT.GEMM/FORMAT.MARLIN.
• ✨ GPTQ, AWQ, ParoQuant, QQQ, VPTQ, GGUF, FP8, EXL3, NF12, GPTAQ, and FOEM quantization support, plus opt-in GSQ refinement for compatible GPTQ, AWQ, RTN, QQQ, FP8, and ParoQuant paths.
• ✨ All current GGUF quantized tensor types are supported end to end for CPU quantization and inference: every Q*_0/Q*_1, Q2_K-Q8_K, IQ1_S-IQ4_XS, TQ1_0/TQ2_0, MXFP4, NVFP4, Q1_0, Q2_0, and the Q1_0_g128 sign-only variant. Install the bindings with pip install gptqmodel[gguf] (gguf>=0.19.0, llama-cpp-python>=0.3.36). CPU inference uses llama.cpp’s native SIMD kernels (BACKEND.GGUF_CPP_CPU) with a per-shape autotuned Torch fallback. Native ggml dot kernels also quantize activations, so outputs can differ from weight-only Torch inference; set GPTQMODEL_GGUF_CPP_NATIVE_MATMUL=0 to keep the Torch path. Q8_1/Q8_K always use Torch, including when native execution is forced, since upstream ggml exposes no matmul for them. CPU dequantization reconstructs weights in FP32 before casting to FP16/BF16. See the GGUF compatibility notes.
• 🚀 Routing-aware MoE quantization controls for extreme activation bias via Moe.Routing and/or FailSafe.
• 🚀 Multi-GPU data-parallel quantization with GIL-safe peer copies and device-aware scheduling.
• ✨ CUDA (NVIDIA), ROCm (AMD), Tenstorrent (Torch/XLA and TTNN), NPU (Huawei Ascend), XPU (Intel), MPS/MLX (Apple silicon), and CPU support. See the platform matrix below for OS and kernel availability.
• ✨ Dynamic per-module mixed quantization control: each layer/module can have a unique quantization config or be excluded from quantization.
• 🚀 Torch fused kernels for Intel XPU and CPU, plus native AVX512-BF16 GPTQ/AWQ CPU kernels.
• 🚀 Free-threaded Python support for multi-GPU quantization, with explicit PyTorch GIL=0 safety contracts.
• ✨ Asymmetric Sym=False support.
• ✨ lm_head module quant inference support for further VRAM reduction.
• 🚀 Microsoft/BITBLAS optimized tile based inference.
• Unit, kernel, model-plan, and post-quantization quality regression tests. See accelerator test gates for hardware requirements and skip/failure rules.

Organizations and projects using GPT-QModel 🌐

Selected public references where teams or companies explicitly mention GPT-QModel in documentation, integration notes, or quantized model usage. This is not an exhaustive customer list.

• Hugging Face
• Intel
• NVIDIA
• Alibaba Cloud

Quality: GPTQ 4-bit evaluation 🏆

🤗 ModelCloud quantized Vortex models on HF

Model Support 🤖

The table mirrors every explicit registration in gptqmodel.models.auto.MODEL_MAP, including text-model and backward-compatibility aliases. Qwen 3.5 registrations require Transformers 5.2.0 or newer.

Model family	Registered Transformers model_type values
A.X-K2	axk2
AfMoE / Trinity	afmoe
Agnes 3.0 Flash	agnes
Apertus 1 / 1.5	apertus, apertus1p5, apertus1p5_text
Baichuan	baichuan
Bailing MoE / Hybrid (LING / RING)	bailing_moe, bailing_hybrid
Bloom	bloom
Brumby	brumby
ChatGLM	chatglm
CodeGen	codegen
Cohere 1 / 2 / 2 MoE / Compass (North Mini / Micro Vision)	cohere, cohere2, cohere2_moe, cohere_compass
DBRX / DBRX Converted	dbrx, dbrx_converted
DeciLM	deci
DeepSeek V2 / V3 / V3.2 / V4 / VL / VL2 / OCR2	deepseek_v2, deepseek_v3, deepseek_v32, deepseek_v4, deepseek_vl, deepseek_vl_v2, deepseek_ocr2
KRAFTON Raon-1 mini	raon1_mini
DiffusionGemma	diffusion_gemma
Dots1 / Dots3 Note	dots1, dots3_note
Dream	dream
ERNIE 4.5 / MoE / VL MoE	ernie4_5, ernie4_5_moe, ernie4_5_moe_vl, ernie4_5_vl_moe
EXAONE 3 / 4 / 4.5	exaone, exaone4, exaone4_5, exaone4_5_text
Falcon / Falcon H1 / Falcon Mamba	falcon, falcon_h1, falcon_mamba
RecursiveSI Flint	flint
Gemma 1-4 / 3n / Unified	gemma, gemma2, gemma3, gemma3_text, gemma3n, gemma3n_text, gemma4, gemma4_text, gemma4_unified, gemma4_unified_text
GLM / GLM4 / GLM4V / GLM5 / OCR / ASR	glm, glm4, glm4_moe, glm4_moe_lite, glm4v, glm4v_moe, glm4v_moe_text, glm5_next, glm_moe_dsa, glm_ocr, glmasr
GPT-2	gpt2
GPT BigCode	gpt_bigcode
GPT-Neo / GPT-NeoX	gpt_neo, gpt_neox
GPT-OSS	gpt_oss
GPT-J	gptj
Granite / Granite SWA / Granite MoE / Granite MoE SWA / Granite MoE Hybrid	granite, granite_swa, granitemoe, granitemoe_swa, granitemoehybrid
GRIN-MoE	grinmoe
Helium	helium
HRM	hrm_text
Hunyuan V1 / VL / OCR	hunyuan_v1_dense, hunyuan_v1_moe, hunyuan_vl
HY-V3	hy_v3
Hymba	hymba
Inkling	inkling_mm_model
Instella	instella
Intern S1 / S2 Preview	interns1, intern_s2_preview
InternLM 1 / 2 / 2.5	internlm, internlm2
InternVL 3 / InternVL Chat	internvl, internvl_chat
Jamba / Jamba2	jamba
K2-Horizon (Dense / MoVA)	k2_horizon
Kimi K2 / K2.5	kimi_k2, kimi_k25
Kimi K3 / Linear (bounded native-source calibration)	kimi_k3, kimi_linear
Klear	klear
Laguna	laguna
LFM2 / LFM2 MoE / LFM2-VL	lfm2, lfm2_moe, lfm2_vl
LFM3 / LFM3 Omni / LFM3-VL	lfm3, lfm3_omni, lfm3_vl
LLaDA2 MoE	llada2_moe
Llama 1-4 / TinyLlama / Nemotron Ultra	llama, llama4, llama4_text
Llama 3.2 VL	mllama, mllama_text_model
LLaVA-1.5	llava
FastVLM / LLaVA-Qwen2	llava_qwen2
LocateAnything	locateanything
LongCat Flash	longcat_flash
LongLLaMA	longllama
Mage-VL	mage_vl
Marin	marin
MiMo / MiMo V2	mimo, mimo_v2
MiniCPM / MiniCPM3 / MiniCPM-O / MiniCPM-V	minicpm, minicpm3, minicpmo, minicpmv, minicpmv4_6
MiniMax M2 / M3-VL	minimax, minimax_m2, minimax_m3_vl
Mistral / Mistral3 / Ministral3	mistral, mistral3, ministral3
Mixtral	mixtral
MobileLLM	mobilellm
MOSS	moss
MPT	mpt
Muse Glimmer	muse_glimmer
Nanbeige 4.2	nanbeige
Nemotron NAS / H / H Puzzle / Omni / Labs Diffusion	nemotron-nas, nemotron_h, nemotron_h_puzzle, nemotronh_nano_omni_reasoning_v3, nemotron_labs_diffusion
OLMo / OLMo 2 / 3 / Hybrid / OLMoE	olmo, olmo2, olmo3, olmo_hybrid, olmoe
OpenVLA	openvla
OPT	opt
Ouro	ouro
Ovis 1.6 / 2 / 2.5 / 2.6 MoE / 2.6 Next	ovis, ovis2, ovis2_5, ovis2_6_moe, ovis2_6_next
PanGu-α	gpt_pangu
Phi 1-4 / Phi MoE	phi, phi3, phi4mm, phimoe
Qwen 1-4 / 3.5 / 3.6 / 3.8 / MoE / Next	qwen, qwen2, qwen2_moe, qwen3, qwen3_moe, qwen3_next, qwen3_5, qwen3_5_text, qwen3_5_moe, qwen3_5_moe_text, qwen4_exp, qwen4_exp_text
Qwen 2 / 2.5 / 3 VL / 3 VL MoE	qwen2_vl, qwen2_vl_text, qwen2_5_vl, qwen2_5_vl_text, qwen3_vl, qwen3_vl_moe
Qwen 2.5 / 3 Omni	qwen2_5_omni, qwen3_omni_moe
Qwen2-Audio	qwen2_audio
Qwen-Drive 1.0	qwen_drive
RefinedWeb	refinedWeb, refinedWebModel
Ricursive Ripple (cached generation)	ricursive
Seed-OSS	seed_oss
SmolLM3	smollm3
SmolVLM	smolvlm
Solar Open / Open 2	solar_open, solar_open2
Spark 2.5	spark2_5
StableLM	stablelm, stablelm_epoch
StarCoder2	starcoder2
TeleChat2	telechat
Unlimited-OCR	unlimited_ocr, unlimited-ocr
Voxtral	voxtral
Xing4.0 (29B-A4B)	xing4_0
XVERSE	xverse
Yi	yi
Zamba / Zamba2	zamba, zamba2
ZDTaichu 5.0	zdtaichu5_0

Kimi K3 / Linear quantization requires the bounded native-source runner to preserve AttnRes prefix/residual state; the ordinary GPTQModel.quantize() looper rejects it. See the Kimi K3 calibration contract.

Qwen2.5-Omni supports quantizing its text output head with lm_head=True. The head is thinker.lm_head; GPTQ defaults it to 8-bit, group size 32. To use a 4-bit head, configure an explicit dynamic override:

from gptqmodel import QuantizeConfig

quantize_config = QuantizeConfig(
    bits=4,
    group_size=128,
    lm_head=True,
    dynamic={r"thinker\.lm_head$": {"bits": 4, "group_size": 32, "desc_act": False}},
)

This option quantizes the thinker’s text head; the talker and audio decoder retain their original weights. Tied output-head weights must be untied first.

Platform and HW Support 🖥️

GPT-QModel is validated on Linux, macOS, and Windows 11:

Platform	Device		Optimized Arch	Kernels
🐧 Linux	NVIDIA GPU	✅	Turing+ (sm_75+); NF12 and block-float CUDA sm_80+; backend-specific limits apply	Machete, Marlin, Swordfish, AllSpark, Exllama V3 / EXL3, Exllama V2, AWQ GEMM/GEMV, ParoQuant CUDA/Triton, GGUF CUDA/Triton, QQQ, VPTQ, NF12, bf4_b/bf8_b CUDA, BitBLAS, Triton, BitsAndBytes, Torch
🐧 Linux	AMD GPU	✅	ROCm 10.1+, gfx950 (CDNA4), gfx1200/1201/1250 (RDNA4); backend-specific limits apply	AWQ Triton GEMM/GEMV, ParoQuant Triton, GGUF Triton, BitsAndBytes 4-bit Triton, QQQ, NF12 HIP (RDNA4), AllSpark (gfx1200/1201), bf4_b/bf8_b HIP, FP8 Torch, Torch
🐧 Linux	Tenstorrent	✅	Blackhole p300c, DEVICE.XLA	Packed GPTQ via Torch/XLA/PJRT or native TTNN; opt-in bf8_b TTNN; multi-card execution
🐧 Linux	Huawei Ascend NPU	✅	Ascend 910B, torch-npu / CANN	Native Torch kernels for GPTQ, AWQ, ParoQuant, GGUF, QQQ, and EXL3
🐧 Linux	Intel XPU	✅	Arc, Datacenter Max	TorchFused, TorchFusedAWQ, FP8 Torch, Torch
🐧 Linux	Intel/AMD CPU	✅	x86-64; AVX2, AVX512-BF16, or AMX for applicable optimized paths	GPTQ/AWQ AVX512-BF16, TorchFused, TorchFusedAWQ, TorchAten int4, GPTQ INT8 (8-bit sources), GGUF C++, BitsAndBytes, VPTQ, NF12 Torch, ttbfp block floats, Torch
🍎 macOS	GPU (Metal) / CPU	✅	Apple Silicon, M1+	Native MLX quantization and packed inference; MPS/Torch, FP8 Torch, NF12 Torch
🪟 Windows	GPU (NVIDIA) / CPU	✅	NVIDIA	Torch

Availability depends on each backend’s architecture, dtype, shape, and runtime requirements; a platform row does not imply that every listed kernel runs on every device in that family. Tenstorrent hardware inference and CPU block-float execution are separate paths; the CPU kernels do not require TTNN or PJRT.

ROCm 10.1 or newer is required. GPT-QModel only supports and validates ROCm >= 10.1; older releases are unsupported and import gptqmodel or model loading logs a warning asking you to upgrade. Use the latest ROCm release and a matching ROCm PyTorch build. ROCm 10.1 ships HIP 7.16; torch.version.hip reports the HIP runtime version, not the ROCm release, so version checks use torch.version.rocm.

ROCm pip SDK installations that expose _rocm_sdk_core as ROCM_HOME use the matching expanded rocm-sdk-devel tree, and otherwise fall back to the version-matched /opt/rocm/core-* SDK for the template headers, the libamdhip64.so symlink and the device bitcode. See docs/rocm-jit-toolkit-paths.md for the resolution order and failure paths. JIT builds on ROCm always use the HIP/torch headers only: NVIDIA nvidia wheel headers are never added to a hipcc command line, even when those packages are installed in the environment, because their vector_types.h conflicts with HIP’s vector type aliases.

On gfx950 (CDNA4) and the gfx120*/gfx125* RDNA4 family, the Torch GPTQ backend uses a fused Triton GEMV for single-token 2/4/8-bit inference and a fused small-M GEMM for M=2…16, and falls back to the existing path for other shapes or unregistered architectures. Launch constants are selected per architecture by gptqmodel/utils/arch.py. On gfx1201 (R9700) the fused small-M GEMM measured 1.1x-3.2x faster than dequantize+matmul for M=2…16 in FP16 up to K=17408, and its .amdgcn contains v_wmma_f32_16x16x16_f16. The fused small-M GEMM is FP16-only on RDNA4: BF16 measured 0.43x-1.42x of the split path there (median 0.85x), so BF16 falls back to dequantize+matmul (GEMV keeps both dtypes). Set GPTQ_TORCH_ROCM_GEMV=0 / GPTQ_TORCH_ROCM_GEMM=0 to disable the fused paths. See docs/rocm-rdna4.md for the per-backend RDNA4 matrix and known limitations, and docs/rocm-default-environment.md for the ROCm defaults GPT-QModel installs (expandable_segments is not enabled; ROCBLAS_USE_HIPBLASLT=0 makes quantization reproducible across processes).

On Ada (SM 8.9) and Blackwell (SM 10.x), TritonV2Linear reuses the same fused GEMV / small-M GEMM dispatch and the packed-load kernel covers M=2..64 with no K/N caps: decode and small-batch forward measured 2.4x-6.8x faster than the previous TritonV2 path. See the GPTQ Triton decode and small-M dispatch flow for the gate, split-K and bitwise contracts.

bitsandbytes 4-bit (NF4/FP4) inference on ROCm routes M >= 16 through a fused Triton GEMM that dequantizes the packed codes inside the matmul, because the vendor HIP kernel switches to a much slower path at that row count. Measured on gfx1201 (R9700, FP16) it is 1.4x-6.3x faster than bnb.matmul_4bit; below M = 16 bitsandbytes’ own GEMV is faster and is kept. CUDA/Ada keeps the vendor kernels, which are faster there; set GPTQMODEL_BNB_TRITON=1 to force the Triton path for A/B runs.

Marlin supports NVIDIA Turing and newer (sm_75+); other JIT CUDA kernels enforce their own minimum architecture rather than sharing a universal cutoff. Huawei Ascend NPU support uses native Torch kernels through torch-npu / CANN.

Tenstorrent block floats on CPU

gptqmodel_ext/ttbfp/ provides a bit-exact CPU implementation of Tenstorrent’s bf8_b / bf4_b / bf2_b block-float formats, so models and kernels that use them can run without Tenstorrent hardware. It ships a quantizer validated against the upstream tt-llk helpers, fused dequantize+GEMM microkernels for scalar / AVX2+FMA / AVX-512, block-major and bf8_b-only int8 kernel-native layouts, and bitwise-parity tests. See the CPU block-float guide for dtype naming, dispatch flow and measured performance.

Native MLX quantization on Apple silicon

Install the optional MLX dependency and quantize a dense model directly to an MLX-LM checkpoint:

pip install 'gptqmodel[mlx]'

from gptqmodel import GPTQModel

GPTQModel.quantize_mlx(
    "Qwen/Qwen3-0.6B",
    "./qwen3-mlx-gptq",
    method="gptq",  # or "awq"
    bits=4,
    group_size=64,
    num_samples=128,
)

GPTQ uses MLX for calibration and weight updates, with one fused Metal dispatch per weight group. MLX currently performs the Cholesky factorization on its CPU stream. AWQ uses MLX-LM’s native scale and clip search. The output uses MLX-LM’s quantized model format and can be loaded with mlx_lm.load. Pass an MLX token array as calibration_data to use your own calibration set. To drive a GPTQ/AWQ checkpoint through the MLX runtime with the standard generate() API, or to export one with GPTQModel.export(..., format="mlx"), see the MLX runtime and export flow.

Install 💾

PIP/UV 💿

# vLLM and SGLang publish their own Torch/Transformers pins. Install one engine
# in a dedicated environment aligned to that engine's release.
# Example optional module: pip install -v "gptqmodel[bitblas]"
pip install -v gptqmodel
uv pip install -v gptqmodel

Compiled package backends build and cache their extensions on first use (JIT), not during pip install. The package depends on ninja for these builds; install a compatible compiler and the CUDA/ROCm toolkit required by the chosen backend. Pure Torch paths do not require building an extension. NumPy is not a base dependency: the quantization core is NumPy-free, and the gguf, bitblas, and mlx extras declare it for the paths that need it. See NumPy-free core.

AMD ROCm support requires ROCm >= 10.1 and a matching ROCm PyTorch build. Older ROCm releases are unsupported; GPT-QModel logs a warning at import/model load time and asks you to upgrade.

PyTorch 2.13.x or >= 2.15.0 is required. Importing GPT-QModel on a torch release below 2.13.0 raises UnsafeTorchVersionError because that range is affected by GHSA-rrmf-rvhw-rf47 (CVE-2025-3000, memory corruption in torch.jit.script), and 2.14.x raises because it is affected by GHSA-v32g-mr63-3j83 (torch.load(weights_only=True) arbitrary file read, patched in 2.15.0). The check runs before the CUDA allocator is initialized. Python 3.10 has no patched wheel and therefore installs the 2.13.x line. See PyTorch runtime safety gate for the affected ranges, the bypass, and why the comparison uses the PEP 440 public version.

Install from source 🛠️

# clone repo
git clone https://github.com/ModelCloud/GPTQModel.git && cd GPTQModel

# python3-dev is required for some source installs
apt install python3-dev

# pip: install from source
# Example optional module: pip install -v ".[bitblas]"
pip install -v .

Inference 🔮

Three-line API to use GPT-QModel for GPTQ model inference:

from gptqmodel import GPTQModel

model = GPTQModel.load("ModelCloud/Llama-3.2-1B-Instruct-gptqmodel-4bit-vortex-v2.5")
result = model.generate("Uncover

> README 内容过长，已截取前 64 KB。完整内容请前往 GitHub 仓库查看。