GPT-QModel ⚡
An extensible platform for LLM quantization, validation, and deployment.
Compress, evaluate, and run models across supported GPUs, accelerators, and CPUs.
Documentation ·
Quantization guide ·
Reference index ·
Releases
Latest News 🗞️🚀
7.11.0
- MACaQ: Native MXFP4 W4A4 quantization for low-precision weights and activations, with CUDA, ROCm, and Apple silicon inference. Includes packed checkpoints and a portable Torch fallback.
- NF12: Lossless 12-bit compression of BF16 weights reduces source-weight storage during calibration without changing the restored values. Packed inference now includes native Metal support alongside CUDA, HIP, and Torch paths. MLX · MoE capture contract
- Tenstorrent: Packed GPTQ inference through Torch/XLA and TTNN, multi-card execution, and block-float quantization. Long-context Llama serving adds paged caching and chunked prefill; compatible block-float checkpoints also run on CUDA, ROCm, and CPU. Guide · Serving · Device ownership
- Quantization: Resumable distributed EXL3 jobs (projection publication recovery), broader mixed-bit allocation across AWQ/EXL3/GGUF, and SmoothQuant preprocessing improve large-model workflows and quality control. Quantizer calibration flows document the shared Hessian-failure and zero-sample fallback contracts. MoE routing override and bypass defines when each mode is honoured and when it fails closed. Shared-input Hessian dedup covers QQQ/VPTQ/MACaQ/EXL3 and GPTAQ/FOEM. Optional calibration attention-sink exclusion drops BOS-token rows from the calibration statistics.
- Inference: Opt-in attention and MLP projection fusion, faster quantized embedding lookups, and improved small-batch inference. Native backend selection and packed-weight storage are preserved. Fusion guide
- Configuration: Typed backend configurations unify backend selection and tuning, with explicit numerical and worker controls instead of environment switches.
- Calibration: Broader dataset input support, document-isolated AWQ replay, and opt-in attention-sink exclusion improve calibration coverage and control.
- Memory: Reduced packed-weight duplication, bounded EXL3 scratch, and shorter-lived inference workspaces lower memory use.
- Models: Added OLMo Hybrid, Granite SWA/MoE SWA, and Dots3 Note quantization, plus broader decoder-weight coverage and Transformers 5.22 compatibility. Model notes
- Compat: Major fixes across dense/MoE calibration, multimodal replay, checkpoint round trips, Transformers model loading, multi-GPU memory use, and ROCm execution. Unsupported combinations fail clearly rather than loading incomplete weights. Full notes
- Security: Runtime checks for vulnerable PyTorch versions, explicit checkpoint-code trust, stronger checkpoint validation, and bounded server/media requests. Runtime safety · Serving security
| Recent release |
Highlights |
| 7.10.0 |
Ricursive Ripple support, faster decode, checkpoint resharding, and multi-GPU fixes. |
| 7.9.0 |
Kernel tuning, MLX improvements, new models, and numerical fixes. |
| 7.8.0 |
GSQ/RCO training and allocation, expanded model support, and compatibility fixes. |
Detailed release notes · Older changelog
GPT-QModel brings calibration, quantization, evaluation, conversion, and accelerated
inference into one workflow. Start from a pretrained model, choose a compression
recipe, validate its quality, and save a checkpoint for native inference or a
compatible deployment engine.
Methods and compression 🎛️
Quantized kernels keep only packed weights resident; per-call QQQ and EXL3 kernel scratch, one quantization stream and one tile-scratch pair per device, per-device plan lifetimes and the single GGUF CUDA weight copy are documented in kernel scratch lifecycle. CPU layer packing quantizes and packs each bounded input block in one shift/reduce pass; see CPU block packing.
Control, scale, and quality ⚙️
- Choose where to spend precision. Mix bit widths, keep sensitive modules
unquantized, and use activation-quantization or preprocessing policies where
supported. Quantization protocols let you organize multi-stage recipes.
- Keep floating activation searches finite. NVFP4 block-scale ranking
normalizes candidate error by an exact power of two so large finite
activations cannot overflow the comparison and silently return zeros; see
NVFP4 candidate ranking.
- Fit larger models. Multi-GPU calibration, CPU/disk offload, and checkpointed
execution help manage memory and recover interrupted jobs. Disk-offload run
directories
are per-run scratch space and are removed at teardown or failure, and the
in-session model
is fully materialized when
quantize() returns;
reference offload
keeps untouched base modules as source-checkpoint references instead of
second copies, and releases pinned staging memory. Quantized MoE reloads park
fused expert projections on meta until the packed weights are read
(meta fused-expert shell).
Distributed EXL3 workflows support durable replay and resume.
- Recover accuracy. Use GSQ refinement, EoRA low-rank compensation, rotation,
smoothing, and norm tuning for supported recipes. LoRA training and adapter
loading support further task adaptation; exported EoRA
rank_pattern keys follow the adapter rank-pattern contract so PEFT builds the trained rank per module. Rotation model support lists the exact definitions that opt in. AWQ GSQ uses the
packed-code scoring contract to evaluate exported candidates.
The fused relaxation kernel keeps the eager precision contract
for BF16 and FP32 logits, and per-linear refinement searches the exported
codes and refits scales under the scalar refinement
contract.
- Trust the fold. AWQ divides a scale only into the norm that actually feeds
a quantized block, including multimodal wrappers and hybrid layer shapes; see
the AWQ scaling groups contract. Scale search keeps
routing in the original-input coordinates, validates replay coverage, and
restores temporary weights on failure; see search safety.
- Reject unrepresentable AWQ pack scales. The native GEMM/GEMV packers
validate the actual FP16 storage cast before replacing module state; see
AWQ pack-time scale representability.
- Decode AWQ weights in range. The portable Torch AWQ forward picks its
decode dtype from the stored scale range, so a wide or tiny checkpoint scale
no longer becomes
inf/0 before the product; see
AWQ Torch forward decode range.
- Keep the AWQ bias out of the scale dtype. A non-quantized bias is added
after the reduction, so packing keeps its source precision instead of casting
it to FP16; see
AWQ pack-time bias storage.
- Keep the Marlin bias in the checkpoint. The native Marlin kernel adds the
bias to its FP32 accumulator, so the buffer keeps its checkpoint width and
order and the padded, permuted operand is derived per launch; see
Marlin bias epilogue and storage.
- Measure before deploying. Evaluate perplexity and downstream tasks, compare
against a non-quantized baseline, and test prompts representative of your users.
Compression settings trade model size, calibration cost, speed, and quality.
- Preserve causal replay. Bloom AWQ keeps captured attention context and
rejects unsafe residual folds; see Bloom replay safety.
- Capture masks a model passes positionally. GPT-2 hands its causal mask to
each block as a positional argument; reading only keyword arguments dropped it
and replayed every layer without causality. See
positional replay mask capture.
- Replay each calibration document separately. AWQ token-row replay rebuilds
a block-diagonal per-document mask and keeps each row’s original position;
additive masks treat
0 as attended. See
AWQ calibration aggregation.
- Resolve the rotary label from the attention. AWQ scale search rebuilds
position_embeddings with the attention’s own rope label and keeps a
per-rope-type mapping intact; see
AWQ rotary replay selection.
- Trim padding before the position budget. Over-long calibration rows are
measured on their real tokens, not their padded width, so a left-padded row
cannot lose real tokens to its own padding. See
calibration row trimming.
- Keep VLM calibration rows intact. Pre-tokenized rows (
input_ids) take the
base text path instead of a chat template, columnar mappings are unpacked, and
conversations reject CalibrationConfig.concat_size. See
VLM calibration input normalization.
Inference and deployment 🚢
Run compatible checkpoints directly with GPT-QModel, integrate with Transformers
(import gptqmodel first so packed MoE experts load through the
native bridge),
or deploy supported GPTQ/AWQ formats through vLLM and SGLang. Apple silicon has
native MLX quantization and packed inference. Optional projection fusion can
combine supported attention and MLP operations without changing the saved format.
EoRA/LoRA adapters, mixed-precision checkpoints, and external engines have their
own compatibility limits; support for a method does not imply support in every engine.
Start with the quantization guide.
External runtimes support only specific method/format combinations; check the
integration and backend tables before exporting.
| Platform |
Hardware |
Runtime overview |
| Linux |
NVIDIA GPUs |
CUDA kernels; Turing or newer, with backend-specific minimums |
| Linux |
AMD GPUs |
ROCm 10.1+; architecture-specific Torch/Triton/HIP paths |
| Linux |
Tenstorrent |
Torch/XLA/PJRT and native TTNN |
| Linux |
Huawei Ascend |
Native Torch through torch-npu / CANN |
| Linux |
Intel XPU |
Torch and TorchFused |
| Linux |
Intel/AMD CPUs |
Torch and optimized x86 kernels |
| macOS |
Apple silicon |
Native MLX/Metal quantization and inference; MPS/Torch |
| Windows 11 |
NVIDIA GPUs / CPU |
Torch |
Support depends on the chosen method, backend, dtype, and device architecture.
See the full hardware and kernel matrix,
ROCm guide, Tenstorrent guide,
and Apple silicon guide.
Tenstorrent-compatible block-float formats also run on CPU without Tenstorrent
hardware. See the CPU block-float guide.
Model Support 🤖
Support spans dense, mixture-of-experts (MoE), vision-language, and audio-language
families. The compact list below keeps model coverage visible; the
complete registry lists exact model-type aliases and
model-specific limitations.
| Families |
Supported models and variants |
| A |
A.X-K2; AfMoE / Trinity; Agnes 3.0 Flash; Apertus 1 / 1.5 |
| B–C |
Baichuan; Bailing MoE / Hybrid (LING / RING); Bloom; Brumby; ChatGLM; CodeGen; Cohere 1 / 2 / 2 MoE / Compass (North Mini / Micro Vision) |
| D |
DBRX / DBRX Converted; DeciLM; DeepSeek V2 / V3 / V3.2 / V4 / VL / VL2 / OCR2; DiffusionGemma; Dots1 / Dots3 Note; Dream |
| E–F |
ERNIE 4.5 / MoE / VL MoE; EXAONE 3 / 4 / 4.5; Falcon / Falcon H1 / Falcon Mamba; FastVLM / LLaVA-Qwen2 |
| G |
Gemma 1–4 / 3n / Unified; GLM / GLM4 / GLM4V / GLM5 / OCR / ASR; GPT-2; GPT BigCode; GPT-Neo / GPT-NeoX; GPT-OSS; GPT-J; Granite / Granite SWA / MoE / MoE SWA / MoE Hybrid; GRIN-MoE |
| H |
Helium; HRM; Hunyuan V1 / VL / OCR; HY-V3; Hymba |
| I–J |
Inkling; Instella; Intern S1 / S2 Preview; InternLM 1 / 2 / 2.5; InternVL 3 / InternVL Chat; Jamba / Jamba2 |
| K |
K2-Horizon (Dense / MoVA); Kimi K2 / K2.5 / K3 / Linear; Klear; KRAFTON Raon-1 mini |
| L |
Laguna; LFM2 / LFM2 MoE / LFM2-VL; LFM3 / LFM3 Omni / LFM3-VL; LLaDA2 MoE; Llama 1–4 / TinyLlama / Nemotron Ultra; Llama 3.2 VL; LLaVA-1.5; LocateAnything; LongCat Flash; LongLLaMA |
| M |
Mage-VL; Marin; MiMo / MiMo V2 / MiMo V2 Flash; MiniCPM / MiniCPM3 / MiniCPM-O / MiniCPM-V; MiniMax M2 / M3-VL; Mistral / Mistral3 / Ministral3; Mixtral; MobileLLM; MOSS; MPT; Muse Glimmer |
| N |
Nanbeige 4.2; Nemotron NAS / H / H Puzzle / Omni / Labs Diffusion |
| O |
OLMo / OLMo 2 / 3 / Hybrid / OLMoE; OpenVLA; OPT; Ouro; Ovis 1.6 / 2 / 2.5 / 2.6 MoE / 2.6 Next |
| P |
PanGu-α; Phi 1–4 / Phi MoE |
| Q |
Qwen 1–4 / 3.5 / 3.6 / 3.8 / MoE / Next; Qwen 2 / 2.5 / 3 VL / 3 VL MoE; Qwen 2.5 / 3 Omni; Qwen2-Audio; Qwen-Drive 1.0 |
| R |
RecursiveSI Flint; RefinedWeb; Ricursive Ripple |
| S |
Seed-OSS; SmolLM3; SmolVLM; Solar Open / Open 2; Spark 2.5; StableLM; StarCoder2 |
| T–V |
TeleChat2; Unlimited-OCR; Voxtral |
| X–Z |
Xing4.0 (29B-A4B); XVERSE; Yi; Zamba / Zamba2; ZDTaichu 5.0 |
For multimodal checkpoints, support may target the language decoder rather than
every vision/audio component. Kimi K3 / Linear requires the dedicated native-source
calibration workflow; Qwen 3.5 requires Transformers 5.2.0 or newer. Check the
model notes before starting a large run.
Browse quantized models on Hugging Face
Install 💾
Use a Python environment with a compatible PyTorch build for your hardware:
pip install gptqmodel
# Or: uv pip install gptqmodel
PyTorch 2.13.x or >= 2.15.0 is required; known-vulnerable versions are
blocked by the runtime safety check.
Compiled package backends build and cache extensions on first use (JIT),
so install the compiler and CUDA/ROCm toolkit required by your backend. HIP
builds stage their sources under the cache; see
HIP JIT source staging.
See installation and optional extras
for source installs and platform setup. Install vLLM or SGLang in a separate
environment aligned with that engine’s dependency pins.
Quick start 🚀
Inference 💬
Load an existing quantized checkpoint and generate text. GPT-QModel selects a
compatible installed backend for the checkpoint and your device; you can choose
an explicit backend when tuning a deployment.
from gptqmodel import GPTQModel
model = GPTQModel.load("ModelCloud/Llama-3.2-1B-Instruct-gptqmodel-4bit-vortex-v2.5")
tokens = model.generate("Uncovering deep insights begins with", max_new_tokens=64)[0]
print(model.tokenizer.decode(tokens))
Loading and inference
covers backend selection, generation, and adapters. Only enable
trust_remote_code=True for checkpoint code you trust. For model types the
installed Transformers implements natively, the native class is used unless the
checkpoint declares a remote-only layout; see
docs/trust-remote-code-isolation.md.
Generation settings preserved during quantized checkpoint re-save are described
in docs/generation-config-round-trip.md.
Re-saving a loaded fused-expert MoE defuses the export shell and fails closed on
dropped quantization payloads; see
docs/quantized-moe-resave-defuse.md.
In-session QQQ kernel rehoming and cache initialization are documented in
docs/qqq-kernel-selection.md.
QQQ’s shared-memory staging of the per-group scales and the reduction-scratch
ordering are documented in
docs/qqq-group-scale-staging.md.
Quantized modules load their scales and other quantization buffers at the dtype
the checkpoint stores, whatever model dtype you request; see
checkpoint buffer dtype.
Quantization 🗜️
This example creates a 4-bit GPTQ checkpoint using 256 streamed C4 samples.
Use calibration text representative of your workload; the dataset and precision
settings affect the quality of the resulting model.
from datasets import load_dataset
from gptqmodel import CalibrationConfig, GPTQConfig, GPTQModel
dataset = load_dataset(
"allenai/c4",
data_files="en/c4-train.00001-of-01024.json.gz",
split="train",
streaming=True,
)
calibration = [row["text"] for row in dataset.take(256)]
model = GPTQModel.load(
"Qwen/Qwen3-0.6B", GPTQConfig(bits=4, group_size=128),
)
model.quantize(CalibrationConfig(dataset=calibration, batch_size=1))
model.save("Qwen3-0.6B-GPTQ-4bit")
quantize() takes every calibration choice in one CalibrationConfig: dataset,
concat_size, sort ("desc"), batch_size (1), min_length (10),
concat_separator, exclude_attention_sinks (False) and the EoRA
adapter_dataset. See
the calibration API.
The end-to-end guide covers other methods,
calibration, saving, reloading, and validation.
For Apple silicon, use the native MLX workflow.
What to tune next 🔧
- Quality versus size: Start from a supported recipe, then compare bit widths
and group sizes. Use mixed precision or exclusions for sensitive layers rather
than assuming one configuration suits the whole model.
- Calibration: Choose representative text and enough examples to cover your
use case. MoE and multimodal models may need specialized data preparation.
- Memory and recovery: Configure multi-GPU placement and offload before a
large run, and use checkpoint/resume where supported.
- Deployment: Reload the saved checkpoint, evaluate it, and verify its format
in the intended engine. A successful native load is not an external-engine
compatibility guarantee.
See the configuration reference
and quantization recipes for the available controls.
Guides and reference 🧭
Kernel contracts, implementation flows, and focused technical notes are in the
documentation index, not repeated on this page.
Quality: GPTQ 4-bit evaluation 🏆
Quantization quality depends on the source model, recipe, and calibration data.
The Vortex collection provides GPTQ examples; the charts below illustrate selected
evaluations, not a guarantee for every supported model or method.
ModelCloud quantized Vortex models

Organizations and projects using GPT-QModel 🌐
Selected public references include Hugging Face, Intel, NVIDIA, Alibaba Cloud,
Fujitsu, and IBM Research.
This is not an exhaustive customer list.
Contributions welcome 🛠️
We welcome all contributors, including people working with human-guided AI
agents. Code, model and hardware support, documentation, bug reports, reviews,
and testing all help move GPT-QModel forward. See the
developer guide
to get started.
Support ongoing development 💛
Supporting a wide range of hardware and models requires continuous CI testing,
accuracy validation, and compatibility work. That takes substantial compute and
AI-agent tokens. Contributions of compute, model/API credits, or financial
support of any size help sustain project maintenance and future development.
Interested in sponsoring? Message Qubitium on X.
If you find GPT-QModel useful, sending even a couple of dollars through X is
also appreciated. Ongoing sponsors will be credited on this README page.
License 📜
GPT-QModel is licensed under Apache-2.0. The optional, JIT-loaded
Swordfish kernel is AGPL-3.0-or-later.
Attributions and citations 📚
GPT-QModel builds on the research and open-source projects below. Please cite
GPT-QModel and the methods you use; full author lists, BibTeX, and attribution
details are available in the citation reference.
Cite GPT-QModel (BibTeX)
@misc{qubitium2024gptqmodel,
author = {ModelCloud.ai and [email protected]},
title = {GPT-QModel},
year = {2024},
howpublished = {\url{https://github.com/ModelCloud/GPTQModel}}
}
| Work |
Authors / project |
Paper or reference |
| GPT-QModel |
ModelCloud.ai and contributors |
Project |
| GPTQ |
Elias Frantar et al., IST-DASLab |
Paper |
| AWQ |
Ji Lin et al. |
Paper |
| Paro |
Yesheng Liang et al., Z-Lab |
Paper · Code |
| MACaQ |
Lorenz K. Muller et al., Huawei CSL |
Paper · Code |
| QQQ |
Ying Zhang et al., Meituan |
Paper |
| VPTQ |
Yifei Liu et al., Microsoft |
Paper · Code |
| GGUF / llama.cpp |
Georgi Gerganov and ggml-org contributors |
Project |
| ExLlama V3 / EXL3 |
turboderp and contributors |
Project |
| NF12 / NyoomFloat12 |
Liberman et al., Together AI |
Research and codec |
| EoRA |
Shih-Yang Liu et al., NVIDIA |
Paper |
| GSQ |
Alireza Dadgarnia et al., IST-DASLab |
Paper · Code |
| RCO |
Michael Helcig and Dan Alistarh, IST-DASLab |
Paper · Code |
| GPTAQ |
Yuhang Li et al., Yale Intelligent Computing Lab |
Paper |
| GAR |
T. Gafni, A. Karnieli, and Y. Hanani, Intel |
Paper |
| FOEM |
Xingyu Zheng et al. |
Paper |
| Norm Tweaking |
Liang Li, Qingyuan Li, Bo Zhang, and Xiangxiang Chu |
Paper |
| Fast Hadamard Transform |
Tri Dao, Dao-AILab |
Code |
| Marlin |
Elias Frantar et al. |
Paper |
| Machete |
vLLM contributors |
Code |
| Swordfish |
AlpinDale |
Article |
评论