GPT-QModel ⚡
An extensible platform for LLM quantization, validation, and deployment.
Compress, evaluate, and run models across supported GPUs, accelerators, and CPUs.
Documentation ·
Quantization guide ·
Reference index ·
Releases
Latest News 🗞️🚀
7.11.0
- MACaQ: Native MXFP4 W4A4 quantization for low-precision weights and activations, with CUDA, ROCm, and Apple silicon inference. Includes packed checkpoints and a portable Torch fallback.
- NF12: Lossless 12-bit compression of BF16 weights reduces source-weight storage during calibration without changing the restored values. Packed inference now includes native Metal support alongside CUDA, HIP, and Torch paths. MLX
- Tenstorrent: Packed GPTQ inference through Torch/XLA and TTNN, multi-card execution, and block-float quantization. Long-context Llama serving adds paged caching and chunked prefill; compatible block-float checkpoints also run on CUDA, ROCm, and CPU. Guide · Serving
- Quantization: Resumable distributed EXL3 jobs, broader mixed-bit allocation across AWQ/EXL3/GGUF, and SmoothQuant preprocessing improve large-model workflows and quality control. Quantizer calibration flows document the shared Hessian-failure and zero-sample fallback contracts.
- Inference: Opt-in attention and MLP projection fusion, faster quantized embedding lookups, and improved small-batch inference. Native backend selection and packed-weight storage are preserved. Fusion guide
- Compat: Major fixes across dense/MoE calibration, multimodal replay, checkpoint round trips, Transformers model loading, multi-GPU memory use, and ROCm execution. Unsupported combinations fail clearly rather than loading incomplete weights. Full notes
- Security: Runtime checks for vulnerable PyTorch versions, explicit checkpoint-code trust, stronger checkpoint validation, and bounded server/media requests. Runtime safety · Serving security
| Recent release |
Highlights |
| 7.10.0 |
Ricursive Ripple support, faster decode, checkpoint resharding, and multi-GPU fixes. |
| 7.9.0 |
Kernel tuning, MLX improvements, new models, and numerical fixes. |
| 7.8.0 |
GSQ/RCO training and allocation, expanded model support, and compatibility fixes. |
Detailed release notes · Older changelog
GPT-QModel brings calibration, quantization, evaluation, conversion, and accelerated
inference into one workflow. Start from a pretrained model, choose a compression
recipe, validate its quality, and save a checkpoint for native inference or a
compatible deployment engine.
Methods and compression 🎛️
| Method |
What it provides |
| GPTQ |
Calibration-driven, low-bit weight quantization with mixed precision and broad checkpoint/runtime support. |
| AWQ |
Activation-aware weight quantization, including dense and MoE workflows. |
| RTN |
Round-to-nearest weight quantization when a calibration-free baseline is useful. |
| ParoQuant |
Low-bit weight quantization with learned rotations for accuracy recovery. Affine fitting and optimization controls. |
| MACaQ |
Calibrated MXFP4 weights and activations, with native W4A4 inference and a W4A16 option. |
| QQQ |
INT4 weights with dynamic INT8 activation quantization for W4A8 inference. |
| GGUF |
Block-quantized checkpoints for GGUF-oriented workflows and native inference. |
| FP8 |
Floating-point weight quantization with scaled E4M3/E5M2 formats. Native inference is weight-only W8A16; W8A8 is an explicit per-token opt-in. |
| EXL3 |
Variable-rate compression with target average bits per weight and mixed-tier allocation. |
| VPTQ |
Vector-codebook quantization for low-bit model compression. |
| BitsAndBytes |
NF4/FP4 and INT8 quantization options. |
| NF12 |
Lossless BF16 weight compression, distinct from lossy low-bit quantization. |
Control, scale, and quality ⚙️
- Choose where to spend precision. Mix bit widths, keep sensitive modules
unquantized, and use activation-quantization or preprocessing policies where
supported. Quantization protocols let you organize multi-stage recipes.
- Keep floating activation searches finite. NVFP4 block-scale ranking
normalizes candidate error by an exact power of two so large finite
activations cannot overflow the comparison and silently return zeros; see
NVFP4 candidate ranking.
- Fit larger models. Multi-GPU calibration, CPU/disk offload, and checkpointed
execution help manage memory and recover interrupted jobs. Distributed EXL3
workflows support durable replay and resume.
- Recover accuracy. Use GSQ refinement, EoRA low-rank compensation, rotation,
smoothing, and norm tuning for supported recipes. LoRA training and adapter
loading support further task adaptation; exported EoRA
rank_pattern keys follow the adapter rank-pattern contract so PEFT builds the trained rank per module. Rotation model support lists the exact definitions that opt in. AWQ GSQ uses the
packed-code scoring contract to evaluate exported candidates.
The fused relaxation kernel keeps the eager precision contract
for BF16 and FP32 logits.
- Trust the fold. AWQ divides a scale only into the norm that actually feeds
a quantized block, including multimodal wrappers and hybrid layer shapes; see
the AWQ scaling groups contract. Scale search keeps
routing in the original-input coordinates, validates replay coverage, and
restores temporary weights on failure; see search safety.
- Reject unrepresentable AWQ pack scales. The native GEMM/GEMV packers
validate the actual FP16 storage cast before replacing module state; see
AWQ pack-time scale representability.
- Measure before deploying. Evaluate perplexity and downstream tasks, compare
against a non-quantized baseline, and test prompts representative of your users.
Compression settings trade model size, calibration cost, speed, and quality.
- Preserve causal replay. Bloom AWQ keeps captured attention context and
rejects unsafe residual folds; see Bloom replay safety.
- Replay each calibration document separately. AWQ token-row replay rebuilds
a block-diagonal per-document mask and keeps each row’s original position;
additive masks treat
0 as attended. See
AWQ calibration aggregation.
- Trim padding before the position budget. Over-long calibration rows are
measured on their real tokens, not their padded width, so a left-padded row
cannot lose real tokens to its own padding. See
calibration row trimming.
Inference and deployment 🚢
Run compatible checkpoints directly with GPT-QModel, integrate with Transformers,
or deploy supported GPTQ/AWQ formats through vLLM and SGLang. Apple silicon has
native MLX quantization and packed inference. Optional projection fusion can
combine supported attention and MLP operations without changing the saved format.
EoRA/LoRA adapters, mixed-precision checkpoints, and external engines have their
own compatibility limits; support for a method does not imply support in every engine.
Start with the quantization guide.
External runtimes support only specific method/format combinations; check the
integration and backend tables before exporting.
| Platform |
Hardware |
Runtime overview |
| Linux |
NVIDIA GPUs |
CUDA kernels; Turing or newer, with backend-specific minimums |
| Linux |
AMD GPUs |
ROCm 10.1+; architecture-specific Torch/Triton/HIP paths |
| Linux |
Tenstorrent |
Torch/XLA/PJRT and native TTNN |
| Linux |
Huawei Ascend |
Native Torch through torch-npu / CANN |
| Linux |
Intel XPU |
Torch and TorchFused |
| Linux |
Intel/AMD CPUs |
Torch and optimized x86 kernels |
| macOS |
Apple silicon |
Native MLX/Metal quantization and inference; MPS/Torch |
| Windows 11 |
NVIDIA GPUs / CPU |
Torch |
Support depends on the chosen method, backend, dtype, and device architecture.
See the full hardware and kernel matrix,
ROCm guide, Tenstorrent guide,
and Apple silicon guide.
Tenstorrent-compatible block-float formats also run on CPU without Tenstorrent
hardware. See the CPU block-float guide.
Model Support 🤖
Support spans dense, mixture-of-experts (MoE), vision-language, and audio-language
families. The compact list below keeps model coverage visible; the
complete registry lists exact model-type aliases and
model-specific limitations.
| Families |
Supported models and variants |
| A |
A.X-K2; AfMoE / Trinity; Agnes 3.0 Flash; Apertus 1 / 1.5 |
| B–C |
Baichuan; Bailing MoE / Hybrid (LING / RING); Bloom; Brumby; ChatGLM; CodeGen; Cohere 1 / 2 / 2 MoE / Compass (North Mini / Micro Vision) |
| D |
DBRX / DBRX Converted; DeciLM; DeepSeek V2 / V3 / V3.2 / V4 / VL / VL2 / OCR2; DiffusionGemma; Dots1 / Dots3 Note; Dream |
| E–F |
ERNIE 4.5 / MoE / VL MoE; EXAONE 3 / 4 / 4.5; Falcon / Falcon H1 / Falcon Mamba; FastVLM / LLaVA-Qwen2 |
| G |
Gemma 1–4 / 3n / Unified; GLM / GLM4 / GLM4V / GLM5 / OCR / ASR; GPT-2; GPT BigCode; GPT-Neo / GPT-NeoX; GPT-OSS; GPT-J; Granite / Granite SWA / MoE / MoE SWA / MoE Hybrid; GRIN-MoE |
| H |
Helium; HRM; Hunyuan V1 / VL / OCR; HY-V3; Hymba |
| I–J |
Inkling; Instella; Intern S1 / S2 Preview; InternLM 1 / 2 / 2.5; InternVL 3 / InternVL Chat; Jamba / Jamba2 |
| K |
K2-Horizon (Dense / MoVA); Kimi K2 / K2.5 / K3 / Linear; Klear; KRAFTON Raon-1 mini |
| L |
Laguna; LFM2 / LFM2 MoE / LFM2-VL; LFM3 / LFM3 Omni / LFM3-VL; LLaDA2 MoE; Llama 1–4 / TinyLlama / Nemotron Ultra; Llama 3.2 VL; LLaVA-1.5; LocateAnything; LongCat Flash; LongLLaMA |
| M |
Mage-VL; Marin; MiMo / MiMo V2 / MiMo V2 Flash; MiniCPM / MiniCPM3 / MiniCPM-O / MiniCPM-V; MiniMax M2 / M3-VL; Mistral / Mistral3 / Ministral3; Mixtral; MobileLLM; MOSS; MPT; Muse Glimmer |
| N |
Nanbeige 4.2; Nemotron NAS / H / H Puzzle / Omni / Labs Diffusion |
| O |
OLMo / OLMo 2 / 3 / Hybrid / OLMoE; OpenVLA; OPT; Ouro; Ovis 1.6 / 2 / 2.5 / 2.6 MoE / 2.6 Next |
| P |
PanGu-α; Phi 1–4 / Phi MoE |
| Q |
Qwen 1–4 / 3.5 / 3.6 / 3.8 / MoE / Next; Qwen 2 / 2.5 / 3 VL / 3 VL MoE; Qwen 2.5 / 3 Omni; Qwen2-Audio; Qwen-Drive 1.0 |
| R |
RecursiveSI Flint; RefinedWeb; Ricursive Ripple |
| S |
Seed-OSS; SmolLM3; SmolVLM; Solar Open / Open 2; Spark 2.5; StableLM; StarCoder2 |
| T–V |
TeleChat2; Unlimited-OCR; Voxtral |
| X–Z |
Xing4.0 (29B-A4B); XVERSE; Yi; Zamba / Zamba2; ZDTaichu 5.0 |
For multimodal checkpoints, support may target the language decoder rather than
every vision/audio component. Kimi K3 / Linear requires the dedicated native-source
calibration workflow; Qwen 3.5 requires Transformers 5.2.0 or newer. Check the
model notes before starting a large run.
Browse quantized models on Hugging Face
Install 💾
Use a Python environment with a compatible PyTorch build for your hardware:
pip install gptqmodel
# Or: uv pip install gptqmodel
PyTorch 2.13.x or >= 2.15.0 is required; known-vulnerable versions are
blocked by the runtime safety check.
Compiled package backends build and cache extensions on first use (JIT),
so install the compiler and CUDA/ROCm toolkit required by your backend.
See installation and optional extras
for source installs and platform setup. Install vLLM or SGLang in a separate
environment aligned with that engine’s dependency pins.
Quick start 🚀
Inference 💬
Load an existing quantized checkpoint and generate text. GPT-QModel selects a
compatible installed backend for the checkpoint and your device; you can choose
an explicit backend when tuning a deployment.
from gptqmodel import GPTQModel
model = GPTQModel.load("ModelCloud/Llama-3.2-1B-Instruct-gptqmodel-4bit-vortex-v2.5")
tokens = model.generate("Uncovering deep insights begins with", max_new_tokens=64)[0]
print(model.tokenizer.decode(tokens))
Loading and inference
covers backend selection, generation, and adapters. Only enable
trust_remote_code=True for checkpoint code you trust.
Quantization 🗜️
This example creates a 4-bit GPTQ checkpoint using 256 streamed C4 samples.
Use calibration text representative of your workload; the dataset and precision
settings affect the quality of the resulting model.
from datasets import load_dataset
from gptqmodel import GPTQConfig, GPTQModel
dataset = load_dataset(
"allenai/c4",
data_files="en/c4-train.00001-of-01024.json.gz",
split="train",
streaming=True,
)
calibration = [row["text"] for row in dataset.take(256)]
model = GPTQModel.load(
"Qwen/Qwen3-0.6B", GPTQConfig(bits=4, group_size=128),
)
model.quantize(calibration, batch_size=1)
model.save("Qwen3-0.6B-GPTQ-4bit")
The end-to-end guide covers other methods,
calibration, saving, reloading, and validation.
For Apple silicon, use the native MLX workflow.
What to tune next 🔧
- Quality versus size: Start from a supported recipe, then compare bit widths
and group sizes. Use mixed precision or exclusions for sensitive layers rather
than assuming one configuration suits the whole model.
- Calibration: Choose representative text and enough examples to cover your
use case. MoE and multimodal models may need specialized data preparation.
- Memory and recovery: Configure multi-GPU placement and offload before a
large run, and use checkpoint/resume where supported.
- Deployment: Reload the saved checkpoint, evaluate it, and verify its format
in the intended engine. A successful native load is not an external-engine
compatibility guarantee.
See the configuration reference
and quantization recipes for the available controls.
Guides and reference 🧭
Kernel contracts, implementation flows, and focused technical notes are in the
documentation index, not repeated on this page.
Quality: GPTQ 4-bit evaluation 🏆
Quantization quality depends on the source model, recipe, and calibration data.
The Vortex collection provides GPTQ examples; the charts below illustrate selected
evaluations, not a guarantee for every supported model or method.
ModelCloud quantized Vortex models

Organizations and projects using GPT-QModel 🌐
Selected public references include Hugging Face, Intel, NVIDIA, Alibaba Cloud,
Fujitsu, and IBM Research.
This is not an exhaustive customer list.
Contributions welcome 🛠️
We welcome all contributors, including people working with human-guided AI
agents. Code, model and hardware support, documentation, bug reports, reviews,
and testing all help move GPT-QModel forward. See the
developer guide
to get started.
Support ongoing development 💛
Supporting a wide range of hardware and models requires continuous CI testing,
accuracy validation, and compatibility work. That takes substantial compute and
AI-agent tokens. Contributions of compute, model/API credits, or financial
support of any size help sustain project maintenance and future development.
Interested in sponsoring? Message Qubitium on X.
If you find GPT-QModel useful, sending even a couple of dollars through X is
also appreciated. Ongoing sponsors will be credited on this README page.
License 📜
GPT-QModel is licensed under Apache-2.0. The optional, JIT-loaded
Swordfish kernel is AGPL-3.0-or-later.
Attributions and citations 📚
GPT-QModel builds on the research and open-source projects below. Please cite
GPT-QModel and the methods you use; full author lists, BibTeX, and attribution
details are available in the citation reference.
Cite GPT-QModel (BibTeX)
@misc{qubitium2024gptqmodel,
author = {ModelCloud.ai and [email protected]},
title = {GPT-QModel},
year = {2024},
howpublished = {\url{https://github.com/ModelCloud/GPTQModel}}
}
| Work |
Authors / project |
Paper or reference |
| GPT-QModel |
ModelCloud.ai and contributors |
Project |
| GPTQ |
Elias Frantar et al., IST-DASLab |
Paper |
| AWQ |
Ji Lin et al. |
Paper |
| ParoQuant |
Yesheng Liang et al., Z-Lab |
Paper · Code |
| MACaQ |
Lorenz K. Muller et al., Huawei CSL |
Paper · Code |
| QQQ |
Ying Zhang et al., Meituan |
Paper |
| VPTQ |
Yifei Liu et al., Microsoft |
Paper · Code |
| GGUF / llama.cpp |
Georgi Gerganov and ggml-org contributors |
Project |
| ExLlama V3 / EXL3 |
turboderp and contributors |
Project |
| NF12 / NyoomFloat12 |
Liberman et al., Together AI |
Research and codec |
| EoRA |
Shih-Yang Liu et al., NVIDIA |
Paper |
| GSQ |
Alireza Dadgarnia et al., IST-DASLab |
Paper · Code |
| RCO |
Michael Helcig and Dan Alistarh, IST-DASLab |
Paper · Code |
| GPTAQ |
Yuhang Li et al., Yale Intelligent Computing Lab |
Paper |
| GAR |
T. Gafni, A. Karnieli, and Y. Hanani, Intel |
Paper |
| FOEM |
Xingyu Zheng et al. |
Paper |
| Norm Tweaking |
Liang Li, Qingyuan Li, Bo Zhang, and Xiangxiang Chu |
Paper |
| Fast Hadamard Transform |
Tri Dao, Dao-AILab |
Code |
| Marlin |
Elias Frantar et al. |
Paper |
| Machete |
vLLM contributors |
Code |
| Swordfish |
AlpinDale |
Article |
评论