auto-round

🚀 What is AutoRound?

AutoRound is an easy-to-use post-training quantization toolkit for Large Language Models (LLMs) and Vision-Language Models (VLMs). It supports several practical low-bit and adaptive-bit quantization algorithms, with a particular focus on SignRoundV1 and SignRoundV2. AutoRound is designed to deliver strong accuracy at ultra-low bit widths while keeping quantization costs reasonable. It supports widely used data types and quantization schemes, including WNA16, MXFP4, NVFP4, and FP8, as well as low-bit model formats such as GGUF. It integrates with popular ecosystems including Transformers, vLLM, and SGLang, and supports deployment across Intel CPUs and XPUs, Gaudi accelerators, and NVIDIA GPUs.

🆕 What’s New

• [WIP] For quantization at 4 bits and above, we currently recommend mixing 75% UltraChat with 25% OpenCode to improve reasoning capabilities and agent compatibility. With the default nsamples=128, use --dataset "HuggingFaceH4/ultrachat_200k:num=96,nvidia/OpenCodeInstruct:num=32". See the doc for details. We will update the default calibration dataset after the evaluation is complete.

• [2026/10] We now show that W2A16 accuracy improves steadily as more tuning cost is invested. See the W2A16 accuracy results.

• [2026/09] We improved MoE tuning speed by more than 3× and significantly reduced RAM usage. See the doc for detailed results. Please note that this optimization may introduce some regressions or issues.

• [2026/09] We now support 5/6/7-bit WOQ models in vLLM and Transformers on CUDA devices, thanks to Humming Kernel.

• [2026/09] The NeUQI grid search (arXiv 2505.17595) is available for calibration-free quantization: --enable_neuqi with --iters 0 runs a joint (scale, zero-point) search for asymmetric layers and a two-stage scale search for symmetric layers, and anchors the tuning grid when iters > 0. Results

• [2026/08] We experimentally support algorithm composition (e.g., --algs awq,signround or --algs hadamard,awq,signround) to improve accuracy Overview. We welcome any practical, deployable algorithms that are ready for real-world use. Feel free to submit a PR or leave a comment in Issues.

• [2026/06] AutoScheme has been refined to improve accuracy for gguf format. See AutoScheme Accuracy for details. This enhancement incurs additional tuning cost.

• [2026/06] Community adoption update. AutoRound is now fully integrated into vLLM-Omni vLLM blog, following its adoption by Transformers, vLLM, and SGLang. See the vLLM blog.

• [2026/05] We provide free devices for quantization; please visit Intel Low Bit Open LLM Leaderboard for more details.

• [2025/12] The SignRoundV2 paper is available. Turn on the algorithm extension (--algorithm auto_round_v2, formerly enable_alg_ext) and use the AutoScheme API for mixed-precision quantization to reproduce the results: Paper, Notes for evaluating LLaMA models.

• [2025/10] AutoScheme is available to generate mixed-bit recipes in minutes: Usage, Accuracy.

• [2025/08] An improved INT2 algorithm is available via the algorithm extension (--algorithm auto_round_v2, formerly --enable_alg_ext): Accuracy

• [2025/09] MXFP4 and NVFP4 dtypes is available: Accuracy.

• [2025/07] GGUF format is supported: Usage.

• [2025/03] The INT2-mixed DeepSeek-R1 model (~200GB) retains 97.9% accuracy: Model.

✨ Key Features

✅ Superior Accuracy Delivers strong performance even at 2–3 bits example models, with leading results at 4 bits benchmark.

✅ Ecosystem Integration Seamlessly works with Transformers, vLLM, SGLang and more.

✅ Multiple Formats Export Support AutoRound, AutoAWQ, AutoGPTQ, and GGUF for maximum compatibility. Details are shown in export formats

✅ Fast Mixed Bits/Dtypes Scheme Generation Automatically configure in minutes, with about 1.1X-1.5X the model’s BF16 RAM size as overhead. Accuracy results and user guide.

✅ Optimized Round-to-Nearest Mode Use --iters 0 for fast quantization with some accuracy drop for 4 bits; an optional NeUQI grid search (--enable_neuqi) improves the zero-shot scales. Details are shown in opt_rtn mode

✅ Affordable Quantization Cost Quantize 7B models in about 10 minutes on a single GPU. Details are shown in quantization costs

✅ 10+ VLMs Support Out-of-the-box quantization for 10+ vision-language models example models, support matrix

✅ Multiple Recipes Choose from auto-round-best, auto-round, auto-round-light, auto-round-opt-rtn (optimized RTN) and auto-round-rtn (pure RTN, fastest baseline) to suit your needs. Details are shown in quantization recipes

Use the same plans in Python with AutoRound(..., recipe="auto-round-best") or in the CLI with --recipe auto-round-best. Omitting the recipe selects auto-round; legacy commands remain shortcuts. See built-in recipes and overrides.

✅ Advanced Utilities Includes multiple gpus quantization, multiple calibration datasets and support for 10+ runtime backends.

✅ Beyond weight only quantization. We are actively expanding support for additional datatypes such as MXFP, NVFP, W8A8, and more.

Installation

Install from pypi

# CPU(Xeon)/GPU(CUDA)
pip install auto-round

# CPU(Xeon)/GPU(CUDA) nightly
pip install auto-round-nightly

# HPU(Gaudi)
# install inside the hpu docker container, e.g. vault.habana.ai/gaudi-docker/1.23.0/ubuntu24.04/habanalabs/pytorch-installer-2.9.0:latest  
pip install auto-round-hpu

# XPU(Intel GPU)
pip install torch --index-url https://download.pytorch.org/whl/xpu
pip install auto-round

# CPU(Xeon)/GPU(CUDA)
pip install .

# HPU(Gaudi)
python setup.py install hpu

# XPU(Intel GPU)
pip install torch --index-url https://download.pytorch.org/whl/xpu
pip install .

Model Quantization (CPU/Intel GPU/Gaudi/CUDA)

If you encounter issues during quantization, try using pure RTN mode with iters=0, disable_opt_rtn=True. Additionally, using group_size=32 or mixed bits is recommended for better results.

CLI Usage

The full list of supported arguments is provided by calling auto-round -h on the terminal.

ModelScope is supported for model downloads, simply set AR_USE_MODELSCOPE=1.

auto-round \
    --model Qwen/Qwen3-0.6B \
    --scheme "W4A16" \
    --format "auto_round" \
    --output_dir ./tmp_autoround

We offer another two recipes, auto-round-best and auto-round-light, designed for optimal accuracy and improved speed, respectively. Details are as follows.

# Best accuracy, 3X slower, low_gpu_mem_usage could save ~20G but ~30%-100% slower
auto-round-best \
  --model Qwen/Qwen3-0.6B \
  --scheme "W4A16" \
  --low_gpu_mem_usage 

# 2-3X speedup, slight accuracy drop at W4 and larger accuracy drop at W2
auto-round-light \
  --model Qwen/Qwen3-0.6B \
  --scheme "W4A16" 

# Optimized RTN (iters=0, opt_rtn enabled); fast baseline
auto-round-opt-rtn \
  --model Qwen/Qwen3-0.6B \
  --scheme "W4A16"

# Pure RTN (iters=0, no AutoRound optimization); fastest, lowest memory
# auto-routes to model-free mode for supported INT WOQ schemes
auto-round-rtn \
  --model Qwen/Qwen3-0.6B \
  --scheme "W4A16"

In conclusion, we recommend using auto-round for W4A16 and auto-round-best with --algorithm auto_round_v2 for W2A16. However, you may adjust the configuration to suit your specific requirements and available resources.

API Usage

from auto_round import AutoRound

# Load a model (supports FP8/BF16/FP16/FP32)
model_name_or_path = "Qwen/Qwen3-0.6B"

# Available schemes: "W2A16", "W3A16", "W4A16", "W8A16", "NVFP4", "MXFP4" (no real kernels), "GGUF:Q4_K_M", etc.
ar = AutoRound(model_name_or_path, scheme="W4A16")

# Highest accuracy (4–5× slower).
# `low_gpu_mem_usage=True` saves ~20GB VRAM but runs ~30%-100% slower.
# ar = AutoRound(model_name_or_path, nsamples=512, iters=1000, low_gpu_mem_usage=True)

# Faster quantization (2–3× speedup) with slight accuracy drop at W4G128.
# ar = AutoRound(model_name_or_path, nsamples=128, iters=50, lr=5e-3)

# Supported formats: "auto_round" (default), "auto_gptq", "auto_awq", "llm_compressor", "gguf:q4_k_m", etc.
ar.quantize_and_save(output_dir="./qmodel", format="auto_round")

Supported Schemes

Adaptive Schemes (Experimental Feature)

AutoScheme provides an automatic algorithm to generate adaptive mixed bits/data-type quantization recipes. Please refer to the user guide for more details on AutoScheme.

from auto_round import AutoRound

model_name = "Qwen/Qwen3-8B"
layer_config = {"lm_head": "GGUF:Q6_K"}

# Change iters to 200 for non-GGUF schemes
ar = AutoRound(
    model=model_name,
    schemes=("GGUF:Q2_K_S", "GGUF:Q4_K_S"),
    bits=3.0,
    ignore_scale_zp_bits=True,
    layer_config=layer_config,
    iters=0,
)
ar.quantize_and_save()

API Usage for VLMs

from auto_round import AutoRound

# Load the model
model_name_or_path = "Qwen/Qwen2.5-VL-7B-Instruct"
# Quantize the model
ar = AutoRound(model_name_or_path, scheme="W4A16")
output_dir = "./qmodel"
ar.quantize_and_save(output_dir)

Model Inference

vLLM (CPU/Intel GPU/CUDA)

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
]
sampling_params = SamplingParams(temperature=0.6, top_p=0.95)
model_name = "Intel/DeepSeek-R1-0528-Qwen3-8B-int4-AutoRound"
llm = LLM(model=model_name)

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

SGLang (Intel GPU/CUDA)

Please note that support for the MoE models and visual language models is currently limited.

import sglang as sgl

llm = sgl.Engine(model_path="Intel/DeepSeek-R1-0528-Qwen3-8B-int4-AutoRound")
prompts = [
    "Hello, my name is",
]
sampling_params = {"temperature": 0.6, "top_p": 0.95}

outputs = llm.generate(prompts, sampling_params)
for prompt, output in zip(prompts, outputs):
    print(f"Prompt: {prompt}\nGenerated text: {output['text']}")

Transformers (CPU/Intel GPU/Gaudi/CUDA)

AutoRound supports 10+ backends and automatically selects the best available backend based on the installed libraries and prompts the user to install additional libraries when a better backend is found.

Please avoid manually moving the quantized model to a different device (e.g., model.to(‘cpu’)) during inference, as this may cause unexpected exceptions.

The support for Gaudi device is limited.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Intel/DeepSeek-R1-0528-Qwen3-8B-int4-AutoRound"
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)
text = "There is a girl who likes adventure,"
inputs = tokenizer(text, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))

Publications & Events

SignRoundV2: Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMs (2025.12 paper)

Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLM (2023.09 paper)

TEQ: Trainable Equivalent Transformation for Quantization of LLMs (2023.10 paper)

Effective Post-Training Quantization for Large Language Models (2023.04 blog)

Check out Full Publication List.

Acknowledgement

Special thanks to open-source low precision libraries such as AutoGPTQ, AutoAWQ, GPTQModel, Triton, Marlin, and ExLLaMAV2 for providing low-precision CUDA kernels, which are leveraged in AutoRound.

🌟 Support Us

If you find AutoRound helpful, please ⭐ star the repo and share it with your community!