Xinference (Xorbits Inference) is an open-source distributed inference platform.
Deploy and run models on heterogeneous hardware, from a local machine to a distributed cluster. Xinference includes 300+ built-in models for language, vision, embeddings, reranking, image, audio, video, and experimental world generation. Register your own models, including traditional ML models through the Flexible type. Manage deployments in the Web UI and use models through OpenAI-compatible APIs, the Python SDK, or the CLI.
Quick start · Key features · Comparison · Self-hosting · Documentation · Model catalog · Releases · Xinference Enterprise
Languages · English · 日本語 · 한국어 · Deutsch · Français · Español · Italiano · Português · 繁體中文 · 简体中文
Quick start
1. Start the server
Xinference downloads the model and installs any needed engine dependencies on the first launch.
pip install xinference
xinference-local --host 127.0.0.1
Open http://127.0.0.1:9997. On a fresh instance, use the setup token printed in the server log to create an administrator account, then sign in. First-run setup
2. Launch a chat model
In Launch Model, search for qwen3.5 and open its launch form. Click Recommend configuration to fill in settings for your hardware. Review the recommendation and click Deploy.
3. Make your first request
Open the deployed model in Running Models to try a prompt in the Web UI. The first instance uses qwen3.5 as its Model UID. If your instance shows a different UID, use that value for model below. Create a key under API Key Management to call the API.
export XINFERENCE_API_KEY="<your-api-key>"
curl --fail-with-body http://127.0.0.1:9997/v1/chat/completions \
-H "Authorization: Bearer $XINFERENCE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen3.5",
"messages": [{"role": "user", "content": "Translate this sentence into Spanish: The model is ready."}],
"enable_thinking": false,
"max_tokens": 96
}'
Key features
| Area |
What Xinference supports |
| Model catalog |
300+ built-in models across language, vision, embeddings, reranking, images, audio, video, and experimental world generation. Register your own models, including Flexible models for traditional ML. Model catalog |
| Engines and hardware |
Choose vLLM, SGLang, TRT-LLM, Transformers, llama.cpp, or MLX for supported LLMs. Hardware support varies by engine and includes CPU, GPUs, Apple Silicon, and supported MLU configurations. Backends · MLU support |
| Distributed serving |
Run on one machine, add workers and replicas, or run a large model across workers with a supported engine. Compatible deployments support heterogeneous prefill/decode separation. Distributed inference · P/D separation |
| APIs and tools |
OpenAI- and Anthropic-compatible endpoints, with streaming and tool calling where supported. Deploy and manage models through the Web UI, CLI, or Python SDK. Client guide · Tool calling |
| Performance and operations |
Benchmark running chat models in the Web UI. Supported deployments can share prefix caches with Xavier, change selected runtime settings, and reload replicas one at a time. Performance tests · Runtime configuration |
| Languages |
Use the Web UI in English, Simplified Chinese, Traditional Chinese, Japanese, or Korean. The README is available in ten languages. |
How Xinference compares
| Capability |
Xinference |
LocalAI |
KServe |
NVIDIA Dynamo |
| Standalone (no Kubernetes) |
 |
|
 |
|
| Multi-node inference |
|
|
|
|
| CPU / GPU / Apple Metal |
|
|
 |
|
| llama.cpp / vLLM / SGLang |
|
|
|
|
| CUDA ↔ Metal prefill/decode |
|
|
|
|
| Mixed engines under one model ID |
|
|
|
|
| Rolling reconfiguration with GPU weight reuse |
|
|
|
|
| Model catalog |
300+ |
|
|
|
| Model deployment Web UI |
|
|
|
|
| Embeddings and reranking |
|
|
|
|
| Image and video generation |
|
|
|
|
| Speech recognition and synthesis |
|
|
|
|
Supported · Partial or via integration · No documented support
Scope and sources
Based on documented features in the open-source releases. Engine and hardware support varies by model. CUDA ↔ Metal prefill/decode requires a compatible model and engine pair.
Xinference can serve replicas with different engines concurrently under one model ID. LocalAI uses an ordered failover chain under one model name and sends requests to the first available target. KServe and Dynamo can route across separate deployments. GPU weight reuse applies to supported vLLM and SGLang configurations.
Recent updates
Read the v4.2.0 release notes
Built-in models added in the last two releases (v4.1 and v4.2)
Open Performance Tests, pick a running chat model, and set the concurrency levels you want to test. Compare throughput, time to first token, and time per output token before choosing a serving configuration. Benchmark guide
Example from a local Qwen3.5-0.8B run. Performance depends on the hardware, engine, and workload.
Using Xinference
Apps and frameworks · LangChain · LlamaIndex · FastGPT · RAGFlow · MaxKB · Chatbox
Xinference Enterprise
Deploy Xinference Enterprise in your own cloud or data center. Xinference Cloud offers managed dedicated deployments, and the Model API provides hosted model endpoints. Enterprise overview
Business enquiries · [email protected]
Getting involved
| Platform |
What it’s for |
| GitHub Issues |
Report bugs and request features. |
| Discord |
Ask questions and discuss Xinference with other users. |
| Telegram |
Chat with the Xinference community. |
| X |
Follow project news and releases. |
Citation
If you use Xinference in your research, please cite Xinference: Making Large Model Serving Easy.
@inproceedings{lu2024xinference,
title = "Xinference: Making Large Model Serving Easy",
author = "Lu, Weizheng and Xiong, Lingfeng and Zhang, Feng and Qin, Xuye and Chen, Yueguo",
booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
month = nov,
year = "2024",
address = "Miami, Florida, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.emnlp-demo.30",
pages = "291--300",
}
Contributors
Thank you to everyone who helps build Xinference. Open your first pull request to add a model, improve the docs, or fix a bug. Read the contribution guide.
评论