mlx-vlm

MLX-VLM

Run and fine-tune multimodal models on Apple Silicon. Text, images, video, and audio in — any out. Built on MLX.

Installation · Documentation · Releases

macOS · Apple silicon

Built by Neywa Labs, the local AI company behind Nativ.

Table of Contents

• Installation
• Usage
  • Command Line Interface (CLI)
    • Thinking Budget
  • Speculative Decoding
    • DFlash, DFlash2, and DSpark
    • Gemma 4 MTP
    • Gemma 4 EAGLE-3
    • MiniMax M3 EAGLE-3
  • Chat UI with Gradio
  • Python Script
  • Server (FastAPI)
    • Continuous Batching
    • Automatic Prefix Caching (APC)
    • KV Cache Quantization
• 1-bit Affine Inference
• Activation Quantization (CUDA)
• Multi-Image Chat Support
  • Supported Models
  • Usage Examples
• Model-Specific Documentation
• Vision Feature Caching
• TurboQuant KV Cache
• Distributed Inference
• Fine-tuning

Model-Specific Documentation

Some models have detailed documentation with prompt formats, examples, and best practices:

Model	Documentation
LFM3 / LFM3-VL	Docs
LFM3-Audio	Docs
LFM3-Omni	Docs
DeepSeek-OCR	Docs
DeepSeek-OCR-2	Docs
Unlimited-OCR	Docs
DOTS-OCR	Docs
DOTS-MOCR	Docs
ERNIE 4.5 VL	Docs
GLM-OCR	Docs
Phi-4 Reasoning Vision	Docs
MiniCPM-o	Docs
PaddleOCR-VL	Docs
OXR	Docs
Phi-4 Multimodal	Docs
MolmoPoint	Docs
LocateAnything	Docs
Moondream2	Docs
Moondream3	Docs
EmbeddingGemma 2	Docs
Gemma 4	Docs
Muse Spark Open	Docs
MiniMax M3 / M3.1 Flash	Docs
Falcon-OCR	Docs
IndicOCR	Docs
PP-DocLayoutV3	Docs
Granite Vision 3.2	Docs
Granite Vision (4.0, 4.1)	Docs
MiniCPM-OCR 1.0	Docs
MiniCPM-OCR Jev	Docs
MiniCPM-V 4.7	Docs
MiniCPM-V 4.6	Docs
GLiNER2.5	Docs
BERT	Docs
LLaVA-OneVision	Docs
K2-Horizon	Docs
Z1T-0	Docs
Spark-X2.5	Docs
Ternary Bonsai 2	Docs
Flint	Docs
Nemotron 3.5 Super VL / Bolt / Nano Omni	Docs
Idol (image to 3D Gaussian splats)	Docs

Installation

The easiest way to get started is to install the mlx-vlm package using pip:

pip install -U mlx-vlm

The Gradio chat UI needs an extra dependency that is not part of the base install:

pip install -U 'mlx-vlm[ui]'

Quote the package name so that shells which expand square brackets, such as zsh, do not treat [ui] as a glob pattern.

Agent Skills

This repo ships an agent-skills bundle under skills/ for common MLX-VLM workflows — usage, conversion, development, and support. Skills load into a coding agent (Claude Code, Codex, Gemini) so it follows the right project conventions instead of guessing.

Skill	Description
cli-inference	Run and debug command-line inference (mlx_vlm.generate) — text/image/audio inputs and image-generation flags.
server-inference	Run and debug the local server across the models, chat, responses, messages, audio, image, cache, and metrics endpoints.
convert-quantize	Convert and quantize Hugging Face models to MLX (mlx_vlm.convert) — bits/group size, quant modes, RTN/AWQ, mixed recipes.
add-new-model	Port a new architecture into mlx_vlm/models — config, weight-name mapping, reuse a similar model, add a test class.
benchmarking	Produce credible, reproducible perf numbers and fork-vs-main A/B tables for PRs.
contributing	Shape a change to pass review — code/config/test placement, pre-commit hooks, and PR expectations.
hf-cache-models	List cached, custom, and loaded models through the server’s model discovery endpoint.
reproducible-github-issues	Turn CLI or server failures into concise, reproducible GitHub issues.

Validate the bundle at any time:

python3 skills/scripts/validate_skills.py

Install from a local checkout:

# Claude Code
/plugin marketplace add /path/to/mlx-vlm
/plugin install mlx-vlm-skills@mlx-vlm

# Codex CLI
codex plugin marketplace add /path/to/mlx-vlm
codex plugin add mlx-vlm-skills@mlx-vlm

# Gemini CLI
gemini extensions install /path/to/mlx-vlm/skills

Usage

Command Line Interface (CLI)

Generate output from a model using the CLI:

# Text generation
mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --prompt "Hello, how are you?"

# Image generation
mlx_vlm.generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --temperature 0.0 --image http://images.cocodataset.org/val2017/000000039769.jpg

# Audio understanding
mlx_vlm.generate --model mlx-community/gemma-3n-E2B-it-4bit --max-tokens 100 --prompt "Describe what you hear" --audio /path/to/audio.wav

# Multi-modal understanding (Image + Audio)
mlx_vlm.generate --model mlx-community/gemma-3n-E2B-it-4bit --max-tokens 100 --prompt "Describe what you see and hear" --image /path/to/image.jpg --audio /path/to/audio.wav

# Speech generation
mlx_vlm.generate --model openbmb/MiniCPM-o-4_5 --output-modality audio --prompt "Say hello." --ref-audio /path/to/voice.wav --output speech.wav --max-tokens 256

# Image to 3D Gaussian splats
mlx_vlm.generate --model speridlabs/idol --output-modality 3d --image object.png --output object.ply

Thinking Budget

For thinking models (e.g., Qwen3.5), you can limit the number of tokens spent in the thinking block:

mlx_vlm.generate --model mlx-community/Qwen3.5-2B-4bit \
  --thinking-budget 50 \
  --thinking-start-token "<think>" \
  --thinking-end-token "</think>" \
  --enable-thinking \
  --prompt "Solve 2+2"

Flag	Description
--enable-thinking	Activate thinking mode in the chat template
--thinking-budget	Max tokens allowed inside the thinking block
--thinking-start-token	Token that opens a thinking block (default: <think>)
--thinking-end-token	Token that closes a thinking block (default: </think>)

When the budget is exceeded, the model is forced to emit \n</think> and transition to the answer. If --enable-thinking is passed but the model’s chat template does not support it, the budget is applied only if the model generates the start token on its own.

On the server, thinking mode is disabled by default. Start the server with --enable-thinking to make thinking mode the default for requests that do not specify it:

mlx_vlm.server --model Qwen/Qwen3.5-4B --enable-thinking

You can also set server defaults for the thinking budget and delimiter tokens:

mlx_vlm.server --model Qwen/Qwen3.5-4B \
  --enable-thinking \
  --thinking-budget 512 \
  --thinking-start-token "<think>" \
  --thinking-end-token "</think>"

Requests can override the server defaults with enable_thinking, thinking_budget, thinking_start_token, or thinking_end_token. chat_template_kwargs.enable_thinking is also accepted; an explicit top-level enable_thinking takes precedence.

Speculative Decoding

Speed up generation by drafting several candidate tokens with a small “drafter” model and verifying them in a single target forward pass. Three drafter families are supported.

Flag	Description
--draft-model	HuggingFace repo or local path for the drafter
--draft-kind	Drafter family — dflash (default), eagle3, or mtp (native/assistant MTP)
--draft-block-size	Override the drafter’s configured block size

See docs/usage.md for Python API examples including batch generation.

DFlash, DFlash2, and DSpark

A lightweight block-diffusion drafter that predicts multiple tokens per round, typically 2–3× faster.

# Text generation with speculative decoding
mlx_vlm.generate --model Qwen/Qwen3.5-4B \
  --draft-model z-lab/Qwen3.5-4B-DFlash \
  --prompt "Write a quicksort in Python." \
  --max-tokens 512 --temperature 0 --enable-thinking

# Also works with images
mlx_vlm.generate --model Qwen/Qwen3.5-4B \
  --draft-model z-lab/Qwen3.5-4B-DFlash \
  --image examples/images/cats.jpg \
  --prompt "Describe this image." \
  --max-tokens 256 --temperature 0 --enable-thinking

# Server with speculative decoding
mlx_vlm.server --model Qwen/Qwen3.5-4B \
  --draft-model z-lab/Qwen3.5-4B-DFlash

DFlash2 adds dynamic convolutions and a candidate-path selector. The published Qwen3.8-27B checkpoint is auto-detected and uses the shared exact DFlash target verification path. For the fastest quantized setup, convert the drafter to 4-bit; the verifier adapts between three and five rows from recent acceptance:

mlx_vlm.convert --hf-path z-lab/Qwen3.8-27B-DFlash2 \
  --mlx-path Qwen3.8-27B-DFlash2-4bit \
  --quantize --q-bits 4 --q-group-size 64

mlx_vlm.generate --model mlx-community/Qwen3.8-27B-4bit \
  --draft-model Qwen3.8-27B-DFlash2-4bit \
  --prompt "Write a quicksort in Python." \
  --max-tokens 512 --temperature 0

mlx_vlm.server --model mlx-community/Qwen3.8-27B-4bit \
  --draft-model Qwen3.8-27B-DFlash2-4bit

Liquid AI’s DSpark checkpoint uses a Qwen3-style block drafter plus a learned Markov correction head. It is auto-detected and runs through the exact target verification path:

mlx_vlm.generate --model LiquidAI/LFM2.5-2.6B \
  --draft-model LiquidAI/LFM2.5-2.6B-DSpark \
  --prompt "Explain speculative decoding in three sentences." \
  --max-tokens 256 --temperature 0

mlx_vlm.server --model LiquidAI/LFM2.5-2.6B \
  --draft-model LiquidAI/LFM2.5-2.6B-DSpark

The published DSpark block_size: 9 means nine proposals, or ten target rows after adding the anchor token. On MLX, DSpark verifies seven proposals plus the anchor by default: eight rows exactly fill the verifier threadgroup, while nine or ten rows pad to sixteen and run slower. The trained width remains available with --draft-block-size 10. The checkpoint’s confidence head is loaded for parity, and DSpark decoding currently requires greedy sampling (temperature=0).

Muse Glimmer’s published assistant checkpoint is auto-detected as DFlash:

mlx_vlm.generate --model meta-models/Muse-Glimmer-30B \
  --draft-model meta-models/Muse-Glimmer-30B-assistant \
  --prompt "Write a quicksort in Python." \
  --max-tokens 512 --temperature 0

mlx_vlm.server --model meta-models/Muse-Glimmer-30B \
  --draft-model meta-models/Muse-Glimmer-30B-assistant

DFlash draft-cache windowing is available from the Python API. During speculative decoding the target model still verifies every proposed token with its full KV cache; this knob only changes the DFlash drafter cache. When draft_window_size is set, the drafter keeps at most that many recent committed tokens in its own KV cache instead of attending over the full generated prefix. That reduces draft-side cache length and memory, but it can lower acceptance because the drafter has less context than the target verifier. On MLX, the full draft cache is usually faster for Qwen3.5 DFlash, so windowing defaults to None; set it only when you want to experiment with this compact recent-token cache tradeoff:

from mlx_vlm import load
from mlx_vlm.generate import generate
from mlx_vlm.speculative.drafters import load_drafter

model, processor = load("Qwen/Qwen3.5-4B")
draft_model, draft_kind = load_drafter("z-lab/Qwen3.5-4B-DFlash")
draft_model.config.draft_window_size = 256  # None disables windowing

result = generate(
    model,
    processor,
    "Write a quicksort in Python.",
    max_tokens=512,
    temperature=0,
    draft_model=draft_model,
    draft_kind=draft_kind,
)

Gemma 4 MTP

Multi-Token Prediction: Google’s 4-layer “assistant” drafter that shares K/V with the target and drafts multiple tokens autoregressively from a constant position. Pass --draft-kind mtp to dispatch the MTP round-loop.

mlx_vlm.generate --model mlx-community/gemma-4-31B-it-bf16 \
  --draft-model mlx-community/gemma-4-31B-it-assistant-bf16 \
  --draft-kind mtp --draft-block-size 4 \
  --prompt "Explain speculative decoding in 3 sentences." \
  --max-tokens 256 --temperature 0

# Server
mlx_vlm.server --model mlx-community/gemma-4-31B-it-bf16 \
  --draft-model mlx-community/gemma-4-31B-it-assistant-bf16 \
  --draft-kind mtp --draft-block-size 4

Supported pairings (target ↔ drafter):

Target	Drafter
mlx-community/gemma-4-E2B-it-bf16	mlx-community/gemma-4-E2B-it-assistant-bf16
mlx-community/gemma-4-E4B-it-bf16	mlx-community/gemma-4-E4B-it-assistant-bf16
mlx-community/gemma-4-26B-A4B-it-bf16	mlx-community/gemma-4-26B-A4B-it-assistant-bf16
mlx-community/gemma-4-31B-it-bf16	mlx-community/gemma-4-31B-it-assistant-bf16

Measured speedups (greedy, byte-identical output): up to 3.94× on 26B-A4B and 2.29× on 31B at B=4. See mlx_vlm/speculative/drafters/gemma4_assistant/README.md for full sweeps and architecture notes.

Gemma 4 EAGLE-3

EAGLE-3 drafts from three target hidden-state captures with a lightweight one-layer speculator. The Red Hat Speculators checkpoint auto-detects as --draft-kind eagle3.

mlx_vlm.generate --model mlx-community/gemma-4-31B-it-bf16 \
  --draft-model RedHatAI/gemma-4-31B-it-speculator.eagle3 \
  --prompt "Explain speculative decoding in 3 sentences." \
  --max-tokens 256 --temperature 0

# Server
mlx_vlm.server --model mlx-community/gemma-4-31B-it-bf16 \
  --draft-model RedHatAI/gemma-4-31B-it-speculator.eagle3

MiniMax M3 EAGLE-3

MiniMax M3 supports the released Inferact/MiniMax-M3-EAGLE3 drafter. Convert the target with mlx_vlm.convert because mlx_lm.convert does not know the minimax_m3_vl model type.

mlx_vlm.convert \
  --hf-path MiniMaxAI/MiniMax-M3 \
  --mlx-path ~/MiniMax-M3-4bit \
  --quantize --q-bits 4 \
  --trust-remote-code

mlx_vlm.convert \
  --hf-path Inferact/MiniMax-M3-EAGLE3 \
  --mlx-path ~/MiniMax-M3-EAGLE3

mlx_vlm.generate \
  --model ~/MiniMax-M3-4bit \
  --draft-model ~/MiniMax-M3-EAGLE3 \
  --draft-kind eagle3 \
  --draft-block-size 3 \
  --prompt "Explain MiniMax Sparse Attention in one paragraph." \
  --max-tokens 256 --temperature 0

The public MiniMax M3 BF16 checkpoint advertises MTP metadata but does not publish mtp or nextn tensors, so use the released EAGLE-3 drafter for that checkpoint.

MiniMax M3 also supports image/video prompts, MiniMax thinking tags, MiniMax tool-call parsing, MSA index caches, and MXFP8 config loading. See mlx_vlm/models/minimax_m3_vl/README.md for model-specific conversion and runtime notes.

Reading the server’s draft counters

With a drafter loaded, mlx_vlm.server adds four fields to a response’s timings, named after llama.cpp’s:

Field	Meaning
draft_kind	The drafter family that ran
draft_rounds	Verification rounds: target forward passes after the prefill
draft_n	Tokens proposed, --draft-block-size minus one per round (the block includes the anchor token)
draft_n_accepted	Proposed tokens the target’s greedy choice matched

draft_n_accepted is a measure of the drafter, not of the output. A round is recorded when the target verifies it, before the stop token or max_tokens cuts the reply, so matches past the end of the reply are counted and the field can exceed predicted_n. predicted_n - draft_n_accepted is therefore not the number of target forward passes. Use draft_rounds:

accept_length = (predicted_n - 1) / draft_rounds

The first token comes from the prefill, before any draft exists, so it is left out of the numerator. The three counters are differences of a tally kept on the drafter, so they belong to one request only when one request is in flight; with concurrent requests they are shared across the batch.

Chat UI with Gradio

The Gradio chat UI requires the optional ui extra, which the base mlx-vlm install does not include:

pip install -U 'mlx-vlm[ui]'

Then launch the chat interface:

mlx_vlm.chat_ui --model mlx-community/Qwen2-VL-2B-Instruct-4bit

Python Script

Here’s an example of how to use MLX-VLM in a Python script:

import mlx.core as mx
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

# Load the model
model_path = "mlx-community/Qwen2-VL-2B-Instruct-4bit"
model, processor = load(model_path)
config = load_config(model_path)

# Prepare input
image = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
# image = [Image.open("...")] can also be used with PIL.Image.Image objects
prompt = "Describe this image."

# Apply chat template
formatted_prompt = apply_chat_template(
    processor, config, prompt, num_images=len(image)
)

# Generate output
output = generate(model, processor, formatted_prompt, image, verbose=False)
print(output)

Audio Example

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

# Load model with audio support
model_path = "mlx-community/gemma-3n-E2B-it-4bit"
model, processor = load(model_path)
config = model.config

# Prepare audio input
audio = ["/path/to/audio1.wav", "/path/to/audio2.mp3"]
prompt = "Describe what you hear in these audio files."

# Apply chat template with audio
formatted_prompt = apply_chat_template(
    processor, config, prompt, num_audios=len(audio)
)

# Generate output with audio
output = generate(model, processor, formatted_prompt, audio=audio, verbose=False)
print(output)

Multi-Modal Example (Image + Audio)

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

# Load multi-modal model
model_path = "mlx-community/gemma-3n-E2B-it-4bit"
model, processor = load(model_path)
config = model.config

# Prepare inputs
image = ["/path/to/image.jpg"]
audio = ["/path/to/audio.wav"]
prompt = ""

# Apply chat template
formatted_prompt = apply_chat_template(
    processor, config, prompt,
    num_images=len(image),
    num_audios=len(audio)
)

# Generate output
output = generate(model, processor, formatted_prompt, image, audio=audio, verbose=False)
print(output)

Decision models

Run shared typed decision prediction from the CLI:

python -m mlx_vlm decide --model nativ-community/decider-2b \
  --state "Please refund my duplicate charge" \
  --questions '{"department":{"type":"choice","instructions":"Which team should handle this ticket?","criteria":["billing","technical","sales"]}}'

Use --state-file request.txt for UTF-8 text and --questions-file questions.json for named questions. mlx_vlm.decide runs the same command.

Models that read images take a repeatable --image, and models that read speech take one --audio (each a path or URL). The state is optional when media are given:

python -m mlx_vlm decide --model LiquidAI/d1-omni-600M \
  --image http://images.cocodataset.org/val2017/000000039769.jpg \
  --questions '{"cats":{"type":"choice","instructions":"How many cats are there?","criteria":["one","two","more"]}}'
python -m mlx_vlm decide --model LiquidAI/d1-omni-600M \
  --audio https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/1.flac \
  --questions '{"speaking":{"type":"bool","instructions":"Is anyone speaking in this clip?"}}'

Preload a decision model in the existing server:

python -m mlx_vlm server --decision-model nativ-community/decider-2b
curl http://localhost:8080/v1/decisions \
  -H 'Content-Type: application/json' \
  -d '{"state":"Please refund my duplicate charge","questions":{"department":{"type":"choice","instructions":"Which team should handle this ticket?","criteria":["billing","technical","sales"]}}}'

An explicit model in the request selects a checkpoint; otherwise the currently loaded decision model is used. Without either, the endpoint returns HTTP 400. Requests may add "images", a list of URLs, paths, or data URIs, or "audio", one URL, path, or base64 data URI; state may then be null. Decision requests are non-streaming and use the server’s API-key authentication.

Both interfaces return the shared predict() result and preserve model-specific scoring. Decider, Laya, d1-3B, and d1-omni-600M support choice, bool, and score; the two LiquidAI d1 models also read images, and d1-omni reads speech. Consult each model’s README for criteria and calibration details.

Server (FastAPI)

Start the server:

mlx_vlm.server --port 8080

# Preload a model at startup (Hugging Face repo or local path)
mlx_vlm.server --model <hf_repo_or_local_path>

# Preload separate model kinds at startup
mlx_vlm.server --model <language_model> \
  --image-model <image_generation_model> \
  --tts-model <text_to_speech_model> \
  --stt-model <speech_to_text_model>

# Preload a model with adapter
mlx_vlm.server --model <hf_repo_or_local_path> --adapter-path <adapter_path>

# With trust remote code enabled (required for some models)
mlx_vlm.server --trust-remote-code

# Enable thinking mode by default for requests that do not override it
mlx_vlm.server --model Qwen/Qwen3.5-4B --enable-thinking

# Configure thinking defaults at startup
mlx_vlm.server --model Qwen/Qwen3.5-4B \
  --enable-thinking \
  --thinking-budget 512 \
  --thinking-start-token "<think>" \
  --thinking-end-token "</think>"

# Require bearer authentication for API endpoints
mlx_vlm.server --api-key <secret-token>

# Include local models outside the Hugging Face cache
mlx_vlm.server --model-dir /Volumes/Models --model-dir ~/my-custom-model

Server Options

• --model: Preload a language model at server startup, accepts a Hugging Face repo ID or local path (optional, loads lazily on first request if omitted)
• --image-model: Preload an image generation model at server startup
• --mesh-model: Preload an image-to-3D model for asynchronous PLY generation
• --tts-model: Preload a text-to-speech model at server startup
• --stt-model: Preload a speech-to-text model at server startup
• --embedding-model: Preload an embedding model at server startup
• --decision-model: Preload a decision model at server startup
• --reranker-model: Preload a supported reranker model at server startup
• --model-dir: Additional model folder or parent containing model folders to include in discovery; repeat for multiple paths. Overrides MLX_VLM_MODEL_PATHS (paths separated by os.pathsep, : on macOS/Linux)
• --adapter-path: Path for adapter weights to use with the preloaded model
• --draft-model: Speculative drafter path or HF id (e.g. z-lab/Qwen3.8-27B-DFlash2, z-lab/Qwen3.5-4B-DFlash, RedHatAI/gemma-4-31B-it-speculator.eagle3, google/gemma-4-31B-it-assistant, Inferact/MiniMax-M3-EAGLE3) — enables speculative decoding for ~2× or higher throughput
• --draft-kind: Drafter family — dflash (default), eagle3, or mtp (native/assistant MTP)
• --draft-block-size: Override the drafter’s configured block size
• --host: Host address (default: 0.0.0.0)
• --port: Port number (default: 8080)
• --trust-remote-code: Trust remote code when loading models from Hugging Face Hub
• --enable-thinking: Enable thinking mode by default for requests that do not set enable_thinking
• --thinking-budget: Default maximum number of tokens allowed inside a thinking block
• --thinking-start-token: Default token that opens a thinking block
• --thinking-end-token: Default token that closes a thinking block (--thinking-eos-token is also accepted)
• --kv-bits: Number of bits for KV cache quantization (e.g. 8 for uniform, 3.5 for TurboQuant)
• --kv-quant-scheme: KV cache quantization backend (uniform or turboquant)
• --kv-key-bits / --kv-value-bits: Override the bit-width for keys or values individually (see Per-tensor KV quantization)
• --kv-key-scheme / --kv-value-scheme: Override the quantization backend for keys or values individually
• --kv-group-size: Group size for uniform KV cache quantization (default: 64)
• --max-kv-size: Maximum KV cache size in tokens
• --vision-cache-size: Max number of cached vision features (default: 20)
• --min-pixels / --max-pixels: Default minimum/maximum input image pixel area for processors that support these options (also MLX_VLM_MIN_PIXELS / MLX_VLM_MAX_PIXELS). Unset values use the model’s preprocessing defaults.
• --log-progress-interval: Decoded tokens between progress log messages; 0 disables periodic decode progress (default: 10)
• --api-key: Bearer token required for inference, model discovery, and management endpoints
• --log-level: Logging level — DEBUG, INFO, WARNING, ERROR, CRITICAL (default: INFO)

Streaming /v1/chat/completions and /v1/responses send invisible SSE comments every 15 seconds while idle, keeping connections alive during queueing and long prefills. Set MLX_VLM_STREAM_KEEPALIVE_INTERVAL to a different interval in seconds or 0 to disable it. This does not change model output or generation timeouts.

For higher-resolution image input, including OCR with Bonsai 2, set pixel-area defaults at startup:

mlx_vlm.server --model prism-ml/Ternary-Bonsai-2-2B-mlx-2bit \
  --min-pixels 1048576 --max-pixels 4194304

Chat Completions, Responses, and Messages requests can override these defaults with top-level min_pixels and max_pixels fields (use extra_body with an OpenAI client). For example, "min_pixels": 1048576 requests at least 1024 × 1024 pixels of area, preserving aspect ratio and rounding to the processor’s patch size. On Bonsai 2 this corresponds to approximately 1024 image tokens. Higher resolution uses more context, memory, and processing time. Both limits must be positive and the effective minimum must not exceed the maximum. Omitted or null request values inherit the server defaults.

Change these defaults on a running server without restarting or reloading the model:

curl -X PATCH http://localhost:8080/v1/settings \
  -H 'Content-Type: application/json' \
  -d '{"min_pixels": 1048576, "max_pixels": 4194304}'

Use GET /v1/settings to inspect the current values. New requests inherit the updated limits; requests already queued for generation keep their resolved options, and per-request overrides still take precedence. The pair is updated atomically; invalid limits appear in rejected and leave both limits unchanged. Set either setting to null to use that limit from the model’s preprocessing configuration. Updates last until the server restarts; the {"op": "replace", "values": {}} form resets all runtime settings to their startup defaults, including CLI pixel limits. Include Authorization: Bearer <key> when the server has an API key configured.

The existing resize_shape request field also applies before image processing in the batching server. Processor pixel limits still apply afterward. Token counting and streaming context checks use the same options as generation.

At INFO, the server logs request start/completion, chunked-prefill progress, time to first token, periodic decode throughput, and the final token counts. Set --log-level DEBUG to emit decode progress for every token and add its token number, token ID, and decoded text to the same log entry. Decode progress uses rate for the instantaneous inter-token rate; decode completion uses the same field name for aggregate decode throughput measured across completed token intervals.

OpenAI-compatible streaming responses expose throughput under timings.predicted_per_second. Token-bearing SSE chunks report the instantaneous inter-token rate, while terminal and usage chunks report the aggregate rate as (tokens - 1) / (last_token_time - first_token_time). The first token reports null because it has no preceding token interval.

You can also set trust remote code via environment variable:

MLX_TRUST_REMOTE_CODE=true mlx_vlm.server

The server provides multiple endpoints for different use cases and supports dynamic model loading/unloading with caching (one model at a time).

Continuous Batching

The server supports continuous batching for higher throughput when handling multiple concurrent requests. New requests join the active batch immediately without waiting for existing requests to finish, and mixed batches of image and text-only requests are supported.

Prompt processing and decoding are interleaved, with decoding taking priority. The server’s --prefill-step-size (default: 2048) is a shared prompt-token budget per step, including padding, so increasing the batch size does not multiply the prompt work between decode steps. While peers decode, the scheduler also caps prompt work at 256 tokens per step. Lower the configured budget to reduce pauses further. Models that require unchunked prefill do not use this budget.

Ready decode tokens are published before the next prefill step. Pending prompts are grouped by comparable length within the oldest candidate window to limit padding without starving long prompts. APC retains its cached-suffix grouping and memory admission checks. Python streaming integrations can use BatchGenerator.decode_step() and prefill_step() separately; next() continues to advance both phases for existing callers.

Continuous batching is enabled automatically when the server loads a model. You can pre-load a model at startup so it’s ready to serve immediately:

mlx_vlm.server --port 8080 --model mlx-community/Qwen2.5-VL-3B-Instruct-4bit

Verify via the health endpoint:

curl http://localhost:8080/health
# {"status":"healthy","loaded_model":"...","apc_enabled":false}

If --model is omitted, the model is loaded on the first request.

Automatic Prefix Caching (APC)

Automatic Prefix Caching reuses model cache state across requests that share the same prefix. It is useful for repeated long documents, long chat histories, or retrieval contexts where each request appends a short new suffix.

APC builds a cache plan from model.make_cache() in the same style as vLLM’s hybrid cache manager: each layer gets a cache spec, compatible specs form cache groups, and one coordinator selects a common reusable prefix across the groups. Dense attention models use pageable K/V blocks. Hybrid full/sliding-attention, recurrent/SSM, MLA/composite, VLM, and Omni layouts use restorable state checkpoints for the components that cannot be concatenated safely. Generation code uses the same coordinator API for both paths.

APC has two tiers:

• Warm memory: keeps reusable APCBlock tensors in process memory. This is the fastest path, but it keeps both the reusable block pool and the runtime KVCache.
• Warm disk: persists cached prefixes as safetensors shards so they survive process restarts. Warm-disk reads build the layer-major prompt cache directly without promoting restored blocks into the APCBlock pool; writes can still populate both memory and disk tiers.

Python Script

Use APCManager directly when calling stream_generate:

from pathlib import Path

from mlx_vlm import load, stream_generate
from mlx_vlm.apc import APCManager, DiskBlockStore
from mlx_vlm.prompt_utils import apply_chat_template

model_id = "Qwen/Qwen3-VL-4B-Instruct"
model, processor = load(model_id)

disk = DiskBlockStore(
    Path("~/.cache/mlx-vlm/caching").expanduser(),
    namespace=model_id,
    max_bytes=3 * (1 << 30),  # 3 GB disk cap; use None for uncapped
)
apc = APCManager(num_blocks=4096, block_size=16, disk=disk)

# Optional diagnostics: inspect the automatically inferred cache groups.
print(apc.coordinator(model).plan.describe())

document = Path("long_document.txt").read_text()

try:
    # First request computes the full prefix and stores reusable K/V blocks.
    prompt1 = apply_chat_template(
        processor,
        model.config,
        prompt=f"{document}\n\nSummarize the key decisions.",
        num_images=0,
    )
    for _ in stream_generate(
        model, processor, prompt1, max_tokens=128, temperature=0.0, apc_manager=apc
    ):
        pass

    # Second request shares the same document prefix and only prefills the suffix.
    prompt2 = apply_chat_template(
        processor,
        model.config,
        prompt=f"{document}\n\nList the open engineering risks.",
        num_images=0,
    )
    for chunk in stream_generate(
        model, processor, prompt2, max_tokens=128, temperature=0.0, apc_manager=apc
    ):
        print(chunk.text, end="", flush=True)

    print(apc.stats_snapshot())
finally:
    apc.close()

To compare cold, warm-memory, warm-disk, and disk-eviction behavior with a model, use the same direct API path:

import os
import tempfile
import time
from pathlib import Path

from mlx_vlm import load, stream_generate
from mlx_vlm.apc import APCManager, DiskBlockStore
from mlx_vlm.prompt_utils import apply_chat_template

model_id = "Qwen/Qwen3-VL-4B-Instruct"
contexts = [8000, 20000, 50000, 100000]
disk_cap_gb = 0  # 0 means uncapped
shard_max_blocks = 256
context_sweep_max_tokens = 1  # one token is enough to measure prefill reuse

test_prompt_tokens = 8000
fill_prompts = 80
eviction_disk_cap_gb = 3.0

os.environ["APC_DISK_SHARD_MAX_BLOCKS"] = str(shard_max_blocks)

model, processor = load(model_id)
tokenizer = processor.tokenizer if hasattr(processor, "tokenizer") else processor


def disk_cap_bytes(gb: float):
    return None if gb <= 0 else int(gb * (1 << 30))


def make_context(target_tokens: int, seed: int = 0) -> str:
    line = (
        f"Document {seed}: APC benchmark content with deterministic facts, "
        "dates, identifiers, and repeated technical notes.\n"
    )
    line_tokens = max(1, len(tokenizer.encode(line, add_special_tokens=False)))
    text = line * max(1, target_tokens // line_tokens)
    while len(tokenizer.encode(text, add_special_tokens=False)) < target_tokens:
        text += line
    return text


def make_prompt(context: str, question: str) -> str:
    return apply_chat_template(
        processor,
        model.config,
        prompt=f"{context}\n\n{question}",
        num_images=0,
    )


def run_once(apc: APCManager, context: str, question: str, max_tokens: int = 32):
    prompt = make_prompt(context, question)
    apc.reset_stats()

    last = None
    output = []
    start = time.perf_counter()
    for chunk in stream_generate(
        model,
        processor,
        prompt,
        max_tokens=max_tokens,
        temperature=0.0,
        apc_manager=apc,
    ):
        output.append(chunk.text)
        last = chunk

    if last is None:
        raise RuntimeError("generation returned no chunks")

    return {
        "wall_s": time.perf_counter() - start,
        "prompt_tokens": last.prompt_tokens,
        "prompt_tps": last.prompt_tps,
        "generation_tps": last.generation_tps,
        "apc": apc.stats_snapshot(),
        "text": "".join(output).strip(),
    }


def print_result(label: str, result: dict) -> None:
    stats = result["apc"]
    print(
        f"{label:<12} "
        f"prompt_tokens={result['prompt_tokens']:>7} "
        f"prompt_tps={result['prompt_tps']:>8.1f} "
        f"gen_tps={result['generation_tps']:>7.1f} "
        f"matched={stats.get('matched_tokens', 0):>7} "
        f"disk_hits={stats.get('disk_hits', 0):>5} "
        f"disk_evictions={stats.get('disk_evictions', 0):>5}"
    )


def open_apc(cache_root: Path, namespace: str, disk_gb: float) -> APCManager:
    disk = DiskBlockStore(
        cache_root,
        namespace=namespace,
        max_bytes=disk_cap_bytes(disk_gb),
    )
    return APCManager(num_blocks=4096, block_size=16, disk=disk)


def run_context_sweep() -> None:
    print("cold / warm-memory / warm-disk")
    with tempfile.TemporaryDirectory() as tmp:
        cache_root = Path(tmp)
        for target_tokens in contexts:
            context = make_context(target_tokens)
            namespace = f"{model_id}-context-{target_tokens}"
            apc = open_apc(cache_root, namespace, disk_cap_gb)
            try:
                print(f"\ncontext ~= {target_tokens} text tokens")
                print_result(
                    "cold",
                    run_once(
                        apc,
                        context,
                        "Summarize the key decisions.",
                        max_tokens=context_sweep_max_tokens,
                    ),
                )
                print_result(
                    "warm-memory",
                    run_once(
                        apc,
                        context,
                        "List the open engineering risks.",
                        max_tokens=context_sweep_max_tokens,
                    ),
                )
            finally:
                # Closing waits for queued disk writes before reopening the disk tier.
                apc.close()

            apc = open_apc(cache_root, namespace, disk_cap_gb)
            try:
                print_result(
                    "warm-disk",
                    run_once(
                        apc,
                        context,
                        "Extract the implementation timeline.",
                        max_tokens=context_sweep_max_tokens,
                    ),
                )
            finally:
                apc.close()


def run_disk_eviction_workload() -> None:
    print("\ndisk eviction workload")
    with tempfile.TemporaryDirectory() as tmp:
        cache_root = Path(tmp)
        namespace = f"{model_id}-eviction"
        test_context = make_context(test_prompt_tokens, seed=0)

        apc = open_apc(cache_root, namespace, eviction_disk_cap_gb)
        try:
            print_resul

提示：代码过长已自动截断

Server

Enable in-memory APC for the server with environment variables:

APC_ENABLED=1 \
APC_NUM_BLOCKS=4096 \
mlx_vlm.server --model Qwen/Qwen3-VL-4B-Instruct --port 8080

APC works with KV-cache quantization (--kv-bits):

APC_ENABLED=1 \
APC_NUM_BLOCKS=4096 \
mlx_vlm.server --model Qwen/Qwen3-VL-4B-Instruct --kv-bits 8 --port 8080

APC persists caches to disk by default when enabled, under $MLX_VLM_CACHE_HOME/apc (or ~/.cache/mlx-vlm/apc), with a 20 GiB cap per model namespace. Customize the location and cap:

APC_ENABLED=1 \
APC_NUM_BLOCKS=4096 \
APC_DISK_PATH=~/.cache/mlx-vlm/caching \
APC_DISK_MAX_GB=3 \
APC_DISK_SHARD_MAX_BLOCKS=256 \
mlx_vlm.server --model Qwen/Qwen3-VL-4B-Instruct --port 8080

Repeated requests with the same long prefix will hit APC automatically:

curl -X POST "http://localhost:8080/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "X-APC-Tenant: demo" \
  -d '{
    "model": "Qwen/Qwen3-VL-4B-Instruct",
    "messages": [{
      "role": "user",
      "content": "Paste a long shared document here.\n\nNow answer question A."
    }],
    "max_tokens": 128
  }'

Use the same X-APC-Tenant value for requests that may share cached prefixes. Use different tenant values to isolate cache entries between users or workspaces.

Inspect and reset APC state:

curl http://localhost:8080/v1/cache/stats
curl -X POST http://localhost:8080/v1/cache/reset

restored_tokens counts prefix tokens successfully materialized for generation across block and exact caches, in memory or on disk. A lookup alone does not increase it, nor does a failed batch restore. A successful restore remains counted if the request later fails or is cancelled. The previous served_tokens block-storage count is now stored_tokens; token_hit_rate retains its existing calculation, matched_tokens / (matched_tokens + stored_tokens).

Configure APC on a running server with PATCH /v1/settings. Use GET /v1/settings to discover supported settings and read their current values:

curl http://localhost:8080/v1/settings

curl -X PATCH http://localhost:8080/v1/settings \
  -H 'Content-Type: application/json' \
  -d '{
    "apc_enabled": true,
    "apc_disk_enabled": true,
    "apc_memory_max_gb": 2,
    "apc_disk_max_gb": 20,
    "apc_checkpoint_interval_tokens": 1024
  }'

The endpoint also accepts apc_memory_reserve_gb, apc_disk_queue_max_gb, apc_disk_path, apc_disk_shard_max_blocks, apc_block_size, apc_num_blocks, apc_checkpoint_entries, and apc_checkpoint_guard_tokens. When MLX_VLM_SERVER_API_KEY is configured, include Authorization: Bearer <key>.

Settings apply on the next text-generation request through the existing model reload path; the server process stays running. Reloading clears resident caches and retires the previous disk writer after its generation worker finishes. Existing disk files remain available. Updating APC options while APC is disabled stages them for the next enable. The response reports applied, rejected, reload_kinds, and the resulting settings; invalid sizes are rejected without changing those values. Unchanged settings do not trigger a reload.

Use null for automatic memory budgets or the default disk path/cap. Zero has specific meanings: apc_memory_max_gb: 0 retains caches only on disk, apc_disk_max_gb: 0 removes the disk cap, and apc_disk_queue_max_gb: 0 writes synchronously. An empty apc_disk_path or apc_disk_enabled: false disables persistence. PATCH merges with current settings; the optional {"op": "replace", "values": {...}} form first restores startup environment defaults. /v1/cache/stats shows the active manager after the next request.

Common APC environment variables:

Variable	Default	Description
APC_ENABLED	0	Set to 1 to enable APC
APC_NUM_BLOCKS	2048	Number of in-memory APC blocks
APC_BLOCK_SIZE	16	Tokens per APC block
APC_CHECKPOINT_ENTRIES	2	In-memory checkpoint entries; also bounds snapshots captured per hybrid prompt (disk-only mode captures two)
APC_CHECKPOINT_GUARD_TOKENS	1	Tokens retained after a reusable hybrid checkpoint boundary; the default preserves the normal final-token prefill boundary
APC_CHECKPOINT_INTERVAL_TOKENS	2048	Spacing of intermediate hybrid checkpoints, rounded up to a multiple of APC_BLOCK_SIZE; observed prompt divergence takes priority within the entry budget; 0 keeps only the final checkpoint
APC_MEMORY_MAX_GB	auto	Resident block and checkpoint budget in GiB: 10% of Metal’s recommended working set, capped at 8 GiB; 0 retains caches only on disk
APC_MEMORY_RESERVE_GB	auto	Additional memory headroom in GiB: 10% of Metal’s recommended working set, at least 1 GiB
APC_DISK_ENABLED	1	Set to 0 to disable disk persistence
MLX_VLM_CACHE_HOME	~/.cache/mlx-vlm	Base cache directory; APC uses its apc subdirectory unless APC_DISK_PATH is set
APC_DISK_PATH	cache directory above	Directory for persistent disk shards; an empty value disables persistence
APC_DISK_MAX_GB	20	Disk cap per model namespace in GiB; 0 means uncapped
APC_DISK_QUEUE_MAX_GB	1	Maximum tensor bytes held by queued disk writes in GiB; larger writes run synchronously; 0 makes all writes synchronous
APC_DISK_SHARD_MAX_BLOCKS	256	Max blocks per disk segment shard
APC_MAX_POOL_TENSORS	450000	Stops adding memory blocks before the Metal resource limit; disk writes continue
APC_LAYER_MAJOR_MEMORY_MIN_TOKENS	50000	Store long warm-memory prefixes as compact layer-major snapshots instead of per-block tensors
APC_HASH	fast	Set to sha256 for a stable cryptographic hash
APC_TRACE	unset	Set to 1 for greppable store/reject/self-check log lines

Custom cache layouts can opt in without APC model-name checks by implementing prefix_cache_snapshot() and prefix_cache_restore(snapshot). In-tree dense, sliding-window, recurrent, composite, VLM, and Omni cache layouts are detected automatically. APC works with --kv-bits (including TurboQuant): the live KV cache stays quantized; pageable APC K/V blocks are stored as dequantized float K/V, so block-pool size does not shrink with quant. When APC is enabled on the server, a non-fatal layout self-check runs at model load.

Requests with a shared document and different questions can reuse their common prefix. Dense K/V caches, including compact layer-major memory snapshots, match complete blocks before the first differing token. Hybrid, recurrent and sliding window caches must restore a state captured before that divergence: their final state cannot be rolled back by slicing K/V tensors.

Hybrid prefill now captures intermediate checkpoints as well as the final guard checkpoint, in streaming, continuous batching and DiffusionGemma generation. For text API requests with transient_context, the server verifies and prioritizes the token boundary before that mutable suffix. Otherwise, observed divergence from resident prompts takes priority. With neither hint, each prompt stores its latest 2,048-token boundary before the final checkpoint and the final checkpoint itself. For example, a 30,000-token document followed by 20 instruction tokens can reuse 28,672 tokens when the instructions change. The same checkpoints persist across server restarts when the disk tier is enabled.

Captures are bounded by APC_CHECKPOINT_ENTRIES to avoid copying the growing hybrid cache at every prefill chunk. More entries retain more earlier boundaries; a smaller interval gives finer reuse near the end. Reuse still requires a retained boundary before the divergence. Short prompts below the interval, divergence before the earliest retained checkpoint, and evicted checkpoints can miss. Media checkpoints include all media tokens so the remaining suffix is text. Changing checkpoint boundaries can change floating-point execution shapes, as with other chunked or cached prefill paths.

The resident byte budget includes exact snapshots as well as pageable blocks. Before embeddings and prefill, APC drains pending disk writes and evicts idle checkpoints and blocks in LRU order within each tier. Admission uses current Metal allocations, available system RAM, and the incoming prompt’s estimated cache growth, based on observed bytes per token. Leased blocks remain valid for active requests. Snapshots that exceed the budget are written directly to disk without making another resident copy, and disk restores stay out of the memory LRU when promotion would exceed the budget. Disk pressure may therefore trade latency for lower memory use.

These controls limit APC’s memory overhead; model weights and an individual request must still fit in memory. Tune the reserve for models with larger prefill or vision temporaries. /v1/cache/stats reports resident bytes, the memory budget, prefill reserve, memory evictions, and pending disk bytes.

KV Cache Quantization

Reduce KV cache memory during continuous batching with --kv-bits. Both uniform quantization and TurboQuant are supported. Compatible with Automatic Prefix Caching (APC_ENABLED=1).

# Uniform 8-bit KV cache quantization
mlx_vlm.server --model google/gemma-4-26b-a4b-it --kv-bits 8

# TurboQuant 3.5-bit (3-bit keys + 4-bit values)
mlx_vlm.server --model google/gemma-4-26b-a4b-it --kv-bits 3.5 --kv-quant-scheme turboquant

Full-attention layers use quantized batch caches while sliding-window layers keep their fixed-size rotating caches. The last full-attention layer stays unquantized (sensitive in deep models).

Per-tensor KV quantization

Keys and values do not have to share a bit-width or a backend. A fractional --kv-bits already splits the budget — 3.5 gives 3-bit keys and 4-bit values — and --kv-key-bits / --kv-value-bits override either side:

# 8-bit keys, 3-bit values, both TurboQuant
mlx_vlm.generate --model mlx-community/Qwen3.5-9B-MLX-4bit \
  --kv-bits 3.5 --kv-quant-scheme turboquant \
  --kv-key-bits 8 --kv-value-bits 3

--kv-key-scheme / --kv-value-scheme go further and select a different backend per tensor, which builds a hybrid cache:

# uniform 8-bit keys beside TurboQuant 3-bit values
mlx_vlm.generate --model mlx-community/Qwen3.5-9B-MLX-4bit \
  --kv-bits 8 --kv-quant-scheme uniform \
  --kv-value-bits 3 --kv-value-scheme turboquant

Two limitations apply to mixed schemes specifically:

• The hybrid cache dequantizes on every step instead of using a fused kernel, so it is slower than either homogeneous path.
• Mixed schemes are not supported during continuous batching or for batch prefix caches, and raise NotImplementedError there. Mixed bit-widths under a single scheme work everywhere.

Note that values are given the extra bit by default for a reason: value error passes straight through the attention output, whereas key error is partly reabsorbed by the softmax. Measured on Qwen3.5, spending an equal budget key-heavy was consistently worse than value-heavy, so prefer measuring before overriding.

Tested with gemma-4-26b-a4b-it at 20K context:

Config	Gen tok/s	KV Cache	KV Reduction
No quant	50.3	0.624 GB	1x
Uniform 8-bit	52.6	0.469 GB	1.33x
TurboQuant 3.5-bit	25.6	0.365 GB	1.71x

Models with all full-attention layers (e.g. Qwen, LLaMA) see larger reductions — up to 3.6x at 8-bit and 6.4x at 4-bit.

Log Probabilities

The /chat/completions endpoint supports OpenAI-compatible per-token log probabilities. Pass logprobs: true (and optionally top_logprobs: N, up to 20) in the request:

curl -X POST "http://localhost:8080/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen2-VL-2B-Instruct-4bit",
    "messages": [{"role":"user","content":"Say hi in 3 words."}],
    "max_tokens": 8,
    "logprobs": true,
    "top_logprobs": 3
  }'

Each choice gets a logprobs.content[] list with one entry per generated token: {token, logprob, bytes, top_logprobs: [{token, logprob, bytes}, ...]}. Works for both streaming and non-streaming.

top_logprobs requires the server to be started with a non-zero cap on how many alternatives it will compute per token (default 0 = disabled, max 20). Set it via the --top-logprobs-k flag or the TOP_LOGPROBS_K env var:

mlx_vlm.server --model mlx-community/Qwen2-VL-2B-Instruct-4bit --top-logprobs-k 5
# or
TOP_LOGPROBS_K=5 mlx_vlm.server --model mlx-community/Qwen2-VL-2B-Instruct-4bit

Per-request top_logprobs is clamped to TOP_LOGPROBS_K. When TOP_LOGPROBS_K=0, requests with logprobs: true still return chosen-token logprobs; only the top_logprobs list stays empty. Leaving the cap at 0 keeps the vocab-wide sort out of the decode graph, so deployments that don’t need logprobs pay zero overhead.

Structured Outputs

The /v1/chat/completions and /v1/responses endpoints support OpenAI-compatible json_schema structured outputs. The server constrains generation to the supplied JSON schema and supports both streaming and non-streaming responses.

You can define the schema with Pydantic:

from typing import Literal

from pydantic import BaseModel, ConfigDict, Field


class AnimalResult(BaseModel):
    model_config = ConfigDict(extra="forbid")

    animal: Literal["dog", "cat", "bird", "unknown"]
    species: str = Field(max_length=60)
    description: str = Field(max_length=200)


schema = AnimalResult.model_json_schema()

Call the local server with the OpenAI Python client:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="mlx-community/Qwen3.5-4B-MLX-4bit",
    messages=[
        {"role": "user", "content": "Return a dog object."},
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "AnimalResult",
            "strict": True,
            "schema": schema,
        },
    },
)

result = AnimalResult.model_validate_json(response.choices[0].message.content)
print(result)

Example output:

animal='dog' species='Canis lupus familiaris' description='A domesticated canine known for companionship and loyalty.'

Chat completions use top-level response_format. The same format works for text-only and multimodal requests:

curl -X POST "http://localhost:8080/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3.5-4B-MLX-4bit",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "Identify the main animal in this image."},
        {"type": "image_url", "image_url": {"url": "/path/to/image.jpg"}}
      ]
    }],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "AnimalResult",
        "strict": true,
        "schema": {
          "type": "object",
          "properties": {
            "animal": {"type": "string", "enum": ["dog", "cat", "bird", "unknown"]},
            "species": {"type": "string", "maxLength": 60},
            "description": {"type": "string", "maxLength": 200}
          },
          "required": ["animal", "species", "description"],
          "additionalProperties": false
        }
      }
    },
    "max_tokens": 256
  }'

Structured outputs are also supported with:

• Streaming chat completions by setting "stream": true
• The responses API via text.format on /v1/responses
• Text-only requests using the same response_format shape

Structured outputs are not currently supported with speculative decoding.

How It Works

• A dedicated generation thread runs a BatchGenerator that processes multiple requests in parallel
• Image requests are prefilled individually with their own vision embeddings, then join the shared decoding batch
• Text-only requests are batched together for efficient prefill
• After prefill, all requests decode together in a single batch, sharing GPU compute

Available Endpoints

• /models and /v1/models - Discover cached and local models, including their loaded status; accepts repeated model_dir query parameters
• /chat/completions and /v1/chat/completions - OpenAI-compatible chat-style interaction endpoint with support for images, audio, and text
• /responses and /v1/responses - OpenAI-compatible responses endpoint
• /responses/compact and /v1/responses/compact - Compact conversation history into replayable state; see compaction and APC
• /embeddings and /v1/embeddings - OpenAI-compatible embeddings endpoint backed by native MLX embedding models
• /v1/decisions - Predict named typed decisions using the shared decision API
• /v1/meshes - Asynchronous image-to-3D jobs, with polling and downloads at /v1/meshes/{id} and /v1/meshes/{id}/content; see 3D serving
• /v1/rerank - Rank text or multimodal documents by relevance to a query
• /audio/speech and /v1/audio/speech - OpenAI-compatible text-to-speech endpoint backed by mlx-audio TTS models
• /audio/transcriptions and /v1/audio/transcriptions - OpenAI-compatible speech-to-text endpoint backed by mlx-audio STT models
• /audio/translations and /v1/audio/translations - OpenAI-compatible audio translation endpoint for STT models that expose a translation task
• /audio/diarizations and /v1/audio/diarizations - Recording-wide speaker turns using MLX-Audio Nemotron diarization
• /audio/speaker_embeddings and /v1/audio/speaker_embeddings - Normalized ECAPA speaker vectors from separate speech windows
• /v1/realtime - WebSocket-based realtime full-duplex speech. See the Nemotron VoiceChat guide for supported models and usage.
• /health - Check server status
• /metrics and /v1/metrics - Inspect rolling request metrics, throughput, and runtime counters
• /unload - Unload all loaded model caches from memory

Usage Examples

List available models

curl "http://localhost:8080/models"

/models and /v1/models list loaded models and local model candidates from the Hugging Face cache and any configured or request-specific model directories. Each entry includes loaded: true when the model is currently loaded in this server process, or loaded: false when it is discovered but not loaded. Discovery does not load models or guarantee that every discovered architecture is supported.

Discovery reads model metadata (config.json, or model_index.json for image pipelines) and checks for nonempty safetensors files. Sharded checkpoints must have every shard referenced by their weight index. Tokenizer metadata is not required, and checkpoint Python code is never executed by discovery.

All cached revisions are considered, preferring main. If only another revision is usable, the entry’s ID is its absolute snapshot path so selecting it uses that local revision. Custom models also use absolute paths. Symlink aliases and loaded local models are deduplicated by their resolved directory.

Each --model-dir can be a model folder or a parent whose immediate children are model folders; it does not recursively scan arbitrary directories. Missing paths and invalid model folders are skipped. Hugging Face’s HF_HUB_CACHE and HF_HOME settings select a custom Hugging Face cache directory as usual.

You can also pass directories directly to either models endpoint:

curl --get "http://localhost:8080/v1/models" \
  --data-urlencode "model_dir=/Volumes/Models" \
  --data-urlencode "model_dir=/Users/me/my custom model"

These paths refer to the server’s filesystem. They supplement the cache and configured directories for this request only; they do not change server settings. Use repeated model_dir parameters for multiple paths. Empty values are ignored.

Embeddings

curl -X POST "http://localhost:8080/v1/embeddings" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "sentence-transformers/all-MiniLM-L6-v2",
    "input": ["The quick brown fox.", "A fast auburn fox."]
  }'

Preload a default with --embedding-model <repo-or-path>. Supported architectures: BERT, XLM-RoBERTa, ModernBERT, Qwen3-Embedding, EmbeddingGemma (gemma3), LFM2, SigLIP (text), Qwen3-VL-Embedding, and Llama-Nemotron-VL, plus LLM2Vec bidirectional Llama. ColBERT-style multi-vector models (ColIdefics3, ColQwen2.5, LFM2-ColBERT) a</target_tokens:></adapter_path></hf_repo_or_local_path></speech_to_text_model></text_to_speech_model></image_generation_model></language_model></hf_repo_or_local_path>

README 内容过长，已截取前 64 KB。完整内容请前往 GitHub 仓库查看。