VibeVoice-FastAPI

VibeVoice FastAPI Server

A production-ready FastAPI server that exposes the VibeVoice TTS model as an OpenAI-compatible API, with Docker support and comprehensive voice management.

🚀 Features

• Docker-First Deployment: Production-ready Docker setup with GPU support (recommended)
• OpenAI-Compatible API: Drop-in replacement for OpenAI’s TTS API (/v1/audio/speech)
• Unlimited Custom Voices: Automatically load any voice from a directory - just drop audio files and restart
• Multi-Format Support: MP3, WAV, FLAC, AAC, M4A, Opus, PCM
• Streaming Support: Real-time audio streaming for long-form content
• Voice Management: Dynamic voice loading, OpenAI voice mapping, and custom voice presets
• AWQ-INT4 Quantization: Drop-in ncoder-ai/VibeVoice-Large-AWQ — 8.4 GB VRAM at RTF ~0.70 on RTX 3090, faster + smaller than bnb-Q8 with no audible quality loss. See AWQ quickstart below.
• Production Ready: Health checks, error handling, CORS support, and comprehensive logging

⚡ AWQ Quickstart (recommended for single 3090)

For best speed-to-VRAM ratio on a single 24 GB GPU, just point at the unified AWQ model:

# In your .env (or docker-env file):
VIBEVOICE_MODEL_PATH=ncoder-ai/VibeVoice-Large-AWQ
VIBEVOICE_INFERENCE_STEPS=7
TORCH_COMPILE=true

That’s it — no extra env vars. Transformers reads quantization_config from the checkpoint and wires AWQ kernels automatically. Start with docker compose up -d --build. First request downloads the model (~9 GB); subsequent runs reuse the cache.

Benchmarks (RTX 3090, 277-char prompt, 7 steps):

Setup	VRAM	Generation Time	RTF
rsxdalv/VibeVoice-Large (FP16)	17.4 GB	~9 s for 16 s audio	0.54
FabioSarracino/VibeVoice-Large-Q8 (bnb-Q8)	10.8 GB	~20 s for 16 s audio	1.22
ncoder-ai/VibeVoice-Large-AWQ	8.4 GB	~11 s for 16 s audio	0.70

The unified model has its Qwen2 LLM quantized to INT4 with AWQ + Marlin GEMM kernels. The audio tokenizer + diffusion head stay FP16 inside the same checkpoint, so audio quality is indistinguishable from FP16 at INFERENCE_STEPS=7.

📋 Quick Start

One-Click Install (Recommended)

Linux / macOS:

curl -fsSL https://raw.githubusercontent.com/ncoder-ai/VibeVoice-FastAPI/main/install.sh | bash

Windows (PowerShell):

Invoke-WebRequest -Uri "https://raw.githubusercontent.com/ncoder-ai/VibeVoice-FastAPI/main/install.bat" -OutFile "install.bat"; .\install.bat

This clones the repo, detects your GPU, walks you through setup, and starts the server automatically.

Or if you already have the repo cloned:

python3 install.py

Docker Deployment (Manual)

If you prefer to set things up manually with Docker:

# Clone the repository
git clone https://github.com/ncoder-ai/VibeVoice-FastAPI.git
cd VibeVoice-FastAPI

# Copy and configure environment
cp docker-env.example .env
# Edit .env - set VOICES_DIR to your voice files path

# Build and run
docker-compose up -d

# Check logs
docker-compose logs -f

The API will be available at http://localhost:8001

See DOCKER_QUICKSTART.md for detailed Docker instructions.

Local Installation (Linux/macOS)

For development or if you prefer bare-metal installation:

# Clone the repository
git clone https://github.com/ncoder-ai/VibeVoice-FastAPI.git
cd VibeVoice-FastAPI

# Run setup script
./setup.sh

# Configure environment
cp env.example .env
# Edit .env with your settings

# Start server
./start.sh

Local Installation (Windows)

Windows baremetal installation requires manual setup (the .sh scripts are Linux/macOS only):

# Clone the repository
git clone https://github.com/ncoder-ai/VibeVoice-FastAPI.git
cd VibeVoice-FastAPI

# Create and activate virtual environment
python -m venv venv
venv\Scripts\activate

# Install PyTorch with CUDA (check https://pytorch.org for your CUDA version)
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu128

# Install VibeVoice and API dependencies
pip install -e .
pip install -r requirements-api.txt

# Configure environment
copy env.example .env
# Edit .env with your settings (use notepad, vscode, etc.)

# Start server
start.bat

Note: Flash-attention does not have pre-built Windows wheels. The API will automatically fall back to SDPA attention, which works well. Also ensure ffmpeg is installed and on your PATH for audio format conversion.

📖 Documentation

• API README - Complete API documentation with examples, voice management, and troubleshooting
• Docker Quickstart - Docker deployment quickstart guide

🎯 API Endpoints

OpenAI-Compatible Endpoints

• POST /v1/audio/speech - Generate speech from text (OpenAI-compatible)
• GET /v1/audio/voices - List all available voices

VibeVoice-Specific Endpoints

• POST /v1/vibevoice/generate - Advanced generation with multi-speaker support
• GET /v1/vibevoice/voices - List all voices with detailed info
• GET /v1/vibevoice/health - Detailed health check

Example: Generate Speech

curl -X POST http://localhost:8001/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "tts-1",
    "input": "Hello, this is a test of the VibeVoice API",
    "voice": "alloy",
    "response_format": "mp3"
  }' \
  --output speech.mp3

Example: List Voices

# List all available voices
curl http://localhost:8001/v1/audio/voices

# List with OpenAI format
curl http://localhost:8001/v1/audio/voices | jq

MODEL MANAGEMENT

VibeVoice Large (AWQ-INT4, recommended): Huggingface: ncoder-ai/VibeVoice-Large-AWQ [https://huggingface.co/ncoder-ai/VibeVoice-Large-AWQ] VibeVoice Large (FP16): Huggingface: rsxdalv/VibeVoice-Large [https://huggingface.co/rsxdalv/VibeVoice-Large]

VibeVoice 1.5B: Huggingface microsoft/VibeVoice-1.5B [https://huggingface.co/microsoft/VibeVoice-1.5B]

🎤 Voice Management

Using OpenAI-Compatible Voices

The API includes 6 OpenAI-compatible voice mappings:

• alloy, echo, fable, onyx, nova, shimmer

Using Custom Voices

Simply place audio files (.wav, .mp3, .flac, .m4a, etc.) in your VOICES_DIR and restart the server. All files are automatically loaded as voice presets!

# Add a custom voice
cp my_voice.wav /path/to/voices/custom_voice.wav
# Restart server - voice is now available!

Direct Voice Usage

You can use any voice name directly in API requests:

curl -X POST http://localhost:8001/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "tts-1",
    "input": "Testing custom voice",
    "voice": "custom_voice",
    "response_format": "wav"
  }'

⚙️ Configuration

Key environment variables (see env.example for full list):

# Model Configuration
VIBEVOICE_MODEL_PATH=ncoder-ai/VibeVoice-Large-AWQ   # AWQ-INT4 drop-in, ~8.4 GB VRAM
VIBEVOICE_DEVICE=cuda                                # cuda, cpu, or mps
VIBEVOICE_INFERENCE_STEPS=7                          # 5-50; 7 is the sweet spot
VIBEVOICE_DTYPE=float16                              # match AWQ's compute path
VIBEVOICE_ATTN_IMPLEMENTATION=flash_attention_2      # falls back to sdpa if unavailable

# Voice Configuration
VOICES_DIR=demo/voices                               # Directory with voice files

# API Configuration
API_PORT=8001
API_CORS_ORIGINS=*

# Performance Optimization
TORCH_COMPILE=true                                   # ~40% speedup (slower first request)
TORCH_COMPILE_MODE=default                           # default, reduce-overhead, or max-autotune
TORCH_CACHE_DIR=~/.cache/torch_compile_vibevoice     # persist compile cache across restarts
# VIBEVOICE_QUANTIZATION=int8_torchao                # Runtime torchao quant on top of FP16 base

# Generation Defaults
DEFAULT_CFG_SCALE=1.8                                # 1.0-3.0
DEFAULT_RESPONSE_FORMAT=mp3

🐳 Docker Deployment

Docker is the recommended and preferred deployment method. It provides:

• ✅ Consistent environment across all systems
• ✅ No dependency conflicts
• ✅ Easy GPU configuration
• ✅ Production-ready setup
• ✅ Simplified updates and maintenance

Requirements

• Docker and Docker Compose
• NVIDIA Container Toolkit (for GPU support)
• NVIDIA GPU with 12GB+ VRAM recommended (8GB minimum with the AWQ-INT4 model)

Quick Start

# Copy and configure environment
cp docker-env.example .env
# Edit .env - set VOICES_DIR to your voice files path

# Build and run
docker-compose up -d

# Check logs
docker-compose logs -f

See DOCKER_QUICKSTART.md for complete Docker deployment guide.

🔧 Development

Setup Development Environment

# Create virtual environment
python3.12 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -e .
pip install -r requirements-api.txt

# Install PyTorch (CUDA)
pip install torch --index-url https://download.pytorch.org/whl/cu128

# Optional: Install flash-attn for faster inference
# See setup.sh for pre-built wheel installation

Running Tests

# Start server
./start.sh

# Test API
curl http://localhost:8001/health
curl http://localhost:8001/v1/audio/voices

📊 Supported Models

Model	Size	Context	VRAM	RTF (RTX 3090)	Notes
ncoder-ai/VibeVoice-Large-AWQ	7B (INT4)	32K	~8.4 GB	~0.70	Default. AWQ-INT4 LLM + FP16 audio in one drop-in checkpoint.
rsxdalv/VibeVoice-Large	7B	32K	~17 GB	~0.54	Full FP16, fastest on a 24 GB card.
FabioSarracino/VibeVoice-Large-Q8	7B (INT8)	32K	~11 GB	~1.22	bnb-Q8. Slower than AWQ on Ampere.
microsoft/VibeVoice-1.5B	1.5B	64K	~8 GB	n/a	Single-speaker only, no voice cloning.
microsoft/VibeVoice-Realtime-0.5B	0.5B	—	~4 GB	<0.3	Single-speaker realtime variant.

Models are automatically downloaded from HuggingFace on first use.

🛠️ System Requirements

For Docker Deployment (Recommended):

• Docker: Docker and Docker Compose installed
• GPU: NVIDIA GPU with 12GB+ VRAM recommended (8GB minimum with the AWQ-INT4 default)
• NVIDIA Container Toolkit: Required for GPU support
• RAM: 16GB minimum, 32GB recommended
• Storage: 10GB minimum, 50GB recommended (with model cache)
• OS: Linux (recommended), macOS, or Windows (with Docker Desktop + WSL2)

For Local Installation:

• Python: 3.12
• GPU: NVIDIA GPU with 8GB+ VRAM
• RAM: 16GB minimum, 32GB recommended
• Storage: 10GB minimum, 50GB recommended
• OS: Linux, macOS, or Windows
• ffmpeg: Required for audio format conversion (download)

🔐 Security Notes

• The API does not include authentication by default. For production use, add authentication middleware or deploy behind a reverse proxy with authentication.
• Voice files are loaded from the configured directory - ensure proper file permissions.
• Model weights are downloaded from HuggingFace - verify model integrity in production.

📝 License

This project maintains the original VibeVoice model codebase. Please refer to the original VibeVoice license for model usage terms.

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

🙏 Acknowledgments

• VibeVoice Team at Microsoft for the original model
• shijincai for maintaining a backup of the original VibeVoice codebase
• FastAPI for the excellent web framework
• HuggingFace for model hosting and transformers library

📚 Additional Resources

• VibeVoice Original Paper
• VibeVoice HuggingFace Collection
• FastAPI Documentation

⚠️ Limitations

• Language Support: Primarily English and Chinese. Other languages may produce unexpected results.
• Non-Speech Audio: The model focuses on speech synthesis and may generate background music or sounds spontaneously.
• Commercial Use: This model is intended for research and development. Test thoroughly before production use.

🆘 Troubleshooting

Server won’t start

• Check GPU availability: nvidia-smi
• Verify Python version: python3 --version (should be 3.12)
• Check dependencies: pip list | grep torch

Out of memory errors

• Switch to the AWQ-INT4 model (~8.4 GB VRAM, no quality loss): VIBEVOICE_MODEL_PATH=ncoder-ai/VibeVoice-Large-AWQ
• Or fall back to single-speaker 1.5B: VIBEVOICE_MODEL_PATH=microsoft/VibeVoice-1.5B
• Reduce inference steps: VIBEVOICE_INFERENCE_STEPS=5
• Use CPU mode: VIBEVOICE_DEVICE=cpu (much slower)

Voice not found errors

• Verify VOICES_DIR path in .env
• Check file permissions
• Ensure audio files are in supported formats

For more help, see the API README or open an issue.

---

Made with ❤️ for the VibeVoice community