audio.cpp

audio.cpp

audio.cpp is a high-performance C++ audio inference framework built on top of ggml, designed to make modern local audio models practical, portable, and fast.

Tired of juggling a dozen Conda environments, hundreds of Python packages, and dependency conflicts just to try a few audio models? audio.cpp gives those paths a shared native runtime instead. Runs on Windows, Linux, and macOS, with support for NVIDIA, AMD, Apple Silicon, and CPU-only machines.

Lost in a sea of audio model names, papers, and codebases? Explore 100+ audio models in the Audio Model Architecture Atlas to see how they’re built, what they share, and where they differ, whether you’re learning or building your own.

Supported backends:

audio.cpp is built for real end-to-end execution rather than one-off model demos. Its optimized, reusable building blocks support speech generation and recognition, separation, alignment, and multi-model workflows. Shared components make new models easier to integrate and let improvements benefit multiple model families while keeping inference efficient, maintainable, and portable.

Features

⭐ 143 audio model families and 250+ variants. Run TTS, ASR, music generation, voice cloning, conversion, separation, and more locally through a shared CLI and server API.

⭐ Built-in WebUI. Generate and clone voices in Studio, compare models in Arena, manage models and voice libraries, and assemble narration with reusable speakers and passage editing in the TTS app.

⭐ Performance. Reusable sessions, cached graphs, and batched offline inference. Selected TTS paths run 1.8–8× faster than Python; tested Q8 packages deliver up to 1.53× speedup and 37% lower peak VRAM versus 16-bit weights (report). Supertonic 3 generates 10 hours of audio in 3 minutes on RTX 5090 (demo), while Nemotron 3.5 ASR matches WER at roughly ¼ the wall time in the TranscrIA benchmark. audio.cpp is among the fastest inference engines for streaming TTS latency on the Open TTS Leaderboard.

⭐ Portability. A native ggml runtime supporting CUDA, HIP/ROCm, Vulkan, Metal, WebGPU, and CPU through common CLI and server interfaces.

⭐ Hardware-specific optimizations. ARM NEON acceleration, Rockchip RKNN component execution, experimental HIP kernels, and CUDA, Metal, and Vulkan optimizations. Model-specific perf_mode presets expose optional fast paths for supported hardware.

⭐ Parity. Validation tooling for comparing model outputs against Python reference implementations.

⭐ Integrations and deployment. Powers speech, music, and transcription in Unsloth Studio. Fun-ASR-Nano and SenseVoice can also be deployed through the FunASR platform.

Contributing

audio.cpp would not be moving this quickly without generous contributors bringing in real fixes, new capabilities, and careful polish. See CONTRIBUTING.md for how to contribute and for a shout-out to the people already helping shape the project. Before starting a new model port, check the supported model table because several families are already implemented or under testing, and read the New Model PRs section in CONTRIBUTING.md. New ports should start under the community model source tree, where review is lighter than core models but still needs reproducible validation.

News

2026-10-26 — Release v0.9.3: New speech models including Drama3, Ming 3.0 TTS/Audio Tiny, XTTS v2, Plapre 3, Gradium Phonon, and Earshot VAD, plus experimental LatentSync lip sync. Highlights include LoRA across 11 models, a TTS workspace, voice-cloning and dubbing recipes, perf_mode presets, lower memory use, and expanded HIP, ARM NEON, and Rockchip NPU support. Thanks to our contributors!

2026-10-15 — Release v0.9.2: New models include Voxtral TTS, TF-GridNet, MossFormer2, Hviske v6, FireRed VAD, AST, CED, and wake-word detection models. This release expands TTS streaming, adds multi-turn LFM2.5-Audio conversations, and improves model performance and compatibility. The WebUI gains a shared Voice Library and layout improvements. Downloadable C API libraries, Windows Vulkan portable builds, and packaged server configuration examples make integration easier. Thanks to everyone contributing code, testing, and feedback!

2026-09-30 - Release v0.9.0: 100+ audio models, an enhancement and denoising WebUI tab, speaker-tagged ASR, and performance and memory improvements. Explore the Architecture Atlas. Thanks to our community!

2026-06-25 to 2026-09-23 (releases 0.1 to 0.8): audio.cpp grew to 80+ model families and 120+ variants across TTS, ASR, music generation, voice conversion, separation, and speech analysis. Major additions included the native WebUI and Arena, GGUF packaging, LoRA adapters, streaming, and CUDA/Metal performance improvements, alongside Vulkan and experimental HIP support. The C ABI, Colab integration, MP3/HTTPS frontends, and models such as YuE2 and SheetSage2 expanded integration and audio-to-score workflows.

Supported Models

Task tags: TTS text to speech, Clone voice cloning, VC voice conversion, S2S speech-to-speech, ASR speech recognition, Classify audio classification, Wake wake-word detection, Align forced alignment, VAD voice activity detection, Diar speaker diarization, Codec audio codec, Sep source separation, MIDI audio-to-symbolic MIDI/events, Music music/song generation, SFX sound effects, Video video generation, Edit audio/music editing, Design voice design, Dialogue multi-speaker dialogue TTS, Ctrl TTS/clone voice control such as emotion, style, instruction, caption, or non-verbal tag control.

Runtime tags summarize the supported loading paths. GGUF package precision varies by model and release; check the audio.cpp GGUF repo or docs/gguf.md for the exact package list. Bundled means the tiny runtime asset ships under assets/framework/models and needs no separate model download. Stream means the family exposes a streaming server/session path. LoRA means adapter loading is supported in audio.cpp; see the LoRA guide for supported formats and usage.

Model weights keep the license of their original release, which is separate from audio.cpp’s own license. docs/model_licenses.md lists that license per family, and whether it allows commercial use.

Speech Generation And Conversation

Speech Recognition And Analysis

Audio Conversion And Processing

Music, Media, And Editing

Some model families in the supported table started as outside contributions before being promoted into the core release surface. Thanks to Mirek @mirek190 for BS-RoFormer, @justinjohn0306 for MOSS-TTS-Local, and @LauraGPT from the official FunASR team for Fun-ASR-Nano.

Community Models

Community model ports live under community_models to make the ownership boundary clear while keeping them available through the normal audio.cpp CLI and server paths. Some community-contributed models graduate into the core model tree when they become part of the main release surface. Huge thanks to the contributors who bring these models in, test them, and keep pushing the framework into new territory. See CONTRIBUTING.md for community-model expectations.

Prebuilt Binaries

Official packages for every release are attached to the Releases page:

The Windows CUDA packages ship the CUDA runtime in a separate cudart archive. Extract it next to the binaries so the CUDA backend can load ggml-cuda.dll and the CUDA runtime DLLs (cudart, cuBLAS, cuFFT).

• HIP/ROCm (AMD GPUs): Official Windows and Ubuntu test builds are available on the Releases page. GPU compatibility and model coverage are still being validated; please report issues with your GPU, driver version, and reproduction command.
• Older community Windows HIP packages: @IIIIIllllIIIIIlllll’s fork Releases provides v0.9.0 packages with the ROCm runtime bundled, in ROCm 6.4 and 7.1 tracks. See docs/build/windows-hip-distribution.md for packaging details.

Homebrew Install

On macOS, audio.cpp can be installed from the Homebrew tap:

brew tap 0xShug0/audio-cpp
brew trust 0xShug0/audio-cpp
brew install audio-cpp

WebUI

audiocpp_server includes an embedded SvelteKit/TypeScript WebUI for running local TTS, cloning, ASR, generation, conversion, separation, VAD, diarization, and alignment workflows. The production UI is compiled into the server binary, so using it requires neither Python nor separate frontend files:

audiocpp_server --ui --ui-management --backend cuda

Open http://127.0.0.1:8080. --ui-management enables catalog browsing, downloads, temporary uploads, and dynamic model switching, so this is the easiest way to try audio.cpp without writing a server config first. For a locked-down config-driven instance, start with --ui and a server config instead; in that mode the UI only offers models declared by the config unless management is enabled.

For Docker UI downloads, mount a writable models directory at /app/models and use --ui-management. See docker.md.

The native UI also exposes background model download/preparation, long-text split-and-merge synthesis, a browser-local saved voice library, microphone recording, and near-live ASR input. Some model preparation jobs invoke the repository’s Python model manager because those packages require Hugging Face download or checkpoint conversion; model inference and the embedded UI remain Python-free. See webui/README.md for launch commands, model notes, and frontend development instructions.

Docker

Docker CUDA and CPU images are available for both CLI and server use. See docker.md for available images, build commands and working Docker examples.

Model Manager and GGUF Downloads

Use tools/model_manager_v2.py for normal model downloads. It reads model_specs/*.json and installs the default package for each family, preferring ready-to-use GGUF packages when they are available.

Native builds configured with -DAUDIOCPP_BUILD_NATIVE_MODEL_MANAGER=ON also provide audiocpp_model_manager, a standalone headless frontend over the same reusable C++ package-management library used by the server and embedded WebUI. It supports list, info, install, clean, and remove without starting audiocpp_server; this is the preferred native path for CLI, Docker, remote provisioning, and other scripted environments. The Python v2 manager remains available as an alternative during migration. The native path uses bundled TLS by default and does not require libcurl.

audiocpp_model_manager list
audiocpp_model_manager install qwen3_asr_0_6b_q8_0 --models-dir models

The old safetensors/converter catalog has been renamed to tools/model_manager_deprecated.py. Use it only for legacy model layouts that have not moved to spec-backed GGUF packages yet.

GGUF downloads:

• Official model packages: audio-cpp on Hugging Face
• Community-hosted model packages: see the individual model guides linked in the supported model tables for repositories and download instructions.

See the Model Manager guide for model-manager usage and package notes.

Build yourself

scripts/build_linux.sh --backend cuda --model-set custom --models qwen3_tts,pocket_tts,qwen3_asr --target audiocpp_cli

cmake -S . -B build/debug -DCMAKE_BUILD_TYPE=Debug -DAUDIOCPP_MODEL_SET=custom -DAUDIOCPP_MODELS=qwen3_tts,pocket_tts,qwen3_asr
cmake --build build/debug --target audiocpp_cli -j 8

scripts/build_linux.sh --backend cuda --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend vulkan --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend hip --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend cpu --target audiocpp_cli --target audiocpp_server

scripts/build_linux.sh --backend cuda --cuda-arch native --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend cuda --cuda-arch "86;89" --target audiocpp_cli --target audiocpp_server

scripts/build_linux.sh --backend cuda --cuda-arch native --ccache --target audiocpp_cli

scripts/build_linux.sh --backend cuda --model-set full --target audiocpp_cli
scripts/build_linux.sh --backend cuda --model-set custom --models qwen3_tts,pocket_tts,qwen3_asr --target audiocpp_cli
scripts/build_linux.sh --backend cpu --model-set core --target audiocpp_cli

scripts/build_linux.sh --backend cuda --native-cpu OFF --target audiocpp_cli --target audiocpp_server

scripts/build_linux.sh --backend cuda --deployment-build --target audiocpp_cli --target audiocpp_server

scripts/build_linux.sh --backend cuda --native-model-manager --target audiocpp_server

powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\scripts\build_windows.ps1

.\scripts\build_windows.ps1 -Preset windows-cuda-release -Target audiocpp_cli
.\scripts\build_windows.ps1 -Preset windows-vulkan-release -Target audiocpp_cli
.\scripts\build_windows.ps1 -Preset windows-cpu-release -Target audiocpp_cli
.\scripts\build_windows.ps1 -Target audiocpp_server -Jobs 16
.\scripts\build_windows.ps1 -Preset windows-cuda-release -ModelSet custom -Models "qwen3_tts,pocket_tts,qwen3_asr" -Target audiocpp_cli

scripts\build_windows.cmd

.\scripts\build_windows.ps1 -DeploymentBuild -Target audiocpp_cli

.\scripts\build_windows.ps1 -NativeModelManager -Target audiocpp_server

cmake -S . -B build/macos-cpu-release \
  -DCMAKE_BUILD_TYPE=Release \
  -DENGINE_ENABLE_CUDA=OFF \
  -DENGINE_ENABLE_VULKAN=OFF \
  -DENGINE_ENABLE_METAL=OFF \
  -DENGINE_ENABLE_OPENMP=OFF \
  -DGGML_OPENMP=OFF
cmake --build build/macos-cpu-release \
  --parallel "$(sysctl -n hw.logicalcpu)" \
  --target audiocpp_cli audiocpp_server audiocpp_gguf

build/macos-cpu-release/bin/audiocpp_cli --list-devices

scripts/build_metal.sh --target audiocpp_cli

scripts/build_metal.sh --target audiocpp_server
scripts/build_metal.sh --build-type Release --archs arm64 --target audiocpp_cli
scripts/build_metal.sh --model-set custom --models qwen3_tts,pocket_tts --target audiocpp_cli
scripts/build_metal.sh --with-tests --target audio_dsp_test
scripts/build_metal.sh --openmp auto --target audiocpp_cli
scripts/build_metal.sh --native-cpu OFF --target audiocpp_cli
scripts/build_metal.sh --deployment-build --target audiocpp_cli

build/macos-metal-release/bin/audiocpp_cli

scripts/build_linux.sh --backend hip --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend hip --gpu-targets "gfx1100;gfx1103" --target audiocpp_cli

cmake -S . -B build_hip \
  -DENGINE_ENABLE_HIP=ON \
  -DGPU_TARGETS=gfx1151 \
  -DCMAKE_C_COMPILER="$(hipconfig -l)/clang" \
  -DCMAKE_CXX_COMPILER="$(hipconfig -l)/clang++" \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build_hip -j$(nproc)

powershell -ExecutionPolicy Bypass -File scripts\build_windows_hip.ps1

cmake -S . -B build/debug \
  -DCMAKE_BUILD_TYPE=Debug \
  -DENGINE_ENABLE_WEBGPU=ON \
  -DDawn_DIR=/path/to/dawn/lib/cmake/Dawn
cmake --build build/debug --target audiocpp_cli -j 8

git -c 'url.https://github.com/[email protected]:' submodule update --init external/audio.cpp-server-frontends

cmake -S . -B build/debug \
  -DCMAKE_BUILD_TYPE=Debug \
  -DAUDIOCPP_BUILD_SERVER_FRONTENDS=ON \
  -DAUDIOCPP_SERVER_FRONTENDS_DIR=external/audio.cpp-server-frontends \
  -DAUDIOCPP_SERVER_FRONTEND_MODULES="audio_decode;mp3_encode"
cmake --build build/debug --target audiocpp_server --parallel 8

Usage

CLI

The main CLI binary is:

build/bin/audiocpp_cli

High-level command shape:

audiocpp_cli --task <task> --model <path> --family <family> [--backend <backend>] [--mode <mode>] [options]

Always specify --family for model commands, including model-specific help. Treat it as required; omission is supported only for backward compatibility. Automatic loader discovery without it can substantially slow startup and --model ... --help.

Core selectors:

• --task vad|asr|diar|sep|gen|tts|clon|vc|s2s|align|vdes|spk|svc|midi|wake|cls
• --model <path>
• --family <name> explicitly selects the model family
• --backend cpu|cuda|vulkan|metal|best
• --mode offline|streaming; streaming is available for models whose docs list streaming support

Common interface options:

• --load-option key=value passes model-load options, such as PocketTTS language selection
• --session-option key=value passes session/runtime options, such as backend-specific weight controls
• --request-option key=value passes per-request model options
• --config <id> selects a discovered config asset
• --weight <id> selects a discovered weight asset
• --device <n> selects the backend device
• --threads <n> sets backend and OpenMP worker threads

Examples:

Text-to-speech:

build/bin/audiocpp_cli \
  --task tts \
  --family pocket_tts \
  --model /path/to/model \
  --backend cuda \
  --text "audio.cpp is running PocketTTS locally." \
  --voice-ref assets/resources/sample.wav \
  --out build/out/pocket_tts.wav

PocketTTS with another language and a built-in voice:

build/bin/audiocpp_cli \
  --task tts \
  --family pocket_tts \
  --model /path/to/models/pocket-tts \
  --backend cuda \
  --load-option language=spanish \
  --text "Hola, esta es una prueba corta de Pocket TTS." \
  --voice-id alba \
  --out build/out/pocket_tts_spanish.wav

ASR:

build/bin/audiocpp_cli \
  --task asr \
  --family qwen3_asr \
  --model /path/to/model \
  --backend cuda \
  --audio assets/resources/sample_16k.wav

Voice conversion:

build/bin/audiocpp_cli \
  --task vc \
  --family seed_vc \
  --model /path/to/model \
  --backend cuda \
  --audio assets/resources/a.wav \
  --voice-ref assets/resources/b.wav \
  --out build/out/seed_vc.wav

Useful CLI features:

• --help with --task shows task-oriented help
• --help with --model <path> and --family <family> shows model-owned request, session, and load options
• --inspect prints discovered configs, weights, and capabilities
• --list-loaders prints registered model families (--json for the machine-readable contract)
• --list-devices prints backend devices and the --backend/--device values that select them (--json adds PCI ids and free/total memory)
• python tools/model_manager_v2.py list --json prints installable packages from model_specs/*.json
• --batch-text-file <txt> runs one offline request per non-empty line
• --batch-text-dir <dir> runs one offline request per .txt, .md, or .json file, normalizing each file as one paragraph
• --batch-audio-dir <dir> runs one offline request per .wav
• --audio-chunk-mode auto lets ASR/alignment models choose their safe long-audio policy; expert users can override with fixed, vad, or none where supported
• --request-sequence <json> runs a multi-request offline session
• --batch-merge-audio none|concat controls batch audio merge behavior
• --batch-manifest-out <json> writes a batch output manifest
• --metrics prints compact offline wall time, audio duration, RTF, realtime speed, sample rate, and channel metrics
• Use --request-sequence <json> --metrics for per-request metrics from one long-lived offline session
• --pipeline <json> runs a workflow instead of a raw task
• --list-pipelines prints registered workflows
• --workflow-input key=value overrides pipeline inputs
• --recipe <name> runs a built-in multi-model composition without a workflow file; see Built-in recipes
• --log streams framework logs to stdout
• --log-file <path> streams framework logs to a file in real time
• --segments-out, --turns-out, and --words-out write structured JSON outputs
• --vad-chunks-out writes offline VAD-based chunk windows; tune them with --vad-chunk-max-seconds, --vad-chunk-merge-gap-seconds, and --vad-chunk-padding-seconds

Server

The server binary is:

build/bin/audiocpp_server

Build:

cmake --build build -j$(nproc) --target audiocpp_server

Create a config file with your own model paths:

cat > server.json <<'JSON'
{
  "host": "127.0.0.1",
  "port": 8080,
  "backend": "cuda",
  "device": 0,
  "threads": 1,
  "lazy_load": true,
  "models": [
    {
      "id": "pocket-tts",
      "family": "pocket_tts",
      "path": "/path/to/models/pocket-tts",
      "task": "tts",
      "mode": "offline",
      "load_options": {
        "language": "english"
      },
      "session_options": {
        "language": "english"
      }
    },
    {
      "id": "qwen3-asr",
      "family": "qwen3_asr",
      "path": "/path/to/models/Qwen3-ASR-0.6B",
      "task": "asr",
      "mode": "offline"
    }
  ]
}
JSON

Set "lazy_load": true to register configured model ids at startup while loading each model only on first use. Use per-model "lazy": true or "lazy": false to override that default.

Set top-level "backend" to "cuda", "cpu", "vulkan", "metal", or "hip". CUDA is the optimized path for audio.cpp; CPU, Vulkan, Metal, and HIP are intended for portability and testing when the binary is built with that backend, but performance and model coverage may be lower.

Set top-level "max_loaded_models" (or start with --max-loaded-models <n>) to bound how many models stay resident in memory at once: loading one more past the limit first unloads the least recently used idle model, and 1 enforces a single loaded model at a time. The default 0 keeps every used model in memory. See app/server/README.md for details.

Start:

build/bin/audiocpp_server --config server.json

The server exposes:

• GET /health
• GET /v1/models
• POST /v1/audio/speech
• POST /v1/audio/transcriptions
• POST /v1/audio/transcriptions/details
• POST /v1/audio/alignments
• POST /v1/tasks/run
• POST /v1/tasks/batch

More server examples are in app/server/README.md.

Pipelines

Pipelines are an experimental JSON workflow feature for chaining multiple model and audio-processing steps behind one CLI command. A pipeline can define default inputs, let users override them with --workflow-input key=value, split long media into model-sized chunks, merge text or audio outputs back together, write intermediate artifacts under --out-dir, and copy the declared final_audio to --out.

This is the higher-level layer for production-style audio jobs: redubbing, batch cleanup, long-form narration, voice conversion, source-separation workflows, transcription-plus-alignment, and future workflows that combine translation, diarization, denoise, enhancement, or review steps as those model surfaces are wired into the framework.

The included same-language speech redub pipeline transcribes long speech in chunks with Qwen3 ASR, merges the transcript, then regenerates the speech in a target reference voice with Qwen3 TTS. The default test input assets/resources/speech.wav is about 418 seconds long and was generated from an 8,091-character speech text, so it exercises long-audio split and merge behavior rather than a short one-shot request:

build/bin/audiocpp_cli \
  --pipeline assets/pipeline/speech_redub.json \
  --backend cuda \
  --out-dir build/out/speech_redub_pipeline \
  --out build/out/speech_redub_pipeline.wav

Override the source speech or target voice without editing the JSON:

build/bin/audiocpp_cli \
  --pipeline assets/pipeline/speech_redub.json \
  --backend cuda \
  --workflow-input source_audio=/path/to/speech.wav \
  --workflow-input target_voice=/path/to/voice.wav \
  --workflow-input language=English \
  --out-dir build/out/speech_redub_pipeline \
  --out build/out/speech_redub_pipeline.wav

Tests

The repository includes both framework-level parity validation and app-level end-to-end path checks. At a high level, the flow is:

README 内容过长，已截取前 64 KB。完整内容请前往 GitHub 仓库查看。