ComfyUI-VibeVoice

• Features
• Models
• Getting started
• Usage
  • Script format
  • Voice modes
  • Realtime models
  • Node inputs
  • Loading external models
    • Host RAM during an external load
• Performance & stability
  • Feature compatibility & VRAM matrix
  • transformers version support
  • Attention backends
• Changelog
• License
• Acknowledgments
• Star history

cd ComfyUI/custom_nodes
git clone https://github.com/wildminder/ComfyUI-VibeVoice.git
cd ComfyUI-VibeVoice
pip install -r requirements.txt

Restart ComfyUI. Nodes appear under WMNodes/sound/tts.

Setup:

1. Copy an official voice prompt (e.g. en-Carter_man.pt) to ComfyUI/models/tts/VibeVoice/voices/ — or any folder registered under ComfyUI’s vibevoice_voices key. Prompts are discovered recursively by file stem, so the official voices/streaming_model/*.pt layout can be copied intact.
2. Add VibeVoice TTS; set model_name to VibeVoice-Realtime-0.5B and voice_preset to the prompt stem (e.g. en-Carter_man).
3. Write a single-speaker script, leave speaker_*_voice unconnected, queue.

Length budget (max_new_tokens). 0 means auto, not “model maximum”. The auto budget charges prefilled text tokens and generated speech latents against one allowance (each window = 5 text + 6 latent units) and adds a tail so the model can finish its last clause and reach end-of-speech; the chosen budget is logged and an explicit positive value always wins. If the console reports “stopped on the length budget … without the model signalling end of speech”, the clip was cut off mid-script — raise max_new_tokens or shorten the script.

[!IMPORTANT] VibeVoice TTS is the only TTS node — it serves standard and realtime model families alike. The former VibeVoice Realtime TTS node has been removed; workflows saved against it only need the node swapped for VibeVoice TTS, with the model selection and script carried over unchanged.

1. Place the weight file in models/diffusion_models/ (a.k.a. models/unet/).
2. (Optional but recommended) Bind configs with sidecar JSONs next to the weights:

3. Select the file in model_file; keep config_name on Auto-detect unless needed — it reads the weight file’s embedding fingerprint, recognizes all four packaged families (VibeVoice-1.5B, VibeVoice-7B, VibeVoice-Realtime-0.5B, VibeVoice-ASR), never guesses an unmatched shape, and auto-corrects an explicit choice that contradicts the weights with a warning.
4. Connect VIBEVOICE_MODEL → external_model.

Notes:

• The model is built entirely on CPU; the single host-to-device transfer is owned by ComfyUI’s VRAM arbitration.
• Type guards reject mis-wiring (an ASR model on the TTS node and vice versa) with an error naming the correct node; a realtime bundle on the TTS node is accepted and routed through the realtime path.
• quantize_llm_4bit applies to TTS / realtime models only; ASR always loads at full precision.
• Realtime key counts: a VibeVoice-Realtime-0.5B export is decoder-only — the released weights deliberately omit the 276 model.acoustic_tokenizer.encoder.* tensors (unlike 1.5B / 7B, which carry them). The loader treats that block as intentionally absent only while the checkpoint supplies none of it, so a truncated export still reports its real missing keys. Diffing key counts against 1.5B is expected, not a bug.
• GGUF: .gguf files are listed in model_file and dequantized via the gguf package — ComfyUI’s stock diffusion loader cannot parse GGUF. The package is optional: safetensors and other formats never need it, and it is installed only on request (pip install gguf, or the gguf-extra extra for registry installs), never forced by the node. Without it, the node loads fine, and selecting a .gguf model file shows an in-app notice offering to install the missing package (one click through ComfyUI-Manager, or a copyable command when Manager is unavailable). Both models/diffusion_models/ and the ComfyUI-GGUF unet_gguf folder are discovered.
• Mismatch guard: a config_name contradicting the weights fails fast with the offending tensor names and the correct config_name suggested — no raw size mismatch stack trace.
• Windows Pin error.: under RAM pressure ComfyUI core can log a harmless flood of [WARNING] Pin error. while pinning partially offloaded weights. Quant-resident loads (GGUF / ConvRot INT8 / fp8) never offload, so they avoid it; otherwise start ComfyUI with --disable-pinned-memory.

ComfyUI’s ModelPatcherDynamic gates its entire demand-paging mechanism on a single per-module attribute, comfy_cast_weights (comfy/model_patcher.py:1967); only modules carrying it get an m._v = vbar.alloc(...) handle (:1993) and are paged. Modules without it take the branch at :2000-2009, which stashes the host tensor in self.backup and makes a full eager device copy.

This route suppresses convert_tree_for_streaming (modules/external_loader.py:1904, :2239) to avoid two owners of one weight — but convert_tree_for_streaming is the only thing in this pack that sets comfy_cast_weights on these modules. Core sets it solely on comfy.ops.manual_cast.* (comfy/ops.py:821-853), classes this pack’s transformers trees never use. So the suppression removes the attribute that unlocks paging, and the load lands entirely on the eager branch.

Consequences, all in core’s source, none measured here:

• Every original host tensor is retained by reference in ModelPatcher.backup for the patcher’s lifetime — the full 16.66 GB stays in host RAM. This is the spike the work set out to remove, unchanged.
• A full device copy is made on top, and model_loaded_weight_memory reports full residency, so the low-VRAM budget is defeated rather than honoured.
• Core’s per-module lazy H2D (driven from hasattr(s, "_v") in comfy/ops.py:374) never happens, because no _v is ever allocated.

A second, deeper obstacle is visible in the same source: even with comfy_cast_weights set, ModelPatcherDynamic never releases the host parameter — setup_param allocates _v but does not replace the parameter (comfy/model_patcher.py:1921-1953). Core’s host-RAM economy for its own models comes from the loader (comfy/utils.py:165-166, aimdo load_safetensors against a meta-initialised tree), not from the patcher. This pack instead materialises the whole model in ordinary host memory first (modules/loader.py:895-903: safe_open.get_tensor returns an owned copy under MMAP_TORCH_FILES=False). So the patcher port alone is unlikely to remove the spike even once paging is reachable.

The port therefore needs re-planning; it is not a defect that can be closed by adjusting the existing steps. Until it is re-done and measured, treat the dense route’s host-RAM behaviour as unchanged from the legacy patcher.

• Turn on DEBUG logging for the loader and read the dtype line it now emits, e.g. Model cast torch.float32 -> torch.bfloat16 (812 mismatched params). A cast is the one thing that can double the footprint: it allocates a second copy of every mismatched parameter before the old one is released. Both source dtype and parameter count are on the line for exactly that reason.

• For a timeline of a full external ASR load — peak private commit, peak working set, CUDA allocation, per-phase timings and how many times each load phase ran — use the scratch probe .dev/scratch/probe_external_asr_ram.py (Windows; needs COMFYUI_ROOT set; it loads the real checkpoint, so run it yourself):

  python .dev/scratch/probe_external_asr_ram.py "<path-to-checkpoint>" --config-name VibeVoice-ASR
  

  It loads the checkpoint twice by default (--passes), keeping the first copy alive across the second — that is the situation a live graph is in between two generations, and a second full load is the usual reason a load costs twice the file size. Pass --passes 1 for a single load. Add --drive-consumer to also push the bundle through the ASR node’s cache layer, which is where a re-load caused by a cache-key mismatch shows up. It is also required for the PATCHER SELECTION block: only the consumer path constructs a patcher, so without it that block reports NONE OBSERVED.

• Paste the timeline back — the TIMELINE, PEAK, PHASE COUNTS and PATCHER SELECTION blocks — when reporting a load that uses far more RAM than the file size. The counts are the diagnosis: with --passes N, each phase should show exactly N. More than N means the checkpoint is being loaded more times than you asked for, which is a different bug from a slow or cast-heavy single load. PATCHER SELECTION answers the separate question of which patcher served the load: is_dynamic=True means the weights are demand-paged, and is_dynamic=False on a machine that should have DynamicVRAM means the silent fallback to legacy fired.

◆ Changes

• One TTS node. VibeVoice TTS now handles realtime models as well: just pick VibeVoice-Realtime-0.5B in model_name. The separate realtime TTS node is gone; existing saved workflows load unchanged.
• All nodes moved under WMNodes. WMNodes/sound/tts for the TTS and loader nodes, WMNodes/sound/asr for ASR. Node IDs are unchanged, so saved workflows still load — only the menu location moved.

• Fixes

• The bundled test suite now works from a fresh clone; it no longer references development-only files.

Full release history: CHANGELOG.md.