QwenVL for ComfyUI
The ComfyUI-QwenVL suite brings Alibaba Cloud’s next-generation vision-language and reasoning models into ComfyUI. Featuring native support for the full Qwen3-VL family (2B to 32B, Instruct & Thinking variants), Qwen3.5-VL, Qwen3.6-VL (MoE), Qwen3.8-VL, and Qwen2.5-VL, alongside text-only Qwen3 prompt enhancers. Powered by a high-performance dual-engine architecture (GGUF via llama.cpp & PyTorch via Transformers + SageAttention), it delivers ultra-fast image perception, long-sequence video analysis with intelligent token-budget auto-scaling, and zero-config custom model expansion.
📰 News & Updates
-
2026/09/13: v2.3.2 Custom File Directories & Mirror Download Acceleration! [Update]
- Multi-Drive & Local Folder Support: Directly load HuggingFace and GGUF models stored on other drives without re-downloading via seamless
extra_model_paths.yaml integration (case-insensitive LLM/llm and gguf) or directory links.
- hf-mirror.com Download Acceleration: Added one-click download source toggle in
AILab_HuggingFaceDownloader for fast downloads in mainland China.
-
2026/08/31: v2.3.1 Fix several bugs! [Update]
-
2026/08/26: v2.3.0 Intelligent Video Scaling & Custom Models Upgrade!
- Intelligent Video Auto-Scaling & Token Budget Safeguard: Solves context slot overflow (
failed to find a memory slot) and CUDA OOM. Automatically computes safe per-frame token budget based on ctx and frame_count; preserves full native resolution for small video frames while gracefully downscaling high-res videos (1080p/4K). Added video_frame_size control to all Advanced nodes.
- Unified Custom Models Architecture (
custom_models.json): Simplified 2-section design (hf_models & gguf_models) with full support for custom GGUF and HF models across vision and text nodes.
- Enhanced Model Downloader (
AILab_HuggingFaceDownloader): Standalone workflow execution (OUTPUT_NODE = True), smart auto-routing (save_folder: "auto"), automated mmproj visual projector discovery/downloading, and color-coded status UI card.
- Modular Inference Engine (
qwenvl_engine.py & qwenvl_cli.py): Clean Python API and CLI tool for cross-plugin integration.
- Centralized Media Utilities (
AILab_Utils.py): Shared tensor conversion, frame sampling, and dynamic budget estimation across both GGUF and Transformers backends.
- Complete User Guide: In-depth documentation in
docs/USER_GUIDE.md and docs/USER_GUIDE_zh.md. [Update]
-
2026/08/20: v2.2.0 Added native GGUF support for Qwen3.5, 3.6 (MoE), and 3.8. Implemented native comfy.model_management for memory clearing. [Update]
-
2026/02/08: v2.1.1 Fixed compatibility for Transformers 4.x and 5.x [Update]
-
2026/02/05: v2.1.0 Added SageAttention support with per-GPU architecture optimization, improved FP8 model handling, and automatic attention mode selection. [Update]
- SageAttention Support: New attention mode with per-GPU optimized kernels (SM80, SM89, SM90, SM120)
- Improved FP8 Handling: Better support for pre-quantized FP8 models with automatic SDPA fallback
- Smart Attention Selection: Auto mode now tries Sage → Flash → SDPA for optimal performance
- Progress Bar: Added ComfyUI progress bar for model loading and generation stages
- Better Memory Management: Improved cache clearing when changing attention modes or quantization
-
2025/12/22: v2.0.0 Added GGUF supported nodes and Prompt Enhancer nodes. [Update]
[!IMPORTANT]
Install llama-cpp-python before running GGUF nodes instruction
- 2025/11/10: v1.1.0 Runtime overhaul with attention-mode selector, flash-attn auto detection, smarter caching, and quantization/torch.compile controls in both nodes. [Update]
- 2025/10/31: v1.0.4 Custom Models Supported [Update]
- 2025/10/22: v1.0.3 Models list updated [Update]
- 2025/10/17: v1.0.0 Initial Release
- Support for Qwen3-VL and Qwen2.5-VL series models.
- Automatic model downloading from Hugging Face.
- On-the-fly quantization (4-bit, 8-bit, FP16).
- Preset and Custom Prompt system for flexible and easy use.
- Includes both a standard and an advanced node for users of all levels.
- Hardware-aware safeguards for FP8 model compatibility.
- Image and Video (frame sequence) input support.
- “Keep Model Loaded” option for improved performance on sequential runs.
- Seed parameter for reproducible generation.
✨ Features
- Standard & Advanced Nodes: Includes a simple QwenVL node for quick use and a QwenVL (Advanced) node with fine-grained control over generation.
- Prompt Enhancers: Dedicated text-only prompt enhancers for both HF and GGUF backends.
- Preset & Custom Prompts: Choose from a list of convenient preset prompts or write your own for full control.
- Multi-Model Support: Easily switch between various official Qwen-VL models.
- Automatic Model Download: Models are downloaded automatically on first use.
- Smart Quantization: Balance VRAM and performance with 4-bit, 8-bit, and FP16 options.
- Hardware-Aware: Automatically detects GPU capabilities and prevents errors with incompatible models (e.g., FP8).
- Reproducible Generation: Use the seed parameter to get consistent outputs.
- Memory Management: “Keep Model Loaded” option to retain the model in VRAM for faster processing.
- Image & Video Support: Accepts both single images and video frame sequences as input.
- Intelligent Video Auto-Scaling: Dynamic context token budget estimation for video inputs, preserving native resolution for small clips while preventing context slot overflow and CUDA OOM on high-res videos.
- Robust Error Handling: Provides clear error messages for hardware or memory issues.
- Clean Console Output: Minimal and informative console logs during operation.
- SageAttention Support: GPU-optimized attention mechanism with per-architecture kernels (Ampere, Ada, Hopper, Blackwell).
- Progress Bar: Visual feedback during model loading and generation stages.
- Intelligent Cache Management: Automatically clears VRAM when changing attention modes or quantization settings.
🚀 Installation
-
Clone this repository to your ComfyUI/custom_nodes directory:
cd ComfyUI/custom\_nodes
git clone https://github.com/1038lab/ComfyUI-QwenVL.git
-
Install the required dependencies:
cd ComfyUI/custom\_nodes/ComfyUI-QwenVL
pip install \-r requirements.txt
-
Restart ComfyUI.
Optional: SageAttention Support
For optimal performance on supported GPUs, install SageAttention:
pip install sageattention
🧭 Node Overview
- QwenVL: Quick vision-language inference (image/video + preset/custom prompts).
- QwenVL (Advanced): Full control over sampling, device, and performance settings.
- QwenVL Prompt Enhancer: Text-only prompt enhancement (supports both Qwen3 text models and QwenVL models in text mode).
Utilities Nodes
- HuggingFace Downloader: Directly download GGUF/HF models or entire repositories from HuggingFace to your ComfyUI directories.
GGUF (llama.cpp) Nodes
- QwenVL (GGUF): GGUF vision-language inference.
- QwenVL (GGUF Advanced): Extended GGUF controls (context, GPU layers, etc.).
- QwenVL Prompt Enhancer (GGUF): GGUF text-only prompt enhancement.
🧩 GGUF Nodes (llama.cpp backend)
This repo includes GGUF nodes powered by llama-cpp-python (separate from the Transformers-based nodes).
- Nodes:
QwenVL (GGUF), QwenVL (GGUF Advanced), QwenVL Prompt Enhancer (GGUF)
- Model folder (default):
ComfyUI/models/llm/GGUF/ (configurable via gguf_models.json)
- Vision requirement: install a vision-capable
llama-cpp-python wheel that provides Qwen3VLChatHandler / Qwen25VLChatHandler
See docs/LLAMA_CPP_PYTHON_VISION_INSTALL.md
🗂️ Config Files
- HF models:
hf_models.json
hf_vl_models: vision-language models (used by QwenVL nodes).hf_text_models: text-only models (used by Prompt Enhancer).
- GGUF models:
gguf_models.json
- System prompts:
system_prompts.json (includes both VL prompts and prompt-enhancer styles).
📥 Download Models
The models will be automatically downloaded on first use. If you prefer to download them manually, place them in the ComfyUI/models/LLM/Qwen-VL/ directory.
HF Vision Models (Qwen-VL)
HF Text Models (Qwen3)
GGUF Models (Manual Download)
| Group |
Model |
Repo |
Alt Repo |
Model Files |
MMProj |
| Qwen text (GGUF) |
Qwen3-4B-GGUF |
Qwen/Qwen3-4B-GGUF |
|
Qwen3-4B-Q4_K_M.gguf, Qwen3-4B-Q5_0.gguf, Qwen3-4B-Q5_K_M.gguf, Qwen3-4B-Q6_K.gguf, Qwen3-4B-Q8_0.gguf |
|
| Qwen-VL (GGUF) |
Qwen3-VL-4B-Instruct-GGUF |
Qwen/Qwen3-VL-4B-Instruct-GGUF |
|
Qwen3VL-4B-Instruct-F16.gguf, Qwen3VL-4B-Instruct-Q4_K_M.gguf, Qwen3VL-4B-Instruct-Q8_0.gguf |
mmproj-Qwen3VL-4B-Instruct-F16.gguf |
| Qwen-VL (GGUF) |
Qwen3-VL-8B-Instruct-GGUF |
Qwen/Qwen3-VL-8B-Instruct-GGUF |
|
Qwen3VL-8B-Instruct-F16.gguf, Qwen3VL-8B-Instruct-Q4_K_M.gguf, Qwen3VL-8B-Instruct-Q8_0.gguf |
mmproj-Qwen3VL-8B-Instruct-F16.gguf |
| Qwen-VL (GGUF) |
Qwen3-VL-4B-Thinking-GGUF |
Qwen/Qwen3-VL-4B-Thinking-GGUF |
|
Qwen3VL-4B-Thinking-F16.gguf, Qwen3VL-4B-Thinking-Q4_K_M.gguf, Qwen3VL-4B-Thinking-Q8_0.gguf |
mmproj-Qwen3VL-4B-Thinking-F16.gguf |
| Qwen-VL (GGUF) |
Qwen3-VL-8B-Thinking-GGUF |
Qwen/Qwen3-VL-8B-Thinking-GGUF |
|
Qwen3VL-8B-Thinking-F16.gguf, Qwen3VL-8B-Thinking-Q4_K_M.gguf, Qwen3VL-8B-Thinking-Q8_0.gguf |
mmproj-Qwen3VL-8B-Thinking-F16.gguf |
| Qwen-VL (GGUF) |
Qwen3.5-VL-7B-Instruct-GGUF |
Qwen/Qwen3.5-VL-7B-Instruct-GGUF |
|
Qwen3.5VL-7B-Instruct-F16.gguf, Qwen3.5VL-7B-Instruct-Q4_K_M.gguf, Qwen3.5VL-7B-Instruct-Q8_0.gguf |
mmproj-Qwen3.5VL-7B-Instruct-F16.gguf |
| Qwen-VL (GGUF) |
Qwen3.6-VL-MoE-Instruct-GGUF |
Qwen/Qwen3.6-VL-MoE-Instruct-GGUF |
|
Qwen3.6VL-MoE-Instruct-F16.gguf, Qwen3.6VL-MoE-Instruct-Q4_K_M.gguf, Qwen3.6VL-MoE-Instruct-Q8_0.gguf |
mmproj-Qwen3.6VL-MoE-Instruct-F16.gguf |
| Qwen-VL (GGUF) |
Qwen3.8-VL-14B-Instruct-GGUF |
Qwen/Qwen3.8-VL-14B-Instruct-GGUF |
|
Qwen3.8VL-14B-Instruct-F16.gguf, Qwen3.8VL-14B-Instruct-Q4_K_M.gguf, Qwen3.8VL-14B-Instruct-Q8_0.gguf |
mmproj-Qwen3.8VL-14B-Instruct-F16.gguf |
📖 Usage
Basic Usage
- Add the “QwenVL” node from the 🧪AILab/QwenVL category.
- Select the model_name you wish to use.
- Connect an image or video (image sequence) source to the node.
- Write your prompt using the preset or custom field.
- Run the workflow.
Advanced Usage
For more control, use the “QwenVL (Advanced)” node. This gives you access to detailed generation parameters like temperature, top_p, beam search, and device selection.
⚙️ Parameters
| Parameter |
Description |
Default |
Range |
Node(s) |
| model_name |
The Qwen-VL model to use. |
Qwen3-VL-4B-Instruct |
- |
Standard & Advanced |
| quantization |
On-the-fly quantization. Ignored for pre-quantized models (e.g., FP8). |
8-bit (Balanced) |
4-bit, 8-bit, None |
Standard & Advanced |
| attention_mode |
Attention mechanism: auto (Sage→Flash→SDPA), sage, flash_attention_2, sdpa |
auto |
auto, sage, flash_attention_2, sdpa |
Standard & Advanced |
| preset_prompt |
A selection of pre-defined prompts for common tasks. |
“Describe this…” |
Any text |
Standard & Advanced |
| custom_prompt |
Overrides the preset prompt if provided. |
|
Any text |
Standard & Advanced |
| max_tokens |
Maximum number of new tokens to generate. |
1024 |
64-2048 |
Standard & Advanced |
| keep_model_loaded |
Keep the model in VRAM for faster subsequent runs. |
True |
True/False |
Standard & Advanced |
| seed |
A seed for reproducible results. |
1 |
1 - 2^64-1 |
Standard & Advanced |
| temperature |
Controls randomness. Higher values = more creative. (Used when num_beams is 1). |
0.6 |
0.1-1.0 |
Advanced Only |
| top_p |
Nucleus sampling threshold. (Used when num_beams is 1). |
0.9 |
0.0-1.0 |
Advanced Only |
| num_beams |
Number of beams for beam search. > 1 disables temperature/top_p sampling. |
1 |
1-10 |
Advanced Only |
| repetition_penalty |
Discourages repeating tokens. |
1.2 |
0.0-2.0 |
Advanced Only |
| frame_count |
Number of frames to sample from the video input. |
16 |
1-64 |
Advanced Only |
| device |
Override automatic device selection. |
auto |
auto, cuda, cpu |
Advanced Only |
| use_torch_compile |
Enable torch.compile optimization for faster inference. |
False |
True/False |
Advanced Only |
💡 Quantization Options
| Mode |
Precision |
Memory Usage |
Speed |
Quality |
Recommended For |
| None (FP16) |
16-bit Float |
High |
Fastest |
Best |
High VRAM GPUs (16GB+) |
| 8-bit (Balanced) |
8-bit Integer |
Medium |
Fast |
Very Good |
Balanced performance (8GB+) |
| 4-bit (VRAM-friendly) |
4-bit Integer |
Low |
Slower* |
Good |
Low VRAM GPUs (<8GB) |
* Note on 4-bit Speed: 4-bit quantization significantly reduces VRAM usage but may result in slower performance on some systems due to the computational overhead of real-time dequantization.
🎯 Attention Mode Guide
| Mode |
Description |
Best For |
| auto |
Automatically selects best available: Sage → Flash → SDPA |
Most users (recommended) |
| sage |
SageAttention with GPU-optimized kernels |
Speed on modern GPUs (RTX 40 series, Hopper, Blackwell) |
| flash_attention_2 |
Flash Attention 2 |
Speed when Sage unavailable |
| sdpa |
PyTorch SDPA (default) |
Compatibility, FP8/BitsAndBytes models |
Note: FP8 models and BitsAndBytes quantization automatically use SDPA regardless of selection.
🤔 Setting Tips
| Setting |
Recommendation |
| Model Choice |
For most users, Qwen3-VL-4B-Instruct is a great starting point. If you have a 40-series GPU, try the -FP8 version for better performance. |
| Memory Mode |
Keep keep_model_loaded enabled (True) for the best performance if you plan to run the node multiple times. Disable it only if you are running out of VRAM for other nodes. |
| Quantization |
Start with the default 8-bit. If you have plenty of VRAM (>16GB), switch to None (FP16) for the best speed and quality. If you are low on VRAM, use 4-bit. |
| Attention Mode |
Use “auto” for best performance. SageAttention provides fastest inference on supported GPUs. |
| Performance |
The first time a model is loaded with a specific quantization, it may be slow. Subsequent runs (with keep_model_loaded enabled) will be much faster. |
🧠 About Model
This node utilizes the Qwen-VL series of models, developed by the Qwen Team at Alibaba Cloud. These are powerful, open-source large vision-language models (LVLMs) designed to understand and process both visual and textual information, making them ideal for tasks like detailed image and video description.
🗺️ Roadmap
✅ Completed (v2.1.0)
- ✅ SageAttention support with per-GPU architecture optimization
- ✅ Improved FP8 model handling with automatic SDPA fallback
- ✅ Smart attention selection (auto: Sage → Flash → SDPA)
- ✅ Progress bar for model loading and generation
- ✅ Better memory management and cache clearing
✅ Completed (v2.0.0)
- ✅ GGUF model support via llama.cpp backend
- ✅ Prompt Enhancer nodes for text-only optimization
✅ Completed (v1.0.0)
- ✅ Support for Qwen3-VL and Qwen2.5-VL models.
- ✅ Automatic model downloading and management.
- ✅ On-the-fly 4-bit, 8-bit, and FP16 quantization.
- ✅ Hardware compatibility checks for FP8 models.
- ✅ Image and Video (frame sequence) input support.
🙏 Credits
- Qwen Team: Alibaba Cloud - For developing and open-sourcing the powerful Qwen-VL models.
- ComfyUI: comfyanonymous - For the incredible and extensible ComfyUI platform.
- llama-cpp-python: JamePeng/llama-cpp-python - GGUF backend with vision support used by the GGUF nodes.
- SageAttention: SageAttention - Efficient attention implementation with GPU-optimized kernels.
- ComfyUI Integration: 1038lab - Developer of this custom node.
📜 License
This repository’s code is released under the GPL-3.0 License.
评论