ID-LoRA-LTX2.3-ComfyUI

This package wraps the inference pipelines as ComfyUI nodes. Supports both one-stage (single resolution) and two-stage (2x spatial upsampling) pipelines.

One-stage — generates at a single resolution (lower quality):

IDLoraModelLoader --> IDLoraPromptEncoder --> IDLoraOneStageSampler --> SaveVideo
                                                  ^          ^
                                            first_frame  reference_audio

Two-stage — generates at target resolution then refines at 2x with spatial upsampling (higher quality):

IDLoraTwoStageModelLoader --> IDLoraPromptEncoder --> IDLoraTwoStageSampler --> SaveVideo
                                                          ^          ^
                                                    first_frame  reference_audio

The two-stage pipeline produces higher-resolution output (e.g. 512x512 -> 1024x1024) by:

1. Stage 1: Generating video+audio at target resolution with ID-LoRA + full guidance (CFG, STG, identity guidance, A/V bimodal)
2. Transition: Freeing stage-1 models, 2x spatial upsampling the video latent, re-encoding the first frame at 2x
3. Stage 2: Refining at 2x resolution with a distilled LoRA only (no guidance), audio frozen from stage 1

An optional HQ mode uses the res2s second-order sampler instead of Euler for higher quality.

Requirements

• GPU: NVIDIA GPU with >=24 GB VRAM (48 GB recommended for non-quantized; 80 GB recommended for two-stage non-quantized at high resolutions)
• Python: 3.10+
• ComfyUI: Recent version with comfy_api.latest support
• Disk: ~67 GB for one-stage models, ~75 GB for two-stage models (additional upsampler + distilled LoRA)

Installation

0. Install ComfyUI dependencies

If you haven’t done so already, install ComfyUI’s own requirements first:

pip install -r ComfyUI/requirements.txt

Important: ComfyUI may install transformers 5.x, which is incompatible with ID-LoRA. After installing ComfyUI requirements, pin to the latest 4.x release:

pip install 'transformers>=4.52,<5'

1. Clone ID-LoRA

This package depends on the ID-LoRA repo for its Python packages and model download script. ID-LoRA-2.3 is a subdirectory inside it.

git clone https://github.com/ID-LoRA/ID-LoRA.git

2. Download models

Run the download script from the repository root:

bash ID-LoRA/ID-LoRA-2.3/scripts/download_models.sh models/

This downloads all required models from HuggingFace:

For two-stage, also download (included in the download script):

Note: Gemma requires accepting the license at https://huggingface.co/google/gemma-3-12b-it-qat-q4_0-unquantized and logging in with huggingface-cli login before downloading.

3. Symlink models into ComfyUI

The nodes use ComfyUI’s standard model folder system. Create symlinks so ComfyUI can find the downloaded models (adjust the models/ path if you downloaded elsewhere):

# Checkpoint
ln -sf "$(pwd)/models/ltx-2.3-22b-dev.safetensors" ComfyUI/models/checkpoints/

# LoRAs
ln -sf "$(pwd)/models/id-lora-celebvhq-ltx2.3/lora_weights.safetensors" ComfyUI/models/loras/id-lora-celebvhq-ltx2.3.safetensors
ln -sf "$(pwd)/models/id-lora-talkvid-ltx2.3/lora_weights.safetensors" ComfyUI/models/loras/id-lora-talkvid-ltx2.3.safetensors
ln -sf "$(pwd)/models/ltx-2.3-22b-distilled-lora-384.safetensors" ComfyUI/models/loras/

# Upscaler
ln -sf "$(pwd)/models/ltx-2.3-spatial-upscaler-x2-1.1.safetensors" ComfyUI/models/upscale_models/

# Text encoder
ln -sf "$(pwd)/models/gemma-3-12b-it-qat-q4_0-unquantized" ComfyUI/models/text_encoders/

If you already have these models in ComfyUI’s model directories (e.g. from other LTX nodes), the symlinks are not needed — the nodes will find them automatically via the dropdown selectors.

After setup, ComfyUI’s model directories should contain:

ComfyUI/models/
├── checkpoints/
│   └── ltx-2.3-22b-dev.safetensors -> .../models/ltx-2.3-22b-dev.safetensors
├── loras/
│   ├── id-lora-celebvhq-ltx2.3.safetensors -> .../models/id-lora-celebvhq-ltx2.3/lora_weights.safetensors
│   ├── id-lora-talkvid-ltx2.3.safetensors -> .../models/id-lora-talkvid-ltx2.3/lora_weights.safetensors
│   └── ltx-2.3-22b-distilled-lora-384.safetensors -> .../models/ltx-2.3-22b-distilled-lora-384.safetensors
├── upscale_models/
│   └── ltx-2.3-spatial-upscaler-x2-1.1.safetensors -> .../models/ltx-2.3-spatial-upscaler-x2-1.1.safetensors
└── text_encoders/
    └── gemma-3-12b-it-qat-q4_0-unquantized -> .../models/gemma-3-12b-it-qat-q4_0-unquantized

4. Install the ltx packages

pip install -e ID-LoRA/ID-LoRA-2.3/packages/ltx-core
pip install -e ID-LoRA/ID-LoRA-2.3/packages/ltx-pipelines
pip install -e ID-LoRA/ID-LoRA-2.3/packages/ltx-trainer

5. Clone this repo into ComfyUI custom_nodes

cd ComfyUI/custom_nodes
git clone <this-repo-url> comfyui-id-lora-ltx

6. Start ComfyUI

python ComfyUI/main.py

Then open http://127.0.0.1:8188 in your browser. The nodes will appear under the ID-LoRA category in the node menu.

Example workflow templates

Two ready-to-use workflows are included in the example_workflows/ directory:

• id_lora_one_stage.json — one-stage pipeline
• id_lora_two_stage.json — two-stage pipeline with 2x upsampling

In ComfyUI, go to Browse Templates and look for comfyui-id-lora-ltx to load them. They wire all nodes together with LoadImage, LoadAudio, and SaveVideo.

The example workflows use poster_image.png (first frame) and reference.mp3 (speaker reference audio), which are included in the example_inputs/ directory. Copy them into ComfyUI’s input/ folder so the LoadImage and LoadAudio nodes can find them:

cp ComfyUI/custom_nodes/ID-LoRA-LTX2.3-ComfyUI/example_inputs/poster_image.png ComfyUI/input/
cp ComfyUI/custom_nodes/ID-LoRA-LTX2.3-ComfyUI/example_inputs/reference.mp3 ComfyUI/input/

Then load either workflow template from ComfyUI and click Queue Prompt to run.

Nodes

ID-LoRA Model Loader

Loads the base LTX-2.3 checkpoint, Gemma text encoder, and ID-LoRA weights into a one-stage pipeline object. This is the slow/expensive node — ComfyUI caches its output when inputs don’t change.

Output: ID_LORA_PIPELINE

ID-LoRA Two-Stage Model Loader

Loads the base checkpoint, text encoder, ID-LoRA, spatial upsampler, and distilled LoRA for two-stage generation. Like the one-stage loader, ComfyUI caches this node.

Output: ID_LORA_PIPELINE

ID-LoRA Prompt Encoder

Encodes positive and negative text prompts into conditioning tensors. Works with both one-stage and two-stage model loaders. Loads the text encoder temporarily, encodes, then frees it to save VRAM.

Output: ID_LORA_CONDITIONING

ID-LoRA One-Stage Sampler

Runs the one-stage generation pipeline: denoising loop with identity guidance, then decodes video and audio. Outputs a ComfyUI VIDEO that you wire directly to the built-in Save Video node.

Output: VIDEO (wire to Save Video)

ID-LoRA Two-Stage Sampler

Runs the two-stage generation pipeline. Stage 1 generates at the specified resolution with full guidance, then stage 2 refines at 2x resolution with the distilled LoRA. Output resolution is 2x the specified height/width.

Output: VIDEO (wire to Save Video)

Prompt format

Prompts use a structured [VISUAL] / [SPEECH] / [SOUNDS] format:

[VISUAL]: A medium shot of a young man with curly brown hair, sitting on a beige couch.
He is wearing a light blue shirt and speaking warmly.
[SPEECH]: We are proud to introduce ID-LoRA.
[SOUNDS]: The speaker has a moderate volume and conversational tone. Light instrumental
background music plays softly.

All three sections are optional but recommended for best results.

Example workflows

One-stage

1. Add ID-LoRA Model Loader — set paths to checkpoint, text encoder, and LoRA
2. Add ID-LoRA Prompt Encoder — connect the pipeline, write your prompt
3. Add Load Image + Load Audio — load face image and reference audio
4. Add ID-LoRA One-Stage Sampler — connect everything
5. Add Save Video — connect the video output

Two-stage (2x upsampling)

1. Add ID-LoRA Two-Stage Model Loader — set paths (including upsampler and distilled LoRA)
2. Add ID-LoRA Prompt Encoder — connect the pipeline, write your prompt
3. Add Load Image + Load Audio — load face image and reference audio
4. Add ID-LoRA Two-Stage Sampler — connect everything, optionally enable hq_mode
5. Add Save Video — connect the video output

The output resolution will be 2x the stage 1 resolution (e.g. 512x512 stage 1 -> 1024x1024 output).

Reducing memory usage

The full pipeline is memory-intensive — the 22B parameter transformer alone requires significant VRAM. If you run into out-of-memory errors, here are several ways to reduce usage, roughly ordered from least to most impact on quality:

1. Enable quantization — Set quantize to int8 or fp8 on the Model Loader node. This reduces the transformer’s memory footprint substantially with a small quality tradeoff. int8 is recommended as a good balance; fp8 saves slightly more memory but may introduce more artifacts.

2. Lower max_resolution — Reducing from 768 to 512 (or lower) significantly reduces VRAM during generation, since memory scales quadratically with resolution. For two-stage, this also reduces the stage-2 resolution proportionally (e.g. 512 → 1024 output instead of 768 → 1536).

3. Reduce num_frames — Fewer frames means less latent state in memory. Try 121 instead of 242 if memory is tight.

4. Disable HQ mode (two-stage only) — The res2s sampler uses more memory than Euler due to its second-order midpoint evaluation. Turning off hq_mode saves memory at the cost of some quality.

5. Use one-stage instead of two-stage — The two-stage pipeline requires loading two separate transformers sequentially and upsampling the latent to 2x, which has higher peak memory than one-stage.

These options can be combined. For example, int8 quantization + max_resolution=512 + one-stage should run on 24 GB GPUs.

Notes

• Model paths use ComfyUI’s standard models/ folder system. Models are selected via dropdown menus that scan ComfyUI/models/checkpoints/, loras/, upscale_models/, and text_encoders/. The text encoder path is a string field — leave it empty to auto-detect a Gemma directory from text_encoders/.
• Auto-resolution (enabled by default) matches the first frame’s aspect ratio while capping the long side at max_resolution (default 512px). To generate at higher resolution, increase max_resolution on the sampler node — the aspect ratio is always preserved. For the two-stage pipeline, this controls stage 1; the final output will be 2x that (e.g. max_resolution=768 produces up to 1536px output).
• Two-stage VRAM: Stage-1 models (transformer, audio encoder) are fully freed before loading stage-2 models. The video encoder is shared for upsampling then freed.
• HQ mode (two-stage only): Uses the res2s second-order sampler instead of Euler in both stages for higher quality at the cost of speed.
• Reusable pipeline (one-stage only): The video encoder is stashed to CPU (not deleted) after encoding, so re-running with different prompts/seeds doesn’t require a full reload.

File structure

comfyui-id-lora-ltx/
├── __init__.py              # Extension entry point (registers all 5 nodes)
├── nodes_model_loader.py    # IDLoraModelLoader + IDLoraTwoStageModelLoader nodes
├── nodes_prompt_encoder.py  # IDLoraPromptEncoder node (shared)
├── nodes_sampler.py         # IDLoraOneStageSampler + IDLoraTwoStageSampler nodes
├── pipeline_wrapper.py      # _IDLoraBase, IDLoraOneStagePipeline, IDLoraTwoStagePipeline
├── example_inputs/
│   ├── poster_image.png         # Example first-frame image
│   └── reference.mp3            # Example reference audio
├── example_outputs/
│   └── two_stage_demo.mp4       # Sample two-stage output
├── example_workflows/
│   ├── id_lora_one_stage.json   # One-stage workflow template
│   └── id_lora_two_stage.json   # Two-stage workflow template
├── pyproject.toml           # Package metadata
├── requirements.txt         # Dependencies
└── README.md