vk_gltf_renderer

Vulkan RTX Path Tracer — glTF Scene Editor & PBR Material Reference

Open-source Vulkan RTX path tracer and PBR material reference for glTF 2.0 — with a built-in glTF scene editor for non-destructive hierarchy manipulation, material authoring, and save-back to glTF; AI denoising, and support for 36+ glTF extensions.

Vulkan RTX Path Tracer & glTF Scene Editor

A production-quality Vulkan ray tracing renderer and glTF PBR material reference for glTF 2.0 scenes. Comes with a built-in glTF scene editor — edit scene hierarchies, author PBR materials, apply transforms with a gizmo, and save back to glTF — backed by a high-fidelity RTX path tracer with DLSS Ray Reconstruction and OptiX AI Denoiser. Built for graphics developers who want a reference they can study, profile, and extend.

Built in C++ on nvpro_core2 with Slang shaders. Successor to vk_raytrace.

Highlights

• glTF scene editor — Non-destructive scene authoring: hierarchy manipulation, transform gizmo, node/material/light editing, undo/redo, and save back to glTF without touching the original assets.
• glTF PBR material reference — Monte Carlo path tracer with NEE, MIS, and adaptive sampling for physically accurate glTF PBR material evaluation.
• 35 glTF extensions — Anisotropy, clearcoat, transmission, volume, sheen, iridescence, dispersion, retroreflection, Draco, interactivity, opacity micromaps, IES light profiles, and more.
• glTF 2.1 complex scenes (preview) — Compose multi-file scenes with External Assets: reference glTF/GLB files as instances that share geometry, with nested references, cycle detection, and file aliases — re-externalized on save.
• Caustics (beta) — The bright, focused light that glass, water and polished metal throw on the surfaces around them, which a plain path tracer cannot make. A per-frame photon map adds it to the path-traced image (off by default; see Caustics).
• AI denoising — DLSS Ray Reconstruction, OptiX AI Denoiser and Open Image Denoise (on Vulkan, through vk_oidn) produce clean images at interactive rates.
• Agentic AI generation (optional) — Local ComfyUI bridge that beautifies the current render or generates an HDRI environment from a text prompt.
• Rasterizer preview — Fast PBR rasterizer shares scene resources for instant iteration during editing.
• Developer tools — GPU profiler, memory tracker, shader hot-reload (Ctrl+Shift+R), headless batch mode.

Quick Tour

The demo shows a short end-to-end workflow: switching renderer modes, tuning settings, and inspecting scene content in the integrated editor.

Build and Run

Requirements

Requirement	Minimum	Recommended
OS	Windows 10 / Linux	Windows 11 / Ubuntu 22.04+
GPU	NVIDIA RTX 20-series (Turing)	NVIDIA RTX 40-series (Ada)
Driver	535+	Latest Game Ready / Studio
CMake	3.22	3.28+
C++ Compiler	C++20 (MSVC 2022 / GCC 12 / Clang 15)	MSVC 2022 17.8+
Vulkan SDK	1.4	Latest

Quick start

# Clone (recommended: siblings; CMake auto-downloads nvpro_core2 if missing)
git clone https://github.com/nvpro-samples/nvpro_core2.git
git clone https://github.com/nvpro-samples/vk_gltf_renderer.git
cd vk_gltf_renderer

# Windows
cmake -B build -S . -A x64
cmake --build build --config Release -j 8
.\_bin\Release\vk_gltf_renderer.exe

# Linux
cmake -B build -S . -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build
./_bin/Release/vk_gltf_renderer

Common CMake options

Option	Default	Description
USE_DLSS	ON	Enable DLSS Ray Reconstruction integration
USE_OPTIX_DENOISER	ON	Enable OptiX AI Denoiser (requires CUDA Toolkit)
USE_OIDN_DENOISER	ON	Enable the Open Image Denoise denoiser (downloads the vk_oidn SDK package)
USE_DRACO	ON	Enable Draco mesh compression support
BUILD_TESTING	OFF	Build unit tests and benchmarks
SHADER_DEBUG_LEVEL	2	Slang debug info in the embedded shaders. 2+ enables Nsight source-level shader debugging but crashes AMD’s Vulkan driver; release binaries are built with 0

Features

• Ray tracing: High-quality path tracing reference for glTF PBR materials — Monte Carlo global illumination, next event estimation, multiple importance sampling, and adaptive sampling.
• Caustics (beta): light focused by glass, water, or polished metal, from a per-frame photon map (Settings > Path Tracer > Caustics; see Caustics).
• Profiler, GPU monitor, GPU memory tracking, and statistics.
• Shader hot-reload (Ctrl+Shift+R) for developers who want to experiment.
• AI denoisers: DLSS Ray Reconstruction, OptiX AI Denoiser and Open Image Denoise (vk_oidn).
• Optional agentic AI generation via a local ComfyUI bridge: beautify the current render, or generate an HDRI environment from a text prompt (see ComfyUI Agentic Setup).
• Rasterizer fallback for fast scene interaction and editing.
• A scene asset editor with hierarchy operations, a transform gizmo, material editing, merging, and saving back to glTF (non-destructive).
• glTF 2.1 complex-scene composition (preview): reference external glTF/GLB assets, instance them (shared geometry), resolve nested references with cycle detection, and re-externalize on save.
• Support for 36 glTF extensions, including anisotropy, clearcoat, transmission, volume, sheen, iridescence, dispersion, diffuse transmission, retroreflection, opacity micromaps, material variant, scattering, IES light profiles, and interactivity (behavior graphs).
• HDR environments, a physical sun and sky model, depth of field, and multiple tone mappers.
• Animation support includes skeletal, morph targets, and KHR_animation_pointer.
• GPU compute accelerates both skinning/morphing and per-level world-matrix propagation.

More features --> User Guide

Showcase

Feature	Preview
Showcase
Material features
Lighting and camera
Caustics (beta), off then on

For a full walkthrough of rendering modes, editor workflows, and feature screenshots, see the User Guide.

For headless timing and optional scripted GPU benchmarks, see Benchmarking (utils/benchmark/).

To let an AI agent measure a shader change against the running renderer, launch with --mcp and see MCP shader timing.

Agentic — AI-Assisted Generation

Drive local generative AI from inside the renderer through an optional ComfyUI bridge. Two tools are wired into the in-app Agentic window (press F7):

• Beautify Last Render — send the current path-traced frame to a diffusion model and get it back photorealistic, or restyled by prompt (pencil, cartoon, watercolor, blueprint, and more).
• Generate HDRI From Prompt — type a prompt and get a full 360° HDR environment, applied directly as the scene’s lighting.

Generation is entirely optional — the renderer runs without it.

Showcase

Path-traced renders (left), beautified or restyled by prompt (right):

Original render	AI result

Setup

Generation needs a local ComfyUI install, the diffusion models, and a small Python bridge that connects it to the renderer. The ComfyUI Agentic Setup guide walks through it end to end:

1. Install ComfyUI and download the models.
2. Start ComfyUI.
3. Open the Agentic window (F7) and use Copy start command to launch the bridge in a terminal; once the Bridge and ComfyUI status lights are green, generate.

For the architecture, bridge directory layout, and request/response protocol behind it, see Agentic Workflow.

glTF Support

Reference scope: The list below reflects what this renderer loads and displays. The path tracer is the authoritative PBR implementation — especially for ray-traced material evaluation, sampling, and new extensions (e.g. KHR_materials_retroreflection). The rasterizer is a preview path for interaction, not the primary material reference.

Core

• ✅ glTF 2.0 (.gltf/.glb)
• ✅ Images (HDR, PNG, JPEG, KTX, KTX2, DDS, WebP, OpenEXR)
• ✅ Buffers (geometry, animation, skinning)
• ✅ Textures and samplers
• ✅ Materials (PBR metallic-roughness and specular-glossiness)
• ✅ Animations (keyframe, skeletal)
• ✅ Skins
• ✅ Morph targets
• ✅ Cameras (perspective and orthographic)
• ✅ Punctual lights (directional, point, spot)
• ✅ Nodes and scene hierarchy
• ✅ Multiple scenes

Extensions

✅ Supported · 🧪 Implemented against a specification outside the official glTF extension registry (draft, third-party, or specific to this renderer); it may change

• ✅ KHR_accessor_float64
• ✅ KHR_animation_pointer
• ✅ KHR_draco_mesh_compression
• ✅ KHR_interactivity
• ✅ KHR_lights_punctual
• ✅ KHR_materials_anisotropy
• ✅ KHR_materials_clearcoat
• ✅ KHR_materials_diffuse_transmission
• ✅ KHR_materials_dispersion
• ✅ KHR_materials_emissive_strength
• ✅ KHR_materials_ior
• ✅ KHR_materials_iridescence
• ✅ KHR_materials_pbrSpecularGlossiness
• 🧪 KHR_materials_retroreflection
• 🧪 KHR_materials_scatter
• ✅ KHR_materials_sheen
• ✅ KHR_materials_specular
• ✅ KHR_materials_transmission
• ✅ KHR_materials_unlit
• ✅ KHR_materials_variants
• ✅ KHR_materials_volume
• ✅ KHR_mesh_quantization
• ✅ KHR_meshopt_compression
• ✅ KHR_node_hoverability
• ✅ KHR_node_selectability
• ✅ KHR_node_visibility
• ✅ KHR_texture_basisu
• ✅ KHR_texture_transform
• ✅ KHR_xmp_json_ld
• ✅ EXT_lights_ies
• ✅ EXT_mesh_gpu_instancing
• 🧪 EXT_mesh_opacity_micromap
• ✅ EXT_meshopt_compression
• ✅ EXT_texture_webp
• ✅ MSFT_texture_dds
• 🧪 OMI_environment_sky — plain, gradient and physical are rendered and round-trip on save; panorama needs a URI, which the extension does not define, so it is read from the sibling NV_environment_sky_panorama below
• 🧪 NV_environment_sky_panorama — vendor sibling carrying the panorama image URI for OMI_environment_sky
• 🧪 NV_node_shadow_casting — vendor node extension, {"castShadows": false}: the node’s mesh is skipped by the path tracer’s shadow rays but stays visible to every other ray (USD primvars:doNotCastShadows). It applies to the node’s own mesh, not its children
• 🧪 NV_environment_sky_atmosphere — vendor sibling carrying the rest of the physical sky’s atmosphere (solar spectrum, planet size, scale heights, aerosol albedo, ozone layer), which OMI_environment_sky has no fields for

glTF 2.1 (Complex Scenes — preview)

Early support for the glTF 2.1 complex-scene composition features — the standardized successor to the earlier glTFX / External Reference proposals:

• ✅ External Assets — reference other glTF/GLB files from scene nodes (node.externalAsset) and instantiate them at load time
• ✅ Multiple instances of the same asset share geometry (meshes / BLAS), across every file that references it: each file is loaded once per scene, and static content is drawn without copying its nodes
• ✅ Nested external assets — recursive resolution with cycle detection
• ✅ Unified file references — top-level files array (external uri)
• ✅ File aliases — inner-URI redirection for shared or overridden resources
• ✅ Re-externalized on save (references preserved); “Make Editable” embeds an asset inline
• 🚧 Packaging — embedded external assets (bufferView / data: URI) not yet resolved
• 🚧 Shapes and scene-level bounding volumes

See External Assets for design details.

Documentation

For users

• User Guide — renderer settings, PBR materials, scene editor, camera, environment, tone mapping, common CLI flags, and troubleshooting.
• ComfyUI Agentic Setup — set up the local ComfyUI install, models, and bridge to beautify renders and generate HDRIs from prompts.
• Caustics (beta) — what caustics are, how the photon map adds them to the path-traced image, its settings and cost, and its limits.
• glTF Resources — curated collection of glTF models, HDR environments, specifications, and tools.

For contributors

• Developer Guide — architecture overview, source map, material system, and testing.
• Rendering Architecture — data flow from glTF model to GPU, BLAS/TLAS acceleration structures, and render nodes.
• External Assets — glTF 2.1 complex scenes: reference / merge / edit / save mechanics.
• glTF 2.1 — what the renderer supports so far, how it follows a draft spec, the tinygltf fork, and the 2.1 test fixtures.
• Agentic Workflow — architecture, filesystem bridge protocol, and roadmap behind the optional AI generation feature.
• Benchmarking — headless timing and scripted GPU benchmarks.

These docs explain concepts and workflows and point at where things live; enumerated facts (extension lists, CLI flags, enum values) are owned by the code.

License

Apache License 2.0 - Copyright © 2023-2026, NVIDIA CORPORATION.