cudnn-frontend

cuDNN Frontend (FE)

cuDNN Frontend is NVIDIA’s modern, open-source entry point to the cuDNN library and a growing collection of high-performance open-source kernels — scaled dot-product attention (SDPA / Flash Attention), grouped GEMM fusions for Mixture-of-Experts (MoE) training, fused normalization + activation, and more.

It provides a header-only C++ API and a Python interface (with native PyTorch integration) to the cuDNN Graph API, targeting NVIDIA Hopper (H100/H200) and Blackwell (B200/GB200/GB300) GPUs across FP16, BF16, FP8, and MXFP8 precision.

Links: Documentation · Blog & Deep Dives · PyPI · Release Notes · Samples

🚀 Latest news:

We will begin open-sourcing kernels based on customer needs, with the goal to educate developers and enable them to customize as needed.

We are now shipping OSS kernels, allowing you to inspect, modify, and contribute to the core logic. Check out our latest implementations:

• FROST GEMM engine: JIT-compiled Blackwell GEMM engine reachable through the ordinary cudnn.pygraph API — matmul, grouped (MoE) matmul, block-scaled FP4/FP8, and chained pointwise epilogues are fused into one kernel from the graph you already built. Opt in with CUDNN_FRONTEND_ENABLE_FROST_ENGINES=1; it is then a candidate for every matmul graph it can serve, ranked against the backend’s own plans.
• GEMM + Amax: Optimized FP8 matrix multiplication with absolute maximum calculation.
• GEMM + SwiGLU: High-performance implementation of the SwiGLU activation fused with GEMM.
• GEMM + sReLU: High-performance implementation of squared-ReLU fused with GEMM.
• GEMM + dsReLU: High-performance implementation of dsquared-ReLU fused with GEMM.
• Grouped GEMM (BF16): Unfused BF16 grouped GEMM with dense and discrete MoE weight layouts.
• Grouped GEMM + GLU: Unified BF16 and legacy block-scaled grouped GEMM GLU API supporting dense and discrete MoE weight layouts.
• Grouped GEMM + GLU + Hadamard: Dense grouped GEMM GLU forward fusion with a fused Hadamard transform and per-expert AMAX reduction.
• Grouped GEMM + dGLU: Unified BF16 and legacy block-scaled grouped GEMM dGLU backward API supporting dense and discrete MoE weight layouts.
• Grouped GEMM + SwiGLU: SwiGLU activation fused with Grouped GEMM.
• Grouped GEMM + dSwiglu: dSwiglu activation fused with Grouped GEMM.
• Grouped GEMM + sReLU: Contiguous grouped squared-ReLU GEMM for MoE workloads.
• Grouped GEMM + dsReLU: Contiguous grouped dsquared-ReLU GEMM for MoE workloads.
• Discrete Grouped GEMM + SwiGLU: Per-expert-pointer SwiGLU grouped GEMM for MoE workloads without weight packing.
• Discrete Grouped GEMM + dSwiGLU: Per-expert-pointer dSwiGLU backward grouped GEMM for MoE workloads without weight packing.
• Grouped GEMM + Quant: Legacy dense-only grouped GEMM quant API for MoE FC2/dFC1 workloads.
• Grouped GEMM + Quant (Unified): Unified grouped GEMM quant API with per-row gating for MoE FC2/dFC1 workloads.
• Grouped GEMM + Wgrad: Unified BF16 and legacy block-scaled grouped GEMM weight-gradient API supporting dense and discrete output layouts for MoE workloads.
• BSA: Block-sparse attention forward and backward CuTe DSL kernels for block-level routing metadata.
• Flex Attention: SM90/SM100/SM103 CuTe DSL attention with reusable arbitrary interval-mask plans and PyTorch autograd.
• HSTU Attention: Packed variable-length HSTU attention forward and backward CuTe DSL kernels for Blackwell SM100/SM103 GPUs, using SiLU scores without softmax.
• NSA: Native Sparse attention as described in the Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention.
• SDPA Backward: SM100, D=256: SDPA Backward pass for D=256 on SM100.
• Fused RMSNorm + SiLU: Implementation of a fused kernel of RMS normalization followed by SiLU (Swish) activation.
• DSA: DSA/CSA kernels for DSv4 and DSv3.2 for fprop and bprop.

Contributor credits for these OSS CuTe DSL kernels are listed in Acknowledgements.

Tech talks

• See our latest talk on GPU-Mode

  ▶ Watch on YouTube

🔥🔥🔥 SOTA Attention Kernels from cudnn backend

Llama 3.1 style Forward and Bprop with causal masking (GB300)

Deepseek v3 style Forward and Bprop with causal masking (GB300)

New OSS Linear Attention Kernels

GDN Forward and Bprop (GB300)

KDA Forward and Bprop (GB300)

GDN-2 Forward and Bprop (GB300)

GDP Forward and Bprop (GB300)

Mamba-2 Forward and Bprop (GB300)

Key Features

• Unified Graph API: Create reusable, persistent cudnn_frontend::graph::Graph objects to describe complex subgraphs.
• Ease of Use: Simplified C++ and Python bindings (via nanobind) that abstract away the boilerplate of the backend API.
• Performance: Built-in autotuning and support for the latest NVIDIA GPU architectures.

Installation

🐍 Python

The easiest way to get started is via pip:

pip install nvidia-cudnn-frontend

Requirements:

• Python 3.10+
• NVIDIA driver and CUDA Toolkit
• NVIDIA cuDNN (minimum 8.5.0)

⚙️ C++ (Header Only)

Since the C++ API is header-only, integration is seamless. Simply include the header in your compilation unit:

#include <cudnn_frontend.h>

Ensure your include path points to the include/ directory of this repository.

Building from Source

If you want to build the Python bindings from source or run the C++ samples:

1. Dependencies

• python-dev (e.g., apt-get install python-dev)
• Dependencies listed in requirements.txt (pip install -r requirements.txt)

2. Python Source Build

pip install -v git+https://github.com/NVIDIA/cudnn-frontend.git

Environment variables CUDAToolkit_ROOT and CUDNN_PATH can be used to override default paths.

3. C++ Samples Build

mkdir build && cd build
cmake -DCUDNN_PATH=/path/to/cudnn -DCUDAToolkit_ROOT=/path/to/cuda ../
cmake --build . -j16
./bin/samples

Documentation & Examples

• Developer Guide: Official NVIDIA Documentation (latest)
• Blog & Deep Dives: nvidia.github.io/cudnn-frontend — release notes, installation guides, and technical deep-dives (MXFP8 attention, FP8 scale layouts, etc.)
• C++ Samples: See samples/cpp for end-to-end examples covering convolution, matmul, SDPA / Flash Attention, normalization, and more.
• Python Samples: See samples/python for Jupyter notebooks and PyTorch integration patterns.
• OSS Kernels: See python/cudnn/ for source of SDPA, Flex Attention, grouped GEMM + SwiGLU/GLU, RMSNorm + SiLU, Native Sparse Attention, and other open-sourced kernels.
• PyTorch Custom Ops: See python/cudnn/experimental/ops for torch.compile-compatible wrappers around cuDNN kernels.

🤝 Contributing

We strictly welcome contributions! Whether you are fixing a bug, improving documentation, or optimizing one of our new OSS kernels, your help makes cuDNN better for everyone.

1. Check the Contribution Guide for details.
2. Fork the repo and create your branch.
3. Submit a Pull Request.

Debugging

To view the execution flow and debug issues, you can enable logging via environment variables:

# Log to stdout
export CUDNN_FRONTEND_LOG_INFO=1
export CUDNN_FRONTEND_LOG_FILE=stdout

# Log to a file
export CUDNN_FRONTEND_LOG_INFO=1
export CUDNN_FRONTEND_LOG_FILE=execution_log.txt

Logging Levels:

• CUDNN_FRONTEND_LOG_INFO=0: No logging
• CUDNN_FRONTEND_LOG_INFO=1: Full logging with tensor dumps
• CUDNN_FRONTEND_LOG_INFO=10: Basic logging (safe for CUDA graph capture)

Alternatively, you can control logging programmatically via cudnn_frontend::isLoggingEnabled().

OSS engine selection:

The FROST SDPA forward engines for SM100 and SM120 (f16/bf16) are candidates by default, ranked against the cuDNN backend’s own engines per measured shape class (python/cudnn/sdpa/fwd/placement.py: FROST first where it is timed ahead, the backend first where it is not). The other open-source engines are opt-in while they mature: set the flag below and they become candidates for every graph they can serve; the flag also ranks every FROST engine first. Engines that are the only implementation of their operation (GDN/KDA) need no flag.

# Offer the maturing open-source engines (FROST GEMM / SDPA) as plan candidates.
export CUDNN_FRONTEND_ENABLE_FROST_ENGINES=1

# Where JIT-compiled kernels are cached (default: $XDG_CACHE_HOME/cudnn_gemm/kernel_cache).
export CUDNN_FRONTEND_GEMM_KERNEL_CACHE=/path/to/cache

graph.plans is the ranked list and graph.get_plan_name_at_index(i) names each entry — an OSS engine reports its engine name, the backend reports the backend plan name. Pin one with graph.select_plan(i) (strict: that plan runs or the build fails) and exclude by name with the classic graph.deselect_engines([...]). graph.selected_engine is the engine that ran, or None when the backend served the graph.

Environment report

When filing a bug, include the output of the environment collector — it reports the frontend/backend versions, GPU/driver properties, and every cuDNN/CUDA library copy on the system (loaded vs on disk):

python -m cudnn.collect_env

If import cudnn itself fails, download collect_env.py and run it standalone with any Python.

CUDA graph retention

A CUDA graph recorded from a graph (stream capture of execute(), or populate_cuda_graph()) holds references to the execution plans and kernel code it launches, so it stays valid after the graph is destroyed. Launching a CUDA graph that holds such references takes slightly longer. Applications can turn the references off if they keep each graph’s execution plans and host-copy data alive and unchanged until every CUDA graph recorded from it (and every clone and graph exec of one) has finished and been destroyed. Keeping the graph object alive is not enough if it is rebuilt or deserialized again in the meantime, since that replaces them:

graph.set_cuda_graph_retention(cudnn_frontend::CudaGraphRetention_t::NONE);  // before building plans

graph = cudnn.pygraph(..., cuda_graph_retention=cudnn.cuda_graph_retention.NONE)

or for the whole process, for graphs that do not set it:

export CUDNN_CUDA_GRAPH_RETENTION=none   # default: per_graph

The variable is shared with the cuDNN backend, so with cuDNN 9.28 or newer it turns off the backend’s references too; the graph setting is also passed to the backend. With cuDNN 9.27, whose backend always keeps its references, only the frontend’s are removed.

Overriding the CUDA runtime library

On Linux, when the frontend is built with dynamic loading enabled, it locates the CUDA runtime (libcudart.so.*) by searching CUDA 12 followed by CUDA 13. In some environments (for example, containers such as GKE where the TCPXO NCCL plugin mounts a different libcudart major version from the host) multiple versions of libcudart may be visible on the library search path. Automatic detection warns with Multiple libcudart libraries found and keeps the first match.

To select a particular runtime, set CUDNN_FRONTEND_CUDART_LIB_NAME to the library name or full path. This bypasses automatic detection:

export CUDNN_FRONTEND_CUDART_LIB_NAME=libcudart.so.13
# or an absolute path
export CUDNN_FRONTEND_CUDART_LIB_NAME=/usr/local/cuda/lib64/libcudart.so.13

On Windows, dynamic loading currently requires CUDA 13 headers and a CUDA 13 runtime (cudart64_13.dll). The loader calls cudaRuntimeGetVersion and rejects other runtime majors, including when using an explicit override. Support for additional Windows CUDA versions can be added separately. The loader searches registered DLL directories, including those added through Python’s os.add_dll_directory(), and retains a PATH fallback. Keep Python DLL-directory handles alive for as long as library discovery may occur.

$env:CUDNN_FRONTEND_CUDART_LIB_NAME = "cudart64_13.dll"
# or the full path to the selected CUDA 13 runtime DLL

License

This project is distributed primarily under the Apache License 2.0. A subset of files remain under the MIT License; each source file declares its license with an SPDX SPDX-License-Identifier: tag. See LICENSING.md for the full list of MIT-licensed files and the rationale, and THIRD_PARTY_LICENSES.txt for third-party attributions.