coderonion/awesome-cuda-and-hpc 开源项目中文解读：技术栈、适用场景与上手建议

项目介绍

coderonion/awesome-cuda-and-hpc 是一个托管在 GitHub 上的开源项目，目前获得 524 个 Star、50 次 Fork。

仓库全名: coderonion/awesome-cuda-and-hpc

README 摘要

Awesome-CUDA-and-HPC 🚀🚀🚀 This repository lists some awesome public CUDA, cuda-python, cuBLAS, CUTLASS, cuDNN, NCCL, TensorRT, TensorRT-LLM, Triton, TileLang, TVM, MLIR, LLVM, PTX and High Performance Computing (HPC) projects. Contents Awesome-CUDA-and-HPC Official Version Awesome List Learning Resources CUDA Learning TensorRT Learning Triton Learning TileLang Learning TVM Learning MLIR Learning LLVM Learning HPC…

完整内容见 GitHub 原始 README 或本页。

常见问题

如何获取 awesome-cuda-and-hpc 的源码？

可以直接克隆仓库：git clone https://github.com/coderonion/awesome-cuda-and-hpc.git，默认分支为 main；安装与运行步骤见仓库 README。

awesome-cuda-and-hpc 最近还在维护吗？

仓库最近一次更新为 2025年8月2日，当前有 6 个未关闭 Issue；Star 数为 524，可作为社区活跃度的参考。

关于

🚀🚀🚀 This repository lists some awesome public CUDA, cuda-python, cuBLAS, cuDNN, CUTLASS, TensorRT, TensorRT-LLM, Triton, TVM, MLIR, PTX and High Performance Computing (HPC) projects.

awesome blas cublas cuda cudnn cutlass deepseek gemm gpu hpc llama llm mlir ptx pytorch tensorrt tensorrt-llm triton tvm vlm

524 Stars
50 Forks
7 Watching

描述取自 GitHub 官方 API，中文内容由 GitRead 中文社区编辑团队整理。

简要分析

热度分

631

价值分

329

活跃状态

活跃

GitHub

贡献者

coderonion

Awesome-CUDA-and-HPC

🚀🚀🚀 This repository lists some awesome public CUDA, cuda-python, cuBLAS, CUTLASS, cuDNN, NCCL, TensorRT, TensorRT-LLM, Triton, TileLang, TVM, MLIR, LLVM, PTX and High Performance Computing (HPC) projects.

Awesome-CUDA-and-HPC

Official Version

CUDA : CUDA is a parallel computing platform and programming model developed by NVIDIA for general computing on graphical processing units (GPUs).
NVIDIA/cuda-python : CUDA Python: Performance meets Productivity. nvidia.github.io/cuda-python/
cuBLAS : Basic Linear Algebra on NVIDIA GPUs. NVIDIA cuBLAS is a GPU-accelerated library for accelerating AI and HPC applications. It includes several API extensions for providing drop-in industry standard BLAS APIs and GEMM APIs with support for fusions that are highly optimized for NVIDIA GPUs. The cuBLAS library also contains extensions for batched operations, execution across multiple GPUs, and mixed- and low-precision execution with additional tuning for the best performance.
cuDNN : The NVIDIA CUDA Deep Neural Network library (cuDNN) is a GPU-accelerated library of primitives for deep neural networks. cuDNN provides highly tuned implementations for standard routines such as forward and backward convolution, attention, matmul, pooling, and normalization.
CUTLASS : CUDA Templates for Linear Algebra Subroutines. CUTLASS is a collection of CUDA C++ template abstractions for implementing high-performance matrix-matrix multiplication (GEMM) and related computations at all levels and scales within CUDA. It incorporates strategies for hierarchical decomposition and data movement similar to those used to implement cuBLAS and cuDNN.
TensorRT : NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT. developer.nvidia.com/tensorrt
TensorRT-LLM : TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines. nvidia.github.io/TensorRT-LLM
Triton : Development repository for the Triton language and compiler. This is the development repository of Triton, a language and compiler for writing highly efficient custom Deep-Learning primitives. The aim of Triton is to provide an open-source environment to write fast code at higher productivity than CUDA, but also with higher flexibility than other existing DSLs. triton-lang.org/
TileLang : Domain-specific language designed to streamline the development of high-performance GPU/CPU/Accelerators kernels. Tile Language (tile-lang) is a concise domain-specific language designed to streamline the development of high-performance GPU/CPU kernels (e.g., GEMM, Dequant GEMM, FlashAttention, LinearAttention). By employing a Pythonic syntax with an underlying compiler infrastructure on top of TVM, tile-lang allows developers to focus on productivity without sacrificing the low-level optimizations necessary for state-of-the-art performance. tilelang.com/
TVM : Open deep learning compiler stack for cpu, gpu and specialized accelerators. tvm.apache.org/
MLIR : Multi-Level Intermediate Representation Compiler Framework. The MLIR project is a novel approach to building reusable and extensible compiler infrastructure. MLIR aims to address software fragmentation, improve compilation for heterogeneous hardware, significantly reduce the cost of building domain specific compilers, and aid in connecting existing compilers together.
LLVM : The LLVM Project is a collection of modular and reusable compiler and toolchain technologies.
PTX : PTX, a low-level parallel thread execution virtual machine and instruction set architecture (ISA).

Awesome List

awesome-cuda-and-hpc : some awesome public CUDA, cuda-python, cuBLAS, cuDNN, CUTLASS, TensorRT, TensorRT-LLM, Triton, TVM, MLIR, LLVM, PTX and High Performance Computing (HPC) projects.
Erkaman/Awesome-CUDA : This is a list of useful libraries and resources for CUDA development.
jslee02/awesome-gpgpu : 😎 A curated list of awesome GPGPU (CUDA/OpenCL/Vulkan) resources.
mikeroyal/CUDA-Guide : A guide covering CUDA including the applications and tools that will make you a better and more efficient CUDA developer.
rkinas/triton-resources : A curated list of resources for learning and exploring Triton, OpenAI’s programming language for writing efficient GPU code.

Learning Resources

chenzomi12/AISystem : AISystem 主要是指AI系统，包括AI芯片、AI编译器、AI推理和训练框架等AI全栈底层技术。
chenzomi12/AIFoundation : AIFoundation 主要是指AI系统遇到大模型，从底层到上层如何系统级地支持大模型训练和推理，全栈的核心技术。
CUDA Learning
- NVIDIA CUDA Toolkit Documentation : CUDA Toolkit Documentation.
- NVIDIA CUDA C++ Programming Guide : CUDA C++ Programming Guide.
- NVIDIA CUDA C++ Best Practices Guide : CUDA C++ Best Practices Guide.
- NVIDIA cuBLAS Programming Guide : NVIDIA cuBLAS Documentation.
- NVIDIA CUTLASS Programming Guide : NVIDIA CUTLASS Documentation.
- NVIDIA NCCL Programming Guide : NVIDIA Collective Communication Library (NCCL) Documentation.
- NVIDIA PTX(Parallel Thread Execution) Programming Guide : NVIDIA PTX (Parallel Thread Execution) Programming Guide.
- NVIDIA/cuda-samples : Samples for CUDA Developers which demonstrates features in CUDA Toolkit.
- NVIDIA/CUDALibrarySamples : CUDA Library Samples.
- NVIDIA/cuda-python : CUDA Python: Performance meets Productivity. nvidia.github.io/cuda-python/
- CuPy : CuPy : NumPy & SciPy for GPU. cupy.dev. CuPy User Guide
- NVIDIA-developer-blog/code-samples : Source code examples from the Parallel Forall Blog.
- HeKun-NVIDIA/CUDA-Programming-Guide-in-Chinese : This is a Chinese translation of the CUDA programming guide. 本项目为 CUDA C Programming Guide 的中文翻译版。
- RichardAns/CUDA-Programs : Examples from Programming in Parallel with CUDA.
- brucefan1983/CUDA-Programming : Sample codes for my CUDA programming book.
- QINZHAOYU/CudaSteps : 基于《cuda编程-基础与实践》（樊哲勇著）的cuda学习之路。
- MAhaitao999/CUDA_Programming : 《CUDA编程基础与实践》一书的代码。
- YouQixiaowu/CUDA-Programming-with-Python : 关于书籍CUDA Programming使用了pycuda模块的Python版本的示例代码。
- xlite-dev/LeetCUDA : 📚LeetCUDA: Modern CUDA Learn Notes with PyTorch for Beginners🐑, 200+ CUDA/Tensor Cores Kernels, HGEMM, FA-2 MMA etc.🔥 https://github.com/xlite-dev/LeetCUDA
- BBuf/how-to-optim-algorithm-in-cuda : how to optimize some algorithm in cuda.
- PaddleJitLab/CUDATutorial : A self-learning tutorail for CUDA High Performance Programing. 从零开始学习 CUDA 高性能编程。
- RussWong/CUDATutorial : A CUDA tutorial to make people learn CUDA program from 0.
- Tongkaio/CUDA_Kernel_Samples : CUDA 算子手撕与面试指南。
- bertmaher/simplegemm : Pingpong GEMM from scratch. I’ve been really excited to learn the lowest-level details of GPU matrix multiplication recently, so I was really inspired to read Pranjal Shankhdhar’s fantastic blog post Outperforming cuBLAS on H100, which implements a fast gemm from first principles in CUDA, and actually outperforms cuBLAS. In a similar vein, I wanted to understand the pingpong gemm algorithm in detail. So, I used https://github.com/pranjalssh/fast.cu as a starting point, and wrote this kernel to see if I could match CUTLASS’s pingpong implementation myself, using hand-written CUDA.
- pranjalssh/fast.cu : Fastest GPU kernels, written from scratch. Matrix multiplication of square bf16 matrices, accumulated in fp32. Explanation in https://cudaforfun.substack.com/p/outperforming-cublas-on-h100-a-worklog
- gpu-mode/lectures : Material for gpu-mode lectures. www.youtube.com/@GPUMODE
- gpu-mode/resource-stream :GPU programming related news and material links. discord.gg/gpumode
- ifromeast/cuda_learning : learning how CUDA works.
- a-hamdi/cuda : 100 days of building Cuda kernels! This document serves as a log of the progress and knowledge I gained while working on CUDA programming and studying the PMPP (Parallel Programming and Optimization) book. Mentor: https://github.com/hkproj/. Bro in the 100 days challenge: https://github.com/1y33/100Days.
- SwekeR-463/100kernels : 100 days of learning & making kernels in cuda / triton.
- leimao/CUDA-GEMM-Optimization : CUDA Matrix Multiplication Optimization. This repository contains the CUDA kernels for general matrix-matrix multiplication (GEMM) and the corresponding performance analysis.
- interestingLSY/CUDA-From-Correctness-To-Performance-Code : Codes & examples for “CUDA - From Correctness to Performance”. The lecture can be found at https://wiki.lcpu.dev/zh/hpc/from-scratch/cuda.
- Liu-xiandong/How_to_optimize_in_GPU : This is a series of GPU optimization topics. Here we will introduce how to optimize the CUDA kernel in detail. I will introduce several basic kernel optimizations, including: elementwise, reduce, sgemv, sgemm, etc. The performance of these kernels is basically at or near the theoretical limit.
- tpoisonooo/how-to-optimize-gemm : row-major matmul optimization. zhuanlan.zhihu.com/p/65436463.
- Bruce-Lee-LY/matrix_multiply : Several common methods of matrix multiplication are implemented on CPU and Nvidia GPU using C++11 and CUDA.
- Bruce-Lee-LY/cuda_hgemm : Several optimization methods of half-precision general matrix multiplication (HGEMM) using tensor core with WMMA API and MMA PTX instruction.
- Bruce-Lee-LY/cuda_hgemv : Several optimization methods of half-precision general matrix vector multiplication (HGEMV) using CUDA core.
- enp1s0/ozIMMU : FP64 equivalent GEMM via Int8 Tensor Cores using the Ozaki scheme. arxiv.org/abs/2306.11975
- Cjkkkk/CUDA_gemm : A simple high performance CUDA GEMM implementation.
- AyakaGEMM/Hands-on-GEMM : A GEMM tutorial.
- zpzim/MSplitGEMM : Large matrix multiplication in CUDA.
- jundaf2/CUDA-INT8-GEMM : CUDA 8-bit Tensor Core Matrix Multiplication based on m16n16k16 WMMA API.
- chanzhennan/cuda_gemm_benchmark : Base on gtest/benchmark, refer to https://github.com/Liu-xiandong/How_to_optimize_in_GPU.
- YuxueYang1204/CudaDemo : Implement custom operators in PyTorch with cuda/c++.
- CoffeeBeforeArch/cuda_programming : Code from the “CUDA Crash Course” YouTube series by CoffeeBeforeArch.
- rbaygildin/learn-gpgpu : Algorithms implemented in CUDA + resources about GPGPU.
- godweiyang/NN-CUDA-Example : Several simple examples for popular neural network toolkits calling custom CUDA operators.
- yhwang-hub/Matrix_Multiplication_Performance_Optimization : Matrix Multiplication Performance Optimization.
- caiwanxianhust/ClusteringByCUDA : 使用 CUDA C++ 实现的一系列聚类算法。
- ulrichstern/cuda-convnet : Alex Krizhevsky’s original code from Google Code. “微信公众号「人工智能大讲堂」《找到了AlexNet当年的源代码，没用框架，从零手撸CUDA/C++》”。
- PacktPublishing/Learn-CUDA-Programming : Learn CUDA Programming, published by Packt.
- PacktPublishing/Hands-On-GPU-Programming-with-Python-and-CUDA : Hands-On GPU Programming with Python and CUDA, published by Packt.
- PacktPublishing/Hands-On-GPU-Accelerated-Computer-Vision-with-OpenCV-and-CUDA : Hands-On GPU Accelerated Computer Vision with OpenCV and CUDA, published by Packt.
- BobMcDear/neural-network-cuda : Neural network from scratch in CUDA/C++.
- zjhellofss/KuiperLLama : 《动手自制大模型推理框架》。KuiperLLama 动手自制大模型推理框架，支持LLama2/3和Qwen2.5。校招、秋招、春招、实习好项目，带你从零动手实现支持LLama2/3和Qwen2.5的大模型推理框架。
- zjhellofss/KuiperInfer : 校招、秋招、春招、实习好项目！带你从零实现一个高性能的深度学习推理库，支持大模型 llama2 、Unet、Yolov5、Resnet等模型的推理。Implement a high-performance deep learning inference library step by step。
- zjhellofss/kuiperdatawhale : 从零自制深度学习推理框架。
- MarioSieg/magnetron : (WIP) A small but powerful, homemade PyTorch from scratch. Minimalistic homemade PyTorch alternative, written in C99 and Python.
- lucasdelimanogueira/PyNorch : Recreating PyTorch from scratch (C/C++, CUDA, NCCL and Python, with multi-GPU support and automatic differentiation!)
- YuxueYang1204/CudaDemo : A simple demo to show how to implement custom operators in PyTorch with cuda/c++. More details can be seen in 知乎: 用Cuda实现PyTorch算子.
- xgqdut2016/cuda_code : easy cuda code. CUDA代码简单入门。
- xgqdut2016/hpc_project : some hpc project for learning.
- xgqdut2016/hpc2torch : 这个仓库打算搭建一个高性能底层库的测试框架，将会针对onnx的算子编写相关的高性能kernel，作为pytorch的补充，从python端对比手写kernel和pytorch库函数的性能以及精度对比。
- zoheth/yan : Yan (炎) is a high-performance CUDA operator library designed for learning purposes while emphasizing clean code and maximum performance.
TensorRT Learning
- NVIDIA TensorRT Docs : NVIDIA Deep Learning TensorRT Documentation.
- TensorRT : NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT. developer.nvidia.com/tensorrt
- TensorRT-LLM : TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines. nvidia.github.io/TensorRT-LLM
- HeKun-NVIDIA/TensorRT-Developer_Guide_in_Chinese : 本项目是NVIDIA TensorRT的中文版开发手册，有个人翻译并添加自己的理解。
- kalfazed/tensorrt_starter : This repository give a guidline to learn CUDA and TensorRT from the beginning.
- LitLeo/TensorRT_Tutorial : TensorRT_Tutorial.
Triton Learning
- Triton : Development repository for the Triton language and compiler. triton-lang.org/
- Triton Docs : Triton Documentation.
- hyperai/triton-cn : Triton Documentation in Chinese Simplified / Triton 中文文档. triton.hyper.ai
TileLang Learning
- TileLang : Domain-specific language designed to streamline the development of high-performance GPU/CPU/Accelerators kernels. tilelang.com/
- TileLang Docs : TileLang Documentation.
TVM Learning
- Apache TVM 中文站 : Apache TVM 中文文档！
MLIR Learning
- MLIR Docs : MLIR Documentation.
- BBuf/tvm_mlir_learn : compiler learning resources collect.
- j2kun/mlir-tutorial : This is the code repository for a series of articles on the MLIR framework for building compilers.
- KEKE046/mlir-tutorial : Hands-On Practical MLIR Tutorial.
- AyakaGEMM/Hands-on-MLIR : Hands-on-MLIR.
- yao-jiashu/KernelCodeGen : GEMM/Conv2d CUDA/HIP kernel code generation using MLIR.
LLVM Learning
- LLVM Docs : LLVM Documentation.
HPC Learning
- LAFF-On-PfHP : LAFF-On Programming for High Performance.
- flame/how-to-optimize-gemm : How To Optimize Gemm wiki pages. https://github.com/flame/how-to-optimize-gemm/wiki
- flame/blislab : BLISlab: A Sandbox for Optimizing GEMM. Check the tutorial for more details.
- tpoisonooo/how-to-optimize-gemm : row-major matmul optimization. zhuanlan.zhihu.com/p/65436463.
- YichengDWu/matmul.mojo : High Performance Matrix Multiplication in Pure Mojo 🔥

Frameworks

CUDA Frameworks
- GPU Interface
  
  GPU接口
  - CPP Version
    - CCCL : CUDA C++ Core Libraries. The concept for the CUDA C++ Core Libraries (CCCL) grew organically out of the Thrust, CUB, and libcudacxx projects that were developed independently over the years with a similar goal: to provide high-quality, high-performance, and easy-to-use C++ abstractions for CUDA developers.
    - HIP : HIP: C++ Heterogeneous-Compute Interface for Portability. HIP is a C++ Runtime API and Kernel Language that allows developers to create portable applications for AMD and NVIDIA GPUs from single source code. rocmdocs.amd.com/projects/HIP/
  - Python Version
    - NVIDIA/cuda-python : CUDA Python is the home for accessing NVIDIA’s CUDA platform from Python. CUDA Python Low-level Bindings. nvidia.github.io/cuda-python/
    - CuPy : CuPy : NumPy & SciPy for GPU. cupy.dev
    - PyCUDA : PyCUDA: Pythonic Access to CUDA, with Arrays and Algorithms. mathema.tician.de/software/pycuda
  - Rust Version
    - jessfraz/advent-of-cuda : Doing advent of code with CUDA and rust.
    - Bend : A massively parallel, high-level programming language.higherorderco.com
    - HVM : A massively parallel, optimal functional runtime in Rust.higherorderco.com
    - ZLUDA : CUDA on AMD GPUs.
    - Rust-CUDA : Ecosystem of libraries and tools for writing and executing fast GPU code fully in Rust.
    - cudarc : cudarc: minimal and safe api over the cuda toolkit.
    - bindgen_cuda : Similar crate than bindgen in philosophy. It will help create automatic bindgen to cuda kernels source files and make them easier to use directly from Rust.
    - cuda-driver : 基于 CUDA Driver API 的 cuda 运行时环境。
    - async-cuda : Asynchronous CUDA for Rust.
    - async-tensorrt : Asynchronous TensorRT for Rust.
    - krnl : Safe, portable, high performance compute (GPGPU) kernels.
    - custos : A minimal OpenCL, CUDA, WGPU and host CPU array manipulation engine / framework.
    - spinorml/nvlib : Rust interoperability with NVIDIA CUDA NVRTC and Driver.
    - DoeringChristian/cuda-rs : Cuda Bindings for rust generated with bindgen-cli (similar to cust_raw).
    - romankoblov/rust-nvrtc : NVRTC bindings for RUST.
    - solkitten/astro-cuda : CUDA Driver API bindings for Rust.
    - bokutotu/curs : cuda&cublas&cudnn wrapper for Rust.
    - rust-cuda/cuda-sys : Rust binding to CUDA APIs.
    - bheisler/RustaCUDA : Rusty wrapper for the CUDA Driver API.
    - tmrob2/cuda2rust_sandpit : Minimal examples to get CUDA linear algebra programs working with Rust using CC & FFI.
    - PhDP/rust-cuda-template : Simple template for Rust + CUDA.
    - neka-nat/cuimage : Rust implementation of image processing library with CUDA.
    - yanghaku/cuda-driver-sys : Rust binding to CUDA Driver APIs.
    - Canyon-ml/canyon-sys : Rust Bindings for Cuda, CuDNN.
    - cea-hpc/HARP : Small tool for profiling the performance of hardware-accelerated Rust code using OpenCL and CUDA.
    - Conqueror712/CUDA-Simulator : A self-developed version of the user-mode CUDA emulator project and a learning repository for Rust.
    - cszach/rust-cuda-template : A Rust CUDA template with detailed instructions.
    - exor2008/fluid-simulator : Rust CUDA fluid simulator.
    - chichieinstein/rustycuda : Convenience functions for generic handling of CUDA resources on the Rust side.
    - Jafagervik/cruda : CRUDA - Writing rust with cuda.
    - lennyerik/cutransform : CUDA kernels in any language supported by LLVM.
    - cjordan/hip-sys : Rust bindings for HIP.
    - rust-gpu : 🐉 Making Rust a first-class language and ecosystem for GPU shaders 🚧 shader.rs
    - wgpu : Safe and portable GPU abstraction in Rust, implementing WebGPU API. wgpu.rs
    - Vulkano : Safe and rich Rust wrapper around the Vulkan API. Vulkano is a Rust wrapper around the Vulkan graphics API. It follows the Rust philosophy, which is that as long as you don’t use unsafe code you shouldn’t be able to trigger any undefined behavior. In the case of Vulkan, this means that non-unsafe code should always conform to valid API usage.
    - Ash : Vulkan bindings for Rust.
    - ocl : OpenCL for Rust.
    - opencl3 : A Rust implementation of the Khronos OpenCL 3.0 API.
  - Julia Version
    - CUDA.jl : CUDA programming in Julia. juliagpu.org/
    - AMDGPU.jl : AMD GPU (ROCm) programming in Julia.
- Performance Benchmark
  - NVIDIA/nvbench : CUDA Kernel Benchmarking Library.
  - FlagPerf : FlagPerf is an open-source software platform for benchmarking AI chips. FlagPerf是智源研究院联合AI硬件厂商共建的一体化AI硬件评测引擎，旨在建立以产业实践为导向的指标体系，评测AI硬件在软件栈组合（模型+框架+编译器）下的实际能力。
  - te42kyfo/gpu-benches : collection of benchmarks to measure basic GPU capabilities.
- Scientific Computing Framework
  
  科学计算框架
  - cuBLAS : Basic Linear Algebra on NVIDIA GPUs. NVIDIA cuBLAS is a GPU-accelerated library for accelerating AI and HPC applications. It includes several API extensions for providing drop-in industry standard BLAS APIs and GEMM APIs with support for fusions that are highly optimized for NVIDIA GPUs. The cuBLAS library also contains extensions for batched operations, execution across multiple GPUs, and mixed- and low-precision execution with additional tuning for the best performance.
  - CUTLASS : CUDA Templates for Linear Algebra Subroutines.
  - MatX : MatX - GPU-Accelerated Numerical Computing in Modern C++. An efficient C++17 GPU numerical computing library with Python-like syntax. nvidia.github.io/MatX
  - DeepGEMM : DeepGEMM: clean and efficient FP8 GEMM kernels with fine-grained scaling.
  - MUTLASS : MUSA Templates for Linear Algebra Subroutines.
  - CuPy : CuPy : NumPy & SciPy for GPU. cupy.dev
  - GenericLinearAlgebra.jl : Generic numerical linear algebra in Julia.
  - custos-math : This crate provides CUDA, OpenCL, CPU (and Stack) based matrix operations using custos.
- Attention and Transformer Framework
  - FlashAttention : Fast and memory-efficient exact attention. “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness”. (arXiv 2022).
  - SageAttention : Quantized Attention that achieves speedups of 2.1-3.1x and 2.7-5.1x compared to FlashAttention2 and xformers, respectively, without lossing end-to-end metrics across various models. “SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration”. (arXiv 2024). “SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization”. (arXiv 2024).
  - fla-org/flash-linear-attention : 🚀 Efficient implementations of state-of-the-art linear attention models in Pytorch and Triton.
  - 66RING/tiny-flash-attention : flash attention tutorial written in python, triton, cuda, cutlass.
  - weishengying/tiny-flash-attention : 使用 cutlass 实现 flash-attention 精简版，具有教学意义。
  - jepeake/tiny-flash-attention : flash attention in ~20 lines.
- Machine Learning Framework
  - cuDNN : The NVIDIA CUDA® Deep Neural Network library (cuDNN) is a GPU-accelerated library of primitives for deep neural networks. cuDNN provides highly tuned implementations for standard routines such as forward and backward convolution, attention, matmul, pooling, and normalization.
  - PyTorch : Tensors and Dynamic neural networks in Python with strong GPU acceleration. pytorch.org
  - MooreThreads/torch_musa : torch_musa is an open source repository based on PyTorch, which can make full use of the super computing power of MooreThreads graphics cards.
  - PaddlePaddle : PArallel Distributed Deep LEarning: Machine Learning Framework from Industrial Practice （『飞桨』核心框架，深度学习&机器学习高性能单机、分布式训练和跨平台部署）. www.paddlepaddle.org/
  - flashlight/flashlight : A C++ standalone library for machine learning. fl.readthedocs.io/en/latest/
  - yhwang-hub/dl_model_infer : his is a c++ version of the AI reasoning library. Currently, it only supports the reasoning of the tensorrt model. The follow-up plan supports the c++ reasoning of frameworks such as Openvino, NCNN, and MNN. There are two versions for pre- and post-processing, c++ version and cuda version. It is recommended to use the cuda version., This repository provides accelerated deployment cases of deep learning CV popular models, and cuda c supports dynamic-batch image process, infer, decode, NMS.
  - NVlabs/tiny-cuda-nn : Lightning fast C++/CUDA neural network framework.
  - zjhellofss/KuiperLLama : 《动手自制大模型推理框架》。KuiperLLama 动手自制大模型推理框架，支持LLama2/3和Qwen2.5。校招、秋招、春招、实习好项目，带你从零动手实现支持LLama2/3和Qwen2.5的大模型推理框架。
  - zjhellofss/KuiperInfer : 校招、秋招、春招、实习好项目！带你从零实现一个高性能的深度学习推理库，支持大模型 llama2 、Unet、Yolov5、Resnet等模型的推理。Implement a high-performance deep learning inference library step by step。
  - zjhellofss/kuiperdatawhale : 从零自制深度学习推理框架。
  - MarioSieg/magnetron : (WIP) A small but powerful, homemade PyTorch from scratch. Minimalistic homemade PyTorch alternative, written in C99 and Python.
  - lucasdelimanogueira/PyNorch : Recreating PyTorch from scratch (C/C++, CUDA, NCCL and Python, with multi-GPU support and automatic differentiation!)
- AI Inference Framework
  
  AI推理框架
  - LLM Inference and Serving Engine
    - TensorRT : NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT. developer.nvidia.com/tensorrt
    - TensorRT-LLM : TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines. nvidia.github.io/TensorRT-LLM
    - NVIDIA/TensorRT-Model-Optimizer : TensorRT Model Optimizer is a unified library of state-of-the-art model optimization techniques such as quantization, pruning, distillation, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM or TensorRT to optimize inference speed on NVIDIA GPUs. nvidia.github.io/TensorRT-Model-Optimizer
    - Ollama : Get up and running with Llama 3.3, DeepSeek-R1, Phi-4, Gemma 2, and other large language models. ollama.com
    - vLLM : A high-throughput and memory-efficient inference and serving engine for LLMs. docs.vllm.ai
    - Nano-vLLM : A lightweight vLLM implementation built from scratch.
    - SGLang : SGLang is a fast serving framework for large language models and vision language models. docs.sglang.ai/
    - sgl-project/sgl-learning-materials : Materials for learning SGLang.
    - llama.cpp : LLM inference in C/C++.
    - FlashInfer : FlashInfer: Kernel Library for LLM Serving . flashinfer.ai
    - Chitu（赤兔） : High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability. Chitu (赤兔) 是一个专注于效率、灵活性和可用性的高性能大语言模型推理框架。
    - ztxz16/fastllm : fastllm是c++实现自有算子替代Pytorch的高性能全功能大模型推理库，可以推理Qwen, Llama, Phi等稠密模型，以及DeepSeek, Qwen-moe等moe模型。fastllm是后端无依赖的高性能大模型推理库。同时支持张量并行推理稠密模型和混合模式推理MOE模型，任意10G以上显卡即可推理满血DeepSeek。双路9004/9005服务器+单显卡部署DeepSeek满血满精度原版模型，单并发20tps；INT4量化模型单并发30tps，多并发可达60+。
    - MLC LLM : Universal LLM Deployment Engine with ML Compilation. llm.mlc.ai/
    - KTransformers : A Flexible Framework for Experiencing Cutting-edge LLM Inference Optimizations. kvcache-ai.github.io/ktransformers/
    - Aphrodite : Large-scale LLM inference engine. aphrodite.pygmalion.chat
    - GPUStack : GPUStack is an open-source GPU cluster manager for running AI models. Manage GPU clusters for running AI models. gpustack.ai
    - Lamini : The Official Python Client for Lamini’s API. lamini.ai/
    - triton-inference-server/tensorrtllm_backend : The Triton TensorRT-LLM Backend.
    - datawhalechina/self-llm : 《开源大模型食用指南》基于Linux环境快速部署开源大模型，更适合中国宝宝的部署教程。
    - ninehills/llm-inference-benchmark : LLM Inference benchmark.
    - csbench/csbench : “CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery”. (arXiv 2024).
    - MooreThreads/vllm_musa : A high-throughput and memory-efficient inference and serving engine for LLMs. docs.vllm.ai
    - dusty-nv/jetson-inference : Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA Jetson.
    - m0dulo/InferSpore : 🌱 A fully independent Large Language Model (LLM) inference engine, built leveraging cuBLAS and cub. 🧩
  - High Performance Kernel Library
    - Liger-Kernel : Efficient Triton Kernels for LLM Training. arxiv.org/pdf/2410.10989
    - FlashInfer : FlashInfer: Kernel Library for LLM Serving . flashinfer.ai
    - DeepGEMM : DeepGEMM: clean and efficient FP8 GEMM kernels with fine-grained scaling.
    - FlashMLA : FlashMLA: Efficient MLA Decoding Kernel for Hopper GPUs.
    - DeepEP : DeepEP: an efficient expert-parallel communication library.
    - FlagGems : FlagGems is a high-performance general operator library implemented in OpenAI Triton. It aims to provide a suite of kernel functions to accelerate LLM training and inference.
  - C Implementation
    - llm.c : LLM training in simple, pure C/CUDA. There is no need for 245MB of PyTorch or 107MB of cPython. For example, training GPT-2 (CPU, fp32) is ~1,000 lines of clean code in a single file. It compiles and runs instantly, and exactly matches the PyTorch reference implementation.
    - llama2.c : Inference Llama 2 in one file of pure C. Train the Llama 2 LLM architecture in PyTorch then inference it with one simple 700-line C file (run.c).
  - CPP Implementation
    - llama.cpp