gpu-camera-sample
Camera sample application with realtime GPU image processing performance (Windows, Linux, Jetson)
That software is based on the following image processing pipeline for camera applications that includes:
- Raw image capture (monochrome or bayer 8-bit, 12-bit packed/unpacked, 16-bit)
- Import to GPU
- Raw data conversion and unpacking
- Linearization curve
- Bad Pixel Correction
- Dark frame subtraction
- Flat-Field Correction
- White Balance
- Exposure Correction (brightness control)
- Debayer with HQLI (5×5 window), L7 (7×7 window), DFPD (11×11), MG (23×23) algorithms
- Color correction with 3×3 matrix
- Bilaterea or NLM denoiser
- Crop / Resize / Flip / Flop / Rotate
- Gamma (linear, sRGB)
- JPEG / MJPEG encoding/decoding
- H.264 and HEVC encoding/decoding
- Output to monitor via OpenGL
- Export from GPU to CPU memory
- MJPEG and H.264/H.265 streaming
- Storage of compressed images/video to SSD
Processing is done on NVIDIA GPU to speedup the performance. The software could also work with raw bayer images in PGM format and you can utilize these images for testing or if you don’t have a camera, or if your camera is not supported. More info about that project you can find here.
From the benchmarks on NVIDIA Quadro RTX 6000 or GeForce RTX 4090 we can see that GPU-based raw image processing is very fast and it could offer high image quality at the same time. The total performance could reach 4 GPix/s for color cameras. The performance strongly depends on complexity of the pipeline. Multiple GPU solutions could significantly improve the performance.
Currently the software is working with XIMEA cameras via XIMEA SDK. FLIR cameras are supported via Spinnaker SDK. We can work with Imperx cameras via Imperx SDK. LUCID Vision Labs cameras are supported via Arena SDK.
Via GenICam the software can work with XIMEA, MATRIX VISION, Basler, FLIR, Imperx, JAI, LUCID Vision Labs, Daheng Imaging cameras.
The software is also working with Leopard Imaging mipi csi cameras on Jetson. You need to have a proper driver to be able to acquire raw frames from mipi camera for further image processing on the GPU with 16/32-bit precision. The software doesn’t use NVIDIA ISP via libargus.
Soon we are going to add support for Baumer, Kaya Instruments, IDS Imaging Development Systems cameras. Cameras from Emergent Vision Technologies, IO Industries and Hikrobot are already supported (see the roadmap below). You can add support for desired cameras by yourself. The software is working with demo version of Fastvideo SDK, that is why you can see a watermark on the screen. To get a Fastvideo SDK license for development and for deployment, please contact Fastvideo company.
All figures below are on a single NVIDIA GeForce RTX 4090 unless noted. They match the numbers published on Fastvideo’s product and benchmark pages (linked in the last column).
| Task |
Performance |
Source |
| Full color-camera ISP (this application) |
up to 4 GPix/s |
this repository |
| FastVCR, 65 MPix camera (9344×7000) |
70 fps (8-bit) / 60 fps (10-bit) |
FastVCR for XIMEA |
| Debayer, MG algorithm (62 MPix, 16-bit) |
up to 20 GPix/s, ~40.5 dB PSNR (Kodak) |
GPU Debayer |
| Debayer, HQLI algorithm (fastest) |
up to 69 GPix/s |
GPU Debayer |
| Motion JPEG encoding (Full HD) |
up to 5500 fps |
Motion JPEG on CUDA |
| Bilateral denoiser (8.9 MPix, 16-bit RGB) |
0.13 ms (61 GPix/s) |
GPU Denoiser |
| RawZipper RAW Bayer codec (12-bit, 12 MPix) |
lossy compression ratio ~10 at up to ~1 GPix/s |
RawZipper RAW codec |
| Adaptive local tone mapping (65 MPix, 16-bit) |
6.3 ms (10 GPix/s) |
ALTMapper |
| Fast CinemaDNG RAW-to-RGB (12 MPix DNG, on RTX 2080ti) |
3000 frames/min = 50 fps |
Fast CinemaDNG Processor |
| Glass-to-glass latency (XIMEA USB3, 3 MPix, 120 fps, 144 Hz monitor) |
~35–40 ms |
FastVCR |
The whole pipeline runs in 16/32-bit precision on the NVIDIA GPU. The software does not use the NVIDIA Jetson hardware ISP (libargus), so every stage of the ISP is tunable and quality-critical steps — demosaicing, denoising, and tone mapping — use high-quality algorithms rather than fixed-function approximations.
How to build gpu-camera-sample
Requirements for Windows
- Camera SDK or GenICam package + camera vendor GenTL producer (.cti). Сurrently XIMEA, MATRIX VISION, Basler, FLIR, Imperx, JAI, LUCID Vision Labs, Daheng Imaging cameras are supported
- Fastvideo SDK (demo) ver.0.22.0.0
- NVIDIA CUDA-12.6
- Qt ver.5.13.1
- Compiler MSVC 2022 or later
Requirements for Linux
- Ubuntu 22.04 for x64 platform, Ubuntu 20.04 for Arm64 platform with CUDA 11.4 / CUDA 12.2, Ubuntu 18.04 for Arm64 platform with CUDA 10.2
- Camera SDK or GenICam package + camera vendor GenTL producer (.cti). Currently XIMEA, MATRIX VISION, Basler, FLIR, Imperx, JAI, Daheng Imaging cameras are supported
- Fastvideo SDK (demo) ver.0.22.0.0
- NVIDIA CUDA-12.6 for x64, CUDA-11.4/12.6 (Jetson AGX Xavier, Orin) or CUDA-10.2 (Jetson TX2, NX) for ARM64 platform
- Compiler gcc 7.4 or later
- Qt 5 (qtbase5-dev)
sudo apt-get install qtbase5-dev qtbase5-dev-tools qtcreator git
sudo apt-get install libavutil-dev libavcodec-dev libavdevice-dev libavfilter-dev libavformat-dev libavresample-dev libx264-dev
Jetson users have to build FFmpeg libraries from sources. See this shell script for details.
- Libjpeg and zlib libraries
sudo apt-get install libjpeg-dev zlib1g-dev
Build instructions
git clone https://github.com/fastvideo/gpu-camera-sample.git
For Windows users
- Create OtherLibs folder in the project root folder. This folder will contains external libraries, used in gpu-camera-sample application.
- Download Fastvideo SDK from Fastvideo SDK (demo) for Windows-10, 64-bit, unpack it into <ProjectRoot>/OtherLibs/FastvideoSDK folder. If the trial period is expired, please send an inquiry to Fastvideo to obtain the latest version.
- If you need direct XIMEA camera support, download XiAPI from https://www.ximea.com/support/wiki/apis/XIMEA_Windows_Software_Package. Install downloaded package (by default into C:\XIMEA). Copy API folder from XIAPI installation folder into <ProjectRoot>/OtherLibs folder.
- To work with FLIR cameras
- To work with Imperx cameras, download Imperx SDK (IpxCameraSdk) from https://www.imperx.com/downloads/. Unpack and copy archive content to <ProjectRoot>/OtherLibs/Imperx folder.
- To work with LUCID Vision Labs cameras, download Arena SDK from https://thinklucid.com/downloads-hub/. Install it and copy “C:\Program Files\Lucid Vision Labs\Arena SDK” folder into <ProjectRoot>/OtherLibs.
- If you need GenICam support
You also can download precompiled libs.
- By default the application will be built with no camera support. The only option is camera simulator which is working with PGM files.
- Open <ProjectRoot>/src/GPUCameraSample.pro in Qt Creator.
- Open common_defs.pri
- To enable GenICam support, uncomment DEFINES += SUPPORT_GENICAM
- To enable XIMEA camera support, uncomment DEFINES += SUPPORT_XIMEA
- To enable Lucid Vision Labs camera support, uncomment DEFINES += SUPPORT_LUCID
- To enable FLIR camera support, uncomment DEFINES += SUPPORT_FLIR
- To enable Imperx camera support, uncomment DEFINES += SUPPORT_IMPERX.
- Build the project
- Binaries will be placed into <ProjectRoot>/GPUCameraSample_x64 folder.
For Linux users
Here and after we assume you put source code into home directory, so project root is ~/gpu-camera-sample
- Make sure file <ProjectRoot>/Scripts/make_links.sh is executable
chmod 755 ~/gpu-camera-sample/Scripts/make_links.sh
- Create OtherLibsLinux folder in the project root folder. This folder will contain external libraries, used in gpu-camera-sample application.
- Download Fastvideo SDK x64 platform for Windows-10/11, Linux Ubuntu 22.04, or Arm64 for NVIDIA Jetson Nano, TX2, Xavier, Orin and unpack it into <ProjectRoot>/OtherLibsLinux/FastvideoSDK folder. Copy all files from <ProjectRoot>/OtherLibsLinux/FastvideoSDK/fastvideo_sdk/lib to <ProjectRoot>/OtherLibsLinux/FastvideoSDK/fastvideo_sdk/lib/Linux64 for x64 platform and to <ProjectRoot>/OtherLibsLinux/FastvideoSDK/fastvideo_sdk/lib/Arm64 for Arm64 platform.
- Create links to Fastvideo SDK *.so files
cd ~/gpu-camera-sample/Scripts
./make_links.sh ~/gpu-camera-sample/OtherLibsLinux/FastvideoSDK/fastvideo_sdk/lib/Linux64
- If you need direct XIMEA camera support, download XiAPI from https://www.ximea.com/support/wiki/apis/XIMEA_Linux_Software_Package. Unpack and install downloaded package.
- If you need GenICam support
- Download GenICamTM Package Version 2019.11 (https://www.emva.org/wp-content/uploads/GenICam_Package_2019.11.zip).
- Unpack it to a temporary folder and cd to Reference Implementation folder.
- Create <ProjectRoot>/OtherLibsLinux/GenICam folder.
- Unpack GenICam_V3_2_0-Linux64_x64_gcc48-Runtime.tgz or GenICam_V3_2_0-Linux64_ARM_gcc49-Runtime.tgz into <ProjectRoot>/OtherLibsLinux/GenICam folder.
- Unpack GenICam_V3_2_0-Linux64_x64_gcc48-SDK.tgz or GenICam_V3_2_0-Linux64_ARM_gcc49-SDK.tgz into <ProjectRoot>/OtherLibsLinux/GenICam/library/CPP
- Create <ProjectRoot>OtherLibsLinux\GenICam\library\CPP\include\GenTL folder and copy there
- GenICam_Package_2019.11.zip\GenICam_Package_2019.11\GenTL\GenTL_v1_6.zip\GenTL.h
- GenICam_Package_2019.11.zip\GenICam_Package_2019.11\SFNC\PFNC.h
- Ensure Qt uses gcc, not clang to build project.
- If you need LUCID Vision Labs cameras support.
- Download Arena SDK for Linux from https://thinklucid.com/downloads-hub/ and unpack it into OtherLibsLinux/Arena SDK/ArenaSDK_Linux_x64/ for x64 platform and OtherLibsLinux/Arena SDK/ArenaSDK_Linux_ARM64 for arm64 platform.
- Run
sudo /.Arena_SDK_Linux_x64.conf
for x64 platform and
sudo /.Arena_SDK_ARM64.conf
for arm64 platform
cd to precompiledExamples and run
./IpConfigUtility /list
if everything is OK you will see something like that
Scanning for devices...
index MAC IP SUBNET GATEWAY IP CONFIG
0 1C0FAF5A908A 169.254.139.144 255.255.0.0 0.0.0.0 DHCP= 1 Persistent Ip= 0 LLA = 1
run
./Cpp_Acquisition
to test that camera is working.
- If you need built-in MIPI cameras support on Jetson TX2 or Leopard Imaging IMX477 MIPI SCI camera support on Jetson AGX Xavier, read this guide how to setup units and install required drivers.
- By default the application will be built with no camera support. The only option is camera simulator which is working with PGM files.
- Open <ProjectRoot>/src/GPUCameraSample.pro in Qt Creator.
- Open common_defs.pri
- To enable GenICam support, uncomment DEFINES += SUPPORT_GENICAM.
- To enable XIMEA camera support, uncomment DEFINES += SUPPORT_XIMEA
- To enable Imperx camera support, uncomment DEFINES += SUPPORT_IMPERX.
- To enable Lucid Vision Labs camera support, uncomment DEFINES += SUPPORT_LUCID
- FLIR support is experimental at the moment. Use it on your own risk.
- Build the project.
- If GenICam support is enabled, set environment variable GENICAM_GENTL64_PATH with full path to the camera vendor GenTL producer (.cti) library, before run the application.
- Binaries will be placed into <ProjectRoot>/GPUCameraSample_Arm64 or GPUCameraSample_Linux64 folder. To run the application from the terminal run GPUCameraSample.sh. Necessary symbolic links will be made during compile time.
NVIDIA Jetson provides many features related to power management, thermal management, and electrical management. These features deliver the best user experience possible given the constraints of a particular platform. The target user experience ensures the perception that the device provides:
- Uniformly high performance
- Excellent battery life
- Perfect stability
Utility nvpmodel has to been used to change the power mode. Mode with power consumption is MAXN. To activate this mode call
sudo /usr/sbin/nvpmodel –m 0
Also you have to call jetson_clocks script to maximize Jetson performance by setting the static maximum frequencies of the CPU, GPU, and EMC clocks. You can also use the script to show current clock settings, store current clock settings into a file, and restore clock settings from a file.
sudo /usr/bin/jetson_clocks
NVIDIA Jetson TX2 has two CPU core types. These are Denver2 and A57. During benchmarking of Fastvideo SDK we have realized that better performance for J2K encoder and decoder could be achieved with A57 core type. Affinity mask has to be assigned to run only on A57 cores. Linux command taskset assign process affinity.
taskset -c 3,4,5 myprogram
TX2 has the following core numbers: 0 – A57; 1, 2 – Denver2; 3, 4, 5 – A57. Core 0 is used by Linux for interrupt processing. We do not recommend include it in the user affinity mask.
Glass-to-Glass Time Measurements
To check system latency we’ve implemented the software to run G2G tests in the gpu-camera-sample application.
We have the following choices for G2G tests:
- Camera captures frame with current time from high resolution timer at the monitor, we send data from camera to the software, do image processing on GPU and then show processed image at the same monitor close to the window with the timer. If we stop the software, we see two different times and their difference is system latency.
- We have implemented more complicated solution: after image processing on GPU we’ve done JPEG encoding (MJPEG on CPU or on GPU), then send MJPEG stream to receiver process, where we do MJPEG parsing and decoding, then frame output to the monitor. Both processes (sender and receiver) are running at the same PC.
- The same solution as in the previous approach, but with H.264 encoding/decoding (CPU or GPU), both processes are at the same PC.
We can also measure the latency for the case when we stream compressed data from one PC to another over network. Latency depends on camera frame rate, monitor fps, NVIDIA GPU performance, network bandwidth, complexity of image processing pipeline, etc.
Software architecture
gpu-camera-sample is a multithreaded application. It consists of the following threads:
- Main application thread to control app GUI and other threads.
- Image acquisition from a camera thread which controls camera data acquisition and CUDA-based image processing thread.
- CUDA-based image processing thread. Controls RAW data processing, async data writing thread, and OpenGL renderer thread.
- OpenGL rendering thread. Renders processed data into OpenGL surface.
- Async data writing thread. Writes processed JPEG/MJPEG data to SSD or streams processed video.
Here we’ve implemented the simplest approach for camera application. Camera driver is writing raw data to memory ring buffer, then we copy data from that ring buffer to GPU for computations. Full image processing pipeline is done on GPU, so we need just to collect processed frames at the output.
In general case, Fastvideo SDK can import/export data from/to SSD / CPU memory / GPU memory. This is done to ensure compatibility with third-party libraries on CPU and GPU. You can get more info at Fastvideo SDK Manual.
Using gpu-camera-sample
- Run GPUCameraSample.exe
- Press Open button on the toolbar. This will open the first camera in the system or ask to open PGM file (bayer or grayscale) if application was built with no camera support.
- Press Play button. This will start data acquisition from the camera and display it on the screen.
- Adjust zoom with Zoom slider or toggle Fit check box if requires.
- Select appropriate output format in the Recording pane (please check that output folder exists in the file system, otherwise nothing will be recorded) and press Record button to start recording to disk.
- Press Record button again to stop the recording.
Minimum Hardware ans Software Requirements for desktop application
- Windows-10/11, Ubuntu 22.04 or 24.04 64-bit
- The latest NVIDIA driver
- NVIDIA GPU 10xx series minimum
- NVIDIA GPU with 4-8-12 GB memory or better
- Intel Core i5 or better
- NVIDIA CUDA-12.6
- Compiler MSVC 2019 for Windows or gcc 7.4.0 for Linux
We also recommend to check PCI-Express bandwidth for Host-to-Device and Device-to-Host transfers. For GPU with Gen3 x16 it should be in the range of 10-12 GB/s, and for GPU with Gen4 x16 it should be in the range of 20-24 GB/s. GPU memory size could be a bottleneck for image processing from high resolution cameras, so please check GPU memory usage in the software.
If you are working with images which reside on HDD, please place them on SSD or M2.
For testing purposes you can utilize the latest NVIDIA GeForce RTX 3070/3080ti/3090, 4080/4090, 5080/5090 or Jetson TX2, NX and AGX Xavier / Orin.
For continuous high performance applications we recommend professional NVIDIA Quadro RTX Ada GPUs.
Multi-camera applications
To run the software for multi-camera setups, we recommend to run one process per camera. If you have enough GPU memory and processing performance is ok, this is the simplest solution, which was tested in many applications. This is also a good choice for Linux solutions, please don’t forget to turn on CUDA MPS.
You can also create a software module to collect frames from different cameras and process them at the same pipeline with gpu-camera-sample application. In that case you will need less GPU memory which could be important for embedded solutions.
Please bear in mind that this is just a sample application. It’s intended to show how machine vision cameras can work with Fastvideo SDK to get high perfromance image processing on the NVIDIA GPU.
To test a real application with XIMEA cameras (USB3 or PCIe), please have a look at the following page and download FastVCR software. That software with GenICam (GenTL) support will be released soon.
Roadmap
- GPU pipeline for monochrome cameras - done
- GenICam Standard support - done
- Linux version - done
- Software for NVIDIA Jetson hardware and L4T for CUDA-10.2 (Jetson Nano, TX2, Xavier/Orin AGX and NX) - done
- Glass-to-Glass (G2G) test for latency measurements - done
- Support for XIMEA, Bulluf (MATRIX VISION), Basler, FLIR, Imperx, JAI, Daheng Imaging cameras - done
- MJPEG and H.264 streaming with or without FFmpeg RTSP - done
- HEVC (H.265) encoder/decoder - done
- Real-time Image Processing on NVIDIA GPU with Basler pylon - done
- Benchmarks for Jetson Xavier NX - done
- CUDA-11.4 support - done
- Support for MIPI CSI-2 camera which is embedded into TX2 - done
- Support for LUCID Vision Labs cameras - done
- Support for Leopard Imaging MIPI CSI-2 camera with IMX477 image sensor (4K/60fps) - done
- Defringe - done
- FastVCR software for XIMEA cameras - done
- Support for Jetson AGX Orin and CUDA-11.4 - done
- HDR image processing on GPU for automotive 16/20/24-bit image sensors (IMX490, IMX728, AR0820, OX08) - done
- High performance JPEG-XS decoder on GPU - done
- Support for XIMEA MU181CR-ON camera - done
- Support for Jetson AGX Orin CUDA-11.4 and CUDA-12.6 - done
- CUDA-12.6 support - done
- Support for cameras from Emergent Vision Technologies, IO Industries, Hikrobot - done
- Fast undistortion on GPU with precise and compact maps - in progress
- High performance chromatic aberration suppression in RAW domain - in progress
- Support for DALSA, Baumer, Kaya Instruments, SVS-Vistek, MindVision cameras - in progress
- RAW Bayer codec - in progress
- GPU denoiser (NLM and Bilateral algorithms) - in progress
- ALTMapper software (adaptive local tone mapping algorithm) - in progress
- JPEG2000 encoder and decoder on GPU for camera applications
- Interoperability with FFmpeg, UltraGrid, and GStreamer
Frequently asked questions
What is gpu-camera-sample?
A cross-platform (Windows, Linux, NVIDIA Jetson) sample camera application that runs the full image signal processor (ISP) on the NVIDIA GPU in CUDA — from raw capture and debayer to denoising, tone mapping, JPEG / H.264 / H.265 encoding, OpenGL display and RTSP streaming. It is built on the Fastvideo SDK.
Which cameras does it support?
XIMEA (XIMEA SDK), FLIR (Spinnaker), Imperx, LUCID Vision Labs (Arena), and — via the GenICam (GenTL) standard — Basler, MATRIX VISION, JAI, Daheng Imaging and more; Leopard Imaging MIPI CSI-2 cameras on NVIDIA Jetson. Emergent Vision Technologies, IO Industries and Hikrobot cameras are supported as well. A PGM/RAW file simulator lets you test the pipeline without a camera.
How fast is GPU raw image processing?
The full color pipeline reaches up to 4 GPix/s, and a 65 MPix machine-vision camera is processed at 70 fps on a GeForce RTX 4090. See the table above and the Fastvideo SDK benchmarks.
Does it use the NVIDIA Jetson hardware ISP or libargus?
No. gpu-camera-sample / FastVCR implement a full software ISP in CUDA at 16/32-bit precision — a tunable alternative to the fixed-function hardware ISP and libargus. See FastVCR for MIPI CSI2 cameras.
Is it free?
The application runs with a demo Fastvideo SDK (a watermark is shown on the image). For a development or deployment license, contact Fastvideo.
- Fastvideo SDK — CUDA image & video processing SDK (the core of this application)
- FastVCR camera software — XIMEA (USB3/PCIe), MIPI CSI2 on Jetson, Emergent Vision (10/25/50/100-GigE), Hikrobot
- Fast CinemaDNG Processor — RAW / CinemaDNG / MLV processing on GPU · product site fastcinemadng.com
- Codecs & modules — GPU JPEG2000, CUDA JPEG, Motion JPEG, GPU Debayer, GPU Denoiser, RawZipper RAW codec, ALTMapper tone mapping, GPU Transcoder
- All products · Solutions · Blog
Info
Fastvideo SDK Benchmarks
Downloads
评论