DCGM-Exporter
This repository contains the DCGM-Exporter project. It exposes GPU metrics exporter for Prometheus leveraging NVIDIA DCGM.
Documentation
Official documentation for DCGM-Exporter can be found on docs.nvidia.com.
Quickstart
To gather metrics on a GPU node, simply start the dcgm-exporter container:
docker run -d --gpus all --cap-add SYS_ADMIN --rm -p 9400:9400 nvcr.io/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless
curl localhost:9400/metrics
# HELP DCGM_FI_DEV_SM_CLOCK SM clock frequency (in MHz).
# TYPE DCGM_FI_DEV_SM_CLOCK gauge
# HELP DCGM_FI_DEV_MEM_CLOCK Memory clock frequency (in MHz).
# TYPE DCGM_FI_DEV_MEM_CLOCK gauge
# HELP DCGM_FI_DEV_MEMORY_TEMP Memory temperature (in C).
# TYPE DCGM_FI_DEV_MEMORY_TEMP gauge
...
DCGM_FI_DEV_SM_CLOCK{gpu="0", UUID="GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52"} 139
DCGM_FI_DEV_MEM_CLOCK{gpu="0", UUID="GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52"} 405
DCGM_FI_DEV_MEMORY_TEMP{gpu="0", UUID="GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52"} 9223372036854775794
...
Quickstart on Kubernetes
Note: Consider using the NVIDIA GPU Operator rather than DCGM-Exporter directly.
Ensure you have already setup your cluster with the default runtime as NVIDIA.
The recommended way to install DCGM-Exporter is to use the Helm chart:
helm repo add gpu-helm-charts \
https://nvidia.github.io/dcgm-exporter/helm-charts
Update the repo:
helm repo update
And install the chart:
helm install \
--generate-name \
gpu-helm-charts/dcgm-exporter
Once the dcgm-exporter pod is deployed, you can use port forwarding to obtain metrics quickly:
kubectl create -f https://raw.githubusercontent.com/NVIDIA/dcgm-exporter/master/dcgm-exporter.yaml
# Let's get the output of a random pod:
NAME=$(kubectl get pods -l "app.kubernetes.io/name=dcgm-exporter" \
-o "jsonpath={ .items[0].metadata.name}")
kubectl port-forward $NAME 8080:9400 &
curl -sL http://127.0.0.1:8080/metrics
# HELP DCGM_FI_DEV_SM_CLOCK SM clock frequency (in MHz).
# TYPE DCGM_FI_DEV_SM_CLOCK gauge
# HELP DCGM_FI_DEV_MEM_CLOCK Memory clock frequency (in MHz).
# TYPE DCGM_FI_DEV_MEM_CLOCK gauge
# HELP DCGM_FI_DEV_MEMORY_TEMP Memory temperature (in C).
# TYPE DCGM_FI_DEV_MEMORY_TEMP gauge
...
DCGM_FI_DEV_SM_CLOCK{gpu="0", UUID="GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52",container="",namespace="",pod=""} 139
DCGM_FI_DEV_MEM_CLOCK{gpu="0", UUID="GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52",container="",namespace="",pod=""} 405
DCGM_FI_DEV_MEMORY_TEMP{gpu="0", UUID="GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52",container="",namespace="",pod=""} 9223372036854775794
...
To integrate DCGM-Exporter with Prometheus and Grafana, see the full instructions in the user guide.
dcgm-exporter is deployed as part of the GPU Operator. To get started with integrating with Prometheus, check the Operator user guide.
TLS and Basic Auth
Exporter supports TLS and basic auth using exporter-toolkit. To use TLS and/or basic auth, users need to use --web-config-file CLI flag as follows
dcgm-exporter --web-config-file=web-config.yaml
A sample web-config.yaml file can be fetched from exporter-toolkit repository. The reference of the web-config.yaml file can be consulted in the docs.
IPv6 Support
DCGM-Exporter supports IPv6 addresses for both the remote hostengine connection (-r) and the metrics listen address (-a). IPv6 addresses must use bracket notation when combined with a port.
Remote Hostengine (CLI)
dcgm-exporter -r "[::1]:5555"
Remote Hostengine (Environment Variable)
export DCGM_REMOTE_HOSTENGINE_INFO="[::1]:5555"
dcgm-exporter
Metrics Listen Address
dcgm-exporter -a "[::]:9400"
Note: The brackets in [::1]:5555 are required by the DCGM connection protocol. When using the CLI, the shell requires quoting (double or single quotes) around the address to prevent bracket interpretation.
Prerequisites
The remote nv-hostengine must be configured to listen on IPv6. Refer to the DCGM documentation for configuring nv-hostengine bind address options.
How to include HPC jobs in metric labels
The DCGM-exporter can include High-Performance Computing (HPC) job information into its metric labels. To achieve this, HPC environment administrators must configure their HPC environment to generate files that map GPUs to HPC jobs.
File Conventions
These mapping files follow a specific format:
- Each file is named after either a unique GPU ID or a unique GPU ID and a GPU instance (MIG) ID separated with a “.” (e.g., 0, 1, 2.0, 2.1, 3, etc.).
- Each line in the file contains JOB IDs that run on the corresponding GPU/MIG instance.
Enabling HPC Job Mapping on DCGM-Exporter
To enable GPU-to-job mapping on the DCGM-exporter side, users must run the DCGM-exporter with the --hpc-job-mapping-dir command-line parameter, pointing to a directory where the HPC cluster creates job mapping files. Or, users can set the environment variable DCGM_HPC_JOB_MAPPING_DIR to achieve the same result.
Building from Source
In order to build dcgm-exporter ensure you have the following:
git clone https://github.com/NVIDIA/dcgm-exporter.git
cd dcgm-exporter
make binary
sudo make install
...
dcgm-exporter &
curl localhost:9400/metrics
# HELP DCGM_FI_DEV_SM_CLOCK SM clock frequency (in MHz).
# TYPE DCGM_FI_DEV_SM_CLOCK gauge
# HELP DCGM_FI_DEV_MEM_CLOCK Memory clock frequency (in MHz).
# TYPE DCGM_FI_DEV_MEM_CLOCK gauge
# HELP DCGM_FI_DEV_MEMORY_TEMP Memory temperature (in C).
# TYPE DCGM_FI_DEV_MEMORY_TEMP gauge
...
DCGM_FI_DEV_SM_CLOCK{gpu="0", UUID="GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52"} 139
DCGM_FI_DEV_MEM_CLOCK{gpu="0", UUID="GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52"} 405
DCGM_FI_DEV_MEMORY_TEMP{gpu="0", UUID="GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52"} 9223372036854775794
...
Changing Metrics
With dcgm-exporter you can configure which fields are collected by specifying a custom CSV file.
You will find the default CSV file under etc/default-counters.csv in the repository, which is copied on your system or container to /etc/dcgm-exporter/default-counters.csv
The layout and format of this file is as follows:
# Format
# If line starts with a '#' it is considered a comment
# DCGM FIELD, Prometheus metric type, help message
# Clocks
DCGM_FI_DEV_SM_CLOCK, gauge, SM clock frequency (in MHz).
DCGM_FI_DEV_MEM_CLOCK, gauge, Memory clock frequency (in MHz).
A custom csv file can be specified using the -f option or --collectors as follows:
dcgm-exporter -f /tmp/custom-collectors.csv
Notes:
Profiling Metrics
Please note that for Ampere and earlier generation GPUs, profiling metrics depend on the datacenter-gpu-manager-4-proprietary package. This package is included in the container.
What about a Grafana Dashboard?
You can find the official NVIDIA DCGM-Exporter dashboard here: https://grafana.com/grafana/dashboards/12239
You will also find the json file on this repo under grafana/dcgm-exporter-dashboard.json
You can find the DCGM-Exporter OpenObserve dashboard here
You can find the NVIDIA DCGM-Exporter dashboard here: https://github.com/openobserve/dashboards/tree/main/NVIDIA GPU Monitoring
To integrate DCGM-Exporter with OpenObserve, follow the blog monitoring GPU with OpenObserve
Pull requests are accepted!
Building the containers
This project uses docker buildx for multi-arch image creation. Follow the instructions on that page to get a working builder instance for creating these containers. Some other useful build options follow.
Builds local images based on the machine architecture and makes them available in ‘docker images’
make local
Build the distroless image and export to ‘docker images’
make distroless PLATFORMS=linux/amd64 OUTPUT=type=docker
Build and push the images to some other ‘private_registry’
make REGISTRY=<private_registry> push
Issues and Contributing
Checkout the Contributing document!
Reporting Security Issues
We ask that all community members and users of DCGM Exporter follow the standard NVIDIA process for reporting security vulnerabilities. This process is documented at the NVIDIA Product Security website.
Following the process will result in any needed CVE being created as well as appropriate notifications being communicated
to the entire DCGM Exporter community. NVIDIA reserves the right to delete vulnerability reports until they’re fixed.
Please refer to the policies listed there to answer questions related to reporting security issues.
评论