GPU Monitoring: How to Track GPU Performance on Linux Servers

For a quick NVIDIA GPU check on Linux, start with nvidia-smi. For an interactive view of GPU processes and VRAM over SSH, use nvtop. For trends and notifications across servers, add a metrics collector and storage or a hosted monitoring platform.

This guide covers Linux GPU monitoring from terminal diagnosis to ongoing collection. It distinguishes VRAM capacity from memory-controller activity and explains where GPU readings need application or hardware diagnostics alongside them.

If you're already comfortable with basic Linux server monitoring, this is the natural next step for GPU-heavy infrastructure.

Quick Answer: Monitor VRAM on Debian or Ubuntu

For NVIDIA, nvidia-smi shows VRAM at the device level. Use nvtop for an interactive process view. If htop does not show GPU activity, check with a GPU-aware tool rather than reading host RAM as VRAM.

nvidia-smi --query-gpu=index,name,memory.used,memory.total --format=csv
sudo apt install nvtop
nvtop

Run these checks where the GPU is visible. For a passed-through device that normally means inside the guest, with its NVIDIA driver installed.

Why GPU Monitoring Matters

A GPU job can slow down because of device saturation, memory pressure, data loading or cooling. Start by observing the GPU and host together; a utilization percentage alone does not identify the cause or prove that a job is healthy.

Proper GPU monitoring catches a few categories of problems that are really hard to debug otherwise.

Track temperature against the limits for your GPU model and server design. Compare readings under a repeatable workload with clock and throttle information before treating a temperature change as a cooling fault.

Track VRAM used and free across a job. Growing allocation can reflect caching, changing batch sizes or a leak; investigate it in the application. A GPU allocation failure is distinct from the Linux host running out of system RAM.

Compare GPU activity with CPU, storage and input-pipeline behavior when a workload underuses a device. Idle periods may be expected between jobs; the useful baseline depends on the service you run.

Keep driver errors and supported hardware-health counters alongside load measurements. Those records support diagnosis; no single sensor reliably predicts the date of a hardware failure.

nvidia-smi: The Starting Point for NVIDIA GPUs

For NVIDIA hardware, first verify that the driver is installed and that the account running your checks can access the GPU. nvidia-smi is supplied with NVIDIA driver tooling. Unsupported fields can report N/A.

Run it without arguments to get a snapshot of all GPUs in the system:

nvidia-smi

This shows you GPU utilization, memory usage, temperature, power draw, and running processes. Useful for a quick check, but the real power comes from its programmatic options.

Continuous Monitoring with nvidia-smi

For repeated terminal samples, use device-monitoring mode. It runs until interrupted; it is not a persistent background service:

nvidia-smi dmon -s pucvmet -d 5

Here p includes power and supported temperatures; u is utilization; c is clocks; v is power/thermal violations; m is memory; e is ECC/PCIe errors; t is PCIe throughput. Field availability varies. See the NVIDIA command reference.

For CSV output that's easier to work with in scripts:

nvidia-smi --query-gpu=timestamp,name,pci.bus_id,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.used,memory.free,power.draw,clocks.current.sm --format=csv -l 10

This gives you a clean CSV with timestamps, GPU names, temperatures, utilization percentages, memory usage, power draw, and clock speeds, refreshed every 10 seconds.

Monitoring per-Process GPU Usage

One of nvidia-smi's most useful features is showing which processes are using GPU resources:

nvidia-smi pmon -s um -d 5

This reports supported per-process activity and framebuffer use. It does not assign processes to a customer, container or training run for you, and some device or virtualization modes omit process metrics.

Querying Specific Metrics

Need just the temperature of GPU 0? nvidia-smi can give you exactly that:

nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader,nounits -i 0

This returns a single number, which makes it perfect for shell scripts and monitoring integrations. Here are some commonly queried fields:

Setting Up nvidia-smi Alerts

The following one-shot example flags temperature and VRAM-capacity thresholds. The numbers are illustrative: choose values for your GPU and workload. It reports missing readings separately and does not send notifications:

#!/usr/bin/env bash
set -euo pipefail
readings=$(nvidia-smi --query-gpu=index,temperature.gpu,memory.used,memory.total --format=csv,noheader,nounits)
printf '%s\n' "$readings" | awk -F, -v temp_limit=85 -v vram_limit=90 '
{
  for (i = 1; i <= NF; i++) gsub(/^[[:space:]]+|[[:space:]]+$/, "", $i)
  if (NF != 4 || $1 !~ /^[0-9]+$/ ||
      $2 !~ /^[0-9]+([.][0-9]+)?$/ ||
      $3 !~ /^[0-9]+([.][0-9]+)?$/ ||
      $4 !~ /^[0-9]+([.][0-9]+)?$/ || $4 <= 0) {
    print "Unavailable GPU reading: " $0 > "/dev/stderr"
    next
  }
  pct = 100 * $3 / $4
  if ($2 >= temp_limit || pct >= vram_limit)
    printf "GPU %s: temperature %s C, VRAM %.1f%%\n", $1, $2, pct
}'

This is a diagnostic building block. Production alerting also needs a scheduler, collection-failure handling, sustained conditions, notification delivery and recovery behavior. To keep history, write timestamped readings to storage rather than relying on terminal output.

nvtop: Real-Time GPU Monitoring in Your Terminal

If you searched for htop for GPU or a Debian app to monitor VRAM, start with nvtop. It provides a terminal view of supported GPUs and their processes. The available metrics and vendors depend on the installed version, build, driver and device permissions.

Install it on Ubuntu/Debian:

sudo apt install nvtop

On Fedora, the package is also available through DNF. RHEL, Rocky and AlmaLinux may require the appropriate EPEL repository first; follow your distribution policy:

sudo dnf install nvtop

Then just run nvtop. You'll get a color-coded dashboard showing utilization graphs, memory bars, temperature readings, and a process list sorted by GPU usage. It supports multiple GPUs and updates in real time.

Use nvtop for live diagnosis; it does not supply a central alerting and retention service. NVTOP installation and compatibility notes describe the supported backends. For a combined CPU/GPU terminal view, btop is another option, but its binary must be built with GPU support and have the required driver libraries.

gpustat: Quick GPU Status at a Glance

gpustat gives you a compact, colorful summary of NVIDIA GPU status. Think of it as nvidia-smi with better formatting:

python3 -m venv .venv-gpustat
. .venv-gpustat/bin/activate
python -m pip install gpustat
gpustat --color --watch

The output is clean: one line per GPU showing utilization, temperature, memory usage, and which users/processes are running. The --watch flag refreshes automatically.

For JSON output you can pipe into other tools:

gpustat --json

It's minimal and focused. Perfect for a quick ssh + gpustat check across your fleet.

NVML: Programmatic GPU Monitoring

NVML is NVIDIA's management library. Python bindings let an agent read supported device metrics directly rather than parsing command output. Support varies across metrics and devices.

Python bindings make this accessible:

python3 -m venv .venv-gpu
. .venv-gpu/bin/activate
python -m pip install nvidia-ml-py

import pynvml

pynvml.nvmlInit()
try:
    for i in range(pynvml.nvmlDeviceGetCount()):
        handle = pynvml.nvmlDeviceGetHandleByIndex(i)
        try:
            memory = pynvml.nvmlDeviceGetMemoryInfo(handle)
            print(f"GPU {i}: {memory.used / 1024**2:.0f} / "
                  f"{memory.total / 1024**2:.0f} MiB VRAM")
        except pynvml.NVMLError as error:
            print(f"GPU {i}: memory reading unavailable ({error})")
finally:
    pynvml.nvmlShutdown()

The maintained package is nvidia-ml-py; its import name is pynvml. Handle unavailable readings explicitly in an agent. The example above reads VRAM capacity only, not process attribution or memory bandwidth.

AMD GPU Monitoring with ROCm

For supported AMD hardware, use the tools matched to your ROCm installation. AMD SMI provides device monitoring, including VRAM usage. Check AMD SMI CLI usage and the supported-device list for your release:

amd-smi monitor --vram-usage

Use the installed CLI help to choose the additional fields and interval available in your version:

amd-smi monitor --help

Older deployments may still use rocm-smi. NVTOP also offers AMD support through its supported Linux backend, subject to kernel, build and permission requirements.

Choose the collector for the GPU vendor you actually run. The Fivenines GPU integration described below collects NVIDIA telemetry; it does not provide AMD or Intel GPU metrics.

Prometheus and Grafana for GPU Monitoring

DCGM Exporter exposes supported NVIDIA telemetry to Prometheus. This is useful when you already operate Prometheus and Grafana or need NVIDIA data center metrics beyond utilization, temperature and VRAM.

Setting Up DCGM Exporter

Follow the current DCGM Exporter installation guide to select a compatible image and permissions. The example below requires Docker with the NVIDIA Container Toolkit and a tag you have selected. It binds the endpoint to loopback for a local check:

# Set DCGM_EXPORTER_TAG to a compatible tag from NVIDIA's container catalog.
: "${DCGM_EXPORTER_TAG:?Set a compatible DCGM Exporter image tag first}"
docker run -d --rm --name dcgm-exporter \
  --gpus all --cap-add SYS_ADMIN \
  -p 127.0.0.1:9400:9400 \
  "nvcr.io/nvidia/k8s/dcgm-exporter:${DCGM_EXPORTER_TAG}"

Verify it's working:

curl localhost:9400/metrics | grep DCGM_FI_DEV_GPU_TEMP

After verifying the local endpoint, Prometheus in the same host network can scrape it as below. A remote Prometheus server cannot use this loopback binding: expose the endpoint on a controlled private interface with access restricted to the scraper. Inside a container, 127.0.0.1 refers to that container, not the Docker host.

scrape_configs:
  - job_name: 'gpu-metrics'
    static_configs:
      - targets: ['127.0.0.1:9400']

Key DCGM Metrics to Track

The Reality of Prometheus GPU Monitoring

Setting up Prometheus and Grafana for GPU monitoring works, but the effort is real. You need to deploy and maintain Prometheus, configure storage retention and resource limits, set up Grafana with dashboards (or find community dashboards that mostly work), configure alerting rules in PromQL, and manage this infrastructure across your fleet.

If your team already maintains Prometheus, adding GPU metrics can fit that existing workflow. Otherwise, account for storage, dashboards and notification operations when comparing it with a hosted server monitoring tool. Setup effort depends on your environment.

Monitoring GPU Health: Beyond Utilization

GPU utilization and temperature are the obvious metrics, but production GPU monitoring needs to go deeper.

ECC Memory Errors

ECC telemetry is available only on supported hardware and configurations. Corrected and uncorrected errors have different meanings; interpret counts together with driver logs, workload impact and NVIDIA's guidance for that GPU generation.

nvidia-smi --query-gpu=ecc.errors.corrected.volatile.total,ecc.errors.uncorrected.volatile.total --format=csv

Investigate new or recurring errors and retain diagnostic evidence. A corrected-error counter alone is not a replacement instruction. Follow NVIDIA GPU debug guidance for the observed error and device.

XID Errors

XID errors are NVIDIA's error codes logged to the kernel. They cover everything from driver bugs to hardware failures:

dmesg | grep -i "NVRM: Xid"

Examples from NVIDIA's Xid documentation are listed below. The code narrows the investigation; it does not by itself prove a failed GPU:

PCIe Link Speed and Width

GPU bus monitoring can mean PCIe link state or actual traffic. Read the current link generation and width first. Gen3 x8 has roughly one quarter of Gen4 x16's theoretical one-direction bandwidth, but application performance does not scale by that ratio automatically:

nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current --format=csv

Compare link state under load with the topology and slot configuration you expect; idle power management can lower the current generation. For supported PCIe receive/transmit throughput, use nvidia-smi dmon -s t -d 1. This is distinct from VRAM capacity and memory-controller utilization.

Power Limit and Thermal Throttling

Check if your GPUs are power-limited or thermal-limited:

nvidia-smi --query-gpu=power.draw,power.limit,clocks_throttle_reasons.hw_thermal_slowdown,clocks_throttle_reasons.hw_power_brake_slowdown --format=csv

A reported slowdown reason is an investigation lead. Compare it with temperature, clocks, workload and configured power limits. Query names and availability vary by driver: inspect nvidia-smi --help-query-gpu if a field is rejected.

GPU Monitoring in Docker and Kubernetes

If your GPU workloads run in containers, monitoring requires some additional setup.

Docker GPU Monitoring

With the NVIDIA Container Toolkit configured, verify GPU visibility using the NVIDIA sample workload. This is a diagnostic test, not deployment of a monitoring service:

docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

GPU process IDs and container process IDs must be compared in the same PID namespace. Inspect the host-side GPU process list, then inspect processes for a specific running container:

nvidia-smi pmon -c 1 -s um
# Replace your-container with the actual running container name.
docker top your-container -eo pid,comm

If you're already using Docker container monitoring for CPU and memory, GPU metrics are the missing piece for GPU-accelerated workloads.

Kubernetes GPU Monitoring

For Kubernetes, use the maintained DCGM Exporter Helm chart or the monitoring components managed by NVIDIA GPU Operator. Deployment depends on the cluster runtime, device plugin and whether DCGM already runs on the nodes.

Follow the DCGM Exporter deployment documentation for a compatible chart and its values. Do not deploy a second exporter if GPU Operator already manages one.

Pod and namespace attribution requires the corresponding Kubernetes integration and allocation metadata. A per-GPU metrics endpoint alone does not establish which pod owns a GPU. Verify those labels in the exported metrics before building pod-level dashboards.

Best Practices for GPU Monitoring

Now that we've covered the tools, here's what actually matters when you go to implement this.

Set temperature thresholds from the GPU model's limits and the server's operating baseline, not a universal 80°C/85°C rule. Check clocks and throttle information alongside temperature.

For VRAM capacity, calculate 100 × memory.used / memory.total and consider absolute free space too. utilization.memory measures memory-controller activity; it is not the fraction of VRAM occupied. An application can fail a large allocation even with some free VRAM.

Use windows appropriate to the workload. A short idle gap can be normal; prolonged low activity may indicate a stalled input pipeline, an empty queue or a completed job. Confirm the cause with application signals.

Use power readings for operational context. They do not directly measure useful work or equal a cloud invoice, and their meaning depends on GPU architecture and workload.

Where supported, retain ECC and driver-error evidence from the start. Compare new events with the workload and use vendor diagnostics before deciding on recovery or replacement.

Choose a collection interval that balances diagnostic needs, overhead and storage. Shorter intervals are useful for transient investigation; hosted periodic collection is better suited to ongoing trends than subsecond profiling.

GPU Monitoring Without the Infrastructure Overhead

For teams that want hosted charts and notifications alongside their server metrics, Fivenines collects NVIDIA GPU readings through NVML. It uses a default 60-second collection interval, with history according to your plan.

The agent needs an accessible NVIDIA driver, its Python NVML binding and GPU collection enabled. It records supported utilization, temperature, VRAM, power, fan, SM/memory clocks and process-memory readings. It does not collect ECC, Xid, PCIe/NVLink or per-MIG-instance metrics. GPU health workflows cover temperature, GPU and memory-controller activity, and power relative to the reported limit; add host and application checks for service failures.

Review the requirements for Fivenines NVIDIA GPU monitoring or start a 14-day trial. No credit card is required. For the broader choice between local utilities and monitoring platforms, see our GPU monitoring software comparison.