GPU temperature
GPU Temp Monitor NVIDIA: Setup, Logs and Alerts
Run nvidia-smi -q -d TEMPERATURE to retrieve the current GPU temperature and the card's thermal limits, then read the GPU Current Temp field. For a timed workload trace, use nvidia-smi query logging at a one-second interval and compare temperature with clock behaviour. A training job has slowed down,