How to Monitor CPU Usage in Linux Like an SRE

At 03:14, a CPU alert fires across a Linux fleet. The dashboard shows 92% utilization, yet requests still meet their service objective. A quick glance at top suggests an emergency, while the application looks healthy. That mismatch doesn't mean Linux is lying. It means the headline number needs context.

Linux CPU monitoring is built on cumulative kernel counters, sampling intervals, and aggregation choices. The useful question isn't only whether CPU is high. It's which CPU, which state, which workload, and for how long. A host can have spare capacity overall while one core is saturated, or appear comfortable while a container is being throttled by its cgroup limit.

This guide follows the path an SRE uses during an incident, from direct /proc/stat inspection to retained history, per-core diagnosis, container visibility, and alert routing that doesn't wake someone for every short-lived batch spike.

Table of Contents

• The Real Story Behind Linux CPU Numbers
  • Read the state, not only the total
• One-Off CLI Inspection With top htop mpstat and ps
  • Use mpstat when the headline looks wrong
• Keeping CPU History With sar and sysstat
  • What sar keeps, and what it doesn't
• Per-Core cgroup and GPU-Aware CPU Visibility
  • Per-core saturation changes the diagnosis
  • Cgroups explain healthy hosts with slow workloads
  • GPU workloads need a shared resource view
• From DIY Scripts to a Monitoring Platform
  • Three practical operating models
• Alerting Thresholds and Notification Routing That Actually Work
  • Build the signal from multiple dimensions
  • Route by consequence and ownership
• A Tiered Monitoring Routine for Busy Linux Teams
  • Tier one answers what is happening now
  • Tier two preserves local evidence
  • Tier three supports fleet decisions

The Real Story Behind Linux CPU Numbers

The kernel exposes CPU time through /proc/stat and its documented CPU fields. The aggregate cpu line and individual cpuN lines contain cumulative time spent in user, nice, system, idle, iowait, interrupt, softirq, steal, and guest-related states. Those counters are measured in USER_HZ, so a monitoring tool must compare two samples, calculate the deltas, and estimate how time was divided during the interval.

top, mpstat, and sar therefore don't read a magical instantaneous CPU percentage. They sample the same underlying counters at different intervals and present different calculations. A brief burst can dominate a short sample, while a broad average can conceal a pinned core. The aggregate line is useful for host capacity, but the per-core lines are essential when a single execution path is limiting latency.

Read the state, not only the total

User time represents application work. System time reflects kernel activity. Iowait indicates time associated with CPUs waiting on I/O, while steal identifies time a virtual machine was ready to run but didn't receive a physical CPU allocation. Idle is the remaining capacity, but even idle time needs interpretation when runnable work is stuck behind scheduling, device, or cgroup constraints.

CPU pressure information adds another perspective. The kernel's pressure interface, including /proc/pressure/cpu, helps show whether tasks are delayed waiting for CPU, rather than merely reporting how much processor time was busy. That distinction matters during latency incidents where utilization alone looks ordinary.

Practical rule: A CPU percentage is a calculation over a window. Always record the interval, the CPU state, and whether the value is aggregated across cores.

For teams establishing a repeatable baseline, a documented Linux performance baseline gives those readings a reference point. Without a baseline, an alert compares current behavior with an arbitrary threshold instead of with the workload's normal operating envelope.

One-Off CLI Inspection With top htop mpstat and ps

During a live incident, the fastest useful sequence is layered. top answers what is happening now, htop makes the same information easier to scan, mpstat exposes the per-core breakdown, and ps turns process ranking into a scriptable output.

Start with:

top

top gives a dynamic process list, load averages, and CPU state totals. It's excellent first-response tooling because it's commonly available and requires no separate monitoring service. It isn't an alerting system, though, and closing the terminal discards the view.

htop is better for a human scan. Its per-core bars, process tree, sorting controls, and interactive signals make it easier to connect a hot process with its parent service. On a multi-socket system, those individual bars can expose one saturated execution lane hidden by the host average.

Use mpstat when the headline looks wrong

For a deliberate sample, use:

mpstat -P ALL 2 5

The -P ALL option prints each logical CPU, while the interval and count make the observation explicit. The output separates user, nice, system, iowait, interrupt, softirq, steal, guest, and idle states. The first report can represent accumulated time since boot, so the later interval reports are the useful ones for current diagnosis.

The command is especially valuable when an application feels slow but the aggregate CPU line appears moderate. A single busy core can be the limiting resource even when the machine has substantial idle capacity elsewhere. Oracle-oriented sysstat guidance also recommends mpstat -P ALL and sar -u -P ALL for per-core statistics, with load average interpreted relative to the number of cores rather than as a universal percentage.

For automation or a quick incident paste, use:

ps -eo pid,ppid,pcpu,pmem,comm --sort=-pcpu

That output identifies likely process offenders, but it doesn't prove causality. A hot process may be the victim of retries, lock contention, or downstream delay. Pair it with per-core and state-level readings.

A practical CPU usage command reference for Linux can help standardize the first commands in an incident runbook. The recommended order remains simple: use top to orient, htop to inspect interactively, mpstat to validate the signal, and ps when another tool needs structured process data.

Keeping CPU History With sar and sysstat

Live tools answer what the server is doing now. They can't answer when saturation started, whether a deployment changed the pattern, or whether the same behavior appeared during an earlier batch window. sar, supplied by the sysstat package, fills that local history gap.

For a live sample:

sar -u 1 3

This records three one-second intervals and reports the CPU states in a format related to mpstat. For per-core history, use:

sar -u -P ALL

The collector stores daily binary files, commonly under /var/log/sa or /var/log/sysstat, depending on the distribution. Archived files such as sa24 can be queried for an earlier day, which makes sar useful after deployments, batch jobs, and incidents that weren't actively watched.

What sar keeps, and what it doesn't

Default local retention is 7 days on Debian and Ubuntu and 28 days on RHEL, as documented in this Linux CPU monitoring history guide. Those defaults are useful for short forensic work, but they're not a long-term capacity archive. The oldest data is rotated out, and the files aren't convenient relational datasets that several engineers can query concurrently.

The same collector can provide neighboring signals:

• CPU context: sar -u and sar -P ALL show aggregate and per-CPU states.
• Memory context: sar -r helps determine whether CPU behavior accompanies memory pressure.
• Block activity: sar -b adds block I/O context when iowait rises.
• Network context: sar -n DEV helps correlate interrupt or softirq activity with interface traffic.

Use sadf when the archived data needs export for another system or visualization format. Configuration controls collection and retention, so operators should verify that the collector is enabled and that rotation matches the incident investigation window.

Local sar history is a valuable black box, but it isn't a fleet observability platform.

Teams should document those limits in their data retention policy. If an investigation may span months, regions, or multiple hosts, the metrics need to leave the individual server and enter a shared store with access controls, durable retention, and cross-host queries.

Per-Core cgroup and GPU-Aware CPU Visibility

Host-wide CPU averages hide several failure modes that repeatedly appear in production. One thread can pin a core while other cores remain mostly idle. A container can hit its CPU quota and be throttled while the host still has capacity. An inference workload can compete across CPU preprocessing, device execution, and result handling while a dashboard shows only one side of the system.

Per-core saturation changes the diagnosis

mpstat -P ALL exposes the first blind spot. turbostat can add processor and package-level detail where frequency, power behavior, and socket topology matter. The key is to compare the aggregate line with each logical CPU, then connect the busy core to process placement, thread affinity, and scheduler behavior.

A multi-threaded service doesn't automatically use all available cores effectively. Thread pinning, locks, runtime constraints, and serial sections can leave one execution path at its ceiling. A host average may therefore look safe while request latency rises.

Cgroups explain healthy hosts with slow workloads

Container-aware monitoring must inspect the workload's cgroup rather than stopping at the node. Linux exposes CPU accounting and throttling information through cgroup files such as cpu.stat, including usage and throttling counters. In Kubernetes, Podman, and systemd-managed services, those values reveal whether a workload is being limited by policy or competing with neighboring workloads.

Watch for increasing throttled time or nr_throttled alongside application latency. A node-level graph can remain calm because unused CPU elsewhere doesn't help a container that has reached its assigned quota. Neutral container monitoring documentation describes this distinction in its Podman container metrics reference.

GPU workloads need a shared resource view

Inference and media pipelines often move between CPU preprocessing, kernel launches, memory transfers, and post-processing. nvidia-smi dmon provides a direct device-side view, while DCGM supports more systematic NVIDIA telemetry. Those readings should sit beside CPU per-core, cgroup, memory, and network signals.

The operational principle is straightforward: host average, core, cgroup, and device views answer different questions. During an escalating incident, omitting any one of them can send remediation toward the wrong resource.

From DIY Scripts to a Monitoring Platform

A shell loop that samples /proc/stat and ships values to a time-series store is a reasonable starting point for a small lab. It teaches the counter-delta calculation and makes the collection path visible. It becomes fragile when operators need shared dashboards, durable retention, permissions, on-call routing, and consistent labels across a fleet.

The first break usually isn't the math. It's everything around the math. Scripts need process supervision, retry handling, authentication, schema changes, clock considerations, local buffering, and a way to prevent duplicate alerts. A cron job can collect data, but it doesn't automatically provide a reliable incident workflow.

Three practical operating models

DIY collection keeps ownership close to the server. It suits experiments and tightly bounded environments, but every operational feature becomes another script or service to maintain.

An open-source agent stack typically combines node_exporter, Prometheus, Grafana, and Alertmanager. This route offers flexibility and a large ecosystem, but the team still owns upgrades, storage, access control, dashboard design, alert deduplication, and notification integration.

A managed platform shifts those operational responsibilities to a service. Fivenines is one option in this category, providing an open-source Linux agent, CPU and host metrics, process visibility, container and NVIDIA GPU insights, dashboards, alert workflows, and external uptime checks in the same monitoring environment. It can replace several separately operated components for teams that prefer a unified control plane.

The choice depends on the boundary an organization wants to own. A platform doesn't make mpstat or top obsolete. Those commands remain faster for immediate host triage, especially when the agent is unavailable or the question concerns a single process.

Keep the CLI for diagnosis. Use continuous collection when the answer must survive the SSH session.

A platform earns its place when engineers need to correlate CPU with latency, compare hosts, retain trends, assign roles, and route alerts without asking one operator to maintain every storage and notification detail. The trade-off is less control over internals than a fully self-managed stack, balanced against less infrastructure to operate.

Alerting Thresholds and Notification Routing That Actually Work

A useful CPU alert must distinguish busy, blocked, throttled, and user-visible. A host average alone can't make that distinction. Alert logic should combine sustained behavior with per-core saturation, iowait, load or run-queue context, cgroup throttling, and service health.

A practical rule set can begin with sustained conditions rather than single samples. Use a warning for sustained high utilization and a critical condition for sustained severe utilization, but calibrate those levels against the service's baseline and latency objective. The specific thresholds in an alert rule should be treated as policy, not as universal Linux constants.

Build the signal from multiple dimensions

Prometheus users can examine node_cpu_seconds_total by mode and evaluate iowait over a sustained window. Pair that with per-core utilization so one saturated logical CPU isn't diluted by idle neighbors. For workload limits, include cgroup throttling metrics and throttled duration where the runtime exposes them.

Load and run-queue context helps separate genuine scheduling pressure from CPU time spent waiting on storage. CPU pressure stall information adds another view of delayed work. A rate-of-change condition can suppress short batch spikes, while a sustained condition catches a workload that remains constrained.

Route by consequence and ownership

Low-severity warnings can go to a team channel for review. A critical condition should page the service's primary on-call only when the signal remains sustained or coincides with an SLO symptom. Resolved events should close the associated incident or ticket automatically, otherwise the notification system creates administrative noise after recovery.

Every alert needs a runbook link and a dashboard link. The runbook should tell the responder to check per-core output, CPU states, pressure, cgroup throttling, recent deployments, and application latency. For teams implementing this workflow, a focused guide to setting up infrastructure alerts can help connect threshold logic with notification routing.

Burn-rate alerting tied to an SLO is often more meaningful than a static CPU threshold. It asks whether the service is consuming its error budget quickly, then uses CPU as diagnostic context rather than treating utilization as the customer impact itself.

A Tiered Monitoring Routine for Busy Linux Teams

A durable routine assigns each CPU question to the smallest toolset that can answer it accurately. That keeps incident response fast without pretending that a terminal snapshot can provide historical or fleet-wide evidence.

Tier one answers what is happening now

During live triage, start with top or htop to identify active processes and load context. Use mpstat -P ALL to test for a pinned core, inspect iowait and steal, then read /proc/stat directly when a script or exporter needs the raw counters. ps provides a compact process ranking for incident notes and automation.

The question at this tier is immediate: is something on fire right now?

Tier two preserves local evidence

Enable sysstat and use sar for scheduled local retention. Archived CPU, memory, block, and network data can show when a pattern began and whether it aligns with a deployment or scheduled workload. This tier is inexpensive and useful, but its history remains tied to the host and its configured rotation.

The question changes to: when did saturation start, and what changed around that time?

Tier three supports fleet decisions

Continuous agent collection adds shared dashboards, alert routing, cross-host comparisons, and longer-horizon capacity analysis. Teams can evaluate whether a service is approaching a capacity cliff, whether one workload repeatedly triggers throttling, and whether CPU behavior correlates with user-facing latency.

The question becomes: is the current growth pattern creating a future capacity risk?

AI-assisted anomaly detection can add another layer to continuous streams by identifying seasonal behavior, silent regressions, and unusual combinations of CPU pressure, throttling, and latency that static thresholds miss. It doesn't replace the underlying counters or the runbook. It helps operators find the deviations worth investigating before a customer-facing failure becomes obvious.

Fivenines combines Linux CPU, memory, disk, network, process, container, and NVIDIA GPU visibility with alert workflows and uptime monitoring through an open-source agent. Visit Fivenines to evaluate whether a unified dashboard can replace manual CPU checks and give the operations team durable, actionable history.