How to Monitor CPU Usage in Linux Like an SRE

At 03:14, a CPU alert fires across a Linux fleet. The dashboard shows 92% utilization, yet requests still meet their service objective. A quick glance at top suggests an emergency, while the application looks healthy. That mismatch doesn't mean Linux is lying. It means the headline number needs context.

Linux CPU monitoring is built on cumulative kernel counters, sampling intervals, and aggregation choices. The useful question isn't only whether CPU is high. It's which CPU, which state, which workload, and for how long. A host can have spare capacity overall while one core is saturated, or appear comfortable while a container is being throttled by its cgroup limit.

This guide focuses on Linux CPU usage history: collect interval samples, retain them with sar and sysstat, and compare a past incident with per-core and cgroup evidence. Historical data must be collected before the incident; installing a tool afterwards cannot reconstruct missing samples. For a quick check of a host that is slow right now, use the Linux CPU command guide.

Table of Contents

• The Real Story Behind Linux CPU Numbers
  • Read the state, not only the total
• One-Off CLI Inspection With top htop mpstat and ps
  • Use mpstat when the headline looks wrong
• Keeping CPU History With sar and sysstat
  • What sar keeps, and what it doesn't
• Per-Core cgroup and GPU-Aware CPU Visibility
  • Per-core saturation changes the diagnosis
  • Cgroups explain healthy hosts with slow workloads
  • GPU workloads need a shared resource view
• From DIY Scripts to a Monitoring Platform
  • Three practical operating models
• Alerting Thresholds and Notification Routing That Actually Work
  • Build the signal from multiple dimensions
  • Route by consequence and ownership
• A Tiered Monitoring Routine for Busy Linux Teams
  • Tier one answers what is happening now
  • Tier two preserves local evidence
  • Tier three supports fleet decisions

The Real Story Behind Linux CPU Numbers

The kernel exposes CPU time through /proc/stat and its documented CPU fields. The aggregate cpu line and individual cpuN lines contain cumulative time spent in user, nice, system, idle, iowait, interrupt, softirq, steal, and guest-related states. Those counters are measured in USER_HZ, so a monitoring tool must compare two samples, calculate the deltas, and estimate how time was divided during the interval.

top, mpstat, and sar therefore don't read a magical instantaneous CPU percentage. They sample the same underlying counters at different intervals and present different calculations. A brief burst can dominate a short sample, while a broad average can conceal a pinned core. The aggregate line is useful for host capacity, but the per-core lines are essential when a single execution path is limiting latency.

Read the state, not only the total

User time represents application work. System time reflects kernel activity. Iowait accounts for idle CPU time while disk I/O is outstanding; it does not mean the CPU itself is blocked. Linux documents limitations in this accounting, so check storage latency and application behavior before assigning a cause. Steal is involuntary virtual CPU wait while the hypervisor services another virtual processor. Spare host capacity also does not remove a workload's CPU quota or affinity restrictions.

CPU pressure information adds another perspective. The kernel's pressure interface, including /proc/pressure/cpu, helps show whether tasks are delayed waiting for CPU, rather than merely reporting how much processor time was busy. That distinction matters during latency incidents where utilization alone looks ordinary.

Practical rule: A CPU percentage is a calculation over a window. Always record the interval, the CPU state, and whether the value is aggregated across cores.

For teams establishing a repeatable baseline, a documented Linux performance baseline gives those readings a reference point. Without a baseline, an alert compares current behavior with an arbitrary threshold instead of with the workload's normal operating envelope.

One-Off CLI Inspection With top htop mpstat and ps

During a live incident, the fastest useful sequence is layered. top answers what is happening now, htop makes the same information easier to scan, mpstat exposes the per-core breakdown, and ps turns process ranking into a scriptable output.

Start with:

top

top gives a dynamic process list, load averages, and CPU state totals. It's excellent first-response tooling because it's commonly available and requires no separate monitoring service. It isn't an alerting system, though, and closing the terminal discards the view.

htop is better for a human scan. Its per-core bars, process tree, sorting controls, and interactive signals make it easier to connect a hot process with its parent service. On a multi-socket system, those individual bars can expose one saturated execution lane hidden by the host average.

Use mpstat when the headline looks wrong

For a deliberate sample, use:

mpstat -P ALL 2 5

The -P ALL option prints the aggregate and each logical CPU. In this command, 2 5 requests five reports covering two-second intervals. The output separates user, nice, system, iowait, interrupt, softirq, steal, guest, and idle states. With mpstat, an omitted or zero interval instead gives statistics since boot; do not confuse that with the interval reports requested here.

The command is useful when an application feels slow but aggregate CPU looks moderate: one busy core can limit progress while others remain idle. The official mpstat manual documents -P ALL for per-CPU reports. Keep load averages separate from utilization percentages, and compare the same time window when moving from the host summary to individual cores.

For automation or a quick incident paste, use:

ps -eo pid,ppid,pcpu,pmem,comm --sort=-pcpu

This ps ranking uses CPU time divided by each process's elapsed lifetime. It identifies candidates, but it is not a measurement of the last second and does not prove causality. Follow up with pidstat -u 1 5 for five one-second process reports, then compare them with per-core and CPU-state readings.

A practical CPU usage command reference for Linux can help standardize the first commands in an incident runbook. The recommended order remains simple: use top to orient, htop to inspect interactively, mpstat to validate the signal, and ps when another tool needs structured process data.

Keeping CPU History With sar and sysstat

Live tools answer what the server is doing now. They can't answer when saturation started, whether a deployment changed the pattern, or whether the same behavior appeared during an earlier batch window. sar, supplied by the sysstat package, fills that local history gap.

For a live sample:

sar -u 1 3

This displays three one-second reports. It does not create a persistent archive by itself: sysstat collection must already be enabled, or data must be written explicitly with sar -o. To read today's stored per-core history, when those samples exist, use:

sar -u -P ALL

The collector stores daily binary files, commonly under /var/log/sa or /var/log/sysstat, depending on the distribution. Archived files such as sa24 can be queried for an earlier day, which makes sar useful after deployments, batch jobs, and incidents that weren't actively watched.

What sar keeps, and what it doesn't

Check the collector schedule, data directory, and HISTORY setting on the actual host. Retention is configurable, and distribution packages can choose different defaults; a distribution name alone is not a reliable retention policy. The sysstat configuration reference documents the setting. Local files rotate, so confirm the incident window is still present before relying on them for a review.

The same collector can provide neighboring signals:

• CPU context: sar -u and sar -P ALL show aggregate and per-CPU states.
• Memory context: sar -r helps determine whether CPU behavior accompanies memory pressure.
• Block activity: sar -b adds block I/O context when iowait rises.
• Network context: sar -n DEV helps correlate interrupt or softirq activity with interface traffic.

Use sadf when the archived data needs export for another system or visualization format. Configuration controls collection and retention, so operators should verify that the collector is enabled and that rotation matches the incident investigation window.

Local sar history is a valuable black box, but it isn't a fleet observability platform.

Teams should document those limits in their data retention policy. If an investigation may span months, regions, or multiple hosts, the metrics need to leave the individual server and enter a shared store with access controls, durable retention, and cross-host queries.

Per-Core cgroup and GPU-Aware CPU Visibility

Host-wide CPU averages hide several failure modes that repeatedly appear in production. One thread can pin a core while other cores remain mostly idle. A container can hit its CPU quota and be throttled while the host still has capacity. An inference workload can compete across CPU preprocessing, device execution, and result handling while a dashboard shows only one side of the system.

Per-core saturation changes the diagnosis

mpstat -P ALL 1 5 exposes the first blind spot. turbostat can add processor and package-level detail where frequency, power behavior, and socket topology matter. The key is to compare the aggregate line with each logical CPU, then connect the busy core to process placement, thread affinity, and scheduler behavior.

A multi-threaded service doesn't automatically use all available cores effectively. Thread pinning, locks, runtime constraints, and serial sections can leave one execution path at its ceiling. A host average may therefore look safe while request latency rises.

Cgroups explain healthy hosts with slow workloads

Container-aware monitoring must inspect the workload's cgroup rather than stopping at the node. On cgroup v2, cpu.stat reports cumulative CPU usage in microseconds and, when the CPU controller is enabled, bandwidth-throttling counters. Compare successive samples rather than treating cumulative totals as a current percentage. These are per-cgroup totals, not a per-core breakdown.

Compare changes in nr_throttled and throttled_usec with latency, and inspect cpu.max for the quota and period. A value such as 200000 100000 permits two CPU-equivalents of bandwidth; it does not reserve two particular cores. Also check parent cgroups and CPU placement, because a workload can be constrained above its own cgroup. The kernel cgroup v2 documentation explains these controls.

GPU workloads need a shared resource view

Inference and media pipelines often move between CPU preprocessing, kernel launches, memory transfers, and post-processing. nvidia-smi dmon provides a direct device-side view, while DCGM supports more systematic NVIDIA telemetry. Those readings should sit beside CPU per-core, cgroup, memory, and network signals.

The operational principle is straightforward: host average, core, cgroup, and device views answer different questions. During an escalating incident, omitting any one of them can send remediation toward the wrong resource.

From DIY Scripts to a Monitoring Platform

A shell loop that samples /proc/stat and ships values to a time-series store is a reasonable starting point for a small lab. It teaches the counter-delta calculation and makes the collection path visible. It becomes fragile when operators need shared dashboards, durable retention, permissions, on-call routing, and consistent labels across a fleet.

The first break usually isn't the math. It's everything around the math. Scripts need process supervision, retry handling, authentication, schema changes, clock considerations, local buffering, and a way to prevent duplicate alerts. A cron job can collect data, but it doesn't automatically provide a reliable incident workflow.

Three practical operating models

DIY collection keeps ownership close to the server. It suits experiments and tightly bounded environments, but every operational feature becomes another script or service to maintain.

An open-source agent stack typically combines node_exporter, Prometheus, Grafana, and Alertmanager. This route offers flexibility and a large ecosystem, but the team still owns upgrades, storage, access control, dashboard design, alert deduplication, and notification integration.

A managed platform shifts those operational responsibilities to a service. Fivenines is one option in this category, providing an open-source Linux agent, CPU and host metrics, process visibility, container and NVIDIA GPU insights, dashboards, alert workflows, and external uptime checks in the same monitoring environment. It can replace several separately operated components for teams that prefer a unified control plane.

The choice depends on the boundary an organization wants to own. A platform doesn't make mpstat or top obsolete. Those commands remain faster for immediate host triage, especially when the agent is unavailable or the question concerns a single process.

Keep the CLI for diagnosis. Use continuous collection when the answer must survive the SSH session.

A platform earns its place when engineers need to correlate CPU with latency, compare hosts, retain trends, assign roles, and route alerts without asking one operator to maintain every storage and notification detail. The trade-off is less control over internals than a fully self-managed stack, balanced against less infrastructure to operate.

Alerting Thresholds and Notification Routing That Actually Work

A useful CPU alert must distinguish busy, blocked, throttled, and user-visible. A host average alone can't make that distinction. Alert logic should combine sustained behavior with per-core saturation, iowait, load or run-queue context, cgroup throttling, and service health.

A practical rule set can begin with sustained conditions rather than single samples. Use a warning for sustained high utilization and a critical condition for sustained severe utilization, but calibrate those levels against the service's baseline and latency objective. The specific thresholds in an alert rule should be treated as policy, not as universal Linux constants.

Build the signal from multiple dimensions

Prometheus users can examine node_cpu_seconds_total by mode and evaluate iowait over a sustained window. Pair that with per-core utilization so one saturated logical CPU isn't diluted by idle neighbors. For workload limits, include cgroup throttling metrics and throttled duration where the runtime exposes them.

Load and run-queue context helps distinguish runnable work from other causes of delay. CPU pressure stall information adds a view of time tasks spend waiting for CPU. To avoid paging on brief batch spikes, use a sustained-duration condition or a recheck, then correlate it with latency. A rate-of-change rule detects rapid change; it does not inherently suppress short spikes.

Route by consequence and ownership

Low-severity warnings can go to a team channel for review. A critical condition should page the service's primary on-call only when the signal remains sustained or coincides with an SLO symptom. Resolved events should close the associated incident or ticket automatically, otherwise the notification system creates administrative noise after recovery.

Every alert needs a runbook link and a dashboard link. The runbook should tell the responder to check per-core output, CPU states, pressure, cgroup throttling, recent deployments, and application latency. For teams implementing this workflow, a focused guide to setting up infrastructure alerts can help connect threshold logic with notification routing.

Burn-rate alerting tied to an SLO is often more meaningful than a static CPU threshold. It asks whether the service is consuming its error budget quickly, then uses CPU as diagnostic context rather than treating utilization as the customer impact itself.

A Tiered Monitoring Routine for Busy Linux Teams

A durable routine assigns each CPU question to the smallest toolset that can answer it accurately. That keeps incident response fast without pretending that a terminal snapshot can provide historical or fleet-wide evidence.

Tier one answers what is happening now

During live triage, start with top or htop for process activity and load context. Use mpstat -P ALL 1 5 to compare cores over explicit intervals, inspect iowait and steal, and read /proc/stat when a script needs the raw host counters. Use ps for a process lifetime-average ranking and pidstat -u 1 5 for recent process CPU activity.

The question at this tier is immediate: is something on fire right now?

Tier two preserves local evidence

Enable sysstat and use sar for scheduled local retention. Archived CPU, memory, block, and network data can show when a pattern began and whether it aligns with a deployment or scheduled workload. This tier is inexpensive and useful, but its history remains tied to the host and its configured rotation.

The question changes to: when did saturation start, and what changed around that time?

Tier three supports fleet decisions

Continuous agent collection adds shared dashboards, alert routing, cross-host comparisons, and longer-horizon capacity analysis. Teams can evaluate whether a service is approaching a capacity cliff, whether one workload repeatedly triggers throttling, and whether CPU behavior correlates with user-facing latency.

The question becomes: is the current growth pattern creating a future capacity risk?

AI-assisted anomaly detection can add another layer to continuous streams by identifying seasonal behavior, silent regressions, and unusual combinations of CPU pressure, throttling, and latency that static thresholds miss. It doesn't replace the underlying counters or the runbook. It helps operators find the deviations worth investigating before a customer-facing failure becomes obvious.

Fivenines combines Linux CPU, memory, disk, network, process, container, and NVIDIA GPU visibility with alert workflows and uptime monitoring through an open-source agent. Visit Fivenines to evaluate whether a unified dashboard can replace manual CPU checks and give the operations team durable, actionable history.

To retain Linux server metrics beyond a terminal session and review them with alerts, explore Fivenines Linux server monitoring.