Observability vs Monitoring for Real Failures
Monitoring tracks system behavior and raises alerts for conditions you define. Observability is the ability to understand internal behavior from the evidence a system exposes. Investigation uses both: the distinction is not a hard boundary between tools, and a distributed trace is not required to explain every infrastructure failure.
A public checkout API starts returning errors at 14:05. The dashboard stays green: CPU near idle, memory flat, host up, agent reporting. Customers can't pay, and nothing on the screen names the dependency that broke.
That gap is where observability vs monitoring stops being a definitions argument and becomes a decision about which signal to read next. The useful question for an operator is which signal can answer the question in front of them, and where that signal's evidence runs out.
Table of Contents
- Monitoring Detects and Observability Explains
- Match Each Failure Question to a Signal
- Compare Goals, Evidence and Cost
- Walk One Hypothetical Checkout Failure
- Keep Alerts Useful When Data Goes Missing
- Decide When Traces Are Actually Required
- Pilot One Host Before You Add Scope
Monitoring Detects and Observability Explains
Treat the checkout outage as a labelled hypothetical, not a case study. The API returns errors for about ten minutes. The host serving it reports CPU near idle and plenty of free memory. Payment fails for real customers while every capacity chart looks normal.
In this example, monitoring checks a predefined availability condition. An external HTTPS probe fails against a representative endpoint, which is a useful symptom to investigate. It speaks for that endpoint from the probe's location, not for every customer transaction. A TCP connection proves only that a port accepted a connection; neither a TCP success nor a shallow health-page HTTP 200 proves that checkout completes.

Observability lets an operator move from the external symptom toward internal evidence, including failure modes nobody anticipated. The OpenTelemetry observability primer connects this ability to instrumentation and telemetry. Monitoring data can support that investigation too. More panels or more data do not automatically make a system explainable: the evidence needs enough context to distinguish competing explanations.
A useful distinction: monitoring tracks behavior and detects configured conditions; observability is the ability to explain behavior using available evidence. The same metrics, logs and traces can serve both.
That framing kills two bad habits. The first is treating observability as a bigger dashboard with more panels on it. The second is insisting that metrics, logs and traces must all exist before an outage becomes explainable, which turns a ten-minute incident into a tooling project.
The signals here include uptime and response time, host CPU, memory and disk, scheduled-job check-ins, supported database metrics, and selected journald error summaries. A distributed trace, where application instrumentation provides one, connects operations along a request path. Tracing complements these signals; whether you need it depends on the unanswered question. Teams that want the platform-level picture can read the guide to a unified observability platform; this article focuses on choosing evidence.
Match Each Failure Question to a Signal
The checkout outage generates a short list of questions, and each question has a first signal worth opening, plus a point past which that signal proves nothing. The table below maps the two. Before relying on any row, confirm the prerequisites.
- A representative test host. Choose a non-critical Linux machine with a workload and resource limits you understand.
- Explicit collection configuration and least-privilege access. Enable journald monitoring and choose the units; configure database access for the metrics you need. A credential or collection error is not evidence that the application is down.
- A representative, safe endpoint. Confirm what the check actually exercises. Do not run a payment or other state-changing transaction merely to test monitoring.
| Operator question | First signal | Evidence it can give | Explicit limit |
|---|---|---|---|
| Does the monitored endpoint respond correctly? | External HTTPS check | HTTP status and elapsed time from each probe location | A shallow endpoint does not establish successful checkout; TCP proves only connection reachability |
| Is this host under resource pressure? | CPU, memory, disk and process context | Utilization and capacity trends | Host averages can hide a constrained process, one busy core or I/O waits |
| Did the expected job check in? | Scheduled-job monitor | A received completion ping, missing ping or explicit failure report | A missing ping cannot distinguish a failed job from a failed delivery path |
| Is the database under pressure? | Configured engine metrics | Connections, replication, cache or other engine-specific readings | A high count alone does not identify the request or root cause |
| What error did this unit record? | Enabled journald monitoring for selected units | Error counts, patterns and bounded redacted incident context | Not full log ingestion, indexing or arbitrary historical search |
| Which downstream operation slowed a request? | Instrumented distributed trace | Related spans and their timing | Missing instrumentation, propagation or sampled-out traces leave gaps |
| Has collection stopped? | Last successful collection and collector health | Whether the evidence is recent enough to use | A missing sample is not zero and does not by itself prove a host outage |
Pick the Question Before the Tool
Read one row at a time. Start with the row that matches the question being asked out loud in the incident channel, and stop when that row's evidence ends. An on-call engineer who opens a CPU chart to answer "can users reach checkout" is not investigating, just scrolling.
The Fivenines database modules collect engine-specific metrics through the agent, without a separate exporter. Supported families currently include PostgreSQL, MySQL/MariaDB, MongoDB, Redis/Valkey and Memcached. Connections, replication and cache measurements differ by engine and configuration; do not assume every metric exists for every database. Configure the required read permissions before using a chart as incident evidence.
Where Each Row Stops
Idle CPU does not clear the host. Low average utilization argues against sustained whole-host CPU saturation, but it can hide one busy core, a process limit, lock contention, or a workload blocked on disk or the network. Check the affected process and the same time window. On Linux, pressure stall information can help distinguish CPU, memory and I/O contention where the kernel exposes it; this is a host diagnostic, not a promise that every dashboard includes PSI.
Fivenines journald monitoring must be enabled for selected units. It records counts and error patterns and can attach bounded, redacted log context to an incident. A log line does not itself open an incident in this feature. It is not a searchable archive, so a question such as "find every matching historical line" needs a different log tool.
Network devices follow the same discipline as everything else here. A switch, router or firewall is polled only after someone adds it explicitly, and there's no subnet sweep that finds it for you.
Compare Goals, Evidence and Cost
Monitoring and observability overlap in the data they use. Monitoring includes collection, visualization and alerting; it can examine both external symptoms and internal state. Observability describes how well that evidence supports understanding behavior. Request rate, errors, latency and resource measurements can contribute to both detection and diagnosis.
The Google SRE chapter on monitoring distributed systems distinguishes symptoms from causes and black-box from white-box monitoring. Use actionable user-facing failures for urgent paging, then investigate internal signals. A capacity or safety condition may also justify a page before user impact when it predicts an imminent, actionable failure; internal metrics are not categorically excluded.
| Criterion | Monitoring practice | Observability capability |
|---|---|---|
| Main question | Is behavior within the conditions we track? | Can we explain the behavior with the available evidence? |
| Useful for | Detection, trends, dashboards and response triggers | Investigation and testing explanations, including unexpected failures |
| Evidence | Checks, metrics, logs or derived signals selected for the task | The same telemetry with sufficient context and correlations |
| Common gap | Uncovered conditions or unnoticed collection failures | Missing instrumentation or context; correlation without proof of cause |
| Cost to control | Collection, storage, operations and alert noise | Collection, storage, operations and the detail needed for investigation |
Paging test: name the risk, the owner and the action required now. If no immediate action is needed, a ticket or dashboard may be more appropriate. Set urgency from service impact and remaining response time, not a universal ten-minute rule.
Both practices need cost controls. Excess alerts consume attention; excessive telemetry volume and cardinality consume collection, network and storage resources. Keep data that answers a defined operational question, and measure the cost of collecting and retaining it.
An incomplete monitoring system misses important failures, while an unnecessarily complicated one adds operational work. The guide to the real cost of monitoring explores that trade-off and when consolidation is useful.
Walk One Hypothetical Checkout Failure
The timeline below is illustrative, constructed to show which signal answers which question at which moment. No customer, no quote and no measured duration is being reported here.
At minute 0, the representative checkout check fails. Record the failure type and time: an HTTP 503 returned quickly, a DNS failure and a timeout are different observations. A failed request can still have useful timing. Keep successful-request latency separate from error and timeout timings, and do not turn missing timing into zero. Follow the configured confirmation and paging policy; this illustrative timeline assumes the failure has been confirmed.
At minute 1, inspect the host and affected process over the same interval. Normal aggregate CPU and memory make whole-host exhaustion less likely, but do not rule out a constrained process, lock or I/O bottleneck. Check disk space and I/O evidence as well.
At minute 2, suppose the database metrics show connections near the limit verified in that database's configuration. That supports a connection-capacity hypothesis, not a conclusion about a particular query, lock or caller. Check which connections are active or idle and whether reserved capacity or application pooling changes the effective limit.
At minute 3, suppose the application's selected journald unit reports a repeated "too many connections" error. Correlate it with the database and endpoint timestamps. A "connection refused" error would suggest a different set of checks, such as the listener and network path; the messages are not interchangeable. Where enabled, bounded redacted incident excerpts add context, as discussed in the guide to event correlation. They do not by themselves prove the causal chain.
At minute 4, a required job has not checked in before its deadline. Record it as a possible contributor, not a cause: investigate whether the job started, completed, failed explicitly or merely could not deliver its ping. Task monitoring needs a completion ping or explicit failure report from the job; installing a host agent alone does not monitor every cron task.
At minute 5, if the unresolved question is which downstream operation delayed an affected request, inspect an existing distributed trace or arrange suitable instrumentation for recurrence. A trace that was not collected during the incident cannot be recovered afterward. APM is one option for analyzing traces; OpenTelemetry and a compatible backend are another. Host metrics alone do not supply a request trace.
Logs, metrics and traces stay different evidence rather than interchangeable copies. OpenTelemetry's description of the log signal is precise about it: a log is a timestamped record, a metric is a runtime measurement, and a trace is the path of one request through an application. Trace and span identifiers added to log records are what let an operator move from an aggregate symptom to an affected request path.
Read-only timing check. Replace this reserved example URL with an endpoint you own and are authorized to probe:
curl --connect-timeout 3 --max-time 10 --silent --show-error \
--output /dev/null \
--write-out 'http_code=%{http_code} time_total=%{time_total}s\n' \
'https://example.com/health'
The curl reference defines the status and elapsed-time fields. With these options, an HTTP 503 can still give curl exit status 0: the transfer completed, but the HTTP response indicates an error. Inspect both the HTTP code and the command's exit status. http_code=000 means no HTTP response code was obtained, not a real HTTP status. A timeout duration is elapsed time to failure, not successful application latency. Keep each result's timestamp and probe location when comparing it with server data.
What Would Falsify This Story
- The check recovers while the database remains near its limit. The simple causal story is incomplete. Confirm the relevant user journey and recovery criteria; one green endpoint is not proof that all users recovered.
- The job completed on time but its ping failed. The job-delay hypothesis loses support; investigate telemetry delivery separately.
- A restart clears connections without a code change. That shows a temporary state change, not proof of a leak. Pool sizing, traffic, long transactions and leaks remain competing explanations. Preserve diagnostics before any approved restart.
Optional third-party example: RunaLabs' video, Debug a Broken Checkout with Rill | Before and After. It uses a different tool and is not a demonstration of Fivenines or a record of the hypothetical incident above.
Keep Alerts Useful When Data Goes Missing
Alert design breaks into three separate conditions, and collapsing them into one threshold is how on-call rotations burn out. Every window below is an illustrative example, not a tuned value for any real estate.
Set detection time from the service risk. Confirm a user-visible failure or a sustained breach of its response budget without delaying response beyond what the service can tolerate. The Prometheus alerting-rule reference uses for to require an expression to remain active across evaluations before firing. for: 10m is an example, not a recommended delay for checkout. Regional confirmation, repeated checks and time windows are different controls.
Define recovery separately. Choose an appropriate recovery condition and, if needed, hysteresis or a hold period. It need not match the firing window. Prometheus keep_firing_for holds a firing alert after its condition was last met; it does not establish that successful samples arrived throughout that period. Verify recovery and missing-data behavior independently.
Monitor collection health. Compare sample age with the expected interval plus an appropriate delay allowance. Distinguish stale telemetry, a failed probe, an unreachable host and planned maintenance. Choose paging or a ticket from the monitoring gap's urgency. Missing is not zero.
| Rule | Illustrative condition | Confirmation | Recovery evidence |
|---|---|---|---|
| Page on the user symptom | Representative endpoint fails or violates its budget | Risk-based policy; distinguish region quorum, repeated failures and elapsed time | Representative successful responses under the chosen recovery policy |
| Diagnose resource pressure | CPU, disk or database capacity is abnormal | Check sustained pressure, affected workload and time to impact | Resource and service behavior recover; a single average is insufficient |
| Detect missing telemetry | Sample age exceeds the expected interval plus delay allowance | Check collector health and maintenance context | Fresh samples arrive; separately assess what they report |
| Add context without duplicate pages | Related database or disk symptoms | Link evidence to the incident with timestamps | Confirm or reject the causal hypothesis |
The OpenTelemetry Collector follow-up survey analysis, published in January 2026 about the 2025 survey, reports that about 23% of respondents did not monitor their Collectors. It also reports collecting internal metrics (83%), logs (61%) and traces (25%). These are survey responses about Collector monitoring, not a census of all infrastructure teams or proof that any specific installation is healthy.
A quiet dashboard is ambiguous until something checks the freshness of the data behind it.
Map collection dependencies explicitly. Losing a host agent hides that host's telemetry and may also hide the services or SNMP devices collected through it. It does not automatically hide unrelated hosts with independent agents. Assign ownership to the collector and affected monitors so one collection failure does not generate unexplained gaps across a client account.
Route diagnostic symptoms as context when that avoids duplicate pages, but keep actionable imminent-capacity and safety alerts. Fivenines uptime checks use regional confirmation, while workflows can add rechecks, delays and routing. Neither should be described as a universal ten-minute delay. Test a notification in a non-production workflow and verify the current channel entitlement. Keep TLS verification enabled. The monitoring data quality guide covers interpreting gaps and stale evidence.
Decide When Traces Are Actually Required
Decide from an unanswered diagnostic question, not estate size alone. A small service can benefit from tracing; a large estate can also contain failures explained by host and database evidence. Grafana Labs' 2025 observability survey reports that 39% of its 1,255 respondents selected complexity/overhead as their biggest obstacle. Responses came from its community and industry events. That is useful context about complexity, not evidence that a particular team should avoid tracing.
When Infrastructure Signals Are Enough
Infrastructure signals carry the incident when the failing unit is a host, a guest, a disk, a database process, an explicitly added network device, a website or API check, or a scheduled job. The open question is availability, saturation, or whether required work ran, and each of those has a direct reading.
For a particular incident, infrastructure and database evidence may explain a monolith's failure without a trace. Even then, "the database ran out of connections" describes a failure mechanism; explaining why may require pool, query, lock or application evidence. Stop collecting more data only when the evidence answers the operational question with adequate confidence.
When They Genuinely Aren't
Tracing is especially useful when a request crosses several instrumented components and you need to locate latency, errors or retries along that path. It can also help inside a single application. Correlation requires propagated context and appropriate instrumentation; a missing span does not prove that an operation never happened.
Measure instrumentation overhead on a representative workload rather than reusing a generic percentage. Compare throughput, response-time percentiles, CPU, memory, export volume and dropped telemetry with and without instrumentation under equivalent conditions. The OpenTelemetry sampling guide explains choices that trade data volume against detail. Sampling can omit the request you need, and a downstream sampling decision does not remove all upstream instrumentation cost. Record the sampling policy with any conclusions.
As the vendor publishing this article, Fivenines is scoped to infrastructure observability: servers, guests, explicitly added devices, checks, jobs and supported database modules, with journald summaries for diagnosis. It is not full APM, not distributed tracing, not a general log archive, and not a replacement for remote access or patch deployment in an RMM. Teams that do run JVM services and want to see what a trace buys them can read the guide to tracing in Java before adding another collector.
One boundary worth repeating: a guest reported as running doesn't prove the application works. Pair any internal reading with an external check before calling a service healthy.
Pilot One Host Before You Add Scope
If the unresolved question is which application operation failed, evaluate tracing while retaining availability and host checks. If the immediate gap concerns hosts, services or jobs, start a bounded infrastructure pilot with an explicit success criterion.
Fivenines is the vendor of this guide. Linux Server Monitoring supplies host metrics through an open-source agent. Choose one non-critical representative host, a safe service URL and an owner for the test. Configure endpoint checks, any task pings and alert routing separately; they do not all appear merely because an agent was installed.
- Connect the test host and record freshness. Confirm timestamps advance for CPU, memory and disk; note the expected interval.
- Configure one external HTTPS check against an endpoint whose meaning you understand. Keep TLS verification enabled and avoid a state-changing transaction.
- Configure and test notification routing. Use a test channel supported by the plan. Record who receives a failure and a recovery.
- Test missing data only on the isolated test host. Record the expected offline/freshness detection behavior first. Briefly stop its agent with an agreed restart plan, confirm the gap and configured alert, then restart and confirm fresh data and recovery. Do not stop a production agent or a polling host shared by other monitors.
- Test a job independently if it is in scope. Configure its completion ping and deadline; verify a safe test success, a missed ping and recovery.
- Keep existing monitoring until these tests pass. Record timestamps and actual outcomes rather than assuming a restart sent a notification.
The pilot passes when fresh readings, the representative endpoint check, the configured notification and the deliberately tested recovery all behave as expected. A stopped agent tests telemetry loss, not an application outage. If notification or recovery does not occur under the configuration you intended, diagnose that gap before extending the pilot.
If you manage monitoring resources with Terraform, terraform plan normally refreshes tracked resources and compares observed state with the configuration to propose changes. It neither applies those changes nor continuously prevents drift, and it covers only resources represented by the configuration and provider. Confirm current trial and plan limits on the pricing page before extending the pilot.
Start with one host, one representative check and a verified alert path. Add task monitoring or request tracing when it answers a specific remaining question. The aim is evidence you can use during an incident, with clear limits and a tested collection path.