Observability vs Monitoring: What Engineers Must Know

In 2016, Charity Majors and the team at Honeycomb articulated what became the industry's working definition of observability: a system is observable if you can understand its internal state from its external outputs, without having to ship new code or add new log fields to answer a novel question. That definition has become the clearest line separating observability from traditional monitoring—and it has become increasingly important as microservice architectures have fractured what used to be a monolith into dozens or hundreds of independently deployed services, each generating its own stream of signals.

Traditional monitoring checks whether predefined thresholds are breached. A CPU alert fires at 90%; a latency alert fires above 500ms; a disk alert fires at 85% usage. These known-unknowns are essential—but they only catch failures you anticipated when you wrote the alert rule. Observability fills the space of unknown-unknowns: the emergent failure mode you did not predict, the cascading latency from a dependency you do not own, the one user cohort whose requests are silently returning corrupt data. Understanding the difference is not academic—it directly determines how fast you can resolve incidents in production.

Image: Opsview Monitor 5.0 Dashboard.png — Goojannii (CC BY-SA 4.0), via Wikimedia Commons

The Core Difference: Reactive vs Exploratory

Monitoring is fundamentally reactive and known-state: you define what "healthy" looks like in advance and alert when that definition is violated. It is excellent for well-understood failure modes, capacity planning, and SLA tracking. A Prometheus alert that fires when error rate exceeds 1% over a 5-minute window is monitoring. An on-call page from PagerDuty when a synthetic health check fails is monitoring. These tools work reliably for the scenarios they were designed for.

Observability is exploratory and unknown-state: it gives engineers the ability to ask arbitrary questions about system behavior after the fact, using high-cardinality, high-dimensionality data captured during normal operation. The key phrase is arbitrary questions—not just the questions you had when you wrote the instrumentation, but questions you could not have anticipated. In a distributed system where a single user request touches fifteen services before returning, the ability to ask "show me all requests where payment-service latency exceeded 300ms and the user was on a mobile device in Southeast Asia" requires both high-cardinality attributes per event and causality links that connect events across service boundaries.

The Three Pillars of Observability

The industry has converged on three signal types as the core of any observability stack:

• Metrics: Numerical measurements aggregated over time. Prometheus counters, gauges, histograms. Low storage cost, excellent for dashboards and alerting. The weakness: aggregation destroys cardinality. A p99 latency metric tells you something is slow but not which user, endpoint, or dependency caused it.
• Logs: Discrete event records emitted by services. Structured logs in JSON are vastly more useful than unstructured text. Logs capture rich context but are expensive to store and query at scale, and correlating logs across services requires consistent trace and span IDs in every log line.
• Traces: Distributed traces represent the full lifecycle of a single request as it travels through multiple services. Each trace is composed of spans—one per service or operation—with timing, error state, and attributes attached. Traces are the primary tool for identifying which service in a call chain is introducing latency or errors.

These three pillars are not independent: the real power comes from correlating them. A metrics alert fires, you drill into logs filtered by trace ID, you follow the trace to the offending span, you inspect the attributes on that span to understand the exact condition. Without correlation, you are context-switching between tools and losing critical time.

Where Traditional Monitoring Falls Short in Distributed Systems

Monolithic applications were relatively well-served by traditional monitoring. You had one process, one set of logs, one CPU and memory profile to watch. Failure modes were largely local and predictable. Microservices shatter all of that.

Consider a checkout flow that calls: API gateway → auth service → inventory service → pricing service → payment service → notification service. Each service has its own health check, its own error rate, its own latency. A 400ms degradation in the overall checkout time might originate in any one of these six services, in the network between them, or in an upstream database dependency of the pricing service. A traditional monitoring dashboard shows you six green health checks and one slightly elevated latency metric with no attribution. You do not know which service to look at first.

This is the distributed systems observability problem in concrete form: the failure is visible at the edge but the cause is internal. Without distributed tracing, every incident begins with the same manual process—grep logs on each service in turn, try to reconstruct the timeline, work backwards from symptoms to cause. With distributed tracing, you have the full call graph for the slow request in seconds.

OpenTelemetry: The Emerging Standard

Until 2019, every observability vendor had proprietary instrumentation: you used Datadog's SDK to send to Datadog, Jaeger's client to send to Jaeger, Zipkin's tracer to send to Zipkin. Migrating vendors required re-instrumenting every service. OpenTelemetry (OTel) solved this by defining a vendor-neutral standard for telemetry instrumentation across traces, metrics, and logs.

OpenTelemetry consists of three layers:

• API: The interface your application code calls. Stable and vendor-neutral. Changing backends does not require changing application code.
• SDK: The implementation that collects, batches, and exports telemetry. Configurable via environment variables or YAML without code changes.
• Collector: An optional agent or gateway that receives telemetry from your services and exports it to one or more backends—Prometheus, Jaeger, Datadog, Honeycomb, etc. The Collector allows central configuration of sampling rates, redaction of sensitive attributes, and fan-out to multiple destinations.

Auto-instrumentation is OpenTelemetry's most practical entry point: for languages like Python, Java, and Node.js, you can attach the OTel SDK to an existing application at startup without modifying a single line of application code and immediately begin receiving traces and metrics. Manual instrumentation—adding custom spans and attributes to business-critical code paths—then layers on top to add the high-cardinality context that makes traces actionable.

Building a Practical Observability Stack

The most common mistake teams make when building this stack is treating instrumentation as an afterthought. Adding OTel to an existing service after an incident is reactive engineering; adding it during feature development is what gives you the data you need when the next novel failure occurs. The time to add trace context to your checkout service is not when checkout is degraded at 2am.

Sampling strategy is the other critical design decision. Capturing 100% of traces at production scale is prohibitively expensive for most teams. Head-based sampling—probabilistic, decided at the start of the request—is simple but blindly drops slow or error traces with the same probability as fast ones. Tail-based sampling—decided after the trace is complete, at the Collector—retains 100% of interesting traces (errors, slow requests, traces with specific attributes) and samples down only the boring successful ones. Tail-based sampling requires an OTel Collector with sufficient memory to buffer complete traces before making the sampling decision, but it dramatically improves the signal-to-noise ratio in your trace backend.

Frequently Asked Questions

Can we just use logging instead of distributed tracing?

Structured logs with consistent request IDs can partially substitute for distributed tracing in simple two- or three-service systems, but they break down as service count grows. Finding the logs for a specific request across fifteen services requires querying fifteen separate log streams and manually correlating by request ID. Distributed tracing assembles this automatically. For systems with more than three or four services, dedicated tracing pays for itself quickly in reduced mean-time-to-resolution during incidents.

What is the difference between SLIs, SLOs, and SLAs?

A Service Level Indicator (SLI) is a measurable metric that reflects service health—typically availability, latency, or error rate. A Service Level Objective (SLO) is an internal target for that metric: "99.9% of requests served within 200ms over a 30-day window." A Service Level Agreement (SLA) is an external contractual commitment, usually with financial penalties for breach. SLOs are the operational target you design your monitoring and alerting around; SLAs are the business consequence of missing that target. Running an error budget—the allowable error margin within the SLO window—is the standard mechanism for balancing reliability work against feature development.

Is Prometheus enough for observability, or do we need distributed tracing too?

Prometheus is an excellent metrics system but it is not an observability platform by itself. Metrics aggregate away cardinality: a p99 latency histogram cannot tell you which specific requests were slow or which downstream service caused the slowness. For systems with more than one service communicating over the network, distributed tracing is the essential complement to Prometheus metrics. The two are not redundant—they answer fundamentally different questions at different levels of resolution.

Bottom Line

We recommend treating observability and monitoring as complementary layers, not competing philosophies. Run Prometheus-based alerting for your known failure modes and SLO tracking. Run OpenTelemetry instrumentation across all services from day one of development—not after your first major incident. Store traces in a backend that supports high-cardinality exploration. Connect all three signal types with Grafana or an equivalent unified frontend. Teams that invest in this foundation before an incident consistently resolve production issues faster and spend less time in post-mortem archaeology and more time shipping. The infrastructure cost of a complete observability stack has fallen dramatically with open-source options; the organizational cost of building it after a major outage is always higher.

Disclaimer: This article is for informational purposes only. Technology landscapes change rapidly; verify information with official sources before making technical decisions.