Observability

Production has its own personality. Observability is the set of senses that let you listen to it, ask it new questions, and find out what hurts before users do.

Three pillars Logs · Metrics · Traces

It’s 3 a.m. on a Sunday. Your phone goes off. An alert says "payments service: high error rate."

You open the dashboard. The error rate is up. That’s all the dashboard knows. It doesn’t know which errors, which users, which dependency, which code path. Eight minutes later you’re still searching logs with grep, refilling a glass of water, and silently regretting your career choices.

Now imagine the same alert with a single click that filters every log, metric and trace to only the failing requests, joined by trace ID. The cause — a third-party API timing out for South-American payment cards — is in front of you in 90 seconds. You disable that path with a feature flag and go back to bed.

That second scenario is what observability is for.

01Monitoring vs observability

The two words get used interchangeably, but they’re different jobs. Monitoring is a subset of observability. Knowing the difference helps you spend tooling money in the right place.

Monitoring

Watching pre-defined dashboards and alerts for known failure modes. Great at "is the disk full?", "is the queue backed up?", "did the CPU pass 90%?". Built for failures you have seen before.

Observability

The property of a system that lets you ask new questions about its behavior, after the fact, without shipping new code. Built for failures you have never seen before.

A useful analogy

Monitoring is the dashboard in your car. Observability is the diagnostic port the mechanic plugs into when the dashboard isn’t telling them enough.

02The three pillars

Observability stands on three types of data — logs, metrics and traces. Each one is good at a different question. You need all three to answer "what is happening, and why?"

Logs — "what happened?"

Discrete events: "user 42 paid 19.99 EUR". Cheap to write, expensive to store and search at scale. Structure them as JSON, attach trace IDs, and log only what you’d actually want at 3 a.m.

Metrics — "how much, how often?"

Numerical time series: requests per second, error rate, p99 latency, memory usage. Tiny per data point, perfect for dashboards and alerts. Beware of high-cardinality labels — they explode storage.

Traces — "where did the time go?"

The journey of one request across many services. The killer feature of distributed systems. A trace shows that the 800 ms total latency was 700 ms inside one downstream call — and you’d never know without it.

03The four golden signals

Coined by Google’s SRE team and now industry-standard, these are the four numbers every service should display on its dashboard before anything else.

RPS
Traffic
Volume reaching the service right now.
%
Errors
Non-2xx responses, panics, exceptions.
p99
Latency
The tail. Average lies; p99 tells the truth.
%
Saturation
How "full" the service is — CPU, memory, threads, queues.
i

Why p99, not average?

Averages hide tail pain. Imagine 99 users get a response in 50 ms and one waits 10 seconds. The average is around 150 ms — looks fine. The p99 says 10 seconds — tells the real story. Use p99 (or p99.9) for any latency that users feel.

04Turning reliability into a number

"Stable" is an opinion. SLIs and SLOs turn it into a measurable, negotiable contract between the engineering team and the rest of the business.

SLI — Service Level Indicator
The thing you measure. Successful requests / total requests over the last 5 minutes.
SLO — Service Level Objective
The target. "We will succeed at least 99.9% of the time, over rolling 30 days."
Error budget
100% minus the SLO. At 99.9%, that’s ~43 minutes of allowed unreliability per month — permission to break things.
Burn-rate alert
Fires when you’re spending the error budget faster than the window can absorb. Far quieter than "every error pages someone".
!

SLOs are a culture, not a config

An SLO is only useful if breaking it actually changes behavior. If your error budget runs out, the team should slow down feature work and pay down reliability debt. If nothing changes when the budget burns, the SLO is theatre.

05OpenTelemetry — one SDK, any backend

Before OpenTelemetry, every backend (Datadog, New Relic, Splunk, Grafana, …) had its own SDK. Switching vendors meant re-instrumenting everything. OpenTelemetry is the vendor-neutral standard the whole industry settled on.

Your app uses the OpenTelemetry SDK. The SDK ships data to the OpenTelemetry Collector. The Collector forwards it to any backend you choose — or several at once.

otel-collector.yamlYAML
receivers:
  otlp:
    protocols:
      grpc: {}
      http: {}

processors:
  batch: {}
  memory_limiter:
    limit_mib: 400

exporters:
  otlphttp/grafana:
    endpoint: https://otlp.grafana.net
  otlphttp/datadog:
    endpoint: https://otlp.datadoghq.com

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, batch]
      exporters:  [otlphttp/grafana, otlphttp/datadog]
    metrics:
      receivers:  [otlp]
      processors: [memory_limiter, batch]
      exporters:  [otlphttp/grafana]
    logs:
      receivers:  [otlp]
      processors: [memory_limiter, batch]
      exporters:  [otlphttp/grafana]

06Habits that turn data into answers

Most observability problems aren’t "we don’t have data" — they’re "we have too much of the wrong kind". These habits clean it up.

Anti-pattern
Unstructured logs. "user 42 paid 19.99" is hard to query. JSON with proper fields ({"user":42,"amount":19.99}) is filterable, aggregatable, joinable with traces.
Anti-pattern
Metrics with high-cardinality labels. Putting user_id as a Prometheus label creates one time series per user. The store explodes. Use traces for per-user investigation; reserve metric labels for low-cardinality dimensions.
Anti-pattern
Alert on causes, not symptoms. "CPU is at 80%" tells you nothing about whether users are suffering. Alert on golden signals (latency, error rate, saturation relative to capacity) and on SLO burn rate.
Anti-pattern
Alert fatigue. When the pager goes off ten times a day, humans stop reading it. Aim for fewer than five actionable pages per on-call shift. Everything else goes to a queue or a dashboard.
Healthy
Trace ID everywhere. Every log line, every error event, every metric exemplar carries the trace ID of the request that produced it. One click jumps you from "something broke" to "the exact request that broke it".

07Putting it together

Take-aways

1. Monitoring answers known questions; observability lets you ask new ones in the middle of an incident.

2. Three pillars (logs, metrics, traces) plus four golden signals (traffic, errors, latency, saturation) cover most of the work.

3. SLIs measure, SLOs target, error budgets give permission. If breaking the SLO doesn’t change behavior, the SLO is theatre.

4. Instrument with OpenTelemetry once; route to any backend forever.

Next chapter: the version-control system that holds the whole thing together — Git.