Observability Wiki

Metrics

Computing SLOs and Error Budgets from Metrics

The PromQL patterns for turning request-rate and latency metrics into a working SLI, SLO, and burn-rate alert.

Last updated

A service level objective is only as good as the metric feeding it. This article covers the practical, metrics-layer side of that: which raw counters and histograms turn into a valid SLI, and how to query them. For the full definitions of SLI, SLO, SLA, and how error budgets are used organizationally, see SRE Fundamentals — this article assumes that vocabulary and focuses on the query patterns.

Why the metrics layer matters here

  • The SLI is a query, not a concept. “99.9% of requests succeed” is only meaningful once it’s backed by a specific PromQL (or equivalent) expression that both engineering and the business agree measures the right thing.
  • Small query mistakes silently break the SLO. Counting client-cancelled requests as failures, measuring at the wrong aggregation level, or using an average instead of a percentile for latency can all make an SLO report a number that doesn’t match what users experience.
  • Burn-rate alerting depends entirely on the underlying metric’s resolution and labels. A success-rate metric without enough label granularity to isolate a single failing dependency makes burn-rate alerts accurate but not actionable.

PromQL patterns for SLIs

  • Request success rate, the standard availability SLI, is typically computed as sum(rate(http_requests_total{code!~"5.."}[5m])) / sum(rate(http_requests_total[5m])) — the proportion of successful requests over a rolling window, built directly on the counters described in metric types.
  • Latency percentiles, the standard performance SLI, come from a histogram via histogram_quantile() (or an equivalent function on other backends), since percentiles capture tail latency in a way an average cannot — a p99 SLO of “300ms” is a very different promise than an average of 300ms.
  • Burn-rate alerts compare the current rate of error-budget consumption against multiple time windows at once — for example a fast 1-hour window and a slower 6-hour confirming window — so both sudden outages and slow leaks get caught before the budget is fully gone, rather than alerting only after the SLO is already breached.
  • Measurement window and event boundaries have to be decided in the query, not just in policy. Whether a client-cancelled request counts as a failure, and whether the window is a rolling 28 or 30 days, changes what the PromQL expression actually returns — settle this before treating the number as authoritative.

Most metrics platforms — Prometheus with recording rules, or commercial tools like Datadog and New Relic with built-in SLO features — now offer dedicated support for defining an SLO, tracking its error budget in real time, and alerting on burn rate, which has made this pattern a default rather than a bespoke build for most teams adopting it.

Related tools