The Cost of Observability (and How to Control It)

Observability is not free, and the cost curve is easy to underestimate because it grows with exactly the things that make a system worth observing: more services, more detail, more history. Left unmanaged, telemetry volume, cardinality, and retention tend to grow faster than the infrastructure they’re monitoring, and teams that don’t treat cost as a deliberate design constraint from the start often end up with a bill that forces a painful, reactive cutback later. Each of the four pillars has its own dominant cost driver and its own set of practical controls.

The pattern that shows up across all of them is the same: cost scales with volume, cardinality/uniqueness, and retention, and the fix is almost always to reduce one of those three deliberately rather than to buy more storage. What differs is where in the pipeline that reduction is applied — at collection time, at ingestion, or at query time — and how much fidelity a team is willing to trade away to get there.

Why it matters

• Metrics cardinality is a multiplier, not an additive cost. Every unique combination of label values creates a new time series; a label with unbounded values (user ID, raw pod name, full URL path) can silently multiply storage and query cost by orders of magnitude compared to a bounded label set, often without anyone noticing until a query starts timing out or a bill spikes.
• Log volume and retention compound because logs are the least structured, highest-volume signal. Unlike metrics, raw log lines don’t compress into aggregates — every line ingested is stored roughly as-is, so verbose logging at scale directly drives both ingestion and storage cost, and long retention windows (often required for compliance) keep that cost accruing indefinitely.
• Trace volume grows with traffic, and traces are large relative to metrics. A single trace carries many spans, each with attributes; capturing every trace for every request at real production traffic is rarely affordable, which is why sampling strategy is one of the first cost decisions a team makes when adopting distributed tracing.
• Profiling data adds a continuous storage cost on top of the other three. Continuous profiling collects stack samples at a steady cadence across a fleet, and while individual profiles compress well, running it fleet-wide and retaining history for regression comparison is an additional, often underestimated, storage line item.

How to approach it

• Control metrics cardinality at the source and with recording rules. Avoid putting unbounded values in labels; where high-cardinality dimensions are genuinely needed for debugging, keep them in traces or logs instead of metrics. Use recording rules to pre-aggregate frequently queried, high-cardinality series into cheaper rollups, and run periodic cardinality audits to catch unintentional label explosions early.
• Tier log retention and reduce volume at the source. Not every log line needs the same retention or the same storage tier — keep a short hot window for active debugging and move older data to cheaper storage, or drop it, once it’s outlived operational usefulness. Reduce noisy, low-value log lines (debug-level logging left on in production is a common offender) and adopt structured logging so filtering and reduction can be automated at collection time.
• Use tail-based sampling for traces rather than sampling purely at random. Head-based sampling (a fixed percentage of all traces) is simple but tends to throw away exactly the rare, slow, or error traces that matter most for debugging. Tail-based sampling defers the sampling decision until a trace completes, so it can preferentially keep error traces and outlier-latency traces while discarding routine, fast, successful ones — commonly cutting trace volume substantially while preserving debugging value.
• Right-size profiling collection frequency and retention. Continuous profiling doesn’t need every service profiled continuously at maximum resolution — sampling frequency and which services are profiled can both be scoped to where the diagnostic value justifies the storage cost.
• Treat cost as a first-class dashboard, not an annual surprise. Track ingestion volume, cardinality growth, and retention-driven storage cost the same way you’d track any other operational metric, so a regression in cost is visible and attributable before it becomes a budget conversation.

None of these controls require giving up visibility where it matters — they shift effort toward keeping high-value, high-resolution data and deliberately discarding the rest. See metrics and traces for more on cardinality and sampling mechanics specifically.