Log Retention, Sampling, and Cost Control

Log volume grows roughly in proportion to request volume: double the traffic, and — absent any intervention — you roughly double the number of log lines written, shipped, indexed, and stored. Because logs are also the most detailed and least compressed of the observability signals, this usually makes logging the single largest line item in an observability budget, ahead of metrics or traces. Retention policy — how long logs are kept, at what detail, and how they’re indexed — is the primary lever for controlling that cost.

Why it matters

• Most log volume is never queried. The overwhelming majority of INFO and DEBUG lines are written and never looked at again; only a small fraction, usually tied to an incident or an audit request, is ever retrieved. Paying full indexing and hot-storage cost for all of it is rarely justified.
• Full-text indexing costs scale with content, not just volume. Backends that build an inverted index over every token in every log line (classic Elasticsearch/Splunk-style indexing) pay storage and CPU cost proportional to how much text there is, not just how many lines — which is exactly why index-light alternatives changed the economics of logging.
• Retention needs differ by use case. Debugging a live incident needs recent, high-detail logs; compliance and audit needs long-term retention but is rarely queried and doesn’t need sub-second full-text search; almost nothing needs both high detail and multi-year retention at the same query performance.
• Uncontrolled retention is a silent, compounding cost. Because logs accumulate continuously, a retention setting that’s too generous doesn’t cause an obvious one-time spike — it just makes every subsequent month’s storage bill larger than it needs to be.

Strategies

• Tiered retention (hot/warm/cold). Keep recent logs (hours to a few weeks) in fast, fully indexed hot storage for active debugging; move older data to cheaper warm storage with reduced indexing; archive or delete beyond that. Elasticsearch’s index lifecycle management and Splunk’s storage tiers formalize this pattern.
• Short high-detail retention, longer low-detail retention. Keep DEBUG/INFO-level detail for days, not months, but retain ERROR-level or business-audit events — which are far lower volume — for much longer. This captures most compliance and audit needs without paying full retention cost on the noisy majority of log lines.
• Log sampling. Rather than dropping detail uniformly, sample: keep a representative percentage of high-volume, low-value log lines (health checks, successful routine requests) while always keeping WARN/ERROR lines and logs tied to flagged or unusual requests. A common pattern is sampling DEBUG/INFO at a low fixed rate while retaining 100% of WARN and above.
• Index-light architectures. Grafana Loki’s core design difference from Elasticsearch-style systems is indexing only a small set of labels (service, environment, level) per log stream, and storing the actual log content compressed in cheap object storage without a full-text index. Queries filter by label first, then scan the relevant compressed chunks — trading some query flexibility for a large reduction in index size and cost, which is why it’s often positioned as a cheaper backend at high volume.
• Push detail toward traces and metrics where possible. If a piece of information can be captured as a span attribute or a metric label instead of a repeated log line, it usually should be — traces and metrics compress and aggregate far better than equivalent free-form log volume.

None of these strategies work well in isolation from structured logging: sampling and tiered retention both depend on being able to reliably identify level, service, and event type per log line, which is much harder with unstructured free text. Getting the schema right first is what makes retention and sampling policy practical to implement at all.