Observability for Kubernetes

Kubernetes changes the shape of the observability problem more than almost any other infrastructure shift before it. On a static fleet of VMs, a host lives for months, has a fixed hostname, and can be monitored by installing an agent once and leaving it alone. On Kubernetes, pods are created and destroyed constantly, IP addresses and hostnames are recycled within minutes, and the thing you actually care about — a deployment, a service, a workload — is an abstraction spread across a shifting set of ephemeral containers. Observability tooling built for long-lived hosts doesn’t map cleanly onto that model, which is why Kubernetes observability is usually treated as its own discipline rather than an extension of traditional server monitoring.

Two problems dominate in practice: cardinality and collection topology. Every pod restart, rolling deployment, or autoscaling event mints new pod names, new IPs, and often new labels, and each of those becomes a distinct time series if metrics are labeled naively. Left unmanaged, this cardinality growth is one of the most common sources of runaway costs and slow queries in a Kubernetes-based observability stack. Collection topology matters because there’s no single obvious place to run an agent — it can run once per node, once per pod, or centrally, and each choice has different tradeoffs for isolation, resource overhead, and operational simplicity.

Why it matters

• Ephemerality breaks identity-based monitoring. A pod that existed five minutes ago and crashed is gone, along with any state an agent held about it; observability has to be built around continuously discovering current pods rather than tracking a fixed host list, which is why Kubernetes-aware service discovery (label and annotation-based) is central to tools like Prometheus.
• Cardinality from pod and deployment names compounds fast. Labeling metrics with raw pod names or container IDs instead of stable workload names (deployment, service) causes time series counts to grow unbounded as pods churn, driving up storage costs and slowing queries.
• Cluster-level and application-level signals answer different questions. Node CPU pressure, kubelet health, and scheduling failures are a different layer of the system than application request latency or error rates — you need both, and conflating them makes root-causing an incident slower.
• The collection layer itself becomes infrastructure you have to run and scale. Whatever collects metrics, logs, and traces off every node now runs on the cluster it’s monitoring, so its resource usage, failure modes, and upgrade path are now part of the platform you operate.

How to approach it

• Use label-stable metrics, not identity-stable ones. Aggregate or relabel away pod-specific identifiers (pod name, IP) in favor of workload-level labels (deployment, namespace, container) before they hit long-term storage, and apply this at the collection or recording-rule layer rather than trying to fix it after ingestion.
• Pick a collection pattern deliberately: DaemonSet, sidecar, or both. A DaemonSet runs one collector per node and is the most common pattern for gathering node-level and container-level telemetry with a single deployment — applications just send data to a local endpoint. A sidecar runs a collector container inside each application pod, which is necessary on platforms without DaemonSet support (like AWS Fargate) and useful when different workloads need different processing rules. Many production setups combine both: DaemonSet agents per node forwarding to a central gateway deployment that handles routing, sampling, and export.
• Separate cluster-level signals from application-level ones, but correlate them. Kubernetes state metrics (kube-state-metrics), node metrics (node exporter, cAdvisor), and control-plane health are their own signal set; instrument applications separately with OpenTelemetry or a vendor agent, and make sure both sets share consistent labels (namespace, pod, node) so they can be joined during an incident.
• Treat the OpenTelemetry Operator or equivalent as part of your platform, not an afterthought. Auto-instrumentation injection, collector lifecycle management, and configuration rollout are ongoing operational responsibilities on a cluster that changes shape daily, not a one-time install.
• Budget for the collector’s own resource footprint. A DaemonSet or sidecar running on every node or pod consumes CPU and memory that competes with application workloads; size and rate-limit it explicitly rather than letting it grow with cluster size unchecked.

In practice, most Kubernetes-native stacks pair a metrics backend built for high cardinality (Prometheus feeding a longer-term store, or a managed equivalent) with a visualization layer like Grafana, a log pipeline that tags every line with pod and namespace metadata, and a tracing backend that can follow a request across the many small services Kubernetes makes it easy to run. See metrics, logs, and traces for the pillar-level detail this article assumes.