Centralized Logging Architecture Patterns

Centralized logging is the practice of shipping logs from every service and host to one queryable place, rather than leaving them scattered across individual machines and containers. Almost every implementation, regardless of the specific tools involved, follows the same basic pipeline shape: collect logs at the edge, optionally buffer them, process and enrich them, store them, and query or visualize the result.

The standard pipeline

• Collect. A lightweight agent runs close to where logs are generated — on each host or, in Kubernetes, on each node — and picks up log output (stdout/stderr, log files) as it’s written. Fluent Bit, Fluentd, and Vector are common choices here because they’re built to run with a small resource footprint at this stage.
• Buffer (optional). At high volume, an intermediate queue — commonly Kafka — sits between collection and processing so that a downstream outage or slowdown doesn’t cause data loss or back-pressure all the way to the application. This is an addition most teams make only once collection volume or reliability requirements justify the added operational complexity.
• Process and enrich. Logs get parsed into structured fields, have metadata attached (Kubernetes pod labels, environment, region), get filtered or sampled, and are sometimes redacted for sensitive fields before storage. Fluentd and Vector both do this stage natively; a separate aggregation layer is also common for heavier transformation.
• Store. The processed logs land in a backend built for log storage and search — Elasticsearch, Grafana Loki, Splunk, or a managed SaaS equivalent — chosen based on query needs, retention requirements, and cost profile.
• Query and visualize. Engineers search, filter, and build dashboards on top of the stored logs, typically through the storage backend’s own UI (Kibana for Elasticsearch, Grafana for Loki, Splunk’s own interface) or a unified observability frontend that also shows traces and metrics alongside.

Architectural choices

• DaemonSet vs. sidecar collection in Kubernetes. Running the collector as a DaemonSet — one instance per node, reading every container’s logs on that node — is the default and lowest-overhead pattern, and is enough when applications log to stdout/stderr as expected. A sidecar collector, running inside the same pod as the application, is used instead when an application writes logs to a file rather than stdout, or needs per-application processing that shouldn’t be shared across every workload on the node.
• Kafka as a buffer at scale. Introducing Kafka (or a similar queue) between collection and processing decouples the two: a burst in log volume, or a temporary outage in the storage backend, is absorbed by the queue rather than causing collectors to drop logs or back up into the applications generating them. The trade-off is real operational cost — Kafka itself needs to be run, scaled, and monitored — so it’s typically adopted only once direct-to-backend shipping starts causing reliability problems.
• Aggregator layer vs. direct shipping. Some architectures route node-level collectors (e.g. Fluent Bit) to a smaller number of central aggregator instances (e.g. Fluentd or Vector) that do the heavier parsing, enrichment, and fan-out to storage — keeping the per-node footprint minimal while centralizing the more expensive processing logic.
• Self-hosted pipeline vs. shipping directly to a managed SaaS backend. Sending logs straight from a lightweight agent to a managed backend (Datadog, Splunk Cloud, Sumo Logic, and similar) skips building and operating the buffer/process/store stages yourself, trading operational effort for both a recurring cost tied to volume and less control over exactly how processing and retention are handled. Self-hosting the pipeline (e.g. Fluent Bit or Vector feeding Loki or self-managed Elasticsearch) is more operational work but gives direct control over cost levers like retention tiering and sampling.
• Consistency of format before storage. Regardless of which shape is chosen, the processing stage is where structured logging conventions actually get enforced across services that may not agree on format natively — normalizing field names and parsing unstructured lines into structured ones before they hit storage is usually cheaper than trying to fix it at query time.

The right shape depends mostly on scale and existing infrastructure: a small system can ship directly from an agent to a managed backend or a single Loki/Elasticsearch instance with no buffer at all, while a system logging at very high volume typically needs the full collect-buffer-process-store shape to stay reliable and keep cost under control. See the logs hub for how this pipeline choice connects to retention and cost strategy.