How Fluentd Works: The Unified Logging Layer

Scenario context

Fluentd is an open-source data collector that decouples log sources from destinations. It ingests logs from 100+ sources (Nginx, Docker, syslog), normalizes them to JSON, applies filters and transformations, and routes them to 100+ outputs (Elasticsearch, S3, Kafka). Tag-based routing is the core concept.

Every piece of data flowing through Fluentd is called an event. Each event has exactly three component…

This walkthrough focuses on Understanding Log Events: Tag, Time, and Record.

Before making a design choice, clarify:

• Every piece of data flowing through Fluentd is called an event .
• Each event has exactly three components: Think of it like a letter in a postal system: the tag is the address, the time is the postmark, and the record is the contents.
• Fluentd's routing engine reads only the tag to decide where the letter goes — it never needs to open the envelope.

Concepts in play: jvm-internals-and-garbage-collection, api-gateway, caching, database-indexing

Stage 1 — Tags and Routing: Fluentd's Core Concept

• Every event in Fluentd has a tag — a dot-separated string that determines where the event is routed.
• • web.nginx events match <match web.
• → go to Elasticsearch - app.backend events match <match app.

<source>
  @type tail
  path /var/log/nginx/access.log
  tag web.nginx
  format nginx
</source>

<source>
  @type tail
  path /var/log/app/app.log
  tag app.backend
  format json
</source>

<match web.**>
  @type elasticsearch
  host elastic.local
  port 9200
  index_name nginx-logs
</match>

<match app.**>
  @type s3
  s3_bucket my-log-archive
  path logs/%Y/%m/%d/
</match>

flowchart TD
    Nginx[Nginx access.log tag: web.nginx] --> Fluentd
    App[App log tag: app.backend] --> Fluentd
    Sys[Syslog tag: system.kernel] --> Fluentd

    Fluentd -->|match web.**| ES[Elasticsearch]
    Fluentd -->|match app.**| S3[Amazon S3]
    Fluentd -->|match system.**| Kafka[Kafka topic]

Observable outcome: → go to Elasticsearch - app.backend events match <match app.

Stage 2 — End-to-End Pipeline: From Log Line to Storage

• Understanding the tag concept is one thing — seeing data flow from a raw log line all the way to a storage backend makes it concrete.
• The diagram below traces a single Nginx request through every stage of a typical Fluentd pipeline: Here is what happens at each step: 1.
• Input (tail) — Fluentd watches /var/log/nginx/access.log for new lines, stamping each with the tag web.nginx and a Unix timestamp.
• Parser (nginx) — The raw text line ( 127.0.0.1 - [date] "GET /api" 200 512 ) is parsed into a structured JSON record using Nginx's known log format.

flowchart LR
    A[ Nginx access.log] -->|tail plugin reads new lines| B[ Input: tail tag: web.nginx]
    B -->|raw text  JSON| C[ Parser: nginx {status, path, method}]
    C -->|add hostname + env fields| D[ Filter: record_transformer {status, path, hostname, env}]
    D -->|batch to disk first| E[ Buffer: file /var/log/fluentd-buffer]
    E -->|flush every 5 s| F[ Output: elasticsearch nginx-logs index]

Observable outcome: Parser (nginx) — The raw text line ( 127.0.0.1 - [date] "GET /api" 200 512 ) is parsed into a structured JSON record using Nginx's known log format.

Stage 3 — The Plugin Architecture: Input → Filter → Buffer → Output

• Fluentd's power comes from its plugin model: Buffer plugins are critical for reliability.
• Without buffering, a downstream outage (e.g., Elasticsearch restart) causes log loss.
• With a file buffer: - Events are written to disk first.
• • Flushed to the output on schedule (or when the buffer fills).

<match **>
  @type elasticsearch
  host elastic.local

  <buffer>
    @type file
    path /var/log/fluentd-buffer
    flush_interval 5s
    retry_max_times 10
  </buffer>
</match>

Observable outcome: - Flushed to the output on schedule (or when the buffer fills).

Stage 4 — Deep Dive: Filter Enriching and Transforming Events

• Filters run in the pipeline between input and output: This adds environment , hostname , and a derived log level to every Nginx event before sending to Elasticsearch.

<filter web.nginx>
  @type record_transformer
  enable_ruby true
  <record>
    environment "production"
    hostname "#{Socket.gethostname}"
    log_level ${record["status"].to_i >= 500 ? "ERROR" : "INFO"}
  </record>
</filter>

Observable outcome: Filters run in the pipeline between input and output: This adds environment , hostname , and a derived log level to every Nginx event before sending to Elasticsearch.

Stage 5 — Practical: Collecting Kubernetes Container Logs

• In Kubernetes every container's stdout/stderr lands in /var/log/containers/ on the host node.
• A Fluentd DaemonSet reads these files and ships them to a central store.
• The ConfigMap below shows a minimal but production-ready pipeline: Three design decisions are worth calling out: - pos file records the last-read byte offset for every log file.
• If the Fluentd pod restarts (node eviction, OOM, rolling upgrade), it resumes from exactly where it left off — no duplicate lines, no gaps.

# fluentd-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
data:
  fluent.conf: |
    <source>
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos   # ← resume after restart
      tag kubernetes.*
      read_from_head true
      <parse>
        @type json
      </parse>
    </source>

    <filter kubernetes.**>
      @type kubernetes_metadata   # ← enriches with pod name, namespace, labels
    </filter>

    <match kubernetes.**>
      @type elasticsearch
      host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
      port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
      logstash_format true        # ← creates logstash-YYYY.MM.DD indices for Kibana
      logstash_prefix k8s
      <buffer>
        @type file
        path /var/log/fluentd-buffer
        flush_interval 5s
        retry_max_times 17
      </buffer>
    </match>

Observable outcome: If the Fluentd pod restarts (node eviction, OOM, rolling upgrade), it resumes from exactly where it left off — no duplicate lines, no gaps.

Stage 6 — Logback Fluentd Appender: Shipping Structured Logs from Spring Boot to Fluentd

• Spring Boot applications log via SLF4J + Logback by default.
• The fluent-logger-java Logback appender encodes each log event as a MessagePack record and forwards it over TCP to Fluentd's @type forward input — the same Tag, Time, Record model described in this post.
• Structured field arguments (via logstash-logback-encoder ) become JSON keys in the Fluentd record, making them fully queryable in Kibana without any Fluentd filter plugin.
• The tag: app.payments in Logback maps directly to Fluentd's <match app.

<!-- pom.xml -->
<dependency>
    <groupId>org.fluentd</groupId>
    <artifactId>fluent-logger</artifactId>
    <version>0.3.4</version>
</dependency>
<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>7.4</version>
</dependency>

<!-- src/main/resources/logback-spring.xml -->
<configuration>
    <!-- Console output for local development -->
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
    </appender>

    <!-- Fluentd forward appender for production -->
    <appender name="FLUENTD"
              class="org.fluentd.logger.logback.FluentLogbackAppender">
        <tag>app.payments</tag>           <!-- routes to <match app.**> in Fluentd -->
        <remoteHost>fluentd-aggregator</remoteHost>
        <port>24224</port>
        <maxQueueSize>500</maxQueueSize>  <!-- in-memory buffer before forward -->
    </appender>

    <springProfile name="prod">
        <root level="INFO">
            <appender-ref ref="FLUENTD"/>
        </root>
    </springProfile>
    <springProfile name="!prod">
        <root level="INFO">
            <appender-ref ref="CONSOLE"/>
        </root>
    </springProfile>
</configuration>

Observable outcome: The tag: app.payments in Logback maps directly to Fluentd's <match app.

End-to-end outcome

Every piece of data flowing through Fluentd is called an event .

Alternatives considered

Real-World Applications: Fluentd vs Logstash vs Fluent Bit

The common production pattern: Fluent Bit as a lightweight DaemonSet on every node, forwarding to a central Fluentd aggregation layer, then to Elasticsearch/S3.

Decision Guide: When to Choose Fluentd

Decision Guide: When to Choose Fluentd

Failure and degraded modes

Trade-offs & Failure Modes: Fluentd Configuration Pitfalls

• Detection: Trade-offs & Failure Modes: Fluentd Configuration Pitfalls
• User or operator impact: Trade-offs & Failure Modes: Fluentd Configuration Pitfalls