Eloquent

Configuration Options

Fluentd uses a flexible and powerful configuration language that allows you to specify how it should collect, transform, and output logs. The configuration file is written in XML-like syntax and comprises several types of elements: <source>, <match>, <filter>, and <system>, among others.

The <source> element defines the input sources of log data. Each <source> element corresponds to an input plugin and its configuration, for example:

<source>
  @type forward
  port 24224
</source>

In this example, Fluentd is configured to use the forward input plugin to listen for incoming log events on port 24224.

The <match> element defines the output destinations of log data, and each element corresponds to an output plugin and its configuration, for example:

<match myapp.access>
  @type file
  path /var/log/fluent/myapp.access.log
  format json
</match>

As you can guess, in this case Fluentd is configured to use the file output plugin for logs tagged with myapp.access. The logs are written to the specified file in JSON format.

The <filter> element defines the transformations applied to log data. Each <filter> element corresponds to a filter plugin and its configuration, for example:

<filter myapp.access>
  @type record_transformer
  <record>
    hostname "#{Socket.gethostname}"
  </record>
</filter>

In this example, Fluentd uses the record_transformer filter plugin for logs tagged with myapp.access. The filter adds a hostname field to each log record.

The <system> element defines system-wide configuration options, such as

<system>
  log_level info
</system>

In this example, Fluentd is configured to log at the info level.

These simple configurations are just short examples; I'll discuss the various filters in more detail later on.

Deployment

Once Fluentd is installed and configured, the next step is to deploy it in your environment. The deployment strategy you choose depends on your specific needs and the nature of your environment. The simplest way to deploy Fluentd is as a standalone service on a single machine, which is a good choice for small environments or for testing and development purposes:

fluentd -c /<path/to>/fluentd.conf -d /<path/to>/fluentd.pid

In this command, -c specifies the path to the configuration file, and -d defines the path to the process ID (PID) file, which Fluentd creates when it starts.

Although a standalone deployment is simple to set up, it might not be sufficient for larger environments or for setups that require high availability or fault tolerance.

For large environments, you might want to deploy Fluentd on multiple nodes, which can help distribute the load and provide redundancy in case of node failures. In a multinode deployment, you typically have one or more aggregator nodes that collect logs from multiple forwarder nodes.

An example of a forwarder node configuration is shown in Listing 2, whereas Listing 3 shows an aggregator node. In these examples, the forwarder node tails a logfile and forwards the log events to the aggregator node, which writes the log events to a file.

<source>
  @type tail
  path /var/log/myapp.log
  tag myapp.log
  format json
</source>
<match myapp.log>
  @type forward
  <server>
    host aggregator.example.com
    port 24224
  </server>
</match>

<source>
  @type forward
  port 24224
</source>
<match myapp.log>
  @type file
  path /var/log/myapp.log
  format json
</match>

If you're using containerization technologies like Docker or Kubernetes, you can deploy Fluentd as a container. This method allows you to take advantage of the isolation, scalability, and deployment management features that these technologies provide.

In this Docker command that runs Fluentd,

docker run -d -p 24224:24224 -v /<path/to>/fluentd.conf:/fluentd/etc/fluent.conf fluent/fluentd

the -d runs the container in detached mode, -p maps container port 24224 to host port 24224, and -v mounts the host's configuration file at the specified path in the container.

You can also create a Dockerfile to build your own Fluentd image with custom plugins:

FROM fluent/fluentd:v1.16-debian-1
USER root
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document"]
USER fluent

This Dockerfile starts from the official Fluentd image, switches to the root user to install the Elasticsearch plugin, and then switches back to the Fluentd user for security.

Kubernetes provides several features that can help manage Fluentd deployments. One common pattern is to deploy Fluentd as a DaemonSet, which ensures that a Fluentd pod runs on each node in the Kubernetes cluster (Listing 4). In this example, the Fluentd DaemonSet uses a ConfigMap to provide the Fluentd configuration file. A command such as

kubectl create configmap fluentd-config --from-file=fluent.conf=/<path/to>/fluentd.conf

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: fluentd
  template:
    metadata:
      labels:
        name: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd:v1.16-debian-1
        volumeMounts:
        - name: config-volume
          mountPath: /fluentd/etc/fluent.conf
          subPath: fluent.conf
      volumes:
      - name: config-volume
        configMap:
          name: fluentd-config

creates the ConfigMap.

Input Plugins

I already mentioned plugins when describing a simple configuration. Among these plugins, input plugins play a crucial role in defining how Fluentd collects log data. Fluentd input plugins are responsible for collecting log data from various sources. They define where Fluentd should look for log data and how it should interpret it. The software comes with a variety of input plugins for common log sources, and you can also create your own if you have unique needs.

To use an input plugin, you need to include a <source> element in your Fluentd configuration file. This element should specify the type of plugin and any necessary configuration options:

<source>
  @type tail
  path /var/log/myapp.log
  tag myapp.log
  format json
</source>

In this example, Fluentd is configured to use the tail input plugin, which reads log data from a file. The path option specifies the file to read, the tag option specifies the tag to assign to the log events, and the format option specifies the format of the log data.

The tail input plugin, which you've already seen, is one of the most commonly used plugins. It reads log data from a file, making it a good choice for collecting logs from applications that write to logfiles. The forward input plugin listens for incoming log events over a network protocol and is useful for collecting logs from other Fluentd instances or from applications that can send logs over the network. The http input plugin listens for incoming log events over HTTP and is useful for collecting logs from applications that can send logs over HTTP.