Output Plugins

Fluentd output plugins are responsible for sending the log data that Fluentd collects to various destinations. These destinations can range from files on a local system to databases, cloud storage, or other log analysis tools. Fluentd comes with a variety of output plugins for common log destinations, and, as with input plugins, you can create your own if you have unique requirements.

To use an output plugin, you need to include a <match> element in your Fluentd configuration file. This element should specify the type of the plugin, any necessary configuration options, and a pattern that matches the tags of the log events you want to send. For example, in Listing 3, Fluentd is configured to use the file output plugin, which writes log data to a file. The path option specifies the file to write to, the format option specifies the format of the log data, and the pattern myapp.log matches any log events whose tag equals myapp.log.

The file output plugin, already discussed, is one of the most commonly used. It writes log data to a file, making it a good choice for storing logs locally for later analysis. The forward output plugin, presented in Listings  2 and 3, sends log events to another Fluentd instance over a network protocol and is useful for sending logs to a central Fluentd instance for aggregation. The elasticsearch output plugin sends log events to an Elasticsearch cluster; it is useful for storing logs in a searchable and scalable database.

Filter Plugins

This presentation of plugins would be incomplete without mentioning filter plugins, which are one of the reasons behind the flexibility of Fluentd. Fluentd filter plugins process the collected log data by modifying or adding information, or even by filtering out certain records. Fluentd comes with a variety of filter plugins for common data processing tasks, and you can also create your own if you have unique requirements.

To use a filter plugin, you need to include a <filter> element in your Fluentd configuration file that specifies the type of the plugin, any necessary configuration options, and a pattern that matches the tags of the log events you want to filter, for example:

<filter myapp.**>
  @type record_transformer
  <record>
    hostname "#{Socket.gethostname}"
  </record>
</filter>

In this example, Fluentd is configured to use the record_transformer filter plugin, which modifies log records. The pattern myapp.** matches any log events whose tag starts with myapp., and the <record> element specifies that a hostname field should be added to each log record.

The grep filter plugin allows you to filter out log records by content. You can specify a regular expression, and any log records that don't match the expression will be filtered out. In this example configuration, any log records whose message field doesn't contain the string error will be filtered out:

<filter myapp.**>
  @type grep
  <regexp>
    key message
    pattern /error/
  </regexp>
</filter>

The parser filter plugin allows you to parse a field in each log record and is useful if your log data is in a structured format like JSON or XML. In this example configuration,

<filter myapp.**>
  @type parser
  key_name log
  format json
</filter>

the parser plugin is configured to parse the log field of each record as JSON.

Fluent Bit

Although critical parts of Fluentd are written in C, the rest is written Ruby, which is great for flexibility but has an effect on performance and resource usage. If you need something lighter, you will probably use Fluent Bit. Although the number of available plugins is an order of magnitude smaller, Fluent Bit is very light and will run on devices that would have trouble running Ruby itself.

A typical scenario in which Fluentd and Fluent Bit are used is container logging. With the ascendancy of Kubernetes and similar container orchestration systems, applications are frequently dispersed across numerous containers. Fluent Bit, given its lightweight structure, is especially favored for collecting logs in these contexts and then transmitting them to Fluentd or alternative logging back ends.

Simply speaking, Fluent Bit prioritizes performance and a lightweight footprint, whereas Fluentd focuses on being highly extensible, with a vast plugin ecosystem. Therefore, Fluent Bit is ideal for scenarios requiring quick startups and low resource usage (e.g., Kubernetes logging or Internet of Things devices). On the other hand, Fluentd, with its richer set of plugins, is often used in more complex log aggregation setups, where data transformation and enrichment are necessary.

Listing 5 is a simple configuration file that presents a glimpse of the simplicity and power of Fluent Bit. The [SERVICE] section defines service-wide settings. Here, Flush is set to 5, meaning data is flushed every five seconds. When Daemon is set to Off, Fluent Bit runs in the foreground.

[SERVICE]
    Flush     5
    Daemon   Off
[INPUT]
    Name     tail
    Path     /var/log/syslog
[OUTPUT]
    Name     stdout
    Match    *

The [INPUT] section defines where the logs originate. With the tail input plugin, you're reading from the system's syslog. Finally, the [OUTPUT] section decides where the logs go. Here, they're being sent to the console (stdout).