A modern logging solution
Eloquent
Output Plugins
Fluentd output plugins are responsible for sending the log data that Fluentd collects to various destinations. These destinations can range from files on a local system to databases, cloud storage, or other log analysis tools. Fluentd comes with a variety of output plugins for common log destinations, and, as with input plugins, you can create your own if you have unique requirements.
To use an output plugin, you need to include a <match>
element in your Fluentd configuration file. This element should specify the type of the plugin, any necessary configuration options, and a pattern that matches the tags of the log events you want to send. For example, in Listing 3, Fluentd is configured to use the file
output plugin, which writes log data to a file. The path
option specifies the file to write to, the format
option specifies the format of the log data, and the pattern myapp.log
matches any log events whose tag equals myapp.log
.
The file
output plugin, already discussed, is one of the most commonly used. It writes log data to a file, making it a good choice for storing logs locally for later analysis. The forward
output plugin, presented in Listings 2 and 3, sends log events to another Fluentd instance over a network protocol and is useful for sending logs to a central Fluentd instance for aggregation. The elasticsearch
output plugin sends log events to an Elasticsearch cluster; it is useful for storing logs in a searchable and scalable database.
Filter Plugins
This presentation of plugins would be incomplete without mentioning filter plugins, which are one of the reasons behind the flexibility of Fluentd. Fluentd filter plugins process the collected log data by modifying or adding information, or even by filtering out certain records. Fluentd comes with a variety of filter plugins for common data processing tasks, and you can also create your own if you have unique requirements.
To use a filter plugin, you need to include a <filter>
element in your Fluentd configuration file that specifies the type of the plugin, any necessary configuration options, and a pattern that matches the tags of the log events you want to filter, for example:
<filter myapp.**> @type record_transformer <record> hostname "#{Socket.gethostname}" </record> </filter>
In this example, Fluentd is configured to use the record_transformer
filter plugin, which modifies log records. The pattern myapp.**
matches any log events whose tag starts with myapp.
, and the <record>
element specifies that a hostname
field should be added to each log record.
The grep
filter plugin allows you to filter out log records by content. You can specify a regular expression, and any log records that don't match the expression will be filtered out. In this example configuration, any log records whose message
field doesn't contain the string error
will be filtered out:
<filter myapp.**> @type grep <regexp> key message pattern /error/ </regexp> </filter>
The parser
filter plugin allows you to parse a field in each log record and is useful if your log data is in a structured format like JSON or XML. In this example configuration,
<filter myapp.**> @type parser key_name log format json </filter>
the parser
plugin is configured to parse the log
field of each record as JSON.
Fluent Bit
Although critical parts of Fluentd are written in C, the rest is written Ruby, which is great for flexibility but has an effect on performance and resource usage. If you need something lighter, you will probably use Fluent Bit. Although the number of available plugins is an order of magnitude smaller, Fluent Bit is very light and will run on devices that would have trouble running Ruby itself.
A typical scenario in which Fluentd and Fluent Bit are used is container logging. With the ascendancy of Kubernetes and similar container orchestration systems, applications are frequently dispersed across numerous containers. Fluent Bit, given its lightweight structure, is especially favored for collecting logs in these contexts and then transmitting them to Fluentd or alternative logging back ends.
Simply speaking, Fluent Bit prioritizes performance and a lightweight footprint, whereas Fluentd focuses on being highly extensible, with a vast plugin ecosystem. Therefore, Fluent Bit is ideal for scenarios requiring quick startups and low resource usage (e.g., Kubernetes logging or Internet of Things devices). On the other hand, Fluentd, with its richer set of plugins, is often used in more complex log aggregation setups, where data transformation and enrichment are necessary.
Listing 5 is a simple configuration file that presents a glimpse of the simplicity and power of Fluent Bit. The [SERVICE]
section defines service-wide settings. Here, Flush
is set to 5
, meaning data is flushed every five seconds. When Daemon
is set to Off
, Fluent Bit runs in the foreground.
Listing 5
Simple Fluent Bit Config File
[SERVICE] Flush 5 Daemon Off [INPUT] Name tail Path /var/log/syslog [OUTPUT] Name stdout Match *
The [INPUT]
section defines where the logs originate. With the tail
input plugin, you're reading from the system's syslog. Finally, the [OUTPUT]
section decides where the logs go. Here, they're being sent to the console (stdout
).
Buy this article as PDF
(incl. VAT)
Buy ADMIN Magazine
Subscribe to our ADMIN Newsletters
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Most Popular
Support Our Work
ADMIN content is made possible with support from readers like you. Please consider contributing when you've found an article to be beneficial.
![Learn More”>
</a>
<hr>
</div>
</div>
<div class=](https://www.admin-magazine.com/var/ezflow_site/storage/images/media/images/learn-more/211417-1-eng-US/Learn-More.png)