Eloquent

Fluentd is an open source data collector designed to simplify the process of log management. It serves as a unified logging layer that sits between your applications and various log outputs. Its primary function is to ingest logs from different sources, transform them as needed, and then send them to the appropriate destinations.

One of the key strengths of Fluentd is its flexibility. It can collect logs from a wide variety of sources, including logfiles, system logs, and network protocols. It can also output these logs to a similarly wide variety of destinations, including databases, cloud storage, and other log analysis tools.

In a typical system, logs might be produced in different formats and sent to different locations, which can make it difficult to aggregate and analyze the logs in a meaningful way. Fluentd solves this problem by providing a single, consistent interface for handling logs, making it much easier to aggregate and analyze them because they're all in one place and in a consistent format.

This approach has several benefits. First, it simplifies log management because you only need to interact with the unified logging layer, rather than with each log source individually. Second, it is easier to analyze your logs because they're all in a consistent format. Third, it is easier to ensure that all your logs are stored securely and reliably because the unified logging layer can handle things like buffering and retrying failed log transmissions. Moreover, Fluentd (and Fluent Bit) can apply various transformations along the way (e.g., anonymization). In this way the same data can be used for different purposes depending on the case – and with minimum friction.

In dynamic cloud settings, applications might produce logs spread over an array of virtual machines, serverless platforms, and various services. Fluentd and Fluent Bit stand out in these situations, helping centralize this dispersed data and providing a unified log perspective. Practically all services from major public cloud providers have at least one Fluentd or Fluent Bit plugin, and in the realm of hybrid and multicloud scenarios, the versatility of Fluentd and Fluent Bit becomes even more evident. Companies operating their applications across numerous cloud providers or even a mix of on-site and cloud infrastructures can bank on Fluentd to gather logs from all these environments. This centralization ensures that logs aren't scattered, making management and review easier.

Suffice it to say that, at present, Fluentd boasts around 1,000 plugins for various applications. Whatever cloud service you use, chances are Fluentd has a plugin for it. [1].

Better Together

You probably already have a solution for metric collection and alerting, such as Prometheus. However, Prometheus doesn't handle logs, so Fluentd and Fluent Bit can fill this gap. In this way you get a complete picture of your system's state from logs, metrics, and alerts. Practically speaking, if Prometheus issues an alert triggered by a specific metric threshold, logs collected and processed by Fluentd or Fluent Bit can provide the contextual information needed to diagnose the root cause of the alert. If you lack some specific information in the log data (e.g., an availability zone or other AWS metadata), you can modify your setup relatively easily so that the next time the alert is triggered, you will receive the complete set of information.

OpenTelemetry [2], on the other hand, primarily focuses on distributed tracing and metrics. By adding Fluentd or Fluent Bit, you can bring logging into the observability mix, resulting in a comprehensive view of system performance, traces, and logs. Again, when issues arise, correlating trace data with log data is beneficial. By using both tools, you can link a specific request or transaction (captured by OpenTelemetry) with detailed log events (captured by Fluentd or Fluent Bit) to facilitate in-depth troubleshooting.

Installation

Before beginning the installation process, it's important to ensure that your system meets the necessary prerequisites. Fluentd is designed to be lightweight and can run on nearly any modern system. However, it does require a few things. It is written in Ruby, and you'll need Ruby version 2.1 or later to run it. You can check your Ruby version by running

ruby -v

in your terminal. If you don't have Ruby installed, you can install it with your favorite package manager, for example:

apt install ruby ruby-dev

For best performance, you should have at least 1GB of RAM available. On low-end systems with very little memory you should use Fluent Bit, which I discuss later.

With the prerequisites out of the way, you can move on to the installation process. The easiest way to install Fluentd is to use the Ruby package manager, RubyGems:

gem install fluentd

This command downloads and installs the latest version of Fluentd. After the installation process is complete, you can verify that Fluentd was installed correctly by running the command

fluentd --version

which should print the version of Fluentd you have installed.

Apart from Fluentd, in the repository of your distribution you will probably find td-agent which is a stable version packaged by Treasure Data.

A Simple Configuration

After installing Fluentd, the next step is to create a configuration file. Fluentd uses a flexible and powerful configuration language that allows you to specify how it should collect, transform, and output logs. A simple example of a Fluentd configuration file is shown in Listing 1.

<source>
  @type forward
  port 24224
</source>
<match *.**>
  @type stdout
</match>

This configuration tells Fluentd to listen for logs on port 24224 and output them to the console. Then, you can start up by running

fluentd -c fluentd.conf

which is the Fluentd configuration file you created in the previous step.