Eloquent

Installing Fluent Bit

Depending on your use case, you can install Fluent Bit with a package manager, for example, by issuing the Debian/Ubuntu command:

sudo apt-get install td-agent-bit

or running a Docker image:

docker pull fluent/fluent-bit
docker run fluent/fluent-bit

as an example of the simplest case.

Configuration

Now construct a configuration file that will allow you to monitor CPU usage and send the metrics to an Elasticsearch instance on localhost. You can use the same [SERVICE] section as in Listing 5, setting the flush interval of five seconds and logging the events at the info level (Listing  6).

[SERVICE]
    Flush        5
    Log_Level    info
[INPUT]
    Name        cpu
    Tag         cpu.local
[OUTPUT]
    Name        es
    Match       *
    Host        localhost
    Port        9200
    Index       cpu_metrics

For [INPUT], you simply use cpu as the name and cpu.local as the tag, which tells Fluent Bit to collect CPU metrics and tag them with cpu.local. In the [OUTPUT] section, instead of just displaying data to stdout, as in Listing 5, you define Host and Port, pointing Fluent Bit to a local Elasticsearch instance listening on port 9200. The logs are stored in the index called cpu_metrics. The es plugin has more options (e.g., Buffer_Size and Retry_Limit) that can influence the efficiency of the whole setup.

Elasticsearch is a popular option, but many more are possible. If you use AWS, you might want to use the s3 output plugin,

[OUTPUT]
    Name        s3
    Match       *
    bucket      my-log-bucket
    region      us-west-1
    store_dir   /tmp/

to send your logs directly to a Simple Storage Service (S3) bucket.

Input Sources

The primary responsibility of Fluent Bit input plugins is to act as gatekeepers, ushering data into the Fluent Bit system. They determine the source of your data, be it system metrics, files, network traffic, or custom sources. One of Fluent Bit's most versatile plugins is tail, an input plugin. It's designed to read from the end of files, making it particularly useful for tracking logs. With the following setup, Fluent Bit can monitor an application's logfile:

[INPUT]
    Name    tail
    Path    /var/log/myapp.log

You have already seen the cpu plugin. Fluent Bit's ability to adapt to various data sources extends farther with plugins such as netif, which collects network interface metrics, or mem, which is designed to fetch memory usage statistics.

In a more complex scenario, assume you're running a web server and you want to track both the access logs and error logs. Fluent Bit allows you to use multiple input plugins within the same configuration (Listing 7). The dual configuration ensures that both access and error logs flow into Fluent Bit. The distinct tags nginx.access and nginx.error allow for easy differentiation and processing downstream.

[INPUT]
    Name    tail
    Path    /var/log/nginx/access.log
    Tag     nginx.access
[INPUT]
    Name    tail
    Path    /var/log/nginx/error.log
    Tag     nginx.error