On Call

Log Monitoring by Message Bus

For this practical example, EDA monitors the system logs of multiple servers and responds to a wide variety of log entries. To access the logs, EDA could use the ansible.eda.journald source plugin; however, it would then have to keep active connections to all monitored servers open. In many places, administrators collect all logfiles in a centralized database anyway.

To ensure that the log data is not only available to a single service such as Logstash, you can deploy a message bus in the middle – in the best scale-out style: Apache Kafka. All log information passes through the message broker and target systems such as the log collection database, and EDA also connects to the message bus as a "subscriber."

If you are not yet familiar with this message bus technology, just think of Kafka as WhatsApp for applications. The logging systems join a chat group (topic) and post all their log entries in this group. Now "subscribers" such as EDA can also join the group and view and evaluate all the messages. In parallel, a log aggregator such as Logstash can retrieve all the messages in the group and forward them to a database such as Elasticsearch for archiving.

In this example, the Filebeat [7] tool first runs as a log shipper on the servers to be monitored. It takes care of turning unstructured log entries of system services into semi-structured messages such as Service=sshd or Message=Failed Login before Filebeat forwards them. The tool also caches all messages if the connection to the message bus goes down so that nothing is lost.

In many installations, Filebeat sends the data directly to Logstash, so it is not available to other services. However, switching to the message bus as a middleman does not involve too much overhead (Figure 1). It makes sense to do this in distributed, hybrid environments with multiple clouds and data center setups because it reduces the connections needed between the separate networks. Filebeat-Logstash turns into Filebeat-Kafka-Logstash, with the option for other services to listen in on Kafka.

Figure 1: Logstash can retrieve logs from the Kafka message bus and back them up to Elasticsearch.

From Filebeat to Kafka

Although large installations run Kafka as a cluster, a single container is fine for the test setup. You can optionally log the raw messages of the bus to a persistent volume. Messages usually end up with a log aggregator such as Logstash anyway, which is why the Kafka container works well in a stateless setup (i.e., without permanent storage).

Follow the instructions on the page of the image [8] and use the setup from the "Using the command line" section, which will set up kafka with "KRaft" as quorum instead of Zookeeper. Unlike the example, enter the external IP address of your server where you run the Kafka container with Docker or Podman. Add the an additional configuration line to advertise the external IP of your Kafka-Server.

KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://192.168.2.12:9092

(if your Docker/Podman server is running on 192.168.2.12). In a production environment, of course, the Kafka setup would not run in plaintext and would only allow authorized and encrypted connections. Once the Kafka container is running, configure the Filebeat clients on the servers to send their log information to Kafka. The /etc/filebeat/filebeat.yml file then looks something like:

filebeat.inputs:
- type: journald
  id: everything
 **
output.kafka:
  hosts: ["192.168.2.12:9092"]
  topic: 'journals'
  partition.round_robin:
    reachable_only: false

Filebeat sends all the log messages to the message broker journals topic. Alternatively, Filebeat could send messages to different topics depending on the content, such as with the optional entry:

topics:
  - topic: "critical"
    when.contains:
      message: "CRITICAL"

As soon as Filebeat sends log data you can check – on the host with the Kafka container – whether the information is reaching the message bus correctly. To do this, simply run the console consumer in the active Kafka container, kafka:

podman exec -it kafka kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic journals --from-beginning

Now you should see the JSON-formatted log entries of your systems with the Filebeat setup. If you optionally want to collect all the log information with Logstash, the lines:

input {
  kafka{
    codec => json
    bootstrap_servers => "192.168.2.12:9092"
    topics => ["journals"]
  }
}

in the Logstash configuration are all it takes.

Creating an Inventory

In a terminal, you can start the EDA container interactively with:

podman run -it --name eda --volume /home/user/rules:rules:Z quay.io/ansible/ansible-rulebook:main /bin/bash

Now you can edit the rulebooks and playbooks on the host machine and test them in the interactive container. By the way, this even works on Windows if you use a WSL distribution with Podman.

To begin, create an inventory in /home/user/rules. EDA currently only supports the YAML format for inventories and not the old INI format. In the test, two systems provide log information to Kafka, so the inventory.yml file looks like:

all:
  children:
    servers:
    hosts:
      srv1.local.ip:
        ansible_host: 192.168.2.1
      srv2.local.ip:
        ansible_host: 192.168.2.2
  vars:
    ansible_user: root

For the tests, Ansible logs in to the remote systems as the root user. In a production environment, there would be a user directory with sudo privileges and the playbook would use become.

I list the hosts with their fully qualified domain names (FQDNs) in the inventory because that is how they are listed in the log entries on the Kafka bus and can therefore be assigned to the inventory. However, you need to specify the IP address for the Ansible connection. The example here shows how EDA can react in case of a DNS failure; in that case, it has to contact the host by the IP address, of course.