Don't Like Reading Logs? Try the ELK Stack

You and your boss probably don't want to spend a lot of time reading raw logfiles. Users often combine Bro with data visualization tools for more effective presentation. One popular set of open source tools for data collection and presentation is the so-called "ELK" stack, which comprises:

• Elasticsearch: Enables sophisticated searches of large amounts of volatile data.
• Logstash: Collects, stores, and parses logfiles from remote hosts.
• Kibana: Visualizes data so that it appears less abstract and has higher impact.

These applications provide a graphical visualization of the logfiles you've captured with Bro. For example, Figure 8 shows Kibana's output of a Bro logfile. Instead of reviewing overly technical data, such as bad_TCP_checksum data, Kibana can visualize this information so that you can identify essential trends on the network.

Figure 8: Kibana visualizing Bro data.

To set up the ELK stack, start by installing Java 8, or the latest stable version. In my system, I used the commands in Listing 1 to set up the ELK stack and install Elasticsearch. I can then set up the Elasticsearch initialization script:

# Set up ELK stack
$ sudo add-apt-repository -y ppa:webupd8team/java
$ sudo apt-get update
$ echo debconf shared/accepted-oracle-license-v1-1 select true | sudo debconf-set-selections
$ echo debconf shared/accepted-oracle-license-v1-1 seen true | sudo debconf-set-selections
$ sudo apt-get -y install oracle-java8-installer
# Install Elasticsearch
$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
$ echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
$ sudo apt-get update && sudo apt-get install elasticsearch

$ sudo update-rc.d elasticsearch defaults 95 10

Once Elasticsearch is installed, I then installed Logstash:

$ echo "deb https://packages.elastic.co/logstash/2.3/ debian stable main" | sudo tee -a /etc/apt/sources.list
$ sudo apt-get update && sudo apt-get install logstash

Make sure Logstash is part of the startup scripts:

$ sudo update-rc.d logstash defaults 95 10

Finally, you can install Kibana:

$ echo "deb http://packages.elastic.co/kibana/4.5/debian stable main" | sudo tee -a /etc/apt/sources.list
$ sudo apt-get update && sudo apt-get install kibana

Once again, I can then create the System V scripts:

$ sudo update-rc.d kibana defaults 95 10

Once I have Kibana running, I can then use the web interface to point it toward my Bro log directories (e.g., those in the current directory), then I can start parsing and visualizing data.

Conclusion

Network security monitoring software has been around a long time. But now, we're starting to see software, such as Bro, that has a bit more capability. Instead of merely looking for pre-defined traffic patterns, Bro has the ability to identify trends. Using visualization software such as the ELK stack, it is possible to sift through all of this data to discover truly useful trends. The key, of course, is in properly configuring Bro to process relevant information. This takes a bit of fine tuning of scripts, as well as quite a bit of trial and error. But with some time, you'll be able to identify key security issues and trends quickly. Although you might not think you're doing "big data," with Bro, you are, in a very real sense. You're taking unstructured data and quickly discovering meaningful patterns, and these trends represent information that you will find truly useful in your work.