Good for You!

Riemann Clients

After making sure the Riemann server can receive streams and events, it is time to set up clients. The Riemann project provides tools that collect and send common metrics for observing CPU load, processes, and disk fill levels and to monitor common daemons, such as web, proxy, and NTP. All of these tools are bundled in the riemann-tools package and can be installed by typing on the client machine:

sudo gem install riemann-tools

Less frequently used or more complex tools, such as monitoring AWS instances (riemann-aws), Elasticsearch (riemann-elasticsearch), and Docker installations (riemann docker), are available as separate Gems. A search using gem search riemann- discovers a number of clients, and the project page also lists interesting external applications [13]. GitHub has clients for Windows, and thanks to Homebrew, Riemann also runs on OS X [14].

The individual tools do not have separate man pages, but the -help parameter reveals the most important options. For example, the riemann-health program bundled in the riemann-tools gem monitors CPU, hard disk, memory, and load (Figure 1). The -h switch names the Riemann server, and -e specifies the client name; optional specifications on warning values and critical values for the CPU, hard disks, and load follow:

riemann-health -h <Server> -e <Client> -u 0.2 -r 0.5 -d 0.2 -s 0.3 -o l &

Figure 1: The -help parameter lists the known options for all Riemann programs.

Each metric has a process that passes its values to the Riemann server. Depending on the number of system parameters you want to monitor, you will see that a number of processes were created just for monitoring.

You will search in vain for startup scripts that save you from entering these long commands at every boot; however, individual systemd unit files (services here) under /etc/systemd/system are a potential solution. Listing 6 shows an example of the riemann-health.service unit on client bert that keeps an eye on the processor, disks, and memory.

[Unit]
Description=Riemann Health
After=network.target
[Service]
ExecStart=/usr/local/bin/riemann-health --host dc_monitoring --event-host=bert -u 0.2 -r 0.5 -d 0.2 -s 0.3 -o 1 --load-critical=2
PidFile=/var/run/riemann-health.pid
Restart=on-failure
[Install]
Alias=riemann-health.service

Now that the Riemann server is receiving data from clients, you need to think about notification options. Listing 2 shows what email alerts might look like. The Clojure configuration supports other approaches, such as SMS, IRC, or cooperation with external services [10] such as HipChat, PagerDuty, Blueflood, or Campfire.

To capture a coherent current state of all clients, you can install the riemann-dash Ruby gem. Implemented in the Sinatra web application framework, the tool pulls in some dependencies. After starting the command of the same name, you reach the web interface on http:/localhost:4567 . For an example of the configuration file that adapts this and other options for your own server, see the Riemann GitHub page [15].

The config.rb file must be located in the same directory from which you launch riemann-dash. Alternatively, you can set the RIEMANN_DASH_CONFIG variable appropriately. Again, you'll find no startup script, but you can create a systemd service unit.

Visualized

At first glance, the web interface does not offer much, with only two widgets on view: Riemann and online help, which gives you tips on how to set up. On closer inspection, the interface is quite flexible – once you get used to the rather convoluted controls. Queries follow the same syntax used by the command-line programs (Figure 2).

Figure 2: Press Ctrl+E to edit a widget in the dash and select a grid or list view, charts, and so on. Define your filter in the Query field.

Pressing the Ctrl + arrow keys splits the view as needed (Figure 3). Pressing + and - changes the dimensions, and pressing S saves the current layout. At the top left you will find buttons for adding more tabs for custom dashboards. A double-click in the box lets you change the name. At top right is the current load and an input box. If your Riemann server is not running on the local system, you need enter the correct IP address here.

Figure 3: A dashboard accommodates any number of widgets of different sizes. You can easily create more by clicking the plus icon.

Teamwork with Graphite

All widgets take their data directly from the Riemann index, which they query via websockets. Because this index resides exclusively in RAM, it is not possible to visualize history data, as mentioned earlier. Integrating an external tool is a potential solution. In the test environment, Graphite [16] handled this job to document long-term trends. Listing 2 shows how to integrate the Graphite service and forward all Riemann events to it. If you want a more selective approach, you can configure a corresponding stream in the Riemann configuration file; the following example only sends events from the PostgreSQL service:

(streams
  (where (service "postgres")
    graph))

Graphite itself requires no special settings to handle Riemann server events. An article on the Linux Administration website describes how to install and use Graphite [17]. You can access the web console on http://localhost:8080 . The clients are listed in a tree view on the left. To view an entry in Graphite Composer on the right (Figure 4), just click on it. You can define specific time periods and customize the chart's appearance.

Figure 4: Graphite provides Riemann metrics over a long period.