Painting by Numbers

statsd

If you are not interested in operating system metrics, but in the performance of your own applications, statsd is your friend. This daemon can count events, measure their duration, and buffer and forward the resulting values on a regular basis to a back end like Graphite.

Libraries for a number of programming languages can increment counters, start timers, and send the results to statsd. To start statsd from /opt/statsd via the Node.js JavaScript server, you can enter the command:

/usr/bin/node ./stats.js ./localConfig.js

Before you do this, however, you need to generate or edit the configuration file localConfig.js, which should contain at least the following lines (you will need to adjust the IP address):

{
   graphitePort: 2003,
   graphiteHost: "192.168.56.102",
   port: 8125
}

In this example, I use the Perl package Net::Statsd to instrument a benchmark (Listing 2).

01 #!/usr/bin/perl
02
03 use Time::HiRes qw(gettimeofday tv_interval);
04 use Net::Statsd;
05
06 $Net::Statsd::HOST = 'localhost';    # Default
07 $Net::Statsd::PORT = 8125;           # Default
08
09 my $t0 = [gettimeofday];
10
11 my($one, $two, $three) = map { $_ x 4096 } 'a'..'c';
12
13 $s .= $one;
14 $s .= $two;
15 $s .= $three;
16
17 my $temp;
18 for (my $i=0; $i<12288; $i++) {
19    $temp=substr($s,length($s)-1,1);
20    $s=$temp.$s;
21    $s = substr($s,0,12288);
22 }
23
24 $elapsed = tv_interval ( $t0, [gettimeofday]);
25 $elapsed = int($elapsed * 1000 * 1000);
26
27 Net::Statsd::timing('charbench', $elapsed);

This trivial script first creates a 4KB string of each of the letters a , b , and c ; then, it puts the last character of the string into the first position over and over until the string reaches its initial state.

The script also measures the duration of each pass and sends the value via UDP to statsd. Statsd subsequently forwards the value to Graphite. By starting the script as a cron job once a minute, you can create a chart as in Figure 3.

Figure 3: Values passed from the instrumented benchmark, to statsd, and then to Graphite.

Web GUI

The web GUI contains the tree view of all configured metrics in the left frame and shows a resulting graph in the right frame. If multiple metrics have a similar range of values, you can superimpose graphs.

In all other cases, use the Dashboard button, which leads to an area in which you can arrange a number of graphs in the same window by choosing metrics either from the tree view at the left or from a menu in a top frame (Figure 4).

Figure 4: Choose the metrics for the dashboard..

Every metric you choose gets its own graph. If you drag one chart on top of another, the graphs are combined (Figure  5).

Figure 5: Dashboard with superimposed charts.

You can save dashboards and reload them later, and a search function lets you search for saved dashboards. Chart sizes, periods of time displayed, and automatic refresh intervals can all be customized.

Conclusion

Graphite is released under the Apache 2.0 license. It was written by Chris Davis and is maintained and actively developed by Orbitz.com to "… visualize a variety of operations-critical data including application metrics, database metrics, sales, etc.," and "… handle approximately 160,000 distinct metrics per minute running on two niagra-2 Sun servers on a very fast SAN" [6]. Graphite is thus best used in environments that need to monitor thousands of regularly updated metrics in real time.