It’s the Little Things

vmstat

One of those *nix commands that gets no respect is vmstat . However, it can be an extremely useful command, particularly for HPC. Vmstat reports Linux system virtual memory statistics. Although it has several “modes,” I find the default mode to be extremely useful. Listing 2 is a quick snapshot of a Linux laptop.

Listing 2: vmstat on a Laptop

[laytonjb@laytonjb-Lenovo-G50-45 ~]$ vmstat 1 5
procs  -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd    free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 5279852   2256 668972    0    0  1724    25  965 1042 17  9 71  2  0
 1  0      0 5269008   2256 669004    0    0     0     0 2667 1679 28  3 69  0  0
 1  0      0 5260976   2256 669004    0    0     0   504 1916  933 25  1 74  0  0
 2  0      0 5266288   2256 668980    0    0     0    36 4523 2941 29  4 67  0  0
 0  0      0 5276056   2256 668960    0    0     4     4 9104 6262 36  5 58  0  0

Each line of output corresponds to a system snapshot at a particular time (Table 1), and you can control the amount of time between snapshots. The first line of numbers are the metrics since the system was rebooted. Lines of output after that correspond to their current values. A number of system metrics are very important. The first thing is to look at is the number of processes (r  and b ). If these numbers start moving up, something unusual might be happening on the node, such as processes waiting for run time or sleeping.

Table 1: vmstat Output

The metrics listed as m emory can be useful, particularly as the kernel grabs and releases memory. You shouldn't be too worried about these values unless the values in the next section (swap ), are non-zero. If you see non-zero si and so values, excluding the first row, you should be concerned, because it indicates that the system is swapping, and swapping memory to disk and can really kill performance. If a user is complaining about performance and you see a node running really slowly with a very large load, then it’s a good possibility that the node is swapping.

The metrics listed in the io section are also good to watch. They list either blocks sent to a block device or blocks received from a block device. If these numbers are both large, the application running on the nodes is likely doing something unusual by reading and writing to the device at the same time. This situation too can hurt performance.

The other metrics can be very useful, but I tend to focus on those mentioned first before scanning the others. You can also send this data to a file for later postprocessing or plotting – for example, for debugging user problems on nodes.