Stressing Out

I have discussed load-generating tools on these pages from the very beginning [1] out of simple practical necessity. Setting aside the benchmarks themselves, the facility generating a load consistently over and over again is the most important tool in a performance tester's arsenal. Amos Waterland's stress [2] has long been my go-to tool because of its simple, reliable effectiveness, but I have been watching with interest Colin King's promising stress-ng [3] since our time together on the Ubuntu Server team. Time to take the challenger out for a spin!

The New King

The newer stress-ng tool can be promptly installed with

sudo apt install stress-ng

in Ubuntu Server versions dating as far back as 2015. Insuring a system is exercised consistently during testing is the tool's primary mission, validating software response as well as hardware reliability and its behavior under load. The latest release provides a menagerie of 300 stressors crafted to load the CPU (with a variety of more than 80 tests), memory and CPU caches, filesystems, and even the virtual memory system.

The original stress syntax remains valid, and one can start up resource stressors in the same fashion. Figure 1 shows one such example, including the first notable difference. The tool itself originated as a load generator, but it also outputs results that can be construed as a benchmark of sorts. The maintainer advises against using stress-ng as a precise benchmark on the project's web page, but he also points out that these metrics can be useful to monitor relative regression between operating system (OS) versions, particularly when used on the same hardware.

Figure 1: The new tool can include load statistics in its output.

Load figures are provided in arbitrary "Bogo Ops" [4] units (bogus operations per second) and will run for a default period of 24 hours unless instructed otherwise (with the -t or --timeout options). Bogo Ops are defined in the context of a specific stressor and should not be compared across different resource loads, but they absolutely can be compared within the same stressor. While not aspiring to be perfect benchmarks, they do make a convenient starting point.

Complex Loads

I am not running the new tool to use the old loads. A good example of what the 300 stressors include is matrix, notorious for being the stressor expected to heat Intel processors the most by stressing the CPU with floating-point arithmetic while exercising its caches and the system's memory:

stress-ng --matrix 0 -t 1m --tz --times

The zero count syntax requests one stressor to run on each CPU in the system, --times generates statistics on userland and kernel time (Figure 2), and the --tz option collects CPU temperature data where available. It is also possible to collect detailed statistics with perf [5] by using the --perf option:

stress-ng --matrix 0 -t 1m --perf

Figure 2: Measuring the results of the best load to raise CPU temperature.

Figure 3 shows the resulting output. This type of analysis is more interesting when tuning code as a developer rather than as an operator, and attempting to improve cache performance and keeping page faults in check is of interest. The stress-ng tool is remarkably good at tailored cache stress plans – better than anything else I have seen to date.

Figure 3: Branch misses, page faults, cache misses, and even alignment faults are detailed.

Another interesting aspect is the possibility to load a stressor to a fixed share of CPU core, as opposed to maxing out a core with 100 percent load. For example,

stress-ng -t 1m -c 0 -l 40

will burden all cores in the system (-c 0) with a 40 percent load (Figure 4). Loading all cores to 40 percent is something not trivially accomplished without tooling. To torment the hardware, stress-ng can vary its approach within two abstractions: methods and classes. Methods are located within a given stressor and are executed round-robin as time permits. The command

stress-ng --cpu-method which

Figure 4: stress-ng brackets the target load when looking through top.

lists more than 50 methods to load a CPU. When consistency is desirable, you can select a given method individually (--cpu-method).

Classes are the second such facet. Stressors are grouped together in different classes according to the subsystems they load the most – CPU, memory, I/O, network, and so on. All tests in the class can be run in parallel or, perhaps more usefully, sequentially, as in this example,

stress-ng --class network --seq 0

which runs all network tests sequentially, one instance per CPU. One last consideration is that stress-ng is great at creating memory pressure. Try running the following command with swap turned off (swapoff -a) and watch the fireworks:

stress-ng --brk 2 --stack 2 --bigheap 2

Colin has presented his work at several Linux Foundation events, and slides [6] and recordings [7] are readily available to those who want to learn more about this great tool.