Lead Image © Lucy Baldwin, 123RF.com

Lead Image © Lucy Baldwin, 123RF.com

Logical Bombs for Fun and Benchmarking

Explosive Code

Article from ADMIN 53/2019
By
The admin can generate load a number of ways with shell commands.

In this column, I often make use of the stress [1] utility as a convenient way to generate load on a system's memory, CPU, or storage subsystem. Although not the most sophisticated of tools, load generators like stress are simple to use and effective, because they provide a convenient way to load a certain number of CPU cores or to fill a predefined amount of RAM and do so in a manner that can be reproduced with consistency. However, you can find other minimalist ways to generate load easily with shell commands. For example, with

$ yes > /dev/null &

the yes [2] command repeatedly outputs a string until killed and would normally be bound by the I/O speed of the terminal. Because the output is redirected into the oblivion that is /dev/null before it ever reaches the screen buffers, each such invocation is essentially a pure processor workload that will maximally use up to one CPU core while taking up close to zero I/O or memory resources.

The top [3] command displays a perfect 1.00 load average after one minute (Figure 1). Amazon CloudWatch [4] data for this test instance more slowly converges on 100% CPU, consistent with the single vCPU configuration of a t2.micro instance (Figure 2).

Figure 1: The one-minute load average quickly reaches a perfect 1.00 value.
Figure 2: CloudWatch data is also showing 100% CPU load over its longer time base.

Fork Bomb

The fork [5] command, perhaps more than any other system call, is distinctive of Unix system design. The call enters kernel space in one process, but returns in two, the original process having been duplicated into a copy. The only internal distinction between the two processes is the return value of the call itself, enabling code to distinguish the parent and child processes.

As many computer science students first learn accidentally in their systems programming class, fork and its variants can also be used to stress a system by generating a very large number of processes. A fork bomb [6] spawns so many processes so quickly that it often results in a denial-of-service attack against the machine it is running on. Once a fork bomb has been launched, it might actually be impossible to recover interactive control of the system to kill all of these processes, forcing the operator to reboot. (See the "Stopping Fork Bombs" box.)

Stopping Fork Bombs

The effect of a fork bomb can be mitigated by limiting the number of processes a user is allowed to start on the system. The ulimit [7] command exposes the maximum number of processes available to a single user. To test my Ubuntu 18.04 stock instance, I enter:

$ ulimit -u
3841

The limit is set with the -S option:

$ ulimit -S -u 500

Once the limit is reached, the system will generate an error. Setting too low a limit will not only inconvenience users, but also render the system unable to start. Testing with the Bash fork bomb produces this output:

bash: fork: Resource temporarily unavailable
bash: fork: Resource temporarily unavailable

The restriction can be made permanent by configuring the nproc value in /etc/security/limits.conf [8].

The shell provides a straightforward way to initiate a fork bomb, in the form of a short but cryptic Bash function:

:(){ :|:& };:

This shell scripting one-liner defines and launches a recursive Bash function named : that does nothing but execute itself in the background (twice!). This charming piece of code is best not executed on your computer, as it will likely crash it. Following that advice, I used a virtual instance on Amazon EC2 to carry out the experiment. Because the terminal session immediately froze, I used CloudWatch again to observe the CPU load shoot up (Figure 3).

Figure 3: The fork bomb not only maxes out CPU, but also consumes all available RAM in the process.

RAM does not fare any better, because free memory is the limiting factor to the creation of more processes. Recovering the instance could be performed by killing all processes belonging to the user, if you could manage to log in. Fortunately, the system is watching for this situation, after a fashion.

Return of the OOM Killer

In a previous issue, I detailed the design of the Linux kernel's OOM logic [9] that went mainstream when Ubuntu 12.04 shipped version 3.2 of the Linux kernel. Processes are assigned a badness score [10], primarily based on their memory footprint, that is combined by the OOM killer with minimally configurable heuristics (enabling operators to designate preferred victims should the system run out of memory). The OOM killer's current design is not well-suited to this situation: The fork bomb is launching a lot of processes (Figure 4), each with a small memory footprint not standing out on its own. Even so, eventually the OOM killer gets its PID (Figure 5).

Figure 4: A bash fork bomb generates a lot of processes, each with a small memory footprint.
Figure 5: /var/log/kern.log recorded the reaper process firing three times.

It did take a few tries and almost four hours before the instance returned to normal operation (Figure 6). Sometimes, you just have to get lucky! Third time is the charm, and with the third kill the system fully righted itself. Older OOM killer implementations accounted for the memory footprint of a process' spawned children (or a fraction of it), mitigating fork bombs explicitly [11], but as I read through the code, that does not appear to be the case any longer.

Figure 6: The OOM killer to the rescue, restoring the system to normality without a reboot.

Infos

  1. stress (1) man page: https://manpages.ubuntu.com/manpages/bionic/en/man1/stress.1.html
  2. yes (1) man page: https://manpages.ubuntu.com/manpages/bionic/en/man1/yes.1.html
  3. top (1) man page: https://manpages.ubuntu.com/manpages/bionic/en/man1/top.1.html
  4. Amazon CloudWatch: https://aws.amazon.com/cloudwatch/
  5. fork (2) man page: https://manpages.ubuntu.com/manpages/bionic/en/man2/fork.2.html
  6. Fork bomb: https://en.wikipedia.org/wiki/Fork_bomb
  7. bash (1) man page: https://manpages.ubuntu.com/manpages/bionic/en/man1/bash.1.html
  8. Setting nproc to limit maximum number of processes for a user: https://red.ht/325Byl5
  9. "Tune kernel overcommit behavior to avoid the dread OOM Killer" by Federico Lucifredi, ADMIN , issue 10, pg. 94
  10. oom_kill.c: https://elixir.bootlin.com/linux/v4.15/source/mm/oom_kill.c
  11. Linux memory management: https://linux-mm.org/OOM_Killer

The Author

Federico Lucifredi (@0xf2) is the Product Management Director for Ceph Storage at Red Hat and was formerly the Ubuntu Server Project Manager at Canonical and the Linux "Systems Management Czar" at SUSE. He enjoys arcane hardware issues and shell-scripting mysteries and takes his McFlurry shaken, not stirred. You can read more from him in the new O'Reilly title AWS System Administration .

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy ADMIN Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

comments powered by Disqus