Cursed Monitor

The venerable curses [1] library first enabled plotting graphics on a Unix terminal, allowing the emergence of dynamic, interactive tools such as top [2] and its brood of descendants to improve greatly the ability to monitor a system's state continuously. The transition from static tables of numbers to charts and sometimes even dynamic data representations was followed by new implementations of terminal user interface (TUI) libraries, including Ncurses [3] and Newt [4]. Jeff Layton has previously graced these pages with a tutorial on how TUI tools are developed today [5], and yours truly followed in the next issue with a survey of network monitoring tools adopting in-terminal graphics [6]. Two years later, I revisit the subject to witness the impressive improvement of the state of the art taking place since.

Not Your Father's Curses

The bashtop [7] utility is one of the most remarkable new entrants – so much so it has been featured by the likes of Forbes magazine [8], hardly what I would call conventional treatment for a Linux utility. Originally written in Bash (hence its name), it was re-implemented in Python [9] last year, reducing its CPU consumption by two-thirds and simultaneously improving speed and extending functionality. A third rewrite, this time in C++, is in progress at the time of this writing. The performance enhancements are welcome for administrators who leave the tool running in the background, but all versions of the tool are equally suitable for intermittent use – just keep in mind that newer ports provide additional functionality not found in the original; for example, mouse support absent in the Bash version has been added with the Python port.

Bashtop (Figure 1) supports both Linux and macOS (when used with iTerm2 [10]) and is controlled from the keyboard. (Menu entries are keyed by letters highlighted in grey.) The Python port is available as part of the Ubuntu standard repositories [11] starting with Ubuntu 20.10 (Groovy Gorilla), whereas the original Bash implementation is easily built on stock 20.04 (Focal Fossa) from the author's repository with sudo make install.

Figure 1: Bashtop makes good use of terminals wider than 80 columns, dynamically adjusting its layout.

In its default configuration, Bashtop provides a ranked process listing including memory and CPU usage, as well as system-wide aggregate status for CPU, memory, permanent storage, and network connections. The configuration screen features a retro arcade game look, speaking to the author's attention to detail (Figure 2).

Figure 2: The 1980s coin-op arcade vibe of Bashtop's configuration screen.

A more practical nod to UX is the fading out of process listing entries as you get farther from the top, and the increased brightness of entries as they approach 100 percent. Although the tool can be used to monitor systems with many disks or network connections, the most striking visualization is reserved for CPU load history (Figure 3). It presents a history ribbon view of the CPU, breaks out individual cores, and includes load average [12] figures. Network interfaces benefit from a similar visualization (Figure 4), with the Python implementation bpytop extending the treatment to memory use, as well (Figure 5).

Figure 3: Detail of the CPU ribbon display, detailing aggregate and per-core histories.

Figure 4: Network traffic history can span multiple interfaces (if present).

Figure 5: Memory use history is a great addition found in the Python port.

At a Glance

The older glances [13] tool might not push the limits of TUI graphics as aggressively as its newer challenger, but it more than makes it up in portability and extensibility. Supporting Linux, Solaris, *BSD, macOS, and even Windows, glances is my go-to system monitoring: I quite simply install it everywhere (Figure 6). Take note to use binaries where a build is available, because like many other highly portable applications, a build of Glances can take quite some time to complete on account of its many dependencies. On macOS, I use brew install glances as a stress generator for new hosts, akin to running a kernel build on Linux. Glances can also provide remote monitoring when needed (both as a web application and in client-server mode), although I choose SSH when the occasion calls for remote access.

Figure 6: Glances employs a simple but effective bar graph load visualization style.

Alongside processes listing and network, disk, and CPU system metrics, a glances summary screen can display sensor data, notably for CPU and disk temperatures, which is critical information for servers. Sensor instrumentation can include other plugins, such as battery charge levels on laptops. A unique capability of glances is its built-in container monitoring, putting Docker containers on equal footing with Linux processes, making it very interesting in modern environments with containerized applications (Figure 7). Glances itself is available as a Docker image with the one-liner docker pull nicolargo/glances.

Figure 7: Glances can integrate with Docker's API to include a container view.

Network Watch

The venerable ping [14] tool has also received a TUI facelift – two in fact. The prettyping [15] Bash and Awk wrapper for the original tool requires no installation and no special permissions but greatly enhances the visibility of latency variance or the appearance of spurious errors while tracking aggregate statistics for the last 60 packets (Figure 8). On packet loss, prettyping breaks to a new line and highlights latency changes with a predefined color scale.

Figure 8: prettyping colorizes ping output to highlight packet loss and latency spikes.

Less apt at highlighting failures, but impressive at charting change in latency, is the newfangled gping [16], which charts latency in what is perhaps the best dynamically resizing in-terminal graph to date (Figure 9). The gping tool pushes the limits of in-terminal graphs, and it is perfectly suited to examining continuous changes in network performance, as opposed to highlighting spurious events.

Figure 9: gping charting the performance of a network connection.