Traffic Cops

Even if you have elaborate Elasticsearch-Logstash-Kibana (ELK)-based logging and metrics boasting gloriously detailed dashboards, if all you want is to SSH into a server and check which IP address is doing what over which interface, then the ELK stack is overkill. In this article, I demonstrate two tools that allow me to diagnose a misbehaving application on one occasion and odd routing behavior on another.

Sandwiches

The first tool I look at has changed its name recently from what to bandwhich and has had a bit of a facelift. A comprehensive README file [1] offers some insight into the tool written by Aram Drevekenin, who describes the tool as a "Terminal bandwidth utilization tool … formerly known as 'what'."

For this exercise, I used Linux Mint (based on Ubuntu 18.04) to see how the clever tool keeps a close eye on what the /proc pseudo filesystem is seeing, while producing useful, terminal-friendly output.

bandwhich is written in the Rust programming language [2]. To install a Rust package, you need the Rust package manager Cargo. However, if you use the apt install cargo command, you'll see that it needs a not-so-trivial 328MB of disk storage for Cargo and its libraries – just to be able to install bandwhich. My objection to the extra package installations comes from a security perspective. I just don't want unnecessary lines of code on my machines unless I need them. More package updates also means more admin time and a greater exposure to security issues.

The good news is that the author has kindly created some prebuilt binaries (which should be scanned for malware) to get around this problem, so I decided to take that route instead of installing Cargo.

From the Releases page [3] on the GitHub, repository you can see the latest build. In my case, that was version 0.9.0 at the time of testing. To proceed, I clicked the tar.gz link to download a minuscule 2MB file and then ran the command:

$ tar xvfz bandwhich-<version>-unknown-linux-musl.tar.gz

Having extracted the file, I can see that, uncompressed, it sits at only 6MB, which is still a far cry from 328MB that the Cargo route would have introduced to my laptop. To see whether the binary was compatible with my system, I ran as root:

$ ./bandwhich

On the right-hand side of Figure 1, you can see that my wireless network (wlp1s0) and localhost (127.0.0.1) interfaces both register traffic. The left-hand side lists any processes that are causing traffic; in this case, that is Google Chrome (chrome ). Both sides of the screen show Rate Up/Down columns.

Figure 1: Happiness is a binary that just works as expected.

If it looks like your bandwhich binary is working well, I'll now look at some of the options for customizing your setup. Hitting Ctrl+C returns you to the command-line prompt. If you don't want graphical output, you can get what the author describes as "machine friendly" output by simply adding the -r switch to the command for "raw" output (Listing 1):

$ ./bandwhich -r

128/196 process: "<UNKNOWN>"
connection: <1578840731> <lo>:55535 => 127.0.0.53:53 (udp) up/down Bps: 128/328 process: "<UNKNOWN>"
connection: <1578840731> <lo>:61233 => 127.0.0.53:53 (udp) up/down Bps: 128/128 process: "<UNKNOWN>"
remote_address: <1578840731> 127.0.0.53 up/down Bps: 3312/3934 connections: 16
process: <1578840732> "chrome" up/down Bps: 180/320 connections: 2
connection: <1578840732> <lo>:6225 => 127.0.0.53:53 (udp) up/down Bps: 114/114 process: "<UNKNOWN>"
connection: <1578840732> <lo>:21874 => 127.0.0.53:53 (udp) up/down Bps: 114/114 process: "<UNKNOWN>"
connection: <1578840732> <lo>:25497 => 127.0.0.53:53 (udp) up/down Bps: connection: <1578840732> <lo>:50008 => 127.0.0.53:53 (tcp) up/down Bps: 566/438 process: "<UNKNOWN>"
connection: <1578840732> <wlp1s0>:47741 => 192.168.1.2:1900 (udp) up/down Bps: 0/320 process: "chrome"
connection: <1578840732> <wlp1s0>:47741 => 239.255.255.250:1900 (udp) up/down Bps: 180/0 process: "chrome"

The output continues to scroll steadily up the terminal until you quit with Ctrl+C and provides a good level of detail relating to network connections.

Root Cause

Access to your network stack requires superuser permissions; however, you can, very carefully (because of security pitfalls), offer permanent privileges to the bandwhich binary so that becoming root is no longer necessary. If you get the error

$ ./bandwhich
Error: Insufficient permissions to listen on network interface(s). Try running with sudo.

fear not; I'll show how you can allow a non-root user access to the binary. If, for example, you move the bandwhich binary to your home directory, you can use (as root) the command

$ setcap cap_net_raw,cap_net_admin+ep "/home/chrisbinnie/bandwhich"

to open up permissions on a file located at /home/chrisbinnie/bandwhich. This command allows root-level permissions to the user. Note, however, that anybody with access to that binary inherits the potential to do bad things to your machine. Caution is therefore required. Now you should be able to execute the ./bandwhich command successfully (as user chrisbinnie in this case).

Before looking at the second network tool that I always turn to when in a jam, I'll offer a quick note about the -i and -n switches you can use with bandwhich. If you want to monitor the eth0 network interface instead of a WiFi interface, then use the -i switch. If you want to turn off DNS resolution (which on a router's interface could cause extra load), then just use the -n switch.

All in all, even with the slightly scary non-root user setcap trick, bandwhich is a nice-looking and handy tool. Having used it for a little while, I think if I had a feature request it would be to allow keystrokes to change some options while it was running.

if, then, fi

My favorite tool for watching traffic flow is iftop [4], which displays bandwidth usage on an interface by host. Without waxing lyrical for too long about this fantastic, lightweight tool, I have used it faithfully for longer than I care to remember: certainly somewhere in the region of 15 years. To my mind, barring packet sniffing, it covers all the requirements that a network monitor should (regex, reliability, detailed live updates in a lightweight package). What's more, if you can get the package installed on a server, it runs in a terminal without breaking a sweat. On Linux Mint and other Ubuntu flavors, you can install it with:

$ apt-get install iftop

Running the help command iftop -h shows the options available (Table 1). To begin, I'll take a look at the default options, how to create a custom configuration file, and how to change the configuration while it's live.

On busy systems I tend to turn off DNS lookups, but I usually leave them running on a laptop, for example. To affect that setting and just show IP addresses while iftop is running, simply press n .

I also like seeing port numbers, which requires pressing the lowercase p key while iftop is running – not to be confused with the P key, which is extremely useful when you've spotted something you need to record and want to pause the display.

Additionally, I prefer adjusting the t key options, which is useful for seeing bidirectional traffic (shown as <=> ) on one line. If you toggle the t option again you'll see received traffic only; hit that key once more and, you guessed correctly, you'll only see traffic that's been sent from your local machine's network interface. Another press of the powerful t option shows the default two lines of traffic per connection (Figure 2), so you can categorically see how much traffic is ingress and how much is egress, and not the aggregated amount on one line.

Figure 2: The mighty iftop, slightly tweaked to my preferences, ports and all.

If you run iftop from the command line, the options shown in Table 1 apply, but you can also add lots of filters while it's running live. For example, to set the refresh rate to the fastest two seconds at execution time on my wireless interface wlp1s0 enter:

$ iftop -i wlp1s0 -o2s

If you're familiar with ngrep or tcpdump, the language for a filter option that can be toggled while iftop is running will be familiar, because you can use all kinds of regex and network options. Pressing the f key brings up the Net filter prompt, from which you can focus on specific networks, ports, or hosts.

For example, to see traffic from a specific host only, use the dst host filter:

Net filter> dst host 10.1.1.10

Be aware that it's apparently better for stability to use the config file to choose these filters, if possible, which I'll talk about in a moment.

In addition to filter option dst (destination), you can use src (source) and the self-described port, net, and host filters. Getting familiar with how traffic displays when you use different options is well worth the time, because it will almost certainly get you out of a bind at some point in the future.

For more filter options, you can check out the iftop man page (man iftop).