Small embedded devices running Linux often face a trade-off between limited resources in CPU and RAM and the abysmal write performance of SD card storage with abundant capacity. The first rule, as I discussed back in 2018 [1] is to make exclusive use of high-quality SD cards for boot devices – a class 10 device or better for starters – with a reputable brand name printed on it. (I recommend SanDisk or Samsung SD cards.) Lower quality SD cards simply cannot satisfy the write pressure of system boot or other peak logging moments of operation, and while suitable for data, they cannot be used as boot devices. These are the basic table stakes, but this being Linux, you can of course optimize further!

Test Bench

I am going to discuss a specific board for this column to try out the theories in practice. I will be using a Marvell ESPRESSObin [2] v7 (Figure 1), a low-cost, low-power board made by Globalscale Technologies. The ESPRESSObin was one of the earliest single-board computers (SBCs) to sport a 64-bit ARM CPU and remains remarkable today for its inclusion of a SATA port and PCI expansion. The full spec of the board is interesting in its own right (Table 1), but here I will focus on the setup for my tests. I use the external 12V power supply provided and the USB interface for serial access.

Figure 1: ESPRESSObin v7 SBC shown in the 2GB of RAM and no embedded MultiMediaCard (eMMC) variety.

On the client side, I employ the handy Serial [3] Mac program, which includes userspace implementations of driver stacks for the two common USB-to-serial chipsets, eliminating the hassle of installing or updating drivers on multiple computers (and the reboots that go with those Mach kernel modules). The connection runs at 115,200 baud, with no flow control and no parity.

The board bootstraps with U-Boot [4], and from there it loads the operating system. Because the board was first released in 2017, the natural places to build current OS images are the Yocto project [5] and Armbian [6], a community-managed distribution that builds board-specific images of Ubuntu and Debian. I chose to use the stable Armbian build of Debian Stretch for the ESPRESSObin, and the results are shown in Figure 2.

Figure 2: Armbian bootup screen on ESPRESSObin v7 right after login.

The Right Trade-off

On systems with lots of storage and not enough RAM, one is naturally inclined to think about turning some of the permanent storage into swap space [7]. Unfortunately, the low write performance of SD card storage precludes this avenue to most embedded SBCs. The sensible strategy here is to turn some of the RAM into a write-friendly device and to stretch the amount of RAM allocated with the use of inline data compression for that partition.

I chose Armbian partly because it is a distribution that turns to this strategy by default. Figure 3 shows Armbian using a zram [8] device to store /var/log, offloading the primary source of boot-time writes from the physical device to a 50MB RAM drive.

Figure 3: Three zram partitions were found in my system: two for Linux swap and one for logs.

Zram, introduced by kernel 3.14 and considered stable, creates a block device in RAM and compresses writes to it on the fly. Default zram configurations on Debian and Ubuntu (controlled by the zram-config package) also create one swap device per CPU core, as seen in Figure 3. The increase in CPU load is counterbalanced by the availability of more memory, as existing RAM is stretched with the use of compression.

Starting with kernel 4.14, zram can be configured as a write-back cache committing data to permanent storage in the background. Until then, this was commonly accomplished with zswap [9], a lightweight memory page compression driver that operates exclusively as a writeback cache for swap.

Swap Usage

Some important workloads, most notably Kubernetes, are openly averse to swap, preferring to address resource limits early rather than accept performance degradation and inconsistency. At the edge or in the Internet of Things (IOT), you have to work with the hardware you have and compromises are more likely. Either way, the free [10] command provides a simple avenue to check swap use (Listing 1). Currently the system described is not using swap.

root@espressobin:~# free
              total        used        free      shared  buff/cache   available
Mem:        2046088       82796     1834332        5548      128960     1891416
Swap:       1023040           0     1023040
root@espressobin:~#