The team here at the Dragon Propulsion Laboratory has kept busy building multiple Linux clusters as of late [1]. Some of the designs rely on spinning disks or SSD drives, whereas others use low-cost USB storage or even SD cards as boot media. In the process, I was hastily reminded of the limits of external storage media: not all flash is created equal, and in some crucial ways external drives, SD cards, and USB keys can be fundamentally different.

Turtles All the Way Down

Mass storage performance lags that of working memory in the Von Neumann architecture [2], with the need to persist data leading to the rise of caches at multiple levels in the memory hierarchy. An access speed gap three orders of magnitude between levels makes this design decision essentially inevitable where performance is at all a concern. (See Brendan Gregg's table of computer speed in human time [3].) The operating system itself provides the most visible manifestation of this design in Linux: Any RAM not allocated to a running program is used by the kernel to cache the reads from and buffer the writes to the storage subsystem [4], leading to the often repeated quip that there is really no such thing as "free memory" in a Linux system.

An easy way to observe the operating system (OS) buffering a write operation is to write the right amount of data to a disk in a system with lots of RAM, as shown in Figure 1, in which a rather improbable half a gigabyte worth of zeros is being written to a generic, low-cost USB key in half a second, but then experiences a 30-second delay when forcing the system to sync [5] to disk. Most of the data was still residing in the kernel's page cache [6], waiting eventually to be persisted to disk. This resulted in an impressive (and impossible for the hardware) 885MBps transfer rate, but not to disk. The best way to address the flaw in my basic benchmark is to use the oflag=sync setting of dd [7], which forces synchronous data and metadata writes. A more likely result is shown in Figure 2.

Figure 1: Naive dd test showcasing OS write buffering.

Figure 2: Sequentially testing synchronous writes to a low-cost USB key, a SanDisk Cruzer Fit 32 (USB 2), and a Samsung Fit Plus 32 (USB 3).

Disk Caches

The OS is not the only player in the caching business. Examining a Samsung 750 SATA SSD drive, you cannot but notice a 256MB RAM buffer [8] stacked under the controller's packaging, a way to smooth over the flash's garbage collection pauses I examined in this summer's column [9]. This kind of cache is an advantage of spinning media and SSD drives over plain SD cards and USB keys.

The hdparm command [10] includes a built-in benchmark that is aware of OS caching and buffering, conveniently comparing the results of the Samsung 750 device with OS caching on and off. Listing 1 shows the output of performing just such a comparison on the SSD drive. Initially, the amazing performance boost provided by the operating system's write buffering is showcased, elevating a physical USB 3 transfer rate of about 434MBps to a stunning 12.8GBps. The performance advantage of write buffering is of course balanced by the need to unmount devices before their physical removal or face inevitable file corruption. One further comparison is provided by disabling the disk's SATA write cache (-W), dropping the physical rate 17% to 360MBps while the buffered rate remains substantially unchanged.

federico@cybertron:~$ sudo hdparm -tT /dev/sdb
/dev/sdb:
 Timing cached reads:   25662 MB in  1.99 seconds = 12879.79 MB/sec
 Timing buffered disk reads: 1306 MB in  3.00 seconds = 434.77 MB/sec
federico@cybertron:~$ sudo hdparm -W /dev/sdb
/dev/sdb:
 write-caching =  1 (on)
federico@cybertron:~$ sudo hdparm -W 0 /dev/sdb
/dev/sdb:
 write-caching =  0 (off)
federico@cybertron:~$ sudo hdparm -tT /dev/sdb
/dev/sdb:
 Timing cached reads:   25252 MB in  1.99 seconds = 12672.90 MB/sec
 Timing buffered disk reads: 1080 MB in  3.00 seconds = 359.86 MB/sec
federico@cybertron:~$

Not All USB Keys Are Equal

In one cluster design in the lab, we repeatedly observed SATA timeouts during system startup while using low-cost USB keys for bootable storage. During boot, a significant amount of log writing takes place alongside all of the reading necessary to start the system, exceeding the cheaper device's ability to keep up. Figure 2 sequentially compares the cheap USB storage with more expensive devices from SanDisk (Cruzer Fit 32) and Samsung (Fit Plus 32), all trading places as device /dev/sdb. Although all three devices have 32GB of storage, the Samsung drive is almost twice as fast as the cheap, unbranded disk. The SanDisk drive is connecting as a USB 2 device, so despite its high quality, it falls behind in the 10MB test because the other two devices connected in USB 3 SuperSpeed mode. (See the "Checking USB Speed" box.)

$ lsusb -v -s 2:6 | grep bcdUSB
  bcdUSB               3.10

federico@cybertron:~$ lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 5000M
    |__ Port 1: Dev 6, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/12p, 480M
    |__ Port 5: Dev 2, If 3, Class=Video, Driver=uvcvideo, 480M
    |__ Port 5: Dev 2, If 1, Class=Video, Driver=uvcvideo, 480M
    |__ Port 5: Dev 2, If 2, Class=Video, Driver=uvcvideo, 480M
    |__ Port 5: Dev 2, If 0, Class=Video, Driver=uvcvideo, 480M
    |__ Port 7: Dev 3, If 0, Class=Wireless, Driver=btusb, 12M
    |__ Port 7: Dev 3, If 1, Class=Wireless, Driver=btusb, 12M
    |__ Port 10: Dev 4, If 0, Class=Communications, Driver=cdc_acm, 12M
    |__ Port 10: Dev 4, If 1, Class=CDC Data, Driver=cdc_acm, 12M

I further compared the two SuperSpeed devices in Figures 3 and 4 with a much larger 1GB write test, showing a wider difference: The Samsung disk is delivering four times as high a write rate as the unbranded flash in this case. The "Gnome Disks" write-speed graph (jumpy red line on graph) pattern also shows how the garbage collection algorithm on the Samsung device intermittently recovers during a sustained write period, whereas the cheaper flash is overwhelmed and hoping for a break. Remarkably, the higher quality SanDisk device posts almost twice the write performance of the cheap USB key despite the handicap of lower USB bus performance, thanks to its superior garbage collection implementation (Figure 5).

Figure 3: Gnome Disks benchmark (1,000, 10MiB samples) running on the low-cost USB key. Note garbage collection slowdown.

Figure 4: Gnome Disks benchmark running on a Samsung Fit Plus 32 USB key. Note garbage collection recovery.

Figure 5: Gnome Disks benchmark running on a SanDisk Cruzer Fit 32 USB key. This older device is connecting at USB 2 speeds.

Take note of how the tests used the physical USB device directly, ignoring the filesystem and in fact trashing it in the write tests. By operating this way, you can avoid considerations about filesystem type and partition alignment and are free to assess those variables separately. One last trick in the arsenal is to monitor activity in real time with iotop [14] – try running it in a separate terminal while going over the other benchmarks (see Figure 6).

Figure 6: iotop showing real-time input/output operations per second (IOPS) load on the system during the dd tests.