Assess USB performance while exploring storage caching
Plug and Play
The team here at the Dragon Propulsion Laboratory has kept busy building multiple Linux clusters as of late [1]. Some of the designs rely on spinning disks or SSD drives, whereas others use low-cost USB storage or even SD cards as boot media. In the process, I was hastily reminded of the limits of external storage media: not all flash is created equal, and in some crucial ways external drives, SD cards, and USB keys can be fundamentally different.
Turtles All the Way Down
Mass storage performance lags that of working memory in the Von Neumann architecture [2], with the need to persist data leading to the rise of caches at multiple levels in the memory hierarchy. An access speed gap three orders of magnitude between levels makes this design decision essentially inevitable where performance is at all a concern. (See Brendan Gregg's table of computer speed in human time [3].) The operating system itself provides the most visible manifestation of this design in Linux: Any RAM not allocated to a running program is used by the kernel to cache the reads from and buffer the writes to the storage subsystem [4], leading to the often repeated quip that there is really no such thing as "free memory" in a Linux system.
An easy way to observe the operating system (OS) buffering a write operation is to write the right amount of data to a disk in a system with lots of RAM, as shown in Figure 1, in which a rather improbable half a gigabyte worth of zeros is being written to a generic, low-cost USB key in half a second, but then experiences a 30-second delay when forcing the system to sync
[5] to disk. Most of the data was still residing in the kernel's page cache [6], waiting eventually to be persisted to disk. This resulted in an impressive (and impossible for the hardware) 885MBps transfer rate, but not to disk. The best way to address the flaw in my basic benchmark is to use the oflag=sync
setting of dd
[7], which forces synchronous data and metadata writes. A more likely result is shown in Figure 2.
Disk Caches
The OS is not the only player in the caching business. Examining a Samsung 750 SATA SSD drive, you cannot but notice a 256MB RAM buffer [8] stacked under the controller's packaging, a way to smooth over the flash's garbage collection pauses I examined in this summer's column [9]. This kind of cache is an advantage of spinning media and SSD drives over plain SD cards and USB keys.
The hdparm
command [10] includes a built-in benchmark that is aware of OS caching and buffering, conveniently comparing the results of the Samsung 750 device with OS caching on and off. Listing 1 shows the output of performing just such a comparison on the SSD drive. Initially, the amazing performance boost provided by the operating system's write buffering is showcased, elevating a physical USB 3 transfer rate of about 434MBps to a stunning 12.8GBps. The performance advantage of write buffering is of course balanced by the need to unmount devices before their physical removal or face inevitable file corruption. One further comparison is provided by disabling the disk's SATA write cache (-W
), dropping the physical rate 17% to 360MBps while the buffered rate remains substantially unchanged.
Listing 1
hdparm
federico@cybertron:~$ sudo hdparm -tT /dev/sdb /dev/sdb: Timing cached reads: 25662 MB in 1.99 seconds = 12879.79 MB/sec Timing buffered disk reads: 1306 MB in 3.00 seconds = 434.77 MB/sec federico@cybertron:~$ sudo hdparm -W /dev/sdb /dev/sdb: write-caching = 1 (on) federico@cybertron:~$ sudo hdparm -W 0 /dev/sdb /dev/sdb: write-caching = 0 (off) federico@cybertron:~$ sudo hdparm -tT /dev/sdb /dev/sdb: Timing cached reads: 25252 MB in 1.99 seconds = 12672.90 MB/sec Timing buffered disk reads: 1080 MB in 3.00 seconds = 359.86 MB/sec federico@cybertron:~$
Not All USB Keys Are Equal
In one cluster design in the lab, we repeatedly observed SATA timeouts during system startup while using low-cost USB keys for bootable storage. During boot, a significant amount of log writing takes place alongside all of the reading necessary to start the system, exceeding the cheaper device's ability to keep up. Figure 2 sequentially compares the cheap USB storage with more expensive devices from SanDisk (Cruzer Fit 32) and Samsung (Fit Plus 32), all trading places as device /dev/sdb
. Although all three devices have 32GB of storage, the Samsung drive is almost twice as fast as the cheap, unbranded disk. The SanDisk drive is connecting as a USB 2 device, so despite its high quality, it falls behind in the 10MB test because the other two devices connected in USB 3 SuperSpeed mode. (See the "Checking USB Speed" box.)
Checking USB Speed
Any device using a bus or network connection involving a speed negotiation may accidentally be handshaking at a rate other than is expected. A savvy performance engineer will always validate the connection rate early on, rather than assume it is correct until proven otherwise. The current USB specification provides three likely transfer rates, with version 1 of the protocol having fallen out of fashion and 3.2 hardware soon to add a fourth.
The speed of a device's USB connection can be checked with dmesg
[11] right after connecting – the log entry may include one of the rates [12] listed in the first column of Table 1. Even when that information is not included, finding the USB device's bus
and devnum
enables you to ask the system directly:
$ lsusb -v -s 2:6 | grep bcdUSB bcdUSB 3.10
Comparing the results with Table 1 shows that the cheap USB drive and the branded model are both operating at USB 3.1 speeds.
Another approach is to use the -t
option of lsusb
[13] to list devices in a tree view, including rates. You can easily identify the drive in Listing 2 by looking at the driver section, which identifies it as usb-storage
. It is worth noting that net speed may vary depending on which internal USB hub a device is attached to, if a system has more than one.
Table 1
USB Transfer Rates
Label | Speed (Mbps) | Version | Marketed Name |
---|---|---|---|
1.5M | 1.5 | 1 | Low speed |
12M | 12 | 1 | Full speed |
480M | 480 | 2 | High speed |
5000M | 5,000 | 3 | SuperSpeed |
10000M | 10,000 | 3.1 | SuperSpeed+ |
Listing 2
lsusb -t
federico@cybertron:~$ lsusb -t /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 5000M |__ Port 1: Dev 6, If 0, Class=Mass Storage, Driver=usb-storage, 5000M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/12p, 480M |__ Port 5: Dev 2, If 3, Class=Video, Driver=uvcvideo, 480M |__ Port 5: Dev 2, If 1, Class=Video, Driver=uvcvideo, 480M |__ Port 5: Dev 2, If 2, Class=Video, Driver=uvcvideo, 480M |__ Port 5: Dev 2, If 0, Class=Video, Driver=uvcvideo, 480M |__ Port 7: Dev 3, If 0, Class=Wireless, Driver=btusb, 12M |__ Port 7: Dev 3, If 1, Class=Wireless, Driver=btusb, 12M |__ Port 10: Dev 4, If 0, Class=Communications, Driver=cdc_acm, 12M |__ Port 10: Dev 4, If 1, Class=CDC Data, Driver=cdc_acm, 12M
I further compared the two SuperSpeed devices in Figures 3 and 4 with a much larger 1GB write test, showing a wider difference: The Samsung disk is delivering four times as high a write rate as the unbranded flash in this case. The "Gnome Disks" write-speed graph (jumpy red line on graph) pattern also shows how the garbage collection algorithm on the Samsung device intermittently recovers during a sustained write period, whereas the cheaper flash is overwhelmed and hoping for a break. Remarkably, the higher quality SanDisk device posts almost twice the write performance of the cheap USB key despite the handicap of lower USB bus performance, thanks to its superior garbage collection implementation (Figure 5).
Take note of how the tests used the physical USB device directly, ignoring the filesystem and in fact trashing it in the write tests. By operating this way, you can avoid considerations about filesystem type and partition alignment and are free to assess those variables separately. One last trick in the arsenal is to monitor activity in real time with iotop
[14] – try running it in a separate terminal while going over the other benchmarks (see Figure 6).
Infos
- "Low-Cost Linux Clusters" by Federico Lucifredi: https://f2.svbtle.com/low-cost-linux-clusters
- The Von Neumann architecture: https://en.wikipedia.org/wiki/Von_Neumann_architecture
- "The Infinite Space Between Words" by Jeff Atwood (table from Brendon Gregg's Systems performance Enterprise and the Cloud): https://blog.codinghorror.com/the-infinite-space-between-words/
- "Tuning Your Filesystem's Cache" by Federico Lucifredi, ADMIN , issue 7, 2012, pg. 81, http://www.admin-magazine.com/Articles/Tuning-Your-Filesystem-s-Cache
- sync(1) man page: https://linux.die.net/man/1/sync
- Cesati, Marco, and Daniel Bovet. "The Page Cache," chapter 15. In: Understanding the Linux Kernel , 3rd ed. O'Reilly Media, 2005, https://www.oreilly.com/library/view/understanding-the-linux/0596005652/ch15s01.html
- dd(1) man page: https://linux.die.net/man/1/dd
- "Samsung 750 EVO SSD Review" by Chris Ramseyer: https://www.tomshardware.com/reviews/samsung-750-evo-ssd,4467.html
- "Testing the Samsung MU-PA500B 500GB SSD" by Federico Lucifredi, ADMIN , issue 47, 2018, pg. 92, http://www.admin-magazine.com/Archive/2018/47/Testing-the-Samsung-MU-PA500B-500GB-SSD
- hdparm(1) man page: https://linux.die.net/man/8/hdparm
- dmesg(1) man page: https://linux.die.net/man/1/dmesg
- sysfs file attributes for USB devices and interfaces: https://github.com/torvalds/linux/blob/master/drivers/usb/core/sysfs.c
- lsusb(8) man page: https://linux.die.net/man/8/lsusb
- iotop(1) man page: https://linux.die.net/man/1/iotop
Buy this article as PDF
(incl. VAT)