Lead Image © Lucy Baldwin, 123RF.com

Lead Image © Lucy Baldwin, 123RF.com

Assess USB performance while exploring storage caching

Plug and Play

Article from ADMIN 48/2018
By
Use dd fluently to compare disk performance.

The team here at the Dragon Propulsion Laboratory has kept busy building multiple Linux clusters as of late [1]. Some of the designs rely on spinning disks or SSD drives, whereas others use low-cost USB storage or even SD cards as boot media. In the process, I was hastily reminded of the limits of external storage media: not all flash is created equal, and in some crucial ways external drives, SD cards, and USB keys can be fundamentally different.

Turtles All the Way Down

Mass storage performance lags that of working memory in the Von Neumann architecture [2], with the need to persist data leading to the rise of caches at multiple levels in the memory hierarchy. An access speed gap three orders of magnitude between levels makes this design decision essentially inevitable where performance is at all a concern. (See Brendan Gregg's table of computer speed in human time [3].) The operating system itself provides the most visible manifestation of this design in Linux: Any RAM not allocated to a running program is used by the kernel to cache the reads from and buffer the writes to the storage subsystem [4], leading to the often repeated quip that there is really no such thing as "free memory" in a Linux system.

An easy way to observe the operating system (OS) buffering a write operation is to write the right amount of data to a disk in a system with lots of RAM, as shown in Figure 1, in which a rather improbable half a gigabyte worth of zeros is being written to a generic, low-cost USB key in half a second, but then experiences a 30-second delay when forcing the system to sync [5] to disk. Most of the data was still residing in the kernel's page cache [6], waiting eventually to be persisted to disk. This resulted in an impressive (and impossible for the hardware) 885MBps transfer rate, but not to disk. The best way to address the flaw in my basic benchmark is to use the oflag=sync setting of dd [7], which forces synchronous data and metadata writes. A more likely result is shown in Figure 2.

Figure 1: Naive dd test showcasing OS write buffering.
Figure 2: Sequentially testing synchronous writes to a low-cost USB key, a SanDisk Cruzer Fit 32 (USB 2), and a Samsung Fit Plus 32 (USB 3).

Disk Caches

The OS is not the only player in the caching business. Examining a Samsung 750 SATA SSD drive, you cannot but notice a 256MB RAM buffer [8] stacked under the controller's packaging, a way to smooth over the flash's garbage collection pauses I examined in this summer's column [9]. This kind of cache is an advantage of spinning media and SSD drives over plain SD cards and USB keys.

The hdparm command [10] includes a built-in benchmark that is aware of OS caching and buffering, conveniently comparing the results of the Samsung 750 device with OS caching on and off. Listing 1 shows the output of performing just such a comparison on the SSD drive. Initially, the amazing performance boost provided by the operating system's write buffering is showcased, elevating a physical USB 3 transfer rate of about 434MBps to a stunning 12.8GBps. The performance advantage of write buffering is of course balanced by the need to unmount devices before their physical removal or face inevitable file corruption. One further comparison is provided by disabling the disk's SATA write cache (-W), dropping the physical rate 17% to 360MBps while the buffered rate remains substantially unchanged.

Listing 1

hdparm

federico@cybertron:~$ sudo hdparm -tT /dev/sdb
/dev/sdb:
 Timing cached reads:   25662 MB in  1.99 seconds = 12879.79 MB/sec
 Timing buffered disk reads: 1306 MB in  3.00 seconds = 434.77 MB/sec
federico@cybertron:~$ sudo hdparm -W /dev/sdb
/dev/sdb:
 write-caching =  1 (on)
federico@cybertron:~$ sudo hdparm -W 0 /dev/sdb
/dev/sdb:
 write-caching =  0 (off)
federico@cybertron:~$ sudo hdparm -tT /dev/sdb
/dev/sdb:
 Timing cached reads:   25252 MB in  1.99 seconds = 12672.90 MB/sec
 Timing buffered disk reads: 1080 MB in  3.00 seconds = 359.86 MB/sec
federico@cybertron:~$

Not All USB Keys Are Equal

In one cluster design in the lab, we repeatedly observed SATA timeouts during system startup while using low-cost USB keys for bootable storage. During boot, a significant amount of log writing takes place alongside all of the reading necessary to start the system, exceeding the cheaper device's ability to keep up. Figure 2 sequentially compares the cheap USB storage with more expensive devices from SanDisk (Cruzer Fit 32) and Samsung (Fit Plus 32), all trading places as device /dev/sdb. Although all three devices have 32GB of storage, the Samsung drive is almost twice as fast as the cheap, unbranded disk. The SanDisk drive is connecting as a USB 2 device, so despite its high quality, it falls behind in the 10MB test because the other two devices connected in USB 3 SuperSpeed mode. (See the "Checking USB Speed" box.)

Checking USB Speed

Any device using a bus or network connection involving a speed negotiation may accidentally be handshaking at a rate other than is expected. A savvy performance engineer will always validate the connection rate early on, rather than assume it is correct until proven otherwise. The current USB specification provides three likely transfer rates, with version 1 of the protocol having fallen out of fashion and 3.2 hardware soon to add a fourth.

The speed of a device's USB connection can be checked with dmesg [11] right after connecting – the log entry may include one of the rates [12] listed in the first column of Table 1. Even when that information is not included, finding the USB device's bus and devnum enables you to ask the system directly:

$ lsusb -v -s 2:6 | grep bcdUSB
  bcdUSB               3.10

Comparing the results with Table 1 shows that the cheap USB drive and the branded model are both operating at USB 3.1 speeds.

Another approach is to use the -t option of lsusb [13] to list devices in a tree view, including rates. You can easily identify the drive in Listing 2 by looking at the driver section, which identifies it as usb-storage . It is worth noting that net speed may vary depending on which internal USB hub a device is attached to, if a system has more than one.

Table 1

USB Transfer Rates

Label Speed (Mbps) Version Marketed Name
1.5M 1.5 1 Low speed
12M 12 1 Full speed
480M 480 2 High speed
5000M 5,000 3 SuperSpeed
10000M 10,000 3.1 SuperSpeed+

Listing 2

lsusb -t

federico@cybertron:~$ lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 5000M
    |__ Port 1: Dev 6, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/12p, 480M
    |__ Port 5: Dev 2, If 3, Class=Video, Driver=uvcvideo, 480M
    |__ Port 5: Dev 2, If 1, Class=Video, Driver=uvcvideo, 480M
    |__ Port 5: Dev 2, If 2, Class=Video, Driver=uvcvideo, 480M
    |__ Port 5: Dev 2, If 0, Class=Video, Driver=uvcvideo, 480M
    |__ Port 7: Dev 3, If 0, Class=Wireless, Driver=btusb, 12M
    |__ Port 7: Dev 3, If 1, Class=Wireless, Driver=btusb, 12M
    |__ Port 10: Dev 4, If 0, Class=Communications, Driver=cdc_acm, 12M
    |__ Port 10: Dev 4, If 1, Class=CDC Data, Driver=cdc_acm, 12M

I further compared the two SuperSpeed devices in Figures 3 and 4 with a much larger 1GB write test, showing a wider difference: The Samsung disk is delivering four times as high a write rate as the unbranded flash in this case. The "Gnome Disks" write-speed graph (jumpy red line on graph) pattern also shows how the garbage collection algorithm on the Samsung device intermittently recovers during a sustained write period, whereas the cheaper flash is overwhelmed and hoping for a break. Remarkably, the higher quality SanDisk device posts almost twice the write performance of the cheap USB key despite the handicap of lower USB bus performance, thanks to its superior garbage collection implementation (Figure 5).

Figure 3: Gnome Disks benchmark (1,000, 10MiB samples) running on the low-cost USB key. Note garbage collection slowdown.
Figure 4: Gnome Disks benchmark running on a Samsung Fit Plus 32 USB key. Note garbage collection recovery.
Figure 5: Gnome Disks benchmark running on a SanDisk Cruzer Fit 32 USB key. This older device is connecting at USB 2 speeds.

Take note of how the tests used the physical USB device directly, ignoring the filesystem and in fact trashing it in the write tests. By operating this way, you can avoid considerations about filesystem type and partition alignment and are free to assess those variables separately. One last trick in the arsenal is to monitor activity in real time with iotop [14] – try running it in a separate terminal while going over the other benchmarks (see Figure 6).

Figure 6: iotop showing real-time input/output operations per second (IOPS) load on the system during the dd tests.

Infos

  1. "Low-Cost Linux Clusters" by Federico Lucifredi: https://f2.svbtle.com/low-cost-linux-clusters
  2. The Von Neumann architecture: https://en.wikipedia.org/wiki/Von_Neumann_architecture
  3. "The Infinite Space Between Words" by Jeff Atwood (table from Brendon Gregg's Systems performance Enterprise and the Cloud): https://blog.codinghorror.com/the-infinite-space-between-words/
  4. "Tuning Your Filesystem's Cache" by Federico Lucifredi, ADMIN , issue 7, 2012, pg. 81, http://www.admin-magazine.com/Articles/Tuning-Your-Filesystem-s-Cache
  5. sync(1) man page: https://linux.die.net/man/1/sync
  6. Cesati, Marco, and Daniel Bovet. "The Page Cache," chapter 15. In: Understanding the Linux Kernel , 3rd ed. O'Reilly Media, 2005, https://www.oreilly.com/library/view/understanding-the-linux/0596005652/ch15s01.html
  7. dd(1) man page: https://linux.die.net/man/1/dd
  8. "Samsung 750 EVO SSD Review" by Chris Ramseyer: https://www.tomshardware.com/reviews/samsung-750-evo-ssd,4467.html
  9. "Testing the Samsung MU-PA500B 500GB SSD" by Federico Lucifredi, ADMIN , issue 47, 2018, pg. 92, http://www.admin-magazine.com/Archive/2018/47/Testing-the-Samsung-MU-PA500B-500GB-SSD
  10. hdparm(1) man page: https://linux.die.net/man/8/hdparm
  11. dmesg(1) man page: https://linux.die.net/man/1/dmesg
  12. sysfs file attributes for USB devices and interfaces: https://github.com/torvalds/linux/blob/master/drivers/usb/core/sysfs.c
  13. lsusb(8) man page: https://linux.die.net/man/8/lsusb
  14. iotop(1) man page: https://linux.die.net/man/1/iotop

The Author

Federico Lucifredi is the Product Management Director for Ceph Storage at Red Hat, formerly the Ubuntu Server PM at Canonical, and the Linux "Systems Management Czar" at SUSE. He enjoys arcane hardware issues and shell scripting mysteries, and he takes his McFlurry shaken, not stirred. You can read more from him in the new O'Reilly title AWS System Administration .

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy ADMIN Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

comments powered by Disqus