We look at how to label and partition proprietary and Linux storage servers for high-performance computing.
Proprietary and Open Source Storage Options
How to manage storage is a very popular question with a simple, single answer: It depends. Let me explain. Because of the many aspects of managing storage – from design, to monitoring, to user management, to troubleshooting – defining what “manage” means is virtually impossible. Moreover, with the different types of storage from various vendors and open source origins, the combination of these intersecting aspects creates literately infinite possibilities, so it’s impossible to define a single way to manage storage. Therefore, you must address the question as to how to manage storage with the further question: “What are your goals?” In answering that question, you can start to develop how you will manage your storage.
Defining your goals requires a good examination of performance, capacity, growth, monitoring, (of course) management, and any other aspects for the targeted systems. To reduce the discussion to something more manageable, I split the discussion into two parts: proprietary storage solutions and Linux storage solutions that you either build yourself or specify to vendors that build with open source Linux tools. I’m not advocating one over the other; I’m just trying to create a reasonable narrative.
This discussion about Linux storage solutions is longer than you think, even when discussing only high-level concepts. Consequently, I’m creating a series of articles. The first part, this article, starts with a quick overview of proprietary and open source solutions, starting with the beginning of Linux storage device solutions (open source) and ending with partitioning of these devices.
Proprietary Storage Solutions
Proprietary storage solutions will always come with a set of tools for managing and monitoring their storage solution. Most of the time these will be tools that the storage vendor has developed. Be sure to use those tools because if you have any problems, the vendor will likely want information you glean with the tools developed for their storage solution, so they can quickly look through the data to solve any problems.
Consequently, I won’t discuss proprietary software here. I encourage you to talk directly to the source, but only after you do one thing: Take the time to create a stack rank list of the most important aspects of storage for your system. Know where you are willing to trade priorities in that list. For example, if you can get much less expensive storage ($/GB) in exchange for performance, would you be willing to take it?
Coupled with a stack rank list of priorities, you should also know what tests or benchmarks best represent the applications on your system. Try to find those that vendors have been likely to run already. Asking the storage vendor to build storage and run your specific benchmarks is a big and expensive ask, but if you know the common storage tests a vendor is likely to have and that even somewhat match the I/O of your main applications, then you will understand their solution, understand the trade-offs, and quickly come to a recommended solution.
Although I recommend you use the tools the vendor provides, in subsequent articles I will talk about Linux tools that you can use even with proprietary solutions.
Linux Storage Solutions
The second solution option is to build the Linux storage solution yourself and use the open source tools that Linux offers. On the other hand, you could specify to a vendor that you require a Linux-based storage solution built with standard Linux tools and let them build the solution for you.
To begin this discussion, I want to set some boundaries and expectations. The discussion will not be a tutorial for building and managing Linux solutions. Instead, it will cover various storage topics, specifically the tools for creating and managing storage on a Linux system but at a higher level to help you get started with understanding Linux storage solutions. Don’t be surprised if this article misses some topics entirely. I apologize for these gaps.
How Linux Labels Storage Devices
I don’t discuss, compare, or contrast Linux filesystems, but to begin the discussion, you need to know how storage devices are labeled within Linux. To create a common basis for discussion, I start with how Linux views storage devices (hard drives and solid-state drives (SSDs). What I won’t talk about are removable media such as USB drives or sticks; CD, DVD, and Blu-ray drives; or tape.
When a Linux system boots, it discovers storage devices that are attached to the system (server). As they are found, a representative file is created in the /dev directory, which contains mostly character devices, storage devices, and sometimes other devices.
The name of the file created for a storage device depends on the type of device and the order in which the devices are discovered. The first SATA device encountered will be sda. The next one would be sdb, and so on.
In the case of NVMe, the device names (i.e., the file name in /dev/) can be rather long, but logical. The first NVMe controller is referred to as nvme0, the second nvme1, and so on.
Several devices can be attached to an NVMe controller. The number of the device is added to the NVMe controller beginning with an n. The first device connected to a controller starts with 0, the second device is 1, and so on. Therefore, the first device on the first controller would be nvme0n0. The next device on that controller would be nvme0n1. The first device on the second NVMe controller would be nvme1n0. I hope you see the pattern,
nvme[controller]n[device]
where [controller] is the number of the controller (e.g., 0, 1, 2) and [device] is the device number (e.g., 0, 1, 2) on the controller.
You can use the device labels with other tools to create complicated devices and use these devices for building filesystems. Additionally, you can partition these devices to achieve better management, better performance, and better data reliability.
Partitioning – Containers for Filesystems
Rarely will you use an entire block storage device for the filesystem, unless it is for simple storage configurations such as a one-drive laptop. Typically these block devices are split into partitions that are then used to house filesystems. Technically, a partition is a region of multiple contiguous blocks (or sectors) that are then managed by the partition table on the block device. If you like, you can think of these partitions as “logical containers” that are then used for filesystems.
Partitions come in two types: primary and extended. A primary partition is bootable, and an extended partition is not. You are allowed up to four primary partitions or three primary partitions and one extended partition. However, you can create more than four partitions by taking a primary partition and creating an extended partition; then, inside the extended partition you can create logical partitions. In this way, you can get up to 15 partitions per device.
Partitioning block storage devices allows you to control what part of Linux is put on which partition. In turn, this can help control the amount of storage space used by that partition (limiting filesystem size), which prevents filesystems from filling up and causing Linux to crash.
Classically, the storage device that holds the operating system (OS) has a few partitions. The first is /boot, which holds the kernel and the files needed for the bootstrapping process. This partition doesn’t have to be too large, and you can make it a primary partition.
The root filesystem classically has a partition, as well, referred to as /. The amount of space needed for the root filesystem varies and depends on the packages installed and the number of packages installed as part of the OS. You should do some research on what OS and distribution you’ll be using and how much space is needed for an installation.
The next “classic” partition, which could be a bit controversial, is for swap space. Some people advocate that you no longer need swap space. Others recommend making a swap “file” instead of using a partition dedicated to swap space. The guidance many years ago was to make the swap space twice as large as system memory. I think that wisdom is pretty old, but I still create a dedicated swap partition because it gives me some time to determine what is happening to a system before it crashes.
Finally, for desktops and laptops, the classic approach is to create a partition for user data (/home). Following these traditional recommendations, this would be the fourth primary partition for a single storage device, preventing the addition of any partitions to the device. Therefore, it’s probably not a bad idea to use the fourth primary partition to create an extended partition where you can create several logical partitions.
The reason to dedicate a partition to /home is so that filling up user data space won’t cause problems. For larger systems /home is usually provided by external storage from a storage vendor.
Commands to Create Partitions
Two tools let you create partitions in Linux: fdisk and parted. I tend to use fdisk, but I think that is more from habit than anything substantial. The first command option you are likely to use is -l to list the partitions on a device (Listing 1). I removed the non-storage devices from the output. Note the partitions on some devices (e.g., /dev/nvme0n1) and not on others (e.g., /dev/sda).
Listing 1: Listing Partitions on a Device
$ sudo fdisk -l ... Disk /dev/nvme1n1: 953.87 GiB, 1024209543168 bytes, 2000409264 sectors Disk model: Samsung SSD 970 PRO 1TB Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x8c344631 Device Boot Start End Sectors Size Id Type /dev/nvme1n1p1 2048 2000408575 2000406528 953.9G 83 Linux Disk /dev/nvme0n1: 476.94 GiB, 512110190592 bytes, 1000215216 sectors Disk model: Samsung SSD 970 PRO 512GB Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: DD46F2B6-9DDE-4810-AA43-905AB60C656D Device Start End Sectors Size Type /dev/nvme0n1p1 2048 1050623 1048576 512M EFI System /dev/nvme0n1p2 1050624 1000214527 999163904 476.4G Linux filesystem Disk /dev/sda: 5.46 TiB, 6001175126016 bytes, 11721045168 sectors Disk model: ST6000NM0115-1YZ Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt Disk identifier: B20C5813-F7F2-45BF-89C0-A88CE5D5329F Device Start End Sectors Size Type /dev/sda1 2048 11721043967 11721041920 5.5T Linux filesystem ...
If you create a partition on an NVMe drive, the partition number is added to the end of the device name after a p. Partitions start counting with 1. Therefore, something like /dev/nvme0n1p2 is the second partition (p2) of the first device (n1) on the first NVMe controller (nvme0).
To create a partition, run the fdisk command with the device name but no options to put you in “command” mode (Listing 2).
Listing 2: Entering Command Mode
$ sudo fdisk /dev/sda Welcome to fdisk (util-linux 2.37.2). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Command (m for help): m Help: GPT M enter protective/hybrid MBR Generic d delete a partition F list free unpartitioned space l list known partition types n add a new partition p print the partition table t change a partition type v verify the partition table i print information about a partition Misc m print this menu x extra functionality (experts only) Script I load disk layout from sfdisk script file O dump disk layout to sfdisk script file Save & Exit w write table to disk and exit q quit without saving changes Create a new label g create a new empty GPT partition table G create a new empty SGI (IRIX) partition table o create a new empty DOS partition table s create a new empty Sun partition table
You can then create a new partition with the n option, which will ask whether you want a primary or logical partition. Fortunately, fdisk gives good hints, such as what the next partition number will be, so you don’t overwrite something you already did without first removing it.
Next, you are asked for the first and last sector of the partition. You can use sectors or kibibytes (K), mebibytes (M), gibibytes (G), tebibytes (T), or pebibytes (P). If this is the first partition, accept the default first sector. For the last sector, enter the size you want the partition.
Now you can create the partition type. If you type L it lists all of the partition types it understands. Select the one you want – probably one of the Linux partition types.
After the partition type you reach the main menu. At this point all changes are in memory and have not been committed to the devices. To get a detailed list of the partitions, enter p. If you want to commit these partitions, type w to write the partitions to the drives; otherwise, enter q.
Other fdisk commands are useful. I would recommend finding a tutorial for more details. I recommend that you don’t try to partition a device that is mounted because it will cause all kinds of mayhem. If you want to repartition a device, be sure to check what you are doing beforehand. Also, if that device has data on it, be sure there is at least one copy of it outside of the device.
Listing Block Devices
One command that will help you at this point is lsblk, which lists block devices (Listing 3). The output includes the loop devices as well as the storage devices and creates something like a tree structure for the device partitions . It also labels these lines part instead of disk. Note that the SIZE column lists the size of the partitions in kibibyte, mebibyte, gibibyte, tebibyte, or pebibyte.
Listing 3: Listing Block Devices
$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 4K 1 loop /snap/bare/5 loop1 7:1 0 63.4M 1 loop /snap/core20/1974 loop2 7:2 0 63.5M 1 loop /snap/core20/2015 loop3 7:3 0 73.9M 1 loop /snap/core22/864 loop4 7:4 0 237.2M 1 loop /snap/firefox/3026 loop5 7:5 0 236.9M 1 loop /snap/firefox/3131 loop6 7:6 0 346.3M 1 loop /snap/gnome-3-38-2004/119 loop7 7:7 0 349.7M 1 loop /snap/gnome-3-38-2004/143 loop8 7:8 0 485.5M 1 loop /snap/gnome-42-2204/126 loop9 7:9 0 496.9M 1 loop /snap/gnome-42-2204/132 loop10 7:10 0 91.7M 1 loop /snap/gtk-common-themes/1535 loop11 7:11 0 40.8M 1 loop /snap/snapd/19993 loop12 7:12 0 40.8M 1 loop /snap/snapd/20092 sda 8:0 0 5.5T 0 disk |---sda1 8:1 0 5.5T 0 part /home2 nvme1n1 259:0 0 953.9G 0 disk |---nvme1n1p1 259:2 0 953.9G 0 part /home nvme0n1 259:1 0 476.9G 0 disk |---nvme0n1p1 259:3 0 512M 0 part /boot/efi |---nvme0n1p2 259:4 0 476.4G 0 part /var/snap/firefox/common/host-hunspell /
The lsblk command has some really great options, such as,
- listing devices with byte information (size),
- printing the device owner, group, and mode of the block devices with the m option (lsblk -m),
and many others options. If you are talking about storage, lsblk is a nifty command to have.
Now you can create a filesystem on top of these partitions, modify /etc/fstab to mount them, and be happy. However, could you better use the partitions to improve performance or make things more flexible for the inevitable requirements for more storage space? (How would you “grow” /home if it was locked to a single partition on a single storage device?) The next installment of this series of articles briefly presents a tool that can help in this regard.
Just the Beginning
How to manage storage is an immense topic and is too much to cover in one article or even a book. However, I really want to tackle this question because it has been asked so often – thus the need for a series of articles that address the question from a high level.
This first article in the series discussed the two main options for storage: proprietary and open source (Linux based). I do not compare them, merely point out the differences. If you buy a proprietary solution, be sure to use their tools. The vendor carefully designed them for their solution, and they understand how the tools work and what the output means. You can use some Linux commands with them, but that's a topic for another article.
Linux has quite a bit of capability for building storage solutions. I began this discussion with instructions on labeling and partitioning Linux storage devices, which are straightforward but definitely foundational topics.