Playing with Blocks

Time is money, and sometimes that means you need a faster way to process data. Solid state drives (SSDs) and, more specifically, non-volatile memory express (NVMe) devices have helped alleviate the burden of processing data to and from a backing store. However, at times, even SSD technology is not quite fast enough, which is where the RAM drive comes into the picture.

Typically, the RAM drive is used as temporary storage for two reasons: Its capacities tend to be lower (because the technology is more expensive), and more importantly, it is a volatile technology; that is, if the system were to lose power or go into an unstable state, the contents of that RAM drive would disappear. Depending on the type of data being processed, the reward can often outweigh the risks, which is why the RAM drive can potentially be the better option.

In this article, I rely on the RapidDisk suite to create and manage RAM drives. The RapidDisk software project [1] provides an advanced set of Linux kernel RAM drive and caching modules with which you can dynamically create and remove RAM drives of any size or map them as a temporary read cache to slower devices.

The system used in this article is an older system with limited memory clocked at a slow speed. More modern and faster systems with faster memory will produce significantly different results than those found here. The dmidecode command summarizes the configuration and capabilities of memory DIMMs and revealed that my system has four DDR3 RAM devices of 2048MB configured at speeds of 1333MTps (mega transfers per second).

Playing with RAM Drives

To begin, you need to download and build the RapidDisk suite from source code by cloning the latest stable code from the RapidDisk repository [2] (Listing 1).

$ git clone https://github.com/pkoutoupis/rapiddisk.git
Cloning into 'rapiddisk'...
remote: Enumerating objects: 1560, done.
remote: Counting objects: 100% (241/241), done.
remote: Compressing objects: 100% (158/158), done.
remote: Total 1560 (delta 149), reused 156 (delta 82), pack-reused 1319
Receiving objects: 100% (1560/1560), 762.11 KiB | 5.82 MiB/s, done.
Resolving deltas: 100% (949/949), done.

Next, change into the repository's root directory and build and install both the kernel modules and userspace utilities:

$ cd rapiddisk
$ make && sudo make install

Assuming that all libraries and package dependencies have been installed, both the build and installation should have completed without failure. Now insert the kernel modules:

$ sudo modprobe rapiddisk
$ sudo modprobe rapiddisk-cache

At this point, if you invoke the rapiddisk utility to list all RAM drive targets, none should be listed:

$ sudo rapiddisk -l
rapiddisk 7.2.0
Copyright 2011 - 2021 Petros Koutoupis
 **
Unable to locate any RapidDisk devices.

The amount of memory to use for your RAM drive needs to be determined carefully. The rapiddisk module allocates memory pages as they are requested. In theory, you can create a RAM drive that is much larger than the amount of system memory, but in good practice, this should never be done. As a RAM drive continues to be filled with data, it will eventually run out of free memory pages to allocate, and it could potentially panic the kernel, requiring a reboot of the system. In this example, the system has 8GB of total memory and about 6.5GB of "free" memory (Listing 2).

$ free -m
              total        used        free      shared  buff/cache   available
Mem:           7951         202        6678           1        1069        7479
Swap:          4095           0        4095

The following example creates a 1GB RAM drive:

$ sudo rapiddisk -a 1024
rapiddisk 7.2.0
Copyright 2011 - 2021 Petros Koutoupis
 **
Attached device rd0 of size 1024 Mbytes

As mentioned earlier, the RapidDisk suite is designed to create and remove RAM drives of any size dynamically. For instance, if you want to create an additional 32MB RAM drive, you would rerun the same command with a different size (Listing 3). The output from the second command verifies that the RAM drives were created.

$ sudo rapiddisk -a 32
rapiddisk 7.2.0
Copyright 2011 - 2021 Petros Koutoupis
 **
Attached device rd1 of size 32 Mbytes
$ sudo rapiddisk -l
rapiddisk 7.2.0
Copyright 2011 - 2021 Petros Koutoupis
List of RapidDisk device(s):
 RapidDisk Device 1: rd1   Size (KB): 32768
 RapidDisk Device 2: rd0   Size (KB): 1048576
List of RapidDisk-Cache mapping(s):
  None

Just as easily as it was created, the RAM drive can be removed:

$ sudo rapiddisk -d rd1
rapiddisk 7.2.0
Copyright 2011 - 2021 Petros Koutoupis
Detached device rd1

To compare the RAM drive with a local 12Gb Serial Attached SCSI (SAS) spinning hard disk drive (HDD) connected to a 6Gb Host Bus Adaptor (HBA), write 1GB worth of sequential data in 1MB transfers to the HDD with dd:

$ sudo dd if=/dev/zero of=/dev/sdf bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.76685 s, 225 MB/s

The result, 225MBps throughput, is not bad at all for an HDD. To compare its performance with a memory-backed volume, enter:

$

Wow! The output shows 1.4GBps. That is nearly six times the throughput of the HDD. On a more modern and faster system, that number should and would be higher: about 16 times the throughput or more.

Random access I/O is where the memory device shines. Now, test your performance with the fio performance benchmarking utility by running a random write test on the original HDD with 4KB transfers (Listing 4). The output shows 1.5MBps, which isn't fast at all. Now, run the same random write test to the RAM drive (Listing 5). Here, you can see an impressive 1GBps; again, on a modern system that number would be much higher.

$ sudo fio --bs=4k --ioengine=libaio --iodepth=32 --size=500m --direct=1 --runtime=60 --filename=/dev/sdf --rw=randwrite --numjobs=1 --name=test
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=1733KiB/s][w=433 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=12944: Sat Jun 19 14:49:58 2021
  write: IOPS=421, BW=1685KiB/s (1725kB/s)(98.9MiB/60118msec); 0 zone resets
    [ ... ]
Run status group 0 (all jobs):
  WRITE: bw=1685KiB/s (1725kB/s), 1685KiB/s-1685KiB/s (1725kB/s-1725kB/s), io=98.9MiB (104MB), run=60118-60118msec
Disk stats (read/write):
  sdf: ios=51/25253, merge=0/0, ticks=7/1913272, in_queue=1862556, util=99.90%

$ sudo fio --bs=4k --ioengine=libaio --iodepth=32 --size=500m --direct=1 --runtime=60 --filename=/dev/rd0 --rw=randwrite --numjobs=1 --name=test
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=12936: Sat Jun 19 14:48:48 2021
  write: IOPS=250k, BW=977MiB/s (1024MB/s)(500MiB/512msec); 0 zone resets
    [ ... ]
Run status group 0 (all jobs):
  WRITE: bw=977MiB/s (1024MB/s), 977MiB/s-977MiB/s (1024MB/s-1024MB/s), io=500MiB (524MB), run=512-512msec
Disk stats (read/write):
  rd0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

You will observe similar results with random read operations (Listings 6 and 7). The HDD produces about 2.5MBps, whereas the RAM drive is up to an impressive 1.2GBps.

$ sudo fio --bs=4k --ioengine=libaio --iodepth=32 --size=500m --direct=1 --runtime=60 --filename=/dev/sdf --rw=randread --numjobs=1 --name=test
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=2320KiB/s][r=580 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=12975: Sat Jun 19 14:51:27 2021
  read: IOPS=622, BW=2488KiB/s (2548kB/s)(146MiB/60127msec)
    [ ... ]
Run status group 0 (all jobs):
   READ: bw=2488KiB/s (2548kB/s), 2488KiB/s-2488KiB/s (2548kB/s-2548kB/s), io=146MiB (153MB), run=60127-60127msec
Disk stats (read/write):
  sdf: ios=37305/0, merge=0/0, ticks=1913563/0, in_queue=1838228, util=99.89%

$ sudo fio --bs=4k --ioengine=libaio --iodepth=32 --size=500m --direct=1 --runtime=60 --filename=/dev/rd0 --rw=randread --numjobs=1 --name=test
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
test: (groupid=0, jobs=1): err= 0: pid=12967: Sat Jun 19 14:50:18 2021
  read: IOPS=307k, BW=1199MiB/s (1257MB/s)(500MiB/417msec)
    [ ... ]
Run status group 0 (all jobs):
   READ: bw=1199MiB/s (1257MB/s), 1199MiB/s-1199MiB/s (1257MB/s-1257MB/s), io=500MiB (524MB), run=417-417msec
Disk stats (read/write):
  rd0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%

NVMe over Fabrics Network

Sometimes it might be necessary to export those performant volumes across a network so that other compute nodes can take advantage of the high speed. In the following example, I rely on the NVMe over Fabrics concept and, more specifically, the NVMe target modules provided by the Linux kernel to export the RAM drive and, in turn, import it to another server where it will look and operate like a local storage volume.

Most modern distributions will have the NVMe target modules installed and available for use. To insert the NVMe and NVMe TCP target modules, enter:

$ sudo modprobe nvmet
$ sudo modprobe nvmet-tcp

The NVMe target directory tree will need to be made available by the kernel user configuration filesystem, which will provide access to the entire NVMe target configuration environment. To begin, mount the kernel user configuration filesystem and verify that it has been mounted:

$ sudo /bin/mount -t configfs none /sys/kernel/config/
$ mount|grep configfs
configfs on /sys/kernel/config type configfs (rw,relatime)

Now, create an NVMe target directory for the RAM drive under the target subsystem and change to that directory (which will host the NVMe target volume plus its attributes):

$ sudo mkdir /sys/kernel/config/nvmet/subsystems/nvmet-rd0
$ cd /sys/kernel/config/nvmet/subsystems/nvmet-rd0

Because this is an example of general usage, you do not necessarily care about which initiators (i.e., hosts) connect to the exported target:

$ echo 1 |sudo tee -a attr_allow_any_host > /dev/null

Next, create a namespace, change into the directory, set the RAM drive volume as the device for the NVMe target, and enable the namespace:

$ sudo mkdir namespaces/1
$ cd namespaces/1/
$ echo -n /dev/rd0 |sudo tee -a device_path > /dev/null
$ echo 1|sudo tee -a enable > /dev/null

Now that you have defined the target block device, you need to switch your focus and define the target (network) port. To create a port directory in the NVMe target tree, change into that directory, and set the local IP address from which the export will be visible, enter:

$ sudo mkdir /sys/kernel/config/nvmet/ports/1
$ cd /sys/kernel/config/nvmet/ports/1
$ echo 10.0.0.185 |sudo tee -a addr_traddr > /dev/null

(The IP address in the last command will need to reflect your server configuration.) Now, set the transport type, port number, and protocol version:

$ echo tcp|sudo tee -a addr_trtype > /dev/null
$ echo 4420|sudo tee -a addr_trsvcid > /dev/null
$ echo ipv4|sudo tee -a addr_adrfam > /dev/null

Note that for any of this to work, both the target and initiator will need port 4420 to be open in its input/output firewall rules.

To tell the NVMe target tree that the port just created will export the block device defined in the subsystem section above, the commands

$ sudo ln -s /sys/kernel/config/nvmet/subsystems/nvmet-rd0/ /sys/kernel/config/nvmet/ports/1/subsystems/nvmet-rd0
$ dmesg |grep "nvmet_tcp"
[14798.568843] nvmet_tcp: enabling port 1 (10.0.0.185:4420)

link the target subsystem to the target port and verify the export.

Importing to a Remote Server

To use the RAM drive as if it were a locally attached device, move onto a secondary server (i.e., the server that will connect to the exported target). Because most modern distributions will have the proper NVMe modules installed and available for use, load the initiator or host-side kernel modules:

$ sudo modprobe nvme
$ sudo modprobe nvme-tcp

Again, for any of this to work, both the target and initiator will need to have port 4420 open in its input/output firewall rules.

Use the nvme command-line utility [3] to discover the NVMe target exported by the target server (Listing 8), connect to the target server, and import the NVMe device(s) it is exporting (in this case, you should see just the one),

$ sudo nvme discover -t tcp -a 10.0.0.185 -s 4420
Discovery Log Number of Records 1, Generation counter 2
=====Discovery Log Entry 0======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified, sq flow control disable supported
portid:  1
trsvcid: 4420
subnqn:  nvmet-rd0
traddr:  10.0.0.185
sectype: none

$ sudo nvme connect -t tcp -n nvmet-rd0 -a 10.0.0.185 -s 4420

and verify that the NVMe subsystem sees the NVMe target (Listing 9).

$ sudo nvme list
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1     S3ESNX0JA48075E      Samsung SSD 960 EVO 250GB                1          22.41  GB / 250.06  GB    512   B +  0 B   2B7QCXE7
/dev/nvme1n1     07b4753784e26c18     Linux                                    1           1.07  GB /   1.07  GB    512   B +  0 B   5.4.0-74

You will notice that the RAM drive is enumerated as the second NVMe drive in the list (i.e., /dev/nvme1n1). Now, verify that the volume is listed in the local device listing:

$ cat /proc/partitions |grep nvme
 259        0  244198584 nvme0n1
 259        1  244197543 nvme0n1p1
 259        3    1048576 nvme1n1

You are now able to read from and write to /dev/nvme1n1 as if the RAM drive were a locally attached device:

$ sudo dd if=/dev/zero of=/dev/nvme1n1 bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.868222 s, 1.2 GB/s

Finally, type

$ sudo nvme disconnect -d /dev/nvme1n1

which will disconnect the NVMe target volume.