As part of my life experience, I have discovered that people like to keep pretty much every piece of data that’s crossed their hard drive. That is, the rm command is rarely, if ever, used. I am no exception. I have lots of data that I want to keep available, yet rarely touch.

Even though you can now get 10TB hard drives and HPC systems routinely have more than 1PB of storage, it is fairly easy to run out of space. Users don’t take the time to compress their data files to save space, and possibly for good reasons. A compressed file has to be uncompressed and then examined to discover its contents. If it is used, then it needs to be compressed again, which means several commands have to be performed just to examine the data.

What could be more useful would be the use of a compressed filesystem. Linux has several options, including the definitely underestimated SquashFS.

Compressing Data

The concept behind data compression, which has been around for a long time, is to encode data by using various techniques that save storage space. Data compression also reduces the size of data that is to be transmitted. The most common example in Linux is gzip, which is used to compress data files. Here’s a quick example illustrating the change in file size:

$ ls -lsah FS_scan.csv 
3.2M -rw-r--r-- 1 laytonjb laytonjb 3.2M 2014-06-09 20:31 FS_scan.csv
$ gzip -9 FS_scan.csv 
$ ls -lsah FS_scan.csv.gz 
268K -rw-r--r-- 1 laytonjb laytonjb 261K 2014-06-09 20:31 FS_scan.csv.gz

The original file is 3.2MB, but after using gzip with the -9 option (i.e., maximum compression), the resulting file is 268KB. The .gz extension indicates that the file has been compressed with gzip. The compression ratio, which is the ratio of the original size to the compressed size, is 11.9:1. This compression ratio is very dependent on the uncompressed data (i.e., how compressible the data is) and the compression algorithm.

Not all data can be compressed as much as a CSV file, which is pure text and highly compressible. Binary data files usually cannot be compressed as much because they are already just about as small as possible.

A huge number of compression algorithms can take data and find a new encoding that results in much smaller files. Classically, this involves looking for patterns in the data. The algorithms vary by how they search for patterns and create the encoding. Sometimes they require a great deal of memory to store the various patterns, and sometimes they require lots of CPU time, which leads to the need to find a balance between compression level, the amount of memory required, and the amount of time it takes to complete the compression. However, the goal of all compression programs remains the same: reduce the size of data to save space.

The primary trade-off for compressed filesystems is the number of CPU cycles and the time it takes to compress and uncompress data in return for reduced storage space. If you have the cycles and don’t need a fast filesystem, you can save space. Alternatively, if your memory or storage space is severely constrained, then compressed filesystems may be your only choice. Severe storage restrictions is most common in embedded systems.

Compressed Filesystems in Linux

Through the growth and development of Linux, a fair number of filesystems have focused on compressing data, including:

• JFFS
• JFFS2
• LogFS
• Cramfs
• Cromfs
• Fusecompress
• SquashFS

Other filesystems such as Btrfs, ZFS, and ReiserFS have compression capability, but they have had to make some pretty serious compromises. So that filesystem performance is not overly penalized, they cannot perform compressions that require a great deal of time, and they really can only compress the data chunks they are given. Although they can achieve some level of compression, they cannot obtain the levels that focused filesystems, such as those in the list, do.

SquashFS

SquashFS is a compressed read-only filesystem for Linux. It takes data and creates something like a compressed “archive” that can be mounted on Linux systems. You can then read data from the filesystem as needed, but you can’t write to it.

The exciting thing about SquashFS is that it has a wide variety of compression algorithms available:

• gzip
• LZMA
• LZO
• xz

You can experiment with all of them to find the one that compresses the most, the fastest, or according to whatever metric you value.

SquashFS has been in the kernel for a long time (since 2.6.29). The tools for managing SquashFS are available in almost all Linux distributions. A number of features from among the many are summarized here:

• Maximum filesystem size is 2^64.
• Maximum file size is 2TiB.
• Can be NFS-exported (read-only).
• Compresses metadata.
• Has extended file attribute (xattr) support (but not ACL support).
• Supports an unlimited number of directories, files, and entries per directory.
• Can support sparse files.
• Can use a 1MiB block size (default is 128KiB).

The larger block sizes can producer greater compression ratios (i.e., smaller size filesystems), although using block sizes other than the typical 4KB has created some difficulties for SquashFS.

SquashFS doesn’t decompress the blocks into the kernel pagecache. This means that SquashFS has its own caches: one for metadata and one for fragments (i.e., two small caches). The cache is not used for file data blocks, which are decompressed and cached in the kernel pagecache in the typical fashion; rather, it is used for fragment or metadata blocks that have been read as a result of a metadata or fragment access. The blocks are decompressed and temporarily placed into the SquashFS cache. SquashFS packs metadata and fragments together into blocks for maximum compression so that when a read access of a fragment or metadata occurs, the retrieval also obtains other metadata and fragment data. Rather than discard these additional pieces of information, they are placed into the temporary cache, so they do not have to be retrieved and decompressed in the case of a near future access.

Using SquashFS

Using SquashFS is not difficult, comprising only two steps. The first step is to create a filesystem image using the SquashFS tools. You can create an image of an entire filesystem, a directory, or even a single file. This image, then, can be mounted directly (if it is a device) or mounted using a loopback device (if it is a file).

The tool that creates the image is called mksquashfs. It has a number of options that allow control over virtually all aspects of the image. The man page is not very long, and it’s definitely worth a look at the various options. Any user can create an image of any part of their data they desire. However, mounting it requires root access (or at least sudo access).

As an example, I'll take a directory (/home/laytonjb/20170502) on my desktop where I have stored PDFs, ZIP files, and other bits of information and articles that I collect throughout the month (I’m a digital hoarder). I want to compress this directory and all its subdirectories and files. Then, I want to mount it read-only so I can access the information but still save some space.

Before compression the directory was about 358MB:

$ du -sh
358M    .

The first step is to create the image file, which can be done by the user as long as the resulting image is stored somewhere the user has permission:

$ time mksquashfs /home/laytonjb/20170502 /home/laytonjb/squashfs/20170502.sqsh
Parallel mksquashfs: Using 4 processors
Creating 4.0 filesystem on /home/laytonjb/squashfs/20170502.sqsh, block size 131072.
[================================================-] 2904/2904 100%
 
Exportable Squashfs 4.0 filesystem, gzip compressed, data block size 131072
        compressed data, compressed metadata, compressed fragments, compressed xattrs
        duplicates are removed
Filesystem size 335196.73 Kbytes (327.34 Mbytes)
        91.53% of uncompressed filesystem size (366234.01 Kbytes)
Inode table size 8424 bytes (8.23 Kbytes)
        50.01% of uncompressed inode table size (16846 bytes)
Directory table size 2199 bytes (2.15 Kbytes)
        63.72% of uncompressed directory table size (3451 bytes)
Xattr table size 54 bytes (0.05 Kbytes)
        100.00% of uncompressed xattr table size (54 bytes)
Number of duplicate files found 1
Number of inodes 94
Number of files 93
Number of fragments 5
Number of symbolic links  0
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 1
Number of ids (unique uids + gids) 1
Number of uids 1
        laytonjb (1000)
Number of gids 1
        laytonjb (1000)

Notice that the command gives a reasonable amount of output without being too verbose.

I used the command defaults, which means a block size of 128KiB (131,072 bytes) and the use of gzip to compress the data. In the output, SquashFS states that it was able to compress the data to 91.53% of its uncompressed size, or to 328MB (327.34MB).

Notice that I used the time command to time how long it took to run the command. The results were:

real    0m7.675s
user   0m29.074s
sys     0m1.002s

This looks to be pretty fast for compressing 358MB of data (on an SSD).

The next step is to mount the SquashFS image as you would any other filesystem. Out of the box, root needs to do this because the user does not have access to the mount command.

$ mount -t squashfs /home/laytonjb/squashfs/20170502.sqsh /home/laytonjb/20170502_new -o loop
$ mount
...
/home/laytonjb/squashfs/20170502.sqsh on /home/laytonjb/20170502_new type squashfs (ro,relatime,seclabel)

It all looks good. Now to look at /home/laytonjb/20170502_new to make sure everything is there and permissions are as expected:

$ ls -lsat
...
  830 -rw-r--r--.  1 laytonjb laytonjb   848854 Jun 10 13:58 mesos.pdf
  535 -rw-r--r--.  1 laytonjb laytonjb   546505 Jun 10 13:58 Martins2003CSD.pdf
 8803 -rw-r--r--.  1 laytonjb laytonjb  9013307 Jun 10 13:58 Hwang2012c.pdf
...

I can look at the files, and they are owned by me.

Optimization Study

The two major options you are likely to use are -comp [comp] and -b [bsize]. The first option allows you to specify the compression algorithm used (from the current options listed earlier). The second option allows you to control the block size (from the default of 128KiB to the maximum of 1MiB). Larger block sizes can help improve the amount of compression.

The simple command that uses the lzma compression and a 1MiB block size would be:

$ mksquashfs /home/laytonjb/20170502 /home/laytonjb/squashfs/20170502.sqsh -comp lzma -b 1048576

The directory I’ve used in the examples is full of PDF and ZIP files. I didn’t expect it to compress too much, but I did get some compression. As an experiment, I tried all four compression techniques with the default block size, 128KiB, and the maximum block size, 1MiB. The results are shown in the table.

Pretty obviously, the fastest compression technique is gzip, with little difference in the user time it took for either block size (two-second difference, or a little less than 10%). The large block size did give a very tiny bit of extra compression.

The xz and lzma algorithms result in the most compression and take the longest – much longer than gzip – but even for the default block size, they can compress the data by about 10%. Using the largest block size, they can get a little more compression: a little over 11%.

You might scoff at 10%, but remember that the files are binary. If you have 100TB of data, 10% is 1TB. Not too bad. If you have 1PB, then 10% is 100TB, which is quite a bit of space.

Summary

Even though data storage has gotten inexpensive, data consumption grows at a faster rate than storage. I don’t think I’ve ever heard anyone ask for less storage space. Finding ways to reduce the amount of data is a key function in the life of an HPC administrator.

One way to conserve space is to compress data that is not used very often. Although you can do this on a file-by-file basis, a better way is to collect all of the data into a single directory and create a compressed filesystem image. SquashFS is probably the best tool for the job. It’s very easy to use and comes with virtually every Linux distribution out there. Give it a try; you won’t be disappointed.