Containers provide reproducible, portable applications. In a sense, they are an evolution of virtual machines (VMs), but without the entire operating system (OS). Therefore, containers are a bit smaller, easier to move around, possibly easier to create, and typically run on bare metal, dispensing with the hypervisor abstraction.

Many people in the high-performance computing (HPC) world are adopting containers to help users and application developers package their software. The containers can then be shared with others, who then don’t have to rebuild an application and test it on their system – they just use the prebuilt application in the container.

Containers can also be used as a way to reproduce work. A container can package everything, starting with the application, datasets, output, and any written reports or papers. Including datasets can be problematic because they can inflate the size of the container, but if you are archiving the container, size isn’t as big a consideration.

The dominant container, Docker, was created for developers to share their work easily. Because developers usually have root access on their development system, not much thought was given to security, so Docker needs to be run as root.

However, in the HPC world, where many people share a single system, this lack of security is a non-starter. The first, and arguably most popular, container for HPC is Singularity, which provides the security of running containers as a user rather than as root. It also works well with parallel filesystems, InfiniBand, and Message Passing Interface (MPI) libraries, something that Docker has struggled to do.

Despite Docker’s need to run with root access, it is still probably the most popular container overall. At the very least, the Docker container format is used by a number of other containers or can be converted to another format, such as Singularity.

This article is the first in a series about best practices in building and using containers. The focus of these articles will be on HPC with Docker and Singularity container technologies, although other containers may be mentioned. In this first article, I discuss some of the initial best practices that apply to both Docker and Singularity. Subsequent articles will cover other best practices.

Building Containers

Before building a container, you usually start by creating a specification file (specfile) that is then used to build your container. In general, you define the base OS image and then add packages, code, libraries, and other things you want to include in your container.

The specfile is just a text file that the container reads to create the container. The formats for Docker and Singularity have differences, but they are fairly close. The sample Docker specfile in Listing 1 was taken from the Nvidia Developer Blog. A sample Singularity specfile is shown in Listing 2.

Listing 1: Docker Specfile

FROM nvidia/cuda:9.0-devel AS devel
 
# OpenMPI version 3.0.0
RUN apt-get update -y && \
    apt-get install -y --no-install-recommends \
        file \
        hwloc \
        openssh-client \
        wget && \
    rm -rf /var/lib/apt/lists/*
RUN mkdir -p /tmp && wget -q --no-check-certificate -P /tmp https://www.open-mpi.org/software/ompi/v3.0/downloads/openmpi-3.0.0.tar.bz2 && \
    tar -x -f /tmp/openmpi-3.0.0.tar.bz2 -C /tmp -j && \
    cd /tmp/openmpi-3.0.0 && ./configure --prefix=/usr/local/openmpi --disable-getpwuid --enable-orterun-prefix-by-default --with-cuda --with-verbs && \
    make -j4 && \
    make -j4 install && \
    rm -rf /tmp/openmpi-3.0.0.tar.bz2 /tmp/openmpi-3.0.0
ENV PATH=/usr/local/openmpi/bin:$PATH \
    LD_LIBRARY_PATH=/usr/local/openmpi/lib:$LD_LIBRARY_PATH

Listing 2: Singularity Specfile

BootStrap: docker
From: nvidia/cuda:9.0-devel
 
# OpenMPI version 3.0.0
%post
    apt-get update -y
    apt-get install -y --no-install-recommends \
        file \
        hwloc \
        openssh-client \
        wget
    rm -rf /var/lib/apt/lists/*
%post
    mkdir -p /tmp && wget -q --no-check-certificate -P /tmp https://www.open-mpi.org/software/ompi/v3.0/downloads/openmpi-3.0.0.tar.bz2
    tar -x -f /tmp/openmpi-3.0.0.tar.bz2 -C /tmp -j
    cd /tmp/openmpi-3.0.0 && ./configure --prefix=/usr/local/openmpi --disable-getpwuid --enable-orterun-prefix-by-default --with-cuda --with-verbs
    make -j4
    make -j4 install
    rm -rf /tmp/openmpi-3.0.0.tar.bz2 /tmp/openmpi-3.0.0
%environment
    export PATH=/usr/local/openmpi/bin:$PATH
    export LD_LIBRARY_PATH=/usr/local/openmpi/lib:$LD_LIBRARY_PATH

The length of the specfile can be fairly simple or it can be long and complex. Maintaining these specfiles or editing them can be difficult.

HPC Container Maker

To create or build a container, you can write a specfile that contains the steps needed to build a container. You can find a number of tutorials online for writing these specfiles for both Docker and Singularity. As you get more experience writing them, you discover that they get longer and longer, and sometimes you forget what the specfile does or why you included certain steps. Moreover, you might not include best practices for certain tools or libraries you use in the container, giving up performance.

One best practice I recommend for building specification files is to use HPC Container Maker (HPCCM), which I’ve written about in the past. Written in Python, HPCCM allows you to write a very simple Python script that describes your container; then, it creates a Docker or Singularity specfile. Also, you can create a single HPCCM recipe that has branching to allow you to create multiple container specifications.

HPCCM has some notable features:

• It collects and codifies best practices.
• It makes recipe file creation easy, repeatable, and modular.
• It becomes a reference and a vehicle to drive collaboration.
• It is container-implementation-neutral.

Rather than create yet another specific language, HPCCM relies on Python for the “recipe” of the container you want to build, regardless of the target container type. The HPCCM recipe has the steps you want to take in your container build. You can take advantage of the Python language within the recipe by creating variables and using if/elif/else statements, loops, functions, or almost anything else from Python.

The example in Listing 3 from my previous article on HPCCM shows a sample HPCCM recipe written in Python. The recipe starts with a base image (Ubuntu 16.04) then installs some packages that weren’t included in that image (i.e., make, wget, bzip2, and tar) with the built-in apt_get recipe, which has the ospackages option that adds these packages to the container. The recipe then does the same thing for the GNU compilers. Notice that because the compiler versions aren’t specified, HPCCM defaults to using the latest versions for that OS.

Listing 3: HPCCM Recipe

"""This example demonstrates recipe basics.
 
Usage:
$ hpccm.py --recipe test2.py --format docker
# hpccm.py --recipe test2.py --format singularity
"""
 
# Choose a base image
Stage0.baseimage('ubuntu:16.04')
 
ospackages = ['make', 'wget', 'bzip2', 'tar']
Stage0 += apt_get(ospackages=ospackages)
 
# Install GNU compilers (upstream)
Stage0 += apt_get(ospackages=['gcc', 'g++', 'gfortran'])
 
Stage0 += openmpi(cuda=False, infiniband=False,
                  prefix='/usr/local/openmpi', version='3.1.0')

The last line in the recipe adds the OpenMP MPI library to the specfile. In this case, it’s telling HPCCM not to include the commands for building with GPUs or InfiniBand and to build the library and install it into /usr/local/openmpi. Finally, it tells HPCCM to use OpenMPI version 3.1. When you use HPCCM to create a specfile, all of the specfile options to download the source code, including the dependencies, are added to the specfile. All of the build commands are also included.

Note that the recipe uses “stages,” starting with Stage0, that allow you to create multistage or multilayer builds. You can keep adding items to the stage (e.g., specific packages) or even have it build packages for a particular stage. If you want to create a multistage specfile, you would start with Stage0, then move to Stage1, Stage2, and so on.

Before applying the HPCCM recipe, save it to a file (e.g., test2.py); then, HPCCM can take that recipe and create a Docker or Singularity specfile. The command for creating a Singularity specfile (Listing 4) is:

$ hpccm --recipe ./test2.py --format singularity > Singularity2

Listing 4: The Singularity2 Specfile

BootStrap: docker
From: ubuntu:16.04
 
%post
    apt-get update -y
    apt-get install -y --no-install-recommends \
        make \
        wget \
        bzip2 \
        tar
    rm -rf /var/lib/apt/lists/*
 
%post
    apt-get update -y
    apt-get install -y --no-install-recommends \
        gcc \
        g++ \
        gfortran
    rm -rf /var/lib/apt/lists/*
 
# OpenMPI version 3.1.0
%post
    apt-get update -y
    apt-get install -y --no-install-recommends \
        file \
        hwloc \
        openssh-client \
        wget
    rm -rf /var/lib/apt/lists/*
%post
    mkdir -p /tmp && wget -q --no-check-certificate 
        -P /tmp https://www.open-mpi.org/software/ompi/v3.1/
         downloads/openmpi-3.1.0.tar.bz2
    tar -x -f /tmp/openmpi-3.1.0.tar.bz2 -C /tmp -j
    cd /tmp/openmpi-3.1.0 && ./configure 
        --prefix=/usr/local/openmpi --disable-getpwuid 
        --enable-orterun-prefix-by-default 
        --without-cuda --without-verbs
    make -j4
    make -j4 install
    rm -rf /tmp/openmpi-3.1.0.tar.bz2 /tmp/openmpi-3.1.0
%environment
    export LD_LIBRARY_PATH=
        /usr/local/openmpi/lib:$LD_LIBRARY_PATH
    export PATH=/usr/local/openmpi/bin:$PATH
%post
    export Misc
LD_LIBRARY_PATH=/usr/local/openmpi/lib:$LD_LIBRARY_PATH
    export PATH=/usr/local/openmpi/bin:$PATH

If you want to create a Singularity container, you can run the command:

% singularity build test2.simg Singularity2

HPCCM is a great help in writing specfiles for Singularity and Docker, making it much easier to add and build packages for your containers. A single HPCCM recipe file can work for both Docker and Singularity and a variety of distributions. These recipe files are simple and much easier to understand than other types of specfiles, and I’ve found them to be indispensable for building containers.

General Practices

Although it will take a few articles to cover the best practices for creating and maintaining containers, I thought I would share some general practices I’ve collected myself, learned from others, and gathered from articles:

• Create a help section: Being able to query a container for help information, without having to run the container and shell, is a great help to users. Singularity has the capability.
• Include the Dockerfile or Singularity recipe in the container: Very often, people want to know how you built the container, so they can inspect the container to understand what’s in it, discover who created it and when, or just to learn about containers. I think putting the Dockerfile or Singularity recipe file in the container is a great idea.
• Include a test section: Including a test section that can be executed to test the application in the container is a great way to be sure that the container is working properly. Ideally this section of the container should be able to be run without having to “shell” into the container. Singularity currently has this capability.
• Don't put data in a container: I see people violate this important best practice all the time. Putting data in the container can greatly increase its size, making it difficult to work with. Instead, you want the datasets to be mounted in the container in a volume.
• Keep to a single application per container: Although perhaps a controversial best practice, containers originally were designed to contain a single application. To run multiple applications you coordinated various containers. Theoretically, you can put multiple applications in a single container, but you greatly increase the complexity and size of the container.
• Build the smallest container possible: If you build a container that is as small as possible, it’s easier to store and move around and will contain fewer packages, reducing the container’s attack surface. However, don’t get crazy and eliminate almost everything in a container; it’s a balance between size and usability.
• Remove any unnecessary tools and files: This best practice is related to the previous item. You should only use the tools, libraries, and packages that are needed in the container, so you can reduce its size and the attack surface.
• Use multistage builds: Multistage builds have a number of advantages, but one stands out: If you build the application in one layer, you can then, in a subsequent layer, remove the build tools, reducing the size of the container.
• Think of containers as ephemeral objects: You should be able to stop, destroy, and rebuild a container with little effort. Once you are finished using the container, you can archive it for reproducibility. During the development phase, though, think of the container as ephemeral. HPCCM can be a great help in this regard.
• Use official images when possible: Companies and organizations put out “official” containers that have been created, scanned, and updated with security patches, so you should use these containers as a starting point rather than try to build your own. However, be careful of the origin of a container. Just because a container is listed somewhere doesn’t mean it’s safe.

This sampling of best practices are not hard and fast rules; rather, they are guidelines you should consider when you start building and using containers.

In the next section, I talk about what I consider to be the most important guideline when creating containers: datasets in the container.

Container Datasets

When I first started using containers, I quickly learned that if you aren’t careful, the size of a container can grow quite large. One of the first containers I built was almost 10GB because I put all of my data in the container.

In many situations, people believe that containers have to be completely self-contained; that is, the container holds the application and the dataset for the application. The obvious advantage of this approach is that everything needed is in the container. Wherever you run the container, you will have everything you need, and you know you have the desired data.

Depending on the size of the dataset, the container can be very large, which increases the time to transfer the container to a node for execution. Also, the container will be slow to start. Both of these issues, although they don’t affect container execution, can drive users crazy.

Over time, containers also have a tendency to grow as code and libraries are added, which exacerbates all of the issues around container size and execution time and can force the specfile out of control (use HPCCM to limit this issue).

Also, the dataset included in the container might not be the data you want. Moreover, you are moving around a potentially large container, so maintaining the container and the data within becomes unwieldy.

A best practice is to separate the container with the application from the data, so you can control the size of the container. Instead, keep the datasets on the host filesystem. This practice is perfect while you are developing the code, trying different datasets, or using the code to solve different problems. Granted, separating the data from the application doesn’t maintain the container as the sole source of the application and data, but it does make code development and maintaining the application much easier.

An HPC filesystem such a Lustre, BeeGFS, or IBM Spectrum Scale allows you store very large datasets. If you run the container on a node that mounts one of these filesystems, you can access the data quite easily.

Test Data

Nonetheless, people might argue that having some data in the container could prove to be useful. One good argument for putting datasets in the container that many users do not take into account is to be able to check that the application produces a correct answer.

A best practice is to include some “small-ish” datasets in the container that will exercise the primary code paths. You should also include the “correct” output when the included data is used. The key is that the datasets should be fairly small, run fairly quickly, and not produce much output, but exercise the important code paths; although this might seem to be a difficult proposition, it greatly enhances the utility of the container.

An alternative to including smaller data sets is to write a script that can download the data. Making the data available on the web allows a simple wgetcommand to pull down the dataset into the container.

Singularity allows you to define a test section in the specfile with%test that is executed after the container build process is complete. However, you can also execute it any time you want, and is a great way to build so that you can test an application with data. To execute the test section of the container, use the command:

$ singularity image.simg test

Inside the test section you can put almost anything you want. For example, you could could download data and execute the application or even put in code to compare your output to the “correct” output to make sure you are getting the correct answers. The Singularity documentation has a simple example:

%test
    /usr/local/bin/mpirun --allow-run-as-root /usr/bin/mpi_test

This simple test runs the mpi_test application, which you could create in a script, put in the container, and execute in the test section. The possibilities for testing are many.

Summary

Containers are a very fast up-and-coming technology for HPC. Many sites are already using them to help users and code developers. Two strengths of containers are portability and reproducibility. As with any software technology, people have developed best practices for containers.

This article is the first in a series that will present and discuss the current best practices for containers. Although many of these practices are logical, some (e.g., not including datasets in a container) seem limiting, but they always end up helping container creators and users.