Containers are a great solution for the problem of reproducibility, repeatability, portability, resource isolation, and archiving. They free users from being tied to a specific distribution, so they can build and use the distribution they know and use. In the previous article, I discussed general best practices by focusing on “high-level” techniques for containers. In this article, I dive deeper into specific best practices for building and running Docker and Singularity containers.

Building Containers

Building containers is not too difficult, especially if you use HPC Container Maker (HPCCM), which I discussed in the last article. However, as with many things, the devil is in the details.

Docker

As mentioned previously, Docker was not originally designed for HPC containers, but rather for people developing tools, applications, and libraries to share with each other and to run on their laptops. Assuming they were root, access was a given. If you are root, you can do anything.

Docker is still important to HPC because it was the first container. Used heavily in artificial intelligence, deep learning, and machine learning, it is even used for classic HPC applications.

To begin, I assume you have Docker or Docker Desktop installed on the system on which you want to build your container – which also assumes that you have root access to the system. The next step is that Docker requires a docker group to be on the system. If you want to build or run containers, you need to be part of that group. Adding someone to an existing group is not difficult:

$ sudo usermod -a -G docker layton

Chris Hoffman wrote an article with more details on how to add an existing user to a group.

Next, create your Dockerfile, which defines the image you want to create (again, using HPCCM). When Docker creates an image, it will look for a file named dockerfile, so I recommend creating a directory for each container you want to build and putting that file and the HPCCM recipe in that directory.

The next step is to build the image with the minimal command:

$ sudo docker build -t name:tag .

The first part of the command, docker build, tells Docker to create the image from the local dockerfile. The -t option lets you add a “tag” to Docker images to help identify them. These tags are extremely important because they label the image. Think of it as a file name: A good file name prevents you from searching every file to find the data you want. A viewable label for an image comprises a base name followed by a semicolon followed by a tag.

The base name should be something very familiar to anyone using your container. For example, it could be centos-7.5, ubuntu-19.10, ubuntu-19.10-tensorflow-2.0, or almost anything your want (but watch special characters and blanks). It doesn’t have to have the complete version in the name, but it should be recognizable and describe the base OS.

The tag allows you to be much more descriptive or to call out the name of the application(s) in the image. For example, the tag can be used for defining the version (e.g., 2.0), or a “purpose” for the container (e.g., data-science). Putting a date in the tag (e.g., data-science-0212020) is also a good practice. Combining all of these practices can create a long tag (e.g., centos-7.6:tensorflow-2.0-0212020-Layton), but such tags can be very useful. The previous example informs you that the distribution in the image is CentOS 7.6 and the image has TensorFlow 2.0, was created on February 12, 2020, and was created by user Layton. Overall, the important point is to make the tag useful so someone could look at it and have a good idea what the image is about.

Although the name:tag combination seems long, it contains a fair amount of information, and you could even add more information if you like, such as moving the main application to the name and using the tag for other information (e.g., other applications, build time, the origin of the container, etc.).

On the other hand, don’t go crazy. Here are a few best practices to follow for the name:tag:

• Don’t make it impossibly long.
• Don’t make it cryptic so no one else understands it.
• If you add special characters or phrases indicating the image has been “certified” in some way, be sure to inform the users.
• Don’t make it too short, or it will not be descriptive enough and is therefore useless.
• Personally, I like putting dates and creator initials in the name:tag, but not everyone does.
• Be consistent when creating your name:tag.

Take a look around at image names on your own Dockerhub, the public Dockerhub, or one at work. See what people have (or haven’t) done with name:tag. The keys are to make it simple yet descriptive and make it consistent.

The docker build command has many options you can use to control building the image, but I don’t use too many of them, and I don’t know of others who do, either. The only two I’ve really used are:

• -f – allows you to use a file with a name other than dockerfile
• --squash – allows you to squash newly built layers into a new single layer

The --squash command is still marked as experimental, although I think it’s been experimental for as long as Docker as been around. I’ll talk about it later in the article.

The very last option on the example docker build command line is a dot (.), which tells Docker to put the image in the current directory – a good practice when you are first learning Docker or when you first build an image.

Singularity

Singularity is not conceptually much different from Docker. Remember that it is really focused on HPC containers. Although rather easy to install, it is very dependent on a specific version of Go, which could cause issues, so be sure to read the installation documents and look at the release notes. However, once you have the matching version of Go, the installation is very easy.

In general, Singularity requires that you have root access to build your image (hold on, though, there is an exception). The HPC world could argue that root access is either a so-so thing or a bad thing. For the HPC users who have root access to their laptops, or even desktops, this is a non-issue. Work laptops, desktops, or workstations, however, typically won’t give you root access or even sudo access (maybe sudo for a few commands). Your work site will have policies for this issue.

If you don’t have root access, you can always create your image on a home machine or boot a laptop to a Linux distribution on a USB stick.

The other option is to use the build option --fakeroot. I’ll discuss this option more later on in the article.

Building a Singularity image locally is very straightforward and not unlike that of Docker. A sample build command as root might be:

$ sudo singularity build container.sif container.def

The command takes the Singularity definition file container.def and builds the resulting image container.sif (where sif stands for Singularity Image Format).

A number of build options are listed on the Singularity website. The two I recommend are:

• --encrypt
• --fakeroot

A very important Singularity feature is the ability to build encrypted images. The -e or --encrypt option during a build encrypts the filesystem image within a Singularity image. This encryption can be done with either a passphrase or asymmetrically with an RSA key pair in Privacy-Enhanced Mail (PEM/PKCS1) format.

The great thing about the encryption is that the image is encrypted at rest. A container based on the image is encrypted in transit and while running. At no time is a decrypted version on disk. The decryption occurs at run time, completely in kernel space. The details of how you encrypt an image during the build is explained in the documentation.

A definite best practice for building Singularity images is always to encrypt. You can pick the encryption method you want, but the advantages of encryption far outweigh the extra effort needed to decrypt them.

Singularity has a unique feature called fakeroot for building images and running containers. The feature is called with the option --fakeroot and is commonly referred to as “rootless,” allowing an unprivileged user to run a container by leveraging user namespace UID/GID mapping.

A fakeroot user inside a container and the requested namespaces has almost the same admin rights as root. The first implication is that you don’t have to have root access to build an image. The other implications are that the fakeroot user:

• can set UID/GID ownership for files or directories they own,
• can change user:group identity with the su or sudo commands, and
• have full privileges inside the requested namespace.

The Singularity documentation discusses the implications of encryption for filesystems and networking. If you struggle to understand fakeroot, then understanding the filesystem implications might help.

A fakeroot user can neither access nor modify files and directories on the host system, where they don’t have access or rights. For example, you wouldn’t be able to access root-only files such as /usr/shadow or the host /root directory, just as a normal user cannot.

Almost everyone should be running a Linux distribution that can use this feature: Linux kernel greater than 3.8. Singularity recommends a kernel greater than 3.18.

If you don’t have root access or, for some reason, can’t use the --fakeroot option, how can you build an image? Sylabs, the company behind Singularity, has a service named Remote Builder that allows people without root or the ability to use --fakeroot to build a Singularity image. Conceptually, it’s very simple:

$ singularity build --remote output.sif /home/laytonjb/TEST/singularity.def

The --remote option tells Singularity to use Remote Builder. (You have to have an account first.) Remote Builder takes the image definition file, singularity.def, and copies it to the Remote Builder virtual machine (VM) that then creates the image. Afterward, Remote Builder copies the image back to the user’s specified location, and the VM is destroyed.

The documentation for Remote Builder seems to be a little sparse right now, but it appears that the first 30 days are free. Apparently, you are limited to 2GB images and a 30-minute build time. Presumably, you can pay to use the service longer, create larger images, and have longer build times, but you would have to contact Sylabs for more information.

The Remote Builder system offers a very cool solution to the root access requirement to build an image.

Running Containers

Running a container should be seen as a separate function from building the image (recall that a container is a running instance of an image). You can build images with Singularity or Docker or in other ways, and in some cases, other container run times can be used to “run” the container.

Singularity

A Singularity container can be run four ways: (1) as a native command, (2) with the run subcommand, (3) with the exec subcommand, or (4) with the shell subcommand. Probably the easiest way to run the container is as a native command (i.e., execute the container’s runscript):

$ ./hello-world_latest.sif

This command runs the script in the %runscript section in the container. A huge point to be made of the command is that you don’t need to be root to run the container.

The %runscript section of a Singularity container will be executed when the container is run. (You define the script in the Singularity specification file.) A simple example of a %runscript in a specification file is,

%runscript
    echo "Container was created $NOW"
    echo "Arguments received: $*"
    exec echo "$@"

which echos two lines to stdout, as well as the command-line arguments. In the command to run the container, you can pass arguments after the container name, which are used in the %runscript section.

The second way Singularity can run a container is with the run subcommand:

$ singularity run hello-world_latest.sif

This command runs the %runscript section of the container as with the previous command. To run a container from a remote hub – for example, the Singularity hub – enter:

$ singularity run shub://singularityhub/hello-world

You can also use a Docker container:

$ singularity run docker://godlovedc/lolcow

Singularity pulls the Docker image from Dockerhub and converts it to a Singularity image, pulling the Docker layers and combining them into a single layer in a final Singularity image that is then executed. Normally for a Singularity container, the %runscript section of the container is executed. However, Docker does not have this feature, so Singularity runs the script in the ENTRYPOINT of the Docker container.

The third way to run a Singularity container is with the exec subcommand, which lets you run a command different from those in %runscript:

$ singularity exec hello-world_latest.sif cat /etc/os-release

This command “executes” the container, which then runs the command cat /etc/os-release.

The fourth way to run a Singularity container is to use the shell subcommand, which starts a Singularity container and a shell in the container:

$ singularity shell lolcow.sif
 
Singularity lolcow:~> whoami
eduardo
 
Singularity lolcow:~> hostname
eduardo-laptop

Note that if you use the shell subcommand for a Docker container, once you exit the container, it disappears. The converted container is ephemeral by default unless you do something to save it.

Some important options can be used with Singularity when running containers. The first is for using GPUs. For Nvidia GPUs, add the option --nv to the command line:

$ singularity exec --nv hello-world.sif /bin/bash

For Singularity versions 3.0+, this option causes Singularity to look for Nvidia libraries on the host system and automatically bind mount them to the container.

Singularity can “bind mount” files or directories from the host system to mountpoints in the container. A best practice is for containers not to store their data inside the container (possibly making it very large). Instead, it is highly recommended that you leave the data outside the container and bind mount it to the container from the host system.

With Singularity, the bind option has the form:

--bind src[:dest[:opts]]

The options (opts) are pretty simple: ro, read-only, and rw, read-write (the default); for example:

--bind /scratch:/mnt/scratch:rw

If you need to bind mount several files or directories, you can use a comma-delimited string of bind path specifications, or you can just use a number of --bind options on the command line.

Running an encrypted Singularity image is the same as running an unencrypted image: You can use run, shell, or exec and either a passphrase, a PEM file, or an environment variable. You just use the appropriate option on the command line.

You can also use --fakeroot to run a Singularity image with the following options,

• shell
• exec
• run
• instance start
• build

which also works when Docker containers are used.

Docker

Docker is a bit different from Singularity when running images, because it has a single run subcommand with a number of options.

Before getting into the details of running a container, I should mention some preliminaries. To run a Docker container, you should be part of the docker group, which is inherently insecure because any user who can send commands to the Docker engine can escalate privilege and run root user operations. But such is the nature of Docker.

A good example of best practices for the docker run command is:

$ sudo docker run --gpus all -it --name testing \
  -u $(id -u):$(id -g) -e HOME=$HOME -e USER=$USER \
  -v $HOME:$HOME --rm nvcr.io/nvidia/digits:17.05

Running a Docker image requires root access, for which this example uses sudo.

The option --gpus, which tells Docker which GPU devices to add to the container, is followed by the option all, which tells Docker to use all of the GPUs available. Note that you don’t have to use this option if the applications in the container don’t use GPUs.

To specify a specific GPU with the device name, enter:

$ sudo docker run -it --gpus device=GPU-3a23c669-1f69-c64e-cf85-44e9b07e7a2a ...

Another option for GPUs is to call them out by device numbers, beginning with zero. A simple comma-delimited list is used to specify the devices:

$ sudo docker run -it  --gpus device=0,2

This command tells the container to use the first GPU, device=0, and the third GPU,device=2.

The -it option is really two commands that are typically used when you need interactive access in the container. The option -i tells Docker you want an interactive session. The option -t allocates a pseudo-TTY for the session. You almost always see these options together. If you don’t need interactive access to the Docker container, you don’t need this option.

When running a Docker container, it is handy if the running process has a name so you can locate the specific container in the process table. Otherwise, the container process has the same name as the container, making it very difficult to identify a specific container. The --name testing option gives the process name testing to the container. You can specify whatever you like for the name. For example, you could create a name that includes $USER, $GROUP, or both. A best practice is to always assign a process name to the container that is different from the container name.

The next options in the sample command line allow you to set the username and group:

-u $(id -u):$(id -g)

The general form of the option is:

name|uid]:[group|gid]

This best practice sets the GID and UID in the container to match those on the host.

Docker also allows you to set environment variables as part of the docker run command. For the sample command, the home directory and the username are all set in the container by environment variables:

-e HOME=$HOME -e USER=$USER

This best practice for running Docker containers sets the user’s home directory and the name in the container.

You can define container environment variables on the command line with the -e option:

-e VARIABLE='value'

VARIABLE is the environment you want to set in the container, and value is the value of the environment variable.

The -v option lets you bind mount files or directories from the host system into the container. As mentioned in the previous article, you should really never put data into your container unless you are done with the container and will be archiving it. The recommended approach is to bind mount the data from the host system to the container:

-v (host file/directory):(container mount point):(option)

The first part after the switch is the file or directory on the host, followed by a colon and the mountpoint in the container. The :(option) is an optional field to specify details about the mount, such as ro (read-only) or rw (read-write).

In the sample docker run command, the user’s home directory on the host is mounted in the container:

-v $HOME:$HOME

A definite best practice is to mount the directories you need in the container. To make things easy, I always mount my /home directory. If your home directory doesn’t contain the data you need, then a second mount option might be needed to mount the data directory into the container, for example:

-v /data/testcase1:/data

The last best practice is to use the --rm option, which automatically removes the container when it exits. It does not remove the image on which it is based. Remember that containers are merely an instance of an image. If you don’t use this command, you will still have containers hanging around, and you will have to stop and kill them.

Summary of Best Practices

The primary best practice in this article is pretty subtle but very important. As you read through best practices to build and run images and containers, you should design the container for the use cases and any goals you have in mind: Will the container be interactive? Will it run without a user logging in to it? Who will use the container? To keep all of the design goals in mind, you can create a simple text file that describes the goals and use cases for the container. Perhaps you could even copy the text file into the final version of the container.

Although this article is a bit long, it mentions some important best practices:

(1) Use a good name:tag combination for all Docker images you build.

• Don’t make it impossibly long.
• Don’t make it cryptic so no one else understands it
• If you add special characters or phrases indicating the image has been “certified” in some fashion, be sure to tell the users.
• Don’t make it too short. People won’t understand it.
• I personally like adding dates and creator initials, but not everyone does.

(2) Singularity doesn’t have the concept of an image name:tag. Unlike Docker, the container is just a file, so you can name it whatever you want, as long as it’s a legal file name. For Singularity, just be sure to use a good file name that has some evidence of when and who built the container, as well as some information about the container, and you might want to indicate that the image is encrypted. (Yes, the file name will be long.) You could even mimic the Docker name:tag combination, but without the semicolon.

(3) Use bind mounts for both Docker and Singularity! As mentioned in the first article, keeping all application data outside the container and on the host file system is highly recommended.

• For Docker, use the -v option to mount the file or directory in the container (e.g., -v /data/testcase1:/data). Note that Docker is slowly moving to the --mount option, but be assured they are going to support -v for a long time, because there are too many docker run commands floating around to just kill -v in a new version of Docker.
• For Singularity, use the --bind option (e.g., --bind /data/testcase1:data:rw).
• Both Docker and Singularity have options for mounting the file or directory in the container.

(4) Even though the --squash Docker option is still marked experimental, it has been around for years. I use it all the time; however, if it worries you, feel free not to use it.

(5) For Singularity, always encrypt your containers with the --encrypt command-line option and either a passphrase or PEM key; however, encrypt them, especially if you intend to share your container with others.

(6) For Docker, always set the UID:GID inside the container to be the same as outside the container. The run-time option is -u $(id -u):$(id -g).

(7) If you use environment variables with Docker, you are put into your /home directory when running the container, and you are the same $USER as outside the container (-e HOME=$HOME -e USER=$USER). Note that you need to mount your /home directory into the container.

(8) For Singularity, always use the --fakeroot option to build and run the container. If you are going to run a Docker image with Singularity, you can still use --fakeroot.

(9) For Singularity, remember that if the container isn’t run interactively, it will run the script in the %runscript section.

(10) If you are going to run a Docker container with Singularity and run it non-interactively, Singularity will run the script in the Docker container’s ENTRYPOINT. Be sure you have a script in that section of the container. (Docker does not have a %runscipt section.)

(11) For Docker, always assign a process name to the running container that is different from the container name, so you know exactly which container you are running.

Please take these practices into account as you create images and run containers.