Building Containers with HPC Container Maker

Containers package entire workflows, including software, libraries, and even data, into a single file. The container can then be run on any compatible hardware that can run the container type, regardless of the underlying operating system.

Containers are finding increased utility in the worlds of scientific computing, deep learning, HPC, machine learning, and artificial intelligence, because they are reproducible, portable (mobility of compute), user friendly (admins don’t have to install everything), and simple, and they isolate resources, reduce complexity (reduction in dependencies), and make it easy to distribute the application and dependencies.

Using containers, you have virtually everything you need in a single file, including a base operating system (OS), the application or workflow (multiple applications), and all of the dependencies. Sometimes the data is also included in the container, although it is not strictly necessary because you can mount filesystems with the data from the container.

To run or execute the container, you just need an OS that accommodates the specific container technology, a container run time, and a compatible hardware system. It does not have to be the same OS as in the container. For example, you could create a container using something like Docker on a Linux Mint 18.2 system and run it on a CentOS 7.2 system. Besides hardware compatibility, the only requirement is that the Linux Mint system have the ability to create a container correctly and that the CentOS 7.2 system be able to run the container correctly.

The creator of the container on the Linux Mint system includes everything needed in the container, depending on the function of the container. The container then can be distributed and executed on any system that can accommodate that container technology, has an appropriate run time, and has compatible hardware. For users, the container could have an application or a prebuilt workflow, so it can be executed. The user does not have to worry about having the tools to build everything; it has already been done for them. Moreover, the application or workflow could be optimized for performance, so the user doesn’t have to attempt to tune it.

For developers, the containers could be base systems with the basic tools for creating and building applications. The container could be something as simple as a base OS (pick your favorite distribution), with tools like compilers and libraries for building the application(s) and its dependencies.

For both users and developers, the containers can be executed as they’ve been created, or you can open a shell into the container. From that point, you can install software, build or run applications, import or export data, or do just about anything you want. Once you are done, you can save the container, which is called “extending” the container.

When you want to share the application or data, you simply give the entire container to the recipient. If their system hardware is compatible and they have the ability to run the container, they can execute the container and the applications. Theoretically, this means you can save the container and in perhaps five years run it again and get the exact same answers.

Contrast this to the classical way of distributing applications by making the source code available along with some basic instructions on how to build the application, with perhaps test cases to reproduce the output of the application. It becomes the user’s responsibility to build and execute the application. However, they might not follow the instructions correctly, or they might use a different compiler or a different set of dependencies or a slightly different CPU that could introduce differences. As a result, it could be difficult to reproduce the original application and output exactly.

Current Practice

Tools such as Environment Modules or LMOD are used in HPC systems to control your environment. They allow you to specify a certain compiler, libraries built with that compiler, and applications built using the specified compiler and libraries to build your own application. Modules have been used successfully for a number of years.

Over time, the number of versions of compilers, libraries, tools, and applications has increased dramatically, which means you will need or want to keep older versions available for a long period of time, which results in a large number of combinations leading to a combinatorial explosion of software components.

Other tools, such as EasyBuild and Spack, act sort of like package managers for HPC applications, allowing you build and install applications on HPC systems.

Neither Environment Modules, LMOD, EasyBuild, or Spack, are really designed to create and manage containers.

Spec Files

Several container technologies have quickly become an essential technology for HPC (e.g., Docker, Singularity, Charliecloud, Shifter, rkt, LXC, OpenVZ, Virtuozzo, Hyper-V containers, and Windows Server containers), all of which are in some level of use right now.

Typically, building a container requires the creation of a specification (spec) file that defines the contents of the container, including what needs to be built from source in the container. The process is not too difficult, and writing a spec file is straightforward in general. However, for users unaccustomed to building foundational libraries and tools from source, some steps could be daunting. For complicated containers that have very complex applications or workflows, the spec file could become quite large and complex itself.

These issues can be multiplied when GPU drivers, tools, and libraries are included. A container when first created usually has a combination of a GPU driver, libraries, and tools. When the container is executed, the host system’s combination of a GPU driver, libraries, and tools can be different from those in the container, which can cause problems when the container is executed – considering containers are designed to be portable.

Another concern is the separation of what goes into a container from how you specify it. When containers are created, you will select a base OS for the container, but you will likely need to add other packages. Rather than specifying the exact details of how the packages are installed in the spec file, it would be great just to specify that you want package X installed in the container.