The past two articles in this newsletter have been about containers and HPC, and I would like to write one more article in that vein. In the HPC Containers Survey, HPC Container Maker (HPCCM) was mentioned a couple of times. I wrote about HPCCM in an article on best practices for containers and in another article about building containers, but I want to address the container spec files that are used to build container images with Docker and Podman.

HPCCM is one of those tools you didn’t know you needed, until you needed it. Building a container image is not difficult because, in the case of a Docker image, you can start with an existing image, install some packages or build code, and then save a running container to a new image.

If you want to get fancier and have a container image that builds an application, but you don’t want to save all of the build tools because the image would be HUGE, or if you want to build an image that, when invoked, has all kinds of entry points that can’t be used in complicated ways, then spec files can get long and complicated. On the other hand, you might want to build container images for Docker and Singularity, which means you would need to know both spec file formats. All of these issues can be addressed by HPCCM.

The tool is written in Python, which allows you to create a simple “recipe” for your container image and then turn that recipe into a container specification file for either Docker or Singularity/Apptainer. You can write just a few lines of Python and create hundreds of lines of directions in the specification file. Moreover, this recipe is much easier to maintain than a long specification file, and because it’s Python, you can use any Python code or tools as part of the recipe.

The recipe uses “building blocks” for capabilities such as compilers, libraries, and applications. The container options for building these components also contain some best practices. A very powerful feature of HPCCM is that it makes multistage spec files extremely easy to create. An example of a multistage build has an application in the container image, but not the build tools, leaving just the runtime. HPCCM allows you to install the tools, build the application, and then remove the build tools, except for the necessary runtime components.

In this article, I focus on creating a simple recipe for a development container image that contains a compiler set, as well as a Message Passing Interface (MPI) library. Moreover, I want to take the resulting Dockerfile that HPCCM creates and use Docker and Podman to build the final container image.

Development Container

One of the better ways to use a container is as a development environment for building applications. You can use whatever operating system (OS) you like as your base for the container image; add in whatever development tools you like, including compilers, libraries, and tools; and create a container image. You can then run that image, mount your home directory into the running container, giving you access to your source code, and start to create and edit the code and build and test your application: in other words, a development environment.

However, installing all the development tools you want into the container image can be somewhat of a pain. Your container spec file can grow very quickly, making editing it a bit challenging, to say the least. I’ll show you how to use HPCCM to make the creation and maintenance of the spec file easier.

One quick point to make is that unless you know how to run a container to allow X applications (GUI applications) run in the container and display on the host, then you probably won’t install GUI editors or other tools in the container. You will run those on the host, save the code, but build (compile) the code in the container. Although you can’t just compile with your integrated development environment (IDE) tool, you can just switch to the running container terminal window and build the code with Make, CMake, or another build tool.

Note that nothing stops you from installing GUI applications in the container, but in this article, I do not explain how to run them from inside a container. Otherwise, feel free to install your favorite ASCII-based editors such as Vi, Emacs, or something else.

For the sake of argument, I’ll start with Ubuntu 24.04 as the base OS. I want to install the GNU compiler chain for the latest versions of C, C++, and Fortran. Additionally, I want to install the latest released version of Open MPI. To be complete, I will also install Vi (Vim), because the base container doesn’t include it, and it is the text-based editor I use the most.

One more note about container “design” for development environments: You could install multiple libraries and use environment modules within the container to switch between them, but that’s not really the point of a simple development environment. The same is true for multiple compilers. Just make a container for a single combination of compiler and libraries. If you want more options, just create a container for the specific combinations you want. Remember, with HPCCM, you maintain a simple recipe (Python script) for the specific container.

I asked Scott McMillan, the HPCCM developer, if he had any tips on how to maintain the various recipes, and he had a good suggestion: You can take recipes for different but related containers and combine them into a single file so you can specify a compiler of library on the command line and create a container that matches the command-line interface (CLI) options. In his following example recipe, you can chose to build with Open MPI or MPICH:

Stage0 += baseimage(image='ubuntu:24.04')
Stage0 += gnu(version='14')
 
if USERARG.get('mpi') == 'mpich':
    Stage0 += mpich()
else:
    Stage0 += openmpi()
# end if

Then, you invoke HPCCM with the command:

$ hpccm --recipe recipe.py --userarg mpi=mpich --format Docker > Dockerfile

The sample recipe might not work as written; it is just a sample. You could also use USERARGs to call a single recipe to create a customized container. Don’t forget that an HPCCM recipe is just a Python script, so you could use all kinds of Python tools in the recipe.

dev1 – Base OS

I like to build up elements in steps starting with the smallest, the OS. In the following recipe, I create my base container with Ubuntu 24.04 and add Vim to the mix:

Stage0 += baseimage(image='ubuntu:24.04')
Stage0 += apt_get(ospackages=['vim'])

Note that I used apt-get to install vim and that I specified the packages in a Python list. This allows you to specify several packages in a single command in the recipe.

Note that both apt-get and yum are available as building blocks in HPCCM for installing packages. If you need some other package manager, you can use the code for either an existing package manager to create your own or open an issue on the HPCCM GitHub site. You could also use the generic_build building block to compile what you need from scratch.

The HPCCM syntax is fairly easy to understand and is explained in the documentation. The syntax allows you to create a multistage build if you want (not covered here) with the use of “stages.” With building blocks, you add the components you want to the base object (Stage0), where a baseimage has been specified.

For the sake of argument, I create a Dockerfile spec file from this recipe,

$ hpccm --recipe dev1.py --format docker > Dockerfile

with the resulting Dockerfile:

FROM ubuntu:24.04
RUN apt-get update -y && \
  DEBIAN_FRONTEND=noninteractive \
  apt-get install -y --no-install-recommends vim && \
  rm -rf /var/lib/apt/lists/*

The command to build the Docker container image is:

$ docker build -f Dockerfile -t ubuntu-24.04-dev1 .

After a bit, the container is done building and can be seen in the list of Docker container images:

$ docker images
REPOSITORY          TAG       IMAGE ID       CREATED          SIZE
ubuntu-24.04-dev1   latest    2358ab17d70c   13 minutes ago   149MB

Note that the image is only 149MB, which is pretty small compared with the multigigbyte installations on desktops and laptops.

To confirm that the container image works and that Vi is present, create an interactive container:

$ docker run --rm -it \
  -v /home/laytonjb:/home/laytonjb \
  --user $(id -u):$(id -g) \
  --workdir=/home/laytonjb ubuntu-24.04-dev1 /bin/bash

I should clarify something about this command: The part of the command that reads --user $(id -u):$(id -g) takes the user and group identifiers (UID and GID) from your account and uses them in the container. This step helps a great deal with manipulating files. If you didn’t take this step, you would get a strange UID and GID on the files that were created. Believe me, taking this step makes life much simpler in the container world.

Note that I mount my home directory, /home/laytonjb, in the container as /home/laytonjb, which is what you want when creating a development environment, because all your source trees appear in the container if you mount the host directory into the container. Moreover, when the container starts, it changes directory to /home/laytonjb with the option --workdir=/home/laytonjb.

I check that Vi is in the container, indicating that the container was built successfully:

ubuntu@5e8f3323fb36:/home/laytonjb$ whereis vi
vi: /usr/bin/vi
ubuntu@5e8f3323fb36:/home/laytonjb$ pwd
/home/laytonjb

This container image appears to be a good start toward creating a development environment.

dev2 – Add Compilers

The container image just built is ready for adding compilers or whatever tools you need to develop applications. You have several options, and with HPCCM, you can install common compiler suites including GNU, NVIDIA, and Flang compilers. You could also use the generic_build building block if you are building from source or using a package manager building block to install them. For this example, I install version 14 of the C, C++, and Fortran GNU compilers.

The HPCCM building block is simply named gnu. I add this to the recipe from the previous section with one small exception: Rather than start with the base Ubuntu 24.04, I start with the dev1 container image from the previous section because it has everything I want. The HPCCM recipe I used is:

Stage0 += baseimage(image='ubuntu-24.04-dev1')
Stage0 += apt_get(ospackages=['vim'])
compiler = gnu(version='14')
Stage0 += compiler

I chose to add the latest GNU compilers to the container image, so I had to specify the version number (14).

The command to take the recipe file and convert it to a Dockerfile is:

$ hpccm --recipe dev2.py --format docker > Dockerfile

The resulting Dockerfile is shown in Listing 1. Now you can see where an HPCCM recipe is shorter than a Dockerfile.

Listing 1: dev2 Dockerfile

FROM ubuntu-24.04-dev1
 
RUN apt-get update -y && \
  DEBIAN_FRONTEND=noninteractive \
  apt-get install -y --no-install-recommends vim && \
  rm -rf /var/lib/apt/lists/*
 
# GNU compiler
RUN apt-get update -y && \
  DEBIAN_FRONTEND=noninteractive \
  apt-get install -y --no-install-recommends g++-14 gcc-14 gfortran-14 && \
  rm -rf /var/lib/apt/lists/*
RUN update-alternatives \
  --install /usr/bin/g++ g++ $(which g++-14) 30 && \
  update-alternatives --install /usr/bin/gcc gcc $(which gcc-14) 30 && \
  update-alternatives --install /usr/bin/gcov gcov $(which gcov-14) 30 && \
  update-alternatives --install /usr/bin/gfortran gfortran $(which gfortran-14) 30

The container image built just fine:

$ docker images
REPOSITORY          TAG       IMAGE ID       CREATED          SIZE
ubuntu-24.04-dev2   latest    8f311b6fbb1e   9 seconds ago    1.82GB
ubuntu-24.04-dev1   latest    2358ab17d70c   35 minutes ago   149MB

Notice the very large jump in the size of the container image.

To prove the container image was built correctly, run and check it. To run the container, I used the following command (virtually identical to the previous one):

$ docker run --rm -it \
  -v /home/laytonjb:/home/laytonjb \
  --user $(id -u):$(id -g) \
  --workdir=/home/laytonjb ubuntu-24.04-dev2 /bin/bash

Once in the container, I checked whether the compilers were there, starting with GCC, then GFortran (the most important building block) and G++ (Listing 2).

Listing 2: Checking for Compilers

ubuntu@e45200a49bf1:/home/laytonjb$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-linux-gnu/14/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 14-20240412-0ubuntu1' 
...
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 14.0.1 20240412 (experimental) [master r14-9935-g67e1433a94f] (Ubuntu 14-20240412-0ubuntu1) 
 
ubuntu@e45200a49bf1:/home/laytonjb$ gfortran -v
Using built-in specs.
COLLECT_GCC=gfortran
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-linux-gnu/14/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 14-20240412-0ubuntu1' 
...
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 14.0.1 20240412 (experimental) [master r14-9935-g67e1433a94f] (Ubuntu 14-20240412-0ubuntu1)
 
ubuntu@e45200a49bf1:/home/laytonjb$ g++ -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-linux-gnu/14/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 14-20240412-0ubuntu1' 
...
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 14.0.1 20240412 (experimental) [master r14-9935-g67e1433a94f] (Ubuntu 14-20240412-0ubuntu1)

dev3 – Adding Open MPI

The last step is to install Open MPI with an HPCCM building block that adds it to the Dockerfile (or Singularity spec file):

Stage0 += baseimage(image='ubuntu:24.04')
Stage0 += apt_get(ospackages=['vim'])
compiler = gnu(version='14')
Stage0 += compiler
Stage0 += openmpi(cuda=False, environment=True, ldconfig=True, infiniband=False, prefix='/usr/local/openmpi', version='4.0.3')

For this containment image, I went back to the starting Ubuntu base image. I could have easily started with the dev2 container image I had just built.

For Open MPI, I specified a few things, starting with the version (4.0.3, not the latest one); then, I specified that I did not want to build in CUDA (the test system doesn’t have a discrete GPU) or InfiniBand support (it’s only TCP in the home lab) and that I wanted to install Open MPI in /usr/local/openmpi. I also specified environment=True, which specifies that LD_LIBRARY_PATH and PATH should be modified to include Open MPI. This step allows you NOT to have to specify the fully qualified path to the binaries, libraries, and include files. Finally, I set ldconfig=True, which specifies that the Open MPI library directory should be added to the dynamic liner cache, which can help with compiling and is something I like to include.

Now I can create the Dockerfile from the HPCCM recipe:

$ hpccm --recipe dev3.py --format docker > Dockerfile

The resulting Dockerfile is shown in Listing 3. The obvious benefit of a very short HPCCM recipe file versus this much longer Dockerfile is obvious.

Listing 3: dev3 Dockerfile

FROM ubuntu:24.04
 
RUN apt-get update -y && \
  DEBIAN_FRONTEND=noninteractive \
  apt-get install -y --no-install-recommends vim && \
  rm -rf /var/lib/apt/lists/*
 
# GNU compiler
RUN apt-get update -y && \
  DEBIAN_FRONTEND=noninteractive \
  apt-get install -y --no-install-recommends \
        g++-14 \
        gcc-14 \
        gfortran-14 && \
    rm -rf /var/lib/apt/lists/*
RUN update-alternatives --install /usr/bin/g++ g++ $(which g++-14) 30 && \
    update-alternatives --install /usr/bin/gcc gcc $(which gcc-14) 30 && \
    update-alternatives --install /usr/bin/gcov gcov $(which gcov-14) 30 && \
    update-alternatives --install /usr/bin/gfortran gfortran $(which gfortran-14) 30
 
# OpenMPI version 4.0.3
RUN apt-get update -y && \
    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
        bzip2 \
        file \
        hwloc \
        libnuma-dev \
        make \
        openssh-client \
        perl \
        tar \
        wget && \
    rm -rf /var/lib/apt/lists/*
RUN mkdir -p /var/tmp && wget -q -nc --no-check-certificate -P /var/tmp https://www.open-mpi.org/software/ompi/v4.0/dow
nloads/openmpi-4.0.3.tar.bz2 && \
    mkdir -p /var/tmp && tar -x -f /var/tmp/openmpi-4.0.3.tar.bz2 -C /var/tmp -j && \
    cd /var/tmp/openmpi-4.0.3 &&   ./configure --prefix=/usr/local/openmpi --disable-getpwuid --enable-orterun-prefix-b
y-default --without-cuda --without-verbs && \
    make -j$(nproc) && \
    make -j$(nproc) install && \
    echo "/usr/local/openmpi/lib" >> /etc/ld.so.conf.d/hpccm.conf && ldconfig && \
    rm -rf /var/tmp/openmpi-4.0.3 /var/tmp/openmpi-4.0.3.tar.bz2
ENV PATH=/usr/local/openmpi/bin:$PATH

Next, I built the container with the command:

$ docker build -f Dockerfile -t ubuntu-24.04-dev3 .

After the build finished, I checked the container images:

$ docker images
REPOSITORY          TAG       IMAGE ID       CREATED         SIZE
ubuntu-24.04-dev3   latest    c6c30ca1386d   3 minutes ago   1.91GB
ubuntu-24.04-dev2   latest    8f311b6fbb1e   17 hours ago    1.82GB
ubuntu-24.04-dev1   latest    2358ab17d70c   17 hours ago    149MB

Interestingly, adding Open MPI didn’t add that much to the size of the container image. The proof is in the pudding, so time run the container:

$ docker run --rm -it \
  -v /home/laytonjb:/home/laytonjb \
  --user $(id -u):$(id -g) \
  --workdir=/home/laytonjb ubuntu-24.04-dev3 /bin/bash

After starting the container, I then check on Open MPI:

Listing 4: Checking Open MPI

ubuntu@4255aefc4a94:/home/laytonjb$ which mpirun
/usr/local/openmpi/bin/mpirun
ubuntu@4255aefc4a94:/home/laytonjb$ mpicc -v
Using built-in specs.
COLLECT_GCC=/usr/bin/gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-linux-gnu/14/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 14-20240412-0ubuntu1'
...
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 14.0.1 20240412 (experimental) [master r14-9935-g67e1433a94f] (Ubuntu 14-20240412-0ubuntu1) 

Everything looks good: mpirun is there and mpicc points to gcc-14.0.1 (the host system is Ubuntu 22.04, for which gcc-14 does not exist, so you know it is pointing to the correct gcc).

Podman – dev1

I wanted to gain some more Podman experience after my last article, so I tried building a container image with Podman to match the ones I built with Docker. I started the experiments with dev1, which is just the base image with Vi. You’ll find that most of the Podman commands are almost identical to Docker.

For the Podman experiments, I used the exact same Dockerfiles I used for the Docker container images. I made no changes – and I shouldn’t have to according to the Podman documentation I've read so far.

The first step is to build the Podman container image from the dev1 Dockerfile:

$ podman build -f Dockerfile -t ubuntu-24.04-dev1 .

Note that the Podman container images are different from the Docker container images, so they will not conflict. The list of Podman images is:

$ podman images
REPOSITORY                   TAG         IMAGE ID      CREATED         SIZE
localhost/ubuntu-24.04-dev1  latest      0797eec2258f  55 seconds ago  154 MB

Next, I run the container and check the OS release (Listing 5). Looks good to me! +1 for Podman.

Listing 5: Checking the OS Release

$ podman run --rm -it -v /home/laytonjb:/home/laytonjb \
  --user $(id -u):$(id -g) \
  --workdir=/home/laytonjb ubuntu-24.04-dev1 /bin/bash
ubuntu@fa9e0b5b0895:/home/laytonjb$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 24.04 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

Podman – dev2

Next, I’ll build the dev2 container image that has the added compilers:

$ podman build -f Dockerfile -t ubuntu-24.04-dev2 .

Note that this command looks exactly like the Docker command. The Podman container image list is now:

$ podman images
REPOSITORY                   TAG         IMAGE ID      CREATED             SIZE
localhost/ubuntu-24.04-dev2  latest      3604d85bbd50  About a minute ago  1.83 GB
localhost/ubuntu-24.04-dev1  latest      0797eec2258f  8 minutes ago       154 MB

Running the container and checking the compilers produces the same output as for the Docker container. I have not included it here to reduce the length of the article. I built this container image exactly like the Docker container image with the dev1 starting image.

Podman -- dev3

The dev3 container image added Open MPI. I used the exact same Dockerfile as the Docker container image, but this time, Podman failed to build the container. I kept getting errors about not being able to change permissions on files when untarring the openmpi TAR file. My Google searching did not point to anything solid, but it may have something to do with SELinux. My first wart with Podman.

Summary

In this article I showed you simple ways to use HPCCM to build a development container image. The HPCCM recipes are simple but create powerful container spec files. The HPCCM recipe is just Python, which allows you to use Python for other code and use Python tools in your recipe.

Docker had no trouble with the Docker spec files. (I didn’t test Singularity or Apptainer in this article in the interest of length.) Podman worked just fine on the first two examples but couldn’t handle the last attempt when the MPI library was added.

If you read through the HPCCM documentation, you will see other really cool things you can do with HPCCM. For example, you can add multiarchitecture support in a recipe. You can use code version tagging for better reproducibility by using a specific commit or tags in GitHub. I also learned a bit more about the generic_build building block. It looks very powerful for building applications in your container image.

If you have not heard of HPCCM, give it a try. I have been using it for a few years and have found it very useful.