MPI Apps with Singularity and Docker

Creating Container Specification Files

The first step in creating the container specification file is to create the HPCCM recipe, for which you can find the instructions in the HPCCM tutorial. The recipe file used in this article (Listing 1) also can be found in that tutorial.

Listing 1: poisson.py

import hpccm
 
Stage0 += baseimage(image='ubuntu:18.04')
Stage0 += pgi(eula=True, mpi=True)
 
Stage0 += copy(src='poisson_mpi.f90', dest='/var/tmp/poisson_mpi.f90')
Stage0 += shell(commands=['mpif90 /var/tmp/poisson_mpi.f90 -o /usr/local/bin/poisson_mpi'])

From the recipe, you can see that Ubuntu 18.04 is the base image and that the PGI compilers, version 19.10, are installed. Note that the PGI compilers require you to accept the end-user license agreement (EULA). (If you install the compilers on your home system, for example, you are presented the EULA to read and accept.) Also notice that I’m installing the prebuilt MPI libraries that come with the compiler.

The next line in the recipe copies the application source code into the image, and the last line is a shell directive to compile the code and put the binary in /usr/local/bin . The location of the binary was chosen arbitrarily.

Taking the poisson.py recipe in Listing 1, you can create a Singularity definition file or Dockerfile with just one command:

$ hpccm --recipe poisson.py --format singularity > Singularity.def
$ hpccm --recipe poisson.py --format docker > Dockerfile

After the specification files are created, be sure to examine them, even if you aren’t modifying anything. Alternatively, if you remove the redirection to an output file, HPCCM prints the specification file to the screen.

For Singularity, I made a small modification to the recipe file because I wanted to try pulling the PGI compilers from the local host rather than over the Internet (Listing 2). The resulting Singularity definition file is shown in Listing 3, and the Dockerfile is shown in Listing 4.

Listing 2: Modified poisson.py File

import hpccm
 
Stage0 += baseimage(image='ubuntu:18.04')
Stage0 += pgi(eula=True, tarball='/home/laytonjb/pgilinux-2019-1910-x86-64.tar.gz', mpi=True)
 
Stage0 += copy(src='poisson_mpi.f90', dest='/var/tmp/poisson_mpi.f90')
Stage0 += shell(commands=['mpif90 /var/tmp/poisson_mpi.f90 -o /usr/local/bin/poisson_mpi'])

Listing 3: Singularity Definition File

From: ubuntu:18.04
%post
    . /.singularity.d/env/10-docker*.sh
 
# PGI compiler version 19.10
%files
    /home/laytonjb/pgilinux-2019-1910-x86-64.tar /var/tmp/pgilinux-2019-1910-x86-64.tar
%post
    apt-get update -y
    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
        g++ \
        gcc \
        libnuma1 \
        openssh-client \
        perl
    rm -rf /var/lib/apt/lists/*
%post
    cd /
    mkdir -p /var/tmp/pgi && tar -x -f /var/tmp/pgilinux-2019-1910-x86-64.tar -C /var/tmp/pgi
    cd /var/tmp/pgi && PGI_ACCEPT_EULA=accept PGI_INSTALL_DIR=/opt/pgi PGI_INSTALL_MPI=true PGI_INSTALL_NVIDIA
      =true PGI_MPI_GPU_SUPPORT=true PGI_SILENT=true ./install
    echo "variable LIBRARY_PATH is environment(LIBRARY_PATH);" >> /opt/pgi/linux86-64/19.10/bin/siterc
    echo "variable library_path is default(\$if(\$LIBRARY_PATH,\$foreach(ll,\$replace(\$LIBRARY_PATH,":",), -L \$ll)));" >> /opt/pgi/linux86-64/19.10/bin/siterc
    echo "append LDLIBARGS=\$library_path;" >> /opt/pgi/linux86-64/19.10/bin/siterc
    ln -sf /usr/lib/x86_64-linux-gnu/libnuma.so.1 /opt/pgi/linux86-64/19.10/lib/libnuma.so
    ln -sf /usr/lib/x86_64-linux-gnu/libnuma.so.1 /opt/pgi/linux86-64/19.10/lib/libnuma.so.1
    rm -rf /var/tmp/pgilinux-2019-1910-x86-64.tar /var/tmp/pgi
%environment
    export LD_LIBRARY_PATH=/opt/pgi/linux86-64/19.10/mpi/openmpi-3.1.3/lib:/opt/pgi/linux86-64/19.10/lib:$LD_LIBRARY_PATH
    export PATH=/opt/pgi/linux86-64/19.10/mpi/openmpi-3.1.3/bin:/opt/pgi/linux86-64/19.10/bin:$PATH
%post
    export LD_LIBRARY_PATH=/opt/pgi/linux86-64/19.10/mpi/openmpi-3.1.3/lib:/opt/pgi/linux86-64/19.10/lib:$LD_LIBRARY_PATH
    export PATH=/opt/pgi/linux86-64/19.10/mpi/openmpi-3.1.3/bin:/opt/pgi/linux86-64/19.10/bin:$PATH
 
%files
    poisson_mpi.f90 /var/tmp/poisson_mpi.f90
 
%post
    cd /
    mpif90 /var/tmp/poisson_mpi.f90 -o /usr/local/bin/poisson_mpi

Listing 4: Dockerfile

FROM ubuntu:18.04
 
# PGI compiler version 19.10
RUN apt-get update -y && \
    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
        g++ \
        gcc \
        libnuma1 \
        openssh-client \
        perl \
        wget && \
    rm -rf /var/lib/apt/lists/*
RUN mkdir -p /var/tmp && wget -q -nc --no-check-certificate -O /var/tmp/pgi-community-linux-x64-latest.tar.gz 
--referer https://www.pgroup.com/products/community.htm?utm_source=hpccm\&utm_medium=wgt\&utm_campaign=CE\&nvi
d=nv-int-14-39155 -P /var/tmp https://www.pgroup.com/support/downloader.php?file=pgi-community-linux-x64 && \
    mkdir -p /var/tmp/pgi && tar -x -f /var/tmp/pgi-community-linux-x64-latest.tar.gz -C /var/tmp/pgi -z && \
    cd /var/tmp/pgi && PGI_ACCEPT_EULA=accept PGI_INSTALL_DIR=/opt/pgi PGI_INSTALL_MPI=true PGI_INSTALL_NVIDIA
      =true PGI_MPI_GPU_SUPPORT=true PGI_SILENT=true ./install && \
    echo "variable LIBRARY_PATH is environment(LIBRARY_PATH);" >> /opt/pgi/linux86-64/19.10/bin/siterc && \
    echo "variable library_path is default(\$if(\$LIBRARY_PATH,\$foreach(ll,\$replace(\$LIBRARY_PATH,":",), -L\$ll)));" >> /opt/pgi/linux86-64/19.10/bin/siterc && \
    echo "append LDLIBARGS=\$library_path;" >> /opt/pgi/linux86-64/19.10/bin/siterc && \
    ln -sf /usr/lib/x86_64-linux-gnu/libnuma.so.1 /opt/pgi/linux86-64/19.10/lib/libnuma.so && \
    ln -sf /usr/lib/x86_64-linux-gnu/libnuma.so.1 /opt/pgi/linux86-64/19.10/lib/libnuma.so.1 && \
    rm -rf /var/tmp/pgi-community-linux-x64-latest.tar.gz /var/tmp/pgi
ENV LD_LIBRARY_PATH=/opt/pgi/linux86-64/19.10/mpi/openmpi-3.1.3/lib:/opt/pgi/linux86-64/19.10/lib:$LD_LIBRARY_PATH \
    PATH=/opt/pgi/linux86-64/19.10/mpi/openmpi-3.1.3/bin:/opt/pgi/linux86-64/19.10/bin:$PATH
 
COPY poisson_mpi.f90 /var/tmp/poisson_mpi.f90
 
RUN mpif90 /var/tmp/poisson_mpi.f90 -o /usr/local/bin/poisson_mpi

Building the Images

I leave the details on how to build images to you. The build command for the Singularity image is:

$ sudo singularity build poisson.sif Singularity.def

Note that sudo was used instead of fakeroot , which would have allowed the image to be built as a non-privileged user. At the time of writing, I did not build fakeroot capability into Singularity. I also did not use encryption, so the steps for running MPI applications in containers would be more clear.

You can easily check whether the build was successful by listing the files in the directory (Listing 5).

Listing 5: Checking the Singularity Build

$ ls -s
total 2444376
      4 buildit.docker        4 poisson-docker.py        4 runit.docker
      4 buildit.sing         20 poisson_mpi.f90          4 runit.sing
      4 Dockerfile            4 poisson_mpi.txt          4 Singularity.def
      4 hosts                 4 poisson.py              16 summary-notes.txt
      4 ompi.env        2444296 poisson.sif

The build command for the Docker image is:

$ sudo docker build -t poisson -f Dockerfile .

Notice that a version was not used with the image tag (the image is named poisson ).To check whether the build was successful, display a list of images on the host system (Listing 6).

Listing 6: Checking the Docker Build

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
poisson             latest              0a7e2fad652e        2 hours ago         9.83GB
                                        49cbd14ae32f        3 hours ago         269MB
ubuntu              18.04               72300a873c2c        3 weeks ago         64.2MB
hello-world         latest              fce289e99eb9        14 months ago       1.84kB