Under the Module File Hood

Everything works just great with Lmod so far. Modules can be loaded, unloaded, deleted, purged, and so on. However, without good module files, Lmod would execute whatever commands you put in the file, which could cause problems. To understand what is happening with module files, the GCC 8.1 compiler module (it, too, is written in Lua) is shown in Listing 4.

-- -*- lua -*-
------------------------------------------------------------------------
-- GCC 8.1 compilers - gcc, g++, and gfortran. (Version 8.1)
------------------------------------------------------------------------
help(
[[
This module loads the gcc-8.1.0 compilers (8.1.0). The
following additional environment variables are defined:
CC   (path to gcc compiler wrapper      )
CXX  (path to g++ compiler wrapper      )
F77  (path to gfortran compiler wrapper )
F90  (path to gfortran compiler wrapper )
FC   (path to gfortran compiler wrapper )
See the man pages for gcc, g++, gfortran (f77, f90). For
more detailed information on available compiler options and
command-line syntax.
]])
-- Local variables
local version = "8.1"
local base = "/home/laytonjb/bin/gcc-8.1.0/"
-- Whatis description
whatis("Description: GCC 8.1.0 compilers")
whatis("URL: www.gnu.org")
-- Take care of $PATH, $LD_LIBRARY_PATH, $MANPATH
prepend_path("PATH", pathJoin(base,"bin"))
prepend_path("PATH", pathJoin(base,"sbin"))
prepend_path("PATH", pathJoin(base,"include"))
prepend_path("LD_LIBRARY_PATH", pathJoin(base,"lib"))
prepend_path("LD_LIBRARY_PATH", pathJoin(base,"lib64"))
prepend_path("MANPATH", pathJoin(base,"share/man"))
-- Environment Variables
pushenv("CC", pathJoin(base,"bin","gcc"))
pushenv("CXX", pathJoin(base,"bin","g++"))
pushenv("F77", pathJoin(base,"bin","gfortran"))
pushenv("FORT", pathJoin(base,"bin","gfortran"))
pushenv("cc", pathJoin(base,"bin","gcc"))
pushenv("cxx", pathJoin(base,"bin","g++"))
pushenv("f77", pathJoin(base,"bin","gfortran"))
pushenv("fort", pathJoin(base,"bin","gfortran"))
pushenv("FC", pathJoin(base,"bin","gfortran"))
pushenv("fc", pathJoin(base,"bin","gfortran"))
-- Setup Modulepath for packages built by this compiler
local mroot = os.getenv("MODULEPATH_ROOT")
local mdir = pathJoin(mroot,"compiler/gcc", version)
prepend_path("MODULEPATH", mdir)
-- Set family for this module
family("compiler")

The GCC 8.1 compilers were installed in my home directory on my laptop, so I am not too worried about where exactly the compilers are installed. However, for clusters, I would install them on an NFS-shared filesystem, such as /usr/local/ or /opt/. Installing them individually on each node is ripe for problems.

The module file can be broken down into several sections. The first part of the file is the help function, which is printed to stdout when you ask for module help. The next section defines the major environment variables $PATH, $LD_LIBRARY_PATH, and $MANPATH. Notice that the function prepend_path is used to put the compiler "first" in these environment variables.

The third major section of the module file is where the specific environment variables for the compiler are defined. For this module, the variables are pretty straightforward: CC, cc, f77, F77, and so on. These variables are specific to the compiler and are defined with the pushenv function, which pushes the variables into the environment. It also uses the pathJoin function, which helps creates the correct paths for these variables.

The last section is key to Lmod, with the definition of two environment variables: $MODULEPATH and $MODULEPATH_ROOT. The line

local mdir = pathJoin(mroot,"compiler/gcc", version)

creates a local variable named mdir, which is a concatenation of the mroot variable ($MODULEPATH_ROOT) and compiler/gcc. It tells Lmod that subsequent module avail commands should look at the compiler/gcc subdirectory under the main module directory corresponding to the compilers just loaded (gcc/8.1). As the writer of the modules, you control where the module files that depend on the compilers are located. This step is the key to module hierarchy. You can control what modules are subsequently available by manipulating the mdir variable. This Lmod attribute gives you great flexibility.

The very last line in the module file, the statement family("compiler"), although optional, simplifies everything for users (i.e., it is a best practice). The function family tells Lmod to which family the module belongs. A user can only have one module per family loaded at a time. In this case, the family is compiler, so that means no other compilers can be loaded. (You would hope all other compiler modules also use this family statement.) Adding this line helps users prevent self-inflicted problems. Even though the statement is somewhat optional, I highly recommend using it.

If the GCC 8.1 compiler is loaded, then the diagram of the module layout should look something like Figure 2. The green labels indicate the compiler module that is loaded. The red labels indicate the path to the modules that depend on it (the MPI modules). Note that the MPI modules are under the compiler directory, because they depend on the compiler module that is loaded.

Figure 2: Active path after gcc/8.1 is loaded.

In the previous section, I loaded the mpich/3.2 module associated with the GCC 8.1 compiler. Listing 5 for the mpich/3.2 module file was built with the GCC 8.1 compiler.

-- -*- lua -*-
------------------------------------------------------------------------
-- mpich-3.2 (3.2.1) support. Built with gcc-8.1 (8.1.0)
------------------------------------------------------------------------
help(
[[
This module loads the mpich-3.2 MPI library built with gcc-8.1.
compilers (8.1.0). It updates the PATH, LD_LIBRARY_PATH,
and MANPATH environment variables to access the tools for
building MPI applications using MPICH, libraries, and
available man pages, respectively.
This was built using the GCC compilers, version 8.1.0.
The following additional environment variables are also defined:
MPICC   (path to mpicc compiler wrapper   )
MPICXX  (path to mpicxx compiler wrapper  )
MPIF77  (path to mpif77 compiler wrapper  )
MPIF90  (path to mpif90 compiler wrapper  )
MPIFORT (path to mpifort compiler wrapper )
See the man pages for mpicc, mpicxx, mpif77, and mpif90. For
more detailed information on available compiler options and
command-line syntax. Also see the man pages for mpirun or
mpiexec on executing MPI applications.
]])
-- Local variables
local version = "3.2"
local base = "/home/laytonjb/bin/gcc-8.1-mpich-3.2.1"
-- Whatis description
whatis("Description: MPICH-3.2 with GNU 8.1 compilers")
whatis("URL: www.mpich.org")
-- Take care of $PATH, $LD_LIBRARY_PATH, $MANPATH
prepend_path("PATH", pathJoin(base,"bin"))
prepend_path("PATH", pathJoin(base,"include"))
prepend_path("LD_LIBRARY_PATH", pathJoin(base,"lib"))
prepend_path("MANPATH", pathJoin(base,"share/man"))
-- Environment Variables
pushenv("MPICC", pathJoin(base,"bin","mpicc"))
pushenv("MPICXX", pathJoin(base,"bin","mpic++"))
pushenv("MPIF90", pathJoin(base,"bin","mpif90"))
pushenv("MPIF77", pathJoin(base,"bin","mpif77"))
pushenv("MPIFORT", pathJoin(base,"bin","mpifort"))
pushenv("mpicc", pathJoin(base,"bin","mpicc"))
pushenv("mpicxx", pathJoin(base,"bin","mpic++"))
pushenv("mpif90", pathJoin(base,"bin","mpif90"))
pushenv("mpif77", pathJoin(base,"bin","mpif77"))
pushenv("mpifort", pathJoin(base,"bin","mpifort"))
-- Setup Modulepath for packages built by this compiler/mpi
local mroot = os.getenv("MODULEPATH_ROOT")
local mdir = pathJoin(mroot,"mpi/gcc", "8.1","mpich","3.2")
prepend_path("MODULEPATH", mdir)
-- Set family for this module (mpi)
family("mpi")

If you compare this module file to the compiler module file, you will see many similarities. The classic environment variables, $PATH, $LD_LIBRARY_PATH, and $MANPATH, are modified and certain environment variables are defined. Because you want the MPI tools that are associated with the module to be "first" in $PATH, the Lmod module command prepend_path is used again.

Toward the end of the file, examine the code for Modulepath. The local variable mdir points to the "new" module subdirectory, which is mpi/gcc/8.1/mpich/3.2. (Technically, the full path is /usr/local/modulefiles/mpi/gcc/8.1/mpich/3.2, because $MODULEPATH_ROOT is /usr/local/modulefiles.) In this subdirectory, you should place all modules that point to tools that have been built with both the gcc/8.1 compilers and the mpich/3.2 tools. Examples of module files that depend on both a compiler and an MPI tool are applications or libraries such as PETSc. Although not shown here, it is not too difficult to extend the MPI module file to depend on both a compiler and an MPI library.

Also notice that the mpich/3.2 module uses the family() function so that the user cannot load a second MPI module. You could even have a family() function for libraries such as PETSc.

Module Usage for the Admin

In an article [9] from a couple of years ago, I presented a way to gather logs about Tcl/C Environment Modules usage. It was a bit of a kludge, but it did allow me to gather data. Lmod brings this ability to the forefront.

Tracking module usage is conceptually fairly easy, but a number of steps are involved. Having this information can be amazingly important, because it allows you to track which tools are used the most. (I associate one tool with one module.) If you have various versions of a specific tool, it allows you to track the usage of each, so you can either deprecate an older version or justify keeping it around and maintaining it. You can also see which modules are used as a function of time, which helps you understand when people run their jobs and what modules they use.

Summary

Although I have written about Lmod before, I continue to come back to it, because it is so useful. It greatly helps users sort out their environment so that they do not accidentally load conflicting libraries and tools. The first time you have to debug a user's code when they have mixed MPI implementations, you will be thankful for Lmod.

Environment Modules in general, and Lmod specifically, allow you to keep multiple versions of the same package on a system to service applications that have been built with older versions of a compiler, MPI, or library, or even old libraries that are needed. I even saw a somewhat recent posting [10] to the Open MPI mailing list asking about LAM-MPI, even though it basically has been dead for a decade. You would be surprised how long applications stick around and bring their dependencies with them.

Because Lmod can read Tcl/C module files in addition to Lua (the preferred language), you can move easily from Tcl/C Environment Modules to Lmod. As you can see from the Lua module file examples here, the syntax is very clean and simple, making them very easy to read.

Finally, Lmod is developing tools that allow you to collect module usage and put it into a database that you can mine – which is very cool stuff, indeed.