New Release of Lmod Environment Modules System

Lmod Hierarchical Modules

One of the key capabilities of Lmod is module hierarchy. With TCL/C modules, you can load pretty much any modules you want, even if they are not compatible, whereas Lmod doesn't allow you to see all possible modules, so you might not be aware of what is available. However, the Lmod module spider command lets you see all modules. Figure 1 illustrates the module hierarchy of the module files for one of my systems.

Figure 1: Example module file layout.

At the top of the diagram, /cluster/modulefiles is the directory for all module files. Below this directory are three main subdirectories: Core , compiler , and mpi . These directories indicate the dependencies of the various modules. For example, everything in the compiler directory depends on a specific compiler (e.g., GCC 4.8). Everything in the mpi directory is dependent on a specific MPI and compiler combination.

By default, Lmod reads module files in /cluster/modulefiles/Core , so a best practice is to put any module files in this directory that do not depend on either a compiler or an MPI. This means you also put the compiler module files in the Core directory.

The gnu subdirectory under Core is where all of the module files for the GNU family of compilers are stored. A best practice from the developer of Lmod, Dr. Robert McLay at the University of Texas Advanced Computing Center (TACC), is to make all subdirectories beneath Core , compiler , and mpi lowercase. In McLay’s own words, “Lmod is designed to be easy to use interactively and be easy to type. So I like lower case names where ever possible.” He continues: “I know of some sites that try very hard to match the case of the software: OpenMPI, PETSc, etc. All I can say is that I’m glad I don’t have to work on those systems.”

In the gnu subdirectory is a module file named 4.8.lua , which in Figure 1 is labeled 4.8 (f) , with the (f) meaning that it is a file and not a directory. This file contains the details for version 4.8 of the GNU compilers. The extension .lua , although not shown in Figure 1, indicates that the module file is written in Lua; however, as I mentioned before, it could be written in TCL.

If you had version 4.9 and 5.1 of the GNU compilers, you would have module files named 4.9.lua and 5.1.lua in the /cluster/modulefiles/Core/gnu directory. If you had a different set of compilers (e.g., those from PGI), you would create a directory under /cluster/modulefiles named pgi (i.e., /cluster/modulefiles/Core/pgi )and then place the module files in this subdirectory.

You can find what modules are available in Core with the avail command:

[laytonjb@home4 ~]$ module avail
 
-------------------------- /cluster/modulefiles/Core ---------------------------
   gnu/4.8    lmod/6.0.1    settarg/6.0.1
 
Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching
any of the "keys".

For the example module file layout, you see the compiler module (only one in this example) called gnu/4.8 . Because it's the only one available, you can go ahead and load it:

[laytonjb@home4 ~]$ module load gnu/4.8
[laytonjb@home4 ~]$ module list
 
Currently Loaded Modules:
  1) gnu/4.8

The compiler module files modify the Lmod environment variables to point to the MPI libraries so that only the MPI tools that depend on the loaded compiler are available to the user. That is because everything in the mpi subdirectory depends on having a compiler loaded. Now if you type module avail , you get the following response:

[laytonjb@home4 ~]$ module avail
 
-------------------- /cluster/modulefiles/compiler/gnu/4.8 ---------------------
   mpich/3.1    openmpi/1.8
 
-------------------------- /cluster/modulefiles/Core ---------------------------
   gnu/4.8    lmod/6.0.1    settarg/6.0.1
 
Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching
any of the "keys".

Notice that Lmod now lists the two MPI libraries that have been built with the compiler you loaded. Now you can load one of the MPI libraries. For example, load MPICH and then see which MPI scripts are in $PATH .

[laytonjb@home4 ~]$ module load mpich/3.1
[laytonjb@home4 ~]$ module list
 
Currently Loaded Modules:
  1) gnu/4.8   2) mpich/3.1
 
[laytonjb@home4 ~]$ which mpicc
/cluster/mpich-3.1.4-gnu-4.8.2/bin/mpicc
[laytonjb@home4 ~]$ which mpif77
/cluster/mpich-3.1.4-gnu-4.8.2/bin/mpif77

Notice that mpicc and mpif77 point to the correct scripts.

An important key to making everything work is in the module files. To better understand these module files, I’ll take a deeper look at them.

Under the Module File Hood

Everything works just great with Lmod so far. I can load, unload, delete, and purge modules and so on. Lmod handles everything quite well; however, without good module files, Lmod would blindly execute your commands, which could cause problems. To understand what's happening with the module files, I’ll take a look at the gnu/4.8 compiler module:

-- -*- lua -*-
------------------------------------------------------------------------
-- GNU 4.8.2 compilers - gcc, g++, and gfortran. (Version 4.8.2)
------------------------------------------------------------------------
 
 
help(
[[
This module loads the gcc-4.8 compilers (4.8.2). The 
following additional environment variables are defined:
 
CC   (path to gcc compiler wrapper      )
CXX  (path to g++ compiler wrapper      )
F77  (path to gfortran compiler wrapper )
F90  (path to gfortran compiler wrapper )
 
See the man pages for gcc, g++, gfortran (f77, f90). For 
more detailed information on available compiler options and 
command-line syntax.
]])     
 
 
-- Local variables
local version = "4.8"
local base = "/usr/bin/"
 
 
-- Whatis description
whatis("Description: GNU 4.8 compilers (4.8.2)")
whatis("URL: www.gnu.org")
 
 
-- Normally - Take care of $PATH, $LD_LIBRARY_PATH, $MANPATH
-- Don't add "/usr/bin" to path since it will get removed if module
-- is removed
 
 
-- Environment Variables
pushenv("CC", pathJoin(base,"gcc"))
pushenv("FC", pathJoin(base,"gfortran"))
pushenv("CXX", pathJoin(base,"g++"))
pushenv("F90", pathJoin(base,"gfortran"))
pushenv("cc", pathJoin(base,"gcc"))
pushenv("fc", pathJoin(base,"gfortran"))
pushenv("cxx", pathJoin(base,"g++"))
pushenv("f90", pathJoin(base,"gfortran"))
 
 
-- Setup Modulepath for packages built by this compiler
local mroot = os.getenv("MODULEPATH_ROOT")
local mdir = pathJoin(mroot,"compiler/gnu", version)
prepend_path("MODULEPATH", mdir)
 
 
-- Set family for this module
family("compiler")

For this module file (recall that it’s written in Lua), I use the standard GNU compilers that came with the distribution. Therefore they are installed in /usr/bin and /usr/lib – basically the standard $PATH .

The module file can be broken down into several sections. The first part of the file is the help function, which is what is printed to stdout when you ask for help with the module. The next major section is where the environment variables are defined. For this module, the variables are pretty straightforward: CC , cc , fc , FC , and so on.

Notice that I didn't define any changes to $PATH or $LD_LIBRARY_PATH . If I had, then if I were to unload the module, it would erase parts of $PATH , such as /usr/bin . This isn't a good idea, so I didn't modify $PATH , $LD_LIBRARY_PATH , or $MANPATH . This guideline only extends to packages that are installed in the standard $PATH and is not true if the application has been installed somewhere else.

The next to last section, Setup Modulepath for packages built by this compiler , is very important. Here, I define two environment variables for Lmod: $MODULEPATH and $MODULE PATH _ROOT . The line

local mdir = pathJoin(mroot,"compiler/gnu", version)

creates a local variable named mdir , which is a concatenation of the variable mroot ($MODULEPATH_ROOT ) and compiler/gnu . This tells Lmod that subsequent module avail commands should look at the compiler/gnu subdirectory under the main modules directory corresponding to the compilers just loaded (gnu/4.8 ). As the writer of the modules, you control where the module files that depend on the compilers are located. This step is the key to module hierarchy. You can control what modules are subsequently available by manipulating the mdir variable (in this case). This key attribute of Lmod gives you great flexibility.

The very last line in the module file, the statement family("compiler") , although optional, can make life better and easier for users (i.e., a best practice). The function family tells Lmod to which family the module belongs. A user can only have one family loaded at a time. In this case, the family is compiler , so that means no other compilers can be loaded. (You would hope all other compiler modules also use the family statement.) Adding this line helps users prevent self-inflicted problems. Even though the statement is somewhat optional, I highly recommend using it.

If the gnu/4.8 compiler is loaded, then the diagram of the module layout should look something like Figure 2. The green labels indicate the module that is loaded and the path to the modules that depend upon it (the MPI modules). Note that the MPI modules are under the compiler directory because they depend on the compiler module that is loaded.

Figure 2: Active path after gnu/4.8 is loaded.

In the previous section, I loaded the mpich-3.1 module. The listing here is for the mpich/3.1 module that was built with the gnu/4.8 compilers.

-- -*- lua -*-
------------------------------------------------------------------------
-- mpich-3.1 (3.1.4) support. Built with gcc-4.8 (4.8.2)
------------------------------------------------------------------------
 
 
help(
[[
This module loads the mpich-3.1 MPI library built with gcc-4.8.
compilers (4.8.2). It updates the PATH, LD_LIBRARY_PATH, 
and MANPATH environment variables to access the tools for
building MPI applications using MPICH, libraries, and
available man pages, respectively.
 
This was built using the GNU compilers, version 4.8.2.
 
The following additional environment variables are also defined:
 
MPICC   (path to mpicc compiler wrapper   )
MPICXX  (path to mpicxx compiler wrapper  )
MPIF77  (path to mpif77 compiler wrapper  )
MPIF90  (path to mpif90 compiler wrapper  )
MPIFORT (path to mpifort compiler wrapper )
 
See the man pages for mpicc, mpicxx, mpif77, and mpif90. For 
more detailed information on available compiler options and 
command-line syntax. Also see the man pages for mpirun or
mpiexec on executing MPI applications.
]])     
 
 
-- Local variables
local version = "3.1.4"
local base = "/cluster/mpich-3.1.4-gnu-4.8.2"
 
 
-- Whatis description
whatis("Description: MPICH-3.1.4 with GNU 4.8 compilers")
whatis("URL: www.mpich.org")
 
 
-- Take care of $PATH, $LD_LIBRARY_PATH, $MANPATH
prepend_path("PATH", pathJoin(base,"bin"))
prepend_path("PATH", pathJoin(base,"include"))
prepend_path("LD_LIBRARY_PATH", pathJoin(base,"lib"))
prepend_path("MANPATH", pathJoin(base,"share/man"))
 
 
-- Environment Variables
pushenv("MPICC", pathJoin(base,"bin","mpicc"))
pushenv("MPICXX", pathJoin(base,"bin","mpic++"))
pushenv("MPIF90", pathJoin(base,"bin","mpif90"))
pushenv("MPIF77", pathJoin(base,"bin","mpif77"))
pushenv("MPIFORT", pathJoin(base,"bin","mpifort"))
pushenv("mpicc", pathJoin(base,"bin","mpicc"))
pushenv("mpicxx", pathJoin(base,"bin","mpic++"))
pushenv("mpif90", pathJoin(base,"bin","mpif90"))
pushenv("mpif77", pathJoin(base,"bin","mpif77"))
pushenv("mpifort", pathJoin(base,"bin","mpifort"))
 
 
-- Setup Modulepath for packages built by this compiler/mpi
local mroot = os.getenv("MODULEPATH_ROOT")
local mdir = pathJoin(mroot,"mpi/gnu", "4.8","mpich","3.1")
prepend_path("MODULEPATH", mdir)
 
 
-- Set family for this module (mpi)
family("mpi")

If you compare this module file to the compiler module file, you will see many similarities. However, in this module file, the classic environment variables, $PATH , $LD_LIBRARY_PATH , and $MANPATH , are modified. Because you want the MPI tools associated with the module to be “first” in $PATH , the Lmod module command prepend_path is used.

Toward the end of the file, examine the code for the Modulepath. The local variable mdir points to the “new” module subdirectory, which is mpi/gnu/4.8/mpich/3.1 . (Technically, the full path is /cluster/modulefiles/mpi/gnu/4.8/mpich/3.1 because $MODULEPATH_ROOT is /cluster/ modulefiles.) In this subdirectory, you should place all modules that point to tools that have been built with both the gnu/4.8 compilers and the mpich/3.1 tools. Examples of module files that depend on both a compiler and an MPI tool are applications or libraries such as PETSc.

Also notice that the mpich/3.1 module also uses the family() function so that the user cannot load a second MPI module. You could even have a family() function for libraries such as PETSc.