Software tools or packages are created and updated all of the time. Ideally, you want to have updated versions of packages for users that need the latest and greatest (perhaps a bug fix or two as well), but you might also have users who need previous versions of these packages. This problem is compounded by having multiple compilers and multiple MPI libraries, resulting in a large number of possible combinations for building packages. Moreover, users will build their own libraries and applications using certain compilers and libraries, and it is much easier for them to continue using those tools without having to upgrade their entire application set to the latest compiler version. Environment Modules are one of those indispensable tools for HPC systems that help solve this package management problem. They allow you to mix and match compilers, libraries (including MPI libraries), applications, and almost anything you want to control from a version or management perspective.

I first wrote about Environment Modules in HPC Online, explaining how to integrate it with Warewulf. Then I wrote an article about Environment Modules itself, followed by an article about how you can instrument Environment Modules to gather information on what modules users are using. Most recently, my HPC cohort, Doug Eadline, wrote a more in-depth article on writing modules for Environment Modules. I have used the classic Environment Modules, sometimes referred to as TCL/C Modules because the modules are written in TCL, and I have been very pleased with it, but lately I have found some tasks difficult or impossible to do with it. Furthermore, the article about gathering Environment Module data, while functional, has some issues that are not exactly appealing to the average user and can make things a little more difficult. (Note that lowercase “environment module” indicates a generic capability and capitalized “Environment Module” indicates a particular implementation of that capability).

The particular capability I’m chasing is the ability to do software hierarchy with the packages on my system. For example, if I use a particular library with my code, I need to make sure I use the proper compilers and libraries for linking to my application. This also includes MPI libraries. More specifically, I need to make sure I link with the correct MPI library/compiler combination. Among the several ways to achieve this are two approaches called “Flat Naming Scheme” or “Hierarchical Naming Scheme.”

In the Flat Naming Scheme, you create modules with names that have all of the details on the build tools in the name itself. For example, for a library that uses a specific compiler and MPI library, you would end up with a module names like atlas-3.10.0-opempi-1.6.2-open64-5.0. The name is useful because it tells the user the library version (atlas-3.10.0), the MPI used in the library (Open MPI 1.6.2), and the compiler used to build everything (Open64 5.0). However, this approach has a few problems. The first is that if the user “loads” the module, it doesn’t necessarily mean that the Open MPI 1.6.2 module and the Open64 5.0 modules are loaded (note: there are some ways around this problem). The second problem is that if I change compilers to something like GNU 4.6.2 but I still want to use Atlas 3.10.0, I need to unload all three modules and reload the correct three modules I need. If you don’t load the correct matching modules, some mysterious failures can occur that are difficult to track down. The Flat Naming Scheme approach has pushed the responsibility of making sure the correct modules are loaded to the user rather than the module software itself.

The Hierarchical Naming Scheme approach is one I talked about in my previous two articles, even though I didn’t specifically call it out. In this approach, you create subdirectories for the compilers because they are usually at the top of the “package food chain.” Then, you create subdirectories for the MPI packages and the libraries that depend on the various other modules. In essence, you are creating a “tree” of modules. This can be a little vague, so let me explain with an example. Assume I have two compilers, Open64 5.0 and GNU 4.6.2; two MPI libraries, Open MPI 1.6.2 and MPICH2 3.0.10; and two libraries, Atlas 3.10.0 and Atlas 3.11.4. I start by putting modules for the two compilers in subdirectories from the root directory for modules (I’m assuming it is /opt/Modules).

• GNU 4.6.2:
  • /opt/Modules/compiler/GNU/4.6.2
• Open64 5.0:
  • /opt/Modules/compiler/Open64/5.0

Note that the modules in this case have a common root directory of /opt/Modules and that I have created the subdirectory compiler to hold the compiler directories and modules. Underneath each compiler subdirectory, I create a subdirectory for the MPI library modules.

• GNU 4.6.2:
  • Open MPI 1.6.2:
    • /opt/Modules/compiler/GNU/4.6.2/openmpi/1.6.2
  • MPICH2 3.0.1:
    • /opt/Modules/compiler/GNU/4.6.2/mpich2/3.0.1
• Open64 5.0:
  • Open MPI 1.6.2:
    • /opt/Modules/compiler/Open64/5.0/openmpi/1.6.2
  • MPICH2 3.0.1:
    • /opt/Modules/compiler/Open64/5.0/mpich2/3.0.1

Because I have two compilers and two MPI libraries, at this point I have four subdirectories that describe the compiler/MPI combination.

Finally, I need to add further subdirectories for the two versions of the Atlas library, 3.10.0 and 3.11.4 (a developer version).

• GNU 4.6.2:
  • Open MPI 1.6.2:
    • Atlas 3.10.0:
      • /opt/Modules/compiler/GNU/4.6.2/openmpi/1.6.2/atlas/3.10.0
    • Atlas 3.11.4:
      • /opt/Modules/compiler/GNU/4.6.2/openmpi/1.6.2/atlas/3.11.4
  • MPICH3 3.0.1:
    • Atlas 3.10.0:
      • /opt/Modules/compiler/GNU/4.6.2/mpich2/3.0.1/atlas/3.10.0
    • Atlas 3.11.4:
      • /opt/Modules/compiler/GNU/4.6.2/mpich2/3.0.1/atlas/3.11.4
• Open64 5.0:
  • Open MPI 1.6.2:
    • Atlas 3.10.0:
      • /opt/Modules/compiler/open64/5.0/openmpi/1.6.2/atlas/3.10.0
    • Atlas 3.11.4:
      • /opt/Modules/compiler/open64/5.0/openmpi/1.6.2/atlas/3.11.4
  • MPICH3 3.0.1:
    • Atlas 3.10.0:
      • /opt/Modules/compiler/open64/5.0/mpich2/3.0.1/atlas/3.10.0
    • Atlas 3.11.4:
      • /opt/Modules/compiler/open64/5.0/mpich2/3.0.1/atlas/3.11.4

This creates a total of eight subdirectories (two compilers * two MPI libraries * two library versions, or 2*2*2 = 8).

Using this approach, if I am a user and I load the compiler module open64/5.0 (module load open64/5.0), then I can only see the MPIs and libraries built with that compiler. Furthermore, because I cannot “see” the modules, I cannot load them, but I might want to see what other libraries or other MPI libraries are on the system because I might want to use them (I might be willing to switch compilers), and I can’t do that with this approach. Moreover, I might know that certain libraries are compatible with each other, but the Software Hierarchy approach does not give me the flexibility to load them. However, the Software Hierarchy approach does allow users to load the correct modules without fear that they will use the wrong one, so it is better than the Flat Naming Scheme.

I also do a great deal of testing with other compilers, MPI libraries, technical libraries, applications, and so on, so I want something that gives me flexibility but also tells me which combinations of compiler and libraries exist and work together. Although TCL/C modules have been great for a number of years, I have been on the hunt for alternatives. In this article, I examine one called Lmod.

Introduction to Lmod

Lmod is an environment modules implementation that provides all of the functionality of TCL/C Environment Modules. The “L” in Lmod really stands for lua, which is the primary language for Lmod and its modules. Lmod can read modules written in TCL and in lua, which allows you to reuse the modules you might have written for TCL/C Modules.

Lua is a very lightweight scripting language on the scale of Perl or Python, but it is lightweight enough to be embedded in other applications. You might have been using lua without knowing it because it is the scripting language for World of Warcraft. Even better, it has a small footprint of about 182KB of memory for the base interpreter and about 243KB for the library. The Wikipedia entry for lua has a pretty good description of the syntax with a few examples if you are interested.

To understand how Lmod works and why it is potentially better, you need to get into the details. Lmod uses MODULEPATH, an environment variable, just as TCL/C Modules does. However, Lmod remembers the current value for MODULEPATH, so when it changes, Lmod can unload any modules not in the current search path (it marks them as inactive and remembers them). It will also activate any inactive modules as needed.

For example, if you load the GNU 4.6.2 compilers and MPICH2 1.6.2, this adds the compiler to MODULEPATH, which in this case is /opt/Modules/compiler/GNU/4.6.2, which is loaded by:

module load compiler/gnu/4.6.2

Then you can load MPICH2 3.0.1 for that compiler with the command:

module load compiler/gnu/4.6.2/mpich2/3.0.1

If you then unload or remove the GNU compiler module, it marks the MPICH2 1.6.2 module as “inactive” and remembers it, and if you load the Open64 5.0 compiler (module load compiler/open64/5.0), it is added to MODULEPATH. When this happens, Lmod is also smart enough to “activate” the module compiler/open64/5.0/mpich2/3.0.1 to match your environment. This keeps the environment sane and tries to keep users from stepping on their own toes.

Lmod also allows you to load only one version of a package, also reducing problems. If I have the GNU 4.6.2 and the GNU 4.7.0 compilers, I can only load one at a time. For example, I can run the command

module load compiler/gnu/4.6.2

but if I then tried

module load compiler/gnu/4.7.0

it would unload compiler/gnu/4.6.2. In the case of TCL/C Modules, it would happily let you do this, even if you used a software hierarchy. The last compiler module loaded wins in this case.

Overall, Lmod provides all of the functionality that TCL/C Modules does and supports modules written in TCL and/or lua. It also directly supports software hierarchy and has a cool feature called “spider” (module spider) that allows Lmod to find all modules, so if you write any personal modules, it can find those. Another useful feature of Lmod is that users can “save” their current set of modules, so they don’t have to reload the set every time they log into the system.

Installing Lmod

I’m going to install Lmod on my test system (see the Test System section of the Warewulf article), which is a Scientific Linux 6.2 system. I’m going to build it from source because I will need to install it in my Warewulf VNFS, allowing it to work on all nodes in the cluster. I’m also going to install it into /usr/local on each node rather than use a centrally shared filesystem (over NFS). I could have used a central filesystem, but I chose to install it locally on each node so that I can maintain some flexibility in my testing.

Lmod has just a few dependencies. The first set of dependencies is for two lua libraries, luaposix and LuaFileSystem. Luaposix requires Lua BitOp as well. These libraries might work with lua 5.2 (the current version), but to make things easier, the author of Lmod, Robert McLay at the Texas Advanced Computing Center (TACC), has gathered a complete package of lua and the associated libraries and their dependencies into a package, so I downloaded that package and Lmod-4.1.4 (the latest version as of the writing of this article).

The first step is to build lua and the libraries, which is really easy to do (Listing 1).

Listing 1: Building and Installing lua for Lmod

[root@test1 ~]# ./configure --prefix=/usr/local
...
[root@test1 ~]# make
...
[root@test1 ~]# make install
...
[root@test1 ~]# cd /usr/local/bin
[root@test1 bin]# ls -s
total 336
196 lua  136 luac    4 summary.lua
[root@test1 bin]# ./lua
Lua 5.1.4.5-rtm  Copyright (C) 1994-2008 Lua.org, PUC-Rio
> 

Lua appears to be working. I added /usr/local/bin/ to $PATH for root so that Lmod could find lua when building.

Building Lmod is the next step. It is built similarly to lua using autoconf (Listing 2).

Listing 2: Building and Installing Lmod

[root@test1 ~]# ./configure --prefix=/usr/local
...
[root@test1 ~]# make
...
[root@test1 ~]# make install
...
[root@test1 local]# ls -s
total 48
4 bin  4 etc  4 games  4 include  4 lib  4 lib64  4 libexec  4 lmod  4 man  4 sbin  4 share  4 src
[root@test1 local]# cd lmod
[root@test1 lmod]# ls -s
total 4
4 4.1.4  0 lmod
[root@test1 lmod]# cd lmod
[root@test1 lmod]# ls -s
total 12
4 init  4 lib  4 libexec

Before I can use Lmod, I need to do a little more post-configuration work (but only a little). Most importantly, I need to make sure the Lmod init scripts can be executed by users. This is very, very similar to the way the TCL/C Modules function. For Lmod, I will create symlinks from the init scripts to their proper places (Listing 3).

Listing 3: Configuring Lmod init Scripts

[root@test1 profile.d]# ln -s /usr/local/lmod/lmod/init/profile /etc/profile.d/modules.sh
[root@test1 profile.d]# ln -s /usr/local/lmod/lmod/init/cshrc /etc/profile.d/modules.csh

The next step is to add a simple command in my .bashrc file to execute the proper init script (Listing 4).

Listing 4: Executing Lmod init Script in .bashrc

# User specific aliases and functions
. /etc/profile.d/modules.sh

The next time I log in to a shell, the modules.sh script will be executed.

Lmod Module Files

The next step is to configure the module files for Lmod. I’m going to reuse the module files I built for TCL/C Modules that I discussed in a previous article. But, as you shall see, I will have to make a few modifications to them to integrate them with Lmod so they use the Lmod features. To get started, all module files will be put under the directory /usr/local/modulefiles. Although you can arrange the module files under this directory however you like, I will follow the general pattern that Lmod discusses.

Fundamentally, all applications are built with a compiler. This includes applications such as Perl, Python, lua, and so on. Therefore, compilers are at the “core” of all applications. After compilers, at least in the HPC world, the next set of important tools are MPI libraries. Many applications and even libraries are built with a compiler/MPI combination. Given these concepts, the Lmod module file placement and creation starts by defining three subdirectories under /usr/local/modulefiles. These three subdirectories are: Core, Compiler, and MPI. In my thought processes, these directories reflect not where the module files reside, but rather the module dependencies. This will become more clear as the article progresses.

For the module files I want to create, I have two compilers: gcc 4.4.6 and Open64 5.0. I also have two MPI libraries: mpich2-1.5b1 and Open MPI 1.6. This results in a total of four module files. The placement of the module files is shown in Figure 1.

Figure 1: Module file hierarchy.

At the bottom of each branch are the module files, denoted with “(f)” indicating that it’s a file (the name next to this is the module file name). The other labels are subdirectories of the main module file directory /usr/local/modulefiles.

As a user when I run the command module avail when no modules are loaded, I will only see the module files in the Core subdirectory, as in Listing 5.

Listing 5: Output from “module avail”

[laytonjb@test1 ~]$ module avail
Using your spider cache file

--------------------------- /usr/local/modulefiles/Core ----------------------------
   gcc/4.4.6       open64/5.0

Use "module spider" to find all possible modules. 
Use "module keyword key1 key2 ..." to search for all possible modules matching any 
of the "keys".

Figure 2 shows the module files I can access as a user, highlighted in green.

Figure 2: Modules in “module avail” before any modules are loaded.

Notice that I only “see” the two compilers, gcc 4.4.6 and Open64 5.0. The way the hierarchy is created, I would put other gcc compiler versions underneath the subdirectory gcc (such as 4.7.2), which is under the Core subdirectory. I would put other versions of Open64 underneath the open64 directory.

I’m going to first load the Open64 5.0 compiler module file. To do this, I need to make a couple of modifications to my original TCL/C module so that it can be adapted to take advantage of Lmod. Listing 6 shows the module listing with the modifications preceded by ‘>>’.

Listing 6: Open64/5.0 Module File

#%Module1.0#####################################################################
##
## modules compilers/open64/5.0
##
## modulefiles/compilers/open64/5.0  Written by Jeff Layton
##
proc ModulesHelp { } {
   global version modroot

   puts stderr ""
   puts stderr "The compilers/open64/5.0 module enables the Open64 family of"
   puts stderr "compilers. It updates the \$PATH, "\$LD_LIBRARY_PATH, and"
   puts stderr "\$MANPATH environment variables to access the compiler and"
   puts stderr "libraries for compiling applications."
   puts stderr " "
   puts stderr "The following additional environment variables are also defined:"
   puts stderr ""
   puts stderr "\$CC\t\t(uses opencc for compiling C          )"
   puts stderr "\$CXX\t\t(uses openCC for compiling C++        )"
   puts stderr "\$FC\t\t(uses openf90 for compiling F90   )"
   puts stderr "\$F90\t\t(uses openf90 for compiling F90   )"
   puts stderr "\$F95\t\t(uses openf95 for compiling F95   )"
   puts stderr "\$F77\t\t(uses openf90 for compiling F77   )"
   puts stderr " "
   puts stderr "See the man pages for gcc, g++, and gfortran for more"
   puts stderr "detailed information on available compiler options and"
   puts stderr "command-line syntax."
   puts stderr " "
}

module-whatis "Name: Open64 Environment"
module-whatis "Version: 5.0"
module-whatis "Category: compiler, runtime support"
module-whatis "Description: open64 Compiler Family (C/C++/Fortran90/Fortran95 for x86_64)"
module-whatis "URL: http://www.open64.net/"

# for Tcl script use only
set     topdir          /opt/open64
set     version         5.0
set     sys             linux86

>># For Lmod:
>>set MODULEPATH_ROOT /usr/local/modulefiles
>>set open64_version 5.0

prepend-path    PATH            $topdir/lib/gcc-lib/x86_64-open64-linux/5.0
prepend-path    PATH            $topdir/include/5.0
prepend-path    PATH            $topdir/bin
prepend-path    MANPATH         $topdir/open64-gcc-4.2.0/man
prepend-path    LD_LIBRARY_PATH $topdir/lib/gcc-lib/x86_64-open64-linux/5.0

setenv          CC              $topdir/bin/opencc
setenv          CXX             $topdir/bin/openCC
setenv          FC              $topdir/bin/openf90
setenv          F77             $topdir/bin/openf90

>># For Lmod:
>>prepend-path MODULEPATH $MODULEPATH_ROOT/Compiler/open64/$open64_version

The first set of three modified lines defines and sets two variables, MODULEPATH_ROOT and open64_version. These variables are used in the last two modified lines. The command prepend-path does exactly that – it puts a new path before the rest of the existing path. In this case, it’s the path for MODULEPATH.

Look at the last line carefully. It adds a new path to MODULEPATH. This path works out to /usr/local/modulefiles/Compiler/open64/5.0. In particular, notice the subdirectory Compiler in the path. Under this subdirectory, I put any modules that are dependent on the compiler open64/5.0 that is just loaded. If I had used the gcc 4.4.6 compilers instead of the Open64 5.0 compilers, I would have prepended the path /usr/local/modulefiles/Compiler/gcc/4.4.6 in the appropriate module file in the Core subdirectory.

Listing 7 shows loading the Open64 5.0 compiler and checking that it works, as well as the output of module avail.

Listing 7: Loading the open64/5.0 Compiler

[laytonjb@test1 ~]$ module load open64/5.0
[laytonjb@test1 ~]$ openf90 -v
Open64 Compiler Suite: Version 5.0
Built on: 2011-11-09 11:16:36 +0800
Thread model: posix
GNU gcc version 4.2.0 (Open64 5.0 driver)
[laytonjb@test1 ~]$ module avail
Using your spider cache file

-------------------- /usr/local/modulefiles/Compiler/open64/5.0 --------------------
   mpich2/1.5b1       openmpi/1.6

--------------------------- /usr/local/modulefiles/Core ----------------------------
   gcc/4.4.6       open64/5.0

Use "module spider" to find all possible modules. 
Use "module keyword key1 key2 ..." to search for all possible modules matching any 
of the "keys". 

You can see that the correct module was loaded because I could correctly run the Open64 F90 compiler (openf90), but pay particular attention to the output of the module avail command.

The tree hierarchy in Figure 1 has changed because MODULEPATH changed when the Open64 5.0 module was loaded. Figure 3 shows the additional parts of the tree in red that I can now access.

Figure 3: Modules in “module avail” after the open64/5.0 module is loaded.

If you compare the module files in Figure 3 to the output in Listing 7, you will see that they match. More precisely, the MPI libraries built with Open64 5.0 are now listed.

For the openmpi 1.6 module to work properly with Lmod, I made some small changes to the open64/5.0/openmpi/1.6 module (Listing 8) so that it points to the correct subdirectory under modulefiles.

Listing 8: Loading the open64/5.0 Compiler

#%Module1.0#####################################################################
##
## modules mpi/openmpi/1.6-open64-5.0
##
## modulefiles/mpi/openmpi/1.6-open64-5.0  Written by Jeff Layton
##
proc ModulesHelp { } {
   global version modroot

   puts stderr ""
   puts stderr "The mpi/opempi/1.6-open64-5.0 module enables the Open MPI"
   puts stderr "library and tools for version 1.6 as built with the Open64"
   puts stderr "compilers. It updates the \$PATH, \$LD_LIBRARY_PATH, and "
   puts stderr "\$MANPATH environment variables to access the scripts for"
   puts stderr "building MPI applications using Open MPI, libraries, and"
   puts stderr "available man pages, respectively."
   puts stderr ""
   puts stderr "This was built using the Open64 compilers, version 5.0."
   puts stderr ""
   puts stderr "The following additional environment variables are also defined:"
   puts stderr ""
   puts stderr "\$MPICC\t\t(path to mpicc compiler wrapper  )"
   puts stderr "\$MPICXX\t\t(path to mpicxx compiler wrapper )"
   puts stderr "\$MPIF77\t\t(path to mpif77 compiler wrapper )"
   puts stderr "\$MPIF90\t\t(path to mpif90 compiler wrapper )"
   puts stderr "\$CC\t\t(uses mpicc for compiling C      )"
   puts stderr "\$CXX\t\t(uses mpicc for compiling C++    )"
   puts stderr "\$FC\t\t(uses mpif90 for compiling F90   )"
   puts stderr "\$F90\t\t(uses mpif90 for compiling F90   )"
   puts stderr "\$F77\t\t(uses mpif90 for compiling F77   )"
   puts stderr " "
   puts stderr "See the man pages for mpicc, mpicxx, mpif77, and mpif90. For "
   puts stderr "more detailed information on available compiler options and "
   puts stderr "command-line syntax. Also see the man pages for mpirun or"
   puts stderr "mpiexec on executing MPI applications."
}

module-whatis "Name: Open MPI Environment"
module-whatis "Version: 1.6 Built with the Open64 compilers"
module-whatis "Category: mpi/openmpi"
module-whatis "Description: Open MPI library and tools for MPI applications"
module-whatis "URL: http://www.open-mpi.org/"
module-whatis "URL: http://www.open64.net/"

# for Tcl script use only
set     topdir          /opt/openmpi-1.6-open64-5.0
set     version         1.6
set     sys             linux86

>># For Lmod:
>>set MODULEPATH_ROOT /usr/local/modulefiles

prepend-path    PATH            $topdir/bin
prepend-path    PATH            $topdir/include
prepend-path    MANPATH         $topdir/share/man
prepend-path    LD_LIBRARY_PATH $topdir/lib

setenv          MPICC           $topdir/bin/mpicc
setenv          MPICXX          $topdir/bin/mpicxx
setenv          CC              $topdir/bin/mpicc
setenv          CXX             $topdir/bin/mpicxx
setenv          FC              $topdir/bin/mpif90
setenv          F77             $topdir/bin/mpic77
setenv          F90             $topdir/bin/mpif90
setenv          MPIF77          $topdir/bin/mpic77
setenv          MPIF90          $topdir/bin/mpif90

>># For Lmod:
>>prepend-path MODULEPATH $MODULEPATH_ROOT/MPI/open64/5.0/openmpi/1.6

The changed lines are preceded by ‘>>’, and the last line should be examined carefully. Now that a compiler and an MPI library are loaded, the MODULEPATH points to the MPI subdirectory that will contain module files for applications or libraries that are built with a specific compiler/MPI combination. This is reflected in the prepend path to MODULEPATH, /usr/local/modulefiles/MPI/open64/5.0/openmpi/1.6.

Now I’ll load the Open MPI 1.6 module and check that it is loads correctly and points to the correct tools (Listing 9).

Listing 9: Loading the open64/5.0 Compiler

[laytonjb@test1 ~]$ module load openmpi/1.6
[laytonjb@test1 ~]$ which mpicc
/opt/openmpi-1.6-open64-5.0/bin/mpicc
[laytonjb@test1 ~]$ mpicc -v
Open64 Compiler Suite: Version 5.0
Built on: 2011-11-09 11:16:36 +0800
Thread model: posix
GNU gcc version 4.2.0 (Open64 5.0 driver)

From this, I can conclude that the compiler/MPI pair are working correctly.

One of the cool features of Lmod is that when I swap modules, Lmod will correctly unload any incorrect modules and load the correct ones. To demonstrate this, I’ll swap out the open64/5.0 module for the gcc/4.4.6 module while I have the openmpi/1.6 module loaded (the one for the Open64 5.0 compiler). Listing 10 illustrates what happens when the swap is made and checks that the swap happened correctly.

Listing 10: Swapping the gcc Module for the Open64 Module and the Resulting Changes by Lmod

[laytonjb@test1 ~]$ module avail
Using your spider cache file

-------------------- /usr/local/modulefiles/Compiler/open64/5.0 --------------------
   mpich2/1.5b1       openmpi/1.6

--------------------------- /usr/local/modulefiles/Core ----------------------------
   gcc/4.4.6       open64/5.0

Use "module spider" to find all possible modules. 
Use "module keyword key1 key2 ..." to search for all possible modules matching any 
of the "keys". 

[laytonjb@test1 ~]$ module list
Currently Loaded Modules:
  1) open64/5.0    2) openmpi/1.6
[laytonjb@test1 ~]$ module swap open64/5.0 gcc/4.4.6

Due to MODULEPATH changes the following have been reloaded:
  1) openmpi/1.6

[laytonjb@test1 ~]$ openf90 -v
bash: openf90: command not found

[laytonjb@test1 ~]$ gcc -v
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man 
--infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla 
--enable-bootstrap --enable-shared --enable-threads=posix 
--enable-checking=release --with-system-zlib --enable-__cxa_atexit 
--disable-libunwind-exceptions --enable-gnu-unique-object 
--enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk
--disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre
--enable-libgcj-multifile --enable-java-maintainer-mode
--with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib
--with-ppl --with-cloog --with-tune=generic --with-arch_32=i686
--build=x86_64-redhat-linux
Thread model: posix
gcc version 4.4.6 20110731 (Red Hat 4.4.6-3) (GCC) 

[laytonjb@test1 ~]$ which mpicc
/opt/openmpi-1.6/bin/mpicc

[laytonjb@test1 ~]$ mpicc -v
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla
--enable-bootstrap --enable-shared --enable-threads=posix
--enable-checking=release --with-system-zlib --enable-__cxa_atexit
--disable-libunwind-exceptions --enable-gnu-unique-object
--enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk
--disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre
--enable-libgcj-multifile --enable-java-maintainer-mode
--with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib
--with-ppl --with-cloog --with-tune=generic --with-arch_32=i686
--build=x86_64-redhat-linux
Thread model: posix
gcc version 4.4.6 20110731 (Red Hat 4.4.6-3) (GCC) 

[laytonjb@test1 ~]$ module list
Currently Loaded Modules:
  1) gcc/4.4.6    2) openmpi/1.6

You can see that when the open64/5.0 module is swapped out for the gcc/4.4.6 module, Lmod switches the openmpi/1.6 module to the correct one. I checked this by seeing if I could run the openf90 compiler, then running the gcc compiler and the mpicc compiler wrapper for Open MPI 1.6. Notice that the Open MPI 1.6 wrapper points to the correct compiler (gcc 4.4.6).

Summary

Environment Modules are one of those “go-to” tools for HPC, or, for that matter, any situation in which you need to manage applications. They have been in use for quite some time, and I absolutely swear by them. I remember a time when they did not exist and the sticky situations that resulted. However, I must admit that I’m a junkie for performance and improvements, so I’m always looking for ways to improve upon existing tools. This has led me to examine Lmod.

Lmod is an “implementation” of environment modules (the generic purpose of the tool) written in lua (a cool language in its own right). It has some features that I’ve wanted and that are really useful for my situation. The biggest one for me is the ability to unload modules and load correct modules depending on what modules are loaded. However, I also like the feature that lets me save my current modules to a file so I can load them the next time I log on. (I didn’t show this feature here because it’s self-explanatory.)

Lmod can work with TCL module files, lua module files, or both, even at the same time. This gives some flexibility because, honestly, TCL has passed its prime (many scripting languages tend to do this). I was able to reuse my TCL module files with just a very few additions.

Before I end the article, I want to thank Dr. Robert McLay from TACC who is the author of Lmod. He provided a great deal of help to me in building and understanding Lmod, including when I got confused. He also did this while he was under the weather. I greatly appreciate his help.