Customizing Your Computing Environment with Modules

1. Introduction

In this lesson, you will learn about a utility called Environment Modules, or Modules, that simplifies managing the environment variables you need to run your applications. At the end of the lesson, you will be able to find and use pre-written modulefiles to set up your software environment.

2. Environment Modules

Many HPC clusters provide a utility for managing your software environment called Environment Modules, or Modules. Modules makes it easier for you to manage collections of environment variables associated with the software packages you need to run your applications. It also enables you to automatically modify environment variables when switching between software packages.

Modules is a command-line interface comprised of two components - modulefiles and the module command.

Modulefiles are the basic building blocks of Modules. A modulefile contains the information needed to configure the shell for an application.

The module command interprets the information in a modulefile to modify the computing environment. You can also use it to find information about modulefiles available on the system.

Is the following statement True or False?

The Modules package makes it easy to install software packages onto an HPC cluster.

3. Modulefile Contents

Typically, a modulefile contains commands for extending the PATH, MANPATH, INCLUDE, and LD_LIBRARY_PATH variables for local applications and tools. It may also include commands to create variables, aliases, and functions that facilitate using specialized hardware.

Modulefiles are written in the Tool Command Language, Tcl(3). A sample modulefile used to configure an environment using MPI is shown below.

#########################################
##
## mpi4py
##

set _module_name  [module-info name]

proc ModulesHelp { } {
global _module_name
puts stderr "The $_module_name modulefile defines the environment for the numpy
}

set _module_name  [module-info name]
set is_module_rm  [module-info mode remove]
set sys           [uname sysname]
set os            [uname release]
set mach          [uname machine]

set mpi4py_1_2_2 /usr/local/packages/mpi4py-1.2.2
set mpi4py_1_2_2_BIN $mpi4py_1_2_2/build/exe.linux-x86_64-2.7
set mpi4py_1_2_2_LIB $mpi4py_1_2_2/lib
#################################################

setenv          MPI4PY_VERSION 1.2.2
setenv          MPI4PY_DIR $mpi4py_1_2_2
module load python/2.7.1
prepend-path    PATH $mpi4py_1_2_2_BIN
prepend-path    LD_LIBRARY_PATH $mpi4py_1_2_2_LIB
append-path    PYTHONPATH       $mpi4py_1_2_2/lib/python2.7/site-packages/

Don't worry if this looks a bit complicated. HPC clusters typically provide pre-written modulefiles you can use to easily set up your environment without becoming an expert in writing modulefiles. However, if you want to learn how to code your modulefiles, see the Modulefile manpage.

4. Pre-Written Modulefiles

Your account probably includes pre-written modulefiles that set up a basic environment for your default compilers, tools, and libraries when you log in. You can use these modulefiles as they are provided or modify them to customize your environment. 

Pre-written modulefiles are available for most commonly used applications such as Matlab and Python. 

Next, you will learn how to use the Module Command to find and load pre-written modulefiles.

5. Module Commands

The module command is the user interface to the Modules Package. Below is a listing of some commonly used module commands and their descriptions.

These commands will enable you to get started using modules. If you are curious about the additional capabilities of the module command, see the SourceForge Module Command manpage.

Exercises

Given the module commands listed above, complete the following exercises by entering the appropriate command on your system's command line:

Exercise 1: List all of the modulefiles available on your system.

Exercise 2: List the modulefiles available on your system whose pathname begins with the letter 'm'.

Exercise 3: List the modulefiles that have already been loaded into your shell environment.

Exercise 4: Select one of the modulefiles available on your system and display the help information for it.

Exercise 5: Select one of the modulefiles on your system and display the changes it will make to the environment when it is loaded

Next, you will learn how to use the module command to configure your software environment.

6. Configuring Your Software Environment

The following steps outline a basic process you can use to configure your software environment using the module command and pre-written modulefiles:

1. Determine if the modulefile you want has already been loaded into your environment or not. Even if you have not loaded it yourself, it might have been loaded automatically when you logged into your resource.
2. If it isn't already loaded, find the modulefile(s) you want to use from those available on your resource.
3. Load the modulefile(s) into your environment.

The following example shows how to configure your shell environment using this process and the commands you learned in the previous section.

Example

Let's say you want to use the Matlab software on your HPC Cluster.

Step 1:

First, check to see if the Matlab modulefile has already been loaded or not. To do this, enter the module list command to display the modulefiles currently loaded in your shell environment. A sample session is shown below:

$ module list
Currently Loaded Modulefiles:
  1) torque/6.1.2   2) moab/9.1.2     3) env/ncsa       4) env/taub

From the output, we see that there are four modulefiles currently loaded, with none of them being Matlab.

Step 2:

Next, we'll use the module avail command to see if it is available and, if so, find its pathname. We could do this by displaying a list of all available modules using the module avail command with no search argument. However, the list may be long, so we'll use a search term to shorten it. For our example, we'll use the module avail command with the search term 'matlab.'

$ module avail matlab

---------------------------------------- /usr/local/modulefiles --------------------------------------
matlab/9.4 matlab/9.5 matlab/9.7

Step 3:

Finally, we'll load version 9.7 of Matlab into our environment by entering the module load command and the modulefile's pathname. 

$ module load matlab/9.7

To be sure everything loaded alright, we enter the module list command. From the output, we see that matlab/9.7 was loaded.

$ module list
Currently Loaded Modulefiles:
  1) torque/6.1.2   2) moab/9.1.2     3) env/ncsa       4) env/taub       5) matlab/9.7

Exercise

Complete the following exercise by entering the appropriate module command on your system's command line:

Load a modulefile of your choice into your shell environment and show that it was indeed loaded.

$ module load java
$ module list
Currently Loaded Modulefiles:
  1) torque/6.1.2   2) moab/9.1.2     3) env/ncsa       4) env/taub       5) matlab/9.7     6) java/1.8

7. Switching Application Environments

When switching among applications, you may need to remove the changes made by a previously loaded modulefile(s) before loading another one. There are two ways you can do this. 

First, you can use the module unload and module load commands that you learned about previously. Recall that the module unload command removes the environment changes made by a modulefile. Therefore, when switching applications, you can use the module unload command to remove the changes made by your previously loaded modulefile(s) and then load your new modulefile using the module load command. 

The second way is to use a single Modules command that lets you switch between applications by unloading one modulefile and loading another. This command is appropriately called module switch. Note that the module swap command and the module switch commands are equivalent, and only the name differs. If you try one of them on your system and you get an error message, use the other one. On some systems, you can use either one.

The syntax for the module switch command is:

module switch modulefile1 modulefile2

where modulefile1 is the currently loaded modulefile, and modulefile2 is the modulefile you want to load.

For example, to switch to another version of Matlab, say version 9.4, you would enter:

$ module switch matlab/9.7 matlab/9.4

Entering the module list command, we see that the switch was made.

$ module list
Currently Loaded Modulefiles:
  1) torque/6.1.2   2) moab/9.1.2     3) env/ncsa       4) env/taub       5) matlab/9.4     6) java/1.8

Exercises

Complete the following exercises by entering the appropriate module command on your system's command line:

Exercise 1: In the last section's exercises, you used the module load command to load a modulefile of your choice. Now, remove it and show that it is no longer in your shell environment.

$ module unload matlab/9.7
$ module list
Currently Loaded Modulefiles:
  1) torque/6.1.2   2) moab/9.1.2     3) env/ncsa       4) env/taub       5) java/1.8

Exercise 2:  You have java/1.8 loaded into your software environment. What single module command would you enter to switch it to another version?

$ module avail java

------------------------------------------------ /usr/local/modulefiles ------------------------------------------------
java/1.8(default) java/10           java/11           java/12           java/13           java/9

$ module switch java java/9

$ module list
Currently Loaded Modulefiles:
  1) torque/6.1.2   2) moab/9.1.2     3) env/ncsa       4) env/taub       5) java/9

8. Loading Modules at Startup

The changes you make to your application environment using Modules are not saved when you log off.  To make them persistent, you can add commands to a file in your home directory named .modules. If this file exists, its contents will be automatically sourced by the Environment Modules Package whenever you start a login shell. You can modify this file as needed to load, unload, or change the modules you would like enabled automatically.

A sample .modules file may have been provided for you in your home directory. This file usually contains example module load commands that are commented out. To activate any of them, remove the leading # character.

If a sample .modules file has not been provided, you can create one and add your module commands.

9. Summary

In this lesson, you learned about a tool called Modules to help you simplify the process of setting up your user environment to meet your specific needs. Modules makes it easier for you to manage collections of environment variables associated with the software packages you need to run your applications. It also enables you to automatically modify environment variables when switching between software packages.