Software Modules Lmod

Software Module System - Lmod

Preface

This guide describes the software environment and tools used to run applications on JUSTUS 2 system. It is intended as a general overview and introduction to the software system management on JUSTUS 2 for new users as well as for experienced users coming, e.g. from sites planted with different Environment modules package, as for example JUSTUS 1 system.

For users migrating from JUSTUS 1 to JUSTUS 2

There exist several software implementations for a dynamical management of environmental variables throughout the HPC world. JUSTUS 1 hosted Environment Modules package based on Tcl, while there is a Lua based module system Lmod (https://lmod.readthedocs.io/en/latest/#) installed on JUSTUS 2. Despite of the various software solutions, the difference in handling of the modules from users' perspective of view stays tiny. The biggest change lies in the introducing of the partial software hierarchy implemented in modules system on JUSTUS 2. Besides that, Lmod extends the basic environment modules by adding predefined shortcuts and a more powerful search capability.

Lmod commnads

The much of module commands and its functionality has remained same on JUSTUS 2, although Lmod offers new functions as for example module spider as an extension of "old" module avail functionality.
Thus, e.g. the default Gaussian module is initialized by

$ module load chem/gaussian

which is pretty much the same as on JUSTUS 1 or with using of the Lmod shorthand version

$ ml chem/gaussian

Full list of the Lmod module commands is available online on https://lmod.readthedocs.io/en/latest/010_user.html

Rather than the divergence of module commands, JUSTUS 1 users has to get a grasp of the modulefile system hierarchy built-in JUSTUS 2 software layout. The layout is no more flat as on JUSTUS 1, however JUSTUS 2 adopts a partial module hierarchy for mpi modules.

Consequences of the partial module hierarchy for mpi modules

Due the flat layout of software modules on JUSTUS 1 module avail command showed all pre-installed software packages including those directly depended on another modules. This is no more true on JUSTUS 2, in particular for mpi modules. mpi modules remains invisible for a user (prompted module avail) until some compiler module has been loaded. Once the compiler module has been activated corresponding mpi modules, i.e. built with the particular compiler, become visible.

E.g., with the originally empty list of the loaded modules, the module command

$ module avail

or its shorthand analogue

 
$ ml av

displays no mpi module available. After running

ml compiler/intel/19.1
ml av

mpi packages compatible with the intel 19.1 compiler becomes visible

------------ /opt/bwhpc/common/modulefiles/Compiler/intel/19.1 --------------------
   mpi/impi/2019.7    mpi/openmpi/4.0

in the list of the available software.

Lmod Module System on JUSTUS 2

JUSTUS 2 system uses Lmod (the Lua Based Module System) environment management package to support dynamical modification of the user environment via modulefiles. Each modulefile contains information needed to configure the shell, i.e. its environmental variables, required to run a particular software application. To make major changes in the users' environment, such as modifications of PATH, LD_LIBRARY_PATH, CC or FC variables for, e.g. a switching to a different compiler, use the appropriate single or sequence of modules commands instead of a reconfiguration of the shell by hand.

E.g. in order to activate the default vasp module run a single command:

$ module load chem/vasp

or its shorthand variant

$ml chem/vasp

For an activation of the module of intel mpi built with the corresponding intel compiler a sequence of two module commands

$ module load compiler/intel
$ module load mpi/impi 

has to be run.

Online user guide of Lmod

The complete user guide can be found on Lmod websites https://lmod.readthedocs.io/en/latest/010_user.html

Which shells supports module commands?

So far Bash is only supported shell on JUSTUS 2 to interpret module commands.

Basic Module System tasks

Lmod offers more than 25 sub-commands plus various options to manage the modulefile system installed on JUSTUS 2. See, e.g. output of "module --help" command. Large majority of users will use only couple of them. A complete list of module sub-commands can be displayed by entering "module --help" command or in Lmod online documentation. The following text lists only a couple of them.

Lmod shortcuts

Lmod extends the basic environment modules by adding simple shortcuts. The ml shortcut can be used in place of module, av in instead of avail, ml instead of module list, or - can alternate unload.
For instance, following combinations are interchangeble:

module list      <==>   ml

module avail      <==>   ml av

module load compiler/intel      <==>   ml compiler/intel

module unload compiler/intel      <==>   ml -compiler/intel

Loading and unloading modulefiles

To load and unload a module of a specific package use "module load category/package" and "module unload category/package", respectively. Thus, e.g, to load default version of VASP which belongs into category chem, enter

$ module load chem/VASP

and for unloading of the package use

$ module unload chem/VASP

Complete unloading of all modulefiles

To clean the environment from all entries set by lmod use "module purge" command.

$ module purge

Beware that the "module purge" does not always restore the environmental variables that existed prior entering first "module load" command. I.e., some of the entries in PATH, or environmental variables, etc. originally existed in the environment might be cleaned by "module purge" if those would be duplicates of the variables set by any modulefiles that was loaded.

What is Loaded Now?

To see which modules are currently loaded in your environment, enter the command "module list".

$ module list

Which modulefiles/software is available?

To see what modulefiles are available on JUSTUS 2, you can enter the following command:

$ module avail

or alternatively

$ ml av

which comes with lmod to save typewriting.

Module specific help

To access the module specific help, use the "module help" command. For example, to see the module help associated with the default VASP module, enter this command:

$ module help chem/vasp

It is always a good practice to read through all the info provided by a specific module help since this serves as an important communication channel between a user and the person who install the piece of a software.

Semi hierarchical layout of modules on JUSTUS 2

Module hierarchy in Lmod

The structure of software modules on JUSTUS 2 exploits a "semi" hierarchical structure. This is slightly different from what can be seen on another HPC systems with "full" hierarchical structure. The typical systems with full hierarchy put compiler modules (i.e., intel, gcc) in the uppermost (Core) level, depending libraries (e.g., MPI) on the second level, and more depending libraries on a third level. As a consequence, not all the modules contained in the module system are initially visible, namely the modules putted in the second and third layer. Only after a loading a compiler module, the modules of the second layer directly depending on the particular compiler will become available. And similarly, loading an MPI module will make the modules of the third layer depending on the loaded MPI library visible.

Semi hierarchy of software stack on JUSTUS 2

JUSTUS 2 adopted the hierarchical structure of the modules layout only partially. In particular, there is only "Core" and the "second" level presented and there are only mpi modules contained in the second level. All other modules, i.e. for example those from the "chem" sub-cathegory such as vasp, turbomole, or gaussian, or those located in the "numlib" sub-cathegory such as mkl or python_numpy, are embodied in the "Core" level.

Module dependency

The adopted hierarchy models is not the only tool handling the module dependency. As a matter of fact, most of the modules on JUSTUS 2 require a provision of functionalities from another modules, albeit located in the "Core" level. Such provisioning is implemented in a modulefile either automatically without a need of any action from the user (the depending modulefile, while loading, loads all additional modules automatically) or the depending modulefile, while loading, informs the user about necessity to pre-load additional modules if those has not been activated yet (in this case the user must repeat the loading operation). Which of the solution is applied rests with the decision of the person who built the particular module.

An example of module with the implemented automated pre-loading is orca module. With the pre-emptied list of the loading modules, i.e.

$ ml

shows

No modules loaded

, the command sequence

$ ml chem/orca
$ ml

shows

Currently Loaded Modules:
  1) compiler/intel/19.1   2) chem/orca/4.2.1 

I.e., loading of the intel compiler is built-in the orca module.

Advanced topics

Complete list of Lmod options and sub-commands

The whole list of module options and all commands available can be displayed by running

man module

with the output showing

Usage: module [options] sub-command [args ...]

Options:
  -h -? -H --help                   This help message
  -s availStyle --style=availStyle  Site controlled avail style: system
                                    (default: system)
  --regression_testing              Lmod regression testing
  -D                                Program tracing written to stderr
  --debug=dbglvl                    Program tracing written to stderr (where
                                    dbglvl is a number 1,2,3)
  --pin_versions=pinVersions        When doing a restore use specified
                                    version, do not follow defaults
  -d --default                      List default modules only when used with
                                    avail
  -q --quiet                        Do not print out warnings
  --expert                          Expert mode
  -t --terse                        Write out in machine readable format for
                                    commands: list, avail, spider, savelist
  --initial_load                    loading Lmod for first time in a user
                                    shell
  --latest                          Load latest (ignore default)
  --ignore_cache                    Treat the cache file(s) as out-of-date
  --novice                          Turn off expert and quiet flag
  --raw                             Print modulefile in raw output when used
                                    with show
  -w twidth --width=twidth          Use this as max term width
  -v --version                      Print version info and quit
  -r --regexp                       use regular expression match
  --gitversion                      Dump git version in a machine readable
                                    way and quit
  --dumpversion                     Dump version in a machine readable way
                                    and quit
  --check_syntax --checkSyntax      Checking module command syntax: do not
                                    load
  --config                          Report Lmod Configuration
  --config_json                     Report Lmod Configuration in json format
  --mt                              Report Module Table State
  --timer                           report run times
  --force                           force removal of a sticky module or save
                                    an empty collection
  --redirect                        Send the output of list, avail, spider
                                    to stdout (not stderr)
  --no_redirect                     Force output of list, avail and spider
                                    to stderr
  --show_hidden                     Avail and spider will report hidden
                                    modules
  --spider_timeout=timeout          a timeout for spider
  -T --trace

module [options] sub-command [args ...]

Help sub-commands:
------------------
  help                              prints this message
  help                module [...]  print help message from module(s)

Loading/Unloading sub-commands:
-------------------------------
  load | add          module [...]  load module(s)
  try-load | try-add  module [...]  Add module(s), do not complain if not
                                    found
  del | unload        module [...]  Remove module(s), do not complain if not
                                    found
  swap | sw | switch  m1 m2         unload m1 and load m2
  purge                             unload all modules
  refresh                           reload aliases from current list of
                                    modules.
  update                            reload all currently loaded modules.

Listing / Searching sub-commands:
---------------------------------
  list                              List loaded modules
  list                s1 s2 ...     List loaded modules that match the
                                    pattern
  avail | av                        List available modules
  avail | av          string        List available modules that contain
                                    "string".
  spider                            List all possible modules
  spider              module        List all possible version of that module
                                    file
  spider              string        List all module that contain the
                                    "string".
  spider              name/version  Detailed information about that version
                                    of the module.
  whatis              module        Print whatis information about module
  keyword | key       string        Search all name and whatis that contain
                                    "string".

Searching with Lmod:
--------------------
  All searching (spider, list, avail, keyword) support regular expressions:

  -r spider           '^p'          Finds all the modules that start with
                                    `p' or `P'
  -r spider           mpi           Finds all modules that have "mpi" in
                                    their name.
  -r spider           'mpi$         Finds all modules that end with "mpi" in
                                    their name.

Handling a collection of modules:
--------------------------------
  save | s                          Save the current list of modules to a
                                    user defined "default" collection.
  save | s            name          Save the current list of modules to
                                    "name" collection.
  reset                             The same as "restore system"
  restore | r                       Restore modules from the user's
                                    "default" or system default.
  restore | r         name          Restore modules from "name" collection.
  restore             system        Restore module state to system defaults.
  savelist                          List of saved collections.
  describe | mcc      name          Describe the contents of a module
                                    collection.
  disable             name          Disable (i.e. remove) a collection.

Deprecated commands:
--------------------
  getdefault          [name]        load name collection of modules or
                                    user's "default" if no name given.
                                    ===> Use "restore" instead <====
  setdefault          [name]        Save current list of modules to name if
                                    given, otherwise save as the default
                                    list for you the user.
                                    ===> Use "save" instead. <====

Miscellaneous sub-commands:
---------------------------
  is-loaded           modulefile    return a true status if module is loaded
  is-avail            modulefile    return a true status if module can be
                                    loaded
  show                modulefile    show the commands in the module file.
  use [-a]            path          Prepend or Append path to MODULEPATH.
  unuse               path          remove path from MODULEPATH.
  tablelist                         output list of active modules as a lua
                                    table.

Important Environment Variables:
--------------------------------
  LMOD_COLORIZE                     If defined to be "YES" then Lmod prints
                                    properties and warning in color.

--------------------------------------------------------------------------

Lmod Web Sites

  Documentation:    http://lmod.readthedocs.org
  Github:           https://github.com/TACC/Lmod
  Sourceforge:      https://lmod.sf.net
  TACC Homepage: https://www.tacc.utexas.edu/research-development/tacc-projects/lmod

  To report a bug please read http://lmod.readthedocs.io/en/latest/075_bug_reporting.html
--------------------------------------------------------------------------

Modules based on Lua: Version 8.2.7  2019-11-30 09:41 -06:00
    by Robert McLay mclay@tacc.utexas.edu

Module categories, versions and defaults

Software stack on bwHPC systems is commonly classified into following categories:

• chem
• compiler
• devel
• lib
• numlib
• phys
• system
• vis
• math

Each the category is further divided according to software packages and those finally according to software versions. Similarly, the module identifier has the format: category/softwarename/version For instance, gnu compiler of the version 10.1 is unabiguosly addressed as compiler/gnu/10.1 in, e.g. the load command:

$ ml compiler/gnu/10.1

In case of there is multiple software versions, one version is always pre-determined as the default version. To address a default version, version can be omitted in the module identifier. For example, the loading of the default intel compiler module is realized via

$ ml compiler/intel

Selective searching

It is possible to perform searching through only selected category by adding name of the cathegory to module avail or ml av command. For example, to only explore the compilers is realized with

$ module avail compiler/

Searching capabilities of module avail commands

The command module avail has a search capabilities. For example

$ ml av gnu

will list for any modulefile with the name containing the string "gnu", i.e. for instance

--------------------------- /opt/bwhpc/common/modulefiles/Core ---------------------------------
   compiler/gnu/system    compiler/gnu/9.3    compiler/gnu/10.1 (D)    vis/gnuplot/5.2

Loading Modules

You can load a Module software in to your environment to enable easier access to software that you want to use by executing:

$ module load category/softwarename/version

or

$ module add category/softwarename/version

Loading a Module in this manner affects ONLY your environment for the current session.

Loading conflicts

You can not load different versions of the same software at the same time! Loading the Intel compiler in version X while Intel compiler in version Y is loaded leads to an automatic unloading of Intel compiler in version Y.

Showing the changes introduced by a Module

Loading a Module will change the environment of the current shell session. For instance the $PATH variable will be expanded by the software's binary directory. Other Module variables may even change the behavior of the current shell session or the software program(s) in a more drastic way.

All the changes to the current shell session to be invoked by loading the Module can be reviewed using
'module show category/softwarename/version'.

Example (Intel compiler)

$ module show compiler/intel
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   /opt/bwhpc/common/modulefiles/Core/compiler/intel/19.1.lua:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
setenv("INTEL_LICENSE_FILE","28518@scclic1.scc.kit.edu")
setenv("AR","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64/xiar")
setenv("CC","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64/icc")
setenv("CXX","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64/icpc")
setenv("F77","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64/ifort")
setenv("FC","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64/ifort")
setenv("CFLAGS","-O2 -xCORE-AVX2")
setenv("CXXFLAGS","-O2 -xCORE-AVX2")
setenv("FFLAGS","-O2 -xCORE-AVX2")
setenv("FCFLAGS","-O2 -xCORE-AVX2")
setenv("INTEL_VERSION","19.1.0.166")
setenv("INTEL_HOME","/opt/intel/compilers_and_libraries_2020/linux")
setenv("INTEL_BIN_DIR","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64")
setenv("INTEL_LIB_DIR","/opt/intel/compilers_and_libraries_2020/linux/lib/intel64")
setenv("INTEL_INC_DIR","/opt/intel/compilers_and_libraries_2020/linux/include")
setenv("INTEL_MAN_DIR","/opt/intel/compilers_and_libraries_2020/linux/man/common")
setenv("INTEL_DOC_DIR","/opt/intel/compilers_and_libraries_2020/linux/documentation/en")
setenv("GDB_VERSION","19.1.0.166")
setenv("GDB_HOME","/opt/intel/debugger_2020/gdb/intel64")
setenv("GDB_BIN_DIR","/opt/intel/debugger_2020/gdb/intel64/bin")
setenv("GDB_LIB_DIR","/opt/intel/debugger_2020/libipt/intel64/lib")
setenv("GDB_INC_DIR","/opt/intel/debugger_2020/gdb/intel64/include")
setenv("GDB_INF_DIR","/opt/intel/documentation_2020/en/debugger/gdb-ia/info")
setenv("GDB_MAN_DIR","/opt/intel/documentation_2020/en/debugger/gdb-ia/man")
setenv("KMP_AFFINITY","noverbose,granularity=core,respect,warnings,compact,1")
prepend_path("PATH","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64")
prepend_path("MANPATH","/opt/intel/compilers_and_libraries_2020/linux/man/common")
prepend_path("LD_LIBRARY_PATH","/opt/intel/compilers_and_libraries_2020/linux/lib/intel64")
whatis("Sets up Intel C/C++ and Fortran compiler version 19.1 (Intel(R) Compilers 19.1 for Linux*) - supported by SCC till 2022-12-31!")
help([[Intel(R) Compilers 19.1 for Linux*
For details see: https://software.intel.com/en-us/intel-compilers
In case of problems, please contact: Hartmut Häfner <hartmut.haefner@kit.edu>
SCC support end: 2022-12-31]])
prepend_path("MODULEPATH","/software/bwhpc/common/modulefiles/Compiler/intel/19.1")
family("compiler")

Modules depending on Modules

Some program Modules depend on libraries to be loaded to the user environment. Therefore the corresponding Modules of the software must be loaded together with the Modules of the libraries.
By default such software Modules try to load required Modules and corresponding versions automatically.

Unloading Modules

To unload or remove a software Module execute:

$ module unload category/softwarename/version

or

$ module remove category/softwarename/version

Unloading all loaded modules

Purge

Unloading a Module that has been loaded by default makes it inactive for the current session only - it will be reloaded the next time you log in.
In order to remove all previously loaded software modules from your environment issue the command 'module purge'.
Example

$ module list
Currently Loaded Modules:
  1) compiler/intel/19.1   2) mpi/impi/2019   3) numlib/mkl/2019
$
$ module purge
$ module list
No modules loaded
$

Beware!
'module purge' is working without any further inquiry.

Display your loaded Modules

All Modules that are currently loaded for you can be displayed by the command 'module list'. See example above.
Note: You only have to load further Modules, if you want to use additional software packages or to change the version of an already loaded software.

Display all possible Modules

Modulefiles can be searched by the user. You can dipslay all possible modules by executing:

$ module spider

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The following is a list of the modules and extensions currently available:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  cae/abaqus: cae/abaqus/2018, cae/abaqus/2019

  cae/adina: cae/adina/9.1.2

  cae/ansys: cae/ansys/19.2, cae/ansys/2019R3, cae/ansys/2020R1

  cae/comsol: cae/comsol/5.4, cae/comsol/5.5

  cae/cst: cae/cst/2018

  cae/lsdyna: cae/lsdyna/901

  cae/openfoam: cae/openfoam/v1912, cae/openfoam/2.4.x, cae/openfoam/6, cae/openfoam/7

  cae/paraview: cae/paraview/5.8

  cae/starccm+: cae/starccm+/14.02.010, cae/starccm+/2019.2.1

  cae/starcd: cae/starcd/4.28

  compiler/clang: compiler/clang/9.0

  compiler/gnu: compiler/gnu/9.2

  compiler/intel: compiler/intel/18.0, compiler/intel/19.0, compiler/intel/19.1

  compiler/pgi: compiler/pgi/2019

  devel/cmake: devel/cmake/3.16

  devel/cuda: devel/cuda/9.2, devel/cuda/10.0, devel/cuda/10.2

  devel/gdb: devel/gdb/9.1

  devel/python: devel/python/3.7.4_gnu_9.2, devel/python/3.8.1_gnu_9.2, devel/python/3.8.1_intel_19.1

  math/R: math/R/3.6.3

  math/julia: math/julia/1.3.1

  mpi/impi: mpi/impi/2018, mpi/impi/2019, mpi/impi/2020

  mpi/openmpi: mpi/openmpi/4.0

  numlib/mkl: numlib/mkl/2018, numlib/mkl/2019, numlib/mkl/2020

  numlib/python_numpy: numlib/python_numpy/1.17.2_python_3.7.4_gnu_9.2

  numlib/python_scipy: numlib/python_scipy/1.3.1_numpy_1.17.2_python_3.7.4_gnu_9.2

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

To learn more about a package execute:

   $ module spider Foo

where "Foo" is the name of a module.

To find detailed information about a particular package you
must specify the version if there is more than one version:

   $ module spider Foo/11.1

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

'module spider name/version' : If you search the full name and version of the module, the search gives detailed information about that module version.

$ module spider devel/cmake

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  devel/cmake: devel/cmake/3.16
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    This module can be loaded directly: module load devel/cmake/3.16

    Help:
      Home page:            https://www.cmake.org
      Online Documentation: https://www.cmake.org/HTML/Documentation.html
      Local Documentation:  /opt/bwhpc/common/devel/cmake/3.16.4/docFAQ:                  https://gitlab.kitware.com/cmake/community/wikis/FAQ

Moreover, you can see the dependencies of the module with using the same command. For example, if the following is executed, you can see which modules need to be loaded before loading the module mpi/impi/2019

$ module spider mpi/impi/2019

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  mpi/impi: mpi/impi/2019
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    You will need to load all module(s) on any one of the lines below before the "mpi/impi/2019" module is available to load.

      compiler/clang/9.0
      compiler/gnu/9.2
      compiler/intel/18.0
      compiler/intel/19.0
      compiler/intel/19.1

    Help:
      Intel(R) MPI Library

How do Modules work?

The default shell on the bwHPC clusters is bash, so explanations and examples will be shown for bash. In general, programs cannot modify the environment of the shell they are being run from, so how can the module command do exactly that?
The module command is not a program, but a bash-function. You can view its content using:

$ type module

and you will get the following result:

$ type module
module is a function
module ()
{
    eval $($LMOD_CMD bash "$@");
    [ $? = 0 ] && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}

In this function, lmod is called. Its output to stdout is then executed inside your current shell using the bash-internal eval command. As a consequence, all output that you see from the module is transmitted via stderr (output handle 2) or in so

---

Back to top