Software Modules Lmod
Software Module System - Lmod
PREFACE
This guide describes the software environment and tools used to run applications on JUSTUS 2 system. It is intended as a general overview and introduction to the software system management on JUSTUS 2 for new users as well as for experienced users coming, e.g. from sites planted with different Environment modules package, as for example JUSTUS 1 system.
Essential "JUSTUS 1 to JUSTUS 2" migration guide
There exist several software implementations for a dynamical management of environmental variables throughout the HPC world.
JUSTUS 1 hosted Environment Modules package based on Tcl, while there is a Lua based module system Lmod (https://lmod.readthedocs.io/en/latest/#) installed on JUSTUS 2.
Despite of the various software solutions, the difference in handling of the modules from users' perspective of view stays tiny. The biggest change lies in the introducing of the partial software hierarchy implemented in modules system on JUSTUS 2.
Lmod commnads
The much of module commands and its functionality has remained same on JUSTUS 2, although Lmod offers new functions as for example module spider as an extension of "old" module avail functionality.
Thus, the default Gaussian module is initialized by
$ module load chem/gaussian
or with using of the shorthand version
$ ml chem/gaussian
Full list of the Lmod module commands is available online on https://lmod.readthedocs.io/en/latest/010_user.html
Rather than the divergence of module commands, JUSTUS 1 users has to get a grasp of the modulefile system hierarchy built-in JUSTUS 2 software layout.
The layout is no more flat as on JUSTUS 1, however JUSTUS 2 adopts a partial module hierarchy for mpi modules.
Consequences of the partial module hierarchy for mpi modules
Due the flat layout of software modules on JUSTUS 1 module avail command showed all pre-installed software packages including those directly depended on another modules.
This is no more true on JUSTUS 2, in particular for mpi modules. mpi modules remains invisible for a user (prompted module avail) until some compiler module has been loaded. Once the compiler module has been activated corresponding mpi modules, i.e. built with the particular compiler, become visible.
E.g., with the originally empty list of the loaded modules, the module command
$ module avail
or its shorthand analogue
$ ml av
displays no mpi module available. After running
ml compiler/intel/19.1 ml av
mpi packages compatible with the intel 19.1 compiler becomes visible
--------------------------------------------------------------- /opt/bwhpc/common/modulefiles/Compiler/intel/19.1 --------------------------------------------------------------- mpi/impi/2019.7 mpi/openmpi/4.0
in the list of the available software.
MINIMALISTIC DESCRIPTION OF LMOD MODULE SYSTEM ON JUSTUS 2
JUSTUS 2 system uses Lmod (the Lua Based Module System) environment management package to support dynamical modification of the user environment via modulefiles. Each modulefile contains information needed to configure the shell, i.e. its environmental variables, required to run a particular software application. To make major changes in the users' environment, such as modifications of PATH, LD_LIBRARY_PATH, CC or FC variables for, e.g. a switching to a different compiler, use the appropriate single or sequence of modules commands instead of a reconfiguration of the shell by hand.
E.g. in order to activate the default vasp module run a single command:
$ module load chem/vasp
or its shorthand variant
$ml chem/vasp
For an activation of the module of intel mpi built with the corresponding intel compiler a sequence of two module commands
$ module load compiler/intel $ module load mpi/impi
has to be run.
Online user guide of Lmod
The complete user guide can be found on Lmod websites https://lmod.readthedocs.io/en/latest/010_user.html
Which shells supports module commands?
So far Bash is only supported shell on JUSTUS 2 to interpret module commands.
Most basic tasks for Module System
Lmod offers more than 25 sub-commands plus various options to manage the modulefile system installed on JUSTUS 2. See, e.g. output of "module --help" command. Large majority of users will use only couple of them. A complete list of module sub-commands can be displayed by entering "module --help" command.
Loading and unloading modulefiles
To load and unload a module of a specific package use "module load category/package" and "module unload category/package", respectively. Thus, e.g, to load default version of VASP which belongs into category chem, enter
$ module load chem/VASP
and for unloading of the package use
$ module unload chem/VASP
Complete unloading of all modulefiles
To clean the environment from all entries set by lmod use "module purge" command.
$ module purge
Beware that the "module purge" does not always restore the environmental variables that existed prior entering first "module load" command. I.e., some of the entries in PATH, or environmental variables, etc. originally existed in the environment might be cleaned by "module purge" if those would be duplicates of the variables set by any modulefiles that was loaded.
What is Loaded Now?
To see which modules are currently loaded in your environment, enter the command "module list".
$ module list
Which modulefiles/software is available?
To see what modulefiles are available on JUSTUS 2, you can enter the following command:
$ module avail
or alternatively
$ ml av
which comes with lmod to save typewriting.
Module specific help
To access the module specific help, use the "module help" command. For example, to see the module help associated with the default VASP module, enter this command:
$ module help chem/vasp
It is always a good practice to read through all the info provided by a specific module help since this serves as an important communication channel between a user and the person who install the piece of a software.
JUSTUS 2 specific configuration of module systems
The structure of software modules on JUSTUS 2 exploits a "semi" hierarchical structure. This is slightly different from what can be seen on another HPC systems with "full" hierarchical structure. Systems with full hierarchy typically compiler modules (i.e., intel, gcc) are located on the uppermost level, depending libraries (e.g., MPI) on the second level, and more depending libraries on a third level. This means that not all the modules are visible initially: only after loading a compiler module, the modules directly depending on that compiler will become available. And similarly, loading an MPI module in addition will make the modules depending on the MPI library available.
Software module semi-hierarchy
Not all modules even though installed are visible upon entering of "module avail" command. In particular modules falling into MPI category. As a matter of fact, available MPI packages become visible not until there is some activated compiler module. This is due site specific module hierarchy.
Many modules depends on another modules. Except the modules in compiler category, it is the case for almost all software modules installed on JUSTUS 2. In another words, for a correct functioning of the most modules another modules must be simultaneously activated. Most of the modules manage that automatically or the user is instructed to reload a particular module/s to fulfill a specific module dependency.
General Introduction
Environment Modules, or short Modules are the means by which most of the installed scientific software is provided on JUSTUS 2.
The use of different compilers, libraries and software packages requires users to set up a specific session environment suited for the program they want to run. JUSTUS 2 provides users with the possibility to load and unload complete environments for compilers, libraries and software packages by a single command.
General Description
The Environment Modules package enables dynamic modification of your environment by the
use of so-called modulefiles. A modulefile contains information to configure the shell
for a program/software . Typically, a modulefile contains instructions that alter or set shell
environment variables, such as PATH and MANPATH, to enable access to various installed
software.
One of the key features of using the Environment Modules software is to allow multiple versions of the same software to be used in your environment in a controlled manner.
For example, two different versions of the Intel C compiler can be installed on the system at the same time - the version used is based upon which Intel C compiler modulefile is loaded.
The software stack of JUSTUS 2 provides a number of modulefiles. You can also
create your own modulefiles. Modulefiles may be shared by many users on a system, and
users may have their own collection of modulefiles to supplement or replace the shared
modulefiles.
A modulefile does not provide configuration of your environment until it is explicitly loaded,
i.e., the specific modulefile for a software product or application must be loaded in your environment before the configuration information in the modulefile is effective.
If you want to see which modules are loaded you must execute
'module list'.
$ module list Currently Loaded Modules: 1) compiler/intel/19.1 2) mpi/impi/2019 3) numlib/mkl/2019
Usage
Lmod on JUSTUS 2: A New Environment Module System from http://lmod.readthedocs.org/en/latest/ is installed.
Documentation
Execute 'module help' or 'man module' for help on how to use Modules software.
$ module help
Usage: module [options] sub-command [args ...]
Options:
-h -? -H --help This help message
-s availStyle --style=availStyle Site controlled avail style: system (default: system)
--regression_testing Lmod regression testing
-D Program tracing written to stderr
--debug=dbglvl Program tracing written to stderr (where dbglvl is a number 1,2,3)
--pin_versions=pinVersions When doing a restore use specified version, do not follow defaults
-d --default List default modules only when used with avail
-q --quiet Do not print out warnings
--expert Expert mode
-t --terse Write out in machine readable format for commands: list, avail, spider, savelist
--initial_load loading Lmod for first time in a user shell
--latest Load latest (ignore default)
--ignore_cache Treat the cache file(s) as out-of-date
--novice Turn off expert and quiet flag
--raw Print modulefile in raw output when used with show
-w twidth --width=twidth Use this as max term width
-v --version Print version info and quit
-r --regexp use regular expression match
--gitversion Dump git version in a machine readable way and quit
--dumpversion Dump version in a machine readable way and quit
--check_syntax --checkSyntax Checking module command syntax: do not load
--config Report Lmod Configuration
--config_json Report Lmod Configuration in json format
--mt Report Module Table State
--timer report run times
--force force removal of a sticky module or save an empty collection
--redirect Send the output of list, avail, spider to stdout (not stderr)
--no_redirect Force output of list, avail and spider to stderr
--show_hidden Avail and spider will report hidden modules
--spider_timeout=timeout a timeout for spider
-T --trace
module [options] sub-command [args ...]
Help sub-commands:
------------------
help prints this message
help module [...] print help message from module(s)
Loading/Unloading sub-commands:
-------------------------------
load | add module [...] load module(s)
try-load | try-add module [...] Add module(s), do not complain if not found
del | unload module [...] Remove module(s), do not complain if not found
swap | sw | switch m1 m2 unload m1 and load m2
purge unload all modules
refresh reload aliases from current list of modules.
update reload all currently loaded modules.
Listing / Searching sub-commands:
---------------------------------
list List loaded modules
list s1 s2 ... List loaded modules that match the pattern
avail | av List available modules
avail | av string List available modules that contain "string".
spider List all possible modules
spider module List all possible version of that module file
spider string List all module that contain the "string".
spider name/version Detailed information about that version of the module.
whatis module Print whatis information about module
keyword | key string Search all name and whatis that contain "string".
Searching with Lmod:
--------------------
All searching (spider, list, avail, keyword) support regular expressions:
-r spider '^p' Finds all the modules that start with `p' or `P'
-r spider mpi Finds all modules that have "mpi" in their name.
-r spider 'mpi$ Finds all modules that end with "mpi" in their name.
Handling a collection of modules:
--------------------------------
save | s Save the current list of modules to a user defined "default" collection.
save | s name Save the current list of modules to "name" collection.
reset The same as "restore system"
restore | r Restore modules from the user's "default" or system default.
restore | r name Restore modules from "name" collection.
restore system Restore module state to system defaults.
savelist List of saved collections.
describe | mcc name Describe the contents of a module collection.
disable name Disable (i.e. remove) a collection.
Deprecated commands:
--------------------
getdefault [name] load name collection of modules or user's "default" if no name given.
===> Use "restore" instead <====
setdefault [name] Save current list of modules to name if given, otherwise save as the default list for you the user.
===> Use "save" instead. <====
Miscellaneous sub-commands:
---------------------------
is-loaded modulefile return a true status if module is loaded
is-avail modulefile return a true status if module can be loaded
show modulefile show the commands in the module file.
use [-a] path Prepend or Append path to MODULEPATH.
unuse path remove path from MODULEPATH.
tablelist output list of active modules as a lua table.
Important Environment Variables:
--------------------------------
LMOD_COLORIZE If defined to be "YES" then Lmod prints properties and warning in color.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Lmod Web Sites
Documentation: http://lmod.readthedocs.org
Github: https://github.com/TACC/Lmod
Sourceforge: https://lmod.sf.net
TACC Homepage: https://www.tacc.utexas.edu/research-development/tacc-projects/lmod
To report a bug please read http://lmod.readthedocs.io/en/latest/075_bug_reporting.html
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Modules based on Lua: Version 8.2 (8.2-1-g9c98036c) 2019-10-30 11:17 -05:00
by Robert McLay mclay@tacc.utexas.edu
For help on particular version of Module, e.g. Intel default compiler version, execute 'module help compiler/intel'.
$ module help compiler/intel ---------------------- Module Specific Help for "compiler/intel/19.1" ---------------------- Intel(R) Compilers 19.1 for Linux* For details see: https://software.intel.com/en-us/intel-compilers In case of problems, please contact: Hartmut Häfner <hartmut.haefner@kit.edu> SCC support end: 2022-12-31
Online Documentation
Lmod: A New Environment Module System
Display all available Modules
Available Module are modulefiles that can be loaded by the user. A Module must be loaded before it provides changes to your environment, as described in the introduction to this section. You can display all available Modules on the system by executing:
$ module avail
The short form the command is:
$ module av
Available Modules can be also displayed in different modes, such as
- each Module per one line
$ module -t avail
Some modules may not be available right now, because their requirements are not met. To get a complete list of all possible modules use the module spider command.
Module categories, versions and defaults
The ForHLR clusters traditionally provide a large variety of software and software versions. Therefore Module are divided in category folders containing subfolders of modulefiles again containing modulefile versions, and must be addressed as follows:
category/softwarename/version
For instance all versions of the Intel compiler belong to the category of compilers, thus the corresponding modulefiles are placed under the category compiler and intel.
In case of multiple software versions, one version will be always defined as the default
version. The Module of the default can be addressed by simply omitting the version number:
category/softwarename
Finding software Modules
Currently all JUSTUS 2 software packages are assigned to the following Module categories (???):
- chem
- compiler
- devel
- lib
- numlib
- phys
- system
- vis
You can selectively list software in one of those categories using, e.g. for the category "compiler"
$ module avail compiler/
Searches are looking for a substring starting at the begin of the name, so this would list all software in categories starting with a "c"
$ module avail c
while this would find nothing
$ module avail hem
Loading Modules
You can load a Module software in to your environment to enable easier access to software that you want to use by executing:
$ module load category/softwarename/version
or
$ module add category/softwarename/version
Loading a Module in this manner affects ONLY your environment for the current session.
Loading conflicts
You can not load different versions of the same software at the same time! Loading the Intel compiler in version X while Intel compiler in version Y is loaded leads to an automatic unloading of Intel compiler in version Y.
Showing the changes introduced by a Module
Loading a Module will change the environment of the current shell session. For instance the $PATH variable will be expanded by the software's binary directory. Other Module variables may even change the behavior of the current shell session or the software program(s) in a more drastic way.
All the changes to the current shell session to be invoked by loading the Module can be reviewed using
'module show category/softwarename/version'.
Example (Intel compiler)
$ module show compiler/intel
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
/opt/bwhpc/common/modulefiles/Core/compiler/intel/19.1.lua:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
setenv("INTEL_LICENSE_FILE","28518@scclic1.scc.kit.edu")
setenv("AR","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64/xiar")
setenv("CC","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64/icc")
setenv("CXX","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64/icpc")
setenv("F77","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64/ifort")
setenv("FC","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64/ifort")
setenv("CFLAGS","-O2 -xCORE-AVX2")
setenv("CXXFLAGS","-O2 -xCORE-AVX2")
setenv("FFLAGS","-O2 -xCORE-AVX2")
setenv("FCFLAGS","-O2 -xCORE-AVX2")
setenv("INTEL_VERSION","19.1.0.166")
setenv("INTEL_HOME","/opt/intel/compilers_and_libraries_2020/linux")
setenv("INTEL_BIN_DIR","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64")
setenv("INTEL_LIB_DIR","/opt/intel/compilers_and_libraries_2020/linux/lib/intel64")
setenv("INTEL_INC_DIR","/opt/intel/compilers_and_libraries_2020/linux/include")
setenv("INTEL_MAN_DIR","/opt/intel/compilers_and_libraries_2020/linux/man/common")
setenv("INTEL_DOC_DIR","/opt/intel/compilers_and_libraries_2020/linux/documentation/en")
setenv("GDB_VERSION","19.1.0.166")
setenv("GDB_HOME","/opt/intel/debugger_2020/gdb/intel64")
setenv("GDB_BIN_DIR","/opt/intel/debugger_2020/gdb/intel64/bin")
setenv("GDB_LIB_DIR","/opt/intel/debugger_2020/libipt/intel64/lib")
setenv("GDB_INC_DIR","/opt/intel/debugger_2020/gdb/intel64/include")
setenv("GDB_INF_DIR","/opt/intel/documentation_2020/en/debugger/gdb-ia/info")
setenv("GDB_MAN_DIR","/opt/intel/documentation_2020/en/debugger/gdb-ia/man")
setenv("KMP_AFFINITY","noverbose,granularity=core,respect,warnings,compact,1")
prepend_path("PATH","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64")
prepend_path("MANPATH","/opt/intel/compilers_and_libraries_2020/linux/man/common")
prepend_path("LD_LIBRARY_PATH","/opt/intel/compilers_and_libraries_2020/linux/lib/intel64")
whatis("Sets up Intel C/C++ and Fortran compiler version 19.1 (Intel(R) Compilers 19.1 for Linux*) - supported by SCC till 2022-12-31!")
help([[Intel(R) Compilers 19.1 for Linux*
For details see: https://software.intel.com/en-us/intel-compilers
In case of problems, please contact: Hartmut Häfner <hartmut.haefner@kit.edu>
SCC support end: 2022-12-31]])
prepend_path("MODULEPATH","/software/bwhpc/common/modulefiles/Compiler/intel/19.1")
family("compiler")
Modules depending on Modules
Some program Modules depend on libraries to be loaded to the user environment. Therefore the
corresponding Modules of the software must be loaded together with the Modules of
the libraries.
By default such software Modules try to load required Modules and corresponding versions automatically.
Unloading Modules
To unload or remove a software Module execute:
$ module unload category/softwarename/version
or
$ module remove category/softwarename/version
Unloading all loaded modules
Purge
Unloading a Module that has been loaded by default makes it inactive for the current session only - it will be reloaded the next time you log in.
In order to remove all previously loaded software modules from your environment issue the command 'module purge'.
Example
$ module list Currently Loaded Modules: 1) compiler/intel/19.1 2) mpi/impi/2019 3) numlib/mkl/2019 $ $ module purge $ module list No modules loaded $
Beware!
'module purge' is working without any further inquiry.
Display your loaded Modules
All Modules that are currently loaded for you can be displayed by the
command 'module list'. See example above.
Note: You only have to load further Modules, if you want to use additional software
packages or to change the version of an already loaded software.
Display all possible Modules
Modulefiles can be searched by the user. You can dipslay all possible modules by executing:
$ module spider ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The following is a list of the modules and extensions currently available: ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cae/abaqus: cae/abaqus/2018, cae/abaqus/2019 cae/adina: cae/adina/9.1.2 cae/ansys: cae/ansys/19.2, cae/ansys/2019R3, cae/ansys/2020R1 cae/comsol: cae/comsol/5.4, cae/comsol/5.5 cae/cst: cae/cst/2018 cae/lsdyna: cae/lsdyna/901 cae/openfoam: cae/openfoam/v1912, cae/openfoam/2.4.x, cae/openfoam/6, cae/openfoam/7 cae/paraview: cae/paraview/5.8 cae/starccm+: cae/starccm+/14.02.010, cae/starccm+/2019.2.1 cae/starcd: cae/starcd/4.28 compiler/clang: compiler/clang/9.0 compiler/gnu: compiler/gnu/9.2 compiler/intel: compiler/intel/18.0, compiler/intel/19.0, compiler/intel/19.1 compiler/pgi: compiler/pgi/2019 devel/cmake: devel/cmake/3.16 devel/cuda: devel/cuda/9.2, devel/cuda/10.0, devel/cuda/10.2 devel/gdb: devel/gdb/9.1 devel/python: devel/python/3.7.4_gnu_9.2, devel/python/3.8.1_gnu_9.2, devel/python/3.8.1_intel_19.1 math/R: math/R/3.6.3 math/julia: math/julia/1.3.1 mpi/impi: mpi/impi/2018, mpi/impi/2019, mpi/impi/2020 mpi/openmpi: mpi/openmpi/4.0 numlib/mkl: numlib/mkl/2018, numlib/mkl/2019, numlib/mkl/2020 numlib/python_numpy: numlib/python_numpy/1.17.2_python_3.7.4_gnu_9.2 numlib/python_scipy: numlib/python_scipy/1.3.1_numpy_1.17.2_python_3.7.4_gnu_9.2 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To learn more about a package execute: $ module spider Foo where "Foo" is the name of a module. To find detailed information about a particular package you must specify the version if there is more than one version: $ module spider Foo/11.1 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
'module spider name/version' : If you search the full name and version of the module, the search gives detailed information about that module version.
$ module spider devel/cmake
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
devel/cmake: devel/cmake/3.16
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
This module can be loaded directly: module load devel/cmake/3.16
Help:
Home page: https://www.cmake.org
Online Documentation: https://www.cmake.org/HTML/Documentation.html
Local Documentation: /opt/bwhpc/common/devel/cmake/3.16.4/docFAQ: https://gitlab.kitware.com/cmake/community/wikis/FAQ
Moreover, you can see the dependencies of the module with using the same command. For example, if the following is executed, you can see which modules need to be loaded before loading the module mpi/impi/2019
$ module spider mpi/impi/2019
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
mpi/impi: mpi/impi/2019
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
You will need to load all module(s) on any one of the lines below before the "mpi/impi/2019" module is available to load.
compiler/clang/9.0
compiler/gnu/9.2
compiler/intel/18.0
compiler/intel/19.0
compiler/intel/19.1
Help:
Intel(R) MPI Library
How do Modules work?
The default shell on the bwHPC clusters is bash, so explanations and examples will be shown for bash. In general, programs cannot modify the environment of the shell they are being run from, so how can the module command do exactly that?
The module command is not a program, but a bash-function.
You can view its content using:
$ type module
and you will get the following result:
$ type module
module is a function
module ()
{
eval $($LMOD_CMD bash "$@");
[ $? = 0 ] && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}
In this function, lmod is called. Its output to stdout is then executed inside your current shell using the bash-internal eval command. As a consequence, all output that you see from the module is transmitted via stderr (output handle 2) or in so