BwUniCluster3.0/Software Modules

Introduction

Software (Environment) Modules, or short modules are the means by which most of the installed scientific software is provided on bwUniCluster 3.0.

Different compilers, libraries and software packages requires users to set up a specific session environment suited for the program they want to run. bwUniCluster 3.0 provides users with the possibility to load and unload complete environments for compilers, libraries and software packages using a single command.

Basics

The Environment modules package enables dynamic modification of shell environments. Typically, a modulefile contains instructions that alter or set shell environment variables, such as PATH and MANPATH, to enable access to various installed software.

One of the key features of Environment Modules is the ability to have multiple versions of the same software available in a controlled manner. For example, two different versions of the Intel C compiler can be installed on the system at the same time - the version used is based upon which Intel C compiler modulefile is loaded.

The software stack of bwUniCluster 3.0 provides a number of modules. Users can also provide modules for their own software. Modules may be shared between users on a system, and users may have their own collection of modules to supplement or replace the global modules.

To see which modules are currently loaded/enabled one must execute

$ module list
No modules loaded

If the user has just logged in no modules should be loaded.

If a user has setup his/her environment for an Intel toolchain the output may look like this

$ module list

Currently Loaded Modules:
  1) compiler/intel/2023.1.0   2) numlib/mkl/2022.2.1   3) mpi/impi/2021.11

Usage

bwUniCluster 3.0 has been set up to use the Lmod module system. Lmod is a modern replacement of the older tcl-based environment modules. The complete Lmod documentation can be found [[http://lmod.readthedocs.org/en/latest/][here]].

Getting help

The Lmod module command provides a manual page (man module) and a help command line option. Execute 'module help' or 'man module' for help on how to use Lmod software.

$ module help
Usage: module [options] sub-command [args ...]

Options:
  -h -? -H --help                   This help message
...
a lot of help
...
Lmod Web Sites

  Documentation:    https://lmod.readthedocs.org
  GitHub:           https://github.com/TACC/Lmod
  SourceForge:      https://lmod.sf.net
  TACC Homepage:    https://www.tacc.utexas.edu/research-development/tacc-projects/lmod

  To report a bug please read https://lmod.readthedocs.io/en/latest/075_bug_reporting.html

Modules based on Lua: Version 8.7.55 2024-12-13 12:24 -07:00
    by Robert McLay mclay@tacc.utexas.edu

To get help for a particular module, e.g. the default GNU compiler, run

$ module help compiler/gnu

------------- Module Specific Help for "compiler/gnu/14.2" -------------------------------
This module provides the GNU compiler collection version 14.2.0 via commands
gcc, g++, gfortran and gccgo. The GNU compiler has been build with the libraries
gmp (6.3.0), mpfr (4.2.1), mpc (1.3.1) and isl (0.27).
This compiler supports offloating OpenACC and OpenMP to NVIDIA CUDA using -foffload.

cpp      - GNU pre processor
gcc      - GNU C compiler
g++      - GNU C++ compiler
gfortran - GNU Fortran compiler (Fortran 95/2003/2008 plus legacy Fortran 77)
gccgo    - GNU Google's Go language

Libraries can be found in
  \$GNU_HOME/lib64 = $env(GNU_HOME)/lib64
One may require additionally libraries installed in
  \$GNU_HOME/lib   = $env(GNU_HOME)/lib

Local documentation:
  See commands 'man cpp', 'man gcc', 'man g++' and 'man gfortran'.

Online documentation:
  https://gcc.gnu.org/onlinedocs/

For details on library and include dirs please call
    module show

The man pages, environment variables and compiler commands
are available after loading '[module-info name]'.

In case of problems, submit a trouble ticket at 'https://bw-support.scc.kit.edu'.

The full version is: compiler/gnu/14.2.0

Display all available Modules

bwUniCluster3.0 comes with a variety of modules pre-installed. Some of these modules may not be available for all users of all institutions, due to licensing restrictions. You can display all available modules by running:

$ module avail

Due to the fact that some modules depend on other modules, some modules may not be shown as available all the time. To get a complete list of all possible modules use the module spider command.

Module categories, versions and defaults

The bwHPC clusters traditionally provide a large variety of software and software versions. Therefore modules are divided into category folders containing subfolders of modulefiles again containing modulefile versions, and must be addressed as follows:

category/softwarename/version

For instance all versions of the Intel compiler belong to the category compiler, thus the corresponding modulefiles are placed under the category compiler and intel.

In case of multiple software versions, one version will be defined as the default version. The Module of the default can be addressed by simply omitting the version number:

category/softwarename

Please be aware, that the default version may change over time.

Finding software Modules

Currently all bwUniCluster 3.0 software packages are assigned to the following Module categories:

• bio
• cae
• chem
• compiler
• devel
• lib
• math
• mpi
• numlib
• phys
• system
• toolkit
• vis

You can selectively list software in one of those categories using, e.g. for the category "compiler"

$ module avail compiler

Searches with module avail are substring searches and will return any module with the string in its name.

Loading Modules

To use a software provided through modules in your environment, run:

$ module load category/softwarename/version

or

$ module add category/softwarename/version

Loading a Module in this manner affects your environment only for the current session or until you unload the module again

module conflicts

For some software, it doesn't make sense to provide multiple versions of the same software in the same environment. For this reason, module will automatically unload a conflicting version of the same software, e.g. loading the Intel compiler in version X while Intel compiler in version Y is loaded leads to an automatic unloading of Intel compiler in version Y.

Showing the changes introduced by a Module

Loading a module will change the environment of the current shell session. For instance the $PATH variable will be expanded by the software's bin directory. Other module variables may even change the behavior of the current shell session or the software program(s) in a more drastic way.

All the changes to the current shell session to be invoked by loading the module can be reviewed using 'module show category/softwarename/version', if version is omitted, information for the default version will be shown, e.g. for the default GNU compiler

$ module show compiler/gnu
-------------------------------------------------------------------------------------------------------
   /opt/bwhpc/common/modulefiles/Core/compiler/gnu/13.3.lua:
-------------------------------------------------------------------------------------------------------
setenv("GNU_VERSION","13.3.0")
setenv("GNU_HOME","/opt/bwhpc/common/compiler/gnu/13.3.0")
setenv("GNU_BIN_DIR","/opt/bwhpc/common/compiler/gnu/13.3.0/bin")
setenv("GNU_LIB_DIR","/opt/bwhpc/common/compiler/gnu/13.3.0/lib64")
setenv("GNU_INC_DIR","/opt/bwhpc/common/compiler/gnu/13.3.0/include")
setenv("GNU_MAN_DIR","/opt/bwhpc/common/compiler/gnu/13.3.0/share/man")
prepend_path("PATH","/opt/bwhpc/common/compiler/gnu/13.3.0/bin")
prepend_path("LD_LIBRARY_PATH","/opt/bwhpc/common/compiler/gnu/13.3.0/lib64")
prepend_path("MANPATH","/opt/bwhpc/common/compiler/gnu/13.3.0/share/man")
conflict("compiler/intel")
conflict("compiler/pgi")
whatis("GNU compiler suite version 13.3.0  (gcc, g++, gfortran, gccgo) including LTO and JIT ")
help([[This module provides the GNU compiler collection version 13.3.0 via commands
gcc, g++, gfortran and gccgo. The GNU compiler has been build with the libraries
gmp (6.3.0), mpfr (4.2.1), mpc (1.3.1) and isl (0.26).
This compiler supports offloating OpenACC and OpenMP to NVIDIA CUDA using -foffload.

cpp      - GNU pre processor
gcc      - GNU C compiler
g++      - GNU C++ compiler
gfortran - GNU Fortran compiler (Fortran 95/2003/2008 plus legacy Fortran 77)
gccgo    - GNU Google's Go language

Libraries can be found in
  \$GNU_HOME/lib64 = $env(GNU_HOME)/lib64
One may require additionally libraries installed in
  \$GNU_HOME/lib   = $env(GNU_HOME)/lib

Local documentation:
  See commands 'man cpp', 'man gcc', 'man g++' and 'man gfortran'.

Online documentation:
  https://gcc.gnu.org/onlinedocs/

For details on library and include dirs please call
    module show [module-info name]

The man pages, environment variables and compiler commands
are available after loading '[module-info name]'.

In case of problems, submit a trouble ticket at 'https://bw-support.scc.kit.edu'.

The full version is: compiler/gnu/13.3.0

]])
prepend_path("MODULEPATH","/software/bwhpc/common/modulefiles/Compiler/gnu/13.3")
family("compiler")

Dependencies between modules

Most modern software depends on shared libraries. Shared libraries must be accesible during the runtime of the software. This is true for basic OS libraries as well as for more specialized libraries like math or communication libraries. For basic OS libraries, one version is usually enough for all software running on a system. For the more specialized libraries, different user may prefer different versions or even implementations of these libraries. Theses situations can be represented using modules. The modules contain the information, what tools are necessary to run a specific software and load all required software automatically.

Unloading Modules

To unload or remove a module run:

$ module unload category/softwarename/version

or

$ module remove category/softwarename/version

Unloading all loaded modules

1. Purge

   Loading different modules may lead to a expansive list of modules, especially when using EasyBuild. To unload all currently loaded modules use the 'module purge' command like this:

   $ module list
   Currently Loaded Modules:
     1) compiler/intel/19.1   2) mpi/impi/2019   3) numlib/mkl/2019
   $ module purge
   $ module list
   No modules loaded
   

   Be aware that 'module purge' is working without any further inquiry.

Display your loaded Modules

All currently loaded modules can be displayed using the 'module list' command. See example above.

Display all possible Modules

Some of the modules may not be visible with the 'module avail' command to due to unmet dependencies. All available and currently not available modules can be listed using the 'module spider' command:

$ module spider
-----------------------------------------------------------------------------------------------
The following is a list of the modules and extensions currently available:
-----------------------------------------------------------------------------------------------
  compiler/gnu: compiler/gnu/11.4, compiler/gnu/14.2

  devel/code-server: devel/code-server/4.96.4

  devel/cuda: devel/cuda/12.8

  ... long list of modules and versions ...

To learn more about a package execute:

   $ module spider Foo

where "Foo" is the name of a module.

To find detailed information about a particular package you
must specify the version if there is more than one version:

   $ module spider Foo/11.1

-----------------------------------------------------------------------------------------------

To see all versions of a software in a category use 'module spider category/name/'

$ module spider compiler/gnu

-----------------------------------------------------------------------------------------------
  compiler/gnu:
-----------------------------------------------------------------------------------------------
     Versions:
        compiler/gnu/11.4
        compiler/gnu/14.2

-----------------------------------------------------------------------------------------------
  For detailed information about a specific "compiler/gnu" package (including how to load the modules) use the module's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.
  For example:

     $ module spider compiler/gnu/14.2
-----------------------------------------------------------------------------------------------

module spider may be used to show the dependencies of the module. For example, you can see which modules need to be loaded before loading the module mpi/impi/2021.11 is possible

$ module spider mpi/impi/2021.11

----------------------------------------------------------------------
  mpi/impi: mpi/impi/2021.11
----------------------------------------------------------------------

    You will need to load all module(s) on any one of the lines below before the "mpi/impi/2021.11" module is available to load.

      compiler/intel/2021.4.0
      compiler/intel/2021.4.0_llvm
      compiler/intel/2022.2.1
      compiler/intel/2022.2.1_llvm
      compiler/intel/2023.1.0
      compiler/intel/2023.1.0_llvm
      compiler/intel/2024.0_llvm

    Help:
      Intel MPI Library version 2021.11

      From the product website
      (https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/mpi-library.html):

      Intel MPI Library
      Deliver flexible, efficient, and scalable cluster messaging.

      One Library with Multiple Fabric Support
      Intel MPI Library is a multifabric message-passing library that implements the
      open-source MPICH specification. Use the library to create, maintain, and test
      advanced, complex applications that perform better on high-performance
      computing (HPC) clusters based on Intel processors.

      * Develop applications that can run on multiple cluster interconnects that you
        choose at run time.
      * Quickly deliver maximum end-user performance without having to change the
        software or operating environment.
      * Achieve the best latency, bandwidth, and scalability through automatic tuning
        for the latest Intel platforms.
      * Reduce the time to market by linking to one library and deploying on the
        latest optimized fabrics.

EasyBuild modules

With bwUniCluster 3.0 we start an official beta-test to provide toolchains and scientific software built with EasyBuild. EasyBuild is a well established and consistent way to provide scientific software at many HPC sites all over the world. Since the naming of Easybuild modules is consistent users can use modules with the same name at different sites. To make this possible, Easybuild software minimizes the resources used by the local operating system and provides most software independently. This makes scientific results more reproducible.

Using EasyBuild modules

Users interested in using easyBuild modules need to add the Easybuild module paths to their module environment. This can be done using

source /opt/bwhpc/common/etc/easybuild/enable_eb_modules

Now users have acces to a comlete Easybuild module tree, providing compiler toolchains, libraries, and scientific software. The same module commands described above can be used for EasyBuild modules. They can be loaded and unloaded in the same way. Please try not to mix Easybuild modules with the standard modules. There may be unintended consequences. It's always a good idea to purge all modules before loading new ones. If you want to use easybuild modules in your jobscripts, please make sure to enable them in your jobscript as well.

To hide the EasyBuild modules again, run:

source /opt/bwhpc/common/etc/easybuild/disable_eb_modules

or start again with a clean shell

EasyBuild conventions

In theory one could build every software against every library and every compiler. This would lead to an unmaintainable amount of combinations. For this reason the EasyBuild team has decided to provide two so called toolchains a year for the most common compiler suits. Each toolchain defines one version of compiler, MPI library, math libraries and other packages as default for all software build aginst this toolchain. On bwUniCluster 3.0 we start with 3 foss, 3 intel and one nvhpc toolchains. For a complete list of EasyBuild toolchains, please see the EasyBuild website. Currently the 2023a, 2023b, and 2024a toolchains are available, with 2024a being the default one. Some software may not be available for the latest toolchain yet, but may be available for an older toolchain.

EasyBuild currently supports almost 4000 software packages. We can and will only provide a subset, of course. As a start, we provide current versions of popular software packages form bwUniCluster 2.0. Updating and extending EasyBuild packages is a community effort, which should make providing new software easier. We try to provide additional packages if users request the installation. If we can't provide the software you can still use the advantages of EasyBuild by using it to build your private version.

Build individual EasyBuild packages as a user

EasyBuild can also be used to compile your own or niche software, based on the available EasyBuild files by creating an independent EasyBuild software and module tree. The bwUnicluster 3.0 EasyBuild tree can be used as a base to build additional software in your home directory.

We assume the following

• the sofware will be installed in your home directory in the sub-directory eb
• you have already cloned the git repository from https://github.com/easybuilders/easybuild-easyconfigs to ~/eb/easybuild-easyconfigs
• create an easybuild configuration file in your home at ~/.config/easybuild/config.cfg

[basic]
robot=<path_to_your_home>/eb/easybuild-easyconfigs/easybuild/easyconfigs
[config]
modules-tool: Lmod
prefix=<path_to_your_home>/eb
buildpath=/scratch/build
[override]
allow-modules-tool-mismatch=True

• add system-wide easybuild module path

source /opt/bwhpc/common/etc/easybuild/enable_eb_modules

• add local module path

mkdir -p ~/eb/modules/all
module use ~/eb/modules/all

• find or create easybuild file, you can use any of the easybild

ls ~/eb/easybuild-easyconfigs/easybuild/easyconfigs/*/* | grep eb$

• build your software with EasyBuild

eb cowsay-3.04.eb

• use your software

$ module load cowsay
$ cowsay "Hello EasyBuild"
 _________________
< Hello EasyBuild >
 -----------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

How do Modules work?

The default shell on the bwUniCluster 3.0 is bash, so explanations and examples will be shown for bash. In general, programs cannot modify the environment of the shell they are being run from, so how can the module command do exactly that? The module command is not a program, but a bash-function. You can view its content using:

$ type module

and you will get the following result:

$ type module
module is a function
module ()
{
    eval $($LMOD_CMD bash "$@");
    [ $? = 0 ] && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}

In this function, lmod is called. Its output to stdout is then executed inside your current shell using the bash-internal eval command. As a consequence, all output that you see from the module is transmitted via stderr (output handle 2) or in some cases even stdin (output handle 0). back to top