Difference between revisions of "Environment Modules"

From bwHPC Wiki
Jump to: navigation, search
(Showing the changes introduced by a Module)
m
Line 1: Line 1:
  +
{| width=600px class="wikitable"
'''Environment Modules''', or short '''Modules''' are the means by which most of the installed scientific software is provided on the bwHPC clusters.
 
 
The use of different compilers, libraries and software packages requires users to set up a specific session environment suited for the program they want to run. The bwHPC clusters provide users with the possibility to load and unload complete environments
 
for compilers, libraries and software packages by a single command.
 
 
<!--{| align="right" {{Table|width=40%}} -->
 
{|{{Softwarebox}}
 
 
|-
 
|-
  +
! Description !! Content
! colspan="2" style="text-align:center" | Modules
 
<!--
 
 
|-
 
|-
 
| module load
 
| module load
| category/name
+
| category/name &#124; category/name/version (optional)
-->
 
 
|-
 
|-
  +
| Availability
| License
 
  +
| [[bwUniCluster]] &#124; [[BwForCluster_Chemistry]] &#124; bwGRiD_tu
| GPL
 
 
|-
 
|-
 
| Links
 
| Links
| http://modules.sourceforge.net/
+
| [http://modules.sourceforge.net/ Environment Modules Project]
|
 
|}
 
 
<!--
 
{| style="width: 100%; border-spacing: 5px;"
 
| style="text-align:left; color:#000;vertical-align:top;" |__TOC__
 
|<br><br>[[File:BwUniCluster_modulefiles.png|center|border|400px]]
 
 
|-
 
|-
  +
| License
  +
| [http://www.gnu.org/licenses/old-licenses/gpl-2.0.html GNU General Public License]]
 
|}
 
|}
-->
+
<br>
  +
= Introduction =
 
  +
'''Environment Modules''', or short '''Modules''' are the means by which most of the installed scientific software is provided on the bwHPC clusters.
 
  +
<br>
  +
The use of different compilers, libraries and software packages requires users to set up a specific session environment suited for the program they want to run. The bwHPC clusters provide users with the possibility to load and unload complete environments for compilers, libraries and software packages by a single command.
  +
<br>
  +
<br>
 
= Description =
 
= Description =
 
 
The Environment ''Modules'' package enables dynamic modification of your environment by the
 
The Environment ''Modules'' package enables dynamic modification of your environment by the
 
use of so-called ''modulefiles''. A ''modulefile'' contains information to configure the shell
 
use of so-called ''modulefiles''. A ''modulefile'' contains information to configure the shell
Line 38: Line 28:
 
environment variables, such as PATH and MANPATH, to enable access to various installed
 
environment variables, such as PATH and MANPATH, to enable access to various installed
 
software.
 
software.
  +
<br>
 
 
One of the key features of using the Environment ''Modules'' software is to allow multiple versions of the same software to be used in your environment in a controlled manner.
 
One of the key features of using the Environment ''Modules'' software is to allow multiple versions of the same software to be used in your environment in a controlled manner.
 
For example, two different versions of the Intel C compiler can be installed on the system at the same time - the version used is based upon which Intel C compiler modulefile is loaded.
 
For example, two different versions of the Intel C compiler can be installed on the system at the same time - the version used is based upon which Intel C compiler modulefile is loaded.
  +
<br>
 
 
The software stack of bwHPC clusters provides a number of modulefiles. You can also
 
The software stack of bwHPC clusters provides a number of modulefiles. You can also
 
create your own modulefiles. ''Modulefiles'' may be shared by many users on a system, and
 
create your own modulefiles. ''Modulefiles'' may be shared by many users on a system, and
 
users may have their own collection of modulefiles to supplement or replace the shared
 
users may have their own collection of modulefiles to supplement or replace the shared
 
modulefiles.
 
modulefiles.
  +
<br>
 
 
A modulefile does not provide configuration of your environment until it is explicitly loaded,
 
A modulefile does not provide configuration of your environment until it is explicitly loaded,
 
i.e., the specific modulefile for a software product or application must be loaded in your environment before the configuration information in the modulefile is effective. For instance
 
i.e., the specific modulefile for a software product or application must be loaded in your environment before the configuration information in the modulefile is effective. For instance
Line 53: Line 43:
 
$ module load compiler/intel
 
$ module load compiler/intel
 
</pre>
 
</pre>
  +
<br>
 
 
 
= Usage =
 
= Usage =
 
 
== Documentation ==
 
== Documentation ==
 
 
For help on how to use ''Modules'' software, i.e., the command '''module''',
 
For help on how to use ''Modules'' software, i.e., the command '''module''',
 
execute:
 
execute:
Line 68: Line 55:
 
$ man module
 
$ man module
 
</pre>
 
</pre>
 
 
For help on particular version of ''Module'', e.g. Intel compiler version X.Y, execute:
 
For help on particular version of ''Module'', e.g. Intel compiler version X.Y, execute:
 
<pre>
 
<pre>
 
$ module help compiler/intel/X.Y
 
$ module help compiler/intel/X.Y
 
</pre>
 
</pre>
  +
=== Online Documentation ===
 
  +
[[https://sourceforge.net/p/modules/wiki/FAQ/ Frequently Asked Questions (FAQ)]]
 
 
== Display all available Modules ==
 
== Display all available Modules ==
 
 
Available ''Module'' are modulefiles that can be loaded by the user. A ''Module'' must be loaded before it provides changes to your environment, as described in the introduction to this
 
Available ''Module'' are modulefiles that can be loaded by the user. A ''Module'' must be loaded before it provides changes to your environment, as described in the introduction to this
 
section. You can display all available ''Modules'' on the system by executing:
 
section. You can display all available ''Modules'' on the system by executing:
Line 86: Line 71:
 
$ module av
 
$ module av
 
</pre>
 
</pre>
 
 
Available ''Modules'' can be also displayed in different modes, such as
 
Available ''Modules'' can be also displayed in different modes, such as
 
* each ''Module'' per one line
 
* each ''Module'' per one line
Line 96: Line 80:
 
$ module -l avail
 
$ module -l avail
 
</pre>
 
</pre>
  +
== bwHPC CLuster Information System (CIS) ==
 
== Display all available and scheduled Modules using a browser GUI ==
+
A '''GUI-Version of all available and scheduled modules''' is available with our
  +
<br>
 
A '''GUI-Version of all available and scheduled modules''' is available with our '''CIS''' (Cluster Information System).
+
'''CIS''' (Cluster Information System).
 
<br>
 
<br>
 
Software Admins are able to '''announce new modules''', versions and complete new software, too.
 
Software Admins are able to '''announce new modules''', versions and complete new software, too.
 
<br>
 
<br>
 
Academic users and other interested parties can get a '''summay of all installed modules and module-help informations'''.
 
Academic users and other interested parties can get a '''summay of all installed modules and module-help informations'''.
<br><big>
+
<big>
[https://cis-hpc.uni-konstanz.de/prod.cis/ CIS: Cluster Information System]
+
[https://cis-hpc.uni-konstanz.de CIS: Cluster Information System]
</big><br>
+
</big>
  +
<br><br>
 
  +
[[File: cis.jpg]]
  +
<br>
 
== Module categories, versions and defaults ==
 
== Module categories, versions and defaults ==
 
 
The bwHPC clusters (such as [[bwUniCluster]]) traditionally provide a large variety of
 
The bwHPC clusters (such as [[bwUniCluster]]) traditionally provide a large variety of
 
software and software versions. Therefore ''Module'' are divided in category folders
 
software and software versions. Therefore ''Module'' are divided in category folders
Line 117: Line 102:
 
For instance the Intel compiler X.Y belongs to the category of compilers, therefore the
 
For instance the Intel compiler X.Y belongs to the category of compilers, therefore the
 
modulefile ''X.Y'' is placed under the category ''compiler'' and ''intel''.
 
modulefile ''X.Y'' is placed under the category ''compiler'' and ''intel''.
  +
<br>
 
 
In case of multiple software versions, one version will be always defined as the '''default'''
 
In case of multiple software versions, one version will be always defined as the '''default'''
 
version. The ''Module'' of the default can be addressed by simply omitting the version number:
 
version. The ''Module'' of the default can be addressed by simply omitting the version number:
 
category/softwarename
 
category/softwarename
 
 
== Finding software Modules ==
 
== Finding software Modules ==
 
 
Currently all bwHPC software packages are assigned to the following ''Module'' categories:
 
Currently all bwHPC software packages are assigned to the following ''Module'' categories:
 
<!-- add wiki category for each of those, possibly just as a link -->
 
<!-- add wiki category for each of those, possibly just as a link -->
Line 137: Line 120:
 
* [[:Category:System software|system]]
 
* [[:Category:System software|system]]
 
* [[:Category:Visualization|vis]]
 
* [[:Category:Visualization|vis]]
 
 
You can selectively list software in one of those categories using, e.g. for the category "compiler"
 
You can selectively list software in one of those categories using, e.g. for the category "compiler"
 
<pre>
 
<pre>
Line 150: Line 132:
 
$ module avail hem
 
$ module avail hem
 
</pre>
 
</pre>
<br>
 
 
 
== Loading Modules ==
 
== Loading Modules ==
 
You can load a ''Module'' software in to your environment to enable easier access to software that
 
You can load a ''Module'' software in to your environment to enable easier access to software that
Line 163: Line 143:
 
</pre>
 
</pre>
 
Loading a ''Module'' in this manner affects ONLY your environment for the current session.
 
Loading a ''Module'' in this manner affects ONLY your environment for the current session.
 
 
 
=== Loading conflicts ===
 
=== Loading conflicts ===
 
By default you can not load different versions of same software ''Module'' in same session. Loading for example Intel compiler version X while Intel compiler version Y is loaded results in error message as follows:
 
By default you can not load different versions of same software ''Module'' in same session. Loading for example Intel compiler version X while Intel compiler version Y is loaded results in error message as follows:
Line 171: Line 149:
 
</source>
 
</source>
 
The solution is [[#Unloading Modules|unloading]] or switching ''Modules''.
 
The solution is [[#Unloading Modules|unloading]] or switching ''Modules''.
<br><br>
 
 
 
=== Showing the changes introduced by a Module ===
 
=== Showing the changes introduced by a Module ===
 
Loading a ''Module'' will change the environment of the current shell session. For instance the $PATH variable will be expanded by the software's binary directory. Other ''Module'' variables may even change the behavior of the current shell session or the software program(s) in a more drastic way.
 
Loading a ''Module'' will change the environment of the current shell session. For instance the $PATH variable will be expanded by the software's binary directory. Other ''Module'' variables may even change the behavior of the current shell session or the software program(s) in a more drastic way.
  +
<br>
 
 
Loaded ''Modules'' may also invoke an additional set of environment variables, which e.g. point to directories or destinations of documentation and examples. Their nomenclature is systematic:
 
Loaded ''Modules'' may also invoke an additional set of environment variables, which e.g. point to directories or destinations of documentation and examples. Their nomenclature is systematic:
 
{|{{Table|width=50%}}
 
{|{{Table|width=50%}}
Line 196: Line 172:
 
|}
 
|}
 
with FOO being the placeholder of the software ''Module'' name.
 
with FOO being the placeholder of the software ''Module'' name.
  +
<br>
 
 
All the changes to the current shell session to be invoked by loading the ''Module'' can be reviewed using
 
All the changes to the current shell session to be invoked by loading the ''Module'' can be reviewed using
 
<pre>
 
<pre>
Line 204: Line 180:
 
<br>
 
<br>
 
<br>
 
<br>
<!-- 2014-05-08, RB: Depreciated, if really necessary then only for login, i.e. to be placed in .profile
 
=== Automatic Loading ===
 
-->
 
 
 
=== Modules depending on Modules ===
 
=== Modules depending on Modules ===
 
Some program ''Modules'' depend on libraries to be loaded to the user environment. Therefore the
 
Some program ''Modules'' depend on libraries to be loaded to the user environment. Therefore the
 
corresponding ''Modules'' of the software must be loaded together with the ''Modules'' of
 
corresponding ''Modules'' of the software must be loaded together with the ''Modules'' of
 
the libraries.
 
the libraries.
  +
<br>
 
 
By default such software ''Modules'' try to load required ''Modules'' and corresponding versions automatically. However, automatic loading might fail if a different version of that required ''Module''
 
By default such software ''Modules'' try to load required ''Modules'' and corresponding versions automatically. However, automatic loading might fail if a different version of that required ''Module''
 
is already loaded (cf. [[#Loading conflicts|Loading conflicts]]).
 
is already loaded (cf. [[#Loading conflicts|Loading conflicts]]).
  +
<br>
 
 
 
== Unloading Modules ==
 
== Unloading Modules ==
 
To unload or to remove a software ''Module'' execute:
 
To unload or to remove a software ''Module'' execute:
Line 226: Line 197:
 
$ module remove category/softwarename/version
 
$ module remove category/softwarename/version
 
</pre>
 
</pre>
 
 
Unloading a ''Module'' that has been loaded by default makes it inactive for the current session only - it will be reloaded the next time you log in.
 
Unloading a ''Module'' that has been loaded by default makes it inactive for the current session only - it will be reloaded the next time you log in.
  +
<br>
 
 
In order to remove all previously loaded software modules from your environment issue the following command:
 
In order to remove all previously loaded software modules from your environment issue the following command:
 
<pre>
 
<pre>
 
$ module purge
 
$ module purge
 
</pre>
 
</pre>
 
 
 
== Display your loaded Modules ==
 
== Display your loaded Modules ==
 
All ''Modules'' that are currently loaded for you can be displayed by the
 
All ''Modules'' that are currently loaded for you can be displayed by the
Line 243: Line 211:
 
Note: you only have to load further ''Modules'', if you want to use additional software
 
Note: you only have to load further ''Modules'', if you want to use additional software
 
packages or to change the version of an already loaded software.
 
packages or to change the version of an already loaded software.
  +
<br>
 
  +
<br>
 
 
= Software job examples =
 
= Software job examples =
 
 
The ''Modules'' installed on bwHPC systems provide job examples to help you get started using the software or submitting jobs with this software. Examples can be found via a convenient
 
The ''Modules'' installed on bwHPC systems provide job examples to help you get started using the software or submitting jobs with this software. Examples can be found via a convenient
 
variable $FOO_EXA_DIR (for a ''Module'' called '''foo'''). It is advisable to copy the whole example folder to your $HOME directory, so you can edit those job examples.
 
variable $FOO_EXA_DIR (for a ''Module'' called '''foo'''). It is advisable to copy the whole example folder to your $HOME directory, so you can edit those job examples.
  +
<br>
 
 
For copying the entire job examples folder of software '''foo''' to your working directory, execute:
 
For copying the entire job examples folder of software '''foo''' to your working directory, execute:
 
<pre>
 
<pre>
Line 255: Line 222:
 
$ cp -R $FOO_EXA_DIR .
 
$ cp -R $FOO_EXA_DIR .
 
</pre>
 
</pre>
  +
<br>
 
 
 
= How do Modules work? =
 
= How do Modules work? =
 
 
The default shell on the bwHPC clusters is bash, so explanations and examples will be shown for bash. In general, programs cannot modify the environment of the shell they are being run from, so how can the module command do exactly that?
 
The default shell on the bwHPC clusters is bash, so explanations and examples will be shown for bash. In general, programs cannot modify the environment of the shell they are being run from, so how can the module command do exactly that?
  +
<br>
 
 
The module command is not a program, but a bash-function.
 
The module command is not a program, but a bash-function.
 
You can view its content using
 
You can view its content using
Line 275: Line 240:
 
}
 
}
 
</pre>
 
</pre>
 
 
In this function, modulecmd is called. Its output to stdout is then executed inside your current shell using the bash-internal ''eval'' command. As a consequence, all output that you see from the module is transmitted via stderr (output handle 2) or in some cases even stdin (output handle 0).
 
In this function, modulecmd is called. Its output to stdout is then executed inside your current shell using the bash-internal ''eval'' command. As a consequence, all output that you see from the module is transmitted via stderr (output handle 2) or in some cases even stdin (output handle 0).
  +
<br>
 
  +
<br>
 
 
 
 
----
 
----
 
[[Category:System software]][[Category:bwUniCluster|Environment Modules]][[Category:ForHLR Phase I|Environment Modules]]
 
[[Category:System software]][[Category:bwUniCluster|Environment Modules]][[Category:ForHLR Phase I|Environment Modules]]
<!--[[Category:bwHPC|User Environment]]
 
[[Category:bwUniCluster|Environment Modules]]-->
 

Revision as of 17:31, 21 December 2015

Description Content
module load category/name | category/name/version (optional)
Availability bwUniCluster | BwForCluster_Chemistry | bwGRiD_tu
Links Environment Modules Project
License GNU General Public License]


1 Introduction

Environment Modules, or short Modules are the means by which most of the installed scientific software is provided on the bwHPC clusters.
The use of different compilers, libraries and software packages requires users to set up a specific session environment suited for the program they want to run. The bwHPC clusters provide users with the possibility to load and unload complete environments for compilers, libraries and software packages by a single command.

2 Description

The Environment Modules package enables dynamic modification of your environment by the use of so-called modulefiles. A modulefile contains information to configure the shell for a program/software . Typically, a modulefile contains instructions that alter or set shell environment variables, such as PATH and MANPATH, to enable access to various installed software.
One of the key features of using the Environment Modules software is to allow multiple versions of the same software to be used in your environment in a controlled manner. For example, two different versions of the Intel C compiler can be installed on the system at the same time - the version used is based upon which Intel C compiler modulefile is loaded.
The software stack of bwHPC clusters provides a number of modulefiles. You can also create your own modulefiles. Modulefiles may be shared by many users on a system, and users may have their own collection of modulefiles to supplement or replace the shared modulefiles.
A modulefile does not provide configuration of your environment until it is explicitly loaded, i.e., the specific modulefile for a software product or application must be loaded in your environment before the configuration information in the modulefile is effective. For instance loading the default Intel C and Fortran compiler you must execute:

$ module load compiler/intel


3 Usage

3.1 Documentation

For help on how to use Modules software, i.e., the command module, execute:

$ module help

or

$ man module

For help on particular version of Module, e.g. Intel compiler version X.Y, execute:

$ module help compiler/intel/X.Y

3.1.1 Online Documentation

[Frequently Asked Questions (FAQ)]

3.2 Display all available Modules

Available Module are modulefiles that can be loaded by the user. A Module must be loaded before it provides changes to your environment, as described in the introduction to this section. You can display all available Modules on the system by executing:

$ module avail

The short form the command is:

$ module av

Available Modules can be also displayed in different modes, such as

  • each Module per one line
$ module -t avail
  • long
$ module -l avail

3.3 bwHPC CLuster Information System (CIS)

A GUI-Version of all available and scheduled modules is available with our
CIS (Cluster Information System).
Software Admins are able to announce new modules, versions and complete new software, too.
Academic users and other interested parties can get a summay of all installed modules and module-help informations. CIS: Cluster Information System

Cis.jpg

3.4 Module categories, versions and defaults

The bwHPC clusters (such as bwUniCluster) traditionally provide a large variety of software and software versions. Therefore Module are divided in category folders containing subfolders of modulefiles again containing modulefile versions, and must be addressed as follows:

category/softwarename/version

For instance the Intel compiler X.Y belongs to the category of compilers, therefore the modulefile X.Y is placed under the category compiler and intel.
In case of multiple software versions, one version will be always defined as the default version. The Module of the default can be addressed by simply omitting the version number:

category/softwarename

3.5 Finding software Modules

Currently all bwHPC software packages are assigned to the following Module categories:

You can selectively list software in one of those categories using, e.g. for the category "compiler"

$ module avail compiler/

Searches are looking for a substring starting at the begin of the name, so this would list all software in categories starting with a "c"

$ module avail c

while this would find nothing

$ module avail hem

3.6 Loading Modules

You can load a Module software in to your environment to enable easier access to software that you want to use by executing:

$ module load category/softwarename/version

or

$ module add category/softwarename/version

Loading a Module in this manner affects ONLY your environment for the current session.

3.6.1 Loading conflicts

By default you can not load different versions of same software Module in same session. Loading for example Intel compiler version X while Intel compiler version Y is loaded results in error message as follows:

ERROR:150: Module 'compiler/intel/X' conflicts with the currently loaded module(s) 'compiler/intel/Y'

The solution is unloading or switching Modules.

3.6.2 Showing the changes introduced by a Module

Loading a Module will change the environment of the current shell session. For instance the $PATH variable will be expanded by the software's binary directory. Other Module variables may even change the behavior of the current shell session or the software program(s) in a more drastic way.
Loaded Modules may also invoke an additional set of environment variables, which e.g. point to directories or destinations of documentation and examples. Their nomenclature is systematic:

Variable Pointing to
$FOO_HOME Root directory of the software package
$FOO_DOC_DIR Documentation
$FOO_EXA_DIR Examples
$FOO_BPR_URL URL of software's Wiki article

with FOO being the placeholder of the software Module name.
All the changes to the current shell session to be invoked by loading the Module can be reviewed using

$ module show category/softwarename/0.0.0

while it does not load the Module.

3.6.3 Modules depending on Modules

Some program Modules depend on libraries to be loaded to the user environment. Therefore the corresponding Modules of the software must be loaded together with the Modules of the libraries.
By default such software Modules try to load required Modules and corresponding versions automatically. However, automatic loading might fail if a different version of that required Module is already loaded (cf. Loading conflicts).

3.7 Unloading Modules

To unload or to remove a software Module execute:

$ module unload category/softwarename/version

or

$ module remove category/softwarename/version

Unloading a Module that has been loaded by default makes it inactive for the current session only - it will be reloaded the next time you log in.
In order to remove all previously loaded software modules from your environment issue the following command:

$ module purge

3.8 Display your loaded Modules

All Modules that are currently loaded for you can be displayed by the command:

$ module list

Note: you only have to load further Modules, if you want to use additional software packages or to change the version of an already loaded software.

4 Software job examples

The Modules installed on bwHPC systems provide job examples to help you get started using the software or submitting jobs with this software. Examples can be found via a convenient variable $FOO_EXA_DIR (for a Module called foo). It is advisable to copy the whole example folder to your $HOME directory, so you can edit those job examples.
For copying the entire job examples folder of software foo to your working directory, execute:

$ module load catogory/softwarename
$ cp -R $FOO_EXA_DIR .


5 How do Modules work?

The default shell on the bwHPC clusters is bash, so explanations and examples will be shown for bash. In general, programs cannot modify the environment of the shell they are being run from, so how can the module command do exactly that?
The module command is not a program, but a bash-function. You can view its content using

$ type module

and you will get a result like this:

$ type module
module is a function
module () 
{ 
    eval `/usr/bin/modulecmd bash $*`
}

In this function, modulecmd is called. Its output to stdout is then executed inside your current shell using the bash-internal eval command. As a consequence, all output that you see from the module is transmitted via stderr (output handle 2) or in some cases even stdin (output handle 0).