bwHPC Wiki - User contributions [en]

BwUniCluster3.0/Hardware and Architecture/Filesystem Details

2026-05-13T08:23:02Z

A Flachmueller: /* Workspaces */

= File System Details =

On bwUniCluster 3.0 the parallel file system Lustre is used for most globally visible user data. Lustre is open source and Lustre solutions and support are available from different vendors. Nowadays, most of the biggest HPC systems are using Lustre. An initial home directory on a Lustre file system is created automatically after account activation, and the environment variable $HOME holds its name. Users can create so-called workspaces on another Lustre file system for non-permanent data with temporary lifetime. There is another workspace file system based on flash storage for special requirements available.

Within a batch job further file systems are available:
* The directory $TMPDIR is only available and visible on the local node. It is located on fast SSD storage devices.
* On request a parallel on-demand file system (BeeOND) is created which uses the SSDs of the nodes which were allocated to the batch job.
* On request the external LSDF Online Storage is mounted on the nodes which were allocated to the batch job. This file system is based on the parallel file system IBM Storage Scale.

Some of the characteristics of the file systems are shown in the following table.

{| class="wikitable"
|-
! scope="column" style="width:9%"| Property
! style="width:13%"| $TMPDIR
! style="width:13%"| BeeOND
! style="width:13%"| $HOME
! style="width:13%"| Workspace
! style="width:13%"| Workspace on flash
|-
! scope="column" style="height=20px; padding:3px"| Visibility
| style="height=20px; text-align:left;padding:3px"| local node
| style="height=20px; text-align:left;padding:3px"| nodes of batch job
| style="height=20px; text-align:left;padding:3px"| global
| style="height=20px; text-align:left;padding:3px"| global
| style="height=20px; text-align:left;padding:3px"| global
|-
! scope="column" style="height=20px; padding:3px"| Lifetime
| style="height=20px; text-align:left;padding:3px"| batch job runtime
| style="height=20px; text-align:left;padding:3px"| batch job runtime
| style="height=20px; text-align:left;padding:3px"| permanent
| style="height=20px; text-align:left;padding:3px"| max. 240 days
| style="height=20px; text-align:left;padding:3px"| max. 240 days
|-
! scope="column" style="height=20px; padding:3px"| Disk space
| style="height=20px; text-align:left;padding:3px"| 960 GB - 6.4 TB details see table 1
| style="height=20px; text-align:left;padding:3px"| n*750 GB
| style="height=20px; text-align:left;padding:3px"| 1.1 PiB
| style="height=20px; text-align:left;padding:3px"| 4.6 PiB
| style="height=20px; text-align:left;padding:3px"| 236 TiB
|-
! scope="column" style="height=20px; padding:3px"| Capacity Quotas
| style="height=20px; text-align:left;padding:3px"| no
| style="height=20px; text-align:left;padding:3px"| no
| style="height=20px; text-align:left;padding:3px"| yes 500 GiB per user, for MA users 250 GiB also per organization
| style="height=20px; text-align:left;padding:3px"| yes 40 TiB per user
| style="height=20px; text-align:left;padding:3px"| yes 1 TiB per user
|-
! scope="column" style="height=20px; padding:3px"| Inode Quotas
| style="height=20px; text-align:left;padding:3px"| no
| style="height=20px; text-align:left;padding:3px"| no
| style="height=20px; text-align:left;padding:3px"| yes 5 million per user for MA users 2.5 million
| style="height=20px; text-align:left;padding:3px"| yes 20 million per user
| style="height=20px; text-align:left;padding:3px"| yes 5 million per user
|-
! scope="column" style="height=20px; padding:3px"| Backup
| style="height=20px; text-align:left;padding:3px"| no
| style="height=20px; text-align:left;padding:3px"| no
| style="height=20px; text-align:left;padding:3px"| yes
| style="height=20px; text-align:left;padding:3px"| no
| style="height=20px; text-align:left;padding:3px"| no
|-
! scope="column" style="height=20px; padding:3px"| Read perf./node
| style="height=20px; text-align:left;padding:3px"| 500 MB/s - 6 GB/s depends on type of local SSD / job queue
| style="height=20px; text-align:left;padding:3px"| 400 MB/s - 500 MB/s depends on type of local SSDs / job queue
| style="height=20px; text-align:left;padding:3px"| 5 GB/s
| style="height=20px; text-align:left;padding:3px"| 5 GB/s
| style="height=20px; text-align:left;padding:3px"| 1 GB/s
|-
! scope="column" style="height=20px; padding:3px"| Write perf./node
| style="height=20px; text-align:left;padding:3px"| 500 MB/s - 4 GB/s depends on type of local SSD / job queue
| style="height=20px; text-align:left;padding:3px"| 250 MB/s - 350 MB/s depends on type of local SSDs / job queue
| style="height=20px; text-align:left;padding:3px"| 5 GB/s
| style="height=20px; text-align:left;padding:3px"| 5 GB/s
| style="height=20px; text-align:left;padding:3px"| 1 GB/s
|-
! scope="column" style="height=20px;padding:3px"| Total read perf.
| style="height=20px; text-align:left;padding:3px"| n*500-6000 MB/s
| style="height=20px; text-align:left;padding:3px"| n*400-500 MB/s
| style="height=20px; text-align:left;padding:3px"| 63 GB/s
| style="height=20px; text-align:left;padding:3px"| 45 GB/s
| style="height=20px; text-align:left;padding:3px"| 45 GB/s
|-
! scope="column" style="height=20px;padding:3px"| Total write perf.
| style="height=20px; text-align:left;padding:3px"| n*500-4000 MB/s
| style="height=20px; text-align:left;padding:3px"| n*250-350 MB/s
| style="height=20px; text-align:left;padding:3px"| 63 GB/s
| style="height=20px; text-align:left;padding:3px"| 40 GB/s
| style="height=20px; text-align:left;padding:3px"| 38 GB/s
|}
---------------------------------------------------------------------------------------------------------
global: all nodes of UniCluster access the same file system;
local: each node has its own file system;
permanent: files are stored permanently;
batch job: files are removed at end of the batch job.
---------------------------------------------------------------------------------------------------------
Table 2: Properties of the file systems

== Selecting the appropriate file system ==

In general, you should separate your data and store it on the appropriate file system.
Permanently needed data like software or important results should be stored below $HOME
but capacity restrictions (quotas) apply. In case you accidentally deleted data on $HOME
there is a chance that we can restore it from backup. Permanent data which is not needed for months
or exceeds the capacity restrictions should be sent to the LSDF Online Storage
or to the archive and deleted from the file systems. Temporary data which is only needed on a single
node and which does not exceed the disk space shown in [[BwUniCluster3.0/Hardware_and_Architecture#Compute_nodes|Table 1]]
should be stored below $TMPDIR. Data which is read many times on a single node, e.g. if you are doing AI training,
should be copied to $TMPDIR and read from there. Temporary data which is used from many nodes
of your batch job and which is only needed during job runtime should be stored on a
parallel on-demand file system. Temporary data which can be recomputed or which is the
result of one job and input for another job should be stored in workspaces. The lifetime
of data in workspaces is limited and depends on the lifetime of the workspace which can be
several months.

For further details please check the chapters below.

== $HOME ==

The home directories of bwUniCluster 3.0 (uc3) users are located in the parallel file system Lustre.
You have access to your home directory from all nodes of uc3. A regular backup of these directories
to a tape library is done automatically. The directory $HOME should be used to hold permanently used data like
source code, configuration files, executable programs etc.

On uc3 there is a default user quota limit of 500 GiB and 5 million inodes (files and directories) per user.
For users of University of Mannheim the limit is 250 GiB and 2.5 million inodes.
You can check your current usage and limits with the command
<pre>
$ lfs quota -uh $(whoami) $HOME
</pre>
Note that the quota limits mentioned above are soft quota limits. The hard limits (shown on the ''limits''
columns) are 10 percent higher. If you are above the soft limit and below the hard limit
during the grace period (7 days) your I/O operations will show a warning message. If the grace period has
passed or if you are above the hard limit your I/O operations will abort.

In addition to the user limit there is a limit of your organization (e.g. university) which depends on the financial share. This limit is enforced with so-called Lustre project quotas. You can show the current usage and limits of your organization with the following command:
<pre>
lfs quota -ph $(grep $(echo $HOME | sed -e "s|/[^/]*/[^/]*$||") /pfs/data6/project_ids.txt | cut -f 1 -d\ ) $HOME
</pre>

== Workspaces ==

On uc3 workspaces can be used to store large non-permanent data sets, e.g. restart files or output
data that has to be post-processed. The file system used for workspaces is also the parallel file system Lustre. This file system is especially designed for parallel access and for a high throughput to large
files. It is able to provide high data transfer rates of up to 40 GB/s write and read performance when data access is parallel.

On uc3 there is a default user quota limit of 40 TiB and 20 million inodes (files and directories) per user.
You can chek your current usage and limits with the command
<pre>
$ lfs quota -uh $(whoami) /pfs/work9
</pre>
Note that the quotas include data and inodes for all of your workspaces and all of your expired workspaces (as long as they are not yet completely removed).

Workspaces have a lifetime and the data on a workspace expires as a whole after a fixed period. The '''maximum lifetime''' of a workspace on uc3 is '''60 days''', but it can be '''renewed''' at the end of that period '''3 times to a total maximum of 240 days''' after workspace generation.

Creating, deleting, finding, extending and sharing workspaces is explained on the [[workspace]] page.

=== Reminder for workspace deletion ===

Normally you will get an email every day starting 7 days before a workspace expires. You can send yourself a calender entry which reminds you when a workspace will be automatically deleted:

$ ws_send_ical <workspace> <email>

=== Restoring expired Workspaces ===

At expiration time your workspace will be moved to a special, hidden directory. On uc3 expired workspaces are currently kept for 30 days. During this time you can still restore your data into a valid workspace. The same is true for released workspaces but they are only kept until the next night. In order to restore an expired workspace, use
<pre>
ws_restore -l
</pre>
to get a list of your expired workspaces, and then restore them into an '''existing, active workspace''' (here with name <code>my_restored</code>):
<pre>
ws_restore <full_name_of_expired_workspace> my_restored
</pre>
NOTE: The expired workspace has to be specified using the full name as listed by <code>ws_restore -l</code>, including username prefix and timestamp suffix (otherwise, it cannot be uniquely identified).
The target workspace, on the other hand, must be given with just its short name as listed by <code>ws_list</code>, without the username prefix.

NOTE: <code>ws_restore</code> can only work on the same filesystem. So you have to ensure that the new workspace allocated with <code>ws_allocate</code> is placed on the same filesystem as the expired workspace. Therefore, you can use <code>-F <filesystem></code> flag if needed.

=== Linking workspaces in Home ===

It might be valuable to have links to personal workspaces within a certain directory, e.g. below the user home directory. The command
<pre>
ws_register <DIR>
</pre>
will create and manage links to all personal workspaces within in the directory <DIR>. Calling this command will do the following:

* The directory <DIR> will be created if necessary
* Links to all personal workspaces will be managed:
** Creates links to all available workspaces if not already present
** Removes links to released or expired workspaces

== Improving Performance on $HOME and workspaces ==

The following recommendations might help to improve throughput and metadata
performance on Lustre filesystems.

=== Improving Throughput Performance ===

Depending on your application some adaptations might be necessary if you want to reach
the full bandwidth of the filesystems.

When you are designing your application you should consider that the performance of
parallel filesystems is generally better if data is transferred in large blocks and stored in
few large files. In more detail, to increase throughput performance of a parallel application
following aspects should be considered:

* collect large chunks of data and write them sequentially at once,

* to exploit complete filesystem bandwidth use several clients,

* avoid competitive file access by different tasks or use blocks with boundaries at stripe size (default is 1MB),

* if files are small enough for the SSDs and are only used from one node store them on $TMPDIR.

Parallel filesystems typically stripe files over storage subsystems, i.e. large files are separated into stripes and distributed to different storage subsystems. In Lustre, the size of these stripes (sometimes also mentioned as chunks) is called stripe size and the number of used storage subsystems is called stripe count.

With previous Lustre versions adapting the Lustre stripe count was the most important optimization. However, for the workspaces of uc3 the Lustre feature Progressive File Layout is used to define file striping parameters. This means that the stripe count is adapted if the file size is growing. In normal cases users no longer need to adapt file striping parameters in case they have very huge files or in order to reach better performance. If you know what you are doing you can still change striping parameters but further explanation is beyond the scope of this documentation.

=== Improving Metadata Performance ===

Metadata performance on parallel file systems is usually not as good as with local
filesystems. In addition, it is usually not scalable, i.e. a limited resource. Therefore,
you should omit metadata operations whenever possible. For example, it is much better
to have few large files than lots of small files. In more detail, to increase metadata
performance of a parallel application following aspects should be considered:

* avoid creating many small files

* avoid competitive directory access, e.g. by creating files in separate subdirectories for each task

* if many small files are only used within a batch job and accessed by one node store them on $TMPDIR

== Workspaces on flash storage ==

There is another workspace file system for special requirements available. The file system is called ''full flash pfs'' and is based on the parallel file system Lustre.

=== Advantages of this file system ===

# From the Ice Lake nodes of bwUniCluster 3.0 (queue ''cpu_il'') the network distance and latency is low compared to the normal workspace file system.
# All storage devices are based on flash (no hard disks) with low access times. Hence performance is better compared to other parallel file systems for read and write access with small blocks and with small files, i.e. IOPS rates are improved.
# The file system is mounted on bwUniCluster 3.0 and HoreKa, i.e. it can be used to share data between these clusters.

=== Access restrictions ===

Only HoreKa users or KIT users of bwUniCluster 3.0 can use this file system.

=== Using the file system ===

As KIT or HoreKa user you can use the file system in the same way as a normal workspace. You just have to specify the name of the flash-based workspace file system using the option ''-F'' to all the commands that manage workspaces. On bwUniCluster 3.0 it is called ''ffuc'', on HoreKa it is ''ffhk''. For example, to create a workspace with name myws and a lifetime of 60 days on bwUniCluster 3.0 execute:
ws_allocate -F ffuc myws 60

If you want to use the full flash pfs on bwUniCluster 3.0 '''and''' HoreKa at the same time, please note that you only have to manage a particular workspace on one of the clusters since the name of the workspace directory is different. However, the path to each workspace is visible and can be used on both clusters.

Other features are similar to normal workspaces. For example, you can restore expired workspaces during 30 days after workspace expiration. There are quota limits with a default limit of 1 TiB capacity and 5 millions inodes per user. You can check your current usage with
lfs quota -uh $(whoami) /pfs/work8

== $TMPDIR ==

The environment variable $TMPDIR contains the name of a directory which is located on the local SSD of each node. This means
that different tasks of a parallel application use different directories when they do not utilize the same node.
Although $TMPDIR points to the same path name for different nodes of a batch job, the physical location and the
content of this directory path on these nodes is different.

This directory should be used for temporary files being accessed from the local node during job runtime. It should
also be used if you read the same data many times from a single node, e.g. if you are doing AI training. In this
case you should copy the data at the beginning of your batch job to $TMPDIR and read the data from there, see usage example below.

The $TMPDIR directory is located on extremely fast local SSD storage devices. This means that performance
on small files is much better than on the parallel file systems. The capacity of the local SSDs for each node type
is different and can be checked in [[BwUniCluster3.0/Hardware_and_Architecture#Compute_nodes|Table 1]].
The capacity of $TMPDIR is at least 1400 GB.

Each time a batch job is started, a subdirectory is created on the SSD of each node and assigned to the job.
$TMPDIR is set to the name of the subdirectory and this name contains the job ID so that it is unique
for each job. At the end of the job the subdirectory is removed.

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#ffa500; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#ffa500; text-align:left"|
'''All data on $TMPDIR will be deleted when your job completes.''' 
Make sure you have copied your results back to a global filesystem, e.g., $HOME or a workspace, within your job.
|}

On login nodes $TMPDIR also points to a fast directory on a local SSD disk but this directory is not unique.
It is recommended to create your own unique subdirectory on these nodes. This directory should be used for the
installation of software packages. This means that the software package to be installed should be unpacked,
compiled and linked in a subdirectory of $TMPDIR. The real installation of the package (e.g. make install)
should be made into the $HOME folder.

{|style="background:#ffdeee; width:100%;"
|style="padding:5px; background:#f2cece; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#f2cece; text-align:left"|
Note that you should '''not''' use /tmp or /scratch! Please use '''$TMPDIR''' instead. 
The reason is that an automatic cleanup on /tmp or /scratch is not possible because another job could be still using data below these directories. Hence the corresponding file systems could fill up and this can cause issues for you and for other users. On the other hand, $TMPDIR is created when the job starts and removed when the job completes, i.e. a cleanup is automatically done.
|}

=== Usage example for $TMPDIR ===

We will provide an example for using $TMPDIR and describe efficient data transfer to and from $TMPDIR.

If you have a data set with many files which is frequently used by batch jobs you should create
a compressed archive on a workspace. This archive can be extracted on $TMPDIR inside your batch jobs.
Such an archive can be read efficiently from a parallel file system since it is a single huge file.
On a login node you can create such an archive with the following steps:
<syntaxhighlight lang="bash">
# Create a workspace to store the archive
[ab1234@uc3n991 ~]$ ws_allocate data-ssd 60
# Create the archive from a local dataset folder (example)
[ab1234@uc3n991 ~]$ tar -cvzf $(ws_find data-ssd)/dataset.tgz dataset/
</syntaxhighlight>

Inside a batch job extract the archive on $TMPDIR, read input data from $TMPDIR, store results on $TMPDIR
and save the results on a workspace:
<syntaxhighlight lang="bash">
#!/bin/bash
# very simple example on how to use local $TMPDIR
#SBATCH -N 1
#SBATCH -t 24:00:00

# Extract compressed input dataset on local SSD
tar -C $TMPDIR/ -xvzf $(ws_find data-ssd)/dataset.tgz

# The application reads data from dataset on $TMPDIR and writes results to $TMPDIR
myapp -input $TMPDIR/dataset/myinput.csv -outputdir $TMPDIR/results

# Before job completes save results on a workspace
rsync -av $TMPDIR/results $(ws_find data-ssd)/results-${SLURM_JOB_ID}/
</syntaxhighlight>

== LSDF Online Storage ==

The LSDF Online Storage allows dedicated users to store scientific measurement data and simulation results. BwUniCluster 3.0 has an extremely fast network connection to the LSDF Online Storage. This file system provides external access via different protocols. It is only available for certain users. For information how request storage projects on the LSDF Online Storage see [[https://www.scc.kit.edu/en/services/lsdf]].

The LSDF Online Storage is mounted on the login nodes. It will also be mounted on the compute nodes of your batch job request it with the constraint flag ''LSDF''. You have one of the following options:

1. Add after the initial lines of your script job.sh the line <code>#SBATCH --constraint=LSDF</code>:
<pre>
#!/bin/bash
#SBATCH --ntasks=1
#SBATCH --time=120
#SBATCH --mem=200
#SBATCH --constraint=LSDF
</pre>
2. Add the constraint on command line:
<pre>
$ sbatch -p <queue> -n 1 -t 2:00:00 --mem 200 job.sh -C LSDF
</pre>

In order to access the LSDF Online Storage the following environment variables are available:
<code>$LSDF</code>, <code>$LSDFPROJECTS</code>, <code>$LSDFHOME</code>

==BeeOND (BeeGFS On-Demand)==

Users have the possibility to request a private BeeOND (on-demand BeeGFS) parallel filesystem for each job. The file system is created during job startup and purged when your job completes.

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#ffa500; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#ffa500; text-align:left"|
'''All data on the private BeeOND filesystem will be deleted when your job completes.''' 
Make sure you have copied your results back to a global filesystem, e.g., $HOME or a workspace, within your job.
|}

BeeOND/BeeGFS can be used like any other parallel file system. All nodes of the batch job have acess to the same data below the same path. Tools like cp or rsync can be used to copy data in and out.

Starting and stopping BeeOND is integrated in the prolog and epilog of the cluster batch system Slurm. It can be used during job runtime if compute nodes are exclusive used, meaning that no other jobs are running on the same node. To achieve this, use the batch option "--exclusive" in your batch script or on command line. This is particularly important in shared partitions.

You can request the creation of a BeeOND file system with the constraint flags ''BEEOND'', ''BEEOND_4MDS'' or ''BEEOND_MAXMDS''.
* BEEOND: one metadata server is started on the first node
* BEEOND_4MDS: 4 metadata servers are started within your job. If you have less than 4 nodes less metadata servers are started.
* BEEOND_MAXMDS: on every node of your job a metadata server for the on_demand file system is started

As starting point we recommend using the constraint ''BEEOND''. You have one of the following options to request the constraint:

1. Add the line <code>#SBATCH --constraint=BEEOND</code> to your job script:
<source lang="bash">
#!/bin/bash
#SBATCH ...
#SBATCH --constraint=BEEOND # or BEEOND_4MDS or BEEOND_MAXMDS
#SBATCH --exclusive
</source>
2. Add the constraint on command line:
<pre>
$ sbatch -p <queue> -N <# of nodes> -t <runtime> --mem <mem> --exclusive -C BEEOND job.sh
</pre>

After your job has started you can find the private on-demand file system in '''/mnt/odfs/${SLURM_JOB_ID}''' directory. The mountpoint comes with five pre-configured directories:
<source lang="bash">
# For small files (stripe count = 1)
/mnt/odfs/${SLURM_JOB_ID}/stripe_1
# Stripe count = 4
/mnt/odfs/${SLURM_JOB_ID}/stripe_default
# or
/mnt/odfs/${SLURM_JOB_ID}/stripe_4
# Stripe count = 8, 16 or 32, use this directories for medium sized and large files or when using MPI-IO
/mnt/odfs/${SLURM_JOB_ID}/stripe_8
/mnt/odfs/${SLURM_JOB_ID}/stripe_16
# or
/mnt/odfs/${SLURM_JOB_ID}/stripe_32
</source>

If you request less nodes than stripe count, the stripe count will be the number of nodes. For example, if you only request 8 nodes the directory stripe_16 has only a stripe count 8.

; '''Attention:''' 
:Use always the directory with the greatest stripe count for large files. E.g. if your largest file is 3.1 Tb, then you have to use a stripe count greater than 4 (4 x 750 GB), otherwise the used disk space is exceeded.

The capacity of the private file system depends on the number of nodes. For each node you get 750 Gbyte.
If you request 100 nodes for your job, the private file system is 100 * 750 Gbyte ~ 75 Tbyte (approx) capacity.

==Backup and Archiving==

There are regular backups of all data of the home directories, whereas ACLs and extended attributes will
not be saved in backups.

Please open a ticket if you have the need to restore backup data.

BwUniCluster3.0/Support

2026-05-12T13:53:52Z

A Flachmueller:

== Registration ==

{| style="vertical-align:top;background:#f5fffa;border:2px solid #000000;"
| The primary support channel for all inquiries is the ticket system at

* '''[https://www.bwhpc.de/supportportal bwSupport Portal]'''
|}

If you are having issues connected to your local home institution, e.g. getting the entitlement for registration on bwUniCluster 3.0, you may also contact your local hotline:

{| style="vertical-align:top;"
! University !! Hotline
|-
| Albert Ludwig University of Freiburg || hpc-support (ät) hpc.uni-freiburg.de
|-
| Eberhard Karls University, Tübingen || hpcmaster (ät) uni-tuebingen.de
|-
| Hochschule Esslingen || cluster-support (ät) hs-esslingen.de
|-
| Karlsruhe Institute of Technology || servicedesk (ät) scc.kit.edu
|-
| Ruprecht-Karls-Universität Heidelberg || hpc-support (ät) urz.uni-heidelberg.de
|-
| Ulm University || helpdesk (ät) uni-ulm.de
|-
| University of Hohenheim || kim-bw-projekt (ät) uni-hohenheim.de
|-
| University of Konstanz || bwhpc (ät) uni-konstanz.de
|-
| University of Mannheim ||hpc-support (ät) mailman.uni-mannheim.de
|-
| University of Stuttgart || bwunicluster (ät) hlrs.de
|}

== Problems with Login/2-Factor-Authentication ==

'''Q: How do I register/deactivate a token?'''

A: Please refer to the [[Registration/2FA|2FA documentation wiki page]].

 

'''Q: I have lost my 2FA device or can no longer generate One-Time Passwords using my software token.'''

A: KIT users can contact the [https://www.scc.kit.edu/en/services/servicedesk.php KIT ServiceDesk], users from all other institutions should open a support ticket via the [https://www.bwhpc.de/supportportal bwSupport Portal].

== Deregistration ==

If you are no longer using the bwUniCluster 3.0 and want to deregister yourself, please follow the [[Registration/Deregistration|Deregistration instructions]]. This will also automatically remove your e-mail address from the user mailing list and you will no longer receive user announcements for this system.

----

BwForCluster User Access Members Uni Konstanz

2026-05-12T13:48:35Z

A Flachmueller:

'''A valid account for the University of Konstanz is required to access to bwForCluster. So you need at least an employee- or student ID (Matrikelnummer).'''

== ==

Your [[Registration/bwForCluster/RV | registration request]] for a new ''rechenvorhaben'' will be delivered to your local support team at the University of Konstanz. They will automatically proof your request and set the bwForCluster-Entitlement or contact you directly if further information is necessary.

In case of joining an existing ''rechenvorhaben'', please contact the [https://www.kim.uni-konstanz.de/services/forschen-und-lehren/high-performance-computing/ local support] to obtain the bwForCluster-Entitlement.

== ==

For more information about registration visit the related web pages and follow the instructions documented on this page.

German version: [https://www.kim.uni-konstanz.de/services/forschen-und-lehren/high-performance-computing/zugang-bwforcluster/ Zugang bwForCluster]

English version: [https://www.kim.uni-konstanz.de/en/services/research-and-teaching/high-performance-computing/access-bwforcluster/ Access bwForCluster]

BwUniCluster3.0/Software Modules

2026-05-11T09:51:05Z

A Flachmueller: /* Build individual EasyBuild packages as a user */

= Introduction =

'''Software (Environment) Modules''', or short '''modules''' are the means by which most of the installed scientific software is provided on ''bwUniCluster 3.0''.

Different compilers, libraries and software packages requires users to set up a specific session environment suited for the program they want to run. ''bwUniCluster 3.0'' provides users with the possibility to load and unload complete environments for compilers, libraries and software packages using a single command.


= Basics =

The Environment ''modules'' package enables dynamic modification of shell environments. Typically, a modulefile contains instructions that alter or set shell environment variables, such as PATH and MANPATH, to enable access to various installed software.

One of the key features of Environment ''Modules'' is the ability to have multiple versions of the same software available in a controlled manner. For example, two different versions of the Intel C compiler can be installed on the system at the same time - the version used is based upon which Intel C compiler modulefile is loaded.

The software stack of bwUniCluster 3.0 provides a number of modules. Users can also provide modules for their own software. ''Modules'' may be shared between users on a system, and users may have their own collection of ''modules'' to supplement or replace the global ''modules''.

To see which modules are currently loaded/enabled one must execute

<pre class="example">$ module list
No modules loaded
</pre>
If the user has just logged in no ''modules'' should be loaded.

If a user has setup his/her environment for an Intel toolchain the output may look like this

<pre class="example">$ module list

Currently Loaded Modules:
1) compiler/intel/2023.1.0 2) numlib/mkl/2022.2.1 3) mpi/impi/2021.11

</pre>

= Usage =

bwUniCluster 3.0 has been set up to use the ''Lmod'' module system. ''Lmod'' is a modern replacement of the older tcl-based environment modules. The complete ''Lmod'' documentation can be found [[<nowiki/>http://lmod.readthedocs.org/en/latest/][here]].


== Getting help ==

The ''Lmod'' module command provides a manual page (man module) and a help command line option. Execute ''''module help'''' or ''''man module'''' for help on how to use ''Lmod'' software.

<pre class="example">$ module help
Usage: module [options] sub-command [args ...]

Options:
-h -? -H --help This help message
...
a lot of help
...
Lmod Web Sites

Documentation: https://lmod.readthedocs.org
GitHub: https://github.com/TACC/Lmod
SourceForge: https://lmod.sf.net
TACC Homepage: https://www.tacc.utexas.edu/research-development/tacc-projects/lmod

To report a bug please read https://lmod.readthedocs.io/en/latest/075_bug_reporting.html

Modules based on Lua: Version 8.7.55 2024-12-13 12:24 -07:00
by Robert McLay mclay@tacc.utexas.edu

</pre>
To get help for a particular ''module'', e.g. the default GNU compiler, run

<pre class="example">$ module help compiler/gnu

------------- Module Specific Help for "compiler/gnu/14.2" -------------------------------
This module provides the GNU compiler collection version 14.2.0 via commands
gcc, g++, gfortran and gccgo. The GNU compiler has been build with the libraries
gmp (6.3.0), mpfr (4.2.1), mpc (1.3.1) and isl (0.27).
This compiler supports offloating OpenACC and OpenMP to NVIDIA CUDA using -foffload.

cpp - GNU pre processor
gcc - GNU C compiler
g++ - GNU C++ compiler
gfortran - GNU Fortran compiler (Fortran 95/2003/2008 plus legacy Fortran 77)
gccgo - GNU Google's Go language

Libraries can be found in
\$GNU_HOME/lib64 = $env(GNU_HOME)/lib64
One may require additionally libraries installed in
\$GNU_HOME/lib = $env(GNU_HOME)/lib

Local documentation:
See commands 'man cpp', 'man gcc', 'man g++' and 'man gfortran'.

Online documentation:
https://gcc.gnu.org/onlinedocs/

For details on library and include dirs please call
module show

The man pages, environment variables and compiler commands
are available after loading '[module-info name]'.

In case of problems, submit a trouble ticket at 'https://bw-support.scc.kit.edu'.

The full version is: compiler/gnu/14.2.0
</pre>

== Display all available Modules ==

bwUniCluster3.0 comes with a variety of modules pre-installed. Some of these modules may not be available for all users of all institutions, due to licensing restrictions. You can display all available ''modules'' by running:

<pre class="example">$ module avail
</pre>
Due to the fact that some modules depend on other modules, some modules may not be shown as available all the time. To get a complete list of all possible modules use the [[#displayallmodules|module spider command]].


== Module categories, versions and defaults ==

The bwHPC clusters traditionally provide a large variety of software and software versions. Therefore ''modules'' are divided into category folders containing subfolders of modulefiles again containing modulefile versions, and must be addressed as follows:

<code>category/softwarename/version</code>

For instance all versions of the Intel compiler belong to the category compiler, thus the corresponding modulefiles are placed under the category ''compiler'' and ''intel''.

In case of multiple software versions, one version will be defined as the '''default''' version. The ''Module'' of the default can be addressed by simply omitting the version number:

<code>category/softwarename</code>

Please be aware, that the '''default''' version may change over time.


== Finding software Modules ==

Currently all bwUniCluster 3.0 software packages are assigned to the following ''Module'' categories:

* bio
* cae
* chem
* compiler
* devel
* lib
* math
* mpi
* numlib
* phys
* system
* toolkit
* vis

You can selectively list software in one of those categories using, e.g. for the category "compiler"

<pre class="example">$ module avail compiler
</pre>
Searches with module avail are substring searches and will return any module with the string in its name.


== Loading Modules ==

To use a software provided through modules in your environment, run:

<pre class="example">$ module load category/softwarename/version
</pre>
or

<pre class="example">$ module add category/softwarename/version
</pre>
Loading a ''Module'' in this manner affects your environment only for the current session or until you [[#unloadingmodules|unload]] the module again


=== module conflicts ===

For some software, it doesn't make sense to provide multiple versions of the same software in the same environment. For this reason, module will automatically unload a conflicting version of the same software, e.g. loading the Intel compiler in version X while Intel compiler in version Y is loaded leads to an automatic unloading of Intel compiler in version Y.


=== Showing the changes introduced by a Module ===

Loading a module will change the environment of the current shell session. For instance the $PATH variable will be expanded by the software's bin directory. Other module variables may even change the behavior of the current shell session or the software program(s) in a more drastic way.

All the changes to the current shell session to be invoked by loading the module can be reviewed using 'module show category/softwarename/version', if version is omitted, information for the default version will be shown, e.g. for the default GNU compiler

<pre class="example">$ module show compiler/gnu
-------------------------------------------------------------------------------------------------------
/opt/bwhpc/common/modulefiles/Core/compiler/gnu/13.3.lua:
-------------------------------------------------------------------------------------------------------
setenv("GNU_VERSION","13.3.0")
setenv("GNU_HOME","/opt/bwhpc/common/compiler/gnu/13.3.0")
setenv("GNU_BIN_DIR","/opt/bwhpc/common/compiler/gnu/13.3.0/bin")
setenv("GNU_LIB_DIR","/opt/bwhpc/common/compiler/gnu/13.3.0/lib64")
setenv("GNU_INC_DIR","/opt/bwhpc/common/compiler/gnu/13.3.0/include")
setenv("GNU_MAN_DIR","/opt/bwhpc/common/compiler/gnu/13.3.0/share/man")
prepend_path("PATH","/opt/bwhpc/common/compiler/gnu/13.3.0/bin")
prepend_path("LD_LIBRARY_PATH","/opt/bwhpc/common/compiler/gnu/13.3.0/lib64")
prepend_path("MANPATH","/opt/bwhpc/common/compiler/gnu/13.3.0/share/man")
conflict("compiler/intel")
conflict("compiler/pgi")
whatis("GNU compiler suite version 13.3.0 (gcc, g++, gfortran, gccgo) including LTO and JIT ")
help([[This module provides the GNU compiler collection version 13.3.0 via commands
gcc, g++, gfortran and gccgo. The GNU compiler has been build with the libraries
gmp (6.3.0), mpfr (4.2.1), mpc (1.3.1) and isl (0.26).
This compiler supports offloating OpenACC and OpenMP to NVIDIA CUDA using -foffload.

cpp - GNU pre processor
gcc - GNU C compiler
g++ - GNU C++ compiler
gfortran - GNU Fortran compiler (Fortran 95/2003/2008 plus legacy Fortran 77)
gccgo - GNU Google's Go language

Libraries can be found in
\$GNU_HOME/lib64 = $env(GNU_HOME)/lib64
One may require additionally libraries installed in
\$GNU_HOME/lib = $env(GNU_HOME)/lib

Local documentation:
See commands 'man cpp', 'man gcc', 'man g++' and 'man gfortran'.

Online documentation:
https://gcc.gnu.org/onlinedocs/

For details on library and include dirs please call
module show [module-info name]

The man pages, environment variables and compiler commands
are available after loading '[module-info name]'.

In case of problems, submit a trouble ticket at 'https://bw-support.scc.kit.edu'.

The full version is: compiler/gnu/13.3.0

]])
prepend_path("MODULEPATH","/software/bwhpc/common/modulefiles/Compiler/gnu/13.3")
family("compiler")
</pre>

=== Dependencies between modules ===

Most modern software depends on shared libraries. Shared libraries must be accesible during the runtime of the software. This is true for basic OS libraries as well as for more specialized libraries like math or communication libraries. For basic OS libraries, one version is usually enough for all software running on a system. For the more specialized libraries, different user may prefer different versions or even implementations of these libraries. Theses situations can be represented using modules. The modules contain the information, what tools are necessary to run a specific software and load all required software automatically.


== Unloading Modules ==

To unload or remove a module run:

<pre class="example">$ module unload category/softwarename/version
</pre>
or

<pre class="example">$ module remove category/softwarename/version
</pre>

=== Unloading all loaded modules ===

<ol>
<li>Purge
Loading different modules may lead to a expansive list of modules, especially when using [[#EasyBuild|EasyBuild]]. To unload all currently loaded modules use the 'module purge' command like this:
<pre class="example">$ module list
Currently Loaded Modules:
1) compiler/intel/19.1 2) mpi/impi/2019 3) numlib/mkl/2019
$ module purge
$ module list
No modules loaded
</pre>
Be aware that 'module purge' is working without any further inquiry.</li></ol>


== Display your loaded Modules ==

All currently loaded modules can be displayed using the 'module list' command. [[#basics|See example above]].


== Display all possible Modules ==

Some of the modules may not be visible with the 'module avail' command to due to unmet dependencies. All available and currently not available modules can be listed using the 'module spider' command:

<pre class="example">$ module spider
-----------------------------------------------------------------------------------------------
The following is a list of the modules and extensions currently available:
-----------------------------------------------------------------------------------------------
compiler/gnu: compiler/gnu/11.4, compiler/gnu/14.2

devel/code-server: devel/code-server/4.96.4

devel/cuda: devel/cuda/12.8

... long list of modules and versions ...

To learn more about a package execute:

$ module spider Foo

where "Foo" is the name of a module.

To find detailed information about a particular package you
must specify the version if there is more than one version:

$ module spider Foo/11.1

-----------------------------------------------------------------------------------------------
</pre>
To see all versions of a software in a category use 'module spider category/name/'

<pre class="example">$ module spider compiler/gnu

-----------------------------------------------------------------------------------------------
compiler/gnu:
-----------------------------------------------------------------------------------------------
Versions:
compiler/gnu/11.4
compiler/gnu/14.2

-----------------------------------------------------------------------------------------------
For detailed information about a specific "compiler/gnu" package (including how to load the modules) use the module's full name.
Note that names that have a trailing (E) are extensions provided by other modules.
For example:

$ module spider compiler/gnu/14.2
-----------------------------------------------------------------------------------------------

</pre>
module spider may be used to show the dependencies of the module. For example, you can see which modules need to be loaded before loading the module mpi/impi/2021.11 is possible

<pre class="example">$ module spider mpi/impi/2021.11

----------------------------------------------------------------------
mpi/impi: mpi/impi/2021.11
----------------------------------------------------------------------

You will need to load all module(s) on any one of the lines below before the "mpi/impi/2021.11" module is available to load.

compiler/intel/2021.4.0
compiler/intel/2021.4.0_llvm
compiler/intel/2022.2.1
compiler/intel/2022.2.1_llvm
compiler/intel/2023.1.0
compiler/intel/2023.1.0_llvm
compiler/intel/2024.0_llvm

Help:
Intel MPI Library version 2021.11

From the product website
(https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/mpi-library.html):

Intel MPI Library
Deliver flexible, efficient, and scalable cluster messaging.

One Library with Multiple Fabric Support
Intel MPI Library is a multifabric message-passing library that implements the
open-source MPICH specification. Use the library to create, maintain, and test
advanced, complex applications that perform better on high-performance
computing (HPC) clusters based on Intel processors.

* Develop applications that can run on multiple cluster interconnects that you
choose at run time.
* Quickly deliver maximum end-user performance without having to change the
software or operating environment.
* Achieve the best latency, bandwidth, and scalability through automatic tuning
for the latest Intel platforms.
* Reduce the time to market by linking to one library and deploying on the
latest optimized fabrics.
</pre>

= EasyBuild modules =

With bwUniCluster 3.0 we start an official beta-test to provide toolchains and scientific software built with [https://docs.easybuild.io/ EasyBuild]. EasyBuild is a well established and consistent way to provide scientific software at many HPC sites all over the world. Since the naming of EasyBuild modules is consistent users can use modules with the same name at different sites. To make this possible, Easybuild software minimizes the resources used by the local operating system and provides most software independently. This makes scientific results more reproducible.


== Using EasyBuild modules ==

Users interested in using EasyBuild modules need to add the EasyBuild module paths to their module environment. This can be done using:

<pre class="example">source /opt/bwhpc/common/etc/easybuild/enable_eb_modules
</pre>
Now users have acces to a complete EasyBuild module tree, providing compiler toolchains, libraries, and scientific software. The same module commands described above (Lmod) can be used for EasyBuild modules. They can be loaded and unloaded in the same way. Please try not to mix EasyBuild modules with the standard modules. There may be unintended consequences. It's always a good idea to [[#purge|purge]] all modules before loading new ones. If you want to use EasyBuild modules in your job scripts, please make sure to enable them in your job script as well.

To hide the EasyBuild modules again, run:

<pre class="example">source /opt/bwhpc/common/etc/easybuild/disable_eb_modules
</pre>
or start again with a clean shell.



== EasyBuild conventions ==

In theory one could build every software against every library and every compiler. This would lead to an unmaintainable amount of combinations. For this reason the EasyBuild team has decided to provide two so called toolchains a year for the most common compiler suits. Each toolchain defines one version of compiler, MPI library, math libraries and other packages as default for all software built aginst this toolchain. On bwUniCluster 3.0 we start with 3 FOSS, 3 Intel and one NVHPC toolchains. For a complete list of EasyBuild toolchains, please see [https://docs.easybuild.io/version-specific/toolchains/ the EasyBuild website]. Currently the 2023a, 2023b, and 2024a toolchains are available, with 2024a being the default one. Some software may not be available for the latest toolchain yet, but may be available for an older toolchain.

EasyBuild currently supports almost 4000 software packages. We can and will only provide a subset, of course. As a start, we provide current versions of popular software packages from bwUniCluster 2.0. Updating and extending EasyBuild packages is a community effort, which should make providing new software easier. We try to provide additional packages if users request the installation. If we can't provide the software you can still use the advantages of EasyBuild by using it to build your private version.


== Build individual EasyBuild packages as a user ==

EasyBuild can also be used to compile your own or niche software, based on the available EasyBuild files by creating an independent EasyBuild software and module tree. The bwUnicluster 3.0 EasyBuild tree can be used as a base to build additional software in your HOME directory.

We assume the following

* the sofware will be installed in your HOME directory in the sub-directory <code>eb</code>
* you have already cloned the git repository from https://github.com/easybuilders/easybuild-easyconfigs to <code>~/eb/easybuild-easyconfigs</code>
* create an EasyBuild configuration file in your home at <code>~/.config/easybuild/config.cfg</code>

<pre class="example">[basic]
robot=<path_to_your_home>/eb/easybuild-easyconfigs/easybuild/easyconfigs
[config]
modules-tool: Lmod
prefix=<path_to_your_home>/eb
buildpath=/scratch/build
[override]
allow-modules-tool-mismatch=True
</pre>
* add system-wide EasyBuild module path

<pre class="example">source /opt/bwhpc/common/etc/easybuild/enable_eb_modules
</pre>

*Check provided EasyBuild installations:

<pre class="example">module spider EasyBuild</pre>

*You have to load an EasyBuild module, e.g.:

<pre class="example">module load EasyBuild/5.3.0</pre>

* add local module path

<pre class="example">mkdir -p ~/eb/modules/all
module use ~/eb/modules/all
</pre>
* find or create EasyBuild file, you can use any of the EasyBuild

<pre class="example">ls ~/eb/easybuild-easyconfigs/easybuild/easyconfigs/*/* | grep eb$
</pre>
* build your software with EasyBuild

<pre class="example">eb cowsay-3.04.eb
</pre>
* use your software

<pre class="example">$ module load cowsay
$ cowsay "Hello EasyBuild"
_________________
< Hello EasyBuild >
-----------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||

</pre>


= How do Modules work? =

The default shell on the bwUniCluster 3.0 is bash, so explanations and examples will be shown for bash. In general, programs cannot modify the environment of the shell they are being run from, so how can the module command do exactly that? The module command is not a program, but a bash-function. You can view its content using:

<pre class="example">$ type module
</pre>
and you will get the following result:

<pre class="example">$ type module
module is a function
module ()
{
eval $($LMOD_CMD bash "$@");
[ $? = 0 ] && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}
</pre>
In this function, lmod is called. Its output to stdout is then executed inside your current shell using the bash-internal ''eval'' command. As a consequence, all output that you see from the module is transmitted via stderr (output handle 2) or in some cases even stdin (output handle 0). [[#introduction|back to top]]

BwUniCluster3.0/Software Modules

2026-05-11T09:29:54Z

A Flachmueller: /* Build individual EasyBuild packages as a user */

= Introduction =

'''Software (Environment) Modules''', or short '''modules''' are the means by which most of the installed scientific software is provided on ''bwUniCluster 3.0''.

Different compilers, libraries and software packages requires users to set up a specific session environment suited for the program they want to run. ''bwUniCluster 3.0'' provides users with the possibility to load and unload complete environments for compilers, libraries and software packages using a single command.


= Basics =

The Environment ''modules'' package enables dynamic modification of shell environments. Typically, a modulefile contains instructions that alter or set shell environment variables, such as PATH and MANPATH, to enable access to various installed software.

One of the key features of Environment ''Modules'' is the ability to have multiple versions of the same software available in a controlled manner. For example, two different versions of the Intel C compiler can be installed on the system at the same time - the version used is based upon which Intel C compiler modulefile is loaded.

The software stack of bwUniCluster 3.0 provides a number of modules. Users can also provide modules for their own software. ''Modules'' may be shared between users on a system, and users may have their own collection of ''modules'' to supplement or replace the global ''modules''.

To see which modules are currently loaded/enabled one must execute

<pre class="example">$ module list
No modules loaded
</pre>
If the user has just logged in no ''modules'' should be loaded.

If a user has setup his/her environment for an Intel toolchain the output may look like this

<pre class="example">$ module list

Currently Loaded Modules:
1) compiler/intel/2023.1.0 2) numlib/mkl/2022.2.1 3) mpi/impi/2021.11

</pre>

= Usage =

bwUniCluster 3.0 has been set up to use the ''Lmod'' module system. ''Lmod'' is a modern replacement of the older tcl-based environment modules. The complete ''Lmod'' documentation can be found [[<nowiki/>http://lmod.readthedocs.org/en/latest/][here]].


== Getting help ==

The ''Lmod'' module command provides a manual page (man module) and a help command line option. Execute ''''module help'''' or ''''man module'''' for help on how to use ''Lmod'' software.

<pre class="example">$ module help
Usage: module [options] sub-command [args ...]

Options:
-h -? -H --help This help message
...
a lot of help
...
Lmod Web Sites

Documentation: https://lmod.readthedocs.org
GitHub: https://github.com/TACC/Lmod
SourceForge: https://lmod.sf.net
TACC Homepage: https://www.tacc.utexas.edu/research-development/tacc-projects/lmod

To report a bug please read https://lmod.readthedocs.io/en/latest/075_bug_reporting.html

Modules based on Lua: Version 8.7.55 2024-12-13 12:24 -07:00
by Robert McLay mclay@tacc.utexas.edu

</pre>
To get help for a particular ''module'', e.g. the default GNU compiler, run

<pre class="example">$ module help compiler/gnu

------------- Module Specific Help for "compiler/gnu/14.2" -------------------------------
This module provides the GNU compiler collection version 14.2.0 via commands
gcc, g++, gfortran and gccgo. The GNU compiler has been build with the libraries
gmp (6.3.0), mpfr (4.2.1), mpc (1.3.1) and isl (0.27).
This compiler supports offloating OpenACC and OpenMP to NVIDIA CUDA using -foffload.

cpp - GNU pre processor
gcc - GNU C compiler
g++ - GNU C++ compiler
gfortran - GNU Fortran compiler (Fortran 95/2003/2008 plus legacy Fortran 77)
gccgo - GNU Google's Go language

Libraries can be found in
\$GNU_HOME/lib64 = $env(GNU_HOME)/lib64
One may require additionally libraries installed in
\$GNU_HOME/lib = $env(GNU_HOME)/lib

Local documentation:
See commands 'man cpp', 'man gcc', 'man g++' and 'man gfortran'.

Online documentation:
https://gcc.gnu.org/onlinedocs/

For details on library and include dirs please call
module show

The man pages, environment variables and compiler commands
are available after loading '[module-info name]'.

In case of problems, submit a trouble ticket at 'https://bw-support.scc.kit.edu'.

The full version is: compiler/gnu/14.2.0
</pre>

== Display all available Modules ==

bwUniCluster3.0 comes with a variety of modules pre-installed. Some of these modules may not be available for all users of all institutions, due to licensing restrictions. You can display all available ''modules'' by running:

<pre class="example">$ module avail
</pre>
Due to the fact that some modules depend on other modules, some modules may not be shown as available all the time. To get a complete list of all possible modules use the [[#displayallmodules|module spider command]].


== Module categories, versions and defaults ==

The bwHPC clusters traditionally provide a large variety of software and software versions. Therefore ''modules'' are divided into category folders containing subfolders of modulefiles again containing modulefile versions, and must be addressed as follows:

<code>category/softwarename/version</code>

For instance all versions of the Intel compiler belong to the category compiler, thus the corresponding modulefiles are placed under the category ''compiler'' and ''intel''.

In case of multiple software versions, one version will be defined as the '''default''' version. The ''Module'' of the default can be addressed by simply omitting the version number:

<code>category/softwarename</code>

Please be aware, that the '''default''' version may change over time.


== Finding software Modules ==

Currently all bwUniCluster 3.0 software packages are assigned to the following ''Module'' categories:

* bio
* cae
* chem
* compiler
* devel
* lib
* math
* mpi
* numlib
* phys
* system
* toolkit
* vis

You can selectively list software in one of those categories using, e.g. for the category "compiler"

<pre class="example">$ module avail compiler
</pre>
Searches with module avail are substring searches and will return any module with the string in its name.


== Loading Modules ==

To use a software provided through modules in your environment, run:

<pre class="example">$ module load category/softwarename/version
</pre>
or

<pre class="example">$ module add category/softwarename/version
</pre>
Loading a ''Module'' in this manner affects your environment only for the current session or until you [[#unloadingmodules|unload]] the module again


=== module conflicts ===

For some software, it doesn't make sense to provide multiple versions of the same software in the same environment. For this reason, module will automatically unload a conflicting version of the same software, e.g. loading the Intel compiler in version X while Intel compiler in version Y is loaded leads to an automatic unloading of Intel compiler in version Y.


=== Showing the changes introduced by a Module ===

Loading a module will change the environment of the current shell session. For instance the $PATH variable will be expanded by the software's bin directory. Other module variables may even change the behavior of the current shell session or the software program(s) in a more drastic way.

All the changes to the current shell session to be invoked by loading the module can be reviewed using 'module show category/softwarename/version', if version is omitted, information for the default version will be shown, e.g. for the default GNU compiler

<pre class="example">$ module show compiler/gnu
-------------------------------------------------------------------------------------------------------
/opt/bwhpc/common/modulefiles/Core/compiler/gnu/13.3.lua:
-------------------------------------------------------------------------------------------------------
setenv("GNU_VERSION","13.3.0")
setenv("GNU_HOME","/opt/bwhpc/common/compiler/gnu/13.3.0")
setenv("GNU_BIN_DIR","/opt/bwhpc/common/compiler/gnu/13.3.0/bin")
setenv("GNU_LIB_DIR","/opt/bwhpc/common/compiler/gnu/13.3.0/lib64")
setenv("GNU_INC_DIR","/opt/bwhpc/common/compiler/gnu/13.3.0/include")
setenv("GNU_MAN_DIR","/opt/bwhpc/common/compiler/gnu/13.3.0/share/man")
prepend_path("PATH","/opt/bwhpc/common/compiler/gnu/13.3.0/bin")
prepend_path("LD_LIBRARY_PATH","/opt/bwhpc/common/compiler/gnu/13.3.0/lib64")
prepend_path("MANPATH","/opt/bwhpc/common/compiler/gnu/13.3.0/share/man")
conflict("compiler/intel")
conflict("compiler/pgi")
whatis("GNU compiler suite version 13.3.0 (gcc, g++, gfortran, gccgo) including LTO and JIT ")
help([[This module provides the GNU compiler collection version 13.3.0 via commands
gcc, g++, gfortran and gccgo. The GNU compiler has been build with the libraries
gmp (6.3.0), mpfr (4.2.1), mpc (1.3.1) and isl (0.26).
This compiler supports offloating OpenACC and OpenMP to NVIDIA CUDA using -foffload.

cpp - GNU pre processor
gcc - GNU C compiler
g++ - GNU C++ compiler
gfortran - GNU Fortran compiler (Fortran 95/2003/2008 plus legacy Fortran 77)
gccgo - GNU Google's Go language

Libraries can be found in
\$GNU_HOME/lib64 = $env(GNU_HOME)/lib64
One may require additionally libraries installed in
\$GNU_HOME/lib = $env(GNU_HOME)/lib

Local documentation:
See commands 'man cpp', 'man gcc', 'man g++' and 'man gfortran'.

Online documentation:
https://gcc.gnu.org/onlinedocs/

For details on library and include dirs please call
module show [module-info name]

The man pages, environment variables and compiler commands
are available after loading '[module-info name]'.

In case of problems, submit a trouble ticket at 'https://bw-support.scc.kit.edu'.

The full version is: compiler/gnu/13.3.0

]])
prepend_path("MODULEPATH","/software/bwhpc/common/modulefiles/Compiler/gnu/13.3")
family("compiler")
</pre>

=== Dependencies between modules ===

Most modern software depends on shared libraries. Shared libraries must be accesible during the runtime of the software. This is true for basic OS libraries as well as for more specialized libraries like math or communication libraries. For basic OS libraries, one version is usually enough for all software running on a system. For the more specialized libraries, different user may prefer different versions or even implementations of these libraries. Theses situations can be represented using modules. The modules contain the information, what tools are necessary to run a specific software and load all required software automatically.


== Unloading Modules ==

To unload or remove a module run:

<pre class="example">$ module unload category/softwarename/version
</pre>
or

<pre class="example">$ module remove category/softwarename/version
</pre>

=== Unloading all loaded modules ===

<ol>
<li>Purge
Loading different modules may lead to a expansive list of modules, especially when using [[#EasyBuild|EasyBuild]]. To unload all currently loaded modules use the 'module purge' command like this:
<pre class="example">$ module list
Currently Loaded Modules:
1) compiler/intel/19.1 2) mpi/impi/2019 3) numlib/mkl/2019
$ module purge
$ module list
No modules loaded
</pre>
Be aware that 'module purge' is working without any further inquiry.</li></ol>


== Display your loaded Modules ==

All currently loaded modules can be displayed using the 'module list' command. [[#basics|See example above]].


== Display all possible Modules ==

Some of the modules may not be visible with the 'module avail' command to due to unmet dependencies. All available and currently not available modules can be listed using the 'module spider' command:

<pre class="example">$ module spider
-----------------------------------------------------------------------------------------------
The following is a list of the modules and extensions currently available:
-----------------------------------------------------------------------------------------------
compiler/gnu: compiler/gnu/11.4, compiler/gnu/14.2

devel/code-server: devel/code-server/4.96.4

devel/cuda: devel/cuda/12.8

... long list of modules and versions ...

To learn more about a package execute:

$ module spider Foo

where "Foo" is the name of a module.

To find detailed information about a particular package you
must specify the version if there is more than one version:

$ module spider Foo/11.1

-----------------------------------------------------------------------------------------------
</pre>
To see all versions of a software in a category use 'module spider category/name/'

<pre class="example">$ module spider compiler/gnu

-----------------------------------------------------------------------------------------------
compiler/gnu:
-----------------------------------------------------------------------------------------------
Versions:
compiler/gnu/11.4
compiler/gnu/14.2

-----------------------------------------------------------------------------------------------
For detailed information about a specific "compiler/gnu" package (including how to load the modules) use the module's full name.
Note that names that have a trailing (E) are extensions provided by other modules.
For example:

$ module spider compiler/gnu/14.2
-----------------------------------------------------------------------------------------------

</pre>
module spider may be used to show the dependencies of the module. For example, you can see which modules need to be loaded before loading the module mpi/impi/2021.11 is possible

<pre class="example">$ module spider mpi/impi/2021.11

----------------------------------------------------------------------
mpi/impi: mpi/impi/2021.11
----------------------------------------------------------------------

You will need to load all module(s) on any one of the lines below before the "mpi/impi/2021.11" module is available to load.

compiler/intel/2021.4.0
compiler/intel/2021.4.0_llvm
compiler/intel/2022.2.1
compiler/intel/2022.2.1_llvm
compiler/intel/2023.1.0
compiler/intel/2023.1.0_llvm
compiler/intel/2024.0_llvm

Help:
Intel MPI Library version 2021.11

From the product website
(https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/mpi-library.html):

Intel MPI Library
Deliver flexible, efficient, and scalable cluster messaging.

One Library with Multiple Fabric Support
Intel MPI Library is a multifabric message-passing library that implements the
open-source MPICH specification. Use the library to create, maintain, and test
advanced, complex applications that perform better on high-performance
computing (HPC) clusters based on Intel processors.

* Develop applications that can run on multiple cluster interconnects that you
choose at run time.
* Quickly deliver maximum end-user performance without having to change the
software or operating environment.
* Achieve the best latency, bandwidth, and scalability through automatic tuning
for the latest Intel platforms.
* Reduce the time to market by linking to one library and deploying on the
latest optimized fabrics.
</pre>

= EasyBuild modules =

With bwUniCluster 3.0 we start an official beta-test to provide toolchains and scientific software built with [https://docs.easybuild.io/ EasyBuild]. EasyBuild is a well established and consistent way to provide scientific software at many HPC sites all over the world. Since the naming of EasyBuild modules is consistent users can use modules with the same name at different sites. To make this possible, Easybuild software minimizes the resources used by the local operating system and provides most software independently. This makes scientific results more reproducible.


== Using EasyBuild modules ==

Users interested in using EasyBuild modules need to add the EasyBuild module paths to their module environment. This can be done using:

<pre class="example">source /opt/bwhpc/common/etc/easybuild/enable_eb_modules
</pre>
Now users have acces to a complete EasyBuild module tree, providing compiler toolchains, libraries, and scientific software. The same module commands described above (Lmod) can be used for EasyBuild modules. They can be loaded and unloaded in the same way. Please try not to mix EasyBuild modules with the standard modules. There may be unintended consequences. It's always a good idea to [[#purge|purge]] all modules before loading new ones. If you want to use EasyBuild modules in your job scripts, please make sure to enable them in your job script as well.

To hide the EasyBuild modules again, run:

<pre class="example">source /opt/bwhpc/common/etc/easybuild/disable_eb_modules
</pre>
or start again with a clean shell.



== EasyBuild conventions ==

In theory one could build every software against every library and every compiler. This would lead to an unmaintainable amount of combinations. For this reason the EasyBuild team has decided to provide two so called toolchains a year for the most common compiler suits. Each toolchain defines one version of compiler, MPI library, math libraries and other packages as default for all software built aginst this toolchain. On bwUniCluster 3.0 we start with 3 FOSS, 3 Intel and one NVHPC toolchains. For a complete list of EasyBuild toolchains, please see [https://docs.easybuild.io/version-specific/toolchains/ the EasyBuild website]. Currently the 2023a, 2023b, and 2024a toolchains are available, with 2024a being the default one. Some software may not be available for the latest toolchain yet, but may be available for an older toolchain.

EasyBuild currently supports almost 4000 software packages. We can and will only provide a subset, of course. As a start, we provide current versions of popular software packages from bwUniCluster 2.0. Updating and extending EasyBuild packages is a community effort, which should make providing new software easier. We try to provide additional packages if users request the installation. If we can't provide the software you can still use the advantages of EasyBuild by using it to build your private version.


== Build individual EasyBuild packages as a user ==

EasyBuild can also be used to compile your own or niche software, based on the available EasyBuild files by creating an independent EasyBuild software and module tree. The bwUnicluster 3.0 EasyBuild tree can be used as a base to build additional software in your HOME directory.

Check provided EasyBuild installations:

<pre class="example">module spider EasyBuild</pre>

You have to load an EasyBuild module, e.g.:

<pre class="example">module load EasyBuild/5.3.0</pre>

We assume the following

* the sofware will be installed in your HOME directory in the sub-directory <code>eb</code>
* you have already cloned the git repository from https://github.com/easybuilders/easybuild-easyconfigs to <code>~/eb/easybuild-easyconfigs</code>
* create an EasyBuild configuration file in your home at <code>~/.config/easybuild/config.cfg</code>

<pre class="example">[basic]
robot=<path_to_your_home>/eb/easybuild-easyconfigs/easybuild/easyconfigs
[config]
modules-tool: Lmod
prefix=<path_to_your_home>/eb
buildpath=/scratch/build
[override]
allow-modules-tool-mismatch=True
</pre>
* add system-wide EasyBuild module path

<pre class="example">source /opt/bwhpc/common/etc/easybuild/enable_eb_modules
</pre>
* add local module path

<pre class="example">mkdir -p ~/eb/modules/all
module use ~/eb/modules/all
</pre>
* find or create EasyBuild file, you can use any of the EasyBuild

<pre class="example">ls ~/eb/easybuild-easyconfigs/easybuild/easyconfigs/*/* | grep eb$
</pre>
* build your software with EasyBuild

<pre class="example">eb cowsay-3.04.eb
</pre>
* use your software

<pre class="example">$ module load cowsay
$ cowsay "Hello EasyBuild"
_________________
< Hello EasyBuild >
-----------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||

</pre>


= How do Modules work? =

The default shell on the bwUniCluster 3.0 is bash, so explanations and examples will be shown for bash. In general, programs cannot modify the environment of the shell they are being run from, so how can the module command do exactly that? The module command is not a program, but a bash-function. You can view its content using:

<pre class="example">$ type module
</pre>
and you will get the following result:

<pre class="example">$ type module
module is a function
module ()
{
eval $($LMOD_CMD bash "$@");
[ $? = 0 ] && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}
</pre>
In this function, lmod is called. Its output to stdout is then executed inside your current shell using the bash-internal ''eval'' command. As a consequence, all output that you see from the module is transmitted via stderr (output handle 2) or in some cases even stdin (output handle 0). [[#introduction|back to top]]

BwUniCluster3.0/Software Modules

2026-05-11T09:29:30Z

A Flachmueller: /* Build individual EasyBuild packages as a user */

= Introduction =

'''Software (Environment) Modules''', or short '''modules''' are the means by which most of the installed scientific software is provided on ''bwUniCluster 3.0''.

Different compilers, libraries and software packages requires users to set up a specific session environment suited for the program they want to run. ''bwUniCluster 3.0'' provides users with the possibility to load and unload complete environments for compilers, libraries and software packages using a single command.


= Basics =

The Environment ''modules'' package enables dynamic modification of shell environments. Typically, a modulefile contains instructions that alter or set shell environment variables, such as PATH and MANPATH, to enable access to various installed software.

One of the key features of Environment ''Modules'' is the ability to have multiple versions of the same software available in a controlled manner. For example, two different versions of the Intel C compiler can be installed on the system at the same time - the version used is based upon which Intel C compiler modulefile is loaded.

The software stack of bwUniCluster 3.0 provides a number of modules. Users can also provide modules for their own software. ''Modules'' may be shared between users on a system, and users may have their own collection of ''modules'' to supplement or replace the global ''modules''.

To see which modules are currently loaded/enabled one must execute

<pre class="example">$ module list
No modules loaded
</pre>
If the user has just logged in no ''modules'' should be loaded.

If a user has setup his/her environment for an Intel toolchain the output may look like this

<pre class="example">$ module list

Currently Loaded Modules:
1) compiler/intel/2023.1.0 2) numlib/mkl/2022.2.1 3) mpi/impi/2021.11

</pre>

= Usage =

bwUniCluster 3.0 has been set up to use the ''Lmod'' module system. ''Lmod'' is a modern replacement of the older tcl-based environment modules. The complete ''Lmod'' documentation can be found [[<nowiki/>http://lmod.readthedocs.org/en/latest/][here]].


== Getting help ==

The ''Lmod'' module command provides a manual page (man module) and a help command line option. Execute ''''module help'''' or ''''man module'''' for help on how to use ''Lmod'' software.

<pre class="example">$ module help
Usage: module [options] sub-command [args ...]

Options:
-h -? -H --help This help message
...
a lot of help
...
Lmod Web Sites

Documentation: https://lmod.readthedocs.org
GitHub: https://github.com/TACC/Lmod
SourceForge: https://lmod.sf.net
TACC Homepage: https://www.tacc.utexas.edu/research-development/tacc-projects/lmod

To report a bug please read https://lmod.readthedocs.io/en/latest/075_bug_reporting.html

Modules based on Lua: Version 8.7.55 2024-12-13 12:24 -07:00
by Robert McLay mclay@tacc.utexas.edu

</pre>
To get help for a particular ''module'', e.g. the default GNU compiler, run

<pre class="example">$ module help compiler/gnu

------------- Module Specific Help for "compiler/gnu/14.2" -------------------------------
This module provides the GNU compiler collection version 14.2.0 via commands
gcc, g++, gfortran and gccgo. The GNU compiler has been build with the libraries
gmp (6.3.0), mpfr (4.2.1), mpc (1.3.1) and isl (0.27).
This compiler supports offloating OpenACC and OpenMP to NVIDIA CUDA using -foffload.

cpp - GNU pre processor
gcc - GNU C compiler
g++ - GNU C++ compiler
gfortran - GNU Fortran compiler (Fortran 95/2003/2008 plus legacy Fortran 77)
gccgo - GNU Google's Go language

Libraries can be found in
\$GNU_HOME/lib64 = $env(GNU_HOME)/lib64
One may require additionally libraries installed in
\$GNU_HOME/lib = $env(GNU_HOME)/lib

Local documentation:
See commands 'man cpp', 'man gcc', 'man g++' and 'man gfortran'.

Online documentation:
https://gcc.gnu.org/onlinedocs/

For details on library and include dirs please call
module show

The man pages, environment variables and compiler commands
are available after loading '[module-info name]'.

In case of problems, submit a trouble ticket at 'https://bw-support.scc.kit.edu'.

The full version is: compiler/gnu/14.2.0
</pre>

== Display all available Modules ==

bwUniCluster3.0 comes with a variety of modules pre-installed. Some of these modules may not be available for all users of all institutions, due to licensing restrictions. You can display all available ''modules'' by running:

<pre class="example">$ module avail
</pre>
Due to the fact that some modules depend on other modules, some modules may not be shown as available all the time. To get a complete list of all possible modules use the [[#displayallmodules|module spider command]].


== Module categories, versions and defaults ==

The bwHPC clusters traditionally provide a large variety of software and software versions. Therefore ''modules'' are divided into category folders containing subfolders of modulefiles again containing modulefile versions, and must be addressed as follows:

<code>category/softwarename/version</code>

For instance all versions of the Intel compiler belong to the category compiler, thus the corresponding modulefiles are placed under the category ''compiler'' and ''intel''.

In case of multiple software versions, one version will be defined as the '''default''' version. The ''Module'' of the default can be addressed by simply omitting the version number:

<code>category/softwarename</code>

Please be aware, that the '''default''' version may change over time.


== Finding software Modules ==

Currently all bwUniCluster 3.0 software packages are assigned to the following ''Module'' categories:

* bio
* cae
* chem
* compiler
* devel
* lib
* math
* mpi
* numlib
* phys
* system
* toolkit
* vis

You can selectively list software in one of those categories using, e.g. for the category "compiler"

<pre class="example">$ module avail compiler
</pre>
Searches with module avail are substring searches and will return any module with the string in its name.


== Loading Modules ==

To use a software provided through modules in your environment, run:

<pre class="example">$ module load category/softwarename/version
</pre>
or

<pre class="example">$ module add category/softwarename/version
</pre>
Loading a ''Module'' in this manner affects your environment only for the current session or until you [[#unloadingmodules|unload]] the module again


=== module conflicts ===

For some software, it doesn't make sense to provide multiple versions of the same software in the same environment. For this reason, module will automatically unload a conflicting version of the same software, e.g. loading the Intel compiler in version X while Intel compiler in version Y is loaded leads to an automatic unloading of Intel compiler in version Y.


=== Showing the changes introduced by a Module ===

Loading a module will change the environment of the current shell session. For instance the $PATH variable will be expanded by the software's bin directory. Other module variables may even change the behavior of the current shell session or the software program(s) in a more drastic way.

All the changes to the current shell session to be invoked by loading the module can be reviewed using 'module show category/softwarename/version', if version is omitted, information for the default version will be shown, e.g. for the default GNU compiler

<pre class="example">$ module show compiler/gnu
-------------------------------------------------------------------------------------------------------
/opt/bwhpc/common/modulefiles/Core/compiler/gnu/13.3.lua:
-------------------------------------------------------------------------------------------------------
setenv("GNU_VERSION","13.3.0")
setenv("GNU_HOME","/opt/bwhpc/common/compiler/gnu/13.3.0")
setenv("GNU_BIN_DIR","/opt/bwhpc/common/compiler/gnu/13.3.0/bin")
setenv("GNU_LIB_DIR","/opt/bwhpc/common/compiler/gnu/13.3.0/lib64")
setenv("GNU_INC_DIR","/opt/bwhpc/common/compiler/gnu/13.3.0/include")
setenv("GNU_MAN_DIR","/opt/bwhpc/common/compiler/gnu/13.3.0/share/man")
prepend_path("PATH","/opt/bwhpc/common/compiler/gnu/13.3.0/bin")
prepend_path("LD_LIBRARY_PATH","/opt/bwhpc/common/compiler/gnu/13.3.0/lib64")
prepend_path("MANPATH","/opt/bwhpc/common/compiler/gnu/13.3.0/share/man")
conflict("compiler/intel")
conflict("compiler/pgi")
whatis("GNU compiler suite version 13.3.0 (gcc, g++, gfortran, gccgo) including LTO and JIT ")
help([[This module provides the GNU compiler collection version 13.3.0 via commands
gcc, g++, gfortran and gccgo. The GNU compiler has been build with the libraries
gmp (6.3.0), mpfr (4.2.1), mpc (1.3.1) and isl (0.26).
This compiler supports offloating OpenACC and OpenMP to NVIDIA CUDA using -foffload.

cpp - GNU pre processor
gcc - GNU C compiler
g++ - GNU C++ compiler
gfortran - GNU Fortran compiler (Fortran 95/2003/2008 plus legacy Fortran 77)
gccgo - GNU Google's Go language

Libraries can be found in
\$GNU_HOME/lib64 = $env(GNU_HOME)/lib64
One may require additionally libraries installed in
\$GNU_HOME/lib = $env(GNU_HOME)/lib

Local documentation:
See commands 'man cpp', 'man gcc', 'man g++' and 'man gfortran'.

Online documentation:
https://gcc.gnu.org/onlinedocs/

For details on library and include dirs please call
module show [module-info name]

The man pages, environment variables and compiler commands
are available after loading '[module-info name]'.

In case of problems, submit a trouble ticket at 'https://bw-support.scc.kit.edu'.

The full version is: compiler/gnu/13.3.0

]])
prepend_path("MODULEPATH","/software/bwhpc/common/modulefiles/Compiler/gnu/13.3")
family("compiler")
</pre>

=== Dependencies between modules ===

Most modern software depends on shared libraries. Shared libraries must be accesible during the runtime of the software. This is true for basic OS libraries as well as for more specialized libraries like math or communication libraries. For basic OS libraries, one version is usually enough for all software running on a system. For the more specialized libraries, different user may prefer different versions or even implementations of these libraries. Theses situations can be represented using modules. The modules contain the information, what tools are necessary to run a specific software and load all required software automatically.


== Unloading Modules ==

To unload or remove a module run:

<pre class="example">$ module unload category/softwarename/version
</pre>
or

<pre class="example">$ module remove category/softwarename/version
</pre>

=== Unloading all loaded modules ===

<ol>
<li>Purge
Loading different modules may lead to a expansive list of modules, especially when using [[#EasyBuild|EasyBuild]]. To unload all currently loaded modules use the 'module purge' command like this:
<pre class="example">$ module list
Currently Loaded Modules:
1) compiler/intel/19.1 2) mpi/impi/2019 3) numlib/mkl/2019
$ module purge
$ module list
No modules loaded
</pre>
Be aware that 'module purge' is working without any further inquiry.</li></ol>


== Display your loaded Modules ==

All currently loaded modules can be displayed using the 'module list' command. [[#basics|See example above]].


== Display all possible Modules ==

Some of the modules may not be visible with the 'module avail' command to due to unmet dependencies. All available and currently not available modules can be listed using the 'module spider' command:

<pre class="example">$ module spider
-----------------------------------------------------------------------------------------------
The following is a list of the modules and extensions currently available:
-----------------------------------------------------------------------------------------------
compiler/gnu: compiler/gnu/11.4, compiler/gnu/14.2

devel/code-server: devel/code-server/4.96.4

devel/cuda: devel/cuda/12.8

... long list of modules and versions ...

To learn more about a package execute:

$ module spider Foo

where "Foo" is the name of a module.

To find detailed information about a particular package you
must specify the version if there is more than one version:

$ module spider Foo/11.1

-----------------------------------------------------------------------------------------------
</pre>
To see all versions of a software in a category use 'module spider category/name/'

<pre class="example">$ module spider compiler/gnu

-----------------------------------------------------------------------------------------------
compiler/gnu:
-----------------------------------------------------------------------------------------------
Versions:
compiler/gnu/11.4
compiler/gnu/14.2

-----------------------------------------------------------------------------------------------
For detailed information about a specific "compiler/gnu" package (including how to load the modules) use the module's full name.
Note that names that have a trailing (E) are extensions provided by other modules.
For example:

$ module spider compiler/gnu/14.2
-----------------------------------------------------------------------------------------------

</pre>
module spider may be used to show the dependencies of the module. For example, you can see which modules need to be loaded before loading the module mpi/impi/2021.11 is possible

<pre class="example">$ module spider mpi/impi/2021.11

----------------------------------------------------------------------
mpi/impi: mpi/impi/2021.11
----------------------------------------------------------------------

You will need to load all module(s) on any one of the lines below before the "mpi/impi/2021.11" module is available to load.

compiler/intel/2021.4.0
compiler/intel/2021.4.0_llvm
compiler/intel/2022.2.1
compiler/intel/2022.2.1_llvm
compiler/intel/2023.1.0
compiler/intel/2023.1.0_llvm
compiler/intel/2024.0_llvm

Help:
Intel MPI Library version 2021.11

From the product website
(https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/mpi-library.html):

Intel MPI Library
Deliver flexible, efficient, and scalable cluster messaging.

One Library with Multiple Fabric Support
Intel MPI Library is a multifabric message-passing library that implements the
open-source MPICH specification. Use the library to create, maintain, and test
advanced, complex applications that perform better on high-performance
computing (HPC) clusters based on Intel processors.

* Develop applications that can run on multiple cluster interconnects that you
choose at run time.
* Quickly deliver maximum end-user performance without having to change the
software or operating environment.
* Achieve the best latency, bandwidth, and scalability through automatic tuning
for the latest Intel platforms.
* Reduce the time to market by linking to one library and deploying on the
latest optimized fabrics.
</pre>

= EasyBuild modules =

With bwUniCluster 3.0 we start an official beta-test to provide toolchains and scientific software built with [https://docs.easybuild.io/ EasyBuild]. EasyBuild is a well established and consistent way to provide scientific software at many HPC sites all over the world. Since the naming of EasyBuild modules is consistent users can use modules with the same name at different sites. To make this possible, Easybuild software minimizes the resources used by the local operating system and provides most software independently. This makes scientific results more reproducible.


== Using EasyBuild modules ==

Users interested in using EasyBuild modules need to add the EasyBuild module paths to their module environment. This can be done using:

<pre class="example">source /opt/bwhpc/common/etc/easybuild/enable_eb_modules
</pre>
Now users have acces to a complete EasyBuild module tree, providing compiler toolchains, libraries, and scientific software. The same module commands described above (Lmod) can be used for EasyBuild modules. They can be loaded and unloaded in the same way. Please try not to mix EasyBuild modules with the standard modules. There may be unintended consequences. It's always a good idea to [[#purge|purge]] all modules before loading new ones. If you want to use EasyBuild modules in your job scripts, please make sure to enable them in your job script as well.

To hide the EasyBuild modules again, run:

<pre class="example">source /opt/bwhpc/common/etc/easybuild/disable_eb_modules
</pre>
or start again with a clean shell.



== EasyBuild conventions ==

In theory one could build every software against every library and every compiler. This would lead to an unmaintainable amount of combinations. For this reason the EasyBuild team has decided to provide two so called toolchains a year for the most common compiler suits. Each toolchain defines one version of compiler, MPI library, math libraries and other packages as default for all software built aginst this toolchain. On bwUniCluster 3.0 we start with 3 FOSS, 3 Intel and one NVHPC toolchains. For a complete list of EasyBuild toolchains, please see [https://docs.easybuild.io/version-specific/toolchains/ the EasyBuild website]. Currently the 2023a, 2023b, and 2024a toolchains are available, with 2024a being the default one. Some software may not be available for the latest toolchain yet, but may be available for an older toolchain.

EasyBuild currently supports almost 4000 software packages. We can and will only provide a subset, of course. As a start, we provide current versions of popular software packages from bwUniCluster 2.0. Updating and extending EasyBuild packages is a community effort, which should make providing new software easier. We try to provide additional packages if users request the installation. If we can't provide the software you can still use the advantages of EasyBuild by using it to build your private version.


== Build individual EasyBuild packages as a user ==

EasyBuild can also be used to compile your own or niche software, based on the available EasyBuild files by creating an independent EasyBuild software and module tree. The bwUnicluster 3.0 EasyBuild tree can be used as a base to build additional software in your HOME directory.

Check provided EasyBuild installations:

<pre class="example">module spider EasyBuild</pre>

You have to load the EasyBuild module:

<pre class="example">module load EasyBuild/5.3.0</pre>

We assume the following

* the sofware will be installed in your HOME directory in the sub-directory <code>eb</code>
* you have already cloned the git repository from https://github.com/easybuilders/easybuild-easyconfigs to <code>~/eb/easybuild-easyconfigs</code>
* create an EasyBuild configuration file in your home at <code>~/.config/easybuild/config.cfg</code>

<pre class="example">[basic]
robot=<path_to_your_home>/eb/easybuild-easyconfigs/easybuild/easyconfigs
[config]
modules-tool: Lmod
prefix=<path_to_your_home>/eb
buildpath=/scratch/build
[override]
allow-modules-tool-mismatch=True
</pre>
* add system-wide EasyBuild module path

<pre class="example">source /opt/bwhpc/common/etc/easybuild/enable_eb_modules
</pre>
* add local module path

<pre class="example">mkdir -p ~/eb/modules/all
module use ~/eb/modules/all
</pre>
* find or create EasyBuild file, you can use any of the EasyBuild

<pre class="example">ls ~/eb/easybuild-easyconfigs/easybuild/easyconfigs/*/* | grep eb$
</pre>
* build your software with EasyBuild

<pre class="example">eb cowsay-3.04.eb
</pre>
* use your software

<pre class="example">$ module load cowsay
$ cowsay "Hello EasyBuild"
_________________
< Hello EasyBuild >
-----------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||

</pre>


= How do Modules work? =

The default shell on the bwUniCluster 3.0 is bash, so explanations and examples will be shown for bash. In general, programs cannot modify the environment of the shell they are being run from, so how can the module command do exactly that? The module command is not a program, but a bash-function. You can view its content using:

<pre class="example">$ type module
</pre>
and you will get the following result:

<pre class="example">$ type module
module is a function
module ()
{
eval $($LMOD_CMD bash "$@");
[ $? = 0 ] && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}
</pre>
In this function, lmod is called. Its output to stdout is then executed inside your current shell using the bash-internal ''eval'' command. As a consequence, all output that you see from the module is transmitted via stderr (output handle 2) or in some cases even stdin (output handle 0). [[#introduction|back to top]]

Registration/bwForCluster/Entitlement

2026-04-28T15:18:31Z

A Flachmueller: /* Request an Entitlement */

<div style="border: 3px solid #e2e3e5; padding: 15px; background-color: #ffffff; margin: 10px 0;">
The bwForCluster Entitlement is issued by the university you are a member of. It tells the cluster-operator that you are currently a member of this university. It also means that the university checked that your actions comply with the German Foreign Trade Act (Außenwirtschaftsgesetz - AWG) and German Foreign Trade Regulations (Außenwirtschaftsverordnung - AWV) (see [https://www.bwidm.de/attribute.php#Berechtigung eduPersonEntitlement]).
</div>

= Step A: bwForCluster Entitlement =

<div style="border: 3px solid #0066cc; padding: 15px; background-color: #e7f3ff; margin: 10px 0;">
To register for a bwForCluster you need the '''bwForCluster Entitlement''' issued by your university.
</div>

The full name of the entitlement is <code><nowiki>http://bwidm.de/entitlement/bwForCluster</nowiki></code>.

Note that this is the name of an identity management attribute and not a link.

More informations on Entitlements and other attributes in general: [https://www.bwidm.de/dienste.php Entitlement (Dienste)]

== Check your Entitlements ==

First check if you already have the Entitlement, some universities automatically assign the entitlement to some accounts:

→ '''[https://login.bwidm.de/user/index.xhtml?show=entitlement&highlight=bwForCluster Registration Server bwForCluster Entitlement Check]'''



== Request an Entitlement ==

If you need the entitlement, please follow the link for your institution or contact your local service desk if no information is provided:

* [[Registration/bwIDM-Entitlements-Uni-Freiburg|Universität Freiburg]]
* [https://heiservices.uni-heidelberg.de/entitlement Universität Heidelberg] (access only within Uni Heidelberg network)
* [https://kim.uni-hohenheim.de/bwhpc-account/ Universität Hohenheim]
* [https://www.scc.kit.edu/downloads/ISM/SD-HPC-Formulare/Accessform_bwForCluster_v1_DE_EN_2026.pdf Karlsruhe Institute of Technology (KIT)]
* [https://www.kim.uni-konstanz.de/services/forschen-und-lehren/high-performance-computing/zugang-bwforcluster/ Universität Konstanz]
* [[BwForCluster User Access Members Uni Mannheim|Universität Mannheim]]
* [https://www.hlrs.de/apply-for-computing-time/bwforcluster Universität Stuttgart]
* [[BwForCluster User Access Members Uni Tübingen|Universität Tübingen]]
* [[BwForCluster User Access Members Uni Ulm|Universität Ulm]]
* [[Registration/HAW|HAW BW e.V.]]: Please contact your local service desk

<div style="border: 3px solid #ffc107; padding: 15px; background-color: #fff3cd; margin: 10px 0;">
'''Entitlement Synchronization:'''

After your university assigns the entitlement, it takes some time for it to synchronize across the system.
* '''If the entitlement does not appear within 24 hours,''' contact your local service desk
</div>

----

[[Registration/bwForCluster/RV|Go to step B]]

Energy Efficient Cluster Usage

2026-04-22T10:35:53Z

A Flachmueller: /* What do I want to do and why do I need an HPC Cluster for it? */

= Introduction =

Energy consumption of data centers has been increasing continuously throughout the last decade. In 2020, the energy consumption of all data centers in Germany amounted to around [https://www.bundestag.de/resource/blob/863850/423c11968fcb5c9995e9ef9090edf9e6/WD-8-070-21-pdf-data.pdf 3 percent] of the total electricity produced. Accompanying this large energy consumption are large-scale emissions of CO2 to the atmosphere and thus significant contributions to climate change.
To illustrate this, an average compute job running on a single node for one day may easily consume 10 kWh or even more. That translates roughly to brewing 700 cups of coffee.
Assuming that a typical bwHPC cluster has a few hundred compute nodes, this amounts to the energy consumption of a village for each cluster.

Although a large amount of this energy consumption is an intrinsic requirement of running large HPC clusters (even when it's processors are idle, a cluster uses a lot of energy), efficient use of the available resources is important. Using as many resources as possible does not make a power user. Using them wisely does.
In the following, a basic introduction to some of the most important aspects of energy-efficient HPC usage from a user perspective is given.

We can generally distinguish three tasks when optimizing for running HPC jobs efficiently.

→ What do I want to do and why do I need an HPC Cluster for it?

→ How many and which kind of hardware resources do I require for it?

→ How do I optimize my code to use these resources most efficiently?

= What do I want to do and why do I need an HPC Cluster for it? =

The bwHPC clusters are used to almost full capacity, and running a job on an HPC node consumes a lot of energy, as shown above.
Therefore, users are requested to run only necessary jobs.

Please consider testing new setups and their output for validity prior to submitting jobs that require lots of resources. This also includes projects where a lot of (smaller) similar jobs are submitted.

Make sure to double-check your jobs prior to the submission, as having to discard the output data of an HPC project due to faulty input files is wasting a lot of computational resources.

Finally, identifying the specific resource requirements for a given job is important to allocate the optimal amount for your compute job, and to decide if an HPC cluster is needed at all.

= How many and which kind of hardware resources do I require for it =

Resource allocation is a crucial part when working on an HPC cluster.
As this is dependent on both the job as well as the specific cluster hardware and architecture available.

'''
A small number of jobs and few resources'''

→ Submit to the scheduler. No extended testing and resource scaling analysis are needed.

'''
Medium-sized projects'''

→ Run only necessary jobs: Please consider testing new setups and their output for validity prior to submitting a huge amount of similar jobs

→ Start small: Run your problem on a small set of resources first.

→ Use the proper tools for development: If you develop your own code, please use the proper tools for debugging and parallel performance analysis. See: [[Development#Documentation_in_the_Wiki|Development]].

→ A look at the job feedback can help you determine if you are using the cluster efficiently

'''Large projects'''

→ Same approach as for medium-sized projects.

→ Run a scaling analysis for your project with regard to how many resources work best. See: [[Scaling]].

'''Many short jobs'''

→ Handling via the scheduler is inefficient.

→ Simple parallelization by hand is advisable. See: A basic introduction to [[Development/Parallel_Programming | Parallel Programming]].

= How do I optimize my code to use these resources most efficiently? =

The above recommendations will help use the cluster resources more efficiently.
Regarding software development, power efficiency correlates obviously heavily with '''computing performance''', but also with memory usage, i.e. the amount of memory used, but also memory efficiency.

Here, we have gathered a few results based on other research:
→ Use an efficient programming language such as Rust, C, and C++ -- well any compiled language. Do not use any interpreted language like Perl or Python. Since Machine Learning is a hot topic, this deserves a few words: Any ML-Python code using Tensorflow or other libraries will make heavy usage of NumPy and other math packages, which will use C-based implementations. Please make sure, you use the provided Python modules, which are optimized to use Intel MKL and other mathematical libraries.

Further reading:
Rui Pereira, et al: "''Energy efficiency across programming languages: how do energy, time, and memory relate?''", SLE 2017: Proc. of the 10th ACM SIGPLAN Int. Conf. on SW Language Eng., Oct. 2017, pp. 256–267, [https://doi.org/10.1145/3136014.3136031 doi:10.1145/3136014.3136031]

→ Analyse memory access patterns

→ For small tight loops checking for locks, use the <code>pause</code> instruction.

= Summary: General Recommendations =

* Choose the most '''efficient algorithms''' for the given problem
* Run only '''necessary''' jobs: Please consider testing new setups and their output for validity prior to submitting a huge amount of similar jobs
* Start '''small''': Run Your problem on a small number of parallel entities (be it processes or threads) first.
* '''Estimate''' the runtime of the parallel job as '''exactly''' as possible to increase the efficiency of the scheduling of the whole system
* Use the proper tools for development: If You develop your own code, please use the proper tools for debugging and parallel performance analysis. More information is available on the bwHPC Wiki.
* A look at the '''job feedback''' can help you determine if you are using the cluster efficiently

Energy Efficient Cluster Usage

2026-04-22T10:30:03Z

A Flachmueller: /* Introduction */

= Introduction =

Energy consumption of data centers has been increasing continuously throughout the last decade. In 2020, the energy consumption of all data centers in Germany amounted to around [https://www.bundestag.de/resource/blob/863850/423c11968fcb5c9995e9ef9090edf9e6/WD-8-070-21-pdf-data.pdf 3 percent] of the total electricity produced. Accompanying this large energy consumption are large-scale emissions of CO2 to the atmosphere and thus significant contributions to climate change.
To illustrate this, an average compute job running on a single node for one day may easily consume 10 kWh or even more. That translates roughly to brewing 700 cups of coffee.
Assuming that a typical bwHPC cluster has a few hundred compute nodes, this amounts to the energy consumption of a village for each cluster.

Although a large amount of this energy consumption is an intrinsic requirement of running large HPC clusters (even when it's processors are idle, a cluster uses a lot of energy), efficient use of the available resources is important. Using as many resources as possible does not make a power user. Using them wisely does.
In the following, a basic introduction to some of the most important aspects of energy-efficient HPC usage from a user perspective is given.

We can generally distinguish three tasks when optimizing for running HPC jobs efficiently.

→ What do I want to do and why do I need an HPC Cluster for it?

→ How many and which kind of hardware resources do I require for it?

→ How do I optimize my code to use these resources most efficiently?

= What do I want to do and why do I need an HPC Cluster for it? =

The bwHPC clusters are used to almost full capacity, and running a job on an HPC node consumes a lot of energy, as shown above.
Therefore, users are requested to run only necessary jobs.

Please consider testing new setups and their output for validity prior to submitting jobs that require lots of resources. This also includes projects where a lot of (smaller) similar jobs are submitted.

Make sure to double-check your jobs prior to the submission, having to discard the output data of an HPC project due to faulty input files is wasting a lot of computational resources.

Finally, identifying the specific resource requirements for a given job is important to allocate the optimal your compute job, and to decide if an HPC cluster is needed at all.

= How many and which kind of hardware resources do I require for it =

Resource allocation is a crucial part when working on an HPC cluster.
As this is dependent on both the job as well as the specific cluster hardware and architecture available.

'''
A small number of jobs and few resources'''

→ Submit to the scheduler. No extended testing and resource scaling analysis are needed.

'''
Medium-sized projects'''

→ Run only necessary jobs: Please consider testing new setups and their output for validity prior to submitting a huge amount of similar jobs

→ Start small: Run your problem on a small set of resources first.

→ Use the proper tools for development: If you develop your own code, please use the proper tools for debugging and parallel performance analysis. See: [[Development#Documentation_in_the_Wiki|Development]].

→ A look at the job feedback can help you determine if you are using the cluster efficiently

'''Large projects'''

→ Same approach as for medium-sized projects.

→ Run a scaling analysis for your project with regard to how many resources work best. See: [[Scaling]].

'''Many short jobs'''

→ Handling via the scheduler is inefficient.

→ Simple parallelization by hand is advisable. See: A basic introduction to [[Development/Parallel_Programming | Parallel Programming]].

= How do I optimize my code to use these resources most efficiently? =

The above recommendations will help use the cluster resources more efficiently.
Regarding software development, power efficiency correlates obviously heavily with '''computing performance''', but also with memory usage, i.e. the amount of memory used, but also memory efficiency.

Here, we have gathered a few results based on other research:
→ Use an efficient programming language such as Rust, C, and C++ -- well any compiled language. Do not use any interpreted language like Perl or Python. Since Machine Learning is a hot topic, this deserves a few words: Any ML-Python code using Tensorflow or other libraries will make heavy usage of NumPy and other math packages, which will use C-based implementations. Please make sure, you use the provided Python modules, which are optimized to use Intel MKL and other mathematical libraries.

Further reading:
Rui Pereira, et al: "''Energy efficiency across programming languages: how do energy, time, and memory relate?''", SLE 2017: Proc. of the 10th ACM SIGPLAN Int. Conf. on SW Language Eng., Oct. 2017, pp. 256–267, [https://doi.org/10.1145/3136014.3136031 doi:10.1145/3136014.3136031]

→ Analyse memory access patterns

→ For small tight loops checking for locks, use the <code>pause</code> instruction.

= Summary: General Recommendations =

* Choose the most '''efficient algorithms''' for the given problem
* Run only '''necessary''' jobs: Please consider testing new setups and their output for validity prior to submitting a huge amount of similar jobs
* Start '''small''': Run Your problem on a small number of parallel entities (be it processes or threads) first.
* '''Estimate''' the runtime of the parallel job as '''exactly''' as possible to increase the efficiency of the scheduling of the whole system
* Use the proper tools for development: If You develop your own code, please use the proper tools for debugging and parallel performance analysis. More information is available on the bwHPC Wiki.
* A look at the '''job feedback''' can help you determine if you are using the cluster efficiently

Running Calculations

2026-03-11T11:46:16Z

A Flachmueller: /* Life Cycle of a Calculation (Job) */

← This page is used in the [[HPC Glossary]] to explain the term "Batch Scheduler" and "Batch System"
== Life Cycle of a Calculation (Job) ==
[[File:running_calculations_on_cluster.svg|thumb|upright=0.4]]
On your desktop computer you start your calculations and they start immediately, running until they are finished. Then your desktop does mostly nothing, until you start another calculation. A compute cluster has several hundred, maybe a thousand computers (compute nodes), all of them are busy most of the time and many people want to run a great number of calculations. So running your job has to include some extra steps:

# prepare a script (a set commands to run - usually as a shell script), with all the commands that are necessary to run your calculation from start to finish. In addition to the commands necessary to run the calculation, this ''[[batch script]]'' has a header section, in which you specify details like required compute cores (processing units witin a computer), estimated runtime, memory requirements, disk space needed, etc.
# ''Submit'' the script into a queue, where your ''job'' (calculation)
# gets asigned an inital priority, is queued and waits in row with other compute jobs until the resources you requested in the header become available. (Requested time is also a resource!)
# Execution: Once a suitable resource slot is available and your job is in the front of the queue of suitable jobs (suitable with regards to the resource slot), your script is executed on (a) compute node(s). Your calculation runs on that/those node(s) until it is finished or reaches the specified time limit.
# Save results: Include commands to save the calculation results back to a long term storage (e.g. your home directory), at least at the end of your script. What you have not saved until the job finishes won't be saved!
# If your job reaches the specified time limit, all your running processes will be killed and the resources get cleared. So any data that has not been saved will be lost!

The software that distributes jobs on compute nodes is called a '''[[batch system]]''' or '''batch scheduler'''. The software currently used as a [[batch system]] on bwHPC clusters is "Slurm".

Learn more about the functioning of job distribution in

→ '''[[batch system]]'''

== Example Jobs ==

For most software that a bwHPC project installed on the cluster, we have prepared an example job script running some example calculation with that exact software.

How to access these examples is described in the "Software job examples" section of the page

→ '''[[Environment Modules]]'''

== Link to Batch System per Cluster ==

Because of differences in configuration (partly due to different available hardware), each cluster has their own batch system documentation:

→ '''[[BwUniCluster3.0/Running_Jobs|Slurm bwUniCluster 3.0]]'''

→ '''[[JUSTUS2/Jobscripts: Running Your Calculations | Slurm JUSTUS 2]]'''

→ '''[[Helix/Slurm | Slurm Helix]]'''

→ '''[[NEMO2/Slurm | Slurm NEMO2]]'''

→ '''[[BinAC2/Slurm | Slurm BinAC2]]'''

== How to Use Computing Ressources Efficiently ==

When you are running your calculations, you will have to decide on how many compute-cores your job will be simultaneously calculated.
For this, your computational problem will have to be divided into pieces, which always causes some overhead.

How to find a reasonable number of how many compute cores to use for your calculation can be found under

→ '''[[Scaling]]'''

Running calculations on an HPC node consumes a lot of energy. To make the most of the available resources and keep cluster and energy use as efficient as possible please also see our advice for

→ '''[[Energy Efficient Cluster Usage]]'''

Running Calculations

2026-03-11T11:15:34Z

A Flachmueller: /* Life Cycle of a Calculation (Job) */

← This page is used in the [[HPC Glossary]] to explain the term "Batch Scheduler" and "Batch System"
== Life Cycle of a Calculation (Job) ==
[[File:running_calculations_on_cluster.svg|thumb|upright=0.4]]
On your desktop computer you start your calculations and they start immediately, running until they are finished. Then your desktop does mostly nothing, until you start another calculation. A compute cluster has several hundred, maybe a thousand computers (compute nodes), all of them are busy most of the time and many people want to run a great number of calculations. So running your job has to include some extra steps:

# prepare a script (a set commands to run - usually as a shell script), with all the commands that are necessary to run your calculation from start to finish. In addition to the commands necessary to run the calculation, this ''[[batch script]]'' has a header section, in which you specify details like required compute cores (processing units witin a computer), estimated runtime, memory requirements, disk space needed, etc.
# ''Submit'' the script into a queue, where your ''job'' (calculation)
# is queued and waits in row with other compute jobs until the resources you requested in the header become available.
# Execution: Once a suitable resource slot is available and your job is in the front of the queue of suitable jobs (suitable with regards to the resource slot), your script is executed on (a) compute node(s). Your calculation runs on that/those node(s) until it is finished or reaches the specified time limit.
# Save results: Include commands to save the calculation results back to a long term storage (e.g. your home directory), at least at the end of your script. What you have not saved until the job finishes won't be saved!
# If your job reaches the specified time limit, all your running processes will be killed and the resources get cleared. So any data that has not been saved will be lost!

The software that distributes jobs on compute nodes is called a '''[[batch system]]''' or '''batch scheduler'''. The software currently used as a [[batch system]] on bwHPC clusters is "Slurm".

Learn more about the functioning of job distribution in

→ '''[[batch system]]'''

== Example Jobs ==

For most software that a bwHPC project installed on the cluster, we have prepared an example job script running some example calculation with that exact software.

How to access these examples is described in the "Software job examples" section of the page

→ '''[[Environment Modules]]'''

== Link to Batch System per Cluster ==

Because of differences in configuration (partly due to different available hardware), each cluster has their own batch system documentation:

→ '''[[BwUniCluster3.0/Running_Jobs|Slurm bwUniCluster 3.0]]'''

→ '''[[JUSTUS2/Jobscripts: Running Your Calculations | Slurm JUSTUS 2]]'''

→ '''[[Helix/Slurm | Slurm Helix]]'''

→ '''[[NEMO2/Slurm | Slurm NEMO2]]'''

→ '''[[BinAC2/Slurm | Slurm BinAC2]]'''

== How to Use Computing Ressources Efficiently ==

When you are running your calculations, you will have to decide on how many compute-cores your job will be simultaneously calculated.
For this, your computational problem will have to be divided into pieces, which always causes some overhead.

How to find a reasonable number of how many compute cores to use for your calculation can be found under

→ '''[[Scaling]]'''

Running calculations on an HPC node consumes a lot of energy. To make the most of the available resources and keep cluster and energy use as efficient as possible please also see our advice for

→ '''[[Energy Efficient Cluster Usage]]'''

Running Calculations

2026-03-10T16:34:04Z

A Flachmueller: Execution step is expressed more broadly + minor corrections

← This page is used in the [[HPC Glossary]] to explain the term "Batch Scheduler" and "Batch System"
== Life Cycle of a Calculation (Job) ==
[[File:running_calculations_on_cluster.svg|thumb|upright=0.4]]
On your desktop computer you start your calculations and they start immediately, running until they are finished. Then your desktop does mostly nothing, until you start another calculation. A compute cluster has several hundred, maybe a thousand computers (compute nodes), all of them are busy most of the time and many people want to run a great number of calculations. So running your job has to include some extra steps:

# prepare a script (a set commands to run - usually as a shell script), with all the commands that are necessary to run your calculation from start to finish. In addition to the commands necessary to run the calculation, this ''[[batch script]]'' has a header section, in which you specify details like required compute cores (processing units witin a computer), estimated runtime, memory requirements, disk space needed, etc.
# ''Submit'' the script into a queue, where your ''job'' (calculation)
# is queued and waits in row with other compute jobs until the resources you requested in the header become available.
# Execution: Once a suitable resource slot is available and your job is in the front of the queue of suitable jobs (suitable with regards to the resource slot), your script is executed on (a) compute node(s). Your calculation runs on that/those node(s) until it is finished or reaches the specified time limit.
# Save results: Include commands to save the calculation results back to a long term storage (e.g. your home directory), at least at the end of your script.

The software that distributes jobs on compute nodes is called a '''[[batch system]]''' or '''batch scheduler'''. The software currently used as a [[batch system]] on bwHPC clusters is "Slurm".

Learn more about the functioning of job distribution in

→ '''[[batch system]]'''

== Example Jobs ==

For most software that a bwHPC project installed on the cluster, we have prepared an example job script running some example calculation with that exact software.

How to access these examples is described in the "Software job examples" section of the page

→ '''[[Environment Modules]]'''

== Link to Batch System per Cluster ==

Because of differences in configuration (partly due to different available hardware), each cluster has their own batch system documentation:

→ '''[[BwUniCluster3.0/Running_Jobs|Slurm bwUniCluster 3.0]]'''

→ '''[[JUSTUS2/Jobscripts: Running Your Calculations | Slurm JUSTUS 2]]'''

→ '''[[Helix/Slurm | Slurm Helix]]'''

→ '''[[NEMO2/Slurm | Slurm NEMO2]]'''

→ '''[[BinAC2/Slurm | Slurm BinAC2]]'''

== How to Use Computing Ressources Efficiently ==

When you are running your calculations, you will have to decide on how many compute-cores your job will be simultaneously calculated.
For this, your computational problem will have to be divided into pieces, which always causes some overhead.

How to find a reasonable number of how many compute cores to use for your calculation can be found under

→ '''[[Scaling]]'''

Running calculations on an HPC node consumes a lot of energy. To make the most of the available resources and keep cluster and energy use as efficient as possible please also see our advice for

→ '''[[Energy Efficient Cluster Usage]]'''