bwHPC Wiki - User contributions [en]

Registration/HAW

2026-06-01T07:39:30Z

R Barthel:

The [https://hochschulen-bw.de/haw-bw/ HAW BW e.V.] is the voluntary association of the Universities of Applied Sciences Baden-Württemberg.

Currently the following colleges own a share in the bwUniCluster 3.0 and are therefore allowed to use it:

* Hochschule Aalen
* Hochschule Albstadt-Sigmaringen
* Hochschule Esslingen
* Hochschule Heilbronn
* Hochschule Konstanz (HTWG)
* Hochschule Mannheim
* Hochschule Offenburg
* Hochschule Reutlingen
* Hochschule Rottenburg
* Hochschule Stuttgart (HfT)
* Hochschule Ulm
* All sites of the Dualen Hochschule Baden-Württemberg

If you want to become a shareholder of bwUniCluster 3.0, please contact the [https://www.bwhpc.de/supportportal support ticket system].

This does not apply to the bwForClusters. However, [https://www.bwhpc.de/bwhpc-domains.php research in a supported field] is needed here.

Registration/HAW

2026-06-01T07:38:38Z

R Barthel:

The [https://hochschulen-bw.de/haw-bw/ HAW BW e.V.] is the voluntary association of the Universities of Applied Sciences Baden-Württemberg.

Currently the following colleges own a share in the bwUniCluster 2.0 and are therefore allowed to use it:

* Hochschule Aalen
* Hochschule Albstadt-Sigmaringen
* Hochschule Esslingen
* Hochschule Heilbronn
* Hochschule Konstanz (HTWG)
* Hochschule Mannheim
* Hochschule Offenburg
* Hochschule Reutlingen
* Hochschule Rottenburg
* Hochschule Stuttgart (HfT)
* Hochschule Ulm
* All sites of the Dualen Hochschule Baden-Württemberg

If you want to become a shareholder of bwUniCluster 3.0, please contact the [https://www.bwhpc.de/supportportal support ticket system].

This does not apply to the bwForClusters. However, [https://www.bwhpc.de/bwhpc-domains.php research in a supported field] is needed here.

BwUniCluster3.0/Hardware and Architecture

2025-09-03T22:45:34Z

R Barthel:

= Architecture of bwUniCluster 3.0 =

The '''bwUniCluster 3.0''' is a parallel computer with distributed memory.
It consists of the bwUniCluster 3.0 components procured in 2024 and also includes the additional compute nodes which were procured as an extension to the bwUniCluster 2.0 in 2022.

Each node of the system consists of two Intel Xeon or AMD EPYC processors, local memory, local storage, network adapters and optional accelerators (NVIDIA A100 and H100, AMD Instinct MI300A). All nodes are connected via a fast InfiniBand interconnect.

The parallel file system (Lustre) is connected to the InfiniBand switch of the compute cluster. This provides a fast and scalable parallel file
system.

The operating system on each node is Red Hat Enterprise Linux (RHEL) 9.4.

The individual nodes of the system act in different roles. From an end users point of view the different groups of nodes are login nodes and compute nodes. File server nodes and administrative server nodes are not accessible by users.

'''Login Nodes''' 
The login nodes are the only nodes directly accessible by end users. These nodes are used for interactive login, file management, program development, and interactive pre- and post-processing.
There are two nodes dedicated to this service, but they can all be reached from a single address: <code>uc3.scc.kit.edu</code>. A DNS round-robin alias distributes login sessions to the login nodes.
To prevent login nodes from being used for activities that are not permitted there and that affect the user experience of other users, '''long-running and/or compute-intensive tasks are periodically terminated without any prior warning'''. Please refer to [[BwUniCluster3.0/Login#Allowed_Activities_on_Login_Nodes|Allowed Activities on Login Nodes]].

'''Compute Nodes''' 
The majority of nodes are compute nodes which are managed by a batch system. Users submit their jobs to the SLURM batch system and a job is executed when the required resources become available (depending on its fair-share priority).

'''File Systems''' 
bwUniCluster 3.0 comprises two parallel file systems based on Lustre.

[[File:uc3.png|Optionen|center|Überschrift|800px]]

= Compute Resources =

== Login nodes ==

After a successful [[BwUniCluster3.0/Login|login]], users find themselves on one of the so called login nodes. Technically, these largely correspond to a standard CPU node, i.e. users have two AMD EPYC 9454 processors with a total of 96 cores at their disposal. Login nodes are the bridgehead for accessing computing resources.
Data and software are organized here, computing jobs are initiated and managed, and computing resources allocated for interactive use can also be accessed from here.
{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#ffa500; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#ffa500; text-align:left"|
'''Any compute intensive job running on the login nodes will be terminated without any notice.''' 
Please refer to [[BwUniCluster3.0/Login#Allowed_Activities_on_Login_Nodes|Allowed Activities on Login Nodes]].
|}

== Compute nodes ==
All compute activities on bwUniCluster 3.0 have to be performed on the compute nodes. Compute nodes are only available by requesting the corresponding resources via the queuing system. As soon as the requested resources are available, automated tasks are executed via a batch script or they can be accessed interactively. Please refer to [[BwUniCluster3.0/Running_Jobs|Running Jobs]] on how to request resources. 
The following compute node types are available: 
CPU nodes
* '''Standard''': Two AMD EPYC 9454 processors per node with a total of 96 physical CPU cores or 192 logical cores (Hyper-Threading) per node. The nodes have been procured in 2024.
* '''Ice Lake''': Two Intel Xeon Platinum 8358 processors per node with a total of 64 physical CPU cores or 128 logical cores (Hyper-Threading) per node. The nodes have been procured in 2022 as an extension to bwUniCluster 2.0.
* '''High Memory''': Similar to the standard nodes, but with six times larger memory.
GPU nodes
* '''NVIDIA GPU x4''': Similar to the standard nodes, but with larger memory and four NVIDIA H100 GPUs.
* '''AMD GPU x4''': AMD's accelerated processing unit (APU) MI300A with 4 CPU sockets and 4 compute units which share the same high-bandwidth memory (HBM).
* '''Ice Lake NVIDIA GPU x4''': Similar to the Ice Lake nodes, but with larger memory and four NVIDIA A100 or H100 GPUs.
* '''Cascade Lake NVIDIA GPU x4''': Nodes with four NVIDIA A100 GPUs.
{| class="wikitable"
|-
! style="width:10%"| Node Type
! style="width:10%"| CPU nodes Ice Lake
! style="width:10%"| CPU nodes Standard
! style="width:10%"| CPU nodes High Memory
! style="width:10%"| GPU nodes NVIDIA GPU x4
! style="width:10%"| GPU node AMD GPU x4
! style="width:10%"| GPU nodes Ice Lake NVIDIA GPU x4
! style="width:10%"| GPU nodes Cascade Lake NVIDIA GPU x4
! style="width:10%"| Login nodes
|-
!scope="column"| Availability in [[BwUniCluster3.0/Running_Jobs#Queues_on_bwUniCluster_3.0| queues]]
| <code>cpu_il</code>, <code>dev_cpu_il</code>
| <code>cpu</code>, <code>dev_cpu</code>
| <code>highmem</code>, <code>dev_highmem</code>
| <code>gpu_h100</code>, <code>dev_gpu_h100</code>
| <code>gpu_mi300</code>
| <code>gpu_a100_il</code> / <code>gpu_h100_il</code>
| <code>gpu_a100_short</code></code>
| -
|-
!scope="column"| Number of nodes
| 272
| 70
| 4
| 12
| 1
| 15
| 19
| 2
|-
!scope="column"| Processors
| Intel Xeon Platinum 8358
| AMD EPYC 9454
| AMD EPYC 9454
| AMD EPYC 9454
| AMD Zen 4
| Intel Xeon Platinum 8358
| Intel Xeon Gold 6248R
| AMD EPYC 9454
|-
!scope="column"| Number of sockets
| 2
| 2
| 2
| 2
| 4
| 2
| 2
| 2
|-
!scope="column"| Total number of cores
| 64
| 96
| 96
| 96
| 96 (4x 24)
| 64
| 48
| 96
|-
!scope="column"| Main memory
| 256 GiB
| 384 GiB
| 2304 GiB
| 768 GiB
| 4x 128 GiB HBM3
| 512 GiB
| 384 GiB
| 384 GiB
|-
!scope="column"| Local SSD
| 1.8 TB NVMe
| 3.84 TB NVMe
| 15.36 TB NVMe
| 15.36 TB NVMe
| 7.68 TB NVMe
| 6.4 TB NVMe
| 1.92 TB SATA SSD
| 7.68 TB SATA SSD
|-
!scope="column"| Accelerators
| -
| -
| -
| 4x NVIDIA H100
| 4x AMD Instinct MI300A
| 4x NVIDIA A100 / H100
| 4x NVIDIA A100
| -
|-
!scope="column"| Accelerator memory
| -
| -
| -
| 94 GiB
| APU
| 80 GiB / 94 GiB
| 40 GiB
| -
|-
!scope="column"| Interconnect
| IB HDR200
| IB 2x NDR200
| IB 2x NDR200
| IB 4x NDR200
| IB 2x NDR200
| IB 2x HDR200
| IB 4x EDR
| IB 1x NDR200
|}
Table 1: Hardware overview and properties

= File Systems =

On bwUniCluster 3.0 the following file systems are available:

* '''$HOME''' The HOME directory is created automatically after account activation, and the environment variable $HOME holds its name. HOME is the place, where users find themselves after login.
* '''Workspaces''' Users can create so-called workspaces for non-permanent data with temporary lifetime.
* '''Workspaces on flash storage''' A further workspace file system based on flash-only storage is available for special requirements and certain users.
* '''$TMPDIR''' The directory $TMPDIR is only available and visible on the local node during the runtime of a compute job. It is located on fast SSD storage devices.
* '''BeeOND''' (BeeGFS On-Demand) On request a parallel on-demand file system (BeeOND) is created which uses the SSDs of the nodes which were allocated to the batch job.
* '''LSDF Online Storage''' On request the external LSDF Online Storage is mounted on the nodes which were allocated to the batch job. On the login nodes, LSDF is automatically mounted.

'''Which file system to use?'''

You should separate your data and store it on the appropriate file system.
Permanently needed data like software or important results should be stored in $HOME but capacity restrictions (quotas) apply.
In case you accidentally deleted data on $HOME there is a chance that we can restore it from backup.
Permanent data which is not needed for months or exceeds the capacity restrictions should be sent to the LSDF Online Storage or to the archive and deleted from the file systems. Temporary data which is only needed on a single node and which does not exceed the disk space shown in Table 1 above should be stored
below $TMPDIR. Data which is read many times on a single node, e.g. if you are doing AI training,
should be copied to $TMPDIR and read from there. Temporary data which is used from many nodes
of your batch job and which is only needed during job runtime should be stored on a
parallel on-demand file system BeeOND. Temporary data which can be recomputed or which is the
result of one job and input for another job should be stored in workspaces. The lifetime
of data in workspaces is limited and depends on the lifetime of the workspace which can be
several months.

For further details please check: [[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details|File System Details]]

== $HOME ==

The $HOME directories of bwUniCluster 3.0 users are located on the parallel file system Lustre.
You have access to your $HOME directory from all nodes of UC3. A regular backup of these directories
to tape archive is done automatically. The directory $HOME is used to hold those files that are
permanently used like source codes, configuration files, executable programs etc.

[[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#$HOME|Detailed information on $HOME]]

== Workspaces ==

On UC3 workspaces should be used to store large non-permanent data sets, e.g. restart files or output
data that has to be post-processed. The file system used for workspaces is also the parallel file system Lustre. This file system is especially designed for parallel access and for a high throughput to large
files. It is able to provide high data transfer rates of up to 40 GB/s write and read performance when data access is parallel.

On UC3 there is a default user quota limit of 40 TiB and 20 million inodes (files and directories) per user.

[[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#Workspaces|Detailed information on Workspaces]]

== Workspaces on flash storage ==

Another workspace file system based on flash-only storage is available for special requirements and certain users.
If possible, this file system should be used from the Ice Lake nodes of bwUniCluster 3.0 (queue ''cpu_il'').
It provides high IOPS rates and better performance for small files. The quota limts are lower than on the
normal workspace file system.

[[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#Workspaces_on_flash_storage|Detailed information on Workspaces on flash storage]]

== $TMPDIR ==

The environment variable $TMPDIR contains the name of a directory which is located on the local SSD of each node.
This directory should be used for temporary files being accessed from the local node. It should
also be used if you read the same data many times from a single node, e.g. if you are doing AI training.
Because of the extremely fast local SSD storage devices performance with small files is much better than on the parallel file systems.

[[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#$TMPDIR|Detailed information on $TMPDIR]]

== BeeOND (BeeGFS On-Demand) ==

Users have the possibility to request a private BeeOND (on-demand BeeGFS) parallel filesystem for each job. The file system is created during job startup and purged when your job completes.

[[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#BeeOND_(BeeGFS_On-Demand)|Detailed information on BeeOND]]

== LSDF Online Storage ==

The LSDF Online Storage allows dedicated users to store scientific measurement data and simulation results. BwUniCluster 3.0 has an extremely fast network connection to the LSDF Online Storage. This file system provides external access via different protocols and is only available for certain users.

[[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#LSDF_Online_Storage|Detailed information on LSDF Online Storage]]

BwUniCluster3.0/Hardware and Architecture

2025-09-03T22:44:22Z

R Barthel:

= Architecture of bwUniCluster 3.0 =

The '''bwUniCluster 3.0''' is a parallel computer with distributed memory.
It consists of the bwUniCluster 3.0 components procured in 2024 and also includes the additional compute nodes which were procured as an extension to the bwUniCluster 2.0 in 2022.

Each node of the system consists of two Intel Xeon or AMD EPYC processors, local memory, local storage, network adapters and optional accelerators (NVIDIA A100 and H100, AMD Instinct MI300A). All nodes are connected via a fast InfiniBand interconnect.

The parallel file system (Lustre) is connected to the InfiniBand switch of the compute cluster. This provides a fast and scalable parallel file
system.

The operating system on each node is Red Hat Enterprise Linux (RHEL) 9.4.

The individual nodes of the system act in different roles. From an end users point of view the different groups of nodes are login nodes and compute nodes. File server nodes and administrative server nodes are not accessible by users.

'''Login Nodes''' 
The login nodes are the only nodes directly accessible by end users. These nodes are used for interactive login, file management, program development, and interactive pre- and post-processing.
There are two nodes dedicated to this service, but they can all be reached from a single address: <code>uc3.scc.kit.edu</code>. A DNS round-robin alias distributes login sessions to the login nodes.
To prevent login nodes from being used for activities that are not permitted there and that affect the user experience of other users, '''long-running and/or compute-intensive tasks are periodically terminated without any prior warning'''. Please refer to [[BwUniCluster3.0/Login#Allowed_Activities_on_Login_Nodes|Allowed Activities on Login Nodes]].

'''Compute Nodes''' 
The majority of nodes are compute nodes which are managed by a batch system. Users submit their jobs to the SLURM batch system and a job is executed when the required resources become available (depending on its fair-share priority).

'''File Systems''' 
bwUniCluster 3.0 comprises two parallel file systems based on Lustre.

[[File:uc3.png|Optionen|center|Überschrift|800px]]

= Compute Resources =

== Login nodes ==

After a successful [[BwUniCluster3.0/Login|login]], users find themselves on one of the so called login nodes. Technically, these largely correspond to a standard CPU node, i.e. users have two AMD EPYC 9454 processors with a total of 96 cores at their disposal. Login nodes are the bridgehead for accessing computing resources.
Data and software are organized here, computing jobs are initiated and managed, and computing resources allocated for interactive use can also be accessed from here.
{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#ffa500; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#ffa500; text-align:left"|
'''Any compute intensive job running on the login nodes will be terminated without any notice.''' 
Please refer to [[BwUniCluster3.0/Login#Allowed_Activities_on_Login_Nodes|Allowed Activities on Login Nodes]].
|}

== Compute nodes ==
All compute activities on bwUniCluster 3.0 have to be performed on the compute nodes. Compute nodes are only available by requesting the corresponding resources via the queuing system. As soon as the requested resources are available, automated tasks are executed via a batch script or they can be accessed interactively. Please refer to [[BwUniCluster3.0/Running_Jobs|Running Jobs]] on how to request resources. 
The following compute node types are available: 
CPU nodes
* '''Standard''': Two AMD EPYC 9454 processors per node with a total of 96 physical CPU cores or 192 logical cores (Hyper-Threading) per node. The nodes have been procured in 2024.
* '''Ice Lake''': Two Intel Xeon Platinum 8358 processors per node with a total of 64 physical CPU cores or 128 logical cores (Hyper-Threading) per node. The nodes have been procured in 2022 as an extension to bwUniCluster 2.0.
* '''High Memory''': Similar to the standard nodes, but with six times larger memory.
GPU nodes
* '''NVIDIA GPU x4''': Similar to the standard nodes, but with larger memory and four NVIDIA H100 GPUs.
* '''AMD GPU x4''': AMD's accelerated processing unit (APU) MI300A with 4 CPU sockets and 4 compute units which share the same high-bandwidth memory (HBM).
* '''Ice Lake NVIDIA GPU x4''': Similar to the Ice Lake nodes, but with larger memory and four NVIDIA A100 or H100 GPUs.
* '''Cascade Lake NVIDIA GPU x4''': Nodes with four NVIDIA A100 GPUs.
{| class="wikitable"
|-
! style="width:10%"| Node Type
! style="width:10%"| CPU nodes Ice Lake
! style="width:10%"| CPU nodes Standard
! style="width:10%"| CPU nodes High Memory
! style="width:10%"| GPU nodes NVIDIA GPU x4
! style="width:10%"| GPU node AMD GPU x4
! style="width:10%"| GPU nodes Ice Lake NVIDIA GPU x4
! style="width:10%"| GPU nodes Cascade Lake NVIDIA GPU x4
! style="width:10%"| Login nodes
|-
!scope="column"| Availability in [[BwUniCluster3.0/Running_Jobs#Queues_on_bwUniCluster_3.0| queues]]
| <code>cpu_il</code>, <code>dev_cpu_il</code>
| <code>cpu</code>, <code>dev_cpu</code>
| <code>highmem</code>, <code>dev_highmem</code>
| <code>gpu_h100</code>, <code>dev_gpu_h100</code>
| <code>gpu_mi300</code>
| <code>gpu_a100_il</code> / <code>gpu_h100_il</code>
| <code>gpu_a100_short</code></code>
| -
|-
!scope="column"| Number of nodes
| 272
| 70
| 4
| 12
| 1
| 15
| 19
| 2
|-
!scope="column"| Processors
| Intel Xeon Platinum 8358
| AMD EPYC 9454
| AMD EPYC 9454
| AMD EPYC 9454
| AMD Zen 4
| Intel Xeon Platinum 8358
| Intel Xeon Gold 6248R
| AMD EPYC 9454
|-
!scope="column"| Number of sockets
| 2
| 2
| 2
| 2
| 4
| 2
| 2
| 2
|-
!scope="column"| Total number of cores
| 64
| 96
| 96
| 96
| 96 (4x 24)
| 64
| 48
| 96
|-
!scope="column"| Main memory
| 256 GiB
| 384 GiB
| 2304 GiB
| 768 GiB
| 4x 128 GiB HBM3
| 512 GiB
| 384 GiB
| 384 GiB
|-
!scope="column"| Local SSD
| 1.8 TB NVMe
| 3.84 TB NVMe
| 15.36 TB NVMe
| 15.36 TB NVMe
| 7.68 TB NVMe
| 6.4 TB NVMe
| 1.92 TB SATA SSD
| 7.68 TB SATA SSD
|-
!scope="column"| Accelerators
| -
| -
| -
| 4x NVIDIA H100
| 4x AMD Instinct MI300A
| 4x NVIDIA A100 / H100
| 4x NVIDIA A100
| -
|-
!scope="column"| Accelerator memory
| -
| -
| -
| 94 GB
| APU
| 80 GB / 94 GB
| 40 GB
| -
|-
!scope="column"| Interconnect
| IB HDR200
| IB 2x NDR200
| IB 2x NDR200
| IB 4x NDR200
| IB 2x NDR200
| IB 2x HDR200
| IB 4x EDR
| IB 1x NDR200
|}
Table 1: Hardware overview and properties

= File Systems =

On bwUniCluster 3.0 the following file systems are available:

* '''$HOME''' The HOME directory is created automatically after account activation, and the environment variable $HOME holds its name. HOME is the place, where users find themselves after login.
* '''Workspaces''' Users can create so-called workspaces for non-permanent data with temporary lifetime.
* '''Workspaces on flash storage''' A further workspace file system based on flash-only storage is available for special requirements and certain users.
* '''$TMPDIR''' The directory $TMPDIR is only available and visible on the local node during the runtime of a compute job. It is located on fast SSD storage devices.
* '''BeeOND''' (BeeGFS On-Demand) On request a parallel on-demand file system (BeeOND) is created which uses the SSDs of the nodes which were allocated to the batch job.
* '''LSDF Online Storage''' On request the external LSDF Online Storage is mounted on the nodes which were allocated to the batch job. On the login nodes, LSDF is automatically mounted.

'''Which file system to use?'''

You should separate your data and store it on the appropriate file system.
Permanently needed data like software or important results should be stored in $HOME but capacity restrictions (quotas) apply.
In case you accidentally deleted data on $HOME there is a chance that we can restore it from backup.
Permanent data which is not needed for months or exceeds the capacity restrictions should be sent to the LSDF Online Storage or to the archive and deleted from the file systems. Temporary data which is only needed on a single node and which does not exceed the disk space shown in Table 1 above should be stored
below $TMPDIR. Data which is read many times on a single node, e.g. if you are doing AI training,
should be copied to $TMPDIR and read from there. Temporary data which is used from many nodes
of your batch job and which is only needed during job runtime should be stored on a
parallel on-demand file system BeeOND. Temporary data which can be recomputed or which is the
result of one job and input for another job should be stored in workspaces. The lifetime
of data in workspaces is limited and depends on the lifetime of the workspace which can be
several months.

For further details please check: [[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details|File System Details]]

== $HOME ==

The $HOME directories of bwUniCluster 3.0 users are located on the parallel file system Lustre.
You have access to your $HOME directory from all nodes of UC3. A regular backup of these directories
to tape archive is done automatically. The directory $HOME is used to hold those files that are
permanently used like source codes, configuration files, executable programs etc.

[[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#$HOME|Detailed information on $HOME]]

== Workspaces ==

On UC3 workspaces should be used to store large non-permanent data sets, e.g. restart files or output
data that has to be post-processed. The file system used for workspaces is also the parallel file system Lustre. This file system is especially designed for parallel access and for a high throughput to large
files. It is able to provide high data transfer rates of up to 40 GB/s write and read performance when data access is parallel.

On UC3 there is a default user quota limit of 40 TiB and 20 million inodes (files and directories) per user.

[[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#Workspaces|Detailed information on Workspaces]]

== Workspaces on flash storage ==

Another workspace file system based on flash-only storage is available for special requirements and certain users.
If possible, this file system should be used from the Ice Lake nodes of bwUniCluster 3.0 (queue ''cpu_il'').
It provides high IOPS rates and better performance for small files. The quota limts are lower than on the
normal workspace file system.

[[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#Workspaces_on_flash_storage|Detailed information on Workspaces on flash storage]]

== $TMPDIR ==

The environment variable $TMPDIR contains the name of a directory which is located on the local SSD of each node.
This directory should be used for temporary files being accessed from the local node. It should
also be used if you read the same data many times from a single node, e.g. if you are doing AI training.
Because of the extremely fast local SSD storage devices performance with small files is much better than on the parallel file systems.

[[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#$TMPDIR|Detailed information on $TMPDIR]]

== BeeOND (BeeGFS On-Demand) ==

Users have the possibility to request a private BeeOND (on-demand BeeGFS) parallel filesystem for each job. The file system is created during job startup and purged when your job completes.

[[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#BeeOND_(BeeGFS_On-Demand)|Detailed information on BeeOND]]

== LSDF Online Storage ==

The LSDF Online Storage allows dedicated users to store scientific measurement data and simulation results. BwUniCluster 3.0 has an extremely fast network connection to the LSDF Online Storage. This file system provides external access via different protocols and is only available for certain users.

[[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#LSDF_Online_Storage|Detailed information on LSDF Online Storage]]

MediaWiki:Sidebar

2025-04-06T18:54:19Z

R Barthel:

Main Page

2025-04-06T18:52:46Z

R Barthel:

{| style=" background:#FEF4AB; width:100%;"
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News
|-
|
Our ticketing system has been upgraded. Please check out the visual guide on how to create tickets [[BwSupportPortal|here]].

|}

'''Welcome to the bwHPC Wiki.'''

bwHPC represents services and resources in the State of '''B'''aden-'''W'''ürttemberg, Germany, for High Performance Computing ('''HPC'''), Data Intensive Computing ('''DIC''') and Large Scale Scientific Data Management ('''LS2DM''').

The main bwHPC web page is at [https://www.bwhpc.de/ https://www.bwhpc.de/].

Many topics depend on the cluster system you use.
First choose the cluster you use, then select the correct topic.

{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Courses / eLearning
|-
|
* [https://training.bwhpc.de/ eLearning and Online Courses]
* [https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC Introduction to Linux in HPC (external resource)]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | Need Access to a Cluster?
|-
|
* [[When (not) to use a HPC Cluster]]
* [[Registration]]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | HPC System Specific Documentation
|-
|
bwHPC Clusters are dedicated to [https://www.bwhpc.de/bwhpc-domains.php specific research domains].
Documentation differs between compute clusters, please see cluster specific overview pages:
{|
|-
| style="padding:5px; width:30%" | [[BwUniCluster3.0|bwUniCluster 3.0]]
| style="padding-left:20px;" | General Purpose, Teaching
|-
| style="padding:5px; width:30%" | [[:JUSTUS2| bwForCluster JUSTUS 2]]
| style="padding-left:20px;" | Theoretical Chemistry, Condensed Matter Physics, and Quantum Sciences
|-
| style="padding:5px; width:30%" | [[Helix|bwForCluster Helix]]
| style="padding-left:20px;" | Structural and Systems Biology, Medical Science, Soft Matter, Computational Humanities, and Mathematics and Computer Science
|-
| style="padding:5px; width:30%" | [[NEMO|bwForCluster NEMO]]
| style="padding-left:20px;" | Neurosciences, Particle Physics, Materials Science, and Microsystems Engineering
|-
| style="padding:5px; width:30%" | [[BinAC|bwForCluster BinAC]]
| style="padding-left:20px;" | Bioinformatics, Geosciences and Astrophysics.
|}
|-
|bwHPC Clusters: [https://www.bwhpc.de/cluster.php operational status]
Further Compute Clusters in Baden-Württemberg (separate access policies):
* Datenanalyse Cluster der Hochschulen (DACHS): [[DACHS | Datenanalyse Cluster der Hochschulen (DACHS)]]
* bwHPC tier 1: [https://kb.hlrs.de/platforms/index.php/Hunter_(HPE) Hunter] ([https://www.hlrs.de/apply-for-computing-time getting access])
* bwHPC tier 2: [https://www.nhr.kit.edu/userdocs/horeka HoreKa] ([https://www.nhr.kit.edu/userdocs/horeka/projects/ getting access])
|}

{| style="background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | Documentation valid for all Clusters
|-
|
* [[Environment Modules| Software Modules]] and software documentation explained
* [https://www.bwhpc.de/software.html List of Software] on all clusters
* [[Development| Software Development and Parallel Programming]]
* [[Energy Efficient Cluster Usage]]
* [[HPC Glossary]]

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | Scientific Data Storage
|-
|
For user guides of the scientific data storage services:
* [[SDS@hd]]
* [https://www.rda.kit.edu/english bwDataArchive]
* [https://zas.bwsfs.uni-tuebingen.de/info/uebersicht bwSFS]
Associated, but local scientific storage services are:
* [https://wiki.scc.kit.edu/lsdf/index.php/Category:LSDF_Online_Storage LSDF Online Storage] (only for KIT and KIT partners)
|}

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | Data Management
|-
|
* [[Data Transfer|Data Transfer]]
* [https://www.forschungsdaten.org/index.php/FDM-Kontakte#Deutschland Research Data Management (RDM)] contact persons
* [https://www.forschungsdaten.info Portal for Research Data Management] (Forschungsdaten.info)
|}
{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Support
|-
|
Support is provided by the [https://www.bwhpc.de/teams.php bwHPC Competence Centers]:
* [https://www.bwhpc.de/supportportal.php Submit a Ticket]
|}
{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Acknowledgement
|-
|
* Please [[Acknowledgement|acknowledge]] our resources in your publications.
|}

Monitor

2022-12-12T13:11:45Z

R Barthel:

__FORCETOC__

== Capacity utilization rate of bwHPC clusters ==

=== bwUniCluster 2.0 ===

per organisation:
* https://www.bwhpc.de/bwhpc-stats/k.7/a/oe.php#uc2
per bwHPC domain
* https://www.bwhpc.de/bwhpc-stats/k.7/a/sf.php#uc2
----

=== bwForCluster JUSTUS 2 ===

per organisation:
* https://www.bwhpc.de/bwhpc-stats/k.7/a/oe.php#justus2
per bwHPC domain
* https://www.bwhpc.de/bwhpc-stats/k.7/a/sf.php#justus2
----

=== bwForCluster MLS&WISO ===

per organisation:
* https://www.bwhpc.de/bwhpc-stats/k.7/a/oe.php#mlswiso
per bwHPC domain
* https://www.bwhpc.de/bwhpc-stats/k.7/a/sf.php#mlswiso
----

=== bwForCluster NEMO ===

per organisation:
* https://www.bwhpc.de/bwhpc-stats/k.7/a/oe.php#nemo
per bwHPC domain
* https://www.bwhpc.de/bwhpc-stats/k.7/a/sf.php#nemo
----

=== bwForCluster BinAC ===

per organisation:
* https://www.bwhpc.de/bwhpc-stats/k.7/a/oe.php#binac
per bwHPC domain
* https://www.bwhpc.de/bwhpc-stats/k.7/a/sf.php#binac
----

MediaWiki:Sidebar

2022-11-15T08:29:26Z

R Barthel:

BwUniCluster2.0/Slurm

2022-10-18T23:34:30Z

R Barthel: /* BeeOND (BeeGFS On-Demand) */

<div id="top"></div>
= Slurm HPC Workload Manager =
== Specification ==
Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters. Slurm requires no kernel modifications for its operation and is relatively self-contained. As a cluster workload manager, Slurm has three key functions. First, it allocates exclusive and/or non-exclusive access to resources (compute nodes) to users for some duration of time so they can perform work. Second, it provides a framework for starting, executing, and monitoring work (normally a parallel job) on the set of allocated nodes. Finally, it arbitrates contention for resources by managing a queue of pending work.
 
 
Any kind of calculation on the compute nodes of [[bwUniCluster 2.0|bwUniCluster 2.0]] requires the user to define calculations as a sequence of commands or single command together with required run time, number of CPU cores and main memory and submit all, i.e., the '''batch job''', to a resource and workload managing software. bwUniCluster 2.0 has installed the workload managing software Slurm. Therefore any job submission by the user is to be executed by commands of the Slurm software. Slurm queues and runs user jobs based on fair sharing policies.
 
 

== Slurm Commands (excerpt) ==
Some of the most used Slurm commands for non-administrators working on bwUniCluster 2.0.
{| width=750px class="wikitable"
! Slurm commands !! Brief explanation
|-
| [[#Job Submission : sbatch|sbatch]] || Submits a job and queues it in an input queue [[https://slurm.schedmd.com/sbatch.html sbatch]]
|-
| [[#Detailed job information : scontrol show job|scontrol show job]] || Displays detailed job state information [[https://slurm.schedmd.com/scontrol.html scontrol]]
|-
| [[#List of your submitted jo/bs : squeue|squeue]] || Displays information about active, eligible, blocked, and/or recently completed jobs [[https://slurm.schedmd.com/squeue.html squeue]]
|-
| [[#Start time of job or resources : squeue|squeue --start]] || Returns start time of submitted job or requested resources [[https://slurm.schedmd.com/squeue.html squeue]]
|-
| [[#Shows free resources : sinfo_t_idle|sinfo_t_idle]] || Shows what resources are available for immediate use [[https://slurm.schedmd.com/sinfo.html sinfo]]
|-
| [[#Canceling own jobs : scancel|scancel]] || Cancels a job (obsoleted!) [[https://slurm.schedmd.com/scancel.html scancel]]
|}
If your job was submitted to the "multiple" queue you can log into the allocated nodes via SSH as soon as the job is running.

 
* [https://slurm.schedmd.com/tutorials.html Slurm Tutorials]
* [https://slurm.schedmd.com/pdfs/summary.pdf Slurm command/option summary (2 pages)]
* [https://slurm.schedmd.com/man_index.html Slurm Commands]
 

== Job Submission : sbatch ==
Batch jobs are submitted by using the command '''sbatch'''. The main purpose of the '''sbatch''' command is to specify the resources that are needed to run the job. '''sbatch''' will then queue the batch job. However, starting of batch job depends on the availability of the requested resources and the fair sharing value.
 
 
=== sbatch Command Parameters ===
The syntax and use of '''sbatch''' can be displayed via:
<pre>
$ man sbatch
</pre>
'''sbatch''' options can be used from the command line or in your job script.
{| width=750px class="wikitable"
! colspan="3" | sbatch Options
|-
! Command line
! Script
! Purpose
|- style="vertical-align:top;"
| -t ''time'' or --time=''time''
| #SBATCH --time=''time''
| Wall clock time limit. 
|-
|- style="vertical-align:top;"
| -N ''count'' or --nodes=''count''
| #SBATCH --nodes=''count''
| Number of nodes to be used.
|- style="vertical-align:top;"
| -n ''count'' or --ntasks=''count''
| #SBATCH --ntasks=''count''
| Number of tasks to be launched.
|-
|- style="vertical-align:top;"
| --ntasks-per-node=''count''
| #SBATCH --ntasks-per-node=''count''
| Maximum count (<= 28 and <= 40 resp.) of tasks per node. (Replaces the option ppn of MOAB.)
|-
|- style="vertical-align:top;"
| -c ''count'' or --cpus-per-task=''count''
| #SBATCH --cpus-per-task=''count''
| Number of CPUs required per (MPI-)task.
|-
|- style="vertical-align:top;"
| --mem=''value_in_MB''
| #SBATCH --mem=''value_in_MB''
| Memory in MegaByte per node. (Default value is 128000 and 96000 MB resp., i.e. you should omit the setting of this option.)
|-
|- style="vertical-align:top;"
| --mem-per-cpu=''value_in_MB''
| #SBATCH --mem-per-cpu=''value_in_MB''
| Minimum Memory required per allocated CPU. (Replaces the option pmem of MOAB. You should omit the setting of this option.)
|-
|- style="vertical-align:top;"
| --mail-type=''type''
| #SBATCH --mail-type=''type''
| Notify user by email when certain event types occur. Valid type values are NONE, BEGIN, END, FAIL, REQUEUE, ALL.
|-
|- style="vertical-align:top;"
| --mail-user=''mail-address''
| #SBATCH --mail-user=''mail-address''
| The specified mail-address receives email notification of state changes as defined by --mail-type.
|-
|- style="vertical-align:top;"
| --output=''name''
| #SBATCH --output=''name''
| File in which job output is stored.
|-
|- style="vertical-align:top;"
| --error=''name''
| #SBATCH --error=''name''
| File in which job error messages are stored.
|-
|- style="vertical-align:top;"
| -J ''name'' or --job-name=''name''
| #SBATCH --job-name=''name''
| Job name.
|-
|- style="vertical-align:top;"
| --export=[ALL,] ''env-variables''
| #SBATCH --export=[ALL,] ''env-variables''
| Identifies which environment variables from the submission environment are propagated to the launched application. Default is ALL. If adding an environment variable to the submission environment is intended, the argument ALL must be added.
|-
|- style="vertical-align:top;"
| -A ''group-name'' or --account=''group-name''
| #SBATCH --account=''group-name''
| Change resources used by this job to specified group. You may need this option if your account is assigned to more than one group. By command "scontrol show job" the project group the job is accounted on can be seen behind "Account=".
|-
|- style="vertical-align:top;"
| -p ''queue-name'' or --partition=''queue-name''
| #SBATCH --partition=''queue-name''
| Request a specific queue for the resource allocation.
|-
|- style="vertical-align:top;"
| --reservation=''reservation-name''
| #SBATCH --reservation=''reservation-name''
| Use a specific reservation for the resource allocation.
|-
|- style="vertical-align:top;"
| -C ''LSDF'' or --constraint=''LSDF''
| #SBATCH --constraint=LSDF
| Job constraint LSDF Filesystems.
|-
|- style="vertical-align:top;"
| -C ''BEEOND (BEEOND_4MDS, BEEOND_MAXMDS)'' or --constraint=''BEEOND (BEEOND_4MDS, BEEOND_MAXMDS''
| #SBATCH --constraint=BEEOND (BEEOND_4MDS, BEEOND_MAXMDS)
| Job constraint BeeOND file system.
|-
|}
 

==== sbatch --partition ''queues'' ====
Queue classes define maximum resources such as walltime, nodes and processes per node and queue of the compute system. Details can be found here:
* [[BwUniCluster_2.0_Batch_Queues#sbatch_-p_queue|bwUniCluster 2.0 queue settings]]
 

=== sbatch Examples ===
==== Serial Programs ====
To submit a serial job that runs the script '''job.sh''' and that requires 5000 MB of main memory and 10 minutes of wall clock time

a) execute:
<pre>
$ sbatch -p dev_single -n 1 -t 10:00 --mem=5000 job.sh
</pre>
or
b) add after the initial line of your script '''job.sh''' the lines (here with a high memory request):
<source lang="bash">
#SBATCH --ntasks=1
#SBATCH --time=10
#SBATCH --mem=200gb
#SBATCH --job-name=simple
</source>
and execute the modified script with the command line option ''--partition=fat'' (with ''--partition=(dev_)single'' maximum ''--mem=96gb'' is possible):
<pre>
$ sbatch --partition=fat job.sh
</pre>
Note, that sbatch command line options overrule script options.
 
 

==== Multithreaded Programs ====
Multithreaded programs operate faster than serial programs on CPUs with multiple cores. 
Moreover, multiple threads of one process share resources such as memory.
 
For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) a number of threads is defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).
 
'''Because hyperthreading is switched on on bwUniCluster 2.0, the option --cpus-per-task (-c) must be set to 2*n, if you want to use n threads.'''
 
To submit a batch job called ''OpenMP_Test'' that runs a 40-fold threaded program ''omp_exe'' which requires 6000 MByte of total physical memory and total wall clock time of 40 minutes:
 
a) execute:
<pre>
$ sbatch -p single --export=ALL,OMP_NUM_THREADS=40 -J OpenMP_Test -N 1 -c 80 -t 40 --mem=6000 ./omp_exe
</pre>
or
-->
* generate the script '''job_omp.sh''' containing the following lines:
<source lang="bash">
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --cpus-per-task=80
#SBATCH --time=40:00
#SBATCH --mem=6000mb
#SBATCH --export=ALL,EXECUTABLE=./omp_exe
#SBATCH -J OpenMP_Test

#Usually you should set
export KMP_AFFINITY=compact,1,0
#export KMP_AFFINITY=verbose,compact,1,0 prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

export OMP_NUM_THREADS=$((${SLURM_JOB_CPUS_PER_NODE}/2))
echo "Executable ${EXECUTABLE} running on ${SLURM_JOB_CPUS_PER_NODE} cores with ${OMP_NUM_THREADS} threads"
startexe=${EXECUTABLE}
echo $startexe
exec $startexe
</source>
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores and, if necessary, replace <placeholder> with the required modulefile to enable the OpenMP environment and execute the script '''job_omp.sh''' adding the queue class ''single'' as sbatch option:
<pre>
$ sbatch -p single job_omp.sh
</pre>
Note, that sbatch command line options overrule script options, e.g.,
<pre>
$ sbatch --partition=single --mem=200 job_omp.sh
</pre>
overwrites the script setting of 6000 MByte with 200 MByte.
 
 

==== MPI Parallel Programs ====
MPI parallel programs run faster than serial programs on multi CPU and multi core systems. N-fold spawned processes of the MPI program, i.e., '''MPI tasks''', run simultaneously and communicate via the Message Passing Interface (MPI) paradigm. MPI tasks do not share memory but can be spawned over different nodes.
 
Multiple MPI tasks must be launched via '''mpirun''', e.g. 4 MPI tasks of ''my_par_program'':
<pre>
$ mpirun -n 4 my_par_program
</pre>
This command runs 4 MPI tasks of ''my_par_program'' on the node you are logged in.
To run this command with a loaded Intel MPI the environment variable I_MPI_HYDRA_BOOTSTRAP must be unset ( --> $ unset I_MPI_HYDRA_BOOTSTRAP).

Running MPI parallel programs in a batch job the interactive environment - particularly the loaded modules - will also be set in the batch job. If you want to set a defined module environment in your batch job you have to purge all modules before setting the wished modules.
 
 
===== OpenMPI =====

If you want to run jobs on batch nodes, generate a wrapper script ''job_ompi.sh'' for '''OpenMPI''' containing the following lines:
<source lang="bash">
#!/bin/bash
# Use when a defined module environment related to OpenMPI is wished
module load mpi/openmpi/<placeholder_for_version>
mpirun --bind-to core --map-by core -report-bindings my_par_program
</source>
'''Attention:''' Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since Slurm instructs mpirun about number of processes and node hostnames. Use '''ALWAYS''' the MPI options '''''--bind-to core''''' and '''''--map-by core|socket|node'''''. Please type ''mpirun --help'' for an explanation of the meaning of the different options of mpirun option ''--map-by''.
 
Considering 4 OpenMPI tasks on a single node, each requiring 2000 MByte, and running for 1 hour, execute:
<pre>
$ sbatch -p single -N 1 -n 4 --mem-per-cpu=2000 --time=01:00:00 ./job_ompi.sh
</pre>
 

===== Intel MPI =====

Generate a wrapper script for '''Intel MPI''', ''job_impi.sh'' containing the following lines:
<source lang="bash">
#!/bin/bash
# Use when a defined module environment related to Intel MPI is wished
module load mpi/impi/<placeholder_for_version>
mpiexec.hydra -bootstrap slurm my_par_program
</source>
'''Attention:''' 
Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since Slurm instructs mpirun about number of processes and node hostnames.
 
Launching and running 200 Intel MPI tasks on 5 nodes, each requiring 80 GByte, and running for 5 hours, execute:
<pre>
$ sbatch --partition=multiple -N 5 --ntasks-per-node=40 --mem=80gb -t 300 ./job_impi.sh
</pre>
 
If you want to use 128 or more nodes, you must also set the environment variable as follows: 
export I_MPI_HYDRA_BRANCH_COUNT=-1
 
If you want to use the options perhost, ppn or rr, you must additionally set the environment variable I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=off.
 
 

==== Multithreaded + MPI parallel Programs ====
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes. '''Because hyperthreading is switched on on bwUniCluster 2.0, the option --cpus-per-task (-c) must be set to 2*n, if you want to use n threads.'''
 
 
===== OpenMPI with Multithreading =====
Multiple MPI tasks using '''OpenMPI''' must be launched by the MPI parallel program '''mpirun'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).
 
'''For OpenMPI''' a job-script to submit a batch job called ''job_ompi_omp.sh'' that runs a MPI program with 4 tasks and a 28-fold threaded program ''ompi_omp_program'' requiring 3000 MByte of physical memory per thread (using 28 threads per MPI task you will get 28*3000 MByte = 84000 MByte per MPI task) and total wall clock time of 3 hours looks like:

<source lang="bash">
#!/bin/bash
#SBATCH --ntasks=4
#SBATCH --cpus-per-task=56
#SBATCH --time=03:00:00
#SBATCH --mem=83gb # 84000 MB = 84000/1024 GB = 82.1 GB
#SBATCH --export=ALL,MPI_MODULE=mpi/openmpi/3.1,EXECUTABLE=./ompi_omp_program
#SBATCH --output="parprog_hybrid_%j.out"

# Use when a defined module environment related to OpenMPI is wished
module load ${MPI_MODULE}
export OMP_NUM_THREADS=$((${SLURM_CPUS_PER_TASK}/2))
export MPIRUN_OPTIONS="--bind-to core --map-by socket:PE=${OMP_NUM_THREADS} -report-bindings"
export NUM_CORES=${SLURM_NTASKS}*${OMP_NUM_THREADS}
echo "${EXECUTABLE} running on ${NUM_CORES} cores with ${SLURM_NTASKS} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpirun -n ${SLURM_NTASKS} ${MPIRUN_OPTIONS} ${EXECUTABLE}"
echo $startexe
exec $startexe
</source>
Execute the script '''job_ompi_omp.sh''' by command sbatch:
<pre>
$ sbatch -p multiple_e ./job_ompi_omp.sh
</pre>
* With the mpirun option ''--bind-to core'' MPI tasks and OpenMP threads are bound to physical cores.
* With the option ''--map-by node:PE=<value>'' (neighbored) MPI tasks will be attached to different nodes and each MPI task is bound to the first core of a node. <value> must be set to ${OMP_NUM_THREADS}.
* The option ''-report-bindings'' shows the bindings between MPI tasks and physical cores.
* The mpirun-options '''--bind-to core''', '''--map-by socket|...|node:PE=<value>''' should always be used when running a multithreaded MPI program.
 

===== Intel MPI with Multithreading =====
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes.

Multiple Intel MPI tasks must be launched by the MPI parallel program '''mpiexec.hydra'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).

'''For Intel MPI''' a job-script to submit a batch job called ''job_impi_omp.sh'' that runs a Intel MPI program with 10 tasks and a 40-fold threaded program ''impi_omp_program'' requiring 96000 MByte of total physical memory per task and total wall clock time of 1 hours looks like:


<source lang="bash">
#!/bin/bash
#SBATCH --ntasks=10
#SBATCH --cpus-per-task=80
#SBATCH --time=60
#SBATCH --mem=96000
#SBATCH --export=ALL,MPI_MODULE=mpi/impi,EXE=./impi_omp_program
#SBATCH --output="parprog_impi_omp_%j.out"

#If using more than one MPI task per node please set
export KMP_AFFINITY=compact,1,0
#export KMP_AFFINITY=verbose,scatter prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

# Use when a defined module environment related to Intel MPI is wished
module load ${MPI_MODULE}
export OMP_NUM_THREADS=$((${SLURM_CPUS_PER_TASK}/2))
export MPIRUN_OPTIONS="-binding "domain=omp:compact" -print-rank-map -envall"
export NUM_PROCS=eval(${SLURM_NTASKS}*${OMP_NUM_THREADS})
echo "${EXE} running on ${NUM_PROCS} cores with ${SLURM_NTASKS} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpiexec.hydra -bootstrap slurm ${MPIRUN_OPTIONS} -n ${SLURM_NTASKS} ${EXE}"
echo $startexe
exec $startexe
</source>
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores. If you only run one MPI task per node please set KMP_AFFINITY=compact,1,0.
 
If you want to use 128 or more nodes, you must also set the environment variable as follows: 
export I_MPI_HYDRA_BRANCH_COUNT=-1
 
If you want to use the options perhost, ppn or rr, you must additionally set the environment variable I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=off.
 
 
Execute the script '''job_impi_omp.sh''' by command sbatch:
<pre>
$ sbatch -p multiple ./job_impi_omp.sh
</pre>
 
The mpirun option ''-print-rank-map'' shows the bindings between MPI tasks and nodes (not very beneficial). The option ''-binding'' binds MPI tasks (processes) to a particular processor; ''domain=omp'' means that the domain size is determined by the number of threads. If you would choose 2 MPI tasks per node, you should choose ''-binding "cell=unit;map=bunch"''; this binding maps one MPI process to each socket.
 
 

==== Chain jobs ====
The CPU time requirements of many applications exceed the limits of the job classes. In those situations it is recommended to solve the problem by a job chain. A job chain is a sequence of jobs where each job automatically starts its successor.
<source lang="bash">
#!/bin/bash
####################################
## simple Slurm submitter script to setup ##
## a chain of jobs using Slurm ##
####################################
## ver. : 2018-11-27, KIT, SCC

## Define maximum number of jobs via positional parameter 1, default is 5
max_nojob=${1:-5}

## Define your jobscript (e.g. "~/chain_job.sh")
chain_link_job=${PWD}/chain_job.sh

## Define type of dependency via positional parameter 2, default is 'afterok'
dep_type="${2:-afterok}"
## -> List of all dependencies:
## https://slurm.schedmd.com/sbatch.html

myloop_counter=1
## Submit loop
while [ ${myloop_counter} -le ${max_nojob} ] ; do
##
## Differ msub_opt depending on chain link number
if [ ${myloop_counter} -eq 1 ] ; then
slurm_opt=""
else
slurm_opt="-d ${dep_type}:${jobID}"
fi
##
## Print current iteration number and sbatch command
echo "Chain job iteration = ${myloop_counter}"
echo " sbatch --export=myloop_counter=${myloop_counter} ${slurm_opt} ${chain_link_job}"
## Store job ID for next iteration by storing output of sbatch command with empty lines
jobID=$(sbatch -p <queue> --export=ALL,myloop_counter=${myloop_counter} ${slurm_opt} ${chain_link_job} 2>&1 | sed 's/[S,a-z]* //g')
##
## Check if ERROR occured
if [[ "${jobID}" =~ "ERROR" ]] ; then
echo " -> submission failed!" ; exit 1
else
echo " -> job number = ${jobID}"
fi
##
## Increase counter
let myloop_counter+=1
done
</source>
 

==== GPU jobs ====

The nodes in the gpu_4 and gpu_8 queues have 4 or 8 NVIDIA Tesla V100 GPUs. Just submitting a job to these queues is not enough to also allocate one or more GPUs, you have to do so using the "--gres=gpu" parameter. You have to specifiy how many GPUs your job needs, e.g. "--gres=gpu:2" will request two GPUs.

The GPU nodes are shared between multiple jobs if the jobs don't request all the GPUs in a node and there are enough ressources to run more than one job. The individual GPUs are always bound to a single job and will not be shared between different jobs.

a) add after the initial line of your script job.sh the line including the
information about the LSDF Online Storage usage: #SBATCH --constraint=LSDF
<pre>
#!/bin/bash
#SBATCH --ntasks=40
#SBATCH --time=02:00:00
#SBATCH --mem=4000
#SBATCH --gres=gpu:2
</pre>

or b) execute:
<pre>
$ sbatch -p <queue> -n 40 -t 02:00:00 --mem 4000 --gres=gpu:2 job.sh
</pre>
 
If you start an interactive session on of the GPU nodes, you can use the "nvidia-smi" command to list the GPUs allocated to your job:
<pre>
$ nvidia-smi
Sun Mar 29 15:20:05 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2... Off | 00000000:3A:00.0 Off | 0 |
| N/A 29C P0 39W / 300W | 9MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla V100-SXM2... Off | 00000000:3B:00.0 Off | 0 |
| N/A 30C P0 41W / 300W | 8MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 14228 G /usr/bin/X 8MiB |
| 1 14228 G /usr/bin/X 8MiB |
+-----------------------------------------------------------------------------+
</pre>

 
In case of using Open MPI, the underlying communication infrastructure (UCX and Open MPI's BTL) is CUDA-aware.
However, there may be warnings, e.g. when running
<pre>
$ module load compiler/gnu/10.3 mpi/openmpi devel/cuad
$ mpirun mpirun -np 2 ./mpi_cuda_app
--------------------------------------
WARNING: There are more than one active ports on host 'uc2n520', but the
default subnet GID prefix was detected on more than one of these
ports. If these ports are connected to different physical IB
networks, this configuration will fail in Open MPI. This version of
Open MPI requires that every physically separate IB subnet that is
used between connected MPI processes must have different subnet ID
values.

Please see this FAQ entry for more details:

http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid

NOTE: You can turn off this warning by setting the MCA parameter
btl_openib_warn_default_gid_prefix to 0.
--------------------------------------------------------------------------
</pre>

Please run Open MPI's mpirun using the following command:
<pre>
$ mpirun --mca pml ucx --mca btl_openib_warn_default_gid_prefix 0 -np 2 ./mpi_cuda_app
</pre>
or disabling the (older) communication layer Bit-Transfer-Layer (short BTL) alltogether:
<pre>
$ mpirun --mca pml ucx --mca btl ^openib -np 2 ./mpi_cuda_app
</pre>

(Please note, that CUDA per v11.4 is only available with up to GCC-10)
 
 

==== LSDF Online Storage ====
On bwUniCluster 2.0 you can use for special cases the LSDF Online Storage on the HPC cluster nodes. Please request for this service separately ([https://www.lsdf.kit.edu/os/storagerequest/: LSDF Storage Request]).
To mount the LSDF Online Storage on the compute nodes during the job runtime the
the constraint flag "LSDF" has to be set.

a) add after the initial line of your script job.sh the line including the
information about the LSDF Online Storage usage: #SBATCH --constraint=LSDF
<pre>
#!/bin/bash
#SBATCH --ntasks=1
#SBATCH --time=120
#SBATCH --mem=200
#SBATCH --constraint=LSDF
</pre>

or b) execute:
<pre>
$ sbatch -p <queue> -n 1 -t 2:00:00 --mem 200 job.sh -C LSDF
</pre>
 
For the usage of the LSDF Online Storage
the following environment variables are available: $LSDF, $LSDFPROJECTS, $LSDFHOME.
 
 

====BeeOND (BeeGFS On-Demand)====

BeeOND instances are integrated into the prolog and epilog script of the cluster batch system Slurm. It can be used on the compute nodes during the job runtime with the constraint flag "BEEOND", "BEEOND_4MDS" or "BEEOND_MAXMDS" ([[BwUniCluster_2.0_Slurm_common_Features#sbatch_Command_Parameters|Slurm Command Parameters]])
* BEEOND: one metadata server is started on the first node
* BEEOND_4MDS: 4 metadata servers are started within your job. If you have less than 4 nodes less metadata servers are started.
* BEEOND_MAXMDS: on every node of your job a metadata server for the on_demand file system is started

As starting point we recommend using the "BEEOND" option. If you are unsure if this is sufficient for you feel free to contact the support team.
<source lang="bash">
#!/bin/bash
#SBATCH ...
#SBATCH --constraint=BEEOND # or BEEOND_4MDS or BEEOND_MAXMDS
</source>

After your job has started you can find the private on-demand file system in '''/mnt/odfs/${SLURM_JOB_ID}''' directory. The mountpoint comes with five pre-configured directories:
<source lang="bash">
# For small files (stripe count = 1)
/mnt/odfs/${SLURM_JOB_ID}/stripe_1
# Stripe count = 4
/mnt/odfs/${SLURM_JOB_ID}/stripe_default
# or
/mnt/odfs/${SLURM_JOB_ID}/stripe_4
# Stripe count = 8, 16 or 32, use this directories for medium sized and large files or when using MPI-IO
/mnt/odfs/${SLURM_JOB_ID}/stripe_8
/mnt/odfs/${SLURM_JOB_ID}/stripe_16
# or
/mnt/odfs/${SLURM_JOB_ID}/stripe_32
</source>

If you request less nodes than stripe count, the stripe count will be the number of nodes. For example, if you only request 8 nodes the directory stripe_16 has only a stripe count 8.

The capacity of the private file system depends on the number of nodes. For each node you get 250 Gbyte.

!!! Be careful when creating large files: use always the directory with the max stripe count for large files.
If you create large files use a higher stripe count. For example, if your largest file is 1.1 Tb, then you have to use a stripe count larger than 2, otherwise the used disk space is exceeded.

If you request 100 nodes for your job, the private file system is 100 * 250 Gbyte ~ 25 Tbyte (approx) capacity.

'''Possible optimization:'''

A possible optimization is that the private file system uses its own metadata server. With the constraint BEEOND the metadata server is started on the first node. Depending on your application, the metadata server could consume a decent amount of CPU power. In this case, adding a extra node to your job could improve the performance of the on-demand file system and the total runtime of your application. In order to use this option, start your application with the MPI option:
<pre>
mpirun -nolocal myapplication
</pre>
With the -nolocal option the node where mpirun is initiated is not used for your application. This node is fully available for the meta data server of your requested on-demand file system.

Example job script:
<source lang="bash">
#!/bin/bash
# Very simple example on how to use a private on-demand file system.
#SBATCH -N 10
#SBATCH --constraint=BEEOND

# Create a workspace.
ws_allocate myresults-${SLURM_JOB_ID} 90
RESULTDIR=$(ws_find myresults-${SLURM_JOB_ID})

# Set ENV variable to on-demand file system.
ODFSDIR=/mnt/odfs/${SLURM_JOB_ID}/stripe_16/

# Start application and write results to on-demand file system.
mpirun -nolocal myapplication -o $ODFSDIR/results

# Copy back data after your job application end.
rsync -av $ODFSDIR/results $RESULTDIR
</source>
 
 

== Start time of job or resources : squeue --start ==
The command can be used by any user to displays the estimated start time of a job based a number of analysis types based on historical usage, earliest available reservable resources, and priority based backlog. The command squeue is explained in detail on the webpage https://slurm.schedmd.com/squeue.html or via manpage (man squeue).
 
 
=== Access ===
By default, this command can be run by '''any user'''.
 
 

== List of your submitted jobs : squeue ==
Displays information about YOUR active, pending and/or recently completed jobs. The command displays all own active and pending jobs. The command squeue is explained in detail on the webpage https://slurm.schedmd.com/squeue.html or via manpage (man squeue).
 
 
=== Access ===
By default, this command can be run by any user.
 
 

=== Flags ===
{| width=750px class="wikitable"
|-
! Flag !! Description
|-
| -l, --long
| Report more of the available information for the selected jobs or job steps, subject to any constraints specified.
|}
 

=== Examples ===
''squeue'' example on bwUniCluster 2.0 (Only your own jobs are displayed!).
<pre>
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
18088744 single CPV.sbat ab1234 PD 0:00 1 (Priority)
18098414 multiple CPV.sbat ab1234 PD 0:00 2 (Priority)
18090089 multiple CPV.sbat ab1234 R 2:27 2 uc2n[127-128]
$ squeue -l
JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON)
18088654 single CPV.sbat ab1234 COMPLETI 4:29 2:00:00 1 uc2n374
18088785 single CPV.sbat ab1234 PENDING 0:00 2:00:00 1 (Priority)
18098414 multiple CPV.sbat ab1234 PENDING 0:00 2:00:00 2 (Priority)
18088683 single CPV.sbat ab1234 RUNNING 0:14 2:00:00 1 uc2n413
</pre>
* The output of ''squeue'' shows how many jobs of yours are running or pending and how many nodes are in use by your jobs.
 

== Shows free resources : sinfo_t_idle ==
The Slurm command sinfo is used to view partition and node information for a system running Slurm. It incorporates down time, reservations, and node state information in determining the available backfill window. The sinfo command can only be used by the administrator.
 
SCC has prepared a special script (sinfo_t_idle) to find out how many processors are available for immediate use on the system. It is anticipated that users will use this information to submit jobs that meet these criteria and thus obtain quick job turnaround times.
 
 
=== Access ===
By default, this command can be used by any user or administrator.
 
 
=== Example ===
* The following command displays what resources are available for immediate use for the whole partition.
<pre>$ sinfo_t_idle
Partition dev_multiple : 8 nodes idle
Partition multiple : 332 nodes idle
Partition dev_single : 4 nodes idle
Partition single : 76 nodes idle
Partition long : 80 nodes idle
Partition fat : 5 nodes idle
Partition dev_special : 342 nodes idle
Partition special : 342 nodes idle
Partition dev_multiple_e: 7 nodes idle
Partition multiple_e : 335 nodes idle
Partition gpu_4 : 12 nodes idle
Partition gpu_8 : 6 nodes idle
</pre>
* For the above example jobs in all partitions can be run immediately.
 

== Detailed job information : scontrol show job ==
scontrol show job displays detailed job state information and diagnostic output for all or a specified job of yours. Detailed information is available for active, pending and recently completed jobs. The command scontrol is explained in detail on the webpage https://slurm.schedmd.com/scontrol.html or via manpage (man scontrol).
 
Display the state of all your jobs in normal mode: scontrol show job
 
Display the state of a job with <jobid> in normal mode: scontrol show job <jobid>
 
 
=== Access ===
* End users can use scontrol show job to view the status of their '''own jobs''' only.
 

=== Arguments ===
{| width=750px class="wikitable"
|-
! Option !! Default !! Description !! Example
|- style="vertical-align:top;"
|- style="width:12%;"
| -d
| (n/a)
| Detailed mode
| Example: Display the state with jobid 18089884 in detailed mode. <pre>scontrol -d show job 18089884</pre>
|}
 
 

=== Scontrol show job Example ===
Here is an example from bwUniCluster 2.0.
<pre>
squeue # show my own jobs (here the userid is replaced!)
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
18089884 multiple CPV.sbat bq0742 R 33:44 2 uc2n[165-166]

$
$ # now, see what's up with my pending job with jobid 18089884
$
$ scontrol show job 18089884

JobId=18089884 JobName=CPV.sbatch
UserId=bq0742(8946) GroupId=scc(12345) MCS_label=N/A
Priority=3 Nice=0 Account=kit QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:35:06 TimeLimit=02:00:00 TimeMin=N/A
SubmitTime=2020-03-16T14:14:54 EligibleTime=2020-03-16T14:14:54
AccrueTime=2020-03-16T14:14:54
StartTime=2020-03-16T15:12:51 EndTime=2020-03-16T17:12:51 Deadline=N/A
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2020-03-16T15:12:51
Partition=multiple AllocNode:Sid=uc2n995:5064
ReqNodeList=(null) ExcNodeList=(null)
NodeList=uc2n[165-166]
BatchHost=uc2n165
NumNodes=2 NumCPUs=160 NumTasks=80 CPUs/Task=1 ReqB:S:C:T=0:0:*:1
TRES=cpu=160,mem=96320M,node=2,billing=160
Socks/Node=* NtasksPerN:B:S:C=40:0:*:1 CoreSpec=*
MinCPUsNode=40 MinMemoryCPU=1204M MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
Command=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/CPV.sbatch
WorkDir=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin
StdErr=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/slurm-18089884.out
StdIn=/dev/null
StdOut=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/slurm-18089884.out
Power=
MailUser=(null) MailType=NONE
</pre>
 
You can use standard Linux pipe commands to filter the very detailed scontrol show job output.
* In which state the job is?
<pre>$ scontrol show job 18089884 | grep -i State
JobState=COMPLETED Reason=None Dependency=(null)
</pre>
 

== Cancel Slurm Jobs ==
The scancel command is used to cancel jobs. The command scancel is explained in detail on the webpage https://slurm.schedmd.com/scancel.html or via manpage (man scancel).
 
 
=== Canceling own jobs : scancel ===
scancel is used to signal or cancel jobs, job arrays or job steps. The command is:
<pre>
$ scancel [-i] <job-id>
$ scancel -t <job_state_name>
</pre>

 
{| width=750px class="wikitable"
! Flag !! Default !! Description !! Example
|- style="vertical-align:top;"
| -i, --interactive
| (n/a)
| Interactive mode.
| Cancel the job 987654 interactively. <pre> scancel -i 987654 </pre>
|-
| -t, --state
| (n/a)
| Restrict the scancel operation to jobs in a certain state. "job_state_name" may have a value of either "PENDING", "RUNNING" or "SUSPENDED".
| Cancel all jobs in state "PENDING". <pre> scancel -t "PENDING" </pre>
|}
 

= Resource Managers =
=== Batch Job (Slurm) Variables ===
The following environment variables of Slurm are added to your environment once your job has started
(only an excerpt of the most important ones).
{| width=750px class="wikitable"
! Environment !! Brief explanation
|-
| SLURM_JOB_CPUS_PER_NODE
| Number of processes per node dedicated to the job
|-
| SLURM_JOB_NODELIST
| List of nodes dedicated to the job
|-
| SLURM_JOB_NUM_NODES
| Number of nodes dedicated to the job
|-
| SLURM_MEM_PER_NODE
| Memory per node dedicated to the job
|-
| SLURM_NPROCS
| Total number of processes dedicated to the job
|-
| SLURM_CLUSTER_NAME
| Name of the cluster executing the job
|-
| SLURM_CPUS_PER_TASK
| Number of CPUs requested per task
|-
| SLURM_JOB_ACCOUNT
| Account name
|-
| SLURM_JOB_ID
| Job ID
|-
| SLURM_JOB_NAME
| Job Name
|-
| SLURM_JOB_PARTITION
| Partition/queue running the job
|-
| SLURM_JOB_UID
| User ID of the job's owner
|-
| SLURM_SUBMIT_DIR
| Job submit folder. The directory from which msub was invoked.
|-
| SLURM_JOB_USER
| User name of the job's owner
|-
| SLURM_RESTART_COUNT
| Number of times job has restarted
|-
| SLURM_PROCID
| Task ID (MPI rank)
|-
| SLURM_NTASKS
| The total number of tasks available for the job
|-
| SLURM_STEP_ID
| Job step ID
|-
| SLURM_STEP_NUM_TASKS
| Task count (number of PI ranks)
|-
| SLURM_JOB_CONSTRAINT
| Job constraints
|}
See also:
* [https://slurm.schedmd.com/sbatch.html#lbAI Slurm input and output environment variables]
 

=== Job Exit Codes ===
A job's exit code (also known as exit status, return code and completion code) is captured by SLURM and saved as part of the job record.
 
Any non-zero exit code will be assumed to be a job failure and will result in a Job State of FAILED with a reason of "NonZeroExitCode".
 
The exit code is an 8 bit unsigned number ranging between 0 and 255. While it is possible for a job to return a negative exit code, SLURM will display it as an unsigned value in the 0 - 255 range.
 
 
==== Displaying Exit Codes and Signals ====
SLURM displays a job's exit code in the output of the '''scontrol show job''' and the sview utility.
 
When a signal was responsible for a job or step's termination, the signal number will be displayed after the exit code, delineated by a colon(:).
 
 
==== Submitting Termination Signal ====
Here is an example, how to 'save' a Slurm termination signal in a typical jobscript.
<source lang="bash">
[...]
exit_code=$?
mpirun -np <#cores> <EXE_BIN_DIR>/<executable> ... (options) 2>&1
[ "$exit_code" -eq 0 ] && echo "all clean..." || \
echo "Executable <EXE_BIN_DIR>/<executable> finished with exit code ${$exit_code}"
[...]
</source>
* Do not use ''''time'''' mpirun! The exit code will be the one submitted by the first (time) program.
* You do not need an '''exit $exit_code''' in the scripts.
 
 
----
[[Category:bwUniCluster 2.0|bwUniCluster 2.0]]
[[#top|Back to top]]

JUSTUS2/Software

2022-10-10T09:33:21Z

R Barthel: /* Documentation in the Wiki */

== Environment Modules ==
Most software is provided as Modules.

Required reading to use: [[Environment Modules]]

== Available Software ==
Visit [https://www.bwhpc.de/software.php https://www.bwhpc.de/software.php], select <code>Cluster → bwForCluster JUSTUS2</code>

On cluster: <code>module avail</code>
== Software in Containers ==

Instructions for loading software in containers: [[Singularity]]

== Documentation ==
Documentation by the cluster operators:
{| style="width:100%;"
|- style="background:#eeeeee;" |
| <code>module help</code> || See section: [[Environment_Modules#module_help]]
|- style="background:#dddddd; " |
| examples in <code>$SOFTNAME_EXA_DIR</code> || See section: [[Environment_Modules#Software_job_examples]]
|- style="background:#eeeeee; " |
| this wiki || See below
|}

== Documentation in the Wiki ==
Modules with additional documentation here in the wiki:



* [[ADF]]

* [[JUSTUS2/Software/Dalton|Dalton]]

* [[Gaussian]]

* [[Gaussview]]

* [[JUSTUS2/Software/Molden|Molden]]

* [[JUSTUS2/Software/Schrodinger|Schrodinger]]

* [[JUSTUS2/Software/Turbomole|Turbomole]]

* [[JUSTUS2/Software/VASP|VASP]]

BwUniCluster2.0/Software/Ansys

2022-10-10T09:02:50Z

R Barthel:

{{Softwarepage|cae/ansys}}

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| cae/ansys
|-
| License
| Academic. See: [http://www.ansys.com/Academic/educator-tools/Licensing+&+Terms+of+Use Licensing and Terms-of-Use].
|-
| Citing
| [http://www.ansys.com/academic/educator-tools/ Citations]
|-
| Links
| [http://www.ansys.com/ Ansys Homepage] | [http://www.ansys.com/Academic/educator-tools/Support+Resources Support and Resources]
|-
| Graphical Interface
| Yes
|}

= Description =
'''ANSYS''' is a general purpose software to simulate interactions of all disciplines of physics, structural, fluid dynamics, heat transfer, electromagnetic etc. For more information about ANSYS products please visit [http://www.ansys.com/Industries/Academic/ http://www.ansys.com/Industries/Academic/]
 
 

= Versions and Availability =

</pre>
The cae/ansys modules are using the KIT license server and are reserved for members of the KIT only.
 

= Usage =
== ANSYS Fluent batch jobs ==
see http://www.scc.kit.edu/produkte/6724.php > "Kurzanleitung" > "#FLUENT-Aufruf_auf_den_Linux-Clustern_des_SCC"

== ANSYS CFX batch jobs ==
see http://www.scc.kit.edu/produkte/3852.php > "Kurzanleitung" > "#CFX_auf_den_Linux-Clustern_des_SCC"

----
[[Category:Engineering software]][[Category:bwUniCluster_2.0]]

Category:BwUniCluster 2.0

2022-10-08T01:02:22Z

R Barthel:

This page lists all articles that are assigned to the high-performance computer '''[[BwUniCluster2.0|bwUniCluster 2.0]]'''.


[[File:BwUniCluster_2.0_Feb2020_1024x423.jpg|right|frameless|thumb|alt=bwUniCluster2.0 |upright=1| bwUniCluster 2.0 ]]

* '''Overview of all its topics''': → [[bwUniCluster2.0]]
* On 17.03.2020, the Steinbuch Centre for Computing (SCC) at Karlsruhe Institute of Technology (KIT) commissioned a new parallel computer system called "bwUniCluster 2.0+GFB-HPC" as a state service within the bwHPC framework. The bwUniCluster 2.0 replaces the predecessor system [[:Category:BwUniCluster | bwUniCluster]] and also includes the additional compute nodes which were procured as an extension to the bwUniCluster in November 2016.
* The modern bwUniCluster 2.0 system consists of more than 840 SMP nodes with 64-bit Intel Xeon processors. It provides the universities of the state of Baden-Württemberg with general compute resources and can be used free of charge by the staff of all universities in Baden-Württemberg. Users who currently have access to bwUniCluster will automatically also have access to bwUniCluster 2.0. There is no need to apply for new entitlements or to re-register.







<!--
* [[FAQ - bwUniCluster_broadwell_partition|FAQ - bwUniCluster 2.0 Broadwell partition]]
|-
|{{Blue}}| Miscellaneous
|-
|
* [[bwUniCluster_Acknowledgement|Acknowledgement]] of work performed on bwUniCluster (2.0)
* [[BwUniCluster_2.0_File_System_Migration_Guide|File system migration guide]] and [[BwUniCluster_2.0_Batch_System_Migration_Guide|Batch system migration guide]] for users migrating from the former bwUniCluster 1
|}
|}
->

 
-----
 
 
[[Category:bwHPC_infrastructure]][[Category:bwHPC_Cluster]][[Category:bwCluster]]

BwUniCluster3.0/Software/Start vnc desktop

2022-10-08T00:58:45Z

R Barthel: R Barthel moved page Start vnc desktop to BwUniCluster2.0/Software/Start vnc desktop

The Linux 3D graphics stack is based on ''X11'' and ''OpenGL''. This has some
drawbacks in conjunction with remote visualization:

* Rendering takes place on the client, not the cluster
* Whole 3D model must be transferred via network to the client
* Some OpenGL extensions are not supported when using indirect / client side rendering instead of direct / hardware based rendering
* Many round trips in the X11 protocol negatively influence interactivity
* X11 is not available on non-Linux platforms
* Compatibility problems between client and cluster can occur

To avoid these drawbacks, <code>start_vnc_desktop</code> is provided.
It combines the three open source products [http://www.turbovnc.org/ TurboVNC], [http://www.virtualgl.org/ VirtualGL] and [http://openswr.org/ OpenSWR].

''Virtual Network Computing (VNC)'' is a graphical desktop sharing system.
VNC is platform-independent - there are clients and servers for many
GUI-based operating systems. The VNC server is the program on the
machine that shares its screen. The VNC client (or viewer) is the
program that watches, controls, and interacts with the server. For more
details see: [https://en.wikipedia.org/wiki/VNC Wikipedia]

''VirtualGL'' redirects the 3D rendering commands from Linux OpenGL
applications to 3D accelerator hardware in the cluster. For more details
see: [https://en.wikipedia.org/wiki/VirtualGL Wikipedia]

When no 3D accelerator hardware is available ''OpenSWR'', a high
performance, highly scalable software rasterizer for OpenGL can carry
out the rendering task. For more details see: [http://openswr.org OpenSWR]

This script takes a two step approach to start a VNC server in the
cluster environment:

In the first step the batch system is used to allocate resources where a
VNC server can be started.

In the second step the VNC server is launched on the resources granted
by the batch system. When VNC server is successfully started all
required login credentials and connection parameters will be reported.
To connect to this VNC server a VNC client installation on the local
desktop is required.

= Script usage =

* After login the script can simply be called from the command line:<pre>start_vnc_desktop</pre>
* To get help on the available options use:<pre>start_vnc_desktop --help</pre>
* Hardware rendering is currently only available on FH2 and bwUniCluster, it can be requested with:<pre>start_vnc_desktop --hw-rendering</pre>
* Software rendering is available on all clusters, it can be requested with: <pre>start_vnc_desktop --sw-rendering</pre>
* There is only a limited number of nodes with hardware rendering support, software rendering runs on all nodes.
* For large 3D data sets the software renderer may be faster.
* If neither <code>--hw-rendering</code> nor <code>--sw-rendering</code> is selected no 3D rendering support is available.

= VNC client =

In general every VNC client can be used to connect to the VNC server.
However for best performance and compatibility the use of the
[http://www.turbovnc.org/ TurboVNC] client is recommended.
Below you find the necessary steps for different client operation systems.

; Debian, Ubuntu:
* Download: [https://sourceforge.net/projects/turbovnc/files Download Site] -> latest version -> turbovnc_<VERSION>_amd64.deb
* Install: <pre> sudo apt-get install ./turbovnc_<VERSION>_amd64.deb</pre>
* Execute: <pre>/opt/TurboVNC/bin/vncviewer</pre>

; Red Hat Enterprise Linux, Fedora:
* Download: [https://sourceforge.net/projects/turbovnc/files Download Site] -> latest version -> turbovnc-<VERSION>.x86_64.rpm
* Install: <pre>sudo yum install ./turbovnc-<VERSION>.x86_64.rpm</pre>
* Execute: <pre>/opt/TurboVNC/bin/vncviewer</pre>

; SUSE Linux Enterprise, openSUSE:
* Download [https://sourceforge.net/projects/turbovnc/files Download Site] -> latest version -> turbovnc-<VERSION>.x86_64.rpm
* Install: <pre>sudo zypper install ./turbovnc-<VERSION>.x86_64.rpm</pre>
* Execute: <pre>/opt/TurboVNC/bin/vncviewer</pre>

; ArchLinux:
* Download: Can be installed from the AUR
* Install: <pre>pacaur -S turbovnc</pre>
* Execute: <pre>vncviewer</pre>

; Windows:
* Download: [https://sourceforge.net/projects/turbovnc/files Download Site] -> latest version -> TurboVNC64-<VERSION>.exe for 64-bit, TurboVNC-<VERSION>.exe for 32-bit
* Install: Double click on TurboVNC64-<VERSION>.exe / TurboVNC-<VERSION>.exe. Install in default directory (or chose a different one, if preferred)
* Execute: Java TurboVNCviewer (vncviewer-javaw.bat in installation directory)

[[Category:Visualization]]
[[Category:bwUniCluster_2.0]]

Category:BwUniCluster 2.0

2022-10-08T00:58:02Z

R Barthel:

This page lists all articles that are assigned to high-performance computer '''bwUniCluster 2.0'''.


[[File:BwUniCluster_2.0_Feb2020_1024x423.jpg|right|frameless|thumb|alt=bwUniCluster2.0 |upright=1| bwUniCluster 2.0 ]]

* '''Overview of all its topics''': → [[bwUniCluster2.0]]
* On 17.03.2020, the Steinbuch Centre for Computing (SCC) at Karlsruhe Institute of Technology (KIT) commissioned a new parallel computer system called "bwUniCluster 2.0+GFB-HPC" as a state service within the bwHPC framework. The bwUniCluster 2.0 replaces the predecessor system [[:Category:BwUniCluster | bwUniCluster]] and also includes the additional compute nodes which were procured as an extension to the bwUniCluster in November 2016.
* The modern bwUniCluster 2.0 system consists of more than 840 SMP nodes with 64-bit Intel Xeon processors. It provides the universities of the state of Baden-Württemberg with general compute resources and can be used free of charge by the staff of all universities in Baden-Württemberg. Users who currently have access to bwUniCluster will automatically also have access to bwUniCluster 2.0. There is no need to apply for new entitlements or to re-register.







<!--
* [[FAQ - bwUniCluster_broadwell_partition|FAQ - bwUniCluster 2.0 Broadwell partition]]
|-
|{{Blue}}| Miscellaneous
|-
|
* [[bwUniCluster_Acknowledgement|Acknowledgement]] of work performed on bwUniCluster (2.0)
* [[BwUniCluster_2.0_File_System_Migration_Guide|File system migration guide]] and [[BwUniCluster_2.0_Batch_System_Migration_Guide|Batch system migration guide]] for users migrating from the former bwUniCluster 1
|}
|}
->

 
-----
 
 
[[Category:bwHPC_infrastructure]][[Category:bwHPC_Cluster]][[Category:bwCluster]]

JUSTUS2/Software/Molden

2022-10-08T00:47:42Z

R Barthel:

{{Softwarepage|chem/molden}}

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| chem/molden
|-
| License
| Free. See: [http://www.cmbi.ru.nl/molden/CopyRight.html Copyright CMBI]
|-
| Citing
| [http://www.cmbi.ru.nl/molden/ref.html Molden-Reference]
|-
| Links
| [http://www.cmbi.ru.nl/molden/ Homepage] | [http://www.cmbi.ru.nl/molden Documentation]
|-
| Graphical Interface
| [[#Graphical User Interface (GUI)|Yes]]
|}

= Description =
'''Molden''' is a software package for displaying molecular coordinates, molecular orbitals, electron densities from various quantum chemical software packages and supports contour and 3-D grid plots. Moreover, '''Molden''' supports many output formats including postscript, povray, OpenGL and hpgl.
 

= Usage =
Once the module ''Molden'' is loaded, the binaries '''molden''' (Command-Line) and '''gmolden''' (GUI) can be directly executed:
== Command-Line ==
<pre>
$ molden
</pre>
== Graphical User Interface (GUI) ==
<pre>
$ gmolden &
</pre>
[[File:molden_gui.jpg]]

----
[[Category:Chemistry software]][[Category:bwUniCluster_2.0|Molden]][[Category:bwForCluster_Chemistry]]

JUSTUS2/Software/Molden

2022-10-08T00:45:56Z

R Barthel:

BwUniCluster3.0/Software/Start vnc desktop

2022-10-08T00:40:36Z

R Barthel:

Talk:Ansys

2022-10-08T00:38:03Z

R Barthel: R Barthel moved page Talk:Ansys to Talk:BwUniCluster2.0/Software/Ansys

#REDIRECT [[Talk:BwUniCluster2.0/Software/Ansys]]

Talk:BwUniCluster2.0/Software/Ansys

2022-10-08T00:38:03Z

R Barthel: R Barthel moved page Talk:Ansys to Talk:BwUniCluster2.0/Software/Ansys

Ansys

2022-10-08T00:38:03Z

R Barthel: R Barthel moved page Ansys to BwUniCluster2.0/Software/Ansys

#REDIRECT [[BwUniCluster2.0/Software/Ansys]]

BwUniCluster2.0/Software/Ansys

2022-10-08T00:38:03Z

R Barthel: R Barthel moved page Ansys to BwUniCluster2.0/Software/Ansys

BwUniCluster2.0/Software

2022-10-08T00:33:12Z

R Barthel:

== Environment Modules ==
Most software is provided as Modules.

Required reading to use: [[BwUniCluster2.0/Software Modules|Software Environment Modules]]

== Available Software ==
Visit [https://www.bwhpc.de/software.php https://www.bwhpc.de/software.php], select <code>Cluster → bwUniCluster 2.0</code>

On cluster: <code>module avail</code>
== Software in Containers ==

Instructions for loading software in containers: [[BwUniCluster2.0/Containers]]

== Documentation ==
Documentation for environment modules is available on the cluster:
* with command <code>module help</code>
* examples in <code>$SOFTNAME_EXA_DIR</code>

== Documentation in the Wiki ==
For some environment modules additional documentation is provided here.

{{Special:PrefixIndex/BwUniCluster2.0/Software/|stripprefix=yes}}

BwUniCluster2.0/Software/Molden

2022-10-08T00:32:57Z

R Barthel: Redirected page to JUSTUS2/Software/Molden

#REDIRECT [[JUSTUS2/Software/Molden]]

Molden

2022-10-08T00:32:07Z

R Barthel: R Barthel moved page Molden to JUSTUS2/Software/Molden

#REDIRECT [[JUSTUS2/Software/Molden]]

JUSTUS2/Software/Molden

2022-10-08T00:32:07Z

R Barthel: R Barthel moved page Molden to JUSTUS2/Software/Molden

BwUniCluster2.0/Software

2022-10-08T00:24:33Z

R Barthel:

BwUniCluster2.0/Software/Turbomole

2022-10-08T00:24:02Z

R Barthel: Redirected page to JUSTUS2/Software/Turbomole

#REDIRECT [[JUSTUS2/Software/Turbomole]]

BwUniCluster2.0/Software

2022-10-08T00:22:03Z

R Barthel: Created page with "== Environment Modules == Most software is provided as Modules. Required reading to use: Software Environment Modules == Available Softw..."

BwUniCluster2.0/Software Modules

2022-10-08T00:12:22Z

R Barthel:

<div id="top"></div>
 
= Introduction =
'''Software (Environment) Modules''', or short '''Modules''' are the means by which most of the installed scientific software is provided on bwUniCluster 2.0.
 
The use of different compilers, libraries and software packages requires users to set up a specific session environment suited for the program they want to run. bwUniCluster 2.0 provides users with the possibility to load and unload complete environments for compilers, libraries and software packages by a single command.
 
 

= Description =
The Environment ''Modules'' package enables dynamic modification of your environment by the
use of so-called ''modulefiles''. A ''modulefile'' contains information to configure the shell
for a program/software . Typically, a modulefile contains instructions that alter or set shell
environment variables, such as PATH and MANPATH, to enable access to various installed
software.
 
One of the key features of using the Environment ''Modules'' software is to allow multiple versions of the same software to be used in your environment in a controlled manner.
For example, two different versions of the Intel C compiler can be installed on the system at the same time - the version used is based upon which Intel C compiler modulefile is loaded.
 
The software stack of bwUniCluster 2.0 provides a number of modulefiles. You can also
create your own modulefiles. ''Modulefiles'' may be shared by many users on a system, and
users may have their own collection of modulefiles to supplement or replace the shared
modulefiles.
 
A modulefile does not provide configuration of your environment until it is explicitly loaded,
i.e., the specific modulefile for a software product or application must be loaded in your environment before the configuration information in the modulefile is effective.
 
If you want to see which modules are loaded you must execute
''''module list''''.
 
<pre>
$ module list
Currently Loaded Modules:
1) compiler/intel/19.1 2) mpi/impi/2019 3) numlib/mkl/2019
</pre>
 

= Usage =
Lmod on bwUniCluster 2.0: A New Environment Module System from http://lmod.readthedocs.org/en/latest/ is installed.
== Documentation ==
Execute ''''module help'''' or ''''man module'''' for help on how to use ''Modules'' software.
<pre>
$ module help
Usage: module [options] sub-command [args ...]

Options:
-h -? -H --help This help message
-s availStyle --style=availStyle Site controlled avail style: system (default: system)
--regression_testing Lmod regression testing
-D Program tracing written to stderr
--debug=dbglvl Program tracing written to stderr (where dbglvl is a number 1,2,3)
--pin_versions=pinVersions When doing a restore use specified version, do not follow defaults
-d --default List default modules only when used with avail
-q --quiet Do not print out warnings
--expert Expert mode
-t --terse Write out in machine readable format for commands: list, avail, spider, savelist
--initial_load loading Lmod for first time in a user shell
--latest Load latest (ignore default)
--ignore_cache Treat the cache file(s) as out-of-date
--novice Turn off expert and quiet flag
--raw Print modulefile in raw output when used with show
-w twidth --width=twidth Use this as max term width
-v --version Print version info and quit
-r --regexp use regular expression match
--gitversion Dump git version in a machine readable way and quit
--dumpversion Dump version in a machine readable way and quit
--check_syntax --checkSyntax Checking module command syntax: do not load
--config Report Lmod Configuration
--config_json Report Lmod Configuration in json format
--mt Report Module Table State
--timer report run times
--force force removal of a sticky module or save an empty collection
--redirect Send the output of list, avail, spider to stdout (not stderr)
--no_redirect Force output of list, avail and spider to stderr
--show_hidden Avail and spider will report hidden modules
--spider_timeout=timeout a timeout for spider
-T --trace

module [options] sub-command [args ...]

Help sub-commands:
------------------
help prints this message
help module [...] print help message from module(s)

Loading/Unloading sub-commands:
-------------------------------
load | add module [...] load module(s)
try-load | try-add module [...] Add module(s), do not complain if not found
del | unload module [...] Remove module(s), do not complain if not found
swap | sw | switch m1 m2 unload m1 and load m2
purge unload all modules
refresh reload aliases from current list of modules.
update reload all currently loaded modules.

Listing / Searching sub-commands:
---------------------------------
list List loaded modules
list s1 s2 ... List loaded modules that match the pattern
avail | av List available modules
avail | av string List available modules that contain "string".
spider List all possible modules
spider module List all possible version of that module file
spider string List all module that contain the "string".
spider name/version Detailed information about that version of the module.
whatis module Print whatis information about module
keyword | key string Search all name and whatis that contain "string".

Searching with Lmod:
--------------------
All searching (spider, list, avail, keyword) support regular expressions:

-r spider '^p' Finds all the modules that start with `p' or `P'
-r spider mpi Finds all modules that have "mpi" in their name.
-r spider 'mpi$ Finds all modules that end with "mpi" in their name.

Handling a collection of modules:
--------------------------------
save | s Save the current list of modules to a user defined "default" collection.
save | s name Save the current list of modules to "name" collection.
reset The same as "restore system"
restore | r Restore modules from the user's "default" or system default.
restore | r name Restore modules from "name" collection.
restore system Restore module state to system defaults.
savelist List of saved collections.
describe | mcc name Describe the contents of a module collection.
disable name Disable (i.e. remove) a collection.

Deprecated commands:
--------------------
getdefault [name] load name collection of modules or user's "default" if no name given.
===> Use "restore" instead <====
setdefault [name] Save current list of modules to name if given, otherwise save as the default list for you the user.
===> Use "save" instead. <====

Miscellaneous sub-commands:
---------------------------
is-loaded modulefile return a true status if module is loaded
is-avail modulefile return a true status if module can be loaded
show modulefile show the commands in the module file.
use [-a] path Prepend or Append path to MODULEPATH.
unuse path remove path from MODULEPATH.
tablelist output list of active modules as a lua table.

Important Environment Variables:
--------------------------------
LMOD_COLORIZE If defined to be "YES" then Lmod prints properties and warning in color.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Lmod Web Sites

Documentation: http://lmod.readthedocs.org
Github: https://github.com/TACC/Lmod
Sourceforge: https://lmod.sf.net
TACC Homepage: https://www.tacc.utexas.edu/research-development/tacc-projects/lmod

To report a bug please read http://lmod.readthedocs.io/en/latest/075_bug_reporting.html
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Modules based on Lua: Version 8.2 (8.2-1-g9c98036c) 2019-10-30 11:17 -05:00
by Robert McLay mclay@tacc.utexas.edu

</pre>
For help on particular version of ''Module'', e.g. Intel default compiler version, execute
''''module help compiler/intel''''.
<pre>
$ module help compiler/intel
---------------------- Module Specific Help for "compiler/intel/19.1" ----------------------
Intel(R) Compilers 19.1 for Linux*
For details see: https://software.intel.com/en-us/intel-compilers
In case of problems, please contact: Hartmut Häfner <hartmut.haefner@kit.edu>
SCC support end: 2022-12-31
</pre>
 
=== Online Documentation ===
[http://lmod.readthedocs.org Lmod: A New Environment Module System]
 
 

== Display all available Modules ==
Available ''Module'' are modulefiles that can be loaded by the user. A ''Module'' must be loaded before it provides changes to your environment, as described in the introduction to this section. You can display all available ''Modules'' on the system by executing:
<pre>
$ module avail
</pre>
The short form the command is:
<pre>
$ module av
</pre>
Available ''Modules'' can be also displayed in different modes, such as
* each ''Module'' per one line
<pre>
$ module -t avail
</pre>
Some modules may not be available right now, because their requirements are not met. To get a complete list of all possible modules use the [[#Display all possible Modules|module spider command]].
 
 

== Module categories, versions and defaults ==
The bwHPC clusters traditionally provide a large variety of software and software versions. Therefore ''Module'' are divided in category folders containing subfolders of modulefiles again containing modulefile versions, and must be addressed as follows:
category/softwarename/version
For instance all versions of the Intel compiler belong to the category of compilers, thus the corresponding modulefiles are placed under the category ''compiler'' and ''intel''.
 
In case of multiple software versions, one version will be always defined as the '''default'''
version. The ''Module'' of the default can be addressed by simply omitting the version number:
category/softwarename
 

== Finding software Modules ==
Currently all bwUniCluster 2.0 software packages are assigned to the following ''Module'' categories (???):


* bio

* cae

* chem

* compiler

* devel

* lib

* math
* mpi

* numlib

* phys

* system

* toolkit

* vis
You can selectively list software in one of those categories using, e.g. for the category "compiler"
<pre>
$ module avail compiler/
</pre>
Searches are looking for a substring starting at the begin of the name, so this would list all software in categories starting with a "c"
<pre>
$ module avail c
</pre>
while this would find nothing
<pre>
$ module avail hem
</pre>
 

== Loading Modules ==
You can load a ''Module'' software in to your environment to enable easier access to software that
you want to use by executing:
<pre>
$ module load category/softwarename/version
</pre>
or
<pre>
$ module add category/softwarename/version
</pre>
Loading a ''Module'' in this manner affects ONLY your environment for the current session.
 
 
=== Loading conflicts ===
You can not load different versions of the same software at the same time! Loading the Intel compiler in version X while Intel compiler in version Y is loaded leads to an automatic unloading of Intel compiler in version Y.
 
 

=== Showing the changes introduced by a Module ===
Loading a ''Module'' will change the environment of the current shell session. For instance the $PATH variable will be expanded by the software's binary directory. Other ''Module'' variables may even change the behavior of the current shell session or the software program(s) in a more drastic way.
 
 
All the changes to the current shell session to be invoked by loading the ''Module'' can be reviewed using
 
''''module show category/softwarename/version''''.
 
 
'''Example (Intel compiler)'''
<pre>
$ module show compiler/intel
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
/opt/bwhpc/common/modulefiles/Core/compiler/intel/19.1.lua:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
setenv("INTEL_LICENSE_FILE","28518@scclic1.scc.kit.edu")
setenv("AR","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64/xiar")
setenv("CC","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64/icc")
setenv("CXX","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64/icpc")
setenv("F77","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64/ifort")
setenv("FC","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64/ifort")
setenv("CFLAGS","-O2 -xCORE-AVX2")
setenv("CXXFLAGS","-O2 -xCORE-AVX2")
setenv("FFLAGS","-O2 -xCORE-AVX2")
setenv("FCFLAGS","-O2 -xCORE-AVX2")
setenv("INTEL_VERSION","19.1.0.166")
setenv("INTEL_HOME","/opt/intel/compilers_and_libraries_2020/linux")
setenv("INTEL_BIN_DIR","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64")
setenv("INTEL_LIB_DIR","/opt/intel/compilers_and_libraries_2020/linux/lib/intel64")
setenv("INTEL_INC_DIR","/opt/intel/compilers_and_libraries_2020/linux/include")
setenv("INTEL_MAN_DIR","/opt/intel/compilers_and_libraries_2020/linux/man/common")
setenv("INTEL_DOC_DIR","/opt/intel/compilers_and_libraries_2020/linux/documentation/en")
setenv("GDB_VERSION","19.1.0.166")
setenv("GDB_HOME","/opt/intel/debugger_2020/gdb/intel64")
setenv("GDB_BIN_DIR","/opt/intel/debugger_2020/gdb/intel64/bin")
setenv("GDB_LIB_DIR","/opt/intel/debugger_2020/libipt/intel64/lib")
setenv("GDB_INC_DIR","/opt/intel/debugger_2020/gdb/intel64/include")
setenv("GDB_INF_DIR","/opt/intel/documentation_2020/en/debugger/gdb-ia/info")
setenv("GDB_MAN_DIR","/opt/intel/documentation_2020/en/debugger/gdb-ia/man")
setenv("KMP_AFFINITY","noverbose,granularity=core,respect,warnings,compact,1")
prepend_path("PATH","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64")
prepend_path("MANPATH","/opt/intel/compilers_and_libraries_2020/linux/man/common")
prepend_path("LD_LIBRARY_PATH","/opt/intel/compilers_and_libraries_2020/linux/lib/intel64")
whatis("Sets up Intel C/C++ and Fortran compiler version 19.1 (Intel(R) Compilers 19.1 for Linux*) - supported by SCC till 2022-12-31!")
help([[Intel(R) Compilers 19.1 for Linux*
For details see: https://software.intel.com/en-us/intel-compilers
In case of problems, please contact: Hartmut Häfner <hartmut.haefner@kit.edu>
SCC support end: 2022-12-31]])
prepend_path("MODULEPATH","/software/bwhpc/common/modulefiles/Compiler/intel/19.1")
family("compiler")
</pre>
 
 

=== Modules depending on Modules ===
Some program ''Modules'' depend on libraries to be loaded to the user environment. Therefore the
corresponding ''Modules'' of the software must be loaded together with the ''Modules'' of
the libraries.
 
By default such software ''Modules'' try to load required ''Modules'' and corresponding versions automatically.
 
 
 

== Unloading Modules ==
To unload or remove a software ''Module'' execute:
<pre>
$ module unload category/softwarename/version
</pre>
or
<pre>
$ module remove category/softwarename/version
</pre>
 

=== Unloading all loaded modules ===
==== Purge ====
Unloading a ''Module'' that has been loaded by default makes it inactive for the current session only - it will be reloaded the next time you log in.
 
In order to remove all previously loaded software modules from your environment issue the command 'module purge'.
 
Example
<pre>
$ module list
Currently Loaded Modules:
1) compiler/intel/19.1 2) mpi/impi/2019 3) numlib/mkl/2019
$
$ module purge
$ module list
No modules loaded
$
</pre>
Beware!
 
'module purge' is working without any further inquiry.
 
 

== Display your loaded Modules ==
All ''Modules'' that are currently loaded for you can be displayed by the
command ''''module list''''. [[#Purge|See example above]].
 
Note: You only have to load further ''Modules'', if you want to use additional software
packages or to change the version of an already loaded software.
 
 

== Display all possible Modules ==
Modulefiles can be searched by the user. You can dipslay all possible modules by executing:
<pre>
$ module spider

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The following is a list of the modules and extensions currently available:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
cae/abaqus: cae/abaqus/2018, cae/abaqus/2019

cae/adina: cae/adina/9.1.2

cae/ansys: cae/ansys/19.2, cae/ansys/2019R3, cae/ansys/2020R1

cae/comsol: cae/comsol/5.4, cae/comsol/5.5

cae/cst: cae/cst/2018

cae/lsdyna: cae/lsdyna/901

cae/openfoam: cae/openfoam/v1912, cae/openfoam/2.4.x, cae/openfoam/6, cae/openfoam/7

cae/paraview: cae/paraview/5.8

cae/starccm+: cae/starccm+/14.02.010, cae/starccm+/2019.2.1

cae/starcd: cae/starcd/4.28

compiler/clang: compiler/clang/9.0

compiler/gnu: compiler/gnu/9.2

compiler/intel: compiler/intel/18.0, compiler/intel/19.0, compiler/intel/19.1

compiler/pgi: compiler/pgi/2019

devel/cmake: devel/cmake/3.16

devel/cuda: devel/cuda/9.2, devel/cuda/10.0, devel/cuda/10.2

devel/gdb: devel/gdb/9.1

devel/python: devel/python/3.7.4_gnu_9.2, devel/python/3.8.1_gnu_9.2, devel/python/3.8.1_intel_19.1

math/R: math/R/3.6.3

math/julia: math/julia/1.3.1

mpi/impi: mpi/impi/2018, mpi/impi/2019, mpi/impi/2020

mpi/openmpi: mpi/openmpi/4.0

numlib/mkl: numlib/mkl/2018, numlib/mkl/2019, numlib/mkl/2020

numlib/python_numpy: numlib/python_numpy/1.17.2_python_3.7.4_gnu_9.2

numlib/python_scipy: numlib/python_scipy/1.3.1_numpy_1.17.2_python_3.7.4_gnu_9.2

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

To learn more about a package execute:

$ module spider Foo

where "Foo" is the name of a module.

To find detailed information about a particular package you
must specify the version if there is more than one version:

$ module spider Foo/11.1

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
</pre>
''''module spider name/version'''' : If you search the full name and version of the module, the search gives detailed information about that module version.
<pre>
$ module spider devel/cmake

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
devel/cmake: devel/cmake/3.16
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

This module can be loaded directly: module load devel/cmake/3.16

Help:
Home page: https://www.cmake.org
Online Documentation: https://www.cmake.org/HTML/Documentation.html
Local Documentation: /opt/bwhpc/common/devel/cmake/3.16.4/docFAQ: https://gitlab.kitware.com/cmake/community/wikis/FAQ

In case of problems, please contact 'bwunicluster-hotline (at) lists.kit.edu'
or submit a trouble ticket at http://www.support.bwhpc-c5.de.

</pre>
Moreover, you can see the dependencies of the module with using the same command. For example, if the following is executed, you can see which modules need to be loaded before loading the module mpi/impi/2019
<pre>
$ module spider mpi/impi/2019

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
mpi/impi: mpi/impi/2019
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

You will need to load all module(s) on any one of the lines below before the "mpi/impi/2019" module is available to load.

compiler/clang/9.0
compiler/gnu/9.2
compiler/intel/18.0
compiler/intel/19.0
compiler/intel/19.1

Help:
Intel(R) MPI Library

</pre>
 

= How do Modules work? =
The default shell on the bwHPC clusters is bash, so explanations and examples will be shown for bash. In general, programs cannot modify the environment of the shell they are being run from, so how can the module command do exactly that?
 
The module command is not a program, but a bash-function.
You can view its content using:
<pre>
$ type module
</pre>
and you will get the following result:
<pre>
$ type module
module is a function
module ()
{
eval $($LMOD_CMD bash "$@");
[ $? = 0 ] && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}
</pre>
In this function, lmod is called. Its output to stdout is then executed inside your current shell using the bash-internal ''eval'' command. As a consequence, all output that you see from the module is transmitted via stderr (output handle 2) or in some cases even stdin (output handle 0).
 
 
----
[[Category:bwUniCluster_2.0|bwUniCluster 2.0]]
[[#top|Back to top]]

BwUniCluster2.0/Software Modules

2022-10-08T00:05:00Z

R Barthel: R Barthel moved page BwUniCluster2.0/Software to BwUniCluster2.0/Software Modules without leaving a redirect

<div id="top"></div>
 
= Introduction =
'''Environment Modules''', or short '''Modules''' are the means by which most of the installed scientific software is provided on bwUniCluster 2.0.
 
The use of different compilers, libraries and software packages requires users to set up a specific session environment suited for the program they want to run. bwUniCluster 2.0 provides users with the possibility to load and unload complete environments for compilers, libraries and software packages by a single command.
 
 

= Description =
The Environment ''Modules'' package enables dynamic modification of your environment by the
use of so-called ''modulefiles''. A ''modulefile'' contains information to configure the shell
for a program/software . Typically, a modulefile contains instructions that alter or set shell
environment variables, such as PATH and MANPATH, to enable access to various installed
software.
 
One of the key features of using the Environment ''Modules'' software is to allow multiple versions of the same software to be used in your environment in a controlled manner.
For example, two different versions of the Intel C compiler can be installed on the system at the same time - the version used is based upon which Intel C compiler modulefile is loaded.
 
The software stack of bwUniCluster 2.0 provides a number of modulefiles. You can also
create your own modulefiles. ''Modulefiles'' may be shared by many users on a system, and
users may have their own collection of modulefiles to supplement or replace the shared
modulefiles.
 
A modulefile does not provide configuration of your environment until it is explicitly loaded,
i.e., the specific modulefile for a software product or application must be loaded in your environment before the configuration information in the modulefile is effective.
 
If you want to see which modules are loaded you must execute
''''module list''''.
 
<pre>
$ module list
Currently Loaded Modules:
1) compiler/intel/19.1 2) mpi/impi/2019 3) numlib/mkl/2019
</pre>
 

= Usage =
Lmod on bwUniCluster 2.0: A New Environment Module System from http://lmod.readthedocs.org/en/latest/ is installed.
== Documentation ==
Execute ''''module help'''' or ''''man module'''' for help on how to use ''Modules'' software.
<pre>
$ module help
Usage: module [options] sub-command [args ...]

Options:
-h -? -H --help This help message
-s availStyle --style=availStyle Site controlled avail style: system (default: system)
--regression_testing Lmod regression testing
-D Program tracing written to stderr
--debug=dbglvl Program tracing written to stderr (where dbglvl is a number 1,2,3)
--pin_versions=pinVersions When doing a restore use specified version, do not follow defaults
-d --default List default modules only when used with avail
-q --quiet Do not print out warnings
--expert Expert mode
-t --terse Write out in machine readable format for commands: list, avail, spider, savelist
--initial_load loading Lmod for first time in a user shell
--latest Load latest (ignore default)
--ignore_cache Treat the cache file(s) as out-of-date
--novice Turn off expert and quiet flag
--raw Print modulefile in raw output when used with show
-w twidth --width=twidth Use this as max term width
-v --version Print version info and quit
-r --regexp use regular expression match
--gitversion Dump git version in a machine readable way and quit
--dumpversion Dump version in a machine readable way and quit
--check_syntax --checkSyntax Checking module command syntax: do not load
--config Report Lmod Configuration
--config_json Report Lmod Configuration in json format
--mt Report Module Table State
--timer report run times
--force force removal of a sticky module or save an empty collection
--redirect Send the output of list, avail, spider to stdout (not stderr)
--no_redirect Force output of list, avail and spider to stderr
--show_hidden Avail and spider will report hidden modules
--spider_timeout=timeout a timeout for spider
-T --trace

module [options] sub-command [args ...]

Help sub-commands:
------------------
help prints this message
help module [...] print help message from module(s)

Loading/Unloading sub-commands:
-------------------------------
load | add module [...] load module(s)
try-load | try-add module [...] Add module(s), do not complain if not found
del | unload module [...] Remove module(s), do not complain if not found
swap | sw | switch m1 m2 unload m1 and load m2
purge unload all modules
refresh reload aliases from current list of modules.
update reload all currently loaded modules.

Listing / Searching sub-commands:
---------------------------------
list List loaded modules
list s1 s2 ... List loaded modules that match the pattern
avail | av List available modules
avail | av string List available modules that contain "string".
spider List all possible modules
spider module List all possible version of that module file
spider string List all module that contain the "string".
spider name/version Detailed information about that version of the module.
whatis module Print whatis information about module
keyword | key string Search all name and whatis that contain "string".

Searching with Lmod:
--------------------
All searching (spider, list, avail, keyword) support regular expressions:

-r spider '^p' Finds all the modules that start with `p' or `P'
-r spider mpi Finds all modules that have "mpi" in their name.
-r spider 'mpi$ Finds all modules that end with "mpi" in their name.

Handling a collection of modules:
--------------------------------
save | s Save the current list of modules to a user defined "default" collection.
save | s name Save the current list of modules to "name" collection.
reset The same as "restore system"
restore | r Restore modules from the user's "default" or system default.
restore | r name Restore modules from "name" collection.
restore system Restore module state to system defaults.
savelist List of saved collections.
describe | mcc name Describe the contents of a module collection.
disable name Disable (i.e. remove) a collection.

Deprecated commands:
--------------------
getdefault [name] load name collection of modules or user's "default" if no name given.
===> Use "restore" instead <====
setdefault [name] Save current list of modules to name if given, otherwise save as the default list for you the user.
===> Use "save" instead. <====

Miscellaneous sub-commands:
---------------------------
is-loaded modulefile return a true status if module is loaded
is-avail modulefile return a true status if module can be loaded
show modulefile show the commands in the module file.
use [-a] path Prepend or Append path to MODULEPATH.
unuse path remove path from MODULEPATH.
tablelist output list of active modules as a lua table.

Important Environment Variables:
--------------------------------
LMOD_COLORIZE If defined to be "YES" then Lmod prints properties and warning in color.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Lmod Web Sites

Documentation: http://lmod.readthedocs.org
Github: https://github.com/TACC/Lmod
Sourceforge: https://lmod.sf.net
TACC Homepage: https://www.tacc.utexas.edu/research-development/tacc-projects/lmod

To report a bug please read http://lmod.readthedocs.io/en/latest/075_bug_reporting.html
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Modules based on Lua: Version 8.2 (8.2-1-g9c98036c) 2019-10-30 11:17 -05:00
by Robert McLay mclay@tacc.utexas.edu

</pre>
For help on particular version of ''Module'', e.g. Intel default compiler version, execute
''''module help compiler/intel''''.
<pre>
$ module help compiler/intel
---------------------- Module Specific Help for "compiler/intel/19.1" ----------------------
Intel(R) Compilers 19.1 for Linux*
For details see: https://software.intel.com/en-us/intel-compilers
In case of problems, please contact: Hartmut Häfner <hartmut.haefner@kit.edu>
SCC support end: 2022-12-31
</pre>
 
=== Online Documentation ===
[http://lmod.readthedocs.org Lmod: A New Environment Module System]
 
 

== Display all available Modules ==
Available ''Module'' are modulefiles that can be loaded by the user. A ''Module'' must be loaded before it provides changes to your environment, as described in the introduction to this section. You can display all available ''Modules'' on the system by executing:
<pre>
$ module avail
</pre>
The short form the command is:
<pre>
$ module av
</pre>
Available ''Modules'' can be also displayed in different modes, such as
* each ''Module'' per one line
<pre>
$ module -t avail
</pre>
Some modules may not be available right now, because their requirements are not met. To get a complete list of all possible modules use the [[#Display all possible Modules|module spider command]].
 
 

== Module categories, versions and defaults ==
The bwHPC clusters traditionally provide a large variety of software and software versions. Therefore ''Module'' are divided in category folders containing subfolders of modulefiles again containing modulefile versions, and must be addressed as follows:
category/softwarename/version
For instance all versions of the Intel compiler belong to the category of compilers, thus the corresponding modulefiles are placed under the category ''compiler'' and ''intel''.
 
In case of multiple software versions, one version will be always defined as the '''default'''
version. The ''Module'' of the default can be addressed by simply omitting the version number:
category/softwarename
 

== Finding software Modules ==
Currently all bwUniCluster 2.0 software packages are assigned to the following ''Module'' categories (???):


* bio

* cae

* chem

* compiler

* devel

* lib

* math
* mpi

* numlib

* phys

* system

* toolkit

* vis
You can selectively list software in one of those categories using, e.g. for the category "compiler"
<pre>
$ module avail compiler/
</pre>
Searches are looking for a substring starting at the begin of the name, so this would list all software in categories starting with a "c"
<pre>
$ module avail c
</pre>
while this would find nothing
<pre>
$ module avail hem
</pre>
 

== Loading Modules ==
You can load a ''Module'' software in to your environment to enable easier access to software that
you want to use by executing:
<pre>
$ module load category/softwarename/version
</pre>
or
<pre>
$ module add category/softwarename/version
</pre>
Loading a ''Module'' in this manner affects ONLY your environment for the current session.
 
 
=== Loading conflicts ===
You can not load different versions of the same software at the same time! Loading the Intel compiler in version X while Intel compiler in version Y is loaded leads to an automatic unloading of Intel compiler in version Y.
 
 

=== Showing the changes introduced by a Module ===
Loading a ''Module'' will change the environment of the current shell session. For instance the $PATH variable will be expanded by the software's binary directory. Other ''Module'' variables may even change the behavior of the current shell session or the software program(s) in a more drastic way.
 
 
All the changes to the current shell session to be invoked by loading the ''Module'' can be reviewed using
 
''''module show category/softwarename/version''''.
 
 
'''Example (Intel compiler)'''
<pre>
$ module show compiler/intel
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
/opt/bwhpc/common/modulefiles/Core/compiler/intel/19.1.lua:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
setenv("INTEL_LICENSE_FILE","28518@scclic1.scc.kit.edu")
setenv("AR","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64/xiar")
setenv("CC","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64/icc")
setenv("CXX","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64/icpc")
setenv("F77","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64/ifort")
setenv("FC","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64/ifort")
setenv("CFLAGS","-O2 -xCORE-AVX2")
setenv("CXXFLAGS","-O2 -xCORE-AVX2")
setenv("FFLAGS","-O2 -xCORE-AVX2")
setenv("FCFLAGS","-O2 -xCORE-AVX2")
setenv("INTEL_VERSION","19.1.0.166")
setenv("INTEL_HOME","/opt/intel/compilers_and_libraries_2020/linux")
setenv("INTEL_BIN_DIR","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64")
setenv("INTEL_LIB_DIR","/opt/intel/compilers_and_libraries_2020/linux/lib/intel64")
setenv("INTEL_INC_DIR","/opt/intel/compilers_and_libraries_2020/linux/include")
setenv("INTEL_MAN_DIR","/opt/intel/compilers_and_libraries_2020/linux/man/common")
setenv("INTEL_DOC_DIR","/opt/intel/compilers_and_libraries_2020/linux/documentation/en")
setenv("GDB_VERSION","19.1.0.166")
setenv("GDB_HOME","/opt/intel/debugger_2020/gdb/intel64")
setenv("GDB_BIN_DIR","/opt/intel/debugger_2020/gdb/intel64/bin")
setenv("GDB_LIB_DIR","/opt/intel/debugger_2020/libipt/intel64/lib")
setenv("GDB_INC_DIR","/opt/intel/debugger_2020/gdb/intel64/include")
setenv("GDB_INF_DIR","/opt/intel/documentation_2020/en/debugger/gdb-ia/info")
setenv("GDB_MAN_DIR","/opt/intel/documentation_2020/en/debugger/gdb-ia/man")
setenv("KMP_AFFINITY","noverbose,granularity=core,respect,warnings,compact,1")
prepend_path("PATH","/opt/intel/compilers_and_libraries_2020/linux/bin/intel64")
prepend_path("MANPATH","/opt/intel/compilers_and_libraries_2020/linux/man/common")
prepend_path("LD_LIBRARY_PATH","/opt/intel/compilers_and_libraries_2020/linux/lib/intel64")
whatis("Sets up Intel C/C++ and Fortran compiler version 19.1 (Intel(R) Compilers 19.1 for Linux*) - supported by SCC till 2022-12-31!")
help([[Intel(R) Compilers 19.1 for Linux*
For details see: https://software.intel.com/en-us/intel-compilers
In case of problems, please contact: Hartmut Häfner <hartmut.haefner@kit.edu>
SCC support end: 2022-12-31]])
prepend_path("MODULEPATH","/software/bwhpc/common/modulefiles/Compiler/intel/19.1")
family("compiler")
</pre>
 
 

=== Modules depending on Modules ===
Some program ''Modules'' depend on libraries to be loaded to the user environment. Therefore the
corresponding ''Modules'' of the software must be loaded together with the ''Modules'' of
the libraries.
 
By default such software ''Modules'' try to load required ''Modules'' and corresponding versions automatically.
 
 
 

== Unloading Modules ==
To unload or remove a software ''Module'' execute:
<pre>
$ module unload category/softwarename/version
</pre>
or
<pre>
$ module remove category/softwarename/version
</pre>
 

=== Unloading all loaded modules ===
==== Purge ====
Unloading a ''Module'' that has been loaded by default makes it inactive for the current session only - it will be reloaded the next time you log in.
 
In order to remove all previously loaded software modules from your environment issue the command 'module purge'.
 
Example
<pre>
$ module list
Currently Loaded Modules:
1) compiler/intel/19.1 2) mpi/impi/2019 3) numlib/mkl/2019
$
$ module purge
$ module list
No modules loaded
$
</pre>
Beware!
 
'module purge' is working without any further inquiry.
 
 

== Display your loaded Modules ==
All ''Modules'' that are currently loaded for you can be displayed by the
command ''''module list''''. [[#Purge|See example above]].
 
Note: You only have to load further ''Modules'', if you want to use additional software
packages or to change the version of an already loaded software.
 
 

== Display all possible Modules ==
Modulefiles can be searched by the user. You can dipslay all possible modules by executing:
<pre>
$ module spider

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The following is a list of the modules and extensions currently available:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
cae/abaqus: cae/abaqus/2018, cae/abaqus/2019

cae/adina: cae/adina/9.1.2

cae/ansys: cae/ansys/19.2, cae/ansys/2019R3, cae/ansys/2020R1

cae/comsol: cae/comsol/5.4, cae/comsol/5.5

cae/cst: cae/cst/2018

cae/lsdyna: cae/lsdyna/901

cae/openfoam: cae/openfoam/v1912, cae/openfoam/2.4.x, cae/openfoam/6, cae/openfoam/7

cae/paraview: cae/paraview/5.8

cae/starccm+: cae/starccm+/14.02.010, cae/starccm+/2019.2.1

cae/starcd: cae/starcd/4.28

compiler/clang: compiler/clang/9.0

compiler/gnu: compiler/gnu/9.2

compiler/intel: compiler/intel/18.0, compiler/intel/19.0, compiler/intel/19.1

compiler/pgi: compiler/pgi/2019

devel/cmake: devel/cmake/3.16

devel/cuda: devel/cuda/9.2, devel/cuda/10.0, devel/cuda/10.2

devel/gdb: devel/gdb/9.1

devel/python: devel/python/3.7.4_gnu_9.2, devel/python/3.8.1_gnu_9.2, devel/python/3.8.1_intel_19.1

math/R: math/R/3.6.3

math/julia: math/julia/1.3.1

mpi/impi: mpi/impi/2018, mpi/impi/2019, mpi/impi/2020

mpi/openmpi: mpi/openmpi/4.0

numlib/mkl: numlib/mkl/2018, numlib/mkl/2019, numlib/mkl/2020

numlib/python_numpy: numlib/python_numpy/1.17.2_python_3.7.4_gnu_9.2

numlib/python_scipy: numlib/python_scipy/1.3.1_numpy_1.17.2_python_3.7.4_gnu_9.2

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

To learn more about a package execute:

$ module spider Foo

where "Foo" is the name of a module.

To find detailed information about a particular package you
must specify the version if there is more than one version:

$ module spider Foo/11.1

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
</pre>
''''module spider name/version'''' : If you search the full name and version of the module, the search gives detailed information about that module version.
<pre>
$ module spider devel/cmake

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
devel/cmake: devel/cmake/3.16
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

This module can be loaded directly: module load devel/cmake/3.16

Help:
Home page: https://www.cmake.org
Online Documentation: https://www.cmake.org/HTML/Documentation.html
Local Documentation: /opt/bwhpc/common/devel/cmake/3.16.4/docFAQ: https://gitlab.kitware.com/cmake/community/wikis/FAQ

In case of problems, please contact 'bwunicluster-hotline (at) lists.kit.edu'
or submit a trouble ticket at http://www.support.bwhpc-c5.de.

</pre>
Moreover, you can see the dependencies of the module with using the same command. For example, if the following is executed, you can see which modules need to be loaded before loading the module mpi/impi/2019
<pre>
$ module spider mpi/impi/2019

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
mpi/impi: mpi/impi/2019
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

You will need to load all module(s) on any one of the lines below before the "mpi/impi/2019" module is available to load.

compiler/clang/9.0
compiler/gnu/9.2
compiler/intel/18.0
compiler/intel/19.0
compiler/intel/19.1

Help:
Intel(R) MPI Library

</pre>
 

= How do Modules work? =
The default shell on the bwHPC clusters is bash, so explanations and examples will be shown for bash. In general, programs cannot modify the environment of the shell they are being run from, so how can the module command do exactly that?
 
The module command is not a program, but a bash-function.
You can view its content using:
<pre>
$ type module
</pre>
and you will get the following result:
<pre>
$ type module
module is a function
module ()
{
eval $($LMOD_CMD bash "$@");
[ $? = 0 ] && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}
</pre>
In this function, lmod is called. Its output to stdout is then executed inside your current shell using the bash-internal ''eval'' command. As a consequence, all output that you see from the module is transmitted via stderr (output handle 2) or in some cases even stdin (output handle 0).
 
 
----
[[Category:bwUniCluster_2.0|bwUniCluster 2.0]]
[[#top|Back to top]]

Turbomole

2022-10-08T00:00:55Z

R Barthel: R Barthel moved page Turbomole to JUSTUS2/Software/Turbomole

#REDIRECT [[JUSTUS2/Software/Turbomole]]

JUSTUS2/Software/Turbomole

2022-10-08T00:00:55Z

R Barthel: R Barthel moved page Turbomole to JUSTUS2/Software/Turbomole

{{Softwarepage|chem/turbomole}}

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| chem/turbomole
|-
| License
| [https://www.turbomole.org/turbomole/order-turbomole/ Commerical]
|-
| Citing
| [https://www.turbomole.org/turbomole/turbomole-documentation/ See Turbomole manual]
|-
| Links
| [https://www.turbomole.com Homepage] | [https://www.turbomole-gmbh.com/turbomole-manuals.html Documentation]
|-
| User Forum
| [https://www.turbo-forum.com/ external]
|}
= Description =
'''Turbomole''' is a general purpose ''quantum chemistry'' software package for ''ab initio'' electronic structure calculations and provides:
* ground state calculations for methods such as Hartree-Fock, DFT, MP2, and CCSD(T);
* excited state calculations at different levels such as full RPA, TDDFT, CIS(D), CC2, an ADC(2);
* geometry optimizations, transition state searches, molecular dynamics calculations;
* property and spectra calculations such as IR, UV/VIS, Raman, and CD;
* approximations like resolution-of-the-identity (RI) to speed-up the calculations without introducing uncontrollable or unknown errors; as well as
* parallel versions for all kind of jobs.
For more information on Turbmole's features please visit [https://www.turbomole-gmbh.com/program-overview.html http://www.turbomole-gmbh.com/program-overview.html].
 
 

= Versions and Availability =

On the command line interface (CLI) of a particular bwHPC cluster a list of all available Turbomole versions can be inquired as followed
<pre>
$ module avail chem/turbomole
</pre>

A current list of the versions available on the bwUniCluster and bwForClusters can be found here:
https://www.bwhpc.de/software.php

== bwUniCluster 2.0 ==

* Turbomole 7.4.1

== bwForCluster JUSTUS 2 ==

* Turbomole 7.5
* Turbomole 7.4.1

== Parallel computing ==
The Turbomole ''Module'' subsumes all available parallel computing variants of Turbomole's binaries. Turbomole defines the following parallel computing variants:
* SMP = Shared-memory parallel computing based on OpenMP and Fork() with the latter using separated address spaces.
* MPI = Message passing interface protocol based parallel computing
However only one of the parallel variants or the sequential variant can be loaded at once and most Turbomole's binaries support only 1 or 2 of the parallelization variants. Like for Turbomole installations without the ''Module'' system of the bwHPC clusters, the variants have to be triggered by the environment variable $PARA_ARCH.
 
 

= Usage =
== Before loading the Module ==
Before loading the Turbomole ''Module'' the parallel computing variant has to be defined via the environment variable $PARA_ARCH using the abbreviations SMP or MPI, e.g.:
<pre>
$ export PARA_ARCH=MPI
</pre>
will [[#Loading the Module|later load]] the '''MPI binary variants'''. If the variable $PARA_ARCH is not defined or empty, the sequential binary variants will be active once the Turbomole Module is loaded.
 
== Loading the Module ==
You can load the default version of ''Turbomole'' with the command:
<pre>
$ module load chem/turbomole
</pre>
The Turbomole ''Module'' does not depend on any other ''Module'', but on the variable $PARA_ARCH. Moreover, Turbomole provides its own libraries regarding ''OpenMP'', ''Fork()'', and ''MPI'' based parallelization.
If you wish to load a specific (older) version you can do so by executing e.g.:
<pre>
$ module load chem/turbomole/7.4.1
</pre>
to load the version 7.4.1
 
== Switching between different parallel variants ==
To switch between the different parallel variants provided by the Turbomole ''Module'', simply define the new parallel variant via $PARA_ARCH and load the ''Module'' again. Note that for switching between the parallel variants unloading of the Turbomole ''Module'' is not required. For instance to change to the MPI variant, execute:
<pre>
$ export PARA_ARCH=MPI
$ module load chem/turbomole
</pre>
 
== Turbomole binaries ==
The '''Turbomole''' software package consists of a set of stand-alone program binaries providing different features and parallelization support:
{| width=750px class="wikitable"
|- style=text-align:left"
! Binary !! Features !! OpenMP !! Fork !! MPI
|- style="vertical-align:top;"
| define
| Interactive input generator
| no
| no
| no
|- style="vertical-align:top;"
| dscf
| Energy calculations
| yes
| yes
| yes
|- style="vertical-align:top;"
| grad
| Gradient calculations
| yes
| yes
| yes
|- style="vertical-align:top;"
| ridft
| Energy calc. with fast Coulomb approximation
| no
| yes
| yes
|- style="vertical-align:top;"
| rdgrad
| Gradient calc. with fast Coulomb approximation
| yes
| yes
| yes
|- style="vertical-align:top;"
| ricc2
| Electronic excitation energies, transition moments and properties of excited states
| yes
| no
| yes
|- style="vertical-align:top;"
| statpt
| Hessian and coordinate update for stationary point search
| no
| no
| no
|- style="vertical-align:top;"
| aoforce
| Analytic calculation of force constants, vibrational frequencies and IR intensities
| yes
| yes
| yes
|- style="vertical-align:top;"
| escf
| Calc. of time dependent and dielectric properties
| yes
| yes
| yes
|- style="vertical-align:top;"
| egrad
| gradients and first-order properties of excited states
| yes
| no
| yes
|- style="vertical-align:top;"
| odft
| Orbital-dependent energy calc.
| yes
| no
| no
|}
For the complete set of binaries and more detailed description of their features read [https://www.turbomole.org/turbomole/turbomole-documentation/ here].
 

== Turbomole tools ==
Turbomole's tool set contains scripts and binaries that help to prepare, execute workflows (such as geometry optimisation) and postprocess results.
{| width=750px class="wikitable"
|- style="text-align:left"
! Script !! Type !! Description
|- style="vertical-align:top;"
| x2t
| Preparation
| Converts XYZ coordinates into ''Turbomole'' coordinates.
|- style="vertical-align:top;"
| sdg
| Preparation
| Shows data group from ''control'' file: for example$ sdg coordshows current Turbomole coordinates used.
|- style="vertical-align:top;"
| jobex
| Optimization workflow
| Shell script that controls and executes automatic optimizations of molecular geometry parameters.

|- style="vertical-align:top;"
| tm2molden
| Postprocessing
| Creates a molden format input file for the [[Molden]] program.
|- style="vertical-align:top;"
| eiger
| Postprocessing
| Script front-end of program ''eigerf'' to display HOMO-LUMO gap and MO eigenvalues.
|}
For the complete set of tools and more detailed description of their features read [http://www.turbomole-gmbh.com/manuals/version_6_5/Documentation_html/node9.html here].
 
== Disk Usage ==
By default, scratch files of Turbomole binaries are placed in the directory of Turbomole binary execution. Please do not run your Turbomole calculations in your $HOME or $WORK directory.
 
 
= Examples =
== Single node jobs ==
=== Template provided by Turbomole Module ===
The Turbomole ''Module'' provides a simple job script example of Cubane (C8H8) that runs an energy relaxation via MPI parallel ''dscf'' using 4 cores on a single node and its local file system. To simply run the example do the following steps:
<pre>
$ module load chem/turbomole
$ mkdir -vp ~/Turbomole-example/
$ cd ~/Turbomole-examples/
$ cp -r $TURBOMOLE_EXA_DIR/* ~/Turbmole-example/
$ sbatch bwHPC_turbomole_single-node_tmpdir_example.sh
</pre>
The last step submits the job example script bwHPC_turbomole_single-node_example.sh to the queueing system. Once started on a compute node, all calculations will be done under an unique directory on the [[BwUniCluster_File_System#File_Systems|local file system]] of that particular compute node.
 
=== Geometry optimization ===
To do a geometry optimization of the [[#Single node job template provided by Turbomole Module|previous job example]] modify bwHPC_turbomole_single-node_tmpdir_example.sh by replacing the following line
<source lang="bash">
time dscf > dscf.out 2>&1
</source>
with
<source lang="bash">
time jobex -dscf -keep 2>&1
</source>
and submit the modified script to the queueing system on bwForCluster JUSTUS 2 via [[Slurm_JUSTUS_2|sbatch]] and bwUniCluster also via [[BwUniCluster_2.0_Slurm_common_Features|sbatch]], respectively.
The Turbomole command ''jobex'' controls the call of all the required Turbomole binaries for the geometry optimization.
 
 
= Turbomole-Specific Environment Variables =
To see a list of all Turbomole environments set by the 'module load'-command do the following:
<pre>
module show chem/turbomole 2>&1 | grep -E '(setenv|prepend-path)'
</pre>

= Version-Specific Information =
For specific information about the Turbomole version (e.g. 7.5) to be loaded do the following:
<pre>
module help chem/turbomole/7.5
</pre>

----
[[Category:Chemistry software]][[Category:bwUniCluster_2.0]][[Category:bwForCluster_JUSTUS_2]]

BwUniCluster2.0/Acknowledgement

2022-10-07T21:58:33Z

R Barthel:

The HPC resource bwUniCluster (2.0) is funded by the Ministry of Science, Research and the Arts Baden-Württemberg and the Universities of the State of Baden-Württemberg:
* Albert Ludwig University of Freiburg
* Eberhard Karls University, Tübingen
* Karlsruhe Institute of Technology (KIT)
* Ruprecht-Karls-Universität Heidelberg
* Ulm University
* University of Hohenheim
* University of Konstanz
* University of Mannheim
* University of Stuttgart,
* several [https://wiki.bwhpc.de/e/BwUniCluster_2.0_User_Access Universities of Applied Science].

Therefore, any work performed on bwUniCluster (2.0) should be appropriately cited:

'''"The authors acknowledge support by the state of Baden-Württemberg through bwHPC."'''

Furthermore, please report any publication that cites bwUniCluster (2.0) to [mailto:publications@bwhpc.de publications@bwhpc.de] stating:
* DOI (if available)
* author(s)
* title ''or'' booktitle
* journal, volume, pages ''or'' editors, address, publisher
* year.
This information will be used for the evaluation of bwUniCluster (2.0).

The publications will be referenced on the bwHPC website:
https://www.bwhpc.de/user_publications.html

----
[[Category:bwUniCluster_2.0|Acknowledgement]]
[[Category:Acknowledgement|bwUniCluster 2.0]]

BwUniCluster 2.0

2022-10-07T21:52:16Z

R Barthel: Changed redirect target from Category:BwUniCluster 2.0 to BwUniCluster2.0

#REDIRECT [[bwUniCluster2.0]]

BwUniCluster2.0/Maintenance

2022-10-07T21:51:13Z

R Barthel:

'''2022'''

* [[BwUniCluster2.0/Maintenance/2022-03]] from 28.03.2022 to 31.03.2022

'''2021'''

* [[BwUniCluster2.0/Maintenance/2021-10]] from 11.10.2021 to 15.10.2021

'''2020'''

* [[BwUniCluster2.0/Maintenance/2020-10]] from 06.10.2020 to 13.10.2020

=== Maintenance records of retired bwUniCluster 1.0 ===

[[Category:BwUniCluster 2.0]]

'''2019'''

* [[BwUniCluster/Maintenance/2019-02]] from 02.02.2019 to 08.02.2019

'''2017'''

* [[BwUniCluster/Maintenance/2017-05]] from 02.05.2017 to 02.05.2017
* [[BwUniCluster/Maintenance/2017-03]] from 20.03.2017 to 21.03.2017

'''2016'''

* [[BwUniCluster/Maintenance/2016-10]] from 17.10.2016 to 21.10.2016

BwUniCluster/Maintenance/2019-02

2022-10-07T21:51:02Z

R Barthel: R Barthel moved page 2019-02-04: bwUniCluster Maintenance to BwUniCluster/Maintenance/2019-02 without leaving a redirect

Dear users of bwUniCluster,

from
February, 4th, 2019
until
February, 8th, 2019
the HPC system bwUniCluster will not be available due to a planned maintenance slot. The system will only start jobs which are expected to have completed before the maintenance slot starts. At the beginning of the maintenance we will cancel all running user processes.

During the maintenance we will update the software and firmware of the file systems and cluster nodes for security reasons.

Best regards
the HPC operations team

----
[[Category:bwHPC News|UniCluster]][[Category:bwUniCluster]]

BwUniCluster/Maintenance/2017-05

2022-10-07T21:48:09Z

R Barthel: R Barthel moved page 2017-05-02: bwUniCluster Maintenance to BwUniCluster/Maintenance/2017-05 without leaving a redirect

Dear users,

we expect to put the bwUniCluster extension partition into production
during the first week of May. Because of this the whole bwUniCluster
system will be shut down for maintenance for a whole day on

May 2, 2017

starting at 9 AM.

All jobs still running at this time will be aborted, and all job queues
will be flushed during the maintenance.

After this maintenance, the "multinode" queue will contain nodes
from the extension partition exclusively. At the same time the maximum
number of nodes for a "multinode" job will be increased to 128. All
other settings and limits will remain the same.

kind regards,
the system administrators

----
[[Category:bwHPC News|UniCluster]][[Category:bwUniCluster]]

BwUniCluster/Maintenance/2017-03

2022-10-07T21:46:05Z

R Barthel: R Barthel moved page 2017-03-20: bwUniCluster Maintenance to BwUniCluster/Maintenance/2017-03 without leaving a redirect

Dear users of bwUniCluster, ForHLR I and ForHLR II,

on March, 20th, and March, 21st, the HPC systems bwUniCluster, ForHLR I and
ForHLR II will not be available because of a planned maintenance. During the maintenance we will again put the original $HOME file system of
all non-KIT users into operation. All data will be copied from the currently
used file system, i.e. afterwards you will have the same data as before.

We will also update software and firmware of file systems and cluster nodes
but there should be no impact on your applications.

After the maintenance a daily email alert will be sent for workspaces which
will expire during the next week.

On bwUniCluster job classes will change and therefore all waiting jobs
will be removed from the job queue.

Best regards, 
HPC admin team @ KIT/SCC

----
[[Category:bwHPC News|UniCluster]][[Category:bwUniCluster]]

BwUniCluster/Maintenance/2016-10

2022-10-07T21:41:42Z

R Barthel: R Barthel moved page 2016-10-17: bwUniCluster Maintenance to BwUniCluster/Maintenance/2016-10 without leaving a redirect

The bwUniCluster system was '''unavailable''' between '''October 17, 2016''' and '''October 21, 2016''' for maintenance which had been completed as planned.

We strongly recommend to recompile your source code after the maintenance due to operating system as well as software and libraries upgrade. This does also apply to binaries that may
have been built as add-ons for existing software modules, e.g. private packages for R and
Octave, Matlab MEX files, etc.

All pending jobs had been canceled due to possibly obsolete dependencies.

== Operational System Upgrade ==
Upgrade from Red Hat Enterprise Linux Server release '''6.8''' (Santiago) to Red Hat Enterprise Linux Server release '''7.2''' (Maipo)

== REPLACED SSH Host Key ==
The original SSH host key of bwUniCluster has been generated under Redhat 6 with [https://en.wikipedia.org/wiki/Digital_Signature_Algorithm DSA]. Redhat 7 and latest SSH clients recommend improved key generation algorithm such as ECDSA and ED25519. Indeed, Redhat 7 has opted out DSA key generation. Following the recommendations the original SSH host key of bwUniCluster has been replaced with a new one based on [https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm ECDSA].
=== HowTo: Putty users ===
If you have used the putty client before the maintenance the original SSH host key has been cached. Putty will pop up the following message to you (cf. screen shot below):
[[File:BwUniCluster_howto_replace_ssh_host_key_putty_2016-10-21.png|650px|center|border|]]
By accepting it (button '''Yes''') putty will stored the new SSH host key.

=== HowTo: Linux and macOS users ===
Remove the SSH host key of the bwUniCluster host name (bwunicluster.scc.kit.edu or uc1.scc.kit.edu):
<pre>
for hostname in uc1.scc.kit.edu bwunicluster.scc.kit.edu uc1-login1.scc.kit.edu uc1-login2.scc.kit.edu; do ssh-keygen -f "${HOME}/.ssh/known_hosts" -R $hostname ; done
</pre>
Afterwards and during login accept the new host key:
<pre>
ECDSA key fingerprint is e0:3b:1e:36:80:44:c5:48:e9:b3:eb:65:34:ba:72:5c.
Are you sure you want to continue connecting (yes/no)?
yes
</pre>

== Updated and new software modules ==
* bio/qiime/1.9.1
* chem/amber/12 with Intel 14 only (Amber 14 available on bwFor-Cluster Justus)
* chem/dalton/2016
* chem/gromacs/5.1.4
* devel/python/2.7.12
* devel/python/3.5.2
* math/R/3.3.1
* math/rstudio/0.99
* math/sagemath/7.3
* numlib/fftw/3.3.5-impi-5.1.3-gnu-4.8
* numlib/fftw/3.3.5-impi-5.1.3-intel-16.0
* numlib/gsl/2.2.1-gnu-4.8
* numlib/gsl/2.2.1-intel-16.0
* numlib/python_numpy/1.11.2-python-2.7.12
* phys/alps/2.1.1-python_numpy-1.11.2-hdf5-1.8-intel-16.0
* vis/tigervnc/1.3.1 (see migration note below)

=== How to migrate remote desktop from GNOME to XFCE or KDE with vis/tigervnc/1.3.1 ===
GNOME 3, the version that ships with RHEL 7, now requires hardware-accelerated
OpenGL. This means that GNOME 3 desktop environment won't work in a plain VNC
session any more. In order to migrate to KDE oder XFCE environment, <s>users will
have to create a file ~/.Xclients with the following contents</s> the new ''run_vncserver''
wrapper will automatically create a file ~/.Xclients with the following contents:

<pre>
unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS
unset XDG_RUNTIME_DIR
# Uncomment next line for XFCE4
exec startxfce4
# Uncomment next line for KDE
# exec startkde
</pre>

Depending on which line is uncommented in ~/.Xclients, this will cause the TigerVNC server
to invoke XFCE or KDE as remote desktop environment (see screenshot below).

[[File:Screenshot_VNC_XFCE.png|300px|thumb|center|VNC session running XFCE4 desktop environment remotely on top of local GNOME desktop.]]

'''Note:''' If you happen to see an error message "''Error when getting information for file ‘~/.gvfs’: Transport endpoint is not connected''" with the XFCE Thunar file manager accessing the
home directory, open a terminal window and run the following command:
<pre>fusermount -u ~/.gvfs </pre>

Alternatively, you can make Nautilus the default XFCE file manager in ''Settings > Preferred Applications > Utilities > File Manager''.

== CHANGED Software Defaults ==
Software modules for which a newer version has been set as the default.
* chem/turbomole/7.0 -> chem/turbomole/7.1
* chem/tmolex/4.1 -> chem/tmolex/4.2
* devel/python/2.7.10 -> devel/python/2.7.12 (actually not the default but newest 2.7 release)
* devel/python/3.5.0 -> devel/python/3.5.2
* math/R/3.1.2 -> math/R/3.3.1
* numlib/fftw/3.3.3-impi-4.1.1-intel-13.1 -> numlib/fftw/3.3.5-impi-5.1.3-intel-16.0
* numlib/gsl/1.16-intel-13.1 -> numlib/gsl/2.2.1-intel-16.0

* compiler/intel/15.0 -> compiler/intel/16.0
* numlib/mkl/11.2.3 -> numlib/mkl/11.3.4
* mpi/impi/5.0.3-intel-15.0 -> mpi/impi/5.1.3-intel-16.0
* devel/itac/9.0.3 -> devel/itac/9.1.2
* vis/tigervnc/1.1.0 -> vis/tigervnc/1.3.1

== RECOMPILED Software Modules==
Software modules which have been recompiled with different compiler or libraries
* phys/qutip/3.1.0
* chem/xcrysden/1.5.60

== REMOVED Software Modules==
=== Intel compiler and related software ===
Very old Intel compiler and related modules ''(which was available to all bwUniCluster users (/opt/bwhpc/common/modulefiles)'':
* compiler/intel/12.1
* compiler/intel/13.1
* numlib/mkl/10.3.12
* numlib/mkl/11.0.5
* mpi/impi/4.1.0-gnu-4.4
* mpi/impi/4.1.0-gnu-4.5
* mpi/impi/4.1.0-intel-12.1
* mpi/impi/4.1.1-gnu-4.4
* mpi/impi/4.1.1-gnu-4.7
* mpi/impi/4.1.1-intel-13.1
* devel/itac/8.1.1
* devel/itac/8.1.2
* mpi/openmpi/1.6.5-intel-12.1
* mpi/openmpi/1.6.5-intel-13.1
* mpi/openmpi/1.8-intel-12.1
* mpi/openmpi/1.8-intel-13.1
* mpi/openmpi/1.10-intel-12.1
* mpi/openmpi/1.10-intel-13.1
'''''Note that all dependent modules and binaries will stop working thereafter. So rebuild your modules and/or binaries with newer versions of the Intel compiler.'''''

=== Common software ===
Software that was available to all bwUniCluster users ''(/opt/bwhpc/common/modulefiles)'':
* bio/qiime/1.8.0
* chem/dalton/2013
* chem/gromacs/4.5.3_gridcount-1.4
* chem/gromacs/4.6.2
* chem/gromacs/5.0
* chem/gromacs/5.0.2
* chem/turbomole/6.5
* chem/turbomole/6.6
* chem/tmolex/4.0
* chem/molden/5.2.1
* devel/cmake/2.8.11 (now system default version)
* lib/hdf5/1.8.13-gnu-4.4
* lib/hdf5/1.8.13-openmpi-1.8-gnu-4.4
* lib/matplotlib/1.3.1
* math/R/3.0.2
* numlib/fftw/3.3.3-impi-4.1.1-gnu-4.4
* numlib/fftw/3.3.3-impi-4.1.1-intel-13.1
* numlib/gsl/1.16-gnu-4.4
* numlib/gsl/1.16-intel-13.1
* phys/alps/2.1.1-python_numpy-1.10.1-hdf-1.8.14-impi-4.1.3
* phys/qutip/2.2.0
* vis/tigervnc/1.1.0
* vis/tigervnc/1.3.0

=== Freiburg software ===
Software that was '''ONLY''' available to users from Freiburg ''(/opt/bwhpc/fr/modulefiles)'':
* chem/lammps/5Sep2014_linux
* chem/lammps/5Sep2014_mkl
* cns/nest/2.2.2-python-2.7.6_cns
* cns/nest/2.8.0-python-2.7.6_cns
* devel/cmake/3.0.1
* devel/python/2.7.6_cns

----
[[Category:bwHPC News|UniCluster]]|[[Category:bwUniCluster]]

BwUniCluster2.0

2022-10-07T21:39:37Z

R Barthel:

[[File:BwUniCluster_2.0_Feb2020_1024x423.jpg|right|frameless|thumb|alt=bwUniCluster2.0 |upright=1| bwUniCluster 2.0 ]]


The '''bwUniCluster 2.0''' is the joint high-performance computer system of Baden-Württemberg's Universities and Universities of Applied Sciences for '''general purpose and teaching''' and located at the Steinbuch Centre for Computing (SCC) at Karlsruhe Institute of Technology (KIT). The bwUniCluster 2.0 complements the four bwForClusters and their dedicated scientific areas.


{| style=" background:#FEF4AB; width:100%;"
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News
|-
|
* 2022-09-22: [https://indico.scc.kit.edu/e/bwhpc_course_2022-10-13 Introductory Course: HPC and Data Mngt. in Baden-Württemberg] on '''2022-10-13'''
|}

{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support
|-
|
* [[BwUniCluster2.0/First_Steps|Geting Started]]
* [https://training.bwhpc.de E-Learning Courses]
* [[BwUniCluster2.0/Support|Support]]
* [[BwUniCluster2.0/FAQ|FAQ]]
* Send [[:Category:Feedback|Feedback]] about Wiki pages
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | User Documentation
|-
|
* Access: [[Registration/bwUniCluster|Registration]], [[Registration/Deregistration|Deregistration]], [[BwUniCluster2.0/Jupyter|Using Jupyter]]
* [[BwUniCluster2.0/Login|Login]]
*
* [[BwUniCluster2.0/Hardware_and_Architecture|Hardware and Architecture]]
** [[BwUniCluster2.0/Hardware_and_Architecture#File_Systems|File Systems and Workspaces]]
* [[BwUniCluster2.0/Software|Cluster Specific Software]]
** [[BwUniCluster2.0/Containers|Using Containers]]
* [[BwUniCluster2.0/Slurm|Batch System]]
** [[BwUniCluster2.0/Batch_Queues|Queues and interactive Jobs]]
* [[BwUniCluster2.0/Maintenance|Operational Changes]]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | Cluster Funding
|-
|
* Please [[BwUniCluster2.0/Acknowledgement|acknowledge]] bwUniCluster 2.0 in your publications.
|}

BwUniCluster2.0

2022-10-07T21:31:10Z

R Barthel:

[[File:BwUniCluster_2.0_Feb2020_1024x423.jpg|right|frameless|thumb|alt=bwUniCluster2.0 |upright=1| bwUniCluster 2.0 ]]


The '''bwUniCluster 2.0''' is the joint high-performance computer system of Baden-Württemberg's Universities and Universities of Applied Sciences for '''general purpose and teaching''' and located at the Steinbuch Centre for Computing (SCC) at Karlsruhe Institute of Technology (KIT). The bwUniCluster 2.0 complements the four bwForClusters and their dedicated scientific areas.


{| style=" background:#FEF4AB; width:100%;"
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News
|-
|
* 2022-09-22: [https://indico.scc.kit.edu/e/bwhpc_course_2022-10-13 Introductory Course: HPC and Data Mngt. in Baden-Württemberg] on '''2022-10-13'''
|}

{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support
|-
|
* [[BwUniCluster2.0/First_Steps|Geting Started]]
* [https://training.bwhpc.de E-Learning Courses]
* [[BwUniCluster2.0/Support|Support]]
* [[BwUniCluster2.0/FAQ|FAQ]]
* Send [[:Category:Feedback|Feedback]] about Wiki pages
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | User Documentation
|-
|
* Access: [[Registration/bwUniCluster|Registration]], [[Registration/Deregistration|Deregistration]], [[BwUniCluster2.0/Jupyter|Using Jupyter]]
* [[BwUniCluster2.0/Login|Login]]
*
* [[BwUniCluster2.0/Hardware_and_Architecture|Hardware and Architecture]]
** [[BwUniCluster2.0/Hardware_and_Architecture#File_Systems|File Systems and Workspaces]]
* [[BwUniCluster2.0/Software|Cluster Specific Software]]
** [[BwUniCluster2.0/Containers|Using Containers]]
* [[BwUniCluster2.0/Slurm|Batch System]]
** [[BwUniCluster2.0/Batch_Queues|Queues and interactive Jobs]]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | Cluster Funding
|-
|
* Please [[BwUniCluster2.0/Acknowledgement|acknowledge]] bwUniCluster 2.0 in your publications.
|}

Main Page

2022-10-07T21:28:20Z

R Barthel:

'''Welcome to the bwHPC Wiki.'''

bwHPC represents services and resources in the State of '''B'''aden-'''W'''ürttemberg, Germany, for High Performance Computing ('''HPC'''), Data Intensive Computing ('''DIC''') and Large Scale Scientific Data Management ('''LS2DM''').

The main bwHPC web page is at [https://www.bwhpc.de/ https://www.bwhpc.de/].

Many topics depend on the cluster system you use.
First choose the cluster you use, then select the correct topic.

{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Courses / eLearning
|-
|
* [https://training.bwhpc.de/ eLearning and Online Courses]
* [https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC Introduction to Linux in HPC (external resource)]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | Need Access to a Cluster?
|-
|
* [[Registration]]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | HPC System Specific Documentation
|-
|
bwHPC Clusters are dedicated to [https://www.bwhpc.de/bwhpc-domains.php specific research domains].
Documentation differs between compute clusters, please see cluster specific overview pages:
{|
|-
| style="padding:5px; width:30%" | [[BwUniCluster2.0|bwUniCluster 2.0]]
| style="padding-left:20px;" | General Purpose, Teaching
|-
| style="padding:5px; width:30%" | [[:Category:bwForCluster_JUSTUS_2| bwForCluster JUSTUS 2]]
| style="padding-left:20px;" | Theoretical Chemistry, Condensed Matter Physics, and Quantum Sciences
|-
| style="padding:5px; width:30%" | [[Helix|bwForCluster Helix]]
| style="padding-left:20px;" | Computational Humanities, Structural and Systems Biology, Medical Science, and Soft Matter
|-
| style="padding:5px; width:30%" | [[:Category:bwForCluster NEMO| bwForCluster NEMO]]
| style="padding-left:20px;" | Neurosciences, Particle Physics, Materials Science, and Microsystems Engineering
|-
| style="padding:5px; width:30%" | [[:Category:bwForCluster BinAC| bwForCluster BinAC]]
| style="padding-left:20px;" | Bioinformatics, Geosciences and Astrophysics.
|}
|-
|
Further Compute Clusters in Baden-Württemberg (separate access policies):
* bwHPC [[:Category:bwHPC infrastructure|tier]] 1: [https://kb.hlrs.de/platforms/index.php/HPE_Hawk Hawk] ([https://www.hlrs.de/solutions-services/academic-users/ getting access])
* bwHPC tier 2: [https://www.nhr.kit.edu/userdocs/horeka HoreKa] ([https://www.nhr.kit.edu/userdocs/horeka/projects/ getting access])
|}

{| style="background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | Documentation valid for all Clusters
|-
|
* [[Environment Modules| Software Modules]] and software documentation explained
* [https://www.bwhpc.de/software.html List of Software] on all clusters
* [[BwHPC_BPG_for_Parallel_Programming| Parallel Programming]]
|}

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | Scientific Data Storage
|-
|
For user guides of the scientific data storage services:
* [[SDS@hd]]
* [https://www.rda.kit.edu/english bwDataArchive] ([http://wiki.scc.kit.edu/lsdf/index.php/BwDataArchiv_FAQs FAQs])
Associated, but local scientific storage services are:
* [https://wiki.scc.kit.edu/lsdf/index.php/Category:LSDF_Online_Storage LSDF Online Storage] (only for KIT and KIT partners)
|}

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | Data Management
|-
|
* [[Data Transfer|Data Transfer]]
* [https://www.forschungsdaten.org/index.php/FDM-Kontakte#Deutschland Research Data Management (RDM)] contact persons
* [https://www.forschungsdaten.info Portal for Research Data Management] (Forschungsdaten.info)
|}
{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Support
|-
|
Support is provided by the [https://www.bwhpc.de/teams.php bwHPC Competence Centers]:
* [https://bw-support.scc.kit.edu/ Submit a Ticket]
* extended Support via [https://zas.bwhpc.de/en/zas_info_tigerteamsupport.php "Tiger Teams"]
|}

MediaWiki:Sidebar

2022-10-07T21:27:34Z

R Barthel:

BwUniCluster2.0/FAQ

2022-10-07T21:26:41Z

R Barthel:

Frequently asked questions (FAQs) concerning best practice of bwUniCluster 2.0

= Login =

* In case of having issues with OTP, service password or denied access → [[BwUniCluster2.0/Login#Troubleshooting|Troubleshooting]]

= Hardware and Architecture =

* Concerning best practice of [[BwUniCluster_2.0_Hardware_and_Architecture#Components_of_bwUniCluster|bwUniCluster broadwell partition]] → [[BwUniCluster2.0/FAQ_-_broadwell_partition|FAQ of broadwell partition]]

* Migration from bwUniCluster 1.0 to 2.0
** Concerning guide to migrate your data → [[BwUniCluster2.0/File_System_Migration_Guide|file system migration guide]]

= Batch System =

* Migration from bwUniCluster 1.0 to 2.0
** Concerning switching from MOAB to Slurm and adapting your job scripts → [[BwUniCluster2.0/Batch_System_Migration_Guide|MOAB to Slurm migration guide]]

BwUniCluster 2.0 Batch System Migration Guide

2022-10-07T21:23:52Z

R Barthel: R Barthel moved page BwUniCluster 2.0 Batch System Migration Guide to BwUniCluster2.0/Batch System Migration Guide

#REDIRECT [[BwUniCluster2.0/Batch System Migration Guide]]

BwUniCluster2.0/Batch System Migration Guide

2022-10-07T21:23:52Z

R Barthel: R Barthel moved page BwUniCluster 2.0 Batch System Migration Guide to BwUniCluster2.0/Batch System Migration Guide

While the former bwUniCluster 1 system used the combination Moab/SLURM for the batch system queue, bwUniCluster 2.0 uses only SLURM. This means that most job scripts and workflows which relied on Moab-specific pragmas and commands have to be changed.

=General Overview=

Job parameters can be passed to SLURM in the same ways which were possible with Moab.

* Instead of the #MOAB or #PBS pragmas, the pragma #SLURM has to be used within job files.

* Instead of the Moab commands, the corresponding SLURM commands have to be used.

A general mapping of Moab to SLURM commands can be found in the following table:

{| class="wikitable"
! Moab command !! SLURM command
|-
| msub || sbatch
|-
| msub -I || salloc
|-
| canceljob || scancel
|-
| showq || squeue
|-
| checkjob $JOBID || scontrol show job $JOBID
|}

In the next table you find the most MOAB job specification flags and environment variables. You have to replace all MOAB flags and environment variables of your batch scripts by their corresponding Slurm counterparts.

'''Commonly used MOAB job specification flags and their Slurm equivalents'''

{| width=750px class="wikitable"
! Option !! Moab (msub) !! Slurm (sbatch)
|-
| Script directive || #MSUB || #SBATCH
|-
| Job name || -N <name> || --job-name=<name> (-J <name>)
|-
| Account || -A <account> || --account=<account> (-A <account>)
|-
| Queue || -q <queue> || --partition=<partition> (-p <partition>)
|-
| Wall time limit || -l walltime=<hh:mm:ss> || --time=<hh:mm:ss> (-t <hh:mm:ss>)
|-
| Node count || -l nodes=<count> || --nodes=<count> (-N <count>)
|-
| Core count || -l procs=<count> || --ntasks=<count> (-n <count>)
|-
| Process count per node || -l ppn=<count> || --ntasks-per-node=<count>
|-
| Core count per process || || --cpus-per-task=<count>
|-
| Memory limit per node || -l mem=<limit> || --mem=<limit>
|-
| Memory limit per process || -l pmem=<limit> || --mem-per-cpu=<limit>
|-
| Job array || -t <array indices> || --array=<indices> (-a <indices>)
|-
| Node exclusive job || -l naccesspolicy=singlejob || --exclusive
|-
| Initial working directory || -d <directory> (default: $HOME) || --chdir=<directory> (-D <directory>) (default: submission directory)
|-
| Standard output file || -o <file path> || --output=<file> (-o <file>)
|-
| Standard error file || -e <file path> || --error=<file> (-e <file>)
|-
| Combine stdout/stderr to stdout || -j oe || --output=<combined stdout/stderr file>
|-
| Mail notification events || -m <event> || --mail-type=<events> (valid types include: NONE, BEGIN, END, FAIL, ALL)
|-
| Export environment to job || -V || --export=ALL (default)
|-
| Don't export environment to job || (default) || --export=NONE
|-
| Export environment variables to job || -v <var[=value][,var2=value2[, ...]]> || --export=<var[=value][,var2=value2[,...]]>
|}

'''Notes:'''
* Default initial job working directory is $HOME for MOAB. For Slurm the default working directory is where you submit your job from.
* By default MOAB does not export any environment variables to the job's runtime environment. With Slurm most of the login environment variables are exported to your job's runtime environment. This includes environment variables from software modules that were loaded at job submission time (and also $HOSTNAME variable).

'''Commonly used MOAB script environment variables and their Slurm equivalents

{| width=750px class="wikitable"
! Information !! MOAB !! Slurm
|-
| Job name || $MOAB_JOBNAME || $SLURM_JOB_NAME
|-
| Job ID || $MOAB_JOBID || $SLURM_JOB_ID
|-
| Submit directory || $MOAB_SUBMITDIR || $SLURM_SUBMIT_DIR
|-
| Number of nodes allocated || $MOAB_NODECOUNT || $SLURM_JOB_NUM_NODES (and: $SLURM_NNODES)
|-
| Node list || $MOAB_NODELIST || $SLURM_JOB_NODELIST
|-
| Number of processes || $MOAB_PROCCOUNT || $SLURM_NTASKS
|-
| Requested tasks per node || --- || $SLURM_NTASKS_PER_NODE
|-
| Requested CPUs per task || --- || $SLURM_CPUS_PER_TASK
|-
| Job array index || $MOAB_JOBARRAYINDEX || $SLURM_ARRAY_TASK_ID
|-
| Job array range || $MOAB_JOBARRAYRANGE || $SLURM_ARRAY_TASK_COUNT
|-
| Queue name || $MOAB_CLASS || $SLURM_JOB_PARTITION
|-
| QOS name || $MOAB_QOS || $SLURM_JOB_QOS
|-
| Number of processes per node || --- || $SLURM_TASKS_PER_NODE
|-
| Job user || $MOAB_USER || $SLURM_JOB_USER
|-
| Hostname || $MOAB_MACHINE || $SLURMD_NODENAME
|}

 

=Serial Programs=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-n 1''' or '''--ntasks=1''' (instead of '''-l nodes=1,ppn=1''').
* Use the option '''-m''' or '''--mem''' (instead of '''-l pmem'''). The default unit is MegaByte.
* If you want to use one node exclusively, you must enter the whole memory ('''-m 96327''' or '''--mem=96327''').
 
'''Example for a serial job'''
<pre>
$ sbatch -p single -t 60 -n 1 -m 96327 ./job.sh
</pre>
The script job.sh (containing the execution of a serial program) is started running 60 minutes exclusively on a batch node.
 
 
 

=Multithreaded Programs=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-N 1''' or '''--nodes=1''' and '''c ''x''''' or '''--cpus-per-task=''x''''' (instead of '''-l nodes=1,ppn=''x'' '''). '''''x''''' can be a number between 1 and 40 (because of 40 cores within one node); it can also be a number between 41 and 80 (because of active hyperthreading).
* Use the option '''-m''' or '''--mem''' (instead of '''-l pmem'''). The default unit is MegaByte.
* Use the option '''--export''' to set the needed environment variable OMP_NUM_THREADS for the batch job. Adding '''ALL''' means to pass all interactively set environment variables to the batch job.
* If you want to use one node exclusively, you must either enter the whole memory (''-m 96327'' or ''--mem=96327'') or set the number of threads greater than 39.
 
'''Example for a multithreaded job'''
<pre>
$ sbatch -p single -t 1:00:00 -N 1 -c 20 -m 50gb --export=ALL,OMP_NUM_THREADS=20 ./job_threaded.sh
</pre>
The script '''job_threaded.sh''' (containing a multithreaded program) is started running 1 hour in shared mode on 20 cores requesting 50GB on one batch node.
 
 
 

=MPI Parallel Programs within one node=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-n ''x''''' or '''--ntasks=''x''''' (instead of '''-l nodes=1,ppn=''x'' '''). '''''x''''' can be a number between 1 and 40 (because of 40 cores within one node); you should'nt utilize hyperthreading.
* Use the option '''-m''' or '''--mem''' (instead of '''-l pmem'''). The default unit is MegaByte.
* If you want to use one node exclusively, you must either enter the whole memory (''-m 96327'' or ''--mem=96327'') or set the number of MPI tasks greater than 39.
* Don't forget to load the appropriate MPI-module in your job script.
* If you are using OpenMPI, the options '''--bind-to core --map-by core|socket|node''' of the command mpirun should be used.
 
'''Example for a MPI job'''
<pre>
$ sbatch -p single -t 600 -n 10 -m 40000 ./job_mpi.sh
</pre>
The script '''job_mpi.sh''' (containing a MPI program after loading the appropriate MPI module) is started running 10 hours in shared mode on 10 cores requesting 40000 MB on one batch node.
 
 
 

=MPI Parallel Programs on many nodes=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-N ''y''''' or '''--nodes=''y''''' and '''--ntasks-per-node=''x''''' (instead of '''-l nodes=''y'',ppn=''x'''''). '''''x''''' can be a number between 1 and 40 (28 for Broadwell nodes) (because of 40 (28) cores within one node); you should'nt utilize hyperthreading.
* You should'nt use the option '''-m''' or '''--mem''' because the nodes are used exclusively.
* You always use the nodes exclusively.
* Don't forget to load the appropriate MPI-module in your job script.
* If you are using OpenMPI, the options '''--bind-to core --map-by core|socket|node''' ofthe command mpirun should be used.
 
'''Example for a MPI job'''
<pre>
$ sbatch -p multiple -t 48:00:00 -N 10 --ntasks-per-node=40 ./job_mpi.sh
</pre>
The script '''job_mpi.sh''' (containing a MPI program after loading the appropriate MPI module) is started running 2 days on 400 cores on ten batch nodes.
 
 
 

=Multithreaded + MPI Parallel Programs on many nodes=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-N ''y''''' or '''--nodes=''y''''' and '''--ntasks-per-node=''x''''' and '''-c ''z''''' or '''--cpus-per-task=''z''''' (instead of '''-l nodes=''y'',ppn=''x+z'''''). '''''x''''' usually should be 1 or 2 and '''''x+z''''' usually 40 (28 on Broadwell nodes); you can utilize hyperthreading if you want.
* You should'nt use the option '''-m''' or '''--mem''' because the nodes are used exclusively.
* You always use the nodes exclusively.
* Don't forget to load the appropriate MPI-module in your job script.
* If you are using OpenMPI, the options '''--bind-to core --map-by socket|node:PE=''z''''' of the command mpirun must be used.
 
'''Example for a MPI job'''
<pre>
$ sbatch -p multiple -t 2-12 -N 10 --ntasks-per-node=2 -c 20 ./job_threaded_mpi.sh
</pre>
The script '''job_threaded_mpi.sh''' (containing a multithreaded MPI program after loading the appropriate MPI module) is started running 2.5 days on 400 cores with 20 MPI tasks and 20 threads per task on ten batch nodes. Here the options '''--bind-to core --map-by socket:PE=10''' of the command mpirun must be used.

BwUniCluster 2.0 File System Migration Guide

2022-10-07T21:19:01Z

R Barthel: R Barthel moved page BwUniCluster 2.0 File System Migration Guide to BwUniCluster2.0/File System Migration Guide

#REDIRECT [[BwUniCluster2.0/File System Migration Guide]]

BwUniCluster2.0/File System Migration Guide

2022-10-07T21:19:01Z

R Barthel: R Barthel moved page BwUniCluster 2.0 File System Migration Guide to BwUniCluster2.0/File System Migration Guide

This page describes changes on the file systems between bwUniCluster 1 and bwUniCluster 2.0.

= Lustre file systems =

With bwUniCluster 2.0 new file systems for $HOME and workspaces with a total capacity of 5 PB and aggregate throughput of 72 GB/s have been purchased. The user data of the corresponding old file systems has been copied by KIT/SCC to these new file systems. This means that you should see the same data on $HOME and on workspaces as on bwUniCluster 1. The content of $WORK has to be migrated manually by the users to workspaces during the next 4 weeks, for details see below.

== Changes on $HOME and workspace directories ==

You should see the same data as before but if this is not true - which is possible in rare cases since the data of expired users has been cleaned up - please contact the hotline or open a ticket. The virtual path to $HOME (/home/ORG/GROUP/USER) is still the same (except for University of Stuttgart users, see below) but the physical path (which can be seen by ''/bin/pwd'') has changed. If one of your scripts includes a physical path (which is not recommended) you should modify these scripts. If applications complain about the old physical path (which started with ''/pfs/data1/'' or ''/pfs/data2/'') you should modify or recompile the application.

=== New user quota limits for $HOME ===

There are new user quota limits of 1 TiB and 10 million files on $HOME. Users which have currently more data have to reduce their data during the next 4 weeks. You can show your current usage and limits with the following command:
<pre>
$ lfs quota -uh $(whoami) $HOME
</pre>
Note that in addition to the user limit there is a limit of your organization which depends on the financial share. This limit existed before, too, but it is now enforced with so-called Lustre project quotas and before Lustre group quotas were used. You can show the current usage and limits of your organization with the following command:
<pre>
lfs quota -ph $(grep $(echo $HOME | sed -e "s|/[^/]*/[^/]*$||") /pfs/data5/project_ids.txt | cut -f 1 -d\ ) $HOME
</pre>
With the new quota limits there is no longer a need to check the quotas of other users of the main group. Hence the diskusage files will no longer be created.

=== New user quota limits for the workspace file system ===

There are new user quota limits of 40 TiB and 30 million files on the workspace file system. Users which have currently more data have to reduce their data during the next 4 weeks. The limits include released and expired workspaces which are internally deleted after few weeks. You can show your current usage and limits with the following command:
<pre>
$ lfs quota -uh $(whoami) /pfs/work7
</pre>
If it is really needed users can request higher limits by opening a ticket or sending an email to the hotline. However, they have to clearly describe why higher limits are needed, how long these extended limits are required, and why other storage opportunities cannot be used.

=== New stripe count settings for the workspace file system ===

For the workspace file system directories the new Lustre feature ''Progressive File Layouts'' has been used to define file striping parameters. This means that the stripe count is adapted if the file size is growing. In normal cases users no longer need to adapt file striping parameters in case they have very huge files or in order to reach better performance.

=== New group membership for users of University of Stuttgart ===

For users of University of Stuttgart the group membership has usually changed and the owning group for all data below $HOME and the old $WORK has been changed accordingly. This means that the virtual path to $HOME (/home/ORG/GROUP/USER) has changed, too. However, if you use environment variables (for example $HOME) in your scripts (which is recommended) nothing needs to be done. Otherwise you have to change your scripts accordingly.

== Hints for migration of $WORK data ==

Workspaces should be used instead of $WORK. Advantages of workspaces are a clear deletion policy and the ability to restore data from expired workspaces for few weeks. The old $WORK will be mounted in read-only mode for 4 weeks. If data on $WORK is still needed it has to be migrated by the users. Example for the migration to a workspace:
<pre>
# create a new workspace
$ ws_allocate newwork 60
# define environment variable for old WORK
OLDWORK=$WORK
# define environment variable for workspace
WORK=$(ws_find newwork)
# Copy the data with rsync
rsync -a -A $OLDWORK/ $WORK/
</pre>
Of course, if only a part of your old $WORK is needed you should only copy that part. As shown above you can redefine the WORK environment variable. For example, such a definition could be added to old scripts and then no further change on the script should be required.

The environment variable $WORK will exist and is set after login while the old $WORK file system is mounted during the next 4 weeks. Afterwards it will no longer exist or point to an empty string.

= LSDF Online Storage =

The LSDF Online Storage is now directly mounted on the Login- and Datamover-Nodes. Furthermore it can be used on the compute nodes during the job runtime with the constraint flag "LSDF" of the slurm batch system.

= $TMP =

All nodes of bwUniCluster 2.0 now have fast SSDs as fast local storage devices. These are especially useful if you have lots of small files or if you are doing random I/O operations (huge IOPS numbers) and if the data is only needed on the local node during the job runtime.

= BeeOND (BeeGFS On-Demand) =

Users now have possibility to request a private BeeOND (BeeGFS) parallel filesystem for each job with the constraint flag "BEEOND" of the slurm batch system. The file system is created during job startup and purged after your job. This is especially useful if you are doing huge amounts of I/O and/or if your I/O is not very efficient since there is no impact on the global Lustre file systems. However, you have to copy input data to this file system after job start and you have to copy data to a globally visible file system before the job completes.

BwUniCluster2.0/FAQ

2022-10-07T21:13:36Z

R Barthel:

BwUniCluster2.0/FAQ

2022-10-07T21:13:25Z

R Barthel: Created page with "Frequently asked questions (FAQs) concerning best practice of bwUniCluster 2.0 = Login = * In case of having issues with OTP, service password or denied access → BwU..."