bwHPC Wiki - User contributions [en]

BwUniCluster 2.0 User Access

2021-03-31T13:31:50Z

H Haefner: /* Step A: bwUniCluster entitlement for registration */

[[bwUniCluster_2.0|bwUniCluster 2.0]] is Baden-Württemberg's general purpose tier 3 high performance computing (HPC)
cluster co-financed by Baden-Württemberg's ministry of science, research and arts and the shareholders:

* Albert Ludwig University of Freiburg
* Eberhard Karls University, Tübingen
* Karlsruhe Institute of Technology (KIT)
* Heidelberg University (Ruprecht-Karls-Universität Heidelberg)
* Ulm University
* University of Hohenheim
* University of Konstanz
* University of Mannheim
* University of Stuttgart
* HAW BW e.V. (an association of several universities of applied sciences in Baden-Württemberg, see below)
 
To '''log on''' [[bwUniCluster_2.0|bwUniCluster 2.0]] a user account is required. All members of the shareholder
universities can apply for an account.
 

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:left; color:#000;vertical-align:top;" |__TOC__
| [[File:bwUniCluster_17Jan2014_p044-rot_t10.10.00.jpg|center|border|250px|bwUniCluster wiring by Holger Obermaier, copyright: KIT (SCC)]] bwUniCluster wiring © KIT (SCC)
|}

= Registration =

Granting access and issuing a user account for '''bwUniCluster 2.0''' requires the registration at the KIT service website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] (step B). However, this registration depends on the '''bwUniCluster entitlement''' issued by your university (step A).
 

== Step A: bwUniCluster entitlement for registration ==
'''The entitlement is called bwUniCluster (not bwUniCluster 2.0)''' and each university issues the bwUniCluster entitlement '''only''' for their own respective members. Some have established on-line processes or provide downloads of the entitlement application forms. If there is no link behind the name of an institution in the following list, please contact the local IT support services:

* [[BwCluster_User_Access_Uni_Freiburg|Albert Ludwig University of Freiburg]]
* [https://bwunicluster.urz.uni-heidelberg.de/ Heidelberg University]
* [https://kim.uni-hohenheim.de/bwhpc-account University of Hohenheim]
* [http://www.scc.kit.edu/downloads/ism/Accessform_bwUniCluster_DE_EN.pdf Karlsruhe Institute of Technology (KIT)]
* [[BWUniCluster_User_Access_Members_Uni_Konstanz|University of Konstanz]]
* [[BWUniCluster_User_Access_Members_Uni_Mannheim|University of Mannheim]]
* [https://www.hlrs.de/solutions-services/academic-users/bwunicluster-access/ University of Stuttgart]
* [https://uni-tuebingen.de/de/155157 Eberhard Karls University Tübingen]
* [[BWUniCluster_User_Access_Members_Uni_Ulm|Ulm University]]
* Hochschule Aalen
* Hochschule Albstadt-Sigmaringen
* Hochschule Esslingen
* Hochschule Heilbronn
* Hochschule Karlsruhe
* Hochschule Konstanz
* Hochschule Mannheim
* Hochschule Offenburg
* Hochschule Reutlingen
* Hochschule Rottenburg
* Hochschule Stuttgart (HfT)
* Hochschule Ulm
 

== Step B: Web Registration, service password and 2-factor authentication ==

After completing step A, i.e., after successfull issueing of the bwUniCluster entitlement, you have to register yourself for the service. To do so please visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and complete the following steps.

1. Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.

[[File:Bwidm-register-red.png|center|border|]]
 

2. You will be directed to the ''Identity Provider'' of your home organisation. Enter the user ID / username and password of your home organisation - this is usually the same password used for your e-mail account and other services - and click on '''Login''', '''Einloggen''' or something similar.
 

3. You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]. If you are logging into bwIDM for the first time, there will be a summary screen which shows the account details your home institution is providing to the central system. Please check that all data is valid and then click on '''Continue''' or '''Weiter'''.
 

4. Once you have successfully logged into the bwIDM system, you will be greeted by a home screen showing all state-wide services you have access to. There will be a box labelled "bwUniCluster". Click on '''Register''' or '''Registrieren''' to start the registration process.

[[File:Bwidm-2-red.png|center|border|]]
 

5. Since August 13, 2020 a '''2-factor authentication''' mechanism (2FA) is being enforced to improve security. If you have never registered a 2FA token on bwIDM before, the following error message will appear:

[[File:Bwidm-3-red.png|center|]]

Click on the [https://bwidm.scc.kit.edu/user/twofa.xhtml Link] or on the '''My Tokens''' link in the main menu. The instructions for registering a new 2FA token can be found on the following page: [[bwUniCluster 2.0 User Access/2FA Tokens]]. Please complete them before proceeding.
 

6. Make sure all requirements are met by checking the '''Requirements''' box at the top. If the requirements are not met you might be able to correct the issure by following the instructions. In all other cases please [[Registration_Support_-_bwUniCluster|contact your local hotline]].

[[File:BwUniCluster 2.0 access login bwidm registration requirements.png|center|border|]]
 

7. Read the Terms of Use ('''Nutzungsbedingungen und -richtlinien'''), check the box besides '''I have read and accepted the terms of use''' and click on '''Register''' or '''Registrieren'''.
 

8. Set a service password for the bwUniCluster and click on '''Save''' or '''Speichern'''. Logging in with the password of your home organisation, like on the former bwUniCluster 1, is no longer possible. Please make sure to use a strong password which is different from any other password you are currently using or have used on other systems. You will also be asked to change the service password regularly.

[[File:Bwidm-5-red.png|center|]]
 
 

== Step C: Fill out the bwUniCluster questionnaire ==

Filling out the bwUniCluster questionaire on

https://zas.bwhpc.de/shib/en/bwunicluster_survey.php

is mandatory for all users. The input is solely used to improve our support activities and for capacity planning of future HPC resources. '''If the questionaire is not filled out, access to bwUniCluster 2.0 is blocked 14 days after the registration.'''
 
 

== Changing the Service Password ==

Your bwUniCluster 2.0 '''password''' is the service password you set during the web registration (compare step 7 of chapter 1.2). At any time, you can set a new bwUniCluster 2.0 password via the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] by carrying out the following steps:
# Go to [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and select your home organization
# Authenticate yourself via the user id / username and password provided by your home institution
# Find the entry '''bwUniCluster''' and select '''Set Service Password'''
# Enter the new password, repeat it and click '''Save''' button.
# If the change was sucessfull, the message "Das Passwort wurde bei dem Dienst geändert" ("Password has been changed") will be shown
# Proceed to log in using the new password
 
 
== Contact / Support ==
If you have questions or problems concerning the bwUniCluster (2.0) registration, please [[bwUniCluster 2.0 Support|contact your local hotline]].
 
 

= Establishing network access =

Access to bwUniCluster 2.0 is '''limited to IP addresses from the so-called BelWü networks'''. All home institutions of our current users are connected to BelWue, so if you are on your campus network (e.g. in your office or on the Campus WiFi) you should be able to connect to bwUniCluster 2.0 without restrictions. If you are outside of one of the BelWue networks (e.g. in your home office instead of in your campus office), a VPN connection to your home institution has to be established first (see e.g. [1] for the KIT).
 
 

= Login =

After finishing the web registration and making sure that you are on a network from which you have access to bwUniCluster 2.0 (e.g. by establishing a VPN connection), the HPC cluster is ready for your '''SSH''' based login. Recommended SSH clients applications are:

* the ssh (OpenSSH) command included in all Linux distributions and macOS, -in command under Linux and macOS using the application ''terminal''
* [http://mobaxterm.mobatek.net/ MobaXterm] under Windows
 

== Hostnames ==

The main hostname required to connect to bwUniCluster 2.0 is '''bwunicluster.scc.kit.edu''' or '''uc2.scc.kit.edu'''. The system has four login nodes and we use so-called ''DNS round-robin scheduling'' to load-balance the incoming connections between the nodes. If you open multiple SSH sessions to bwUniCluster 2.0, these sessions will be established to different login nodes, so processes started in one session might not be visible in other sessions.

The older Broadwell extension partition of the former bwUniCluster 1 is connected to bwUniCluster 2.0. You can use the hostname '''uc1e.scc.kit.edu''' to connect to the login nodes of this partition.

If you need to connect to specific login nodes, you can use the following hostnames:

{| class="wikitable"
! Hostname !! Node type
|-
| '''uc2-login1.scc.kit.edu''' || bwUniCluster 2.0, first login node
|-
| '''uc2-login2.scc.kit.edu''' || bwUniCluster 2.0, second login node
|-
| '''uc2-login3.scc.kit.edu''' || bwUniCluster 2.0, third login node
|-
| '''uc2-login4.scc.kit.edu''' || bwUniCluster 2.0, fourth login node
|-
| '''uc1e-login1.scc.kit.edu''' || Broadwell partition, first login node
|-
| '''uc1e-login2.scc.kit.edu''' || Broadwell partition, second login node
|-
|}

Only the secure shell ''SSH'' is allowed to login. Other protocols like ''telnet'' or ''rlogin'' are not allowed for security reasons.
 

== Usernames ==

Your username will be the same as the one provided by your home institution, but '''prefixed''' with two characters and an underscore indicating your home institution. For example: If you are a member of the university of Konstanz and your local username is ab1234, your username on bwUniCluster 2.0 is kn_ab1234.

The following list contains all prefixes currently in use:

{| class="wikitable"
! Home organization !! <UserID>
|-
| Universität Freiburg || ''fr_''username
|-
| Universität Heidelberg || ''hd_''username
|-
| Universität Hohenheim || ''ho_''username
|-
| KIT || username ''(without any prefix)''
|-
| Universität Konstanz || ''kn_''username
|-
| Universität Mannheim || ''ma_''username
|-
| Universität Stuttgart || ''st_''username
|-
| Universität Tübingen || ''tu_''username
|-
| Universität Ulm || ''ul_''username
|-
| Hochschule Aalen || ''aa_''username
|-
| Hochschule Albstadt-Sigmaringen || ''as_''username
|-
| Hochschule Esslingen || ''es_''username
|-
| Hochschule Heilbronn || ''hn_''username
|-
| Hochschule Karlsruhe || ''hk_''username
|-
| HTWG Konstanz || ''ht_''username
|-
| Hochschule Mannheim || ''mn_''username
|-
| Hochschule Offenburg || ''of_''username
|-
| Hochschule Reutlingen || ''hr_''username
|-
| Hochschule Rottenburg || ''ro_''username
|-
| Hochschule für Technik Stuttgart || ''hs_''username
|-
| Hochschule Ulm || ''hu_''username
|-
|}
 

== Client application: OpenSSH ==

Most Unix and Unix-like operating systems like Linux, macOS and *BSD come with a built-in SSH client provided by the OpenSSH project. More recent versions of Windows 10 and the Windows Subsystem for Linux also come with a built-in OpenSSH client.

To use this client, simply open a command line terminal (the exact process differs on every operating system, but usually involves starting an application called '''Terminal''' or '''Command Prompt''') and enter the following command to connect to bwUniCluster 2.0:

<pre>
$ ssh <UserID>@bwunicluster.scc.kit.edu
</pre>

If you are on a Linux or Unix system running the X Window System (X11) and want to use a GUI-based application on bwUniCluster 2.0, you can use the ''-X'' option for the ssh command to set up X11 forwarding:

<pre>
$ ssh -X <UserID>@uc2.scc.kit.edu
</pre>

Windows users requiring X11 forwarding for graphical applications should use '''MobaXterm''' instead.
 

== Client application: MobaXterm ==

The bwHPC-C5 support team strongly recommends to use [http://mobaxterm.mobatek.net/ MobaXterm] instead of ''PuTTY'' or ''WinSCP'' on Windows. ''MobaXterm'' provides a built-in X11 server allowing to start GUI based software.

Start ''MobaXterm'', fill in the following fields:
<pre>
Remote name : uc2.scc.kit.edu # or uc1e.scc.kit.edu
Specify user name : <UserID>
Port : 22
</pre>

After that click on 'ok'. Then a terminal will be opened and there you can enter your credentials.
 

== Example login process ==

After the connection has been initiated, a successful login process will go through the following three steps:

1. The system asks for a '''One-Time Password'''. Generate one using the Software or Hardware Token registered on the bwIDM system (see [[bwUniCluster 2.0 User Access/2FA Tokens]]) and enter it after the '''Your OTP:''' prompt.

2. The systems asks for your service password. Enter it after the '''Password:''' prompt.

3. You are greeted by the bwUniCluster 2.0 banner followed by a shell.

The result should look like this:

[[File:BwUniCluster 2.0 access login example.png|center|]]
 

== Troubleshooting ==

'''Issue: The "Your OTP:" prompt never appears and the connection hangs/times out instead'''

Likely cause: You are most likely not on a network from which access to the bwUniCluster 2.0 system is allowed. Please check if you might have to establish a VPN connection first.

 

'''Issue: The system asks for the One-Time Password multiple times'''

Likely cause: Make sure you are using the correct Software Token to generate the One-Time Password.

 

'''Issue: The system asks for the service password multiple times'''

Likely cause: Make sure you are using the service password set on bwIDM and not the password valid for your home institution. Unlike the bwUniCluster 1, the bwUniCluster 2.0 only accepts the service password.

 

'''Issue: There is an error message by the pam_ses_open.sh skript'''

Likely cause: Your account is in the "LOST_ACCESS" state because the entitlement is no longer valid, the questionaire was not filled out or there was a problem during the communication between your home institution and the central bwIDM system. Please try the following steps:

* Log into [https://bwidm.scc.kit.edu bwIDM], look for the bwUniCluster entry and click on '''Registry info'''. Your "Status:" should be "ACTIVE". If it is not, please wait for ten minutes since logging into the bwIDM causes a refresh and the problem might fix itself. If the status does not change to ACTIVE after a longer amount of time, please contact the support channels.

* If you have not filled out the questionaire, please do so on [https://zas.bwhpc.de/shib/en/bwunicluster_survey.php https://zas.bwhpc.de/shib/en/bwunicluster_survey.php] and then wait for about ten minutes before attempting to log into the HPC system again.
 
 

== Allowed activities on login nodes ==

The login nodes of bwUniCluster 2.0 are the access point to the compute system and to your bwUniCluster 2.0 $HOME directory. The login nodes are shared with all the users of bwUniCluster 2.0. Therefore, your activities on the login nodes are limited to primarily set up your batch jobs. Your activities may also be:

* '''short''' compilation of your program code and
* '''short''' pre- and postprocessing of your batch jobs.

To guarantee usability for all the users of bwUniCluster 2.0 '''you must not run your compute jobs on the login nodes'''. Compute jobs must be submitted to the
[[bwUniCluster Batch Jobs|queueing system]]. Any compute job running on the login nodes will be terminated without any notice. Any long-running compilation or any long-running pre- or postprocessing of batch jobs must also be submitted to the [[bwUniCluster Batch Jobs|queueing system]].
 
 

== SSH Keys ==

In contrast to the bwUniCluster 1 and many other HPC systems it is '''no longer possible to self-manage your SSH Keys by adding them to the ~/.ssh/authorized_keys file'''. Existing files will no longer be evaluated. SSH Keys have to be managed via the central bwIDM system instead. Please refer to the user guide for this functionality:

[[bwUniCluster 2.0 User Access/SSH Keys]]
 
 

= [[First_Steps_on_bwHPC_cluster|First steps on bwUniCluster]] =

First and some important steps on bwUniCluster 2.0 can be found [[First_Steps_on_bwHPC_cluster|here]].
 
 

= Deregistration =
If you plan to permanently leave the bwUniCluster 2.0, follow the deregister checklist:
# Transfer all your data in $HOME and workspace to your local computer/storage and after that clear off all your data
# Visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
#* Select your home organization from the list and click '''Proceed'''
#* Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
#* You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
#* <div>Select '''Registry Info''' of the service '''bwUniCluster''' (on the left hand side) [[File:bwUniCluster_registration_sidebar.png|center|border|]]</div>
#* Click '''Deregister'''
Note that Step 2 will automatically unsubscribe you from the bwunicluster mail list.

BwUniCluster2.0/Batch Queues

2020-12-18T09:38:14Z

H Haefner: /* sbatch -p queue */

This article contains information on the queues of the [[BwUniCluster_2.0_Slurm_common_Features|batch job system]] and on interactive jobs.

= Job Submission =
== sbatch Command ==

=== sbatch -p ''queue'' ===
Compute resources such as (wall-)time, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, '''you must add the correct queue class to your sbatch command '''. The specification of a queue is obligatory on BwUniCluster 2.0.
 
Details are:

{| width=750px class="wikitable"
! colspan="5" | bwUniCluster 2.0 sbatch -p ''queue''
|- style="text-align:left;"
! queue !! node !! default resources !! minimum resources !! maximum resources
|- style="text-align:left"
| dev_single
| thin
| time=10, mem-per-cpu=1125mb
|
| time=30, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core=2) 6 nodes are reserved for this queue. Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| single
| thin
| time=30, mem-per-cpu=1125mb
|
| time=72:00:00, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core)=2
|- style="text-align:left;"
| dev_multiple
| thin
| time=10, mem-per-cpu=1125mb
| nodes=2
| time=30, nodes=4, mem=90000mb, ntasks-per-node=40, (threads-per-core=2) 8 nodes are reserved for this queue. Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| multiple
| thin
| time=30, mem-per-cpu=1125mb
| nodes=2
| time=72:00:00, mem=90000mb, nodes=128, ntasks-per-node=40, (threads-per-core=2)
|- style="text-align:left;"
| dev_multiple_e
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
| nodes=2
| time=30, nodes=8, mem=122000, ntasks-per-node=28, (threads-per-core=2) 8 nodes are reserved for this queue Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| multiple_e
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
| nodes=2
| time=72:00:00, nodes=128, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="text-align:left;"
| dev_special (restricted access!)
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
|
| time=30, nodes=1, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="text-align:left;"
| special (restricted access!)
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
|
| time=72:00:00, nodes=1, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
| fat
| fat
| time=10, mem-per-cpu=18750mb
|
| time=72:00:00, nodes=1, ntasks-per-node=80, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
| dev_gpu_4
| gpu_4
| time=10, mem-per-gpu=94000mb, cpu-per-gpu=20
|
| time=30, nodes=1, mem=376000, ntasks-per-node=40, (threads-per-core=2) 1 node is reserved for this queue Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| gpu_4
| gpu4
| time=10, mem-per-gpu=94000mb, cpu-per-gpu=20
|
| time=48:00:00, mem=376000, nodes=14, ntasks-per-node=40, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
| gpu_8
| gpu8
| time=10, mem-per-cpu=94000mb, cpu-per-gpu=10
|
| time=48:00:00, mem=752000, nodes=10, ntasks-per-node=40, (threads-per-core=2)
|-
|}

Default resources of a queue class defines time, #tasks and memory if not explicitly given with sbatch command. Resource list acronyms ''--time'', ''--ntasks'', ''--nodes'', ''--mem'' and ''--mem-per-cpu'' are described [[ForHLR_Batch_Jobs_SLURM#sbatch_Command_Parameters|here]].

Access to the "special" and "dev_special" partitions on the bwUniCluster 2.0 is restricted to members of the institutions which participated in the procurement of the extension partition specifically for this purpose. Please contact the support team if your institution participated in the procurement and your account should be able to run jobs in this partition.
 
 
==== Queue class examples ====
To run your batch job on one of the thin nodes, please use:

<pre>
$ sbatch --partition=dev_multiple
or
$ sbatch -p dev_multiple
</pre>
 

==== Interactive Jobs ====
On bwUniCluster 2.0 you are only allowed to run short jobs (<< 1 hour) with little memory requirements (<< 8 GByte) on the logins nodes. If you want to run longer jobs and/or jobs with a request of more than 8 GByte of memory, you must allocate resources for so-called interactive jobs by usage of the command salloc on a login node. Considering a serial application running on a compute node that requires 5000 MByte of memory and limiting the interactive run to 2 hours the following command has to be executed:
<pre>
$ salloc -p single -n 1 -t 120 --mem=5000
</pre>
Then you will get one core on a compute node within the partition "single". After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system Slurm has granted you the requested resources on the compute system. You will be logged in automatically on the granted core! To run a serial program on the granted core you only have to type the name of the executable.
<pre>
$ ./<my_serial_program>
</pre>
Please be aware that your serial job must run less than 2 hours in this example, else the job will be killed during runtime by the system.
 
You can also start now a graphical X11-terminal connecting you to the dedicated resource that is available for 2 hours. You can start it by the command:
<pre>
$ xterm
</pre>
Note that, once the walltime limit has been reached the resources - i.e. the compute node - will automatically be revoked.
 
An interactive parallel application running on one compute node or on many compute nodes (e.g. here 5 nodes) with 40 cores each requires usually an amount of memory in GByte (e.g. 50 GByte) and a maximum time (e.g. 1 hour). E.g. 5 nodes can be allocated by the following command:
<pre>
$ salloc -p multiple -N 5 --ntasks-per-node=40 -t 01:00:00 --mem=50gb
</pre>
Now you can run parallel jobs on 200 cores requiring 50 GByte of memory per node. Please be aware that you will be logged in on core 0 of the first node. If you want to run MPI-programs, you can do it by simply typing mpirun <program_name>. Then your program will be run on 200 cores. A very simple example for starting a parallel job can be:
<pre>
$ mpirun <my_mpi_program>
</pre>
You can also start the debugger ddt by the commands:
<pre>
$ module add devel/ddt
$ ddt <my_mpi_program>
</pre>
The above commands will execute the parallel program <my_mpi_program> on all available cores. You can also start parallel programs on a subset of cores; an example for this can be:
<pre>
$ mpirun -n 50 <my_mpi_program>
</pre>
If you are using Intel MPI you must start <my_mpi_program> by the command mpiexec.hydra (instead of mpirun).
 
 

----
[[Category:bwUniCluster 2.0|Batch Jobs - bwUniCluster 2.0 Features]]

BwUniCluster2.0/Batch Queues

2020-12-18T09:37:10Z

H Haefner: /* sbatch -p queue */

This article contains information on the queues of the [[BwUniCluster_2.0_Slurm_common_Features|batch job system]] and on interactive jobs.

= Job Submission =
== sbatch Command ==

=== sbatch -p ''queue'' ===
Compute resources such as (wall-)time, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, '''you must add the correct queue class to your sbatch command '''. The specification of a queue is obligatory on BwUniCluster 2.0.
 
Details are:

{| width=750px class="wikitable"
! colspan="5" | bwUniCluster 2.0 sbatch -p ''queue''
|- style="text-align:left;"
! queue !! node !! default resources !! minimum resources !! maximum resources
|- style="text-align:left"
| dev_single
| thin
| time=10, mem-per-cpu=1125mb
|
| time=30, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core=2) 6 nodes are reserved for this queue. Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| single
| thin
| time=30, mem-per-cpu=1125mb
|
| time=72:00:00, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core)=2
|- style="text-align:left;"
| dev_multiple
| thin
| time=10, mem-per-cpu=1125mb
| nodes=2
| time=30, nodes=4, mem=90000mb, ntasks-per-node=40, (threads-per-core=2) 8 nodes are reserved for this queue. Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| multiple
| thin
| time=30, mem-per-cpu=1125mb
| nodes=2
| time=72:00:00, mem=90000mb, nodes=128, ntasks-per-node=40, (threads-per-core=2)
|- style="text-align:left;"
| dev_multiple_e
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
| nodes=2
| time=30, nodes=8, mem=122000, ntasks-per-node=28, (threads-per-core=2) 8 nodes are reserved for this queue Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| multiple_e
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
| nodes=2
| time=72:00:00, nodes=128, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="text-align:left;"
| dev_special (restricted access!)
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
|
| time=30, nodes=1, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="text-align:left;"
| special (restricted access!)
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
|
| time=72:00:00, nodes=1, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
| fat
| fat
| time=10, mem-per-cpu=18750mb
|
| time=72:00:00, nodes=1, ntasks-per-node=80, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
| dev_gpu_4
| gpu_4
| time=10, mem-per-gpu=94000mb, cpu-per-gpu=20
|
| time=30, nodes=1, mem=376000, ntasks-per-node=40, (threads-per-core=2) 8 node is reserved for this queue Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| gpu_4
| gpu4
| time=10, mem-per-gpu=94000mb, cpu-per-gpu=20
|
| time=48:00:00, mem=376000, nodes=14, ntasks-per-node=40, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
| gpu_8
| gpu8
| time=10, mem-per-cpu=94000mb, cpu-per-gpu=10
|
| time=48:00:00, mem=752000, nodes=10, ntasks-per-node=40, (threads-per-core=2)
|-
|}

Default resources of a queue class defines time, #tasks and memory if not explicitly given with sbatch command. Resource list acronyms ''--time'', ''--ntasks'', ''--nodes'', ''--mem'' and ''--mem-per-cpu'' are described [[ForHLR_Batch_Jobs_SLURM#sbatch_Command_Parameters|here]].

Access to the "special" and "dev_special" partitions on the bwUniCluster 2.0 is restricted to members of the institutions which participated in the procurement of the extension partition specifically for this purpose. Please contact the support team if your institution participated in the procurement and your account should be able to run jobs in this partition.
 
 
==== Queue class examples ====
To run your batch job on one of the thin nodes, please use:

<pre>
$ sbatch --partition=dev_multiple
or
$ sbatch -p dev_multiple
</pre>
 

==== Interactive Jobs ====
On bwUniCluster 2.0 you are only allowed to run short jobs (<< 1 hour) with little memory requirements (<< 8 GByte) on the logins nodes. If you want to run longer jobs and/or jobs with a request of more than 8 GByte of memory, you must allocate resources for so-called interactive jobs by usage of the command salloc on a login node. Considering a serial application running on a compute node that requires 5000 MByte of memory and limiting the interactive run to 2 hours the following command has to be executed:
<pre>
$ salloc -p single -n 1 -t 120 --mem=5000
</pre>
Then you will get one core on a compute node within the partition "single". After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system Slurm has granted you the requested resources on the compute system. You will be logged in automatically on the granted core! To run a serial program on the granted core you only have to type the name of the executable.
<pre>
$ ./<my_serial_program>
</pre>
Please be aware that your serial job must run less than 2 hours in this example, else the job will be killed during runtime by the system.
 
You can also start now a graphical X11-terminal connecting you to the dedicated resource that is available for 2 hours. You can start it by the command:
<pre>
$ xterm
</pre>
Note that, once the walltime limit has been reached the resources - i.e. the compute node - will automatically be revoked.
 
An interactive parallel application running on one compute node or on many compute nodes (e.g. here 5 nodes) with 40 cores each requires usually an amount of memory in GByte (e.g. 50 GByte) and a maximum time (e.g. 1 hour). E.g. 5 nodes can be allocated by the following command:
<pre>
$ salloc -p multiple -N 5 --ntasks-per-node=40 -t 01:00:00 --mem=50gb
</pre>
Now you can run parallel jobs on 200 cores requiring 50 GByte of memory per node. Please be aware that you will be logged in on core 0 of the first node. If you want to run MPI-programs, you can do it by simply typing mpirun <program_name>. Then your program will be run on 200 cores. A very simple example for starting a parallel job can be:
<pre>
$ mpirun <my_mpi_program>
</pre>
You can also start the debugger ddt by the commands:
<pre>
$ module add devel/ddt
$ ddt <my_mpi_program>
</pre>
The above commands will execute the parallel program <my_mpi_program> on all available cores. You can also start parallel programs on a subset of cores; an example for this can be:
<pre>
$ mpirun -n 50 <my_mpi_program>
</pre>
If you are using Intel MPI you must start <my_mpi_program> by the command mpiexec.hydra (instead of mpirun).
 
 

----
[[Category:bwUniCluster 2.0|Batch Jobs - bwUniCluster 2.0 Features]]

BwUniCluster2.0/Batch Queues

2020-12-18T09:25:46Z

H Haefner: /* sbatch -p queue */

This article contains information on the queues of the [[BwUniCluster_2.0_Slurm_common_Features|batch job system]] and on interactive jobs.

= Job Submission =
== sbatch Command ==

=== sbatch -p ''queue'' ===
Compute resources such as (wall-)time, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, '''you must add the correct queue class to your sbatch command '''. The specification of a queue is obligatory on BwUniCluster 2.0.
 
Details are:

{| width=750px class="wikitable"
! colspan="5" | bwUniCluster 2.0 sbatch -p ''queue''
|- style="text-align:left;"
! queue !! node !! default resources !! minimum resources !! maximum resources
|- style="text-align:left"
| dev_single
| thin
| time=10, mem-per-cpu=1125mb
|
| time=30, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core=2) 6 nodes are reserved for this queue. Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| single
| thin
| time=30, mem-per-cpu=1125mb
|
| time=72:00:00, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core)=2
|- style="text-align:left;"
| dev_multiple
| thin
| time=10, mem-per-cpu=1125mb
| nodes=2
| time=30, nodes=4, mem=90000mb, ntasks-per-node=40, (threads-per-core=2) 8 nodes are reserved for this queue. Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| multiple
| thin
| time=30, mem-per-cpu=1125mb
| nodes=2
| time=72:00:00, mem=90000mb, nodes=128, ntasks-per-node=40, (threads-per-core=2)
|- style="text-align:left;"
| dev_multiple_e
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
| nodes=2
| time=30, nodes=8, mem=122000, ntasks-per-node=28, (threads-per-core=2) 8 nodes are reserved for this queue Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| multiple_e
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
| nodes=2
| time=72:00:00, nodes=128, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="text-align:left;"
| dev_special (restricted access!)
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
|
| time=30, nodes=1, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="text-align:left;"
| special (restricted access!)
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
|
| time=72:00:00, nodes=1, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
| fat
| fat
| time=10, mem-per-cpu=18750mb
|
| time=72:00:00, nodes=1, ntasks-per-node=80, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
| dev_gpu_4
| gpu_4
| time=10, mem-per-cpu=2178mb
|
| time=30, nodes=8, mem=122000, ntasks-per-node=28, (threads-per-core=2) 8 nodes are reserved for this queue Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| gpu_4
| gpu4
| time=10, mem-per-gpu=94000mb, cpu-per-gpu=20
|
| time=48:00:00, mem=376000, nodes=14, ntasks-per-node=40, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
| gpu_8
| gpu8
| time=10, mem-per-cpu=94000mb, cpu-per-gpu=10
|
| time=48:00:00, mem=752000, nodes=10, ntasks-per-node=40, (threads-per-core=2)
|-
|}

Default resources of a queue class defines time, #tasks and memory if not explicitly given with sbatch command. Resource list acronyms ''--time'', ''--ntasks'', ''--nodes'', ''--mem'' and ''--mem-per-cpu'' are described [[ForHLR_Batch_Jobs_SLURM#sbatch_Command_Parameters|here]].

Access to the "special" and "dev_special" partitions on the bwUniCluster 2.0 is restricted to members of the institutions which participated in the procurement of the extension partition specifically for this purpose. Please contact the support team if your institution participated in the procurement and your account should be able to run jobs in this partition.
 
 
==== Queue class examples ====
To run your batch job on one of the thin nodes, please use:

<pre>
$ sbatch --partition=dev_multiple
or
$ sbatch -p dev_multiple
</pre>
 

==== Interactive Jobs ====
On bwUniCluster 2.0 you are only allowed to run short jobs (<< 1 hour) with little memory requirements (<< 8 GByte) on the logins nodes. If you want to run longer jobs and/or jobs with a request of more than 8 GByte of memory, you must allocate resources for so-called interactive jobs by usage of the command salloc on a login node. Considering a serial application running on a compute node that requires 5000 MByte of memory and limiting the interactive run to 2 hours the following command has to be executed:
<pre>
$ salloc -p single -n 1 -t 120 --mem=5000
</pre>
Then you will get one core on a compute node within the partition "single". After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system Slurm has granted you the requested resources on the compute system. You will be logged in automatically on the granted core! To run a serial program on the granted core you only have to type the name of the executable.
<pre>
$ ./<my_serial_program>
</pre>
Please be aware that your serial job must run less than 2 hours in this example, else the job will be killed during runtime by the system.
 
You can also start now a graphical X11-terminal connecting you to the dedicated resource that is available for 2 hours. You can start it by the command:
<pre>
$ xterm
</pre>
Note that, once the walltime limit has been reached the resources - i.e. the compute node - will automatically be revoked.
 
An interactive parallel application running on one compute node or on many compute nodes (e.g. here 5 nodes) with 40 cores each requires usually an amount of memory in GByte (e.g. 50 GByte) and a maximum time (e.g. 1 hour). E.g. 5 nodes can be allocated by the following command:
<pre>
$ salloc -p multiple -N 5 --ntasks-per-node=40 -t 01:00:00 --mem=50gb
</pre>
Now you can run parallel jobs on 200 cores requiring 50 GByte of memory per node. Please be aware that you will be logged in on core 0 of the first node. If you want to run MPI-programs, you can do it by simply typing mpirun <program_name>. Then your program will be run on 200 cores. A very simple example for starting a parallel job can be:
<pre>
$ mpirun <my_mpi_program>
</pre>
You can also start the debugger ddt by the commands:
<pre>
$ module add devel/ddt
$ ddt <my_mpi_program>
</pre>
The above commands will execute the parallel program <my_mpi_program> on all available cores. You can also start parallel programs on a subset of cores; an example for this can be:
<pre>
$ mpirun -n 50 <my_mpi_program>
</pre>
If you are using Intel MPI you must start <my_mpi_program> by the command mpiexec.hydra (instead of mpirun).
 
 

----
[[Category:bwUniCluster 2.0|Batch Jobs - bwUniCluster 2.0 Features]]

BwUniCluster2.0/Batch Queues

2020-12-18T09:24:30Z

H Haefner: /* sbatch -p queue */

This article contains information on the queues of the [[BwUniCluster_2.0_Slurm_common_Features|batch job system]] and on interactive jobs.

= Job Submission =
== sbatch Command ==

=== sbatch -p ''queue'' ===
Compute resources such as (wall-)time, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, '''you must add the correct queue class to your sbatch command '''. The specification of a queue is obligatory on BwUniCluster 2.0.
 
Details are:

{| width=750px class="wikitable"
! colspan="5" | bwUniCluster 2.0 sbatch -p ''queue''
|- style="text-align:left;"
! queue !! node !! default resources !! minimum resources !! maximum resources
|- style="text-align:left"
| dev_single
| thin
| time=10, mem-per-cpu=1125mb
|
| time=30, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core=2) 6 nodes are reserved for this queue. Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| single
| thin
| time=30, mem-per-cpu=1125mb
|
| time=72:00:00, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core)=2
|- style="text-align:left;"
| dev_multiple
| thin
| time=10, mem-per-cpu=1125mb
| nodes=2
| time=30, nodes=4, mem=90000mb, ntasks-per-node=40, (threads-per-core=2) 8 nodes are reserved for this queue. Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| multiple
| thin
| time=30, mem-per-cpu=1125mb
| nodes=2
| time=72:00:00, mem=90000mb, nodes=128, ntasks-per-node=40, (threads-per-core=2)
|- style="text-align:left;"
| dev_multiple_e
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
| nodes=2
| time=30, nodes=8, mem=122000, ntasks-per-node=28, (threads-per-core=2) 8 nodes are reserved for this queue Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| multiple_e
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
| nodes=2
| time=72:00:00, nodes=128, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="text-align:left;"
| dev_special (restricted access!)
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
|
| time=30, nodes=1, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="text-align:left;"
| special (restricted access!)
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
|
| time=72:00:00, nodes=1, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
| fat
| fat
| time=10, mem-per-cpu=18750mb
| nodes=1
| time=72:00:00, nodes=1, ntasks-per-node=80, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
| dev_gpu_4
| gpu_4
| time=10, mem-per-cpu=2178mb
| nodes=2
| time=30, nodes=8, mem=122000, ntasks-per-node=28, (threads-per-core=2) 8 nodes are reserved for this queue Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| gpu_4
| gpu4
| time=10, mem-per-gpu=94000mb, cpu-per-gpu=20
|
| time=48:00:00, mem=376000, nodes=14, ntasks-per-node=40, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
| gpu_8
| gpu8
| time=10, mem-per-cpu=94000mb, cpu-per-gpu=10
|
| time=48:00:00, mem=752000, nodes=10, ntasks-per-node=40, (threads-per-core=2)
|-
|}

Default resources of a queue class defines time, #tasks and memory if not explicitly given with sbatch command. Resource list acronyms ''--time'', ''--ntasks'', ''--nodes'', ''--mem'' and ''--mem-per-cpu'' are described [[ForHLR_Batch_Jobs_SLURM#sbatch_Command_Parameters|here]].

Access to the "special" and "dev_special" partitions on the bwUniCluster 2.0 is restricted to members of the institutions which participated in the procurement of the extension partition specifically for this purpose. Please contact the support team if your institution participated in the procurement and your account should be able to run jobs in this partition.
 
 
==== Queue class examples ====
To run your batch job on one of the thin nodes, please use:

<pre>
$ sbatch --partition=dev_multiple
or
$ sbatch -p dev_multiple
</pre>
 

==== Interactive Jobs ====
On bwUniCluster 2.0 you are only allowed to run short jobs (<< 1 hour) with little memory requirements (<< 8 GByte) on the logins nodes. If you want to run longer jobs and/or jobs with a request of more than 8 GByte of memory, you must allocate resources for so-called interactive jobs by usage of the command salloc on a login node. Considering a serial application running on a compute node that requires 5000 MByte of memory and limiting the interactive run to 2 hours the following command has to be executed:
<pre>
$ salloc -p single -n 1 -t 120 --mem=5000
</pre>
Then you will get one core on a compute node within the partition "single". After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system Slurm has granted you the requested resources on the compute system. You will be logged in automatically on the granted core! To run a serial program on the granted core you only have to type the name of the executable.
<pre>
$ ./<my_serial_program>
</pre>
Please be aware that your serial job must run less than 2 hours in this example, else the job will be killed during runtime by the system.
 
You can also start now a graphical X11-terminal connecting you to the dedicated resource that is available for 2 hours. You can start it by the command:
<pre>
$ xterm
</pre>
Note that, once the walltime limit has been reached the resources - i.e. the compute node - will automatically be revoked.
 
An interactive parallel application running on one compute node or on many compute nodes (e.g. here 5 nodes) with 40 cores each requires usually an amount of memory in GByte (e.g. 50 GByte) and a maximum time (e.g. 1 hour). E.g. 5 nodes can be allocated by the following command:
<pre>
$ salloc -p multiple -N 5 --ntasks-per-node=40 -t 01:00:00 --mem=50gb
</pre>
Now you can run parallel jobs on 200 cores requiring 50 GByte of memory per node. Please be aware that you will be logged in on core 0 of the first node. If you want to run MPI-programs, you can do it by simply typing mpirun <program_name>. Then your program will be run on 200 cores. A very simple example for starting a parallel job can be:
<pre>
$ mpirun <my_mpi_program>
</pre>
You can also start the debugger ddt by the commands:
<pre>
$ module add devel/ddt
$ ddt <my_mpi_program>
</pre>
The above commands will execute the parallel program <my_mpi_program> on all available cores. You can also start parallel programs on a subset of cores; an example for this can be:
<pre>
$ mpirun -n 50 <my_mpi_program>
</pre>
If you are using Intel MPI you must start <my_mpi_program> by the command mpiexec.hydra (instead of mpirun).
 
 

----
[[Category:bwUniCluster 2.0|Batch Jobs - bwUniCluster 2.0 Features]]

BwUniCluster2.0/Batch Queues

2020-12-18T09:23:59Z

H Haefner: /* sbatch -p queue */

This article contains information on the queues of the [[BwUniCluster_2.0_Slurm_common_Features|batch job system]] and on interactive jobs.

= Job Submission =
== sbatch Command ==

=== sbatch -p ''queue'' ===
Compute resources such as (wall-)time, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, '''you must add the correct queue class to your sbatch command '''. The specification of a queue is obligatory on BwUniCluster 2.0.
 
Details are:

{| width=750px class="wikitable"
! colspan="5" | bwUniCluster 2.0 sbatch -p ''queue''
|- style="text-align:left;"
! queue !! node !! default resources !! minimum resources !! maximum resources
|- style="text-align:left"
| dev_single
| thin
| time=10, mem-per-cpu=1125mb
|
| time=30, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core=2) 6 nodes are reserved for this queue. Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| single
| thin
| time=30, mem-per-cpu=1125mb
|
| time=72:00:00, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core)=2
|- style="text-align:left;"
| dev_multiple
| thin
| time=10, mem-per-cpu=1125mb
| nodes=2
| time=30, nodes=4, mem=90000mb, ntasks-per-node=40, (threads-per-core=2) 8 nodes are reserved for this queue. Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| multiple
| thin
| time=30, mem-per-cpu=1125mb
| nodes=2
| time=72:00:00, mem=90000mb, nodes=128, ntasks-per-node=40, (threads-per-core=2)
|- style="text-align:left;"
| dev_multiple_e
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
| nodes=2
| time=30, nodes=8, mem=122000, ntasks-per-node=28, (threads-per-core=2) 8 nodes are reserved for this queue Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| multiple_e
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
| nodes=2
| time=72:00:00, nodes=128, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="text-align:left;"
| dev_special (restricted access!)
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
|
| time=30, nodes=1, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="text-align:left;"
| special (restricted access!)
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
|
| time=72:00:00, nodes=1, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
| fat
| fat
| time=10, mem-per-cpu=18750mb
| nodes=1
| time=72:00:00, nodes=1, ntasks-per-node=80, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
|
| dev_gpu_4
| gpu_4
| time=10, mem-per-cpu=2178mb
| nodes=2
| time=30, nodes=8, mem=122000, ntasks-per-node=28, (threads-per-core=2) 8 nodes are reserved for this queue Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| gpu_4
| gpu4
| time=10, mem-per-gpu=94000mb, cpu-per-gpu=20
|
| time=48:00:00, mem=376000, nodes=14, ntasks-per-node=40, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
| gpu_8
| gpu8
| time=10, mem-per-cpu=94000mb, cpu-per-gpu=10
|
| time=48:00:00, mem=752000, nodes=10, ntasks-per-node=40, (threads-per-core=2)
|-
|}

Default resources of a queue class defines time, #tasks and memory if not explicitly given with sbatch command. Resource list acronyms ''--time'', ''--ntasks'', ''--nodes'', ''--mem'' and ''--mem-per-cpu'' are described [[ForHLR_Batch_Jobs_SLURM#sbatch_Command_Parameters|here]].

Access to the "special" and "dev_special" partitions on the bwUniCluster 2.0 is restricted to members of the institutions which participated in the procurement of the extension partition specifically for this purpose. Please contact the support team if your institution participated in the procurement and your account should be able to run jobs in this partition.
 
 
==== Queue class examples ====
To run your batch job on one of the thin nodes, please use:

<pre>
$ sbatch --partition=dev_multiple
or
$ sbatch -p dev_multiple
</pre>
 

==== Interactive Jobs ====
On bwUniCluster 2.0 you are only allowed to run short jobs (<< 1 hour) with little memory requirements (<< 8 GByte) on the logins nodes. If you want to run longer jobs and/or jobs with a request of more than 8 GByte of memory, you must allocate resources for so-called interactive jobs by usage of the command salloc on a login node. Considering a serial application running on a compute node that requires 5000 MByte of memory and limiting the interactive run to 2 hours the following command has to be executed:
<pre>
$ salloc -p single -n 1 -t 120 --mem=5000
</pre>
Then you will get one core on a compute node within the partition "single". After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system Slurm has granted you the requested resources on the compute system. You will be logged in automatically on the granted core! To run a serial program on the granted core you only have to type the name of the executable.
<pre>
$ ./<my_serial_program>
</pre>
Please be aware that your serial job must run less than 2 hours in this example, else the job will be killed during runtime by the system.
 
You can also start now a graphical X11-terminal connecting you to the dedicated resource that is available for 2 hours. You can start it by the command:
<pre>
$ xterm
</pre>
Note that, once the walltime limit has been reached the resources - i.e. the compute node - will automatically be revoked.
 
An interactive parallel application running on one compute node or on many compute nodes (e.g. here 5 nodes) with 40 cores each requires usually an amount of memory in GByte (e.g. 50 GByte) and a maximum time (e.g. 1 hour). E.g. 5 nodes can be allocated by the following command:
<pre>
$ salloc -p multiple -N 5 --ntasks-per-node=40 -t 01:00:00 --mem=50gb
</pre>
Now you can run parallel jobs on 200 cores requiring 50 GByte of memory per node. Please be aware that you will be logged in on core 0 of the first node. If you want to run MPI-programs, you can do it by simply typing mpirun <program_name>. Then your program will be run on 200 cores. A very simple example for starting a parallel job can be:
<pre>
$ mpirun <my_mpi_program>
</pre>
You can also start the debugger ddt by the commands:
<pre>
$ module add devel/ddt
$ ddt <my_mpi_program>
</pre>
The above commands will execute the parallel program <my_mpi_program> on all available cores. You can also start parallel programs on a subset of cores; an example for this can be:
<pre>
$ mpirun -n 50 <my_mpi_program>
</pre>
If you are using Intel MPI you must start <my_mpi_program> by the command mpiexec.hydra (instead of mpirun).
 
 

----
[[Category:bwUniCluster 2.0|Batch Jobs - bwUniCluster 2.0 Features]]

BwUniCluster2.0/Batch Queues

2020-12-18T09:21:46Z

H Haefner: /* sbatch -p queue */

This article contains information on the queues of the [[BwUniCluster_2.0_Slurm_common_Features|batch job system]] and on interactive jobs.

= Job Submission =
== sbatch Command ==

=== sbatch -p ''queue'' ===
Compute resources such as (wall-)time, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, '''you must add the correct queue class to your sbatch command '''. The specification of a queue is obligatory on BwUniCluster 2.0.
 
Details are:

{| width=750px class="wikitable"
! colspan="5" | bwUniCluster 2.0 sbatch -p ''queue''
|- style="text-align:left;"
! queue !! node !! default resources !! minimum resources !! maximum resources
|- style="text-align:left"
| dev_single
| thin
| time=10, mem-per-cpu=1125mb
|
| time=30, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core=2) 6 nodes are reserved for this queue. Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| single
| thin
| time=30, mem-per-cpu=1125mb
|
| time=72:00:00, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core)=2
|- style="text-align:left;"
| dev_multiple
| thin
| time=10, mem-per-cpu=1125mb
| nodes=2
| time=30, nodes=4, mem=90000mb, ntasks-per-node=40, (threads-per-core=2) 8 nodes are reserved for this queue. Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| multiple
| thin
| time=30, mem-per-cpu=1125mb
| nodes=2
| time=72:00:00, mem=90000mb, nodes=128, ntasks-per-node=40, (threads-per-core=2)
|- style="text-align:left;"
| dev_multiple_e
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
| nodes=2
| time=30, nodes=8, mem=122000, ntasks-per-node=28, (threads-per-core=2) 8 nodes are reserved for this queue Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| multiple_e
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
| nodes=2
| time=72:00:00, nodes=128, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="text-align:left;"
| dev_special (restricted access!)
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
|
| time=30, nodes=1, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="text-align:left;"
| special (restricted access!)
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
|
| time=72:00:00, nodes=1, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
| fat
| fat
| time=10, mem-per-cpu=18750mb
| time=72:00:00, nodes=1, ntasks-per-node=80, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
|
| dev_gpu_4
| gpu_4
| time=10, mem-per-cpu=2178mb
| nodes=2
| time=30, nodes=8, mem=122000, ntasks-per-node=28, (threads-per-core=2) 8 nodes are reserved for this queue Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| gpu_4
| gpu4
| time=10, mem-per-gpu=94000mb, cpu-per-gpu=20
|
| time=48:00:00, mem=376000, nodes=14, ntasks-per-node=40, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
| gpu_8
| gpu8
| time=10, mem-per-cpu=94000mb, cpu-per-gpu=10
|
| time=48:00:00, mem=752000, nodes=10, ntasks-per-node=40, (threads-per-core=2)
|-
|}

Default resources of a queue class defines time, #tasks and memory if not explicitly given with sbatch command. Resource list acronyms ''--time'', ''--ntasks'', ''--nodes'', ''--mem'' and ''--mem-per-cpu'' are described [[ForHLR_Batch_Jobs_SLURM#sbatch_Command_Parameters|here]].

Access to the "special" and "dev_special" partitions on the bwUniCluster 2.0 is restricted to members of the institutions which participated in the procurement of the extension partition specifically for this purpose. Please contact the support team if your institution participated in the procurement and your account should be able to run jobs in this partition.
 
 
==== Queue class examples ====
To run your batch job on one of the thin nodes, please use:

<pre>
$ sbatch --partition=dev_multiple
or
$ sbatch -p dev_multiple
</pre>
 

==== Interactive Jobs ====
On bwUniCluster 2.0 you are only allowed to run short jobs (<< 1 hour) with little memory requirements (<< 8 GByte) on the logins nodes. If you want to run longer jobs and/or jobs with a request of more than 8 GByte of memory, you must allocate resources for so-called interactive jobs by usage of the command salloc on a login node. Considering a serial application running on a compute node that requires 5000 MByte of memory and limiting the interactive run to 2 hours the following command has to be executed:
<pre>
$ salloc -p single -n 1 -t 120 --mem=5000
</pre>
Then you will get one core on a compute node within the partition "single". After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system Slurm has granted you the requested resources on the compute system. You will be logged in automatically on the granted core! To run a serial program on the granted core you only have to type the name of the executable.
<pre>
$ ./<my_serial_program>
</pre>
Please be aware that your serial job must run less than 2 hours in this example, else the job will be killed during runtime by the system.
 
You can also start now a graphical X11-terminal connecting you to the dedicated resource that is available for 2 hours. You can start it by the command:
<pre>
$ xterm
</pre>
Note that, once the walltime limit has been reached the resources - i.e. the compute node - will automatically be revoked.
 
An interactive parallel application running on one compute node or on many compute nodes (e.g. here 5 nodes) with 40 cores each requires usually an amount of memory in GByte (e.g. 50 GByte) and a maximum time (e.g. 1 hour). E.g. 5 nodes can be allocated by the following command:
<pre>
$ salloc -p multiple -N 5 --ntasks-per-node=40 -t 01:00:00 --mem=50gb
</pre>
Now you can run parallel jobs on 200 cores requiring 50 GByte of memory per node. Please be aware that you will be logged in on core 0 of the first node. If you want to run MPI-programs, you can do it by simply typing mpirun <program_name>. Then your program will be run on 200 cores. A very simple example for starting a parallel job can be:
<pre>
$ mpirun <my_mpi_program>
</pre>
You can also start the debugger ddt by the commands:
<pre>
$ module add devel/ddt
$ ddt <my_mpi_program>
</pre>
The above commands will execute the parallel program <my_mpi_program> on all available cores. You can also start parallel programs on a subset of cores; an example for this can be:
<pre>
$ mpirun -n 50 <my_mpi_program>
</pre>
If you are using Intel MPI you must start <my_mpi_program> by the command mpiexec.hydra (instead of mpirun).
 
 

----
[[Category:bwUniCluster 2.0|Batch Jobs - bwUniCluster 2.0 Features]]

BwUniCluster2.0/Batch Queues

2020-12-18T09:17:08Z

H Haefner: /* sbatch -p queue */

This article contains information on the queues of the [[BwUniCluster_2.0_Slurm_common_Features|batch job system]] and on interactive jobs.

= Job Submission =
== sbatch Command ==

=== sbatch -p ''queue'' ===
Compute resources such as (wall-)time, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, '''you must add the correct queue class to your sbatch command '''. The specification of a queue is obligatory on BwUniCluster 2.0.
 
Details are:

{| width=750px class="wikitable"
! colspan="5" | bwUniCluster 2.0 sbatch -p ''queue''
|- style="text-align:left;"
! queue !! node !! default resources !! minimum resources !! maximum resources
|- style="text-align:left"
| dev_single
| thin
| time=10, mem-per-cpu=1125mb
|
| time=30, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core=2) 6 nodes are reserved for this queue. Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| single
| thin
| time=30, mem-per-cpu=1125mb
|
| time=72:00:00, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core)=2
|- style="text-align:left;"
| dev_multiple
| thin
| time=10, mem-per-cpu=1125mb
| nodes=2
| time=30, nodes=4, mem=90000mb, ntasks-per-node=40, (threads-per-core=2) 8 nodes are reserved for this queue. Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| multiple
| thin
| time=30, mem-per-cpu=1125mb
| nodes=2
| time=72:00:00, mem=90000mb, nodes=128, ntasks-per-node=40, (threads-per-core=2)
|- style="text-align:left;"
| dev_multiple_e
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
| nodes=2
| time=30, nodes=8, mem=122000, ntasks-per-node=28, (threads-per-core=2) 8 nodes are reserved for this queue Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| multiple_e
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
| nodes=2
| time=72:00:00, nodes=128, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="text-align:left;"
| dev_special (restricted access!)
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
|
| time=30, nodes=1, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="text-align:left;"
| special (restricted access!)
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
|
| time=72:00:00, nodes=1, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
| fat
| fat
| time=10, mem-per-cpu=18750mb
|
| dev_gpu_4
| gpu_4
| time=10, mem-per-cpu=2178mb
| nodes=2
| time=30, nodes=8, mem=122000, ntasks-per-node=28, (threads-per-core=2) 8 nodes are reserved for this queue Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
|
| time=72:00:00, nodes=1, ntasks-per-node=80, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
| gpu_4
| gpu4
| time=10, mem-per-gpu=94000mb, cpu-per-gpu=20
|
| time=48:00:00, mem=376000, nodes=14, ntasks-per-node=40, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
| gpu_8
| gpu8
| time=10, mem-per-cpu=94000mb, cpu-per-gpu=10
|
| time=48:00:00, mem=752000, nodes=10, ntasks-per-node=40, (threads-per-core=2)
|-
|}

Default resources of a queue class defines time, #tasks and memory if not explicitly given with sbatch command. Resource list acronyms ''--time'', ''--ntasks'', ''--nodes'', ''--mem'' and ''--mem-per-cpu'' are described [[ForHLR_Batch_Jobs_SLURM#sbatch_Command_Parameters|here]].

Access to the "special" and "dev_special" partitions on the bwUniCluster 2.0 is restricted to members of the institutions which participated in the procurement of the extension partition specifically for this purpose. Please contact the support team if your institution participated in the procurement and your account should be able to run jobs in this partition.
 
 
==== Queue class examples ====
To run your batch job on one of the thin nodes, please use:

<pre>
$ sbatch --partition=dev_multiple
or
$ sbatch -p dev_multiple
</pre>
 

==== Interactive Jobs ====
On bwUniCluster 2.0 you are only allowed to run short jobs (<< 1 hour) with little memory requirements (<< 8 GByte) on the logins nodes. If you want to run longer jobs and/or jobs with a request of more than 8 GByte of memory, you must allocate resources for so-called interactive jobs by usage of the command salloc on a login node. Considering a serial application running on a compute node that requires 5000 MByte of memory and limiting the interactive run to 2 hours the following command has to be executed:
<pre>
$ salloc -p single -n 1 -t 120 --mem=5000
</pre>
Then you will get one core on a compute node within the partition "single". After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system Slurm has granted you the requested resources on the compute system. You will be logged in automatically on the granted core! To run a serial program on the granted core you only have to type the name of the executable.
<pre>
$ ./<my_serial_program>
</pre>
Please be aware that your serial job must run less than 2 hours in this example, else the job will be killed during runtime by the system.
 
You can also start now a graphical X11-terminal connecting you to the dedicated resource that is available for 2 hours. You can start it by the command:
<pre>
$ xterm
</pre>
Note that, once the walltime limit has been reached the resources - i.e. the compute node - will automatically be revoked.
 
An interactive parallel application running on one compute node or on many compute nodes (e.g. here 5 nodes) with 40 cores each requires usually an amount of memory in GByte (e.g. 50 GByte) and a maximum time (e.g. 1 hour). E.g. 5 nodes can be allocated by the following command:
<pre>
$ salloc -p multiple -N 5 --ntasks-per-node=40 -t 01:00:00 --mem=50gb
</pre>
Now you can run parallel jobs on 200 cores requiring 50 GByte of memory per node. Please be aware that you will be logged in on core 0 of the first node. If you want to run MPI-programs, you can do it by simply typing mpirun <program_name>. Then your program will be run on 200 cores. A very simple example for starting a parallel job can be:
<pre>
$ mpirun <my_mpi_program>
</pre>
You can also start the debugger ddt by the commands:
<pre>
$ module add devel/ddt
$ ddt <my_mpi_program>
</pre>
The above commands will execute the parallel program <my_mpi_program> on all available cores. You can also start parallel programs on a subset of cores; an example for this can be:
<pre>
$ mpirun -n 50 <my_mpi_program>
</pre>
If you are using Intel MPI you must start <my_mpi_program> by the command mpiexec.hydra (instead of mpirun).
 
 

----
[[Category:bwUniCluster 2.0|Batch Jobs - bwUniCluster 2.0 Features]]

File:Bwidm-register-red.png

2020-12-17T15:28:32Z

H Haefner:

BwUniCluster 2.0 User Access

2020-12-17T15:27:52Z

H Haefner: /* Step B: Web Registration, service password and 2-factor authentication */

[[bwUniCluster_2.0|bwUniCluster 2.0]] is Baden-Württemberg's general purpose tier 3 high performance computing (HPC)
cluster co-financed by Baden-Württemberg's ministry of science, research and arts and the shareholders:

* Albert Ludwig University of Freiburg
* Eberhard Karls University, Tübingen
* Karlsruhe Institute of Technology (KIT)
* Heidelberg University (Ruprecht-Karls-Universität Heidelberg)
* Ulm University
* University of Hohenheim
* University of Konstanz
* University of Mannheim
* University of Stuttgart
* HAW BW e.V. (an association of several universities of applied sciences in Baden-Württemberg, see below)
 
To '''log on''' [[bwUniCluster_2.0|bwUniCluster 2.0]] a user account is required. All members of the shareholder
universities can apply for an account.
 

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:left; color:#000;vertical-align:top;" |__TOC__
| [[File:bwUniCluster_17Jan2014_p044-rot_t10.10.00.jpg|center|border|250px|bwUniCluster wiring by Holger Obermaier, copyright: KIT (SCC)]] bwUniCluster wiring © KIT (SCC)
|}

= Registration =

Granting access and issuing a user account for '''bwUniCluster 2.0''' requires the registration at the KIT service website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] (step B). However, this registration depends on the '''bwUniCluster entitlement''' issued by your university (step A).
 

== Step A: bwUniCluster entitlement for registration ==
'''The entitlement is called bwUniCluster (not bwUniCluster 2.0)''' and each university issues the bwUniCluster entitlement '''only''' for their own respective members. Some have established on-line processes or provide downloads of the entitlement application forms. If there is no link behind the name of an institution in the following list, please contact the local IT support services:

* [[BwCluster_User_Access_Uni_Freiburg|Albert Ludwig University of Freiburg]]
* [https://bwunicluster.urz.uni-heidelberg.de/ Heidelberg University]
* [https://kim.uni-hohenheim.de/bwhpc-account University of Hohenheim]
* [http://www.scc.kit.edu/downloads/sdo/Antrag_Benutzernummer_bwUniCluster.pdf Karlsruhe Institute of Technology (KIT)]
* [[BWUniCluster_User_Access_Members_Uni_Konstanz|University of Konstanz]]
* [[BWUniCluster_User_Access_Members_Uni_Mannheim|University of Mannheim]]
* [https://www.hlrs.de/solutions-services/academic-users/bwunicluster-access/ University of Stuttgart]
* [https://uni-tuebingen.de/de/155157 Eberhard Karls University Tübingen]
* [[BWUniCluster_User_Access_Members_Uni_Ulm|Ulm University]]
* Hochschule Aalen
* Hochschule Albstadt-Sigmaringen
* Hochschule Esslingen
* Hochschule Furtwangen
* Hochschule Heilbronn
* Hochschule Karlsruhe
* Hochschule Konstanz
* Hochschule Reutlingen
* Hochschule Rottenburg
* Hochschule Stuttgart (HfT)
* Hochschule Ulm
 

== Step B: Web Registration, service password and 2-factor authentication ==

After completing step A, i.e., after successfull issueing of the bwUniCluster entitlement, you have to register yourself for the service. To do so please visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and complete the following steps.

1. Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.

[[File:Bwidm-register-red.png|center|border|]]
 

2. You will be directed to the ''Identity Provider'' of your home organisation. Enter the user ID / username and password of your home organisation - this is usually the same password used for your e-mail account and other services - and click on '''Login''', '''Einloggen''' or something similar.
 

3. You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]. If you are logging into bwIDM for the first time, there will be a summary screen which shows the account details your home institution is providing to the central system. Please check that all data is valid and then click on '''Continue''' or '''Weiter'''.
 

4. Once you have successfully logged into the bwIDM system, you will be greeted by a home screen showing all state-wide services you have access to. There will be a box labelled "bwUniCluster". Click on '''Register''' or '''Registrieren''' to start the registration process.

[[File:Bwidm-2-red.png|center|border|]]
 

5. Since August 13, 2020 a '''2-factor authentication''' mechanism (2FA) is being enforced to improve security. If you have never registered a 2FA token on bwIDM before, the following error message will appear:

[[File:Bwidm-3-red.png|center|]]

Click on the [https://bwidm.scc.kit.edu/user/twofa.xhtml Link] or on the '''My Tokens''' link in the main menu. The instructions for registering a new 2FA token can be found on the following page: [[bwUniCluster 2.0 User Access/2FA Tokens]]. Please complete them before proceeding.
 

6. Make sure all requirements are met by checking the '''Requirements''' box at the top. If the requirements are not met you might be able to correct the issure by following the instructions. In all other cases please [[Registration_Support_-_bwUniCluster|contact your local hotline]].

[[File:BwUniCluster 2.0 access login bwidm registration requirements.png|center|border|]]
 

7. Read the Terms of Use ('''Nutzungsbedingungen und -richtlinien'''), check the box besides '''I have read and accepted the terms of use''' and click on '''Register''' or '''Registrieren'''.
 

8. Set a service password for the bwUniCluster and click on '''Save''' or '''Speichern'''. Logging in with the password of your home organisation, like on the former bwUniCluster 1, is no longer possible. Please make sure to use a strong password which is different from any other password you are currently using or have used on other systems. You will also be asked to change the service password regularly.

[[File:Bwidm-5-red.png|center|]]
 
 

== Step C: Fill out the bwUniCluster questionnaire ==

Filling out the bwUniCluster questionaire on

https://zas.bwhpc.de/shib/en/bwunicluster_survey.php

is mandatory for all users. The input is solely used to improve our support activities and for capacity planning of future HPC resources. '''If the questionaire is not filled out, access to bwUniCluster 2.0 is blocked 14 days after the registration.'''
 
 

== Changing the Service Password ==

Your bwUniCluster 2.0 '''password''' is the service password you set during the web registration (compare step 7 of chapter 1.2). At any time, you can set a new bwUniCluster 2.0 password via the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] by carrying out the following steps:
# Go to [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and select your home organization
# Authenticate yourself via the user id / username and password provided by your home institution
# Find the entry '''bwUniCluster''' and select '''Set Service Password'''
# Enter the new password, repeat it and click '''Save''' button.
# If the change was sucessfull, the message "Das Passwort wurde bei dem Dienst geändert" ("Password has been changed") will be shown
# Proceed to log in using the new password
 
 
== Contact / Support ==
If you have questions or problems concerning the bwUniCluster (2.0) registration, please [[bwUniCluster 2.0 Support|contact your local hotline]].
 
 

= Establishing network access =

Access to bwUniCluster 2.0 is '''limited to IP addresses from the so-called BelWü networks'''. All home institutions of our current users are connected to BelWue, so if you are on your campus network (e.g. in your office or on the Campus WiFi) you should be able to connect to bwUniCluster 2.0 without restrictions. If you are outside of one of the BelWue networks (e.g. in your home office instead of in your campus office), a VPN connection to your home institution has to be established first (see e.g. [1] for the KIT).
 
 

= Login =

After finishing the web registration and making sure that you are on a network from which you have access to bwUniCluster 2.0 (e.g. by establishing a VPN connection), the HPC cluster is ready for your '''SSH''' based login. Recommended SSH clients applications are:

* the ssh (OpenSSH) command included in all Linux distributions and macOS, -in command under Linux and macOS using the application ''terminal''
* [http://mobaxterm.mobatek.net/ MobaXterm] under Windows
 

== Hostnames ==

The main hostname required to connect to bwUniCluster 2.0 is '''bwunicluster.scc.kit.edu''' or '''uc2.scc.kit.edu'''. The system has four login nodes and we use so-called ''DNS round-robin scheduling'' to load-balance the incoming connections between the nodes. If you open multiple SSH sessions to bwUniCluster 2.0, these sessions will be established to different login nodes, so processes started in one session might not be visible in other sessions.

The older Broadwell extension partition of the former bwUniCluster 1 is connected to bwUniCluster 2.0. You can use the hostname '''uc1e.scc.kit.edu''' to connect to the login nodes of this partition.

If you need to connect to specific login nodes, you can use the following hostnames:

{| class="wikitable"
! Hostname !! Node type
|-
| '''uc2-login1.scc.kit.edu''' || bwUniCluster 2.0, first login node
|-
| '''uc2-login2.scc.kit.edu''' || bwUniCluster 2.0, second login node
|-
| '''uc2-login3.scc.kit.edu''' || bwUniCluster 2.0, third login node
|-
| '''uc2-login4.scc.kit.edu''' || bwUniCluster 2.0, fourth login node
|-
| '''uc1e-login1.scc.kit.edu''' || Broadwell partition, first login node
|-
| '''uc1e-login2.scc.kit.edu''' || Broadwell partition, second login node
|-
|}

Only the secure shell ''SSH'' is allowed to login. Other protocols like ''telnet'' or ''rlogin'' are not allowed for security reasons.
 

== Usernames ==

Your username will be the same as the one provided by your home institution, but '''prefixed''' with two characters and an underscore indicating your home institution. For example: If you are a member of the university of Konstanz and your local username is ab1234, your username on bwUniCluster 2.0 is kn_ab1234.

The following list contains all prefixes currently in use:

{| class="wikitable"
! Home organization !! <UserID>
|-
| Universität Freiburg || ''fr_''username
|-
| Universität Heidelberg || ''hd_''username
|-
| Universität Hohenheim || ''ho_''username
|-
| KIT || username ''(without any prefix)''
|-
| Universität Konstanz || ''kn_''username
|-
| Universität Mannheim || ''ma_''username
|-
| Universität Stuttgart || ''st_''username
|-
| Universität Tübingen || ''tu_''username
|-
| Universität Ulm || ''ul_''username
|-
| Hochschule Aalen || ''aa_''username
|-
| Hochschule Albstadt-Sigmaringen || ''as_''username
|-
| Hochschule Esslingen || ''es_''username
|-
| Hochschule Furtwangen || ''hf_''username
|-
| Hochschule Heilbronn || ''hn_''username
|-
| Hochschule Karlsruhe || ''hk_''username
|-
| HTWG Konstanz || ''ht_''username
|-
| Hochschule Reutlingen || ''hr_''username
|-
| Hochschule Rottenburg || ''ro_''username
|-
| Hochschule für Technik Stuttgart || ''hs_''username
|-
| Hochschule Ulm || ''hu_''username
|-
|}
 

== Client application: OpenSSH ==

Most Unix and Unix-like operating systems like Linux, macOS and *BSD come with a built-in SSH client provided by the OpenSSH project. More recent versions of Windows 10 and the Windows Subsystem for Linux also come with a built-in OpenSSH client.

To use this client, simply open a command line terminal (the exact process differs on every operating system, but usually involves starting an application called '''Terminal''' or '''Command Prompt''') and enter the following command to connect to bwUniCluster 2.0:

<pre>
$ ssh <UserID>@bwunicluster.scc.kit.edu
</pre>

If you are on a Linux or Unix system running the X Window System (X11) and want to use a GUI-based application on bwUniCluster 2.0, you can use the ''-X'' option for the ssh command to set up X11 forwarding:

<pre>
$ ssh -X <UserID>@uc2.scc.kit.edu
</pre>

Windows users requiring X11 forwarding for graphical applications should use '''MobaXterm''' instead.
 

== Client application: MobaXterm ==

The bwHPC-C5 support team strongly recommends to use [http://mobaxterm.mobatek.net/ MobaXterm] instead of ''PuTTY'' or ''WinSCP'' on Windows. ''MobaXterm'' provides a built-in X11 server allowing to start GUI based software.

Start ''MobaXterm'', fill in the following fields:
<pre>
Remote name : uc2.scc.kit.edu # or uc1e.scc.kit.edu
Specify user name : <UserID>
Port : 22
</pre>

After that click on 'ok'. Then a terminal will be opened and there you can enter your credentials.
 

== Example login process ==

After the connection has been initiated, a successful login process will go through the following three steps:

1. The system asks for a '''One-Time Password'''. Generate one using the Software or Hardware Token registered on the bwIDM system (see [[bwUniCluster 2.0 User Access/2FA Tokens]]) and enter it after the '''Your OTP:''' prompt.

2. The systems asks for your service password. Enter it after the '''Password:''' prompt.

3. You are greeted by the bwUniCluster 2.0 banner followed by a shell.

The result should look like this:

[[File:BwUniCluster 2.0 access login example.png|center|]]
 

== Troubleshooting ==

'''Issue: The "Your OTP:" prompt never appears and the connection hangs/times out instead'''

Likely cause: You are most likely not on a network from which access to the bwUniCluster 2.0 system is allowed. Please check if you might have to establish a VPN connection first.

 

'''Issue: The system asks for the One-Time Password multiple times'''

Likely cause: Make sure you are using the correct Software Token to generate the One-Time Password.

 

'''Issue: The system asks for the service password multiple times'''

Likely cause: Make sure you are using the service password set on bwIDM and not the password valid for your home institution. Unlike the bwUniCluster 1, the bwUniCluster 2.0 only accepts the service password.

 

'''Issue: There is an error message by the pam_ses_open.sh skript'''

Likely cause: Your account is in the "LOST_ACCESS" state because the entitlement is no longer valid, the questionaire was not filled out or there was a problem during the communication between your home institution and the central bwIDM system. Please try the following steps:

* Log into [https://bwidm.scc.kit.edu bwIDM], look for the bwUniCluster entry and click on '''Registry info'''. Your "Status:" should be "ACTIVE". If it is not, please wait for ten minutes since logging into the bwIDM causes a refresh and the problem might fix itself. If the status does not change to ACTIVE after a longer amount of time, please contact the support channels.

* If you have not filled out the questionaire, please do so on [https://zas.bwhpc.de/shib/en/bwunicluster_survey.php https://zas.bwhpc.de/shib/en/bwunicluster_survey.php] and then wait for about ten minutes before attempting to log into the HPC system again.
 
 

== Allowed activities on login nodes ==

The login nodes of bwUniCluster 2.0 are the access point to the compute system and to your bwUniCluster 2.0 $HOME directory. The login nodes are shared with all the users of bwUniCluster 2.0. Therefore, your activities on the login nodes are limited to primarily set up your batch jobs. Your activities may also be:

* '''short''' compilation of your program code and
* '''short''' pre- and postprocessing of your batch jobs.

To guarantee usability for all the users of bwUniCluster 2.0 '''you must not run your compute jobs on the login nodes'''. Compute jobs must be submitted to the
[[bwUniCluster Batch Jobs|queueing system]]. Any compute job running on the login nodes will be terminated without any notice. Any long-running compilation or any long-running pre- or postprocessing of batch jobs must also be submitted to the [[bwUniCluster Batch Jobs|queueing system]].
 
 

== SSH Keys ==

In contrast to the bwUniCluster 1 and many other HPC systems it is '''no longer possible to self-manage your SSH Keys by adding them to the ~/.ssh/authorized_keys file'''. Existing files will no longer be evaluated. SSH Keys have to be managed via the central bwIDM system instead. Please refer to the user guide for this functionality:

[[bwUniCluster 2.0 User Access/SSH Keys]]
 
 

= [[First_Steps_on_bwHPC_cluster|First steps on bwUniCluster]] =

First and some important steps on bwUniCluster 2.0 can be found [[First_Steps_on_bwHPC_cluster|here]].
 
 

= Deregistration =
If you plan to permanently leave the bwUniCluster 2.0, follow the deregister checklist:
# Transfer all your data in $HOME and workspace to your local computer/storage and after that clear off all your data
# Visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
#* Select your home organization from the list and click '''Proceed'''
#* Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
#* You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
#* <div>Select '''Registry Info''' of the service '''bwUniCluster''' (on the left hand side) [[File:bwUniCluster_registration_sidebar.png|center|border|]]</div>
#* Click '''Deregister'''
Note that Step 2 will automatically unsubscribe you from the bwunicluster mail list.

File:Bwidm-5-red.png

2020-12-17T15:20:59Z

H Haefner:

File:Bwidm-3-red.png

2020-12-17T15:19:49Z

H Haefner:

BwUniCluster 2.0 User Access

2020-12-17T15:19:27Z

H Haefner: /* Step B: Web Registration, service password and 2-factor authentication */

[[bwUniCluster_2.0|bwUniCluster 2.0]] is Baden-Württemberg's general purpose tier 3 high performance computing (HPC)
cluster co-financed by Baden-Württemberg's ministry of science, research and arts and the shareholders:

* Albert Ludwig University of Freiburg
* Eberhard Karls University, Tübingen
* Karlsruhe Institute of Technology (KIT)
* Heidelberg University (Ruprecht-Karls-Universität Heidelberg)
* Ulm University
* University of Hohenheim
* University of Konstanz
* University of Mannheim
* University of Stuttgart
* HAW BW e.V. (an association of several universities of applied sciences in Baden-Württemberg, see below)
 
To '''log on''' [[bwUniCluster_2.0|bwUniCluster 2.0]] a user account is required. All members of the shareholder
universities can apply for an account.
 

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:left; color:#000;vertical-align:top;" |__TOC__
| [[File:bwUniCluster_17Jan2014_p044-rot_t10.10.00.jpg|center|border|250px|bwUniCluster wiring by Holger Obermaier, copyright: KIT (SCC)]] bwUniCluster wiring © KIT (SCC)
|}

= Registration =

Granting access and issuing a user account for '''bwUniCluster 2.0''' requires the registration at the KIT service website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] (step B). However, this registration depends on the '''bwUniCluster entitlement''' issued by your university (step A).
 

== Step A: bwUniCluster entitlement for registration ==
'''The entitlement is called bwUniCluster (not bwUniCluster 2.0)''' and each university issues the bwUniCluster entitlement '''only''' for their own respective members. Some have established on-line processes or provide downloads of the entitlement application forms. If there is no link behind the name of an institution in the following list, please contact the local IT support services:

* [[BwCluster_User_Access_Uni_Freiburg|Albert Ludwig University of Freiburg]]
* [https://bwunicluster.urz.uni-heidelberg.de/ Heidelberg University]
* [https://kim.uni-hohenheim.de/bwhpc-account University of Hohenheim]
* [http://www.scc.kit.edu/downloads/sdo/Antrag_Benutzernummer_bwUniCluster.pdf Karlsruhe Institute of Technology (KIT)]
* [[BWUniCluster_User_Access_Members_Uni_Konstanz|University of Konstanz]]
* [[BWUniCluster_User_Access_Members_Uni_Mannheim|University of Mannheim]]
* [https://www.hlrs.de/solutions-services/academic-users/bwunicluster-access/ University of Stuttgart]
* [https://uni-tuebingen.de/de/155157 Eberhard Karls University Tübingen]
* [[BWUniCluster_User_Access_Members_Uni_Ulm|Ulm University]]
* Hochschule Aalen
* Hochschule Albstadt-Sigmaringen
* Hochschule Esslingen
* Hochschule Furtwangen
* Hochschule Heilbronn
* Hochschule Karlsruhe
* Hochschule Konstanz
* Hochschule Reutlingen
* Hochschule Rottenburg
* Hochschule Stuttgart (HfT)
* Hochschule Ulm
 

== Step B: Web Registration, service password and 2-factor authentication ==

After completing step A, i.e., after successfull issueing of the bwUniCluster entitlement, you have to register yourself for the service. To do so please visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and complete the following steps.

1. Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.

[[File:Bwidm-1-red.png|center|border|]]
 

2. You will be directed to the ''Identity Provider'' of your home organisation. Enter the user ID / username and password of your home organisation - this is usually the same password used for your e-mail account and other services - and click on '''Login''', '''Einloggen''' or something similar.
 

3. You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]. If you are logging into bwIDM for the first time, there will be a summary screen which shows the account details your home institution is providing to the central system. Please check that all data is valid and then click on '''Continue''' or '''Weiter'''.
 

4. Once you have successfully logged into the bwIDM system, you will be greeted by a home screen showing all state-wide services you have access to. There will be a box labelled "bwUniCluster". Click on '''Register''' or '''Registrieren''' to start the registration process.

[[File:Bwidm-2-red.png|center|border|]]
 

5. Since August 13, 2020 a '''2-factor authentication''' mechanism (2FA) is being enforced to improve security. If you have never registered a 2FA token on bwIDM before, the following error message will appear:

[[File:Bwidm-3-red.png|center|]]

Click on the [https://bwidm.scc.kit.edu/user/twofa.xhtml Link] or on the '''My Tokens''' link in the main menu. The instructions for registering a new 2FA token can be found on the following page: [[bwUniCluster 2.0 User Access/2FA Tokens]]. Please complete them before proceeding.
 

6. Make sure all requirements are met by checking the '''Requirements''' box at the top. If the requirements are not met you might be able to correct the issure by following the instructions. In all other cases please [[Registration_Support_-_bwUniCluster|contact your local hotline]].

[[File:BwUniCluster 2.0 access login bwidm registration requirements.png|center|border|]]
 

7. Read the Terms of Use ('''Nutzungsbedingungen und -richtlinien'''), check the box besides '''I have read and accepted the terms of use''' and click on '''Register''' or '''Registrieren'''.
 

8. Set a service password for the bwUniCluster and click on '''Save''' or '''Speichern'''. Logging in with the password of your home organisation, like on the former bwUniCluster 1, is no longer possible. Please make sure to use a strong password which is different from any other password you are currently using or have used on other systems. You will also be asked to change the service password regularly.

[[File:Bwidm-5-red.png|center|]]
 
 

== Step C: Fill out the bwUniCluster questionnaire ==

Filling out the bwUniCluster questionaire on

https://zas.bwhpc.de/shib/en/bwunicluster_survey.php

is mandatory for all users. The input is solely used to improve our support activities and for capacity planning of future HPC resources. '''If the questionaire is not filled out, access to bwUniCluster 2.0 is blocked 14 days after the registration.'''
 
 

== Changing the Service Password ==

Your bwUniCluster 2.0 '''password''' is the service password you set during the web registration (compare step 7 of chapter 1.2). At any time, you can set a new bwUniCluster 2.0 password via the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] by carrying out the following steps:
# Go to [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and select your home organization
# Authenticate yourself via the user id / username and password provided by your home institution
# Find the entry '''bwUniCluster''' and select '''Set Service Password'''
# Enter the new password, repeat it and click '''Save''' button.
# If the change was sucessfull, the message "Das Passwort wurde bei dem Dienst geändert" ("Password has been changed") will be shown
# Proceed to log in using the new password
 
 
== Contact / Support ==
If you have questions or problems concerning the bwUniCluster (2.0) registration, please [[bwUniCluster 2.0 Support|contact your local hotline]].
 
 

= Establishing network access =

Access to bwUniCluster 2.0 is '''limited to IP addresses from the so-called BelWü networks'''. All home institutions of our current users are connected to BelWue, so if you are on your campus network (e.g. in your office or on the Campus WiFi) you should be able to connect to bwUniCluster 2.0 without restrictions. If you are outside of one of the BelWue networks (e.g. in your home office instead of in your campus office), a VPN connection to your home institution has to be established first (see e.g. [1] for the KIT).
 
 

= Login =

After finishing the web registration and making sure that you are on a network from which you have access to bwUniCluster 2.0 (e.g. by establishing a VPN connection), the HPC cluster is ready for your '''SSH''' based login. Recommended SSH clients applications are:

* the ssh (OpenSSH) command included in all Linux distributions and macOS, -in command under Linux and macOS using the application ''terminal''
* [http://mobaxterm.mobatek.net/ MobaXterm] under Windows
 

== Hostnames ==

The main hostname required to connect to bwUniCluster 2.0 is '''bwunicluster.scc.kit.edu''' or '''uc2.scc.kit.edu'''. The system has four login nodes and we use so-called ''DNS round-robin scheduling'' to load-balance the incoming connections between the nodes. If you open multiple SSH sessions to bwUniCluster 2.0, these sessions will be established to different login nodes, so processes started in one session might not be visible in other sessions.

The older Broadwell extension partition of the former bwUniCluster 1 is connected to bwUniCluster 2.0. You can use the hostname '''uc1e.scc.kit.edu''' to connect to the login nodes of this partition.

If you need to connect to specific login nodes, you can use the following hostnames:

{| class="wikitable"
! Hostname !! Node type
|-
| '''uc2-login1.scc.kit.edu''' || bwUniCluster 2.0, first login node
|-
| '''uc2-login2.scc.kit.edu''' || bwUniCluster 2.0, second login node
|-
| '''uc2-login3.scc.kit.edu''' || bwUniCluster 2.0, third login node
|-
| '''uc2-login4.scc.kit.edu''' || bwUniCluster 2.0, fourth login node
|-
| '''uc1e-login1.scc.kit.edu''' || Broadwell partition, first login node
|-
| '''uc1e-login2.scc.kit.edu''' || Broadwell partition, second login node
|-
|}

Only the secure shell ''SSH'' is allowed to login. Other protocols like ''telnet'' or ''rlogin'' are not allowed for security reasons.
 

== Usernames ==

Your username will be the same as the one provided by your home institution, but '''prefixed''' with two characters and an underscore indicating your home institution. For example: If you are a member of the university of Konstanz and your local username is ab1234, your username on bwUniCluster 2.0 is kn_ab1234.

The following list contains all prefixes currently in use:

{| class="wikitable"
! Home organization !! <UserID>
|-
| Universität Freiburg || ''fr_''username
|-
| Universität Heidelberg || ''hd_''username
|-
| Universität Hohenheim || ''ho_''username
|-
| KIT || username ''(without any prefix)''
|-
| Universität Konstanz || ''kn_''username
|-
| Universität Mannheim || ''ma_''username
|-
| Universität Stuttgart || ''st_''username
|-
| Universität Tübingen || ''tu_''username
|-
| Universität Ulm || ''ul_''username
|-
| Hochschule Aalen || ''aa_''username
|-
| Hochschule Albstadt-Sigmaringen || ''as_''username
|-
| Hochschule Esslingen || ''es_''username
|-
| Hochschule Furtwangen || ''hf_''username
|-
| Hochschule Heilbronn || ''hn_''username
|-
| Hochschule Karlsruhe || ''hk_''username
|-
| HTWG Konstanz || ''ht_''username
|-
| Hochschule Reutlingen || ''hr_''username
|-
| Hochschule Rottenburg || ''ro_''username
|-
| Hochschule für Technik Stuttgart || ''hs_''username
|-
| Hochschule Ulm || ''hu_''username
|-
|}
 

== Client application: OpenSSH ==

Most Unix and Unix-like operating systems like Linux, macOS and *BSD come with a built-in SSH client provided by the OpenSSH project. More recent versions of Windows 10 and the Windows Subsystem for Linux also come with a built-in OpenSSH client.

To use this client, simply open a command line terminal (the exact process differs on every operating system, but usually involves starting an application called '''Terminal''' or '''Command Prompt''') and enter the following command to connect to bwUniCluster 2.0:

<pre>
$ ssh <UserID>@bwunicluster.scc.kit.edu
</pre>

If you are on a Linux or Unix system running the X Window System (X11) and want to use a GUI-based application on bwUniCluster 2.0, you can use the ''-X'' option for the ssh command to set up X11 forwarding:

<pre>
$ ssh -X <UserID>@uc2.scc.kit.edu
</pre>

Windows users requiring X11 forwarding for graphical applications should use '''MobaXterm''' instead.
 

== Client application: MobaXterm ==

The bwHPC-C5 support team strongly recommends to use [http://mobaxterm.mobatek.net/ MobaXterm] instead of ''PuTTY'' or ''WinSCP'' on Windows. ''MobaXterm'' provides a built-in X11 server allowing to start GUI based software.

Start ''MobaXterm'', fill in the following fields:
<pre>
Remote name : uc2.scc.kit.edu # or uc1e.scc.kit.edu
Specify user name : <UserID>
Port : 22
</pre>

After that click on 'ok'. Then a terminal will be opened and there you can enter your credentials.
 

== Example login process ==

After the connection has been initiated, a successful login process will go through the following three steps:

1. The system asks for a '''One-Time Password'''. Generate one using the Software or Hardware Token registered on the bwIDM system (see [[bwUniCluster 2.0 User Access/2FA Tokens]]) and enter it after the '''Your OTP:''' prompt.

2. The systems asks for your service password. Enter it after the '''Password:''' prompt.

3. You are greeted by the bwUniCluster 2.0 banner followed by a shell.

The result should look like this:

[[File:BwUniCluster 2.0 access login example.png|center|]]
 

== Troubleshooting ==

'''Issue: The "Your OTP:" prompt never appears and the connection hangs/times out instead'''

Likely cause: You are most likely not on a network from which access to the bwUniCluster 2.0 system is allowed. Please check if you might have to establish a VPN connection first.

 

'''Issue: The system asks for the One-Time Password multiple times'''

Likely cause: Make sure you are using the correct Software Token to generate the One-Time Password.

 

'''Issue: The system asks for the service password multiple times'''

Likely cause: Make sure you are using the service password set on bwIDM and not the password valid for your home institution. Unlike the bwUniCluster 1, the bwUniCluster 2.0 only accepts the service password.

 

'''Issue: There is an error message by the pam_ses_open.sh skript'''

Likely cause: Your account is in the "LOST_ACCESS" state because the entitlement is no longer valid, the questionaire was not filled out or there was a problem during the communication between your home institution and the central bwIDM system. Please try the following steps:

* Log into [https://bwidm.scc.kit.edu bwIDM], look for the bwUniCluster entry and click on '''Registry info'''. Your "Status:" should be "ACTIVE". If it is not, please wait for ten minutes since logging into the bwIDM causes a refresh and the problem might fix itself. If the status does not change to ACTIVE after a longer amount of time, please contact the support channels.

* If you have not filled out the questionaire, please do so on [https://zas.bwhpc.de/shib/en/bwunicluster_survey.php https://zas.bwhpc.de/shib/en/bwunicluster_survey.php] and then wait for about ten minutes before attempting to log into the HPC system again.
 
 

== Allowed activities on login nodes ==

The login nodes of bwUniCluster 2.0 are the access point to the compute system and to your bwUniCluster 2.0 $HOME directory. The login nodes are shared with all the users of bwUniCluster 2.0. Therefore, your activities on the login nodes are limited to primarily set up your batch jobs. Your activities may also be:

* '''short''' compilation of your program code and
* '''short''' pre- and postprocessing of your batch jobs.

To guarantee usability for all the users of bwUniCluster 2.0 '''you must not run your compute jobs on the login nodes'''. Compute jobs must be submitted to the
[[bwUniCluster Batch Jobs|queueing system]]. Any compute job running on the login nodes will be terminated without any notice. Any long-running compilation or any long-running pre- or postprocessing of batch jobs must also be submitted to the [[bwUniCluster Batch Jobs|queueing system]].
 
 

== SSH Keys ==

In contrast to the bwUniCluster 1 and many other HPC systems it is '''no longer possible to self-manage your SSH Keys by adding them to the ~/.ssh/authorized_keys file'''. Existing files will no longer be evaluated. SSH Keys have to be managed via the central bwIDM system instead. Please refer to the user guide for this functionality:

[[bwUniCluster 2.0 User Access/SSH Keys]]
 
 

= [[First_Steps_on_bwHPC_cluster|First steps on bwUniCluster]] =

First and some important steps on bwUniCluster 2.0 can be found [[First_Steps_on_bwHPC_cluster|here]].
 
 

= Deregistration =
If you plan to permanently leave the bwUniCluster 2.0, follow the deregister checklist:
# Transfer all your data in $HOME and workspace to your local computer/storage and after that clear off all your data
# Visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
#* Select your home organization from the list and click '''Proceed'''
#* Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
#* You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
#* <div>Select '''Registry Info''' of the service '''bwUniCluster''' (on the left hand side) [[File:bwUniCluster_registration_sidebar.png|center|border|]]</div>
#* Click '''Deregister'''
Note that Step 2 will automatically unsubscribe you from the bwunicluster mail list.

File:Bwidm-1-red.png

2020-12-17T15:14:49Z

H Haefner:

BwUniCluster 2.0 User Access

2020-12-17T15:11:54Z

H Haefner: /* Step B: Web Registration, service password and 2-factor authentication */

[[bwUniCluster_2.0|bwUniCluster 2.0]] is Baden-Württemberg's general purpose tier 3 high performance computing (HPC)
cluster co-financed by Baden-Württemberg's ministry of science, research and arts and the shareholders:

* Albert Ludwig University of Freiburg
* Eberhard Karls University, Tübingen
* Karlsruhe Institute of Technology (KIT)
* Heidelberg University (Ruprecht-Karls-Universität Heidelberg)
* Ulm University
* University of Hohenheim
* University of Konstanz
* University of Mannheim
* University of Stuttgart
* HAW BW e.V. (an association of several universities of applied sciences in Baden-Württemberg, see below)
 
To '''log on''' [[bwUniCluster_2.0|bwUniCluster 2.0]] a user account is required. All members of the shareholder
universities can apply for an account.
 

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:left; color:#000;vertical-align:top;" |__TOC__
| [[File:bwUniCluster_17Jan2014_p044-rot_t10.10.00.jpg|center|border|250px|bwUniCluster wiring by Holger Obermaier, copyright: KIT (SCC)]] bwUniCluster wiring © KIT (SCC)
|}

= Registration =

Granting access and issuing a user account for '''bwUniCluster 2.0''' requires the registration at the KIT service website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] (step B). However, this registration depends on the '''bwUniCluster entitlement''' issued by your university (step A).
 

== Step A: bwUniCluster entitlement for registration ==
'''The entitlement is called bwUniCluster (not bwUniCluster 2.0)''' and each university issues the bwUniCluster entitlement '''only''' for their own respective members. Some have established on-line processes or provide downloads of the entitlement application forms. If there is no link behind the name of an institution in the following list, please contact the local IT support services:

* [[BwCluster_User_Access_Uni_Freiburg|Albert Ludwig University of Freiburg]]
* [https://bwunicluster.urz.uni-heidelberg.de/ Heidelberg University]
* [https://kim.uni-hohenheim.de/bwhpc-account University of Hohenheim]
* [http://www.scc.kit.edu/downloads/sdo/Antrag_Benutzernummer_bwUniCluster.pdf Karlsruhe Institute of Technology (KIT)]
* [[BWUniCluster_User_Access_Members_Uni_Konstanz|University of Konstanz]]
* [[BWUniCluster_User_Access_Members_Uni_Mannheim|University of Mannheim]]
* [https://www.hlrs.de/solutions-services/academic-users/bwunicluster-access/ University of Stuttgart]
* [https://uni-tuebingen.de/de/155157 Eberhard Karls University Tübingen]
* [[BWUniCluster_User_Access_Members_Uni_Ulm|Ulm University]]
* Hochschule Aalen
* Hochschule Albstadt-Sigmaringen
* Hochschule Esslingen
* Hochschule Furtwangen
* Hochschule Heilbronn
* Hochschule Karlsruhe
* Hochschule Konstanz
* Hochschule Reutlingen
* Hochschule Rottenburg
* Hochschule Stuttgart (HfT)
* Hochschule Ulm
 

== Step B: Web Registration, service password and 2-factor authentication ==

After completing step A, i.e., after successfull issueing of the bwUniCluster entitlement, you have to register yourself for the service. To do so please visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and complete the following steps.

1. Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.

[[File:Bwidm-1-red.png|center|border|]]
 

2. You will be directed to the ''Identity Provider'' of your home organisation. Enter the user ID / username and password of your home organisation - this is usually the same password used for your e-mail account and other services - and click on '''Login''', '''Einloggen''' or something similar.
 

3. You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]. If you are logging into bwIDM for the first time, there will be a summary screen which shows the account details your home institution is providing to the central system. Please check that all data is valid and then click on '''Continue''' or '''Weiter'''.
 

4. Once you have successfully logged into the bwIDM system, you will be greeted by a home screen showing all state-wide services you have access to. There will be a box labelled "bwUniCluster". Click on '''Register''' or '''Registrieren''' to start the registration process.

[[File:Bwidm-2-red.png|center|border|]]
 

5. Since August 13, 2020 a '''2-factor authentication''' mechanism (2FA) is being enforced to improve security. If you have never registered a 2FA token on bwIDM before, the following error message will appear:

[[File:BwUniCluster 2.0 access login bwidm registration requirements 2fa.png|center|]]

Click on the [https://bwidm.scc.kit.edu/user/twofa.xhtml Link] or on the '''My Tokens''' link in the main menu. The instructions for registering a new 2FA token can be found on the following page: [[bwUniCluster 2.0 User Access/2FA Tokens]]. Please complete them before proceeding.
 

6. Make sure all requirements are met by checking the '''Requirements''' box at the top. If the requirements are not met you might be able to correct the issure by following the instructions. In all other cases please [[Registration_Support_-_bwUniCluster|contact your local hotline]].

[[File:BwUniCluster 2.0 access login bwidm registration requirements.png|center|border|]]
 

7. Read the Terms of Use ('''Nutzungsbedingungen und -richtlinien'''), check the box besides '''I have read and accepted the terms of use''' and click on '''Register''' or '''Registrieren'''.
 

8. Set a service password for the bwUniCluster and click on '''Save''' or '''Speichern'''. Logging in with the password of your home organisation, like on the former bwUniCluster 1, is no longer possible. Please make sure to use a strong password which is different from any other password you are currently using or have used on other systems. You will also be asked to change the service password regularly.

[[File:BwUniCluster 2.0 access login bwidm service password.png|center|]]
 
 

== Step C: Fill out the bwUniCluster questionnaire ==

Filling out the bwUniCluster questionaire on

https://zas.bwhpc.de/shib/en/bwunicluster_survey.php

is mandatory for all users. The input is solely used to improve our support activities and for capacity planning of future HPC resources. '''If the questionaire is not filled out, access to bwUniCluster 2.0 is blocked 14 days after the registration.'''
 
 

== Changing the Service Password ==

Your bwUniCluster 2.0 '''password''' is the service password you set during the web registration (compare step 7 of chapter 1.2). At any time, you can set a new bwUniCluster 2.0 password via the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] by carrying out the following steps:
# Go to [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and select your home organization
# Authenticate yourself via the user id / username and password provided by your home institution
# Find the entry '''bwUniCluster''' and select '''Set Service Password'''
# Enter the new password, repeat it and click '''Save''' button.
# If the change was sucessfull, the message "Das Passwort wurde bei dem Dienst geändert" ("Password has been changed") will be shown
# Proceed to log in using the new password
 
 
== Contact / Support ==
If you have questions or problems concerning the bwUniCluster (2.0) registration, please [[bwUniCluster 2.0 Support|contact your local hotline]].
 
 

= Establishing network access =

Access to bwUniCluster 2.0 is '''limited to IP addresses from the so-called BelWü networks'''. All home institutions of our current users are connected to BelWue, so if you are on your campus network (e.g. in your office or on the Campus WiFi) you should be able to connect to bwUniCluster 2.0 without restrictions. If you are outside of one of the BelWue networks (e.g. in your home office instead of in your campus office), a VPN connection to your home institution has to be established first (see e.g. [1] for the KIT).
 
 

= Login =

After finishing the web registration and making sure that you are on a network from which you have access to bwUniCluster 2.0 (e.g. by establishing a VPN connection), the HPC cluster is ready for your '''SSH''' based login. Recommended SSH clients applications are:

* the ssh (OpenSSH) command included in all Linux distributions and macOS, -in command under Linux and macOS using the application ''terminal''
* [http://mobaxterm.mobatek.net/ MobaXterm] under Windows
 

== Hostnames ==

The main hostname required to connect to bwUniCluster 2.0 is '''bwunicluster.scc.kit.edu''' or '''uc2.scc.kit.edu'''. The system has four login nodes and we use so-called ''DNS round-robin scheduling'' to load-balance the incoming connections between the nodes. If you open multiple SSH sessions to bwUniCluster 2.0, these sessions will be established to different login nodes, so processes started in one session might not be visible in other sessions.

The older Broadwell extension partition of the former bwUniCluster 1 is connected to bwUniCluster 2.0. You can use the hostname '''uc1e.scc.kit.edu''' to connect to the login nodes of this partition.

If you need to connect to specific login nodes, you can use the following hostnames:

{| class="wikitable"
! Hostname !! Node type
|-
| '''uc2-login1.scc.kit.edu''' || bwUniCluster 2.0, first login node
|-
| '''uc2-login2.scc.kit.edu''' || bwUniCluster 2.0, second login node
|-
| '''uc2-login3.scc.kit.edu''' || bwUniCluster 2.0, third login node
|-
| '''uc2-login4.scc.kit.edu''' || bwUniCluster 2.0, fourth login node
|-
| '''uc1e-login1.scc.kit.edu''' || Broadwell partition, first login node
|-
| '''uc1e-login2.scc.kit.edu''' || Broadwell partition, second login node
|-
|}

Only the secure shell ''SSH'' is allowed to login. Other protocols like ''telnet'' or ''rlogin'' are not allowed for security reasons.
 

== Usernames ==

Your username will be the same as the one provided by your home institution, but '''prefixed''' with two characters and an underscore indicating your home institution. For example: If you are a member of the university of Konstanz and your local username is ab1234, your username on bwUniCluster 2.0 is kn_ab1234.

The following list contains all prefixes currently in use:

{| class="wikitable"
! Home organization !! <UserID>
|-
| Universität Freiburg || ''fr_''username
|-
| Universität Heidelberg || ''hd_''username
|-
| Universität Hohenheim || ''ho_''username
|-
| KIT || username ''(without any prefix)''
|-
| Universität Konstanz || ''kn_''username
|-
| Universität Mannheim || ''ma_''username
|-
| Universität Stuttgart || ''st_''username
|-
| Universität Tübingen || ''tu_''username
|-
| Universität Ulm || ''ul_''username
|-
| Hochschule Aalen || ''aa_''username
|-
| Hochschule Albstadt-Sigmaringen || ''as_''username
|-
| Hochschule Esslingen || ''es_''username
|-
| Hochschule Furtwangen || ''hf_''username
|-
| Hochschule Heilbronn || ''hn_''username
|-
| Hochschule Karlsruhe || ''hk_''username
|-
| HTWG Konstanz || ''ht_''username
|-
| Hochschule Reutlingen || ''hr_''username
|-
| Hochschule Rottenburg || ''ro_''username
|-
| Hochschule für Technik Stuttgart || ''hs_''username
|-
| Hochschule Ulm || ''hu_''username
|-
|}
 

== Client application: OpenSSH ==

Most Unix and Unix-like operating systems like Linux, macOS and *BSD come with a built-in SSH client provided by the OpenSSH project. More recent versions of Windows 10 and the Windows Subsystem for Linux also come with a built-in OpenSSH client.

To use this client, simply open a command line terminal (the exact process differs on every operating system, but usually involves starting an application called '''Terminal''' or '''Command Prompt''') and enter the following command to connect to bwUniCluster 2.0:

<pre>
$ ssh <UserID>@bwunicluster.scc.kit.edu
</pre>

If you are on a Linux or Unix system running the X Window System (X11) and want to use a GUI-based application on bwUniCluster 2.0, you can use the ''-X'' option for the ssh command to set up X11 forwarding:

<pre>
$ ssh -X <UserID>@uc2.scc.kit.edu
</pre>

Windows users requiring X11 forwarding for graphical applications should use '''MobaXterm''' instead.
 

== Client application: MobaXterm ==

The bwHPC-C5 support team strongly recommends to use [http://mobaxterm.mobatek.net/ MobaXterm] instead of ''PuTTY'' or ''WinSCP'' on Windows. ''MobaXterm'' provides a built-in X11 server allowing to start GUI based software.

Start ''MobaXterm'', fill in the following fields:
<pre>
Remote name : uc2.scc.kit.edu # or uc1e.scc.kit.edu
Specify user name : <UserID>
Port : 22
</pre>

After that click on 'ok'. Then a terminal will be opened and there you can enter your credentials.
 

== Example login process ==

After the connection has been initiated, a successful login process will go through the following three steps:

1. The system asks for a '''One-Time Password'''. Generate one using the Software or Hardware Token registered on the bwIDM system (see [[bwUniCluster 2.0 User Access/2FA Tokens]]) and enter it after the '''Your OTP:''' prompt.

2. The systems asks for your service password. Enter it after the '''Password:''' prompt.

3. You are greeted by the bwUniCluster 2.0 banner followed by a shell.

The result should look like this:

[[File:BwUniCluster 2.0 access login example.png|center|]]
 

== Troubleshooting ==

'''Issue: The "Your OTP:" prompt never appears and the connection hangs/times out instead'''

Likely cause: You are most likely not on a network from which access to the bwUniCluster 2.0 system is allowed. Please check if you might have to establish a VPN connection first.

 

'''Issue: The system asks for the One-Time Password multiple times'''

Likely cause: Make sure you are using the correct Software Token to generate the One-Time Password.

 

'''Issue: The system asks for the service password multiple times'''

Likely cause: Make sure you are using the service password set on bwIDM and not the password valid for your home institution. Unlike the bwUniCluster 1, the bwUniCluster 2.0 only accepts the service password.

 

'''Issue: There is an error message by the pam_ses_open.sh skript'''

Likely cause: Your account is in the "LOST_ACCESS" state because the entitlement is no longer valid, the questionaire was not filled out or there was a problem during the communication between your home institution and the central bwIDM system. Please try the following steps:

* Log into [https://bwidm.scc.kit.edu bwIDM], look for the bwUniCluster entry and click on '''Registry info'''. Your "Status:" should be "ACTIVE". If it is not, please wait for ten minutes since logging into the bwIDM causes a refresh and the problem might fix itself. If the status does not change to ACTIVE after a longer amount of time, please contact the support channels.

* If you have not filled out the questionaire, please do so on [https://zas.bwhpc.de/shib/en/bwunicluster_survey.php https://zas.bwhpc.de/shib/en/bwunicluster_survey.php] and then wait for about ten minutes before attempting to log into the HPC system again.
 
 

== Allowed activities on login nodes ==

The login nodes of bwUniCluster 2.0 are the access point to the compute system and to your bwUniCluster 2.0 $HOME directory. The login nodes are shared with all the users of bwUniCluster 2.0. Therefore, your activities on the login nodes are limited to primarily set up your batch jobs. Your activities may also be:

* '''short''' compilation of your program code and
* '''short''' pre- and postprocessing of your batch jobs.

To guarantee usability for all the users of bwUniCluster 2.0 '''you must not run your compute jobs on the login nodes'''. Compute jobs must be submitted to the
[[bwUniCluster Batch Jobs|queueing system]]. Any compute job running on the login nodes will be terminated without any notice. Any long-running compilation or any long-running pre- or postprocessing of batch jobs must also be submitted to the [[bwUniCluster Batch Jobs|queueing system]].
 
 

== SSH Keys ==

In contrast to the bwUniCluster 1 and many other HPC systems it is '''no longer possible to self-manage your SSH Keys by adding them to the ~/.ssh/authorized_keys file'''. Existing files will no longer be evaluated. SSH Keys have to be managed via the central bwIDM system instead. Please refer to the user guide for this functionality:

[[bwUniCluster 2.0 User Access/SSH Keys]]
 
 

= [[First_Steps_on_bwHPC_cluster|First steps on bwUniCluster]] =

First and some important steps on bwUniCluster 2.0 can be found [[First_Steps_on_bwHPC_cluster|here]].
 
 

= Deregistration =
If you plan to permanently leave the bwUniCluster 2.0, follow the deregister checklist:
# Transfer all your data in $HOME and workspace to your local computer/storage and after that clear off all your data
# Visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
#* Select your home organization from the list and click '''Proceed'''
#* Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
#* You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
#* <div>Select '''Registry Info''' of the service '''bwUniCluster''' (on the left hand side) [[File:bwUniCluster_registration_sidebar.png|center|border|]]</div>
#* Click '''Deregister'''
Note that Step 2 will automatically unsubscribe you from the bwunicluster mail list.

BwUniCluster 2.0 User Access

2020-12-17T15:10:24Z

H Haefner: /* Step B: Web Registration, service password and 2-factor authentication */

[[bwUniCluster_2.0|bwUniCluster 2.0]] is Baden-Württemberg's general purpose tier 3 high performance computing (HPC)
cluster co-financed by Baden-Württemberg's ministry of science, research and arts and the shareholders:

* Albert Ludwig University of Freiburg
* Eberhard Karls University, Tübingen
* Karlsruhe Institute of Technology (KIT)
* Heidelberg University (Ruprecht-Karls-Universität Heidelberg)
* Ulm University
* University of Hohenheim
* University of Konstanz
* University of Mannheim
* University of Stuttgart
* HAW BW e.V. (an association of several universities of applied sciences in Baden-Württemberg, see below)
 
To '''log on''' [[bwUniCluster_2.0|bwUniCluster 2.0]] a user account is required. All members of the shareholder
universities can apply for an account.
 

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:left; color:#000;vertical-align:top;" |__TOC__
| [[File:bwUniCluster_17Jan2014_p044-rot_t10.10.00.jpg|center|border|250px|bwUniCluster wiring by Holger Obermaier, copyright: KIT (SCC)]] bwUniCluster wiring © KIT (SCC)
|}

= Registration =

Granting access and issuing a user account for '''bwUniCluster 2.0''' requires the registration at the KIT service website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] (step B). However, this registration depends on the '''bwUniCluster entitlement''' issued by your university (step A).
 

== Step A: bwUniCluster entitlement for registration ==
'''The entitlement is called bwUniCluster (not bwUniCluster 2.0)''' and each university issues the bwUniCluster entitlement '''only''' for their own respective members. Some have established on-line processes or provide downloads of the entitlement application forms. If there is no link behind the name of an institution in the following list, please contact the local IT support services:

* [[BwCluster_User_Access_Uni_Freiburg|Albert Ludwig University of Freiburg]]
* [https://bwunicluster.urz.uni-heidelberg.de/ Heidelberg University]
* [https://kim.uni-hohenheim.de/bwhpc-account University of Hohenheim]
* [http://www.scc.kit.edu/downloads/sdo/Antrag_Benutzernummer_bwUniCluster.pdf Karlsruhe Institute of Technology (KIT)]
* [[BWUniCluster_User_Access_Members_Uni_Konstanz|University of Konstanz]]
* [[BWUniCluster_User_Access_Members_Uni_Mannheim|University of Mannheim]]
* [https://www.hlrs.de/solutions-services/academic-users/bwunicluster-access/ University of Stuttgart]
* [https://uni-tuebingen.de/de/155157 Eberhard Karls University Tübingen]
* [[BWUniCluster_User_Access_Members_Uni_Ulm|Ulm University]]
* Hochschule Aalen
* Hochschule Albstadt-Sigmaringen
* Hochschule Esslingen
* Hochschule Furtwangen
* Hochschule Heilbronn
* Hochschule Karlsruhe
* Hochschule Konstanz
* Hochschule Reutlingen
* Hochschule Rottenburg
* Hochschule Stuttgart (HfT)
* Hochschule Ulm
 

== Step B: Web Registration, service password and 2-factor authentication ==

After completing step A, i.e., after successfull issueing of the bwUniCluster entitlement, you have to register yourself for the service. To do so please visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and complete the following steps.

1. Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.

[[File:BwIDM-1.png|center|border|]]
 

2. You will be directed to the ''Identity Provider'' of your home organisation. Enter the user ID / username and password of your home organisation - this is usually the same password used for your e-mail account and other services - and click on '''Login''', '''Einloggen''' or something similar.
 

3. You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]. If you are logging into bwIDM for the first time, there will be a summary screen which shows the account details your home institution is providing to the central system. Please check that all data is valid and then click on '''Continue''' or '''Weiter'''.
 

4. Once you have successfully logged into the bwIDM system, you will be greeted by a home screen showing all state-wide services you have access to. There will be a box labelled "bwUniCluster". Click on '''Register''' or '''Registrieren''' to start the registration process.

[[File:Bwidm-2-red.png|center|border|]]
 

5. Since August 13, 2020 a '''2-factor authentication''' mechanism (2FA) is being enforced to improve security. If you have never registered a 2FA token on bwIDM before, the following error message will appear:

[[File:BwUniCluster 2.0 access login bwidm registration requirements 2fa.png|center|]]

Click on the [https://bwidm.scc.kit.edu/user/twofa.xhtml Link] or on the '''My Tokens''' link in the main menu. The instructions for registering a new 2FA token can be found on the following page: [[bwUniCluster 2.0 User Access/2FA Tokens]]. Please complete them before proceeding.
 

6. Make sure all requirements are met by checking the '''Requirements''' box at the top. If the requirements are not met you might be able to correct the issure by following the instructions. In all other cases please [[Registration_Support_-_bwUniCluster|contact your local hotline]].

[[File:BwUniCluster 2.0 access login bwidm registration requirements.png|center|border|]]
 

7. Read the Terms of Use ('''Nutzungsbedingungen und -richtlinien'''), check the box besides '''I have read and accepted the terms of use''' and click on '''Register''' or '''Registrieren'''.
 

8. Set a service password for the bwUniCluster and click on '''Save''' or '''Speichern'''. Logging in with the password of your home organisation, like on the former bwUniCluster 1, is no longer possible. Please make sure to use a strong password which is different from any other password you are currently using or have used on other systems. You will also be asked to change the service password regularly.

[[File:BwUniCluster 2.0 access login bwidm service password.png|center|]]
 
 

== Step C: Fill out the bwUniCluster questionnaire ==

Filling out the bwUniCluster questionaire on

https://zas.bwhpc.de/shib/en/bwunicluster_survey.php

is mandatory for all users. The input is solely used to improve our support activities and for capacity planning of future HPC resources. '''If the questionaire is not filled out, access to bwUniCluster 2.0 is blocked 14 days after the registration.'''
 
 

== Changing the Service Password ==

Your bwUniCluster 2.0 '''password''' is the service password you set during the web registration (compare step 7 of chapter 1.2). At any time, you can set a new bwUniCluster 2.0 password via the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] by carrying out the following steps:
# Go to [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and select your home organization
# Authenticate yourself via the user id / username and password provided by your home institution
# Find the entry '''bwUniCluster''' and select '''Set Service Password'''
# Enter the new password, repeat it and click '''Save''' button.
# If the change was sucessfull, the message "Das Passwort wurde bei dem Dienst geändert" ("Password has been changed") will be shown
# Proceed to log in using the new password
 
 
== Contact / Support ==
If you have questions or problems concerning the bwUniCluster (2.0) registration, please [[bwUniCluster 2.0 Support|contact your local hotline]].
 
 

= Establishing network access =

Access to bwUniCluster 2.0 is '''limited to IP addresses from the so-called BelWü networks'''. All home institutions of our current users are connected to BelWue, so if you are on your campus network (e.g. in your office or on the Campus WiFi) you should be able to connect to bwUniCluster 2.0 without restrictions. If you are outside of one of the BelWue networks (e.g. in your home office instead of in your campus office), a VPN connection to your home institution has to be established first (see e.g. [1] for the KIT).
 
 

= Login =

After finishing the web registration and making sure that you are on a network from which you have access to bwUniCluster 2.0 (e.g. by establishing a VPN connection), the HPC cluster is ready for your '''SSH''' based login. Recommended SSH clients applications are:

* the ssh (OpenSSH) command included in all Linux distributions and macOS, -in command under Linux and macOS using the application ''terminal''
* [http://mobaxterm.mobatek.net/ MobaXterm] under Windows
 

== Hostnames ==

The main hostname required to connect to bwUniCluster 2.0 is '''bwunicluster.scc.kit.edu''' or '''uc2.scc.kit.edu'''. The system has four login nodes and we use so-called ''DNS round-robin scheduling'' to load-balance the incoming connections between the nodes. If you open multiple SSH sessions to bwUniCluster 2.0, these sessions will be established to different login nodes, so processes started in one session might not be visible in other sessions.

The older Broadwell extension partition of the former bwUniCluster 1 is connected to bwUniCluster 2.0. You can use the hostname '''uc1e.scc.kit.edu''' to connect to the login nodes of this partition.

If you need to connect to specific login nodes, you can use the following hostnames:

{| class="wikitable"
! Hostname !! Node type
|-
| '''uc2-login1.scc.kit.edu''' || bwUniCluster 2.0, first login node
|-
| '''uc2-login2.scc.kit.edu''' || bwUniCluster 2.0, second login node
|-
| '''uc2-login3.scc.kit.edu''' || bwUniCluster 2.0, third login node
|-
| '''uc2-login4.scc.kit.edu''' || bwUniCluster 2.0, fourth login node
|-
| '''uc1e-login1.scc.kit.edu''' || Broadwell partition, first login node
|-
| '''uc1e-login2.scc.kit.edu''' || Broadwell partition, second login node
|-
|}

Only the secure shell ''SSH'' is allowed to login. Other protocols like ''telnet'' or ''rlogin'' are not allowed for security reasons.
 

== Usernames ==

Your username will be the same as the one provided by your home institution, but '''prefixed''' with two characters and an underscore indicating your home institution. For example: If you are a member of the university of Konstanz and your local username is ab1234, your username on bwUniCluster 2.0 is kn_ab1234.

The following list contains all prefixes currently in use:

{| class="wikitable"
! Home organization !! <UserID>
|-
| Universität Freiburg || ''fr_''username
|-
| Universität Heidelberg || ''hd_''username
|-
| Universität Hohenheim || ''ho_''username
|-
| KIT || username ''(without any prefix)''
|-
| Universität Konstanz || ''kn_''username
|-
| Universität Mannheim || ''ma_''username
|-
| Universität Stuttgart || ''st_''username
|-
| Universität Tübingen || ''tu_''username
|-
| Universität Ulm || ''ul_''username
|-
| Hochschule Aalen || ''aa_''username
|-
| Hochschule Albstadt-Sigmaringen || ''as_''username
|-
| Hochschule Esslingen || ''es_''username
|-
| Hochschule Furtwangen || ''hf_''username
|-
| Hochschule Heilbronn || ''hn_''username
|-
| Hochschule Karlsruhe || ''hk_''username
|-
| HTWG Konstanz || ''ht_''username
|-
| Hochschule Reutlingen || ''hr_''username
|-
| Hochschule Rottenburg || ''ro_''username
|-
| Hochschule für Technik Stuttgart || ''hs_''username
|-
| Hochschule Ulm || ''hu_''username
|-
|}
 

== Client application: OpenSSH ==

Most Unix and Unix-like operating systems like Linux, macOS and *BSD come with a built-in SSH client provided by the OpenSSH project. More recent versions of Windows 10 and the Windows Subsystem for Linux also come with a built-in OpenSSH client.

To use this client, simply open a command line terminal (the exact process differs on every operating system, but usually involves starting an application called '''Terminal''' or '''Command Prompt''') and enter the following command to connect to bwUniCluster 2.0:

<pre>
$ ssh <UserID>@bwunicluster.scc.kit.edu
</pre>

If you are on a Linux or Unix system running the X Window System (X11) and want to use a GUI-based application on bwUniCluster 2.0, you can use the ''-X'' option for the ssh command to set up X11 forwarding:

<pre>
$ ssh -X <UserID>@uc2.scc.kit.edu
</pre>

Windows users requiring X11 forwarding for graphical applications should use '''MobaXterm''' instead.
 

== Client application: MobaXterm ==

The bwHPC-C5 support team strongly recommends to use [http://mobaxterm.mobatek.net/ MobaXterm] instead of ''PuTTY'' or ''WinSCP'' on Windows. ''MobaXterm'' provides a built-in X11 server allowing to start GUI based software.

Start ''MobaXterm'', fill in the following fields:
<pre>
Remote name : uc2.scc.kit.edu # or uc1e.scc.kit.edu
Specify user name : <UserID>
Port : 22
</pre>

After that click on 'ok'. Then a terminal will be opened and there you can enter your credentials.
 

== Example login process ==

After the connection has been initiated, a successful login process will go through the following three steps:

1. The system asks for a '''One-Time Password'''. Generate one using the Software or Hardware Token registered on the bwIDM system (see [[bwUniCluster 2.0 User Access/2FA Tokens]]) and enter it after the '''Your OTP:''' prompt.

2. The systems asks for your service password. Enter it after the '''Password:''' prompt.

3. You are greeted by the bwUniCluster 2.0 banner followed by a shell.

The result should look like this:

[[File:BwUniCluster 2.0 access login example.png|center|]]
 

== Troubleshooting ==

'''Issue: The "Your OTP:" prompt never appears and the connection hangs/times out instead'''

Likely cause: You are most likely not on a network from which access to the bwUniCluster 2.0 system is allowed. Please check if you might have to establish a VPN connection first.

 

'''Issue: The system asks for the One-Time Password multiple times'''

Likely cause: Make sure you are using the correct Software Token to generate the One-Time Password.

 

'''Issue: The system asks for the service password multiple times'''

Likely cause: Make sure you are using the service password set on bwIDM and not the password valid for your home institution. Unlike the bwUniCluster 1, the bwUniCluster 2.0 only accepts the service password.

 

'''Issue: There is an error message by the pam_ses_open.sh skript'''

Likely cause: Your account is in the "LOST_ACCESS" state because the entitlement is no longer valid, the questionaire was not filled out or there was a problem during the communication between your home institution and the central bwIDM system. Please try the following steps:

* Log into [https://bwidm.scc.kit.edu bwIDM], look for the bwUniCluster entry and click on '''Registry info'''. Your "Status:" should be "ACTIVE". If it is not, please wait for ten minutes since logging into the bwIDM causes a refresh and the problem might fix itself. If the status does not change to ACTIVE after a longer amount of time, please contact the support channels.

* If you have not filled out the questionaire, please do so on [https://zas.bwhpc.de/shib/en/bwunicluster_survey.php https://zas.bwhpc.de/shib/en/bwunicluster_survey.php] and then wait for about ten minutes before attempting to log into the HPC system again.
 
 

== Allowed activities on login nodes ==

The login nodes of bwUniCluster 2.0 are the access point to the compute system and to your bwUniCluster 2.0 $HOME directory. The login nodes are shared with all the users of bwUniCluster 2.0. Therefore, your activities on the login nodes are limited to primarily set up your batch jobs. Your activities may also be:

* '''short''' compilation of your program code and
* '''short''' pre- and postprocessing of your batch jobs.

To guarantee usability for all the users of bwUniCluster 2.0 '''you must not run your compute jobs on the login nodes'''. Compute jobs must be submitted to the
[[bwUniCluster Batch Jobs|queueing system]]. Any compute job running on the login nodes will be terminated without any notice. Any long-running compilation or any long-running pre- or postprocessing of batch jobs must also be submitted to the [[bwUniCluster Batch Jobs|queueing system]].
 
 

== SSH Keys ==

In contrast to the bwUniCluster 1 and many other HPC systems it is '''no longer possible to self-manage your SSH Keys by adding them to the ~/.ssh/authorized_keys file'''. Existing files will no longer be evaluated. SSH Keys have to be managed via the central bwIDM system instead. Please refer to the user guide for this functionality:

[[bwUniCluster 2.0 User Access/SSH Keys]]
 
 

= [[First_Steps_on_bwHPC_cluster|First steps on bwUniCluster]] =

First and some important steps on bwUniCluster 2.0 can be found [[First_Steps_on_bwHPC_cluster|here]].
 
 

= Deregistration =
If you plan to permanently leave the bwUniCluster 2.0, follow the deregister checklist:
# Transfer all your data in $HOME and workspace to your local computer/storage and after that clear off all your data
# Visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
#* Select your home organization from the list and click '''Proceed'''
#* Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
#* You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
#* <div>Select '''Registry Info''' of the service '''bwUniCluster''' (on the left hand side) [[File:bwUniCluster_registration_sidebar.png|center|border|]]</div>
#* Click '''Deregister'''
Note that Step 2 will automatically unsubscribe you from the bwunicluster mail list.

BwUniCluster 2.0 User Access

2020-12-17T15:10:07Z

H Haefner: /* Step B: Web Registration, service password and 2-factor authentication */

[[bwUniCluster_2.0|bwUniCluster 2.0]] is Baden-Württemberg's general purpose tier 3 high performance computing (HPC)
cluster co-financed by Baden-Württemberg's ministry of science, research and arts and the shareholders:

* Albert Ludwig University of Freiburg
* Eberhard Karls University, Tübingen
* Karlsruhe Institute of Technology (KIT)
* Heidelberg University (Ruprecht-Karls-Universität Heidelberg)
* Ulm University
* University of Hohenheim
* University of Konstanz
* University of Mannheim
* University of Stuttgart
* HAW BW e.V. (an association of several universities of applied sciences in Baden-Württemberg, see below)
 
To '''log on''' [[bwUniCluster_2.0|bwUniCluster 2.0]] a user account is required. All members of the shareholder
universities can apply for an account.
 

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:left; color:#000;vertical-align:top;" |__TOC__
| [[File:bwUniCluster_17Jan2014_p044-rot_t10.10.00.jpg|center|border|250px|bwUniCluster wiring by Holger Obermaier, copyright: KIT (SCC)]] bwUniCluster wiring © KIT (SCC)
|}

= Registration =

Granting access and issuing a user account for '''bwUniCluster 2.0''' requires the registration at the KIT service website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] (step B). However, this registration depends on the '''bwUniCluster entitlement''' issued by your university (step A).
 

== Step A: bwUniCluster entitlement for registration ==
'''The entitlement is called bwUniCluster (not bwUniCluster 2.0)''' and each university issues the bwUniCluster entitlement '''only''' for their own respective members. Some have established on-line processes or provide downloads of the entitlement application forms. If there is no link behind the name of an institution in the following list, please contact the local IT support services:

* [[BwCluster_User_Access_Uni_Freiburg|Albert Ludwig University of Freiburg]]
* [https://bwunicluster.urz.uni-heidelberg.de/ Heidelberg University]
* [https://kim.uni-hohenheim.de/bwhpc-account University of Hohenheim]
* [http://www.scc.kit.edu/downloads/sdo/Antrag_Benutzernummer_bwUniCluster.pdf Karlsruhe Institute of Technology (KIT)]
* [[BWUniCluster_User_Access_Members_Uni_Konstanz|University of Konstanz]]
* [[BWUniCluster_User_Access_Members_Uni_Mannheim|University of Mannheim]]
* [https://www.hlrs.de/solutions-services/academic-users/bwunicluster-access/ University of Stuttgart]
* [https://uni-tuebingen.de/de/155157 Eberhard Karls University Tübingen]
* [[BWUniCluster_User_Access_Members_Uni_Ulm|Ulm University]]
* Hochschule Aalen
* Hochschule Albstadt-Sigmaringen
* Hochschule Esslingen
* Hochschule Furtwangen
* Hochschule Heilbronn
* Hochschule Karlsruhe
* Hochschule Konstanz
* Hochschule Reutlingen
* Hochschule Rottenburg
* Hochschule Stuttgart (HfT)
* Hochschule Ulm
 

== Step B: Web Registration, service password and 2-factor authentication ==

After completing step A, i.e., after successfull issueing of the bwUniCluster entitlement, you have to register yourself for the service. To do so please visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and complete the following steps.

1. Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.

[[File:Bwidm-1.png|center|border|]]
 

2. You will be directed to the ''Identity Provider'' of your home organisation. Enter the user ID / username and password of your home organisation - this is usually the same password used for your e-mail account and other services - and click on '''Login''', '''Einloggen''' or something similar.
 

3. You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]. If you are logging into bwIDM for the first time, there will be a summary screen which shows the account details your home institution is providing to the central system. Please check that all data is valid and then click on '''Continue''' or '''Weiter'''.
 

4. Once you have successfully logged into the bwIDM system, you will be greeted by a home screen showing all state-wide services you have access to. There will be a box labelled "bwUniCluster". Click on '''Register''' or '''Registrieren''' to start the registration process.

[[File:Bwidm-2-red.png|center|border|]]
 

5. Since August 13, 2020 a '''2-factor authentication''' mechanism (2FA) is being enforced to improve security. If you have never registered a 2FA token on bwIDM before, the following error message will appear:

[[File:BwUniCluster 2.0 access login bwidm registration requirements 2fa.png|center|]]

Click on the [https://bwidm.scc.kit.edu/user/twofa.xhtml Link] or on the '''My Tokens''' link in the main menu. The instructions for registering a new 2FA token can be found on the following page: [[bwUniCluster 2.0 User Access/2FA Tokens]]. Please complete them before proceeding.
 

6. Make sure all requirements are met by checking the '''Requirements''' box at the top. If the requirements are not met you might be able to correct the issure by following the instructions. In all other cases please [[Registration_Support_-_bwUniCluster|contact your local hotline]].

[[File:BwUniCluster 2.0 access login bwidm registration requirements.png|center|border|]]
 

7. Read the Terms of Use ('''Nutzungsbedingungen und -richtlinien'''), check the box besides '''I have read and accepted the terms of use''' and click on '''Register''' or '''Registrieren'''.
 

8. Set a service password for the bwUniCluster and click on '''Save''' or '''Speichern'''. Logging in with the password of your home organisation, like on the former bwUniCluster 1, is no longer possible. Please make sure to use a strong password which is different from any other password you are currently using or have used on other systems. You will also be asked to change the service password regularly.

[[File:BwUniCluster 2.0 access login bwidm service password.png|center|]]
 
 

== Step C: Fill out the bwUniCluster questionnaire ==

Filling out the bwUniCluster questionaire on

https://zas.bwhpc.de/shib/en/bwunicluster_survey.php

is mandatory for all users. The input is solely used to improve our support activities and for capacity planning of future HPC resources. '''If the questionaire is not filled out, access to bwUniCluster 2.0 is blocked 14 days after the registration.'''
 
 

== Changing the Service Password ==

Your bwUniCluster 2.0 '''password''' is the service password you set during the web registration (compare step 7 of chapter 1.2). At any time, you can set a new bwUniCluster 2.0 password via the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] by carrying out the following steps:
# Go to [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and select your home organization
# Authenticate yourself via the user id / username and password provided by your home institution
# Find the entry '''bwUniCluster''' and select '''Set Service Password'''
# Enter the new password, repeat it and click '''Save''' button.
# If the change was sucessfull, the message "Das Passwort wurde bei dem Dienst geändert" ("Password has been changed") will be shown
# Proceed to log in using the new password
 
 
== Contact / Support ==
If you have questions or problems concerning the bwUniCluster (2.0) registration, please [[bwUniCluster 2.0 Support|contact your local hotline]].
 
 

= Establishing network access =

Access to bwUniCluster 2.0 is '''limited to IP addresses from the so-called BelWü networks'''. All home institutions of our current users are connected to BelWue, so if you are on your campus network (e.g. in your office or on the Campus WiFi) you should be able to connect to bwUniCluster 2.0 without restrictions. If you are outside of one of the BelWue networks (e.g. in your home office instead of in your campus office), a VPN connection to your home institution has to be established first (see e.g. [1] for the KIT).
 
 

= Login =

After finishing the web registration and making sure that you are on a network from which you have access to bwUniCluster 2.0 (e.g. by establishing a VPN connection), the HPC cluster is ready for your '''SSH''' based login. Recommended SSH clients applications are:

* the ssh (OpenSSH) command included in all Linux distributions and macOS, -in command under Linux and macOS using the application ''terminal''
* [http://mobaxterm.mobatek.net/ MobaXterm] under Windows
 

== Hostnames ==

The main hostname required to connect to bwUniCluster 2.0 is '''bwunicluster.scc.kit.edu''' or '''uc2.scc.kit.edu'''. The system has four login nodes and we use so-called ''DNS round-robin scheduling'' to load-balance the incoming connections between the nodes. If you open multiple SSH sessions to bwUniCluster 2.0, these sessions will be established to different login nodes, so processes started in one session might not be visible in other sessions.

The older Broadwell extension partition of the former bwUniCluster 1 is connected to bwUniCluster 2.0. You can use the hostname '''uc1e.scc.kit.edu''' to connect to the login nodes of this partition.

If you need to connect to specific login nodes, you can use the following hostnames:

{| class="wikitable"
! Hostname !! Node type
|-
| '''uc2-login1.scc.kit.edu''' || bwUniCluster 2.0, first login node
|-
| '''uc2-login2.scc.kit.edu''' || bwUniCluster 2.0, second login node
|-
| '''uc2-login3.scc.kit.edu''' || bwUniCluster 2.0, third login node
|-
| '''uc2-login4.scc.kit.edu''' || bwUniCluster 2.0, fourth login node
|-
| '''uc1e-login1.scc.kit.edu''' || Broadwell partition, first login node
|-
| '''uc1e-login2.scc.kit.edu''' || Broadwell partition, second login node
|-
|}

Only the secure shell ''SSH'' is allowed to login. Other protocols like ''telnet'' or ''rlogin'' are not allowed for security reasons.
 

== Usernames ==

Your username will be the same as the one provided by your home institution, but '''prefixed''' with two characters and an underscore indicating your home institution. For example: If you are a member of the university of Konstanz and your local username is ab1234, your username on bwUniCluster 2.0 is kn_ab1234.

The following list contains all prefixes currently in use:

{| class="wikitable"
! Home organization !! <UserID>
|-
| Universität Freiburg || ''fr_''username
|-
| Universität Heidelberg || ''hd_''username
|-
| Universität Hohenheim || ''ho_''username
|-
| KIT || username ''(without any prefix)''
|-
| Universität Konstanz || ''kn_''username
|-
| Universität Mannheim || ''ma_''username
|-
| Universität Stuttgart || ''st_''username
|-
| Universität Tübingen || ''tu_''username
|-
| Universität Ulm || ''ul_''username
|-
| Hochschule Aalen || ''aa_''username
|-
| Hochschule Albstadt-Sigmaringen || ''as_''username
|-
| Hochschule Esslingen || ''es_''username
|-
| Hochschule Furtwangen || ''hf_''username
|-
| Hochschule Heilbronn || ''hn_''username
|-
| Hochschule Karlsruhe || ''hk_''username
|-
| HTWG Konstanz || ''ht_''username
|-
| Hochschule Reutlingen || ''hr_''username
|-
| Hochschule Rottenburg || ''ro_''username
|-
| Hochschule für Technik Stuttgart || ''hs_''username
|-
| Hochschule Ulm || ''hu_''username
|-
|}
 

== Client application: OpenSSH ==

Most Unix and Unix-like operating systems like Linux, macOS and *BSD come with a built-in SSH client provided by the OpenSSH project. More recent versions of Windows 10 and the Windows Subsystem for Linux also come with a built-in OpenSSH client.

To use this client, simply open a command line terminal (the exact process differs on every operating system, but usually involves starting an application called '''Terminal''' or '''Command Prompt''') and enter the following command to connect to bwUniCluster 2.0:

<pre>
$ ssh <UserID>@bwunicluster.scc.kit.edu
</pre>

If you are on a Linux or Unix system running the X Window System (X11) and want to use a GUI-based application on bwUniCluster 2.0, you can use the ''-X'' option for the ssh command to set up X11 forwarding:

<pre>
$ ssh -X <UserID>@uc2.scc.kit.edu
</pre>

Windows users requiring X11 forwarding for graphical applications should use '''MobaXterm''' instead.
 

== Client application: MobaXterm ==

The bwHPC-C5 support team strongly recommends to use [http://mobaxterm.mobatek.net/ MobaXterm] instead of ''PuTTY'' or ''WinSCP'' on Windows. ''MobaXterm'' provides a built-in X11 server allowing to start GUI based software.

Start ''MobaXterm'', fill in the following fields:
<pre>
Remote name : uc2.scc.kit.edu # or uc1e.scc.kit.edu
Specify user name : <UserID>
Port : 22
</pre>

After that click on 'ok'. Then a terminal will be opened and there you can enter your credentials.
 

== Example login process ==

After the connection has been initiated, a successful login process will go through the following three steps:

1. The system asks for a '''One-Time Password'''. Generate one using the Software or Hardware Token registered on the bwIDM system (see [[bwUniCluster 2.0 User Access/2FA Tokens]]) and enter it after the '''Your OTP:''' prompt.

2. The systems asks for your service password. Enter it after the '''Password:''' prompt.

3. You are greeted by the bwUniCluster 2.0 banner followed by a shell.

The result should look like this:

[[File:BwUniCluster 2.0 access login example.png|center|]]
 

== Troubleshooting ==

'''Issue: The "Your OTP:" prompt never appears and the connection hangs/times out instead'''

Likely cause: You are most likely not on a network from which access to the bwUniCluster 2.0 system is allowed. Please check if you might have to establish a VPN connection first.

 

'''Issue: The system asks for the One-Time Password multiple times'''

Likely cause: Make sure you are using the correct Software Token to generate the One-Time Password.

 

'''Issue: The system asks for the service password multiple times'''

Likely cause: Make sure you are using the service password set on bwIDM and not the password valid for your home institution. Unlike the bwUniCluster 1, the bwUniCluster 2.0 only accepts the service password.

 

'''Issue: There is an error message by the pam_ses_open.sh skript'''

Likely cause: Your account is in the "LOST_ACCESS" state because the entitlement is no longer valid, the questionaire was not filled out or there was a problem during the communication between your home institution and the central bwIDM system. Please try the following steps:

* Log into [https://bwidm.scc.kit.edu bwIDM], look for the bwUniCluster entry and click on '''Registry info'''. Your "Status:" should be "ACTIVE". If it is not, please wait for ten minutes since logging into the bwIDM causes a refresh and the problem might fix itself. If the status does not change to ACTIVE after a longer amount of time, please contact the support channels.

* If you have not filled out the questionaire, please do so on [https://zas.bwhpc.de/shib/en/bwunicluster_survey.php https://zas.bwhpc.de/shib/en/bwunicluster_survey.php] and then wait for about ten minutes before attempting to log into the HPC system again.
 
 

== Allowed activities on login nodes ==

The login nodes of bwUniCluster 2.0 are the access point to the compute system and to your bwUniCluster 2.0 $HOME directory. The login nodes are shared with all the users of bwUniCluster 2.0. Therefore, your activities on the login nodes are limited to primarily set up your batch jobs. Your activities may also be:

* '''short''' compilation of your program code and
* '''short''' pre- and postprocessing of your batch jobs.

To guarantee usability for all the users of bwUniCluster 2.0 '''you must not run your compute jobs on the login nodes'''. Compute jobs must be submitted to the
[[bwUniCluster Batch Jobs|queueing system]]. Any compute job running on the login nodes will be terminated without any notice. Any long-running compilation or any long-running pre- or postprocessing of batch jobs must also be submitted to the [[bwUniCluster Batch Jobs|queueing system]].
 
 

== SSH Keys ==

In contrast to the bwUniCluster 1 and many other HPC systems it is '''no longer possible to self-manage your SSH Keys by adding them to the ~/.ssh/authorized_keys file'''. Existing files will no longer be evaluated. SSH Keys have to be managed via the central bwIDM system instead. Please refer to the user guide for this functionality:

[[bwUniCluster 2.0 User Access/SSH Keys]]
 
 

= [[First_Steps_on_bwHPC_cluster|First steps on bwUniCluster]] =

First and some important steps on bwUniCluster 2.0 can be found [[First_Steps_on_bwHPC_cluster|here]].
 
 

= Deregistration =
If you plan to permanently leave the bwUniCluster 2.0, follow the deregister checklist:
# Transfer all your data in $HOME and workspace to your local computer/storage and after that clear off all your data
# Visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
#* Select your home organization from the list and click '''Proceed'''
#* Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
#* You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
#* <div>Select '''Registry Info''' of the service '''bwUniCluster''' (on the left hand side) [[File:bwUniCluster_registration_sidebar.png|center|border|]]</div>
#* Click '''Deregister'''
Note that Step 2 will automatically unsubscribe you from the bwunicluster mail list.

File:Bwidm-2-red.png

2020-12-17T15:03:27Z

H Haefner:

BwUniCluster 2.0 User Access

2020-12-17T15:03:10Z

H Haefner: /* Step B: Web Registration, service password and 2-factor authentication */

[[bwUniCluster_2.0|bwUniCluster 2.0]] is Baden-Württemberg's general purpose tier 3 high performance computing (HPC)
cluster co-financed by Baden-Württemberg's ministry of science, research and arts and the shareholders:

* Albert Ludwig University of Freiburg
* Eberhard Karls University, Tübingen
* Karlsruhe Institute of Technology (KIT)
* Heidelberg University (Ruprecht-Karls-Universität Heidelberg)
* Ulm University
* University of Hohenheim
* University of Konstanz
* University of Mannheim
* University of Stuttgart
* HAW BW e.V. (an association of several universities of applied sciences in Baden-Württemberg, see below)
 
To '''log on''' [[bwUniCluster_2.0|bwUniCluster 2.0]] a user account is required. All members of the shareholder
universities can apply for an account.
 

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:left; color:#000;vertical-align:top;" |__TOC__
| [[File:bwUniCluster_17Jan2014_p044-rot_t10.10.00.jpg|center|border|250px|bwUniCluster wiring by Holger Obermaier, copyright: KIT (SCC)]] bwUniCluster wiring © KIT (SCC)
|}

= Registration =

Granting access and issuing a user account for '''bwUniCluster 2.0''' requires the registration at the KIT service website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] (step B). However, this registration depends on the '''bwUniCluster entitlement''' issued by your university (step A).
 

== Step A: bwUniCluster entitlement for registration ==
'''The entitlement is called bwUniCluster (not bwUniCluster 2.0)''' and each university issues the bwUniCluster entitlement '''only''' for their own respective members. Some have established on-line processes or provide downloads of the entitlement application forms. If there is no link behind the name of an institution in the following list, please contact the local IT support services:

* [[BwCluster_User_Access_Uni_Freiburg|Albert Ludwig University of Freiburg]]
* [https://bwunicluster.urz.uni-heidelberg.de/ Heidelberg University]
* [https://kim.uni-hohenheim.de/bwhpc-account University of Hohenheim]
* [http://www.scc.kit.edu/downloads/sdo/Antrag_Benutzernummer_bwUniCluster.pdf Karlsruhe Institute of Technology (KIT)]
* [[BWUniCluster_User_Access_Members_Uni_Konstanz|University of Konstanz]]
* [[BWUniCluster_User_Access_Members_Uni_Mannheim|University of Mannheim]]
* [https://www.hlrs.de/solutions-services/academic-users/bwunicluster-access/ University of Stuttgart]
* [https://uni-tuebingen.de/de/155157 Eberhard Karls University Tübingen]
* [[BWUniCluster_User_Access_Members_Uni_Ulm|Ulm University]]
* Hochschule Aalen
* Hochschule Albstadt-Sigmaringen
* Hochschule Esslingen
* Hochschule Furtwangen
* Hochschule Heilbronn
* Hochschule Karlsruhe
* Hochschule Konstanz
* Hochschule Reutlingen
* Hochschule Rottenburg
* Hochschule Stuttgart (HfT)
* Hochschule Ulm
 

== Step B: Web Registration, service password and 2-factor authentication ==

After completing step A, i.e., after successfull issueing of the bwUniCluster entitlement, you have to register yourself for the service. To do so please visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and complete the following steps.

1. Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.

[[File:Bwidm-1-red.png|center|border|]]
 

2. You will be directed to the ''Identity Provider'' of your home organisation. Enter the user ID / username and password of your home organisation - this is usually the same password used for your e-mail account and other services - and click on '''Login''', '''Einloggen''' or something similar.
 

3. You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]. If you are logging into bwIDM for the first time, there will be a summary screen which shows the account details your home institution is providing to the central system. Please check that all data is valid and then click on '''Continue''' or '''Weiter'''.
 

4. Once you have successfully logged into the bwIDM system, you will be greeted by a home screen showing all state-wide services you have access to. There will be a box labelled "bwUniCluster". Click on '''Register''' or '''Registrieren''' to start the registration process.

[[File:Bwidm-2-red.png|center|border|]]
 

5. Since August 13, 2020 a '''2-factor authentication''' mechanism (2FA) is being enforced to improve security. If you have never registered a 2FA token on bwIDM before, the following error message will appear:

[[File:BwUniCluster 2.0 access login bwidm registration requirements 2fa.png|center|]]

Click on the [https://bwidm.scc.kit.edu/user/twofa.xhtml Link] or on the '''My Tokens''' link in the main menu. The instructions for registering a new 2FA token can be found on the following page: [[bwUniCluster 2.0 User Access/2FA Tokens]]. Please complete them before proceeding.
 

6. Make sure all requirements are met by checking the '''Requirements''' box at the top. If the requirements are not met you might be able to correct the issure by following the instructions. In all other cases please [[Registration_Support_-_bwUniCluster|contact your local hotline]].

[[File:BwUniCluster 2.0 access login bwidm registration requirements.png|center|border|]]
 

7. Read the Terms of Use ('''Nutzungsbedingungen und -richtlinien'''), check the box besides '''I have read and accepted the terms of use''' and click on '''Register''' or '''Registrieren'''.
 

8. Set a service password for the bwUniCluster and click on '''Save''' or '''Speichern'''. Logging in with the password of your home organisation, like on the former bwUniCluster 1, is no longer possible. Please make sure to use a strong password which is different from any other password you are currently using or have used on other systems. You will also be asked to change the service password regularly.

[[File:BwUniCluster 2.0 access login bwidm service password.png|center|]]
 
 

== Step C: Fill out the bwUniCluster questionnaire ==

Filling out the bwUniCluster questionaire on

https://zas.bwhpc.de/shib/en/bwunicluster_survey.php

is mandatory for all users. The input is solely used to improve our support activities and for capacity planning of future HPC resources. '''If the questionaire is not filled out, access to bwUniCluster 2.0 is blocked 14 days after the registration.'''
 
 

== Changing the Service Password ==

Your bwUniCluster 2.0 '''password''' is the service password you set during the web registration (compare step 7 of chapter 1.2). At any time, you can set a new bwUniCluster 2.0 password via the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] by carrying out the following steps:
# Go to [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and select your home organization
# Authenticate yourself via the user id / username and password provided by your home institution
# Find the entry '''bwUniCluster''' and select '''Set Service Password'''
# Enter the new password, repeat it and click '''Save''' button.
# If the change was sucessfull, the message "Das Passwort wurde bei dem Dienst geändert" ("Password has been changed") will be shown
# Proceed to log in using the new password
 
 
== Contact / Support ==
If you have questions or problems concerning the bwUniCluster (2.0) registration, please [[bwUniCluster 2.0 Support|contact your local hotline]].
 
 

= Establishing network access =

Access to bwUniCluster 2.0 is '''limited to IP addresses from the so-called BelWü networks'''. All home institutions of our current users are connected to BelWue, so if you are on your campus network (e.g. in your office or on the Campus WiFi) you should be able to connect to bwUniCluster 2.0 without restrictions. If you are outside of one of the BelWue networks (e.g. in your home office instead of in your campus office), a VPN connection to your home institution has to be established first (see e.g. [1] for the KIT).
 
 

= Login =

After finishing the web registration and making sure that you are on a network from which you have access to bwUniCluster 2.0 (e.g. by establishing a VPN connection), the HPC cluster is ready for your '''SSH''' based login. Recommended SSH clients applications are:

* the ssh (OpenSSH) command included in all Linux distributions and macOS, -in command under Linux and macOS using the application ''terminal''
* [http://mobaxterm.mobatek.net/ MobaXterm] under Windows
 

== Hostnames ==

The main hostname required to connect to bwUniCluster 2.0 is '''bwunicluster.scc.kit.edu''' or '''uc2.scc.kit.edu'''. The system has four login nodes and we use so-called ''DNS round-robin scheduling'' to load-balance the incoming connections between the nodes. If you open multiple SSH sessions to bwUniCluster 2.0, these sessions will be established to different login nodes, so processes started in one session might not be visible in other sessions.

The older Broadwell extension partition of the former bwUniCluster 1 is connected to bwUniCluster 2.0. You can use the hostname '''uc1e.scc.kit.edu''' to connect to the login nodes of this partition.

If you need to connect to specific login nodes, you can use the following hostnames:

{| class="wikitable"
! Hostname !! Node type
|-
| '''uc2-login1.scc.kit.edu''' || bwUniCluster 2.0, first login node
|-
| '''uc2-login2.scc.kit.edu''' || bwUniCluster 2.0, second login node
|-
| '''uc2-login3.scc.kit.edu''' || bwUniCluster 2.0, third login node
|-
| '''uc2-login4.scc.kit.edu''' || bwUniCluster 2.0, fourth login node
|-
| '''uc1e-login1.scc.kit.edu''' || Broadwell partition, first login node
|-
| '''uc1e-login2.scc.kit.edu''' || Broadwell partition, second login node
|-
|}

Only the secure shell ''SSH'' is allowed to login. Other protocols like ''telnet'' or ''rlogin'' are not allowed for security reasons.
 

== Usernames ==

Your username will be the same as the one provided by your home institution, but '''prefixed''' with two characters and an underscore indicating your home institution. For example: If you are a member of the university of Konstanz and your local username is ab1234, your username on bwUniCluster 2.0 is kn_ab1234.

The following list contains all prefixes currently in use:

{| class="wikitable"
! Home organization !! <UserID>
|-
| Universität Freiburg || ''fr_''username
|-
| Universität Heidelberg || ''hd_''username
|-
| Universität Hohenheim || ''ho_''username
|-
| KIT || username ''(without any prefix)''
|-
| Universität Konstanz || ''kn_''username
|-
| Universität Mannheim || ''ma_''username
|-
| Universität Stuttgart || ''st_''username
|-
| Universität Tübingen || ''tu_''username
|-
| Universität Ulm || ''ul_''username
|-
| Hochschule Aalen || ''aa_''username
|-
| Hochschule Albstadt-Sigmaringen || ''as_''username
|-
| Hochschule Esslingen || ''es_''username
|-
| Hochschule Furtwangen || ''hf_''username
|-
| Hochschule Heilbronn || ''hn_''username
|-
| Hochschule Karlsruhe || ''hk_''username
|-
| HTWG Konstanz || ''ht_''username
|-
| Hochschule Reutlingen || ''hr_''username
|-
| Hochschule Rottenburg || ''ro_''username
|-
| Hochschule für Technik Stuttgart || ''hs_''username
|-
| Hochschule Ulm || ''hu_''username
|-
|}
 

== Client application: OpenSSH ==

Most Unix and Unix-like operating systems like Linux, macOS and *BSD come with a built-in SSH client provided by the OpenSSH project. More recent versions of Windows 10 and the Windows Subsystem for Linux also come with a built-in OpenSSH client.

To use this client, simply open a command line terminal (the exact process differs on every operating system, but usually involves starting an application called '''Terminal''' or '''Command Prompt''') and enter the following command to connect to bwUniCluster 2.0:

<pre>
$ ssh <UserID>@bwunicluster.scc.kit.edu
</pre>

If you are on a Linux or Unix system running the X Window System (X11) and want to use a GUI-based application on bwUniCluster 2.0, you can use the ''-X'' option for the ssh command to set up X11 forwarding:

<pre>
$ ssh -X <UserID>@uc2.scc.kit.edu
</pre>

Windows users requiring X11 forwarding for graphical applications should use '''MobaXterm''' instead.
 

== Client application: MobaXterm ==

The bwHPC-C5 support team strongly recommends to use [http://mobaxterm.mobatek.net/ MobaXterm] instead of ''PuTTY'' or ''WinSCP'' on Windows. ''MobaXterm'' provides a built-in X11 server allowing to start GUI based software.

Start ''MobaXterm'', fill in the following fields:
<pre>
Remote name : uc2.scc.kit.edu # or uc1e.scc.kit.edu
Specify user name : <UserID>
Port : 22
</pre>

After that click on 'ok'. Then a terminal will be opened and there you can enter your credentials.
 

== Example login process ==

After the connection has been initiated, a successful login process will go through the following three steps:

1. The system asks for a '''One-Time Password'''. Generate one using the Software or Hardware Token registered on the bwIDM system (see [[bwUniCluster 2.0 User Access/2FA Tokens]]) and enter it after the '''Your OTP:''' prompt.

2. The systems asks for your service password. Enter it after the '''Password:''' prompt.

3. You are greeted by the bwUniCluster 2.0 banner followed by a shell.

The result should look like this:

[[File:BwUniCluster 2.0 access login example.png|center|]]
 

== Troubleshooting ==

'''Issue: The "Your OTP:" prompt never appears and the connection hangs/times out instead'''

Likely cause: You are most likely not on a network from which access to the bwUniCluster 2.0 system is allowed. Please check if you might have to establish a VPN connection first.

 

'''Issue: The system asks for the One-Time Password multiple times'''

Likely cause: Make sure you are using the correct Software Token to generate the One-Time Password.

 

'''Issue: The system asks for the service password multiple times'''

Likely cause: Make sure you are using the service password set on bwIDM and not the password valid for your home institution. Unlike the bwUniCluster 1, the bwUniCluster 2.0 only accepts the service password.

 

'''Issue: There is an error message by the pam_ses_open.sh skript'''

Likely cause: Your account is in the "LOST_ACCESS" state because the entitlement is no longer valid, the questionaire was not filled out or there was a problem during the communication between your home institution and the central bwIDM system. Please try the following steps:

* Log into [https://bwidm.scc.kit.edu bwIDM], look for the bwUniCluster entry and click on '''Registry info'''. Your "Status:" should be "ACTIVE". If it is not, please wait for ten minutes since logging into the bwIDM causes a refresh and the problem might fix itself. If the status does not change to ACTIVE after a longer amount of time, please contact the support channels.

* If you have not filled out the questionaire, please do so on [https://zas.bwhpc.de/shib/en/bwunicluster_survey.php https://zas.bwhpc.de/shib/en/bwunicluster_survey.php] and then wait for about ten minutes before attempting to log into the HPC system again.
 
 

== Allowed activities on login nodes ==

The login nodes of bwUniCluster 2.0 are the access point to the compute system and to your bwUniCluster 2.0 $HOME directory. The login nodes are shared with all the users of bwUniCluster 2.0. Therefore, your activities on the login nodes are limited to primarily set up your batch jobs. Your activities may also be:

* '''short''' compilation of your program code and
* '''short''' pre- and postprocessing of your batch jobs.

To guarantee usability for all the users of bwUniCluster 2.0 '''you must not run your compute jobs on the login nodes'''. Compute jobs must be submitted to the
[[bwUniCluster Batch Jobs|queueing system]]. Any compute job running on the login nodes will be terminated without any notice. Any long-running compilation or any long-running pre- or postprocessing of batch jobs must also be submitted to the [[bwUniCluster Batch Jobs|queueing system]].
 
 

== SSH Keys ==

In contrast to the bwUniCluster 1 and many other HPC systems it is '''no longer possible to self-manage your SSH Keys by adding them to the ~/.ssh/authorized_keys file'''. Existing files will no longer be evaluated. SSH Keys have to be managed via the central bwIDM system instead. Please refer to the user guide for this functionality:

[[bwUniCluster 2.0 User Access/SSH Keys]]
 
 

= [[First_Steps_on_bwHPC_cluster|First steps on bwUniCluster]] =

First and some important steps on bwUniCluster 2.0 can be found [[First_Steps_on_bwHPC_cluster|here]].
 
 

= Deregistration =
If you plan to permanently leave the bwUniCluster 2.0, follow the deregister checklist:
# Transfer all your data in $HOME and workspace to your local computer/storage and after that clear off all your data
# Visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
#* Select your home organization from the list and click '''Proceed'''
#* Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
#* You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
#* <div>Select '''Registry Info''' of the service '''bwUniCluster''' (on the left hand side) [[File:bwUniCluster_registration_sidebar.png|center|border|]]</div>
#* Click '''Deregister'''
Note that Step 2 will automatically unsubscribe you from the bwunicluster mail list.

BwUniCluster 2.0 User Access

2020-12-17T15:01:24Z

H Haefner: /* Step B: Web Registration, service password and 2-factor authentication */

[[bwUniCluster_2.0|bwUniCluster 2.0]] is Baden-Württemberg's general purpose tier 3 high performance computing (HPC)
cluster co-financed by Baden-Württemberg's ministry of science, research and arts and the shareholders:

* Albert Ludwig University of Freiburg
* Eberhard Karls University, Tübingen
* Karlsruhe Institute of Technology (KIT)
* Heidelberg University (Ruprecht-Karls-Universität Heidelberg)
* Ulm University
* University of Hohenheim
* University of Konstanz
* University of Mannheim
* University of Stuttgart
* HAW BW e.V. (an association of several universities of applied sciences in Baden-Württemberg, see below)
 
To '''log on''' [[bwUniCluster_2.0|bwUniCluster 2.0]] a user account is required. All members of the shareholder
universities can apply for an account.
 

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:left; color:#000;vertical-align:top;" |__TOC__
| [[File:bwUniCluster_17Jan2014_p044-rot_t10.10.00.jpg|center|border|250px|bwUniCluster wiring by Holger Obermaier, copyright: KIT (SCC)]] bwUniCluster wiring © KIT (SCC)
|}

= Registration =

Granting access and issuing a user account for '''bwUniCluster 2.0''' requires the registration at the KIT service website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] (step B). However, this registration depends on the '''bwUniCluster entitlement''' issued by your university (step A).
 

== Step A: bwUniCluster entitlement for registration ==
'''The entitlement is called bwUniCluster (not bwUniCluster 2.0)''' and each university issues the bwUniCluster entitlement '''only''' for their own respective members. Some have established on-line processes or provide downloads of the entitlement application forms. If there is no link behind the name of an institution in the following list, please contact the local IT support services:

* [[BwCluster_User_Access_Uni_Freiburg|Albert Ludwig University of Freiburg]]
* [https://bwunicluster.urz.uni-heidelberg.de/ Heidelberg University]
* [https://kim.uni-hohenheim.de/bwhpc-account University of Hohenheim]
* [http://www.scc.kit.edu/downloads/sdo/Antrag_Benutzernummer_bwUniCluster.pdf Karlsruhe Institute of Technology (KIT)]
* [[BWUniCluster_User_Access_Members_Uni_Konstanz|University of Konstanz]]
* [[BWUniCluster_User_Access_Members_Uni_Mannheim|University of Mannheim]]
* [https://www.hlrs.de/solutions-services/academic-users/bwunicluster-access/ University of Stuttgart]
* [https://uni-tuebingen.de/de/155157 Eberhard Karls University Tübingen]
* [[BWUniCluster_User_Access_Members_Uni_Ulm|Ulm University]]
* Hochschule Aalen
* Hochschule Albstadt-Sigmaringen
* Hochschule Esslingen
* Hochschule Furtwangen
* Hochschule Heilbronn
* Hochschule Karlsruhe
* Hochschule Konstanz
* Hochschule Reutlingen
* Hochschule Rottenburg
* Hochschule Stuttgart (HfT)
* Hochschule Ulm
 

== Step B: Web Registration, service password and 2-factor authentication ==

After completing step A, i.e., after successfull issueing of the bwUniCluster entitlement, you have to register yourself for the service. To do so please visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and complete the following steps.

1. Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.

[[File:Bwidm-1-red.png|center|border|]]
 

2. You will be directed to the ''Identity Provider'' of your home organisation. Enter the user ID / username and password of your home organisation - this is usually the same password used for your e-mail account and other services - and click on '''Login''', '''Einloggen''' or something similar.
 

3. You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]. If you are logging into bwIDM for the first time, there will be a summary screen which shows the account details your home institution is providing to the central system. Please check that all data is valid and then click on '''Continue''' or '''Weiter'''.
 

4. Once you have successfully logged into the bwIDM system, you will be greeted by a home screen showing all state-wide services you have access to. There will be a box labelled "bwUniCluster". Click on '''Register''' or '''Registrieren''' to start the registration process.

[[File:BwUniCluster 2.0 access login bwidm service list.png|center|border|]]
 

5. Since August 13, 2020 a '''2-factor authentication''' mechanism (2FA) is being enforced to improve security. If you have never registered a 2FA token on bwIDM before, the following error message will appear:

[[File:BwUniCluster 2.0 access login bwidm registration requirements 2fa.png|center|]]

Click on the [https://bwidm.scc.kit.edu/user/twofa.xhtml Link] or on the '''My Tokens''' link in the main menu. The instructions for registering a new 2FA token can be found on the following page: [[bwUniCluster 2.0 User Access/2FA Tokens]]. Please complete them before proceeding.
 

6. Make sure all requirements are met by checking the '''Requirements''' box at the top. If the requirements are not met you might be able to correct the issure by following the instructions. In all other cases please [[Registration_Support_-_bwUniCluster|contact your local hotline]].

[[File:BwUniCluster 2.0 access login bwidm registration requirements.png|center|border|]]
 

7. Read the Terms of Use ('''Nutzungsbedingungen und -richtlinien'''), check the box besides '''I have read and accepted the terms of use''' and click on '''Register''' or '''Registrieren'''.
 

8. Set a service password for the bwUniCluster and click on '''Save''' or '''Speichern'''. Logging in with the password of your home organisation, like on the former bwUniCluster 1, is no longer possible. Please make sure to use a strong password which is different from any other password you are currently using or have used on other systems. You will also be asked to change the service password regularly.

[[File:BwUniCluster 2.0 access login bwidm service password.png|center|]]
 
 

== Step C: Fill out the bwUniCluster questionnaire ==

Filling out the bwUniCluster questionaire on

https://zas.bwhpc.de/shib/en/bwunicluster_survey.php

is mandatory for all users. The input is solely used to improve our support activities and for capacity planning of future HPC resources. '''If the questionaire is not filled out, access to bwUniCluster 2.0 is blocked 14 days after the registration.'''
 
 

== Changing the Service Password ==

Your bwUniCluster 2.0 '''password''' is the service password you set during the web registration (compare step 7 of chapter 1.2). At any time, you can set a new bwUniCluster 2.0 password via the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] by carrying out the following steps:
# Go to [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and select your home organization
# Authenticate yourself via the user id / username and password provided by your home institution
# Find the entry '''bwUniCluster''' and select '''Set Service Password'''
# Enter the new password, repeat it and click '''Save''' button.
# If the change was sucessfull, the message "Das Passwort wurde bei dem Dienst geändert" ("Password has been changed") will be shown
# Proceed to log in using the new password
 
 
== Contact / Support ==
If you have questions or problems concerning the bwUniCluster (2.0) registration, please [[bwUniCluster 2.0 Support|contact your local hotline]].
 
 

= Establishing network access =

Access to bwUniCluster 2.0 is '''limited to IP addresses from the so-called BelWü networks'''. All home institutions of our current users are connected to BelWue, so if you are on your campus network (e.g. in your office or on the Campus WiFi) you should be able to connect to bwUniCluster 2.0 without restrictions. If you are outside of one of the BelWue networks (e.g. in your home office instead of in your campus office), a VPN connection to your home institution has to be established first (see e.g. [1] for the KIT).
 
 

= Login =

After finishing the web registration and making sure that you are on a network from which you have access to bwUniCluster 2.0 (e.g. by establishing a VPN connection), the HPC cluster is ready for your '''SSH''' based login. Recommended SSH clients applications are:

* the ssh (OpenSSH) command included in all Linux distributions and macOS, -in command under Linux and macOS using the application ''terminal''
* [http://mobaxterm.mobatek.net/ MobaXterm] under Windows
 

== Hostnames ==

The main hostname required to connect to bwUniCluster 2.0 is '''bwunicluster.scc.kit.edu''' or '''uc2.scc.kit.edu'''. The system has four login nodes and we use so-called ''DNS round-robin scheduling'' to load-balance the incoming connections between the nodes. If you open multiple SSH sessions to bwUniCluster 2.0, these sessions will be established to different login nodes, so processes started in one session might not be visible in other sessions.

The older Broadwell extension partition of the former bwUniCluster 1 is connected to bwUniCluster 2.0. You can use the hostname '''uc1e.scc.kit.edu''' to connect to the login nodes of this partition.

If you need to connect to specific login nodes, you can use the following hostnames:

{| class="wikitable"
! Hostname !! Node type
|-
| '''uc2-login1.scc.kit.edu''' || bwUniCluster 2.0, first login node
|-
| '''uc2-login2.scc.kit.edu''' || bwUniCluster 2.0, second login node
|-
| '''uc2-login3.scc.kit.edu''' || bwUniCluster 2.0, third login node
|-
| '''uc2-login4.scc.kit.edu''' || bwUniCluster 2.0, fourth login node
|-
| '''uc1e-login1.scc.kit.edu''' || Broadwell partition, first login node
|-
| '''uc1e-login2.scc.kit.edu''' || Broadwell partition, second login node
|-
|}

Only the secure shell ''SSH'' is allowed to login. Other protocols like ''telnet'' or ''rlogin'' are not allowed for security reasons.
 

== Usernames ==

Your username will be the same as the one provided by your home institution, but '''prefixed''' with two characters and an underscore indicating your home institution. For example: If you are a member of the university of Konstanz and your local username is ab1234, your username on bwUniCluster 2.0 is kn_ab1234.

The following list contains all prefixes currently in use:

{| class="wikitable"
! Home organization !! <UserID>
|-
| Universität Freiburg || ''fr_''username
|-
| Universität Heidelberg || ''hd_''username
|-
| Universität Hohenheim || ''ho_''username
|-
| KIT || username ''(without any prefix)''
|-
| Universität Konstanz || ''kn_''username
|-
| Universität Mannheim || ''ma_''username
|-
| Universität Stuttgart || ''st_''username
|-
| Universität Tübingen || ''tu_''username
|-
| Universität Ulm || ''ul_''username
|-
| Hochschule Aalen || ''aa_''username
|-
| Hochschule Albstadt-Sigmaringen || ''as_''username
|-
| Hochschule Esslingen || ''es_''username
|-
| Hochschule Furtwangen || ''hf_''username
|-
| Hochschule Heilbronn || ''hn_''username
|-
| Hochschule Karlsruhe || ''hk_''username
|-
| HTWG Konstanz || ''ht_''username
|-
| Hochschule Reutlingen || ''hr_''username
|-
| Hochschule Rottenburg || ''ro_''username
|-
| Hochschule für Technik Stuttgart || ''hs_''username
|-
| Hochschule Ulm || ''hu_''username
|-
|}
 

== Client application: OpenSSH ==

Most Unix and Unix-like operating systems like Linux, macOS and *BSD come with a built-in SSH client provided by the OpenSSH project. More recent versions of Windows 10 and the Windows Subsystem for Linux also come with a built-in OpenSSH client.

To use this client, simply open a command line terminal (the exact process differs on every operating system, but usually involves starting an application called '''Terminal''' or '''Command Prompt''') and enter the following command to connect to bwUniCluster 2.0:

<pre>
$ ssh <UserID>@bwunicluster.scc.kit.edu
</pre>

If you are on a Linux or Unix system running the X Window System (X11) and want to use a GUI-based application on bwUniCluster 2.0, you can use the ''-X'' option for the ssh command to set up X11 forwarding:

<pre>
$ ssh -X <UserID>@uc2.scc.kit.edu
</pre>

Windows users requiring X11 forwarding for graphical applications should use '''MobaXterm''' instead.
 

== Client application: MobaXterm ==

The bwHPC-C5 support team strongly recommends to use [http://mobaxterm.mobatek.net/ MobaXterm] instead of ''PuTTY'' or ''WinSCP'' on Windows. ''MobaXterm'' provides a built-in X11 server allowing to start GUI based software.

Start ''MobaXterm'', fill in the following fields:
<pre>
Remote name : uc2.scc.kit.edu # or uc1e.scc.kit.edu
Specify user name : <UserID>
Port : 22
</pre>

After that click on 'ok'. Then a terminal will be opened and there you can enter your credentials.
 

== Example login process ==

After the connection has been initiated, a successful login process will go through the following three steps:

1. The system asks for a '''One-Time Password'''. Generate one using the Software or Hardware Token registered on the bwIDM system (see [[bwUniCluster 2.0 User Access/2FA Tokens]]) and enter it after the '''Your OTP:''' prompt.

2. The systems asks for your service password. Enter it after the '''Password:''' prompt.

3. You are greeted by the bwUniCluster 2.0 banner followed by a shell.

The result should look like this:

[[File:BwUniCluster 2.0 access login example.png|center|]]
 

== Troubleshooting ==

'''Issue: The "Your OTP:" prompt never appears and the connection hangs/times out instead'''

Likely cause: You are most likely not on a network from which access to the bwUniCluster 2.0 system is allowed. Please check if you might have to establish a VPN connection first.

 

'''Issue: The system asks for the One-Time Password multiple times'''

Likely cause: Make sure you are using the correct Software Token to generate the One-Time Password.

 

'''Issue: The system asks for the service password multiple times'''

Likely cause: Make sure you are using the service password set on bwIDM and not the password valid for your home institution. Unlike the bwUniCluster 1, the bwUniCluster 2.0 only accepts the service password.

 

'''Issue: There is an error message by the pam_ses_open.sh skript'''

Likely cause: Your account is in the "LOST_ACCESS" state because the entitlement is no longer valid, the questionaire was not filled out or there was a problem during the communication between your home institution and the central bwIDM system. Please try the following steps:

* Log into [https://bwidm.scc.kit.edu bwIDM], look for the bwUniCluster entry and click on '''Registry info'''. Your "Status:" should be "ACTIVE". If it is not, please wait for ten minutes since logging into the bwIDM causes a refresh and the problem might fix itself. If the status does not change to ACTIVE after a longer amount of time, please contact the support channels.

* If you have not filled out the questionaire, please do so on [https://zas.bwhpc.de/shib/en/bwunicluster_survey.php https://zas.bwhpc.de/shib/en/bwunicluster_survey.php] and then wait for about ten minutes before attempting to log into the HPC system again.
 
 

== Allowed activities on login nodes ==

The login nodes of bwUniCluster 2.0 are the access point to the compute system and to your bwUniCluster 2.0 $HOME directory. The login nodes are shared with all the users of bwUniCluster 2.0. Therefore, your activities on the login nodes are limited to primarily set up your batch jobs. Your activities may also be:

* '''short''' compilation of your program code and
* '''short''' pre- and postprocessing of your batch jobs.

To guarantee usability for all the users of bwUniCluster 2.0 '''you must not run your compute jobs on the login nodes'''. Compute jobs must be submitted to the
[[bwUniCluster Batch Jobs|queueing system]]. Any compute job running on the login nodes will be terminated without any notice. Any long-running compilation or any long-running pre- or postprocessing of batch jobs must also be submitted to the [[bwUniCluster Batch Jobs|queueing system]].
 
 

== SSH Keys ==

In contrast to the bwUniCluster 1 and many other HPC systems it is '''no longer possible to self-manage your SSH Keys by adding them to the ~/.ssh/authorized_keys file'''. Existing files will no longer be evaluated. SSH Keys have to be managed via the central bwIDM system instead. Please refer to the user guide for this functionality:

[[bwUniCluster 2.0 User Access/SSH Keys]]
 
 

= [[First_Steps_on_bwHPC_cluster|First steps on bwUniCluster]] =

First and some important steps on bwUniCluster 2.0 can be found [[First_Steps_on_bwHPC_cluster|here]].
 
 

= Deregistration =
If you plan to permanently leave the bwUniCluster 2.0, follow the deregister checklist:
# Transfer all your data in $HOME and workspace to your local computer/storage and after that clear off all your data
# Visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
#* Select your home organization from the list and click '''Proceed'''
#* Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
#* You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
#* <div>Select '''Registry Info''' of the service '''bwUniCluster''' (on the left hand side) [[File:bwUniCluster_registration_sidebar.png|center|border|]]</div>
#* Click '''Deregister'''
Note that Step 2 will automatically unsubscribe you from the bwunicluster mail list.

BwUniCluster 2.0 User Access

2020-12-17T14:59:00Z

H Haefner: /* Step B: Web Registration, service password and 2-factor authentication */

[[bwUniCluster_2.0|bwUniCluster 2.0]] is Baden-Württemberg's general purpose tier 3 high performance computing (HPC)
cluster co-financed by Baden-Württemberg's ministry of science, research and arts and the shareholders:

* Albert Ludwig University of Freiburg
* Eberhard Karls University, Tübingen
* Karlsruhe Institute of Technology (KIT)
* Heidelberg University (Ruprecht-Karls-Universität Heidelberg)
* Ulm University
* University of Hohenheim
* University of Konstanz
* University of Mannheim
* University of Stuttgart
* HAW BW e.V. (an association of several universities of applied sciences in Baden-Württemberg, see below)
 
To '''log on''' [[bwUniCluster_2.0|bwUniCluster 2.0]] a user account is required. All members of the shareholder
universities can apply for an account.
 

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:left; color:#000;vertical-align:top;" |__TOC__
| [[File:bwUniCluster_17Jan2014_p044-rot_t10.10.00.jpg|center|border|250px|bwUniCluster wiring by Holger Obermaier, copyright: KIT (SCC)]] bwUniCluster wiring © KIT (SCC)
|}

= Registration =

Granting access and issuing a user account for '''bwUniCluster 2.0''' requires the registration at the KIT service website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] (step B). However, this registration depends on the '''bwUniCluster entitlement''' issued by your university (step A).
 

== Step A: bwUniCluster entitlement for registration ==
'''The entitlement is called bwUniCluster (not bwUniCluster 2.0)''' and each university issues the bwUniCluster entitlement '''only''' for their own respective members. Some have established on-line processes or provide downloads of the entitlement application forms. If there is no link behind the name of an institution in the following list, please contact the local IT support services:

* [[BwCluster_User_Access_Uni_Freiburg|Albert Ludwig University of Freiburg]]
* [https://bwunicluster.urz.uni-heidelberg.de/ Heidelberg University]
* [https://kim.uni-hohenheim.de/bwhpc-account University of Hohenheim]
* [http://www.scc.kit.edu/downloads/sdo/Antrag_Benutzernummer_bwUniCluster.pdf Karlsruhe Institute of Technology (KIT)]
* [[BWUniCluster_User_Access_Members_Uni_Konstanz|University of Konstanz]]
* [[BWUniCluster_User_Access_Members_Uni_Mannheim|University of Mannheim]]
* [https://www.hlrs.de/solutions-services/academic-users/bwunicluster-access/ University of Stuttgart]
* [https://uni-tuebingen.de/de/155157 Eberhard Karls University Tübingen]
* [[BWUniCluster_User_Access_Members_Uni_Ulm|Ulm University]]
* Hochschule Aalen
* Hochschule Albstadt-Sigmaringen
* Hochschule Esslingen
* Hochschule Furtwangen
* Hochschule Heilbronn
* Hochschule Karlsruhe
* Hochschule Konstanz
* Hochschule Reutlingen
* Hochschule Rottenburg
* Hochschule Stuttgart (HfT)
* Hochschule Ulm
 

== Step B: Web Registration, service password and 2-factor authentication ==

After completing step A, i.e., after successfull issueing of the bwUniCluster entitlement, you have to register yourself for the service. To do so please visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and complete the following steps.

1. Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.

[[File:BwIDM-1-red.png|center|border|]]
 

2. You will be directed to the ''Identity Provider'' of your home organisation. Enter the user ID / username and password of your home organisation - this is usually the same password used for your e-mail account and other services - and click on '''Login''', '''Einloggen''' or something similar.
 

3. You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]. If you are logging into bwIDM for the first time, there will be a summary screen which shows the account details your home institution is providing to the central system. Please check that all data is valid and then click on '''Continue''' or '''Weiter'''.
 

4. Once you have successfully logged into the bwIDM system, you will be greeted by a home screen showing all state-wide services you have access to. There will be a box labelled "bwUniCluster". Click on '''Register''' or '''Registrieren''' to start the registration process.

[[File:BwUniCluster 2.0 access login bwidm service list.png|center|border|]]
 

5. Since August 13, 2020 a '''2-factor authentication''' mechanism (2FA) is being enforced to improve security. If you have never registered a 2FA token on bwIDM before, the following error message will appear:

[[File:BwUniCluster 2.0 access login bwidm registration requirements 2fa.png|center|]]

Click on the [https://bwidm.scc.kit.edu/user/twofa.xhtml Link] or on the '''My Tokens''' link in the main menu. The instructions for registering a new 2FA token can be found on the following page: [[bwUniCluster 2.0 User Access/2FA Tokens]]. Please complete them before proceeding.
 

6. Make sure all requirements are met by checking the '''Requirements''' box at the top. If the requirements are not met you might be able to correct the issure by following the instructions. In all other cases please [[Registration_Support_-_bwUniCluster|contact your local hotline]].

[[File:BwUniCluster 2.0 access login bwidm registration requirements.png|center|border|]]
 

7. Read the Terms of Use ('''Nutzungsbedingungen und -richtlinien'''), check the box besides '''I have read and accepted the terms of use''' and click on '''Register''' or '''Registrieren'''.
 

8. Set a service password for the bwUniCluster and click on '''Save''' or '''Speichern'''. Logging in with the password of your home organisation, like on the former bwUniCluster 1, is no longer possible. Please make sure to use a strong password which is different from any other password you are currently using or have used on other systems. You will also be asked to change the service password regularly.

[[File:BwUniCluster 2.0 access login bwidm service password.png|center|]]
 
 

== Step C: Fill out the bwUniCluster questionnaire ==

Filling out the bwUniCluster questionaire on

https://zas.bwhpc.de/shib/en/bwunicluster_survey.php

is mandatory for all users. The input is solely used to improve our support activities and for capacity planning of future HPC resources. '''If the questionaire is not filled out, access to bwUniCluster 2.0 is blocked 14 days after the registration.'''
 
 

== Changing the Service Password ==

Your bwUniCluster 2.0 '''password''' is the service password you set during the web registration (compare step 7 of chapter 1.2). At any time, you can set a new bwUniCluster 2.0 password via the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] by carrying out the following steps:
# Go to [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and select your home organization
# Authenticate yourself via the user id / username and password provided by your home institution
# Find the entry '''bwUniCluster''' and select '''Set Service Password'''
# Enter the new password, repeat it and click '''Save''' button.
# If the change was sucessfull, the message "Das Passwort wurde bei dem Dienst geändert" ("Password has been changed") will be shown
# Proceed to log in using the new password
 
 
== Contact / Support ==
If you have questions or problems concerning the bwUniCluster (2.0) registration, please [[bwUniCluster 2.0 Support|contact your local hotline]].
 
 

= Establishing network access =

Access to bwUniCluster 2.0 is '''limited to IP addresses from the so-called BelWü networks'''. All home institutions of our current users are connected to BelWue, so if you are on your campus network (e.g. in your office or on the Campus WiFi) you should be able to connect to bwUniCluster 2.0 without restrictions. If you are outside of one of the BelWue networks (e.g. in your home office instead of in your campus office), a VPN connection to your home institution has to be established first (see e.g. [1] for the KIT).
 
 

= Login =

After finishing the web registration and making sure that you are on a network from which you have access to bwUniCluster 2.0 (e.g. by establishing a VPN connection), the HPC cluster is ready for your '''SSH''' based login. Recommended SSH clients applications are:

* the ssh (OpenSSH) command included in all Linux distributions and macOS, -in command under Linux and macOS using the application ''terminal''
* [http://mobaxterm.mobatek.net/ MobaXterm] under Windows
 

== Hostnames ==

The main hostname required to connect to bwUniCluster 2.0 is '''bwunicluster.scc.kit.edu''' or '''uc2.scc.kit.edu'''. The system has four login nodes and we use so-called ''DNS round-robin scheduling'' to load-balance the incoming connections between the nodes. If you open multiple SSH sessions to bwUniCluster 2.0, these sessions will be established to different login nodes, so processes started in one session might not be visible in other sessions.

The older Broadwell extension partition of the former bwUniCluster 1 is connected to bwUniCluster 2.0. You can use the hostname '''uc1e.scc.kit.edu''' to connect to the login nodes of this partition.

If you need to connect to specific login nodes, you can use the following hostnames:

{| class="wikitable"
! Hostname !! Node type
|-
| '''uc2-login1.scc.kit.edu''' || bwUniCluster 2.0, first login node
|-
| '''uc2-login2.scc.kit.edu''' || bwUniCluster 2.0, second login node
|-
| '''uc2-login3.scc.kit.edu''' || bwUniCluster 2.0, third login node
|-
| '''uc2-login4.scc.kit.edu''' || bwUniCluster 2.0, fourth login node
|-
| '''uc1e-login1.scc.kit.edu''' || Broadwell partition, first login node
|-
| '''uc1e-login2.scc.kit.edu''' || Broadwell partition, second login node
|-
|}

Only the secure shell ''SSH'' is allowed to login. Other protocols like ''telnet'' or ''rlogin'' are not allowed for security reasons.
 

== Usernames ==

Your username will be the same as the one provided by your home institution, but '''prefixed''' with two characters and an underscore indicating your home institution. For example: If you are a member of the university of Konstanz and your local username is ab1234, your username on bwUniCluster 2.0 is kn_ab1234.

The following list contains all prefixes currently in use:

{| class="wikitable"
! Home organization !! <UserID>
|-
| Universität Freiburg || ''fr_''username
|-
| Universität Heidelberg || ''hd_''username
|-
| Universität Hohenheim || ''ho_''username
|-
| KIT || username ''(without any prefix)''
|-
| Universität Konstanz || ''kn_''username
|-
| Universität Mannheim || ''ma_''username
|-
| Universität Stuttgart || ''st_''username
|-
| Universität Tübingen || ''tu_''username
|-
| Universität Ulm || ''ul_''username
|-
| Hochschule Aalen || ''aa_''username
|-
| Hochschule Albstadt-Sigmaringen || ''as_''username
|-
| Hochschule Esslingen || ''es_''username
|-
| Hochschule Furtwangen || ''hf_''username
|-
| Hochschule Heilbronn || ''hn_''username
|-
| Hochschule Karlsruhe || ''hk_''username
|-
| HTWG Konstanz || ''ht_''username
|-
| Hochschule Reutlingen || ''hr_''username
|-
| Hochschule Rottenburg || ''ro_''username
|-
| Hochschule für Technik Stuttgart || ''hs_''username
|-
| Hochschule Ulm || ''hu_''username
|-
|}
 

== Client application: OpenSSH ==

Most Unix and Unix-like operating systems like Linux, macOS and *BSD come with a built-in SSH client provided by the OpenSSH project. More recent versions of Windows 10 and the Windows Subsystem for Linux also come with a built-in OpenSSH client.

To use this client, simply open a command line terminal (the exact process differs on every operating system, but usually involves starting an application called '''Terminal''' or '''Command Prompt''') and enter the following command to connect to bwUniCluster 2.0:

<pre>
$ ssh <UserID>@bwunicluster.scc.kit.edu
</pre>

If you are on a Linux or Unix system running the X Window System (X11) and want to use a GUI-based application on bwUniCluster 2.0, you can use the ''-X'' option for the ssh command to set up X11 forwarding:

<pre>
$ ssh -X <UserID>@uc2.scc.kit.edu
</pre>

Windows users requiring X11 forwarding for graphical applications should use '''MobaXterm''' instead.
 

== Client application: MobaXterm ==

The bwHPC-C5 support team strongly recommends to use [http://mobaxterm.mobatek.net/ MobaXterm] instead of ''PuTTY'' or ''WinSCP'' on Windows. ''MobaXterm'' provides a built-in X11 server allowing to start GUI based software.

Start ''MobaXterm'', fill in the following fields:
<pre>
Remote name : uc2.scc.kit.edu # or uc1e.scc.kit.edu
Specify user name : <UserID>
Port : 22
</pre>

After that click on 'ok'. Then a terminal will be opened and there you can enter your credentials.
 

== Example login process ==

After the connection has been initiated, a successful login process will go through the following three steps:

1. The system asks for a '''One-Time Password'''. Generate one using the Software or Hardware Token registered on the bwIDM system (see [[bwUniCluster 2.0 User Access/2FA Tokens]]) and enter it after the '''Your OTP:''' prompt.

2. The systems asks for your service password. Enter it after the '''Password:''' prompt.

3. You are greeted by the bwUniCluster 2.0 banner followed by a shell.

The result should look like this:

[[File:BwUniCluster 2.0 access login example.png|center|]]
 

== Troubleshooting ==

'''Issue: The "Your OTP:" prompt never appears and the connection hangs/times out instead'''

Likely cause: You are most likely not on a network from which access to the bwUniCluster 2.0 system is allowed. Please check if you might have to establish a VPN connection first.

 

'''Issue: The system asks for the One-Time Password multiple times'''

Likely cause: Make sure you are using the correct Software Token to generate the One-Time Password.

 

'''Issue: The system asks for the service password multiple times'''

Likely cause: Make sure you are using the service password set on bwIDM and not the password valid for your home institution. Unlike the bwUniCluster 1, the bwUniCluster 2.0 only accepts the service password.

 

'''Issue: There is an error message by the pam_ses_open.sh skript'''

Likely cause: Your account is in the "LOST_ACCESS" state because the entitlement is no longer valid, the questionaire was not filled out or there was a problem during the communication between your home institution and the central bwIDM system. Please try the following steps:

* Log into [https://bwidm.scc.kit.edu bwIDM], look for the bwUniCluster entry and click on '''Registry info'''. Your "Status:" should be "ACTIVE". If it is not, please wait for ten minutes since logging into the bwIDM causes a refresh and the problem might fix itself. If the status does not change to ACTIVE after a longer amount of time, please contact the support channels.

* If you have not filled out the questionaire, please do so on [https://zas.bwhpc.de/shib/en/bwunicluster_survey.php https://zas.bwhpc.de/shib/en/bwunicluster_survey.php] and then wait for about ten minutes before attempting to log into the HPC system again.
 
 

== Allowed activities on login nodes ==

The login nodes of bwUniCluster 2.0 are the access point to the compute system and to your bwUniCluster 2.0 $HOME directory. The login nodes are shared with all the users of bwUniCluster 2.0. Therefore, your activities on the login nodes are limited to primarily set up your batch jobs. Your activities may also be:

* '''short''' compilation of your program code and
* '''short''' pre- and postprocessing of your batch jobs.

To guarantee usability for all the users of bwUniCluster 2.0 '''you must not run your compute jobs on the login nodes'''. Compute jobs must be submitted to the
[[bwUniCluster Batch Jobs|queueing system]]. Any compute job running on the login nodes will be terminated without any notice. Any long-running compilation or any long-running pre- or postprocessing of batch jobs must also be submitted to the [[bwUniCluster Batch Jobs|queueing system]].
 
 

== SSH Keys ==

In contrast to the bwUniCluster 1 and many other HPC systems it is '''no longer possible to self-manage your SSH Keys by adding them to the ~/.ssh/authorized_keys file'''. Existing files will no longer be evaluated. SSH Keys have to be managed via the central bwIDM system instead. Please refer to the user guide for this functionality:

[[bwUniCluster 2.0 User Access/SSH Keys]]
 
 

= [[First_Steps_on_bwHPC_cluster|First steps on bwUniCluster]] =

First and some important steps on bwUniCluster 2.0 can be found [[First_Steps_on_bwHPC_cluster|here]].
 
 

= Deregistration =
If you plan to permanently leave the bwUniCluster 2.0, follow the deregister checklist:
# Transfer all your data in $HOME and workspace to your local computer/storage and after that clear off all your data
# Visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
#* Select your home organization from the list and click '''Proceed'''
#* Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
#* You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
#* <div>Select '''Registry Info''' of the service '''bwUniCluster''' (on the left hand side) [[File:bwUniCluster_registration_sidebar.png|center|border|]]</div>
#* Click '''Deregister'''
Note that Step 2 will automatically unsubscribe you from the bwunicluster mail list.

BwUniCluster 2.0 User Access

2020-12-17T14:15:29Z

H Haefner: /* Step B: Web Registration, service password and 2-factor authentication */

[[bwUniCluster_2.0|bwUniCluster 2.0]] is Baden-Württemberg's general purpose tier 3 high performance computing (HPC)
cluster co-financed by Baden-Württemberg's ministry of science, research and arts and the shareholders:

* Albert Ludwig University of Freiburg
* Eberhard Karls University, Tübingen
* Karlsruhe Institute of Technology (KIT)
* Heidelberg University (Ruprecht-Karls-Universität Heidelberg)
* Ulm University
* University of Hohenheim
* University of Konstanz
* University of Mannheim
* University of Stuttgart
* HAW BW e.V. (an association of several universities of applied sciences in Baden-Württemberg, see below)
 
To '''log on''' [[bwUniCluster_2.0|bwUniCluster 2.0]] a user account is required. All members of the shareholder
universities can apply for an account.
 

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:left; color:#000;vertical-align:top;" |__TOC__
| [[File:bwUniCluster_17Jan2014_p044-rot_t10.10.00.jpg|center|border|250px|bwUniCluster wiring by Holger Obermaier, copyright: KIT (SCC)]] bwUniCluster wiring © KIT (SCC)
|}

= Registration =

Granting access and issuing a user account for '''bwUniCluster 2.0''' requires the registration at the KIT service website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] (step B). However, this registration depends on the '''bwUniCluster entitlement''' issued by your university (step A).
 

== Step A: bwUniCluster entitlement for registration ==
'''The entitlement is called bwUniCluster (not bwUniCluster 2.0)''' and each university issues the bwUniCluster entitlement '''only''' for their own respective members. Some have established on-line processes or provide downloads of the entitlement application forms. If there is no link behind the name of an institution in the following list, please contact the local IT support services:

* [[BwCluster_User_Access_Uni_Freiburg|Albert Ludwig University of Freiburg]]
* [https://bwunicluster.urz.uni-heidelberg.de/ Heidelberg University]
* [https://kim.uni-hohenheim.de/bwhpc-account University of Hohenheim]
* [http://www.scc.kit.edu/downloads/sdo/Antrag_Benutzernummer_bwUniCluster.pdf Karlsruhe Institute of Technology (KIT)]
* [[BWUniCluster_User_Access_Members_Uni_Konstanz|University of Konstanz]]
* [[BWUniCluster_User_Access_Members_Uni_Mannheim|University of Mannheim]]
* [https://www.hlrs.de/solutions-services/academic-users/bwunicluster-access/ University of Stuttgart]
* [https://uni-tuebingen.de/de/155157 Eberhard Karls University Tübingen]
* [[BWUniCluster_User_Access_Members_Uni_Ulm|Ulm University]]
* Hochschule Aalen
* Hochschule Albstadt-Sigmaringen
* Hochschule Esslingen
* Hochschule Furtwangen
* Hochschule Heilbronn
* Hochschule Karlsruhe
* Hochschule Konstanz
* Hochschule Reutlingen
* Hochschule Rottenburg
* Hochschule Stuttgart (HfT)
* Hochschule Ulm
 

== Step B: Web Registration, service password and 2-factor authentication ==

After completing step A, i.e., after successfull issueing of the bwUniCluster entitlement, you have to register yourself for the service. To do so please visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and complete the following steps.

1. Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.

[[File:bwIDM-1-red.png|center|border|]]
 

2. You will be directed to the ''Identity Provider'' of your home organisation. Enter the user ID / username and password of your home organisation - this is usually the same password used for your e-mail account and other services - and click on '''Login''', '''Einloggen''' or something similar.
 

3. You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]. If you are logging into bwIDM for the first time, there will be a summary screen which shows the account details your home institution is providing to the central system. Please check that all data is valid and then click on '''Continue''' or '''Weiter'''.
 

4. Once you have successfully logged into the bwIDM system, you will be greeted by a home screen showing all state-wide services you have access to. There will be a box labelled "bwUniCluster". Click on '''Register''' or '''Registrieren''' to start the registration process.

[[File:BwUniCluster 2.0 access login bwidm service list.png|center|border|]]
 

5. Since August 13, 2020 a '''2-factor authentication''' mechanism (2FA) is being enforced to improve security. If you have never registered a 2FA token on bwIDM before, the following error message will appear:

[[File:BwUniCluster 2.0 access login bwidm registration requirements 2fa.png|center|]]

Click on the [https://bwidm.scc.kit.edu/user/twofa.xhtml Link] or on the '''My Tokens''' link in the main menu. The instructions for registering a new 2FA token can be found on the following page: [[bwUniCluster 2.0 User Access/2FA Tokens]]. Please complete them before proceeding.
 

6. Make sure all requirements are met by checking the '''Requirements''' box at the top. If the requirements are not met you might be able to correct the issure by following the instructions. In all other cases please [[Registration_Support_-_bwUniCluster|contact your local hotline]].

[[File:BwUniCluster 2.0 access login bwidm registration requirements.png|center|border|]]
 

7. Read the Terms of Use ('''Nutzungsbedingungen und -richtlinien'''), check the box besides '''I have read and accepted the terms of use''' and click on '''Register''' or '''Registrieren'''.
 

8. Set a service password for the bwUniCluster and click on '''Save''' or '''Speichern'''. Logging in with the password of your home organisation, like on the former bwUniCluster 1, is no longer possible. Please make sure to use a strong password which is different from any other password you are currently using or have used on other systems. You will also be asked to change the service password regularly.

[[File:BwUniCluster 2.0 access login bwidm service password.png|center|]]
 
 

== Step C: Fill out the bwUniCluster questionnaire ==

Filling out the bwUniCluster questionaire on

https://zas.bwhpc.de/shib/en/bwunicluster_survey.php

is mandatory for all users. The input is solely used to improve our support activities and for capacity planning of future HPC resources. '''If the questionaire is not filled out, access to bwUniCluster 2.0 is blocked 14 days after the registration.'''
 
 

== Changing the Service Password ==

Your bwUniCluster 2.0 '''password''' is the service password you set during the web registration (compare step 7 of chapter 1.2). At any time, you can set a new bwUniCluster 2.0 password via the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] by carrying out the following steps:
# Go to [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and select your home organization
# Authenticate yourself via the user id / username and password provided by your home institution
# Find the entry '''bwUniCluster''' and select '''Set Service Password'''
# Enter the new password, repeat it and click '''Save''' button.
# If the change was sucessfull, the message "Das Passwort wurde bei dem Dienst geändert" ("Password has been changed") will be shown
# Proceed to log in using the new password
 
 
== Contact / Support ==
If you have questions or problems concerning the bwUniCluster (2.0) registration, please [[bwUniCluster 2.0 Support|contact your local hotline]].
 
 

= Establishing network access =

Access to bwUniCluster 2.0 is '''limited to IP addresses from the so-called BelWü networks'''. All home institutions of our current users are connected to BelWue, so if you are on your campus network (e.g. in your office or on the Campus WiFi) you should be able to connect to bwUniCluster 2.0 without restrictions. If you are outside of one of the BelWue networks (e.g. in your home office instead of in your campus office), a VPN connection to your home institution has to be established first (see e.g. [1] for the KIT).
 
 

= Login =

After finishing the web registration and making sure that you are on a network from which you have access to bwUniCluster 2.0 (e.g. by establishing a VPN connection), the HPC cluster is ready for your '''SSH''' based login. Recommended SSH clients applications are:

* the ssh (OpenSSH) command included in all Linux distributions and macOS, -in command under Linux and macOS using the application ''terminal''
* [http://mobaxterm.mobatek.net/ MobaXterm] under Windows
 

== Hostnames ==

The main hostname required to connect to bwUniCluster 2.0 is '''bwunicluster.scc.kit.edu''' or '''uc2.scc.kit.edu'''. The system has four login nodes and we use so-called ''DNS round-robin scheduling'' to load-balance the incoming connections between the nodes. If you open multiple SSH sessions to bwUniCluster 2.0, these sessions will be established to different login nodes, so processes started in one session might not be visible in other sessions.

The older Broadwell extension partition of the former bwUniCluster 1 is connected to bwUniCluster 2.0. You can use the hostname '''uc1e.scc.kit.edu''' to connect to the login nodes of this partition.

If you need to connect to specific login nodes, you can use the following hostnames:

{| class="wikitable"
! Hostname !! Node type
|-
| '''uc2-login1.scc.kit.edu''' || bwUniCluster 2.0, first login node
|-
| '''uc2-login2.scc.kit.edu''' || bwUniCluster 2.0, second login node
|-
| '''uc2-login3.scc.kit.edu''' || bwUniCluster 2.0, third login node
|-
| '''uc2-login4.scc.kit.edu''' || bwUniCluster 2.0, fourth login node
|-
| '''uc1e-login1.scc.kit.edu''' || Broadwell partition, first login node
|-
| '''uc1e-login2.scc.kit.edu''' || Broadwell partition, second login node
|-
|}

Only the secure shell ''SSH'' is allowed to login. Other protocols like ''telnet'' or ''rlogin'' are not allowed for security reasons.
 

== Usernames ==

Your username will be the same as the one provided by your home institution, but '''prefixed''' with two characters and an underscore indicating your home institution. For example: If you are a member of the university of Konstanz and your local username is ab1234, your username on bwUniCluster 2.0 is kn_ab1234.

The following list contains all prefixes currently in use:

{| class="wikitable"
! Home organization !! <UserID>
|-
| Universität Freiburg || ''fr_''username
|-
| Universität Heidelberg || ''hd_''username
|-
| Universität Hohenheim || ''ho_''username
|-
| KIT || username ''(without any prefix)''
|-
| Universität Konstanz || ''kn_''username
|-
| Universität Mannheim || ''ma_''username
|-
| Universität Stuttgart || ''st_''username
|-
| Universität Tübingen || ''tu_''username
|-
| Universität Ulm || ''ul_''username
|-
| Hochschule Aalen || ''aa_''username
|-
| Hochschule Albstadt-Sigmaringen || ''as_''username
|-
| Hochschule Esslingen || ''es_''username
|-
| Hochschule Furtwangen || ''hf_''username
|-
| Hochschule Heilbronn || ''hn_''username
|-
| Hochschule Karlsruhe || ''hk_''username
|-
| HTWG Konstanz || ''ht_''username
|-
| Hochschule Reutlingen || ''hr_''username
|-
| Hochschule Rottenburg || ''ro_''username
|-
| Hochschule für Technik Stuttgart || ''hs_''username
|-
| Hochschule Ulm || ''hu_''username
|-
|}
 

== Client application: OpenSSH ==

Most Unix and Unix-like operating systems like Linux, macOS and *BSD come with a built-in SSH client provided by the OpenSSH project. More recent versions of Windows 10 and the Windows Subsystem for Linux also come with a built-in OpenSSH client.

To use this client, simply open a command line terminal (the exact process differs on every operating system, but usually involves starting an application called '''Terminal''' or '''Command Prompt''') and enter the following command to connect to bwUniCluster 2.0:

<pre>
$ ssh <UserID>@bwunicluster.scc.kit.edu
</pre>

If you are on a Linux or Unix system running the X Window System (X11) and want to use a GUI-based application on bwUniCluster 2.0, you can use the ''-X'' option for the ssh command to set up X11 forwarding:

<pre>
$ ssh -X <UserID>@uc2.scc.kit.edu
</pre>

Windows users requiring X11 forwarding for graphical applications should use '''MobaXterm''' instead.
 

== Client application: MobaXterm ==

The bwHPC-C5 support team strongly recommends to use [http://mobaxterm.mobatek.net/ MobaXterm] instead of ''PuTTY'' or ''WinSCP'' on Windows. ''MobaXterm'' provides a built-in X11 server allowing to start GUI based software.

Start ''MobaXterm'', fill in the following fields:
<pre>
Remote name : uc2.scc.kit.edu # or uc1e.scc.kit.edu
Specify user name : <UserID>
Port : 22
</pre>

After that click on 'ok'. Then a terminal will be opened and there you can enter your credentials.
 

== Example login process ==

After the connection has been initiated, a successful login process will go through the following three steps:

1. The system asks for a '''One-Time Password'''. Generate one using the Software or Hardware Token registered on the bwIDM system (see [[bwUniCluster 2.0 User Access/2FA Tokens]]) and enter it after the '''Your OTP:''' prompt.

2. The systems asks for your service password. Enter it after the '''Password:''' prompt.

3. You are greeted by the bwUniCluster 2.0 banner followed by a shell.

The result should look like this:

[[File:BwUniCluster 2.0 access login example.png|center|]]
 

== Troubleshooting ==

'''Issue: The "Your OTP:" prompt never appears and the connection hangs/times out instead'''

Likely cause: You are most likely not on a network from which access to the bwUniCluster 2.0 system is allowed. Please check if you might have to establish a VPN connection first.

 

'''Issue: The system asks for the One-Time Password multiple times'''

Likely cause: Make sure you are using the correct Software Token to generate the One-Time Password.

 

'''Issue: The system asks for the service password multiple times'''

Likely cause: Make sure you are using the service password set on bwIDM and not the password valid for your home institution. Unlike the bwUniCluster 1, the bwUniCluster 2.0 only accepts the service password.

 

'''Issue: There is an error message by the pam_ses_open.sh skript'''

Likely cause: Your account is in the "LOST_ACCESS" state because the entitlement is no longer valid, the questionaire was not filled out or there was a problem during the communication between your home institution and the central bwIDM system. Please try the following steps:

* Log into [https://bwidm.scc.kit.edu bwIDM], look for the bwUniCluster entry and click on '''Registry info'''. Your "Status:" should be "ACTIVE". If it is not, please wait for ten minutes since logging into the bwIDM causes a refresh and the problem might fix itself. If the status does not change to ACTIVE after a longer amount of time, please contact the support channels.

* If you have not filled out the questionaire, please do so on [https://zas.bwhpc.de/shib/en/bwunicluster_survey.php https://zas.bwhpc.de/shib/en/bwunicluster_survey.php] and then wait for about ten minutes before attempting to log into the HPC system again.
 
 

== Allowed activities on login nodes ==

The login nodes of bwUniCluster 2.0 are the access point to the compute system and to your bwUniCluster 2.0 $HOME directory. The login nodes are shared with all the users of bwUniCluster 2.0. Therefore, your activities on the login nodes are limited to primarily set up your batch jobs. Your activities may also be:

* '''short''' compilation of your program code and
* '''short''' pre- and postprocessing of your batch jobs.

To guarantee usability for all the users of bwUniCluster 2.0 '''you must not run your compute jobs on the login nodes'''. Compute jobs must be submitted to the
[[bwUniCluster Batch Jobs|queueing system]]. Any compute job running on the login nodes will be terminated without any notice. Any long-running compilation or any long-running pre- or postprocessing of batch jobs must also be submitted to the [[bwUniCluster Batch Jobs|queueing system]].
 
 

== SSH Keys ==

In contrast to the bwUniCluster 1 and many other HPC systems it is '''no longer possible to self-manage your SSH Keys by adding them to the ~/.ssh/authorized_keys file'''. Existing files will no longer be evaluated. SSH Keys have to be managed via the central bwIDM system instead. Please refer to the user guide for this functionality:

[[bwUniCluster 2.0 User Access/SSH Keys]]
 
 

= [[First_Steps_on_bwHPC_cluster|First steps on bwUniCluster]] =

First and some important steps on bwUniCluster 2.0 can be found [[First_Steps_on_bwHPC_cluster|here]].
 
 

= Deregistration =
If you plan to permanently leave the bwUniCluster 2.0, follow the deregister checklist:
# Transfer all your data in $HOME and workspace to your local computer/storage and after that clear off all your data
# Visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
#* Select your home organization from the list and click '''Proceed'''
#* Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
#* You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
#* <div>Select '''Registry Info''' of the service '''bwUniCluster''' (on the left hand side) [[File:bwUniCluster_registration_sidebar.png|center|border|]]</div>
#* Click '''Deregister'''
Note that Step 2 will automatically unsubscribe you from the bwunicluster mail list.

File:BwIDM-1.png

2020-12-17T14:12:57Z

H Haefner:

BwUniCluster 2.0 User Access

2020-12-17T14:04:42Z

H Haefner: /* Step B: Web Registration, service password and 2-factor authentication */

[[bwUniCluster_2.0|bwUniCluster 2.0]] is Baden-Württemberg's general purpose tier 3 high performance computing (HPC)
cluster co-financed by Baden-Württemberg's ministry of science, research and arts and the shareholders:

* Albert Ludwig University of Freiburg
* Eberhard Karls University, Tübingen
* Karlsruhe Institute of Technology (KIT)
* Heidelberg University (Ruprecht-Karls-Universität Heidelberg)
* Ulm University
* University of Hohenheim
* University of Konstanz
* University of Mannheim
* University of Stuttgart
* HAW BW e.V. (an association of several universities of applied sciences in Baden-Württemberg, see below)
 
To '''log on''' [[bwUniCluster_2.0|bwUniCluster 2.0]] a user account is required. All members of the shareholder
universities can apply for an account.
 

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:left; color:#000;vertical-align:top;" |__TOC__
| [[File:bwUniCluster_17Jan2014_p044-rot_t10.10.00.jpg|center|border|250px|bwUniCluster wiring by Holger Obermaier, copyright: KIT (SCC)]] bwUniCluster wiring © KIT (SCC)
|}

= Registration =

Granting access and issuing a user account for '''bwUniCluster 2.0''' requires the registration at the KIT service website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] (step B). However, this registration depends on the '''bwUniCluster entitlement''' issued by your university (step A).
 

== Step A: bwUniCluster entitlement for registration ==
'''The entitlement is called bwUniCluster (not bwUniCluster 2.0)''' and each university issues the bwUniCluster entitlement '''only''' for their own respective members. Some have established on-line processes or provide downloads of the entitlement application forms. If there is no link behind the name of an institution in the following list, please contact the local IT support services:

* [[BwCluster_User_Access_Uni_Freiburg|Albert Ludwig University of Freiburg]]
* [https://bwunicluster.urz.uni-heidelberg.de/ Heidelberg University]
* [https://kim.uni-hohenheim.de/bwhpc-account University of Hohenheim]
* [http://www.scc.kit.edu/downloads/sdo/Antrag_Benutzernummer_bwUniCluster.pdf Karlsruhe Institute of Technology (KIT)]
* [[BWUniCluster_User_Access_Members_Uni_Konstanz|University of Konstanz]]
* [[BWUniCluster_User_Access_Members_Uni_Mannheim|University of Mannheim]]
* [https://www.hlrs.de/solutions-services/academic-users/bwunicluster-access/ University of Stuttgart]
* [https://uni-tuebingen.de/de/155157 Eberhard Karls University Tübingen]
* [[BWUniCluster_User_Access_Members_Uni_Ulm|Ulm University]]
* Hochschule Aalen
* Hochschule Albstadt-Sigmaringen
* Hochschule Esslingen
* Hochschule Furtwangen
* Hochschule Heilbronn
* Hochschule Karlsruhe
* Hochschule Konstanz
* Hochschule Reutlingen
* Hochschule Rottenburg
* Hochschule Stuttgart (HfT)
* Hochschule Ulm
 

== Step B: Web Registration, service password and 2-factor authentication ==

After completing step A, i.e., after successfull issueing of the bwUniCluster entitlement, you have to register yourself for the service. To do so please visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and complete the following steps.

1. Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.

[[File:BwUniCluster 2.0 access login bwidm select institution_new.png|center|border|]]
 

2. You will be directed to the ''Identity Provider'' of your home organisation. Enter the user ID / username and password of your home organisation - this is usually the same password used for your e-mail account and other services - and click on '''Login''', '''Einloggen''' or something similar.
 

3. You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]. If you are logging into bwIDM for the first time, there will be a summary screen which shows the account details your home institution is providing to the central system. Please check that all data is valid and then click on '''Continue''' or '''Weiter'''.
 

4. Once you have successfully logged into the bwIDM system, you will be greeted by a home screen showing all state-wide services you have access to. There will be a box labelled "bwUniCluster". Click on '''Register''' or '''Registrieren''' to start the registration process.

[[File:BwUniCluster 2.0 access login bwidm service list.png|center|border|]]
 

5. Since August 13, 2020 a '''2-factor authentication''' mechanism (2FA) is being enforced to improve security. If you have never registered a 2FA token on bwIDM before, the following error message will appear:

[[File:BwUniCluster 2.0 access login bwidm registration requirements 2fa.png|center|]]

Click on the [https://bwidm.scc.kit.edu/user/twofa.xhtml Link] or on the '''My Tokens''' link in the main menu. The instructions for registering a new 2FA token can be found on the following page: [[bwUniCluster 2.0 User Access/2FA Tokens]]. Please complete them before proceeding.
 

6. Make sure all requirements are met by checking the '''Requirements''' box at the top. If the requirements are not met you might be able to correct the issure by following the instructions. In all other cases please [[Registration_Support_-_bwUniCluster|contact your local hotline]].

[[File:BwUniCluster 2.0 access login bwidm registration requirements.png|center|border|]]
 

7. Read the Terms of Use ('''Nutzungsbedingungen und -richtlinien'''), check the box besides '''I have read and accepted the terms of use''' and click on '''Register''' or '''Registrieren'''.
 

8. Set a service password for the bwUniCluster and click on '''Save''' or '''Speichern'''. Logging in with the password of your home organisation, like on the former bwUniCluster 1, is no longer possible. Please make sure to use a strong password which is different from any other password you are currently using or have used on other systems. You will also be asked to change the service password regularly.

[[File:BwUniCluster 2.0 access login bwidm service password.png|center|]]
 
 

== Step C: Fill out the bwUniCluster questionnaire ==

Filling out the bwUniCluster questionaire on

https://zas.bwhpc.de/shib/en/bwunicluster_survey.php

is mandatory for all users. The input is solely used to improve our support activities and for capacity planning of future HPC resources. '''If the questionaire is not filled out, access to bwUniCluster 2.0 is blocked 14 days after the registration.'''
 
 

== Changing the Service Password ==

Your bwUniCluster 2.0 '''password''' is the service password you set during the web registration (compare step 7 of chapter 1.2). At any time, you can set a new bwUniCluster 2.0 password via the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] by carrying out the following steps:
# Go to [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and select your home organization
# Authenticate yourself via the user id / username and password provided by your home institution
# Find the entry '''bwUniCluster''' and select '''Set Service Password'''
# Enter the new password, repeat it and click '''Save''' button.
# If the change was sucessfull, the message "Das Passwort wurde bei dem Dienst geändert" ("Password has been changed") will be shown
# Proceed to log in using the new password
 
 
== Contact / Support ==
If you have questions or problems concerning the bwUniCluster (2.0) registration, please [[bwUniCluster 2.0 Support|contact your local hotline]].
 
 

= Establishing network access =

Access to bwUniCluster 2.0 is '''limited to IP addresses from the so-called BelWü networks'''. All home institutions of our current users are connected to BelWue, so if you are on your campus network (e.g. in your office or on the Campus WiFi) you should be able to connect to bwUniCluster 2.0 without restrictions. If you are outside of one of the BelWue networks (e.g. in your home office instead of in your campus office), a VPN connection to your home institution has to be established first (see e.g. [1] for the KIT).
 
 

= Login =

After finishing the web registration and making sure that you are on a network from which you have access to bwUniCluster 2.0 (e.g. by establishing a VPN connection), the HPC cluster is ready for your '''SSH''' based login. Recommended SSH clients applications are:

* the ssh (OpenSSH) command included in all Linux distributions and macOS, -in command under Linux and macOS using the application ''terminal''
* [http://mobaxterm.mobatek.net/ MobaXterm] under Windows
 

== Hostnames ==

The main hostname required to connect to bwUniCluster 2.0 is '''bwunicluster.scc.kit.edu''' or '''uc2.scc.kit.edu'''. The system has four login nodes and we use so-called ''DNS round-robin scheduling'' to load-balance the incoming connections between the nodes. If you open multiple SSH sessions to bwUniCluster 2.0, these sessions will be established to different login nodes, so processes started in one session might not be visible in other sessions.

The older Broadwell extension partition of the former bwUniCluster 1 is connected to bwUniCluster 2.0. You can use the hostname '''uc1e.scc.kit.edu''' to connect to the login nodes of this partition.

If you need to connect to specific login nodes, you can use the following hostnames:

{| class="wikitable"
! Hostname !! Node type
|-
| '''uc2-login1.scc.kit.edu''' || bwUniCluster 2.0, first login node
|-
| '''uc2-login2.scc.kit.edu''' || bwUniCluster 2.0, second login node
|-
| '''uc2-login3.scc.kit.edu''' || bwUniCluster 2.0, third login node
|-
| '''uc2-login4.scc.kit.edu''' || bwUniCluster 2.0, fourth login node
|-
| '''uc1e-login1.scc.kit.edu''' || Broadwell partition, first login node
|-
| '''uc1e-login2.scc.kit.edu''' || Broadwell partition, second login node
|-
|}

Only the secure shell ''SSH'' is allowed to login. Other protocols like ''telnet'' or ''rlogin'' are not allowed for security reasons.
 

== Usernames ==

Your username will be the same as the one provided by your home institution, but '''prefixed''' with two characters and an underscore indicating your home institution. For example: If you are a member of the university of Konstanz and your local username is ab1234, your username on bwUniCluster 2.0 is kn_ab1234.

The following list contains all prefixes currently in use:

{| class="wikitable"
! Home organization !! <UserID>
|-
| Universität Freiburg || ''fr_''username
|-
| Universität Heidelberg || ''hd_''username
|-
| Universität Hohenheim || ''ho_''username
|-
| KIT || username ''(without any prefix)''
|-
| Universität Konstanz || ''kn_''username
|-
| Universität Mannheim || ''ma_''username
|-
| Universität Stuttgart || ''st_''username
|-
| Universität Tübingen || ''tu_''username
|-
| Universität Ulm || ''ul_''username
|-
| Hochschule Aalen || ''aa_''username
|-
| Hochschule Albstadt-Sigmaringen || ''as_''username
|-
| Hochschule Esslingen || ''es_''username
|-
| Hochschule Furtwangen || ''hf_''username
|-
| Hochschule Heilbronn || ''hn_''username
|-
| Hochschule Karlsruhe || ''hk_''username
|-
| HTWG Konstanz || ''ht_''username
|-
| Hochschule Reutlingen || ''hr_''username
|-
| Hochschule Rottenburg || ''ro_''username
|-
| Hochschule für Technik Stuttgart || ''hs_''username
|-
| Hochschule Ulm || ''hu_''username
|-
|}
 

== Client application: OpenSSH ==

Most Unix and Unix-like operating systems like Linux, macOS and *BSD come with a built-in SSH client provided by the OpenSSH project. More recent versions of Windows 10 and the Windows Subsystem for Linux also come with a built-in OpenSSH client.

To use this client, simply open a command line terminal (the exact process differs on every operating system, but usually involves starting an application called '''Terminal''' or '''Command Prompt''') and enter the following command to connect to bwUniCluster 2.0:

<pre>
$ ssh <UserID>@bwunicluster.scc.kit.edu
</pre>

If you are on a Linux or Unix system running the X Window System (X11) and want to use a GUI-based application on bwUniCluster 2.0, you can use the ''-X'' option for the ssh command to set up X11 forwarding:

<pre>
$ ssh -X <UserID>@uc2.scc.kit.edu
</pre>

Windows users requiring X11 forwarding for graphical applications should use '''MobaXterm''' instead.
 

== Client application: MobaXterm ==

The bwHPC-C5 support team strongly recommends to use [http://mobaxterm.mobatek.net/ MobaXterm] instead of ''PuTTY'' or ''WinSCP'' on Windows. ''MobaXterm'' provides a built-in X11 server allowing to start GUI based software.

Start ''MobaXterm'', fill in the following fields:
<pre>
Remote name : uc2.scc.kit.edu # or uc1e.scc.kit.edu
Specify user name : <UserID>
Port : 22
</pre>

After that click on 'ok'. Then a terminal will be opened and there you can enter your credentials.
 

== Example login process ==

After the connection has been initiated, a successful login process will go through the following three steps:

1. The system asks for a '''One-Time Password'''. Generate one using the Software or Hardware Token registered on the bwIDM system (see [[bwUniCluster 2.0 User Access/2FA Tokens]]) and enter it after the '''Your OTP:''' prompt.

2. The systems asks for your service password. Enter it after the '''Password:''' prompt.

3. You are greeted by the bwUniCluster 2.0 banner followed by a shell.

The result should look like this:

[[File:BwUniCluster 2.0 access login example.png|center|]]
 

== Troubleshooting ==

'''Issue: The "Your OTP:" prompt never appears and the connection hangs/times out instead'''

Likely cause: You are most likely not on a network from which access to the bwUniCluster 2.0 system is allowed. Please check if you might have to establish a VPN connection first.

 

'''Issue: The system asks for the One-Time Password multiple times'''

Likely cause: Make sure you are using the correct Software Token to generate the One-Time Password.

 

'''Issue: The system asks for the service password multiple times'''

Likely cause: Make sure you are using the service password set on bwIDM and not the password valid for your home institution. Unlike the bwUniCluster 1, the bwUniCluster 2.0 only accepts the service password.

 

'''Issue: There is an error message by the pam_ses_open.sh skript'''

Likely cause: Your account is in the "LOST_ACCESS" state because the entitlement is no longer valid, the questionaire was not filled out or there was a problem during the communication between your home institution and the central bwIDM system. Please try the following steps:

* Log into [https://bwidm.scc.kit.edu bwIDM], look for the bwUniCluster entry and click on '''Registry info'''. Your "Status:" should be "ACTIVE". If it is not, please wait for ten minutes since logging into the bwIDM causes a refresh and the problem might fix itself. If the status does not change to ACTIVE after a longer amount of time, please contact the support channels.

* If you have not filled out the questionaire, please do so on [https://zas.bwhpc.de/shib/en/bwunicluster_survey.php https://zas.bwhpc.de/shib/en/bwunicluster_survey.php] and then wait for about ten minutes before attempting to log into the HPC system again.
 
 

== Allowed activities on login nodes ==

The login nodes of bwUniCluster 2.0 are the access point to the compute system and to your bwUniCluster 2.0 $HOME directory. The login nodes are shared with all the users of bwUniCluster 2.0. Therefore, your activities on the login nodes are limited to primarily set up your batch jobs. Your activities may also be:

* '''short''' compilation of your program code and
* '''short''' pre- and postprocessing of your batch jobs.

To guarantee usability for all the users of bwUniCluster 2.0 '''you must not run your compute jobs on the login nodes'''. Compute jobs must be submitted to the
[[bwUniCluster Batch Jobs|queueing system]]. Any compute job running on the login nodes will be terminated without any notice. Any long-running compilation or any long-running pre- or postprocessing of batch jobs must also be submitted to the [[bwUniCluster Batch Jobs|queueing system]].
 
 

== SSH Keys ==

In contrast to the bwUniCluster 1 and many other HPC systems it is '''no longer possible to self-manage your SSH Keys by adding them to the ~/.ssh/authorized_keys file'''. Existing files will no longer be evaluated. SSH Keys have to be managed via the central bwIDM system instead. Please refer to the user guide for this functionality:

[[bwUniCluster 2.0 User Access/SSH Keys]]
 
 

= [[First_Steps_on_bwHPC_cluster|First steps on bwUniCluster]] =

First and some important steps on bwUniCluster 2.0 can be found [[First_Steps_on_bwHPC_cluster|here]].
 
 

= Deregistration =
If you plan to permanently leave the bwUniCluster 2.0, follow the deregister checklist:
# Transfer all your data in $HOME and workspace to your local computer/storage and after that clear off all your data
# Visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
#* Select your home organization from the list and click '''Proceed'''
#* Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
#* You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
#* <div>Select '''Registry Info''' of the service '''bwUniCluster''' (on the left hand side) [[File:bwUniCluster_registration_sidebar.png|center|border|]]</div>
#* Click '''Deregister'''
Note that Step 2 will automatically unsubscribe you from the bwunicluster mail list.

File:BwUniCluster 2.0 access login bwidm service list new.png

2020-12-17T14:03:36Z

H Haefner:

BwUniCluster2.0/Maintenance/2020-10

2020-10-01T12:55:03Z

H Haefner: /* Compilers, Libaries and Runtime Environments */

The following changes are planned for the maintenance interval starting on 06.10.2020 (Tuesday) at 10 AM and ending on 13.10.2020 (Tuesday) at 10 AM.

'''This information is still preliminary and subjected to change.'''

= Operating system =

* The operating system version will be upgraded to Red Hat Enterprise Linux (RHEL) 8.2. We recommend to re-compile all applications after the upgrade.

* The Linux kernel version will be upgraded to 4.18 (from 3.10).

* The Mellanox OFED InfiniBand Stack will be upgraded to version 4.9 (from 4.7). If you maintain libraries which directly link against OFED libraried (e.g. MPI or PGAS libraries compiled in your $HOME directory) we recommend to re-compile these libraries after the upgrade.

= Compilers, Libaries and Runtime Environments =

* The ''GNU C Library'' (glibc) will be upgraded to version 2.28 (from 2.17). This version will no longer automatically print backtraces after stack checking failures due to security hardening, which might affect the debugging of applications.

* The default system compiler will be upgraded to ''GCC'' version 8.3.1 (from 4.8.5).

* The Intel compiler module versions will not change.

* The ''Clang'' 8 module will probably be replaced in favor of Clang 10. The final decision on this has not been made yet. Clang 9 will continue to be available in any case.

* The PGI Compiler will be updated to version 20.7 (from 19.1). It includes the NVIDIA environment in version 11.0 (e.g. nvcc).

* The system ''Python'' version will be upgraded to 3.6. If no other Python module is loaded, the ''python'' command will default to Python 3.6, while Python 2.7 can be still used with the ''python2'' command.

* The system ''Perl'' version will be upgraded to 5.26.

= Development tools =

* The default system version of the ''GNU Make'' build utility will be upgraded to 4.2.1. Please note the new ''--output-sync'' command line option that allows better debugging of parallel builds.

* The default system version of the ''autotools'' package will be upgraded to 2.69. The "devel/autotools/2.69" software module is no longer necessary and will be removed.

* The default system version of the ''CMake'' build system will be upgraded to 3.11.

* The default system version of the ''Git'' version control utility will be upgraded to 2.18.4. The "devel/git/2.18.4" software module is no longer necessary and will be removed.

= Userspace tools =

* The ''screen'' utility is no longer be supported by Red Hat. The recommended replacement is ''tmux''.

= Batch system =

* The Slurm version will be upgraded to the most recent minor version available at the start of the maintenance interval. At this moment this would be version 20.02.5 (instead of 20.02.3).

* The maximum total number of jobs of a single user in all queues (running and waiting) will be reduced to 100 (from 200).

= Graphics stack =

* The KDE Plasma desktop environment is no longer supported by Red Hat. ''start_vnc_desktop'' will default to GNOME 3.28 instead.

* The NVIDIA driver will be upgraded to the latest version available at the start of the maintenance interval.

= Containers =

* Network Namespaces will be disabled due to security concerns.

BwUniCluster2.0/Batch Queues

2020-07-03T13:22:43Z

H Haefner: /* sbatch -p queue */

This article contains information on the queues of the [[BwUniCluster_2.0_Slurm_common_Features|batch job system]] and on interactive jobs.

= Job Submission =
== sbatch Command ==

=== sbatch -p ''queue'' ===
Compute resources such as (wall-)time, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, '''you must add the correct queue class to your sbatch command '''. 'The specification of a queue is obligatory on BwUniCluster 2.0.
 
Details are:

{| width=750px class="wikitable"
! colspan="5" | bwUniCluster 2.0 sbatch -p ''queue''
|- style="text-align:left;"
! queue !! node !! default resources !! minimum resources !! maximum resources
|- style="text-align:left"
| dev_single
| thin
| time=10, mem-per-cpu=1125mb
|
| time=30, nodes=1, mem=90000mb, ntasks-per-node=40, (threads-per-core=2) 4 nodes are reserved for this queue. Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| single
| thin
| time=30, mem-per-cpu=1125mb
|
| time=72:00:00, nodes=1, mem=90000mb, ntasks-per-node=40, (threads-per-core)=2
|- style="text-align:left;"
| dev_multiple
| thin
| time=10, mem-per-cpu=1125mb
| nodes=2
| time=30, nodes=4, mem=90000mb, ntasks-per-node=40, (threads-per-core=2) 8 nodes are reserved for this queue. Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| multiple
| thin
| time=30, mem-per-cpu=1125mb
| nodes=2
| time=72:00:00, mem=90000mb, nodes=128, ntasks-per-node=40, (threads-per-core=2)
|- style="text-align:left;"
| dev_multiple_e
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
| nodes=2
| time=30, nodes=8, mem=122000, ntasks-per-node=28, (threads-per-core=2) 8 nodes are reserved for this queue Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| multiple_e
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
| nodes=2
| time=72:00:00, nodes=128, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="text-align:left;"
| dev_special (restricted access!)
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
|
| time=30, nodes=1, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="text-align:left;"
| special (restricted access!)
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
|
| time=72:00:00, nodes=1, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
| fat
| fat
| time=10, mem-per-cpu=18750mb
|
| time=72:00:00, nodes=1, ntasks-per-node=80, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
| gpu_4
| gpu4
| time=10, mem-per-gpu=94000mb, cpu-per-gpu=20
|
| time=48:00:00, mem=376000, nodes=14, ntasks-per-node=40, (threads-per-core=2)
|- style="vertical-align:top; text-align:left"
| gpu_8
| gpu8
| time=10, mem-per-cpu=94000mb, cpu-per-gpu=10
|
| time=48:00:00, mem=752000, nodes=10, ntasks-per-node=40, (threads-per-core=2)
|-
|}

Default resources of a queue class defines time, #tasks and memory if not explicitly given with sbatch command. Resource list acronyms ''--time'', ''--ntasks'', ''--nodes'', ''--mem'' and ''--mem-per-cpu'' are described [[ForHLR_Batch_Jobs_SLURM#sbatch_Command_Parameters|here]].

Access to the "special" and "dev_special" partitions on the bwUniCluster 2.0 is restricted to members of the institutions which participated in the procurement of the extension partition specifically for this purpose. Please contact the support team if your institution participated in the procurement and your account should be able to run jobs in this partition.
 
 
==== Queue class examples ====
To run your batch job on one of the thin nodes, please use:

<pre>
$ sbatch --partition=dev_multiple
or
$ sbatch -p dev_multiple
</pre>
 

==== Interactive Jobs ====
On bwUniCluster 2.0 you are only allowed to run short jobs (<< 1 hour) with little memory requirements (<< 8 GByte) on the logins nodes. If you want to run longer jobs and/or jobs with a request of more than 8 GByte of memory, you must allocate resources for so-called interactive jobs by usage of the command salloc on a login node. Considering a serial application running on a compute node that requires 5000 MByte of memory and limiting the interactive run to 2 hours the following command has to be executed:
<pre>
$ salloc -p single -n 1 -t 120 --mem=5000
</pre>
Then you will get one core on a compute node within the partition "single". After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system Slurm has granted you the requested resources on the compute system. You will be logged in automatically on the granted core! To run a serial program on the granted core you only have to type the name of the executable.
<pre>
$ ./<my_serial_program>
</pre>
Please be aware that your serial job must run less than 2 hours in this example, else the job will be killed during runtime by the system.
 
You can also start now a graphical X11-terminal connecting you to the dedicated resource that is available for 2 hours. You can start it by the command:
<pre>
$ xterm
</pre>
Note that, once the walltime limit has been reached the resources - i.e. the compute node - will automatically be revoked.
 
An interactive parallel application running on one compute node or on many compute nodes (e.g. here 5 nodes) with 40 cores each requires usually an amount of memory in GByte (e.g. 50 GByte) and a maximum time (e.g. 1 hour). E.g. 5 nodes can be allocated by the following command:
<pre>
$ salloc -p multiple -N 5 --ntasks-per-node=40 -t 01:00:00 --mem=50gb
</pre>
Now you can run parallel jobs on 200 cores requiring 50 GByte of memory per node. Please be aware that you will be logged in on core 0 of the first node. If you want to run MPI-programs, you can do it by simply typing mpirun <program_name>. Then your program will be run on 200 cores. A very simple example for starting a parallel job can be:
<pre>
$ mpirun <my_mpi_program>
</pre>
You can also start the debugger ddt by the commands:
<pre>
$ module add devel/ddt
$ ddt <my_mpi_program>
</pre>
The above commands will execute the parallel program <my_mpi_program> on all available cores. You can also start parallel programs on a subset of cores; an example for this can be:
<pre>
$ mpirun -n 50 <my_mpi_program>
</pre>
If you are using Intel MPI you must start <my_mpi_program> by the command mpiexec.hydra (instead of mpirun).
 
 

----
[[Category:bwUniCluster 2.0|Batch Jobs - bwUniCluster 2.0 Features]]

BwUniCluster2.0/Batch Queues

2020-07-03T13:17:52Z

H Haefner: /* sbatch -p queue */

This article contains information on the queues of the [[BwUniCluster_2.0_Slurm_common_Features|batch job system]] and on interactive jobs.

= Job Submission =
== sbatch Command ==

=== sbatch -p ''queue'' ===
Compute resources such as (wall-)time, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, '''you must add the correct queue class to your sbatch command '''. 'The specification of a queue is obligatory on BwUniCluster 2.0.
 
Details are:

{| width=750px class="wikitable"
! colspan="5" | bwUniCluster 2.0 sbatch -p ''queue''
|- style="text-align:left;"
! queue !! node !! default resources !! minimum resources !! maximum resources
|- style="text-align:left"
| dev_single
| thin
| time=10, mem-per-cpu=1125mb
|
| time=30, nodes=1, mem=90000mb, ntasks-per-node=40, (threads-per-core=2) 4 nodes are reserved for this queue. Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| single
| thin
| time=30, mem-per-cpu=1125mb
|
| time=72:00:00, nodes=1, mem=90000mb, ntasks-per-node=80
|- style="text-align:left;"
| dev_multiple
| thin
| time=10, mem-per-cpu=1125mb
| nodes=2
| time=30, nodes=4, mem=90000mb, ntasks-per-node=80 8 nodes are reserved for this queue. Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| multiple
| thin
| time=30, mem-per-cpu=1125mb
| nodes=2
| time=72:00:00, mem=90000mb, nodes=128, ntasks-per-node=80
|- style="text-align:left;"
| dev_multiple_e
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
| nodes=2
| time=30, nodes=8, mem=122000, ntasks-per-node=56 8 nodes are reserved for this queue Only for development, i.e. debugging or performance optimization ...
|- style="text-align:left;"
| multiple_e
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
| nodes=2
| time=72:00:00, nodes=128, mem=122000mb, ntasks-per-node=56
|- style="text-align:left;"
| dev_special (restricted access!)
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
|
| time=30, nodes=1, mem=122000mb, ntasks-per-node=56
|- style="text-align:left;"
| special (restricted access!)
| thin (Broadwell)
| time=10, mem-per-cpu=2178mb
|
| time=72:00:00, nodes=1, mem=122000mb, ntasks-per-node=56
|- style="vertical-align:top; text-align:left"
| fat
| fat
| time=10, mem-per-cpu=18750mb
|
| time=72:00:00, nodes=1, ntasks-per-node=160
|- style="vertical-align:top; text-align:left"
| gpu_4
| gpu4
| time=10, mem-per-gpu=94000mb, cpu-per-gpu=20
|
| time=48:00:00, mem=376000, nodes=14, ntasks-per-node=80
|- style="vertical-align:top; text-align:left"
| gpu_8
| gpu8
| time=10, mem-per-cpu=94000mb, cpu-per-gpu=10
|
| time=48:00:00, mem=752000, nodes=10, ntasks-per-node=80
|-
|}

Default resources of a queue class defines time, #tasks and memory if not explicitly given with sbatch command. Resource list acronyms ''--time'', ''--ntasks'', ''--nodes'', ''--mem'' and ''--mem-per-cpu'' are described [[ForHLR_Batch_Jobs_SLURM#sbatch_Command_Parameters|here]].

Access to the "special" and "dev_special" partitions on the bwUniCluster 2.0 is restricted to members of the institutions which participated in the procurement of the extension partition specifically for this purpose. Please contact the support team if your institution participated in the procurement and your account should be able to run jobs in this partition.
 
 
==== Queue class examples ====
To run your batch job on one of the thin nodes, please use:

<pre>
$ sbatch --partition=dev_multiple
or
$ sbatch -p dev_multiple
</pre>
 

==== Interactive Jobs ====
On bwUniCluster 2.0 you are only allowed to run short jobs (<< 1 hour) with little memory requirements (<< 8 GByte) on the logins nodes. If you want to run longer jobs and/or jobs with a request of more than 8 GByte of memory, you must allocate resources for so-called interactive jobs by usage of the command salloc on a login node. Considering a serial application running on a compute node that requires 5000 MByte of memory and limiting the interactive run to 2 hours the following command has to be executed:
<pre>
$ salloc -p single -n 1 -t 120 --mem=5000
</pre>
Then you will get one core on a compute node within the partition "single". After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system Slurm has granted you the requested resources on the compute system. You will be logged in automatically on the granted core! To run a serial program on the granted core you only have to type the name of the executable.
<pre>
$ ./<my_serial_program>
</pre>
Please be aware that your serial job must run less than 2 hours in this example, else the job will be killed during runtime by the system.
 
You can also start now a graphical X11-terminal connecting you to the dedicated resource that is available for 2 hours. You can start it by the command:
<pre>
$ xterm
</pre>
Note that, once the walltime limit has been reached the resources - i.e. the compute node - will automatically be revoked.
 
An interactive parallel application running on one compute node or on many compute nodes (e.g. here 5 nodes) with 40 cores each requires usually an amount of memory in GByte (e.g. 50 GByte) and a maximum time (e.g. 1 hour). E.g. 5 nodes can be allocated by the following command:
<pre>
$ salloc -p multiple -N 5 --ntasks-per-node=40 -t 01:00:00 --mem=50gb
</pre>
Now you can run parallel jobs on 200 cores requiring 50 GByte of memory per node. Please be aware that you will be logged in on core 0 of the first node. If you want to run MPI-programs, you can do it by simply typing mpirun <program_name>. Then your program will be run on 200 cores. A very simple example for starting a parallel job can be:
<pre>
$ mpirun <my_mpi_program>
</pre>
You can also start the debugger ddt by the commands:
<pre>
$ module add devel/ddt
$ ddt <my_mpi_program>
</pre>
The above commands will execute the parallel program <my_mpi_program> on all available cores. You can also start parallel programs on a subset of cores; an example for this can be:
<pre>
$ mpirun -n 50 <my_mpi_program>
</pre>
If you are using Intel MPI you must start <my_mpi_program> by the command mpiexec.hydra (instead of mpirun).
 
 

----
[[Category:bwUniCluster 2.0|Batch Jobs - bwUniCluster 2.0 Features]]

BwUniCluster2.0/Batch System Migration Guide

2020-04-23T14:39:45Z

H Haefner: /* General Overview */

While the former bwUniCluster 1 system used the combination Moab/SLURM for the batch system queue, bwUniCluster 2.0 uses only SLURM. This means that most job scripts and workflows which relied on Moab-specific pragmas and commands have to be changed.

=General Overview=

Job parameters can be passed to SLURM in the same ways which were possible with Moab.

* Instead of the #MOAB or #PBS pragmas, the pragma #SLURM has to be used within job files.

* Instead of the Moab commands, the corresponding SLURM commands have to be used.

A general mapping of Moab to SLURM commands can be found in the following table:

{| class="wikitable"
! Moab command !! SLURM command
|-
| msub || sbatch
|-
| msub -I || salloc
|-
| canceljob || scancel
|-
| showq || squeue
|-
| checkjob $JOBID || scontrol show job $JOBID
|}

In the next table you find the most MOAB job specification flags and environment variables. You have to replace all MOAB flags and environment variables of your batch scripts by their corresponding Slurm counterparts.

'''Commonly used MOAB job specification flags and their Slurm equivalents'''

{| width=750px class="wikitable"
! Option !! Moab (msub) !! Slurm (sbatch)
|-
| Script directive || #MSUB || #SBATCH
|-
| Job name || -N <name> || --job-name=<name> (-J <name>)
|-
| Account || -A <account> || --account=<account> (-A <account>)
|-
| Queue || -q <queue> || --partition=<partition> (-p <partition>)
|-
| Wall time limit || -l walltime=<hh:mm:ss> || --time=<hh:mm:ss> (-t <hh:mm:ss>)
|-
| Node count || -l nodes=<count> || --nodes=<count> (-N <count>)
|-
| Core count || -l procs=<count> || --ntasks=<count> (-n <count>)
|-
| Process count per node || -l ppn=<count> || --ntasks-per-node=<count>
|-
| Core count per process || || --cpus-per-task=<count>
|-
| Memory limit per node || -l mem=<limit> || --mem=<limit>
|-
| Memory limit per process || -l pmem=<limit> || --mem-per-cpu=<limit>
|-
| Job array || -t <array indices> || --array=<indices> (-a <indices>)
|-
| Node exclusive job || -l naccesspolicy=singlejob || --exclusive
|-
| Initial working directory || -d <directory> (default: $HOME) || --chdir=<directory> (-D <directory>) (default: submission directory)
|-
| Standard output file || -o <file path> || --output=<file> (-o <file>)
|-
| Standard error file || -e <file path> || --error=<file> (-e <file>)
|-
| Combine stdout/stderr to stdout || -j oe || --output=<combined stdout/stderr file>
|-
| Mail notification events || -m <event> || --mail-type=<events> (valid types include: NONE, BEGIN, END, FAIL, ALL)
|-
| Export environment to job || -V || --export=ALL (default)
|-
| Don't export environment to job || (default) || --export=NONE
|-
| Export environment variables to job || -v <var[=value][,var2=value2[, ...]]> || --export=<var[=value][,var2=value2[,...]]>
|}

'''Notes:'''
* Default initial job working directory is $HOME for MOAB. For Slurm the default working directory is where you submit your job from.
* By default MOAB does not export any environment variables to the job's runtime environment. With Slurm most of the login environment variables are exported to your job's runtime environment. This includes environment variables from software modules that were loaded at job submission time (and also $HOSTNAME variable).

'''Commonly used MOAB script environment variables and their Slurm equivalents

{| width=750px class="wikitable"
! Information !! MOAB !! Slurm
|-
| Job name || $MOAB_JOBNAME || $SLURM_JOB_NAME
|-
| Job ID || $MOAB_JOBID || $SLURM_JOB_ID
|-
| Submit directory || $MOAB_SUBMITDIR || $SLURM_SUBMIT_DIR
|-
| Number of nodes allocated || $MOAB_NODECOUNT || $SLURM_JOB_NUM_NODES (and: $SLURM_NNODES)
|-
| Node list || $MOAB_NODELIST || $SLURM_JOB_NODELIST
|-
| Number of processes || $MOAB_PROCCOUNT || $SLURM_NTASKS
|-
| Requested tasks per node || --- || $SLURM_NTASKS_PER_NODE
|-
| Requested CPUs per task || --- || $SLURM_CPUS_PER_TASK
|-
| Job array index || $MOAB_JOBARRAYINDEX || $SLURM_ARRAY_TASK_ID
|-
| Job array range || $MOAB_JOBARRAYRANGE || $SLURM_ARRAY_TASK_COUNT
|-
| Queue name || $MOAB_CLASS || $SLURM_JOB_PARTITION
|-
| QOS name || $MOAB_QOS || $SLURM_JOB_QOS
|-
| Number of processes per node || --- || $SLURM_TASKS_PER_NODE
|-
| Job user || $MOAB_USER || $SLURM_JOB_USER
|-
| Hostname || $MOAB_MACHINE || $SLURMD_NODENAME
|}

 

=Serial Programs=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-n 1''' or '''--ntasks=1''' (instead of '''-l nodes=1,ppn=1''').
* Use the option '''-m''' or '''--mem''' (instead of '''-l pmem'''). The default unit is MegaByte.
* If you want to use one node exclusively, you must enter the whole memory ('''-m 96327''' or '''--mem=96327''').
 
'''Example for a serial job'''
<pre>
$ sbatch -p single -t 60 -n 1 -m 96327 ./job.sh
</pre>
The script job.sh (containing the execution of a serial program) is started running 60 minutes exclusively on a batch node.
 
 
 

=Multithreaded Programs=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-N 1''' or '''--nodes=1''' and '''c ''x''''' or '''--cpus-per-task=''x''''' (instead of '''-l nodes=1,ppn=''x'' '''). '''''x''''' can be a number between 1 and 40 (because of 40 cores within one node); it can also be a number between 41 and 80 (because of active hyperthreading).
* Use the option '''-m''' or '''--mem''' (instead of '''-l pmem'''). The default unit is MegaByte.
* Use the option '''--export''' to set the needed environment variable OMP_NUM_THREADS for the batch job. Adding '''ALL''' means to pass all interactively set environment variables to the batch job.
* If you want to use one node exclusively, you must either enter the whole memory (''-m 96327'' or ''--mem=96327'') or set the number of threads greater than 39.
 
'''Example for a multithreaded job'''
<pre>
$ sbatch -p single -t 1:00:00 -N 1 -c 20 -m 50gb --export=ALL,OMP_NUM_THREADS=20 ./job_threaded.sh
</pre>
The script '''job_threaded.sh''' (containing a multithreaded program) is started running 1 hour in shared mode on 20 cores requesting 50GB on one batch node.
 
 
 

=MPI Parallel Programs within one node=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-n ''x''''' or '''--ntasks=''x''''' (instead of '''-l nodes=1,ppn=''x'' '''). '''''x''''' can be a number between 1 and 40 (because of 40 cores within one node); you should'nt utilize hyperthreading.
* Use the option '''-m''' or '''--mem''' (instead of '''-l pmem'''). The default unit is MegaByte.
* If you want to use one node exclusively, you must either enter the whole memory (''-m 96327'' or ''--mem=96327'') or set the number of MPI tasks greater than 39.
* Don't forget to load the appropriate MPI-module in your job script.
* If you are using OpenMPI, the options '''--bind-to core --map-by core|socket|node''' of the command mpirun should be used.
 
'''Example for a MPI job'''
<pre>
$ sbatch -p single -t 600 -n 10 -m 40000 ./job_mpi.sh
</pre>
The script '''job_mpi.sh''' (containing a MPI program after loading the appropriate MPI module) is started running 10 hours in shared mode on 10 cores requesting 40000 MB on one batch node.
 
 
 

=MPI Parallel Programs on many nodes=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-N ''y''''' or '''--nodes=''y''''' and '''--ntasks-per-node=''x''''' (instead of '''-l nodes=''y'',ppn=''x'''''). '''''x''''' can be a number between 1 and 40 (28 for Broadwell nodes) (because of 40 (28) cores within one node); you should'nt utilize hyperthreading.
* You should'nt use the option '''-m''' or '''--mem''' because the nodes are used exclusively.
* You always use the nodes exclusively.
* Don't forget to load the appropriate MPI-module in your job script.
* If you are using OpenMPI, the options '''--bind-to core --map-by core|socket|node''' ofthe command mpirun should be used.
 
'''Example for a MPI job'''
<pre>
$ sbatch -p multiple -t 48:00:00 -N 10 --ntasks-per-node=40 ./job_mpi.sh
</pre>
The script '''job_mpi.sh''' (containing a MPI program after loading the appropriate MPI module) is started running 2 days on 400 cores on ten batch nodes.
 
 
 

=Multithreaded + MPI Parallel Programs on many nodes=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-N ''y''''' or '''--nodes=''y''''' and '''--ntasks-per-node=''x''''' and '''-c ''z''''' or '''--cpus-per-task=''z''''' (instead of '''-l nodes=''y'',ppn=''x+z'''''). '''''x''''' usually should be 1 or 2 and '''''x+z''''' usually 40 (28 on Broadwell nodes); you can utilize hyperthreading if you want.
* You should'nt use the option '''-m''' or '''--mem''' because the nodes are used exclusively.
* You always use the nodes exclusively.
* Don't forget to load the appropriate MPI-module in your job script.
* If you are using OpenMPI, the options '''--bind-to core --map-by socket|node:PE=''z''''' of the command mpirun must be used.
 
'''Example for a MPI job'''
<pre>
$ sbatch -p multiple -t 2-12 -N 10 --ntasks-per-node=2 -c 20 ./job_threaded_mpi.sh
</pre>
The script '''job_threaded_mpi.sh''' (containing a multithreaded MPI program after loading the appropriate MPI module) is started running 2.5 days on 400 cores with 20 MPI tasks and 20 threads per task on ten batch nodes. Here the options '''--bind-to core --map-by socket:PE=10''' of the command mpirun must be used.

BwUniCluster2.0/Batch System Migration Guide

2020-04-23T14:37:34Z

H Haefner: /* General Overview */

While the former bwUniCluster 1 system used the combination Moab/SLURM for the batch system queue, bwUniCluster 2.0 uses only SLURM. This means that most job scripts and workflows which relied on Moab-specific pragmas and commands have to be changed.

=General Overview=

Job parameters can be passed to SLURM in the same ways which were possible with Moab.

* Instead of the #MOAB or #PBS pragmas, the pragma #SLURM has to be used within job files.

* Instead of the Moab commands, the corresponding SLURM commands have to be used.

A general mapping of Moab to SLURM commands can be found in the following table:

{| class="wikitable"
! Moab command !! SLURM command
|-
| msub || sbatch
|-
| msub -I || salloc
|-
| canceljob || scancel
|-
| showq || squeue
|-
| checkjob $JOBID || scontrol show job $JOBID
|}

In the next table you find the most MOAB job specification flags and environment variables. You have to replace all MOAB flags and environment variables of your batch scripts by their corresponding Slurm counterparts.

'''Commonly used MOAB job specification flags and their Slurm equivalents'''

{| width=750px class="wikitable"
! Option !! Moab (msub) !! Slurm (sbatch)
|-
| Script directive || #MSUB || #SBATCH
|-
| Job name || -N <name> || --job-name=<name> (-J <name>)
|-
| Account || -A <account> || --account=<account> (-A <account>)
|-
| Queue || -q <queue> || --partition=<partition> (-p <partition>)
|-
| Wall time limit || -l walltime=<hh:mm:ss> || --time=<hh:mm:ss> (-t <hh:mm:ss>)
|-
| Node count || -l nodes=<count> || --nodes=<count> (-N <count>)
|-
| Core count || -l procs=<count> || --ntasks=<count> (-n <count>)
|-
| Process count per node || -l ppn=<count> || --ntasks-per-node=<count>
|-
| Core count per process || || --cpus-per-task=<count>
|-
| Memory limit per node || -l mem=<limit> || --mem=<limit>
|-
| Memory limit per process || -l pmem=<limit> || --mem-per-cpu=<limit>
|-
| Job array || -t <array indices> || --array=<indices> (-a <indices>)
|-
| Node exclusive job || -l naccesspolicy=singlejob || --exclusive
|-
| Initial working directory || -d <directory> (default: $HOME) || --chdir=<directory> (-D <directory>) (default: submission directory)
|-
| Standard output file || -o <file path> || --output=<file> (-o <file>)
|-
| Standard error file || -e <file path> || --error=<file> (-e <file>)
|-
| Combine stdout/stderr to stdout || -j oe || --output=<combined stdout/stderr file>
|-
| Mail notification events || -m <event> || --mail-type=<events> (valid types include: NONE, BEGIN, END, FAIL, ALL)
|-
| Export environment to job || -V || --export=ALL (default)
|-
| Don't export environment to job || (default) || --export=NONE
|-
| Export environment variables to job || -v <var[=value][,var2=value2[, ...]]> || --export=<var[=value][,var2=value2[,...]]>
|}

'''Notes:'''
* Default initial job working directory is $HOME for MOAB. For Slurm the default working directory is where you submit your job from.
* By default MOAB does not export any environment variables to the job's runtime environment. With Slurm most of the login environment variables are exported to your job's runtime environment. This includes environment variables from software modules that were loaded at job submission time (and also $HOSTNAME variable).

'''Commonly used MOAB script environment variables and their Slurm equivalents

{| width=750px class="wikitable"
! Information !! MOAB !! Slurm
|-
| Job name || $MOAB_JOBNAME || $SLURM_JOB_NAME
|-
| Job ID || $MOAB_JOBID || $SLURM_JOB_ID
|-
| Submit directory || $MOAB_SUBMITDIR || $SLURM_SUBMIT_DIR
|-
| Number of nodes allocated || $MOAB_NODECOUNT || $SLURM_JOB_NUM_NODES (and: $SLURM_NNODES)
|-
| Node list || $MOAB_NODELIST || $SLURM_JOB_NODELIST
|-
| Number of processes || $MOAB_PROCCOUNT || $SLURM_NTASKS
|-
| Requested tasks per node || --- || $SLURM_NTASKS_PER_NODE
|-
| Requested CPUs per task || --- || $SLURM_CPUS_PER_TASK
|-
| Job array index || $MOAB_JOBARRAYINDEX || $SLURM_ARRAY_TASK_ID
|-
| Job array range || $MOAB_JOBARRAYRANGE || $SLURM_ARRAY_TASK_COUNT
|-
| Queue name || $MOAB_CLASS || $SLURM_JOB_PARTITION
|-
| QOS name || $MOAB_QOS || $SLURM_JOB_QOS
|-
| Number of processes per node | --- || $SLURM_TASKS_PER_NODE
|-
| Job user || $MOAB_USER || $SLURM_JOB_USER
|-
| Hostname || $MOAB_MACHINE || $SLURMD_NODENAME
|}

 

=Serial Programs=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-n 1''' or '''--ntasks=1''' (instead of '''-l nodes=1,ppn=1''').
* Use the option '''-m''' or '''--mem''' (instead of '''-l pmem'''). The default unit is MegaByte.
* If you want to use one node exclusively, you must enter the whole memory ('''-m 96327''' or '''--mem=96327''').
 
'''Example for a serial job'''
<pre>
$ sbatch -p single -t 60 -n 1 -m 96327 ./job.sh
</pre>
The script job.sh (containing the execution of a serial program) is started running 60 minutes exclusively on a batch node.
 
 
 

=Multithreaded Programs=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-N 1''' or '''--nodes=1''' and '''c ''x''''' or '''--cpus-per-task=''x''''' (instead of '''-l nodes=1,ppn=''x'' '''). '''''x''''' can be a number between 1 and 40 (because of 40 cores within one node); it can also be a number between 41 and 80 (because of active hyperthreading).
* Use the option '''-m''' or '''--mem''' (instead of '''-l pmem'''). The default unit is MegaByte.
* Use the option '''--export''' to set the needed environment variable OMP_NUM_THREADS for the batch job. Adding '''ALL''' means to pass all interactively set environment variables to the batch job.
* If you want to use one node exclusively, you must either enter the whole memory (''-m 96327'' or ''--mem=96327'') or set the number of threads greater than 39.
 
'''Example for a multithreaded job'''
<pre>
$ sbatch -p single -t 1:00:00 -N 1 -c 20 -m 50gb --export=ALL,OMP_NUM_THREADS=20 ./job_threaded.sh
</pre>
The script '''job_threaded.sh''' (containing a multithreaded program) is started running 1 hour in shared mode on 20 cores requesting 50GB on one batch node.
 
 
 

=MPI Parallel Programs within one node=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-n ''x''''' or '''--ntasks=''x''''' (instead of '''-l nodes=1,ppn=''x'' '''). '''''x''''' can be a number between 1 and 40 (because of 40 cores within one node); you should'nt utilize hyperthreading.
* Use the option '''-m''' or '''--mem''' (instead of '''-l pmem'''). The default unit is MegaByte.
* If you want to use one node exclusively, you must either enter the whole memory (''-m 96327'' or ''--mem=96327'') or set the number of MPI tasks greater than 39.
* Don't forget to load the appropriate MPI-module in your job script.
* If you are using OpenMPI, the options '''--bind-to core --map-by core|socket|node''' of the command mpirun should be used.
 
'''Example for a MPI job'''
<pre>
$ sbatch -p single -t 600 -n 10 -m 40000 ./job_mpi.sh
</pre>
The script '''job_mpi.sh''' (containing a MPI program after loading the appropriate MPI module) is started running 10 hours in shared mode on 10 cores requesting 40000 MB on one batch node.
 
 
 

=MPI Parallel Programs on many nodes=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-N ''y''''' or '''--nodes=''y''''' and '''--ntasks-per-node=''x''''' (instead of '''-l nodes=''y'',ppn=''x'''''). '''''x''''' can be a number between 1 and 40 (28 for Broadwell nodes) (because of 40 (28) cores within one node); you should'nt utilize hyperthreading.
* You should'nt use the option '''-m''' or '''--mem''' because the nodes are used exclusively.
* You always use the nodes exclusively.
* Don't forget to load the appropriate MPI-module in your job script.
* If you are using OpenMPI, the options '''--bind-to core --map-by core|socket|node''' ofthe command mpirun should be used.
 
'''Example for a MPI job'''
<pre>
$ sbatch -p multiple -t 48:00:00 -N 10 --ntasks-per-node=40 ./job_mpi.sh
</pre>
The script '''job_mpi.sh''' (containing a MPI program after loading the appropriate MPI module) is started running 2 days on 400 cores on ten batch nodes.
 
 
 

=Multithreaded + MPI Parallel Programs on many nodes=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-N ''y''''' or '''--nodes=''y''''' and '''--ntasks-per-node=''x''''' and '''-c ''z''''' or '''--cpus-per-task=''z''''' (instead of '''-l nodes=''y'',ppn=''x+z'''''). '''''x''''' usually should be 1 or 2 and '''''x+z''''' usually 40 (28 on Broadwell nodes); you can utilize hyperthreading if you want.
* You should'nt use the option '''-m''' or '''--mem''' because the nodes are used exclusively.
* You always use the nodes exclusively.
* Don't forget to load the appropriate MPI-module in your job script.
* If you are using OpenMPI, the options '''--bind-to core --map-by socket|node:PE=''z''''' of the command mpirun must be used.
 
'''Example for a MPI job'''
<pre>
$ sbatch -p multiple -t 2-12 -N 10 --ntasks-per-node=2 -c 20 ./job_threaded_mpi.sh
</pre>
The script '''job_threaded_mpi.sh''' (containing a multithreaded MPI program after loading the appropriate MPI module) is started running 2.5 days on 400 cores with 20 MPI tasks and 20 threads per task on ten batch nodes. Here the options '''--bind-to core --map-by socket:PE=10''' of the command mpirun must be used.

BwUniCluster2.0/Batch System Migration Guide

2020-04-23T14:28:18Z

H Haefner: /* General Overview */

While the former bwUniCluster 1 system used the combination Moab/SLURM for the batch system queue, bwUniCluster 2.0 uses only SLURM. This means that most job scripts and workflows which relied on Moab-specific pragmas and commands have to be changed.

=General Overview=

Job parameters can be passed to SLURM in the same ways which were possible with Moab.

* Instead of the #MOAB or #PBS pragmas, the pragma #SLURM has to be used within job files.

* Instead of the Moab commands, the corresponding SLURM commands have to be used.

A general mapping of Moab to SLURM commands can be found in the following table:

{| class="wikitable"
! Moab command !! SLURM command
|-
| msub || sbatch
|-
| msub -I || salloc
|-
| canceljob || scancel
|-
| showq || squeue
|-
| checkjob $JOBID || scontrol show job $JOBID
|}

In the next table you find the most MOAB job specification flags and environment variables. You have to replace all MOAB flags and environment variables of your batch scripts by their corresponding Slurm counterparts.

'''Commonly used MOAB job specification flags and their Slurm equivalents'''

{| width=750px class="wikitable"
! Option !! Moab (msub) !! Slurm (sbatch)
|-
| Script directive || #MSUB || #SBATCH
|-
| Job name || -N <name> || --job-name=<name> (-J <name>)
|-
| Account || -A <account> || --account=<account> (-A <account>)
|-
| Queue || -q <queue> || --partition=<partition> (-p <partition>)
|-
| Wall time limit || -l walltime=<hh:mm:ss> || --time=<hh:mm:ss> (-t <hh:mm:ss>)
|-
| Node count || -l nodes=<count> || --nodes=<count> (-N <count>)
|-
| Core count || -l procs=<count> || --ntasks=<count> (-n <count>)
|-
| Process count per node || -l ppn=<count> || --ntasks-per-node=<count>
|-
| Core count per process || || --cpus-per-task=<count>
|-
| Memory limit per node || -l mem=<limit> || --mem=<limit>
|-
| Memory limit per process || -l pmem=<limit> || --mem-per-cpu=<limit>
|-
| Job array || -t <array indices> || --array=<indices> (-a <indices>)
|-
| Node exclusive job || -l naccesspolicy=singlejob || --exclusive
|-
| Initial working directory || -d <directory> (default: $HOME) || --chdir=<directory> (-D <directory>) (default: submission directory)
|-
| Standard output file || -o <file path> || --output=<file> (-o <file>)
|-
| Standard error file || -e <file path> || --error=<file> (-e <file>)
|-
| Combine stdout/stderr to stdout || -j oe || --output=<combined stdout/stderr file>
|-
| Mail notification events || -m <event> || --mail-type=<events> (valid types include: NONE, BEGIN, END, FAIL, ALL)
|-
| Export environment to job || -V || --export=ALL (default)
|-
| Don't export environment to job || (default) || --export=NONE
|-
| Export environment variables to job || -v <var[=value][,var2=value2[, ...]]> || --export=<var[=value][,var2=value2[,...]]>
|}

'''Notes:'''
* Default initial job working directory is $HOME for Moab. For Slurm the default working directory is where you submit your job from.
* By default Moab does not export any environment variables to the job's runtime environment. With Slurm most of the login environment variables are exported to your job's runtime environment. This includes environment variables from software modules that were loaded at job submission time (and also $HOSTNAME variable).

'''Commonly used Moab/Torque script environment variables and their Slurm equivalents

{| width=750px class="wikitable"
! Information !! Moab !! Torque !! Slurm
|-
| Job name || $MOAB_JOBNAME || $PBS_JOBNAME || $SLURM_JOB_NAME
|-
| Job ID || $MOAB_JOBID || $PBS_JOBID || $SLURM_JOB_ID
|-
| Submit directory || $MOAB_SUBMITDIR || $PBS_O_WORKDIR || $SLURM_SUBMIT_DIR
|-
| Number of nodes allocated || $MOAB_NODECOUNT || $PBS_NUM_NODES || $SLURM_JOB_NUM_NODES (and: $SLURM_NNODES)
|-
| Node list || $MOAB_NODELIST || cat $PBS_NODEFILE || $SLURM_JOB_NODELIST
|-
| Number of processes || $MOAB_PROCCOUNT || $PBS_TASKNUM || $SLURM_NTASKS
|-
| Requested tasks per node || --- || $PBS_NUM_PPN || $SLURM_NTASKS_PER_NODE
|-
| Requested CPUs per task || --- || --- || $SLURM_CPUS_PER_TASK
|-
| Job array index || $MOAB_JOBARRAYINDEX || $PBS_ARRAY_INDEX || $SLURM_ARRAY_TASK_ID
|-
| Job array range || $MOAB_JOBARRAYRANGE || - || $SLURM_ARRAY_TASK_COUNT
|-
| Queue name || $MOAB_CLASS || $PBS_QUEUE || $SLURM_JOB_PARTITION
|-
| QOS name || $MOAB_QOS || --- || $SLURM_JOB_QOS
|-
| Number of processes per node | --- || $PBS_NUM_PPN || $SLURM_TASKS_PER_NODE
|-
| Job user || $MOAB_USER || $PBS_O_LOGNAME || $SLURM_JOB_USER
|-
| Hostname || $MOAB_MACHINE || $PBS_O_HOST || $SLURMD_NODENAME
|}

The following chapters contain short examples for the most common job types. For a full overview of SLURM see the [[BwUniCluster_2.0_Slurm_common_Features|full article]].

=Serial Programs=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-n 1''' or '''--ntasks=1''' (instead of '''-l nodes=1,ppn=1''').
* Use the option '''-m''' or '''--mem''' (instead of '''-l pmem'''). The default unit is MegaByte.
* If you want to use one node exclusively, you must enter the whole memory ('''-m 96327''' or '''--mem=96327''').
 
'''Example for a serial job'''
<pre>
$ sbatch -p single -t 60 -n 1 -m 96327 ./job.sh
</pre>
The script job.sh (containing the execution of a serial program) is started running 60 minutes exclusively on a batch node.
 
 
 

=Multithreaded Programs=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-N 1''' or '''--nodes=1''' and '''c ''x''''' or '''--cpus-per-task=''x''''' (instead of '''-l nodes=1,ppn=''x'' '''). '''''x''''' can be a number between 1 and 40 (because of 40 cores within one node); it can also be a number between 41 and 80 (because of active hyperthreading).
* Use the option '''-m''' or '''--mem''' (instead of '''-l pmem'''). The default unit is MegaByte.
* Use the option '''--export''' to set the needed environment variable OMP_NUM_THREADS for the batch job. Adding '''ALL''' means to pass all interactively set environment variables to the batch job.
* If you want to use one node exclusively, you must either enter the whole memory (''-m 96327'' or ''--mem=96327'') or set the number of threads greater than 39.
 
'''Example for a multithreaded job'''
<pre>
$ sbatch -p single -t 1:00:00 -N 1 -c 20 -m 50gb --export=ALL,OMP_NUM_THREADS=20 ./job_threaded.sh
</pre>
The script '''job_threaded.sh''' (containing a multithreaded program) is started running 1 hour in shared mode on 20 cores requesting 50GB on one batch node.
 
 
 

=MPI Parallel Programs within one node=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-n ''x''''' or '''--ntasks=''x''''' (instead of '''-l nodes=1,ppn=''x'' '''). '''''x''''' can be a number between 1 and 40 (because of 40 cores within one node); you should'nt utilize hyperthreading.
* Use the option '''-m''' or '''--mem''' (instead of '''-l pmem'''). The default unit is MegaByte.
* If you want to use one node exclusively, you must either enter the whole memory (''-m 96327'' or ''--mem=96327'') or set the number of MPI tasks greater than 39.
* Don't forget to load the appropriate MPI-module in your job script.
* If you are using OpenMPI, the options '''--bind-to core --map-by core|socket|node''' of the command mpirun should be used.
 
'''Example for a MPI job'''
<pre>
$ sbatch -p single -t 600 -n 10 -m 40000 ./job_mpi.sh
</pre>
The script '''job_mpi.sh''' (containing a MPI program after loading the appropriate MPI module) is started running 10 hours in shared mode on 10 cores requesting 40000 MB on one batch node.
 
 
 

=MPI Parallel Programs on many nodes=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-N ''y''''' or '''--nodes=''y''''' and '''--ntasks-per-node=''x''''' (instead of '''-l nodes=''y'',ppn=''x'''''). '''''x''''' can be a number between 1 and 40 (28 for Broadwell nodes) (because of 40 (28) cores within one node); you should'nt utilize hyperthreading.
* You should'nt use the option '''-m''' or '''--mem''' because the nodes are used exclusively.
* You always use the nodes exclusively.
* Don't forget to load the appropriate MPI-module in your job script.
* If you are using OpenMPI, the options '''--bind-to core --map-by core|socket|node''' ofthe command mpirun should be used.
 
'''Example for a MPI job'''
<pre>
$ sbatch -p multiple -t 48:00:00 -N 10 --ntasks-per-node=40 ./job_mpi.sh
</pre>
The script '''job_mpi.sh''' (containing a MPI program after loading the appropriate MPI module) is started running 2 days on 400 cores on ten batch nodes.
 
 
 

=Multithreaded + MPI Parallel Programs on many nodes=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-N ''y''''' or '''--nodes=''y''''' and '''--ntasks-per-node=''x''''' and '''-c ''z''''' or '''--cpus-per-task=''z''''' (instead of '''-l nodes=''y'',ppn=''x+z'''''). '''''x''''' usually should be 1 or 2 and '''''x+z''''' usually 40 (28 on Broadwell nodes); you can utilize hyperthreading if you want.
* You should'nt use the option '''-m''' or '''--mem''' because the nodes are used exclusively.
* You always use the nodes exclusively.
* Don't forget to load the appropriate MPI-module in your job script.
* If you are using OpenMPI, the options '''--bind-to core --map-by socket|node:PE=''z''''' of the command mpirun must be used.
 
'''Example for a MPI job'''
<pre>
$ sbatch -p multiple -t 2-12 -N 10 --ntasks-per-node=2 -c 20 ./job_threaded_mpi.sh
</pre>
The script '''job_threaded_mpi.sh''' (containing a multithreaded MPI program after loading the appropriate MPI module) is started running 2.5 days on 400 cores with 20 MPI tasks and 20 threads per task on ten batch nodes. Here the options '''--bind-to core --map-by socket:PE=10''' of the command mpirun must be used.

BwUniCluster2.0/Batch System Migration Guide

2020-04-23T14:27:41Z

H Haefner: /* How to convert Moab batch job scripts to Slurm? */

While the former bwUniCluster 1 system used the combination Moab/SLURM for the batch system queue, bwUniCluster 2.0 uses only SLURM. This means that most job scripts and workflows which relied on Moab-specific pragmas and commands have to be changed.

=General Overview=

Job parameters can be passed to SLURM in the same ways which were possible with Moab.

* Instead of the #MOAB or #PBS pragmas, the pragma #SLURM has to be used within job files.

* Instead of the Moab commands, the corresponding SLURM commands have to be used.

A general mapping of Moab to SLURM commands can be found in the following table:

{| class="wikitable"
! Moab command !! SLURM command
|-
| msub || sbatch
|-
| msub -I || salloc
|-
| canceljob || scancel
|-
| showq || squeue
|-
| checkjob $JOBID || scontrol show job $JOBID
|}

In the next table you find the most MOAB job specification flags and environment variables. You have to replace all MOAB flags and environment variables of your batch scripts by their corresponding Slurm counterparts.

'''Commonly used MOAB job specification flags and their Slurm equivalents'''

{| width=750px class="wikitable"
! Option !! Moab (msub) !! Slurm (sbatch)
|-
| Script directive || #MSUB || #SBATCH
|-
| Job name || -N <name> || --job-name=<name> (-J <name>)
|-
| Account || -A <account> || --account=<account> (-A <account>)
|-
| Queue || -q <queue> || --partition=<partition> (-p <partition>)
|-
| Wall time limit || -l walltime=<hh:mm:ss> || --time=<hh:mm:ss> (-t <hh:mm:ss>)
|-
| Node count || -l nodes=<count> || --nodes=<count> (-N <count>)
|-
| Core count || -l procs=<count> || --ntasks=<count> (-n <count>)
|-
| Process count per node || -l ppn=<count> || --ntasks-per-node=<count>
|-
| Core count per process || || --cpus-per-task=<count>
|-
| Memory limit per node || -l mem=<limit> || --mem=<limit>
|-
| Memory limit per process || -l pmem=<limit> || --mem-per-cpu=<limit>
|-
| Job array || -t <array indices> || --array=<indices> (-a <indices>)
|-
| Node exclusive job || -l naccesspolicy=singlejob || --exclusive
|-
| Initial working directory || -d <directory> (default: $HOME) || --chdir=<directory> (-D <directory>) (default: submission directory)
|-
| Standard output file || -o <file path> || --output=<file> (-o <file>)
|-
| Standard error file || -e <file path> || --error=<file> (-e <file>)
|-
| Combine stdout/stderr to stdout || -j oe || --output=<combined stdout/stderr file>
|-
| Mail notification events || -m <event> || --mail-type=<events> (valid types include: NONE, BEGIN, END, FAIL, ALL)
|-
| Export environment to job || -V || --export=ALL (default)
|-
| Don't export environment to job || (default) || --export=NONE
|-
| Export environment variables to job || -v <var[=value][,var2=value2[, ...]]> || --export=<var[=value][,var2=value2[,...]]>
|}

'''Notes:'''
* Default initial job working directory is $HOME for Moab. For Slurm the default working directory is where you submit your job from.
* By default Moab does not export any environment variables to the job's runtime environment. With Slurm most of the login environment variables are exported to your job's runtime environment. This includes environment variables from software modules that were loaded at job submission time (and also $HOSTNAME variable).

'''Commonly used Moab/Torque script environment variables and their Slurm equivalents

{| width=750px class="wikitable"
! Information !! Moab !! Torque !! Slurm
|-
| Job name || $MOAB_JOBNAME || $PBS_JOBNAME || $SLURM_JOB_NAME
|-
| Job ID || $MOAB_JOBID || $PBS_JOBID || $SLURM_JOB_ID
|-
| Submit directory || $MOAB_SUBMITDIR || $PBS_O_WORKDIR || $SLURM_SUBMIT_DIR
|-
| Number of nodes allocated || $MOAB_NODECOUNT || $PBS_NUM_NODES || $SLURM_JOB_NUM_NODES (and: $SLURM_NNODES)
|-
| Node list || $MOAB_NODELIST || cat $PBS_NODEFILE || $SLURM_JOB_NODELIST
|-
| Number of processes || $MOAB_PROCCOUNT || $PBS_TASKNUM || $SLURM_NTASKS
|-
| Requested tasks per node || --- || $PBS_NUM_PPN || $SLURM_NTASKS_PER_NODE
|-
| Requested CPUs per task || --- || --- || $SLURM_CPUS_PER_TASK
|-
| Job array index || $MOAB_JOBARRAYINDEX || $PBS_ARRAY_INDEX || $SLURM_ARRAY_TASK_ID
|-
| Job array range || $MOAB_JOBARRAYRANGE || - || $SLURM_ARRAY_TASK_COUNT
|-
| Queue name || $MOAB_CLASS || $PBS_QUEUE || $SLURM_JOB_PARTITION
|-
| QOS name || $MOAB_QOS || --- || $SLURM_JOB_QOS
|-
| Number of processes per node | --- || $PBS_NUM_PPN || $SLURM_TASKS_PER_NODE
|-
| Job user || $MOAB_USER || $PBS_O_LOGNAME || $SLURM_JOB_USER
|-
| Hostname || $MOAB_MACHINE || $PBS_O_HOST || $SLURMD_NODENAME
|}

The following chapters contain short examples for the most common job types. For a full overview of SLURM see the [[BwUniCluster_2.0_Slurm_common_Features|full article]].

=Serial Programs=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-n 1''' or '''--ntasks=1''' (instead of '''-l nodes=1,ppn=1''').
* Use the option '''-m''' or '''--mem''' (instead of '''-l pmem'''). The default unit is MegaByte.
* If you want to use one node exclusively, you must enter the whole memory ('''-m 96327''' or '''--mem=96327''').
 
'''Example for a serial job'''
<pre>
$ sbatch -p single -t 60 -n 1 -m 96327 ./job.sh
</pre>
The script job.sh (containing the execution of a serial program) is started running 60 minutes exclusively on a batch node.
 
 
 

=Multithreaded Programs=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-N 1''' or '''--nodes=1''' and '''c ''x''''' or '''--cpus-per-task=''x''''' (instead of '''-l nodes=1,ppn=''x'' '''). '''''x''''' can be a number between 1 and 40 (because of 40 cores within one node); it can also be a number between 41 and 80 (because of active hyperthreading).
* Use the option '''-m''' or '''--mem''' (instead of '''-l pmem'''). The default unit is MegaByte.
* Use the option '''--export''' to set the needed environment variable OMP_NUM_THREADS for the batch job. Adding '''ALL''' means to pass all interactively set environment variables to the batch job.
* If you want to use one node exclusively, you must either enter the whole memory (''-m 96327'' or ''--mem=96327'') or set the number of threads greater than 39.
 
'''Example for a multithreaded job'''
<pre>
$ sbatch -p single -t 1:00:00 -N 1 -c 20 -m 50gb --export=ALL,OMP_NUM_THREADS=20 ./job_threaded.sh
</pre>
The script '''job_threaded.sh''' (containing a multithreaded program) is started running 1 hour in shared mode on 20 cores requesting 50GB on one batch node.
 
 
 

=MPI Parallel Programs within one node=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-n ''x''''' or '''--ntasks=''x''''' (instead of '''-l nodes=1,ppn=''x'' '''). '''''x''''' can be a number between 1 and 40 (because of 40 cores within one node); you should'nt utilize hyperthreading.
* Use the option '''-m''' or '''--mem''' (instead of '''-l pmem'''). The default unit is MegaByte.
* If you want to use one node exclusively, you must either enter the whole memory (''-m 96327'' or ''--mem=96327'') or set the number of MPI tasks greater than 39.
* Don't forget to load the appropriate MPI-module in your job script.
* If you are using OpenMPI, the options '''--bind-to core --map-by core|socket|node''' of the command mpirun should be used.
 
'''Example for a MPI job'''
<pre>
$ sbatch -p single -t 600 -n 10 -m 40000 ./job_mpi.sh
</pre>
The script '''job_mpi.sh''' (containing a MPI program after loading the appropriate MPI module) is started running 10 hours in shared mode on 10 cores requesting 40000 MB on one batch node.
 
 
 

=MPI Parallel Programs on many nodes=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-N ''y''''' or '''--nodes=''y''''' and '''--ntasks-per-node=''x''''' (instead of '''-l nodes=''y'',ppn=''x'''''). '''''x''''' can be a number between 1 and 40 (28 for Broadwell nodes) (because of 40 (28) cores within one node); you should'nt utilize hyperthreading.
* You should'nt use the option '''-m''' or '''--mem''' because the nodes are used exclusively.
* You always use the nodes exclusively.
* Don't forget to load the appropriate MPI-module in your job script.
* If you are using OpenMPI, the options '''--bind-to core --map-by core|socket|node''' ofthe command mpirun should be used.
 
'''Example for a MPI job'''
<pre>
$ sbatch -p multiple -t 48:00:00 -N 10 --ntasks-per-node=40 ./job_mpi.sh
</pre>
The script '''job_mpi.sh''' (containing a MPI program after loading the appropriate MPI module) is started running 2 days on 400 cores on ten batch nodes.
 
 
 

=Multithreaded + MPI Parallel Programs on many nodes=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-N ''y''''' or '''--nodes=''y''''' and '''--ntasks-per-node=''x''''' and '''-c ''z''''' or '''--cpus-per-task=''z''''' (instead of '''-l nodes=''y'',ppn=''x+z'''''). '''''x''''' usually should be 1 or 2 and '''''x+z''''' usually 40 (28 on Broadwell nodes); you can utilize hyperthreading if you want.
* You should'nt use the option '''-m''' or '''--mem''' because the nodes are used exclusively.
* You always use the nodes exclusively.
* Don't forget to load the appropriate MPI-module in your job script.
* If you are using OpenMPI, the options '''--bind-to core --map-by socket|node:PE=''z''''' of the command mpirun must be used.
 
'''Example for a MPI job'''
<pre>
$ sbatch -p multiple -t 2-12 -N 10 --ntasks-per-node=2 -c 20 ./job_threaded_mpi.sh
</pre>
The script '''job_threaded_mpi.sh''' (containing a multithreaded MPI program after loading the appropriate MPI module) is started running 2.5 days on 400 cores with 20 MPI tasks and 20 threads per task on ten batch nodes. Here the options '''--bind-to core --map-by socket:PE=10''' of the command mpirun must be used.

BwUniCluster2.0/Batch System Migration Guide

2020-04-23T14:21:54Z

H Haefner: /* General Overview */

While the former bwUniCluster 1 system used the combination Moab/SLURM for the batch system queue, bwUniCluster 2.0 uses only SLURM. This means that most job scripts and workflows which relied on Moab-specific pragmas and commands have to be changed.

=General Overview=

Job parameters can be passed to SLURM in the same ways which were possible with Moab.

* Instead of the #MOAB or #PBS pragmas, the pragma #SLURM has to be used within job files.

* Instead of the Moab commands, the corresponding SLURM commands have to be used.

A general mapping of Moab to SLURM commands can be found in the following table:

{| class="wikitable"
! Moab command !! SLURM command
|-
| msub || sbatch
|-
| msub -I || salloc
|-
| canceljob || scancel
|-
| showq || squeue
|-
| checkjob $JOBID || scontrol show job $JOBID
|}

== How to convert Moab batch job scripts to Slurm? ==

Replace Moab/Torque job specification flags and environment variables in your job
scripts by their corresponding Slurm counterparts.

'''Commonly used Moab job specification flags and their Slurm equivalents'''

{| width=750px class="wikitable"
! Option !! Moab (msub) !! Slurm (sbatch)
|-
| Script directive || #MSUB || #SBATCH
|-
| Job name || -N <name> || --job-name=<name> (-J <name>)
|-
| Account || -A <account> || --account=<account> (-A <account>)
|-
| Queue || -q <queue> || --partition=<partition> (-p <partition>)
|-
| Wall time limit || -l walltime=<hh:mm:ss> || --time=<hh:mm:ss> (-t <hh:mm:ss>)
|-
| Node count || -l nodes=<count> || --nodes=<count> (-N <count>)
|-
| Core count || -l procs=<count> || --ntasks=<count> (-n <count>)
|-
| Process count per node || -l ppn=<count> || --ntasks-per-node=<count>
|-
| Core count per process || || --cpus-per-task=<count>
|-
| Memory limit per node || -l mem=<limit> || --mem=<limit>
|-
| Memory limit per process || -l pmem=<limit> || --mem-per-cpu=<limit>
|-
| Job array || -t <array indices> || --array=<indices> (-a <indices>)
|-
| Node exclusive job || -l naccesspolicy=singlejob || --exclusive
|-
| Initial working directory || -d <directory> (default: $HOME) || --chdir=<directory> (-D <directory>) (default: submission directory)
|-
| Standard output file || -o <file path> || --output=<file> (-o <file>)
|-
| Standard error file || -e <file path> || --error=<file> (-e <file>)
|-
| Combine stdout/stderr to stdout || -j oe || --output=<combined stdout/stderr file>
|-
| Mail notification events || -m <event> || --mail-type=<events> (valid types include: NONE, BEGIN, END, FAIL, ALL)
|-
| Export environment to job || -V || --export=ALL (default)
|-
| Don't export environment to job || (default) || --export=NONE
|-
| Export environment variables to job || -v <var[=value][,var2=value2[, ...]]> || --export=<var[=value][,var2=value2[,...]]>
|}

'''Notes:'''
* Default initial job working directory is $HOME for Moab. For Slurm the default working directory is where you submit your job from.
* By default Moab does not export any environment variables to the job's runtime environment. With Slurm most of the login environment variables are exported to your job's runtime environment. This includes environment variables from software modules that were loaded at job submission time (and also $HOSTNAME variable).

'''Commonly used Moab/Torque script environment variables and their Slurm equivalents

{| width=750px class="wikitable"
! Information !! Moab !! Torque !! Slurm
|-
| Job name || $MOAB_JOBNAME || $PBS_JOBNAME || $SLURM_JOB_NAME
|-
| Job ID || $MOAB_JOBID || $PBS_JOBID || $SLURM_JOB_ID
|-
| Submit directory || $MOAB_SUBMITDIR || $PBS_O_WORKDIR || $SLURM_SUBMIT_DIR
|-
| Number of nodes allocated || $MOAB_NODECOUNT || $PBS_NUM_NODES || $SLURM_JOB_NUM_NODES (and: $SLURM_NNODES)
|-
| Node list || $MOAB_NODELIST || cat $PBS_NODEFILE || $SLURM_JOB_NODELIST
|-
| Number of processes || $MOAB_PROCCOUNT || $PBS_TASKNUM || $SLURM_NTASKS
|-
| Requested tasks per node || --- || $PBS_NUM_PPN || $SLURM_NTASKS_PER_NODE
|-
| Requested CPUs per task || --- || --- || $SLURM_CPUS_PER_TASK
|-
| Job array index || $MOAB_JOBARRAYINDEX || $PBS_ARRAY_INDEX || $SLURM_ARRAY_TASK_ID
|-
| Job array range || $MOAB_JOBARRAYRANGE || - || $SLURM_ARRAY_TASK_COUNT
|-
| Queue name || $MOAB_CLASS || $PBS_QUEUE || $SLURM_JOB_PARTITION
|-
| QOS name || $MOAB_QOS || --- || $SLURM_JOB_QOS
|-
| Number of processes per node | --- || $PBS_NUM_PPN || $SLURM_TASKS_PER_NODE
|-
| Job user || $MOAB_USER || $PBS_O_LOGNAME || $SLURM_JOB_USER
|-
| Hostname || $MOAB_MACHINE || $PBS_O_HOST || $SLURMD_NODENAME
|}

The following chapters contain short examples for the most common job types. For a full overview of SLURM see the [[BwUniCluster_2.0_Slurm_common_Features|full article]].

=Serial Programs=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-n 1''' or '''--ntasks=1''' (instead of '''-l nodes=1,ppn=1''').
* Use the option '''-m''' or '''--mem''' (instead of '''-l pmem'''). The default unit is MegaByte.
* If you want to use one node exclusively, you must enter the whole memory ('''-m 96327''' or '''--mem=96327''').
 
'''Example for a serial job'''
<pre>
$ sbatch -p single -t 60 -n 1 -m 96327 ./job.sh
</pre>
The script job.sh (containing the execution of a serial program) is started running 60 minutes exclusively on a batch node.
 
 
 

=Multithreaded Programs=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-N 1''' or '''--nodes=1''' and '''c ''x''''' or '''--cpus-per-task=''x''''' (instead of '''-l nodes=1,ppn=''x'' '''). '''''x''''' can be a number between 1 and 40 (because of 40 cores within one node); it can also be a number between 41 and 80 (because of active hyperthreading).
* Use the option '''-m''' or '''--mem''' (instead of '''-l pmem'''). The default unit is MegaByte.
* Use the option '''--export''' to set the needed environment variable OMP_NUM_THREADS for the batch job. Adding '''ALL''' means to pass all interactively set environment variables to the batch job.
* If you want to use one node exclusively, you must either enter the whole memory (''-m 96327'' or ''--mem=96327'') or set the number of threads greater than 39.
 
'''Example for a multithreaded job'''
<pre>
$ sbatch -p single -t 1:00:00 -N 1 -c 20 -m 50gb --export=ALL,OMP_NUM_THREADS=20 ./job_threaded.sh
</pre>
The script '''job_threaded.sh''' (containing a multithreaded program) is started running 1 hour in shared mode on 20 cores requesting 50GB on one batch node.
 
 
 

=MPI Parallel Programs within one node=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-n ''x''''' or '''--ntasks=''x''''' (instead of '''-l nodes=1,ppn=''x'' '''). '''''x''''' can be a number between 1 and 40 (because of 40 cores within one node); you should'nt utilize hyperthreading.
* Use the option '''-m''' or '''--mem''' (instead of '''-l pmem'''). The default unit is MegaByte.
* If you want to use one node exclusively, you must either enter the whole memory (''-m 96327'' or ''--mem=96327'') or set the number of MPI tasks greater than 39.
* Don't forget to load the appropriate MPI-module in your job script.
* If you are using OpenMPI, the options '''--bind-to core --map-by core|socket|node''' of the command mpirun should be used.
 
'''Example for a MPI job'''
<pre>
$ sbatch -p single -t 600 -n 10 -m 40000 ./job_mpi.sh
</pre>
The script '''job_mpi.sh''' (containing a MPI program after loading the appropriate MPI module) is started running 10 hours in shared mode on 10 cores requesting 40000 MB on one batch node.
 
 
 

=MPI Parallel Programs on many nodes=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-N ''y''''' or '''--nodes=''y''''' and '''--ntasks-per-node=''x''''' (instead of '''-l nodes=''y'',ppn=''x'''''). '''''x''''' can be a number between 1 and 40 (28 for Broadwell nodes) (because of 40 (28) cores within one node); you should'nt utilize hyperthreading.
* You should'nt use the option '''-m''' or '''--mem''' because the nodes are used exclusively.
* You always use the nodes exclusively.
* Don't forget to load the appropriate MPI-module in your job script.
* If you are using OpenMPI, the options '''--bind-to core --map-by core|socket|node''' ofthe command mpirun should be used.
 
'''Example for a MPI job'''
<pre>
$ sbatch -p multiple -t 48:00:00 -N 10 --ntasks-per-node=40 ./job_mpi.sh
</pre>
The script '''job_mpi.sh''' (containing a MPI program after loading the appropriate MPI module) is started running 2 days on 400 cores on ten batch nodes.
 
 
 

=Multithreaded + MPI Parallel Programs on many nodes=

* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
* Use the option '''-N ''y''''' or '''--nodes=''y''''' and '''--ntasks-per-node=''x''''' and '''-c ''z''''' or '''--cpus-per-task=''z''''' (instead of '''-l nodes=''y'',ppn=''x+z'''''). '''''x''''' usually should be 1 or 2 and '''''x+z''''' usually 40 (28 on Broadwell nodes); you can utilize hyperthreading if you want.
* You should'nt use the option '''-m''' or '''--mem''' because the nodes are used exclusively.
* You always use the nodes exclusively.
* Don't forget to load the appropriate MPI-module in your job script.
* If you are using OpenMPI, the options '''--bind-to core --map-by socket|node:PE=''z''''' of the command mpirun must be used.
 
'''Example for a MPI job'''
<pre>
$ sbatch -p multiple -t 2-12 -N 10 --ntasks-per-node=2 -c 20 ./job_threaded_mpi.sh
</pre>
The script '''job_threaded_mpi.sh''' (containing a multithreaded MPI program after loading the appropriate MPI module) is started running 2.5 days on 400 cores with 20 MPI tasks and 20 threads per task on ten batch nodes. Here the options '''--bind-to core --map-by socket:PE=10''' of the command mpirun must be used.

BwUniCluster2.0/Slurm

2020-04-20T11:53:04Z

H Haefner: /* Job Submission : sbatch */

<div id="top"></div>
= Slurm HPC Workload Manager =
== Specification ==
Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters. Slurm requires no kernel modifications for its operation and is relatively self-contained. As a cluster workload manager, Slurm has three key functions. First, it allocates exclusive and/or non-exclusive access to resources (compute nodes) to users for some duration of time so they can perform work. Second, it provides a framework for starting, executing, and monitoring work (normally a parallel job) on the set of allocated nodes. Finally, it arbitrates contention for resources by managing a queue of pending work.
 
 
Any kind of calculation on the compute nodes of [[bwUniCluster 2.0|bwUniCluster 2.0]] requires the user to define calculations as a sequence of commands or single command together with required run time, number of CPU cores and main memory and submit all, i.e., the '''batch job''', to a resource and workload managing software. bwUniCluster 2.0 has installed the workload managing software Slurm. Therefore any job submission by the user is to be executed by commands of the Slurm software. Slurm queues and runs user jobs based on fair sharing policies.
 
 

== Slurm Commands (excerpt) ==
Some of the most used Slurm commands for non-administrators working on bwUniCluster 2.0.
{| width=750px class="wikitable"
! Slurm commands !! Brief explanation
|-
| [[#Job Submission : sbatch|sbatch]] || Submits a job and queues it in an input queue [[https://slurm.schedmd.com/sbatch.html sbatch]]
|-
| [[#Detailed job information : scontrol show job|scontrol show job]] || Displays detailed job state information [[https://slurm.schedmd.com/scontrol.html scontrol]]
|-
| [[#List of your submitted jo/bs : squeue|squeue]] || Displays information about active, eligible, blocked, and/or recently completed jobs [[https://slurm.schedmd.com/squeue.html squeue]]
|-
| [[#Start time of job or resources : squeue|squeue --start]] || Returns start time of submitted job or requested resources [[https://slurm.schedmd.com/squeue.html squeue]]
|-
| [[#Shows free resources : sinfo_t_idle|sinfo_t_idle]] || Shows what resources are available for immediate use [[https://slurm.schedmd.com/sinfo.html sinfo]]
|-
| [[#Canceling own jobs : scancel|scancel]] || Cancels a job (obsoleted!) [[https://slurm.schedmd.com/scancel.html scancel]]
|}
If your job was submitted to the "multiple" queue you can log into the allocated nodes via SSH as soon as the job is running.

 
* [https://slurm.schedmd.com/tutorials.html Slurm Tutorials]
* [https://slurm.schedmd.com/pdfs/summary.pdf Slurm command/option summary (2 pages)]
* [https://slurm.schedmd.com/man_index.html Slurm Commands]
 

== Job Submission : sbatch ==
Batch jobs are submitted by using the command '''sbatch'''. The main purpose of the '''sbatch''' command is to specify the resources that are needed to run the job. '''sbatch''' will then queue the batch job. However, starting of batch job depends on the availability of the requested resources and the fair sharing value.
 
 
=== sbatch Command Parameters ===
The syntax and use of '''sbatch''' can be displayed via:
<pre>
$ man sbatch
</pre>
'''sbatch''' options can be used from the command line or in your job script.
{| width=750px class="wikitable"
! colspan="3" | sbatch Options
|-
! Command line
! Script
! Purpose
|- style="vertical-align:top;"
| -t ''time'' or --time=''time''
| #SBATCH --time=''time''
| Wall clock time limit. 
|-
|- style="vertical-align:top;"
| -N ''count'' or --nodes=''count''
| #SBATCH --nodes=''count''
| Number of nodes to be used.
|- style="vertical-align:top;"
| -n ''count'' or --ntasks=''count''
| #SBATCH --ntasks=''count''
| Number of tasks to be launched.
|-
|- style="vertical-align:top;"
| --ntasks-per-node=''count''
| #SBATCH --ntasks-per-node=''count''
| Maximum count (<= 28 and <= 40 resp.) of tasks per node. (Replaces the option ppn of MOAB.)
|-
|- style="vertical-align:top;"
| -c ''count'' or --cpus-per-task=''count''
| #SBATCH --cpus-per-task=''count''
| Number of CPUs required per (MPI-)task.
|-
|- style="vertical-align:top;"
| --mem=''value_in_MB''
| #SBATCH --mem=''value_in_MB''
| Memory in MegaByte per node. (Default value is 128000 and 96000 MB resp., i.e. you should omit the setting of this option.)
|-
|- style="vertical-align:top;"
| --mem-per-cpu=''value_in_MB''
| #SBATCH --mem-per-cpu=''value_in_MB''
| Minimum Memory required per allocated CPU. (Replaces the option pmem of MOAB. You should omit the setting of this option.)
|-
|- style="vertical-align:top;"
| --mail-type=''type''
| #SBATCH --mail-type=''type''
| Notify user by email when certain event types occur. Valid type values are NONE, BEGIN, END, FAIL, REQUEUE, ALL.
|-
|- style="vertical-align:top;"
| --mail-user=''mail-address''
| #SBATCH --mail-user=''mail-address''
| The specified mail-address receives email notification of state changes as defined by --mail-type.
|-
|- style="vertical-align:top;"
| --output=''name''
| #SBATCH --output=''name''
| File in which job output is stored.
|-
|- style="vertical-align:top;"
| --error=''name''
| #SBATCH --error=''name''
| File in which job error messages are stored.
|-
|- style="vertical-align:top;"
| -J ''name'' or --job-name=''name''
| #SBATCH --job-name=''name''
| Job name.
|-
|- style="vertical-align:top;"
| --export=[ALL,] ''env-variables''
| #SBATCH --export=[ALL,] ''env-variables''
| Identifies which environment variables from the submission environment are propagated to the launched application. Default is ALL. If adding an environment variable to the submission environment is intended, the argument ALL must be added.
|-
|- style="vertical-align:top;"
| -A ''group-name'' or --account=''group-name''
| #SBATCH --account=''group-name''
| Change resources used by this job to specified group. You may need this option if your account is assigned to more than one group. By command "scontrol show job" the project group the job is accounted on can be seen behind "Account=".
|-
|- style="vertical-align:top;"
| -p ''queue-name'' or --partition=''queue-name''
| #SBATCH --partition=''queue-name''
| Request a specific queue for the resource allocation.
|-
|- style="vertical-align:top;"
| -C ''LSDF'' or --constraint=''LSDF''
| #SBATCH --constraint=LSDF
| Job constraint LSDF Filesystems.
|-
|- style="vertical-align:top;"
| -C ''BEEOND'' or --constraint=''BEEOND''
| #SBATCH --constraint=BEEOND
| Job constraint BeeOND file system.
|-
|}
 

==== sbatch --partition ''queues'' ====
Queue classes define maximum resources such as walltime, nodes and processes per node and queue of the compute system. Details can be found here:
* [[BwUniCluster_2.0_Batch_Queues#sbatch_-p_queue|bwUniCluster 2.0 queue settings]]
 

=== sbatch Examples ===
==== Serial Programs ====
To submit a serial job that runs the script '''job.sh''' and that requires 5000 MB of main memory and 10 minutes of wall clock time

a) execute:
<pre>
$ sbatch -p dev_single -n 1 -t 10:00 --mem=5000 job.sh
</pre>
or
b) add after the initial line of your script '''job.sh''' the lines (here with a high memory request):
<source lang="bash">
#SBATCH --ntasks=1
#SBATCH --time=10
#SBATCH --mem=200gb
#SBATCH --job-name=simple
</source>
and execute the modified script with the command line option ''--partition=fat'' (with ''--partition=(dev_)single'' maximum ''--mem=96gb'' is possible):
<pre>
$ sbatch --partition=fat job.sh
</pre>
Note, that sbatch command line options overrule script options.
 
 

==== Multithreaded Programs ====
Multithreaded programs operate faster than serial programs on CPUs with multiple cores. 
Moreover, multiple threads of one process share resources such as memory.
 
For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) a number of threads is defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).
 
'''Because hyperthreading is switched on on bwUniCluster 2.0, the option --cpus-per-task (-c) must be set to 2*n, if you want to use n threads.'''
 
To submit a batch job called ''OpenMP_Test'' that runs a 40-fold threaded program ''omp_exe'' which requires 6000 MByte of total physical memory and total wall clock time of 40 minutes:
 
a) execute:
<pre>
$ sbatch -p single --export=ALL,OMP_NUM_THREADS=40 -J OpenMP_Test -N 1 -c 80 -t 40 --mem=6000 ./omp_exe
</pre>
or
-->
* generate the script '''job_omp.sh''' containing the following lines:
<source lang="bash">
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --cpus-per-task=80
#SBATCH --time=40:00
#SBATCH --mem=6000mb
#SBATCH --export=ALL,EXECUTABLE=./omp_exe
#SBATCH -J OpenMP_Test

#Usually you should set
export KMP_AFFINITY=compact,1,0
#export KMP_AFFINITY=verbose,compact,1,0 prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

export OMP_NUM_THREADS=$((${SLURM_JOB_CPUS_PER_NODE}/2))
echo "Executable ${EXECUTABLE} running on ${SLURM_JOB_CPUS_PER_NODE} cores with ${OMP_NUM_THREADS} threads"
startexe=${EXECUTABLE}
echo $startexe
exec $startexe
</source>
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores and, if necessary, replace <placeholder> with the required modulefile to enable the OpenMP environment and execute the script '''job_omp.sh''' adding the queue class ''single'' as sbatch option:
<pre>
$ sbatch -p single job_omp.sh
</pre>
Note, that sbatch command line options overrule script options, e.g.,
<pre>
$ sbatch --partition=single --mem=200 job_omp.sh
</pre>
overwrites the script setting of 6000 MByte with 200 MByte.
 
 

==== MPI Parallel Programs ====
MPI parallel programs run faster than serial programs on multi CPU and multi core systems. N-fold spawned processes of the MPI program, i.e., '''MPI tasks''', run simultaneously and communicate via the Message Passing Interface (MPI) paradigm. MPI tasks do not share memory but can be spawned over different nodes.
 
Multiple MPI tasks must be launched via '''mpirun''', e.g. 4 MPI tasks of ''my_par_program'':
<pre>
$ mpirun -n 4 my_par_program
</pre>
This command runs 4 MPI tasks of ''my_par_program'' on the node you are logged in.
To run this command with a loaded Intel MPI the environment variable I_MPI_HYDRA_BOOTSTRAP must be unset ( --> $ unset I_MPI_HYDRA_BOOTSTRAP).

Running MPI parallel programs in a batch job the interactive environment - particularly the loaded modules - will also be set in the batch job. If you want to set a defined module environment in your batch job you have to purge all modules before setting the wished modules.
 
 
===== OpenMPI =====

If you want to run jobs on batch nodes, generate a wrapper script ''job_ompi.sh'' for '''OpenMPI''' containing the following lines:
<source lang="bash">
#!/bin/bash
# Use when a defined module environment related to OpenMPI is wished
module load mpi/openmpi/<placeholder_for_version>
mpirun --bind-to core --map-by core -report-bindings my_par_program
</source>
'''Attention:''' Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since Slurm instructs mpirun about number of processes and node hostnames. Use '''ALWAYS''' the MPI options '''''--bind-to core''''' and '''''--map-by core|socket|node'''''. Please type ''mpirun --help'' for an explanation of the meaning of the different options of mpirun option ''--map-by''.
 
Considering 4 OpenMPI tasks on a single node, each requiring 2000 MByte, and running for 1 hour, execute:
<pre>
$ sbatch -p single -N 1 -n 4 --mem-per-cpu=2000 --time=01:00:00 ./job_ompi.sh
</pre>
 

===== Intel MPI =====

Generate a wrapper script for '''Intel MPI''', ''job_impi.sh'' containing the following lines:
<source lang="bash">
#!/bin/bash
# Use when a defined module environment related to Intel MPI is wished
module load mpi/impi/<placeholder_for_version>
mpiexec.hydra -bootstrap slurm my_par_program
</source>
'''Attention:''' 
Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since Slurm instructs mpirun about number of processes and node hostnames.
 
Launching and running 200 Intel MPI tasks on 5 nodes, each requiring 80 GByte, and running for 5 hours, execute:
<pre>
$ sbatch --partition=multiple -N 5 --ntasks-per-node=40 --mem=80gb -t 300 ./job_impi.sh
</pre>
 
If you want to use 128 or more nodes, you must also set the environment variable as follows: 
export I_MPI_HYDRA_BRANCH_COUNT=-1
 
If you want to use the options perhost, ppn or rr, you must additionally set the environment variable I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=off.
 
 

==== Multithreaded + MPI parallel Programs ====
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes. '''Because hyperthreading is switched on on bwUniCluster 2.0, the option --cpus-per-task (-c) must be set to 2*n, if you want to use n threads.'''
 
 
===== OpenMPI with Multithreading =====
Multiple MPI tasks using '''OpenMPI''' must be launched by the MPI parallel program '''mpirun'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).
 
'''For OpenMPI''' a job-script to submit a batch job called ''job_ompi_omp.sh'' that runs a MPI program with 4 tasks and a 28-fold threaded program ''ompi_omp_program'' requiring 3000 MByte of physical memory per thread (using 28 threads per MPI task you will get 28*3000 MByte = 84000 MByte per MPI task) and total wall clock time of 3 hours looks like:

<source lang="bash">
#!/bin/bash
#SBATCH --ntasks=4
#SBATCH --cpus-per-task=56
#SBATCH --time=03:00:00
#SBATCH --mem=83gb # 84000 MB = 84000/1024 GB = 82.1 GB
#SBATCH --export=ALL,MPI_MODULE=mpi/openmpi/3.1,EXECUTABLE=./ompi_omp_program
#SBATCH --output="parprog_hybrid_%j.out"

# Use when a defined module environment related to OpenMPI is wished
module load ${MPI_MODULE}
export OMP_NUM_THREADS=$((${SLURM_CPUS_PER_TASK}/2))
export MPIRUN_OPTIONS="--bind-to core --map-by socket:PE=${OMP_NUM_THREADS} -report-bindings"
export NUM_CORES=${SLURM_NTASKS}*${OMP_NUM_THREADS}
echo "${EXECUTABLE} running on ${NUM_CORES} cores with ${SLURM_NTASKS} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpirun -n ${SLURM_NTASKS} ${MPIRUN_OPTIONS} ${EXECUTABLE}"
echo $startexe
exec $startexe
</source>
Execute the script '''job_ompi_omp.sh''' by command sbatch:
<pre>
$ sbatch -p multiple_e ./job_ompi_omp.sh
</pre>
* With the mpirun option ''--bind-to core'' MPI tasks and OpenMP threads are bound to physical cores.
* With the option ''--map-by node:PE=<value>'' (neighbored) MPI tasks will be attached to different nodes and each MPI task is bound to the first core of a node. <value> must be set to ${OMP_NUM_THREADS}.
* The option ''-report-bindings'' shows the bindings between MPI tasks and physical cores.
* The mpirun-options '''--bind-to core''', '''--map-by socket|...|node:PE=<value>''' should always be used when running a multithreaded MPI program.
 

===== Intel MPI with Multithreading =====
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes.

Multiple Intel MPI tasks must be launched by the MPI parallel program '''mpiexec.hydra'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).

'''For Intel MPI''' a job-script to submit a batch job called ''job_impi_omp.sh'' that runs a Intel MPI program with 10 tasks and a 40-fold threaded program ''impi_omp_program'' requiring 96000 MByte of total physical memory per task and total wall clock time of 1 hours looks like:


<source lang="bash">
#!/bin/bash
#SBATCH --ntasks=10
#SBATCH --cpus-per-task=80
#SBATCH --time=60
#SBATCH --mem=96000
#SBATCH --export=ALL,MPI_MODULE=mpi/impi,EXE=./impi_omp_program
#SBATCH --output="parprog_impi_omp_%j.out"

#If using more than one MPI task per node please set
export KMP_AFFINITY=compact,1,0
#export KMP_AFFINITY=verbose,scatter prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

# Use when a defined module environment related to Intel MPI is wished
module load ${MPI_MODULE}
export OMP_NUM_THREADS=$((${SLURM_CPUS_PER_TASK}/2))
export MPIRUN_OPTIONS="-binding "domain=omp:compact" -print-rank-map -envall"
export NUM_PROCS=eval(${SLURM_NTASKS}*${OMP_NUM_THREADS})
echo "${EXE} running on ${NUM_PROCS} cores with ${SLURM_NTASKS} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpiexec.hydra -bootstrap slurm ${MPIRUN_OPTIONS} -n ${SLURM_NTASKS} ${EXE}"
echo $startexe
exec $startexe
</source>
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores. If you only run one MPI task per node please set KMP_AFFINITY=compact,1,0.
 
If you want to use 128 or more nodes, you must also set the environment variable as follows: 
export I_MPI_HYDRA_BRANCH_COUNT=-1
 
If you want to use the options perhost, ppn or rr, you must additionally set the environment variable I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=off.
 
 
Execute the script '''job_impi_omp.sh''' by command sbatch:
<pre>
$ sbatch -p multiple ./job_impi_omp.sh
</pre>
 
The mpirun option ''-print-rank-map'' shows the bindings between MPI tasks and nodes (not very beneficial). The option ''-binding'' binds MPI tasks (processes) to a particular processor; ''domain=omp'' means that the domain size is determined by the number of threads. If you would choose 2 MPI tasks per node, you should choose ''-binding "cell=unit;map=bunch"''; this binding maps one MPI process to each socket.
 
 

==== Chain jobs ====
The CPU time requirements of many applications exceed the limits of the job classes. In those situations it is recommended to solve the problem by a job chain. A job chain is a sequence of jobs where each job automatically starts its successor.
<source lang="bash">
#!/bin/bash
####################################
## simple Slurm submitter script to setup ##
## a chain of jobs using Slurm ##
####################################
## ver. : 2018-11-27, KIT, SCC

## Define maximum number of jobs via positional parameter 1, default is 5
max_nojob=${1:-5}

## Define your jobscript (e.g. "~/chain_job.sh")
chain_link_job=${PWD}/chain_job.sh

## Define type of dependency via positional parameter 2, default is 'afterok'
dep_type="${2:-afterok}"
## -> List of all dependencies:
## https://slurm.schedmd.com/sbatch.html

myloop_counter=1
## Submit loop
while [ ${myloop_counter} -le ${max_nojob} ] ; do
##
## Differ msub_opt depending on chain link number
if [ ${myloop_counter} -eq 1 ] ; then
slurm_opt=""
else
slurm_opt="-d ${dep_type}:${jobID}"
fi
##
## Print current iteration number and sbatch command
echo "Chain job iteration = ${myloop_counter}"
echo " sbatch --export=myloop_counter=${myloop_counter} ${slurm_opt} ${chain_link_job}"
## Store job ID for next iteration by storing output of sbatch command with empty lines
jobID=$(sbatch -p <queue> --export=ALL,myloop_counter=${myloop_counter} ${slurm_opt} ${chain_link_job} 2>&1 | sed 's/[S,a-z]* //g')
##
## Check if ERROR occured
if [[ "${jobID}" =~ "ERROR" ]] ; then
echo " -> submission failed!" ; exit 1
else
echo " -> job number = ${jobID}"
fi
##
## Increase counter
let myloop_counter+=1
done
</source>
 

==== GPU jobs ====

The nodes in the gpu_4 and gpu_8 queues have 4 or 8 NVIDIA Tesla V100 GPUs. Just submitting a job to these queues is not enough to also allocate one or more GPUs, you have to do so using the "--gres=gpu" parameter. You have to specifiy how many GPUs your job needs, e.g. "--gres=gpu:2" will request two GPUs.

The GPU nodes are shared between multiple jobs if the jobs don't request all the GPUs in a node and there are enough ressources to run more than one job. The individual GPUs are always bound to a single job and will not be shared between different jobs.

a) add after the initial line of your script job.sh the line including the
information about the LSDF Online Storage usage: #SBATCH --constraint=LSDF
<pre>
#!/bin/bash
#SBATCH --ntasks=40
#SBATCH --time=02:00:00
#SBATCH --mem=4000
#SBATCH --gres=gpu:2
</pre>

or b) execute:
<pre>
$ sbatch -p <queue> -n 40 -t 02:00:00 --mem 4000 --gres=gpu:2 job.sh
</pre>
 
If you start an interactive session on of the GPU nodes, you can use the "nvidia-smi" command to list the GPUs allocated to your job:
<pre>
$ nvidia-smi
Sun Mar 29 15:20:05 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2... Off | 00000000:3A:00.0 Off | 0 |
| N/A 29C P0 39W / 300W | 9MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla V100-SXM2... Off | 00000000:3B:00.0 Off | 0 |
| N/A 30C P0 41W / 300W | 8MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 14228 G /usr/bin/X 8MiB |
| 1 14228 G /usr/bin/X 8MiB |
+-----------------------------------------------------------------------------+
</pre>

 
 

==== LSDF Online Storage ====
On bwUniCluster 2.0 you can use for special cases the LSDF Online Storage on the HPC cluster nodes. Please request for this service separately ([https://www.lsdf.kit.edu/os/storagerequest/: LSDF Storage Request]).
To mount the LSDF Online Storage on the compute nodes during the job runtime the
the constraint flag "LSDF" has to be set.

a) add after the initial line of your script job.sh the line including the
information about the LSDF Online Storage usage: #SBATCH --constraint=LSDF
<pre>
#!/bin/bash
#SBATCH --ntasks=1
#SBATCH --time=120
#SBATCH --mem=200
#SBATCH --constraint=LSDF
</pre>

or b) execute:
<pre>
$ sbatch -p <queue> -n 1 -t 2:00:00 --mem 200 job.sh -C LSDF
</pre>
 
For the usage of the LSDF Online Storage
the following environment variables are available: $LSDF, $LSDFPROJECTS, $LSDFHOME.
 
 

====BeeOND (BeeGFS On-Demand)====

BeeOND instances are integrated into the prolog and epilog script of the cluster batch system, Slurm. It can be used on the compute nodes during the job runtime with the constraint flag "BEEOND" ([[BwUniCluster_2.0_Slurm_common_Features#sbatch_Command_Parameters|Slurm Command Parameters]])
<pre>
#!/bin/bash
#SBATCH ...
#SBATCH --constraint=BEEOND
</pre>

After your job has started you can find the private on-demand file system in '''/mnt/odfs/$SLURM_JOB_ID''' directory. The mountpoint comes with three pre-configured directories:
<pre>
#for small files (stripe count = 1)
/mnt/odfs/$SLURM_JOB_ID/stripe_1
#stripe count = 4
/mnt/odfs/$SLURM_JOB_ID/stripe_default
#stripe count = 8, 16 or 32, use this directories for medium sized and large files or when using MPI-IO
/mnt/odfs/$SLURM_JOB_ID/stripe_8, /mnt/odfs/$SLURM_JOB_ID/stripe_16 or /mnt/odfs/$SLURM_JOB_ID/stripe_32
</pre>

If you request less nodes than stripe count, the stripe count will be max number of nodes, e.g., You only request 8 nodes , so the directory with stripe count 16 is basically only with a stripe count 8.

The capacity of the private file system depends on the number of nodes. For each node you get 250Gbyte.

!!! Be careful when creating large files, use always the directory with the max stripe count for large files.
If you create large files use a higher stripe count. For example, if your largest file is 1.1 Tb, then you have to use a stripe count larger>4 (4 x 250GB).

If you request 100 nodes for your job, the private file system is 100 * 250 Gbyte ~ 25 Tbyte (approx) capacity.

'''Recommendation:'''

The private file system is using its own metadata server. This metadata server is started on the first nodes. Depending on your application, the metadata server is consuming decent amount of CPU power. Probably adding a extra node to your job could improve the usability of the on-demand file system. Start your application with the MPI option:
<pre>
mpirun -nolocal myapplication
</pre>
With the -nolocal option the node where mpirun is initiated is not used for your application. This node is fully available for the meta data server of your requested on-demand file system.

Example job script:
<pre>
#!/bin/bash
#very simple example on how to use a private on-demand file system
#SBATCH -N 10
#SBATCH --constraint=BEEOND

#create a workspace
ws_allocate myresults-$SLURM_JOB_ID 90
RESULTDIR=`ws_find myresults-$SLURM_JOB_ID`

#Set ENV variable to on-demand file system
ODFSDIR=/mnt/odfs/$SLURM_JOB_ID/stripe_16/

#start application and write results to on-demand file system
mpirun -nolocal myapplication -o $ODFSDIR/results

#Copy back data after your job application end
rsync -av $ODFSDIR/results $RESULTDIR
</pre>
 
 

== Start time of job or resources : squeue --start ==
The command can be used by any user to displays the estimated start time of a job based a number of analysis types based on historical usage, earliest available reservable resources, and priority based backlog. The command squeue is explained in detail on the webpage https://slurm.schedmd.com/squeue.html or via manpage (man squeue).
 
 
=== Access ===
By default, this command can be run by '''any user'''.
 
 

== List of your submitted jobs : squeue ==
Displays information about YOUR active, pending and/or recently completed jobs. The command displays all own active and pending jobs. The command squeue is explained in detail on the webpage https://slurm.schedmd.com/squeue.html or via manpage (man squeue).
 
 
=== Access ===
By default, this command can be run by any user.
 
 

=== Flags ===
{| width=750px class="wikitable"
|-
! Flag !! Description
|-
| -l, --long
| Report more of the available information for the selected jobs or job steps, subject to any constraints specified.
|}
 

=== Examples ===
''squeue'' example on bwUniCluster 2.0 (Only your own jobs are displayed!).
<pre>
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
18088744 single CPV.sbat ab1234 PD 0:00 1 (Priority)
18098414 multiple CPV.sbat ab1234 PD 0:00 2 (Priority)
18090089 multiple CPV.sbat ab1234 R 2:27 2 uc2n[127-128]
$ squeue -l
JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON)
18088654 single CPV.sbat ab1234 COMPLETI 4:29 2:00:00 1 uc2n374
18088785 single CPV.sbat ab1234 PENDING 0:00 2:00:00 1 (Priority)
18098414 multiple CPV.sbat ab1234 PENDING 0:00 2:00:00 2 (Priority)
18088683 single CPV.sbat ab1234 RUNNING 0:14 2:00:00 1 uc2n413
</pre>
* The output of ''squeue'' shows how many jobs of yours are running or pending and how many nodes are in use by your jobs.
 

== Shows free resources : sinfo_t_idle ==
The Slurm command sinfo is used to view partition and node information for a system running Slurm. It incorporates down time, reservations, and node state information in determining the available backfill window. The sinfo command can only be used by the administrator.
 
SCC has prepared a special script (sinfo_t_idle) to find out how many processors are available for immediate use on the system. It is anticipated that users will use this information to submit jobs that meet these criteria and thus obtain quick job turnaround times.
 
 
=== Access ===
By default, this command can be used by any user or administrator.
 
 
=== Example ===
* The following command displays what resources are available for immediate use for the whole partition.
<pre>$ sinfo_t_idle
Partition dev_multiple : 8 nodes idle
Partition multiple : 332 nodes idle
Partition dev_single : 4 nodes idle
Partition single : 76 nodes idle
Partition long : 80 nodes idle
Partition fat : 5 nodes idle
Partition dev_special : 342 nodes idle
Partition special : 342 nodes idle
Partition dev_multiple_e: 7 nodes idle
Partition multiple_e : 335 nodes idle
Partition gpu_4 : 12 nodes idle
Partition gpu_8 : 6 nodes idle
</pre>
* For the above example jobs in all partitions can be run immediately.
 

== Detailed job information : scontrol show job ==
scontrol show job displays detailed job state information and diagnostic output for all or a specified job of yours. Detailed information is available for active, pending and recently completed jobs. The command scontrol is explained in detail on the webpage https://slurm.schedmd.com/scontrol.html or via manpage (man scontrol).
 
Display the state of all your jobs in normal mode: scontrol show job
 
Display the state of a job with <jobid> in normal mode: scontrol show job <jobid>
 
 
=== Access ===
* End users can use scontrol show job to view the status of their '''own jobs''' only.
 

=== Arguments ===
{| width=750px class="wikitable"
|-
! Option !! Default !! Description !! Example
|- style="vertical-align:top;"
|- style="width:12%;"
| -d
| (n/a)
| Detailed mode
| Example: Display the state with jobid 18089884 in detailed mode. <pre>scontrol -d show job 18089884</pre>
|}
 
 

=== Scontrol show job Example ===
Here is an example from bwUniCluster 2.0.
<pre>
squeue # show my own jobs (here the userid is replaced!)
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
18089884 multiple CPV.sbat bq0742 R 33:44 2 uc2n[165-166]

$
$ # now, see what's up with my pending job with jobid 18089884
$
$ scontrol show job 18089884

JobId=18089884 JobName=CPV.sbatch
UserId=bq0742(8946) GroupId=scc(12345) MCS_label=N/A
Priority=3 Nice=0 Account=kit QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:35:06 TimeLimit=02:00:00 TimeMin=N/A
SubmitTime=2020-03-16T14:14:54 EligibleTime=2020-03-16T14:14:54
AccrueTime=2020-03-16T14:14:54
StartTime=2020-03-16T15:12:51 EndTime=2020-03-16T17:12:51 Deadline=N/A
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2020-03-16T15:12:51
Partition=multiple AllocNode:Sid=uc2n995:5064
ReqNodeList=(null) ExcNodeList=(null)
NodeList=uc2n[165-166]
BatchHost=uc2n165
NumNodes=2 NumCPUs=160 NumTasks=80 CPUs/Task=1 ReqB:S:C:T=0:0:*:1
TRES=cpu=160,mem=96320M,node=2,billing=160
Socks/Node=* NtasksPerN:B:S:C=40:0:*:1 CoreSpec=*
MinCPUsNode=40 MinMemoryCPU=1204M MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
Command=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/CPV.sbatch
WorkDir=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin
StdErr=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/slurm-18089884.out
StdIn=/dev/null
StdOut=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/slurm-18089884.out
Power=
MailUser=(null) MailType=NONE
</pre>
 
You can use standard Linux pipe commands to filter the very detailed scontrol show job output.
* In which state the job is?
<pre>$ scontrol show job 18089884 | grep -i State
JobState=COMPLETED Reason=None Dependency=(null)
</pre>
 

== Cancel Slurm Jobs ==
The scancel command is used to cancel jobs. The command scancel is explained in detail on the webpage https://slurm.schedmd.com/scancel.html or via manpage (man scancel).
 
 
=== Canceling own jobs : scancel ===
scancel is used to signal or cancel jobs, job arrays or job steps. The command is:
<pre>
$ scancel [-i] <job-id>
$ scancel -t <job_state_name>
</pre>

 
{| width=750px class="wikitable"
! Flag !! Default !! Description !! Example
|- style="vertical-align:top;"
| -i, --interactive
| (n/a)
| Interactive mode.
| Cancel the job 987654 interactively. <pre> scancel -i 987654 </pre>
|-
| -t, --state
| (n/a)
| Restrict the scancel operation to jobs in a certain state. "job_state_name" may have a value of either "PENDING", "RUNNING" or "SUSPENDED".
| Cancel all jobs in state "PENDING". <pre> scancel -t "PENDING" </pre>
|}
 

= Resource Managers =
=== Batch Job (Slurm) Variables ===
The following environment variables of Slurm are added to your environment once your job has started
(only an excerpt of the most important ones).
{| width=750px class="wikitable"
! Environment !! Brief explanation
|-
| SLURM_JOB_CPUS_PER_NODE
| Number of processes per node dedicated to the job
|-
| SLURM_JOB_NODELIST
| List of nodes dedicated to the job
|-
| SLURM_JOB_NUM_NODES
| Number of nodes dedicated to the job
|-
| SLURM_MEM_PER_NODE
| Memory per node dedicated to the job
|-
| SLURM_NPROCS
| Total number of processes dedicated to the job
|-
| SLURM_CLUSTER_NAME
| Name of the cluster executing the job
|-
| SLURM_CPUS_PER_TASK
| Number of CPUs requested per task
|-
| SLURM_JOB_ACCOUNT
| Account name
|-
| SLURM_JOB_ID
| Job ID
|-
| SLURM_JOB_NAME
| Job Name
|-
| SLURM_JOB_PARTITION
| Partition/queue running the job
|-
| SLURM_JOB_UID
| User ID of the job's owner
|-
| SLURM_SUBMIT_DIR
| Job submit folder. The directory from which msub was invoked.
|-
| SLURM_JOB_USER
| User name of the job's owner
|-
| SLURM_RESTART_COUNT
| Number of times job has restarted
|-
| SLURM_PROCID
| Task ID (MPI rank)
|-
| SLURM_NTASKS
| The total number of tasks available for the job
|-
| SLURM_STEP_ID
| Job step ID
|-
| SLURM_STEP_NUM_TASKS
| Task count (number of PI ranks)
|-
| SLURM_JOB_CONSTRAINT
| Job constraints
|}
See also:
* [https://slurm.schedmd.com/sbatch.html#lbAI Slurm input and output environment variables]
 

=== Job Exit Codes ===
A job's exit code (also known as exit status, return code and completion code) is captured by SLURM and saved as part of the job record.
 
Any non-zero exit code will be assumed to be a job failure and will result in a Job State of FAILED with a reason of "NonZeroExitCode".
 
The exit code is an 8 bit unsigned number ranging between 0 and 255. While it is possible for a job to return a negative exit code, SLURM will display it as an unsigned value in the 0 - 255 range.
 
 
==== Displaying Exit Codes and Signals ====
SLURM displays a job's exit code in the output of the '''scontrol show job''' and the sview utility.
 
When a signal was responsible for a job or step's termination, the signal number will be displayed after the exit code, delineated by a colon(:).
 
 
==== Submitting Termination Signal ====
Here is an example, how to 'save' a Slurm termination signal in a typical jobscript.
<source lang="bash">
[...]
exit_code=$?
mpirun -np <#cores> <EXE_BIN_DIR>/<executable> ... (options) 2>&1
[ "$exit_code" -eq 0 ] && echo "all clean..." || \
echo "Executable <EXE_BIN_DIR>/<executable> finished with exit code ${$exit_code}"
[...]
</source>
* Do not use ''''time'''' mpirun! The exit code will be the one submitted by the first (time) program.
* You do not need an '''exit $exit_code''' in the scripts.
 
 
----
[[Category:bwUniCluster 2.0|bwUniCluster 2.0]]
[[#top|Back to top]]

BwUniCluster2.0/Acknowledgement

2020-04-20T10:57:00Z

H Haefner:

The HPC resource bwUniCluster (2.0) is funded by the Ministry of Science, Research and the Arts Baden-Württemberg and the Universities of the State of Baden-Württemberg:
* Albert Ludwig University of Freiburg
* Eberhard Karls University, Tübingen
* Karlsruhe Institute of Technology (KIT)
* Ruprecht-Karls-Universität Heidelberg
* Ulm University
* University of Hohenheim
* University of Konstanz
* University of Mannheim
* University of Stuttgart.

Therefore, any work performed on bwUniCluster (2.0) should be appropriately cited:

'''"The authors acknowledge support by the state of Baden-Württemberg through bwHPC."'''

Furthermore, please report any publication that cites bwUniCluster (2.0) to [mailto:publications@bwhpc.de publications@bwhpc.de] stating:
* DOI (if available)
* author(s)
* title ''or'' booktitle
* journal, volume, pages ''or'' editors, address, publisher
* year.
This information will be used for the evaluation of bwUniCluster (2.0).

The publications will be referenced on the bwHPC website:
https://www.bwhpc.de/user_publications.html

----
[[Category:bwUniCluster|Acknowledgement]]
[[Category:Acknowledgement|bwUniCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2020-04-20T10:47:05Z

H Haefner: /* How to login to broadwell login nodes? */

FAQs concerning best practice of [[BwUniCluster_Hardware_and_Architecture#Components_of_bwUniCluster|bwUniCluster broadwell partition]] (aka "extension" partition).

__TOC__

= Login =
== Are there separate login nodes for the bwUniCluster broadwell partition? ==
* Yes, but primarily to be used for compiling code.

== How to login to broadwell login nodes? ==
* You can directly login on broadwell partition login nodes using
<pre>
$ ssh username@uc1e.scc.kit.edu
</pre>
* If you are compiling code on broadwell login nodes, your code will not optimally run on the new "Cascade Lake" nodes.
 

= Compilation =
== How to compile code on broadwell (extension) nodes? ==
To use the code only on the partition multiple_e:
<pre>
$ icc/ifort -xHost [-further_options]
</pre>

== How to compile code to be used on ALL partitions? ==
On uc1e (= extension) login nodes:
<pre>
$ icc/ifort -xCORE-AVX2 -axCORE-AVX512 [-further_options]
</pre>
 

= Job execution =
== How to submit jobs to the broadwell (= extension) partition ==
The submitted job will be distributed either way to the broadwell nodes if specified correctly, i.e.:
<pre>
$ sbatch -p multiple_e
</pre>

== Can I use my old multinode job script for the new broadwell partition? ==
Yes, but please note that all broadwell nodes do have '''28 cores per node'''.

----
[[Category:bwUniCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2020-04-20T10:46:46Z

H Haefner: /* How to compile code to be used on ALL partitions? */

FAQs concerning best practice of [[BwUniCluster_Hardware_and_Architecture#Components_of_bwUniCluster|bwUniCluster broadwell partition]] (aka "extension" partition).

__TOC__

= Login =
== Are there separate login nodes for the bwUniCluster broadwell partition? ==
* Yes, but primarily to be used for compiling code.

== How to login to broadwell login nodes? ==
* You can directly login on broadwell partition login nodes using
<pre>
$ ssh username@uc1e.scc.kit.edu
</pre>
* If you are compiling code on broadwell login nodes, your code will not optimally run on the new "Cascade Lake" nodes.

= Compilation =
== How to compile code on broadwell (extension) nodes? ==
To use the code only on the partition multiple_e:
<pre>
$ icc/ifort -xHost [-further_options]
</pre>

== How to compile code to be used on ALL partitions? ==
On uc1e (= extension) login nodes:
<pre>
$ icc/ifort -xCORE-AVX2 -axCORE-AVX512 [-further_options]
</pre>
 

= Job execution =
== How to submit jobs to the broadwell (= extension) partition ==
The submitted job will be distributed either way to the broadwell nodes if specified correctly, i.e.:
<pre>
$ sbatch -p multiple_e
</pre>

== Can I use my old multinode job script for the new broadwell partition? ==
Yes, but please note that all broadwell nodes do have '''28 cores per node'''.

----
[[Category:bwUniCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2020-04-20T10:45:58Z

H Haefner: /* What happens with code compiled for old partition whic is running on the extension partition? */

FAQs concerning best practice of [[BwUniCluster_Hardware_and_Architecture#Components_of_bwUniCluster|bwUniCluster broadwell partition]] (aka "extension" partition).

__TOC__

= Login =
== Are there separate login nodes for the bwUniCluster broadwell partition? ==
* Yes, but primarily to be used for compiling code.

== How to login to broadwell login nodes? ==
* You can directly login on broadwell partition login nodes using
<pre>
$ ssh username@uc1e.scc.kit.edu
</pre>
* If you are compiling code on broadwell login nodes, your code will not optimally run on the new "Cascade Lake" nodes.

= Compilation =
== How to compile code on broadwell (extension) nodes? ==
To use the code only on the partition multiple_e:
<pre>
$ icc/ifort -xHost [-further_options]
</pre>

== How to compile code to be used on ALL partitions? ==
On uc1e (= extension) login nodes:
<pre>
$ icc/ifort -xCORE-AVX2 -axCORE-AVX512 [-further_options]
</pre>

= Job execution =
== How to submit jobs to the broadwell (= extension) partition ==
The submitted job will be distributed either way to the broadwell nodes if specified correctly, i.e.:
<pre>
$ sbatch -p multiple_e
</pre>

== Can I use my old multinode job script for the new broadwell partition? ==
Yes, but please note that all broadwell nodes do have '''28 cores per node'''.

----
[[Category:bwUniCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2020-04-20T10:44:20Z

H Haefner: /* How to compile code on broadwell (extension) nodes? */

FAQs concerning best practice of [[BwUniCluster_Hardware_and_Architecture#Components_of_bwUniCluster|bwUniCluster broadwell partition]] (aka "extension" partition).

__TOC__

= Login =
== Are there separate login nodes for the bwUniCluster broadwell partition? ==
* Yes, but primarily to be used for compiling code.

== How to login to broadwell login nodes? ==
* You can directly login on broadwell partition login nodes using
<pre>
$ ssh username@uc1e.scc.kit.edu
</pre>
* If you are compiling code on broadwell login nodes, your code will not optimally run on the new "Cascade Lake" nodes.

= Compilation =
== How to compile code on broadwell (extension) nodes? ==
To use the code only on the partition multiple_e:
<pre>
$ icc/ifort -xHost [-further_options]
</pre>

== How to compile code to be used on ALL partitions? ==
On uc1e (= extension) login nodes:
<pre>
$ icc/ifort -xCORE-AVX2 -axCORE-AVX512 [-further_options]
</pre>

== What happens with code compiled for old partition whic is running on the extension partition? ==
Code will run but significantly slower since AVX2 will not be used. Please recompile your code accordingly.



= Job execution =
== How to submit jobs to the broadwell (= extension) partition ==
The submitted job will be distributed either way to the broadwell nodes if specified correctly, i.e.:
<pre>
$ sbatch -p multiple_e
</pre>

== Can I use my old multinode job script for the new broadwell partition? ==
Yes, but please note that all broadwell nodes do have '''28 cores per node'''.

----
[[Category:bwUniCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2020-04-20T10:42:55Z

H Haefner: /* How to compile code on broadwell (extension) nodes? */

FAQs concerning best practice of [[BwUniCluster_Hardware_and_Architecture#Components_of_bwUniCluster|bwUniCluster broadwell partition]] (aka "extension" partition).

__TOC__

= Login =
== Are there separate login nodes for the bwUniCluster broadwell partition? ==
* Yes, but primarily to be used for compiling code.

== How to login to broadwell login nodes? ==
* You can directly login on broadwell partition login nodes using
<pre>
$ ssh username@uc1e.scc.kit.edu
</pre>
* If you are compiling code on broadwell login nodes, your code will not optimally run on the new "Cascade Lake" nodes.

= Compilation =
== How to compile code on broadwell (extension) nodes? ==
On uc1 (old) login nodes:
<pre>
$ icc/ifort -axCORE-AVX2 [-further_options]
</pre>
To use the code only on the partition multiple_e:
<pre>
$ icc/ifort -xHost [-further_options]
</pre>

== How to compile code to be used on ALL partitions? ==
On uc1e (= extension) login nodes:
<pre>
$ icc/ifort -xCORE-AVX2 -axCORE-AVX512 [-further_options]
</pre>

== What happens with code compiled for old partition whic is running on the extension partition? ==
Code will run but significantly slower since AVX2 will not be used. Please recompile your code accordingly.



= Job execution =
== How to submit jobs to the broadwell (= extension) partition ==
The submitted job will be distributed either way to the broadwell nodes if specified correctly, i.e.:
<pre>
$ sbatch -p multiple_e
</pre>

== Can I use my old multinode job script for the new broadwell partition? ==
Yes, but please note that all broadwell nodes do have '''28 cores per node'''.

----
[[Category:bwUniCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2020-04-20T10:41:35Z

H Haefner: /* How to compile code tobe used on ALL partitions? */

FAQs concerning best practice of [[BwUniCluster_Hardware_and_Architecture#Components_of_bwUniCluster|bwUniCluster broadwell partition]] (aka "extension" partition).

__TOC__

= Login =
== Are there separate login nodes for the bwUniCluster broadwell partition? ==
* Yes, but primarily to be used for compiling code.

== How to login to broadwell login nodes? ==
* You can directly login on broadwell partition login nodes using
<pre>
$ ssh username@uc1e.scc.kit.edu
</pre>
* If you are compiling code on broadwell login nodes, your code will not optimally run on the new "Cascade Lake" nodes.

= Compilation =
== How to compile code on broadwell (extension) nodes? ==
On uc1 (old) login nodes:
<pre>
$ icc/ifort -axCORE-AVX2 [-further_options]
</pre>
On uc1e (extension) login nodes:
<pre>
$ icc/ifort -xHost [-further_options]
</pre>

== How to compile code to be used on ALL partitions? ==
On uc1e (= extension) login nodes:
<pre>
$ icc/ifort -xCORE-AVX2 -axCORE-AVX512 [-further_options]
</pre>

== What happens with code compiled for old partition whic is running on the extension partition? ==
Code will run but significantly slower since AVX2 will not be used. Please recompile your code accordingly.



= Job execution =
== How to submit jobs to the broadwell (= extension) partition ==
The submitted job will be distributed either way to the broadwell nodes if specified correctly, i.e.:
<pre>
$ sbatch -p multiple_e
</pre>

== Can I use my old multinode job script for the new broadwell partition? ==
Yes, but please note that all broadwell nodes do have '''28 cores per node'''.

----
[[Category:bwUniCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2020-04-20T10:41:22Z

H Haefner: /* How to compile the same code on old and extension partition? */

FAQs concerning best practice of [[BwUniCluster_Hardware_and_Architecture#Components_of_bwUniCluster|bwUniCluster broadwell partition]] (aka "extension" partition).

__TOC__

= Login =
== Are there separate login nodes for the bwUniCluster broadwell partition? ==
* Yes, but primarily to be used for compiling code.

== How to login to broadwell login nodes? ==
* You can directly login on broadwell partition login nodes using
<pre>
$ ssh username@uc1e.scc.kit.edu
</pre>
* If you are compiling code on broadwell login nodes, your code will not optimally run on the new "Cascade Lake" nodes.

= Compilation =
== How to compile code on broadwell (extension) nodes? ==
On uc1 (old) login nodes:
<pre>
$ icc/ifort -axCORE-AVX2 [-further_options]
</pre>
On uc1e (extension) login nodes:
<pre>
$ icc/ifort -xHost [-further_options]
</pre>

== How to compile code tobe used on ALL partitions? ==
On uc1e (= extension) login nodes:
<pre>
$ icc/ifort -xCORE-AVX2 -axCORE-AVX512 [-further_options]
</pre>

== What happens with code compiled for old partition whic is running on the extension partition? ==
Code will run but significantly slower since AVX2 will not be used. Please recompile your code accordingly.



= Job execution =
== How to submit jobs to the broadwell (= extension) partition ==
The submitted job will be distributed either way to the broadwell nodes if specified correctly, i.e.:
<pre>
$ sbatch -p multiple_e
</pre>

== Can I use my old multinode job script for the new broadwell partition? ==
Yes, but please note that all broadwell nodes do have '''28 cores per node'''.

----
[[Category:bwUniCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2020-04-20T10:22:22Z

H Haefner: /* How to login to broadwell login nodes? */

FAQs concerning best practice of [[BwUniCluster_Hardware_and_Architecture#Components_of_bwUniCluster|bwUniCluster broadwell partition]] (aka "extension" partition).

__TOC__

= Login =
== Are there separate login nodes for the bwUniCluster broadwell partition? ==
* Yes, but primarily to be used for compiling code.

== How to login to broadwell login nodes? ==
* You can directly login on broadwell partition login nodes using
<pre>
$ ssh username@uc1e.scc.kit.edu
</pre>
* If you are compiling code on broadwell login nodes, your code will not optimally run on the new "Cascade Lake" nodes.

= Compilation =
== How to compile code on broadwell (extension) nodes? ==
On uc1 (old) login nodes:
<pre>
$ icc/ifort -axCORE-AVX2 [-further_options]
</pre>
On uc1e (extension) login nodes:
<pre>
$ icc/ifort -xHost [-further_options]
</pre>

== How to compile the same code on old and extension partition? ==
On uc1e (= extension) login nodes:
<pre>
$ icc/ifort -xAVX -axCORE-AVX2 [-further_options]
</pre>

== What happens with code compiled for old partition whic is running on the extension partition? ==
Code will run but significantly slower since AVX2 will not be used. Please recompile your code accordingly.



= Job execution =
== How to submit jobs to the broadwell (= extension) partition ==
The submitted job will be distributed either way to the broadwell nodes if specified correctly, i.e.:
<pre>
$ sbatch -p multiple_e
</pre>

== Can I use my old multinode job script for the new broadwell partition? ==
Yes, but please note that all broadwell nodes do have '''28 cores per node'''.

----
[[Category:bwUniCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2020-04-20T09:41:15Z

H Haefner: /* How to submit jobs to the broadwell (= extension) partition */

FAQs concerning best practice of [[BwUniCluster_Hardware_and_Architecture#Components_of_bwUniCluster|bwUniCluster broadwell partition]] (aka "extension" partition).

__TOC__

= Login =
== Are there separate login nodes for the bwUniCluster broadwell partition? ==
* Yes, but primarily to be used for compiling code.

== How to login to broadwell login nodes? ==
* You can directly login on broadwell partition login nodes using
<pre>
$ ssh username@uc1e.scc.kit.edu
</pre>
* If you login to the ''old'' uc1 login nodes, even though you can use broadwell nodes using the same procedure as 'compute nodes'.

= Compilation =
== How to compile code on broadwell (extension) nodes? ==
On uc1 (old) login nodes:
<pre>
$ icc/ifort -axCORE-AVX2 [-further_options]
</pre>
On uc1e (extension) login nodes:
<pre>
$ icc/ifort -xHost [-further_options]
</pre>

== How to compile the same code on old and extension partition? ==
On uc1e (= extension) login nodes:
<pre>
$ icc/ifort -xAVX -axCORE-AVX2 [-further_options]
</pre>

== What happens with code compiled for old partition whic is running on the extension partition? ==
Code will run but significantly slower since AVX2 will not be used. Please recompile your code accordingly.



= Job execution =
== How to submit jobs to the broadwell (= extension) partition ==
The submitted job will be distributed either way to the broadwell nodes if specified correctly, i.e.:
<pre>
$ sbatch -p multiple_e
</pre>

== Can I use my old multinode job script for the new broadwell partition? ==
Yes, but please note that all broadwell nodes do have '''28 cores per node'''.

----
[[Category:bwUniCluster]]

Category:BwUniCluster 2.0

2020-04-20T09:38:46Z

H Haefner:

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:center; color:#000;vertical-align:middle;font-size:75%;" |
[[File:BwUniCluster_2.0_Feb2020.jpg|center|border|550px|Close-up of bwUniCluster by Robert Barthel, copyright: KIT (SCC)]]
|-
| style="text-align:center; color:#000;vertical-align:middle;" |Close-up of bwUniCluster © KIT (SCC)
|}

On 17.03.2020, the Steinbuch Centre for Computing (SCC) at Karlsruhe Institute of Technology (KIT) commissioned a new parallel computer system called "bwUniCluster 2.0+GFB-HPC" as a state service within the bwHPC framework. The bwUniCluster 2.0 replaces the predecessor system [[bwUniCluster]] and also includes the additional compute nodes which were procured as an extension to the bwUniCluster in November 2016.

The modern bwUniCluster 2.0 system consists of more than 840 SMP nodes with 64-bit Intel Xeon processors. It provides the universities of the state of Baden-Württemberg with general compute resources and can be used free of charge by the staff of all universities in Baden-Württemberg. Users who currently have access to bwUniCluster will automatically also have access to bwUniCluster 2.0. There is no need to apply for new entitlements or to re-register.


{| style="width: 100%; margin:4px 0 0 0; background:none; border-spacing: 0px;"
| style="width:100%; border:1px solid #BBBBBB; background:#fff5fa; vertical-align:top; color:#000;" |
{| style="width:100%; vertical-align:top; border:0px solid #BBBBBB; padding:4px;" |
|-
|{{Red}}| Overview: Changes from bwUniCluster 1
|-
|
* Access: (Nearly) no changes. Entitlements are still valid.
* Software: Switch to the Lmod module system, cleanup of older software module versions
* Hardware: Significantly more powerful hardware, 132 Volta V100 GPUs
* File systems: See the [[BwUniCluster_2.0_File_System_Migration_Guide|File system migration guide]]
* Batch system: See the [[BwUniCluster_2.0_Batch_System_Migration_Guide|Batch system migration guide]]
|}
|}


{| style="width: 100%; margin:4px 0 0 0; background:none; border-spacing: 0px;"
| style="width:50%; border:1px solid #BBBBBB; background:#f5fffa; vertical-align:top; color:#000;" |
{| style="width:100%; vertical-align:top; border:0px solid #BBBBBB; padding:4px;" |
|-
|{{Green}}| Access
|-
|
* bwUniCluster [[BwUniCluster_2.0_User_Access|Registration and Login]]
* Registration [[bwUniCluster_2.0_Contact_and_Support|trouble issues]] & [[BwUniCluster_2.0_User_Access#Deregistration|Deregistration]]
* [[First_Steps_on_bwHPC_cluster|First steps on bwUniCluster]]
 
|-
|{{Green}}| Software
|-
|
* [[bwUniCluster_2.0_Software|Software and Environment Modules]]
|}
{| style="width:100%; vertical-align:top; border:0px solid #BBBBBB; padding:4px;" |
|-
|{{Green}}| Hardware
|-
|
* [[bwUniCluster_2.0_Hardware_and_Architecture|Hardware and Architecture]]
* [[BwUniCluster_2.0_Hardware_and_Architecture#File_Systems|File Systems]]
|}

| style="padding:2px;" |

| style="width:50%; border:1px solid #BBBBBB; background:#f5faff; vertical-align:top; color:#000;" |
{| style="width:100%; vertical-align:top; border:0px solid #BBBBBB; padding:4px;" |
|-
|{{Blue}}| Batch/Compute Jobs
|-
|
* [[bwUniCluster_2.0_Slurm_common_Features|Slurm common Features]]
* [[BwUniCluster_2.0_Batch_Queues|Batch Queues and interactive Jobs]]
|-
|{{Blue}}| [[BwHPC_Best_Practices_Repository|bwHPC Best Practice Guides]] / FAQs
|-
|




* [[FAQ - bwUniCluster_broadwell_partition|FAQ - bwUniCluster 2.0 Broadwell partition]]
|-
|{{Blue}}| Miscellaneous
|-
|
* [[bwUniCluster_Acknowledgement|Acknowledgement]] of work performed on bwUniCluster (2.0)

|}
|}

 
-----
 
 
[[Category:bwHPC_infrastructure]][[Category:bwHPC_Cluster]][[Category:bwCluster]]

Category:BwUniCluster 2.0

2020-04-20T09:35:27Z

H Haefner:

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:center; color:#000;vertical-align:middle;font-size:75%;" |
[[File:BwUniCluster_2.0_Feb2020.jpg|center|border|550px|Close-up of bwUniCluster by Robert Barthel, copyright: KIT (SCC)]]
|-
| style="text-align:center; color:#000;vertical-align:middle;" |Close-up of bwUniCluster © KIT (SCC)
|}

On 17.03.2020, the Steinbuch Centre for Computing (SCC) at Karlsruhe Institute of Technology (KIT) commissioned a new parallel computer system called "bwUniCluster 2.0+GFB-HPC" as a state service within the bwHPC framework. The bwUniCluster 2.0 replaces the predecessor system [[bwUniCluster]] and also includes the additional compute nodes which were procured as an extension to the bwUniCluster in November 2016.

The modern bwUniCluster 2.0 system consists of more than 840 SMP nodes with 64-bit Intel Xeon processors. It provides the universities of the state of Baden-Württemberg with general compute resources and can be used free of charge by the staff of all universities in Baden-Württemberg. Users who currently have access to bwUniCluster will automatically also have access to bwUniCluster 2.0. There is no need to apply for new entitlements or to re-register.


{| style="width: 100%; margin:4px 0 0 0; background:none; border-spacing: 0px;"
| style="width:100%; border:1px solid #BBBBBB; background:#fff5fa; vertical-align:top; color:#000;" |
{| style="width:100%; vertical-align:top; border:0px solid #BBBBBB; padding:4px;" |
|-
|{{Red}}| Overview: Changes from bwUniCluster 1
|-
|
* Access: (Nearly) no changes. Entitlements are still valid.
* Software: Switch to the Lmod module system, cleanup of older software module versions
* Hardware: Significantly more powerful hardware, 132 Volta V100 GPUs
* File systems: See the [[BwUniCluster_2.0_File_System_Migration_Guide|File system migration guide]]
* Batch system: See the [[BwUniCluster_2.0_Batch_System_Migration_Guide|Batch system migration guide]]
|}
|}


{| style="width: 100%; margin:4px 0 0 0; background:none; border-spacing: 0px;"
| style="width:50%; border:1px solid #BBBBBB; background:#f5fffa; vertical-align:top; color:#000;" |
{| style="width:100%; vertical-align:top; border:0px solid #BBBBBB; padding:4px;" |
|-
|{{Green}}| Access
|-
|
* bwUniCluster [[BwUniCluster_2.0_User_Access|Registration and Login]]
* Registration [[bwUniCluster_2.0_Contact_and_Support|trouble issues]] & [[BwUniCluster_2.0_User_Access#Deregistration|Deregistration]]
* [[First_Steps_on_bwHPC_cluster|First steps on bwUniCluster]]
 
|-
|{{Green}}| Software
|-
|
* [[bwUniCluster_2.0_Software|Software and Environment Modules]]
|}
{| style="width:100%; vertical-align:top; border:0px solid #BBBBBB; padding:4px;" |
|-
|{{Green}}| Hardware
|-
|
* [[bwUniCluster_2.0_Hardware_and_Architecture|Hardware and Architecture]]
* [[BwUniCluster_2.0_Hardware_and_Architecture#File_Systems|File Systems]]
|}

| style="padding:2px;" |

| style="width:50%; border:1px solid #BBBBBB; background:#f5faff; vertical-align:top; color:#000;" |
{| style="width:100%; vertical-align:top; border:0px solid #BBBBBB; padding:4px;" |
|-
|{{Blue}}| Batch/Compute Jobs
|-
|
* [[bwUniCluster_2.0_Slurm_common_Features|Slurm common Features]]
* [[BwUniCluster_2.0_Batch_Queues|Batch Queues and interactive Jobs]]
|-
|{{Blue}}| [[BwHPC_Best_Practices_Repository|bwHPC Best Practice Guides]] / FAQs
|-
|
* [[Compiler]]
* [[Debugger]]
* [[Numerical Libraries]]
* [[Parallel Programming]]
* [[FAQ - bwUniCluster_broadwell_partition|FAQ - bwUniCluster Broadwell partition]]
|-
|{{Blue}}| Miscellaneous
|-
|
* [[bwUniCluster_Acknowledgement|Acknowledgement]] of work performed on bwUniCluster

|}
|}

 
-----
 
 
[[Category:bwHPC_infrastructure]][[Category:bwHPC_Cluster]][[Category:bwCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2020-04-20T09:30:52Z

H Haefner: H Haefner moved page FAQ - bwUniCluster broadwell partition to FAQ - bwUniCluster 2.0 broadwell partition: bwUniCluster replaced by bwUniCluster 2.0

FAQs concerning best practice of [[BwUniCluster_Hardware_and_Architecture#Components_of_bwUniCluster|bwUniCluster broadwell partition]] (aka "extension" partition).

__TOC__

= Login =
== Are there separate login nodes for the bwUniCluster broadwell partition? ==
* Yes, but primarily to be used for compiling code.

== How to login to broadwell login nodes? ==
* You can directly login on broadwell partition login nodes using
<pre>
$ ssh username@uc1e.scc.kit.edu
</pre>
* If you login to the ''old'' uc1 login nodes, even though you can use broadwell nodes using the same procedure as 'compute nodes'.

= Compilation =
== How to compile code on broadwell (extension) nodes? ==
On uc1 (old) login nodes:
<pre>
$ icc/ifort -axCORE-AVX2 [-further_options]
</pre>
On uc1e (extension) login nodes:
<pre>
$ icc/ifort -xHost [-further_options]
</pre>

== How to compile the same code on old and extension partition? ==
On uc1e (= extension) login nodes:
<pre>
$ icc/ifort -xAVX -axCORE-AVX2 [-further_options]
</pre>

== What happens with code compiled for old partition whic is running on the extension partition? ==
Code will run but significantly slower since AVX2 will not be used. Please recompile your code accordingly.



= Job execution =
== How to submit jobs to the broadwell (= extension) partition ==
The submitted job will be distributed either way to the broadwell nodes if specified correctly, i.e.:
<pre>
$ msub -q multinode
</pre>

== Can I use my old multinode job script for the new broadwell partition? ==
Yes, but please note that all broadwell nodes do have '''28 cores per node'''.

----
[[Category:bwUniCluster]]

BwUniCluster2.0/Slurm

2020-04-02T11:07:30Z

H Haefner: /* Intel MPI with Multithreading */

<div id="top"></div>
= Slurm HPC Workload Manager =
== Specification ==
Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters. Slurm requires no kernel modifications for its operation and is relatively self-contained. As a cluster workload manager, Slurm has three key functions. First, it allocates exclusive and/or non-exclusive access to resources (compute nodes) to users for some duration of time so they can perform work. Second, it provides a framework for starting, executing, and monitoring work (normally a parallel job) on the set of allocated nodes. Finally, it arbitrates contention for resources by managing a queue of pending work.
 
 
Any kind of calculation on the compute nodes of [[bwUniCluster 2.0|bwUniCluster 2.0]] requires the user to define calculations as a sequence of commands or single command together with required run time, number of CPU cores and main memory and submit all, i.e., the '''batch job''', to a resource and workload managing software. bwUniCluster 2.0 has installed the workload managing software Slurm. Therefore any job submission by the user is to be executed by commands of the Slurm software. Slurm queues and runs user jobs based on fair sharing policies.
 
 

== Slurm Commands (excerpt) ==
Some of the most used Slurm commands for non-administrators working on bwUniCluster 2.0.
{| width=750px class="wikitable"
! Slurm commands !! Brief explanation
|-
| [[#Job Submission : sbatch|sbatch]] || Submits a job and queues it in an input queue [[https://slurm.schedmd.com/sbatch.html sbatch]]
|-
| [[#Detailed job information : scontrol show job|scontrol show job]] || Displays detailed job state information [[https://slurm.schedmd.com/scontrol.html scontrol]]
|-
| [[#List of your submitted jo/bs : squeue|squeue]] || Displays information about active, eligible, blocked, and/or recently completed jobs [[https://slurm.schedmd.com/squeue.html squeue]]
|-
| [[#Start time of job or resources : squeue|squeue --start]] || Returns start time of submitted job or requested resources [[https://slurm.schedmd.com/squeue.html squeue]]
|-
| [[#Shows free resources : sinfo_t_idle|sinfo_t_idle]] || Shows what resources are available for immediate use [[https://slurm.schedmd.com/sinfo.html sinfo]]
|-
| [[#Canceling own jobs : scancel|scancel]] || Cancels a job (obsoleted!) [[https://slurm.schedmd.com/scancel.html scancel]]
|}
If your job was submitted to the "multiple" queue you can log into the allocated nodes via SSH as soon as the job is running.

 
* [https://slurm.schedmd.com/tutorials.html Slurm Tutorials]
* [https://slurm.schedmd.com/pdfs/summary.pdf Slurm command/option summary (2 pages)]
* [https://slurm.schedmd.com/man_index.html Slurm Commands]
 

== Job Submission : sbatch ==
Batch jobs are submitted by using the command '''sbatch'''. The main purpose of the '''sbatch''' command is to specify the resources that are needed to run the job. '''sbatch''' will then queue the batch job. However, starting of batch job depends on the availability of the requested resources and the fair sharing value.
 
 
=== sbatch Command Parameters ===
The syntax and use of '''sbatch''' can be displayed via:
<pre>
$ man sbatch
</pre>
'''sbatch''' options can be used from the command line or in your job script.
{| width=750px class="wikitable"
! colspan="3" | sbatch Options
|-
! Command line
! Script
! Purpose
|- style="vertical-align:top;"
| -t ''time'' or --time=''time''
| #SBATCH --time=''time''
| Wall clock time limit. 
|-
|- style="vertical-align:top;"
| -N ''count'' or --nodes=''count''
| #SBATCH --nodes=''count''
| Number of nodes to be used.
|- style="vertical-align:top;"
| -n ''count'' or --ntasks=''count''
| #SBATCH --ntasks=''count''
| Number of tasks to be launched.
|-
|- style="vertical-align:top;"
| --ntasks-per-node=''count''
| #SBATCH --ntasks-per-node=''count''
| Maximum count (<= 28 and <= 40 resp.) of tasks per node. (Replaces the option ppn of MOAB.)
|-
|- style="vertical-align:top;"
| -c ''count'' or --cpus-per-task=''count''
| #SBATCH --cpus-per-task=''count''
| Number of CPUs required per (MPI-)task.
|-
|- style="vertical-align:top;"
| --mem=''value_in_MB''
| #SBATCH --mem=''value_in_MB''
| Memory in MegaByte per node. (Default value is 128000 and 96000 MB resp., i.e. you should omit the setting of this option.)
|-
|- style="vertical-align:top;"
| --mem-per-cpu=''value_in_MB''
| #SBATCH --mem-per-cpu=''value_in_MB''
| Minimum Memory required per allocated CPU. (Replaces the option pmem of MOAB. You should omit the setting of this option.)
|-
|- style="vertical-align:top;"
| --mail-type=''type''
| #SBATCH --mail-type=''type''
| Notify user by email when certain event types occur. Valid type values are NONE, BEGIN, END, FAIL, REQUEUE, ALL.
|-
|- style="vertical-align:top;"
| --mail-user=''mail-address''
| #SBATCH --mail-user=''mail-address''
| The specified mail-address receives email notification of state changes as defined by --mail-type.
|-
|- style="vertical-align:top;"
| --output=''name''
| #SBATCH --output=''name''
| File in which job output is stored.
|-
|- style="vertical-align:top;"
| --error=''name''
| #SBATCH --error=''name''
| File in which job error messages are stored.
|-
|- style="vertical-align:top;"
| -J ''name'' or --job-name=''name''
| #SBATCH --job-name=''name''
| Job name.
|-
|- style="vertical-align:top;"
| --export=[ALL,] ''env-variables''
| #SBATCH --export=[ALL,] ''env-variables''
| Identifies which environment variables from the submission environment are propagated to the launched application. Default is ALL. If adding to the submission environment instead of replacing it is intended, the argument ALL must be added.
|-
|- style="vertical-align:top;"
| -A ''group-name'' or --account=''group-name''
| #SBATCH --account=''group-name''
| Change resources used by this job to specified group. You may need this option if your account is assigned to more than one group. By command "scontrol show job" the project group the job is accounted on can be seen behind "Account=".
|-
|- style="vertical-align:top;"
| -p ''queue-name'' or --partition=''queue-name''
| #SBATCH --partition=''queue-name''
| Request a specific queue for the resource allocation.
|-
|- style="vertical-align:top;"
| -C ''LSDF'' or --constraint=''LSDF''
| #SBATCH --constraint=LSDF
| Job constraint LSDF Filesystems.
|-
|- style="vertical-align:top;"
| -C ''BEEOND'' or --constraint=''BEEOND''
| #SBATCH --constraint=BEEOND
| Job constraint BeeOND file system.
|-
|}
 

==== sbatch --partition ''queues'' ====
Queue classes define maximum resources such as walltime, nodes and processes per node and queue of the compute system. Details can be found here:
* [[BwUniCluster_2.0_Batch_Queues#sbatch_-p_queue|bwUniCluster 2.0 queue settings]]
 

=== sbatch Examples ===
==== Serial Programs ====
To submit a serial job that runs the script '''job.sh''' and that requires 5000 MB of main memory and 10 minutes of wall clock time

a) execute:
<pre>
$ sbatch -p dev_single -n 1 -t 10:00 --mem=5000 job.sh
</pre>
or
b) add after the initial line of your script '''job.sh''' the lines (here with a high memory request):
<source lang="bash">
#SBATCH --ntasks=1
#SBATCH --time=10
#SBATCH --mem=200gb
#SBATCH --job-name=simple
</source>
and execute the modified script with the command line option ''--partition=fat'' (with ''--partition=(dev_)single'' maximum ''--mem=96gb'' is possible):
<pre>
$ sbatch --partition=fat job.sh
</pre>
Note, that sbatch command line options overrule script options.
 
 

==== Multithreaded Programs ====
Multithreaded programs operate faster than serial programs on CPUs with multiple cores. 
Moreover, multiple threads of one process share resources such as memory.
 
For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) a number of threads is defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).
 
'''Because hyperthreading is switched on on bwUniCluster 2.0, the option --cpus-per-task (-c) must be set to 2*n, if you want to use n threads.'''
 
To submit a batch job called ''OpenMP_Test'' that runs a 40-fold threaded program ''omp_exe'' which requires 6000 MByte of total physical memory and total wall clock time of 40 minutes:
 
a) execute:
<pre>
$ sbatch -p single --export=ALL,OMP_NUM_THREADS=40 -J OpenMP_Test -N 1 -c 80 -t 40 --mem=6000 ./omp_exe
</pre>
or
-->
* generate the script '''job_omp.sh''' containing the following lines:
<source lang="bash">
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --cpus-per-task=80
#SBATCH --time=40:00
#SBATCH --mem=6000mb
#SBATCH --export=ALL,EXECUTABLE=./omp_exe
#SBATCH -J OpenMP_Test

#Usually you should set
export KMP_AFFINITY=compact,1,0
#export KMP_AFFINITY=verbose,compact,1,0 prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

export OMP_NUM_THREADS=$((${SLURM_JOB_CPUS_PER_NODE}/2))
echo "Executable ${EXECUTABLE} running on ${SLURM_JOB_CPUS_PER_NODE} cores with ${OMP_NUM_THREADS} threads"
startexe=${EXECUTABLE}
echo $startexe
exec $startexe
</source>
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores and, if necessary, replace <placeholder> with the required modulefile to enable the OpenMP environment and execute the script '''job_omp.sh''' adding the queue class ''single'' as sbatch option:
<pre>
$ sbatch -p single job_omp.sh
</pre>
Note, that sbatch command line options overrule script options, e.g.,
<pre>
$ sbatch --partition=single --mem=200 job_omp.sh
</pre>
overwrites the script setting of 6000 MByte with 200 MByte.
 
 

==== MPI Parallel Programs ====
MPI parallel programs run faster than serial programs on multi CPU and multi core systems. N-fold spawned processes of the MPI program, i.e., '''MPI tasks''', run simultaneously and communicate via the Message Passing Interface (MPI) paradigm. MPI tasks do not share memory but can be spawned over different nodes.
 
Multiple MPI tasks must be launched via '''mpirun''', e.g. 4 MPI tasks of ''my_par_program'':
<pre>
$ mpirun -n 4 my_par_program
</pre>
This command runs 4 MPI tasks of ''my_par_program'' on the node you are logged in.
To run this command with a loaded Intel MPI the environment variable I_MPI_HYDRA_BOOTSTRAP must be unset ( --> $ unset I_MPI_HYDRA_BOOTSTRAP).

Running MPI parallel programs in a batch job the interactive environment - particularly the loaded modules - will also be set in the batch job. If you want to set a defined module environment in your batch job you have to purge all modules before setting the wished modules.
 
 
===== OpenMPI =====

If you want to run jobs on batch nodes, generate a wrapper script ''job_ompi.sh'' for '''OpenMPI''' containing the following lines:
<source lang="bash">
#!/bin/bash
# Use when a defined module environment related to OpenMPI is wished
module load mpi/openmpi/<placeholder_for_version>
mpirun --bind-to core --map-by core -report-bindings my_par_program
</source>
'''Attention:''' Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since Slurm instructs mpirun about number of processes and node hostnames. Use '''ALWAYS''' the MPI options '''''--bind-to core''''' and '''''--map-by core|socket|node'''''. Please type ''mpirun --help'' for an explanation of the meaning of the different options of mpirun option ''--map-by''.
 
Considering 4 OpenMPI tasks on a single node, each requiring 2000 MByte, and running for 1 hour, execute:
<pre>
$ sbatch -p single -N 1 -n 4 --mem-per-cpu=2000 --time=01:00:00 ./job_ompi.sh
</pre>
 

===== Intel MPI =====

Generate a wrapper script for '''Intel MPI''', ''job_impi.sh'' containing the following lines:
<source lang="bash">
#!/bin/bash
# Use when a defined module environment related to Intel MPI is wished
module load mpi/impi/<placeholder_for_version>
mpiexec.hydra -bootstrap slurm my_par_program
</source>
'''Attention:''' 
Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since Slurm instructs mpirun about number of processes and node hostnames.
 
Launching and running 200 Intel MPI tasks on 5 nodes, each requiring 80 GByte, and running for 5 hours, execute:
<pre>
$ sbatch --partition=multiple -N 5 --ntasks-per-node=40 --mem=80gb -t 300 ./job_impi.sh
</pre>
 
If you want to use 128 or more nodes, you must also set the environment variable as follows: 
export I_MPI_HYDRA_BRANCH_COUNT=-1
 
If you want to use the options perhost, ppn or rr, you must additionally set the environment variable I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=off.
 
 

==== Multithreaded + MPI parallel Programs ====
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes. '''Because hyperthreading is switched on on bwUniCluster 2.0, the option --cpus-per-task (-c) must be set to 2*n, if you want to use n threads.'''
 
 
===== OpenMPI with Multithreading =====
Multiple MPI tasks using '''OpenMPI''' must be launched by the MPI parallel program '''mpirun'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).
 
'''For OpenMPI''' a job-script to submit a batch job called ''job_ompi_omp.sh'' that runs a MPI program with 4 tasks and a 28-fold threaded program ''ompi_omp_program'' requiring 3000 MByte of physical memory per thread (using 28 threads per MPI task you will get 28*3000 MByte = 84000 MByte per MPI task) and total wall clock time of 3 hours looks like:

<source lang="bash">
#!/bin/bash
#SBATCH --ntasks=4
#SBATCH --cpus-per-task=56
#SBATCH --time=03:00:00
#SBATCH --mem=83gb # 84000 MB = 84000/1024 GB = 82.1 GB
#SBATCH --export=ALL,MPI_MODULE=mpi/openmpi/3.1,EXECUTABLE=./ompi_omp_program
#SBATCH --output="parprog_hybrid_%j.out"

# Use when a defined module environment related to OpenMPI is wished
module load ${MPI_MODULE}
export OMP_NUM_THREADS=$((${SLURM_CPUS_PER_TASK}/2))
export MPIRUN_OPTIONS="--bind-to core --map-by socket:PE=${OMP_NUM_THREADS} -report-bindings"
export NUM_CORES=${SLURM_NTASKS}*${OMP_NUM_THREADS}
echo "${EXECUTABLE} running on ${NUM_CORES} cores with ${SLURM_NTASKS} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpirun -n ${SLURM_NTASKS} ${MPIRUN_OPTIONS} ${EXECUTABLE}"
echo $startexe
exec $startexe
</source>
Execute the script '''job_ompi_omp.sh''' by command sbatch:
<pre>
$ sbatch -p multiple_e ./job_ompi_omp.sh
</pre>
* With the mpirun option ''--bind-to core'' MPI tasks and OpenMP threads are bound to physical cores.
* With the option ''--map-by node:PE=<value>'' (neighbored) MPI tasks will be attached to different nodes and each MPI task is bound to the first core of a node. <value> must be set to ${OMP_NUM_THREADS}.
* The option ''-report-bindings'' shows the bindings between MPI tasks and physical cores.
* The mpirun-options '''--bind-to core''', '''--map-by socket|...|node:PE=<value>''' should always be used when running a multithreaded MPI program.
 

===== Intel MPI with Multithreading =====
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes.

Multiple Intel MPI tasks must be launched by the MPI parallel program '''mpiexec.hydra'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).

'''For Intel MPI''' a job-script to submit a batch job called ''job_impi_omp.sh'' that runs a Intel MPI program with 10 tasks and a 40-fold threaded program ''impi_omp_program'' requiring 96000 MByte of total physical memory per task and total wall clock time of 1 hours looks like:


<source lang="bash">
#!/bin/bash
#SBATCH --ntasks=10
#SBATCH --cpus-per-task=80
#SBATCH --time=60
#SBATCH --mem=96000
#SBATCH --export=ALL,MPI_MODULE=mpi/impi,EXE=./impi_omp_program
#SBATCH --output="parprog_impi_omp_%j.out"

#If using more than one MPI task per node please set
export KMP_AFFINITY=compact,1,0
#export KMP_AFFINITY=verbose,scatter prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

# Use when a defined module environment related to Intel MPI is wished
module load ${MPI_MODULE}
export OMP_NUM_THREADS=$((${SLURM_CPUS_PER_TASK}/2))
export MPIRUN_OPTIONS="-binding "domain=omp:compact" -print-rank-map -envall"
export NUM_PROCS=eval(${SLURM_NTASKS}*${OMP_NUM_THREADS})
echo "${EXE} running on ${NUM_PROCS} cores with ${SLURM_NTASKS} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpiexec.hydra -bootstrap slurm ${MPIRUN_OPTIONS} -n ${SLURM_NTASKS} ${EXE}"
echo $startexe
exec $startexe
</source>
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores. If you only run one MPI task per node please set KMP_AFFINITY=compact,1,0.
 
If you want to use 128 or more nodes, you must also set the environment variable as follows: 
export I_MPI_HYDRA_BRANCH_COUNT=-1
 
If you want to use the options perhost, ppn or rr, you must additionally set the environment variable I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=off.
 
 
Execute the script '''job_impi_omp.sh''' by command sbatch:
<pre>
$ sbatch -p multiple ./job_impi_omp.sh
</pre>
 
The mpirun option ''-print-rank-map'' shows the bindings between MPI tasks and nodes (not very beneficial). The option ''-binding'' binds MPI tasks (processes) to a particular processor; ''domain=omp'' means that the domain size is determined by the number of threads. If you would choose 2 MPI tasks per node, you should choose ''-binding "cell=unit;map=bunch"''; this binding maps one MPI process to each socket.
 
 

==== Chain jobs ====
The CPU time requirements of many applications exceed the limits of the job classes. In those situations it is recommended to solve the problem by a job chain. A job chain is a sequence of jobs where each job automatically starts its successor.
<source lang="bash">
#!/bin/bash
####################################
## simple Slurm submitter script to setup ##
## a chain of jobs using Slurm ##
####################################
## ver. : 2018-11-27, KIT, SCC

## Define maximum number of jobs via positional parameter 1, default is 5
max_nojob=${1:-5}

## Define your jobscript (e.g. "~/chain_job.sh")
chain_link_job=${PWD}/chain_job.sh

## Define type of dependency via positional parameter 2, default is 'afterok'
dep_type="${2:-afterok}"
## -> List of all dependencies:
## https://slurm.schedmd.com/sbatch.html

myloop_counter=1
## Submit loop
while [ ${myloop_counter} -le ${max_nojob} ] ; do
##
## Differ msub_opt depending on chain link number
if [ ${myloop_counter} -eq 1 ] ; then
slurm_opt=""
else
slurm_opt="-d ${dep_type}:${jobID}"
fi
##
## Print current iteration number and sbatch command
echo "Chain job iteration = ${myloop_counter}"
echo " sbatch --export=myloop_counter=${myloop_counter} ${slurm_opt} ${chain_link_job}"
## Store job ID for next iteration by storing output of sbatch command with empty lines
jobID=$(sbatch -p <queue> --export=ALL,myloop_counter=${myloop_counter} ${slurm_opt} ${chain_link_job} 2>&1 | sed 's/[S,a-z]* //g')
##
## Check if ERROR occured
if [[ "${jobID}" =~ "ERROR" ]] ; then
echo " -> submission failed!" ; exit 1
else
echo " -> job number = ${jobID}"
fi
##
## Increase counter
let myloop_counter+=1
done
</source>
 

==== GPU jobs ====

The nodes in the gpu_4 and gpu_8 queues have 4 or 8 NVIDIA Tesla V100 GPUs. Just submitting a job to these queues is not enough to also allocate one or more GPUs, you have to do so using the "--gres=gpu" parameter. You have to specifiy how many GPUs your job needs, e.g. "--gres=gpu:2" will request two GPUs.

The GPU nodes are shared between multiple jobs if the jobs don't request all the GPUs in a node and there are enough ressources to run more than one job. The individual GPUs are always bound to a single job and will not be shared between different jobs.

a) add after the initial line of your script job.sh the line including the
information about the LSDF Online Storage usage: #SBATCH --constraint=LSDF
<pre>
#!/bin/bash
#SBATCH --ntasks=40
#SBATCH --time=02:00:00
#SBATCH --mem=4000
#SBATCH --gres=gpu:2
</pre>

or b) execute:
<pre>
$ sbatch -p <queue> -n 40 -t 02:00:00 --mem 4000 --gres=gpu:2 job.sh
</pre>
 
If you start an interactive session on of the GPU nodes, you can use the "nvidia-smi" command to list the GPUs allocated to your job:
<pre>
$ nvidia-smi
Sun Mar 29 15:20:05 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2... Off | 00000000:3A:00.0 Off | 0 |
| N/A 29C P0 39W / 300W | 9MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla V100-SXM2... Off | 00000000:3B:00.0 Off | 0 |
| N/A 30C P0 41W / 300W | 8MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 14228 G /usr/bin/X 8MiB |
| 1 14228 G /usr/bin/X 8MiB |
+-----------------------------------------------------------------------------+
</pre>

 
 

==== LSDF Online Storage ====
On bwUniCluster 2.0 you can use for special cases the LSDF Online Storage on the HPC cluster nodes. Please request for this service separately ([https://www.lsdf.kit.edu/os/storagerequest/: LSDF Storage Request]).
To mount the LSDF Online Storage on the compute nodes during the job runtime the
the constraint flag "LSDF" has to be set.

a) add after the initial line of your script job.sh the line including the
information about the LSDF Online Storage usage: #SBATCH --constraint=LSDF
<pre>
#!/bin/bash
#SBATCH --ntasks=1
#SBATCH --time=120
#SBATCH --mem=200
#SBATCH --constraint=LSDF
</pre>

or b) execute:
<pre>
$ sbatch -p <queue> -n 1 -t 2:00:00 --mem 200 job.sh -C LSDF
</pre>
 
For the usage of the LSDF Online Storage
the following environment variables are available: $LSDF, $LSDFPROJECTS, $LSDFHOME.
 
 

====BeeOND (BeeGFS On-Demand)====

BeeOND instances are integrated into the prolog and epilog script of the cluster batch system, Slurm. It can be used on the compute nodes during the job runtime with the constraint flag "BEEOND" ([[BwUniCluster_2.0_Slurm_common_Features#sbatch_Command_Parameters|Slurm Command Parameters]])
<pre>
#!/bin/bash
#SBATCH ...
#SBATCH --constraint=BEEOND
</pre>

After your job has started you can find the private on-demand file system in '''/mnt/odfs/$SLURM_JOB_ID''' directory. The mountpoint comes with three pre-configured directories:
<pre>
#for small files (stripe count = 1)
/mnt/odfs/$SLURM_JOB_ID/stripe_1
#stripe count = 4
/mnt/odfs/$SLURM_JOB_ID/stripe_default
#stripe count = 8, 16 or 32, use this directories for medium sized and large files or when using MPI-IO
/mnt/odfs/$SLURM_JOB_ID/stripe_8, /mnt/odfs/$SLURM_JOB_ID/stripe_16 or /mnt/odfs/$SLURM_JOB_ID/stripe_32
</pre>

If you request less nodes than stripe count, the stripe count will be max number of nodes, e.g., You only request 8 nodes , so the directory with stripe count 16 is basically only with a stripe count 8.

The capacity of the private file system depends on the number of nodes. For each node you get 250Gbyte.

!!! Be careful when creating large files, use always the directory with the max stripe count for large files.
If you create large files use a higher stripe count. For example, if your largest file is 1.1 Tb, then you have to use a stripe count larger>4 (4 x 250GB).

If you request 100 nodes for your job, the private file system is 100 * 250 Gbyte ~ 25 Tbyte (approx) capacity.

'''Recommendation:'''

The private file system is using its own metadata server. This metadata server is started on the first nodes. Depending on your application, the metadata server is consuming decent amount of CPU power. Probably adding a extra node to your job could improve the usability of the on-demand file system. Start your application with the MPI option:
<pre>
mpirun -nolocal myapplication
</pre>
With the -nolocal option the node where mpirun is initiated is not used for your application. This node is fully available for the meta data server of your requested on-demand file system.

Example job script:
<pre>
#!/bin/bash
#very simple example on how to use a private on-demand file system
#SBATCH -N 10
#SBATCH --constraint=BEEOND

#create a workspace
ws_allocate myresults-$SLURM_JOB_ID 90
RESULTDIR=`ws_find myresults-$SLURM_JOB_ID`

#Set ENV variable to on-demand file system
ODFSDIR=/mnt/odfs/$SLURM_JOB_ID/stripe_16/

#start application and write results to on-demand file system
mpirun -nolocal myapplication -o $ODFSDIR/results

#Copy back data after your job application end
rsync -av $ODFSDIR/results $RESULTDIR
</pre>
 
 

== Start time of job or resources : squeue --start ==
The command can be used by any user to displays the estimated start time of a job based a number of analysis types based on historical usage, earliest available reservable resources, and priority based backlog. The command squeue is explained in detail on the webpage https://slurm.schedmd.com/squeue.html or via manpage (man squeue).
 
 
=== Access ===
By default, this command can be run by '''any user'''.
 
 

== List of your submitted jobs : squeue ==
Displays information about YOUR active, pending and/or recently completed jobs. The command displays all own active and pending jobs. The command squeue is explained in detail on the webpage https://slurm.schedmd.com/squeue.html or via manpage (man squeue).
 
 
=== Access ===
By default, this command can be run by any user.
 
 

=== Flags ===
{| width=750px class="wikitable"
|-
! Flag !! Description
|-
| -l, --long
| Report more of the available information for the selected jobs or job steps, subject to any constraints specified.
|}
 

=== Examples ===
''squeue'' example on bwUniCluster 2.0 (Only your own jobs are displayed!).
<pre>
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
18088744 single CPV.sbat ab1234 PD 0:00 1 (Priority)
18098414 multiple CPV.sbat ab1234 PD 0:00 2 (Priority)
18090089 multiple CPV.sbat ab1234 R 2:27 2 uc2n[127-128]
$ squeue -l
JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON)
18088654 single CPV.sbat ab1234 COMPLETI 4:29 2:00:00 1 uc2n374
18088785 single CPV.sbat ab1234 PENDING 0:00 2:00:00 1 (Priority)
18098414 multiple CPV.sbat ab1234 PENDING 0:00 2:00:00 2 (Priority)
18088683 single CPV.sbat ab1234 RUNNING 0:14 2:00:00 1 uc2n413
</pre>
* The output of ''squeue'' shows how many jobs of yours are running or pending and how many nodes are in use by your jobs.
 

== Shows free resources : sinfo_t_idle ==
The Slurm command sinfo is used to view partition and node information for a system running Slurm. It incorporates down time, reservations, and node state information in determining the available backfill window. The sinfo command can only be used by the administrator.
 
SCC has prepared a special script (sinfo_t_idle) to find out how many processors are available for immediate use on the system. It is anticipated that users will use this information to submit jobs that meet these criteria and thus obtain quick job turnaround times.
 
 
=== Access ===
By default, this command can be used by any user or administrator.
 
 
=== Example ===
* The following command displays what resources are available for immediate use for the whole partition.
<pre>$ sinfo_t_idle
Partition dev_multiple : 8 nodes idle
Partition multiple : 332 nodes idle
Partition dev_single : 4 nodes idle
Partition single : 76 nodes idle
Partition long : 80 nodes idle
Partition fat : 5 nodes idle
Partition dev_special : 342 nodes idle
Partition special : 342 nodes idle
Partition dev_multiple_e: 7 nodes idle
Partition multiple_e : 335 nodes idle
Partition gpu_4 : 12 nodes idle
Partition gpu_8 : 6 nodes idle
</pre>
* For the above example jobs in all partitions can be run immediately.
 

== Detailed job information : scontrol show job ==
scontrol show job displays detailed job state information and diagnostic output for all or a specified job of yours. Detailed information is available for active, pending and recently completed jobs. The command scontrol is explained in detail on the webpage https://slurm.schedmd.com/scontrol.html or via manpage (man scontrol).
 
Display the state of all your jobs in normal mode: scontrol show job
 
Display the state of a job with <jobid> in normal mode: scontrol show job <jobid>
 
 
=== Access ===
* End users can use scontrol show job to view the status of their '''own jobs''' only.
 

=== Arguments ===
{| width=750px class="wikitable"
|-
! Option !! Default !! Description !! Example
|- style="vertical-align:top;"
|- style="width:12%;"
| -d
| (n/a)
| Detailed mode
| Example: Display the state with jobid 18089884 in detailed mode. <pre>scontrol -d show job 18089884</pre>
|}
 
 

=== Scontrol show job Example ===
Here is an example from bwUniCluster 2.0.
<pre>
squeue # show my own jobs (here the userid is replaced!)
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
18089884 multiple CPV.sbat bq0742 R 33:44 2 uc2n[165-166]

$
$ # now, see what's up with my pending job with jobid 18089884
$
$ scontrol show job 18089884

JobId=18089884 JobName=CPV.sbatch
UserId=bq0742(8946) GroupId=scc(12345) MCS_label=N/A
Priority=3 Nice=0 Account=kit QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:35:06 TimeLimit=02:00:00 TimeMin=N/A
SubmitTime=2020-03-16T14:14:54 EligibleTime=2020-03-16T14:14:54
AccrueTime=2020-03-16T14:14:54
StartTime=2020-03-16T15:12:51 EndTime=2020-03-16T17:12:51 Deadline=N/A
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2020-03-16T15:12:51
Partition=multiple AllocNode:Sid=uc2n995:5064
ReqNodeList=(null) ExcNodeList=(null)
NodeList=uc2n[165-166]
BatchHost=uc2n165
NumNodes=2 NumCPUs=160 NumTasks=80 CPUs/Task=1 ReqB:S:C:T=0:0:*:1
TRES=cpu=160,mem=96320M,node=2,billing=160
Socks/Node=* NtasksPerN:B:S:C=40:0:*:1 CoreSpec=*
MinCPUsNode=40 MinMemoryCPU=1204M MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
Command=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/CPV.sbatch
WorkDir=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin
StdErr=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/slurm-18089884.out
StdIn=/dev/null
StdOut=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/slurm-18089884.out
Power=
MailUser=(null) MailType=NONE
</pre>
 
You can use standard Linux pipe commands to filter the very detailed scontrol show job output.
* In which state the job is?
<pre>$ scontrol show job 18089884 | grep -i State
JobState=COMPLETED Reason=None Dependency=(null)
</pre>
 

== Cancel Slurm Jobs ==
The scancel command is used to cancel jobs. The command scancel is explained in detail on the webpage https://slurm.schedmd.com/scancel.html or via manpage (man scancel).
 
 
=== Canceling own jobs : scancel ===
scancel is used to signal or cancel jobs, job arrays or job steps. The command is:
<pre>
$ scancel [-i] <job-id>
$ scancel -t <job_state_name>
</pre>

 
{| width=750px class="wikitable"
! Flag !! Default !! Description !! Example
|- style="vertical-align:top;"
| -i, --interactive
| (n/a)
| Interactive mode.
| Cancel the job 987654 interactively. <pre> scancel -i 987654 </pre>
|-
| -t, --state
| (n/a)
| Restrict the scancel operation to jobs in a certain state. "job_state_name" may have a value of either "PENDING", "RUNNING" or "SUSPENDED".
| Cancel all jobs in state "PENDING". <pre> scancel -t "PENDING" </pre>
|}
 

= Resource Managers =
=== Batch Job (Slurm) Variables ===
The following environment variables of Slurm are added to your environment once your job has started
(only an excerpt of the most important ones).
{| width=750px class="wikitable"
! Environment !! Brief explanation
|-
| SLURM_JOB_CPUS_PER_NODE
| Number of processes per node dedicated to the job
|-
| SLURM_JOB_NODELIST
| List of nodes dedicated to the job
|-
| SLURM_JOB_NUM_NODES
| Number of nodes dedicated to the job
|-
| SLURM_MEM_PER_NODE
| Memory per node dedicated to the job
|-
| SLURM_NPROCS
| Total number of processes dedicated to the job
|-
| SLURM_CLUSTER_NAME
| Name of the cluster executing the job
|-
| SLURM_CPUS_PER_TASK
| Number of CPUs requested per task
|-
| SLURM_JOB_ACCOUNT
| Account name
|-
| SLURM_JOB_ID
| Job ID
|-
| SLURM_JOB_NAME
| Job Name
|-
| SLURM_JOB_PARTITION
| Partition/queue running the job
|-
| SLURM_JOB_UID
| User ID of the job's owner
|-
| SLURM_SUBMIT_DIR
| Job submit folder. The directory from which msub was invoked.
|-
| SLURM_JOB_USER
| User name of the job's owner
|-
| SLURM_RESTART_COUNT
| Number of times job has restarted
|-
| SLURM_PROCID
| Task ID (MPI rank)
|-
| SLURM_NTASKS
| The total number of tasks available for the job
|-
| SLURM_STEP_ID
| Job step ID
|-
| SLURM_STEP_NUM_TASKS
| Task count (number of PI ranks)
|-
| SLURM_JOB_CONSTRAINT
| Job constraints
|}
See also:
* [https://slurm.schedmd.com/sbatch.html#lbAI Slurm input and output environment variables]
 

=== Job Exit Codes ===
A job's exit code (also known as exit status, return code and completion code) is captured by SLURM and saved as part of the job record.
 
Any non-zero exit code will be assumed to be a job failure and will result in a Job State of FAILED with a reason of "NonZeroExitCode".
 
The exit code is an 8 bit unsigned number ranging between 0 and 255. While it is possible for a job to return a negative exit code, SLURM will display it as an unsigned value in the 0 - 255 range.
 
 
==== Displaying Exit Codes and Signals ====
SLURM displays a job's exit code in the output of the '''scontrol show job''' and the sview utility.
 
When a signal was responsible for a job or step's termination, the signal number will be displayed after the exit code, delineated by a colon(:).
 
 
==== Submitting Termination Signal ====
Here is an example, how to 'save' a Slurm termination signal in a typical jobscript.
<source lang="bash">
[...]
exit_code=$?
mpirun -np <#cores> <EXE_BIN_DIR>/<executable> ... (options) 2>&1
[ "$exit_code" -eq 0 ] && echo "all clean..." || \
echo "Executable <EXE_BIN_DIR>/<executable> finished with exit code ${$exit_code}"
[...]
</source>
* Do not use ''''time'''' mpirun! The exit code will be the one submitted by the first (time) program.
* You do not need an '''exit $exit_code''' in the scripts.
 
 
----
[[Category:bwUniCluster 2.0|bwUniCluster 2.0]]
[[#top|Back to top]]

BwUniCluster2.0/Slurm

2020-04-02T11:05:29Z

H Haefner: /* OpenMPI with Multithreading */

<div id="top"></div>
= Slurm HPC Workload Manager =
== Specification ==
Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters. Slurm requires no kernel modifications for its operation and is relatively self-contained. As a cluster workload manager, Slurm has three key functions. First, it allocates exclusive and/or non-exclusive access to resources (compute nodes) to users for some duration of time so they can perform work. Second, it provides a framework for starting, executing, and monitoring work (normally a parallel job) on the set of allocated nodes. Finally, it arbitrates contention for resources by managing a queue of pending work.
 
 
Any kind of calculation on the compute nodes of [[bwUniCluster 2.0|bwUniCluster 2.0]] requires the user to define calculations as a sequence of commands or single command together with required run time, number of CPU cores and main memory and submit all, i.e., the '''batch job''', to a resource and workload managing software. bwUniCluster 2.0 has installed the workload managing software Slurm. Therefore any job submission by the user is to be executed by commands of the Slurm software. Slurm queues and runs user jobs based on fair sharing policies.
 
 

== Slurm Commands (excerpt) ==
Some of the most used Slurm commands for non-administrators working on bwUniCluster 2.0.
{| width=750px class="wikitable"
! Slurm commands !! Brief explanation
|-
| [[#Job Submission : sbatch|sbatch]] || Submits a job and queues it in an input queue [[https://slurm.schedmd.com/sbatch.html sbatch]]
|-
| [[#Detailed job information : scontrol show job|scontrol show job]] || Displays detailed job state information [[https://slurm.schedmd.com/scontrol.html scontrol]]
|-
| [[#List of your submitted jo/bs : squeue|squeue]] || Displays information about active, eligible, blocked, and/or recently completed jobs [[https://slurm.schedmd.com/squeue.html squeue]]
|-
| [[#Start time of job or resources : squeue|squeue --start]] || Returns start time of submitted job or requested resources [[https://slurm.schedmd.com/squeue.html squeue]]
|-
| [[#Shows free resources : sinfo_t_idle|sinfo_t_idle]] || Shows what resources are available for immediate use [[https://slurm.schedmd.com/sinfo.html sinfo]]
|-
| [[#Canceling own jobs : scancel|scancel]] || Cancels a job (obsoleted!) [[https://slurm.schedmd.com/scancel.html scancel]]
|}
If your job was submitted to the "multiple" queue you can log into the allocated nodes via SSH as soon as the job is running.

 
* [https://slurm.schedmd.com/tutorials.html Slurm Tutorials]
* [https://slurm.schedmd.com/pdfs/summary.pdf Slurm command/option summary (2 pages)]
* [https://slurm.schedmd.com/man_index.html Slurm Commands]
 

== Job Submission : sbatch ==
Batch jobs are submitted by using the command '''sbatch'''. The main purpose of the '''sbatch''' command is to specify the resources that are needed to run the job. '''sbatch''' will then queue the batch job. However, starting of batch job depends on the availability of the requested resources and the fair sharing value.
 
 
=== sbatch Command Parameters ===
The syntax and use of '''sbatch''' can be displayed via:
<pre>
$ man sbatch
</pre>
'''sbatch''' options can be used from the command line or in your job script.
{| width=750px class="wikitable"
! colspan="3" | sbatch Options
|-
! Command line
! Script
! Purpose
|- style="vertical-align:top;"
| -t ''time'' or --time=''time''
| #SBATCH --time=''time''
| Wall clock time limit. 
|-
|- style="vertical-align:top;"
| -N ''count'' or --nodes=''count''
| #SBATCH --nodes=''count''
| Number of nodes to be used.
|- style="vertical-align:top;"
| -n ''count'' or --ntasks=''count''
| #SBATCH --ntasks=''count''
| Number of tasks to be launched.
|-
|- style="vertical-align:top;"
| --ntasks-per-node=''count''
| #SBATCH --ntasks-per-node=''count''
| Maximum count (<= 28 and <= 40 resp.) of tasks per node. (Replaces the option ppn of MOAB.)
|-
|- style="vertical-align:top;"
| -c ''count'' or --cpus-per-task=''count''
| #SBATCH --cpus-per-task=''count''
| Number of CPUs required per (MPI-)task.
|-
|- style="vertical-align:top;"
| --mem=''value_in_MB''
| #SBATCH --mem=''value_in_MB''
| Memory in MegaByte per node. (Default value is 128000 and 96000 MB resp., i.e. you should omit the setting of this option.)
|-
|- style="vertical-align:top;"
| --mem-per-cpu=''value_in_MB''
| #SBATCH --mem-per-cpu=''value_in_MB''
| Minimum Memory required per allocated CPU. (Replaces the option pmem of MOAB. You should omit the setting of this option.)
|-
|- style="vertical-align:top;"
| --mail-type=''type''
| #SBATCH --mail-type=''type''
| Notify user by email when certain event types occur. Valid type values are NONE, BEGIN, END, FAIL, REQUEUE, ALL.
|-
|- style="vertical-align:top;"
| --mail-user=''mail-address''
| #SBATCH --mail-user=''mail-address''
| The specified mail-address receives email notification of state changes as defined by --mail-type.
|-
|- style="vertical-align:top;"
| --output=''name''
| #SBATCH --output=''name''
| File in which job output is stored.
|-
|- style="vertical-align:top;"
| --error=''name''
| #SBATCH --error=''name''
| File in which job error messages are stored.
|-
|- style="vertical-align:top;"
| -J ''name'' or --job-name=''name''
| #SBATCH --job-name=''name''
| Job name.
|-
|- style="vertical-align:top;"
| --export=[ALL,] ''env-variables''
| #SBATCH --export=[ALL,] ''env-variables''
| Identifies which environment variables from the submission environment are propagated to the launched application. Default is ALL. If adding to the submission environment instead of replacing it is intended, the argument ALL must be added.
|-
|- style="vertical-align:top;"
| -A ''group-name'' or --account=''group-name''
| #SBATCH --account=''group-name''
| Change resources used by this job to specified group. You may need this option if your account is assigned to more than one group. By command "scontrol show job" the project group the job is accounted on can be seen behind "Account=".
|-
|- style="vertical-align:top;"
| -p ''queue-name'' or --partition=''queue-name''
| #SBATCH --partition=''queue-name''
| Request a specific queue for the resource allocation.
|-
|- style="vertical-align:top;"
| -C ''LSDF'' or --constraint=''LSDF''
| #SBATCH --constraint=LSDF
| Job constraint LSDF Filesystems.
|-
|- style="vertical-align:top;"
| -C ''BEEOND'' or --constraint=''BEEOND''
| #SBATCH --constraint=BEEOND
| Job constraint BeeOND file system.
|-
|}
 

==== sbatch --partition ''queues'' ====
Queue classes define maximum resources such as walltime, nodes and processes per node and queue of the compute system. Details can be found here:
* [[BwUniCluster_2.0_Batch_Queues#sbatch_-p_queue|bwUniCluster 2.0 queue settings]]
 

=== sbatch Examples ===
==== Serial Programs ====
To submit a serial job that runs the script '''job.sh''' and that requires 5000 MB of main memory and 10 minutes of wall clock time

a) execute:
<pre>
$ sbatch -p dev_single -n 1 -t 10:00 --mem=5000 job.sh
</pre>
or
b) add after the initial line of your script '''job.sh''' the lines (here with a high memory request):
<source lang="bash">
#SBATCH --ntasks=1
#SBATCH --time=10
#SBATCH --mem=200gb
#SBATCH --job-name=simple
</source>
and execute the modified script with the command line option ''--partition=fat'' (with ''--partition=(dev_)single'' maximum ''--mem=96gb'' is possible):
<pre>
$ sbatch --partition=fat job.sh
</pre>
Note, that sbatch command line options overrule script options.
 
 

==== Multithreaded Programs ====
Multithreaded programs operate faster than serial programs on CPUs with multiple cores. 
Moreover, multiple threads of one process share resources such as memory.
 
For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) a number of threads is defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).
 
'''Because hyperthreading is switched on on bwUniCluster 2.0, the option --cpus-per-task (-c) must be set to 2*n, if you want to use n threads.'''
 
To submit a batch job called ''OpenMP_Test'' that runs a 40-fold threaded program ''omp_exe'' which requires 6000 MByte of total physical memory and total wall clock time of 40 minutes:
 
a) execute:
<pre>
$ sbatch -p single --export=ALL,OMP_NUM_THREADS=40 -J OpenMP_Test -N 1 -c 80 -t 40 --mem=6000 ./omp_exe
</pre>
or
-->
* generate the script '''job_omp.sh''' containing the following lines:
<source lang="bash">
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --cpus-per-task=80
#SBATCH --time=40:00
#SBATCH --mem=6000mb
#SBATCH --export=ALL,EXECUTABLE=./omp_exe
#SBATCH -J OpenMP_Test

#Usually you should set
export KMP_AFFINITY=compact,1,0
#export KMP_AFFINITY=verbose,compact,1,0 prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

export OMP_NUM_THREADS=$((${SLURM_JOB_CPUS_PER_NODE}/2))
echo "Executable ${EXECUTABLE} running on ${SLURM_JOB_CPUS_PER_NODE} cores with ${OMP_NUM_THREADS} threads"
startexe=${EXECUTABLE}
echo $startexe
exec $startexe
</source>
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores and, if necessary, replace <placeholder> with the required modulefile to enable the OpenMP environment and execute the script '''job_omp.sh''' adding the queue class ''single'' as sbatch option:
<pre>
$ sbatch -p single job_omp.sh
</pre>
Note, that sbatch command line options overrule script options, e.g.,
<pre>
$ sbatch --partition=single --mem=200 job_omp.sh
</pre>
overwrites the script setting of 6000 MByte with 200 MByte.
 
 

==== MPI Parallel Programs ====
MPI parallel programs run faster than serial programs on multi CPU and multi core systems. N-fold spawned processes of the MPI program, i.e., '''MPI tasks''', run simultaneously and communicate via the Message Passing Interface (MPI) paradigm. MPI tasks do not share memory but can be spawned over different nodes.
 
Multiple MPI tasks must be launched via '''mpirun''', e.g. 4 MPI tasks of ''my_par_program'':
<pre>
$ mpirun -n 4 my_par_program
</pre>
This command runs 4 MPI tasks of ''my_par_program'' on the node you are logged in.
To run this command with a loaded Intel MPI the environment variable I_MPI_HYDRA_BOOTSTRAP must be unset ( --> $ unset I_MPI_HYDRA_BOOTSTRAP).

Running MPI parallel programs in a batch job the interactive environment - particularly the loaded modules - will also be set in the batch job. If you want to set a defined module environment in your batch job you have to purge all modules before setting the wished modules.
 
 
===== OpenMPI =====

If you want to run jobs on batch nodes, generate a wrapper script ''job_ompi.sh'' for '''OpenMPI''' containing the following lines:
<source lang="bash">
#!/bin/bash
# Use when a defined module environment related to OpenMPI is wished
module load mpi/openmpi/<placeholder_for_version>
mpirun --bind-to core --map-by core -report-bindings my_par_program
</source>
'''Attention:''' Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since Slurm instructs mpirun about number of processes and node hostnames. Use '''ALWAYS''' the MPI options '''''--bind-to core''''' and '''''--map-by core|socket|node'''''. Please type ''mpirun --help'' for an explanation of the meaning of the different options of mpirun option ''--map-by''.
 
Considering 4 OpenMPI tasks on a single node, each requiring 2000 MByte, and running for 1 hour, execute:
<pre>
$ sbatch -p single -N 1 -n 4 --mem-per-cpu=2000 --time=01:00:00 ./job_ompi.sh
</pre>
 

===== Intel MPI =====

Generate a wrapper script for '''Intel MPI''', ''job_impi.sh'' containing the following lines:
<source lang="bash">
#!/bin/bash
# Use when a defined module environment related to Intel MPI is wished
module load mpi/impi/<placeholder_for_version>
mpiexec.hydra -bootstrap slurm my_par_program
</source>
'''Attention:''' 
Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since Slurm instructs mpirun about number of processes and node hostnames.
 
Launching and running 200 Intel MPI tasks on 5 nodes, each requiring 80 GByte, and running for 5 hours, execute:
<pre>
$ sbatch --partition=multiple -N 5 --ntasks-per-node=40 --mem=80gb -t 300 ./job_impi.sh
</pre>
 
If you want to use 128 or more nodes, you must also set the environment variable as follows: 
export I_MPI_HYDRA_BRANCH_COUNT=-1
 
If you want to use the options perhost, ppn or rr, you must additionally set the environment variable I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=off.
 
 

==== Multithreaded + MPI parallel Programs ====
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes. '''Because hyperthreading is switched on on bwUniCluster 2.0, the option --cpus-per-task (-c) must be set to 2*n, if you want to use n threads.'''
 
 
===== OpenMPI with Multithreading =====
Multiple MPI tasks using '''OpenMPI''' must be launched by the MPI parallel program '''mpirun'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).
 
'''For OpenMPI''' a job-script to submit a batch job called ''job_ompi_omp.sh'' that runs a MPI program with 4 tasks and a 28-fold threaded program ''ompi_omp_program'' requiring 3000 MByte of physical memory per thread (using 28 threads per MPI task you will get 28*3000 MByte = 84000 MByte per MPI task) and total wall clock time of 3 hours looks like:

<source lang="bash">
#!/bin/bash
#SBATCH --ntasks=4
#SBATCH --cpus-per-task=56
#SBATCH --time=03:00:00
#SBATCH --mem=83gb # 84000 MB = 84000/1024 GB = 82.1 GB
#SBATCH --export=ALL,MPI_MODULE=mpi/openmpi/3.1,EXECUTABLE=./ompi_omp_program
#SBATCH --output="parprog_hybrid_%j.out"

# Use when a defined module environment related to OpenMPI is wished
module load ${MPI_MODULE}
export OMP_NUM_THREADS=$((${SLURM_CPUS_PER_TASK}/2))
export MPIRUN_OPTIONS="--bind-to core --map-by socket:PE=${OMP_NUM_THREADS} -report-bindings"
export NUM_CORES=${SLURM_NTASKS}*${OMP_NUM_THREADS}
echo "${EXECUTABLE} running on ${NUM_CORES} cores with ${SLURM_NTASKS} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpirun -n ${SLURM_NTASKS} ${MPIRUN_OPTIONS} ${EXECUTABLE}"
echo $startexe
exec $startexe
</source>
Execute the script '''job_ompi_omp.sh''' by command sbatch:
<pre>
$ sbatch -p multiple_e ./job_ompi_omp.sh
</pre>
* With the mpirun option ''--bind-to core'' MPI tasks and OpenMP threads are bound to physical cores.
* With the option ''--map-by node:PE=<value>'' (neighbored) MPI tasks will be attached to different nodes and each MPI task is bound to the first core of a node. <value> must be set to ${OMP_NUM_THREADS}.
* The option ''-report-bindings'' shows the bindings between MPI tasks and physical cores.
* The mpirun-options '''--bind-to core''', '''--map-by socket|...|node:PE=<value>''' should always be used when running a multithreaded MPI program.
 

===== Intel MPI with Multithreading =====
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes.

Multiple Intel MPI tasks must be launched by the MPI parallel program '''mpiexec.hydra'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).

'''For Intel MPI''' a job-script to submit a batch job called ''job_impi_omp.sh'' that runs a Intel MPI program with 10 tasks and a 40-fold threaded program ''impi_omp_program'' requiring 96000 MByte of total physical memory per task and total wall clock time of 1 hours looks like:


<source lang="bash">
#!/bin/bash
#SBATCH --ntasks=10
#SBATCH --cpus-per-task=40
#SBATCH --time=60
#SBATCH --mem=96000
#SBATCH --export=ALL,MPI_MODULE=mpi/impi,EXE=./impi_omp_program
#SBATCH --output="parprog_impi_omp_%j.out"

#If using more than one MPI task per node please set
export KMP_AFFINITY=compact,1,0
#export KMP_AFFINITY=verbose,scatter prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

# Use when a defined module environment related to Intel MPI is wished
module load ${MPI_MODULE}
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
export MPIRUN_OPTIONS="-binding "domain=omp:compact" -print-rank-map -envall"
export NUM_PROCS=eval(${SLURM_NTASKS}*${OMP_NUM_THREADS})
echo "${EXE} running on ${NUM_PROCS} cores with ${SLURM_NTASKS} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpiexec.hydra -bootstrap slurm ${MPIRUN_OPTIONS} -n ${SLURM_NTASKS} ${EXE}"
echo $startexe
exec $startexe
</source>
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores. If you only run one MPI task per node please set KMP_AFFINITY=compact,1,0.
 
If you want to use 128 or more nodes, you must also set the environment variable as follows: 
export I_MPI_HYDRA_BRANCH_COUNT=-1
 
If you want to use the options perhost, ppn or rr, you must additionally set the environment variable I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=off.
 
 
Execute the script '''job_impi_omp.sh''' by command sbatch:
<pre>
$ sbatch -p multiple ./job_impi_omp.sh
</pre>
 
The mpirun option ''-print-rank-map'' shows the bindings between MPI tasks and nodes (not very beneficial). The option ''-binding'' binds MPI tasks (processes) to a particular processor; ''domain=omp'' means that the domain size is determined by the number of threads. If you would choose 2 MPI tasks per node, you should choose ''-binding "cell=unit;map=bunch"''; this binding maps one MPI process to each socket.
 
 

==== Chain jobs ====
The CPU time requirements of many applications exceed the limits of the job classes. In those situations it is recommended to solve the problem by a job chain. A job chain is a sequence of jobs where each job automatically starts its successor.
<source lang="bash">
#!/bin/bash
####################################
## simple Slurm submitter script to setup ##
## a chain of jobs using Slurm ##
####################################
## ver. : 2018-11-27, KIT, SCC

## Define maximum number of jobs via positional parameter 1, default is 5
max_nojob=${1:-5}

## Define your jobscript (e.g. "~/chain_job.sh")
chain_link_job=${PWD}/chain_job.sh

## Define type of dependency via positional parameter 2, default is 'afterok'
dep_type="${2:-afterok}"
## -> List of all dependencies:
## https://slurm.schedmd.com/sbatch.html

myloop_counter=1
## Submit loop
while [ ${myloop_counter} -le ${max_nojob} ] ; do
##
## Differ msub_opt depending on chain link number
if [ ${myloop_counter} -eq 1 ] ; then
slurm_opt=""
else
slurm_opt="-d ${dep_type}:${jobID}"
fi
##
## Print current iteration number and sbatch command
echo "Chain job iteration = ${myloop_counter}"
echo " sbatch --export=myloop_counter=${myloop_counter} ${slurm_opt} ${chain_link_job}"
## Store job ID for next iteration by storing output of sbatch command with empty lines
jobID=$(sbatch -p <queue> --export=ALL,myloop_counter=${myloop_counter} ${slurm_opt} ${chain_link_job} 2>&1 | sed 's/[S,a-z]* //g')
##
## Check if ERROR occured
if [[ "${jobID}" =~ "ERROR" ]] ; then
echo " -> submission failed!" ; exit 1
else
echo " -> job number = ${jobID}"
fi
##
## Increase counter
let myloop_counter+=1
done
</source>
 

==== GPU jobs ====

The nodes in the gpu_4 and gpu_8 queues have 4 or 8 NVIDIA Tesla V100 GPUs. Just submitting a job to these queues is not enough to also allocate one or more GPUs, you have to do so using the "--gres=gpu" parameter. You have to specifiy how many GPUs your job needs, e.g. "--gres=gpu:2" will request two GPUs.

The GPU nodes are shared between multiple jobs if the jobs don't request all the GPUs in a node and there are enough ressources to run more than one job. The individual GPUs are always bound to a single job and will not be shared between different jobs.

a) add after the initial line of your script job.sh the line including the
information about the LSDF Online Storage usage: #SBATCH --constraint=LSDF
<pre>
#!/bin/bash
#SBATCH --ntasks=40
#SBATCH --time=02:00:00
#SBATCH --mem=4000
#SBATCH --gres=gpu:2
</pre>

or b) execute:
<pre>
$ sbatch -p <queue> -n 40 -t 02:00:00 --mem 4000 --gres=gpu:2 job.sh
</pre>
 
If you start an interactive session on of the GPU nodes, you can use the "nvidia-smi" command to list the GPUs allocated to your job:
<pre>
$ nvidia-smi
Sun Mar 29 15:20:05 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2... Off | 00000000:3A:00.0 Off | 0 |
| N/A 29C P0 39W / 300W | 9MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla V100-SXM2... Off | 00000000:3B:00.0 Off | 0 |
| N/A 30C P0 41W / 300W | 8MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 14228 G /usr/bin/X 8MiB |
| 1 14228 G /usr/bin/X 8MiB |
+-----------------------------------------------------------------------------+
</pre>

 
 

==== LSDF Online Storage ====
On bwUniCluster 2.0 you can use for special cases the LSDF Online Storage on the HPC cluster nodes. Please request for this service separately ([https://www.lsdf.kit.edu/os/storagerequest/: LSDF Storage Request]).
To mount the LSDF Online Storage on the compute nodes during the job runtime the
the constraint flag "LSDF" has to be set.

a) add after the initial line of your script job.sh the line including the
information about the LSDF Online Storage usage: #SBATCH --constraint=LSDF
<pre>
#!/bin/bash
#SBATCH --ntasks=1
#SBATCH --time=120
#SBATCH --mem=200
#SBATCH --constraint=LSDF
</pre>

or b) execute:
<pre>
$ sbatch -p <queue> -n 1 -t 2:00:00 --mem 200 job.sh -C LSDF
</pre>
 
For the usage of the LSDF Online Storage
the following environment variables are available: $LSDF, $LSDFPROJECTS, $LSDFHOME.
 
 

====BeeOND (BeeGFS On-Demand)====

BeeOND instances are integrated into the prolog and epilog script of the cluster batch system, Slurm. It can be used on the compute nodes during the job runtime with the constraint flag "BEEOND" ([[BwUniCluster_2.0_Slurm_common_Features#sbatch_Command_Parameters|Slurm Command Parameters]])
<pre>
#!/bin/bash
#SBATCH ...
#SBATCH --constraint=BEEOND
</pre>

After your job has started you can find the private on-demand file system in '''/mnt/odfs/$SLURM_JOB_ID''' directory. The mountpoint comes with three pre-configured directories:
<pre>
#for small files (stripe count = 1)
/mnt/odfs/$SLURM_JOB_ID/stripe_1
#stripe count = 4
/mnt/odfs/$SLURM_JOB_ID/stripe_default
#stripe count = 8, 16 or 32, use this directories for medium sized and large files or when using MPI-IO
/mnt/odfs/$SLURM_JOB_ID/stripe_8, /mnt/odfs/$SLURM_JOB_ID/stripe_16 or /mnt/odfs/$SLURM_JOB_ID/stripe_32
</pre>

If you request less nodes than stripe count, the stripe count will be max number of nodes, e.g., You only request 8 nodes , so the directory with stripe count 16 is basically only with a stripe count 8.

The capacity of the private file system depends on the number of nodes. For each node you get 250Gbyte.

!!! Be careful when creating large files, use always the directory with the max stripe count for large files.
If you create large files use a higher stripe count. For example, if your largest file is 1.1 Tb, then you have to use a stripe count larger>4 (4 x 250GB).

If you request 100 nodes for your job, the private file system is 100 * 250 Gbyte ~ 25 Tbyte (approx) capacity.

'''Recommendation:'''

The private file system is using its own metadata server. This metadata server is started on the first nodes. Depending on your application, the metadata server is consuming decent amount of CPU power. Probably adding a extra node to your job could improve the usability of the on-demand file system. Start your application with the MPI option:
<pre>
mpirun -nolocal myapplication
</pre>
With the -nolocal option the node where mpirun is initiated is not used for your application. This node is fully available for the meta data server of your requested on-demand file system.

Example job script:
<pre>
#!/bin/bash
#very simple example on how to use a private on-demand file system
#SBATCH -N 10
#SBATCH --constraint=BEEOND

#create a workspace
ws_allocate myresults-$SLURM_JOB_ID 90
RESULTDIR=`ws_find myresults-$SLURM_JOB_ID`

#Set ENV variable to on-demand file system
ODFSDIR=/mnt/odfs/$SLURM_JOB_ID/stripe_16/

#start application and write results to on-demand file system
mpirun -nolocal myapplication -o $ODFSDIR/results

#Copy back data after your job application end
rsync -av $ODFSDIR/results $RESULTDIR
</pre>
 
 

== Start time of job or resources : squeue --start ==
The command can be used by any user to displays the estimated start time of a job based a number of analysis types based on historical usage, earliest available reservable resources, and priority based backlog. The command squeue is explained in detail on the webpage https://slurm.schedmd.com/squeue.html or via manpage (man squeue).
 
 
=== Access ===
By default, this command can be run by '''any user'''.
 
 

== List of your submitted jobs : squeue ==
Displays information about YOUR active, pending and/or recently completed jobs. The command displays all own active and pending jobs. The command squeue is explained in detail on the webpage https://slurm.schedmd.com/squeue.html or via manpage (man squeue).
 
 
=== Access ===
By default, this command can be run by any user.
 
 

=== Flags ===
{| width=750px class="wikitable"
|-
! Flag !! Description
|-
| -l, --long
| Report more of the available information for the selected jobs or job steps, subject to any constraints specified.
|}
 

=== Examples ===
''squeue'' example on bwUniCluster 2.0 (Only your own jobs are displayed!).
<pre>
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
18088744 single CPV.sbat ab1234 PD 0:00 1 (Priority)
18098414 multiple CPV.sbat ab1234 PD 0:00 2 (Priority)
18090089 multiple CPV.sbat ab1234 R 2:27 2 uc2n[127-128]
$ squeue -l
JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON)
18088654 single CPV.sbat ab1234 COMPLETI 4:29 2:00:00 1 uc2n374
18088785 single CPV.sbat ab1234 PENDING 0:00 2:00:00 1 (Priority)
18098414 multiple CPV.sbat ab1234 PENDING 0:00 2:00:00 2 (Priority)
18088683 single CPV.sbat ab1234 RUNNING 0:14 2:00:00 1 uc2n413
</pre>
* The output of ''squeue'' shows how many jobs of yours are running or pending and how many nodes are in use by your jobs.
 

== Shows free resources : sinfo_t_idle ==
The Slurm command sinfo is used to view partition and node information for a system running Slurm. It incorporates down time, reservations, and node state information in determining the available backfill window. The sinfo command can only be used by the administrator.
 
SCC has prepared a special script (sinfo_t_idle) to find out how many processors are available for immediate use on the system. It is anticipated that users will use this information to submit jobs that meet these criteria and thus obtain quick job turnaround times.
 
 
=== Access ===
By default, this command can be used by any user or administrator.
 
 
=== Example ===
* The following command displays what resources are available for immediate use for the whole partition.
<pre>$ sinfo_t_idle
Partition dev_multiple : 8 nodes idle
Partition multiple : 332 nodes idle
Partition dev_single : 4 nodes idle
Partition single : 76 nodes idle
Partition long : 80 nodes idle
Partition fat : 5 nodes idle
Partition dev_special : 342 nodes idle
Partition special : 342 nodes idle
Partition dev_multiple_e: 7 nodes idle
Partition multiple_e : 335 nodes idle
Partition gpu_4 : 12 nodes idle
Partition gpu_8 : 6 nodes idle
</pre>
* For the above example jobs in all partitions can be run immediately.
 

== Detailed job information : scontrol show job ==
scontrol show job displays detailed job state information and diagnostic output for all or a specified job of yours. Detailed information is available for active, pending and recently completed jobs. The command scontrol is explained in detail on the webpage https://slurm.schedmd.com/scontrol.html or via manpage (man scontrol).
 
Display the state of all your jobs in normal mode: scontrol show job
 
Display the state of a job with <jobid> in normal mode: scontrol show job <jobid>
 
 
=== Access ===
* End users can use scontrol show job to view the status of their '''own jobs''' only.
 

=== Arguments ===
{| width=750px class="wikitable"
|-
! Option !! Default !! Description !! Example
|- style="vertical-align:top;"
|- style="width:12%;"
| -d
| (n/a)
| Detailed mode
| Example: Display the state with jobid 18089884 in detailed mode. <pre>scontrol -d show job 18089884</pre>
|}
 
 

=== Scontrol show job Example ===
Here is an example from bwUniCluster 2.0.
<pre>
squeue # show my own jobs (here the userid is replaced!)
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
18089884 multiple CPV.sbat bq0742 R 33:44 2 uc2n[165-166]

$
$ # now, see what's up with my pending job with jobid 18089884
$
$ scontrol show job 18089884

JobId=18089884 JobName=CPV.sbatch
UserId=bq0742(8946) GroupId=scc(12345) MCS_label=N/A
Priority=3 Nice=0 Account=kit QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:35:06 TimeLimit=02:00:00 TimeMin=N/A
SubmitTime=2020-03-16T14:14:54 EligibleTime=2020-03-16T14:14:54
AccrueTime=2020-03-16T14:14:54
StartTime=2020-03-16T15:12:51 EndTime=2020-03-16T17:12:51 Deadline=N/A
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2020-03-16T15:12:51
Partition=multiple AllocNode:Sid=uc2n995:5064
ReqNodeList=(null) ExcNodeList=(null)
NodeList=uc2n[165-166]
BatchHost=uc2n165
NumNodes=2 NumCPUs=160 NumTasks=80 CPUs/Task=1 ReqB:S:C:T=0:0:*:1
TRES=cpu=160,mem=96320M,node=2,billing=160
Socks/Node=* NtasksPerN:B:S:C=40:0:*:1 CoreSpec=*
MinCPUsNode=40 MinMemoryCPU=1204M MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
Command=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/CPV.sbatch
WorkDir=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin
StdErr=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/slurm-18089884.out
StdIn=/dev/null
StdOut=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/slurm-18089884.out
Power=
MailUser=(null) MailType=NONE
</pre>
 
You can use standard Linux pipe commands to filter the very detailed scontrol show job output.
* In which state the job is?
<pre>$ scontrol show job 18089884 | grep -i State
JobState=COMPLETED Reason=None Dependency=(null)
</pre>
 

== Cancel Slurm Jobs ==
The scancel command is used to cancel jobs. The command scancel is explained in detail on the webpage https://slurm.schedmd.com/scancel.html or via manpage (man scancel).
 
 
=== Canceling own jobs : scancel ===
scancel is used to signal or cancel jobs, job arrays or job steps. The command is:
<pre>
$ scancel [-i] <job-id>
$ scancel -t <job_state_name>
</pre>

 
{| width=750px class="wikitable"
! Flag !! Default !! Description !! Example
|- style="vertical-align:top;"
| -i, --interactive
| (n/a)
| Interactive mode.
| Cancel the job 987654 interactively. <pre> scancel -i 987654 </pre>
|-
| -t, --state
| (n/a)
| Restrict the scancel operation to jobs in a certain state. "job_state_name" may have a value of either "PENDING", "RUNNING" or "SUSPENDED".
| Cancel all jobs in state "PENDING". <pre> scancel -t "PENDING" </pre>
|}
 

= Resource Managers =
=== Batch Job (Slurm) Variables ===
The following environment variables of Slurm are added to your environment once your job has started
(only an excerpt of the most important ones).
{| width=750px class="wikitable"
! Environment !! Brief explanation
|-
| SLURM_JOB_CPUS_PER_NODE
| Number of processes per node dedicated to the job
|-
| SLURM_JOB_NODELIST
| List of nodes dedicated to the job
|-
| SLURM_JOB_NUM_NODES
| Number of nodes dedicated to the job
|-
| SLURM_MEM_PER_NODE
| Memory per node dedicated to the job
|-
| SLURM_NPROCS
| Total number of processes dedicated to the job
|-
| SLURM_CLUSTER_NAME
| Name of the cluster executing the job
|-
| SLURM_CPUS_PER_TASK
| Number of CPUs requested per task
|-
| SLURM_JOB_ACCOUNT
| Account name
|-
| SLURM_JOB_ID
| Job ID
|-
| SLURM_JOB_NAME
| Job Name
|-
| SLURM_JOB_PARTITION
| Partition/queue running the job
|-
| SLURM_JOB_UID
| User ID of the job's owner
|-
| SLURM_SUBMIT_DIR
| Job submit folder. The directory from which msub was invoked.
|-
| SLURM_JOB_USER
| User name of the job's owner
|-
| SLURM_RESTART_COUNT
| Number of times job has restarted
|-
| SLURM_PROCID
| Task ID (MPI rank)
|-
| SLURM_NTASKS
| The total number of tasks available for the job
|-
| SLURM_STEP_ID
| Job step ID
|-
| SLURM_STEP_NUM_TASKS
| Task count (number of PI ranks)
|-
| SLURM_JOB_CONSTRAINT
| Job constraints
|}
See also:
* [https://slurm.schedmd.com/sbatch.html#lbAI Slurm input and output environment variables]
 

=== Job Exit Codes ===
A job's exit code (also known as exit status, return code and completion code) is captured by SLURM and saved as part of the job record.
 
Any non-zero exit code will be assumed to be a job failure and will result in a Job State of FAILED with a reason of "NonZeroExitCode".
 
The exit code is an 8 bit unsigned number ranging between 0 and 255. While it is possible for a job to return a negative exit code, SLURM will display it as an unsigned value in the 0 - 255 range.
 
 
==== Displaying Exit Codes and Signals ====
SLURM displays a job's exit code in the output of the '''scontrol show job''' and the sview utility.
 
When a signal was responsible for a job or step's termination, the signal number will be displayed after the exit code, delineated by a colon(:).
 
 
==== Submitting Termination Signal ====
Here is an example, how to 'save' a Slurm termination signal in a typical jobscript.
<source lang="bash">
[...]
exit_code=$?
mpirun -np <#cores> <EXE_BIN_DIR>/<executable> ... (options) 2>&1
[ "$exit_code" -eq 0 ] && echo "all clean..." || \
echo "Executable <EXE_BIN_DIR>/<executable> finished with exit code ${$exit_code}"
[...]
</source>
* Do not use ''''time'''' mpirun! The exit code will be the one submitted by the first (time) program.
* You do not need an '''exit $exit_code''' in the scripts.
 
 
----
[[Category:bwUniCluster 2.0|bwUniCluster 2.0]]
[[#top|Back to top]]

BwUniCluster2.0/Slurm

2020-04-02T10:59:34Z

H Haefner: /* Multithreaded + MPI parallel Programs */

<div id="top"></div>
= Slurm HPC Workload Manager =
== Specification ==
Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters. Slurm requires no kernel modifications for its operation and is relatively self-contained. As a cluster workload manager, Slurm has three key functions. First, it allocates exclusive and/or non-exclusive access to resources (compute nodes) to users for some duration of time so they can perform work. Second, it provides a framework for starting, executing, and monitoring work (normally a parallel job) on the set of allocated nodes. Finally, it arbitrates contention for resources by managing a queue of pending work.
 
 
Any kind of calculation on the compute nodes of [[bwUniCluster 2.0|bwUniCluster 2.0]] requires the user to define calculations as a sequence of commands or single command together with required run time, number of CPU cores and main memory and submit all, i.e., the '''batch job''', to a resource and workload managing software. bwUniCluster 2.0 has installed the workload managing software Slurm. Therefore any job submission by the user is to be executed by commands of the Slurm software. Slurm queues and runs user jobs based on fair sharing policies.
 
 

== Slurm Commands (excerpt) ==
Some of the most used Slurm commands for non-administrators working on bwUniCluster 2.0.
{| width=750px class="wikitable"
! Slurm commands !! Brief explanation
|-
| [[#Job Submission : sbatch|sbatch]] || Submits a job and queues it in an input queue [[https://slurm.schedmd.com/sbatch.html sbatch]]
|-
| [[#Detailed job information : scontrol show job|scontrol show job]] || Displays detailed job state information [[https://slurm.schedmd.com/scontrol.html scontrol]]
|-
| [[#List of your submitted jo/bs : squeue|squeue]] || Displays information about active, eligible, blocked, and/or recently completed jobs [[https://slurm.schedmd.com/squeue.html squeue]]
|-
| [[#Start time of job or resources : squeue|squeue --start]] || Returns start time of submitted job or requested resources [[https://slurm.schedmd.com/squeue.html squeue]]
|-
| [[#Shows free resources : sinfo_t_idle|sinfo_t_idle]] || Shows what resources are available for immediate use [[https://slurm.schedmd.com/sinfo.html sinfo]]
|-
| [[#Canceling own jobs : scancel|scancel]] || Cancels a job (obsoleted!) [[https://slurm.schedmd.com/scancel.html scancel]]
|}
If your job was submitted to the "multiple" queue you can log into the allocated nodes via SSH as soon as the job is running.

 
* [https://slurm.schedmd.com/tutorials.html Slurm Tutorials]
* [https://slurm.schedmd.com/pdfs/summary.pdf Slurm command/option summary (2 pages)]
* [https://slurm.schedmd.com/man_index.html Slurm Commands]
 

== Job Submission : sbatch ==
Batch jobs are submitted by using the command '''sbatch'''. The main purpose of the '''sbatch''' command is to specify the resources that are needed to run the job. '''sbatch''' will then queue the batch job. However, starting of batch job depends on the availability of the requested resources and the fair sharing value.
 
 
=== sbatch Command Parameters ===
The syntax and use of '''sbatch''' can be displayed via:
<pre>
$ man sbatch
</pre>
'''sbatch''' options can be used from the command line or in your job script.
{| width=750px class="wikitable"
! colspan="3" | sbatch Options
|-
! Command line
! Script
! Purpose
|- style="vertical-align:top;"
| -t ''time'' or --time=''time''
| #SBATCH --time=''time''
| Wall clock time limit. 
|-
|- style="vertical-align:top;"
| -N ''count'' or --nodes=''count''
| #SBATCH --nodes=''count''
| Number of nodes to be used.
|- style="vertical-align:top;"
| -n ''count'' or --ntasks=''count''
| #SBATCH --ntasks=''count''
| Number of tasks to be launched.
|-
|- style="vertical-align:top;"
| --ntasks-per-node=''count''
| #SBATCH --ntasks-per-node=''count''
| Maximum count (<= 28 and <= 40 resp.) of tasks per node. (Replaces the option ppn of MOAB.)
|-
|- style="vertical-align:top;"
| -c ''count'' or --cpus-per-task=''count''
| #SBATCH --cpus-per-task=''count''
| Number of CPUs required per (MPI-)task.
|-
|- style="vertical-align:top;"
| --mem=''value_in_MB''
| #SBATCH --mem=''value_in_MB''
| Memory in MegaByte per node. (Default value is 128000 and 96000 MB resp., i.e. you should omit the setting of this option.)
|-
|- style="vertical-align:top;"
| --mem-per-cpu=''value_in_MB''
| #SBATCH --mem-per-cpu=''value_in_MB''
| Minimum Memory required per allocated CPU. (Replaces the option pmem of MOAB. You should omit the setting of this option.)
|-
|- style="vertical-align:top;"
| --mail-type=''type''
| #SBATCH --mail-type=''type''
| Notify user by email when certain event types occur. Valid type values are NONE, BEGIN, END, FAIL, REQUEUE, ALL.
|-
|- style="vertical-align:top;"
| --mail-user=''mail-address''
| #SBATCH --mail-user=''mail-address''
| The specified mail-address receives email notification of state changes as defined by --mail-type.
|-
|- style="vertical-align:top;"
| --output=''name''
| #SBATCH --output=''name''
| File in which job output is stored.
|-
|- style="vertical-align:top;"
| --error=''name''
| #SBATCH --error=''name''
| File in which job error messages are stored.
|-
|- style="vertical-align:top;"
| -J ''name'' or --job-name=''name''
| #SBATCH --job-name=''name''
| Job name.
|-
|- style="vertical-align:top;"
| --export=[ALL,] ''env-variables''
| #SBATCH --export=[ALL,] ''env-variables''
| Identifies which environment variables from the submission environment are propagated to the launched application. Default is ALL. If adding to the submission environment instead of replacing it is intended, the argument ALL must be added.
|-
|- style="vertical-align:top;"
| -A ''group-name'' or --account=''group-name''
| #SBATCH --account=''group-name''
| Change resources used by this job to specified group. You may need this option if your account is assigned to more than one group. By command "scontrol show job" the project group the job is accounted on can be seen behind "Account=".
|-
|- style="vertical-align:top;"
| -p ''queue-name'' or --partition=''queue-name''
| #SBATCH --partition=''queue-name''
| Request a specific queue for the resource allocation.
|-
|- style="vertical-align:top;"
| -C ''LSDF'' or --constraint=''LSDF''
| #SBATCH --constraint=LSDF
| Job constraint LSDF Filesystems.
|-
|- style="vertical-align:top;"
| -C ''BEEOND'' or --constraint=''BEEOND''
| #SBATCH --constraint=BEEOND
| Job constraint BeeOND file system.
|-
|}
 

==== sbatch --partition ''queues'' ====
Queue classes define maximum resources such as walltime, nodes and processes per node and queue of the compute system. Details can be found here:
* [[BwUniCluster_2.0_Batch_Queues#sbatch_-p_queue|bwUniCluster 2.0 queue settings]]
 

=== sbatch Examples ===
==== Serial Programs ====
To submit a serial job that runs the script '''job.sh''' and that requires 5000 MB of main memory and 10 minutes of wall clock time

a) execute:
<pre>
$ sbatch -p dev_single -n 1 -t 10:00 --mem=5000 job.sh
</pre>
or
b) add after the initial line of your script '''job.sh''' the lines (here with a high memory request):
<source lang="bash">
#SBATCH --ntasks=1
#SBATCH --time=10
#SBATCH --mem=200gb
#SBATCH --job-name=simple
</source>
and execute the modified script with the command line option ''--partition=fat'' (with ''--partition=(dev_)single'' maximum ''--mem=96gb'' is possible):
<pre>
$ sbatch --partition=fat job.sh
</pre>
Note, that sbatch command line options overrule script options.
 
 

==== Multithreaded Programs ====
Multithreaded programs operate faster than serial programs on CPUs with multiple cores. 
Moreover, multiple threads of one process share resources such as memory.
 
For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) a number of threads is defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).
 
'''Because hyperthreading is switched on on bwUniCluster 2.0, the option --cpus-per-task (-c) must be set to 2*n, if you want to use n threads.'''
 
To submit a batch job called ''OpenMP_Test'' that runs a 40-fold threaded program ''omp_exe'' which requires 6000 MByte of total physical memory and total wall clock time of 40 minutes:
 
a) execute:
<pre>
$ sbatch -p single --export=ALL,OMP_NUM_THREADS=40 -J OpenMP_Test -N 1 -c 80 -t 40 --mem=6000 ./omp_exe
</pre>
or
-->
* generate the script '''job_omp.sh''' containing the following lines:
<source lang="bash">
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --cpus-per-task=80
#SBATCH --time=40:00
#SBATCH --mem=6000mb
#SBATCH --export=ALL,EXECUTABLE=./omp_exe
#SBATCH -J OpenMP_Test

#Usually you should set
export KMP_AFFINITY=compact,1,0
#export KMP_AFFINITY=verbose,compact,1,0 prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

export OMP_NUM_THREADS=$((${SLURM_JOB_CPUS_PER_NODE}/2))
echo "Executable ${EXECUTABLE} running on ${SLURM_JOB_CPUS_PER_NODE} cores with ${OMP_NUM_THREADS} threads"
startexe=${EXECUTABLE}
echo $startexe
exec $startexe
</source>
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores and, if necessary, replace <placeholder> with the required modulefile to enable the OpenMP environment and execute the script '''job_omp.sh''' adding the queue class ''single'' as sbatch option:
<pre>
$ sbatch -p single job_omp.sh
</pre>
Note, that sbatch command line options overrule script options, e.g.,
<pre>
$ sbatch --partition=single --mem=200 job_omp.sh
</pre>
overwrites the script setting of 6000 MByte with 200 MByte.
 
 

==== MPI Parallel Programs ====
MPI parallel programs run faster than serial programs on multi CPU and multi core systems. N-fold spawned processes of the MPI program, i.e., '''MPI tasks''', run simultaneously and communicate via the Message Passing Interface (MPI) paradigm. MPI tasks do not share memory but can be spawned over different nodes.
 
Multiple MPI tasks must be launched via '''mpirun''', e.g. 4 MPI tasks of ''my_par_program'':
<pre>
$ mpirun -n 4 my_par_program
</pre>
This command runs 4 MPI tasks of ''my_par_program'' on the node you are logged in.
To run this command with a loaded Intel MPI the environment variable I_MPI_HYDRA_BOOTSTRAP must be unset ( --> $ unset I_MPI_HYDRA_BOOTSTRAP).

Running MPI parallel programs in a batch job the interactive environment - particularly the loaded modules - will also be set in the batch job. If you want to set a defined module environment in your batch job you have to purge all modules before setting the wished modules.
 
 
===== OpenMPI =====

If you want to run jobs on batch nodes, generate a wrapper script ''job_ompi.sh'' for '''OpenMPI''' containing the following lines:
<source lang="bash">
#!/bin/bash
# Use when a defined module environment related to OpenMPI is wished
module load mpi/openmpi/<placeholder_for_version>
mpirun --bind-to core --map-by core -report-bindings my_par_program
</source>
'''Attention:''' Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since Slurm instructs mpirun about number of processes and node hostnames. Use '''ALWAYS''' the MPI options '''''--bind-to core''''' and '''''--map-by core|socket|node'''''. Please type ''mpirun --help'' for an explanation of the meaning of the different options of mpirun option ''--map-by''.
 
Considering 4 OpenMPI tasks on a single node, each requiring 2000 MByte, and running for 1 hour, execute:
<pre>
$ sbatch -p single -N 1 -n 4 --mem-per-cpu=2000 --time=01:00:00 ./job_ompi.sh
</pre>
 

===== Intel MPI =====

Generate a wrapper script for '''Intel MPI''', ''job_impi.sh'' containing the following lines:
<source lang="bash">
#!/bin/bash
# Use when a defined module environment related to Intel MPI is wished
module load mpi/impi/<placeholder_for_version>
mpiexec.hydra -bootstrap slurm my_par_program
</source>
'''Attention:''' 
Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since Slurm instructs mpirun about number of processes and node hostnames.
 
Launching and running 200 Intel MPI tasks on 5 nodes, each requiring 80 GByte, and running for 5 hours, execute:
<pre>
$ sbatch --partition=multiple -N 5 --ntasks-per-node=40 --mem=80gb -t 300 ./job_impi.sh
</pre>
 
If you want to use 128 or more nodes, you must also set the environment variable as follows: 
export I_MPI_HYDRA_BRANCH_COUNT=-1
 
If you want to use the options perhost, ppn or rr, you must additionally set the environment variable I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=off.
 
 

==== Multithreaded + MPI parallel Programs ====
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes. '''Because hyperthreading is switched on on bwUniCluster 2.0, the option --cpus-per-task (-c) must be set to 2*n, if you want to use n threads.'''
 
 
===== OpenMPI with Multithreading =====
Multiple MPI tasks using '''OpenMPI''' must be launched by the MPI parallel program '''mpirun'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).
 
'''For OpenMPI''' a job-script to submit a batch job called ''job_ompi_omp.sh'' that runs a MPI program with 4 tasks and a 28-fold threaded program ''ompi_omp_program'' requiring 3000 MByte of physical memory per thread (using 28 threads per MPI task you will get 28*3000 MByte = 84000 MByte per MPI task) and total wall clock time of 3 hours looks like:

<source lang="bash">
#!/bin/bash
#SBATCH --ntasks=4
#SBATCH --cpus-per-task=28
#SBATCH --time=03:00:00
#SBATCH --mem=83gb # 84000 MB = 84000/1024 GB = 82.1 GB
#SBATCH --export=ALL,MPI_MODULE=mpi/openmpi/3.1,EXECUTABLE=./ompi_omp_program
#SBATCH --output="parprog_hybrid_%j.out"

# Use when a defined module environment related to OpenMPI is wished
module load ${MPI_MODULE}
export MPIRUN_OPTIONS="--bind-to core --map-by socket:PE=${SLURM_CPUS_PER_TASK} -report-bindings"
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
export NUM_CORES=${SLURM_NTASKS}*${SLURM_CPUS_PER_TASK}
echo "${EXECUTABLE} running on ${NUM_CORES} cores with ${SLURM_NTASKS} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpirun -n ${SLURM_NTASKS} ${MPIRUN_OPTIONS} ${EXECUTABLE}"
echo $startexe
exec $startexe
</source>
Execute the script '''job_ompi_omp.sh''' by command sbatch:
<pre>
$ sbatch -p multiple_e ./job_ompi_omp.sh
</pre>
* With the mpirun option ''--bind-to core'' MPI tasks and OpenMP threads are bound to physical cores.
* With the option ''--map-by node:PE=<value>'' (neighbored) MPI tasks will be attached to different nodes and each MPI task is bound to the first core of a node. <value> must be set to ${OMP_NUM_THREADS}.
* The option ''-report-bindings'' shows the bindings between MPI tasks and physical cores.
* The mpirun-options '''--bind-to core''', '''--map-by socket|...|node:PE=<value>''' should always be used when running a multithreaded MPI program.
 

===== Intel MPI with Multithreading =====
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes.

Multiple Intel MPI tasks must be launched by the MPI parallel program '''mpiexec.hydra'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).

'''For Intel MPI''' a job-script to submit a batch job called ''job_impi_omp.sh'' that runs a Intel MPI program with 10 tasks and a 40-fold threaded program ''impi_omp_program'' requiring 96000 MByte of total physical memory per task and total wall clock time of 1 hours looks like:


<source lang="bash">
#!/bin/bash
#SBATCH --ntasks=10
#SBATCH --cpus-per-task=40
#SBATCH --time=60
#SBATCH --mem=96000
#SBATCH --export=ALL,MPI_MODULE=mpi/impi,EXE=./impi_omp_program
#SBATCH --output="parprog_impi_omp_%j.out"

#If using more than one MPI task per node please set
export KMP_AFFINITY=compact,1,0
#export KMP_AFFINITY=verbose,scatter prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

# Use when a defined module environment related to Intel MPI is wished
module load ${MPI_MODULE}
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
export MPIRUN_OPTIONS="-binding "domain=omp:compact" -print-rank-map -envall"
export NUM_PROCS=eval(${SLURM_NTASKS}*${OMP_NUM_THREADS})
echo "${EXE} running on ${NUM_PROCS} cores with ${SLURM_NTASKS} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpiexec.hydra -bootstrap slurm ${MPIRUN_OPTIONS} -n ${SLURM_NTASKS} ${EXE}"
echo $startexe
exec $startexe
</source>
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores. If you only run one MPI task per node please set KMP_AFFINITY=compact,1,0.
 
If you want to use 128 or more nodes, you must also set the environment variable as follows: 
export I_MPI_HYDRA_BRANCH_COUNT=-1
 
If you want to use the options perhost, ppn or rr, you must additionally set the environment variable I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=off.
 
 
Execute the script '''job_impi_omp.sh''' by command sbatch:
<pre>
$ sbatch -p multiple ./job_impi_omp.sh
</pre>
 
The mpirun option ''-print-rank-map'' shows the bindings between MPI tasks and nodes (not very beneficial). The option ''-binding'' binds MPI tasks (processes) to a particular processor; ''domain=omp'' means that the domain size is determined by the number of threads. If you would choose 2 MPI tasks per node, you should choose ''-binding "cell=unit;map=bunch"''; this binding maps one MPI process to each socket.
 
 

==== Chain jobs ====
The CPU time requirements of many applications exceed the limits of the job classes. In those situations it is recommended to solve the problem by a job chain. A job chain is a sequence of jobs where each job automatically starts its successor.
<source lang="bash">
#!/bin/bash
####################################
## simple Slurm submitter script to setup ##
## a chain of jobs using Slurm ##
####################################
## ver. : 2018-11-27, KIT, SCC

## Define maximum number of jobs via positional parameter 1, default is 5
max_nojob=${1:-5}

## Define your jobscript (e.g. "~/chain_job.sh")
chain_link_job=${PWD}/chain_job.sh

## Define type of dependency via positional parameter 2, default is 'afterok'
dep_type="${2:-afterok}"
## -> List of all dependencies:
## https://slurm.schedmd.com/sbatch.html

myloop_counter=1
## Submit loop
while [ ${myloop_counter} -le ${max_nojob} ] ; do
##
## Differ msub_opt depending on chain link number
if [ ${myloop_counter} -eq 1 ] ; then
slurm_opt=""
else
slurm_opt="-d ${dep_type}:${jobID}"
fi
##
## Print current iteration number and sbatch command
echo "Chain job iteration = ${myloop_counter}"
echo " sbatch --export=myloop_counter=${myloop_counter} ${slurm_opt} ${chain_link_job}"
## Store job ID for next iteration by storing output of sbatch command with empty lines
jobID=$(sbatch -p <queue> --export=ALL,myloop_counter=${myloop_counter} ${slurm_opt} ${chain_link_job} 2>&1 | sed 's/[S,a-z]* //g')
##
## Check if ERROR occured
if [[ "${jobID}" =~ "ERROR" ]] ; then
echo " -> submission failed!" ; exit 1
else
echo " -> job number = ${jobID}"
fi
##
## Increase counter
let myloop_counter+=1
done
</source>
 

==== GPU jobs ====

The nodes in the gpu_4 and gpu_8 queues have 4 or 8 NVIDIA Tesla V100 GPUs. Just submitting a job to these queues is not enough to also allocate one or more GPUs, you have to do so using the "--gres=gpu" parameter. You have to specifiy how many GPUs your job needs, e.g. "--gres=gpu:2" will request two GPUs.

The GPU nodes are shared between multiple jobs if the jobs don't request all the GPUs in a node and there are enough ressources to run more than one job. The individual GPUs are always bound to a single job and will not be shared between different jobs.

a) add after the initial line of your script job.sh the line including the
information about the LSDF Online Storage usage: #SBATCH --constraint=LSDF
<pre>
#!/bin/bash
#SBATCH --ntasks=40
#SBATCH --time=02:00:00
#SBATCH --mem=4000
#SBATCH --gres=gpu:2
</pre>

or b) execute:
<pre>
$ sbatch -p <queue> -n 40 -t 02:00:00 --mem 4000 --gres=gpu:2 job.sh
</pre>
 
If you start an interactive session on of the GPU nodes, you can use the "nvidia-smi" command to list the GPUs allocated to your job:
<pre>
$ nvidia-smi
Sun Mar 29 15:20:05 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2... Off | 00000000:3A:00.0 Off | 0 |
| N/A 29C P0 39W / 300W | 9MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla V100-SXM2... Off | 00000000:3B:00.0 Off | 0 |
| N/A 30C P0 41W / 300W | 8MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 14228 G /usr/bin/X 8MiB |
| 1 14228 G /usr/bin/X 8MiB |
+-----------------------------------------------------------------------------+
</pre>

 
 

==== LSDF Online Storage ====
On bwUniCluster 2.0 you can use for special cases the LSDF Online Storage on the HPC cluster nodes. Please request for this service separately ([https://www.lsdf.kit.edu/os/storagerequest/: LSDF Storage Request]).
To mount the LSDF Online Storage on the compute nodes during the job runtime the
the constraint flag "LSDF" has to be set.

a) add after the initial line of your script job.sh the line including the
information about the LSDF Online Storage usage: #SBATCH --constraint=LSDF
<pre>
#!/bin/bash
#SBATCH --ntasks=1
#SBATCH --time=120
#SBATCH --mem=200
#SBATCH --constraint=LSDF
</pre>

or b) execute:
<pre>
$ sbatch -p <queue> -n 1 -t 2:00:00 --mem 200 job.sh -C LSDF
</pre>
 
For the usage of the LSDF Online Storage
the following environment variables are available: $LSDF, $LSDFPROJECTS, $LSDFHOME.
 
 

====BeeOND (BeeGFS On-Demand)====

BeeOND instances are integrated into the prolog and epilog script of the cluster batch system, Slurm. It can be used on the compute nodes during the job runtime with the constraint flag "BEEOND" ([[BwUniCluster_2.0_Slurm_common_Features#sbatch_Command_Parameters|Slurm Command Parameters]])
<pre>
#!/bin/bash
#SBATCH ...
#SBATCH --constraint=BEEOND
</pre>

After your job has started you can find the private on-demand file system in '''/mnt/odfs/$SLURM_JOB_ID''' directory. The mountpoint comes with three pre-configured directories:
<pre>
#for small files (stripe count = 1)
/mnt/odfs/$SLURM_JOB_ID/stripe_1
#stripe count = 4
/mnt/odfs/$SLURM_JOB_ID/stripe_default
#stripe count = 8, 16 or 32, use this directories for medium sized and large files or when using MPI-IO
/mnt/odfs/$SLURM_JOB_ID/stripe_8, /mnt/odfs/$SLURM_JOB_ID/stripe_16 or /mnt/odfs/$SLURM_JOB_ID/stripe_32
</pre>

If you request less nodes than stripe count, the stripe count will be max number of nodes, e.g., You only request 8 nodes , so the directory with stripe count 16 is basically only with a stripe count 8.

The capacity of the private file system depends on the number of nodes. For each node you get 250Gbyte.

!!! Be careful when creating large files, use always the directory with the max stripe count for large files.
If you create large files use a higher stripe count. For example, if your largest file is 1.1 Tb, then you have to use a stripe count larger>4 (4 x 250GB).

If you request 100 nodes for your job, the private file system is 100 * 250 Gbyte ~ 25 Tbyte (approx) capacity.

'''Recommendation:'''

The private file system is using its own metadata server. This metadata server is started on the first nodes. Depending on your application, the metadata server is consuming decent amount of CPU power. Probably adding a extra node to your job could improve the usability of the on-demand file system. Start your application with the MPI option:
<pre>
mpirun -nolocal myapplication
</pre>
With the -nolocal option the node where mpirun is initiated is not used for your application. This node is fully available for the meta data server of your requested on-demand file system.

Example job script:
<pre>
#!/bin/bash
#very simple example on how to use a private on-demand file system
#SBATCH -N 10
#SBATCH --constraint=BEEOND

#create a workspace
ws_allocate myresults-$SLURM_JOB_ID 90
RESULTDIR=`ws_find myresults-$SLURM_JOB_ID`

#Set ENV variable to on-demand file system
ODFSDIR=/mnt/odfs/$SLURM_JOB_ID/stripe_16/

#start application and write results to on-demand file system
mpirun -nolocal myapplication -o $ODFSDIR/results

#Copy back data after your job application end
rsync -av $ODFSDIR/results $RESULTDIR
</pre>
 
 

== Start time of job or resources : squeue --start ==
The command can be used by any user to displays the estimated start time of a job based a number of analysis types based on historical usage, earliest available reservable resources, and priority based backlog. The command squeue is explained in detail on the webpage https://slurm.schedmd.com/squeue.html or via manpage (man squeue).
 
 
=== Access ===
By default, this command can be run by '''any user'''.
 
 

== List of your submitted jobs : squeue ==
Displays information about YOUR active, pending and/or recently completed jobs. The command displays all own active and pending jobs. The command squeue is explained in detail on the webpage https://slurm.schedmd.com/squeue.html or via manpage (man squeue).
 
 
=== Access ===
By default, this command can be run by any user.
 
 

=== Flags ===
{| width=750px class="wikitable"
|-
! Flag !! Description
|-
| -l, --long
| Report more of the available information for the selected jobs or job steps, subject to any constraints specified.
|}
 

=== Examples ===
''squeue'' example on bwUniCluster 2.0 (Only your own jobs are displayed!).
<pre>
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
18088744 single CPV.sbat ab1234 PD 0:00 1 (Priority)
18098414 multiple CPV.sbat ab1234 PD 0:00 2 (Priority)
18090089 multiple CPV.sbat ab1234 R 2:27 2 uc2n[127-128]
$ squeue -l
JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON)
18088654 single CPV.sbat ab1234 COMPLETI 4:29 2:00:00 1 uc2n374
18088785 single CPV.sbat ab1234 PENDING 0:00 2:00:00 1 (Priority)
18098414 multiple CPV.sbat ab1234 PENDING 0:00 2:00:00 2 (Priority)
18088683 single CPV.sbat ab1234 RUNNING 0:14 2:00:00 1 uc2n413
</pre>
* The output of ''squeue'' shows how many jobs of yours are running or pending and how many nodes are in use by your jobs.
 

== Shows free resources : sinfo_t_idle ==
The Slurm command sinfo is used to view partition and node information for a system running Slurm. It incorporates down time, reservations, and node state information in determining the available backfill window. The sinfo command can only be used by the administrator.
 
SCC has prepared a special script (sinfo_t_idle) to find out how many processors are available for immediate use on the system. It is anticipated that users will use this information to submit jobs that meet these criteria and thus obtain quick job turnaround times.
 
 
=== Access ===
By default, this command can be used by any user or administrator.
 
 
=== Example ===
* The following command displays what resources are available for immediate use for the whole partition.
<pre>$ sinfo_t_idle
Partition dev_multiple : 8 nodes idle
Partition multiple : 332 nodes idle
Partition dev_single : 4 nodes idle
Partition single : 76 nodes idle
Partition long : 80 nodes idle
Partition fat : 5 nodes idle
Partition dev_special : 342 nodes idle
Partition special : 342 nodes idle
Partition dev_multiple_e: 7 nodes idle
Partition multiple_e : 335 nodes idle
Partition gpu_4 : 12 nodes idle
Partition gpu_8 : 6 nodes idle
</pre>
* For the above example jobs in all partitions can be run immediately.
 

== Detailed job information : scontrol show job ==
scontrol show job displays detailed job state information and diagnostic output for all or a specified job of yours. Detailed information is available for active, pending and recently completed jobs. The command scontrol is explained in detail on the webpage https://slurm.schedmd.com/scontrol.html or via manpage (man scontrol).
 
Display the state of all your jobs in normal mode: scontrol show job
 
Display the state of a job with <jobid> in normal mode: scontrol show job <jobid>
 
 
=== Access ===
* End users can use scontrol show job to view the status of their '''own jobs''' only.
 

=== Arguments ===
{| width=750px class="wikitable"
|-
! Option !! Default !! Description !! Example
|- style="vertical-align:top;"
|- style="width:12%;"
| -d
| (n/a)
| Detailed mode
| Example: Display the state with jobid 18089884 in detailed mode. <pre>scontrol -d show job 18089884</pre>
|}
 
 

=== Scontrol show job Example ===
Here is an example from bwUniCluster 2.0.
<pre>
squeue # show my own jobs (here the userid is replaced!)
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
18089884 multiple CPV.sbat bq0742 R 33:44 2 uc2n[165-166]

$
$ # now, see what's up with my pending job with jobid 18089884
$
$ scontrol show job 18089884

JobId=18089884 JobName=CPV.sbatch
UserId=bq0742(8946) GroupId=scc(12345) MCS_label=N/A
Priority=3 Nice=0 Account=kit QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:35:06 TimeLimit=02:00:00 TimeMin=N/A
SubmitTime=2020-03-16T14:14:54 EligibleTime=2020-03-16T14:14:54
AccrueTime=2020-03-16T14:14:54
StartTime=2020-03-16T15:12:51 EndTime=2020-03-16T17:12:51 Deadline=N/A
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2020-03-16T15:12:51
Partition=multiple AllocNode:Sid=uc2n995:5064
ReqNodeList=(null) ExcNodeList=(null)
NodeList=uc2n[165-166]
BatchHost=uc2n165
NumNodes=2 NumCPUs=160 NumTasks=80 CPUs/Task=1 ReqB:S:C:T=0:0:*:1
TRES=cpu=160,mem=96320M,node=2,billing=160
Socks/Node=* NtasksPerN:B:S:C=40:0:*:1 CoreSpec=*
MinCPUsNode=40 MinMemoryCPU=1204M MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
Command=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/CPV.sbatch
WorkDir=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin
StdErr=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/slurm-18089884.out
StdIn=/dev/null
StdOut=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/slurm-18089884.out
Power=
MailUser=(null) MailType=NONE
</pre>
 
You can use standard Linux pipe commands to filter the very detailed scontrol show job output.
* In which state the job is?
<pre>$ scontrol show job 18089884 | grep -i State
JobState=COMPLETED Reason=None Dependency=(null)
</pre>
 

== Cancel Slurm Jobs ==
The scancel command is used to cancel jobs. The command scancel is explained in detail on the webpage https://slurm.schedmd.com/scancel.html or via manpage (man scancel).
 
 
=== Canceling own jobs : scancel ===
scancel is used to signal or cancel jobs, job arrays or job steps. The command is:
<pre>
$ scancel [-i] <job-id>
$ scancel -t <job_state_name>
</pre>

 
{| width=750px class="wikitable"
! Flag !! Default !! Description !! Example
|- style="vertical-align:top;"
| -i, --interactive
| (n/a)
| Interactive mode.
| Cancel the job 987654 interactively. <pre> scancel -i 987654 </pre>
|-
| -t, --state
| (n/a)
| Restrict the scancel operation to jobs in a certain state. "job_state_name" may have a value of either "PENDING", "RUNNING" or "SUSPENDED".
| Cancel all jobs in state "PENDING". <pre> scancel -t "PENDING" </pre>
|}
 

= Resource Managers =
=== Batch Job (Slurm) Variables ===
The following environment variables of Slurm are added to your environment once your job has started
(only an excerpt of the most important ones).
{| width=750px class="wikitable"
! Environment !! Brief explanation
|-
| SLURM_JOB_CPUS_PER_NODE
| Number of processes per node dedicated to the job
|-
| SLURM_JOB_NODELIST
| List of nodes dedicated to the job
|-
| SLURM_JOB_NUM_NODES
| Number of nodes dedicated to the job
|-
| SLURM_MEM_PER_NODE
| Memory per node dedicated to the job
|-
| SLURM_NPROCS
| Total number of processes dedicated to the job
|-
| SLURM_CLUSTER_NAME
| Name of the cluster executing the job
|-
| SLURM_CPUS_PER_TASK
| Number of CPUs requested per task
|-
| SLURM_JOB_ACCOUNT
| Account name
|-
| SLURM_JOB_ID
| Job ID
|-
| SLURM_JOB_NAME
| Job Name
|-
| SLURM_JOB_PARTITION
| Partition/queue running the job
|-
| SLURM_JOB_UID
| User ID of the job's owner
|-
| SLURM_SUBMIT_DIR
| Job submit folder. The directory from which msub was invoked.
|-
| SLURM_JOB_USER
| User name of the job's owner
|-
| SLURM_RESTART_COUNT
| Number of times job has restarted
|-
| SLURM_PROCID
| Task ID (MPI rank)
|-
| SLURM_NTASKS
| The total number of tasks available for the job
|-
| SLURM_STEP_ID
| Job step ID
|-
| SLURM_STEP_NUM_TASKS
| Task count (number of PI ranks)
|-
| SLURM_JOB_CONSTRAINT
| Job constraints
|}
See also:
* [https://slurm.schedmd.com/sbatch.html#lbAI Slurm input and output environment variables]
 

=== Job Exit Codes ===
A job's exit code (also known as exit status, return code and completion code) is captured by SLURM and saved as part of the job record.
 
Any non-zero exit code will be assumed to be a job failure and will result in a Job State of FAILED with a reason of "NonZeroExitCode".
 
The exit code is an 8 bit unsigned number ranging between 0 and 255. While it is possible for a job to return a negative exit code, SLURM will display it as an unsigned value in the 0 - 255 range.
 
 
==== Displaying Exit Codes and Signals ====
SLURM displays a job's exit code in the output of the '''scontrol show job''' and the sview utility.
 
When a signal was responsible for a job or step's termination, the signal number will be displayed after the exit code, delineated by a colon(:).
 
 
==== Submitting Termination Signal ====
Here is an example, how to 'save' a Slurm termination signal in a typical jobscript.
<source lang="bash">
[...]
exit_code=$?
mpirun -np <#cores> <EXE_BIN_DIR>/<executable> ... (options) 2>&1
[ "$exit_code" -eq 0 ] && echo "all clean..." || \
echo "Executable <EXE_BIN_DIR>/<executable> finished with exit code ${$exit_code}"
[...]
</source>
* Do not use ''''time'''' mpirun! The exit code will be the one submitted by the first (time) program.
* You do not need an '''exit $exit_code''' in the scripts.
 
 
----
[[Category:bwUniCluster 2.0|bwUniCluster 2.0]]
[[#top|Back to top]]