bwHPC Wiki - User contributions [en]

BwUniCluster2.0/FAQ - broadwell partition

2017-09-14T10:45:30Z

S Shamsudeen: /* What happens with code compiled for old partition running on the extension partition? */

FAQs concerning best practice of [[BwUniCluster_Hardware_and_Architecture#Components_of_bwUniCluster|bwUniCluster broadwell partition]] (aka "extension" partition).

__TOC__

= Login =
== Are there separate login nodes for the bwUniCluster broadwell partition? ==
* Yes, but primarily to be used for compiling code.

== How to login to broadwell login nodes? ==
* You can directly login on broadwell partition login nodes using
$ username@uc1e.scc.kit.edu
* If you login to the ''old'' uc1 login nodes, even though you can use broadwell nodes using the same procedure as 'compute nodes'.

= Compilation =
== How to compile code on broadwell (extension) nodes? ==
On uc1 (old) login nodes:
<pre>
icc/ifort -axCORE-AVX2
</pre>
On uc1e (extension) login nodes:
<pre>
icc/ifort -xHost
</pre>

== How to compile the same code on old and extension partition? ==
On uc1e (= extension) login nodes:
<pre>
icc/ifort -xAVX -axCORE-AVX2
</pre>

== What happens with code compiled for old partition whic is running on the extension partition? ==
Code will run but significantly slower since AVX2 will not be used. Please recompile your code accordingly.



= Job execution =
== How to submit jobs to the broadwell (= extension) partition ==
The submitted job will be distributed either way to the broadwell nodes if specified correctly, i.e.:
<pre>
msub -q multinode
</pre>

----
[[Category:bwUniCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2017-09-14T10:38:16Z

S Shamsudeen: /* How to compile code working on the old and extension partition? */

FAQs concerning best practice of [[BwUniCluster_Hardware_and_Architecture#Components_of_bwUniCluster|bwUniCluster broadwell partition]] (aka "extension" partition).

__TOC__

= Login =
== Are there separate login nodes for the bwUniCluster broadwell partition? ==
* Yes, but primarily to be used for compiling code.

== How to login to broadwell login nodes? ==
* You can directly login on broadwell partition login nodes using
$ username@uc1e.scc.kit.edu
* If you login to the ''old'' uc1 login nodes, even though you can use broadwell nodes using the same procedure as 'compute nodes'.

= Compilation =
== How to compile code on broadwell (extension) nodes? ==
On uc1 (old) login nodes:
<pre>
icc/ifort -axCORE-AVX2
</pre>
On uc1e (extension) login nodes:
<pre>
icc/ifort -xHost
</pre>

== How to compile the same code on old and extension partition? ==
On uc1e (= extension) login nodes:
<pre>
icc/ifort -xAVX -axCORE-AVX2
</pre>

== What happens with code compiled for old partition running on the extension partition? ==
Code will run but significantly slower since AVX2 will not be used. Please recompile your code accordingly.



= Job execution =
== How to submit jobs to the broadwell (= extension) partition ==
The submitted job will be distributed either way to the broadwell nodes if specified correctly, i.e.:
<pre>
msub -q multinode
</pre>

----
[[Category:bwUniCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2017-09-14T10:36:52Z

S Shamsudeen: /* How to compile code on broadwell (extension) nodes? */

FAQs concerning best practice of [[BwUniCluster_Hardware_and_Architecture#Components_of_bwUniCluster|bwUniCluster broadwell partition]] (aka "extension" partition).

__TOC__

= Login =
== Are there separate login nodes for the bwUniCluster broadwell partition? ==
* Yes, but primarily to be used for compiling code.

== How to login to broadwell login nodes? ==
* You can directly login on broadwell partition login nodes using
$ username@uc1e.scc.kit.edu
* If you login to the ''old'' uc1 login nodes, even though you can use broadwell nodes using the same procedure as 'compute nodes'.

= Compilation =
== How to compile code on broadwell (extension) nodes? ==
On uc1 (old) login nodes:
<pre>
icc/ifort -axCORE-AVX2
</pre>
On uc1e (extension) login nodes:
<pre>
icc/ifort -xHost
</pre>

== How to compile code working on the old and extension partition? ==
On uc1e (= extension) login nodes:
<pre>
icc/ifort -xAVX -axCORE-AVX2
</pre>

== What happens with code compiled for old partition running on the extension partition? ==
Code will run but significantly slower since AVX2 will not be used. Please recompile your code accordingly.



= Job execution =
== How to submit jobs to the broadwell (= extension) partition ==
The submitted job will be distributed either way to the broadwell nodes if specified correctly, i.e.:
<pre>
msub -q multinode
</pre>

----
[[Category:bwUniCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2017-09-14T10:36:08Z

S Shamsudeen: /* How to compile code on broadwell (= extension) nodes? */

FAQs concerning best practice of [[BwUniCluster_Hardware_and_Architecture#Components_of_bwUniCluster|bwUniCluster broadwell partition]] (aka "extension" partition).

__TOC__

= Login =
== Are there separate login nodes for the bwUniCluster broadwell partition? ==
* Yes, but primarily to be used for compiling code.

== How to login to broadwell login nodes? ==
* You can directly login on broadwell partition login nodes using
$ username@uc1e.scc.kit.edu
* If you login to the ''old'' uc1 login nodes, even though you can use broadwell nodes using the same procedure as 'compute nodes'.

= Compilation =
== How to compile code on broadwell (extension) nodes? ==
On uc1 (old) login nodes:
<pre>
icc/ifort -axCORE-AVX2
</pre>
On uc1e (= extension) login nodes:
<pre>
icc/ifort -xHost
</pre>

== How to compile code working on the old and extension partition? ==
On uc1e (= extension) login nodes:
<pre>
icc/ifort -xAVX -axCORE-AVX2
</pre>

== What happens with code compiled for old partition running on the extension partition? ==
Code will run but significantly slower since AVX2 will not be used. Please recompile your code accordingly.



= Job execution =
== How to submit jobs to the broadwell (= extension) partition ==
The submitted job will be distributed either way to the broadwell nodes if specified correctly, i.e.:
<pre>
msub -q multinode
</pre>

----
[[Category:bwUniCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2017-09-14T10:35:40Z

S Shamsudeen: /* Compilation */

FAQs concerning best practice of [[BwUniCluster_Hardware_and_Architecture#Components_of_bwUniCluster|bwUniCluster broadwell partition]] (aka "extension" partition).

__TOC__

= Login =
== Are there separate login nodes for the bwUniCluster broadwell partition? ==
* Yes, but primarily to be used for compiling code.

== How to login to broadwell login nodes? ==
* You can directly login on broadwell partition login nodes using
$ username@uc1e.scc.kit.edu
* If you login to the ''old'' uc1 login nodes, even though you can use broadwell nodes using the same procedure as 'compute nodes'.

= Compilation =
== How to compile code on broadwell (= extension) nodes? ==
On uc1 (old) login nodes:
<pre>
icc/ifort -axCORE-AVX2
</pre>
On uc1e (= extension) login nodes:
<pre>
icc/ifort -xHost
</pre>

== How to compile code working on the old and extension partition? ==
On uc1e (= extension) login nodes:
<pre>
icc/ifort -xAVX -axCORE-AVX2
</pre>

== What happens with code compiled for old partition running on the extension partition? ==
Code will run but significantly slower since AVX2 will not be used. Please recompile your code accordingly.



= Job execution =
== How to submit jobs to the broadwell (= extension) partition ==
The submitted job will be distributed either way to the broadwell nodes if specified correctly, i.e.:
<pre>
msub -q multinode
</pre>

----
[[Category:bwUniCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2017-09-12T11:52:40Z

S Shamsudeen:

FAQs concerning best practice of [[BwUniCluster_Hardware_and_Architecture#Components_of_bwUniCluster|bwUniCluster broadwell partition]] (aka "extension" partition).

__TOC__

= Login =
== Are there separate login nodes for the bwUniCluster broadwell partition? ==
* Yes, but primarily to be used for compiling code.

== How to login to broadwell login nodes? ==
* You can directly login on broadwell partition login nodes using
$ username@uc1e.scc.kit.edu
* If you login to the ''old'' uc1 login nodes, even though you can use broadwell nodes using the same procedure as 'compute nodes'.

= Compilation =
== How to compile code working on broadwell (= extension) nodes? ==
On uc1 (= old) login nodes:
<pre>
icc/ifort -axCORE-AVX2
</pre>
On uc1e (= extension) login nodes:
<pre>
icc/ifort -xHost
</pre>

== How to compile code working on the old and extension partition? ==
On uc1e (= extension) login nodes:
<pre>
icc/ifort -xAVX -axCORE-AVX2
</pre>

== What happens with code compiled for old partition running on the extension partition? ==
Code will run but significantly slower since AVX2 will not be used. Please recompile your code accordingly.



= Job execution =
== How to submit jobs to the broadwell (= extension) partition ==
The submitted job will be distributed either way to the broadwell nodes if specified correctly, i.e.:
<pre>
msub -q multinode
</pre>

----
[[Category:bwUniCluster]]

BwUniCluster 2.0 User Access

2017-09-12T11:29:54Z

S Shamsudeen:

[[bwUniCluster]] is Baden-Württemberg's general purpose tier 3 high performance computing (HPC)
cluster co-financed by Baden-Württemberg's ministry of science, research and arts and
the shareholders:
* Albert Ludwig University of Freiburg
* Eberhard Karls University, Tübingen
* Karlsruhe Institute of Technology (KIT)
* Ruprecht-Karls-Universität Heidelberg
* Ulm University
* University of Hohenheim
* University of Konstanz
* University of Mannheim
* University of Stuttgart
* HAW BW e.V. (an association of university of applied sciences in Baden-Württemberg)
 
To '''log on''' [[bwUniCluster]] a user account is required. All members of the shareholder
universities can apply for an account.
 

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:left; color:#000;vertical-align:top;" |__TOC__
| [[File:bwUniCluster_17Jan2014_p044-rot_t10.10.00.jpg|center|border|250px|bwUniCluster wiring by Holger Obermaier, copyright: KIT (SCC)]] bwUniCluster wiring © KIT (SCC)
|}

= Registration =

Granting access and issuing a user account for '''''bwUniCluster''''' requires the registration at the KIT service website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] (step B). However, this registration depends on the '''bwUniCluster entitlement''' issued by your university (step A).
 

== Step A: bwUniCluster entitlement for registration ==
Each university issues the bwUniCluster entitlement '''only''' for their own members. Details on the issuing process and/or the entitlement application forms are listed hereafter:

* [[BwCluster_User_Access_Uni_Freiburg|Albert Ludwig University of Freiburg]]
* [http://www.zdv.uni-tuebingen.de/dienstleistungen/computing/zugang-zu-den-ressourcen.html Eberhard Karls University Tübingen]
* [http://www.scc.kit.edu/downloads/sdo/Antrag_Benutzernummer_bwUniCluster.pdf Karlsruhe Institute of Technology (KIT)]
* [[‎BWUniCluster_User_Access_Members_Uni_Heidelberg|Ruprecht-Karls-Universität Heidelberg]]
* [[BWUniCluster_User_Access_Members_Uni_Ulm|Ulm University]]
* [https://kim.uni-hohenheim.de/bwhpc-account University of Hohenheim]
* [[BWUniCluster_User_Access_Members_Uni_Konstanz|University of Konstanz]]
* [[BWUniCluster_User_Access_Members_Uni_Mannheim|University of Mannheim]]
* [https://www.hlrs.de/solutions-services/academic-users/bwunicluster-access/ University of Stuttgart]

 

== Step B: Web Registration ==
After step A, i.e., after issueing the bwUniCluster entitlement, please visit:
* [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
*# Select your home organization from the list and click '''Proceed'''
*# You will be directed to the ''Identity Provider'' of your home organisation
*# Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
*# You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
*# <div>Select unter '''The following services are available''' the service '''bwUniCluster''' [[File:bwRegWebApp_avail_services_pic01.png|center|border|]]</div>
*# Click '''Register'''
*# Finally
*#* for all non-KIT members '''mandatorily''' set a service password for authentication on bwUniCluster
*#* for KIT members '''optionally''' set a service password for authentication on bwUniCluster


 

== Step C: Amendment: bwUniCluster questionnaire ==
Starting June 1st, 2015, usage of bwUniCluster mandatorily requires the
questionnaire

https://www.bwhpc-c5.de/en/ZAS/bwunicluster_survey.php

to be answered. The input is solely used for the
elaboration of future support activities and for planning future HPC
resources. From June 1st, 2015, login to bwUniCluster is only permitted
to users who have already answered the questionnaire. '''''Exception''': For the first 14 days after the web registration login to bwUniCluster is permitted without participation in the questionnaire.''
 
 

== Changing Password ==
By default, for KIT members your bwUniCluster '''password''' to log on matches that one of your KIT
account, while for all non-KIT members your bwUniCluster '''password''' is that one you saved during the web registration (compare step 7 of chapter 1.2).
At any time, you can set a new bwUniCluster password via the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] by carrying out the following steps:
# visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and select your home organization
# authenticate yourself via your home-organizational user id / username and your home-organizational password
# find on the left side '''bwUniCluster''' and select '''Set Service Password'''
# set new service, i.e. bwUniCluster, password, repeat it and click '''Save''' button.
# the page answers e.g. "Das Passwort wurde bei dem Dienst geändert" ("password has been changed")
# proceed to log in using the new password in the next step
 

== Contact ==
If you have questions or problems concerning bwUniCluster registration, please [[Registration_Support_-_bwUniCluster|contact your local hotline]].
 
 

= Login =
After finishing the web registration bwUniCluster is ready for your '''SSH''' based login. Available SSH applications are:
* built-in command under Linux and macOS using the application ''terminal''
* [http://mobaxterm.mobatek.net/ MobaXterm] under Windows

bwUniCluster has two dedicated login nodes. The selection of the login node is done automatically and is based on a round-robin scheduling. Logging in another time the session might run
on the other login node.

Only the secure shell ''SSH'' is allowed to login. Other protocols like ''telnet'' or ''rlogin''
are not allowed for security reasons.

== Login applications: Linux and macOS ==

A connection to bwUniCluster can be established by the command:

<pre>
$ ssh <UserID>@bwunicluster.scc.kit.edu
</pre>

If you are using ''openSSH'' (usually installed on Linux based systems) and you want to use a GUI-based application on bwUniCluster you should use the
''ssh'' command with the option ''-X''

<pre>
$ ssh -X <UserID>@bwunicluster.scc.kit.edu
</pre>

== Login applications: Windows ==

=== Hints for using MobaXterm ===

Note: The bwHPC-C5 support team strongly recommends to use [http://mobaxterm.mobatek.net/ MobaXterm] instead of ''PuTTY'' or ''WinSCP''. ''MobaXterm'' provides a X11 server allowing to start GUI based software.

Start ''MobaXterm'', fill in the following fields:
<pre>
Remote name : uc1.scc.kit.edu
Specify user name : <UserID>
Port : 22
</pre>
After that click on 'ok'. Then a terminal will be opened and there you can enter your password.

=== Depreciated: Hints for using PuTTY ===
Start PuTTY, fill in at window ''PuTTY Configuration'' under category ''Session'' the following fields
<pre>
Host Name : <UserID>@bwunicluster.scc.kit.edu
Port : 22
Connection type : SSH
</pre>
and click on ''Open'', enter your password and accept adding ''host key''. Note that you can also name the configured session via ''Save'' and load it later via the given name.

=== Depreciated: Hints for using WinSCP ===

Start WinSCP, fill in the following fields:
<pre>
File Protocol : SCP
Host name : bwunicluster.scc.kit.edu
Port : 22
User name : <UserID>
</pre>
and click on 'Login' and enter your password. Note that you can also name the configured session via ''Save'' and load it later via the given name.

== About UserID / Username ==

'''<UserID>''' of the ssh command is a placeholder for your '''''username''''' at your home
organization together with a '''''prefix''''' as followed:

{| width=450px class="wikitable"
! Home organization !! <UserID>
|-
| Universität Freiburg || ''fr_''username
|-
| Universität Heidelberg || ''hd_''username
|-
| Universität Hohenheim || ''ho_''username
|-
| KIT || username ''(without any prefix)''
|-
| Universität Konstanz || ''kn_''username
|-
| Universität Mannheim || ''ma_''username
|-
| Universität Stuttgart || ''st_''username
|-
| Universität Tübingen || ''tu_''username
|-
| Universität Ulm || ''ul_''username
|-
| Hochschule Aalen || ''aa_''username
|-
| Hochschule Albstadt-Sigmaringen || ''as_''username
|-
| Hochschule Esslingen || ''es_''username
|-
| Hochschule Furtwangen || ''hf_''username
|-
| Hochschule Karlsruhe || ''hk_''username
|-
| HTWG Konstanz || ''ht_''username
|-
| Hochschule Reutlingen || ''hr_''username
|-
| Hochschule Rottenburg || ''ro_''username
|-
| Hochschule für Technik Stuttgart || ''hs_''username
|-
| Hochschule Ulm || ''hu_''username
|-
|}

== Allowed activities on login nodes ==

The login nodes of bwUniCluster are the access point to the compute system and to your bwUniCluster $HOME directory. The login nodes are shared with all the users of bwUniCluster. Therefore, your activities on the login nodes are limited to primarily set up your batch jobs. Your activities may also be:
* '''short''' compilation of your program code and
* '''short''' pre- and postprocessing of your batch jobs.

To guarantee usability for all the users of bwUniCluster '''you must not run your compute jobs on the login nodes'''. Compute jobs must be submitted to the
[[bwUniCluster Batch Jobs|queueing system]]. Any compute job running on the login nodes will be terminated without any notice. Any long-running compilation or any long-running pre- or postprocessing of batch jobs must also be submitted to the [[bwUniCluster Batch Jobs|queueing system]].
 

= [[First_Steps_on_bwHPC_cluster|First steps on bwUniCluster]] =

First and some important steps on bwUniCluster can be found [[First_Steps_on_bwHPC_cluster|here]].

= Deregistration =
If you plan to permanently leave the bwUniCluster, follow the deregister checklist:
# Transfer all your data in $HOME and workspace to your local computer/storage and after that clear off all your data
# Visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
#* Select your home organization from the list and click '''Proceed'''
#* Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
#* You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
#* <div>Select '''Registry Info''' of the service '''bwUniCluster''' (on the left hand side) [[File:bwUniCluster_registration_sidebar.png|center|border|]]</div>
#* Click '''Deregister'''
Note that Step 2 will automatically unsubscribe you from the bwunicluster mail list.

----
[[Category:bwUniCluster|Access]][[Category:Access|bwUniCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2017-09-06T12:58:22Z

S Shamsudeen: /* Job execution */

FAQs concerning best practice of bwUniCluster broadwell (aka "extension") partition.

__TOC__

= Login =
== Are there separate login nodes for the bwUniCluster broadwell partition? ==
* Yes, but primarily to be used for compiling code.
* You can directly login on broadwell partition using
$ username@uc1e.scc.kit.edu
* If you login uc1, even though you can use broadwell nodes using the same procedure as 'compute nodes'. The submitted job will be distributed on broadwell nodes also.
example: msub -q singlenode
msub -q multinode
* But for the compilation you have to use the appropriate flags

= Compilation =
== Q1 ==
* To compile the same jobs on uc1 and uc1e, you have to use appropriate flags. Otherwise the submitted jobs will be crashed. Some of the main flags are:

= Job execution =
== Q2 ==
If the appropriate flags are selected, the submitted jobs can be executed on uc1 and uc1e.
----
[[Category:bwUniCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2017-09-06T12:48:05Z

S Shamsudeen: /* Compilation */

BwUniCluster2.0/FAQ - broadwell partition

2017-09-06T12:46:00Z

S Shamsudeen: /* Are there separate login nodes for the bwUniCluster broadwell partition? */

FAQs concerning best practice of bwUniCluster broadwell (aka "extension") partition.

__TOC__

= Login =
== Are there separate login nodes for the bwUniCluster broadwell partition? ==
* Yes, but primarily to be used for compiling code.
* You can directly login on broadwell partition using
$ username@uc1e.scc.kit.edu
* If you login uc1, even though you can use broadwell nodes using the same procedure as 'compute nodes'. The submitted job will be distributed on broadwell nodes also.
example: msub -q singlenode
msub -q multinode
* But for the compilation you have to use the appropriate flags

= Compilation =
== Q1 ==

= Job execution =
== Q2 ==

----
[[Category:bwUniCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2017-09-06T12:43:09Z

S Shamsudeen: /* Are there separate login nodes for the bwUniCluster broadwell partition? */

FAQs concerning best practice of bwUniCluster broadwell (aka "extension") partition.

__TOC__

= Login =
== Are there separate login nodes for the bwUniCluster broadwell partition? ==
* Yes, but primarily to be used for compiling code.
* You can directly login on broadwell partition using
$ username@uc1e.scc.kit.edu
* If you login uc1, even though you can use broadwell nodes using the same procedure as 'compute nodes'. The submitted job will be distributed on broadwell node also.
example: msub -q singlenode
msub -q multinode
* But for the compilation you have to use the appropriate flags

= Compilation =
== Q1 ==

= Job execution =
== Q2 ==

----
[[Category:bwUniCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2017-09-06T12:42:38Z

S Shamsudeen: /* Are there separate login nodes for the bwUniCluster broadwell partition? */

FAQs concerning best practice of bwUniCluster broadwell (aka "extension") partition.

__TOC__

= Login =
== Are there separate login nodes for the bwUniCluster broadwell partition? ==
* Yes, but primarily to be used for compiling code.
* You can directly login on broadwell partition using
$ username@uc1e.scc.kit.edu
* If you login uc1, even then you can use broadwell nodes using the same procedure as 'compute nodes'. The submitted job will be distributed on broadwell node also.
example: msub -q singlenode
msub -q multinode
* But for the compilation you have to use the appropriate flags

= Compilation =
== Q1 ==

= Job execution =
== Q2 ==

----
[[Category:bwUniCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2017-09-06T12:39:21Z

S Shamsudeen: /* Are there separate login nodes for the bwUniCluster broadwell partition? */

BwUniCluster2.0/FAQ - broadwell partition

2017-09-06T12:38:43Z

S Shamsudeen: /* Are there separate login nodes for the bwUniCluster broadwell partition? */

BwUniCluster2.0/FAQ - broadwell partition

2017-09-06T12:28:04Z

S Shamsudeen: /* Login */

BwUniCluster 2.0 User Access

2017-09-06T09:12:50Z

S Shamsudeen: /* Hints for using PuTTY */

[[bwUniCluster]] is Baden-Württemberg's general purpose tier 3 high performance computing (HPC)
cluster co-financed by Baden-Württemberg's ministry of science, research and arts and
the shareholders:
* Albert Ludwig University of Freiburg
* Eberhard Karls University, Tübingen
* Karlsruhe Institute of Technology (KIT)
* Ruprecht-Karls-Universität Heidelberg
* Ulm University
* University of Hohenheim
* University of Konstanz
* University of Mannheim
* University of Stuttgart
* HAW BW e.V. (an association of university of applied sciences in Baden-Württemberg)
 
To '''log on''' [[bwUniCluster]] a user account is required. All members of the shareholder
universities can apply for an account.
 

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:left; color:#000;vertical-align:top;" |__TOC__
| [[File:bwUniCluster_17Jan2014_p044-rot_t10.10.00.jpg|center|border|250px|bwUniCluster wiring by Holger Obermaier, copyright: KIT (SCC)]] bwUniCluster wiring © KIT (SCC)
|}

= Registration =

Granting access and issuing a user account for '''''bwUniCluster''''' requires the registration at the KIT service website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] (step B). However, this registration depends on the '''bwUniCluster entitlement''' issued by your university (step A).
 

== Step A: bwUniCluster entitlement for registration ==
Each university issues the bwUniCluster entitlement '''only''' for their own members. Details on the issuing process and/or the entitlement application forms are listed hereafter:

* [[BwCluster_User_Access_Uni_Freiburg|Albert Ludwig University of Freiburg]]
* [http://www.zdv.uni-tuebingen.de/dienstleistungen/computing/zugang-zu-den-ressourcen.html Eberhard Karls University Tübingen]
* [http://www.scc.kit.edu/downloads/sdo/Antrag_Benutzernummer_bwUniCluster.pdf Karlsruhe Institute of Technology (KIT)]
* [[‎BWUniCluster_User_Access_Members_Uni_Heidelberg|Ruprecht-Karls-Universität Heidelberg]]
* [[BWUniCluster_User_Access_Members_Uni_Ulm|Ulm University]]
* [https://kim.uni-hohenheim.de/bwhpc-account University of Hohenheim]
* [[BWUniCluster_User_Access_Members_Uni_Konstanz|University of Konstanz]]
* [[BWUniCluster_User_Access_Members_Uni_Mannheim|University of Mannheim]]
* [https://www.hlrs.de/solutions-services/academic-users/bwunicluster-access/ University of Stuttgart]

 

== Step B: Web Registration ==
After step A, i.e., after issueing the bwUniCluster entitlement, please visit:
* [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
*# Select your home organization from the list and click '''Proceed'''
*# You will be directed to the ''Identity Provider'' of your home organisation
*# Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
*# You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
*# <div>Select unter '''The following services are available''' the service '''bwUniCluster''' [[File:bwRegWebApp_avail_services_pic01.png|center|border|]]</div>
*# Click '''Register'''
*# Finally
*#* for all non-KIT members '''mandatorily''' set a service password for authentication on bwUniCluster
*#* for KIT members '''optionally''' set a service password for authentication on bwUniCluster


 

== Step C: Amendment: bwUniCluster questionnaire ==
Starting June 1st, 2015, usage of bwUniCluster mandatorily requires the
questionnaire

https://www.bwhpc-c5.de/en/ZAS/bwunicluster_survey.php

to be answered. The input is solely used for the
elaboration of future support activities and for planning future HPC
resources. From June 1st, 2015, login to bwUniCluster is only permitted
to users who have already answered the questionnaire. '''''Exception''': For the first 14 days after the web registration login to bwUniCluster is permitted without participation in the questionnaire.''
 
 

== Changing Password ==
By default, for KIT members your bwUniCluster '''password''' to log on matches that one of your KIT
account, while for all non-KIT members your bwUniCluster '''password''' is that one you saved during the web registration (compare step 7 of chapter 1.2).
At any time, you can set a new bwUniCluster password via the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] by carrying out the following steps:
# visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and select your home organization
# authenticate yourself via your home-organizational user id / username and your home-organizational password
# find on the left side '''bwUniCluster''' and select '''Set Service Password'''
# set new service, i.e. bwUniCluster, password, repeat it and click '''Save''' button.
# the page answers e.g. "Das Passwort wurde bei dem Dienst geändert" ("password has been changed")
# proceed to log in using the new password in the next step
 

== Contact ==
If you have questions or problems concerning bwUniCluster registration, please [[Registration_Support_-_bwUniCluster|contact your local hotline]].
 
 

= Login =
After finishing the web registration '''''bwUniCluster''''' is ready for your '''SSH''' based login.
'''''bwUniCluster''''' has two dedicated login nodes. The selection of the login node is done automatically and is based on a round-robin scheduling. Logging in another time the session might run
on the other login node.

Only the secure shell '''SSH''' is allowed to login. Other protocols like '''telnet''' or '''rlogin'''
are not allowed for security reasons.

A connection to '''''bwUniCluster''''' can be established by the command:

<pre>
$ ssh <UserID>@bwunicluster.scc.kit.edu
</pre>

If you are using '''OpenSSH''' (usually installed on Linux based systems) and you want to use a GUI-based application on '''''bwUniCluster''''' you should use the
''ssh'' command with the option ''-X''

<pre>
$ ssh -X <UserID>@bwunicluster.scc.kit.edu
</pre>

== Hints for using PuTTY ==
Start PuTTY, fill in at window ''PuTTY Configuration'' under category ''Session'' the following fields
<pre>
Host Name : <UserID>@bwunicluster.scc.kit.edu
Port : 22
Connection type : SSH
</pre>
and click on ''Open'', enter your password and accept adding ''host key''. Note that you can also name the configured session via ''Save'' and load it later via the given name.

== Hints for using MobaXterm ==

Start MobaXterm, fill in the following fields:
<pre>
Remote name : uc1.scc.kit.edu
Specify user name : <UserID>
Port : 22
</pre>
After that click on 'ok'. Then a terminal will be opened and there you can enter your password.

== Hints for using WinSCP ==

Start WinSCP, fill in the following fields:
<pre>
File Protocol : SCP
Host name : bwunicluster.scc.kit.edu
Port : 22
User name : <UserID>
</pre>
and click on 'Login' and enter your password. Note that you can also name the configured session via ''Save'' and load it later via the given name.

== About UserID / Username ==

'''<UserID>''' of the ssh command is a placeholder for your '''''username''''' at your home
organization together with a '''''prefix''''' as followed:

{| width=450px class="wikitable"
! Home organization !! <UserID>
|-
| Universität Freiburg || ''fr_''username
|-
| Universität Heidelberg || ''hd_''username
|-
| Universität Hohenheim || ''ho_''username
|-
| KIT || username ''(without any prefix)''
|-
| Universität Konstanz || ''kn_''username
|-
| Universität Mannheim || ''ma_''username
|-
| Universität Stuttgart || ''st_''username
|-
| Universität Tübingen || ''tu_''username
|-
| Universität Ulm || ''ul_''username
|-
| Hochschule Aalen || ''aa_''username
|-
| Hochschule Albstadt-Sigmaringen || ''as_''username
|-
| Hochschule Esslingen || ''es_''username
|-
| Hochschule Furtwangen || ''hf_''username
|-
| Hochschule Karlsruhe || ''hk_''username
|-
| HTWG Konstanz || ''ht_''username
|-
| Hochschule Reutlingen || ''hr_''username
|-
| Hochschule Rottenburg || ''ro_''username
|-
| Hochschule für Technik Stuttgart || ''hs_''username
|-
| Hochschule Ulm || ''hu_''username
|-
|}

== Allowed activities on login nodes ==

The login nodes of bwUniCluster are the access point to the compute system and to your bwUniCluster $HOME directory. The login nodes are shared with all the users of bwUniCluster. Therefore, your activities on the login nodes are limited to primarily set up your batch jobs. Your activities may also be:
* '''short''' compilation of your program code and
* '''short''' pre- and postprocessing of your batch jobs.

To guarantee usability for all the users of bwUniCluster '''you must not run your compute jobs on the login nodes'''. Compute jobs must be submitted to the
[[bwUniCluster Batch Jobs|queueing system]]. Any compute job running on the login nodes will be terminated without any notice. Any long-running compilation or any long-running pre- or postprocessing of batch jobs must also be submitted to the [[bwUniCluster Batch Jobs|queueing system]].
 

= [[First_Steps_on_bwHPC_cluster|First steps on bwUniCluster]] =

First and some important steps on bwUniCluster can be found [[First_Steps_on_bwHPC_cluster|here]].

= Deregistration =
If you plan to permanently leave the bwUniCluster, follow the deregister checklist:
# Transfer all your data in $HOME and workspace to your local computer/storage and after that clear off all your data
# Visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
#* Select your home organization from the list and click '''Proceed'''
#* Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
#* You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
#* <div>Select '''Registry Info''' of the service '''bwUniCluster''' (on the left hand side) [[File:bwUniCluster_registration_sidebar.png|center|border|]]</div>
#* Click '''Deregister'''
Note that Step 2 will automatically unsubscribe you from the bwunicluster mail list.

----
[[Category:bwUniCluster|Access]][[Category:Access|bwUniCluster]]

BwUniCluster 2.0 User Access

2017-09-06T09:12:15Z

S Shamsudeen: /* Login */

[[bwUniCluster]] is Baden-Württemberg's general purpose tier 3 high performance computing (HPC)
cluster co-financed by Baden-Württemberg's ministry of science, research and arts and
the shareholders:
* Albert Ludwig University of Freiburg
* Eberhard Karls University, Tübingen
* Karlsruhe Institute of Technology (KIT)
* Ruprecht-Karls-Universität Heidelberg
* Ulm University
* University of Hohenheim
* University of Konstanz
* University of Mannheim
* University of Stuttgart
* HAW BW e.V. (an association of university of applied sciences in Baden-Württemberg)
 
To '''log on''' [[bwUniCluster]] a user account is required. All members of the shareholder
universities can apply for an account.
 

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:left; color:#000;vertical-align:top;" |__TOC__
| [[File:bwUniCluster_17Jan2014_p044-rot_t10.10.00.jpg|center|border|250px|bwUniCluster wiring by Holger Obermaier, copyright: KIT (SCC)]] bwUniCluster wiring © KIT (SCC)
|}

= Registration =

Granting access and issuing a user account for '''''bwUniCluster''''' requires the registration at the KIT service website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] (step B). However, this registration depends on the '''bwUniCluster entitlement''' issued by your university (step A).
 

== Step A: bwUniCluster entitlement for registration ==
Each university issues the bwUniCluster entitlement '''only''' for their own members. Details on the issuing process and/or the entitlement application forms are listed hereafter:

* [[BwCluster_User_Access_Uni_Freiburg|Albert Ludwig University of Freiburg]]
* [http://www.zdv.uni-tuebingen.de/dienstleistungen/computing/zugang-zu-den-ressourcen.html Eberhard Karls University Tübingen]
* [http://www.scc.kit.edu/downloads/sdo/Antrag_Benutzernummer_bwUniCluster.pdf Karlsruhe Institute of Technology (KIT)]
* [[‎BWUniCluster_User_Access_Members_Uni_Heidelberg|Ruprecht-Karls-Universität Heidelberg]]
* [[BWUniCluster_User_Access_Members_Uni_Ulm|Ulm University]]
* [https://kim.uni-hohenheim.de/bwhpc-account University of Hohenheim]
* [[BWUniCluster_User_Access_Members_Uni_Konstanz|University of Konstanz]]
* [[BWUniCluster_User_Access_Members_Uni_Mannheim|University of Mannheim]]
* [https://www.hlrs.de/solutions-services/academic-users/bwunicluster-access/ University of Stuttgart]

 

== Step B: Web Registration ==
After step A, i.e., after issueing the bwUniCluster entitlement, please visit:
* [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
*# Select your home organization from the list and click '''Proceed'''
*# You will be directed to the ''Identity Provider'' of your home organisation
*# Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
*# You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
*# <div>Select unter '''The following services are available''' the service '''bwUniCluster''' [[File:bwRegWebApp_avail_services_pic01.png|center|border|]]</div>
*# Click '''Register'''
*# Finally
*#* for all non-KIT members '''mandatorily''' set a service password for authentication on bwUniCluster
*#* for KIT members '''optionally''' set a service password for authentication on bwUniCluster


 

== Step C: Amendment: bwUniCluster questionnaire ==
Starting June 1st, 2015, usage of bwUniCluster mandatorily requires the
questionnaire

https://www.bwhpc-c5.de/en/ZAS/bwunicluster_survey.php

to be answered. The input is solely used for the
elaboration of future support activities and for planning future HPC
resources. From June 1st, 2015, login to bwUniCluster is only permitted
to users who have already answered the questionnaire. '''''Exception''': For the first 14 days after the web registration login to bwUniCluster is permitted without participation in the questionnaire.''
 
 

== Changing Password ==
By default, for KIT members your bwUniCluster '''password''' to log on matches that one of your KIT
account, while for all non-KIT members your bwUniCluster '''password''' is that one you saved during the web registration (compare step 7 of chapter 1.2).
At any time, you can set a new bwUniCluster password via the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] by carrying out the following steps:
# visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and select your home organization
# authenticate yourself via your home-organizational user id / username and your home-organizational password
# find on the left side '''bwUniCluster''' and select '''Set Service Password'''
# set new service, i.e. bwUniCluster, password, repeat it and click '''Save''' button.
# the page answers e.g. "Das Passwort wurde bei dem Dienst geändert" ("password has been changed")
# proceed to log in using the new password in the next step
 

== Contact ==
If you have questions or problems concerning bwUniCluster registration, please [[Registration_Support_-_bwUniCluster|contact your local hotline]].
 
 

= Login =
After finishing the web registration '''''bwUniCluster''''' is ready for your '''SSH''' based login.
'''''bwUniCluster''''' has two dedicated login nodes. The selection of the login node is done automatically and is based on a round-robin scheduling. Logging in another time the session might run
on the other login node.

Only the secure shell '''SSH''' is allowed to login. Other protocols like '''telnet''' or '''rlogin'''
are not allowed for security reasons.

A connection to '''''bwUniCluster''''' can be established by the command:

<pre>
$ ssh <UserID>@bwunicluster.scc.kit.edu
</pre>

If you are using '''OpenSSH''' (usually installed on Linux based systems) and you want to use a GUI-based application on '''''bwUniCluster''''' you should use the
''ssh'' command with the option ''-X''

<pre>
$ ssh -X <UserID>@bwunicluster.scc.kit.edu
</pre>

== Hints for using PuTTY ==
Start PuTTY, fill in at window ''PuTTY Configuration'' under category ''Session'' the following fields
<pre>
Host Name : <UserID>@bwunicluster.scc.kit.edu
Port : 22
Connection type : SSH
</pre>
and click on ''Open'', enter your password and accept adding ''host key''. Note that you can also name the configured session via ''Save'' and load it later via the given name.

== Hints for using MobaXterm ==

Start MobaXterm, fill in the following fields:
<pre>
Remote name : uc1.scc.kit.edu
Specify user name : <UserID>
Port : 22
</pre>
After that click on 'ok'. Then a terminal will be opened and there you can enter your password.

== Hints for using WinSCP ==

Start WinSCP, fill in the following fields:
<pre>
File Protocol : SCP
Host name : bwunicluster.scc.kit.edu
Port : 22
User name : <UserID>
</pre>
and click on 'Login' and enter your password. Note that you can also name the configured session via ''Save'' and load it later via the given name.

== About UserID / Username ==

'''<UserID>''' of the ssh command is a placeholder for your '''''username''''' at your home
organization together with a '''''prefix''''' as followed:

{| width=450px class="wikitable"
! Home organization !! <UserID>
|-
| Universität Freiburg || ''fr_''username
|-
| Universität Heidelberg || ''hd_''username
|-
| Universität Hohenheim || ''ho_''username
|-
| KIT || username ''(without any prefix)''
|-
| Universität Konstanz || ''kn_''username
|-
| Universität Mannheim || ''ma_''username
|-
| Universität Stuttgart || ''st_''username
|-
| Universität Tübingen || ''tu_''username
|-
| Universität Ulm || ''ul_''username
|-
| Hochschule Aalen || ''aa_''username
|-
| Hochschule Albstadt-Sigmaringen || ''as_''username
|-
| Hochschule Esslingen || ''es_''username
|-
| Hochschule Furtwangen || ''hf_''username
|-
| Hochschule Karlsruhe || ''hk_''username
|-
| HTWG Konstanz || ''ht_''username
|-
| Hochschule Reutlingen || ''hr_''username
|-
| Hochschule Rottenburg || ''ro_''username
|-
| Hochschule für Technik Stuttgart || ''hs_''username
|-
| Hochschule Ulm || ''hu_''username
|-
|}

== Allowed activities on login nodes ==

The login nodes of bwUniCluster are the access point to the compute system and to your bwUniCluster $HOME directory. The login nodes are shared with all the users of bwUniCluster. Therefore, your activities on the login nodes are limited to primarily set up your batch jobs. Your activities may also be:
* '''short''' compilation of your program code and
* '''short''' pre- and postprocessing of your batch jobs.

To guarantee usability for all the users of bwUniCluster '''you must not run your compute jobs on the login nodes'''. Compute jobs must be submitted to the
[[bwUniCluster Batch Jobs|queueing system]]. Any compute job running on the login nodes will be terminated without any notice. Any long-running compilation or any long-running pre- or postprocessing of batch jobs must also be submitted to the [[bwUniCluster Batch Jobs|queueing system]].
 

= [[First_Steps_on_bwHPC_cluster|First steps on bwUniCluster]] =

First and some important steps on bwUniCluster can be found [[First_Steps_on_bwHPC_cluster|here]].

= Deregistration =
If you plan to permanently leave the bwUniCluster, follow the deregister checklist:
# Transfer all your data in $HOME and workspace to your local computer/storage and after that clear off all your data
# Visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
#* Select your home organization from the list and click '''Proceed'''
#* Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
#* You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
#* <div>Select '''Registry Info''' of the service '''bwUniCluster''' (on the left hand side) [[File:bwUniCluster_registration_sidebar.png|center|border|]]</div>
#* Click '''Deregister'''
Note that Step 2 will automatically unsubscribe you from the bwunicluster mail list.

----
[[Category:bwUniCluster|Access]][[Category:Access|bwUniCluster]]

BwUniCluster 2.0 User Access

2017-09-06T08:42:12Z

S Shamsudeen: /* Login */

[[bwUniCluster]] is Baden-Württemberg's general purpose tier 3 high performance computing (HPC)
cluster co-financed by Baden-Württemberg's ministry of science, research and arts and
the shareholders:
* Albert Ludwig University of Freiburg
* Eberhard Karls University, Tübingen
* Karlsruhe Institute of Technology (KIT)
* Ruprecht-Karls-Universität Heidelberg
* Ulm University
* University of Hohenheim
* University of Konstanz
* University of Mannheim
* University of Stuttgart
* HAW BW e.V. (an association of university of applied sciences in Baden-Württemberg)
 
To '''log on''' [[bwUniCluster]] a user account is required. All members of the shareholder
universities can apply for an account.
 

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:left; color:#000;vertical-align:top;" |__TOC__
| [[File:bwUniCluster_17Jan2014_p044-rot_t10.10.00.jpg|center|border|250px|bwUniCluster wiring by Holger Obermaier, copyright: KIT (SCC)]] bwUniCluster wiring © KIT (SCC)
|}

= Registration =

Granting access and issuing a user account for '''''bwUniCluster''''' requires the registration at the KIT service website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] (step B). However, this registration depends on the '''bwUniCluster entitlement''' issued by your university (step A).
 

== Step A: bwUniCluster entitlement for registration ==
Each university issues the bwUniCluster entitlement '''only''' for their own members. Details on the issuing process and/or the entitlement application forms are listed hereafter:

* [[BwCluster_User_Access_Uni_Freiburg|Albert Ludwig University of Freiburg]]
* [http://www.zdv.uni-tuebingen.de/dienstleistungen/computing/zugang-zu-den-ressourcen.html Eberhard Karls University Tübingen]
* [http://www.scc.kit.edu/downloads/sdo/Antrag_Benutzernummer_bwUniCluster.pdf Karlsruhe Institute of Technology (KIT)]
* [[‎BWUniCluster_User_Access_Members_Uni_Heidelberg|Ruprecht-Karls-Universität Heidelberg]]
* [[BWUniCluster_User_Access_Members_Uni_Ulm|Ulm University]]
* [https://kim.uni-hohenheim.de/bwhpc-account University of Hohenheim]
* [[BWUniCluster_User_Access_Members_Uni_Konstanz|University of Konstanz]]
* [[BWUniCluster_User_Access_Members_Uni_Mannheim|University of Mannheim]]
* [https://www.hlrs.de/solutions-services/academic-users/bwunicluster-access/ University of Stuttgart]

 

== Step B: Web Registration ==
After step A, i.e., after issueing the bwUniCluster entitlement, please visit:
* [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
*# Select your home organization from the list and click '''Proceed'''
*# You will be directed to the ''Identity Provider'' of your home organisation
*# Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
*# You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
*# <div>Select unter '''The following services are available''' the service '''bwUniCluster''' [[File:bwRegWebApp_avail_services_pic01.png|center|border|]]</div>
*# Click '''Register'''
*# Finally
*#* for all non-KIT members '''mandatorily''' set a service password for authentication on bwUniCluster
*#* for KIT members '''optionally''' set a service password for authentication on bwUniCluster


 

== Step C: Amendment: bwUniCluster questionnaire ==
Starting June 1st, 2015, usage of bwUniCluster mandatorily requires the
questionnaire

https://www.bwhpc-c5.de/en/ZAS/bwunicluster_survey.php

to be answered. The input is solely used for the
elaboration of future support activities and for planning future HPC
resources. From June 1st, 2015, login to bwUniCluster is only permitted
to users who have already answered the questionnaire. '''''Exception''': For the first 14 days after the web registration login to bwUniCluster is permitted without participation in the questionnaire.''
 
 

== Changing Password ==
By default, for KIT members your bwUniCluster '''password''' to log on matches that one of your KIT
account, while for all non-KIT members your bwUniCluster '''password''' is that one you saved during the web registration (compare step 7 of chapter 1.2).
At any time, you can set a new bwUniCluster password via the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] by carrying out the following steps:
# visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and select your home organization
# authenticate yourself via your home-organizational user id / username and your home-organizational password
# find on the left side '''bwUniCluster''' and select '''Set Service Password'''
# set new service, i.e. bwUniCluster, password, repeat it and click '''Save''' button.
# the page answers e.g. "Das Passwort wurde bei dem Dienst geändert" ("password has been changed")
# proceed to log in using the new password in the next step
 

== Contact ==
If you have questions or problems concerning bwUniCluster registration, please [[Registration_Support_-_bwUniCluster|contact your local hotline]].
 
 

= Login =
After finishing the web registration '''''bwUniCluster''''' is ready for your '''SSH''' based login.
'''''bwUniCluster''''' has two dedicated login nodes. The selection of the login node is done automatically and is based on a round-robin scheduling. Logging in another time the session might run
on the other login node.

Only the secure shell '''SSH''' is allowed to login. Other protocols like '''telnet''' or '''rlogin'''
are not allowed for security reasons.

A connection to '''''bwUniCluster''''' can be established by the command:

<pre>
$ ssh <UserID>@bwunicluster.scc.kit.edu
</pre>

If you are using '''OpenSSH''' (usually installed on Linux based systems) and you want to use a GUI-based application on '''''bwUniCluster''''' you should use the
''ssh'' command with the option ''-X''

<pre>
$ ssh -X <UserID>@bwunicluster.scc.kit.edu
</pre>

== Hints for using PuTTY ==
Start PuTTY, fill in at window ''PuTTY Configuration'' under category ''Session'' the following fields
<pre>
Host Name : <UserID>@bwunicluster.scc.kit.edu
Port : 22
Connection type : SSH
</pre>
and click on ''Open'', enter your password and accept adding ''host key''. Note that you can also name the configured session via ''Save'' and load it later via the given name.

== Hints for using MobaXterm ==

Start WinSCP, fill in the following fields:
<pre>
Remote name : uc1.scc.kit.edu
Specify user name : <UserID>
</pre>
and click on 'Login' and enter your password. Note that you can also name the configured session via ''Save'' and load it later via the given name.

== Hints for using WinSCP ==

Start WinSCP, fill in the following fields:
<pre>
File Protocol : SCP
Host name : bwunicluster.scc.kit.edu
Port : 22
User name : <UserID>
</pre>
and click on 'Login' and enter your password. Note that you can also name the configured session via ''Save'' and load it later via the given name.

== About UserID / Username ==

'''<UserID>''' of the ssh command is a placeholder for your '''''username''''' at your home
organization together with a '''''prefix''''' as followed:

{| width=450px class="wikitable"
! Home organization !! <UserID>
|-
| Universität Freiburg || ''fr_''username
|-
| Universität Heidelberg || ''hd_''username
|-
| Universität Hohenheim || ''ho_''username
|-
| KIT || username ''(without any prefix)''
|-
| Universität Konstanz || ''kn_''username
|-
| Universität Mannheim || ''ma_''username
|-
| Universität Stuttgart || ''st_''username
|-
| Universität Tübingen || ''tu_''username
|-
| Universität Ulm || ''ul_''username
|-
| Hochschule Aalen || ''aa_''username
|-
| Hochschule Albstadt-Sigmaringen || ''as_''username
|-
| Hochschule Esslingen || ''es_''username
|-
| Hochschule Furtwangen || ''hf_''username
|-
| Hochschule Karlsruhe || ''hk_''username
|-
| HTWG Konstanz || ''ht_''username
|-
| Hochschule Reutlingen || ''hr_''username
|-
| Hochschule Rottenburg || ''ro_''username
|-
| Hochschule für Technik Stuttgart || ''hs_''username
|-
| Hochschule Ulm || ''hu_''username
|-
|}

== Allowed activities on login nodes ==

The login nodes of bwUniCluster are the access point to the compute system and to your bwUniCluster $HOME directory. The login nodes are shared with all the users of bwUniCluster. Therefore, your activities on the login nodes are limited to primarily set up your batch jobs. Your activities may also be:
* '''short''' compilation of your program code and
* '''short''' pre- and postprocessing of your batch jobs.

To guarantee usability for all the users of bwUniCluster '''you must not run your compute jobs on the login nodes'''. Compute jobs must be submitted to the
[[bwUniCluster Batch Jobs|queueing system]]. Any compute job running on the login nodes will be terminated without any notice. Any long-running compilation or any long-running pre- or postprocessing of batch jobs must also be submitted to the [[bwUniCluster Batch Jobs|queueing system]].
 

= [[First_Steps_on_bwHPC_cluster|First steps on bwUniCluster]] =

First and some important steps on bwUniCluster can be found [[First_Steps_on_bwHPC_cluster|here]].

= Deregistration =
If you plan to permanently leave the bwUniCluster, follow the deregister checklist:
# Transfer all your data in $HOME and workspace to your local computer/storage and after that clear off all your data
# Visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
#* Select your home organization from the list and click '''Proceed'''
#* Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
#* You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
#* <div>Select '''Registry Info''' of the service '''bwUniCluster''' (on the left hand side) [[File:bwUniCluster_registration_sidebar.png|center|border|]]</div>
#* Click '''Deregister'''
Note that Step 2 will automatically unsubscribe you from the bwunicluster mail list.

----
[[Category:bwUniCluster|Access]][[Category:Access|bwUniCluster]]

BwUniCluster 2.0 User Access

2017-09-06T08:34:06Z

S Shamsudeen: /* Login */

[[bwUniCluster]] is Baden-Württemberg's general purpose tier 3 high performance computing (HPC)
cluster co-financed by Baden-Württemberg's ministry of science, research and arts and
the shareholders:
* Albert Ludwig University of Freiburg
* Eberhard Karls University, Tübingen
* Karlsruhe Institute of Technology (KIT)
* Ruprecht-Karls-Universität Heidelberg
* Ulm University
* University of Hohenheim
* University of Konstanz
* University of Mannheim
* University of Stuttgart
* HAW BW e.V. (an association of university of applied sciences in Baden-Württemberg)
 
To '''log on''' [[bwUniCluster]] a user account is required. All members of the shareholder
universities can apply for an account.
 

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:left; color:#000;vertical-align:top;" |__TOC__
| [[File:bwUniCluster_17Jan2014_p044-rot_t10.10.00.jpg|center|border|250px|bwUniCluster wiring by Holger Obermaier, copyright: KIT (SCC)]] bwUniCluster wiring © KIT (SCC)
|}

= Registration =

Granting access and issuing a user account for '''''bwUniCluster''''' requires the registration at the KIT service website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] (step B). However, this registration depends on the '''bwUniCluster entitlement''' issued by your university (step A).
 

== Step A: bwUniCluster entitlement for registration ==
Each university issues the bwUniCluster entitlement '''only''' for their own members. Details on the issuing process and/or the entitlement application forms are listed hereafter:

* [[BwCluster_User_Access_Uni_Freiburg|Albert Ludwig University of Freiburg]]
* [http://www.zdv.uni-tuebingen.de/dienstleistungen/computing/zugang-zu-den-ressourcen.html Eberhard Karls University Tübingen]
* [http://www.scc.kit.edu/downloads/sdo/Antrag_Benutzernummer_bwUniCluster.pdf Karlsruhe Institute of Technology (KIT)]
* [[‎BWUniCluster_User_Access_Members_Uni_Heidelberg|Ruprecht-Karls-Universität Heidelberg]]
* [[BWUniCluster_User_Access_Members_Uni_Ulm|Ulm University]]
* [https://kim.uni-hohenheim.de/bwhpc-account University of Hohenheim]
* [[BWUniCluster_User_Access_Members_Uni_Konstanz|University of Konstanz]]
* [[BWUniCluster_User_Access_Members_Uni_Mannheim|University of Mannheim]]
* [https://www.hlrs.de/solutions-services/academic-users/bwunicluster-access/ University of Stuttgart]

 

== Step B: Web Registration ==
After step A, i.e., after issueing the bwUniCluster entitlement, please visit:
* [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
*# Select your home organization from the list and click '''Proceed'''
*# You will be directed to the ''Identity Provider'' of your home organisation
*# Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
*# You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
*# <div>Select unter '''The following services are available''' the service '''bwUniCluster''' [[File:bwRegWebApp_avail_services_pic01.png|center|border|]]</div>
*# Click '''Register'''
*# Finally
*#* for all non-KIT members '''mandatorily''' set a service password for authentication on bwUniCluster
*#* for KIT members '''optionally''' set a service password for authentication on bwUniCluster


 

== Step C: Amendment: bwUniCluster questionnaire ==
Starting June 1st, 2015, usage of bwUniCluster mandatorily requires the
questionnaire

https://www.bwhpc-c5.de/en/ZAS/bwunicluster_survey.php

to be answered. The input is solely used for the
elaboration of future support activities and for planning future HPC
resources. From June 1st, 2015, login to bwUniCluster is only permitted
to users who have already answered the questionnaire. '''''Exception''': For the first 14 days after the web registration login to bwUniCluster is permitted without participation in the questionnaire.''
 
 

== Changing Password ==
By default, for KIT members your bwUniCluster '''password''' to log on matches that one of your KIT
account, while for all non-KIT members your bwUniCluster '''password''' is that one you saved during the web registration (compare step 7 of chapter 1.2).
At any time, you can set a new bwUniCluster password via the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] by carrying out the following steps:
# visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and select your home organization
# authenticate yourself via your home-organizational user id / username and your home-organizational password
# find on the left side '''bwUniCluster''' and select '''Set Service Password'''
# set new service, i.e. bwUniCluster, password, repeat it and click '''Save''' button.
# the page answers e.g. "Das Passwort wurde bei dem Dienst geändert" ("password has been changed")
# proceed to log in using the new password in the next step
 

== Contact ==
If you have questions or problems concerning bwUniCluster registration, please [[Registration_Support_-_bwUniCluster|contact your local hotline]].
 
 

= Login =
After finishing the web registration '''''bwUniCluster''''' is ready for your '''SSH''' based login.
'''''bwUniCluster''''' has two dedicated login nodes. The selection of the login node is done automatically and is based on a round-robin scheduling. Logging in another time the session might run
on the other login node.

Only the secure shell '''SSH''' is allowed to login. Other protocols like '''telnet''' or '''rlogin'''
are not allowed for security reasons.

A connection to '''''bwUniCluster''''' can be established by the command:

<pre>
$ ssh <UserID>@bwunicluster.scc.kit.edu
</pre>

If you are using '''OpenSSH''' (usually installed on Linux based systems) and you want to use a GUI-based application on '''''bwUniCluster''''' you should use the
''ssh'' command with the option ''-X''

<pre>
$ ssh -X <UserID>@bwunicluster.scc.kit.edu
</pre>

== Hints for using PuTTY ==
Start PuTTY, fill in at window ''PuTTY Configuration'' under category ''Session'' the following fields
<pre>
Host Name : <UserID>@bwunicluster.scc.kit.edu
Port : 22
Connection type : SSH
</pre>
and click on ''Open'', enter your password and accept adding ''host key''. Note that you can also name the configured session via ''Save'' and load it later via the given name.

== Hints for using WinSCP ==

Start WinSCP, fill in the following fields:
<pre>
File Protocol : SCP
Host name : bwunicluster.scc.kit.edu
Port : 22
User name : <UserID>
</pre>
and click on 'Login' and enter your password. Note that you can also name the configured session via ''Save'' and load it later via the given name.

== About UserID / Username ==

'''<UserID>''' of the ssh command is a placeholder for your '''''username''''' at your home
organization together with a '''''prefix''''' as followed:

{| width=450px class="wikitable"
! Home organization !! <UserID>
|-
| Universität Freiburg || ''fr_''username
|-
| Universität Heidelberg || ''hd_''username
|-
| Universität Hohenheim || ''ho_''username
|-
| KIT || username ''(without any prefix)''
|-
| Universität Konstanz || ''kn_''username
|-
| Universität Mannheim || ''ma_''username
|-
| Universität Stuttgart || ''st_''username
|-
| Universität Tübingen || ''tu_''username
|-
| Universität Ulm || ''ul_''username
|-
| Hochschule Aalen || ''aa_''username
|-
| Hochschule Albstadt-Sigmaringen || ''as_''username
|-
| Hochschule Esslingen || ''es_''username
|-
| Hochschule Furtwangen || ''hf_''username
|-
| Hochschule Karlsruhe || ''hk_''username
|-
| HTWG Konstanz || ''ht_''username
|-
| Hochschule Reutlingen || ''hr_''username
|-
| Hochschule Rottenburg || ''ro_''username
|-
| Hochschule für Technik Stuttgart || ''hs_''username
|-
| Hochschule Ulm || ''hu_''username
|-
|}

== Allowed activities on login nodes ==

The login nodes of bwUniCluster are the access point to the compute system and to your bwUniCluster $HOME directory. The login nodes are shared with all the users of bwUniCluster. Therefore, your activities on the login nodes are limited to primarily set up your batch jobs. Your activities may also be:
* '''short''' compilation of your program code and
* '''short''' pre- and postprocessing of your batch jobs.

To guarantee usability for all the users of bwUniCluster '''you must not run your compute jobs on the login nodes'''. Compute jobs must be submitted to the
[[bwUniCluster Batch Jobs|queueing system]]. Any compute job running on the login nodes will be terminated without any notice. Any long-running compilation or any long-running pre- or postprocessing of batch jobs must also be submitted to the [[bwUniCluster Batch Jobs|queueing system]].
 

= [[First_Steps_on_bwHPC_cluster|First steps on bwUniCluster]] =

First and some important steps on bwUniCluster can be found [[First_Steps_on_bwHPC_cluster|here]].

= Deregistration =
If you plan to permanently leave the bwUniCluster, follow the deregister checklist:
# Transfer all your data in $HOME and workspace to your local computer/storage and after that clear off all your data
# Visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
#* Select your home organization from the list and click '''Proceed'''
#* Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
#* You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
#* <div>Select '''Registry Info''' of the service '''bwUniCluster''' (on the left hand side) [[File:bwUniCluster_registration_sidebar.png|center|border|]]</div>
#* Click '''Deregister'''
Note that Step 2 will automatically unsubscribe you from the bwunicluster mail list.

----
[[Category:bwUniCluster|Access]][[Category:Access|bwUniCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2017-08-29T12:22:03Z

S Shamsudeen:

FAQs concerning best practice of bwUniCluster broadwell (aka "extension") partition.

__TOC__

= Login =
== Are there separate login nodes for the bwUniCluster broadwell partition? ==
* Yes, but primarily to be used for compiling code, for just submitting job either login nodes are ok.

= Compilation =
== Q1 ==

= Job execution =
== Q2 ==

----
[[Category:bwUniCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2017-08-29T12:19:44Z

S Shamsudeen:

FAQs concerning best practice of bwUniCluster broadwell (aka "extension") partition.

__TOC__

== Are there separate login nodes for the bwUniCluster broadwell partition? ==
* Yes, but primarily to be used for compiling code, for just submitting job either login nodes are ok.

----
[[Category:bwUniCluster]]

BwUniCluster2.0/FAQ - broadwell partition

2017-08-29T12:18:02Z

S Shamsudeen: Created page with "FAQs concerning best practice of bwUniCluster broadwell (aka "extension") partition. __TOC__ == Are there separate login nodes for the bwUniCluster broadwell partition? == *..."

BwUniCluster 2.0 User Access

2017-07-18T12:33:01Z

S Shamsudeen:

[[bwUniCluster]] is Baden-Württemberg's general purpose tier 3 high performance computing (HPC)
cluster co-financed by Baden-Württemberg's ministry of science, research and arts and
the shareholders:
* Albert Ludwig University of Freiburg
* Eberhard Karls University, Tübingen
* Karlsruhe Institute of Technology (KIT)
* Ruprecht-Karls-Universität Heidelberg
* Ulm University
* University of Hohenheim
* University of Konstanz
* University of Mannheim
* University of Stuttgart
* HAW BW e.V. (an association of university of applied sciences in Baden-Württemberg)
 
To '''log on''' [[bwUniCluster]] a user account is required. All members of the shareholder
universities can apply for an account.
 

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:left; color:#000;vertical-align:top;" |__TOC__
| [[File:bwUniCluster_17Jan2014_p044-rot_t10.10.00.jpg|center|border|250px|bwUniCluster wiring by Holger Obermaier, copyright: KIT (SCC)]] bwUniCluster wiring © KIT (SCC)
|}

= Registration =

Granting access and issuing a user account for '''''bwUniCluster''''' requires the registration at the KIT service website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] (step B). However, this registration depends on the '''bwUniCluster entitlement''' issued by your university (step A).
 

== Step A: bwUniCluster entitlement for registration ==
Each university issues the bwUniCluster entitlement '''only''' for their own members. Details on the issuing process and/or the entitlement application forms are listed hereafter:

* [[BwCluster_User_Access_Uni_Freiburg|Albert Ludwig University of Freiburg]]
* [http://www.zdv.uni-tuebingen.de/dienstleistungen/computing/zugang-zu-den-ressourcen.html Eberhard Karls University Tübingen]
* [http://www.scc.kit.edu/downloads/sdo/Antrag_Benutzernummer_bwUniCluster.pdf Karlsruhe Institute of Technology (KIT)]
* [[‎BWUniCluster_User_Access_Members_Uni_Heidelberg|Ruprecht-Karls-Universität Heidelberg]]
* [[BWUniCluster_User_Access_Members_Uni_Ulm|Ulm University]]
* [http://www.kim.uni-hohenheim.de/bwhpc University of Hohenheim]
* [[BWUniCluster_User_Access_Members_Uni_Konstanz|University of Konstanz]]
* [[BWUniCluster_User_Access_Members_Uni_Mannheim|University of Mannheim]]
* [https://www.hlrs.de/solutions-services/academic-users/bwunicluster-access/ University of Stuttgart]

 

== Step B: Web Registration ==
After step A, i.e., after issueing the bwUniCluster entitlement, please visit:
* [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
*# Select your home organization from the list and click '''Proceed'''
*# You will be directed to the ''Identity Provider'' of your home organisation
*# Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
*# You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
*# <div>Select unter '''The following services are available''' the service '''bwUniCluster''' [[File:bwRegWebApp_avail_services_pic01.png|center|border|]]</div>
*# Click '''Register'''
*# Finally
*#* for all non-KIT members '''mandatorily''' set a service password for authentication on bwUniCluster
*#* for KIT members '''optionally''' set a service password for authentication on bwUniCluster


 

== Step C: Amendment: bwUniCluster questionnaire ==
Starting June 1st, 2015, usage of bwUniCluster mandatorily requires the
questionnaire

https://www.bwhpc-c5.de/en/ZAS/bwunicluster_survey.php

to be answered. The input is solely used for the
elaboration of future support activities and for planning future HPC
resources. From June 1st, 2015, login to bwUniCluster is only permitted
to users who have already answered the questionnaire. '''''Exception''': For the first 14 days after the web registration login to bwUniCluster is permitted without participation in the questionnaire.''
 
 

== Changing Password ==
By default, for KIT members your bwUniCluster '''password''' to log on matches that one of your KIT
account, while for all non-KIT members your bwUniCluster '''password''' is that one you saved during the web registration (compare step 7 of chapter 1.2).
At any time, you can set a new bwUniCluster password via the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] by carrying out the following steps:
# visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and select your home organization
# authenticate yourself via your home-organizational user id / username and your home-organizational password
# find on the left side '''bwUniCluster''' and select '''Set Service Password'''
# set new service, i.e. bwUniCluster, password, repeat it and click '''Save''' button.
# the page answers e.g. "Das Passwort wurde bei dem Dienst geändert" ("password has been changed")
# proceed to log in using the new password in the next step
 

== Contact ==
If you have questions or problems concerning bwUniCluster registration, please [[Registration_Support_-_bwUniCluster|contact your local hotline]].
 
 

= Login =
After finishing the web registration '''''bwUniCluster''''' is ready for your '''SSH''' based login.
'''''bwUniCluster''''' has two dedicated login nodes. The selection of the login node is done automatically and is based on a round-robin scheduling. Logging in another time the session might run
on the other login node.

Only the secure shell '''SSH''' is allowed to login. Other protocols like '''telnet''' or '''rlogin'''
are not allowed for security reasons.

A connection to '''''bwUniCluster''''' can be established by the command:

<pre>
$ ssh <UserID>@bwunicluster.scc.kit.edu
</pre>

If you are using '''OpenSSH''' (usually installed on Linux based systems) and you want to use a GUI-based application on '''''bwUniCluster''''' you should use the
''ssh'' command with the option ''-X''

<pre>
$ ssh -X <UserID>@bwunicluster.scc.kit.edu
</pre>

== Hints for using PuTTY ==
Start PuTTY, fill in at window ''PuTTY Configuration'' under category ''Session'' the following fields
<pre>
Host Name : <UserID>@bwunicluster.scc.kit.edu
Port : 22
Connection type : SSH
</pre>
and click on ''Open'', enter your password and accept adding ''host key''. Note that you can also name the configured session via ''Save'' and load it later via the given name.

== Hints for using WinSCP ==

Start WinSCP, fill in the following fields:
<pre>
File Protocol : SCP
Host name : bwunicluster.scc.kit.edu
Port : 22
User name : <UserID>
</pre>
and click on 'Login' and enter your password. Note that you can also name the configured session via ''Save'' and load it later via the given name.

== About UserID / Username ==

'''<UserID>''' of the ssh command is a placeholder for your '''''username''''' at your home
organization together with a '''''prefix''''' as followed:

{| width=450px class="wikitable"
! Home organization !! <UserID>
|-
| Universität Freiburg || ''fr_''username
|-
| Universität Heidelberg || ''hd_''username
|-
| Universität Hohenheim || ''ho_''username
|-
| KIT || username ''(without any prefix)''
|-
| Universität Konstanz || ''kn_''username
|-
| Universität Mannheim || ''ma_''username
|-
| Universität Stuttgart || ''st_''username
|-
| Universität Tübingen || ''tu_''username
|-
| Universität Ulm || ''ul_''username
|-
| Hochschule Aalen || ''aa_''username
|-
| Hochschule Albstadt-Sigmaringen || ''as_''username
|-
| Hochschule Esslingen || ''es_''username
|-
| Hochschule Furtwangen || ''hf_''username
|-
| Hochschule Karlsruhe || ''hk_''username
|-
| HTWG Konstanz || ''ht_''username
|-
| Hochschule Reutlingen || ''hr_''username
|-
| Hochschule Rottenburg || ''ro_''username
|-
| Hochschule für Technik Stuttgart || ''hs_''username
|-
| Hochschule Ulm || ''hu_''username
|-
|}

== Allowed activities on login nodes ==

The login nodes of bwUniCluster are the access point to the compute system and to your bwUniCluster $HOME directory. The login nodes are shared with all the users of bwUniCluster. Therefore, your activities on the login nodes are limited to primarily set up your batch jobs. Your activities may also be:
* '''short''' compilation of your program code and
* '''short''' pre- and postprocessing of your batch jobs.

To guarantee usability for all the users of bwUniCluster '''you must not run your compute jobs on the login nodes'''. Compute jobs must be submitted to the
[[bwUniCluster Batch Jobs|queueing system]]. Any compute job running on the login nodes will be terminated without any notice. Any long-running compilation or any long-running pre- or postprocessing of batch jobs must also be submitted to the [[bwUniCluster Batch Jobs|queueing system]].
 

= [[First_Steps_on_bwHPC_cluster|First steps on bwUniCluster]] =

First and some important steps on bwUniCluster can be found [[First_Steps_on_bwHPC_cluster|here]].

= Deregistration =
If you plan to permanently leave the bwUniCluster, follow the deregister checklist:
# Transfer all your data in $HOME and workspace to your local computer/storage and after that clear off all your data
# Visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
#* Select your home organization from the list and click '''Proceed'''
#* Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
#* You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
#* <div>Select '''Registry Info''' of the service '''bwUniCluster''' (on the left hand side) [[File:bwUniCluster_registration_sidebar.png|center|border|]]</div>
#* Click '''Deregister'''
Note that Step 2 will automatically unsubscribe you from the bwunicluster mail list.

----
[[Category:bwUniCluster|Access]][[Category:Access|bwUniCluster]]

BwUniCluster2.0/First Steps

2017-07-18T12:25:38Z

S Shamsudeen:

== Command line interface ==

Any work is to be done via command line interface. A quick guide can be found [https://indico.scc.kit.edu/indico/event/278/material/slides/10.pdf here].

== Software setup ==

Softwares provided by bwHPC have to be loaded according to your need via the [[BwUniCluster_Environment_Modules|software module system]].

== Computation ==

Any kind of computation on any HPC cluster is done as [[bwUniCluster Batch Jobs|batch jobs]].

BwUniCluster2.0/First Steps

2017-07-18T12:25:17Z

S Shamsudeen: S Shamsudeen moved page BwUniCluster First Steps to First Steps on bwHPC cluster

= First steps on HPC Cluster =

== Command line interface ==

Any work is to be done via command line interface. A quick guide can be found [https://indico.scc.kit.edu/indico/event/278/material/slides/10.pdf here].

== Software setup ==

Softwares provided by bwHPC have to be loaded according to your need via the [[BwUniCluster_Environment_Modules|software module system]].

== Computation ==

Any kind of computation on any HPC cluster is done as [[bwUniCluster Batch Jobs|batch jobs]].

BwUniCluster2.0/First Steps

2017-07-18T12:24:18Z

S Shamsudeen: Created page with "= First steps on HPC Cluster = == Command line interface == Any work is to be done via command line interface. A quick guide can be found [https://indico.scc.kit.edu/indico/..."

BwUniCluster 2.0 User Access

2017-05-12T12:38:08Z

S Shamsudeen: /* Step B: Amendment: bwUniCluster questionnaire */

[[bwUniCluster]] is Baden-Württemberg's general purpose tier 3 high performance computing (HPC)
cluster co-financed by Baden-Württemberg's ministry of science, research and arts and
the shareholders:
* Albert Ludwig University of Freiburg
* Eberhard Karls University, Tübingen
* Karlsruhe Institute of Technology (KIT)
* Ruprecht-Karls-Universität Heidelberg
* Ulm University
* University of Hohenheim
* University of Konstanz
* University of Mannheim
* University of Stuttgart
* HAW BW e.V. (an association of university of applied sciences in Baden-Württemberg)
 
To '''log on''' [[bwUniCluster]] a user account is required. All members of the shareholder
universities can apply for an account.
 

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:left; color:#000;vertical-align:top;" |__TOC__
| [[File:bwUniCluster_17Jan2014_p044-rot_t10.10.00.jpg|center|border|250px|bwUniCluster wiring by Holger Obermaier, copyright: KIT (SCC)]] bwUniCluster wiring © KIT (SCC)
|}

= Registration =

Granting access and issuing a user account for '''''bwUniCluster''''' requires the registration at the KIT service website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] (step B). However, this registration depends on the '''bwUniCluster entitlement''' issued by your university (step A).
 

== Step A: bwUniCluster entitlement for registration ==
Each university issues the bwUniCluster entitlement '''only''' for their own members. Details on the issuing process and/or the entitlement application forms are listed hereafter:

* [[BwCluster_User_Access_Uni_Freiburg|Albert Ludwig University of Freiburg]]
* [http://www.zdv.uni-tuebingen.de/dienstleistungen/computing/zugang-zu-den-ressourcen.html Eberhard Karls University Tübingen]
* [http://www.scc.kit.edu/downloads/sdo/Antrag_Benutzernummer_bwUniCluster.pdf Karlsruhe Institute of Technology (KIT)]
* [[‎BWUniCluster_User_Access_Members_Uni_Heidelberg|Ruprecht-Karls-Universität Heidelberg]]
* [[BWUniCluster_User_Access_Members_Uni_Ulm|Ulm University]]
* [http://www.kim.uni-hohenheim.de/bwhpc University of Hohenheim]
* [[BWUniCluster_User_Access_Members_Uni_Konstanz|University of Konstanz]]
* [[BWUniCluster_User_Access_Members_Uni_Mannheim|University of Mannheim]]
* [https://www.hlrs.de/solutions-services/academic-users/bwunicluster-access/ University of Stuttgart]

 

== Step B: Web Registration ==
After step A, i.e., after issueing the bwUniCluster entitlement, please visit:
* [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
*# Select your home organization from the list and click '''Proceed'''
*# You will be directed to the ''Identity Provider'' of your home organisation
*# Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
*# You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
*# <div>Select unter '''The following services are available''' the service '''bwUniCluster''' [[File:bwRegWebApp_avail_services_pic01.png|center|border|]]</div>
*# Click '''Register'''
*# Finally
*#* for all non-KIT members '''mandatorily''' set a service password for authentication on bwUniCluster
*#* for KIT members '''optionally''' set a service password for authentication on bwUniCluster


 

== Step C: Amendment: bwUniCluster questionnaire ==
Starting June 1st, 2015, usage of bwUniCluster mandatorily requires the
questionnaire

https://www.bwhpc-c5.de/en/ZAS/bwunicluster_survey.php

to be answered. The input is solely used for the
elaboration of future support activities and for planning future HPC
resources. From June 1st, 2015, login to bwUniCluster is only permitted
to users who have already answered the questionnaire. '''''Exception''': For the first 14 days after the web registration login to bwUniCluster is permitted without participation in the questionnaire.''
 
 

== Changing Password ==
By default, for KIT members your bwUniCluster '''password''' to log on matches that one of your KIT
account, while for all non-KIT members your bwUniCluster '''password''' is that one you saved during the web registration (compare step 7 of chapter 1.2).
At any time, you can set a new bwUniCluster password via the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] by carrying out the following steps:
# visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and select your home organization
# authenticate yourself via your home-organizational user id / username and your home-organizational password
# find on the left side '''bwUniCluster''' and select '''Set Service Password'''
# set new service, i.e. bwUniCluster, password, repeat it and click '''Save''' button.
# the page answers e.g. "Das Passwort wurde bei dem Dienst geändert" ("password has been changed")
# proceed to log in using the new password in the next step
 

== Contact ==
If you have questions or problems concerning bwUniCluster registration, please [[Registration_Support_-_bwUniCluster|contact your local hotline]].
 
 

= Login =
After finishing the web registration '''''bwUniCluster''''' is ready for your '''SSH''' based login.
'''''bwUniCluster''''' has two dedicated login nodes. The selection of the login node is done automatically and is based on a round-robin scheduling. Logging in another time the session might run
on the other login node.

Only the secure shell '''SSH''' is allowed to login. Other protocols like '''telnet''' or '''rlogin'''
are not allowed for security reasons.

A connection to '''''bwUniCluster''''' can be established by the command:

<pre>
$ ssh <UserID>@bwunicluster.scc.kit.edu
</pre>

If you are using '''OpenSSH''' (usually installed on Linux based systems) and you want to use a GUI-based application on '''''bwUniCluster''''' you should use the
''ssh'' command with the option ''-X''

<pre>
$ ssh -X <UserID>@bwunicluster.scc.kit.edu
</pre>

== Hints for using PuTTY ==
Start PuTTY, fill in at window ''PuTTY Configuration'' under category ''Session'' the following fields
<pre>
Host Name : <UserID>@bwunicluster.scc.kit.edu
Port : 22
Connection type : SSH
</pre>
and click on ''Open'', enter your password and accept adding ''host key''. Note that you can also name the configured session via ''Save'' and load it later via the given name.

== Hints for using WinSCP ==

Start WinSCP, fill in the following fields:
<pre>
File Protocol : SCP
Host name : bwunicluster.scc.kit.edu
Port : 22
User name : <UserID>
</pre>
and click on 'Login' and enter your password. Note that you can also name the configured session via ''Save'' and load it later via the given name.

== About UserID / Username ==

'''<UserID>''' of the ssh command is a placeholder for your '''''username''''' at your home
organization together with a '''''prefix''''' as followed:

{| width=450px class="wikitable"
! Home organization !! <UserID>
|-
| Universität Freiburg || ''fr_''username
|-
| Universität Heidelberg || ''hd_''username
|-
| Universität Hohenheim || ''ho_''username
|-
| KIT || username ''(without any prefix)''
|-
| Universität Konstanz || ''kn_''username
|-
| Universität Mannheim || ''ma_''username
|-
| Universität Stuttgart || ''st_''username
|-
| Universität Tübingen || ''tu_''username
|-
| Universität Ulm || ''ul_''username
|-
| Hochschule Aalen || ''aa_''username
|-
| Hochschule Albstadt-Sigmaringen || ''as_''username
|-
| Hochschule Esslingen || ''es_''username
|-
| Hochschule Furtwangen || ''hf_''username
|-
| Hochschule Karlsruhe || ''hk_''username
|-
| HTWG Konstanz || ''ht_''username
|-
| Hochschule Reutlingen || ''hr_''username
|-
| Hochschule Rottenburg || ''ro_''username
|-
| Hochschule für Technik Stuttgart || ''hs_''username
|-
| Hochschule Ulm || ''hu_''username
|-
|}

== Allowed activities on login nodes ==

The login nodes of bwUniCluster are the access point to the compute system and to your bwUniCluster $HOME directory. The login nodes are shared with all the users of bwUniCluster. Therefore, your activities on the login nodes are limited to primarily set up your batch jobs. Your activities may also be:
* '''short''' compilation of your program code and
* '''short''' pre- and postprocessing of your batch jobs.

To guarantee usability for all the users of bwUniCluster '''you must not run your compute jobs on the login nodes'''. Compute jobs must be submitted to the
[[bwUniCluster Batch Jobs|queueing system]]. Any compute job running on the login nodes will be terminated without any notice. Any long-running compilation or any long-running pre- or postprocessing of batch jobs must also be submitted to the [[bwUniCluster Batch Jobs|queueing system]].
 

= First steps on bwUniCluster =
For setting up your environment to use compilers and installed software please visit:
* [[BwUniCluster_Environment_Modules|bwUniCluster user environment]]
For guides on how to submit compute jobs to bwUniCluster please visit:
* [[bwUniCluster Batch Jobs|bwUniCluster batch jobs]]

= Deregistration =
If you plan to permanently leave the bwUniCluster, follow the deregister checklist:
# Transfer all your data in $HOME and workspace to your local computer/storage and after that clear off all your data
# Visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
#* Select your home organization from the list and click '''Proceed'''
#* Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
#* You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
#* <div>Select '''Registry Info''' of the service '''bwUniCluster''' (on the left hand side) [[File:bwUniCluster_registration_sidebar.png|center|border|]]</div>
#* Click '''Deregister'''
Note that Step 2 will automatically unsubscribe you from the bwunicluster mail list.

----
[[Category:bwUniCluster|Access]][[Category:Access|bwUniCluster]]

Batch Jobs - bwUniCluster Features

2017-05-12T08:16:28Z

S Shamsudeen: /* msub -q queues */

This article contains information on features of the [[Batch_Jobs|batch job system]] only applicable on bwUniCluster.
 
= Job Submission =
== msub Command ==
=== Interactive Jobs ===
The bwUniCluster supports the following additional msub option(s):

{| width=750px class="wikitable"
! colspan="3" | msub Options
|-
! Command line
! Script
! Purpose
|- style="vertical-align:top;"
| -I
|
| Declares the job is to be run interactively.
|}

=== msub -l ''resource_list'' ===
No deviation or additional features to general [[Batch_Jobs|batch job]] setting.

=== msub -q ''queues'' ===
Compute resources such as walltime, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, you must add to your msub command the correct queue class. Details are:

{| width=750px class="wikitable"
! colspan="6" style="background-color:#999999;padding:3px"| msub -q ''queue''
|- style="width:10%;height=20px; text-align:left;"
! style="width:10%;padding:3px"| ''queue''
! style="width:5%;padding:3px"| ''node''
! style="width:15%;padding:3px"| ''default resources''
! style="padding:3px"| ''minimum resources''
! style="padding:3px"| ''maximum resources''
! style="padding:3px"| ''node access policy''
|- style="vertical-align:top; height=20px; text-align:left"
| style="padding:3px ; style="color:#00a000"| develop*
| style="padding:3px"| thin
| style="width:15%;padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=00:30:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px ; style="color:#00a000" | singlenode*
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=00:30:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:30:01
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;vertical-align:top;height=20px; text-align:left;padding:3px ; style="color:#00a000" | verylong*
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=3:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=3:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=6:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px ; style="color:#36c" | extralong**
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=6:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=6:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=14:00:00:00
| style="padding:3px"| singlejob (Only one job can run at a time)
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | fat
| style="padding:3px"| fat
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=32000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=32, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px ; style="color:#00a000" | multinode*
| style="padding:3px"| broadwell
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=2
| style="padding:3px"| ''nodes''=128:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| singlejob
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | dev_multinode
| style="padding:3px"| broadwell
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=2
| style="padding:3px"| ''nodes''=16:''ppn''=28, ''walltime''=00:30:00
| style="padding:3px"| singlejob
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px ; style="color:#b32425" | special**
| style="padding:3px"| broadwell
| style="padding:3px"| ''walltime''=00:30:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:30:00
| style="padding:3px"| ''nodes''=1:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px ; style="color:#b32425" | dev_special**
| style="padding:3px"| broadwell
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:10:00
| style="padding:3px"| ''nodes''=1:''ppn''=28, ''walltime''=00:30:00
| style="padding:3px"| shared
|-
|}
 *Automatic routing. 
**Only accessible to predefined user groups. 
 **It can be accessed only limited amount of time. Also only accessible to predefined user groups. 

Note that ''node access policy''=singlejob means that, irrespective of the requested number of cores, node access is exclusive.
Default resources of a queue class defines walltime, processes and memory if not explicitly given with msub command. Resource list acronyms ''walltime'', ''procs'', ''nodes'' and ''ppn'' are described [[Batch_Jobs#msub_-l_resource_list|here]].

==== Queue class examples ====

* To run your batch job longer than 3 days, please use$ msub -q verylong.

* To run your batch job on one of the [[BwUniCluster_File_System#Components_of_bwUniCluster|fat nodes]], please use$ msub -q fat.
 
 

= Environment Variables for Batch Jobs =
== Additional Moab Environments ==
The bwUniCluster expands the [[Batch_Jobs#Moab Environment Variables|common set of MOAB environment variables]] by the following variable:
{| width=700px class="wikitable"
! colspan="3" style="background-color:#999999;padding:3px"| bwUniCluster specific MOAB variables
|-
! Environment variable
! Description
|-
| MOAB_SUBMITDIR
| Directory of job submission
|}

== Additional Slurm Environments ==
Since the work load manager MOAB on [[bwUniCluster]] uses the resource manager SLURM, the following environment variables of SLURM are added to your environment once your job has started:
{| width=750px class="wikitable"
! colspan="3" style="background-color:#999999;padding:3px"| SLURM variables
|- style="width:25%;height=20px; text-align:left;padding:3px"
! style="width:20%;height=20px; text-align:left;padding:3px"| Environment variables
! style="height=20px; text-align:left;padding:3px"| Description
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_CPUS_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Number of processes per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NODELIST
| style="height=20px; text-align:left;padding:3px"| List of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NUM_NODES
| style="height=20px; text-align:left;padding:3px"| Number of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_MEM_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Memory per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_NPROCS
| style="height=20px; text-align:left;padding:3px"| Total number of processes dedicated to the job
|}
See also:
* [[Batch_Jobs#Batch_Job_.28Slurm.29_Variables_:_bwUniCluster|List of almost all important Slurm environments]]

== Interactive Job Monitoring per Node ==
By default nodes are not used exclusive unless they are requested with ''-l naccesspolicy=singlejob'' as described [[Batch_Jobs#msub_-l_resource_list|here]]. 
If a Job runs exclusive on one node you may do a ssh login to that node. The ssh access will be limited by the set walltime. To get the nodes of your job need to read the environment variable SLURM_JOB_NODELIST during the runtime of the job. It contains all nodes in a shortened way e.g. ''uc1n[344,386]'' or ''uc1n[344-345]''. To expand this string to ''uc1n344 uc1n345'' you can you can use the command expandnodes like:

expandnodes $SLURM_JOB_NODELIST > nodelist

 
 
= Intel MPI parallel Programs =
== Intel MPI without Multithreading ==
MPI parallel programs run faster than serial programs on multi CPU and multi core systems. N-fold spawned processes of the MPI program, i.e., '''MPI tasks''', run simultaneously and communicate via the Message Passing Interface (MPI) paradigm. MPI tasks do not share memory but can be spawned over different nodes.

Generate a wrapper script for '''Intel MPI''', ''job_impi.sh'' containing the following lines:
<source lang="bash">
#!/bin/bash
module load mpi/impi/<placeholder_for_version>
mpiexec.hydra -bootstrap slurm my_par_program
</source>
'''Attention:''' 
Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since MOAB instructs mpirun about number of processes and node hostnames.
Moreover, replace <placeholder_for_version> with the wished version of '''Intel MPI''' to enable the MPI environment.
 
Launching and running 32 Intel MPI tasks on 4 nodes, each requiring 1000 MByte, and running for 5 hours, execute:
<pre>
$ msub -q multinode -l nodes=4:ppn=16,pmem=1000mb,walltime=05:00:00 job_impi.sh
</pre>
 

== Intel MPI with Multithreading ==
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes.

Multiple Intel MPI tasks must be launched by the MPI parallel program '''mpiexec.hydra'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).

'''For Intel MPI''' a job-script to submit a batch job called ''job_impi_omp.sh'' that runs a Intel MPI program with 8 tasks and a tenfold threaded program ''impi_omp_program'' requiring 32000 MByte of total physical memory per task and total wall clock time of 6 hours looks like:


<source lang="bash">
#!/bin/bash
#MSUB -l nodes=4:ppn=20
#MSUB -l walltime=06:00:00
#MSUB -l pmem=3200mb
#MSUB -v MPI_MODULE=mpi/impi
#MSUB -v OMP_NUM_THREADS=10
#MSUB -v MPIRUN_OPTIONS="-binding "domain=omp" -print-rank-map -ppn 2 -envall"
#MSUB -v EXE=./impi_omp_program
#MSUB -N test_impi_omp

#If using more than one MPI task per node please set
export KMP_AFFINITY=scatter
#export KMP_AFFINITY=verbose,scatter prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

module load ${MPI_MODULE}
TASK_COUNT=$((${MOAB_PROCCOUNT}/${OMP_NUM_THREADS}))
echo "${EXE} running on ${MOAB_PROCCOUNT} cores with ${TASK_COUNT} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpiexec.hydra -bootstrap slurm ${MPIRUN_OPTIONS} -n ${TASK_COUNT} ${EXE}"
echo $startexe
exec $startexe
</source>
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores. If you only run one MPI task per node please set KMP_AFFINITY=compact,1,0.

 
Execute the script '''job_impi_omp.sh''' adding the queue class ''multinode'' to your msub command:
<pre>
$ msub -q multinode job_impi_omp.sh
</pre>
 
The mpirun option ''-print-rank-map'' shows the bindings between MPI tasks and nodes (not very beneficial). The option ''-binding'' binds MPI tasks (processes) to a particular processor; ''domain=omp'' means that the domain size is determined by the number of threads. In the above examples (2 MPI tasks per node) you could also choose ''-binding "cell=unit;map=bunch"''; this binding maps one MPI process to each socket.
 
 
= Interactive Jobs =
Interactive jobs on bwUniCluster [[BwUniCluster_User_Access#Allowed_activities_on_login_nodes|must '''NOT''' run on the logins nodes]], however resources for interactive jobs can be requested using msub. Considering a serial application with a graphical frontend that requires 5000 MByte of memory and limiting the interactive run to 2 hours execute the following:
<pre>
$ msub -I -V -l nodes=1:ppn=1 -l walltime=0:02:00:00
</pre>
The option -V defines that all environment variables are exported to the compute node of the interactive session.
After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system MOAB has granted you the requested resources on the compute system. Once granted you will be automatically logged on the dedicated resource. Now you have an interactive session with 1 core and 5000 MByte of memory on the compute system for 2 hours. Simply execute now your application:
<pre>
$ cd to_path
$ ./application
</pre>
Note that, once the walltime limit has been reached you will be automatically logged out of the compute system.
 
 
== Chain Jobs ==
The CPU time requirements of many applications exceed the limits of the job classes. In those situations it is recommended to solve the problem by a job chain. A job chain is a sequence of jobs where each job automatically starts its successor.
<source lang="bash">
#!/bin/bash
##################################################
## simple MOAB submitter script to setup ##
## a chain of jobs for bwUniCluster ##
##################################################
## ver. : 2015-09-17, KIT, SCC

## Define maximum number of jobs via positional parameter 1, default is 5
max_nojob=${1:-5}

## Define your jobscript (e.g. "~/chain_link_job.sh")
chain_link_job=${PWD}/chain_link_job.sh

## Define type of dependency via positional parameter 2, default is 'afterok'
dep_type="${2:-afterok}"
## -> List of all dependencies:
## http://docs.adaptivecomputing.com/suite/8-0/enterprise/help.htm#topics/\
## moabWorkloadManager/topics/jobAdministration/jobdependencies.html

myloop_counter=1
## Submit loop
while [ ${myloop_counter} -le ${max_nojob} ] ; do
##
## Differ msub_opt depending on chain link number
if [ ${myloop_counter} -eq 1 ] ; then
msub_opt=""
else
## Attention: do NOT use '-W depend' together with msub
msub_opt="-l depend=${dep_type}:${jobID}"
fi
##
## Print current iteration number and msub command
echo "Chain job iteration = ${myloop_counter}"
echo " msub -v myloop_counter=${myloop_counter} ${msub_opt} ${chain_link_job}"
## Store job ID for next iteration by storing output of msub command with empty lines
jobID=$(msub -v myloop_counter=${myloop_counter} ${msub_opt} ${chain_link_job} 2>&1 | sed '/^$/d')
##
## Check if ERROR occurred
if [[ "${jobID}" =~ "ERROR" ]] ; then
echo " -> submission failed!" ; exit 1
else
echo " -> job number = ${jobID}"
fi
##
## Increase counter
let myloop_counter+=1
done
</source>
 
 
----
[[Category:bwUniCluster|Batch Jobs - bwUniCluster features]]

Batch Jobs - bwUniCluster Features

2017-05-10T10:41:27Z

S Shamsudeen: /* msub -q queues */

This article contains information on features of the [[Batch_Jobs|batch job system]] only applicable on bwUniCluster.
 
= Job Submission =
== msub Command ==
=== Interactive Jobs ===
The bwUniCluster supports the following additional msub option(s):

{| width=750px class="wikitable"
! colspan="3" | msub Options
|-
! Command line
! Script
! Purpose
|- style="vertical-align:top;"
| -I
|
| Declares the job is to be run interactively.
|}

=== msub -l ''resource_list'' ===
No deviation or additional features to general [[Batch_Jobs|batch job]] setting.

=== msub -q ''queues'' ===
Compute resources such as walltime, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, you must add to your msub command the correct queue class. Details are:

{| width=750px class="wikitable"
! colspan="6" style="background-color:#999999;padding:3px"| msub -q ''queue''
|- style="width:10%;height=20px; text-align:left;"
! style="width:10%;padding:3px"| ''queue''
! style="width:5%;padding:3px"| ''node''
! style="width:15%;padding:3px"| ''default resources''
! style="padding:3px"| ''minimum resources''
! style="padding:3px"| ''maximum resources''
! style="padding:3px"| ''node access policy''
|- style="vertical-align:top; height=20px; text-align:left"
| style="padding:3px ; style="color:#FF0000"| develop*
| style="padding:3px"| thin
| style="width:15%;padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=00:30:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px ; style="color:#FF0000" | singlenode*
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=00:30:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:30:01
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;vertical-align:top;height=20px; text-align:left;padding:3px ; style="color:#FF0000" | verylong*
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=3:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=3:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=6:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px ; style="color:#b32425" | extralong**
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=6:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=6:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=14:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | fat
| style="padding:3px"| fat
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=32000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=32, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px ; style="color:#FF0000" | multinode*
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=2
| style="padding:3px"| ''nodes''=128:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| singlejob
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | dev_multinode
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=2
| style="padding:3px"| ''nodes''=16:''ppn''=28, ''walltime''=00:30:00
| style="padding:3px"| singlejob
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px ; style="color:#b32425" | special**
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:30:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:30:00
| style="padding:3px"| ''nodes''=1:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px ; style="color:#b32425" | dev_special**
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:10:00
| style="padding:3px"| ''nodes''=1:''ppn''=28, ''walltime''=00:30:00
| style="padding:3px"| shared
|-
|}
Note that ''node access policy''=singlejob means that, irrespective of the requested number of cores, node access is exclusive.
Default resources of a queue class defines walltime, processes and memory if not explicitly given with msub command. Resource list acronyms ''walltime'', ''procs'', ''nodes'' and ''ppn'' are described [[Batch_Jobs#msub_-l_resource_list|here]].
* *Automatic routing. 
* [[**Only accessible to predefined user groups.]]
==== Queue class examples ====

* To run your batch job longer than 3 days, please use$ msub -q verylong.

* To run your batch job on one of the [[BwUniCluster_File_System#Components_of_bwUniCluster|fat nodes]], please use$ msub -q fat.
 
 

= Environment Variables for Batch Jobs =
== Additional Moab Environments ==
The bwUniCluster expands the [[Batch_Jobs#Moab Environment Variables|common set of MOAB environment variables]] by the following variable:
{| width=700px class="wikitable"
! colspan="3" style="background-color:#999999;padding:3px"| bwUniCluster specific MOAB variables
|-
! Environment variable
! Description
|-
| MOAB_SUBMITDIR
| Directory of job submission
|}

== Additional Slurm Environments ==
Since the work load manager MOAB on [[bwUniCluster]] uses the resource manager SLURM, the following environment variables of SLURM are added to your environment once your job has started:
{| width=750px class="wikitable"
! colspan="3" style="background-color:#999999;padding:3px"| SLURM variables
|- style="width:25%;height=20px; text-align:left;padding:3px"
! style="width:20%;height=20px; text-align:left;padding:3px"| Environment variables
! style="height=20px; text-align:left;padding:3px"| Description
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_CPUS_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Number of processes per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NODELIST
| style="height=20px; text-align:left;padding:3px"| List of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NUM_NODES
| style="height=20px; text-align:left;padding:3px"| Number of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_MEM_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Memory per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_NPROCS
| style="height=20px; text-align:left;padding:3px"| Total number of processes dedicated to the job
|}
See also:
* [[Batch_Jobs#Batch_Job_.28Slurm.29_Variables_:_bwUniCluster|List of almost all important Slurm environments]]

== Interactive Job Monitoring per Node ==
By default nodes are not used exclusive unless they are requested with ''-l naccesspolicy=singlejob'' as described [[Batch_Jobs#msub_-l_resource_list|here]]. 
If a Job runs exclusive on one node you may do a ssh login to that node. The ssh access will be limited by the set walltime. To get the nodes of your job need to read the environment variable SLURM_JOB_NODELIST during the runtime of the job. It contains all nodes in a shortened way e.g. ''uc1n[344,386]'' or ''uc1n[344-345]''. To expand this string to ''uc1n344 uc1n345'' you can you can use the command expandnodes like:

expandnodes $SLURM_JOB_NODELIST > nodelist

 
 
= Intel MPI parallel Programs =
== Intel MPI without Multithreading ==
MPI parallel programs run faster than serial programs on multi CPU and multi core systems. N-fold spawned processes of the MPI program, i.e., '''MPI tasks''', run simultaneously and communicate via the Message Passing Interface (MPI) paradigm. MPI tasks do not share memory but can be spawned over different nodes.

Generate a wrapper script for '''Intel MPI''', ''job_impi.sh'' containing the following lines:
<source lang="bash">
#!/bin/bash
module load mpi/impi/<placeholder_for_version>
mpiexec.hydra -bootstrap slurm my_par_program
</source>
'''Attention:''' 
Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since MOAB instructs mpirun about number of processes and node hostnames.
Moreover, replace <placeholder_for_version> with the wished version of '''Intel MPI''' to enable the MPI environment.
 
Launching and running 32 Intel MPI tasks on 4 nodes, each requiring 1000 MByte, and running for 5 hours, execute:
<pre>
$ msub -q multinode -l nodes=4:ppn=16,pmem=1000mb,walltime=05:00:00 job_impi.sh
</pre>
 

== Intel MPI with Multithreading ==
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes.

Multiple Intel MPI tasks must be launched by the MPI parallel program '''mpiexec.hydra'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).

'''For Intel MPI''' a job-script to submit a batch job called ''job_impi_omp.sh'' that runs a Intel MPI program with 8 tasks and a tenfold threaded program ''impi_omp_program'' requiring 32000 MByte of total physical memory per task and total wall clock time of 6 hours looks like:


<source lang="bash">
#!/bin/bash
#MSUB -l nodes=4:ppn=20
#MSUB -l walltime=06:00:00
#MSUB -l pmem=3200mb
#MSUB -v MPI_MODULE=mpi/impi
#MSUB -v OMP_NUM_THREADS=10
#MSUB -v MPIRUN_OPTIONS="-binding "domain=omp" -print-rank-map -ppn 2 -envall"
#MSUB -v EXE=./impi_omp_program
#MSUB -N test_impi_omp

#If using more than one MPI task per node please set
export KMP_AFFINITY=scatter
#export KMP_AFFINITY=verbose,scatter prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

module load ${MPI_MODULE}
TASK_COUNT=$((${MOAB_PROCCOUNT}/${OMP_NUM_THREADS}))
echo "${EXE} running on ${MOAB_PROCCOUNT} cores with ${TASK_COUNT} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpiexec.hydra -bootstrap slurm ${MPIRUN_OPTIONS} -n ${TASK_COUNT} ${EXE}"
echo $startexe
exec $startexe
</source>
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores. If you only run one MPI task per node please set KMP_AFFINITY=compact,1,0.

 
Execute the script '''job_impi_omp.sh''' adding the queue class ''multinode'' to your msub command:
<pre>
$ msub -q multinode job_impi_omp.sh
</pre>
 
The mpirun option ''-print-rank-map'' shows the bindings between MPI tasks and nodes (not very beneficial). The option ''-binding'' binds MPI tasks (processes) to a particular processor; ''domain=omp'' means that the domain size is determined by the number of threads. In the above examples (2 MPI tasks per node) you could also choose ''-binding "cell=unit;map=bunch"''; this binding maps one MPI process to each socket.
 
 
= Interactive Jobs =
Interactive jobs on bwUniCluster [[BwUniCluster_User_Access#Allowed_activities_on_login_nodes|must '''NOT''' run on the logins nodes]], however resources for interactive jobs can be requested using msub. Considering a serial application with a graphical frontend that requires 5000 MByte of memory and limiting the interactive run to 2 hours execute the following:
<pre>
$ msub -I -V -l nodes=1:ppn=1 -l walltime=0:02:00:00
</pre>
The option -V defines that all environment variables are exported to the compute node of the interactive session.
After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system MOAB has granted you the requested resources on the compute system. Once granted you will be automatically logged on the dedicated resource. Now you have an interactive session with 1 core and 5000 MByte of memory on the compute system for 2 hours. Simply execute now your application:
<pre>
$ cd to_path
$ ./application
</pre>
Note that, once the walltime limit has been reached you will be automatically logged out of the compute system.
 
 
== Chain Jobs ==
The CPU time requirements of many applications exceed the limits of the job classes. In those situations it is recommended to solve the problem by a job chain. A job chain is a sequence of jobs where each job automatically starts its successor.
<source lang="bash">
#!/bin/bash
##################################################
## simple MOAB submitter script to setup ##
## a chain of jobs for bwUniCluster ##
##################################################
## ver. : 2015-09-17, KIT, SCC

## Define maximum number of jobs via positional parameter 1, default is 5
max_nojob=${1:-5}

## Define your jobscript (e.g. "~/chain_link_job.sh")
chain_link_job=${PWD}/chain_link_job.sh

## Define type of dependency via positional parameter 2, default is 'afterok'
dep_type="${2:-afterok}"
## -> List of all dependencies:
## http://docs.adaptivecomputing.com/suite/8-0/enterprise/help.htm#topics/\
## moabWorkloadManager/topics/jobAdministration/jobdependencies.html

myloop_counter=1
## Submit loop
while [ ${myloop_counter} -le ${max_nojob} ] ; do
##
## Differ msub_opt depending on chain link number
if [ ${myloop_counter} -eq 1 ] ; then
msub_opt=""
else
## Attention: do NOT use '-W depend' together with msub
msub_opt="-l depend=${dep_type}:${jobID}"
fi
##
## Print current iteration number and msub command
echo "Chain job iteration = ${myloop_counter}"
echo " msub -v myloop_counter=${myloop_counter} ${msub_opt} ${chain_link_job}"
## Store job ID for next iteration by storing output of msub command with empty lines
jobID=$(msub -v myloop_counter=${myloop_counter} ${msub_opt} ${chain_link_job} 2>&1 | sed '/^$/d')
##
## Check if ERROR occurred
if [[ "${jobID}" =~ "ERROR" ]] ; then
echo " -> submission failed!" ; exit 1
else
echo " -> job number = ${jobID}"
fi
##
## Increase counter
let myloop_counter+=1
done
</source>
 
 
----
[[Category:bwUniCluster|Batch Jobs - bwUniCluster features]]

Batch Jobs - bwUniCluster Features

2017-05-09T08:49:59Z

S Shamsudeen: /* msub -q queues */

This article contains information on features of the [[Batch_Jobs|batch job system]] only applicable on bwUniCluster.
 
= Job Submission =
== msub Command ==
=== Interactive Jobs ===
The bwUniCluster supports the following additional msub option(s):

{| width=750px class="wikitable"
! colspan="3" | msub Options
|-
! Command line
! Script
! Purpose
|- style="vertical-align:top;"
| -I
|
| Declares the job is to be run interactively.
|}

=== msub -l ''resource_list'' ===
No deviation or additional features to general [[Batch_Jobs|batch job]] setting.

=== msub -q ''queues'' ===
Compute resources such as walltime, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, you must add to your msub command the correct queue class. Details are:

{| width=750px class="wikitable"
! colspan="6" style="background-color:#999999;padding:3px"| msub -q ''queue''
|- style="width:10%;height=20px; text-align:left;"
! style="width:10%;padding:3px"| ''queue''
! style="width:5%;padding:3px"| ''node''
! style="width:15%;padding:3px"| ''default resources''
! style="padding:3px"| ''minimum resources''
! style="padding:3px"| ''maximum resources''
! style="padding:3px"| ''node access policy''
|- style="vertical-align:top; height=20px; text-align:left"
| style="padding:3px ; style="color:#FF0000"| develop*
| style="padding:3px"| thin
| style="width:15%;padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=00:30:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px ; style="color:#FF0000" | singlenode*
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=00:30:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:30:01
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;vertical-align:top;height=20px; text-align:left;padding:3px ; style="color:#FF0000" | verylong*
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=3:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=3:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=6:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px ; style="color:#b32425" | extralong**
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=3:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=3:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=6:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | fat
| style="padding:3px"| fat
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=32000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=32, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px ; style="color:#FF0000" | multinode*
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=2
| style="padding:3px"| ''nodes''=128:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| singlejob
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | dev_multinode
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=2
| style="padding:3px"| ''nodes''=16:''ppn''=28, ''walltime''=00:30:00
| style="padding:3px"| singlejob
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px ; style="color:#b32425" | special**
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:30:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:30:00
| style="padding:3px"| ''nodes''=1:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px ; style="color:#b32425" | dev_special**
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:10:00
| style="padding:3px"| ''nodes''=1:''ppn''=28, ''walltime''=00:30:00
| style="padding:3px"| shared
|-
|}
Note that ''node access policy''=singlejob means that, irrespective of the requested number of cores, node access is exclusive.
Default resources of a queue class defines walltime, processes and memory if not explicitly given with msub command. Resource list acronyms ''walltime'', ''procs'', ''nodes'' and ''ppn'' are described [[Batch_Jobs#msub_-l_resource_list|here]].
* *Automatic routing. 
* [[**Only accessible to predefined user groups.]]
==== Queue class examples ====

* To run your batch job longer than 3 days, please use$ msub -q verylong.

* To run your batch job on one of the [[BwUniCluster_File_System#Components_of_bwUniCluster|fat nodes]], please use$ msub -q fat.
 
 

= Environment Variables for Batch Jobs =
== Additional Moab Environments ==
The bwUniCluster expands the [[Batch_Jobs#Moab Environment Variables|common set of MOAB environment variables]] by the following variable:
{| width=700px class="wikitable"
! colspan="3" style="background-color:#999999;padding:3px"| bwUniCluster specific MOAB variables
|-
! Environment variable
! Description
|-
| MOAB_SUBMITDIR
| Directory of job submission
|}

== Additional Slurm Environments ==
Since the work load manager MOAB on [[bwUniCluster]] uses the resource manager SLURM, the following environment variables of SLURM are added to your environment once your job has started:
{| width=750px class="wikitable"
! colspan="3" style="background-color:#999999;padding:3px"| SLURM variables
|- style="width:25%;height=20px; text-align:left;padding:3px"
! style="width:20%;height=20px; text-align:left;padding:3px"| Environment variables
! style="height=20px; text-align:left;padding:3px"| Description
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_CPUS_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Number of processes per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NODELIST
| style="height=20px; text-align:left;padding:3px"| List of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NUM_NODES
| style="height=20px; text-align:left;padding:3px"| Number of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_MEM_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Memory per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_NPROCS
| style="height=20px; text-align:left;padding:3px"| Total number of processes dedicated to the job
|}
See also:
* [[Batch_Jobs#Batch_Job_.28Slurm.29_Variables_:_bwUniCluster|List of almost all important Slurm environments]]

== Interactive Job Monitoring per Node ==
By default nodes are not used exclusive unless they are requested with ''-l naccesspolicy=singlejob'' as described [[Batch_Jobs#msub_-l_resource_list|here]]. 
If a Job runs exclusive on one node you may do a ssh login to that node. The ssh access will be limited by the set walltime. To get the nodes of your job need to read the environment variable SLURM_JOB_NODELIST during the runtime of the job. It contains all nodes in a shortened way e.g. ''uc1n[344,386]'' or ''uc1n[344-345]''. To expand this string to ''uc1n344 uc1n345'' you can you can use the command expandnodes like:

expandnodes $SLURM_JOB_NODELIST > nodelist

 
 
= Intel MPI parallel Programs =
== Intel MPI without Multithreading ==
MPI parallel programs run faster than serial programs on multi CPU and multi core systems. N-fold spawned processes of the MPI program, i.e., '''MPI tasks''', run simultaneously and communicate via the Message Passing Interface (MPI) paradigm. MPI tasks do not share memory but can be spawned over different nodes.

Generate a wrapper script for '''Intel MPI''', ''job_impi.sh'' containing the following lines:
<source lang="bash">
#!/bin/bash
module load mpi/impi/<placeholder_for_version>
mpiexec.hydra -bootstrap slurm my_par_program
</source>
'''Attention:''' 
Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since MOAB instructs mpirun about number of processes and node hostnames.
Moreover, replace <placeholder_for_version> with the wished version of '''Intel MPI''' to enable the MPI environment.
 
Launching and running 32 Intel MPI tasks on 4 nodes, each requiring 1000 MByte, and running for 5 hours, execute:
<pre>
$ msub -q multinode -l nodes=4:ppn=16,pmem=1000mb,walltime=05:00:00 job_impi.sh
</pre>
 

== Intel MPI with Multithreading ==
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes.

Multiple Intel MPI tasks must be launched by the MPI parallel program '''mpiexec.hydra'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).

'''For Intel MPI''' a job-script to submit a batch job called ''job_impi_omp.sh'' that runs a Intel MPI program with 8 tasks and a tenfold threaded program ''impi_omp_program'' requiring 32000 MByte of total physical memory per task and total wall clock time of 6 hours looks like:


<source lang="bash">
#!/bin/bash
#MSUB -l nodes=4:ppn=20
#MSUB -l walltime=06:00:00
#MSUB -l pmem=3200mb
#MSUB -v MPI_MODULE=mpi/impi
#MSUB -v OMP_NUM_THREADS=10
#MSUB -v MPIRUN_OPTIONS="-binding "domain=omp" -print-rank-map -ppn 2 -envall"
#MSUB -v EXE=./impi_omp_program
#MSUB -N test_impi_omp

#If using more than one MPI task per node please set
export KMP_AFFINITY=scatter
#export KMP_AFFINITY=verbose,scatter prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

module load ${MPI_MODULE}
TASK_COUNT=$((${MOAB_PROCCOUNT}/${OMP_NUM_THREADS}))
echo "${EXE} running on ${MOAB_PROCCOUNT} cores with ${TASK_COUNT} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpiexec.hydra -bootstrap slurm ${MPIRUN_OPTIONS} -n ${TASK_COUNT} ${EXE}"
echo $startexe
exec $startexe
</source>
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores. If you only run one MPI task per node please set KMP_AFFINITY=compact,1,0.

 
Execute the script '''job_impi_omp.sh''' adding the queue class ''multinode'' to your msub command:
<pre>
$ msub -q multinode job_impi_omp.sh
</pre>
 
The mpirun option ''-print-rank-map'' shows the bindings between MPI tasks and nodes (not very beneficial). The option ''-binding'' binds MPI tasks (processes) to a particular processor; ''domain=omp'' means that the domain size is determined by the number of threads. In the above examples (2 MPI tasks per node) you could also choose ''-binding "cell=unit;map=bunch"''; this binding maps one MPI process to each socket.
 
 
= Interactive Jobs =
Interactive jobs on bwUniCluster [[BwUniCluster_User_Access#Allowed_activities_on_login_nodes|must '''NOT''' run on the logins nodes]], however resources for interactive jobs can be requested using msub. Considering a serial application with a graphical frontend that requires 5000 MByte of memory and limiting the interactive run to 2 hours execute the following:
<pre>
$ msub -I -V -l nodes=1:ppn=1 -l walltime=0:02:00:00
</pre>
The option -V defines that all environment variables are exported to the compute node of the interactive session.
After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system MOAB has granted you the requested resources on the compute system. Once granted you will be automatically logged on the dedicated resource. Now you have an interactive session with 1 core and 5000 MByte of memory on the compute system for 2 hours. Simply execute now your application:
<pre>
$ cd to_path
$ ./application
</pre>
Note that, once the walltime limit has been reached you will be automatically logged out of the compute system.
 
 
== Chain Jobs ==
The CPU time requirements of many applications exceed the limits of the job classes. In those situations it is recommended to solve the problem by a job chain. A job chain is a sequence of jobs where each job automatically starts its successor.
<source lang="bash">
#!/bin/bash
##################################################
## simple MOAB submitter script to setup ##
## a chain of jobs for bwUniCluster ##
##################################################
## ver. : 2015-09-17, KIT, SCC

## Define maximum number of jobs via positional parameter 1, default is 5
max_nojob=${1:-5}

## Define your jobscript (e.g. "~/chain_link_job.sh")
chain_link_job=${PWD}/chain_link_job.sh

## Define type of dependency via positional parameter 2, default is 'afterok'
dep_type="${2:-afterok}"
## -> List of all dependencies:
## http://docs.adaptivecomputing.com/suite/8-0/enterprise/help.htm#topics/\
## moabWorkloadManager/topics/jobAdministration/jobdependencies.html

myloop_counter=1
## Submit loop
while [ ${myloop_counter} -le ${max_nojob} ] ; do
##
## Differ msub_opt depending on chain link number
if [ ${myloop_counter} -eq 1 ] ; then
msub_opt=""
else
## Attention: do NOT use '-W depend' together with msub
msub_opt="-l depend=${dep_type}:${jobID}"
fi
##
## Print current iteration number and msub command
echo "Chain job iteration = ${myloop_counter}"
echo " msub -v myloop_counter=${myloop_counter} ${msub_opt} ${chain_link_job}"
## Store job ID for next iteration by storing output of msub command with empty lines
jobID=$(msub -v myloop_counter=${myloop_counter} ${msub_opt} ${chain_link_job} 2>&1 | sed '/^$/d')
##
## Check if ERROR occurred
if [[ "${jobID}" =~ "ERROR" ]] ; then
echo " -> submission failed!" ; exit 1
else
echo " -> job number = ${jobID}"
fi
##
## Increase counter
let myloop_counter+=1
done
</source>
 
 
----
[[Category:bwUniCluster|Batch Jobs - bwUniCluster features]]

Batch Jobs - bwUniCluster Features

2017-05-09T08:27:02Z

S Shamsudeen: /* msub -q queues */

This article contains information on features of the [[Batch_Jobs|batch job system]] only applicable on bwUniCluster.
 
= Job Submission =
== msub Command ==
=== Interactive Jobs ===
The bwUniCluster supports the following additional msub option(s):

{| width=750px class="wikitable"
! colspan="3" | msub Options
|-
! Command line
! Script
! Purpose
|- style="vertical-align:top;"
| -I
|
| Declares the job is to be run interactively.
|}

=== msub -l ''resource_list'' ===
No deviation or additional features to general [[Batch_Jobs|batch job]] setting.

=== msub -q ''queues'' ===
Compute resources such as walltime, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, you must add to your msub command the correct queue class. Details are:

{| width=750px class="wikitable"
! colspan="6" style="background-color:#999999;padding:3px"| msub -q ''queue''
|- style="width:10%;height=20px; text-align:left;"
! style="width:10%;padding:3px"| ''queue''
! style="width:5%;padding:3px"| ''node''
! style="width:15%;padding:3px"| ''default resources''
! style="padding:3px"| ''minimum resources''
! style="padding:3px"| ''maximum resources''
! style="padding:3px"| ''node access policy''
|- style="vertical-align:top; height=20px; text-align:left"
| style="padding:3px"| develop
| style="padding:3px"| thin
| style="width:15%;padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=00:30:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | singlenode
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=00:30:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:30:01
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;vertical-align:top;height=20px; text-align:left;padding:3px" | verylong
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=3:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=3:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=6:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | extralong**
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=3:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=3:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=6:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | fat
| style="padding:3px"| fat
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=32000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=32, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | multinode
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=2
| style="padding:3px"| ''nodes''=128:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| singlejob
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | dev_multinode
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=2
| style="padding:3px"| ''nodes''=16:''ppn''=28, ''walltime''=00:30:00
| style="padding:3px"| singlejob
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | special**
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:30:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:30:00
| style="padding:3px"| ''nodes''=1:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | dev_special**
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:10:00
| style="padding:3px"| ''nodes''=1:''ppn''=28, ''walltime''=00:30:00
| style="padding:3px"| shared
|-
|}
Note that ''node access policy''=singlejob means that, irrespective of the requested number of cores, node access is exclusive.
Default resources of a queue class defines walltime, processes and memory if not explicitly given with msub command. Resource list acronyms ''walltime'', ''procs'', ''nodes'' and ''ppn'' are described [[Batch_Jobs#msub_-l_resource_list|here]].
* [[**Only accessible to predefined user groups.]]
==== Queue class examples ====

* To run your batch job longer than 3 days, please use$ msub -q verylong.

* To run your batch job on one of the [[BwUniCluster_File_System#Components_of_bwUniCluster|fat nodes]], please use$ msub -q fat.
 
 

= Environment Variables for Batch Jobs =
== Additional Moab Environments ==
The bwUniCluster expands the [[Batch_Jobs#Moab Environment Variables|common set of MOAB environment variables]] by the following variable:
{| width=700px class="wikitable"
! colspan="3" style="background-color:#999999;padding:3px"| bwUniCluster specific MOAB variables
|-
! Environment variable
! Description
|-
| MOAB_SUBMITDIR
| Directory of job submission
|}

== Additional Slurm Environments ==
Since the work load manager MOAB on [[bwUniCluster]] uses the resource manager SLURM, the following environment variables of SLURM are added to your environment once your job has started:
{| width=750px class="wikitable"
! colspan="3" style="background-color:#999999;padding:3px"| SLURM variables
|- style="width:25%;height=20px; text-align:left;padding:3px"
! style="width:20%;height=20px; text-align:left;padding:3px"| Environment variables
! style="height=20px; text-align:left;padding:3px"| Description
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_CPUS_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Number of processes per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NODELIST
| style="height=20px; text-align:left;padding:3px"| List of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NUM_NODES
| style="height=20px; text-align:left;padding:3px"| Number of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_MEM_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Memory per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_NPROCS
| style="height=20px; text-align:left;padding:3px"| Total number of processes dedicated to the job
|}
See also:
* [[Batch_Jobs#Batch_Job_.28Slurm.29_Variables_:_bwUniCluster|List of almost all important Slurm environments]]

== Interactive Job Monitoring per Node ==
By default nodes are not used exclusive unless they are requested with ''-l naccesspolicy=singlejob'' as described [[Batch_Jobs#msub_-l_resource_list|here]]. 
If a Job runs exclusive on one node you may do a ssh login to that node. The ssh access will be limited by the set walltime. To get the nodes of your job need to read the environment variable SLURM_JOB_NODELIST during the runtime of the job. It contains all nodes in a shortened way e.g. ''uc1n[344,386]'' or ''uc1n[344-345]''. To expand this string to ''uc1n344 uc1n345'' you can you can use the command expandnodes like:

expandnodes $SLURM_JOB_NODELIST > nodelist

 
 
= Intel MPI parallel Programs =
== Intel MPI without Multithreading ==
MPI parallel programs run faster than serial programs on multi CPU and multi core systems. N-fold spawned processes of the MPI program, i.e., '''MPI tasks''', run simultaneously and communicate via the Message Passing Interface (MPI) paradigm. MPI tasks do not share memory but can be spawned over different nodes.

Generate a wrapper script for '''Intel MPI''', ''job_impi.sh'' containing the following lines:
<source lang="bash">
#!/bin/bash
module load mpi/impi/<placeholder_for_version>
mpiexec.hydra -bootstrap slurm my_par_program
</source>
'''Attention:''' 
Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since MOAB instructs mpirun about number of processes and node hostnames.
Moreover, replace <placeholder_for_version> with the wished version of '''Intel MPI''' to enable the MPI environment.
 
Launching and running 32 Intel MPI tasks on 4 nodes, each requiring 1000 MByte, and running for 5 hours, execute:
<pre>
$ msub -q multinode -l nodes=4:ppn=16,pmem=1000mb,walltime=05:00:00 job_impi.sh
</pre>
 

== Intel MPI with Multithreading ==
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes.

Multiple Intel MPI tasks must be launched by the MPI parallel program '''mpiexec.hydra'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).

'''For Intel MPI''' a job-script to submit a batch job called ''job_impi_omp.sh'' that runs a Intel MPI program with 8 tasks and a tenfold threaded program ''impi_omp_program'' requiring 32000 MByte of total physical memory per task and total wall clock time of 6 hours looks like:


<source lang="bash">
#!/bin/bash
#MSUB -l nodes=4:ppn=20
#MSUB -l walltime=06:00:00
#MSUB -l pmem=3200mb
#MSUB -v MPI_MODULE=mpi/impi
#MSUB -v OMP_NUM_THREADS=10
#MSUB -v MPIRUN_OPTIONS="-binding "domain=omp" -print-rank-map -ppn 2 -envall"
#MSUB -v EXE=./impi_omp_program
#MSUB -N test_impi_omp

#If using more than one MPI task per node please set
export KMP_AFFINITY=scatter
#export KMP_AFFINITY=verbose,scatter prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

module load ${MPI_MODULE}
TASK_COUNT=$((${MOAB_PROCCOUNT}/${OMP_NUM_THREADS}))
echo "${EXE} running on ${MOAB_PROCCOUNT} cores with ${TASK_COUNT} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpiexec.hydra -bootstrap slurm ${MPIRUN_OPTIONS} -n ${TASK_COUNT} ${EXE}"
echo $startexe
exec $startexe
</source>
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores. If you only run one MPI task per node please set KMP_AFFINITY=compact,1,0.

 
Execute the script '''job_impi_omp.sh''' adding the queue class ''multinode'' to your msub command:
<pre>
$ msub -q multinode job_impi_omp.sh
</pre>
 
The mpirun option ''-print-rank-map'' shows the bindings between MPI tasks and nodes (not very beneficial). The option ''-binding'' binds MPI tasks (processes) to a particular processor; ''domain=omp'' means that the domain size is determined by the number of threads. In the above examples (2 MPI tasks per node) you could also choose ''-binding "cell=unit;map=bunch"''; this binding maps one MPI process to each socket.
 
 
= Interactive Jobs =
Interactive jobs on bwUniCluster [[BwUniCluster_User_Access#Allowed_activities_on_login_nodes|must '''NOT''' run on the logins nodes]], however resources for interactive jobs can be requested using msub. Considering a serial application with a graphical frontend that requires 5000 MByte of memory and limiting the interactive run to 2 hours execute the following:
<pre>
$ msub -I -V -l nodes=1:ppn=1 -l walltime=0:02:00:00
</pre>
The option -V defines that all environment variables are exported to the compute node of the interactive session.
After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system MOAB has granted you the requested resources on the compute system. Once granted you will be automatically logged on the dedicated resource. Now you have an interactive session with 1 core and 5000 MByte of memory on the compute system for 2 hours. Simply execute now your application:
<pre>
$ cd to_path
$ ./application
</pre>
Note that, once the walltime limit has been reached you will be automatically logged out of the compute system.
 
 
== Chain Jobs ==
The CPU time requirements of many applications exceed the limits of the job classes. In those situations it is recommended to solve the problem by a job chain. A job chain is a sequence of jobs where each job automatically starts its successor.
<source lang="bash">
#!/bin/bash
##################################################
## simple MOAB submitter script to setup ##
## a chain of jobs for bwUniCluster ##
##################################################
## ver. : 2015-09-17, KIT, SCC

## Define maximum number of jobs via positional parameter 1, default is 5
max_nojob=${1:-5}

## Define your jobscript (e.g. "~/chain_link_job.sh")
chain_link_job=${PWD}/chain_link_job.sh

## Define type of dependency via positional parameter 2, default is 'afterok'
dep_type="${2:-afterok}"
## -> List of all dependencies:
## http://docs.adaptivecomputing.com/suite/8-0/enterprise/help.htm#topics/\
## moabWorkloadManager/topics/jobAdministration/jobdependencies.html

myloop_counter=1
## Submit loop
while [ ${myloop_counter} -le ${max_nojob} ] ; do
##
## Differ msub_opt depending on chain link number
if [ ${myloop_counter} -eq 1 ] ; then
msub_opt=""
else
## Attention: do NOT use '-W depend' together with msub
msub_opt="-l depend=${dep_type}:${jobID}"
fi
##
## Print current iteration number and msub command
echo "Chain job iteration = ${myloop_counter}"
echo " msub -v myloop_counter=${myloop_counter} ${msub_opt} ${chain_link_job}"
## Store job ID for next iteration by storing output of msub command with empty lines
jobID=$(msub -v myloop_counter=${myloop_counter} ${msub_opt} ${chain_link_job} 2>&1 | sed '/^$/d')
##
## Check if ERROR occurred
if [[ "${jobID}" =~ "ERROR" ]] ; then
echo " -> submission failed!" ; exit 1
else
echo " -> job number = ${jobID}"
fi
##
## Increase counter
let myloop_counter+=1
done
</source>
 
 
----
[[Category:bwUniCluster|Batch Jobs - bwUniCluster features]]

Batch Jobs - bwUniCluster Features

2017-05-08T12:49:49Z

S Shamsudeen: /* msub -q queues */

This article contains information on features of the [[Batch_Jobs|batch job system]] only applicable on bwUniCluster.
 
= Job Submission =
== msub Command ==
=== Interactive Jobs ===
The bwUniCluster supports the following additional msub option(s):

{| width=750px class="wikitable"
! colspan="3" | msub Options
|-
! Command line
! Script
! Purpose
|- style="vertical-align:top;"
| -I
|
| Declares the job is to be run interactively.
|}

=== msub -l ''resource_list'' ===
No deviation or additional features to general [[Batch_Jobs|batch job]] setting.

=== msub -q ''queues'' ===
Compute resources such as walltime, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, you must add to your msub command the correct queue class. Details are:

{| width=750px class="wikitable"
! colspan="6" style="background-color:#999999;padding:3px"| msub -q ''queue''
|- style="width:10%;height=20px; text-align:left;"
! style="width:10%;padding:3px"| ''queue''
! style="width:5%;padding:3px"| ''node''
! style="width:15%;padding:3px"| ''default resources''
! style="padding:3px"| ''minimum resources''
! style="padding:3px"| ''maximum resources''
! style="padding:3px"| ''node access policy''
|- style="vertical-align:top; height=20px; text-align:left"
| style="padding:3px"| develop
| style="padding:3px"| thin
| style="width:15%;padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=00:30:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | singlenode
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=00:30:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:30:01
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;vertical-align:top;height=20px; text-align:left;padding:3px" | verylong
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=3:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=3:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=6:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | extralong**
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=3:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=3:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=6:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | fat
| style="padding:3px"| fat
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=32000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=32, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | multinode
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=2
| style="padding:3px"| ''nodes''=128:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| singlejob
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | dev_multinode
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=2
| style="padding:3px"| ''nodes''=16:''ppn''=28, ''walltime''=00:30:00
| style="padding:3px"| singlejob
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | special**
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:30:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:30:00
| style="padding:3px"| ''nodes''=1:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | dev_special**
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:10:00
| style="padding:3px"| ''nodes''=1:''ppn''=28, ''walltime''=00:30:00
| style="padding:3px"| shared
|-
|}
Note that ''node access policy''=singlejob means that, irrespective of the requested number of cores, node access is exclusive.
Default resources of a queue class defines walltime, processes and memory if not explicitly given with msub command. Resource list acronyms ''walltime'', ''procs'', ''nodes'' and ''ppn'' are described [[Batch_Jobs#msub_-l_resource_list|here]].

**Only accessible to predefined user groups.

==== Queue class examples ====

* To run your batch job longer than 3 days, please use$ msub -q verylong.

* To run your batch job on one of the [[BwUniCluster_File_System#Components_of_bwUniCluster|fat nodes]], please use$ msub -q fat.
 
 

= Environment Variables for Batch Jobs =
== Additional Moab Environments ==
The bwUniCluster expands the [[Batch_Jobs#Moab Environment Variables|common set of MOAB environment variables]] by the following variable:
{| width=700px class="wikitable"
! colspan="3" style="background-color:#999999;padding:3px"| bwUniCluster specific MOAB variables
|-
! Environment variable
! Description
|-
| MOAB_SUBMITDIR
| Directory of job submission
|}

== Additional Slurm Environments ==
Since the work load manager MOAB on [[bwUniCluster]] uses the resource manager SLURM, the following environment variables of SLURM are added to your environment once your job has started:
{| width=750px class="wikitable"
! colspan="3" style="background-color:#999999;padding:3px"| SLURM variables
|- style="width:25%;height=20px; text-align:left;padding:3px"
! style="width:20%;height=20px; text-align:left;padding:3px"| Environment variables
! style="height=20px; text-align:left;padding:3px"| Description
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_CPUS_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Number of processes per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NODELIST
| style="height=20px; text-align:left;padding:3px"| List of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NUM_NODES
| style="height=20px; text-align:left;padding:3px"| Number of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_MEM_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Memory per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_NPROCS
| style="height=20px; text-align:left;padding:3px"| Total number of processes dedicated to the job
|}
See also:
* [[Batch_Jobs#Batch_Job_.28Slurm.29_Variables_:_bwUniCluster|List of almost all important Slurm environments]]

== Interactive Job Monitoring per Node ==
By default nodes are not used exclusive unless they are requested with ''-l naccesspolicy=singlejob'' as described [[Batch_Jobs#msub_-l_resource_list|here]]. 
If a Job runs exclusive on one node you may do a ssh login to that node. The ssh access will be limited by the set walltime. To get the nodes of your job need to read the environment variable SLURM_JOB_NODELIST during the runtime of the job. It contains all nodes in a shortened way e.g. ''uc1n[344,386]'' or ''uc1n[344-345]''. To expand this string to ''uc1n344 uc1n345'' you can you can use the command expandnodes like:

expandnodes $SLURM_JOB_NODELIST > nodelist

 
 
= Intel MPI parallel Programs =
== Intel MPI without Multithreading ==
MPI parallel programs run faster than serial programs on multi CPU and multi core systems. N-fold spawned processes of the MPI program, i.e., '''MPI tasks''', run simultaneously and communicate via the Message Passing Interface (MPI) paradigm. MPI tasks do not share memory but can be spawned over different nodes.

Generate a wrapper script for '''Intel MPI''', ''job_impi.sh'' containing the following lines:
<source lang="bash">
#!/bin/bash
module load mpi/impi/<placeholder_for_version>
mpiexec.hydra -bootstrap slurm my_par_program
</source>
'''Attention:''' 
Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since MOAB instructs mpirun about number of processes and node hostnames.
Moreover, replace <placeholder_for_version> with the wished version of '''Intel MPI''' to enable the MPI environment.
 
Launching and running 32 Intel MPI tasks on 4 nodes, each requiring 1000 MByte, and running for 5 hours, execute:
<pre>
$ msub -q multinode -l nodes=4:ppn=16,pmem=1000mb,walltime=05:00:00 job_impi.sh
</pre>
 

== Intel MPI with Multithreading ==
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes.

Multiple Intel MPI tasks must be launched by the MPI parallel program '''mpiexec.hydra'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).

'''For Intel MPI''' a job-script to submit a batch job called ''job_impi_omp.sh'' that runs a Intel MPI program with 8 tasks and a tenfold threaded program ''impi_omp_program'' requiring 32000 MByte of total physical memory per task and total wall clock time of 6 hours looks like:


<source lang="bash">
#!/bin/bash
#MSUB -l nodes=4:ppn=20
#MSUB -l walltime=06:00:00
#MSUB -l pmem=3200mb
#MSUB -v MPI_MODULE=mpi/impi
#MSUB -v OMP_NUM_THREADS=10
#MSUB -v MPIRUN_OPTIONS="-binding "domain=omp" -print-rank-map -ppn 2 -envall"
#MSUB -v EXE=./impi_omp_program
#MSUB -N test_impi_omp

#If using more than one MPI task per node please set
export KMP_AFFINITY=scatter
#export KMP_AFFINITY=verbose,scatter prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

module load ${MPI_MODULE}
TASK_COUNT=$((${MOAB_PROCCOUNT}/${OMP_NUM_THREADS}))
echo "${EXE} running on ${MOAB_PROCCOUNT} cores with ${TASK_COUNT} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpiexec.hydra -bootstrap slurm ${MPIRUN_OPTIONS} -n ${TASK_COUNT} ${EXE}"
echo $startexe
exec $startexe
</source>
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores. If you only run one MPI task per node please set KMP_AFFINITY=compact,1,0.

 
Execute the script '''job_impi_omp.sh''' adding the queue class ''multinode'' to your msub command:
<pre>
$ msub -q multinode job_impi_omp.sh
</pre>
 
The mpirun option ''-print-rank-map'' shows the bindings between MPI tasks and nodes (not very beneficial). The option ''-binding'' binds MPI tasks (processes) to a particular processor; ''domain=omp'' means that the domain size is determined by the number of threads. In the above examples (2 MPI tasks per node) you could also choose ''-binding "cell=unit;map=bunch"''; this binding maps one MPI process to each socket.
 
 
= Interactive Jobs =
Interactive jobs on bwUniCluster [[BwUniCluster_User_Access#Allowed_activities_on_login_nodes|must '''NOT''' run on the logins nodes]], however resources for interactive jobs can be requested using msub. Considering a serial application with a graphical frontend that requires 5000 MByte of memory and limiting the interactive run to 2 hours execute the following:
<pre>
$ msub -I -V -l nodes=1:ppn=1 -l walltime=0:02:00:00
</pre>
The option -V defines that all environment variables are exported to the compute node of the interactive session.
After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system MOAB has granted you the requested resources on the compute system. Once granted you will be automatically logged on the dedicated resource. Now you have an interactive session with 1 core and 5000 MByte of memory on the compute system for 2 hours. Simply execute now your application:
<pre>
$ cd to_path
$ ./application
</pre>
Note that, once the walltime limit has been reached you will be automatically logged out of the compute system.
 
 
== Chain Jobs ==
The CPU time requirements of many applications exceed the limits of the job classes. In those situations it is recommended to solve the problem by a job chain. A job chain is a sequence of jobs where each job automatically starts its successor.
<source lang="bash">
#!/bin/bash
##################################################
## simple MOAB submitter script to setup ##
## a chain of jobs for bwUniCluster ##
##################################################
## ver. : 2015-09-17, KIT, SCC

## Define maximum number of jobs via positional parameter 1, default is 5
max_nojob=${1:-5}

## Define your jobscript (e.g. "~/chain_link_job.sh")
chain_link_job=${PWD}/chain_link_job.sh

## Define type of dependency via positional parameter 2, default is 'afterok'
dep_type="${2:-afterok}"
## -> List of all dependencies:
## http://docs.adaptivecomputing.com/suite/8-0/enterprise/help.htm#topics/\
## moabWorkloadManager/topics/jobAdministration/jobdependencies.html

myloop_counter=1
## Submit loop
while [ ${myloop_counter} -le ${max_nojob} ] ; do
##
## Differ msub_opt depending on chain link number
if [ ${myloop_counter} -eq 1 ] ; then
msub_opt=""
else
## Attention: do NOT use '-W depend' together with msub
msub_opt="-l depend=${dep_type}:${jobID}"
fi
##
## Print current iteration number and msub command
echo "Chain job iteration = ${myloop_counter}"
echo " msub -v myloop_counter=${myloop_counter} ${msub_opt} ${chain_link_job}"
## Store job ID for next iteration by storing output of msub command with empty lines
jobID=$(msub -v myloop_counter=${myloop_counter} ${msub_opt} ${chain_link_job} 2>&1 | sed '/^$/d')
##
## Check if ERROR occurred
if [[ "${jobID}" =~ "ERROR" ]] ; then
echo " -> submission failed!" ; exit 1
else
echo " -> job number = ${jobID}"
fi
##
## Increase counter
let myloop_counter+=1
done
</source>
 
 
----
[[Category:bwUniCluster|Batch Jobs - bwUniCluster features]]

Batch Jobs - bwUniCluster Features

2017-05-08T10:46:08Z

S Shamsudeen: /* msub -q queues */

This article contains information on features of the [[Batch_Jobs|batch job system]] only applicable on bwUniCluster.
 
= Job Submission =
== msub Command ==
=== Interactive Jobs ===
The bwUniCluster supports the following additional msub option(s):

{| width=750px class="wikitable"
! colspan="3" | msub Options
|-
! Command line
! Script
! Purpose
|- style="vertical-align:top;"
| -I
|
| Declares the job is to be run interactively.
|}

=== msub -l ''resource_list'' ===
No deviation or additional features to general [[Batch_Jobs|batch job]] setting.

=== msub -q ''queues'' ===
Compute resources such as walltime, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, you must add to your msub command the correct queue class. Details are:

{| width=750px class="wikitable"
! colspan="6" style="background-color:#999999;padding:3px"| msub -q ''queue''
|- style="width:10%;height=20px; text-align:left;"
! style="width:10%;padding:3px"| ''queue''
! style="width:5%;padding:3px"| ''node''
! style="width:15%;padding:3px"| ''default resources''
! style="padding:3px"| ''minimum resources''
! style="padding:3px"| ''maximum resources''
! style="padding:3px"| ''node access policy''
|- style="vertical-align:top; height=20px; text-align:left"
| style="padding:3px"| develop
| style="padding:3px"| thin
| style="width:15%;padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=00:30:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | singlenode
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=00:30:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:30:01
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;vertical-align:top;height=20px; text-align:left;padding:3px" | verylong
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=3:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=3:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=6:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | extralong
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=3:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=3:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=6:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | fat
| style="padding:3px"| fat
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=32000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=32, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | multinode
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=2
| style="padding:3px"| ''nodes''=128:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| singlejob
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | dev_multinode
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=2
| style="padding:3px"| ''nodes''=16:''ppn''=28, ''walltime''=00:30:00
| style="padding:3px"| singlejob
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | special
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:30:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:30:00
| style="padding:3px"| ''nodes''=1:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | dev_special
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:10:00
| style="padding:3px"| ''nodes''=1:''ppn''=28, ''walltime''=00:30:00
| style="padding:3px"| shared
|-
|}
Note that ''node access policy''=singlejob means that, irrespected of the requested number of cores, node access is exclusive.
Default resources of a queue class defines walltime, processes and memory if not explicitly given with msub command. Resource list acronyms ''walltime'', ''procs'', ''nodes'' and ''ppn'' are described [[Batch_Jobs#msub_-l_resource_list|here]].

==== Queue class examples ====

* To run your batch job longer than 3 days, please use$ msub -q verylong.

* To run your batch job on one of the [[BwUniCluster_File_System#Components_of_bwUniCluster|fat nodes]], please use$ msub -q fat.
 
 

= Environment Variables for Batch Jobs =
== Additional Moab Environments ==
The bwUniCluster expands the [[Batch_Jobs#Moab Environment Variables|common set of MOAB environment variables]] by the following variable:
{| width=700px class="wikitable"
! colspan="3" style="background-color:#999999;padding:3px"| bwUniCluster specific MOAB variables
|-
! Environment variable
! Description
|-
| MOAB_SUBMITDIR
| Directory of job submission
|}

== Additional Slurm Environments ==
Since the work load manager MOAB on [[bwUniCluster]] uses the resource manager SLURM, the following environment variables of SLURM are added to your environment once your job has started:
{| width=750px class="wikitable"
! colspan="3" style="background-color:#999999;padding:3px"| SLURM variables
|- style="width:25%;height=20px; text-align:left;padding:3px"
! style="width:20%;height=20px; text-align:left;padding:3px"| Environment variables
! style="height=20px; text-align:left;padding:3px"| Description
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_CPUS_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Number of processes per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NODELIST
| style="height=20px; text-align:left;padding:3px"| List of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NUM_NODES
| style="height=20px; text-align:left;padding:3px"| Number of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_MEM_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Memory per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_NPROCS
| style="height=20px; text-align:left;padding:3px"| Total number of processes dedicated to the job
|}
See also:
* [[Batch_Jobs#Batch_Job_.28Slurm.29_Variables_:_bwUniCluster|List of almost all important Slurm environments]]

== Interactive Job Monitoring per Node ==
By default nodes are not used exclusive unless they are requested with ''-l naccesspolicy=singlejob'' as described [[Batch_Jobs#msub_-l_resource_list|here]]. 
If a Job runs exclusive on one node you may do a ssh login to that node. The ssh access will be limited by the set walltime. To get the nodes of your job need to read the environment variable SLURM_JOB_NODELIST during the runtime of the job. It contains all nodes in a shortened way e.g. ''uc1n[344,386]'' or ''uc1n[344-345]''. To expand this string to ''uc1n344 uc1n345'' you can you can use the command expandnodes like:

expandnodes $SLURM_JOB_NODELIST > nodelist

 
 
= Intel MPI parallel Programs =
== Intel MPI without Multithreading ==
MPI parallel programs run faster than serial programs on multi CPU and multi core systems. N-fold spawned processes of the MPI program, i.e., '''MPI tasks''', run simultaneously and communicate via the Message Passing Interface (MPI) paradigm. MPI tasks do not share memory but can be spawned over different nodes.

Generate a wrapper script for '''Intel MPI''', ''job_impi.sh'' containing the following lines:
<source lang="bash">
#!/bin/bash
module load mpi/impi/<placeholder_for_version>
mpiexec.hydra -bootstrap slurm my_par_program
</source>
'''Attention:''' 
Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since MOAB instructs mpirun about number of processes and node hostnames.
Moreover, replace <placeholder_for_version> with the wished version of '''Intel MPI''' to enable the MPI environment.
 
Launching and running 32 Intel MPI tasks on 4 nodes, each requiring 1000 MByte, and running for 5 hours, execute:
<pre>
$ msub -q multinode -l nodes=4:ppn=16,pmem=1000mb,walltime=05:00:00 job_impi.sh
</pre>
 

== Intel MPI with Multithreading ==
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes.

Multiple Intel MPI tasks must be launched by the MPI parallel program '''mpiexec.hydra'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).

'''For Intel MPI''' a job-script to submit a batch job called ''job_impi_omp.sh'' that runs a Intel MPI program with 8 tasks and a tenfold threaded program ''impi_omp_program'' requiring 32000 MByte of total physical memory per task and total wall clock time of 6 hours looks like:


<source lang="bash">
#!/bin/bash
#MSUB -l nodes=4:ppn=20
#MSUB -l walltime=06:00:00
#MSUB -l pmem=3200mb
#MSUB -v MPI_MODULE=mpi/impi
#MSUB -v OMP_NUM_THREADS=10
#MSUB -v MPIRUN_OPTIONS="-binding "domain=omp" -print-rank-map -ppn 2 -envall"
#MSUB -v EXE=./impi_omp_program
#MSUB -N test_impi_omp

#If using more than one MPI task per node please set
export KMP_AFFINITY=scatter
#export KMP_AFFINITY=verbose,scatter prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

module load ${MPI_MODULE}
TASK_COUNT=$((${MOAB_PROCCOUNT}/${OMP_NUM_THREADS}))
echo "${EXE} running on ${MOAB_PROCCOUNT} cores with ${TASK_COUNT} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpiexec.hydra -bootstrap slurm ${MPIRUN_OPTIONS} -n ${TASK_COUNT} ${EXE}"
echo $startexe
exec $startexe
</source>
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores. If you only run one MPI task per node please set KMP_AFFINITY=compact,1,0.

 
Execute the script '''job_impi_omp.sh''' adding the queue class ''multinode'' to your msub command:
<pre>
$ msub -q multinode job_impi_omp.sh
</pre>
 
The mpirun option ''-print-rank-map'' shows the bindings between MPI tasks and nodes (not very beneficial). The option ''-binding'' binds MPI tasks (processes) to a particular processor; ''domain=omp'' means that the domain size is determined by the number of threads. In the above examples (2 MPI tasks per node) you could also choose ''-binding "cell=unit;map=bunch"''; this binding maps one MPI process to each socket.
 
 
= Interactive Jobs =
Interactive jobs on bwUniCluster [[BwUniCluster_User_Access#Allowed_activities_on_login_nodes|must '''NOT''' run on the logins nodes]], however resources for interactive jobs can be requested using msub. Considering a serial application with a graphical frontend that requires 5000 MByte of memory and limiting the interactive run to 2 hours execute the following:
<pre>
$ msub -I -V -l nodes=1:ppn=1 -l walltime=0:02:00:00
</pre>
The option -V defines that all environment variables are exported to the compute node of the interactive session.
After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system MOAB has granted you the requested resources on the compute system. Once granted you will be automatically logged on the dedicated resource. Now you have an interactive session with 1 core and 5000 MByte of memory on the compute system for 2 hours. Simply execute now your application:
<pre>
$ cd to_path
$ ./application
</pre>
Note that, once the walltime limit has been reached you will be automatically logged out of the compute system.
 
 
== Chain Jobs ==
The CPU time requirements of many applications exceed the limits of the job classes. In those situations it is recommended to solve the problem by a job chain. A job chain is a sequence of jobs where each job automatically starts its successor.
<source lang="bash">
#!/bin/bash
##################################################
## simple MOAB submitter script to setup ##
## a chain of jobs for bwUniCluster ##
##################################################
## ver. : 2015-09-17, KIT, SCC

## Define maximum number of jobs via positional parameter 1, default is 5
max_nojob=${1:-5}

## Define your jobscript (e.g. "~/chain_link_job.sh")
chain_link_job=${PWD}/chain_link_job.sh

## Define type of dependency via positional parameter 2, default is 'afterok'
dep_type="${2:-afterok}"
## -> List of all dependencies:
## http://docs.adaptivecomputing.com/suite/8-0/enterprise/help.htm#topics/\
## moabWorkloadManager/topics/jobAdministration/jobdependencies.html

myloop_counter=1
## Submit loop
while [ ${myloop_counter} -le ${max_nojob} ] ; do
##
## Differ msub_opt depending on chain link number
if [ ${myloop_counter} -eq 1 ] ; then
msub_opt=""
else
## Attention: do NOT use '-W depend' together with msub
msub_opt="-l depend=${dep_type}:${jobID}"
fi
##
## Print current iteration number and msub command
echo "Chain job iteration = ${myloop_counter}"
echo " msub -v myloop_counter=${myloop_counter} ${msub_opt} ${chain_link_job}"
## Store job ID for next iteration by storing output of msub command with empty lines
jobID=$(msub -v myloop_counter=${myloop_counter} ${msub_opt} ${chain_link_job} 2>&1 | sed '/^$/d')
##
## Check if ERROR occurred
if [[ "${jobID}" =~ "ERROR" ]] ; then
echo " -> submission failed!" ; exit 1
else
echo " -> job number = ${jobID}"
fi
##
## Increase counter
let myloop_counter+=1
done
</source>
 
 
----
[[Category:bwUniCluster|Batch Jobs - bwUniCluster features]]

Batch Jobs - bwUniCluster Features

2017-05-08T10:42:56Z

S Shamsudeen: /* msub -q queues */

This article contains information on features of the [[Batch_Jobs|batch job system]] only applicable on bwUniCluster.
 
= Job Submission =
== msub Command ==
=== Interactive Jobs ===
The bwUniCluster supports the following additional msub option(s):

{| width=750px class="wikitable"
! colspan="3" | msub Options
|-
! Command line
! Script
! Purpose
|- style="vertical-align:top;"
| -I
|
| Declares the job is to be run interactively.
|}

=== msub -l ''resource_list'' ===
No deviation or additional features to general [[Batch_Jobs|batch job]] setting.

=== msub -q ''queues'' ===
Compute resources such as walltime, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, you must add to your msub command the correct queue class. Details are:

{| width=750px class="wikitable"
! colspan="6" style="background-color:#999999;padding:3px"| msub -q ''queue''
|- style="width:10%;height=20px; text-align:left;"
! style="width:10%;padding:3px"| ''queue''
! style="width:5%;padding:3px"| ''node''
! style="width:15%;padding:3px"| ''default resources''
! style="padding:3px"| ''minimum resources''
! style="padding:3px"| ''maximum resources''
! style="padding:3px"| ''node access policy''
|- style="vertical-align:top; height=20px; text-align:left"
| style="padding:3px"| develop
| style="padding:3px"| thin
| style="width:15%;padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=00:30:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | singlenode
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=00:30:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:30:01
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;vertical-align:top;height=20px; text-align:left;padding:3px" | verylong
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=3:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=3:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=6:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | extralong
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=3:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=3:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=6:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | fat
| style="padding:3px"| fat
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=32000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=32, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | multinode
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=2
| style="padding:3px"| ''nodes''=128:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| singlejob
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | dev_multinode
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=2
| style="padding:3px"| ''nodes''=16:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| singlejob
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | special
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:30:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:30:00
| style="padding:3px"| ''nodes''=1:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | dev_special
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:10:00
| style="padding:3px"| ''nodes''=1:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| shared
|-
|}
Note that ''node access policy''=singlejob means that, irrespected of the requested number of cores, node access is exclusive.
Default resources of a queue class defines walltime, processes and memory if not explicitly given with msub command. Resource list acronyms ''walltime'', ''procs'', ''nodes'' and ''ppn'' are described [[Batch_Jobs#msub_-l_resource_list|here]].

==== Queue class examples ====

* To run your batch job longer than 3 days, please use$ msub -q verylong.

* To run your batch job on one of the [[BwUniCluster_File_System#Components_of_bwUniCluster|fat nodes]], please use$ msub -q fat.
 
 

= Environment Variables for Batch Jobs =
== Additional Moab Environments ==
The bwUniCluster expands the [[Batch_Jobs#Moab Environment Variables|common set of MOAB environment variables]] by the following variable:
{| width=700px class="wikitable"
! colspan="3" style="background-color:#999999;padding:3px"| bwUniCluster specific MOAB variables
|-
! Environment variable
! Description
|-
| MOAB_SUBMITDIR
| Directory of job submission
|}

== Additional Slurm Environments ==
Since the work load manager MOAB on [[bwUniCluster]] uses the resource manager SLURM, the following environment variables of SLURM are added to your environment once your job has started:
{| width=750px class="wikitable"
! colspan="3" style="background-color:#999999;padding:3px"| SLURM variables
|- style="width:25%;height=20px; text-align:left;padding:3px"
! style="width:20%;height=20px; text-align:left;padding:3px"| Environment variables
! style="height=20px; text-align:left;padding:3px"| Description
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_CPUS_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Number of processes per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NODELIST
| style="height=20px; text-align:left;padding:3px"| List of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NUM_NODES
| style="height=20px; text-align:left;padding:3px"| Number of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_MEM_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Memory per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_NPROCS
| style="height=20px; text-align:left;padding:3px"| Total number of processes dedicated to the job
|}
See also:
* [[Batch_Jobs#Batch_Job_.28Slurm.29_Variables_:_bwUniCluster|List of almost all important Slurm environments]]

== Interactive Job Monitoring per Node ==
By default nodes are not used exclusive unless they are requested with ''-l naccesspolicy=singlejob'' as described [[Batch_Jobs#msub_-l_resource_list|here]]. 
If a Job runs exclusive on one node you may do a ssh login to that node. The ssh access will be limited by the set walltime. To get the nodes of your job need to read the environment variable SLURM_JOB_NODELIST during the runtime of the job. It contains all nodes in a shortened way e.g. ''uc1n[344,386]'' or ''uc1n[344-345]''. To expand this string to ''uc1n344 uc1n345'' you can you can use the command expandnodes like:

expandnodes $SLURM_JOB_NODELIST > nodelist

 
 
= Intel MPI parallel Programs =
== Intel MPI without Multithreading ==
MPI parallel programs run faster than serial programs on multi CPU and multi core systems. N-fold spawned processes of the MPI program, i.e., '''MPI tasks''', run simultaneously and communicate via the Message Passing Interface (MPI) paradigm. MPI tasks do not share memory but can be spawned over different nodes.

Generate a wrapper script for '''Intel MPI''', ''job_impi.sh'' containing the following lines:
<source lang="bash">
#!/bin/bash
module load mpi/impi/<placeholder_for_version>
mpiexec.hydra -bootstrap slurm my_par_program
</source>
'''Attention:''' 
Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since MOAB instructs mpirun about number of processes and node hostnames.
Moreover, replace <placeholder_for_version> with the wished version of '''Intel MPI''' to enable the MPI environment.
 
Launching and running 32 Intel MPI tasks on 4 nodes, each requiring 1000 MByte, and running for 5 hours, execute:
<pre>
$ msub -q multinode -l nodes=4:ppn=16,pmem=1000mb,walltime=05:00:00 job_impi.sh
</pre>
 

== Intel MPI with Multithreading ==
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes.

Multiple Intel MPI tasks must be launched by the MPI parallel program '''mpiexec.hydra'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).

'''For Intel MPI''' a job-script to submit a batch job called ''job_impi_omp.sh'' that runs a Intel MPI program with 8 tasks and a tenfold threaded program ''impi_omp_program'' requiring 32000 MByte of total physical memory per task and total wall clock time of 6 hours looks like:


<source lang="bash">
#!/bin/bash
#MSUB -l nodes=4:ppn=20
#MSUB -l walltime=06:00:00
#MSUB -l pmem=3200mb
#MSUB -v MPI_MODULE=mpi/impi
#MSUB -v OMP_NUM_THREADS=10
#MSUB -v MPIRUN_OPTIONS="-binding "domain=omp" -print-rank-map -ppn 2 -envall"
#MSUB -v EXE=./impi_omp_program
#MSUB -N test_impi_omp

#If using more than one MPI task per node please set
export KMP_AFFINITY=scatter
#export KMP_AFFINITY=verbose,scatter prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

module load ${MPI_MODULE}
TASK_COUNT=$((${MOAB_PROCCOUNT}/${OMP_NUM_THREADS}))
echo "${EXE} running on ${MOAB_PROCCOUNT} cores with ${TASK_COUNT} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpiexec.hydra -bootstrap slurm ${MPIRUN_OPTIONS} -n ${TASK_COUNT} ${EXE}"
echo $startexe
exec $startexe
</source>
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores. If you only run one MPI task per node please set KMP_AFFINITY=compact,1,0.

 
Execute the script '''job_impi_omp.sh''' adding the queue class ''multinode'' to your msub command:
<pre>
$ msub -q multinode job_impi_omp.sh
</pre>
 
The mpirun option ''-print-rank-map'' shows the bindings between MPI tasks and nodes (not very beneficial). The option ''-binding'' binds MPI tasks (processes) to a particular processor; ''domain=omp'' means that the domain size is determined by the number of threads. In the above examples (2 MPI tasks per node) you could also choose ''-binding "cell=unit;map=bunch"''; this binding maps one MPI process to each socket.
 
 
= Interactive Jobs =
Interactive jobs on bwUniCluster [[BwUniCluster_User_Access#Allowed_activities_on_login_nodes|must '''NOT''' run on the logins nodes]], however resources for interactive jobs can be requested using msub. Considering a serial application with a graphical frontend that requires 5000 MByte of memory and limiting the interactive run to 2 hours execute the following:
<pre>
$ msub -I -V -l nodes=1:ppn=1 -l walltime=0:02:00:00
</pre>
The option -V defines that all environment variables are exported to the compute node of the interactive session.
After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system MOAB has granted you the requested resources on the compute system. Once granted you will be automatically logged on the dedicated resource. Now you have an interactive session with 1 core and 5000 MByte of memory on the compute system for 2 hours. Simply execute now your application:
<pre>
$ cd to_path
$ ./application
</pre>
Note that, once the walltime limit has been reached you will be automatically logged out of the compute system.
 
 
== Chain Jobs ==
The CPU time requirements of many applications exceed the limits of the job classes. In those situations it is recommended to solve the problem by a job chain. A job chain is a sequence of jobs where each job automatically starts its successor.
<source lang="bash">
#!/bin/bash
##################################################
## simple MOAB submitter script to setup ##
## a chain of jobs for bwUniCluster ##
##################################################
## ver. : 2015-09-17, KIT, SCC

## Define maximum number of jobs via positional parameter 1, default is 5
max_nojob=${1:-5}

## Define your jobscript (e.g. "~/chain_link_job.sh")
chain_link_job=${PWD}/chain_link_job.sh

## Define type of dependency via positional parameter 2, default is 'afterok'
dep_type="${2:-afterok}"
## -> List of all dependencies:
## http://docs.adaptivecomputing.com/suite/8-0/enterprise/help.htm#topics/\
## moabWorkloadManager/topics/jobAdministration/jobdependencies.html

myloop_counter=1
## Submit loop
while [ ${myloop_counter} -le ${max_nojob} ] ; do
##
## Differ msub_opt depending on chain link number
if [ ${myloop_counter} -eq 1 ] ; then
msub_opt=""
else
## Attention: do NOT use '-W depend' together with msub
msub_opt="-l depend=${dep_type}:${jobID}"
fi
##
## Print current iteration number and msub command
echo "Chain job iteration = ${myloop_counter}"
echo " msub -v myloop_counter=${myloop_counter} ${msub_opt} ${chain_link_job}"
## Store job ID for next iteration by storing output of msub command with empty lines
jobID=$(msub -v myloop_counter=${myloop_counter} ${msub_opt} ${chain_link_job} 2>&1 | sed '/^$/d')
##
## Check if ERROR occurred
if [[ "${jobID}" =~ "ERROR" ]] ; then
echo " -> submission failed!" ; exit 1
else
echo " -> job number = ${jobID}"
fi
##
## Increase counter
let myloop_counter+=1
done
</source>
 
 
----
[[Category:bwUniCluster|Batch Jobs - bwUniCluster features]]

Batch Jobs - bwUniCluster Features

2017-05-08T10:41:08Z

S Shamsudeen: /* msub -q queues */

This article contains information on features of the [[Batch_Jobs|batch job system]] only applicable on bwUniCluster.
 
= Job Submission =
== msub Command ==
=== Interactive Jobs ===
The bwUniCluster supports the following additional msub option(s):

{| width=750px class="wikitable"
! colspan="3" | msub Options
|-
! Command line
! Script
! Purpose
|- style="vertical-align:top;"
| -I
|
| Declares the job is to be run interactively.
|}

=== msub -l ''resource_list'' ===
No deviation or additional features to general [[Batch_Jobs|batch job]] setting.

=== msub -q ''queues'' ===
Compute resources such as walltime, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, you must add to your msub command the correct queue class. Details are:

{| width=750px class="wikitable"
! colspan="6" style="background-color:#999999;padding:3px"| msub -q ''queue''
|- style="width:10%;height=20px; text-align:left;"
! style="width:10%;padding:3px"| ''queue''
! style="width:5%;padding:3px"| ''node''
! style="width:15%;padding:3px"| ''default resources''
! style="padding:3px"| ''minimum resources''
! style="padding:3px"| ''maximum resources''
! style="padding:3px"| ''node access policy''
|- style="vertical-align:top; height=20px; text-align:left"
| style="padding:3px"| develop
| style="padding:3px"| thin
| style="width:15%;padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=00:30:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | singlenode
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=00:30:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:30:01
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;vertical-align:top;height=20px; text-align:left;padding:3px" | verylong
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=3:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=3:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=6:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | extralong
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=3:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=3:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=6:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | fat
| style="padding:3px"| fat
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=32000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=32, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | multinode
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=2
| style="padding:3px"| ''nodes''=128:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| singlejob
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | dev_multinode
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=2
| style="padding:3px"| ''nodes''=16:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | special
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:30:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:30:00
| style="padding:3px"| ''nodes''=1:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | dev_special
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:10:00
| style="padding:3px"| ''nodes''=1:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| shared
|-
|}
Note that ''node access policy''=singlejob means that, irrespected of the requested number of cores, node access is exclusive.
Default resources of a queue class defines walltime, processes and memory if not explicitly given with msub command. Resource list acronyms ''walltime'', ''procs'', ''nodes'' and ''ppn'' are described [[Batch_Jobs#msub_-l_resource_list|here]].

==== Queue class examples ====

* To run your batch job longer than 3 days, please use$ msub -q verylong.

* To run your batch job on one of the [[BwUniCluster_File_System#Components_of_bwUniCluster|fat nodes]], please use$ msub -q fat.
 
 

= Environment Variables for Batch Jobs =
== Additional Moab Environments ==
The bwUniCluster expands the [[Batch_Jobs#Moab Environment Variables|common set of MOAB environment variables]] by the following variable:
{| width=700px class="wikitable"
! colspan="3" style="background-color:#999999;padding:3px"| bwUniCluster specific MOAB variables
|-
! Environment variable
! Description
|-
| MOAB_SUBMITDIR
| Directory of job submission
|}

== Additional Slurm Environments ==
Since the work load manager MOAB on [[bwUniCluster]] uses the resource manager SLURM, the following environment variables of SLURM are added to your environment once your job has started:
{| width=750px class="wikitable"
! colspan="3" style="background-color:#999999;padding:3px"| SLURM variables
|- style="width:25%;height=20px; text-align:left;padding:3px"
! style="width:20%;height=20px; text-align:left;padding:3px"| Environment variables
! style="height=20px; text-align:left;padding:3px"| Description
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_CPUS_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Number of processes per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NODELIST
| style="height=20px; text-align:left;padding:3px"| List of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NUM_NODES
| style="height=20px; text-align:left;padding:3px"| Number of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_MEM_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Memory per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_NPROCS
| style="height=20px; text-align:left;padding:3px"| Total number of processes dedicated to the job
|}
See also:
* [[Batch_Jobs#Batch_Job_.28Slurm.29_Variables_:_bwUniCluster|List of almost all important Slurm environments]]

== Interactive Job Monitoring per Node ==
By default nodes are not used exclusive unless they are requested with ''-l naccesspolicy=singlejob'' as described [[Batch_Jobs#msub_-l_resource_list|here]]. 
If a Job runs exclusive on one node you may do a ssh login to that node. The ssh access will be limited by the set walltime. To get the nodes of your job need to read the environment variable SLURM_JOB_NODELIST during the runtime of the job. It contains all nodes in a shortened way e.g. ''uc1n[344,386]'' or ''uc1n[344-345]''. To expand this string to ''uc1n344 uc1n345'' you can you can use the command expandnodes like:

expandnodes $SLURM_JOB_NODELIST > nodelist

 
 
= Intel MPI parallel Programs =
== Intel MPI without Multithreading ==
MPI parallel programs run faster than serial programs on multi CPU and multi core systems. N-fold spawned processes of the MPI program, i.e., '''MPI tasks''', run simultaneously and communicate via the Message Passing Interface (MPI) paradigm. MPI tasks do not share memory but can be spawned over different nodes.

Generate a wrapper script for '''Intel MPI''', ''job_impi.sh'' containing the following lines:
<source lang="bash">
#!/bin/bash
module load mpi/impi/<placeholder_for_version>
mpiexec.hydra -bootstrap slurm my_par_program
</source>
'''Attention:''' 
Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since MOAB instructs mpirun about number of processes and node hostnames.
Moreover, replace <placeholder_for_version> with the wished version of '''Intel MPI''' to enable the MPI environment.
 
Launching and running 32 Intel MPI tasks on 4 nodes, each requiring 1000 MByte, and running for 5 hours, execute:
<pre>
$ msub -q multinode -l nodes=4:ppn=16,pmem=1000mb,walltime=05:00:00 job_impi.sh
</pre>
 

== Intel MPI with Multithreading ==
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes.

Multiple Intel MPI tasks must be launched by the MPI parallel program '''mpiexec.hydra'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).

'''For Intel MPI''' a job-script to submit a batch job called ''job_impi_omp.sh'' that runs a Intel MPI program with 8 tasks and a tenfold threaded program ''impi_omp_program'' requiring 32000 MByte of total physical memory per task and total wall clock time of 6 hours looks like:


<source lang="bash">
#!/bin/bash
#MSUB -l nodes=4:ppn=20
#MSUB -l walltime=06:00:00
#MSUB -l pmem=3200mb
#MSUB -v MPI_MODULE=mpi/impi
#MSUB -v OMP_NUM_THREADS=10
#MSUB -v MPIRUN_OPTIONS="-binding "domain=omp" -print-rank-map -ppn 2 -envall"
#MSUB -v EXE=./impi_omp_program
#MSUB -N test_impi_omp

#If using more than one MPI task per node please set
export KMP_AFFINITY=scatter
#export KMP_AFFINITY=verbose,scatter prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

module load ${MPI_MODULE}
TASK_COUNT=$((${MOAB_PROCCOUNT}/${OMP_NUM_THREADS}))
echo "${EXE} running on ${MOAB_PROCCOUNT} cores with ${TASK_COUNT} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpiexec.hydra -bootstrap slurm ${MPIRUN_OPTIONS} -n ${TASK_COUNT} ${EXE}"
echo $startexe
exec $startexe
</source>
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores. If you only run one MPI task per node please set KMP_AFFINITY=compact,1,0.

 
Execute the script '''job_impi_omp.sh''' adding the queue class ''multinode'' to your msub command:
<pre>
$ msub -q multinode job_impi_omp.sh
</pre>
 
The mpirun option ''-print-rank-map'' shows the bindings between MPI tasks and nodes (not very beneficial). The option ''-binding'' binds MPI tasks (processes) to a particular processor; ''domain=omp'' means that the domain size is determined by the number of threads. In the above examples (2 MPI tasks per node) you could also choose ''-binding "cell=unit;map=bunch"''; this binding maps one MPI process to each socket.
 
 
= Interactive Jobs =
Interactive jobs on bwUniCluster [[BwUniCluster_User_Access#Allowed_activities_on_login_nodes|must '''NOT''' run on the logins nodes]], however resources for interactive jobs can be requested using msub. Considering a serial application with a graphical frontend that requires 5000 MByte of memory and limiting the interactive run to 2 hours execute the following:
<pre>
$ msub -I -V -l nodes=1:ppn=1 -l walltime=0:02:00:00
</pre>
The option -V defines that all environment variables are exported to the compute node of the interactive session.
After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system MOAB has granted you the requested resources on the compute system. Once granted you will be automatically logged on the dedicated resource. Now you have an interactive session with 1 core and 5000 MByte of memory on the compute system for 2 hours. Simply execute now your application:
<pre>
$ cd to_path
$ ./application
</pre>
Note that, once the walltime limit has been reached you will be automatically logged out of the compute system.
 
 
== Chain Jobs ==
The CPU time requirements of many applications exceed the limits of the job classes. In those situations it is recommended to solve the problem by a job chain. A job chain is a sequence of jobs where each job automatically starts its successor.
<source lang="bash">
#!/bin/bash
##################################################
## simple MOAB submitter script to setup ##
## a chain of jobs for bwUniCluster ##
##################################################
## ver. : 2015-09-17, KIT, SCC

## Define maximum number of jobs via positional parameter 1, default is 5
max_nojob=${1:-5}

## Define your jobscript (e.g. "~/chain_link_job.sh")
chain_link_job=${PWD}/chain_link_job.sh

## Define type of dependency via positional parameter 2, default is 'afterok'
dep_type="${2:-afterok}"
## -> List of all dependencies:
## http://docs.adaptivecomputing.com/suite/8-0/enterprise/help.htm#topics/\
## moabWorkloadManager/topics/jobAdministration/jobdependencies.html

myloop_counter=1
## Submit loop
while [ ${myloop_counter} -le ${max_nojob} ] ; do
##
## Differ msub_opt depending on chain link number
if [ ${myloop_counter} -eq 1 ] ; then
msub_opt=""
else
## Attention: do NOT use '-W depend' together with msub
msub_opt="-l depend=${dep_type}:${jobID}"
fi
##
## Print current iteration number and msub command
echo "Chain job iteration = ${myloop_counter}"
echo " msub -v myloop_counter=${myloop_counter} ${msub_opt} ${chain_link_job}"
## Store job ID for next iteration by storing output of msub command with empty lines
jobID=$(msub -v myloop_counter=${myloop_counter} ${msub_opt} ${chain_link_job} 2>&1 | sed '/^$/d')
##
## Check if ERROR occurred
if [[ "${jobID}" =~ "ERROR" ]] ; then
echo " -> submission failed!" ; exit 1
else
echo " -> job number = ${jobID}"
fi
##
## Increase counter
let myloop_counter+=1
done
</source>
 
 
----
[[Category:bwUniCluster|Batch Jobs - bwUniCluster features]]

Batch Jobs - bwUniCluster Features

2017-05-08T10:35:32Z

S Shamsudeen: /* msub -q queues */

This article contains information on features of the [[Batch_Jobs|batch job system]] only applicable on bwUniCluster.
 
= Job Submission =
== msub Command ==
=== Interactive Jobs ===
The bwUniCluster supports the following additional msub option(s):

{| width=750px class="wikitable"
! colspan="3" | msub Options
|-
! Command line
! Script
! Purpose
|- style="vertical-align:top;"
| -I
|
| Declares the job is to be run interactively.
|}

=== msub -l ''resource_list'' ===
No deviation or additional features to general [[Batch_Jobs|batch job]] setting.

=== msub -q ''queues'' ===
Compute resources such as walltime, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, you must add to your msub command the correct queue class. Details are:

{| width=750px class="wikitable"
! colspan="6" style="background-color:#999999;padding:3px"| msub -q ''queue''
|- style="width:10%;height=20px; text-align:left;"
! style="width:10%;padding:3px"| ''queue''
! style="width:5%;padding:3px"| ''node''
! style="width:15%;padding:3px"| ''default resources''
! style="padding:3px"| ''minimum resources''
! style="padding:3px"| ''maximum resources''
! style="padding:3px"| ''node access policy''
|- style="vertical-align:top; height=20px; text-align:left"
| style="padding:3px"| develop
| style="padding:3px"| thin
| style="width:15%;padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=00:30:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | singlenode
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=00:30:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:30:01
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;vertical-align:top;height=20px; text-align:left;padding:3px" | verylong
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=3:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=3:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=6:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | extralong
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=3:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=3:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=6:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | fat
| style="padding:3px"| fat
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=32000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=32, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | multinode
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=2
| style="padding:3px"| ''nodes''=128:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | dev_multinode
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=2
| style="padding:3px"| ''nodes''=16:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | special
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:30:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:30:00
| style="padding:3px"| ''nodes''=1:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| shared
|-
|}
Note that ''node access policy''=singlejob means that, irrespected of the requested number of cores, node access is exclusive.
Default resources of a queue class defines walltime, processes and memory if not explicitly given with msub command. Resource list acronyms ''walltime'', ''procs'', ''nodes'' and ''ppn'' are described [[Batch_Jobs#msub_-l_resource_list|here]].

==== Queue class examples ====

* To run your batch job longer than 3 days, please use$ msub -q verylong.

* To run your batch job on one of the [[BwUniCluster_File_System#Components_of_bwUniCluster|fat nodes]], please use$ msub -q fat.
 
 

= Environment Variables for Batch Jobs =
== Additional Moab Environments ==
The bwUniCluster expands the [[Batch_Jobs#Moab Environment Variables|common set of MOAB environment variables]] by the following variable:
{| width=700px class="wikitable"
! colspan="3" style="background-color:#999999;padding:3px"| bwUniCluster specific MOAB variables
|-
! Environment variable
! Description
|-
| MOAB_SUBMITDIR
| Directory of job submission
|}

== Additional Slurm Environments ==
Since the work load manager MOAB on [[bwUniCluster]] uses the resource manager SLURM, the following environment variables of SLURM are added to your environment once your job has started:
{| width=750px class="wikitable"
! colspan="3" style="background-color:#999999;padding:3px"| SLURM variables
|- style="width:25%;height=20px; text-align:left;padding:3px"
! style="width:20%;height=20px; text-align:left;padding:3px"| Environment variables
! style="height=20px; text-align:left;padding:3px"| Description
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_CPUS_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Number of processes per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NODELIST
| style="height=20px; text-align:left;padding:3px"| List of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NUM_NODES
| style="height=20px; text-align:left;padding:3px"| Number of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_MEM_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Memory per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_NPROCS
| style="height=20px; text-align:left;padding:3px"| Total number of processes dedicated to the job
|}
See also:
* [[Batch_Jobs#Batch_Job_.28Slurm.29_Variables_:_bwUniCluster|List of almost all important Slurm environments]]

== Interactive Job Monitoring per Node ==
By default nodes are not used exclusive unless they are requested with ''-l naccesspolicy=singlejob'' as described [[Batch_Jobs#msub_-l_resource_list|here]]. 
If a Job runs exclusive on one node you may do a ssh login to that node. The ssh access will be limited by the set walltime. To get the nodes of your job need to read the environment variable SLURM_JOB_NODELIST during the runtime of the job. It contains all nodes in a shortened way e.g. ''uc1n[344,386]'' or ''uc1n[344-345]''. To expand this string to ''uc1n344 uc1n345'' you can you can use the command expandnodes like:

expandnodes $SLURM_JOB_NODELIST > nodelist

 
 
= Intel MPI parallel Programs =
== Intel MPI without Multithreading ==
MPI parallel programs run faster than serial programs on multi CPU and multi core systems. N-fold spawned processes of the MPI program, i.e., '''MPI tasks''', run simultaneously and communicate via the Message Passing Interface (MPI) paradigm. MPI tasks do not share memory but can be spawned over different nodes.

Generate a wrapper script for '''Intel MPI''', ''job_impi.sh'' containing the following lines:
<source lang="bash">
#!/bin/bash
module load mpi/impi/<placeholder_for_version>
mpiexec.hydra -bootstrap slurm my_par_program
</source>
'''Attention:''' 
Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since MOAB instructs mpirun about number of processes and node hostnames.
Moreover, replace <placeholder_for_version> with the wished version of '''Intel MPI''' to enable the MPI environment.
 
Launching and running 32 Intel MPI tasks on 4 nodes, each requiring 1000 MByte, and running for 5 hours, execute:
<pre>
$ msub -q multinode -l nodes=4:ppn=16,pmem=1000mb,walltime=05:00:00 job_impi.sh
</pre>
 

== Intel MPI with Multithreading ==
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes.

Multiple Intel MPI tasks must be launched by the MPI parallel program '''mpiexec.hydra'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).

'''For Intel MPI''' a job-script to submit a batch job called ''job_impi_omp.sh'' that runs a Intel MPI program with 8 tasks and a tenfold threaded program ''impi_omp_program'' requiring 32000 MByte of total physical memory per task and total wall clock time of 6 hours looks like:


<source lang="bash">
#!/bin/bash
#MSUB -l nodes=4:ppn=20
#MSUB -l walltime=06:00:00
#MSUB -l pmem=3200mb
#MSUB -v MPI_MODULE=mpi/impi
#MSUB -v OMP_NUM_THREADS=10
#MSUB -v MPIRUN_OPTIONS="-binding "domain=omp" -print-rank-map -ppn 2 -envall"
#MSUB -v EXE=./impi_omp_program
#MSUB -N test_impi_omp

#If using more than one MPI task per node please set
export KMP_AFFINITY=scatter
#export KMP_AFFINITY=verbose,scatter prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

module load ${MPI_MODULE}
TASK_COUNT=$((${MOAB_PROCCOUNT}/${OMP_NUM_THREADS}))
echo "${EXE} running on ${MOAB_PROCCOUNT} cores with ${TASK_COUNT} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpiexec.hydra -bootstrap slurm ${MPIRUN_OPTIONS} -n ${TASK_COUNT} ${EXE}"
echo $startexe
exec $startexe
</source>
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores. If you only run one MPI task per node please set KMP_AFFINITY=compact,1,0.

 
Execute the script '''job_impi_omp.sh''' adding the queue class ''multinode'' to your msub command:
<pre>
$ msub -q multinode job_impi_omp.sh
</pre>
 
The mpirun option ''-print-rank-map'' shows the bindings between MPI tasks and nodes (not very beneficial). The option ''-binding'' binds MPI tasks (processes) to a particular processor; ''domain=omp'' means that the domain size is determined by the number of threads. In the above examples (2 MPI tasks per node) you could also choose ''-binding "cell=unit;map=bunch"''; this binding maps one MPI process to each socket.
 
 
= Interactive Jobs =
Interactive jobs on bwUniCluster [[BwUniCluster_User_Access#Allowed_activities_on_login_nodes|must '''NOT''' run on the logins nodes]], however resources for interactive jobs can be requested using msub. Considering a serial application with a graphical frontend that requires 5000 MByte of memory and limiting the interactive run to 2 hours execute the following:
<pre>
$ msub -I -V -l nodes=1:ppn=1 -l walltime=0:02:00:00
</pre>
The option -V defines that all environment variables are exported to the compute node of the interactive session.
After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system MOAB has granted you the requested resources on the compute system. Once granted you will be automatically logged on the dedicated resource. Now you have an interactive session with 1 core and 5000 MByte of memory on the compute system for 2 hours. Simply execute now your application:
<pre>
$ cd to_path
$ ./application
</pre>
Note that, once the walltime limit has been reached you will be automatically logged out of the compute system.
 
 
== Chain Jobs ==
The CPU time requirements of many applications exceed the limits of the job classes. In those situations it is recommended to solve the problem by a job chain. A job chain is a sequence of jobs where each job automatically starts its successor.
<source lang="bash">
#!/bin/bash
##################################################
## simple MOAB submitter script to setup ##
## a chain of jobs for bwUniCluster ##
##################################################
## ver. : 2015-09-17, KIT, SCC

## Define maximum number of jobs via positional parameter 1, default is 5
max_nojob=${1:-5}

## Define your jobscript (e.g. "~/chain_link_job.sh")
chain_link_job=${PWD}/chain_link_job.sh

## Define type of dependency via positional parameter 2, default is 'afterok'
dep_type="${2:-afterok}"
## -> List of all dependencies:
## http://docs.adaptivecomputing.com/suite/8-0/enterprise/help.htm#topics/\
## moabWorkloadManager/topics/jobAdministration/jobdependencies.html

myloop_counter=1
## Submit loop
while [ ${myloop_counter} -le ${max_nojob} ] ; do
##
## Differ msub_opt depending on chain link number
if [ ${myloop_counter} -eq 1 ] ; then
msub_opt=""
else
## Attention: do NOT use '-W depend' together with msub
msub_opt="-l depend=${dep_type}:${jobID}"
fi
##
## Print current iteration number and msub command
echo "Chain job iteration = ${myloop_counter}"
echo " msub -v myloop_counter=${myloop_counter} ${msub_opt} ${chain_link_job}"
## Store job ID for next iteration by storing output of msub command with empty lines
jobID=$(msub -v myloop_counter=${myloop_counter} ${msub_opt} ${chain_link_job} 2>&1 | sed '/^$/d')
##
## Check if ERROR occurred
if [[ "${jobID}" =~ "ERROR" ]] ; then
echo " -> submission failed!" ; exit 1
else
echo " -> job number = ${jobID}"
fi
##
## Increase counter
let myloop_counter+=1
done
</source>
 
 
----
[[Category:bwUniCluster|Batch Jobs - bwUniCluster features]]

Batch Jobs - bwUniCluster Features

2017-05-08T09:25:37Z

S Shamsudeen: /* msub -q queues */

This article contains information on features of the [[Batch_Jobs|batch job system]] only applicable on bwUniCluster.
 
= Job Submission =
== msub Command ==
=== Interactive Jobs ===
The bwUniCluster supports the following additional msub option(s):

{| width=750px class="wikitable"
! colspan="3" | msub Options
|-
! Command line
! Script
! Purpose
|- style="vertical-align:top;"
| -I
|
| Declares the job is to be run interactively.
|}

=== msub -l ''resource_list'' ===
No deviation or additional features to general [[Batch_Jobs|batch job]] setting.

=== msub -q ''queues'' ===
Compute resources such as walltime, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, you must add to your msub command the correct queue class. Details are:

{| width=750px class="wikitable"
! colspan="6" style="background-color:#999999;padding:3px"| msub -q ''queue''
|- style="width:10%;height=20px; text-align:left;"
! style="width:10%;padding:3px"| ''queue''
! style="width:5%;padding:3px"| ''node''
! style="width:15%;padding:3px"| ''default resources''
! style="padding:3px"| ''minimum resources''
! style="padding:3px"| ''maximum resources''
! style="padding:3px"| ''node access policy''
|- style="vertical-align:top; height=20px; text-align:left"
| style="padding:3px"| develop
| style="padding:3px"| thin
| style="width:15%;padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=00:30:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | singlenode
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=00:30:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:30:01
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;vertical-align:top;height=20px; text-align:left;padding:3px" | verylong
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=3:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=3:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=6:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | extralong
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=3:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=3:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=6:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | fat
| style="padding:3px"| fat
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=32000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=32, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | multinode
| style="padding:3px"| broad
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4500mb
| style="padding:3px"| ''nodes''=2
| style="padding:3px"| ''nodes''=128:''ppn''=28, ''walltime''=48:00:00
| style="padding:3px"| shared

|-
|}
Note that ''node access policy''=singlejob means that, irrespected of the requested number of cores, node access is exclusive.
Default resources of a queue class defines walltime, processes and memory if not explicitly given with msub command. Resource list acronyms ''walltime'', ''procs'', ''nodes'' and ''ppn'' are described [[Batch_Jobs#msub_-l_resource_list|here]].

==== Queue class examples ====

* To run your batch job longer than 3 days, please use$ msub -q verylong.

* To run your batch job on one of the [[BwUniCluster_File_System#Components_of_bwUniCluster|fat nodes]], please use$ msub -q fat.
 
 

= Environment Variables for Batch Jobs =
== Additional Moab Environments ==
The bwUniCluster expands the [[Batch_Jobs#Moab Environment Variables|common set of MOAB environment variables]] by the following variable:
{| width=700px class="wikitable"
! colspan="3" style="background-color:#999999;padding:3px"| bwUniCluster specific MOAB variables
|-
! Environment variable
! Description
|-
| MOAB_SUBMITDIR
| Directory of job submission
|}

== Additional Slurm Environments ==
Since the work load manager MOAB on [[bwUniCluster]] uses the resource manager SLURM, the following environment variables of SLURM are added to your environment once your job has started:
{| width=750px class="wikitable"
! colspan="3" style="background-color:#999999;padding:3px"| SLURM variables
|- style="width:25%;height=20px; text-align:left;padding:3px"
! style="width:20%;height=20px; text-align:left;padding:3px"| Environment variables
! style="height=20px; text-align:left;padding:3px"| Description
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_CPUS_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Number of processes per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NODELIST
| style="height=20px; text-align:left;padding:3px"| List of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NUM_NODES
| style="height=20px; text-align:left;padding:3px"| Number of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_MEM_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Memory per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_NPROCS
| style="height=20px; text-align:left;padding:3px"| Total number of processes dedicated to the job
|}
See also:
* [[Batch_Jobs#Batch_Job_.28Slurm.29_Variables_:_bwUniCluster|List of almost all important Slurm environments]]

== Interactive Job Monitoring per Node ==
By default nodes are not used exclusive unless they are requested with ''-l naccesspolicy=singlejob'' as described [[Batch_Jobs#msub_-l_resource_list|here]]. 
If a Job runs exclusive on one node you may do a ssh login to that node. The ssh access will be limited by the set walltime. To get the nodes of your job need to read the environment variable SLURM_JOB_NODELIST during the runtime of the job. It contains all nodes in a shortened way e.g. ''uc1n[344,386]'' or ''uc1n[344-345]''. To expand this string to ''uc1n344 uc1n345'' you can you can use the command expandnodes like:

expandnodes $SLURM_JOB_NODELIST > nodelist

 
 
= Intel MPI parallel Programs =
== Intel MPI without Multithreading ==
MPI parallel programs run faster than serial programs on multi CPU and multi core systems. N-fold spawned processes of the MPI program, i.e., '''MPI tasks''', run simultaneously and communicate via the Message Passing Interface (MPI) paradigm. MPI tasks do not share memory but can be spawned over different nodes.

Generate a wrapper script for '''Intel MPI''', ''job_impi.sh'' containing the following lines:
<source lang="bash">
#!/bin/bash
module load mpi/impi/<placeholder_for_version>
mpiexec.hydra -bootstrap slurm my_par_program
</source>
'''Attention:''' 
Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since MOAB instructs mpirun about number of processes and node hostnames.
Moreover, replace <placeholder_for_version> with the wished version of '''Intel MPI''' to enable the MPI environment.
 
Launching and running 32 Intel MPI tasks on 4 nodes, each requiring 1000 MByte, and running for 5 hours, execute:
<pre>
$ msub -q multinode -l nodes=4:ppn=16,pmem=1000mb,walltime=05:00:00 job_impi.sh
</pre>
 

== Intel MPI with Multithreading ==
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes.

Multiple Intel MPI tasks must be launched by the MPI parallel program '''mpiexec.hydra'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).

'''For Intel MPI''' a job-script to submit a batch job called ''job_impi_omp.sh'' that runs a Intel MPI program with 8 tasks and a tenfold threaded program ''impi_omp_program'' requiring 32000 MByte of total physical memory per task and total wall clock time of 6 hours looks like:


<source lang="bash">
#!/bin/bash
#MSUB -l nodes=4:ppn=20
#MSUB -l walltime=06:00:00
#MSUB -l pmem=3200mb
#MSUB -v MPI_MODULE=mpi/impi
#MSUB -v OMP_NUM_THREADS=10
#MSUB -v MPIRUN_OPTIONS="-binding "domain=omp" -print-rank-map -ppn 2 -envall"
#MSUB -v EXE=./impi_omp_program
#MSUB -N test_impi_omp

#If using more than one MPI task per node please set
export KMP_AFFINITY=scatter
#export KMP_AFFINITY=verbose,scatter prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

module load ${MPI_MODULE}
TASK_COUNT=$((${MOAB_PROCCOUNT}/${OMP_NUM_THREADS}))
echo "${EXE} running on ${MOAB_PROCCOUNT} cores with ${TASK_COUNT} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpiexec.hydra -bootstrap slurm ${MPIRUN_OPTIONS} -n ${TASK_COUNT} ${EXE}"
echo $startexe
exec $startexe
</source>
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores. If you only run one MPI task per node please set KMP_AFFINITY=compact,1,0.

 
Execute the script '''job_impi_omp.sh''' adding the queue class ''multinode'' to your msub command:
<pre>
$ msub -q multinode job_impi_omp.sh
</pre>
 
The mpirun option ''-print-rank-map'' shows the bindings between MPI tasks and nodes (not very beneficial). The option ''-binding'' binds MPI tasks (processes) to a particular processor; ''domain=omp'' means that the domain size is determined by the number of threads. In the above examples (2 MPI tasks per node) you could also choose ''-binding "cell=unit;map=bunch"''; this binding maps one MPI process to each socket.
 
 
= Interactive Jobs =
Interactive jobs on bwUniCluster [[BwUniCluster_User_Access#Allowed_activities_on_login_nodes|must '''NOT''' run on the logins nodes]], however resources for interactive jobs can be requested using msub. Considering a serial application with a graphical frontend that requires 5000 MByte of memory and limiting the interactive run to 2 hours execute the following:
<pre>
$ msub -I -V -l nodes=1:ppn=1 -l walltime=0:02:00:00
</pre>
The option -V defines that all environment variables are exported to the compute node of the interactive session.
After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system MOAB has granted you the requested resources on the compute system. Once granted you will be automatically logged on the dedicated resource. Now you have an interactive session with 1 core and 5000 MByte of memory on the compute system for 2 hours. Simply execute now your application:
<pre>
$ cd to_path
$ ./application
</pre>
Note that, once the walltime limit has been reached you will be automatically logged out of the compute system.
 
 
== Chain Jobs ==
The CPU time requirements of many applications exceed the limits of the job classes. In those situations it is recommended to solve the problem by a job chain. A job chain is a sequence of jobs where each job automatically starts its successor.
<source lang="bash">
#!/bin/bash
##################################################
## simple MOAB submitter script to setup ##
## a chain of jobs for bwUniCluster ##
##################################################
## ver. : 2015-09-17, KIT, SCC

## Define maximum number of jobs via positional parameter 1, default is 5
max_nojob=${1:-5}

## Define your jobscript (e.g. "~/chain_link_job.sh")
chain_link_job=${PWD}/chain_link_job.sh

## Define type of dependency via positional parameter 2, default is 'afterok'
dep_type="${2:-afterok}"
## -> List of all dependencies:
## http://docs.adaptivecomputing.com/suite/8-0/enterprise/help.htm#topics/\
## moabWorkloadManager/topics/jobAdministration/jobdependencies.html

myloop_counter=1
## Submit loop
while [ ${myloop_counter} -le ${max_nojob} ] ; do
##
## Differ msub_opt depending on chain link number
if [ ${myloop_counter} -eq 1 ] ; then
msub_opt=""
else
## Attention: do NOT use '-W depend' together with msub
msub_opt="-l depend=${dep_type}:${jobID}"
fi
##
## Print current iteration number and msub command
echo "Chain job iteration = ${myloop_counter}"
echo " msub -v myloop_counter=${myloop_counter} ${msub_opt} ${chain_link_job}"
## Store job ID for next iteration by storing output of msub command with empty lines
jobID=$(msub -v myloop_counter=${myloop_counter} ${msub_opt} ${chain_link_job} 2>&1 | sed '/^$/d')
##
## Check if ERROR occurred
if [[ "${jobID}" =~ "ERROR" ]] ; then
echo " -> submission failed!" ; exit 1
else
echo " -> job number = ${jobID}"
fi
##
## Increase counter
let myloop_counter+=1
done
</source>
 
 
----
[[Category:bwUniCluster|Batch Jobs - bwUniCluster features]]

Batch Jobs - bwUniCluster Features

2017-05-08T06:52:20Z

S Shamsudeen: /* msub -q queues */

This article contains information on features of the [[Batch_Jobs|batch job system]] only applicable on bwUniCluster.
 
= Job Submission =
== msub Command ==
=== Interactive Jobs ===
The bwUniCluster supports the following additional msub option(s):

{| width=750px class="wikitable"
! colspan="3" | msub Options
|-
! Command line
! Script
! Purpose
|- style="vertical-align:top;"
| -I
|
| Declares the job is to be run interactively.
|}

=== msub -l ''resource_list'' ===
No deviation or additional features to general [[Batch_Jobs|batch job]] setting.

=== msub -q ''queues'' ===
Compute resources such as walltime, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, you must add to your msub command the correct queue class. Details are:

{| width=750px class="wikitable"
! colspan="6" style="background-color:#999999;padding:3px"| msub -q ''queue''
|- style="width:10%;height=20px; text-align:left;"
! style="width:10%;padding:3px"| ''queue''
! style="width:5%;padding:3px"| ''node''
! style="width:15%;padding:3px"| ''default resources''
! style="padding:3px"| ''minimum resources''
! style="padding:3px"| ''maximum resources''
! style="padding:3px"| ''node access policy''
|- style="vertical-align:top; height=20px; text-align:left"
| style="padding:3px"| develop
| style="padding:3px"| thin
| style="width:15%;padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=00:30:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | singlenode
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=00:30:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:30:01
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;vertical-align:top;height=20px; text-align:left;padding:3px" | verylong
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=3:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=3:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=6:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | fat
| style="padding:3px"| fat
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=32000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=32, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|-
|}
Note that ''node access policy''=singlejob means that, irrespected of the requested number of cores, node access is exclusive.
Default resources of a queue class defines walltime, processes and memory if not explicitly given with msub command. Resource list acronyms ''walltime'', ''procs'', ''nodes'' and ''ppn'' are described [[Batch_Jobs#msub_-l_resource_list|here]].

==== Queue class examples ====

* To run your batch job longer than 3 days, please use$ msub -q verylong.

* To run your batch job on one of the [[BwUniCluster_File_System#Components_of_bwUniCluster|fat nodes]], please use$ msub -q fat.
 
 

= Environment Variables for Batch Jobs =
== Additional Moab Environments ==
The bwUniCluster expands the [[Batch_Jobs#Moab Environment Variables|common set of MOAB environment variables]] by the following variable:
{| width=700px class="wikitable"
! colspan="3" style="background-color:#999999;padding:3px"| bwUniCluster specific MOAB variables
|-
! Environment variable
! Description
|-
| MOAB_SUBMITDIR
| Directory of job submission
|}

== Additional Slurm Environments ==
Since the work load manager MOAB on [[bwUniCluster]] uses the resource manager SLURM, the following environment variables of SLURM are added to your environment once your job has started:
{| width=750px class="wikitable"
! colspan="3" style="background-color:#999999;padding:3px"| SLURM variables
|- style="width:25%;height=20px; text-align:left;padding:3px"
! style="width:20%;height=20px; text-align:left;padding:3px"| Environment variables
! style="height=20px; text-align:left;padding:3px"| Description
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_CPUS_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Number of processes per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NODELIST
| style="height=20px; text-align:left;padding:3px"| List of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NUM_NODES
| style="height=20px; text-align:left;padding:3px"| Number of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_MEM_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Memory per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_NPROCS
| style="height=20px; text-align:left;padding:3px"| Total number of processes dedicated to the job
|}
See also:
* [[Batch_Jobs#Batch_Job_.28Slurm.29_Variables_:_bwUniCluster|List of almost all important Slurm environments]]

== Interactive Job Monitoring per Node ==
By default nodes are not used exclusive unless they are requested with ''-l naccesspolicy=singlejob'' as described [[Batch_Jobs#msub_-l_resource_list|here]]. 
If a Job runs exclusive on one node you may do a ssh login to that node. The ssh access will be limited by the set walltime. To get the nodes of your job need to read the environment variable SLURM_JOB_NODELIST during the runtime of the job. It contains all nodes in a shortened way e.g. ''uc1n[344,386]'' or ''uc1n[344-345]''. To expand this string to ''uc1n344 uc1n345'' you can you can use the command expandnodes like:

expandnodes $SLURM_JOB_NODELIST > nodelist

 
 
= Intel MPI parallel Programs =
== Intel MPI without Multithreading ==
MPI parallel programs run faster than serial programs on multi CPU and multi core systems. N-fold spawned processes of the MPI program, i.e., '''MPI tasks''', run simultaneously and communicate via the Message Passing Interface (MPI) paradigm. MPI tasks do not share memory but can be spawned over different nodes.

Generate a wrapper script for '''Intel MPI''', ''job_impi.sh'' containing the following lines:
<source lang="bash">
#!/bin/bash
module load mpi/impi/<placeholder_for_version>
mpiexec.hydra -bootstrap slurm my_par_program
</source>
'''Attention:''' 
Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since MOAB instructs mpirun about number of processes and node hostnames.
Moreover, replace <placeholder_for_version> with the wished version of '''Intel MPI''' to enable the MPI environment.
 
Launching and running 32 Intel MPI tasks on 4 nodes, each requiring 1000 MByte, and running for 5 hours, execute:
<pre>
$ msub -q multinode -l nodes=4:ppn=16,pmem=1000mb,walltime=05:00:00 job_impi.sh
</pre>
 

== Intel MPI with Multithreading ==
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes.

Multiple Intel MPI tasks must be launched by the MPI parallel program '''mpiexec.hydra'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).

'''For Intel MPI''' a job-script to submit a batch job called ''job_impi_omp.sh'' that runs a Intel MPI program with 8 tasks and a tenfold threaded program ''impi_omp_program'' requiring 32000 MByte of total physical memory per task and total wall clock time of 6 hours looks like:


<source lang="bash">
#!/bin/bash
#MSUB -l nodes=4:ppn=20
#MSUB -l walltime=06:00:00
#MSUB -l pmem=3200mb
#MSUB -v MPI_MODULE=mpi/impi
#MSUB -v OMP_NUM_THREADS=10
#MSUB -v MPIRUN_OPTIONS="-binding "domain=omp" -print-rank-map -ppn 2 -envall"
#MSUB -v EXE=./impi_omp_program
#MSUB -N test_impi_omp

#If using more than one MPI task per node please set
export KMP_AFFINITY=scatter
#export KMP_AFFINITY=verbose,scatter prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

module load ${MPI_MODULE}
TASK_COUNT=$((${MOAB_PROCCOUNT}/${OMP_NUM_THREADS}))
echo "${EXE} running on ${MOAB_PROCCOUNT} cores with ${TASK_COUNT} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpiexec.hydra -bootstrap slurm ${MPIRUN_OPTIONS} -n ${TASK_COUNT} ${EXE}"
echo $startexe
exec $startexe
</source>
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores. If you only run one MPI task per node please set KMP_AFFINITY=compact,1,0.

 
Execute the script '''job_impi_omp.sh''' adding the queue class ''multinode'' to your msub command:
<pre>
$ msub -q multinode job_impi_omp.sh
</pre>
 
The mpirun option ''-print-rank-map'' shows the bindings between MPI tasks and nodes (not very beneficial). The option ''-binding'' binds MPI tasks (processes) to a particular processor; ''domain=omp'' means that the domain size is determined by the number of threads. In the above examples (2 MPI tasks per node) you could also choose ''-binding "cell=unit;map=bunch"''; this binding maps one MPI process to each socket.
 
 
= Interactive Jobs =
Interactive jobs on bwUniCluster [[BwUniCluster_User_Access#Allowed_activities_on_login_nodes|must '''NOT''' run on the logins nodes]], however resources for interactive jobs can be requested using msub. Considering a serial application with a graphical frontend that requires 5000 MByte of memory and limiting the interactive run to 2 hours execute the following:
<pre>
$ msub -I -V -l nodes=1:ppn=1 -l walltime=0:02:00:00
</pre>
The option -V defines that all environment variables are exported to the compute node of the interactive session.
After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system MOAB has granted you the requested resources on the compute system. Once granted you will be automatically logged on the dedicated resource. Now you have an interactive session with 1 core and 5000 MByte of memory on the compute system for 2 hours. Simply execute now your application:
<pre>
$ cd to_path
$ ./application
</pre>
Note that, once the walltime limit has been reached you will be automatically logged out of the compute system.
 
 
== Chain Jobs ==
The CPU time requirements of many applications exceed the limits of the job classes. In those situations it is recommended to solve the problem by a job chain. A job chain is a sequence of jobs where each job automatically starts its successor.
<source lang="bash">
#!/bin/bash
##################################################
## simple MOAB submitter script to setup ##
## a chain of jobs for bwUniCluster ##
##################################################
## ver. : 2015-09-17, KIT, SCC

## Define maximum number of jobs via positional parameter 1, default is 5
max_nojob=${1:-5}

## Define your jobscript (e.g. "~/chain_link_job.sh")
chain_link_job=${PWD}/chain_link_job.sh

## Define type of dependency via positional parameter 2, default is 'afterok'
dep_type="${2:-afterok}"
## -> List of all dependencies:
## http://docs.adaptivecomputing.com/suite/8-0/enterprise/help.htm#topics/\
## moabWorkloadManager/topics/jobAdministration/jobdependencies.html

myloop_counter=1
## Submit loop
while [ ${myloop_counter} -le ${max_nojob} ] ; do
##
## Differ msub_opt depending on chain link number
if [ ${myloop_counter} -eq 1 ] ; then
msub_opt=""
else
## Attention: do NOT use '-W depend' together with msub
msub_opt="-l depend=${dep_type}:${jobID}"
fi
##
## Print current iteration number and msub command
echo "Chain job iteration = ${myloop_counter}"
echo " msub -v myloop_counter=${myloop_counter} ${msub_opt} ${chain_link_job}"
## Store job ID for next iteration by storing output of msub command with empty lines
jobID=$(msub -v myloop_counter=${myloop_counter} ${msub_opt} ${chain_link_job} 2>&1 | sed '/^$/d')
##
## Check if ERROR occurred
if [[ "${jobID}" =~ "ERROR" ]] ; then
echo " -> submission failed!" ; exit 1
else
echo " -> job number = ${jobID}"
fi
##
## Increase counter
let myloop_counter+=1
done
</source>
 
 
----
[[Category:bwUniCluster|Batch Jobs - bwUniCluster features]]