bwHPC Wiki - User contributions [en]

BwForCluster User Access

2020-03-25T15:37:49Z

T Kienzle: /* Issuing bwForCluster entitlement */

The usage of bwForCluster is free of charge. bwForClusters are customized to the requirements of particular research areas.
Each bwForCluster is/will be financed by the DFG (German Research Foundation) and by the Ministry of Science, Research and Arts of Baden-Württemberg based on scientifc grant proposal (compare proposals guidelines as per Art. 91b GG).

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:center; color:#000;vertical-align:middle;font-size:75%;" |[[File:Bwforreg.svg|center|border|500px|]]
|-
| style="text-align:center; color:#000;vertical-align:middle;" |
|}

Granting user access to a bwForCluster requires 3 steps:

# Registration of a new or joining an already existing ''rechenvorhaben (RV)'' at [https://www.bwhpc.de/zas.html "Zentrale Antragsseite (ZAS)"]. An ''RV'' defines the planned compute activities of a group of researchers. As coworker of this group you only need to register your membership in the corresponding ''RV''.
# Application for a [[#Issuing bwForCluster entitlement | bwForCluster entitlement]] issued by your university.
# [[#Personal registration at bwForCluster | Personal registration at assigned cluster site ]] based on approved ''RV'' [[File:Zas assignment icon.svg|25px|]] and issued bwForCluster entitlement [[File:bwfor entitlement icon.svg|25px|]].

 
Step 1 and 2 are pre-steps and requirements of step 3. Moreover, steps 1 and 2 can be done in parallel. The question, at which bwForCluster site to do step 3, will be answered by the cluster assignment team (CAT) based on the data of step 1, i.e. at ''ZAS''.

= ''RV'' registration at ''ZAS'' =

The ''rechenvorhaben (RV)'' registration at ''ZAS'' does '''not''' correspond to typical compute project proposals as for other HPC clusters, since:
* there is no scientific reviewing process
* the registration inquires only brief details and
* it covers a group of coworkers, and
* only the ''RV'' responsible must submit the ''RV''.
 

== How to register? ==
=== Register a new ''RV'' ===
Typically only the leader of a scientific work group or the senior scientist of a research group/collaboration needs to login at
* https://www.bwhpc-c5.de/en/ZAS/bwforcluster_rv_registration.php
and to fill in all mandatory fields of the given web form. Please note that for your convenience you can also switch to the [https://www.bwhpc-c5.de/ZAS/bwforcluster_rechenvorhaben.php german version] of the web form. The submitter of the ''rechenvorhaben (RV)'' will be assigned the role ''RV'' responsible.

The web form consists of the following fields to be filled:
{| style="width:100%;border:3px solid darkgray; margin: 1em auto 1em auto;"
|-
! style="width:25%;padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;"| Field
! style="padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;" | Explanation
|-
!scope="column" {{Lightgray_small}}| RV Title
|scope="column" {{Tab element}} | Define a short title of your planned compute activities, maximum of 255 characters including spaces.
|-
!scope="column" {{Lightgray_small}}| RV Description
|scope="column" {{Tab element}} | Write a short abstract about your planned compute activities (maximum of 2048 characters including spaces). Research groups that contributed to the DFG grant proposal (Art. 91b GG) of the corresponding bwForCluster only need to give reference to that particular proposal.
|-
!scope="column" {{Lightgray_small}}| Scientific field
|scope="column" {{Tab element}} | Tick one or several scientific fields. Once all bwForClusters are up and running the full list of given scientific fields are supported and hence for ''rechenvorhaben (RV)'' applicable. If your ''RV'' does not match any given scientific field, please state your scientific field in the given text box. Grayed out scientific fields indicate that the corresponding bwForCluster(s) is/are not operational yet.
|-
!scope="column" {{Lightgray_small}}| Field of activity
|scope="column" {{Tab element}} | Define if your ''RV'' is primarily for research and/or teaching. If not applicable, use text box.
|-
!scope="column" {{Lightgray_small}}| Parallel paradigm
|scope="column" {{Tab element}} | State what parallel paradigm your code or application uses. Multiple ticks are allowed. Further information can be provided via text box. If you are not sure about it, please state the software you are using.
|-
!scope="column" {{Lightgray_small}}| Programming language
|scope="column" {{Tab element}} | State the programming language(s) of your code or application.
|-
!scope="column" {{Lightgray_small}}| Numerical methods
|scope="column" {{Tab element}} | State the numerical or "calculation" methods your code or application utilises. If you do not know, write "unknown".
|-
!scope="column" {{Lightgray_small}}| Software packages
|scope="column" {{Tab element}} | State all software package you will or want to use for your ''RV''. Also include compiler, MPI and numerical packages (e.g. MKL, FFTW).
|-
!scope="column" {{Lightgray_small}}| Estimated requested computing resources
|scope="column" {{Tab element}} | Roughly estimate how many CPU hours are required to finish your ''RV''. To calculate "CPU hours" multiply "elapsed parallel computing time" with "number of CPU core" per job.
''Example: Your code uses 4 CPU cores and has a wall time of 1h. This makes 4 CPU hours.''
|-
!scope="column" {{Lightgray_small}}| Planned maximum number of parallel used CPU cores per job
|scope="column" {{Tab element}} | Give an upper limit of how many CPU cores per job are required. Please cross-check with your statement about parallel paradigm.
''Obviously, a sequential code can only use 1 CPU core per job.''
|-
!scope="column" {{Lightgray_small}}| Estimated maximal memory requirements per CPU core
|scope="column" {{Tab element}} | Give an upper limit of how much RAM per CPU core your jobs require. Please give a value in GigaBytes.
|-
!scope="column" {{Lightgray_small}}| Estimated requested persistent disk space for the ''RV''
|scope="column" {{Tab element}} | State how much disk space in total you need for your ''RV''. Please give a value in GigaBytes.
''Example: If your RV has 4 more coworker and each of you produces until the end of your RV 20 GigaByte of output, your maximum disk space is 100 GigaByte.''
|-
!scope="column" {{Lightgray_small}}| Estimated maximal temporary disk space per job
|scope="column" {{Tab element}} | State how much space your intermediate files require during a job run. Please give a value in GigaBytes.
''Example: Your final output of your job is 0.1 GigaByte, but during your job 10 temporary files, each with a size of 1 GigaByte, are written to disk, hence your correct answer is 10 GigaByte.''
|-
!scope="column" {{Lightgray_small}}| Institute
|scope="column" {{Tab element}} | State the full name of your Institute.
|-
|}
Note that the fields - name, firstname, organization, mail, und EPPN - are auto-filled and can not be changed. These are your credentials as given by your home organization.

Response
# After submitting you will get an email from ''ZAS'' confirming your submission. With this email you are given an unique * acronym * password Please keep the password enclosed. Only with both, acronym and password, [[BwForCluster_User_Access#Register_for_an_RV_as_coworker | coworkers ]] can be added to your ''rechenvorhaben''.
# The cluster assignment team will be notified immediately about your submission.
# The cluster assignment team decides based on your submitted data which bwForCluster most fits, and submits its decision to ''ZAS''.
# ''ZAS'' notifies you in an email about your assigned bwForCluster and provides website details for [[BwForCluster_User_Access#Personal_registration_at_bwForCluster | step 3 ]].

 

=== Register for an ''RV'' as coworker ===

Each coworker must add herself/himself to a ''rechenvorhaben (RV)''.
To become coworker of an ''RV'', please login at
* https://www.bwhpc-c5.de/en/ZAS/bwforcluster_collaboration.php
and provide the
* acronym
* password
of the ''RV''. Your ''RV'' responsible will provide you with that information.
You will be assigned to role ''RV'' member.

Response
# After submitting the request for participating in an ''RV'', the coworker will receive an email from ''ZAS'' about the further steps (i.e. [[#Personal registration at bwForCluster | personal registration at assigned bwForCluster]]). The membership status in the ''RV'' is set active by default.
# Each person with role ''RV'' responsible or ''RV'' manager will be notified automatically. They can check the list of ''RV'' members at https://www.bwhpc-c5.de/en/ZAS/info_rv.php by clicking on the ''RV'' acronym.
# The ''RV'' responsible can set any ''RV'' coworker to ''RV'' manager and vice versa as well as deactivate/reactivate any ''RV'' coworker or ''RV'' manager.
# An ''RV'' manager can deactivate/reactivate any ''RV'' coworker.



== Roles in an ''RV'' ==
{| style="width:100%;border:3px solid darkgray; margin: 1em auto 1em auto;"
|-
! style="width:25%;padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;"| Role
! style="padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;" | Explanation
|-
!scope="column" {{Lightgray_small}}| RV responsible
|scope="column" {{Tab element}} | Registers the ''rechenvorhaben (RV)'', i.e. the planned compute activities. Can deactivate but also reactivate ''RV'' managers and coworkers. Can promote coworkers to mangers and vice versa.
|-
!scope="column" {{Lightgray_small}}| RV manager
|scope="column" {{Tab element}} | ''RV'' coworker with elevated rights to deactivate and reactivate other ''RV'' coworkers.
|-
!scope="column" {{Lightgray_small}}| RV coworker
|scope="column" {{Tab element}} | Typical user who has registered her/himself via ''RV'' acronym and password.
|-
|}

 

= Issuing bwForCluster entitlement =

Each university issues the bwForCluster entitlement
<pre>http://bwidm.de/entitlement/bwForCluster</pre>
only for their own members.

The bwForCluster entitlement issued by a university assures the operator of a bwForCluster that its university member's compute activities comply with the German Foreign Trade Act (Außenwirtschaftsgesetz - AWG) und German Foreign Trade Regulations (Außenwirtschaftsverordnung - AWV).

The following unversities have already established a process to issue the bwForCluster entitlement:

* [[BwForCluster User Access Members Hochschule Esslingen|Hochschule Esslingen]]
* [[BwCluster User Access Uni Freiburg|University of Freiburg]]
* [[BwForCluster User Access Members Uni Heidelberg|University of Heidelberg]]
* [https://kim.uni-hohenheim.de/bwhpc-account/ University of Hohenheim]
* [http://www.scc.kit.edu/downloads/sdo/Antrag_Zugangsberechtigung_bwForCluster.pdf Karlsruhe Institute of Technology (KIT)]
* [[BwForCluster User Access Members Uni Konstanz|University of Konstanz]]
* [[BwForCluster User Access Members Uni Mannheim|University of Mannheim]]
* [https://www.hlrs.de/solutions-services/academic-users/bwforcluster-access/ University of Stuttgart]
* [[BwForCluster User Access Members Uni Tübingen|University of Tübingen]]
* [[BwForCluster User Access Members Uni Ulm|University of Ulm]]

If you do not find your university in the list above, please contact your local service desk.

= Personal registration at bwForCluster =

Once you have registered your own RV (''rechenvorhaben'')
or a membership in an RV, the cluster assignment team will provide
you with information about your designated cluster. You will receive
an email with a website to create an account for yourself
on that cluster.

Available bwForCluster registration servers (service providers):
{| style="border:3px solid darkgray; margin: 1em auto 1em auto;" width="72%"
|-
!scope="row" {{Darkgray}} | Cluster topic and location
!scope="row" {{Darkgray}} | Registration server
|-
| bwForCluster Chemistry JUSTUS Ulm
| http://bwidm.rz.uni-ulm.de
|-
| bwForCluster MLS&WISO (Production and Development)
| https://bwservices.uni-heidelberg.de
|-
| bwForCluster NEMO Freiburg
| https://bwservices.uni-freiburg.de
|-
| bwForCluster BinAC Tübingen
| https://bwservices.uni-tuebingen.de
|}

Please note, this step is different to your registration at ZAS since
here you register yourself as a person. Only based on your personal credentials
a user account can be generated. However, resource requests of the RV in ZAS
will be cross-checked with the resources consumed by you and your RV coworkers
on this cluster.

After steps 1 and 2 (RV approval and bwForCluster entitlement) please visit the

* bwForCluster ''service provider'' registration website (see table above or email after RV approval):
*# Select your home organization from the list of organizations and click '''Proceed'''.
*# You will be redirected to the ''Identity Provider'' of your home organization.
*# Enter your home-organizational user ID (might be user name, email, ...) and password and click '''Login''' or '''Anmelden'''.
*# When doing this for the first time you need to accept that your personal data is transferred to the ''service provider''.
*# You will be redirected back to the cluster registration website.
*# Select '''Service Description''' of the designated cluster.
*# Click on '''Register''' link to register for this cluster.
*# Read and accept the terms and conditions of use and click on button '''Register'''.
*# Finally you will receive an email with instructions how to login to the cluster (please wait ~15 minutes before trying).
*# Click on '''Set Service Password''' and set a password for the cluster.
*# '''Note: Users from Tübingen accessing the BinAC do not set a separate service password''' and have to use their local account credentials instead.
*# '''Note:''' You can return to the registration website at any time, in order to review your registration details, change/reset your service password or deregister from the service by yourself.

= Login to bwForCluster =

Personalized details about how to login to the cluster are included
in an email send after registration at the bwForCluster service provider.

General instructions for the bwForCluster login can be found here:
{| style="border:3px solid darkgray; margin: 1em auto 1em auto;" width="73%"
|-
!scope="row" {{Darkgray}} | Cluster topic and location
!scope="row" {{Darkgray}} | Login instructions
|-
| bwForCluster Chemistry JUSTUS Ulm
| [[BwForCluster_Chemistry_Login|bwForCluster Chemistry Login]]
|-
| bwForCluster MLS&WISO Production
| [[BwForCluster_MLS&WISO_Production_Login|bwForCluster MLS&WISO Production Login]]
|-
| bwForCluster MLS&WISO Development
| [[BwForCluster_MLS&WISO_Development_Login|bwForCluster MLS&WISO Development Login]]
|-
| bwForCluster NEMO Freiburg
| [[bwForCluster NEMO Login]]
|-
| bwForCluster BinAC Tübingen
| [[bwForCluster BinAC Login]]

|}

 

----
[[Category:Access|bwForCluster]][[Category:bwForCluster_Chemistry]][[Category:bwForCluster_MLS&WISO_Production]][[Category:bwForCluster_MLS&WISO_Development]]

BwForCluster User Access

2020-03-25T15:37:21Z

T Kienzle: /* Issuing bwForCluster entitlement */

The usage of bwForCluster is free of charge. bwForClusters are customized to the requirements of particular research areas.
Each bwForCluster is/will be financed by the DFG (German Research Foundation) and by the Ministry of Science, Research and Arts of Baden-Württemberg based on scientifc grant proposal (compare proposals guidelines as per Art. 91b GG).

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:center; color:#000;vertical-align:middle;font-size:75%;" |[[File:Bwforreg.svg|center|border|500px|]]
|-
| style="text-align:center; color:#000;vertical-align:middle;" |
|}

Granting user access to a bwForCluster requires 3 steps:

# Registration of a new or joining an already existing ''rechenvorhaben (RV)'' at [https://www.bwhpc.de/zas.html "Zentrale Antragsseite (ZAS)"]. An ''RV'' defines the planned compute activities of a group of researchers. As coworker of this group you only need to register your membership in the corresponding ''RV''.
# Application for a [[#Issuing bwForCluster entitlement | bwForCluster entitlement]] issued by your university.
# [[#Personal registration at bwForCluster | Personal registration at assigned cluster site ]] based on approved ''RV'' [[File:Zas assignment icon.svg|25px|]] and issued bwForCluster entitlement [[File:bwfor entitlement icon.svg|25px|]].

 
Step 1 and 2 are pre-steps and requirements of step 3. Moreover, steps 1 and 2 can be done in parallel. The question, at which bwForCluster site to do step 3, will be answered by the cluster assignment team (CAT) based on the data of step 1, i.e. at ''ZAS''.

= ''RV'' registration at ''ZAS'' =

The ''rechenvorhaben (RV)'' registration at ''ZAS'' does '''not''' correspond to typical compute project proposals as for other HPC clusters, since:
* there is no scientific reviewing process
* the registration inquires only brief details and
* it covers a group of coworkers, and
* only the ''RV'' responsible must submit the ''RV''.
 

== How to register? ==
=== Register a new ''RV'' ===
Typically only the leader of a scientific work group or the senior scientist of a research group/collaboration needs to login at
* https://www.bwhpc-c5.de/en/ZAS/bwforcluster_rv_registration.php
and to fill in all mandatory fields of the given web form. Please note that for your convenience you can also switch to the [https://www.bwhpc-c5.de/ZAS/bwforcluster_rechenvorhaben.php german version] of the web form. The submitter of the ''rechenvorhaben (RV)'' will be assigned the role ''RV'' responsible.

The web form consists of the following fields to be filled:
{| style="width:100%;border:3px solid darkgray; margin: 1em auto 1em auto;"
|-
! style="width:25%;padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;"| Field
! style="padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;" | Explanation
|-
!scope="column" {{Lightgray_small}}| RV Title
|scope="column" {{Tab element}} | Define a short title of your planned compute activities, maximum of 255 characters including spaces.
|-
!scope="column" {{Lightgray_small}}| RV Description
|scope="column" {{Tab element}} | Write a short abstract about your planned compute activities (maximum of 2048 characters including spaces). Research groups that contributed to the DFG grant proposal (Art. 91b GG) of the corresponding bwForCluster only need to give reference to that particular proposal.
|-
!scope="column" {{Lightgray_small}}| Scientific field
|scope="column" {{Tab element}} | Tick one or several scientific fields. Once all bwForClusters are up and running the full list of given scientific fields are supported and hence for ''rechenvorhaben (RV)'' applicable. If your ''RV'' does not match any given scientific field, please state your scientific field in the given text box. Grayed out scientific fields indicate that the corresponding bwForCluster(s) is/are not operational yet.
|-
!scope="column" {{Lightgray_small}}| Field of activity
|scope="column" {{Tab element}} | Define if your ''RV'' is primarily for research and/or teaching. If not applicable, use text box.
|-
!scope="column" {{Lightgray_small}}| Parallel paradigm
|scope="column" {{Tab element}} | State what parallel paradigm your code or application uses. Multiple ticks are allowed. Further information can be provided via text box. If you are not sure about it, please state the software you are using.
|-
!scope="column" {{Lightgray_small}}| Programming language
|scope="column" {{Tab element}} | State the programming language(s) of your code or application.
|-
!scope="column" {{Lightgray_small}}| Numerical methods
|scope="column" {{Tab element}} | State the numerical or "calculation" methods your code or application utilises. If you do not know, write "unknown".
|-
!scope="column" {{Lightgray_small}}| Software packages
|scope="column" {{Tab element}} | State all software package you will or want to use for your ''RV''. Also include compiler, MPI and numerical packages (e.g. MKL, FFTW).
|-
!scope="column" {{Lightgray_small}}| Estimated requested computing resources
|scope="column" {{Tab element}} | Roughly estimate how many CPU hours are required to finish your ''RV''. To calculate "CPU hours" multiply "elapsed parallel computing time" with "number of CPU core" per job.
''Example: Your code uses 4 CPU cores and has a wall time of 1h. This makes 4 CPU hours.''
|-
!scope="column" {{Lightgray_small}}| Planned maximum number of parallel used CPU cores per job
|scope="column" {{Tab element}} | Give an upper limit of how many CPU cores per job are required. Please cross-check with your statement about parallel paradigm.
''Obviously, a sequential code can only use 1 CPU core per job.''
|-
!scope="column" {{Lightgray_small}}| Estimated maximal memory requirements per CPU core
|scope="column" {{Tab element}} | Give an upper limit of how much RAM per CPU core your jobs require. Please give a value in GigaBytes.
|-
!scope="column" {{Lightgray_small}}| Estimated requested persistent disk space for the ''RV''
|scope="column" {{Tab element}} | State how much disk space in total you need for your ''RV''. Please give a value in GigaBytes.
''Example: If your RV has 4 more coworker and each of you produces until the end of your RV 20 GigaByte of output, your maximum disk space is 100 GigaByte.''
|-
!scope="column" {{Lightgray_small}}| Estimated maximal temporary disk space per job
|scope="column" {{Tab element}} | State how much space your intermediate files require during a job run. Please give a value in GigaBytes.
''Example: Your final output of your job is 0.1 GigaByte, but during your job 10 temporary files, each with a size of 1 GigaByte, are written to disk, hence your correct answer is 10 GigaByte.''
|-
!scope="column" {{Lightgray_small}}| Institute
|scope="column" {{Tab element}} | State the full name of your Institute.
|-
|}
Note that the fields - name, firstname, organization, mail, und EPPN - are auto-filled and can not be changed. These are your credentials as given by your home organization.

Response
# After submitting you will get an email from ''ZAS'' confirming your submission. With this email you are given an unique * acronym * password Please keep the password enclosed. Only with both, acronym and password, [[BwForCluster_User_Access#Register_for_an_RV_as_coworker | coworkers ]] can be added to your ''rechenvorhaben''.
# The cluster assignment team will be notified immediately about your submission.
# The cluster assignment team decides based on your submitted data which bwForCluster most fits, and submits its decision to ''ZAS''.
# ''ZAS'' notifies you in an email about your assigned bwForCluster and provides website details for [[BwForCluster_User_Access#Personal_registration_at_bwForCluster | step 3 ]].

 

=== Register for an ''RV'' as coworker ===

Each coworker must add herself/himself to a ''rechenvorhaben (RV)''.
To become coworker of an ''RV'', please login at
* https://www.bwhpc-c5.de/en/ZAS/bwforcluster_collaboration.php
and provide the
* acronym
* password
of the ''RV''. Your ''RV'' responsible will provide you with that information.
You will be assigned to role ''RV'' member.

Response
# After submitting the request for participating in an ''RV'', the coworker will receive an email from ''ZAS'' about the further steps (i.e. [[#Personal registration at bwForCluster | personal registration at assigned bwForCluster]]). The membership status in the ''RV'' is set active by default.
# Each person with role ''RV'' responsible or ''RV'' manager will be notified automatically. They can check the list of ''RV'' members at https://www.bwhpc-c5.de/en/ZAS/info_rv.php by clicking on the ''RV'' acronym.
# The ''RV'' responsible can set any ''RV'' coworker to ''RV'' manager and vice versa as well as deactivate/reactivate any ''RV'' coworker or ''RV'' manager.
# An ''RV'' manager can deactivate/reactivate any ''RV'' coworker.



== Roles in an ''RV'' ==
{| style="width:100%;border:3px solid darkgray; margin: 1em auto 1em auto;"
|-
! style="width:25%;padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;"| Role
! style="padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;" | Explanation
|-
!scope="column" {{Lightgray_small}}| RV responsible
|scope="column" {{Tab element}} | Registers the ''rechenvorhaben (RV)'', i.e. the planned compute activities. Can deactivate but also reactivate ''RV'' managers and coworkers. Can promote coworkers to mangers and vice versa.
|-
!scope="column" {{Lightgray_small}}| RV manager
|scope="column" {{Tab element}} | ''RV'' coworker with elevated rights to deactivate and reactivate other ''RV'' coworkers.
|-
!scope="column" {{Lightgray_small}}| RV coworker
|scope="column" {{Tab element}} | Typical user who has registered her/himself via ''RV'' acronym and password.
|-
|}

 

= Issuing bwForCluster entitlement =

Each university issues the bwForCluster entitlement
<pre>http://bwidm.de/entitlement/bwForCluster</pre>
only for their own members.

The bwForCluster entitlement issued by a university assures the operator of a bwForCluster that its university member's compute activities comply with the German Foreign Trade Act (Außenwirtschaftsgesetz - AWG) und German Foreign Trade Regulations (Außenwirtschaftsverordnung - AWV).

The following unversities have already established a process to issue the bwForCluster entitlement:

* [[BwForCluster User Access Members Hochschule Esslingen|Hochschule Esslingen]]
* [[BwCluster User Access Uni Freiburg|University of Freiburg]]
* [[BwForCluster User Access Members Uni Heidelberg|University of Heidelberg]]
* [https://kim.uni-hohenheim.de/bwhpc-account/|University of Hohenheim]
* [http://www.scc.kit.edu/downloads/sdo/Antrag_Zugangsberechtigung_bwForCluster.pdf Karlsruhe Institute of Technology (KIT)]
* [[BwForCluster User Access Members Uni Konstanz|University of Konstanz]]
* [[BwForCluster User Access Members Uni Mannheim|University of Mannheim]]
* [https://www.hlrs.de/solutions-services/academic-users/bwforcluster-access/ University of Stuttgart]
* [[BwForCluster User Access Members Uni Tübingen|University of Tübingen]]
* [[BwForCluster User Access Members Uni Ulm|University of Ulm]]

If you do not find your university in the list above, please contact your local service desk.

= Personal registration at bwForCluster =

Once you have registered your own RV (''rechenvorhaben'')
or a membership in an RV, the cluster assignment team will provide
you with information about your designated cluster. You will receive
an email with a website to create an account for yourself
on that cluster.

Available bwForCluster registration servers (service providers):
{| style="border:3px solid darkgray; margin: 1em auto 1em auto;" width="72%"
|-
!scope="row" {{Darkgray}} | Cluster topic and location
!scope="row" {{Darkgray}} | Registration server
|-
| bwForCluster Chemistry JUSTUS Ulm
| http://bwidm.rz.uni-ulm.de
|-
| bwForCluster MLS&WISO (Production and Development)
| https://bwservices.uni-heidelberg.de
|-
| bwForCluster NEMO Freiburg
| https://bwservices.uni-freiburg.de
|-
| bwForCluster BinAC Tübingen
| https://bwservices.uni-tuebingen.de
|}

Please note, this step is different to your registration at ZAS since
here you register yourself as a person. Only based on your personal credentials
a user account can be generated. However, resource requests of the RV in ZAS
will be cross-checked with the resources consumed by you and your RV coworkers
on this cluster.

After steps 1 and 2 (RV approval and bwForCluster entitlement) please visit the

* bwForCluster ''service provider'' registration website (see table above or email after RV approval):
*# Select your home organization from the list of organizations and click '''Proceed'''.
*# You will be redirected to the ''Identity Provider'' of your home organization.
*# Enter your home-organizational user ID (might be user name, email, ...) and password and click '''Login''' or '''Anmelden'''.
*# When doing this for the first time you need to accept that your personal data is transferred to the ''service provider''.
*# You will be redirected back to the cluster registration website.
*# Select '''Service Description''' of the designated cluster.
*# Click on '''Register''' link to register for this cluster.
*# Read and accept the terms and conditions of use and click on button '''Register'''.
*# Finally you will receive an email with instructions how to login to the cluster (please wait ~15 minutes before trying).
*# Click on '''Set Service Password''' and set a password for the cluster.
*# '''Note: Users from Tübingen accessing the BinAC do not set a separate service password''' and have to use their local account credentials instead.
*# '''Note:''' You can return to the registration website at any time, in order to review your registration details, change/reset your service password or deregister from the service by yourself.

= Login to bwForCluster =

Personalized details about how to login to the cluster are included
in an email send after registration at the bwForCluster service provider.

General instructions for the bwForCluster login can be found here:
{| style="border:3px solid darkgray; margin: 1em auto 1em auto;" width="73%"
|-
!scope="row" {{Darkgray}} | Cluster topic and location
!scope="row" {{Darkgray}} | Login instructions
|-
| bwForCluster Chemistry JUSTUS Ulm
| [[BwForCluster_Chemistry_Login|bwForCluster Chemistry Login]]
|-
| bwForCluster MLS&WISO Production
| [[BwForCluster_MLS&WISO_Production_Login|bwForCluster MLS&WISO Production Login]]
|-
| bwForCluster MLS&WISO Development
| [[BwForCluster_MLS&WISO_Development_Login|bwForCluster MLS&WISO Development Login]]
|-
| bwForCluster NEMO Freiburg
| [[bwForCluster NEMO Login]]
|-
| bwForCluster BinAC Tübingen
| [[bwForCluster BinAC Login]]

|}

 

----
[[Category:Access|bwForCluster]][[Category:bwForCluster_Chemistry]][[Category:bwForCluster_MLS&WISO_Production]][[Category:bwForCluster_MLS&WISO_Development]]

BwForCluster User Access

2020-03-25T15:36:49Z

T Kienzle: /* Issuing bwForCluster entitlement */ update hohenheim link

The usage of bwForCluster is free of charge. bwForClusters are customized to the requirements of particular research areas.
Each bwForCluster is/will be financed by the DFG (German Research Foundation) and by the Ministry of Science, Research and Arts of Baden-Württemberg based on scientifc grant proposal (compare proposals guidelines as per Art. 91b GG).

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:center; color:#000;vertical-align:middle;font-size:75%;" |[[File:Bwforreg.svg|center|border|500px|]]
|-
| style="text-align:center; color:#000;vertical-align:middle;" |
|}

Granting user access to a bwForCluster requires 3 steps:

# Registration of a new or joining an already existing ''rechenvorhaben (RV)'' at [https://www.bwhpc.de/zas.html "Zentrale Antragsseite (ZAS)"]. An ''RV'' defines the planned compute activities of a group of researchers. As coworker of this group you only need to register your membership in the corresponding ''RV''.
# Application for a [[#Issuing bwForCluster entitlement | bwForCluster entitlement]] issued by your university.
# [[#Personal registration at bwForCluster | Personal registration at assigned cluster site ]] based on approved ''RV'' [[File:Zas assignment icon.svg|25px|]] and issued bwForCluster entitlement [[File:bwfor entitlement icon.svg|25px|]].

 
Step 1 and 2 are pre-steps and requirements of step 3. Moreover, steps 1 and 2 can be done in parallel. The question, at which bwForCluster site to do step 3, will be answered by the cluster assignment team (CAT) based on the data of step 1, i.e. at ''ZAS''.

= ''RV'' registration at ''ZAS'' =

The ''rechenvorhaben (RV)'' registration at ''ZAS'' does '''not''' correspond to typical compute project proposals as for other HPC clusters, since:
* there is no scientific reviewing process
* the registration inquires only brief details and
* it covers a group of coworkers, and
* only the ''RV'' responsible must submit the ''RV''.
 

== How to register? ==
=== Register a new ''RV'' ===
Typically only the leader of a scientific work group or the senior scientist of a research group/collaboration needs to login at
* https://www.bwhpc-c5.de/en/ZAS/bwforcluster_rv_registration.php
and to fill in all mandatory fields of the given web form. Please note that for your convenience you can also switch to the [https://www.bwhpc-c5.de/ZAS/bwforcluster_rechenvorhaben.php german version] of the web form. The submitter of the ''rechenvorhaben (RV)'' will be assigned the role ''RV'' responsible.

The web form consists of the following fields to be filled:
{| style="width:100%;border:3px solid darkgray; margin: 1em auto 1em auto;"
|-
! style="width:25%;padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;"| Field
! style="padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;" | Explanation
|-
!scope="column" {{Lightgray_small}}| RV Title
|scope="column" {{Tab element}} | Define a short title of your planned compute activities, maximum of 255 characters including spaces.
|-
!scope="column" {{Lightgray_small}}| RV Description
|scope="column" {{Tab element}} | Write a short abstract about your planned compute activities (maximum of 2048 characters including spaces). Research groups that contributed to the DFG grant proposal (Art. 91b GG) of the corresponding bwForCluster only need to give reference to that particular proposal.
|-
!scope="column" {{Lightgray_small}}| Scientific field
|scope="column" {{Tab element}} | Tick one or several scientific fields. Once all bwForClusters are up and running the full list of given scientific fields are supported and hence for ''rechenvorhaben (RV)'' applicable. If your ''RV'' does not match any given scientific field, please state your scientific field in the given text box. Grayed out scientific fields indicate that the corresponding bwForCluster(s) is/are not operational yet.
|-
!scope="column" {{Lightgray_small}}| Field of activity
|scope="column" {{Tab element}} | Define if your ''RV'' is primarily for research and/or teaching. If not applicable, use text box.
|-
!scope="column" {{Lightgray_small}}| Parallel paradigm
|scope="column" {{Tab element}} | State what parallel paradigm your code or application uses. Multiple ticks are allowed. Further information can be provided via text box. If you are not sure about it, please state the software you are using.
|-
!scope="column" {{Lightgray_small}}| Programming language
|scope="column" {{Tab element}} | State the programming language(s) of your code or application.
|-
!scope="column" {{Lightgray_small}}| Numerical methods
|scope="column" {{Tab element}} | State the numerical or "calculation" methods your code or application utilises. If you do not know, write "unknown".
|-
!scope="column" {{Lightgray_small}}| Software packages
|scope="column" {{Tab element}} | State all software package you will or want to use for your ''RV''. Also include compiler, MPI and numerical packages (e.g. MKL, FFTW).
|-
!scope="column" {{Lightgray_small}}| Estimated requested computing resources
|scope="column" {{Tab element}} | Roughly estimate how many CPU hours are required to finish your ''RV''. To calculate "CPU hours" multiply "elapsed parallel computing time" with "number of CPU core" per job.
''Example: Your code uses 4 CPU cores and has a wall time of 1h. This makes 4 CPU hours.''
|-
!scope="column" {{Lightgray_small}}| Planned maximum number of parallel used CPU cores per job
|scope="column" {{Tab element}} | Give an upper limit of how many CPU cores per job are required. Please cross-check with your statement about parallel paradigm.
''Obviously, a sequential code can only use 1 CPU core per job.''
|-
!scope="column" {{Lightgray_small}}| Estimated maximal memory requirements per CPU core
|scope="column" {{Tab element}} | Give an upper limit of how much RAM per CPU core your jobs require. Please give a value in GigaBytes.
|-
!scope="column" {{Lightgray_small}}| Estimated requested persistent disk space for the ''RV''
|scope="column" {{Tab element}} | State how much disk space in total you need for your ''RV''. Please give a value in GigaBytes.
''Example: If your RV has 4 more coworker and each of you produces until the end of your RV 20 GigaByte of output, your maximum disk space is 100 GigaByte.''
|-
!scope="column" {{Lightgray_small}}| Estimated maximal temporary disk space per job
|scope="column" {{Tab element}} | State how much space your intermediate files require during a job run. Please give a value in GigaBytes.
''Example: Your final output of your job is 0.1 GigaByte, but during your job 10 temporary files, each with a size of 1 GigaByte, are written to disk, hence your correct answer is 10 GigaByte.''
|-
!scope="column" {{Lightgray_small}}| Institute
|scope="column" {{Tab element}} | State the full name of your Institute.
|-
|}
Note that the fields - name, firstname, organization, mail, und EPPN - are auto-filled and can not be changed. These are your credentials as given by your home organization.

Response
# After submitting you will get an email from ''ZAS'' confirming your submission. With this email you are given an unique * acronym * password Please keep the password enclosed. Only with both, acronym and password, [[BwForCluster_User_Access#Register_for_an_RV_as_coworker | coworkers ]] can be added to your ''rechenvorhaben''.
# The cluster assignment team will be notified immediately about your submission.
# The cluster assignment team decides based on your submitted data which bwForCluster most fits, and submits its decision to ''ZAS''.
# ''ZAS'' notifies you in an email about your assigned bwForCluster and provides website details for [[BwForCluster_User_Access#Personal_registration_at_bwForCluster | step 3 ]].

 

=== Register for an ''RV'' as coworker ===

Each coworker must add herself/himself to a ''rechenvorhaben (RV)''.
To become coworker of an ''RV'', please login at
* https://www.bwhpc-c5.de/en/ZAS/bwforcluster_collaboration.php
and provide the
* acronym
* password
of the ''RV''. Your ''RV'' responsible will provide you with that information.
You will be assigned to role ''RV'' member.

Response
# After submitting the request for participating in an ''RV'', the coworker will receive an email from ''ZAS'' about the further steps (i.e. [[#Personal registration at bwForCluster | personal registration at assigned bwForCluster]]). The membership status in the ''RV'' is set active by default.
# Each person with role ''RV'' responsible or ''RV'' manager will be notified automatically. They can check the list of ''RV'' members at https://www.bwhpc-c5.de/en/ZAS/info_rv.php by clicking on the ''RV'' acronym.
# The ''RV'' responsible can set any ''RV'' coworker to ''RV'' manager and vice versa as well as deactivate/reactivate any ''RV'' coworker or ''RV'' manager.
# An ''RV'' manager can deactivate/reactivate any ''RV'' coworker.



== Roles in an ''RV'' ==
{| style="width:100%;border:3px solid darkgray; margin: 1em auto 1em auto;"
|-
! style="width:25%;padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;"| Role
! style="padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;" | Explanation
|-
!scope="column" {{Lightgray_small}}| RV responsible
|scope="column" {{Tab element}} | Registers the ''rechenvorhaben (RV)'', i.e. the planned compute activities. Can deactivate but also reactivate ''RV'' managers and coworkers. Can promote coworkers to mangers and vice versa.
|-
!scope="column" {{Lightgray_small}}| RV manager
|scope="column" {{Tab element}} | ''RV'' coworker with elevated rights to deactivate and reactivate other ''RV'' coworkers.
|-
!scope="column" {{Lightgray_small}}| RV coworker
|scope="column" {{Tab element}} | Typical user who has registered her/himself via ''RV'' acronym and password.
|-
|}

 

= Issuing bwForCluster entitlement =

Each university issues the bwForCluster entitlement
<pre>http://bwidm.de/entitlement/bwForCluster</pre>
only for their own members.

The bwForCluster entitlement issued by a university assures the operator of a bwForCluster that its university member's compute activities comply with the German Foreign Trade Act (Außenwirtschaftsgesetz - AWG) und German Foreign Trade Regulations (Außenwirtschaftsverordnung - AWV).

The following unversities have already established a process to issue the bwForCluster entitlement:

* [[BwForCluster User Access Members Hochschule Esslingen|Hochschule Esslingen]]
* [[BwCluster User Access Uni Freiburg|University of Freiburg]]
* [[BwForCluster User Access Members Uni Heidelberg|University of Heidelberg]]
* [https://kim.uni-hohenheim.de/bwhpc-account|University of Hohenheim]
* [http://www.scc.kit.edu/downloads/sdo/Antrag_Zugangsberechtigung_bwForCluster.pdf Karlsruhe Institute of Technology (KIT)]
* [[BwForCluster User Access Members Uni Konstanz|University of Konstanz]]
* [[BwForCluster User Access Members Uni Mannheim|University of Mannheim]]
* [https://www.hlrs.de/solutions-services/academic-users/bwforcluster-access/ University of Stuttgart]
* [[BwForCluster User Access Members Uni Tübingen|University of Tübingen]]
* [[BwForCluster User Access Members Uni Ulm|University of Ulm]]

If you do not find your university in the list above, please contact your local service desk.

= Personal registration at bwForCluster =

Once you have registered your own RV (''rechenvorhaben'')
or a membership in an RV, the cluster assignment team will provide
you with information about your designated cluster. You will receive
an email with a website to create an account for yourself
on that cluster.

Available bwForCluster registration servers (service providers):
{| style="border:3px solid darkgray; margin: 1em auto 1em auto;" width="72%"
|-
!scope="row" {{Darkgray}} | Cluster topic and location
!scope="row" {{Darkgray}} | Registration server
|-
| bwForCluster Chemistry JUSTUS Ulm
| http://bwidm.rz.uni-ulm.de
|-
| bwForCluster MLS&WISO (Production and Development)
| https://bwservices.uni-heidelberg.de
|-
| bwForCluster NEMO Freiburg
| https://bwservices.uni-freiburg.de
|-
| bwForCluster BinAC Tübingen
| https://bwservices.uni-tuebingen.de
|}

Please note, this step is different to your registration at ZAS since
here you register yourself as a person. Only based on your personal credentials
a user account can be generated. However, resource requests of the RV in ZAS
will be cross-checked with the resources consumed by you and your RV coworkers
on this cluster.

After steps 1 and 2 (RV approval and bwForCluster entitlement) please visit the

* bwForCluster ''service provider'' registration website (see table above or email after RV approval):
*# Select your home organization from the list of organizations and click '''Proceed'''.
*# You will be redirected to the ''Identity Provider'' of your home organization.
*# Enter your home-organizational user ID (might be user name, email, ...) and password and click '''Login''' or '''Anmelden'''.
*# When doing this for the first time you need to accept that your personal data is transferred to the ''service provider''.
*# You will be redirected back to the cluster registration website.
*# Select '''Service Description''' of the designated cluster.
*# Click on '''Register''' link to register for this cluster.
*# Read and accept the terms and conditions of use and click on button '''Register'''.
*# Finally you will receive an email with instructions how to login to the cluster (please wait ~15 minutes before trying).
*# Click on '''Set Service Password''' and set a password for the cluster.
*# '''Note: Users from Tübingen accessing the BinAC do not set a separate service password''' and have to use their local account credentials instead.
*# '''Note:''' You can return to the registration website at any time, in order to review your registration details, change/reset your service password or deregister from the service by yourself.

= Login to bwForCluster =

Personalized details about how to login to the cluster are included
in an email send after registration at the bwForCluster service provider.

General instructions for the bwForCluster login can be found here:
{| style="border:3px solid darkgray; margin: 1em auto 1em auto;" width="73%"
|-
!scope="row" {{Darkgray}} | Cluster topic and location
!scope="row" {{Darkgray}} | Login instructions
|-
| bwForCluster Chemistry JUSTUS Ulm
| [[BwForCluster_Chemistry_Login|bwForCluster Chemistry Login]]
|-
| bwForCluster MLS&WISO Production
| [[BwForCluster_MLS&WISO_Production_Login|bwForCluster MLS&WISO Production Login]]
|-
| bwForCluster MLS&WISO Development
| [[BwForCluster_MLS&WISO_Development_Login|bwForCluster MLS&WISO Development Login]]
|-
| bwForCluster NEMO Freiburg
| [[bwForCluster NEMO Login]]
|-
| bwForCluster BinAC Tübingen
| [[bwForCluster BinAC Login]]

|}

 

----
[[Category:Access|bwForCluster]][[Category:bwForCluster_Chemistry]][[Category:bwForCluster_MLS&WISO_Production]][[Category:bwForCluster_MLS&WISO_Development]]

User:M Janczyk/Software/Singularity Containers

2019-07-31T08:26:43Z

T Kienzle: /* Introduction */

= Introduction =

Singularity is an open-source software for container-virtualization. Because not every different software configuration can be provided as [[Environment Modules|Modules]] on the clusters, containers offer a way to use a pre-built scientific software in a closed and reproducible space, independent from the environment. A Singularity container contains its own operating system, the intended software and all required dependencies. This also means that you can use software that isn't available for RHEL/CentOS, but is offered for other Linux systems. Singularity containers are easily movable between systems and do not require root-rights for execution (different to Docker). 

For example, a user may build a software package from a library or from source on their own computer, move it to the server (for example with scp) and execute it. This works as long as Singularity is installed on both systems, without having to deal with the environment on the cluster. 

The container generally works as its own closed-off environment. While you can access data and files stored on the host, you generally do not have access to software or modules running there. This means that you usually have to provide software that could otherwise be found in a module inside the container. 

The following tutorial gives a brief introduction to the program to create and run a container on a cluster. 

'''Building containers can be tricky. If you need help, please don't hesitate to contact [https://wiki.bwhpc.de/e/Category:Support us]. [https://wiki.bwhpc.de/e/Category:Support We can help you].'''

== Requirements to build a Container ==

Singularity requires a Linux-system. If no Linux computer is available, a virtual machine with a Linux-OS can be used. Singularity does ''not'' work on the Windows Subsystem for Linux (WSL). 

First, install Singularity 3 from source by following the instructions on the [https://sylabs.io/guides/3.0/user-guide/installation.html# official page]. Singularity 3 also requires the installation of the programming language Go. 

Singularity has to be installed on all systems trying to use the container, and has to be loaded with an appropriate module first on the cluster. 

== Building a Container ==

The command to build a new container is ''singularity build''. A container that should remain writable after building can be created with ''singularity build --sandbox''. A specific home directory can be defined by using ''singularity build --home /your/home/path/''. 

Most building operations require root-rights. If a container has to be made up from scratch, it can be built manually in a writable shell or through a definition file. 

Singularity 3 supports two formats of containers: As a Singularity Image File (''.sif'') and as a directory. In the following, directories will be used in the examples.

=== Building an existing container ===

If an existing container from Docker (''docker://'') or a container library (''library://'' or ''shub://'') should be built using Singularity, it can simply be imported as is. A writable sandbox container can be built by flagging --sandbox: 

<pre>
$ sudo singularity build <containername> library://path/to/file
$ sudo singularity build --sandbox <containername> library://path/to/file
</pre>

 

=== Building a new container with a definition file ===

A definition file, or recipe in older guides, provides the program with a script to build the container from. The same definition-file always reliably produces the same container, so it is encouraged to use definition-files for reproducibility if possible. 

The file has to contain a '''header''', specifying the operating system and its source. If one wants to use CentOS, for example: 

<pre>
Bootstrap: library
From: centos
</pre>

 

After this, various sections with specifications can follow in any order. The most important are '''%post''' and '''%environment'''. 

'''%post''' constitutes the main part of the file, with a set of bash-instructions used to build the container in order. For example, the process might start with an update and the installation of standard tools required later on: 

<pre>
%post
yum -y install update
yum -y install wget tar
cd /
mkdir example
</pre>

 

Keep in mind that the container contains almost no pre-installed packages. 

'''%environment''' can be used to set environment-variables such as paths, for example: 

<pre>
%environment
export PKG_CONFIG_PATH=/usr/bin
</pre>

 

Inputs into '''%environment''' from the definition-file are written to ''/.singularity.d/env/90-environment.sh'', which is sourced at runtime. In ''%post'', commands for starting the container can be written to that file or to ''/.singularity.d/env/91-environment.sh''. This can be useful if no '~/.bashrc' or similar is available. 

Some other options for the definition-file are '''%runtime''', which can include a script to be executed at the start of the ''run'' command, '''%files''', which can be used to copy files from the host-system into the container, and '''%labels''' and '''%help''' to provide the user of the container with credentials and a metadata help-file, respectively. More options can be found on the [https://sylabs.io/guides/3.0/user-guide/definition_files.html Singularity homepage]. 

The file is invoked with 

<pre>
$ sudo singularity build <containername> definition-file.def
</pre>

 

=== Building a new container in the Shell ===

If no definition-file is a available, a container can be built by executing the same commands that would be put in the definition file manually in a shell. Effectively, this approach is the same as building an existing container, just with the intention to write into it immidiately. To do so, a container with the desired OS has to be built as a sandbox and opened as a writable shell: 

<pre>
$ sudo singularity build --sandbox <containername> library://centos
$ sudo singularity shell --writable <containername>
</pre>

 

This shell already behaves like its own OS. Note that if you took a base-container with only an OS, it contains almost no pre-installed packages, and that some paths (e.g. ''~/'') may still reference the host-system. In general, ''cd /'' gets you to the highest parent-folder within singularity. 

<pre>
$ yum -y install update
$ yum -y install wget tar
$ cd /
$ mkdir example
</pre>

 

== Using a container ==

To use a container on a cluster, an available Singularity-module has to be loaded. For example: 

<pre>
$ module load your/singularity/version
</pre>

 

Keep in mind that other modules you may have loaded (that aren't needed for running the container itself) will ''not'' be available inside the container. 

There are three key ways to run services on a regular container, the already known ''shell'', as well as ''exec'' and ''run''. By default, all Singularity containers are read-only. To make them writable, ''--writable'' has to be specified in the command. 

Again, the container can be opened as a shell: 

<pre>
$ singularity shell <containername>
</pre>

 

On the cluster, the easiest way to run one-line commands (including commands to run other files) is passing them to the container through ''exec'': 

<pre>
$ singularity exec --writable <containername> yum -y install update
$ singularity exec <containername> script.sh
</pre>

 

''Exec'' is generally the easiest way to execute scripts. Without further specification, it runs the container in the current directory, making it a good option to run on workspaces, which Singularity might otherwise struggle with if run from ''$HOME''.

The third option, ''run'', allows to execute a '''%runscript''' that was provided in the definition file during the build-process: 

<pre>
$ singularity run <containername>
</pre>

This is for example useful to start an installed program.

 

=== Containers and Batch Jobs ===
[[Batch Jobs]] utilizing Singularity containers are generally built the same way as all other batch jobs, where the job script contains a ''singularity exec'' command. For example: 

<pre>
#!/bin/bash
#MSUB -l nodes=1:gpu
#MSUB -l walltime=3:00:00
#MSUB -l mem=5000mb
#MSUB -N Singularity

module load your/singularity/version
cd your/workspace
singularity exec --nv <containername> script1.sh
singularity exec --nv <containername> python script2.py
</pre>

 

=== Using GPUS ===

Containers can run on GPUs as well. First, make sure that all necessary libraries (for example CUDA) are installed on the container itself and all appropriate drivers are available on the host system. When using the container, both ''exec'' and ''shell'' need another mandatory specification to be able to interact with GPUs - the flag ''--nv'' after the command. 

<pre>
$ singularity exec --nv <containername> script.sh
</pre>

 

== Concluding notes ==

* Keep track of your inputs when using a (writable) shell.
* Modules loaded outside the container don't work on the inside.
* Use ''singularity exec'' for Batch-Jobs.
* Use ''singularity exec --nv'' for GPUs.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

----
[[Category:Scientific_Applications]]

User:M Janczyk/Software/Singularity Containers

2019-07-31T08:25:02Z

T Kienzle: /* Introduction */

= Introduction =

Singularity is an open-source software for container-virtualization. Because not every different software configuration can be provided as [[Environment Modules|Modules]] on the clusters, containers offer a way to use a pre-built scientific software in a closed and reproducible space, independent from the environment. A Singularity container contains its own operating system, the intended software and all required dependencies. This also means that you can use software that isn't available for RHEL/CentOS, but is offered for other Linux systems. Singularity containers are easily movable between systems and do not require root-rights for execution (different to Docker). 

For example, a user may build a software package from a library or from source on their own computer, move it to the server (for example with scp) and execute it. This works as long as Singularity is installed on both systems, without having to deal with the environment on the cluster. 

The container generally works as its own closed-off environment. While you can access data and files stored on the host, you generally do not have access to software or modules running there. This means that you usually have to provide software that could otherwise be found in a module inside the container. 

The following tutorial gives a brief introduction to the program to create and run a container on a cluster. 

'''Building containers can be tricky. If you need help, please don't hesitate to contact [https://wiki.bwhpc.de/e/Category:Support us].'''

== Requirements to build a Container ==

Singularity requires a Linux-system. If no Linux computer is available, a virtual machine with a Linux-OS can be used. Singularity does ''not'' work on the Windows Subsystem for Linux (WSL). 

First, install Singularity 3 from source by following the instructions on the [https://sylabs.io/guides/3.0/user-guide/installation.html# official page]. Singularity 3 also requires the installation of the programming language Go. 

Singularity has to be installed on all systems trying to use the container, and has to be loaded with an appropriate module first on the cluster. 

== Building a Container ==

The command to build a new container is ''singularity build''. A container that should remain writable after building can be created with ''singularity build --sandbox''. A specific home directory can be defined by using ''singularity build --home /your/home/path/''. 

Most building operations require root-rights. If a container has to be made up from scratch, it can be built manually in a writable shell or through a definition file. 

Singularity 3 supports two formats of containers: As a Singularity Image File (''.sif'') and as a directory. In the following, directories will be used in the examples.

=== Building an existing container ===

If an existing container from Docker (''docker://'') or a container library (''library://'' or ''shub://'') should be built using Singularity, it can simply be imported as is. A writable sandbox container can be built by flagging --sandbox: 

<pre>
$ sudo singularity build <containername> library://path/to/file
$ sudo singularity build --sandbox <containername> library://path/to/file
</pre>

 

=== Building a new container with a definition file ===

A definition file, or recipe in older guides, provides the program with a script to build the container from. The same definition-file always reliably produces the same container, so it is encouraged to use definition-files for reproducibility if possible. 

The file has to contain a '''header''', specifying the operating system and its source. If one wants to use CentOS, for example: 

<pre>
Bootstrap: library
From: centos
</pre>

 

After this, various sections with specifications can follow in any order. The most important are '''%post''' and '''%environment'''. 

'''%post''' constitutes the main part of the file, with a set of bash-instructions used to build the container in order. For example, the process might start with an update and the installation of standard tools required later on: 

<pre>
%post
yum -y install update
yum -y install wget tar
cd /
mkdir example
</pre>

 

Keep in mind that the container contains almost no pre-installed packages. 

'''%environment''' can be used to set environment-variables such as paths, for example: 

<pre>
%environment
export PKG_CONFIG_PATH=/usr/bin
</pre>

 

Inputs into '''%environment''' from the definition-file are written to ''/.singularity.d/env/90-environment.sh'', which is sourced at runtime. In ''%post'', commands for starting the container can be written to that file or to ''/.singularity.d/env/91-environment.sh''. This can be useful if no '~/.bashrc' or similar is available. 

Some other options for the definition-file are '''%runtime''', which can include a script to be executed at the start of the ''run'' command, '''%files''', which can be used to copy files from the host-system into the container, and '''%labels''' and '''%help''' to provide the user of the container with credentials and a metadata help-file, respectively. More options can be found on the [https://sylabs.io/guides/3.0/user-guide/definition_files.html Singularity homepage]. 

The file is invoked with 

<pre>
$ sudo singularity build <containername> definition-file.def
</pre>

 

=== Building a new container in the Shell ===

If no definition-file is a available, a container can be built by executing the same commands that would be put in the definition file manually in a shell. Effectively, this approach is the same as building an existing container, just with the intention to write into it immidiately. To do so, a container with the desired OS has to be built as a sandbox and opened as a writable shell: 

<pre>
$ sudo singularity build --sandbox <containername> library://centos
$ sudo singularity shell --writable <containername>
</pre>

 

This shell already behaves like its own OS. Note that if you took a base-container with only an OS, it contains almost no pre-installed packages, and that some paths (e.g. ''~/'') may still reference the host-system. In general, ''cd /'' gets you to the highest parent-folder within singularity. 

<pre>
$ yum -y install update
$ yum -y install wget tar
$ cd /
$ mkdir example
</pre>

 

== Using a container ==

To use a container on a cluster, an available Singularity-module has to be loaded. For example: 

<pre>
$ module load your/singularity/version
</pre>

 

Keep in mind that other modules you may have loaded (that aren't needed for running the container itself) will ''not'' be available inside the container. 

There are three key ways to run services on a regular container, the already known ''shell'', as well as ''exec'' and ''run''. By default, all Singularity containers are read-only. To make them writable, ''--writable'' has to be specified in the command. 

Again, the container can be opened as a shell: 

<pre>
$ singularity shell <containername>
</pre>

 

On the cluster, the easiest way to run one-line commands (including commands to run other files) is passing them to the container through ''exec'': 

<pre>
$ singularity exec --writable <containername> yum -y install update
$ singularity exec <containername> script.sh
</pre>

 

''Exec'' is generally the easiest way to execute scripts. Without further specification, it runs the container in the current directory, making it a good option to run on workspaces, which Singularity might otherwise struggle with if run from ''$HOME''.

The third option, ''run'', allows to execute a '''%runscript''' that was provided in the definition file during the build-process: 

<pre>
$ singularity run <containername>
</pre>

This is for example useful to start an installed program.

 

=== Containers and Batch Jobs ===
[[Batch Jobs]] utilizing Singularity containers are generally built the same way as all other batch jobs, where the job script contains a ''singularity exec'' command. For example: 

<pre>
#!/bin/bash
#MSUB -l nodes=1:gpu
#MSUB -l walltime=3:00:00
#MSUB -l mem=5000mb
#MSUB -N Singularity

module load your/singularity/version
cd your/workspace
singularity exec --nv <containername> script1.sh
singularity exec --nv <containername> python script2.py
</pre>

 

=== Using GPUS ===

Containers can run on GPUs as well. First, make sure that all necessary libraries (for example CUDA) are installed on the container itself and all appropriate drivers are available on the host system. When using the container, both ''exec'' and ''shell'' need another mandatory specification to be able to interact with GPUs - the flag ''--nv'' after the command. 

<pre>
$ singularity exec --nv <containername> script.sh
</pre>

 

== Concluding notes ==

* Keep track of your inputs when using a (writable) shell.
* Modules loaded outside the container don't work on the inside.
* Use ''singularity exec'' for Batch-Jobs.
* Use ''singularity exec --nv'' for GPUs.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

----
[[Category:Scientific_Applications]]

User:M Janczyk/Software/Singularity Containers

2019-07-31T08:21:41Z

T Kienzle: some minor changes

= Introduction =

Singularity is an open-source software for container-virtualization. Because not every different software configuration can be provided as [[Environment Modules|Modules]] on the clusters, containers offer a way to use a pre-built scientific software in a closed and reproducible space, independent from the environment. A Singularity container contains its own operating system, the intended software and all required dependencies. This also means that you can use software that isn't available for RHEL/CentOS, but is offered for other Linux systems. Singularity containers are easily movable between systems and do not require root-rights for execution (different to Docker). 

For example, a user may build a software package from a library or from source on their own computer, move it to the server (for example with scp) and execute it. This works as long as Singularity is installed on both systems, without having to deal with the environment on the cluster. 

The container generally works as its own closed-off environment. While you can access data and files stored on the host, you generally do not have access to software or modules running there. This means that you usually have to provide software that could otherwise be found in a module inside the container. 

The following tutorial gives a brief introduction to the program to create and run a container on a cluster. 

== Requirements to build a Container ==

Singularity requires a Linux-system. If no Linux computer is available, a virtual machine with a Linux-OS can be used. Singularity does ''not'' work on the Windows Subsystem for Linux (WSL). 

First, install Singularity 3 from source by following the instructions on the [https://sylabs.io/guides/3.0/user-guide/installation.html# official page]. Singularity 3 also requires the installation of the programming language Go. 

Singularity has to be installed on all systems trying to use the container, and has to be loaded with an appropriate module first on the cluster. 

== Building a Container ==

The command to build a new container is ''singularity build''. A container that should remain writable after building can be created with ''singularity build --sandbox''. A specific home directory can be defined by using ''singularity build --home /your/home/path/''. 

Most building operations require root-rights. If a container has to be made up from scratch, it can be built manually in a writable shell or through a definition file. 

Singularity 3 supports two formats of containers: As a Singularity Image File (''.sif'') and as a directory. In the following, directories will be used in the examples.

=== Building an existing container ===

If an existing container from Docker (''docker://'') or a container library (''library://'' or ''shub://'') should be built using Singularity, it can simply be imported as is. A writable sandbox container can be built by flagging --sandbox: 

<pre>
$ sudo singularity build <containername> library://path/to/file
$ sudo singularity build --sandbox <containername> library://path/to/file
</pre>

 

=== Building a new container with a definition file ===

A definition file, or recipe in older guides, provides the program with a script to build the container from. The same definition-file always reliably produces the same container, so it is encouraged to use definition-files for reproducibility if possible. 

The file has to contain a '''header''', specifying the operating system and its source. If one wants to use CentOS, for example: 

<pre>
Bootstrap: library
From: centos
</pre>

 

After this, various sections with specifications can follow in any order. The most important are '''%post''' and '''%environment'''. 

'''%post''' constitutes the main part of the file, with a set of bash-instructions used to build the container in order. For example, the process might start with an update and the installation of standard tools required later on: 

<pre>
%post
yum -y install update
yum -y install wget tar
cd /
mkdir example
</pre>

 

Keep in mind that the container contains almost no pre-installed packages. 

'''%environment''' can be used to set environment-variables such as paths, for example: 

<pre>
%environment
export PKG_CONFIG_PATH=/usr/bin
</pre>

 

Inputs into '''%environment''' from the definition-file are written to ''/.singularity.d/env/90-environment.sh'', which is sourced at runtime. In ''%post'', commands for starting the container can be written to that file or to ''/.singularity.d/env/91-environment.sh''. This can be useful if no '~/.bashrc' or similar is available. 

Some other options for the definition-file are '''%runtime''', which can include a script to be executed at the start of the ''run'' command, '''%files''', which can be used to copy files from the host-system into the container, and '''%labels''' and '''%help''' to provide the user of the container with credentials and a metadata help-file, respectively. More options can be found on the [https://sylabs.io/guides/3.0/user-guide/definition_files.html Singularity homepage]. 

The file is invoked with 

<pre>
$ sudo singularity build <containername> definition-file.def
</pre>

 

=== Building a new container in the Shell ===

If no definition-file is a available, a container can be built by executing the same commands that would be put in the definition file manually in a shell. Effectively, this approach is the same as building an existing container, just with the intention to write into it immidiately. To do so, a container with the desired OS has to be built as a sandbox and opened as a writable shell: 

<pre>
$ sudo singularity build --sandbox <containername> library://centos
$ sudo singularity shell --writable <containername>
</pre>

 

This shell already behaves like its own OS. Note that if you took a base-container with only an OS, it contains almost no pre-installed packages, and that some paths (e.g. ''~/'') may still reference the host-system. In general, ''cd /'' gets you to the highest parent-folder within singularity. 

<pre>
$ yum -y install update
$ yum -y install wget tar
$ cd /
$ mkdir example
</pre>

 

== Using a container ==

To use a container on a cluster, an available Singularity-module has to be loaded. For example: 

<pre>
$ module load your/singularity/version
</pre>

 

Keep in mind that other modules you may have loaded (that aren't needed for running the container itself) will ''not'' be available inside the container. 

There are three key ways to run services on a regular container, the already known ''shell'', as well as ''exec'' and ''run''. By default, all Singularity containers are read-only. To make them writable, ''--writable'' has to be specified in the command. 

Again, the container can be opened as a shell: 

<pre>
$ singularity shell <containername>
</pre>

 

On the cluster, the easiest way to run one-line commands (including commands to run other files) is passing them to the container through ''exec'': 

<pre>
$ singularity exec --writable <containername> yum -y install update
$ singularity exec <containername> script.sh
</pre>

 

''Exec'' is generally the easiest way to execute scripts. Without further specification, it runs the container in the current directory, making it a good option to run on workspaces, which Singularity might otherwise struggle with if run from ''$HOME''.

The third option, ''run'', allows to execute a '''%runscript''' that was provided in the definition file during the build-process: 

<pre>
$ singularity run <containername>
</pre>

This is for example useful to start an installed program.

 

=== Containers and Batch Jobs ===
[[Batch Jobs]] utilizing Singularity containers are generally built the same way as all other batch jobs, where the job script contains a ''singularity exec'' command. For example: 

<pre>
#!/bin/bash
#MSUB -l nodes=1:gpu
#MSUB -l walltime=3:00:00
#MSUB -l mem=5000mb
#MSUB -N Singularity

module load your/singularity/version
cd your/workspace
singularity exec --nv <containername> script1.sh
singularity exec --nv <containername> python script2.py
</pre>

 

=== Using GPUS ===

Containers can run on GPUs as well. First, make sure that all necessary libraries (for example CUDA) are installed on the container itself and all appropriate drivers are available on the host system. When using the container, both ''exec'' and ''shell'' need another mandatory specification to be able to interact with GPUs - the flag ''--nv'' after the command. 

<pre>
$ singularity exec --nv <containername> script.sh
</pre>

 

== Concluding notes ==

* Keep track of your inputs when using a (writable) shell.
* Modules loaded outside the container don't work on the inside.
* Use ''singularity exec'' for Batch-Jobs.
* Use ''singularity exec --nv'' for GPUs.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

----
[[Category:Scientific_Applications]]

JUSTUS2/Visualization

2017-11-29T08:19:41Z

T Kienzle:

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| vis/tigervnc
|-
| Availability
| [[bwUniCluster]] | [[BwForCluster_Chemistry]]
|-
| License
| GPL
|-
| Citing
| n/a
|-
| Links
| [http://www.tigervnc.org/ TigerVNC Homepage]
|-
| Graphical Interface
| No
|}
= Introduction to TigerVNC =
'''TigerVNC''' is a high-performance implementation of VNC (Virtual Network Computing), a client/server application that allows
users to launch and interact with graphical applications on remote machines. It should be faster than standard X11 forwarding and thus can
be used if a graphical software feels slow and has bad responsiveness.
 
 
= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/vis/tigervnc
|width=99%
|height=180
|border=0
}}
 

= Start =
First, you need to start the VNC server. The exact startup mechanism depends on the cluster.
* [[Start VNC Server - bwUniCluster]]
* [[Start VNC Server - bwForCluster Chemistry]]
* [[Start VNC Server - bwForCluster Chemistry - 3D Acceleration]]
 
= Login =
The startup script of the VNC server should print detailed instructions on how to establish the connection to the VNC server from your local computer. They depend on whether you
use Windows or Linux and if you work with TurboVNC Java Viewer which is a tool that can simplify the process a little bit but needs the Java Development Kit (JDK) to run. Therefore the next steps are
divided into 3 cases. Each command should be issued on the local computer.

* '''Login with TurboVNC Java Viewer''' Needed Software: [http://turbovnc.org/ TurboVNC 2.0], JDK Open TurboVNC Java Viewer. Go to Options... -> Security -> Gateway and fill in the parameters provided by the ''run_vncserver'' script. You can save these settings in the "Global" tab if you want to. Now you can click "OK", supply the VNC server and connect to the server. You should be prompted for your ssh password and your VNC password and after that the connection is established.

* '''Login without TurboVNC Java Viewer for Linux users''' Needed Software: A VNC viewer such as tigervnc or turbovnc A tunnel must be created with the ssh command given by the ''run_vncserver'' script. Open a new terminal, start a VNC viewer and connect to localhost:n, where n is the display number printed by ''run_vncserver'', using a command such like this <pre>$ vncviewer localhost:1</pre>

* '''Login without TurboVNC Java Viewer for Windows users''' Needed Software: [https://sourceforge.net/projects/tigervnc/files/tigervnc/1.3.0 tigervnc], [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Putty] You need to start Putty and go to Connection -> SSH -> Tunnels. Fill in the parameters provided by ''run_vncserver''. After you clicked "Add" you must navigate to Session and connect to the bwUniCluster with your username and password. Once the connection is established start the tigervnc client and connect to localhost:n where n is the display number printed by ''run_vncserver''.
 

= Shutdown =
To exit your VNC session it is not sufficient to only close the window of the viewer, because this will not terminate the VNC server. The server will keep running and you will run into problems
when you try to start a new VNC session later on. Please use the "log out" function of the desktop environment inside the VNC session, this will terminate the server properly.
[[Category:visualization]][[Category:bwUniCluster]]
 
 
= Starting OpenGL programs =
Starting an OpenGL program in a 3D accelerated VNC session results in an error message. In order to fix this issue and redirect OpenGL commands to the graphics card on the cluster you have to use the vglrun command like in the following example:
$ vglrun glxgears -info

Data Transfer

2017-11-29T08:10:20Z

T Kienzle:

<H1>Using SFTP from Unix client</H1>

'''Example:'''

<pre>
> sftp ka_xy1234@bwfilestorage.lsdf.kit.edu
Connecting to bwfilestorage.lsdf.kit.edu 
ka_xy1234@bwfilestorage.lsdf.kit.edu's password:
sftp> ls
snapshots
temp test
sftp> help
...
sftp> put myfile
sftp> get myfile
</pre>

<H1>Using SFTP from Windows and Mac client</H1>

Windows clients do not have a SCP/SFTP client installed by default, so it needs to be installed before this protocol can be used.

'''Tools for example:'''
* [https://www.openssh.com/ OpenSSH]
*[https://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Putty suite] (for Windows and Unix)
*[https://winscp.net/eng/download.php WinSCP] (for Windows)
*[https://filezilla-project.org/download.php?show_all=1 FileZilla] (for Windows, Mac and Linux)
*[https://cygwin.com/install.html Cygwin] (for Windows)
 
'''network drive over SFTP:'''
*[https://www.southrivertechnologies.com/download/downloadwd.html WebDrive] (for Windows and Mac)
*[https://www.eldos.com/sftp-net-drive/comparison.php SFTP Net Drive (ELDOS)] (for Windows)
*[https://www.netdrive.net/ NetDrive] (for Windows)
*[https://www.expandrive.com/expandrive ExpanDrive] (for Windows and Mac)
<hr>
 
 
 
 
[[Category:bwFileStorage|SFTP]]

Data Transfer

2017-11-29T08:08:51Z

T Kienzle:

<H1>Using SFTP from Unix client</H1>

'''Example:'''

<pre>
> sftp ka_xy1234@bwfilestorage.lsdf.kit.edu
Connecting to bwfilestorage.lsdf.kit.edu 
ka_xy1234@bwfilestorage.lsdf.kit.edu's password:
sftp> ls
snapshots
temp test
sftp> help
...
sftp> put myfile
sftp> get myfile
</pre>

<H1>Using SFTP from Windows and Mac client</H1>

Windows clients do not have a SCP/SFTP client installed by default, so it needs to be installed before this protocol can be used.

'''Tools for example:'''
* [https://www.openssh.com/ OpenSSH]
*[https://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Putty suite] (for Windows and Unix)
*[https://winscp.net/eng/download.php WinSCP] (for Windows)
*[https://filezilla-project.org/download.php?show_all=1 FileZilla] (for Windows, Mac and Linux)
*[http://cygwin.com/install.html Cygwin] (for Windows)
 
'''network drive over SFTP:'''
*[https://www.southrivertechnologies.com/download/downloadwd.html WebDrive] (for Windows and Mac)
*[https://www.eldos.com/sftp-net-drive/comparison.php SFTP Net Drive (ELDOS)] (for Windows)
*[https://www.netdrive.net/ NetDrive] (for Windows)
*[https://www.expandrive.com/expandrive ExpanDrive] (for Windows and Mac)
<hr>
 
 
 
 
[[Category:bwFileStorage|SFTP]]

Data Transfer

2017-11-29T08:07:32Z

T Kienzle:

<H1>Using SFTP from Unix client</H1>

'''Example:'''

<pre>
> sftp ka_xy1234@bwfilestorage.lsdf.kit.edu
Connecting to bwfilestorage.lsdf.kit.edu 
ka_xy1234@bwfilestorage.lsdf.kit.edu's password:
sftp> ls
snapshots
temp test
sftp> help
...
sftp> put myfile
sftp> get myfile
</pre>

<H1>Using SFTP from Windows and Mac client</H1>

Windows clients do not have a SCP/SFTP client installed by default, so it needs to be installed before this protocol can be used.

'''Tools for example:'''
* [http://www.openssh.com/ OpenSSH]
*[http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Putty suite] (for Windows and Unix)
*[https://winscp.net/eng/download.php WinSCP] (for Windows)
*[https://filezilla-project.org/download.php?show_all=1 FileZilla] (for Windows, Mac and Linux)
*[http://cygwin.com/install.html Cygwin] (for Windows)
 
'''network drive over SFTP:'''
*[https://www.southrivertechnologies.com/download/downloadwd.html WebDrive] (for Windows and Mac)
*[https://www.eldos.com/sftp-net-drive/comparison.php SFTP Net Drive (ELDOS)] (for Windows)
*[http://www.netdrive.net/ NetDrive] (for Windows)
*[https://www.expandrive.com/expandrive ExpanDrive] (for Windows and Mac)
<hr>
 
 
 
 
[[Category:bwFileStorage|SFTP]]

JUSTUS2/Software/Schrodinger

2017-11-29T08:03:54Z

T Kienzle:

{| width=600px class="wikitable"
|-
! colspan="2" style="text-align:center" | Schrödinger
|-
| module load
| chem/schrodinger

|-
| Availability
| [[bwUniCluster]] | [[BwForCluster_Chemistry]] | bwGRiD Tuebingen

|-
| License
| commercial
|-
|Citing
| 5. See Schrodinger manual
|-
| Links
| [https://www.schrodinger.com Homepage] | [https://www.schrodinger.com/supportdocs/18/ Documentation]
|-
| Graphical Interface
| Yes, Maestro, Desmond
|}

== Description ==
'''Schrödinger''' is a collection of software chemical or biochemical use. It offers various tools, like Maestro, that acts like an interface to all other schrödinger software and it is supposed to be used for material science, to investigate the structure, the reactivity and properties of chemical systems. Maestro is part of the Materials science suite.

= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/chem/schrodinger
|width=99%
|height=370
|border=0
}}
On the command line interface of any bwHPC cluster you'll get a list of available versions by using
the command
 
''''module avail chem/schrodinger''''.
<pre>
: UniCluster
$ module avail chem/schrodinger
------------------------ /opt/bwhpc/common/modulefiles -------------------------
chem/schrodinger/2014u1 chem/schrodinger/2015u3
chem/schrodinger/2014u2 chem/schrodinger/2015u4
: Justus
$ module avail chem/schrodinger
------------------------ /opt/bwhpc/common/modulefiles -------------------------
chem/schrodinger/2015u4
</pre>
The module is called ''''schrodinger'''' and '''not''' 'schoedinger' or 'schrödinger'!
 

= License and Registering =
Schrödinger, especially Maestro, is available after registering for academical use for no charge. Downloadable here: [https://www.schroedinger.com/downloadcenter/ Download Center] 
Your request will be verified and after that you can download Maestro for academical use. That will take about 48 hours.

== Prerequirements ==
=== Requesting the License Key ===
After installing Maestro you need a license key which you can request from the „Competence Center for Bioinformatics“. Please contact

==== Set environment variable for the license server ====
When you got the licence server information, set it as an environment variable, editing $LICENSE_SERVER$ in the following command.
<pre>export LM_LICENSE_FILE=$LICENSE_SERVER$</pre>

On a Mac system you should additionally execute following tasks
<pre>sudo vi /etc/launchd.conf</pre>

Insert: <pre>setenv LM_LICENSE_FILE $LICENSE_SERVER$</pre> and save the file.

Execute following command and start Maestro from Sportlight:
<pre>launchctl setenv LM_LICENSE_FILE $LICENSE_FILE$</pre>

=== bwGRiD/bwUni/JUSTUS specific installation ===

Follow the instruction for your operating system.

==== Linux/Mac ====

Download [https://raw.githubusercontent.com/marekdynowski/bwSchrodingerHosts/master/post-install.sh this file] and open a terminal in the download folder. Make the bash-script executable
<pre>chmod u+x post-install.sh</pre>
and execute it.
<pre>./post-install.sh</pre>
You will be asked for the License Server, your usernames on the cluster and the schrodinger path/version.

==== Windows ====

$SCHRODINGER ist the path to your schrodinger installation.
Download [https://codeload.github.com/marekdynowski/MOAB-Schroedinger-Submitter/zip/master this zipfile], extract it and copy the MOAB folder to $SCHRODINGER/queues/.

Download the [https://raw.githubusercontent.com/marekdynowski/bwSchrodingerHosts/master/schrodinger.hosts sample schrodinger.hosts]
and change the entries
* %LICENSE_SERVER% the address of the license server
* %USER_BWGRID% your username on the bwhpc cluster
* %USER_BWUNI% your username on the bwUni cluster
* %USER_BWJUSTUS% your username on the Justus cluster
* %SCHROD_VERSION% the schrodinger version (e.g. 2015u4)
You can delete sections of the schrodinger.hosts if you don't have an account on these clusters.

Please execute these commands in your windows shell. (Close Schrodinger before executing these commands!)
<pre>
cd $SCHRODINGER/utilities
feature_flags.exe -d JOBCON_JSERVER_GO
jserver.exe -cleanall
jserver.exe -shutdown
</pre>

== Usage ==

=== Copy your public key ===
Maestro needs a passwordless login for submitting jobs on the cluster.
Login to the cluster and copy your public key of your host system from <tt>.ssh/id_rsa.pub</tt> into <tt>.ssh/authorized_keys</tt> file on the cluster.

=== Using Maestro with the offered example ===
First start Maestro and choose the following menu items: 
[[File:Schrodinger-Usage01.jpg]]

Then load the molecularDynamicsExample.cms from your filesystem: 
[[File:Schrodinger-Usage02.jpg]]

Choose your desired host and click "Run". 
[[File:Schrodinger-Usage03.jpg]]

=== Monitoring your Job ===
Maestro offers an job monitoring interface, which you can start in the left corner of the molecule window.
[[File:Schrodinger-Monitor01.jpg]]
Furthermore you are able to manipulate you job on the cluster.

[[Category:Chemistry software]][[Category:bwUniCluster]][[Category:bwForCluster_Chemistry]]

BwUniCluster3.0/Software/OpenFoam

2017-11-28T13:05:33Z

T Kienzle:

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| cae/openfoam
|-
| Availability
| [[bwUniCluster]] | [[BwForCluster_Chemistry]] | bwGRiD-Tübingen
|-
| License
| [https://www.openfoam.org/licence.php GNU General Public Licence]
|-
| Citing
| n/a
|-
| Links
| [https://www.openfoam.org/ Openfoam Homepage] | [https://www.openfoam.org/docs/ Documentation]
|-
| Graphical Interface
| No
|}
 
 
= Description =
The OpenFOAM® (Open Field Operation and Manipulation) CFD Toolbox is a free, open source CFD software package with an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.
 
 
= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/cae/openfoam
|width=99%
|height=600
|border=0
}}
 
In order to check what OpenFOAM versions are installed on the system, run the following command:
<pre>$ module avail cae/openfoam</pre>
Typically, several OpenFOAM versions might be available.
 
Any available version can be accessed by loading the appropriate module:
<pre>$ module load cae/openfoam/<version></pre>
with <version> specifying the desired version.

To activate the OpenFOAM applications, after the module is loaded, run the following:
<pre>$ source $FOAM_INIT</pre>
or simply:
<pre>$ foamInit</pre>
 

= Parallel run with OpenFOAM =
To improve the concurrently solving process and decrease the error occurrence probability while running an OpenFOAM job in parallel (''on a singlenode''), some modifications have been introduced. Specifically, after the case decomposition is done, it is recommended to save the decomposed data directly on the nodes in a local folder and it run from there. When the calculations are over, the data is moved back to the case folder and reconstructed. It will improve the overall performance, considering that you allocated enough wall-time to decompose and rebuild your cases, as it moves theprocessor*folders to and out of the nodes local work-space. For such a procedure it is necessary to use the following commands for decomposition and reconstruction of the geometry domain:
<pre>$ decomposeParHPC
$ reconstructParHPC
$ reconstructParMeshHPC</pre>
instead of:
<pre>$ decomposePar
$ reconstructPar
$ recontructParMesh</pre>

For example, if you want to runsnappyHexMeshin parallel, you may use the following commands:
<pre>$ decomposeParMeshHPC
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMeshHPC -constant</pre>
instead of:
<pre>$ decomposePar
$ mpiexec snappyHexMesh -overwrite
$ reconstructParMesh -constant</pre>
 

= Building an OpenFOAM batch file for parallel processing =
== General information ==
To run any job in a parallel mode with OpenFOAM, it is necessary to decompose the geometry domain into segments, equal to the number of processors (or threads) you intend to use. That means, for example, if you want to run a case on 8 processors, you will have to decompose the mesh in 8 segments, first. Then, you start the solver in ''parallel'' mode, letting ''OpenFOAM'' to run calculations concurrently on these segments, one processor responding for one segment of the mesh, sharing the data with all other processors in between. There is, of course, a mechanism that connects properly the calculations, so you don't loose your data or generate wrong results. Decomposition and segments building process is handled bydecomposeParutility. In "''system/decomposeParDict''" you may specify, how many "segments" you want your geometry domain to be divided in, and which decomposition method to use. The automatic one is "''scotch''". It trims the mesh, collecting as many cells as possible per processor, trying to avoid having empty segments or segments with few cells. If you want your mesh to be divided in other way, specifying the number of segments it should be cut in x, y or z direction, for example, you can use "simple" or "hierarchical" methods. There are some other ways as well, with more documentation on the internet.
 
== Wrapper script generation ==
'''Attention:''' openfoam module loads automatically the necessary openmpi module for parallel run, do '''NOT''' load another version of mpi, as it may conflict with the loaded openfoam version.

A job-script to submit a batch job called ''job_openfoam.sh'' that runs ''icoFoam'' solver with OpenFoam version 2.4.0, on 8 processors, requiring 6000 MByte of total physical memory per processor and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=1:ppn=8
#MSUB -l walltime=06:00:00
#MSUB -l pmem=6000mb
#MSUB -v FOAM_MODULE="cae/openfoam/2.4.0"
#MSUB -v MPIRUN_OPTIONS="--bind-to core --map-by core -report-bindings"
#MSUB -v EXECUTABLE="icoFoam"
#MSUB -N test_icoFoam
#MSUB -o icoFoam.log
#MSUB -j oe

# openfoam-2.4.0 automatically loads mpi/openmpi/1.8-gnu-4.9
module load ${FOAM_MODULE}
foamInit

# remove decomposePar if you already decomposed your case beforehand
decomposePar &&

# starting the solver in parallel. Name of the solver is given in the "EXECUTABLE" variable,
# in the header
mpirun ${MPIRUN_OPTIONS} ${EXECUTABLE} -parallel &&

# remove reconstructPar and the '&&' operator from the command above
# if you would like to reconstruct the case later
reconstructPar

</source>
|}
 

= Using I/O and reducing the amount of data and files =
In OpenFOAM, you can control which variables or fields are written at specific times. For example, for post-processing purposes, you might need only a subset of variables. In order to control which files will be written, there is a function object called "writeObjects".

An example controlDict file may look like this: At the top of the file (entry "writeControl") you specify that ALL fields (variables) required for restarting are saved every 12 wall-clock hours. Then, additionally, at the bottom of the controlDict in the "functions" block, you can add a function object of type "writeObjects". With this function object, you can control the output of specific fields independent of the entry at the top of the file:

{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="text">
/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 4.1.x |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

startFrom latestTime;
startTime 0;
stopAt endTime;
endTime 1e2;
deltaT 1e-5;

writeControl clockTime;
writeInterval 43200; // write ALL fields necessary to restart your simulation
// every 43200 wall-clock seconds = 12 hours of real time

purgeWrite 0;
writeFormat binary;
writePrecision 10;
writeCompression off;
timeFormat general;
timePrecision 10;
runTimeModifiable false;

functions
{
writeFields // name of the function object
{
type writeObjects;
libs ( "libutilityFunctionObjects.so" );

objects
(
T U rho // list of fields/variables to be written
);

// E.g. write every 1e-5 seconds of simulation time only the specified fields
writeControl runTime;
writeInterval 1e-5; // write every 1e-5 seconds
}
}

</source>
|}
 

You can also define multiple function objects in order to write different subsets of fields at different times. You can also use wildcards in the list of fields- for example, in order to write out all fields starting with "RR_" you can add
<pre>
"RR_.*"
</pre>
to the list of objects. You can get a list of valid field names by writing "banana" in the field list. During the run of the solver all valid field names are printed.
The output time can be changed too. Instead of writing at specific times in the simulation, you can also write after a certain number of time steps or depening on the wall clock time:

<pre>// write every 100th simulation time step
writeControl timeStep;
writeInterval 100;
</pre>

<pre>// every 3600 seconds of real wall clock time
writeControl runtime;
writeInterval 3600;
</pre>

If you use OpenFOAM before version 4.0 or 1606, the type of function object is:
<pre>
type writeRegisteredObject; // (instead of type writeObjects)
</pre>
If you use OpenFOAM before version 3.0, you have to load the library with
<pre>
functionObjectLibs ("libIOFunctionObjects.so"); // (instead of libs ( "libutilityFunctionObjects.so" ))
</pre>
and exchange the entry "writeControl" with "outputControl".

= OpenFOAM and ParaView on bwUniCluster=
ParaView is not directly linked to OpenFOAM installation on the cluster. Therefore, to visualize OpenFOAM jobs with ParaView, they will have to be manually opened within the specific ParaView module.

1. Load the ParaView module. For example:
<pre>$ module load cae/paraview/4.3.1</pre>
2. Create a dummy '*.openfoam' file in the OpenFOAM case folder:
<pre>$ cd <case_folder_path>
$ touch <case_name>.openfoam</pre>
'''NOTICE:''' the name of the dummy file should be the same as the name of the OpenFOAM case folder, with '.openfoam' extension.

3. Open ParaView:
<pre>$ paraview</pre>
'''NOTICE:''' ParaView is a visualization software which requires X-server to run.

4. In Paraview go to 'File' -> 'Open', or press Ctrl+O. Choose to show 'All files (*)', and open your <case_name>.openfoam file. In the pop-up window select OpenFOAM, and press 'Ok'.

5. That's it! Enjoy ParaView and OpenFOAM.

----
[[Category:Engineering software]][[Category:bwUniCluster]][[Category:bwForCluster_Chemistry]]

BwForCluster User Access Members Hochschule Esslingen

2017-11-28T12:57:20Z

T Kienzle:

= Access to bwForCluster for members of Hochschule Esslingen =

== Login prerequisites ==

* A valid Hochschule Esslingen account is required for access to the bwForCluster.

* As part of the login procedure, you will get a page that shows the data on your account that will be transferred from Hochschule Esslingen to the cluster. This page shows your entitlements. It should list the entitlement: <pre>http://bwidm.de/entitlement/bwForCluster</pre> If it does not, please contact [mailto:grid-support@hs-esslingen.de grid-support@hs-esslingen.de] for further information.

== Terms of Use ==

As with any other Hochschule Esslingen service, users of bwForCluster must strictly adhere to the [https://www.hs-esslingen.de/de/hochschule/service/rechenzentrum/organisatorisches/betriebsordnung-des-rechenzentrums.html Betriebsordnung des Rechenzentrums].

Registration Support - bwUniCluster

2017-11-28T12:56:03Z

T Kienzle:

== Registration ==
If you have questions or problems concerning '''registration''', please open a ticket via

* '''[http://www.support.bwhpc-c5.de bwSupport Portal]'''

or contact your local university hotline:

{| style="width:100%; vertical-align:top; background:#f5fffa;border:2px solid #000000;"
! University !! Hotline
|-
| Albert Ludwig University of Freiburg || hpc-support (ät) hpc.uni-freiburg.de bwHPC-Support or visit https://www.hpc.uni-freiburg.de/bwunicluster
|-
| Eberhard Karls University, Tübingen || hpcmaster (ät) uni-tuebingen.de
|-
| Karlsruhe Institute of Technology (KIT) || bwunicluster-hotline (ät) lists.kit.edu
|-
| Ruprecht-Karls-Universität Heidelberg || hpc-support (ät) urz.uni-heidelberg.de
|-
| Ulm University || helpdesk (ät) uni-ulm.de
|-
| University of Hohenheim || kim-bw-projekt (ät) uni-hohenheim.de
|-
| University of Konstanz || support (ät) uni-konstanz.de
|-
| University of Mannheim ||hpc-support (ät) mailman.uni-mannheim.de
|-
| University of Stuttgart || bwunicluster (ät) hlrs.de
|}

----
[[Category:Support]][[Category:bwUniCluster]]

Data Transfer

2017-11-28T12:38:29Z

T Kienzle:

<H1>Using SFTP from Unix client</H1>

'''Example:'''

<pre>
> sftp ka_xy1234@bwfilestorage.lsdf.kit.edu
Connecting to bwfilestorage.lsdf.kit.edu 
ka_xy1234@bwfilestorage.lsdf.kit.edu's password:
sftp> ls
snapshots
temp test
sftp> help
...
sftp> put myfile
sftp> get myfile
</pre>

<H1>Using SFTP from Windows and Mac client</H1>

Windows clients do not have a SCP/SFTP client installed by default, so it needs to be installed before this protocol can be used.

'''Tools for example:'''
* [http://www.openssh.com/ OpenSSH]
*[http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Putty suite] (for Windows and Unix)
*[https://winscp.net/eng/download.php WinSCP] (for Windows)
*[https://filezilla-project.org/download.php?show_all=1 FileZilla] (for Windows, Mac and Linux)
*[http://cygwin.com/install.html Cygwin] (for Windows)
 
'''network drive over SFTP:'''
*[http://www.southrivertechnologies.com/download/downloadwd.html WebDrive] (for Windows and Mac)
*[https://www.eldos.com/sftp-net-drive/comparison.php SFTP Net Drive (ELDOS)] (for Windows)
*[http://www.netdrive.net/ NetDrive] (for Windows)
*[https://www.expandrive.com/expandrive ExpanDrive] (for Windows and Mac)
<hr>
 
 
 
 
[[Category:bwFileStorage|SFTP]]

JUSTUS2/Software/Molden

2017-11-28T12:33:41Z

T Kienzle:

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| chem/molden
|-
| Availability
| [[bwUniCluster]] | [[BwForCluster_Chemistry]]
|-
| License
| Free. See: [http://www.cmbi.ru.nl/molden/CopyRight.html Copyright CMBI]
|-
| Citing
| [http://www.cmbi.ru.nl/molden/ref.html Molden-Reference]
|-
| Links
| [http://www.cmbi.ru.nl/molden/ Homepage] | [http://www.cmbi.ru.nl/molden Documentation]
|-
| Graphical Interface
| [[#Graphical User Interface (GUI)|Yes]]
|}

= Description =
'''Molden''' is a software package for displaying molecular coordinates, molecular orbitals, electron densities from various quantum chemical software packages and supports contour and 3-D grid plots. Moreover, '''Molden''' supports many output formats including postscript, povray, OpenGL and hpgl.
 

= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/chem/molden
|width=99%
|height=150
|border=0
}}
On the command line interface of any bwHPC cluster, a list of the available versions using
'module avail chem/molden/version'.
<pre>
$ module avail chem/molden
------------------------ /opt/bwhpc/common/modulefiles -------------------------
chem/molden/5.2.1
</pre>
 

= Usage =
== Loading the module ==
You can load the default version of ''Molden'' with the command
<pre>
$ module load chem/molden
</pre>
If you wish to load a specific (older) version of ''Molden'', you can do so using e.g.
<pre>
$ module load chem/molden/5.1
</pre>
to load the version 5.1
 
 

== Software Binaries ==
=== Command-Line ===
Once the module ''Molden'' is loaded, the binaries '''molden''' and '''gmolden''' can be directly executed:
<pre>
$ molden
</pre>
=== Graphical User Interface (GUI) ===
<pre>
$ gmolden &
</pre>
[[File:molden_gui.jpg]]

==== Binary options ====
Available binary options can be taken from the binary help:
<pre>
$ molden -h
-a no automatic cartesian -> zmat conversion
-b use orbitals of first point on opt. runs
-c 0.5 change depth of shading, range [0.0-1.0]
-e DMAREL INPUT: Use est set of parameters
-f PDB: build connectivity from cartesian coordinates
-g PDB: always calculate Helix/Sheet information
-geom XXXxYYY-xxx-yyy
XXX and YYY are the size of the window
xxx and yyy are the position of the window
-h print commandline flags
-hoff switch of hydrogen bonds
-hdmin x mininum hydrogen bond distance (Ang)
-hdmax x maximum hydrogen bond distance (Ang)
-hamin x mininum hydrogen bond angle (Degrees)
-hamax x maximum hydrogen bond angle (Degrees)
-i opt fdat files:
opt=1 standardise H-C, H-N
opt=2 1 + standardise phenyl rings
-j num maximum number of gifs to write
-k num select color of labels (0-15)
(gmolden only).
-l dont display molden logo
-n dont add hydrogens to PDB file
-m turn off the beep sounds
-o fname plotfilename (default=plot)
-p 2.0 change perspective value (def: 13.0)
-r fname read file with per line;
atom color(1-15) VandeWaalsRadius, (- = skip)
background color(1-15)
palette red #CF54FD ... (14 colors)
-s 4.0 scale amplitude of normal vibrations
-t read ascii MOPAC93 Chem3D style
-u With GAMESS-US optimisation output, molden
generates a z-matrix, (def: read from output)
-v print verbose information
-w opt write all points of a movie to a file:
opt specifies format; xyz(=1) zmat(=2,mopac)
VRML2.0(=3)
-x file read in file with spherical atomic densities
-y 1.0 threshold for printing displacement vectors
of normal modes to postscript file
-z create high quality opengl coils
-A Keep order of atoms when creating a Z-matrix
-C Color postscript (default=mono, except PDB)
-D opt DMA mode:
0 = atomic sites only (default)
1 = atomic+halfway-bond sites
2 = no shift of overlap dens. of conn. atoms
-E DMAREL input: use coordinates from multipoles
-F gmolden: Use all opengl code, (line drawing)
-G 0.6 Grid width colour coded ESP potential map
-H GAMESS-US: do normal modes when HSSEND=.TRUE.
-I dont use shaders, if available
-J opt Choose format of screen shot:
1 = GIF (default)
2 = RGB
3 = BMP
-L display both neg. and pos. contour in space
plot of the laplacian
-1 use only the lower half of the cubic grid
used for the space type plot
-2 use only the upper half of the cubic grid
used for the space type plot
-M MonoChrome postscript
-N Check for mpi, to run ambfor/ambmd
in parallel
-O switch off multiple structures handling
-P PDB: treat all input files as PDB files
-Q support for older StarNet xwin32 (ver. 6)
-R npts adjust the gridsize in points
-S start with shade off
-T treat all input files as TINKER xyz files
-U do not use opengl shaders
-X use with XMOL cartesian format input
-V fname VRML density filename
-W Write VRML2.0 instead of VRML1.0
-Z Map the Z-matrix file mapfile onto crystal
mapfile contains Z-matrix followed by keyword
MAP and per line an integer that maps a
Z-matrix line onto a cartesian line
-= Use gamess-us dialect of gaussian zmat writing
</pre>
or from the [http://www.cmbi.ru.nl/molden/command.html website].
 
 

= Molden-Specific Environments =
To see a list of all Molden environments set by the 'module load chem/molden/version'-command,
use 'env | grep MOLDEN'
Or try the command 'module show 'chem/molden/version'.
<pre>
$ module load chem/molden
$ env | grep MOLDEN
MOLDEN_VERSION=5.2.1
MOLDEN_AMB_DIR=/opt/bwhpc/common/chem/molden/5.2.1/bin/ambfor
MOLDEN_BIN_DIR=/opt/bwhpc/common/chem/molden/5.2.1/bin
MOLDEN_UTL_DIR=/opt/bwhpc/common/chem/molden/5.2.1/utils
MOLDEN_SRF_DIR=/opt/bwhpc/common/chem/molden/5.2.1/bin/surf
$
</pre>
 

= Examples =
To visualize the cartesian coordinates of one molecule stored in the file ''molecule.xyz'' without automatic conversion to Z-matrix, execute:
<pre>
$ molden -a molecule.xyz
</pre>
 

= Version-Specific Information =
For specific information about a special version , see the information available via the module system with the command
<pre>
$ module help chem/molden/version
</pre>
 

----
[[Category:Chemistry software]][[Category:bwUniCluster]][[Category:bwForCluster_Chemistry]]

User:M Janczyk/Acknowledgement

2017-11-28T12:31:46Z

T Kienzle:

When preparing a publication describing work that involved the usage of a bwForCluster, e.g. the bwForCluster NEMO, please ensure that you reference the bwHPC initiative, the bwHPC-C5 project and – if appropriate – also the bwHPC facility itself. The following sample text is suggested as a starting point.

Acknowledgement:
The authors acknowledge support by the state of Baden-Württemberg through bwHPC
and the German Research Foundation (DFG) through grant no INST 39/963-1 FUGG.

In addition, we kindly ask you to notify us of any reports, conference papers, journal articles, theses, posters, talks which contain results obtained on any bwHPC resource by sending an email to
[mailto:publications@bwhpc-c5.de publications@bwhpc-c5.de] stating:
* cluster facility (e.g. bwForCluster NEMO)
* RV acronym (e.g. bw16A000)
* author(s)
* title ''or'' booktitle
* journal, volume, pages ''or'' editors, address, publisher
* year.

Such recognition is important for acquiring funding for the next generation hardware, support services, data storage and infrastructure.

The publications will be referenced on the [https://www.bwhpc.de/en/user_publications.php bwHPC website].

[[Category:BwForCluster NEMO]]
[[Category:Acknowledgement]]
[[Category:BwForCluster NEMO|Acknowledgement]]
[[Category:Acknowledgement|bwForCluster NEMO]]
[[Category:Acknowledgment|bwForCluster NEMO]]

BwUniCluster2.0/Software/Ansys

2017-11-28T11:37:43Z

T Kienzle:

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| cae/ansys
|-
| Availability
| [[bwUniCluster]] | bwGRiD-Tübingen
|-
| License
| Academic. See: [http://www.ansys.com/Academic/educator-tools/Licensing+&+Terms+of+Use Licensing and Terms-of-Use].
|-
| Citing
| [http://www.ansys.com/academic/educator-tools/ Citations]
|-
| Links
| [http://www.ansys.com/ Ansys Homepage] | [http://www.ansys.com/Academic/educator-tools/Support+Resources Support and Resources]
|-
| Graphical Interface
| Yes
|}

= Description =
'''ANSYS''' is a general purpose software to simulate interactions of all disciplines of physics, structural, fluid dynamics, heat transfer, electromagnetic etc. For more information about ANSYS products please visit [http://www.ansys.com/Industries/Academic/ http://www.ansys.com/Industries/Academic/]
 
 

= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/cae/ansys
|width=99%
|height=350
|border=0
}}
 
On the command line interface of a particular bwHPC cluster a list of all available ANSYS versions can be inquired as followed
<pre>
$ module avail cae/ansys
-------------------------- /opt/bwhpc/kit/modulefiles --------------------------
cae/ansys/15.0

------------------------ /opt/bwhpc/common/modulefiles -------------------------
cae/ansys/15.0_bw
</pre>
There are two licenses. The module <pre>cae/ansys/15.0_bw</pre> uses the BW license server (25 academic research, 69 parallel processes) and <pre>cae/ansys/15.0</pre> uses the KIT license server (only members of the KIT can use it).
 

= Usage =
== Loading the Module ==
If you wish to load a specific version of ANSYS you can do so by executing e.g.:
<pre>
$ module load cae/ansys/15.0_bw
</pre>
to load the version 15.0 with the BW license.

You can load the default version of ANSYS with the command:
<pre>
$ module load cae/ansys
</pre>

== Start commands ==
To start an ANSYS Mechanical session enter
<pre>
$ ansys150
</pre>
To launch an ANSYS FLUENT session enter
<pre>
$ fluent
</pre>
The following command is to run the ANSYS Workbench
<pre>
$ runwb2
</pre>
Online documention is available from the help menu or by using the command
<pre>
$ anshelp150
</pre>
As with all processes that require more than a few minutes to run, non-trivial ANSYS solver jobs must be submitted to the cluster queueing system.
 

= Examples =
== ANSYS Mechanical batch jobs ==

The following script could be submitted to the queueing system to run an ANSYS Mechanical job in parallel:
{{bwFrameA|
<source lang="bash">
#!/bin/bash
module load cae/ansys
export MPIRUN_OPTIONS="-prot"
export MPI_USESRUN=1
cd Arbeitsverzeichnis
export MACHINES=`/software/bwhpc/common/cae/ansys_inc150/scc/machines.pl`
ansys150 -dis -b -j lal -machines $MACHINES < input.f18
</source>
}}

working_dir could start with $HOME or $WORK .

To submit the example script to the queueing system execute the following (32 cores, 1 GB of memory per core, max. time 600 seconds) :
<pre>
msub -l nodes=2:ppn=16,pmem=1000mb,walltime=600 Shell-Script
</pre>

== ANSYS Fluent batch jobs ==
The following script "run_fluent.sh" could be submitted to the queueing system to run an ANSYS Fluent job in parallel using 4 cores on a single node:
{{bwFrameA|
<source lang="bash">
#!/bin/sh
#MSUB -l nodes=1:ppn=4
#MSUB -l walltime=0:10:00
#MSUB -l mem=16000mb

## setup environment
export MPI_USESRUN=1

## generate hosts list
export run_nodes=$(srun hostname)
echo $run_nodes | sed "s/ /\n/g" > fluent.hosts
echo "" >> fluent.hosts

## load ansys module
module load cae/ansys

## start fluent job
time fluent 3d -rsh=ssh -g -pib -cnf=fluent.hosts -i test.inp
</source>
}}
To submit the example script to the queueing system execute the following:
<pre>
$ msub run_fluent.sh
</pre>
 

== ANSYS CFX batch jobs ==
With the script "run_cfx.sh" you can submit a CFX job to the queueing system to run in parallel using 8 cores on two node with the start-method 'Platform MPI Parallel':
{{bwFrameA|
<source lang="bash">
#!/bin/sh
#MSUB -l nodes=2:ppn=4
#MSUB -l walltime=0:10:00
#MSUB -l mem=32000mb

## setup environment
export MPI_USESRUN=1

## load ansys module
module load cae/ansys

## start job
cfx5solve -def test.def -part 8 -start-method 'Platform MPI Parallel'
</source>
}}
To submit the example script to the queueing system execute:
<pre>
$ msub run_cfx.sh
</pre>
 

----
[[Category:Engineering software]][[Category:bwUniCluster]]

Data Transfer

2017-11-28T11:29:53Z

T Kienzle:

<H1>Using SFTP from Unix client</H1>

'''Example:'''

<pre>
> sftp ka_xy1234@bwfilestorage.lsdf.kit.edu
Connecting to bwfilestorage.lsdf.kit.edu 
ka_xy1234@bwfilestorage.lsdf.kit.edu's password:
sftp> ls
snapshots
temp test
sftp> help
...
sftp> put myfile
sftp> get myfile
</pre>

<H1>Using SFTP from Windows and Mac client</H1>

Windows clients do not have a SCP/SFTP client installed by default, so it needs to be installed before this protocol can be used.

'''Tools for example:'''
* [http://www.openssh.com/ OpenSSH]
*[http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Putty suite] (for Windows and Unix)
*[https://winscp.net/eng/download.php WinSCP] (for Windows)
*[https://filezilla-project.org/download.php?show_all=1 FileZilla] (for Windows, Mac and Linux)
*[http://cygwin.com/install.html Cygwin] (for Windows)
 
'''network drive over SFTP:'''
*[http://www.southrivertechnologies.com/download/downloadwd.html WebDrive] (for Windows and Mac)
*[https://www.eldos.com/sftp-net-drive/comparison.php SFTP Net Drive (ELDOS)] (for Windows)
*[http://www.netdrive.net/ NetDrive] (for Windows)
*[http://www.expandrive.com/expandrive ExpanDrive] (for Windows and Mac)
<hr>
 
 
 
 
[[Category:bwFileStorage|SFTP]]

JUSTUS2/Visualization

2017-11-28T11:18:48Z

T Kienzle:

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| vis/tigervnc
|-
| Availability
| [[bwUniCluster]] | [[BwForCluster_Chemistry]]
|-
| License
| GPL
|-
| Citing
| n/a
|-
| Links
| [http://www.tigervnc.org TigerVNC Homepage]
|-
| Graphical Interface
| No
|}
= Introduction to TigerVNC =
'''TigerVNC''' is a high-performance implementation of VNC (Virtual Network Computing), a client/server application that allows
users to launch and interact with graphical applications on remote machines. It should be faster than standard X11 forwarding and thus can
be used if a graphical software feels slow and has bad responsiveness.
 
 
= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/vis/tigervnc
|width=99%
|height=180
|border=0
}}
 

= Start =
First, you need to start the VNC server. The exact startup mechanism depends on the cluster.
* [[Start VNC Server - bwUniCluster]]
* [[Start VNC Server - bwForCluster Chemistry]]
* [[Start VNC Server - bwForCluster Chemistry - 3D Acceleration]]
 
= Login =
The startup script of the VNC server should print detailed instructions on how to establish the connection to the VNC server from your local computer. They depend on whether you
use Windows or Linux and if you work with TurboVNC Java Viewer which is a tool that can simplify the process a little bit but needs the Java Development Kit (JDK) to run. Therefore the next steps are
divided into 3 cases. Each command should be issued on the local computer.

* '''Login with TurboVNC Java Viewer''' Needed Software: [http://turbovnc.org/ TurboVNC 2.0], JDK Open TurboVNC Java Viewer. Go to Options... -> Security -> Gateway and fill in the parameters provided by the ''run_vncserver'' script. You can save these settings in the "Global" tab if you want to. Now you can click "OK", supply the VNC server and connect to the server. You should be prompted for your ssh password and your VNC password and after that the connection is established.

* '''Login without TurboVNC Java Viewer for Linux users''' Needed Software: A VNC viewer such as tigervnc or turbovnc A tunnel must be created with the ssh command given by the ''run_vncserver'' script. Open a new terminal, start a VNC viewer and connect to localhost:n, where n is the display number printed by ''run_vncserver'', using a command such like this <pre>$ vncviewer localhost:1</pre>

* '''Login without TurboVNC Java Viewer for Windows users''' Needed Software: [https://sourceforge.net/projects/tigervnc/files/tigervnc/1.3.0 tigervnc], [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Putty] You need to start Putty and go to Connection -> SSH -> Tunnels. Fill in the parameters provided by ''run_vncserver''. After you clicked "Add" you must navigate to Session and connect to the bwUniCluster with your username and password. Once the connection is established start the tigervnc client and connect to localhost:n where n is the display number printed by ''run_vncserver''.
 

= Shutdown =
To exit your VNC session it is not sufficient to only close the window of the viewer, because this will not terminate the VNC server. The server will keep running and you will run into problems
when you try to start a new VNC session later on. Please use the "log out" function of the desktop environment inside the VNC session, this will terminate the server properly.
[[Category:visualization]][[Category:bwUniCluster]]
 
 
= Starting OpenGL programs =
Starting an OpenGL program in a 3D accelerated VNC session results in an error message. In order to fix this issue and redirect OpenGL commands to the graphics card on the cluster you have to use the vglrun command like in the following example:
$ vglrun glxgears -info

Environment Modules

2017-11-28T08:12:19Z

T Kienzle:

{| width=650px class="wikitable"
|-
! Description !! Content
|-
| module load
| category/name | category/name/version (optional)
|-
| Links
| [http://modules.sourceforge.net/ Environment Modules Project] | [https://sourceforge.net/projects/modules/ Environment Modules]
|-
| License
| [http://www.gnu.org/licenses/old-licenses/gpl-2.0.html GNU General Public License]
|}
 
= Introduction =
'''Environment Modules''', or short '''Modules''' are the means by which most of the installed scientific software is provided on the bwHPC clusters.
 
The use of different compilers, libraries and software packages requires users to set up a specific session environment suited for the program they want to run. The bwHPC clusters provide users with the possibility to load and unload complete environments for compilers, libraries and software packages by a single command.
 
 
= Description =
The Environment ''Modules'' package enables dynamic modification of your environment by the
use of so-called ''modulefiles''. A ''modulefile'' contains information to configure the shell
for a program/software . Typically, a modulefile contains instructions that alter or set shell
environment variables, such as PATH and MANPATH, to enable access to various installed
software.
 
One of the key features of using the Environment ''Modules'' software is to allow multiple versions of the same software to be used in your environment in a controlled manner.
For example, two different versions of the Intel C compiler can be installed on the system at the same time - the version used is based upon which Intel C compiler modulefile is loaded.
 
The software stack of bwHPC clusters provides a number of modulefiles. You can also
create your own modulefiles. ''Modulefiles'' may be shared by many users on a system, and
users may have their own collection of modulefiles to supplement or replace the shared
modulefiles.
 
A modulefile does not provide configuration of your environment until it is explicitly loaded,
i.e., the specific modulefile for a software product or application must be loaded in your environment before the configuration information in the modulefile is effective.
 
For instance loading the default Intel C and Fortran compiler you must execute
''''module load compiler/intel''''.
 
<pre>
$ module load compiler/intel
$ module list
Currently Loaded Modulefiles:
1) compiler/pgi/12.10(default) 2) compiler/intel/15.0(default)
$ : Display all Intel related environments now
$ env | grep INTEL
INTEL_LICENSE_FILE=/opt/bwhpc/common/compiler/intel/compxe.2015.3.187/composer_xe_2015.3.187/licenses
INTEL_LIB_MICMPI=/opt/bwhpc/common/compiler/intel/compxe.2015.3.187/composer_xe_2015.3.187/mpirt/lib/mic
INTEL_HOME=/opt/bwhpc/common/compiler/intel/compxe.2015.3.187/composer_xe_2015.3.187
INTEL_VERSION=15.0.3
INTEL_MAN_DIR=/opt/bwhpc/common/compiler/intel/compxe.2015.3.187/composer_xe_2015.3.187/man/en_US
INTEL_INC_DIR=/opt/bwhpc/common/compiler/intel/compxe.2015.3.187/composer_xe_2015.3.187/compiler/include
INTEL_BIN_DIR=/opt/bwhpc/common/compiler/intel/compxe.2015.3.187/composer_xe_2015.3.187/bin/intel64
INTEL_DOC_DIR=/opt/bwhpc/common/compiler/intel/compxe.2015.3.187/composer_xe_2015.3.187/Documentation/en_US
INTEL_LIB_DIR=/opt/bwhpc/common/compiler/intel/compxe.2015.3.187/composer_xe_2015.3.187/compiler/lib/intel64
INTEL_LIB_MIC=/opt/bwhpc/common/compiler/intel/compxe.2015.3.187/composer_xe_2015.3.187/compiler/lib/mic
INTEL_PYTHONHOME=/opt/bwhpc/common/compiler/intel/compxe.2015.3.187/composer_xe_2015.3.187/debugger/python/intel64
</pre>
 

= Usage =
== Documentation ==
For help on how to use ''Modules'' software, i.e., the command '''module''',
execute ''''module help''''.
<pre>
$ module help
Modules Release 3.2.10 2012-12-21 (Copyright GNU GPL v2 1991):

Usage: module [ switches ] [ subcommand ] [subcommand-args ]

Switches:
-H|--help this usage info
-V|--version modules version & configuration options
-f|--force force active dependency resolution
-t|--terse terse format avail and list format
-l|--long long format avail and list format
-h|--human readable format avail and list format
-v|--verbose enable verbose messages
-s|--silent disable verbose messages
-c|--create create caches for avail and apropos
-i|--icase case insensitive
-u|--userlvl <lvl> set user level to (nov[ice],exp[ert],adv[anced])
Available SubCommands and Args:
+ add|load modulefile [modulefile ...]
+ rm|unload modulefile [modulefile ...]
+ switch|swap [modulefile1] modulefile2
+ display|show modulefile [modulefile ...]
+ avail [modulefile [modulefile ...]]
+ use [-a|--append] dir [dir ...]
+ unuse dir [dir ...]
+ update
+ refresh
+ purge
+ list
+ clear
+ help [modulefile [modulefile ...]]
+ whatis [modulefile [modulefile ...]]
+ apropos|keyword string
+ initadd modulefile [modulefile ...]
+ initprepend modulefile [modulefile ...]
+ initrm modulefile [modulefile ...]
+ initswitch modulefile1 modulefile2
+ initlist
+ initclear
</pre>
or ''''man module''''.
<pre>
MODULE(1) Modules package MODULE(1)

NAME
module - command interface to the Modules package

SYNOPSIS
module [ switches ] [ sub-command ] [ sub-command-args ]

DESCRIPTION
module is a user interface to the Modules package. The Modules package
provides for the dynamic modification of the user's environment via
modulefiles.

Each modulefile contains the information needed to configure the shel
[...]
</pre>
For help on particular version of ''Module'', e.g. Intel compiler version X.Y, execute
''''module help compiler/intel''''.
<pre>
$ module help compiler/intel
----------- Module Specific Help for 'compiler/intel/15.0' --------
This module provides the Intel(R) compiler suite version 15.0.3 via
commands 'icc', 'icpc' and 'ifort' (version 15.0.3), the debugger 'gdb-ia' (version
7.8.3) as well as the Intel(R) Threading Building Blocks TBB (version 4.3.5)
and the Integrated Performance Primitives IPP libraries (version 8.2.2)
(for details see also 'http://software.intel.com/en-us/intel-compilers/').

The related Math Kernel Library MKL module is 'numlib/mkl/11.2.3'.
The related Intel MPI module is 'mpi/impi/5.0.3-intel-15.0'.
The Intel 'icpc' should work well with GNU compiler version 4.4 to 4.8.
Before using TBB or IPP setup the corresponding environment, e.g. for 64bit+bash
source $INTEL_HOME/tbb/bin/tbbvars.sh intel64
source $INTEL_HOME/ipp/bin/ippvars.sh intel64

Commands:
icc # Intel(R) C compiler
icpc # Intel(R) C++ compiler
ifort # Intel(R) Fortran compiler
gdb-ia # Intel version of GNU debugger
# idb is not available anymore in Intel compiler suite 2015.

Local documentation:
Man pages: man icc; man icpc; man ifort; man gdb-ia
firefox $INTEL_DOC_DIR/beginusing_lc.htm
firefox $INTEL_DOC_DIR/beginusing_lf.htm
The html-pages are very detailed and cover TBB and IPP as well as MKL.

For some Intel(R) compiler option examples, hints on how to compile 32bit code
and solutions for less common problems see the tips and troubleshooting doc:
$INTEL_DOC_DIR/intel-compiler-tips-and-troubleshooting.txt

For details on library and include dirs please call
module show compiler/intel/15.0
[...]
</pre>
=== Online Documentation ===
[https://sourceforge.net/p/modules/wiki/FAQ/ Frequently Asked Questions (FAQ)]

== Display all available Modules ==
Available ''Module'' are modulefiles that can be loaded by the user. A ''Module'' must be loaded before it provides changes to your environment, as described in the introduction to this
section. You can display all available ''Modules'' on the system by executing:
<pre>
$ module avail
</pre>
The short form the command is:
<pre>
$ module av
</pre>
Available ''Modules'' can be also displayed in different modes, such as
* each ''Module'' per one line
<pre>
$ module -t avail
</pre>
* long
<pre>
$ module -l avail
</pre>
== bwHPC CLuster Information System (CIS) ==
A '''GUI-Version of all available and scheduled modules''' is available with our
 
'''CIS''' (Cluster Information System).
 
Software Admins are able to '''announce new modules''', versions and complete new software, too.
 
Academic users and other interested parties can get a '''summay of all installed modules and module-help informations'''.
 
<big>
[https://cis-hpc.uni-konstanz.de CIS: Cluster Information System]
</big>
 
[[File: cis.jpg]]
 

== Module categories, versions and defaults ==
The bwHPC clusters (such as [[bwUniCluster]]) traditionally provide a large variety of
software and software versions. Therefore ''Module'' are divided in category folders
containing subfolders of modulefiles again containing modulefile versions, and must be addressed
as follows:
category/softwarename/version
For instance the Intel compiler X.Y belongs to the category of compilers, therefore the
modulefile ''X.Y'' is placed under the category ''compiler'' and ''intel''.
 
In case of multiple software versions, one version will be always defined as the '''default'''
version. The ''Module'' of the default can be addressed by simply omitting the version number:
category/softwarename
== Finding software Modules ==
Currently all bwHPC software packages are assigned to the following ''Module'' categories:

* [[:Category:Biology_software|bio]]
* [[:Category:Engineering_software|cae]]
* [[:Category:Chemistry_software|chem]]
* [[:Category:Compiler_software|compiler]]
* [[:Category:Debugger_software|devel]]
* [[BwHPC_BPG_for_Mathematics|math]]
* mpi
* [[:Category:Numerical libraries|numlib]]
* [[:Category:Physics software|phys]]
* [[:Category:System software|system]]
* [[:Category:Visualization|vis]]
You can selectively list software in one of those categories using, e.g. for the category "compiler"
<pre>
$ module avail compiler/
</pre>
Searches are looking for a substring starting at the begin of the name, so this would list all software in categories starting with a "c"
<pre>
$ module avail c
</pre>
while this would find nothing
<pre>
$ module avail hem
</pre>
== Loading Modules ==
You can load a ''Module'' software in to your environment to enable easier access to software that
you want to use by executing:
<pre>
$ module load category/softwarename/version
</pre>
or
<pre>
$ module add category/softwarename/version
</pre>
Loading a ''Module'' in this manner affects ONLY your environment for the current session.
=== Loading conflicts ===
By default you can not load different versions of same software ''Module'' in same session. Loading for example Intel compiler version X while Intel compiler version Y is loaded results in error message as follows:
<source lang="bash">
Module 'compiler/intel/X' conflicts with the currently loaded module(s) 'compiler/intel/Y'
</source>
The solution is [[#Unloading Modules|unloading]] or switching ''Modules''.

=== Showing the changes introduced by a Module ===
Loading a ''Module'' will change the environment of the current shell session. For instance the $PATH variable will be expanded by the software's binary directory. Other ''Module'' variables may even change the behavior of the current shell session or the software program(s) in a more drastic way.
 
Loaded ''Modules'' may also invoke an additional set of environment variables, which e.g. point to directories or destinations of documentation and examples. Their nomenclature is systematic:
{| width=600px class="wikitable"
|-
! Variable
! Pointing to
|-
| $SWN_HOME
| Root directory of the software package
|-
| $SWN_DOC_DIR
| Documentation
|-
| $SWN_EXA_DIR
| Examples
|-
| $SWN_BPR_URL
| URL of software's Wiki article
|-
| and many many more...
|  
|}
with SWN being the place holder of the software ''Module'' name.
 
All the changes to the current shell session to be invoked by loading the ''Module'' can be reviewed using ''''module show category/softwarename/version''''.
 
Example (Intel compiler)
<pre>
$ module show compiler/intel/14.0
-------------------------------------------------------------------
/opt/bwhpc/common/modulefiles/compiler/intel/14.0:
module-whatis Intel(R) compiler suite (icc, icpc, ifort), debugger (idb), IPP and TBB ver 14.0.4
setenv INTEL_VERSION 14.0.4
setenv INTEL_HOME /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211
setenv INTEL_BIN_DIR /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/bin/intel64
setenv INTEL_LIB_DIR /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/compiler/lib/intel64
setenv INTEL_LIB_MIC /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/compiler/lib/mic
setenv INTEL_INC_DIR /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/compiler/include
setenv INTEL_MAN_DIR /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/man/en_US
setenv INTEL_DOC_DIR /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/Documentation/en_US
setenv ICC_VERSION 14.0.4
setenv ICC_HOME /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211
setenv ICC_BIN_DIR /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/bin/intel64
setenv ICC_LIB_DIR /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/compiler/lib/intel64
setenv ICC_INC_DIR /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/compiler/include
setenv ICC_MAN_DIR /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/man/en_US
setenv ICC_DOC_DIR /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/Documentation/en_US
setenv IFORT_VERSION 14.0.4
setenv IFORT_HOME /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211
setenv IFORT_BIN_DIR /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/bin/intel64
setenv IFORT_LIB_DIR /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/compiler/lib/intel64
setenv IFORT_INC_DIR /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/compiler/include
setenv IFORT_MAN_DIR /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/man/en_US
setenv IFORT_DOC_DIR /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/Documentation/en_US
setenv IDB_VERSION 14.0.4
setenv IDB_HOME /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211
setenv IDB_LIB_DIR /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/debugger/lib/intel64
setenv LANGUAGE_TERRITORY en_US
prepend-path PATH /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/bin/intel64
prepend-path LD_LIBRARY_PATH /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/compiler/lib/intel64
prepend-path LD_RUN_PATH /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/compiler/lib/intel64
prepend-path MIC_LD_LIBRARY_PATH /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/compiler/lib/mic
prepend-path LIBRARY_PATH /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/compiler/lib/intel64
prepend-path MANPATH /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/man/en_US
prepend-path NLSPATH /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/compiler/lib/intel64/locale/%l_%t/%N
prepend-path LD_LIBRARY_PATH /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/debugger/lib/intel64
prepend-path NLSPATH /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/debugger/intel64/locale/%l_%t/%N
prepend-path INTEL_LICENSE_FILE /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composer_xe_2013_sp1.4.211/licenses
setenv IDB_JAVA_ARGUMENTS -Xms512m -Xmx1024m
setenv CC icc
setenv CXX icpc
setenv F77 ifort
setenv FC ifort
setenv F90 ifort
setenv TEST_MODULE_SCRIPT /opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/install-doc/test-compiler-intel.sh
setenv TEST_MODULE_NAME compiler/intel/14.0
conflict compiler/intel
conflict compiler/gnu/4.9
-------------------------------------------------------------------
</pre>
'module show' does '''not''' load the ''Module''!

=== Modules depending on Modules ===
Some program ''Modules'' depend on libraries to be loaded to the user environment. Therefore the
corresponding ''Modules'' of the software must be loaded together with the ''Modules'' of
the libraries.
 
By default such software ''Modules'' try to load required ''Modules'' and corresponding versions automatically. However, automatic loading might fail if a different version of that required ''Module''
is already loaded (cf. [[#Loading conflicts|Loading conflicts]]).
 
== Unloading Modules ==
To unload or to remove a software ''Module'' execute:
<pre>
$ module unload category/softwarename/version
</pre>
or
<pre>
$ module remove category/softwarename/version
</pre>
=== Unloading all loaded modules ===
==== Purge ====
Unloading a ''Module'' that has been loaded by default makes it inactive for the current session only - it will be reloaded the next time you log in.
 
In order to remove all previously loaded software modules from your environment issue the command 'module purge'.
 
Example
<pre>
$ module list
Currently Loaded Modulefiles:
1) devel/gdb/7.7
2) compiler/intel/14.0
3) mpi/openmpi/1.8-intel-14.0(default)
$
$ module purge
$ module list
No Modulefiles Currently Loaded.
$
</pre>
Beware!
 
'module purge' is working without any further inquiry.

==== Clear ====
Use ''''module clear'''' and confirm with "'''y'''" to unload all loaded module, too.
 
Example
<pre>
$ module list
Currently Loaded Modulefiles:
1) compiler/intel/14.0
2) mpi/openmpi/1.8-intel-14.0(default)
3) devel/gdb/7.7
$
$ module clear
Are you sure you want to clear all loaded modules!? [n] y
$
$ module list
No Modulefiles Currently Loaded.
</pre>

== Display your loaded Modules ==
All ''Modules'' that are currently loaded for you can be displayed by the
command ''''module list''''. [[#Purge|See example above]].
 
Note: You only have to load further ''Modules'', if you want to use additional software
packages or to change the version of an already loaded software.
 
 

= Software job examples =
The ''Modules'' installed on bwHPC systems provide job examples to help you get started using the software or submitting jobs with this software. Examples can be found via a convenient
variable $SWN_EXA_DIR (for a ''Module'' called '''SWN'''). It is advisable to copy the whole example folder to your $HOME directory, so you can edit those job examples.
 
For copying the entire job examples folder of software '''swn''' to your working directory, execute:
<pre>
$ module load catogory/softwarename
$ cp -R $SWN_EXA_DIR .
</pre>
 
= How do Modules work? =
The default shell on the bwHPC clusters is bash, so explanations and examples will be shown for bash. In general, programs cannot modify the environment of the shell they are being run from, so how can the module command do exactly that?
 
The module command is not a program, but a bash-function.
You can view its content using
<pre>
$ type module
</pre>
and you will get a result like this:
<pre>
$ type module
module is a function
module ()
{
eval `/usr/bin/modulecmd bash $*`
}
</pre>
In this function, modulecmd is called. Its output to stdout is then executed inside your current shell using the bash-internal ''eval'' command. As a consequence, all output that you see from the module is transmitted via stderr (output handle 2) or in some cases even stdin (output handle 0).
 
 
----
[[Category:System software]][[Category:bwUniCluster|Environment Modules]][[Category:ForHLR Phase I|Environment Modules]]

Development/MKL

2017-11-28T08:11:31Z

T Kienzle:

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| numlib/mkl
|-
| License
| Commercial. See [https://software.intel.com/en-us/articles/end-user-license-agreement EULA].
|-
| Citing
| n/a
|-
| Links
| [https://software.intel.com/en-us/intel-mkl Intel MKL Homepage] | [https://software.intel.com/en-us/articles/intel-math-kernel-library-documentation Online-Documentation]
|-
| Graphical Interface
| No
|}

= Description =
'''Intel MKL (Math Kernel Library)''' is a library of optimized math routines for numerical computations such as linear algebra (using BLAS, LAPACK, ScaLAPACK) and discrete Fourier Transformation.
With its standard interface in matrix computation and the interface of the popular fast Fourier transformation library fftw, MKL can be used to replace other libraries with minimal code changes. In fact a program which uses FFTW without MPI doesn't need to be changed at all. Just recompile it with the MKL linker flags.
 
* [http://software.intel.com/en-us/articles/intel-math-kernel-library-documentation Online-Documentation]

= Versions and Availability =
A list of currently available MKL modules can be obtained from the
 <big>
[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS] </big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/numlib/mkl
|width=99%
|height=700
|border=0
}}
 
Show a list of available versions using 'module avail numlib/mkl' on any HPC-C5 cluster.
<pre>
: EXAMPLE bwUniCluster
$ module avail numlib/mkl
------------------------------ /opt/bwhpc/common/modulefiles ------------------------------
numlib/mkl/10.3.12 numlib/mkl/11.1.4(default)
numlib/mkl/11.0.5 numlib/mkl/11.2.3
</pre>

= Local documentation =
There is some information in the module help file accessible via 'module help numlib/mkl'-
command.
<pre>
: EXCERPT ONLY
$ module help numlib/mkl
----------- Module Specific Help for 'numlib/mkl/11.1.4' ----------
This module provides the Intel(R) Math Kernel Library (MKL)
version 11.1.4, a fast and reliable implementation
of BLAS/LAPACK/FFTW (see also 'http://software.intel.com/en-us/intel-mkl/').

The preferable compiler for this MKL version is 'compiler/intel/14.0'. Linking
with other compilers like GNU, PGI and SUN is possible. The desired compiler
module (exception system GNU compiler) has to be loaded before using MKL.

Local documentation:

Man pages in '$MKL_MAN_DIR/man3', e.g. 'man dotc'.
firefox $MKL_DOC_DIR/mkl_documentation.htm
acroread $MKL_DOC_DIR/l_mkl_11.1.4.211.mklman.pdf
acroread $MKL_DOC_DIR/l_mkl_11.1.4.211.mkl_11.1.4_lnx_userguide.pdf

Linking examples (ifort compiler with support for blas and lapack):

* Dynamic linking of myprog.f and parallel MKL supporting the LP64 interface:

ifort myprog.f -L${MKL_LIB_DIR} -I${MKL_INC_DIR} \
-lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread
[... t.b.c. ...]
</pre>
After loading the module, the environment variable $MKL_DOC_DIR points to the local documentation folder. Various examples can be found in $MKLROOT/examples.

= MKL-Specific Environments =
To see a list of all MKL environments set by the 'module load'-command use 'env | grep MKL'. Or use the command 'module display numlib/mkl/version'.
 
Example (bwUniCluster)
<pre>
$ module load numlib/mkl
$ env | grep MKL
MKLROOT=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/mkl
MKL_LIB_MIC_COM=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/lib/mic
MKL_DOC_DIR=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composerxe/Documentation/en_US/mkl
MKL_NUM_THREADS=1
MKL_HOME=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/mkl
MKL_LIB_MIC=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/mkl/lib/mic
MKL_MAN_DIR=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/man/en_US
MKL_EXA_DIR=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composerxe/Samples/en_US
MKL_STA_DIR=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/mkl/lib/intel64_static
MKL_INC_DIR=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/mkl/include
MKL_BIN_DIR=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/mkl/bin
MKL_LIB_DIR=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/mkl/lib/intel64
MKL_VERSION=11.1.4
MKL_LIB_COM=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/lib/intel64
</pre>

= Compiling and linking =
Compilation is possible with both GCC and Intel compilers but it is easier for Intel compilers, so this case is explained here.
After loading the compiler and the library module with
<pre>
$ module load compiler/intel
$ module load numlib/mkl
</pre>
you can include the MKL header file in your program:
<source lang=cpp>#include <mkl.h></source>
Compilation is simple:
<pre>$ icpc -c example_mkl.c</pre>
When linking the program you have to tell the compiler to link against the mkl library:
<pre>$ icpc example_mkl.o -mkl</pre>
With the -mkl switch the intel compiler automatically sets the correct linker flags but you can specify them explicitly for example to enable static linking or when non-intel compilers are used. Information about the different options can be found at https://software.intel.com/en-us/node/438568 and especially helpful is the MKL link line advisor at https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor.
By default $MKL_NUM_THREADS is set to 1 and so only one thread will be created, but if you feel the need to run the computation on more cores (after benchmarking) you can set $MKL_NUM_THREADS to a higher number.
 
 
== FFTW Interface to Intel Math Kernel Library (MKL) ==
Sometimes, [[FFTW|FFTW]] is not available on your cluster. You can use the MKL library
instead and include the FFTW functions, too.
 
Intel Math Kernel Library (MKL) offers FFTW2 and FFTW3 interfaces to Intel MKL Fast Fourier Transform and Trigonometric Transform functionality. The purpose of these interfaces is to enable applications using FFTW to gain performance with Intel MKL without changing the program source code.
 
Here is an excerpt from 'module help numlib/mkl':
<pre>
Static FFTW2/3 C/Fortran interfaces can be found in dir
${MKL_HOME}/interfaces/
Examples:
Link to FFTW3 Fortran interface with GNU compiler and ilp64 support:
${MKL_HOME}/interfaces/fftw3xf/libfftw3xf_intel64_double_i8_gnu47.a
Link to FFTW3 Fortran interface with Intel compiler and lp64 support:
${MKL_HOME}/interfaces/fftw3xf/libfftw3xf_intel64_double_i4_intel150.a
The Intel FFTW interfaces requires the Intel MKL library (e.g. it does
not work with ACML library). Usually it is not a problem to use a
different compiler version, e.g. to use _gnu41.a with gnu 4.3 compiler.
See dir ${MKL_HOME}/interfaces/ for other interfaces (fftw2/3 Fortran/C).
Compiler option for include files: -I${MKL_INC_DIR}/fftw
</pre>
See the corresponding webpages:
* [https://software.intel.com/en-us/node/471410 FFTW Interface to Intel Math Kernel Library]
* [https://software.intel.com/de-de/node/471414 FFTW2 Interface to Intel Math Kernel Library]
* [https://software.intel.com/en-us/node/471456 FFTW3 Interface to Intel Math Kernel Library]
 

= Examples =
To help getting started we provide two C++ examples. The first one computes the square of a 2x2 matrix:
<source lang="cpp">
#include <iostream>
#include <mkl.h>
using namespace std;

int main()
{
double m[2][2] = {{2,1}, {0,2}};
double c[2][2];

for(int i = 0; i < 2; ++i)
{
for(int j = 0; j < 2; ++j)
cout << m[i][j] << " ";

cout << endl;
}

cblas_dgemm(CblasRowMajor, CblasNoTrans, CblasNoTrans, 2, 2, 2, 1.0, &m[0][0], 2, &m[0][0], 2, 0.0, &c[0][0], 2);

cout << endl;

for(int i = 0; i < 2; ++i)
{
for(int j = 0; j < 2; ++j)
cout << c[i][j] << " ";

cout << endl;
}

return 0;
}
</source>
And the second one does a fast Fourier transformation using the Intel MKL interface (DFTI):
<source lang="cpp">
#include <iostream>
#include <complex>
#include <cmath>
#include <mkl.h>
using namespace std;

int main()
{
const int N = 3;
complex<double> x[N] = {2, -1, 0.5};

cout << "Input: " << endl;

for(int i = 0; i < N; i++)
cout << x[i] << endl;

DFTI_DESCRIPTOR_HANDLE desc;

DftiCreateDescriptor(&desc, DFTI_DOUBLE, DFTI_COMPLEX, 1, N);
DftiCommitDescriptor(desc);
DftiComputeForward(desc, x);
DftiFreeDescriptor(&desc);

cout << "\nOutput: " << endl;

for(int i = 0; i < N; i++)
cout << x[i] << endl;

cout << "\nTest the interpolation function f:" << endl;

for(int i = 0; i < N; i++)
{
double t = i/(double)N;
complex<double> u(0, 2*M_PI*t);
complex<double> z = exp(u);
complex<double> w = 1.0/N * (x[0] + x[1]*z + x[2]*z*z);

cout << "f(" << t << ") = " << w << endl;
}

return 0;
}
</source>

[[Category:Numerical_libraries]][[Category:bwUniCluster]][[Category:bwForCluster_Chemistry]][[Category:BwForCluster_BinAC]][[Category:bwForCluster_MLS&WISO_Production]]

Development/MKL

2017-11-28T07:53:55Z

T Kienzle:

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| numlib/mkl
|-
| License
| Commercial. See [https://software.intel.com/en-us/articles/end-user-license-agreement EULA].
|-
| Citing
| n/a
|-
| Links
| [https://software.intel.com/en-us/intel-mkl Intel MKL Homepage] | [https://software.intel.com/en-us/articles/intel-math-kernel-library-documentation Online-Documentation]
|-
| Graphical Interface
| No
|}

= Description =
'''Intel MKL (Math Kernel Library)''' is a library of optimized math routines for numerical computations such as linear algebra (using BLAS, LAPACK, ScaLAPACK) and discrete Fourier Transformation.
With its standard interface in matrix computation and the interface of the popular fast Fourier transformation library fftw, MKL can be used to replace other libraries with minimal code changes. In fact a program which uses FFTW without MPI doesn't need to be changed at all. Just recompile it with the MKL linker flags.
 
* [http://software.intel.com/en-us/articles/intel-math-kernel-library-documentation Online-Documentation]

= Versions and Availability =
A list of currently available MKL modules can be obtained from the
 <big>
[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS] </big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/numlib/mkl
|width=99%
|height=700
|border=0
}}
 
Show a list of available versions using 'module avail numlib/mkl' on any HPC-C5 cluster.
<pre>
: EXAMPLE bwUniCluster
$ module avail numlib/mkl
------------------------------ /opt/bwhpc/common/modulefiles ------------------------------
numlib/mkl/10.3.12 numlib/mkl/11.1.4(default)
numlib/mkl/11.0.5 numlib/mkl/11.2.3
</pre>

= Local documentation =
There is some information in the module help file accessible via 'module help numlib/mkl'-
command.
<pre>
: EXCERPT ONLY
$ module help numlib/mkl
----------- Module Specific Help for 'numlib/mkl/11.1.4' ----------
This module provides the Intel(R) Math Kernel Library (MKL)
version 11.1.4, a fast and reliable implementation
of BLAS/LAPACK/FFTW (see also 'http://software.intel.com/en-us/intel-mkl/').

The preferable compiler for this MKL version is 'compiler/intel/14.0'. Linking
with other compilers like GNU, PGI and SUN is possible. The desired compiler
module (exception system GNU compiler) has to be loaded before using MKL.

Local documentation:

Man pages in '$MKL_MAN_DIR/man3', e.g. 'man dotc'.
firefox $MKL_DOC_DIR/mkl_documentation.htm
acroread $MKL_DOC_DIR/l_mkl_11.1.4.211.mklman.pdf
acroread $MKL_DOC_DIR/l_mkl_11.1.4.211.mkl_11.1.4_lnx_userguide.pdf

Linking examples (ifort compiler with support for blas and lapack):

* Dynamic linking of myprog.f and parallel MKL supporting the LP64 interface:

ifort myprog.f -L${MKL_LIB_DIR} -I${MKL_INC_DIR} \
-lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread
[... t.b.c. ...]
</pre>
After loading the module, the environment variable $MKL_DOC_DIR points to the local documentation folder. Various examples can be found in $MKLROOT/examples.

= MKL-Specific Environments =
To see a list of all MKL environments set by the 'module load'-command use 'env | grep MKL'. Or use the command 'module display numlib/mkl/version'.
 
Example (bwUniCluster)
<pre>
$ module load numlib/mkl
$ env | grep MKL
MKLROOT=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/mkl
MKL_LIB_MIC_COM=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/lib/mic
MKL_DOC_DIR=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composerxe/Documentation/en_US/mkl
MKL_NUM_THREADS=1
MKL_HOME=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/mkl
MKL_LIB_MIC=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/mkl/lib/mic
MKL_MAN_DIR=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/man/en_US
MKL_EXA_DIR=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composerxe/Samples/en_US
MKL_STA_DIR=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/mkl/lib/intel64_static
MKL_INC_DIR=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/mkl/include
MKL_BIN_DIR=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/mkl/bin
MKL_LIB_DIR=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/mkl/lib/intel64
MKL_VERSION=11.1.4
MKL_LIB_COM=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/lib/intel64
</pre>

= Compiling and linking =
Compilation is possible with both GCC and Intel compilers but it is easier for Intel compilers, so this case is explained here.
After loading the compiler and the library module with
<pre>
$ module load compiler/intel
$ module load numlib/mkl
</pre>
you can include the MKL header file in your program:
<source lang=cpp>#include <mkl.h></source>
Compilation is simple:
<pre>$ icpc -c example_mkl.c</pre>
When linking the program you have to tell the compiler to link against the mkl library:
<pre>$ icpc example_mkl.o -mkl</pre>
With the -mkl switch the intel compiler automatically sets the correct linker flags but you can specify them explicitly for example to enable static linking or when non-intel compilers are used. Information about the different options can be found at http://software.intel.com/en-us/node/438568 and especially helpful is the MKL link line advisor at https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor.
By default $MKL_NUM_THREADS is set to 1 and so only one thread will be created, but if you feel the need to run the computation on more cores (after benchmarking) you can set $MKL_NUM_THREADS to a higher number.
 
 
== FFTW Interface to Intel Math Kernel Library (MKL) ==
Sometimes, [[FFTW|FFTW]] is not available on your cluster. You can use the MKL library
instead and include the FFTW functions, too.
 
Intel Math Kernel Library (MKL) offers FFTW2 and FFTW3 interfaces to Intel MKL Fast Fourier Transform and Trigonometric Transform functionality. The purpose of these interfaces is to enable applications using FFTW to gain performance with Intel MKL without changing the program source code.
 
Here is an excerpt from 'module help numlib/mkl':
<pre>
Static FFTW2/3 C/Fortran interfaces can be found in dir
${MKL_HOME}/interfaces/
Examples:
Link to FFTW3 Fortran interface with GNU compiler and ilp64 support:
${MKL_HOME}/interfaces/fftw3xf/libfftw3xf_intel64_double_i8_gnu47.a
Link to FFTW3 Fortran interface with Intel compiler and lp64 support:
${MKL_HOME}/interfaces/fftw3xf/libfftw3xf_intel64_double_i4_intel150.a
The Intel FFTW interfaces requires the Intel MKL library (e.g. it does
not work with ACML library). Usually it is not a problem to use a
different compiler version, e.g. to use _gnu41.a with gnu 4.3 compiler.
See dir ${MKL_HOME}/interfaces/ for other interfaces (fftw2/3 Fortran/C).
Compiler option for include files: -I${MKL_INC_DIR}/fftw
</pre>
See the corresponding webpages:
* [https://software.intel.com/en-us/node/471410 FFTW Interface to Intel Math Kernel Library]
* [https://software.intel.com/de-de/node/471414 FFTW2 Interface to Intel Math Kernel Library]
* [https://software.intel.com/en-us/node/471456 FFTW3 Interface to Intel Math Kernel Library]
 

= Examples =
To help getting started we provide two C++ examples. The first one computes the square of a 2x2 matrix:
<source lang="cpp">
#include <iostream>
#include <mkl.h>
using namespace std;

int main()
{
double m[2][2] = {{2,1}, {0,2}};
double c[2][2];

for(int i = 0; i < 2; ++i)
{
for(int j = 0; j < 2; ++j)
cout << m[i][j] << " ";

cout << endl;
}

cblas_dgemm(CblasRowMajor, CblasNoTrans, CblasNoTrans, 2, 2, 2, 1.0, &m[0][0], 2, &m[0][0], 2, 0.0, &c[0][0], 2);

cout << endl;

for(int i = 0; i < 2; ++i)
{
for(int j = 0; j < 2; ++j)
cout << c[i][j] << " ";

cout << endl;
}

return 0;
}
</source>
And the second one does a fast Fourier transformation using the Intel MKL interface (DFTI):
<source lang="cpp">
#include <iostream>
#include <complex>
#include <cmath>
#include <mkl.h>
using namespace std;

int main()
{
const int N = 3;
complex<double> x[N] = {2, -1, 0.5};

cout << "Input: " << endl;

for(int i = 0; i < N; i++)
cout << x[i] << endl;

DFTI_DESCRIPTOR_HANDLE desc;

DftiCreateDescriptor(&desc, DFTI_DOUBLE, DFTI_COMPLEX, 1, N);
DftiCommitDescriptor(desc);
DftiComputeForward(desc, x);
DftiFreeDescriptor(&desc);

cout << "\nOutput: " << endl;

for(int i = 0; i < N; i++)
cout << x[i] << endl;

cout << "\nTest the interpolation function f:" << endl;

for(int i = 0; i < N; i++)
{
double t = i/(double)N;
complex<double> u(0, 2*M_PI*t);
complex<double> z = exp(u);
complex<double> w = 1.0/N * (x[0] + x[1]*z + x[2]*z*z);

cout << "f(" << t << ") = " << w << endl;
}

return 0;
}
</source>

[[Category:Numerical_libraries]][[Category:bwUniCluster]][[Category:bwForCluster_Chemistry]][[Category:BwForCluster_BinAC]][[Category:bwForCluster_MLS&WISO_Production]]

Development/MKL

2017-11-28T07:51:05Z

T Kienzle:

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| numlib/mkl
|-
| License
| Commercial. See [https://software.intel.com/en-us/articles/end-user-license-agreement EULA].
|-
| Citing
| n/a
|-
| Links
| [https://software.intel.com/en-us/intel-mkl Intel MKL Homepage] | [https://software.intel.com/en-us/articles/intel-math-kernel-library-documentation Online-Documentation]
|-
| Graphical Interface
| No
|}

= Description =
'''Intel MKL (Math Kernel Library)''' is a library of optimized math routines for numerical computations such as linear algebra (using BLAS, LAPACK, ScaLAPACK) and discrete Fourier Transformation.
With its standard interface in matrix computation and the interface of the popular fast Fourier transformation library fftw, MKL can be used to replace other libraries with minimal code changes. In fact a program which uses FFTW without MPI doesn't need to be changed at all. Just recompile it with the MKL linker flags.
 
* [http://software.intel.com/en-us/articles/intel-math-kernel-library-documentation Online-Documentation]

= Versions and Availability =
A list of currently available MKL modules can be obtained from the
 <big>
[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS] </big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/numlib/mkl
|width=99%
|height=700
|border=0
}}
 
Show a list of available versions using 'module avail numlib/mkl' on any HPC-C5 cluster.
<pre>
: EXAMPLE bwUniCluster
$ module avail numlib/mkl
------------------------------ /opt/bwhpc/common/modulefiles ------------------------------
numlib/mkl/10.3.12 numlib/mkl/11.1.4(default)
numlib/mkl/11.0.5 numlib/mkl/11.2.3
</pre>

= Local documentation =
There is some information in the module help file accessible via 'module help numlib/mkl'-
command.
<pre>
: EXCERPT ONLY
$ module help numlib/mkl
----------- Module Specific Help for 'numlib/mkl/11.1.4' ----------
This module provides the Intel(R) Math Kernel Library (MKL)
version 11.1.4, a fast and reliable implementation
of BLAS/LAPACK/FFTW (see also 'http://software.intel.com/en-us/intel-mkl/').

The preferable compiler for this MKL version is 'compiler/intel/14.0'. Linking
with other compilers like GNU, PGI and SUN is possible. The desired compiler
module (exception system GNU compiler) has to be loaded before using MKL.

Local documentation:

Man pages in '$MKL_MAN_DIR/man3', e.g. 'man dotc'.
firefox $MKL_DOC_DIR/mkl_documentation.htm
acroread $MKL_DOC_DIR/l_mkl_11.1.4.211.mklman.pdf
acroread $MKL_DOC_DIR/l_mkl_11.1.4.211.mkl_11.1.4_lnx_userguide.pdf

Linking examples (ifort compiler with support for blas and lapack):

* Dynamic linking of myprog.f and parallel MKL supporting the LP64 interface:

ifort myprog.f -L${MKL_LIB_DIR} -I${MKL_INC_DIR} \
-lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread
[... t.b.c. ...]
</pre>
After loading the module, the environment variable $MKL_DOC_DIR points to the local documentation folder. Various examples can be found in $MKLROOT/examples.

= MKL-Specific Environments =
To see a list of all MKL environments set by the 'module load'-command use 'env | grep MKL'. Or use the command 'module display numlib/mkl/version'.
 
Example (bwUniCluster)
<pre>
$ module load numlib/mkl
$ env | grep MKL
MKLROOT=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/mkl
MKL_LIB_MIC_COM=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/lib/mic
MKL_DOC_DIR=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composerxe/Documentation/en_US/mkl
MKL_NUM_THREADS=1
MKL_HOME=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/mkl
MKL_LIB_MIC=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/mkl/lib/mic
MKL_MAN_DIR=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/man/en_US
MKL_EXA_DIR=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/composerxe/Samples/en_US
MKL_STA_DIR=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/mkl/lib/intel64_static
MKL_INC_DIR=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/mkl/include
MKL_BIN_DIR=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/mkl/bin
MKL_LIB_DIR=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/mkl/lib/intel64
MKL_VERSION=11.1.4
MKL_LIB_COM=/opt/bwhpc/common/compiler/intel/compxe.2013.sp1.4.211/lib/intel64
</pre>

= Compiling and linking =
Compilation is possible with both GCC and Intel compilers but it is easier for Intel compilers, so this case is explained here.
After loading the compiler and the library module with
<pre>
$ module load compiler/intel
$ module load numlib/mkl
</pre>
you can include the MKL header file in your program:
<source lang=cpp>#include <mkl.h></source>
Compilation is simple:
<pre>$ icpc -c example_mkl.c</pre>
When linking the program you have to tell the compiler to link against the mkl library:
<pre>$ icpc example_mkl.o -mkl</pre>
With the -mkl switch the intel compiler automatically sets the correct linker flags but you can specify them explicitly for example to enable static linking or when non-intel compilers are used. Information about the different options can be found at http://software.intel.com/en-us/node/438568 and especially helpful is the MKL link line advisor at http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor.
By default $MKL_NUM_THREADS is set to 1 and so only one thread will be created, but if you feel the need to run the computation on more cores (after benchmarking) you can set $MKL_NUM_THREADS to a higher number.
 
 
== FFTW Interface to Intel Math Kernel Library (MKL) ==
Sometimes, [[FFTW|FFTW]] is not available on your cluster. You can use the MKL library
instead and include the FFTW functions, too.
 
Intel Math Kernel Library (MKL) offers FFTW2 and FFTW3 interfaces to Intel MKL Fast Fourier Transform and Trigonometric Transform functionality. The purpose of these interfaces is to enable applications using FFTW to gain performance with Intel MKL without changing the program source code.
 
Here is an excerpt from 'module help numlib/mkl':
<pre>
Static FFTW2/3 C/Fortran interfaces can be found in dir
${MKL_HOME}/interfaces/
Examples:
Link to FFTW3 Fortran interface with GNU compiler and ilp64 support:
${MKL_HOME}/interfaces/fftw3xf/libfftw3xf_intel64_double_i8_gnu47.a
Link to FFTW3 Fortran interface with Intel compiler and lp64 support:
${MKL_HOME}/interfaces/fftw3xf/libfftw3xf_intel64_double_i4_intel150.a
The Intel FFTW interfaces requires the Intel MKL library (e.g. it does
not work with ACML library). Usually it is not a problem to use a
different compiler version, e.g. to use _gnu41.a with gnu 4.3 compiler.
See dir ${MKL_HOME}/interfaces/ for other interfaces (fftw2/3 Fortran/C).
Compiler option for include files: -I${MKL_INC_DIR}/fftw
</pre>
See the corresponding webpages:
* [https://software.intel.com/en-us/node/471410 FFTW Interface to Intel Math Kernel Library]
* [https://software.intel.com/de-de/node/471414 FFTW2 Interface to Intel Math Kernel Library]
* [https://software.intel.com/en-us/node/471456 FFTW3 Interface to Intel Math Kernel Library]
 

= Examples =
To help getting started we provide two C++ examples. The first one computes the square of a 2x2 matrix:
<source lang="cpp">
#include <iostream>
#include <mkl.h>
using namespace std;

int main()
{
double m[2][2] = {{2,1}, {0,2}};
double c[2][2];

for(int i = 0; i < 2; ++i)
{
for(int j = 0; j < 2; ++j)
cout << m[i][j] << " ";

cout << endl;
}

cblas_dgemm(CblasRowMajor, CblasNoTrans, CblasNoTrans, 2, 2, 2, 1.0, &m[0][0], 2, &m[0][0], 2, 0.0, &c[0][0], 2);

cout << endl;

for(int i = 0; i < 2; ++i)
{
for(int j = 0; j < 2; ++j)
cout << c[i][j] << " ";

cout << endl;
}

return 0;
}
</source>
And the second one does a fast Fourier transformation using the Intel MKL interface (DFTI):
<source lang="cpp">
#include <iostream>
#include <complex>
#include <cmath>
#include <mkl.h>
using namespace std;

int main()
{
const int N = 3;
complex<double> x[N] = {2, -1, 0.5};

cout << "Input: " << endl;

for(int i = 0; i < N; i++)
cout << x[i] << endl;

DFTI_DESCRIPTOR_HANDLE desc;

DftiCreateDescriptor(&desc, DFTI_DOUBLE, DFTI_COMPLEX, 1, N);
DftiCommitDescriptor(desc);
DftiComputeForward(desc, x);
DftiFreeDescriptor(&desc);

cout << "\nOutput: " << endl;

for(int i = 0; i < N; i++)
cout << x[i] << endl;

cout << "\nTest the interpolation function f:" << endl;

for(int i = 0; i < N; i++)
{
double t = i/(double)N;
complex<double> u(0, 2*M_PI*t);
complex<double> z = exp(u);
complex<double> w = 1.0/N * (x[0] + x[1]*z + x[2]*z*z);

cout << "f(" << t << ") = " << w << endl;
}

return 0;
}
</source>

[[Category:Numerical_libraries]][[Category:bwUniCluster]][[Category:bwForCluster_Chemistry]][[Category:BwForCluster_BinAC]][[Category:bwForCluster_MLS&WISO_Production]]

Development/Intel Compiler

2017-11-28T07:50:30Z

T Kienzle:

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| compiler/intel
|-
| License
| Commercial. See $INTEL_HOME/install-doc/EULA.txt. | [https://software.intel.com/en-us/faq/licensing Intel Product Licensig FAQ]
|-
|Citing
| n/a
|-
| Links
| [https://software.intel.com/en-us/c-compilers Intel C-Compiler Homepage]
|-
| Graphical Interface
| [[#Debugger|Yes (Intel Debugger GUI-Verison)]]
|-
| Included modules
| icc | icpc | ifort | idb | gdb-ia
|}
 
= Introduction =
The '''Intel Compiler''' of the '''Intel Composer XE Suite''' consists of tools to compile and debug C, C++ and Fortran programs:
{| width=500px class="wikitable"
|-
|style="padding:3px"| icc
|style="padding:3px"| Intel C compiler
|-
|style="padding:3px"| icpc
|style="padding:3px"| Intel C++ compiler
|-
|style="padding:3px"| ifort
|style="padding:3px"| [https://software.intel.com/en-us/fortran-compilers Intel Fortran compiler]
|-
|style="padding:3px"| [[#GUI|idb]]
|style="padding:3px"| Intel debugger in GUI mode (until version 14 only)
|-
|style="padding:3px"| [[#Console Mode|gdb-ia]]
|style="padding:3px"| Intel version of GNU debugger in console mode (from version 15)
|-
|style="padding:3px"| [[#Console Mode|idbc]]
|style="padding:3px"| Intel debugger in console mode (until version 14 only)
|}
The intel compiler suite also includes the TBB (Threading Building Blocks) and IPP (Integrated Performance Primitives) libraries.
 
 
More information about the MPI versions of the Intel Compiler is available here:
* [[BwHPC_BPG_for_Parallel_Programming|Best Practices Guide for Parallel Programming]].
 

= Versions and Availability =
A list of versions currently available on all [[:Category:BwHPC Cluster|bwHPC Clusters]] can be obtained from the cluster information system
 
[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System]


On the command line interface of any bwHPC cluster you'll get a list of available versions
by executing:
<pre>
$ module avail compiler/intel
</pre>
 
== Loading the module ==
=== Default Version ===
You can load the default version of the Intel Compiler with the command
<pre>
$ module load compiler/intel
</pre>
If loading the module fails, check if you have already loaded the module
with 'module list'.
 
 
=== Specific (newer or older) Version ===
If you wish to load a specific (older or newer) version (if available),
add the specific version number of the intel compiler, e.g. for loading Intel compiler suite 17.0, execute:
<pre>
$ module unload compiler/intel
$ module load compiler/intel/17.0
</pre>
Note: Only one compiler can be loaded in your active session, hence, before loading a new intel compiler version you must to unload the current loaded version.

For unloading the intel compiler the version number is not required:
<pre>
$ module unload compiler/intel
</pre>
unloads any currently loaded intel compiler version.
 
 

= Intel Compiler Specific Environment Variables =
To see a list of all Intel Compiler environment variables set by the 'module load' command execute:
<pre>
module show compiler/intel
</pre>
 

= Documentation =
== Online documentation ==
* [https://software.intel.com/en-us/articles/intel-c-composer-xe-documentation Intel® C-Compiler Documentation]
* [https://software.intel.com/en-us/intel-software-technical-documentation Intel® Software Documentation Library]

== Local documentation ==
For version specific documentation see the help page of the module. For example
<pre>
$ module help compiler/intel
</pre>
will show the information for the default version.


== Manual Pages ==
For detailed lists of the different program options consult the particular man page
<pre>
$ man icc
$ man icpc
$ man ifort
$ man idb
$ man gdb-ia
</pre>
 

= Debugger =
Please use DDT. It is a parallel debugger with graphical user interface and can also be used for debugging serial programs. The description of the debugger can be found on the website
 
http://www.bwhpc-c5.de/wiki/index.php/DDT
 

= Optimizations =
You can turn on various optimization options to enhance the performance of your program. Which options are the best depends on the specific program and can be determined by benchmarking your code. A command which gives good performance and a decent file size is
'''icc -xHost -O2 ex.c'''.
With the option '''-xHost''' instructions for the highest instruction set available on the compilation host processor are generated. If you want to generate optimal code on bwUniCluster for both nodes with Sandy Bridge architecture and nodes with Broadwell architecture, you must compile your code with the options '''-xAVX -axCORE-AVX2''' (instead of '''-xHost''').
 
There are more aggressive optimization flags and levels (e.g. -O3 or -fast and implied options) but the compiled programs can get quite large due to inlining. Additionally the compilation process will probably take longer. Moreover it may happen that the compiled program is even slower -- or may require installation of additional statically-linked libraries. Such a command would be for example:
'''icc -fast ex.c'''
 
 

= Profiling =
Profiling an application means augmenting the compiled binary with information on execution counts per source-line (and basic blocks) -- e.g. one may see how many times an if-statement has been evaluated to true. To do so, compile your code with the profile flag:
'''icc -p ex.c -o ex'''.
 
Using the gprof tool, one may manually inspect execution count of each executed line of source code.
 
For compiler optimization, recompile your source using
'''icc -prof-gen ex.c -o ex'''
then execute the most co]]mmon and typical use-case of your application, and then recompile using the generated profile count (and using optimization):
'''icc -prof-use -O2 ex.c -o ex'''.
 
== Further literature ==
A tutorial on optimization can be found at [https://software.intel.com/sites/default/files/managed/c1/61/compiler-essentials.1.pdf Compiler-Essentials.pdf]
and to get the different optimization options execute
'''icc -help opt'''
'''icc -help advanced'''
 
or the previously described catch-all option '''''-v --help'''''.

[[Category:Compiler_software]][[Category:bwUniCluster]][[Category:bwForCluster_Chemistry]][[Category:BwForCluster_BinAC]][[Category:bwForCluster_MLS&WISO_Production]]

BwUniCluster2.0/Software/Mathematica

2017-11-28T07:46:25Z

T Kienzle:

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| math/mathematica
|-
| License
| Commercial. See [http://www.wolfram.com/mathematica/pricing/ Mathematica Pricing].
|-
| Citing
| n/a
|-
| Links
| [http://www.wolfram.com/mathematica/ Homepage] | [http://www.wolfram.com/support/?source=nav Documentation]
|-
| Graphical Interface
| Yes (See [[VNC]])
|-
| Comments
| Mathematica might not be available on all locations and clusters.
|}

== Description ==
'''Mathematica''' is a software from Wolfram for symbolic and numerical computation with many features such as powerful visualization and application specific functions.
 
 
== Versions and Availability ==
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>
[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]
</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/math/mathematica
|width=90%
|height=330
|border=0
}}

= Loading =
To check if Mathematica is available execute
$ module avail math/mathematica

If Mathematica is available you can load a specific version or you can load the default version with
$ module load math/mathematica

= General Usage =
Mathematica can be used interactively on the command line or with a graphical front-end.
Alternatively Mathematica can run a script in batch mode which is useful when submitting batch jobs to the cluster.
After loading Mathematica the different modes can be used as follows.

*Interactive with GUI (needs [[X11 forwarding]] or [[VNC]]): <pre>$ mathematica</pre>

*Interactive with command line: <pre>$ math</pre>

*Non-Interactive. Looks like interactive with command line but input is taken from script.m: <pre>$ math < script.m</pre>

*Non-Interactive. You have to explicitly specify what you want to print <pre>$ math -script script.m</pre>However, the output is done in InputForm which is suitable for input in other Mathematica calculations but if you want pretty output you have to change the output format to OutputForm like this <pre>SetOptions[$Output, FormatType -> OutputForm]</pre>

For an introduction to Mathematica we refer to the online documentation ([http://reference.wolfram.com/language/ Mathematica Documentation Center]). Specific information on the use in a compute cluster is in the next section.

= Parallel Computation =
Obviously parallel computation can be useful to speed up time-consuming computations, but it should also be used for multiple computations with different input data files (e.g. for parametric studies). The reason for this
is the license model from Wolfram. There are two types of licenses.
* Each time an instance of Mathematica starts, a so called MathKernel license is used up.
* Each time Mathematica spawns a subprocess, a license called SubMathKernel is used up.
Because usually there are much more SubMathKernel licenses than MathKernel licenses it is recommended to start multiple subprocesses instead of submitting multiple jobs.

Remember to request the correct amount of processors in your [[Batch Jobs]] script but note that Mathematica will not automatically use these processors.
In general you have to adjust your code to benefit from more cores.
To do this you first have to start a number of kernels which are then used by ParallelTable to run the computations in parallel.
This basic example computes the first eight square numbers in parallel.
<pre>
LaunchKernels[8]
f[i_] := i^2
DistributeDefinitions[f]
ParallelTable[f[i], {i,0,7}]
CloseKernels[]
</pre>
Note that the use of DistributeDefinitions is necessary for f is a user defined function and the definition of this function must be available to all kernels.

The next example is the computation of a numerical solution for the following initial value problem
x'(t) = x(t)^2 - x(t)^3
x(0) = d
It is difficult to solve this equation with high accuracy at the point 1/d.
We decrease the step size of the algorithm to see how the execution time and the relative error at the point 1/d change.

<pre>
d = 0.00001

y[t_] := 1/(ProductLog[(1/d-1)*Exp[1/d-1-t]]+1) (* analytical solution *)
relerr[t_, s_] := Abs[(y[t] - x[t]/.s)/y[t]] (* relative error of solution s at time t *)
g[v_] := {v[[1]], v[[2]][[1]], relerr[1/d, v[[2]][[2]][[1]]]} (* helper function *)

(* compute numerical solutions for 6 different step sizes *)
LaunchKernels[6]
tbl = ParallelTable[{step, Timing[NDSolve[{x'[t] == x[t]^2 - x[t]^3, x[0] == d}, x, {t, 0, 2/d}, MaxStepSize->step, MaxSteps->10000000]]}, {step,5,3,-0.4}]
CloseKernels[]

SetOptions[$Output, FormatType -> OutputForm] (* better output when called with -script *)
Print[ Grid[Join[{{"Stepsize", "Time", "Error at 1/d"}}, Map[g, tbl]]] ] (* print the result *)
</pre>
Mathematica features some further functions similar to ParallelTable which are documented on the [http://reference.wolfram.com/mathematica/guide/ParallelComputing.html Mathematica parallel computing webpage] from Wolfram. One such function is ParallelSubmit which can be used to start two (or more) completely unrelated functions in parallel. But bear in mind that the functions should have similar running times because the scheduler reserves the requested resources for the runtime of the whole job. The following code shows how to pack two 1-core jobs into one 2-core job.
<pre>
f[r_] := (
d = 0.000001;
{time, sol} = Timing[NDSolve[{x'[t] == r*x[t]^2 - x[t]^3, x[0] == d}, x, {t, 0, 2/d}, MaxStepSize->3, MaxSteps->10000000]];
{time, x[1/d]/.sol[[1]]}
)

g[r_] := (
{time, sol} = Timing[NDSolve[{y''[t] + (r+1)*y'[t] + r*y[t] == 0, y[0] == 1, y'[0] == 0}, y, {t, 0, 30}, MaxStepSize->0.000002, MaxSteps->10000000000]];
{time, y[30]/.sol[[1]]}
)

kernel1 := (
file = OpenWrite["out1.txt", FormatType -> OutputForm];
$Output = {file};
Print[f[1.2]];
Close[file];
)

kernel2 := (
file = OpenWrite["out2.txt", FormatType -> OutputForm];
$Output = {file};
Print[g[1000]];
Close[file];
)

LaunchKernels[2]
DistributeDefinitions[kernel1, kernel2]
e = {ParallelSubmit[kernel1], ParallelSubmit[kernel2]}
WaitAll[e]
</pre>
The helper functions kernel1 and kernel2 get started in different threads and then they execute f and g. The output of these functions is redirected to different files so the results can clearly be distinguished and examined as soon as one function is done.

[[Category:Mathematics software]] [[Category:bwUniCluster]][[Category:bwForCluster_MLS&WISO_Production]]

Development/Parallel Programming

2017-11-27T14:07:04Z

T Kienzle:

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| mpi/impi | mpi/openmpi
|-
| Links
| [https://software.intel.com/en-us/intel-mpi-library Intel® MPI Library] | [http://www.open-mpi.org/ Open MPI]
|-
| License
| [https://software.intel.com/en-us/articles/intel-mpi-library-licensing-faq Intel MPI Library Licensing FAQ] install-doc/EULA.txt | [http://www.open-mpi.org/community/license.php Open MPI License]
|}
 
= Introduction =
This page will provide information regarding the supported parallel programming paradigms and specific hints on their usage.
Please refer to the [[BwUniCluster_Environment_Modules|Modules Documentation]] how to setup your environment on bwUniCluster to load a specific software installation.
 
More informations about Compilers installed at our clusters is available here:
* [[Intel_Compiler|Intel Compiler Suite]]
* [[GCC|GNU Compiler (GCC)]]
* [[General_compiler_usage|General Compiler Usage (incl. PGI Compiler)]]

= Versions and Availability =
=== impi (Intel) ===
A list of versions currently available compilers on the bwHPC-C5-Clusters can be obtained from the
 
<big>
 
[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]
 </big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/mpi/impi
|width=99%
|height=1200
}}

=== openmpi ===
A list of versions currently available compilers on the bwHPC-C5-Clusters can be obtained from the
 
<big>
 
[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]
 </big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/mpi/openmpi
|width=99%
|height=2100
}}
 

= OpenMP =
== General Information ==
OpenMP is a mature specification [http://www.openmp.org/specifications/] to allow easy, portable, and most importantly incremental node-level parallelisation of code.
Being a thread-based approach, OpenMP is aimed at more fine-grained parallelism than [[BwHPC_Best_Practices_Repository#MPI|MPI]].
Although there have been extensions to extend OpenMP for inter-node parallelisation, it is a node-level approach aimed to make best usage of a node's cores.
 
With regard to ease-of-use, OpenMP is ahead of any other common approach: the source-code is annotated using <tt>#pragma omp</tt> or <tt>!$omp</tt> statements, in C/C++ and Fortran respectively.
Whenever the compiler encompasses a semantic block of code encapsulated in a parallel region, this block of code is transparently compiled into a function, which is passed to a so-called team-of-threads upon entering this semantic block. This fork-join model of execution eases a lot of the programmer's pain involved with Threads.
Being a loop-centric approach, OpenMP is aimed at codes with long/time-consuming loops.
A single combined directive <tt>pragma omp parallel for</tt> will tell the compiler to automatically parallel the ensuing for-loop.
 
The following example is a bit more advanced in that even reductions of variables over multiple threads are easily to parallel:
<source lang="c">
for (int i=0, sum = 0.0; i < VECTOR_LEN; i++)
norm2 += (v[i]*v[i]);
</source>
is parallelized by just adding a single line as in:
<source lang="c">
# pragma omp parallel for reduction(+:norm2)
for (int i=0, sum = 0.0; i < VECTOR_LEN; i++)
norm2 += (v[i]*v[i]);
</source>
With <tt>VECTOR_LENGTH</tt> being large enough, this piece of code compiled with OpenMP will run in parallel, exhibiting very nice speedup.
Compiled without, the code remains as is. Developpers may therefore incrementally parallelize their application based on the profile derived from performance analysis tools, starting with the most time-consuming loops.
Using OpenMP's concise API, one may query the number of running threads, the number of processors, a time to calculate runtime, and even set parameters such as the number of threads to execute a parallel region.
 
The OpenMP-4.0 specification added support for the SIMD-directive to better utilize SIMD-vectorization, as well as integrating directives to offload computation to accelerators using the <tt>target</tt> directive: these are integrated into the Intel Compiler and are actively being worked on for the GNU compiler, some restrictions may apply.
== OpenMP Best Practice Guide ==
The following silly example to calculate the squared Euklidian Norm shows some techniques:
<source lang="c">
#include <stdlib.h>
#include <stdio.h>
#include <omp.h>

#define VECTOR_LENGTH 5

int main (int argc, char * argv[])
{
int len = VECTOR_LENGTH;
int i;
double * v;
double norm2 = 0.0;
double t1, tdiff;

if (argc > 1)
len = atoi (argv[1]);
v = malloc (len * sizeof(double));

t1 = omp_get_wtime();
// Initialization already with (the same number of) threads
#pragma omp parallel for
for (i=0; i < len; i++) {
v[i] = i;
}

// Now aggregate the sum-of-squares by specifying a reduction
#pragma omp parallel for reduction(+:norm2)
for(i=0; i < len; i++) {
norm2 += (v[i]*v[i]);
}
tdiff = omp_get_wtime() - t1;

printf ("norm2: %f Time:%f\n", norm2, tdiff);
return 0;
}
</source>


* Group independent parallel sections together: in the above example, You may combine those two sections into one larger parallel block. This will just once enter the parallel region (in the fork-join model) instead of twice. Especially in inner loops, this will considerably decrease overhead.
* Compile with the Intel compiler's option <tt>-diag-enable sc-parallel3</tt> to get the further warnings on thread-safety, performance, etc. The following code with loop-carried dependency will e.g. compile fine (aka without warning):
<source lang="c">
#pragma omp parallel for reduction(+:norm2)
for(i=1; i < len-1; i++) {
v[i] = v[i-1]+v[i+1];
}
</source>
However the Intel compiler with <tt>-diag-enable sc-parallel3</tt> will produce the following warning:
<tt>warning #12246: variable "v" has loop carried data dependency that may lead to incorrect program execution in parallel mode; see (file:omp_norm2.c line:32)</tt>
* Always specify <tt>default(none)</tt> on larger parallel regions in order to specifically set the visibility of variables to either <tt>shared</tt> or <tt>private</tt>.
* Try to restructure code to allow for <tt>nowait</tt>: OpenMP defines synchronization points (implied barriers) at the end of work sharing constructs such as the <tt>pragma omp for<tt> directive. If the ensuing section of code does not depend on data being generated inside the parallel section, adding the <tt>nowait</tt> clause to the worksharing directive allows the compiler to eliminate this synchronization point. This reduces overhead, allows for better overlap and better utilization of the processor's resources. This might imply however to restructure the code (move portions of independent code in between dependent works-sharing constructs).
== Usage ==
OpenMP is supported by various compilers, here the usage for two main compilers [[BwHPC_BPG_Compiler#GCC|GCC]] and [[BwHPC_BPG_Compiler#Intel Suite|Intel Suite]] are introduced.
For both compilers, You first need to turn on OpenMP support by specifying a parameter on the compiler's command-line.
In case You make function calls to OpenMP's API, You also need to include the header-file <tt>omp.h</tt>.
OpenMP's API allows to query or set the number of threads, query the number of processors, get a wall-clock time to measure execution times, etc.
=== OpenMP with GNU Compiler Collection ===
Starting with version 4.2 the gcc compiler supports OpenMP-2.5.
Since then the analysis capabilities of the GNU compiler have steadily improved.
The installed compilers support OpenMP-3.1.

 
To use OpenMP with the gcc-compiler, pass <tt>-fopenmp</tt> as parameter.
=== OpenMP with Intel Compiler ===
The Intel Compiler's support for OpenMP is more advanced than gcc's -- especially in term of programmer support.
To use OpenMP with the Intel compiler, pass <tt>-openmp</tt> as command-line parameter.
 
One may get very insightful information about OpenMP, when compiling with
* Compiling with <tt>-openmp-report2</tt> to get information, which loops were parallelized and a reason why not.
* Compiling with <tt>-diag-enable sc-parallel3</tt> to get errors and warnings about your sources weaknesses with regard to parallelization (see example below).
 
 

= MPI =
In this section, You will find information regarding the supported installations of the Message-Passing Interface libraries and their usage. 
Due to the Fortran interface ABI, all MPI-libraries are normally bound to a specific compiler-vendor and even the specific compiler version.
Therefore, as listed in [[BwHPC_BPG_Compiler]] two compilers are supported on bwUniCluster: [[GCC|GCC]] and [[Intel_Compiler|Intel Compiler Suite]].
As both compilers are continously improving, the communication libraries will be adopted in lock-step.
 
With a set of different implementations, there comes the problem of choice. These pages should inform the user of the communication libraries, what considerations should be done with regard to performance, maintainability and debugging -- in general tool support -- of the various implementations.
== MPI Introduction ==
The Message-Passing Interface is a standard provided by the [http://www.mpi-forum.org MPI-Forum] which regularly convenes for the [http://mpi-forum.org/meetings/ MPI-Forum Meetings] to update this standard. The current version is MPI-3.0 available as [http://mpi-forum.org/docs/mpi-3.0/mpi30-report.pdf PDF].
This document defines the API of over 300 functions for the C- and the Fortran-language -- however, You will certainly not need all of them to begin with.
 
Every MPI-conforming program needs to call <tt>MPI_Init()</tt> and <tt>MPI_Finalize()</tt> upon start and shutdown -- or <tt>MPI_Abort()</tt> in case of an abnormal termination.
After initialization the programm may call any other MPI-function, specifically communication functions.
However to do so, it is required to find out how many processes the program has been started with, using <tt>MPI_Comm_size()</tt> and what number (here called a rank= this particular process has using <tt>MPI_Comm_rank()</tt>.
Communication is always relative to a so-called communicator -- the default one after initialization being called <tt>MPI_COMM_WORLD</tt>-
 
There's basically three ways of communication:
* two-sided communication using point-to-point (often abbreviated P2P) functions, such as <tt>MPI_Send()</tt> and <tt>MPI_Recv()</tt>, which always involves two participating processes,
* collectice communcation functions (often abbreviated as colls) involve multiple processes, examples are <tt>MPI_Bcast()</tt> and <tt>MPI_Reduce()</tt>,
* one-sided communication, where communication between two processes is initiated by one-process, only. With proper RMA-hardware support and careful programming, this may allow higher performance or scalibility.
 
All parts of the programm, which reference MPI functionality need to be compiled with the '''same''' compiler settings/include files and linked to the same MPI-Library. This is stressed here, since without taking pre-cautions, a different MPI's header may be included, resulting in funny errors: consider that Intel MPI is derived from MPIch, with MPI-datatypes being C <tt>int</tt>s, while Open MPI uses pointers to structures (the former being 4, the latter being 8 bytes on bwUniCluster).
To ease the programmer's life, MPI implementations offer compiler-wrappers, e.g. <tt>mpicc</tt> for C and <tt>mpif90</tt> for Fortran90 for compilation and linking, taking care to include all required libraries.
 
All programs must be started using the <tt>mpirun</tt> or <tt>mpiexec</tt> command. Depending on the actual implementation, it uses different arguments, however the following works with any MPI:
* <tt>mpirun -np 128 ./app</tt> starts 128 processes (with ranks 0 to 127)
* <tt>mpiexec -n 128 -hostfile mynodes.txt ./app</tt> starts 128 processes on only the nodes listed line-by-line in the provided text-file <tt>mynodes.txt</tt>.
* <tt>mpiexec -n 64 ./app1 : -n 64 ./app2</tt> starts 128 processes, 64 of which execute <tt>app1</tt>, the other 64 execute <tt>app2</tt>. All processes however participate in the same <tt>MPI_COMM_WORLD</tt> and therefore must accordingly take care about their respective ranks.
Please note, that process placement (e.g. a round-robin scheme), and specifically process-binding to sockets is MPI-implementation dependant.
== MPI Best Practice Guide ==
Specific performance considerations with regard to MPI (independent of the implementation):
* No communication at all is best: Only communicate between processes if at all necessary. Consider that file-access is "communication" as well.
* If communication is done with multiple processes, try to involve as many processes in just one call: MPI optimizes the communication pattern for so-called "collective communication" to take advantage of the underlying network (with regard to network topology, message sizes, queueing capabilities of the network interconnect, etc.). Therefore try to always think in collective communication, if a communication pattern involves a group of processes.
* Try to group processes together: Function calls like <tt>MPI_Cart_create</tt> will come in handy for applications with cartesian domains but also general communicators derived from <tt>MPI_COMM_WORLD</tt> using <tt>MPI_Comm_split()</tt> may benefit by MPI's knowing the underlying network topology. Use MPI3's <tt>MPI_Comm_split_type()</tt> with <tt>MPI_COMM_TYPE_SHARED</tt> for a sub-communicator with processes having access to the same shared memory region (aka on bwUniCluster the same node).
* File-accesses to load / store data '''must''' be done collectively: Writing to storage, or even reading the initialization data -- all of which involves getting data from/to all MPI processes -- must be done collectively. MPI's Parallel IO offers a rich API to read and distribute the data access -- in order to take advantage of parallel filesystems like Lustre. A many-fold performance improvement may be seen by writing data in large chunks in collective fashion -- and at the same time being nice to other users and applications.
* Try to hide the communication by computation: Try to hide (some) of the cost of communication of Point-to-point communication by using non-blocking / immediate P2P-calls (<tt>MPI_Isend</tt> and <tt>MPI_Irecv</tt> et al, followed by <tt>MPI_Wait</tt> or <tt>MPI_Test</tt> et al). This may allow the MPI-implementation to initiate or even offload communication to the network interconnect and resume executing your application, while data is being transferred. MPI-3 adds non-blocking collectives, e.g. <tt>MPI_Ibcast()</tt> or <tt>MPI_Iallreduce()</tt>. For extra credit, explain the use-cases of <tt>MPI_Ibarrier()</tt>.
* Every call to MPI may trigger an access to physical hardware -- limit it: When calling communication-related functions such as <tt>MPI_Test</tt> to check whether a specific communication has finished, the queue of the network adapter may need to be queried. This memory access or even physical hardware access to query the state will cost cycles. Therefore, the programmer should combine multiple requests with functions such as <tt>MPI_Waitall()</tt> or <tt>MPI_Waitany()</tt> or their Test*-counterparts.
* Make usage of derived datatypes: instead of manually copying data into temporary, even newly allocated memory, describe the data-layout to MPI -- and let the implementation, or even the network HCA's hardware do the data fetching.
* Bind Your processes to sockets: Operating Systems are good in making best use of the ressources -- which sometimes involves moving tasks from one core to another, or even (though more unlikely since the OS' heuristics try to avoid it) to another socket, with the obvious effects: Caches are cold, every memory access to memory allocated on the previous socket "has to travel the bus". This is particularly happening if You have multiple OpenMP parallel regions which are separated by code that does IO -- and threads are sleeping -- the processes doing IO may wander to a different socket... Bind Your processes to at least the socket. All major MPIs support this binding (see below).
* Do not use the C++ interface: First of all, it has been marked as deprecated in the MPI-3.0 standard, since it added little benefit to C++ programmers over the C-interface. Moreover, since MPI implementations are written in C, the interface adds another level of indirection and therefore a bit of overhead in terms of instructions and Cache misses.
== Open MPI ==
The [http://www.open-mpi.org Open MPI] library is an open, flexible and nevertheless performant implementation of MPI-2 and MPI-3. Licensed under BSD, it is being actively developed by an open community of industry and research institutions.
The flexibility comes in handy: using the concept of a [http://www.open-mpi.org/faq/?category=tuning#mca-def MCA] (aka a plugin) Open MPI supports many different network interconnects (Infinband, TCP, Cray, etc.) , on the other hand, a installation may be tailored to suite an installation, e.g. the network (Infiniband with specific settings), the main startup-mechanism, etc.
Furthermore, the [http://www.open-mpi.org/faq/ FAQ] offers hints on [http://www.open-mpi.org/faq/?category=tuning performance tuning].
=== Usage ===
Like other MPI implementations, after loading the [[BwUniCluster_Environment_Modules|module]], Open MPI provides the compiler-wrappers <tt>mpicc</tt>, <tt>mpicxx</tt> and <tt>mpifort</tt> (or for
versions lower than 1.7 <tt>mpif77</tt> and <tt>mpif90</tt>) for the C-, C++ and Fortran compilers respectively. Albeit their usage is not required, these wrappers are handy to not have to use the command-line options for header- or library directories, aka <tt>-I</tt> and <tt>-L</tt>, as well as the actual needed MPI-libraries itselve.
=== Further information ===
Open MPI also features a few specific functionalities that will help users and developpers, alike:
* Open MPI's tool <tt>ompi_info</tt> allows seeing all of Open MPI's installed MCA components and their specific options.
Without any option the user gets a list of flags, the Open MPI installation was compiled for (version of compilers, specific configure-flags, e.g. debugging, or profiling options). Furthermore, using <tt>ompi_info --param all all</tt> one may see all of the MCA's options, e.g. that the default PML-MCA uses an initial free-list of 4 blocks (increased by 64 upon first encountering this limit):
<tt>ompi_info --param ob1 all</tt> -- which may be increased for applications that are certain to benefit from a larger value upon startup.
* Open MPI allows adapting MCA parameters on the command-line: parameters may be supplied, e.g. the above-mentioned parameter <tt>mpirun -np 128 --mca mpirun -np 16 --mca pml_ob1_free_list_num 128 ./mpi_stub</tt>.
* Open MPI internally uses the tool [http://www.open-mpi.org/projects/hwloc/ hwloc] for node-local processor-information, as well as process- and memory-affinity. This tool also is a good tool to get information on the node's processor topology and Cache-information. This may be used to optimize and balance memory usage or for choosing a better ratio of MPI processes per node vs. OpenMP threads per core.
== Intel (i)MPI ==
The Intel MPI Library is a multi-fabric message passing library that implements the Message Passing Interface, v2.2 (MPI-2.2) specification. It provides a standard library across Intel
platforms that enable adoption of MPI-2.2 functions as their needs dictate.
 
The Intel MPI Library enables developers to change or to upgrade processors and interconnects as new technology becomes available without changes to the software or to the operating environment.
 
The library is provided in the following kits:
* The Intel MPI Library Runtime Environment (RTO) has the tools you need to run programs, including Multipurpose Daemon (MPD), Hydra and supporting utilities, shared (.so) libraries, and documentation.
* The Intel MPI Library Development Kit (SDK) includes all of the Runtime Environment components plus compilation tools, including compiler commands such as mpiicc, include files and modules, static (.a) libraries, debug libraries, trace libraries, and test codes.
=== General information ===
All Intel mpi-modules are called ''''mpi/impi''''.
 
These modules provides the Intel Message Passing Interface (mpicc, mpicxx, mpif77
and mpif90) for the Intel compiler suite (icc, icpc and ifort) (see also [http://software.intel.com/en-us/intel-mpi-library Intel MPI Library]).
 
The corresponding Intel compiler module is loaded automatically (if not done before).
 
Compiler and MPI module must fit. Don't mix incongruous versions!
=== Usage ===
The following table lists available MPI compiler commands and the underlying compilers, compiler families, languages, and application binary interfaces (ABIs) that they support.
 
The Intel MPI Library Compiler Drivers
{| width=600px class="wikitable"
|-
! Compiler Command !! Default Compiler !! Supported Language(s) !! Supported ABI's
|-
| colspan=4 style="background-color:#DCDCDC;" | Generic Compilers
|-
| mpicc || gcc, cc || C || 32/64 bit
|-
| mpicxx || g++ || C/C++ || 32/64 bit
|-
| mpifc || gfortran || Fortran77/Fortran 95 || 32/64 bit
|-
| colspan=4 style="background-color:#DCDCDC;" | [[GCC|GNU Compiler]] Versions 3 and higher
|-
| mpigcc || gcc || C || 32/64 bit
|-
| mpigxx || g++ || C/C++ || 32/64 bit
|-
| mpif77 || g77 || Fortran 77 || 32/64 bit
|-
| mpif90 || gfortran || Fortran 95 || 32/64 bit
|-
| colspan=4 style="background-color:#DCDCDC;" | [[Intel_Compiler|'''Intel Fortran, C++ Compilers''']] Versions 13.1 through 14.0 and Higher
|-
| mpiicc || icc || C || 32/64 bit
|-
| mpiicpc || icpc || C++ || 32/64 bit
|-
|impiifort || ifort || Fortran77/Fortran 95 || 32/64 bit
|-
|}
* Compiler commands are available only in the Intel MPI Library Development Kit.
* Compiler commands are in the <installdir>/<arch>/bin directory. Where <installdir> refers to the Intel MPI Library installation directory (depending on the loaded mpi module) and <arch> is one of the following architectures:
* ia32 - IA-32 architecture
* intel64 - Intel 64 architecture
* mic – Intel Xeon Phi™ Coprocessor architecture
* Ensure that the corresponding underlying compilers (32-bit or 64-bit, as appropriate) are already in your PATH. This is normally done by the 'module load' command.
* To port existing MPI-enabled applications to the Intel MPI Library, recompile all sources.
* To display mini-help of a compiler command, execute it without any parameters.

=== Further information ===
==== Intel MPI without multithreading on the bwUniCluster ====
* [[Batch_Jobs_-_bwUniCluster_Features#Intel_MPI_without_Multithreading|Additional Infos about Intel MPI without Multithreading]]
==== Sample MPI programm in "C" ====
The following example code includes a sample MPI program written in C.
<source lang="c">
#******************************************************************************
# Content: (exemplify version)
# Based on a Monto Carlo method, this MPI sample code uses volumes to
# estimate the number PI.
#*****************************************************************************/
#include <stdlib.h>
#include <stdio.h>
#include <math.h>
#include <time.h>
#include <math.h>

// here you'l include the main MPI-library
#include "mpi.h"

#define MASTER 0
#define TAG_HELLO 4
#define TAG_TEST 5
#define TAG_TIME 6

int main(int argc, char *argv[]) {
int i, id, remote_id, num_procs;

MPI_Status stat;
int namelen;
char name[MPI_MAX_PROCESSOR_NAME];

// Start MPI.
if (MPI_Init (&argc, &argv) != MPI_SUCCESS) {
printf ("Failed to initialize MPIn");
return (-1);
}
// Create the communicator, and retrieve the number of processes.
MPI_Comm_size (MPI_COMM_WORLD, &num_procs);

// Determine the rank of the process.
MPI_Comm_rank (MPI_COMM_WORLD, &id);

// Get machine name
MPI_Get_processor_name (name, &namelen);

if (id == MASTER) {
printf ("Hello world: rank %d of %d running on %sn", id, num_procs, name);

for (i = 1; i<num_procs; i++) {
MPI_Recv (&remote_id, 1, MPI_INT, i, TAG_HELLO, MPI_COMM_WORLD, &stat);
MPI_Recv (&num_procs, 1, MPI_INT, i, TAG_HELLO, MPI_COMM_WORLD, &stat);
MPI_Recv (&namelen, 1, MPI_INT, i, TAG_HELLO, MPI_COMM_WORLD, &stat);
MPI_Recv (name, namelen+1, MPI_CHAR, i, TAG_HELLO, MPI_COMM_WORLD, &stat);

printf ("Hello world: rank %d of %d running on %sn", remote_id, num_procs, name);
}
}
else {
MPI_Send (&id, 1, MPI_INT, MASTER, TAG_HELLO, MPI_COMM_WORLD);
MPI_Send (&num_procs, 1, MPI_INT, MASTER, TAG_HELLO, MPI_COMM_WORLD);
MPI_Send (&namelen, 1, MPI_INT, MASTER, TAG_HELLO, MPI_COMM_WORLD);
MPI_Send (name, namelen+1, MPI_CHAR, MASTER, TAG_HELLO, MPI_COMM_WORLD);
}

// Rank 0 distributes seek randomly to all processes.
double startprocess, endprocess;

int distributed_seed = 0;
int *buff;

buff = (int *)malloc(num_procs * sizeof(int));

unsigned int MAX_NUM_POINTS = pow (2,32) - 1;
unsigned int num_local_points = MAX_NUM_POINTS / num_procs;

if (id == MASTER) {
srand (time(NULL));

for (i=0; i<num_procs; i++) {
distributed_seed = rand();
buff[i] = distributed_seed;
}
}
// Broadcast the seed to all processes
MPI_Bcast(buff, num_procs, MPI_INT, MASTER, MPI_COMM_WORLD);

// At this point, every process (including rank 0) has a different seed. Using their seed,
// each process generates N points randomly in the interval [1/n, 1, 1]
startprocess = MPI_Wtime();

srand (buff[id]);

unsigned int point = 0;
unsigned int rand_MAX = 128000;
float p_x, p_y, p_z;
float temp, temp2, pi;
double result;
unsigned int inside = 0, total_inside = 0;

for (point=0; point<num_local_points; point++) {
temp = (rand() % (rand_MAX+1));
p_x = temp / rand_MAX;
p_x = p_x / num_procs;

temp2 = (float)id / num_procs; // id belongs to 0, num_procs-1
p_x += temp2;

temp = (rand() % (rand_MAX+1));
p_y = temp / rand_MAX;

temp = (rand() % (rand_MAX+1));
p_z = temp / rand_MAX;

// Compute the number of points residing inside of the 1/8 of the sphere
result = p_x * p_x + p_y * p_y + p_z * p_z;

if (result <= 1) inside++;
}

double elapsed = MPI_Wtime() - startprocess;

MPI_Reduce (&inside, &total_inside, 1, MPI_UNSIGNED, MPI_SUM, MASTER, MPI_COMM_WORLD);

#if DEBUG
printf ("rank %d counts %u points inside the spheren", id, inside);
#endif
if (id == MASTER) {
double timeprocess[num_procs];

timeprocess[MASTER] = elapsed;
printf("Elapsed time from rank %d: %10.2f (sec) n", MASTER, timeprocess[MASTER]);
for (i=1; i<num_procs; i++) {
// Rank 0 waits for elapsed time value
MPI_Recv (&timeprocess[i], 1, MPI_DOUBLE, i, TAG_TIME, MPI_COMM_WORLD, &stat);
printf("Elapsed time from rank %d: %10.2f (sec) n", i, timeprocess[i]);
}

temp = 6 * (float)total_inside;
pi = temp / MAX_NUM_POINTS;
printf ( "Out of %u points, there are %u points inside the sphere => pi=%16.12fn",
MAX_NUM_POINTS, total_inside, pi);
}
else {
// Send back the processing time (in second)
MPI_Send (&elapsed, 1, MPI_DOUBLE, MASTER, TAG_TIME, MPI_COMM_WORLD);
}

free(buff);

// Terminate MPI.
MPI_Finalize();

return 0;
}</source>

Development/Parallel Programming

2017-11-27T13:56:05Z

T Kienzle:

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| mpi/impi | mpi/openmpi
|-
| Links
| [https://software.intel.com/en-us/intel-mpi-library Intel® MPI Library] | [http://www.open-mpi.org/ Open MPI]
|-
| License
| [https://software.intel.com/en-us/articles/intel-mpi-library-licensing-faq Intel MPI Library Licensing FAQ] install-doc/EULA.txt | [http://www.open-mpi.org/community/license.php Open MPI License]
|}
 
= Introduction =
This page will provide information regarding the supported parallel programming paradigms and specific hints on their usage.
Please refer to the [[BwUniCluster_Environment_Modules|Modules Documentation]] how to setup your environment on bwUniCluster to load a specific software installation.
 
More informations about Compilers installed at our clusters is available here:
* [[Intel_Compiler|Intel Compiler Suite]]
* [[GCC|GNU Compiler (GCC)]]
* [[General_compiler_usage|General Compiler Usage (incl. PGI Compiler)]]

= Versions and Availability =
=== impi (Intel) ===
A list of versions currently available compilers on the bwHPC-C5-Clusters can be obtained from the
 
<big>
 
[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]
 </big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/mpi/impi
|width=99%
|height=1200
}}

=== openmpi ===
A list of versions currently available compilers on the bwHPC-C5-Clusters can be obtained from the
 
<big>
 
[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]
 </big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/mpi/openmpi
|width=99%
|height=2100
}}
 

= OpenMP =
== General Information ==
OpenMP is a mature specification [http://openmp.org/wp/openmp-specifications/] to allow easy, portable, and most importantly incremental node-level parallelisation of code.
Being a thread-based approach, OpenMP is aimed at more fine-grained parallelism than [[BwHPC_Best_Practices_Repository#MPI|MPI]].
Although there have been extensions to extend OpenMP for inter-node parallelisation, it is a node-level approach aimed to make best usage of a node's cores.
 
With regard to ease-of-use, OpenMP is ahead of any other common approach: the source-code is annotated using <tt>#pragma omp</tt> or <tt>!$omp</tt> statements, in C/C++ and Fortran respectively.
Whenever the compiler encompasses a semantic block of code encapsulated in a parallel region, this block of code is transparently compiled into a function, which is passed to a so-called team-of-threads upon entering this semantic block. This fork-join model of execution eases a lot of the programmer's pain involved with Threads.
Being a loop-centric approach, OpenMP is aimed at codes with long/time-consuming loops.
A single combined directive <tt>pragma omp parallel for</tt> will tell the compiler to automatically parallel the ensuing for-loop.
 
The following example is a bit more advanced in that even reductions of variables over multiple threads are easily to parallel:
<source lang="c">
for (int i=0, sum = 0.0; i < VECTOR_LEN; i++)
norm2 += (v[i]*v[i]);
</source>
is parallelized by just adding a single line as in:
<source lang="c">
# pragma omp parallel for reduction(+:norm2)
for (int i=0, sum = 0.0; i < VECTOR_LEN; i++)
norm2 += (v[i]*v[i]);
</source>
With <tt>VECTOR_LENGTH</tt> being large enough, this piece of code compiled with OpenMP will run in parallel, exhibiting very nice speedup.
Compiled without, the code remains as is. Developpers may therefore incrementally parallelize their application based on the profile derived from performance analysis tools, starting with the most time-consuming loops.
Using OpenMP's concise API, one may query the number of running threads, the number of processors, a time to calculate runtime, and even set parameters such as the number of threads to execute a parallel region.
 
The OpenMP-4.0 specification added support for the SIMD-directive to better utilize SIMD-vectorization, as well as integrating directives to offload computation to accelerators using the <tt>target</tt> directive: these are integrated into the Intel Compiler and are actively being worked on for the GNU compiler, some restrictions may apply.
== OpenMP Best Practice Guide ==
The following silly example to calculate the squared Euklidian Norm shows some techniques:
<source lang="c">
#include <stdlib.h>
#include <stdio.h>
#include <omp.h>

#define VECTOR_LENGTH 5

int main (int argc, char * argv[])
{
int len = VECTOR_LENGTH;
int i;
double * v;
double norm2 = 0.0;
double t1, tdiff;

if (argc > 1)
len = atoi (argv[1]);
v = malloc (len * sizeof(double));

t1 = omp_get_wtime();
// Initialization already with (the same number of) threads
#pragma omp parallel for
for (i=0; i < len; i++) {
v[i] = i;
}

// Now aggregate the sum-of-squares by specifying a reduction
#pragma omp parallel for reduction(+:norm2)
for(i=0; i < len; i++) {
norm2 += (v[i]*v[i]);
}
tdiff = omp_get_wtime() - t1;

printf ("norm2: %f Time:%f\n", norm2, tdiff);
return 0;
}
</source>


* Group independent parallel sections together: in the above example, You may combine those two sections into one larger parallel block. This will just once enter the parallel region (in the fork-join model) instead of twice. Especially in inner loops, this will considerably decrease overhead.
* Compile with the Intel compiler's option <tt>-diag-enable sc-parallel3</tt> to get the further warnings on thread-safety, performance, etc. The following code with loop-carried dependency will e.g. compile fine (aka without warning):
<source lang="c">
#pragma omp parallel for reduction(+:norm2)
for(i=1; i < len-1; i++) {
v[i] = v[i-1]+v[i+1];
}
</source>
However the Intel compiler with <tt>-diag-enable sc-parallel3</tt> will produce the following warning:
<tt>warning #12246: variable "v" has loop carried data dependency that may lead to incorrect program execution in parallel mode; see (file:omp_norm2.c line:32)</tt>
* Always specify <tt>default(none)</tt> on larger parallel regions in order to specifically set the visibility of variables to either <tt>shared</tt> or <tt>private</tt>.
* Try to restructure code to allow for <tt>nowait</tt>: OpenMP defines synchronization points (implied barriers) at the end of work sharing constructs such as the <tt>pragma omp for<tt> directive. If the ensuing section of code does not depend on data being generated inside the parallel section, adding the <tt>nowait</tt> clause to the worksharing directive allows the compiler to eliminate this synchronization point. This reduces overhead, allows for better overlap and better utilization of the processor's resources. This might imply however to restructure the code (move portions of independent code in between dependent works-sharing constructs).
== Usage ==
OpenMP is supported by various compilers, here the usage for two main compilers [[BwHPC_BPG_Compiler#GCC|GCC]] and [[BwHPC_BPG_Compiler#Intel Suite|Intel Suite]] are introduced.
For both compilers, You first need to turn on OpenMP support by specifying a parameter on the compiler's command-line.
In case You make function calls to OpenMP's API, You also need to include the header-file <tt>omp.h</tt>.
OpenMP's API allows to query or set the number of threads, query the number of processors, get a wall-clock time to measure execution times, etc.
=== OpenMP with GNU Compiler Collection ===
Starting with version 4.2 the gcc compiler supports OpenMP-2.5.
Since then the analysis capabilities of the GNU compiler have steadily improved.
The installed compilers support OpenMP-3.1.

 
To use OpenMP with the gcc-compiler, pass <tt>-fopenmp</tt> as parameter.
=== OpenMP with Intel Compiler ===
The Intel Compiler's support for OpenMP is more advanced than gcc's -- especially in term of programmer support.
To use OpenMP with the Intel compiler, pass <tt>-openmp</tt> as command-line parameter.
 
One may get very insightful information about OpenMP, when compiling with
* Compiling with <tt>-openmp-report2</tt> to get information, which loops were parallelized and a reason why not.
* Compiling with <tt>-diag-enable sc-parallel3</tt> to get errors and warnings about your sources weaknesses with regard to parallelization (see example below).
 
 

= MPI =
In this section, You will find information regarding the supported installations of the Message-Passing Interface libraries and their usage. 
Due to the Fortran interface ABI, all MPI-libraries are normally bound to a specific compiler-vendor and even the specific compiler version.
Therefore, as listed in [[BwHPC_BPG_Compiler]] two compilers are supported on bwUniCluster: [[GCC|GCC]] and [[Intel_Compiler|Intel Compiler Suite]].
As both compilers are continously improving, the communication libraries will be adopted in lock-step.
 
With a set of different implementations, there comes the problem of choice. These pages should inform the user of the communication libraries, what considerations should be done with regard to performance, maintainability and debugging -- in general tool support -- of the various implementations.
== MPI Introduction ==
The Message-Passing Interface is a standard provided by the [http://www.mpi-forum.org MPI-Forum] which regularly convenes for the [http://mpi-forum.org/meetings/ MPI-Forum Meetings] to update this standard. The current version is MPI-3.0 available as [http://mpi-forum.org/docs/mpi-3.0/mpi30-report.pdf PDF].
This document defines the API of over 300 functions for the C- and the Fortran-language -- however, You will certainly not need all of them to begin with.
 
Every MPI-conforming program needs to call <tt>MPI_Init()</tt> and <tt>MPI_Finalize()</tt> upon start and shutdown -- or <tt>MPI_Abort()</tt> in case of an abnormal termination.
After initialization the programm may call any other MPI-function, specifically communication functions.
However to do so, it is required to find out how many processes the program has been started with, using <tt>MPI_Comm_size()</tt> and what number (here called a rank= this particular process has using <tt>MPI_Comm_rank()</tt>.
Communication is always relative to a so-called communicator -- the default one after initialization being called <tt>MPI_COMM_WORLD</tt>-
 
There's basically three ways of communication:
* two-sided communication using point-to-point (often abbreviated P2P) functions, such as <tt>MPI_Send()</tt> and <tt>MPI_Recv()</tt>, which always involves two participating processes,
* collectice communcation functions (often abbreviated as colls) involve multiple processes, examples are <tt>MPI_Bcast()</tt> and <tt>MPI_Reduce()</tt>,
* one-sided communication, where communication between two processes is initiated by one-process, only. With proper RMA-hardware support and careful programming, this may allow higher performance or scalibility.
 
All parts of the programm, which reference MPI functionality need to be compiled with the '''same''' compiler settings/include files and linked to the same MPI-Library. This is stressed here, since without taking pre-cautions, a different MPI's header may be included, resulting in funny errors: consider that Intel MPI is derived from MPIch, with MPI-datatypes being C <tt>int</tt>s, while Open MPI uses pointers to structures (the former being 4, the latter being 8 bytes on bwUniCluster).
To ease the programmer's life, MPI implementations offer compiler-wrappers, e.g. <tt>mpicc</tt> for C and <tt>mpif90</tt> for Fortran90 for compilation and linking, taking care to include all required libraries.
 
All programs must be started using the <tt>mpirun</tt> or <tt>mpiexec</tt> command. Depending on the actual implementation, it uses different arguments, however the following works with any MPI:
* <tt>mpirun -np 128 ./app</tt> starts 128 processes (with ranks 0 to 127)
* <tt>mpiexec -n 128 -hostfile mynodes.txt ./app</tt> starts 128 processes on only the nodes listed line-by-line in the provided text-file <tt>mynodes.txt</tt>.
* <tt>mpiexec -n 64 ./app1 : -n 64 ./app2</tt> starts 128 processes, 64 of which execute <tt>app1</tt>, the other 64 execute <tt>app2</tt>. All processes however participate in the same <tt>MPI_COMM_WORLD</tt> and therefore must accordingly take care about their respective ranks.
Please note, that process placement (e.g. a round-robin scheme), and specifically process-binding to sockets is MPI-implementation dependant.
== MPI Best Practice Guide ==
Specific performance considerations with regard to MPI (independent of the implementation):
* No communication at all is best: Only communicate between processes if at all necessary. Consider that file-access is "communication" as well.
* If communication is done with multiple processes, try to involve as many processes in just one call: MPI optimizes the communication pattern for so-called "collective communication" to take advantage of the underlying network (with regard to network topology, message sizes, queueing capabilities of the network interconnect, etc.). Therefore try to always think in collective communication, if a communication pattern involves a group of processes.
* Try to group processes together: Function calls like <tt>MPI_Cart_create</tt> will come in handy for applications with cartesian domains but also general communicators derived from <tt>MPI_COMM_WORLD</tt> using <tt>MPI_Comm_split()</tt> may benefit by MPI's knowing the underlying network topology. Use MPI3's <tt>MPI_Comm_split_type()</tt> with <tt>MPI_COMM_TYPE_SHARED</tt> for a sub-communicator with processes having access to the same shared memory region (aka on bwUniCluster the same node).
* File-accesses to load / store data '''must''' be done collectively: Writing to storage, or even reading the initialization data -- all of which involves getting data from/to all MPI processes -- must be done collectively. MPI's Parallel IO offers a rich API to read and distribute the data access -- in order to take advantage of parallel filesystems like Lustre. A many-fold performance improvement may be seen by writing data in large chunks in collective fashion -- and at the same time being nice to other users and applications.
* Try to hide the communication by computation: Try to hide (some) of the cost of communication of Point-to-point communication by using non-blocking / immediate P2P-calls (<tt>MPI_Isend</tt> and <tt>MPI_Irecv</tt> et al, followed by <tt>MPI_Wait</tt> or <tt>MPI_Test</tt> et al). This may allow the MPI-implementation to initiate or even offload communication to the network interconnect and resume executing your application, while data is being transferred. MPI-3 adds non-blocking collectives, e.g. <tt>MPI_Ibcast()</tt> or <tt>MPI_Iallreduce()</tt>. For extra credit, explain the use-cases of <tt>MPI_Ibarrier()</tt>.
* Every call to MPI may trigger an access to physical hardware -- limit it: When calling communication-related functions such as <tt>MPI_Test</tt> to check whether a specific communication has finished, the queue of the network adapter may need to be queried. This memory access or even physical hardware access to query the state will cost cycles. Therefore, the programmer should combine multiple requests with functions such as <tt>MPI_Waitall()</tt> or <tt>MPI_Waitany()</tt> or their Test*-counterparts.
* Make usage of derived datatypes: instead of manually copying data into temporary, even newly allocated memory, describe the data-layout to MPI -- and let the implementation, or even the network HCA's hardware do the data fetching.
* Bind Your processes to sockets: Operating Systems are good in making best use of the ressources -- which sometimes involves moving tasks from one core to another, or even (though more unlikely since the OS' heuristics try to avoid it) to another socket, with the obvious effects: Caches are cold, every memory access to memory allocated on the previous socket "has to travel the bus". This is particularly happening if You have multiple OpenMP parallel regions which are separated by code that does IO -- and threads are sleeping -- the processes doing IO may wander to a different socket... Bind Your processes to at least the socket. All major MPIs support this binding (see below).
* Do not use the C++ interface: First of all, it has been marked as deprecated in the MPI-3.0 standard, since it added little benefit to C++ programmers over the C-interface. Moreover, since MPI implementations are written in C, the interface adds another level of indirection and therefore a bit of overhead in terms of instructions and Cache misses.
== Open MPI ==
The [http://www.open-mpi.org Open MPI] library is an open, flexible and nevertheless performant implementation of MPI-2 and MPI-3. Licensed under BSD, it is being actively developed by an open community of industry and research institutions.
The flexibility comes in handy: using the concept of a [http://www.open-mpi.org/faq/?category=tuning#mca-def MCA] (aka a plugin) Open MPI supports many different network interconnects (Infinband, TCP, Cray, etc.) , on the other hand, a installation may be tailored to suite an installation, e.g. the network (Infiniband with specific settings), the main startup-mechanism, etc.
Furthermore, the [http://www.open-mpi.org/faq/ FAQ] offers hints on [http://www.open-mpi.org/faq/?category=tuning performance tuning].
=== Usage ===
Like other MPI implementations, after loading the [[BwUniCluster_Environment_Modules|module]], Open MPI provides the compiler-wrappers <tt>mpicc</tt>, <tt>mpicxx</tt> and <tt>mpifort</tt> (or for
versions lower than 1.7 <tt>mpif77</tt> and <tt>mpif90</tt>) for the C-, C++ and Fortran compilers respectively. Albeit their usage is not required, these wrappers are handy to not have to use the command-line options for header- or library directories, aka <tt>-I</tt> and <tt>-L</tt>, as well as the actual needed MPI-libraries itselve.
=== Further information ===
Open MPI also features a few specific functionalities that will help users and developpers, alike:
* Open MPI's tool <tt>ompi_info</tt> allows seeing all of Open MPI's installed MCA components and their specific options.
Without any option the user gets a list of flags, the Open MPI installation was compiled for (version of compilers, specific configure-flags, e.g. debugging, or profiling options). Furthermore, using <tt>ompi_info --param all all</tt> one may see all of the MCA's options, e.g. that the default PML-MCA uses an initial free-list of 4 blocks (increased by 64 upon first encountering this limit):
<tt>ompi_info --param ob1 all</tt> -- which may be increased for applications that are certain to benefit from a larger value upon startup.
* Open MPI allows adapting MCA parameters on the command-line: parameters may be supplied, e.g. the above-mentioned parameter <tt>mpirun -np 128 --mca mpirun -np 16 --mca pml_ob1_free_list_num 128 ./mpi_stub</tt>.
* Open MPI internally uses the tool [http://www.open-mpi.org/projects/hwloc/ hwloc] for node-local processor-information, as well as process- and memory-affinity. This tool also is a good tool to get information on the node's processor topology and Cache-information. This may be used to optimize and balance memory usage or for choosing a better ratio of MPI processes per node vs. OpenMP threads per core.
== Intel (i)MPI ==
The Intel MPI Library is a multi-fabric message passing library that implements the Message Passing Interface, v2.2 (MPI-2.2) specification. It provides a standard library across Intel
platforms that enable adoption of MPI-2.2 functions as their needs dictate.
 
The Intel MPI Library enables developers to change or to upgrade processors and interconnects as new technology becomes available without changes to the software or to the operating environment.
 
The library is provided in the following kits:
* The Intel MPI Library Runtime Environment (RTO) has the tools you need to run programs, including Multipurpose Daemon (MPD), Hydra and supporting utilities, shared (.so) libraries, and documentation.
* The Intel MPI Library Development Kit (SDK) includes all of the Runtime Environment components plus compilation tools, including compiler commands such as mpiicc, include files and modules, static (.a) libraries, debug libraries, trace libraries, and test codes.
=== General information ===
All Intel mpi-modules are called ''''mpi/impi''''.
 
These modules provides the Intel Message Passing Interface (mpicc, mpicxx, mpif77
and mpif90) for the Intel compiler suite (icc, icpc and ifort) (see also [http://software.intel.com/en-us/intel-mpi-library Intel MPI Library]).
 
The corresponding Intel compiler module is loaded automatically (if not done before).
 
Compiler and MPI module must fit. Don't mix incongruous versions!
=== Usage ===
The following table lists available MPI compiler commands and the underlying compilers, compiler families, languages, and application binary interfaces (ABIs) that they support.
 
The Intel MPI Library Compiler Drivers
{| width=600px class="wikitable"
|-
! Compiler Command !! Default Compiler !! Supported Language(s) !! Supported ABI's
|-
| colspan=4 style="background-color:#DCDCDC;" | Generic Compilers
|-
| mpicc || gcc, cc || C || 32/64 bit
|-
| mpicxx || g++ || C/C++ || 32/64 bit
|-
| mpifc || gfortran || Fortran77/Fortran 95 || 32/64 bit
|-
| colspan=4 style="background-color:#DCDCDC;" | [[GCC|GNU Compiler]] Versions 3 and higher
|-
| mpigcc || gcc || C || 32/64 bit
|-
| mpigxx || g++ || C/C++ || 32/64 bit
|-
| mpif77 || g77 || Fortran 77 || 32/64 bit
|-
| mpif90 || gfortran || Fortran 95 || 32/64 bit
|-
| colspan=4 style="background-color:#DCDCDC;" | [[Intel_Compiler|'''Intel Fortran, C++ Compilers''']] Versions 13.1 through 14.0 and Higher
|-
| mpiicc || icc || C || 32/64 bit
|-
| mpiicpc || icpc || C++ || 32/64 bit
|-
|impiifort || ifort || Fortran77/Fortran 95 || 32/64 bit
|-
|}
* Compiler commands are available only in the Intel MPI Library Development Kit.
* Compiler commands are in the <installdir>/<arch>/bin directory. Where <installdir> refers to the Intel MPI Library installation directory (depending on the loaded mpi module) and <arch> is one of the following architectures:
* ia32 - IA-32 architecture
* intel64 - Intel 64 architecture
* mic – Intel Xeon Phi™ Coprocessor architecture
* Ensure that the corresponding underlying compilers (32-bit or 64-bit, as appropriate) are already in your PATH. This is normally done by the 'module load' command.
* To port existing MPI-enabled applications to the Intel MPI Library, recompile all sources.
* To display mini-help of a compiler command, execute it without any parameters.

=== Further information ===
==== Intel MPI without multithreading on the bwUniCluster ====
* [[Batch_Jobs_-_bwUniCluster_Features#Intel_MPI_without_Multithreading|Additional Infos about Intel MPI without Multithreading]]
==== Sample MPI programm in "C" ====
The following example code includes a sample MPI program written in C.
<source lang="c">
#******************************************************************************
# Content: (exemplify version)
# Based on a Monto Carlo method, this MPI sample code uses volumes to
# estimate the number PI.
#*****************************************************************************/
#include <stdlib.h>
#include <stdio.h>
#include <math.h>
#include <time.h>
#include <math.h>

// here you'l include the main MPI-library
#include "mpi.h"

#define MASTER 0
#define TAG_HELLO 4
#define TAG_TEST 5
#define TAG_TIME 6

int main(int argc, char *argv[]) {
int i, id, remote_id, num_procs;

MPI_Status stat;
int namelen;
char name[MPI_MAX_PROCESSOR_NAME];

// Start MPI.
if (MPI_Init (&argc, &argv) != MPI_SUCCESS) {
printf ("Failed to initialize MPIn");
return (-1);
}
// Create the communicator, and retrieve the number of processes.
MPI_Comm_size (MPI_COMM_WORLD, &num_procs);

// Determine the rank of the process.
MPI_Comm_rank (MPI_COMM_WORLD, &id);

// Get machine name
MPI_Get_processor_name (name, &namelen);

if (id == MASTER) {
printf ("Hello world: rank %d of %d running on %sn", id, num_procs, name);

for (i = 1; i<num_procs; i++) {
MPI_Recv (&remote_id, 1, MPI_INT, i, TAG_HELLO, MPI_COMM_WORLD, &stat);
MPI_Recv (&num_procs, 1, MPI_INT, i, TAG_HELLO, MPI_COMM_WORLD, &stat);
MPI_Recv (&namelen, 1, MPI_INT, i, TAG_HELLO, MPI_COMM_WORLD, &stat);
MPI_Recv (name, namelen+1, MPI_CHAR, i, TAG_HELLO, MPI_COMM_WORLD, &stat);

printf ("Hello world: rank %d of %d running on %sn", remote_id, num_procs, name);
}
}
else {
MPI_Send (&id, 1, MPI_INT, MASTER, TAG_HELLO, MPI_COMM_WORLD);
MPI_Send (&num_procs, 1, MPI_INT, MASTER, TAG_HELLO, MPI_COMM_WORLD);
MPI_Send (&namelen, 1, MPI_INT, MASTER, TAG_HELLO, MPI_COMM_WORLD);
MPI_Send (name, namelen+1, MPI_CHAR, MASTER, TAG_HELLO, MPI_COMM_WORLD);
}

// Rank 0 distributes seek randomly to all processes.
double startprocess, endprocess;

int distributed_seed = 0;
int *buff;

buff = (int *)malloc(num_procs * sizeof(int));

unsigned int MAX_NUM_POINTS = pow (2,32) - 1;
unsigned int num_local_points = MAX_NUM_POINTS / num_procs;

if (id == MASTER) {
srand (time(NULL));

for (i=0; i<num_procs; i++) {
distributed_seed = rand();
buff[i] = distributed_seed;
}
}
// Broadcast the seed to all processes
MPI_Bcast(buff, num_procs, MPI_INT, MASTER, MPI_COMM_WORLD);

// At this point, every process (including rank 0) has a different seed. Using their seed,
// each process generates N points randomly in the interval [1/n, 1, 1]
startprocess = MPI_Wtime();

srand (buff[id]);

unsigned int point = 0;
unsigned int rand_MAX = 128000;
float p_x, p_y, p_z;
float temp, temp2, pi;
double result;
unsigned int inside = 0, total_inside = 0;

for (point=0; point<num_local_points; point++) {
temp = (rand() % (rand_MAX+1));
p_x = temp / rand_MAX;
p_x = p_x / num_procs;

temp2 = (float)id / num_procs; // id belongs to 0, num_procs-1
p_x += temp2;

temp = (rand() % (rand_MAX+1));
p_y = temp / rand_MAX;

temp = (rand() % (rand_MAX+1));
p_z = temp / rand_MAX;

// Compute the number of points residing inside of the 1/8 of the sphere
result = p_x * p_x + p_y * p_y + p_z * p_z;

if (result <= 1) inside++;
}

double elapsed = MPI_Wtime() - startprocess;

MPI_Reduce (&inside, &total_inside, 1, MPI_UNSIGNED, MPI_SUM, MASTER, MPI_COMM_WORLD);

#if DEBUG
printf ("rank %d counts %u points inside the spheren", id, inside);
#endif
if (id == MASTER) {
double timeprocess[num_procs];

timeprocess[MASTER] = elapsed;
printf("Elapsed time from rank %d: %10.2f (sec) n", MASTER, timeprocess[MASTER]);
for (i=1; i<num_procs; i++) {
// Rank 0 waits for elapsed time value
MPI_Recv (&timeprocess[i], 1, MPI_DOUBLE, i, TAG_TIME, MPI_COMM_WORLD, &stat);
printf("Elapsed time from rank %d: %10.2f (sec) n", i, timeprocess[i]);
}

temp = 6 * (float)total_inside;
pi = temp / MAX_NUM_POINTS;
printf ( "Out of %u points, there are %u points inside the sphere => pi=%16.12fn",
MAX_NUM_POINTS, total_inside, pi);
}
else {
// Send back the processing time (in second)
MPI_Send (&elapsed, 1, MPI_DOUBLE, MASTER, TAG_TIME, MPI_COMM_WORLD);
}

free(buff);

// Terminate MPI.
MPI_Finalize();

return 0;
}</source>

BWUniCluster User Access Members Uni Mannheim

2017-06-27T13:16:49Z

T Kienzle:

= Entitlement for bwUniCluster and bwFileStorage @ Uni Mannheim =

Members of the University of Mannheim need to apply with a valid account for an Entitlement to use bwHPC resources.
Therefor please use this web registration service:

[https://sp-grid-webregistration.uni-mannheim.de https://sp-grid-webregistration.uni-mannheim.de]

The web registration service asks for a project description which is used for internal documentation only. Please also indicate that you want access to '''bwUniCluster''' or '''bwFileStorage'''.

With the approval of your request for the HPC Entitlement you will be able to register for '''bwUniCluster''' and '''bwFileStorage''' too.

In case of questions contact [mailto:hpc-support@mailman.uni-mannheim.de hpc-support@mailman.uni-mannheim.de]

BWUniCluster User Access Members Uni Mannheim

2017-06-27T13:16:08Z

T Kienzle: /* Entitlement for bwUniCluster and bwFileStorage */

= Entitlement for bwUniCluster and bwFileStorage @ Uni Mannheim =

Members of the University of Mannheim need to apply with a valid account for an Entitlement to use bwHPC resources.
Therefor please use this web registration service:

[https://sp-grid-webregistration.uni-mannheim.de https://sp-grid-webregistration.uni-mannheim.de]

The web registration service asks for a project description which is used for internal documentation only. Please also indicate that you want access to bwUniCluster.

With the approval of your request for the HPC Entitlement you will be able to register for bwUniCluster and bwFileStorage too.

In case of questions contact [mailto:hpc-support@mailman.uni-mannheim.de hpc-support@mailman.uni-mannheim.de]

BWUniCluster User Access Members Uni Mannheim

2017-06-27T13:15:27Z

T Kienzle:

= Entitlement for bwUniCluster and bwFileStorage =

Members of the University of Mannheim need to apply with a valid account for an Entitlement to use bwHPC resources.
Therefor please use this web registration service:

[https://sp-grid-webregistration.uni-mannheim.de https://sp-grid-webregistration.uni-mannheim.de]

The web registration service asks for a project description which is used for internal documentation only. Please also indicate that you want access to bwUniCluster.

With the approval of your request for the HPC Entitlement you will be able to register for bwUniCluster and bwFileStorage too.

In case of questions contact [mailto:hpc-support@mailman.uni-mannheim.de hpc-support@mailman.uni-mannheim.de]

BwFileStorage User Access

2017-06-27T13:14:21Z

T Kienzle: /* Step A: bwLSDF-FileService entitlement for registration */

= Registration =

Granting access and issuing a user account for '''''bwFileStorage''''' requires the registration at the KIT service website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] (step B). However, this registration depends on the '''bwLSDF-FileService entitlement''' issued by your university (step A).
 

== Step A: bwLSDF-FileService entitlement for registration ==
Each university issues the bwLSDF-FileService entitlement '''only''' for their own members. Details on the issuing process and/or the entitlement application forms are listed hereafter:



* [[BWUniCluster_User_Access_Members_Uni_Mannheim|University of Mannheim]]

 

== Step B: Web Registration ==
After step A, i.e., after issueing the bwLSDF-FileService entitlement, please visit:
* [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
*# Select your home organization from the list and click '''Proceed'''
*# You will be directed to the ''Identity Provider'' of your home organisation
*# Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
*# You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
*# <div>Select unter '''The following services are available''' the service '''bwFileStorage'''
*# Click '''Register'''
*# Finally
*#* for all non-KIT members '''mandatorily''' set a service password for authentication on bwFileStorage
*#* for KIT members '''optionally''' set a service password for authentication on bwFileStorage


 

== Changing Password ==
By default, for KIT members your bwFileStorage '''password''' to log on matches that one of your KIT
account, while for all non-KIT members your bw '''password''' is that one you saved during the web registration (compare step 7 of chapter 1.2).
At any time, you can set a new bwFileStorage password via the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] by carrying out the following steps:
# visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and select your home organization
# authenticate yourself via your home-organizational user id / username and your home-organizational password
# find on the left side '''bwFileStorage''' and select '''Set Service Password'''
# set new service, i.e. bwFileStorage, password, repeat it and click '''Save''' button.
# the page answers e.g. "Das Passwort wurde bei dem Dienst geändert" ("password has been changed")

BwFileStorage User Access

2017-06-27T13:09:53Z

T Kienzle: Created page with "= Registration = Granting access and issuing a user account for '''''bwFileStorage''''' requires the registration at the KIT service website [https://bwidm.scc.kit.edu/ https..."

= Registration =

Granting access and issuing a user account for '''''bwFileStorage''''' requires the registration at the KIT service website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] (step B). However, this registration depends on the '''bwLSDF-FileService entitlement''' issued by your university (step A).
 

== Step A: bwLSDF-FileService entitlement for registration ==
Each university issues the bwLSDF-FileService entitlement '''only''' for their own members. Details on the issuing process and/or the entitlement application forms are listed hereafter:

* [[BwCluster_User_Access_Uni_Freiburg|Albert Ludwig University of Freiburg]]
* [http://www.zdv.uni-tuebingen.de/dienstleistungen/computing/zugang-zu-den-ressourcen.html Eberhard Karls University Tübingen]
* [http://www.scc.kit.edu/downloads/sdo/Antrag_Benutzernummer_bwUniCluster.pdf Karlsruhe Institute of Technology (KIT)]
* [[‎BWUniCluster_User_Access_Members_Uni_Heidelberg|Ruprecht-Karls-Universität Heidelberg]]
* [[BWUniCluster_User_Access_Members_Uni_Ulm|Ulm University]]
* [http://www.kim.uni-hohenheim.de/bwhpc University of Hohenheim]
* [[BWUniCluster_User_Access_Members_Uni_Konstanz|University of Konstanz]]
* [[BWUniCluster_User_Access_Members_Uni_Mannheim|University of Mannheim]]
* [https://www.hlrs.de/solutions-services/academic-users/bwunicluster-access/ University of Stuttgart]

 

== Step B: Web Registration ==
After step A, i.e., after issueing the bwLSDF-FileService entitlement, please visit:
* [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu]
*# Select your home organization from the list and click '''Proceed'''
*# You will be directed to the ''Identity Provider'' of your home organisation
*# Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
*# You will be redirected back to the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu/]
*# <div>Select unter '''The following services are available''' the service '''bwFileStorage'''
*# Click '''Register'''
*# Finally
*#* for all non-KIT members '''mandatorily''' set a service password for authentication on bwFileStorage
*#* for KIT members '''optionally''' set a service password for authentication on bwFileStorage


 

== Changing Password ==
By default, for KIT members your bwFileStorage '''password''' to log on matches that one of your KIT
account, while for all non-KIT members your bw '''password''' is that one you saved during the web registration (compare step 7 of chapter 1.2).
At any time, you can set a new bwFileStorage password via the registration website [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] by carrying out the following steps:
# visit [https://bwidm.scc.kit.edu/ https://bwidm.scc.kit.edu] and select your home organization
# authenticate yourself via your home-organizational user id / username and your home-organizational password
# find on the left side '''bwFileStorage''' and select '''Set Service Password'''
# set new service, i.e. bwFileStorage, password, repeat it and click '''Save''' button.
# the page answers e.g. "Das Passwort wurde bei dem Dienst geändert" ("password has been changed")

BWUniCluster User Access Members Uni Mannheim

2017-06-27T12:53:21Z

T Kienzle:

Members of the University of Mannheim need to apply with a valid account for an Entitlement to use bwHPC resources.
Therefor please use this web registration service:

[https://sp-grid-webregistration.uni-mannheim.de https://sp-grid-webregistration.uni-mannheim.de]

The web registration service asks for a project description which is used for internal documentation only. Please also indicate that you want access to bwUniCluster.

With the approval of your request for the HPC Entitlement you will be able to register for bwUniCluster and bwFileStorage too.

In case of questions contact [mailto:hpc-support@mailman.uni-mannheim.de hpc-support@mailman.uni-mannheim.de]

BWUniCluster User Access Members Uni Mannheim

2017-02-22T14:15:47Z

T Kienzle:

Members of the University of Mannheim need to apply with a valid account for an Entitlement to use bwHPC resources.
Therefor please use this web registration service:

[https://sp-grid-webregistration.uni-mannheim.de https://sp-grid-webregistration.uni-mannheim.de]

The web registration service asks for a project description which is used for internal documentation only. Please also indicate that you want access to bwUniCluster.

With the approval of your registration for bwGRiD Mannheim/Heidelberg you receive the entitlement to register for bwUniCluster, too.

In case of questions contact [mailto:hpc-support@mailman.uni-mannheim.de hpc-support@mailman.uni-mannheim.de]

Workspace

2016-12-20T16:55:52Z

T Kienzle:

'''Workspace tools''' provide temporary scratch space so calles '''workspaces''' for your calculation on a central file storage. They are meant to keep data for a limited time – but usually longer than the time of a single job run. It is not meant for permanent storage, hence data in workspaces is not backed up and may be lost in case of problems on the storage system. Please copy/move important results to $HOME or some disks outside the cluster.

== Create workspace ==
To create a workspace you need to state ''name'' of your workspace and ''lifetime'' in days. A maximum value for ''lifetime'' and a maximum number of renewals is defined on each cluster. Execution of:

$ ws_allocate blah 30

e.g. returns:

Workspace created. Duration is 720 hours.
Further extensions available: 3
/work/workspace/scratch/username-blah-0

For more information read the program's help, i.e. ''$ ws_allocate -h''.

== List all your workspaces ==
To list all your workspaces, execute:

$ ws_list

which will return:
* Workspace ID
* Workspace location
* available extensions
* creation date and remaining time

== Find workspace location ==

Workspace location/path can be prompted for any workspace ''ID'' using '''ws_find''', in case of workspace ''blah'':

$ ws_find blah

returns the one-liner:

/work/workspace/scratch/username-blah-0

== Extend lifetime of your workspace ==

Any workspace's lifetime can be only extended a cluster-specific number of times. There several commands to extend workspace lifetime
#<pre>$ ws_extend blah 40</pre> which extends workspace ID ''blah'' by ''40'' days from now,
#<pre>$ ws_extend blah</pre> which extends workspace ID ''blah'' by the number days used previously
#<pre>$ ws_allocate -x blah 40</pre> which extends workspace ID ''blah'' by ''40'' days from now.
 

== Delete a workspace ==

$ ws_release blah # Manually erase your workspace blah

BwUniCluster2.0/Software/R

2016-11-11T11:27:03Z

T Kienzle: /* Program Binaries */

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| math/R
|-
| License
| GPL
|-
| Citing
| n/a
|-
| Links
| [http://www.r-project.org/ Homepage] | [http://cran.r-project.org/manuals.html Documentation]
|-
| Graphical Interface
| No
|-
| Plugins
| User dependent
|}

= Description =
'''R''' is a language and environment for statistical computing and graphics. It is a GNU project which is similar to the S language and environment which was developed at Bell Laboratories (formerly AT&T, now Lucent Technologies) by John Chambers and colleagues. R can be considered as a different implementation of S. There are some important differences, but much code written for S runs unaltered under R.

R provides a wide variety of statistical (linear and nonlinear modelling, classical statistical tests, time-series analysis, classification, clustering, …) and graphical techniques, and is highly extensible. The S language is often the vehicle of choice for research in statistical methodology, and R provides an Open Source route to participation in that activity.

One of R’s strengths is the ease with which well-designed publication-quality plots can be produced, including mathematical symbols and formulae where needed. Great care has been taken over the defaults for the minor design choices in graphics, but the user retains full control.

R is available as Free Software under the terms of the Free Software Foundation’s GNU General Public License in source code form. It compiles and runs on a wide variety of UNIX platforms and similar systems (including FreeBSD and Linux), Windows and MacOS.

= Versions and Availability =
A list of versions currently available on all bwHPC-C5-Clusters can be obtained from the
 
<big>

[https://cis-hpc.uni-konstanz.de/prod.cis/ Cluster Information System CIS]

</big>
{{#widget:Iframe
|url=https://cis-hpc.uni-konstanz.de/prod.cis/bwUniCluster/math/R
|width=99%
|height=280
}}
 
On the command line interface of any bwHPC cluster, a list of the available R versions using
<pre>
$ module avail math/R
</pre>

= Usage =
The R installation also provides the standalone library libRmath. This library allows you to access R routines from your own C or C++ programs (see section 9 of the 'R Installation and Administration' manual.)

== Loading the module ==
You can load the default version of R with the command
<pre>
$ module load math/R
</pre>

The module will try to load modules it needs to function (e.g. compiler/intel). If loading the module fails, check if you have already loaded one of those modules, but not in the version needed for R.

If you wish to load a specific (older) version, you can do so using e.g.
<pre>
$ module load math/R/3.1.2
</pre>
to load the version 3.1.2.

== Program Binaries ==
Standard usage:

<pre>
Usage: R [options] [< infile] [> outfile]
R CMD command [arguments]

Example: R CMD BATCH script.R
</pre>

Executing R in batch mode:

<pre>
R CMD BATCH --no-save --no-restore <INPUT_FILE>.R
</pre>

For help run
<pre>
R --help
</pre>

For command help run
<pre>
R CMD command --help
</pre>

Further information and help
<pre>
Man pages: man R man Rscript
Info pages. e.g.: info R-intro info R-FAQ
Manuals: $R_DOC_DIR/manual
</pre>

== Multithreading in R ==

An easy way to use multiple cores on a single node in R is to use the [https://cran.r-project.org/web/packages/doParallel/vignettes/gettingstartedParallel.pdf| doParallel] package in combination with [https://cran.r-project.org/web/packages/foreach/vignettes/foreach.pdf| foreach].

= Examples =
As with all processes that require more than a few minutes to run, non-trivial compute jobs must be submitted to the cluster queuing system.

Example scripts are available in the directory $R_EXA_DIR:
<pre>
$ module show math/R # show environment variables, which will be available after 'module load'
$ module load math/R # load module
$ ls $R_EXA_DIR # show content of directory $R_EXA_DIR
$ cat $R_EXA_DIR/README # show examples README
</pre>

Run a first simple example job
<pre>
$ module load math/R # load module
$ mkdir Rtest # create test directory
$ cp $R_EXA_DIR/bwhpc-r.moab $R_EXA_DIR/fit.R Rtest/ # copy example files to test directory
$ cd Rtest/ # change to directory
$ nano bwhpc-r.moab # change job options, quit with 'CTRL+X'
$ msub bwhpc-r.moab # submit job
$ checkjob -v <JOBID> # check state of job
$ ls # when job finishes the results will be visible in this directory
</pre>

= Installing R-Packages into your home folder =
Since we cannot provide a software module for every R package, we recommend to install special R packages locally into your home folder. One possibility doing this is shown below:

<pre>
cp $HOME/.bashrc $HOME/.bashrc.backup # Make a backup copy of your bashrc
echo "export R_LIBS=\"${HOME}/R_libs\"" >> $HOME/.bashrc # Setting the environment variable R_LIBS permanently in your bashrc
source $HOME/.bashrc # Sourcing bashrc to make R_LIBS available
mkdir $R_LIBS # Create the R_libs folder in your HOME directory
module load math/R # Loading the matlab software module
R # Loading R
install.packages('package_name', repos="http://cran.r-project.org") # Installing your R package and the dependencies
library(package_name) # Loading the package into you R instance
</pre>

The package is now installed permanently in your home folder and is available every time you start R.

You can restore your old .bashrc if something goes wrong with:
<pre>
$ mv $HOME/.bashrc.backup $HOME/.bashrc # Restoring the original bashrc
</pre>

Installed packages can be deleted by deleting the folder ${HOME}/R_libs.

= Version-Specific Information =

For information specific to a single version, see the information available via the module system with the command
<pre>
$ module help math/R
</pre>

= Installed R plugins =
* [[CummeRbund_(R-package)|Bioinformatics: cummeRbund]]
 
 
----
[[Category:Mathematics software]][[Category:bwUniCluster]][[Category:BwForCluster_BinAC]][[Category:bwForCluster_MLS&WISO_Production]]

BwForCluster User Access Members Uni Mannheim

2015-12-03T13:15:00Z

T Kienzle: /* Terms of Use */

= Access to bwForCluster for members of Mannheim University =

== Login prerequisites ==

* A valid RUM account is required for access to the bwForCluster.

* As part of the login procedure, you will get a page that shows the data on your account that will be transferred from Mannheim University to the cluster. This page shows your entitlements. It should list the entitlement: <pre>http://bwidm.de/entitlement/bwForCluster </pre> If it does not, please go to [http://www.uni-mannheim.de/rum/ueber_uns/arbeitsgruppen/zs/hpc/bwgrid_cluster/zugang/ HPC-Zugang] for further information.

== Terms of Use ==

As with any other RUM service, users of bwForCluster must strictly adhere to the [http://www.uni-mannheim.de/ionas/rum/ueber_uns/arbeitsgruppen/zs/webdienste/verwaltungs_und_benutzerordnung/ terms of use Mannheim University].

Batch Jobs - bwUniCluster Features

2015-07-30T18:09:10Z

T Kienzle: /* Interactiv Job Monitoring per Node */

This article contains information on features of the [[Batch_Jobs|batch job system]] only applicable on bwUniCluster.

= Job Submission =
== msub Command ==
The bwUniCluster supports the following additional msub option(s):
{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"
! colspan="3" style="background-color:#999999;padding:3px"| bwUniCluster additional msub Options
|-
! style="width:15%;height=20px; text-align:left;padding:3px"|Command line
! style="width:20%;height=20px; text-align:left;padding:3px"|Script
! style="width:65%;height=20px; text-align:left;padding:3px"|Purpose
|- style="vertical-align:top;"
| style="height=20px; text-align:left;padding:3px" | -I
|
| style="height=20px; text-align:left;padding:3px" | Declares the job is to be run interactively.
|}
 
 

=== msub -l ''resource_list'' ===
No deviation or additional features to general [[Batch_Jobs|batch job]] setting.
=== msub -q ''queues'' ===

Compute resources such as walltime, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, you must add to your msub command the correct queue class. Details are:

{| border="1", style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"
! colspan="6" style="background-color:#999999;padding:3px"| msub -q ''queue''
|- style="width:10%;height=20px; text-align:left;"
! style="width:10%;padding:3px"| ''queue''
! style="width:5%;padding:3px"| ''node''
! style="width:15%;padding:3px"| ''default resources''
! style="padding:3px"| ''minimum resources''
! style="padding:3px"| ''maximum resources''
! style="padding:3px"| ''node access policy''
|- style="vertical-align:top; height=20px; text-align:left"
| style="padding:3px"| develop
| style="padding:3px"| thin
| style="width:15%;padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=00:30:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | singlenode
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=00:30:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:30:01
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;vertical-align:top;height=20px; text-align:left;padding:3px" | multinode
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=2
| style="padding:3px"| ''nodes''=16:''ppn''=16, ''walltime''=2:00:00:00
| style="padding:3px"| singlejob
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;vertical-align:top;height=20px; text-align:left;padding:3px" | verylong
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=3:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=3:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=6:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | fat
| style="padding:3px"| fat
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=32000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=32, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|-
|}
Note that ''node access policy''=singlejob means that, irrespected of the requested number of cores, node access is exclusive.
Default resources of a queue class defines walltime, processes and memory if not explicitly given with msub command. Resource list acronyms ''walltime'', ''procs'', ''nodes'' and ''ppn'' are described [[Batch_Jobs#msub_-l_resource_list|here]].

==== Queue class examples ====

* To run your batch job longer than 3 days, please use$ msub -q verylong.

* To run your batch job on one of the [[BwUniCluster_File_System#Components_of_bwUniCluster|fat nodes]], please use$ msub -q fat.
 
 

= Environment Variables for Batch Jobs =
The bwUniCluster expands the [[Batch_Jobs#Environment Variables for Batch Jobs|common set of MOAB environment variables]] by the following variable(s):
{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"
! colspan="3" style="background-color:#999999;padding:3px"| bwUniCluster specific MOAB variables
|- style="width:25%;height=20px; text-align:left;padding:3px"
! style="width:20%;height=20px; text-align:left;padding:3px"| Environment variables
! style="height=20px; text-align:left;padding:3px"| Description
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | MOAB_SUBMITDIR
| style="height=20px; text-align:left;padding:3px"| Directory of job submission
|}
 
Since the work load manager MOAB on [[bwUniCluster]] uses the resource manager SLURM, the following environment variables of SLURM are added to your environment once your job has started:
{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"
! colspan="3" style="background-color:#999999;padding:3px"| SLURM variables
|- style="width:25%;height=20px; text-align:left;padding:3px"
! style="width:20%;height=20px; text-align:left;padding:3px"| Environment variables
! style="height=20px; text-align:left;padding:3px"| Description
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_CPUS_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Number of processes per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NODELIST
| style="height=20px; text-align:left;padding:3px"| List of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NUM_NODES
| style="height=20px; text-align:left;padding:3px"| Number of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_MEM_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Memory per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_NPROCS
| style="height=20px; text-align:left;padding:3px"| Total number of processes dedicated to the job
|}
 
 
== Interactiv Job Monitoring per Node ==

By default nodes are not used exclusive unless they are requested with ''-l naccesspolicy=singlejob'' as described [[Batch_Jobs#msub_-l_resource_list|here]]. 
If a Job runs exclusive on one node you may do a ssh login to that node. The ssh access will be limited by the set walltime. To get the nodes of your job need to read the environment variable SLURM_JOB_NODELIST during the runtime of the job. It contains all nodes in a shortened way e.g. ''uc1n[344,386]'' or ''uc1n[344-345]''. To expand this string to ''uc1n344 uc1n345'' you can you can use the command expandnodes like:

expandnodes $SLURM_JOB_NODELIST > nodelist

 
 

= Intel MPI parallel Programs =
== Intel MPI without Multithreading ==
MPI parallel programs run faster than serial programs on multi CPU and multi core systems. N-fold spawned processes of the MPI program, i.e., '''MPI tasks''', run simultaneously and communicate via the Message Passing Interface (MPI) paradigm. MPI tasks do not share memory but can be spawned over different nodes.

Generate a wrapper script for '''Intel MPI''', ''job_impi.sh'' containing the following lines:
{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 2px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
module load mpi/impi/<placeholder_for_version>
mpiexec.hydra -bootstrap slurm my_par_program
</source>
|}
'''Attention:''' Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since MOAB instructs mpirun about number of processes and node hostnames.
Moreover, replace <placeholder_for_version> with the wished version of '''Intel MPI''' to enable the MPI environment.
 

Launching and running 32 Intel MPI tasks on 4 nodes, each requiring 1000 MByte, and running for 5 hours, execute:
<pre>
$ msub -q multinode -l nodes=4:ppn=16,pmem=1000mb,walltime=05:00:00 job_impi.sh
</pre>
 

== Intel MPI with Multithreading ==
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes.

Multiple Intel MPI tasks must be launched by the MPI parallel program '''mpiexec.hydra'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).

'''For Intel MPI''' a job-script to submit a batch job called ''job_impi_omp.sh'' that runs a Intel MPI program with 8 tasks and a tenfold threaded program ''impi_omp_program'' requiring 32000 MByte of total physical memory per task and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=4:ppn=20
#MSUB -l walltime=06:00:00
#MSUB -l pmem=3200mb
#MSUB -v MPI_MODULE=mpi/impi
#MSUB -v OMP_NUM_THREADS=10
#MSUB -v MPIRUN_OPTIONS="-binding "domain=omp" -print-rank-map -ppn 2 -envall"
#MSUB -v EXE=./impi_omp_program
#MSUB -N test_impi_omp

#If using more than one MPI task per node please set
export KMP_AFFINITY=scatter
#export KMP_AFFINITY=verbose,scatter prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

module load ${MPI_MODULE}
TASK_COUNT=$((${MOAB_PROCCOUNT}/${OMP_NUM_THREADS}))
echo "${EXE} running on ${MOAB_PROCCOUNT} cores with ${TASK_COUNT} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpiexec.hydra -bootstrap slurm ${MPIRUN_OPTIONS} -n ${TASK_COUNT} ${EXE}"
echo $startexe
exec $startexe
</source>
|}
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores. If you only run one MPI task per node please set KMP_AFFINITY=compact,1,0.

 
Execute the script '''job_impi_omp.sh''' adding the queue class ''multinode'' to your msub command:
<pre>
$ msub -q multinode job_impi_omp.sh
</pre>
 
The mpirun option ''-print-rank-map'' shows the bindings between MPI tasks and nodes (not very beneficial). The option ''-binding'' binds MPI tasks (processes) to a particular processor; ''domain=omp'' means that the domain size is determined by the number of threads. In the above examples (2 MPI tasks per node) you could also choose ''-binding "cell=unit;map=bunch"''; this binding maps one MPI process to each socket.
 
 

= Interactive Jobs =
Interactive jobs on bwUniCluster [[BwUniCluster_User_Access#Allowed_activities_on_login_nodes|must '''NOT''' run on the logins nodes]], however resources for interactive jobs can be requested using msub. Considering a serial application with a graphical frontend that requires 5000 MByte of memory and limiting the interactive run to 2 hours execute the following:
<pre>
$ msub -I -V -l nodes=1:ppn=1 -l walltime=0:02:00:00
</pre>
The option -V defines that all environment variables are exported to the compute node of the interactive session.
After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system MOAB has granted you the requested resources on the compute system. Once granted you will be automatically logged on the dedicated resource. Now you have an interactive session with 1 core and 5000 MByte of memory on the compute system for 2 hours. Simply execute now your application:
<pre>
$ cd to_path
$ ./application
</pre>
Note that, once the walltime limit has been reached you will be automatically logged out of the compute system.
 
 

== Chain Jobs ==

The CPU time requirements of many applications exceed the limits of the job classes. In those situations it is recommended to solve the problem by a job chain. A job chain is a sequence of jobs where each job automatically starts its successor.

{{bwFrameA|
<source lang="bash">
#!/bin/bash
##################################################
## simple job run template for bwUniCluster ##
## to run chain jobs with MOAB ##
##################################################
##
## usage :
## msub -v myloop_counter=0 ./moab_chain_job.sh

#MSUB -l nodes=1:ppn=1
#MSUB -l walltime=00:00:05
#MSUB -l pmem=50mb
#MSUB -q develop
#MSUB -N chain

## Defaults
loop_max=10
cmd='sleep 2'

## Check if counter environment variable is set
if [ -z "${myloop_counter}" ] ; then
echo " ERROR: myloop_counter is undefined, stop chain job"
exit 1
fi

## only continue if below loop_max
if [ ${myloop_counter} -lt ${loop_max} ] ; then

## increase counter
let myloop_counter+=1

## print current Job number
echo " Chain job iteration = ${myloop_counter}"

## Define your command
cmd='sleep 2'
echo " -> executing ${cmd}"
${cmd}

if [ $? -eq 0 ] ; then
## continue only if last command was successful
msub -v myloop_counter=${myloop_counter} ./moab_chain_job.sh
else
## Terminate chain
echo " ERROR: ${cmd} of chain job no. ${myloop_counter} terminated unexpectedly"
exit 1
fi
fi
</source>
}}

----
[[Category:bwUniCluster|Batch Jobs - bwUniCluster features]]

Batch Jobs - bwUniCluster Features

2015-07-30T14:16:12Z

T Kienzle: /* Node Monitoring */

This article contains information on features of the [[Batch_Jobs|batch job system]] only applicable on bwUniCluster.

= Job Submission =
== msub Command ==
The bwUniCluster supports the following additional msub option(s):
{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"
! colspan="3" style="background-color:#999999;padding:3px"| bwUniCluster additional msub Options
|-
! style="width:15%;height=20px; text-align:left;padding:3px"|Command line
! style="width:20%;height=20px; text-align:left;padding:3px"|Script
! style="width:65%;height=20px; text-align:left;padding:3px"|Purpose
|- style="vertical-align:top;"
| style="height=20px; text-align:left;padding:3px" | -I
|
| style="height=20px; text-align:left;padding:3px" | Declares the job is to be run interactively.
|}
 
 

=== msub -l ''resource_list'' ===
No deviation or additional features to general [[Batch_Jobs|batch job]] setting.
=== msub -q ''queues'' ===

Compute resources such as walltime, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, you must add to your msub command the correct queue class. Details are:

{| border="1", style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"
! colspan="6" style="background-color:#999999;padding:3px"| msub -q ''queue''
|- style="width:10%;height=20px; text-align:left;"
! style="width:10%;padding:3px"| ''queue''
! style="width:5%;padding:3px"| ''node''
! style="width:15%;padding:3px"| ''default resources''
! style="padding:3px"| ''minimum resources''
! style="padding:3px"| ''maximum resources''
! style="padding:3px"| ''node access policy''
|- style="vertical-align:top; height=20px; text-align:left"
| style="padding:3px"| develop
| style="padding:3px"| thin
| style="width:15%;padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=00:30:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | singlenode
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=00:30:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1, ''walltime''=00:30:01
| style="padding:3px"| ''nodes''=1:''ppn''=16, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;vertical-align:top;height=20px; text-align:left;padding:3px" | multinode
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=2
| style="padding:3px"| ''nodes''=16:''ppn''=16, ''walltime''=2:00:00:00
| style="padding:3px"| singlejob
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;vertical-align:top;height=20px; text-align:left;padding:3px" | verylong
| style="padding:3px"| thin
| style="padding:3px"| ''walltime''=3:00:00:01,''procs''=1, ''pmem''=4000mb
| style="padding:3px"| ''nodes''=1'', walltime''=3:00:00:01
| style="padding:3px"| ''nodes=1:''ppn''=16, ''walltime''=6:00:00:00
| style="padding:3px"| shared
|- style="vertical-align:top; height=20px; text-align:left"
| style="width:10%;padding:3px" | fat
| style="padding:3px"| fat
| style="padding:3px"| ''walltime''=00:10:00,''procs''=1, ''pmem''=32000mb
| style="padding:3px"| ''nodes''=1
| style="padding:3px"| ''nodes''=1:''ppn''=32, ''walltime''=3:00:00:00
| style="padding:3px"| shared
|-
|}
Note that ''node access policy''=singlejob means that, irrespected of the requested number of cores, node access is exclusive.
Default resources of a queue class defines walltime, processes and memory if not explicitly given with msub command. Resource list acronyms ''walltime'', ''procs'', ''nodes'' and ''ppn'' are described [[Batch_Jobs#msub_-l_resource_list|here]].

==== Queue class examples ====

* To run your batch job longer than 3 days, please use$ msub -q verylong.

* To run your batch job on one of the [[BwUniCluster_File_System#Components_of_bwUniCluster|fat nodes]], please use$ msub -q fat.
 
 

= Environment Variables for Batch Jobs =
The bwUniCluster expands the [[Batch_Jobs#Environment Variables for Batch Jobs|common set of MOAB environment variables]] by the following variable(s):
{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"
! colspan="3" style="background-color:#999999;padding:3px"| bwUniCluster specific MOAB variables
|- style="width:25%;height=20px; text-align:left;padding:3px"
! style="width:20%;height=20px; text-align:left;padding:3px"| Environment variables
! style="height=20px; text-align:left;padding:3px"| Description
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | MOAB_SUBMITDIR
| style="height=20px; text-align:left;padding:3px"| Directory of job submission
|}
 
Since the work load manager MOAB on [[bwUniCluster]] uses the resource manager SLURM, the following environment variables of SLURM are added to your environment once your job has started:
{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"
! colspan="3" style="background-color:#999999;padding:3px"| SLURM variables
|- style="width:25%;height=20px; text-align:left;padding:3px"
! style="width:20%;height=20px; text-align:left;padding:3px"| Environment variables
! style="height=20px; text-align:left;padding:3px"| Description
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_CPUS_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Number of processes per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NODELIST
| style="height=20px; text-align:left;padding:3px"| List of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NUM_NODES
| style="height=20px; text-align:left;padding:3px"| Number of nodes dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_MEM_PER_NODE
| style="height=20px; text-align:left;padding:3px"| Memory per node dedicated to the job
|-
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_NPROCS
| style="height=20px; text-align:left;padding:3px"| Total number of processes dedicated to the job
|}
 
 
== Interactiv Job Monitoring per Node ==

By default nodes are not used exclusive unless they are requested with ''-l naccesspolicy=singlejob'' as described [[Batch_Jobs#msub_-l_resource_list|here]]. 
If a Job runs exclusive on one node you may do a ssh login to that node. The ssh access will be limited by the set walltime. To get the nodes of your job need to read the environment variable SLURM_JOB_NODELIST. It contains node in e.g. uc1n[344,386]. To expand this string you can you can use the command expandnodes.

expandnodes $SLURM_JOB_NODELIST > nodelist

 
 

= Intel MPI parallel Programs =
== Intel MPI without Multithreading ==
MPI parallel programs run faster than serial programs on multi CPU and multi core systems. N-fold spawned processes of the MPI program, i.e., '''MPI tasks''', run simultaneously and communicate via the Message Passing Interface (MPI) paradigm. MPI tasks do not share memory but can be spawned over different nodes.

Generate a wrapper script for '''Intel MPI''', ''job_impi.sh'' containing the following lines:
{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 2px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
module load mpi/impi/<placeholder_for_version>
mpiexec.hydra -bootstrap slurm my_par_program
</source>
|}
'''Attention:''' Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since MOAB instructs mpirun about number of processes and node hostnames.
Moreover, replace <placeholder_for_version> with the wished version of '''Intel MPI''' to enable the MPI environment.
 

Launching and running 32 Intel MPI tasks on 4 nodes, each requiring 1000 MByte, and running for 5 hours, execute:
<pre>
$ msub -q multinode -l nodes=4:ppn=16,pmem=1000mb,walltime=05:00:00 job_impi.sh
</pre>
 

== Intel MPI with Multithreading ==
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes.

Multiple Intel MPI tasks must be launched by the MPI parallel program '''mpiexec.hydra'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).

'''For Intel MPI''' a job-script to submit a batch job called ''job_impi_omp.sh'' that runs a Intel MPI program with 8 tasks and a tenfold threaded program ''impi_omp_program'' requiring 32000 MByte of total physical memory per task and total wall clock time of 6 hours looks like:


{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
| style="width:280px; white-space:nowrap; color:#000;" |
<source lang="bash">
#!/bin/bash
#MSUB -l nodes=4:ppn=20
#MSUB -l walltime=06:00:00
#MSUB -l pmem=3200mb
#MSUB -v MPI_MODULE=mpi/impi
#MSUB -v OMP_NUM_THREADS=10
#MSUB -v MPIRUN_OPTIONS="-binding "domain=omp" -print-rank-map -ppn 2 -envall"
#MSUB -v EXE=./impi_omp_program
#MSUB -N test_impi_omp

#If using more than one MPI task per node please set
export KMP_AFFINITY=scatter
#export KMP_AFFINITY=verbose,scatter prints messages concerning the supported affinity
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE

module load ${MPI_MODULE}
TASK_COUNT=$((${MOAB_PROCCOUNT}/${OMP_NUM_THREADS}))
echo "${EXE} running on ${MOAB_PROCCOUNT} cores with ${TASK_COUNT} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpiexec.hydra -bootstrap slurm ${MPIRUN_OPTIONS} -n ${TASK_COUNT} ${EXE}"
echo $startexe
exec $startexe
</source>
|}
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores. If you only run one MPI task per node please set KMP_AFFINITY=compact,1,0.

 
Execute the script '''job_impi_omp.sh''' adding the queue class ''multinode'' to your msub command:
<pre>
$ msub -q multinode job_impi_omp.sh
</pre>
 
The mpirun option ''-print-rank-map'' shows the bindings between MPI tasks and nodes (not very beneficial). The option ''-binding'' binds MPI tasks (processes) to a particular processor; ''domain=omp'' means that the domain size is determined by the number of threads. In the above examples (2 MPI tasks per node) you could also choose ''-binding "cell=unit;map=bunch"''; this binding maps one MPI process to each socket.
 
 

= Interactive Jobs =
Interactive jobs on bwUniCluster [[BwUniCluster_User_Access#Allowed_activities_on_login_nodes|must '''NOT''' run on the logins nodes]], however resources for interactive jobs can be requested using msub. Considering a serial application with a graphical frontend that requires 5000 MByte of memory and limiting the interactive run to 2 hours execute the following:
<pre>
$ msub -I -V -l nodes=1:ppn=1 -l walltime=0:02:00:00
</pre>
The option -V defines that all environment variables are exported to the compute node of the interactive session.
After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system MOAB has granted you the requested resources on the compute system. Once granted you will be automatically logged on the dedicated resource. Now you have an interactive session with 1 core and 5000 MByte of memory on the compute system for 2 hours. Simply execute now your application:
<pre>
$ cd to_path
$ ./application
</pre>
Note that, once the walltime limit has been reached you will be automatically logged out of the compute system.
 
 

== Chain Jobs ==

The CPU time requirements of many applications exceed the limits of the job classes. In those situations it is recommended to solve the problem by a job chain. A job chain is a sequence of jobs where each job automatically starts its successor.

{{bwFrameA|
<source lang="bash">
#!/bin/bash
##################################################
## simple job run template for bwUniCluster ##
## to run chain jobs with MOAB ##
##################################################
##
## usage :
## msub -v myloop_counter=0 ./moab_chain_job.sh

#MSUB -l nodes=1:ppn=1
#MSUB -l walltime=00:00:05
#MSUB -l pmem=50mb
#MSUB -q develop
#MSUB -N chain

## Defaults
loop_max=10
cmd='sleep 2'

## Check if counter environment variable is set
if [ -z "${myloop_counter}" ] ; then
echo " ERROR: myloop_counter is undefined, stop chain job"
exit 1
fi

## only continue if below loop_max
if [ ${myloop_counter} -lt ${loop_max} ] ; then

## increase counter
let myloop_counter+=1

## print current Job number
echo " Chain job iteration = ${myloop_counter}"

## Define your command
cmd='sleep 2'
echo " -> executing ${cmd}"
${cmd}

if [ $? -eq 0 ] ; then
## continue only if last command was successful
msub -v myloop_counter=${myloop_counter} ./moab_chain_job.sh
else
## Terminate chain
echo " ERROR: ${cmd} of chain job no. ${myloop_counter} terminated unexpectedly"
exit 1
fi
fi
</source>
}}

----
[[Category:bwUniCluster|Batch Jobs - bwUniCluster features]]

BwForCluster User Access Members Uni Mannheim

2014-12-17T15:38:13Z

T Kienzle: Created page with "= Access to bwForCluster for members of Mannheim University = == Login prerequisites == * A valid RUM account is required for access to the bwForCluster. * As part of the..."

= Access to bwForCluster for members of Mannheim University =

== Login prerequisites ==

* A valid RUM account is required for access to the bwForCluster.

* As part of the login procedure, you will get a page that shows the data on your account that will be transferred from Mannheim University to the cluster. This page shows your entitlements. It should list the entitlement: <pre>http://bwidm.de/entitlement/bwForCluster </pre> If it does not, please go to [http://www.uni-mannheim.de/rum/ueber_uns/arbeitsgruppen/zs/hpc/bwgrid_cluster/zugang/ HPC-Zugang] for further information.

== Terms of Use ==

As with any other RUM service, users of bwForCluster must strictly adhere to the [http://www.uni-mannheim.de/rum/ivs/benutzerverwaltung/benutzerkennung/verwaltungs_und_benutzungsordnung.pdf terms of use Mannheim University].

BwForCluster User Access

2014-12-17T15:29:55Z

T Kienzle: /* Issueing bwForCluster entitlement */

The usage of bwForCluster is free of charge. bwForClusters are customized to the requirements of particular research areas.
Each bwForCluster is/will be financed by the DFG (German Research Foundation) and by the Ministry of Science, Research and Arts of Baden-Württemberg based on scientifc grant proposal (compare proposals guidelines as per Art. 91b GG).

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:center; color:#000;vertical-align:middle;font-size:75%;" |[[File:Bwforreg.svg|center|border|500px|]]
|-
| style="text-align:center; color:#000;vertical-align:middle;" |
|}

Granting user access to a bwForCluster requires 3 steps:

# Registration of a new or joining an already existing ''Rechenvorhaben (RV)'' at [https://www.bwhpc-c5.de/en/ZAS/zas_overview.php "Zentrale Antragsseite (ZAS)"]. An ''RV'' defines the planned compute activities of a group of researchers. As coworker of this group you only need to register your membership in the corresponding ''RV''.
# Application for a [[#Issueing bwForCluster entitlement | bwForCluster entitlement]] issued by your university.
# [[#Personal registration at bwForCluster | Personal registration]] at assigned cluster site based on approved ''RV'' [[File:Zas assignment icon.svg|25px|]] and issued bwForCluster entitlement [[File:bwfor entitlement icon.svg|25px|]].

 
Step 1 and 2 are pre-steps and requirements of step 3. Moreover, steps 1 and 2 can be done in parallel. The question, at which bwForCluster site to do step 3, will be answered by the cluster assignment team (CAT) based on the data of step 1, i.e. at ''ZAS''.

= ''RV'' registration at ''ZAS'' =

The ''rechenvorhaben (RV)'' registration at ''ZAS'' does '''NOT''' correspond to typical compute project proposals as for other HPC clusters, since:
* there is no scientific reviewing process
* the registration inquires only brief details and
* it covers a group of coworkers, and
* only the ''RV'' responsible must submit the ''RV''.
 

== How to register? ==
=== Register a new ''RV'' ===
Typically only the leader of a scientific work group or the senior scientist of a research group/collaboration needs to login at
* https://www.bwhpc-c5.de/en/ZAS/bwforcluster_rv_registration.php
and to fill in all mandatory fields of the given web form. Please note that for your convenience you can also switch to the [https://www.bwhpc-c5.de/ZAS/bwforcluster_rechenvorhaben.php german version] of the web form. The submitter of the ''rechenvorhaben (RV)'' will be assigned the role '''RV responsible'''.

The web form consists of the following fields to be filled:
{| style="width:100%;border:3px solid darkgray; margin: 1em auto 1em auto;"
|-
! style="width:25%;padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;"| Field
! style="padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;" | Explanation
|-
!scope="column" {{Lightgray_small}}| RV Title
|scope="column" {{Tab element}} | Define a short title of your planned compute activities, maximum of 255 characters including spaces.
|-
!scope="column" {{Lightgray_small}}| RV Description
|scope="column" {{Tab element}} | Write a short abstract about your planned compute activities (maximum of 2048 characters including spaces). Research groups that contributed to the DFG grant proposal (Art. 91b GG) of the corresponding bwForCluster only need to give reference to that particular proposal.
|-
!scope="column" {{Lightgray_small}}| Scientific field
|scope="column" {{Tab element}} | Tick one or several scientific fields. Once all bwForClusters are up and running the full list of given scientific fields are supported and hence for ''rechenvorhaben (RV)'' applicable. If your ''RV'' does not match any given scientific field, please state your scientific field in the given text box. Grayed out scientific fields indicate that the corresponding bwForCluster(s) is/are not operational yet.
|-
!scope="column" {{Lightgray_small}}| Field of activity
|scope="column" {{Tab element}} | Define if your ''RV'' is primarily for research and/or teaching. If not applicable, use text box.
|-
!scope="column" {{Lightgray_small}}| Parallel paradigm
|scope="column" {{Tab element}} | State what parallel paradigm your code or application uses. Multiple ticks are allowed. Further information can be provided via text box. If you are not sure about it, please state the software you are using.
|-
!scope="column" {{Lightgray_small}}| Programming language
|scope="column" {{Tab element}} | State the programming language(s) of your code or application.
|-
!scope="column" {{Lightgray_small}}| Numerical methods
|scope="column" {{Tab element}} | State the numerical or "calculation" methods your code or application utilises. If you do not know, write "unknown".
|-
!scope="column" {{Lightgray_small}}| Software packages
|scope="column" {{Tab element}} | State all software package you will or want to use for your ''RV''. Also include compiler, MPI and numerical packages (e.g. MKL, FFTW).
|-
!scope="column" {{Lightgray_small}}| Estimated requested computing resources
|scope="column" {{Tab element}} | Roughly estimate how many CPU hours are required to finish your ''RV''. To calculate "CPU hours" multiply "elapsed parallel computing time" with "number of CPU core" per job.
''Example: Your code uses 4 CPU cores and has a wall time of 1h. This makes 4 CPU hours.''
|-
!scope="column" {{Lightgray_small}}| Planned maximum number of parallel used CPU cores per job
|scope="column" {{Tab element}} | Give an upper limit of how many CPU cores per job are required. Please cross-check with your statement about parallel paradigm.
''Obviously, a sequential code can only use 1 CPU core per job.''
|-
!scope="column" {{Lightgray_small}}| Estimated maximal memory requirements per CPU core
|scope="column" {{Tab element}} | Give an upper limit of how much RAM per CPU core your jobs require. Please give a value in GigaBytes.
|-
!scope="column" {{Lightgray_small}}| Estimated requested persistent disk space for the ''RV''
|scope="column" {{Tab element}} | State how much disk space in total you need for your ''RV''. Please give a value in GigaBytes.
''Example: If your RV has 4 more coworker and each of you produces until the
end of your RV 20 GigaByte of output, your maximum disk space is 100 GigaByte.''
|-
!scope="column" {{Lightgray_small}}| Estimated maximal temporary disk space per job
|scope="column" {{Tab element}} | State how much space your intermediate files require during a job run. Please give a value in GigaBytes.
''Example: Your final output of your job is 0.1 GigaByte, but during your job 10 temporary files, each with a size of 1 GigaByte, are written to disk, hence your correct answer is 10 GigaByte.''
|-
!scope="column" {{Lightgray_small}}| Institute
|scope="column" {{Tab element}} | State the full name of your Institute.
|-
|}
Note that the fields - name, firstname, organization, mail, und EPPN - are auto-filled and can not be changed. These are your credentials as given by your home organization.
 
Response
# After submitting you will get an email from ZAS confirming your submission. With this email you are given an unique * acronym * password Please keep the password enclosed. Only with both, acronym and password, [[BwForCluster_User_Access#Coworkers|coworkers]] can be added to your ''rechenvorhaben''.
# The cluster assignment team will be notified immediately about your submission.
# The cluster assignment team decides within 2 working days and based on your submitted data which bwForCluster most fits, and submits its decision to ZAS.
# ZAS notifies you in an email about your assigned bwForCluster and provides website details for [[BwForCluster_User_Access#Personal registration at bwForCluster|step 3]].
 

=== Coworkers ===
Important, each coworker must add herself/himself to a ''rechenvorhaben'', since only via personal authentication the correctness of personal credentials (provided by your home organization) can be guaranteed. To become coworker of a ''rechenvorhaben'', please login at
* https://www.bwhpc-c5.de/en/ZAS/bwforcluster_collaboration.php
and provide correct
* password
* acronym
of the ''rechenvorhaben''. Your RV responsible will provide you with that information. The submitter will be assigned the role '''RV member'''.
 
Response
# After submitting the request for participating a '''RV''', the coworker will receive an email from ZAS about the process. The membership status in the '''RV''' is set active by default.
# Each person with the role of '''RV responsible''' or '''RV manager''' will be notified. Any of these persons can crosscheck via https://www.bwhpc-c5.de/en/ZAS/info_compute_project.php and clicking on the correct RV acronym the correctness of the process.
# The RV responsible can set any '''RV coworker''' to '''RV manager''' and vice versa as well as deactivate/reactivate any '''RV coworker''' or '''RV manager'''.
# Any RV manager can deactivate/reactivate any '''RV coworker'''.
 



== Roles in a RV ==
{| style="width:100%;border:3px solid darkgray; margin: 1em auto 1em auto;"
|-
! style="width:25%;padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;"| Role
! style="padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;" | Explanation
|-
!scope="column" {{Lightgray_small}}| RV responsible
|scope="column" {{Tab element}} | Registers the ''rechenvorhaben'' (RV), i.e. the planned compute activities. Can deactivate but also reactivate RV managers and coworkers. Can promote coworkers to mangers and vice versa.
|-
!scope="column" {{Lightgray_small}}| RV manager
|scope="column" {{Tab element}} | Defines a RV coworker with rights to deactivate and reactivate RV coworkers.
|-
!scope="column" {{Lightgray_small}}| RV coworker
|scope="column" {{Tab element}} | Registers her/his membership via RV acronym and password.
|-
|}

 

= Issueing bwForCluster entitlement =

Each university issues the bwForCluster entitlement
<pre>http://bwidm.de/entitlement/bwForCluster</pre>
only for their own members.

The bwForCluster entitlement issued by a university assures the operator of a bwForCluster that its university member's compute activities comply with the German Foreign Trade Act (Außenwirtschaftsgesetz - AWG) und German Foreign Trade Regulations (Außenwirtschaftsverordnung - AWV).

The following unversities have already established a process to issue the bwForCluster entitlement:
* [http://www.scc.kit.edu/downloads/sdo/Antrag_Zugangsberechtigung_bwForCluster_v2.pdf KIT]
* [[BwForCluster User Access Members Uni Ulm|Ulm University]]
* [[BwForCluster User Access Members Uni Tübingen|Tübingen University]]
* [[BwForCluster User Access Members Uni Mannheim|Mannheim University]]
If you do not find your university in the list above, please contact your local service desk.

= Personal registration at bwForCluster =

Once you have registered your own RV (''rechenvorhaben'')
or a membership in an RV, the cluster assignment team will provide
you with information about your designated cluster. You will receive
an email with a website to create an account for yourself
on that cluster.

Available bwForCluster registration servers (service providers):
{| style="border:3px solid darkgray; margin: 1em auto 1em auto;" width="60%"
|-
!scope="row" {{Darkgray}} | Cluster topic and location
!scope="row" {{Darkgray}} | Registration server
|-
| bwForCluster Chemistry JUSTUS Ulm
| http://bwidm.rz.uni-ulm.de/
|}

Please note, this step is different to your registration at ZAS since
here you register yourself as a person. Only based on your personal credentials
a user account can be generated. However, resource requests of the RV in ZAS
will be cross-checked with the resources consumed by you and your RV coworkers
on this cluster.

After steps 1 and 2 (RV approval and bwForCluster entitlement) please visit the

* bwForCluster ''service provider'' registration website (see table above or email after RV approval):
*# Select your home organization from the list of organizations and click '''Proceed'''.
*# You will be redirected to the ''Identity Provider'' of your home organization.
*# Enter your home-organizational user ID (might be user name, email, ...) and password and click '''Login''' or '''Anmelden'''.
*# When doing this for the first time you need to accept that your personal data is transferred to the ''service provider''.
*# You will be redirected back to the cluster registration website.
*# Select '''Service Description''' of the designated cluster.
*# Click on '''Register''' link to register for this cluster.
*# Read and accept the terms and conditions of use and click on button '''Register'''.
*# Finally you will receive an email with instructions how to login to the cluster (please wait ~15 minutes before trying).
*# Optionally but strongly recommended: Click on '''Set Service Password''' and set a password for the cluster.

= Login to bwForCluster =

Personalized details about how to login to the cluster are included
in an email send after registration at the bwForCluster service provider.

General instructions for the bwForCluster login can be found here:
{| style="border:3px solid darkgray; margin: 1em auto 1em auto;" width="60%"
|-
!scope="row" {{Darkgray}} | Cluster topic and location
!scope="row" {{Darkgray}} | Login instructions
|-
| bwForCluster Chemistry JUSTUS Ulm
| [[BwForCluster_Chemistry_Login|bwForCluster Chemistry Login]]
|}

 

----
[[Category:Access|bwForCluster]][[Category:bwForCluster_Chemistry]]

Batch Jobs - bwUniCluster Features

2014-11-05T15:30:15Z

T Kienzle: /* Environment Variables for Batch Jobs */

Batch Jobs - bwUniCluster Features

2014-11-05T15:09:35Z

T Kienzle: /* msub -q queues */ gb to mb