bwHPC Wiki - User contributions [en]

Justus3

2026-08-20T15:23:55Z

K Siegmund:

{{Template:Justus3}}



The '''bwForCluster Justus 3''' is the state-wide high-performance compute cluster dedicated to '''Computational Chemistry and Quantum Sciences''' in Baden-Württemberg.

{| style=" background:#FEF4AB; width:100%;"
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News
|-
|
* 2026-06-16: Cooling installations for the new cluster has been enabled
|}
{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support
|-
|
* [[Justus3/Migration_from_J2|Migration Guide from JUSTUS2]]
* [[Justus3/Getting Started|Getting Started]]
* Online Tutorial [https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC Introduction to Linux in HPC] (external link)
* [[Justus3/Support|Contact and Support]]
* Send [[Feedback|Feedback]] about Wiki pages
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | User Documentation
|-
|
* [[Registration/bwForCluster|Registration]]
* [[Justus3/Policy Agreement|Policy Agreement]]

* [[Justus3/Login|Login]]
* [[Justus3/Hardware|Hardware and Architecture]]
** [[Justus3/Hardware#Node_Specifications|Node Specifications]]
** [[Justus3/Hardware#Storage_Architecture|File Systems and Workspaces]]

* [[Justus3/Software|Software]]
* [[Justus3/Jobscripts: Running Your Calculations|Jobscripts: Running Your Calculations]]
** [[bwForCluster Justus 3 Slurm HOWTO|Slurm HOWTO]]
* [[Justus3/Visualization|Visualisation]]
* [[Development]]
|}

{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Cluster Funding
|-
|
* Please '''[[Justus3/Acknowledgement|acknowledge]]''' the Justus 3 cluster in your publications.
|}

.bashrc Do's and Don'ts

2026-06-29T15:39:55Z

K Siegmund:

.bashrc and .profiles are files that are sourced when you log in, they set some important variables that make your shell function at a specific site setup.

You can add commands that you want to run every time at login and when starting a job by adding lines to these files, but you can also make mistakes that make your life or the life of the cluster teams harder.

== Don'ts ==

* don't forget to test your changes before you log out (log in on a second shell)
* don't put <code> module load </code> commands in .bashrc, modules may make changes that break the default system environment
* don't put LD_PRELOAD or LIBRARY_PATH in your .bashrc, unless you know exactly what you're doing
*

== Do's ==
* make a copy of your .bashrc before making large changes
* '''test''' your changes to .bashrc before you log out
* add command aliases or bash functions in .bashrc - example: <code> alias l='ls -la' </code>
* add a local ~/bin or similar to $PATH if needed
* improve your terminal command history experience:
<syntaxhighlight lang=bash>
shopt -s histappend # append to the history file, don't overwrite it
shopt -s direxpand # expand dirs before trying to expand filenames
HISTSIZE=1000000 # number of commands stored in one terminal session
HISTFILESIZE=20000000 # number of commmands stored in file
HISTFILE="$HOME"/.bash_history_$USER # prevent file truncation by bash sessions that did not read in .bashrc
PROMPT_COMMAND="history -a" # save history after every command
HISTCONTROL=ignoreboth:erasedups # dont store duplicates and ignore lines starting with white space
HISTIGNORE="?:??" # dont store one and two letter commands
export HISTTIMEFORMAT="%F %H:%M:%S " # show dates on history
</syntaxhighlight>

* maybe make your rm / cp commands safer by aliasing to -i <code> alias rm 'rm -i'</code>

== Autoedits / Conda ==

There is some software (e.g. Conda) that automatically modifies .bashrc. These modifications may interfere with the proper functioning of e.g. Python from Lmod modules or the Jupyter modules. Make sure you have a copy of your .bashrc and test it after installing such software.

Data Transfer/SSHFS

2026-06-29T13:30:11Z

K Siegmund:

Sshfs is a network protocol for mounting data. It uses [[Data Transfer/SFTP|sftp]] under the hood.

== Installation ==

For the necessary installations follow this link:<br /> [https://linuxize.com/post/how-to-use-sshfs-to-mount-remote-directories-over-ssh/ How to use SSHFS to Mount Remote Directories over SSH].<br />
a) '''MacOS after the installation'''<br />
To enable sshfs on your Mac go to: System Preferences -> General tab -> Allow Extensions button<br />
Then follow the instructions that pop up (make sure that you remember the instructions before shutting down your computer).<br />
b) '''Linux installation short commands'''
<ul>
<li>Debian/Ubuntu: <code>apt-get install sshfs</code></li>
<li>CentOS/RedHat: <code>dnf install fuse-sshfs</code></li>
<li>If you want to use /etc/fstab, you can do this with the following entry:
<pre>sshfs#<username>@<hostname>: <your_local_mountpoint> fuse defaults,user,noauto,exec 0 0 </pre></li></ul>

== Usage ==

=== Caveat: Unmounting ===

Note that this creates a virtual drive on your system just as if you would plug in an external drive that stays active until you unmount it. Disk mounts happen on a low level of operating systems, so it is important to unmount the virtual drive cleanly whenever possible.

'''MacOS & Linux:'''<br />
If the connection to the cluster broke and you want to remount, you often have to use umount first. Try to remember to always unmount first before retrying. Otherwise it can happen that unmounting is not possible anymore by cmd. Usually the error message in this case tells you that the process is currently busy and therefore cannot umount.

You should be able to find the process(es) that use the file system with <code>fuser -m <your_local_mountpoint></code> and kill/exit the processes that block unmounting or can kill them directly using <code>fuser -k -m <your_local_mountpoint></code>
In case this fails, you may have to restart your computer which automatically unmounts everything.

<pre>bash
umount -f <localPath>
# For example
umount -f /Users/<username>/Documents/mounted
# or use
fusermount -u <localPath>
</pre>

''' MacOS '''

An automatic umount is highly recommended, because MacOS finder can make thousands of insuccessful login attempts in the worst case leading to you being blocked from the cluster system or even from everything that uses the same OTP.

Example for enabling automatic umount per "sleepwatcher":

<syntaxhighlight lang="bash">
brew install sleepwatcher
brew services start sleepwatcher
brew services info sleepwatcher
cat > ~/.sleep <<EOT
#!/bin/bash
umount -f ~/mountCluster 2>/dev/null
EOT
chmod a+x ~/.sleep
</syntaxhighlight>

'''Windows'''
You should be able to click on the drive and choose "eject" or "unmount". If this is not the case, you can use
net use X: /delete
or if this does not work
taskkill /f /im sshfs.exe
(from: [https://stackoverflow.com/questions/62031900/how-to-unmount-close-the-mount-for-sshfs-windows stackoverflow])

=== Windows ===
''Caution: sshfs on Windows doesn't work with 2FA. An alternative is [[Data_Transfer/Rclone | Rclone with sftp]] + [[Registration/SSH | ssh key]].''
<ol style="list-style-type: decimal;">
<li>Go to "This PC" and select "Connect network drive" ("Netzlaufwerk verbinden")</li>
<li>Select a drive (W:) and click on "connect with webpage on which you can save documents and images" (Verbindung mit website herstellen, auf der Sie Dokumente und Bilder speichern können)</li>
<li>Enter the following as address, to connect to the home or root folder:
<syntaxhighlight lang="bash">\sshfs\<username>@<hostname>/</syntaxhighlight>
If you would want to go two folders up, you would add “''../../''” at the end.</li></ol>

=== MacOS & Linux ===

<ol style="list-style-type: decimal;">
<li>Create an empty directory. The cluster/storage folder is going to be mounted to this directory.</li>
<li>Mount a folder from the cluster/storage:
<syntaxhighlight lang="bash"># Home folder
sshfs <username>@<hostname>: <localPath>
# subfolderX
sshfs <username>@<hostname>:subfolderX <localPath></syntaxhighlight></li></ol>

Data Transfer/SSHFS

2026-06-29T11:36:56Z

K Siegmund: /* MacOS & Linux */

Sshfs is a network protocol for mounting data. It uses [[Data Transfer/SFTP|sftp]] under the hood.

== Installation ==

For the necessary installations follow this link:<br /> [https://linuxize.com/post/how-to-use-sshfs-to-mount-remote-directories-over-ssh/ How to use SSHFS to Mount Remote Directories over SSH].<br />
a) '''MacOS after the installation'''<br />
To enable sshfs on your Mac go to: System Preferences -> General tab -> Allow Extensions button<br />
Then follow the instructions that pop up (make sure that you remember the instructions before shutting down your computer).<br />
b) '''Linux installation short commands'''
<ul>
<li>Debian/Ubuntu: <code>apt-get install sshfs</code></li>
<li>CentOS/RedHat: <code>dnf install fuse-sshfs</code></li>
<li>If you want to use /etc/fstab, you can do this with the following entry:
<pre>sshfs#<username>@<hostname>: <your_local_mountpoint> fuse defaults,user,noauto,exec 0 0 </pre></li></ul>

== Usage ==

=== Caveat: Unmounting ===

Note that this creates a virtual drive on your system just as if you would plug in an external drive that stays active until you unmount it. Disk mounts happen on a low level of operating systems, so it is important to unmount the virtual drive cleanly whenever possible.

'''MacOS & Linux:'''<br />
If the connection to the cluster broke and you want to remount, you often have to use umount first. Try to remember to always unmount first before retrying. Otherwise it can happen that unmounting is not possible anymore by cmd. Usually the error message in this case tells you that the process is currently busy and therefore cannot umount.

You should be able to find the process(es) that use the file system with <code>fuser -m <your_local_mountpoint></code> and kill/exit the processes that block unmounting or can kill them directly using <code>fuser -k -m <your_local_mountpoint></code>
In case this fails, you may have to restart your computer which automatically unmounts everything.

<pre>bash
umount -f <localPath>
# For example
umount -f /Users/<username>/Documents/mounted
# or use
fusermount -u <localPath>
</pre>

'''Windows'''
You should be able to click on the drive and choose "eject" or "unmount". If this is not the case, you can use
net use X: /delete
or if this does not work
taskkill /f /im sshfs.exe
(from: [https://stackoverflow.com/questions/62031900/how-to-unmount-close-the-mount-for-sshfs-windows stackoverflow])

=== Windows ===
''Caution: sshfs on Windows doesn't work with 2FA. An alternative is [[Data_Transfer/Rclone | Rclone with sftp]] + [[Registration/SSH | ssh key]].''
<ol style="list-style-type: decimal;">
<li>Go to "This PC" and select "Connect network drive" ("Netzlaufwerk verbinden")</li>
<li>Select a drive (W:) and click on "connect with webpage on which you can save documents and images" (Verbindung mit website herstellen, auf der Sie Dokumente und Bilder speichern können)</li>
<li>Enter the following as address, to connect to the home or root folder:
<syntaxhighlight lang="bash">\sshfs\<username>@<hostname>/</syntaxhighlight>
If you would want to go two folders up, you would add “''../../''” at the end.</li></ol>

=== MacOS & Linux ===

<ol style="list-style-type: decimal;">
<li>Create an empty directory. The cluster/storage folder is going to be mounted to this directory.</li>
<li>Mount a folder from the cluster/storage:
<syntaxhighlight lang="bash"># Home folder
sshfs <username>@<hostname>: <localPath>
# subfolderX
sshfs <username>@<hostname>:subfolderX <localPath></syntaxhighlight></li></ol>

== MacOS ==

An automatic umount is highly recommended, because MacOS finder can make thousands of insuccessful login attempts in the worst case leading to you being blocked from the cluster system or even from everything that uses the same OTP.

Example for enabling automatic umount per "sleepwatcher":

<syntaxhighlight lang="bash">
brew install sleepwatcher
brew services start sleepwatcher
brew services info sleepwatcher
cat > ~/.sleep <<EOT
#!/bin/bash
umount -f ~/mountCluster 2>/dev/null
EOT
chmod a+x ~/.sleep
</syntaxhighlight>

Justus3/Getting Started

2026-06-26T13:58:30Z

K Siegmund:

Here is a short list of things you may need to do first when you get onto the cluster

== Pre-Requisites ==
* Know how a HPC compute Cluster works: In short: you do not simply run log in and start your software, but you write a "job script" that contains all commands to run and process your job and send this into a waiting queue to be run on one of several hundred computers. → '''[[Running Calculations]]'''
* Get Access: Follow the registration process for the bwForCluster. → [[Registration/bwForCluster|How to Register for a bwForCluster]]

== Basics ==
* know the cluster rules: → '''[[Justus3/Policy_Agreement]]'''
* log in to the cluster (and first set a 2FA secret): '''[[Justus3/Login]]'''
* get accustomed with the linux commandline:
** HPC Wiki (external site) → '''[https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC/The_Command_Line Introduction to Linux Commandline]'''
** Have commands ready → '''[[Media:HPC_User_Linux_command_cheatsheet.pdf|HPC User's Linux Cheat Sheet (PDF)]]'''
** configure your shell → '''[[.bashrc Do's and Don'ts]]'''
** linux course at '''[https://training.bwhpc.de/ training.bwhpc.de]'''

== Running an Example with Preinstalled Software ==
* scientific software: learn how to load '''[[Software Modules]]'''
* continue reading until you found that there are example job scripts: '''[[Environment_Modules#Software_job_examples]]'''
* submit a sample job from a software as mentioned in the job example. Also see: '''[[Justus3/Slurm]]'''
* monitor your job: '''[[Justus3/Monitoring Your Jobs]]'''

== Running Your Own Calculations in a Batch Job ==
{{Template:AttentionBox|Compute jobs on nodes '''must not''' write temporary I/O intensive data to central file systems.}}
* transfer your own data to the cluster → '''[[Data Transfer]]'''
** Learn about available Hardware:
*** → '''[[Justus3/File Systems]]'''
*** → '''[[Justus3/Compute Node Types]]'''
* adapt the sample job script to run your own job
* Submit and monitor your jobs with Slurm commands.
** → '''[[Justus3/Jobscripts: Running Your Calculations|Jobscripts: Running Your Calculations]]'''
** → '''[[BwForCluster_JUSTUS_2_Slurm_HOWTO| extensive Slurm HOWTO on specific tasks]]'''

* Remember to mention the cluster in your publications. → '''[[bwForCluster Justus 3 Acknowledgement|Acknowledgement]]'''

== Do Production Runs with Hundreds of Jobs ==

* Learn about Scaling your Job
** Determine how many compute-cores the job can use? → '''[[Scaling]]'''
** Make especially sure your compute jobs are running efficiently → '''[[Energy Efficient Cluster Usage]]'''

* Specially remember to mention the cluster in your publications. → '''[[bwForCluster Justus 3 Acknowledgement|Acknowledgement]]'''

Justus3/Getting Started

2026-06-26T13:57:07Z

K Siegmund: /* Basics */

Here is a short list of things you may need to do first when you get onto the cluster

== Pre-Requisites ==
* Know how a HPC compute Cluster works: In short: you do not simply run log in and start your software, but you write a "job script" that contains all commands to run and process your job and send this into a waiting queue to be run on one of several hundred computers. → '''[[Running Calculations]]'''
* Get Access: Follow the registration process for the bwForCluster. → [[Registration/bwForCluster|How to Register for a bwForCluster]]

== Basics ==
* know the cluster rules: → '''[[Justus3/Policy_Agreement]]'''
* log in to the cluster (and first set a 2FA secret): '''[[JUSTUS3/Login]]'''
* get accustomed with the linux commandline:
** HPC Wiki (external site) → '''[https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC/The_Command_Line Introduction to Linux Commandline]'''
** Have commands ready → '''[[Media:HPC_User_Linux_command_cheatsheet.pdf|HPC User's Linux Cheat Sheet (PDF)]]'''
** configure your shell → '''[[.bashrc Do's and Don'ts]]'''
** linux course at '''[https://training.bwhpc.de/ training.bwhpc.de]'''

== Running an Example with Preinstalled Software ==
* scientific software: learn how to load '''[[Software Modules]]'''
* continue reading until you found that there are example job scripts: '''[[Environment_Modules#Software_job_examples]]'''
* submit a sample job from a software as mentioned in the job example. Also see: '''[[JUSTUS3/Slurm]]'''
* monitor your job: '''[[JUSTUS3/Monitoring Your Jobs]]'''

== Running Your Own Calculations in a Batch Job ==
{{Template:AttentionBox|Compute jobs on nodes '''must not''' write temporary I/O intensive data to central file systems.}}
* transfer your own data to the cluster → '''[[Data Transfer]]'''
** Learn about available Hardware:
*** → '''[[JUSTUS3/File Systems]]'''
*** → '''[[JUSTUS3/Compute Node Types]]'''
* adapt the sample job script to run your own job

* Submit and monitor your jobs with Slurm commands.
** → '''[[JUSTUS3/Jobscripts: Running Your Calculations|Jobscripts: Running Your Calculations]]'''
** → '''[[BwForCluster_JUSTUS_2_Slurm_HOWTO| extensive Slurm HOWTO on specific tasks]]'''

* Remember to mention the cluster in your publications. → '''[[bwForCluster JUSTUS 3 Acknowledgement|Acknowledgement]]'''

== Do Production Runs with Hundreds of Jobs ==

* Learn about Scaling your Job
** Determine how many compute-cores the job can use? → '''[[Scaling]]'''
** Make especially sure your compute jobs are running efficiently → '''[[Energy Efficient Cluster Usage]]'''

* Specially remember to mention the cluster in your publications. → '''[[bwForCluster JUSTUS 3 Acknowledgement|Acknowledgement]]'''

Justus3/Getting Started

2026-06-26T13:56:14Z

K Siegmund: K Siegmund moved page JUSTUS3/Getting Started to Justus3/Getting Started without leaving a redirect

Here is a short list of things you may need to do first when you get onto the cluster

== Pre-Requisites ==
* Know how a HPC compute Cluster works: In short: you do not simply run log in and start your software, but you write a "job script" that contains all commands to run and process your job and send this into a waiting queue to be run on one of several hundred computers. → '''[[Running Calculations]]'''
* Get Access: Follow the registration process for the bwForCluster. → [[Registration/bwForCluster|How to Register for a bwForCluster]]

== Basics ==
* know the cluster rules: → '''[[JUSTUS3/Policy_Agreement]]'''
* log in to the cluster (and first set a 2FA secret): '''[[JUSTUS3/Login]]'''
* get accustomed with the linux commandline:
** HPC Wiki (external site) → '''[https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC/The_Command_Line Introduction to Linux Commandline]'''
** Have commands ready → '''[[Media:HPC_User_Linux_command_cheatsheet.pdf|HPC User's Linux Cheat Sheet (PDF)]]'''
** configure your shell → '''[[.bashrc Do's and Don'ts]]'''
** linux course at '''[https://training.bwhpc.de/ training.bwhpc.de]'''

== Running an Example with Preinstalled Software ==
* scientific software: learn how to load '''[[Software Modules]]'''
* continue reading until you found that there are example job scripts: '''[[Environment_Modules#Software_job_examples]]'''
* submit a sample job from a software as mentioned in the job example. Also see: '''[[JUSTUS3/Slurm]]'''
* monitor your job: '''[[JUSTUS3/Monitoring Your Jobs]]'''

== Running Your Own Calculations in a Batch Job ==
{{Template:AttentionBox|Compute jobs on nodes '''must not''' write temporary I/O intensive data to central file systems.}}
* transfer your own data to the cluster → '''[[Data Transfer]]'''
** Learn about available Hardware:
*** → '''[[JUSTUS3/File Systems]]'''
*** → '''[[JUSTUS3/Compute Node Types]]'''
* adapt the sample job script to run your own job

* Submit and monitor your jobs with Slurm commands.
** → '''[[JUSTUS3/Jobscripts: Running Your Calculations|Jobscripts: Running Your Calculations]]'''
** → '''[[BwForCluster_JUSTUS_2_Slurm_HOWTO| extensive Slurm HOWTO on specific tasks]]'''

* Remember to mention the cluster in your publications. → '''[[bwForCluster JUSTUS 3 Acknowledgement|Acknowledgement]]'''

== Do Production Runs with Hundreds of Jobs ==

* Learn about Scaling your Job
** Determine how many compute-cores the job can use? → '''[[Scaling]]'''
** Make especially sure your compute jobs are running efficiently → '''[[Energy Efficient Cluster Usage]]'''

* Specially remember to mention the cluster in your publications. → '''[[bwForCluster JUSTUS 3 Acknowledgement|Acknowledgement]]'''

Acknowledgement

2026-06-25T09:36:31Z

K Siegmund:

Remember to acknowledge our resources in your publications!

Such recognition is important for acquiring funding for the next generation of hardware, support services, data storage, and infrastructure.

The publications will be referenced on the bwHPC website: https://www.bwhpc.de/user_publications.html

== HPC Clusters ==

→[[BwUniCluster3.0/Acknowledgement| bwUniCluster Acknowledgement]]

→[[BinAC2/Acknowledgement| bwForCluster BinAC2 Acknowledgement]]

→[[Helix/Acknowledgement| bwForCluster Helix Acknowledgement]]

→[[Justus3/Acknowledgement| bwForCluster Justus 3 Acknowledgement]]

→[[NEMO2/Acknowledgement| bwForCluster NEMO2 Acknowledgement]]

== Data Facilities ==

→[[SDS@hd/Acknowledgement| SDS@hd Acknowledgement]]

Acknowledgement

2026-06-25T09:36:11Z

K Siegmund:

Remember to acknowledge our resources in your publications!

Such recognition is important for acquiring funding for the next generation of hardware, support services, data storage, and infrastructure.

The publications will be referenced on the bwHPC website: https://www.bwhpc.de/user_publications.html

== HPC Clusters ==

→[[BwUniCluster3.0/Acknowledgement| bwUniCluster Acknowledgement]]

→[[BinAC2/Acknowledgement| bwForCluster BinAC2 Acknowledgement]]

→[[Helix/Acknowledgement| bwForCluster Helix Acknowledgement]]

→[[JUSTUS3/Acknowledgement| bwForCluster JUSTUS3 Acknowledgement]]

→[[NEMO2/Acknowledgement| bwForCluster NEMO2 Acknowledgement]]

== Data Facilities ==

→[[SDS@hd/Acknowledgement| SDS@hd Acknowledgement]]

Justus3/Acknowledgement

2026-06-25T09:06:03Z

K Siegmund:

== Acknowledgement ==
When preparing a publication based on work using a bwForCluster (e.g., Justus 3), please ensure you acknowledge the bwHPC initiative, the bwHPC-S5 project, and the specific facility. The following text is suggested.

'''Acknowledgement:'''
<blockquote>
'''The authors acknowledge support by the HPC team at the Communication and Information Center (kiz) of Ulm University, the state of Baden-Württemberg through bwHPC and the German Research Foundation (DFG) for funding under [https://gepris.dfg.de/gepris/projekt/533954541 "Project number 533954541"] HPC Forschungscluster (bwForCluster Justus 3)'''
</blockquote>

== Publication Reporting ==

Please inform us of any fully published publications (e.g., journal articles, books, theses, master or bachelor thesis) resulting from bwHPC resource usage. Such recognition is important for acquiring funding for the next generation hardware, support services, data storage and infrastructure.

For pre-prints, please wait until the paper is fully published and send us only the final record. Please avoid sending publications already reported a second time.

This can happen in one of three ways:

1) with your annual RV renewal (the form will ask for DOIs - most practical, as it will allow to select exactly the publications published in the last year)

2) by emailing to:
* [mailto:publications(ät)bwhpc.de publications(ät)bwhpc.de]

In the email, please state:
* cluster facility (e.g. JUSTUS 3)
* RV acronym (e.g. bw14L001)
* DOI
* author(s)
* title ''or'' booktitle
* journal, volume, pages ''or'' editors, address, publisher
* year.

3) when in the process of writing the DFG report, we will send you a list of your already submitted publications and ask you to update it for the report.

The publications will be referenced on the bwHPC website:
https://www.bwhpc.de/user_publications.html

Justus3/Acknowledgement

2026-06-25T09:00:50Z

K Siegmund:

== Acknowledgement ==
When preparing a publication based on work using a bwForCluster (e.g., Justus 3), please ensure you acknowledge the bwHPC initiative, the bwHPC-S5 project, and the specific facility. The following text is suggested.

'''Acknowledgement:'''
<blockquote>
'''The authors acknowledge support by the HPC team at the Communication and Information Center (kiz) of Ulm University, the state of Baden-Württemberg through bwHPC and the German Research Foundation (DFG) for funding under [https://gepris.dfg.de/gepris/projekt/533954541 "Project number 533954541"] HPC Forschungscluster (bwForCluster Justus 3)'''
</blockquote>

== Publications ==

Please kindly inform us of any publications (e.g., articles, theses, or talks) resulting from bwHPC resource usage.

This can happen in one of three ways:

1) with your annual RV renewal (the form will ask for DOIs)

2) by emailing to:
* [mailto:publications(ät)bwhpc.de publications(ät)bwhpc.de]

Write to stating:
* cluster facility (e.g. JUSTUS 3)
* RV acronym (e.g. bw14L001)
* DOI (if available)
* author(s)
* title ''or'' booktitle
* journal, volume, pages ''or'' editors, address, publisher
* year.

3) when in the process of writing the DFG report, we will send you a list of your already submitted publications and ask you to update it for the report.

Such recognition is important for acquiring funding for the next generation hardware, support services, data storage and infrastructure.

The publications will be referenced on the bwHPC website:
https://www.bwhpc.de/user_publications.html

Justus3/Acknowledgement

2026-06-25T08:56:47Z

K Siegmund:

When preparing a publication based on work using a bwForCluster (e.g., Justus 3), please ensure you acknowledge the bwHPC initiative, the bwHPC-S5 project, and the specific facility. The following text is suggested.

'''Acknowledgement:'''
<blockquote>
'''The authors acknowledge support by the HPC team at the Communication and Information Center (kiz) of Ulm University, the state of Baden-Württemberg through bwHPC and the German Research Foundation (DFG) for funding under [https://gepris.dfg.de/gepris/projekt/533954541 "Project number 533954541"] HPC Forschungscluster (bwForCluster Justus 3)'''
</blockquote>

Please kindly notify us of any publications (e.g., articles, theses, or talks) resulting from bwHPC resource usage by emailing the below email address. '''Note''': If you already provide a complete publication list during your yearly RV renewal, a separate notification is not required.

Write to [mailto:publications(ät)bwhpc.de publications(ät)bwhpc.de] stating:
* cluster facility (e.g. JUSTUS 3)
* RV acronym (e.g. bw14L001)
* DOI (if available)
* author(s)
* title ''or'' booktitle
* journal, volume, pages ''or'' editors, address, publisher
* year.

Such recognition is important for acquiring funding for the next generation hardware, support services, data storage and infrastructure.

The publications will be referenced on the bwHPC website:
https://www.bwhpc.de/user_publications.html

Justus3/Acknowledgement

2026-06-24T17:01:09Z

K Siegmund:

When preparing a publication based on work using a bwForCluster (e.g., Justus 3), please ensure you acknowledge the bwHPC initiative, the bwHPC-S5 project, and the specific facility. The following text is suggested.

'''Acknowledgement:'''
<blockquote>
'''The authors acknowledge support by the HPC team at the Communication and Information Center (kiz) of Ulm University, the state of Baden-Württemberg through bwHPC and the German Research Foundation (DFG) for funding under [https://gepris.dfg.de/gepris/projekt/533954541 "Project number 533954541"] HPC Forschungscluster (bwForCluster Justus 3)'''
</blockquote>

Please kindly notify us of any publications (e.g., articles, theses, or talks) resulting from bwHPC resource usage by emailing the below email address. '''Note''': If you already provide a complete publication list during your yearly RV renewal, a separate notification is not required.

Write to [mailto:publications(ät)bwhpc.de publications(ät)bwhpc.de] stating:
* cluster facility (e.g. JUSTUS 2)
* RV acronym (e.g. bw14L001)
* DOI (if available)
* author(s)
* title ''or'' booktitle
* journal, volume, pages ''or'' editors, address, publisher
* year.

Such recognition is important for acquiring funding for the next generation hardware, support services, data storage and infrastructure.

The publications will be referenced on the bwHPC website:
https://www.bwhpc.de/user_publications.html

Justus3/Acknowledgement

2026-06-24T17:00:58Z

K Siegmund:

When preparing a publication based on work using a bwForCluster (e.g., Justus 3), please ensure you acknowledge the bwHPC initiative, the bwHPC-S5 project, and the specific facility. The following text is suggested.

'''Acknowledgement:'''
<blockquote>
'''The authors acknowledge support by the HPC team at the Communication and Information Center (kiz) of Ulm University, the state of Baden-Württemberg through bwHPC and the German Research Foundation (DFG) for funding under [https://gepris.dfg.de/gepris/projekt/533954541 "Project number 533954541"] HPC Forschungscluster (bwForCluster Justus 3)'''
</blockquote>

"Please kindly notify us of any publications (e.g., articles, theses, or talks) resulting from bwHPC resource usage by emailing the below email address. '''Note''': If you already provide a complete publication list during your yearly RV renewal, a separate notification is not required."

Write to [mailto:publications(ät)bwhpc.de publications(ät)bwhpc.de] stating:
* cluster facility (e.g. JUSTUS 2)
* RV acronym (e.g. bw14L001)
* DOI (if available)
* author(s)
* title ''or'' booktitle
* journal, volume, pages ''or'' editors, address, publisher
* year.

Such recognition is important for acquiring funding for the next generation hardware, support services, data storage and infrastructure.

The publications will be referenced on the bwHPC website:
https://www.bwhpc.de/user_publications.html

Justus3/Acknowledgement

2026-06-24T16:57:13Z

K Siegmund: Created page with "When preparing a publication based on work using a bwForCluster (e.g., Justus 3), please ensure you acknowledge the bwHPC initiative, the bwHPC-S5 project, and the specific facility. The following text is suggested. '''Acknowledgement:''' <blockquote> '''The authors acknowledge support by the HPC team at the Communication and Information Center (kiz) of Ulm University, the state of Baden-Württemberg through bwHPC and the German Research Foundation (DFG) for funding und..."

When preparing a publication based on work using a bwForCluster (e.g., Justus 3), please ensure you acknowledge the bwHPC initiative, the bwHPC-S5 project, and the specific facility. The following text is suggested.

'''Acknowledgement:'''
<blockquote>
'''The authors acknowledge support by the HPC team at the Communication and Information Center (kiz) of Ulm University, the state of Baden-Württemberg through bwHPC and the German Research Foundation (DFG) for funding under [https://gepris.dfg.de/gepris/projekt/533954541 "Project number 533954541"] HPC Forschungscluster (bwForCluster Justus 3)'''
</blockquote>

In addition, we kindly ask you to notify us of any reports, conference papers, journal articles, theses, posters, talks which contain results obtained on any bwHPC resource by sending an email to
[mailto:publications@bwhpc.de publications@bwhpc.de] stating:
* cluster facility (e.g. JUSTUS 2)
* RV acronym (e.g. bw14L001)
* DOI (if available)
* author(s)
* title ''or'' booktitle
* journal, volume, pages ''or'' editors, address, publisher
* year.

Such recognition is important for acquiring funding for the next generation hardware, support services, data storage and infrastructure.

The publications will be referenced on the bwHPC website:
https://www.bwhpc.de/user_publications.html

JUSTUS2/Software/Turbomole

2026-06-24T16:07:12Z

K Siegmund: /* Template provided by Turbomole Module */

{{Softwarepage|chem/turbomole}}

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| chem/turbomole
|-
| License
| [https://www.turbomole.org/turbomole/order-turbomole/ Commerical]
|-
| Citing
| [https://www.turbomole.org/turbomole/turbomole-documentation/ See Turbomole manual]
|-
| Links
| [https://www.turbomole.com Homepage] | [https://www.turbomole-gmbh.com/turbomole-manuals.html Documentation]
|-
| User Forum
| [https://www.turbo-forum.com/ external]
|}
= Description =
'''Turbomole''' is a general purpose ''quantum chemistry'' software package for ''ab initio'' electronic structure calculations and provides:
* ground state calculations for methods such as Hartree-Fock, DFT, MP2, and CCSD(T);
* excited state calculations at different levels such as full RPA, TDDFT, CIS(D), CC2, an ADC(2);
* geometry optimizations, transition state searches, molecular dynamics calculations;
* property and spectra calculations such as IR, UV/VIS, Raman, and CD;
* approximations like resolution-of-the-identity (RI) to speed-up the calculations without introducing uncontrollable or unknown errors; as well as
* parallel versions for all kind of jobs.
For more information on Turbmole's features please visit [https://www.turbomole-gmbh.com/program-overview.html http://www.turbomole-gmbh.com/program-overview.html].
<br>
<br>

= Versions and Availability =

On the command line interface (CLI) of a particular bwHPC cluster a list of all available Turbomole versions can be inquired as followed
<pre>
$ module avail chem/turbomole
</pre>

A current list of the versions available on the bwUniCluster and bwForClusters can be found here:
https://www.bwhpc.de/software.php

== bwUniCluster 3.0 ==

* Turbomole 7.9

== bwForCluster JUSTUS 2 ==

* Turbomole 7.5
* Turbomole 7.4.1

== Parallel computing ==
The Turbomole ''Module'' subsumes all available parallel computing variants of Turbomole's binaries. Turbomole defines the following parallel computing variants:
* SMP = Shared-memory parallel computing based on OpenMP and Fork() with the latter using separated address spaces.
* MPI = Message passing interface protocol based parallel computing
However only one of the parallel variants or the sequential variant can be loaded at once and most Turbomole's binaries support only 1 or 2 of the parallelization variants. Like for Turbomole installations without the ''Module'' system of the bwHPC clusters, the variants have to be triggered by the environment variable $PARA_ARCH.
<br>
<br>

= Usage =
== Before loading the Module ==
Before loading the Turbomole ''Module'' the parallel computing variant has to be defined via the environment variable $PARA_ARCH using the abbreviations SMP or MPI, e.g.:
<pre>
$ export PARA_ARCH=MPI
</pre>
will [[#Loading the Module|later load]] the '''MPI binary variants'''. If the variable $PARA_ARCH is not defined or empty, the sequential binary variants will be active once the Turbomole Module is loaded.
<br>
== Loading the Module ==
You can load the default version of ''Turbomole'' with the command:
<pre>
$ module load chem/turbomole
</pre>
The Turbomole ''Module'' does not depend on any other ''Module'', but on the variable $PARA_ARCH. Moreover, Turbomole provides its own libraries regarding ''OpenMP'', ''Fork()'', and ''MPI'' based parallelization.
If you wish to load a specific (older) version you can do so by executing e.g.:
<pre>
$ module load chem/turbomole/7.4.1
</pre>
to load the version 7.4.1
<br>
== Switching between different parallel variants ==
To switch between the different parallel variants provided by the Turbomole ''Module'', simply define the new parallel variant via $PARA_ARCH and load the ''Module'' again. Note that for switching between the parallel variants unloading of the Turbomole ''Module'' is not required. For instance to change to the MPI variant, execute:
<pre>
$ export PARA_ARCH=MPI
$ module load chem/turbomole
</pre>
<br>
== Turbomole binaries ==
The '''Turbomole''' software package consists of a set of stand-alone program binaries providing different features and parallelization support:
{| width=750px class="wikitable"
|- style=text-align:left"
! Binary !! Features !! OpenMP !! Fork !! MPI
|- style="vertical-align:top;"
| define
| Interactive input generator
| no
| no
| no
|- style="vertical-align:top;"
| dscf
| Energy calculations
| yes
| yes
| yes
|- style="vertical-align:top;"
| grad
| Gradient calculations
| yes
| yes
| yes
|- style="vertical-align:top;"
| ridft
| Energy calc. with fast Coulomb approximation
| no
| yes
| yes
|- style="vertical-align:top;"
| rdgrad
| Gradient calc. with fast Coulomb approximation
| yes
| yes
| yes
|- style="vertical-align:top;"
| ricc2
| Electronic excitation energies, transition moments and properties of excited states
| yes
| no
| yes
|- style="vertical-align:top;"
| statpt
| Hessian and coordinate update for stationary point search
| no
| no
| no
|- style="vertical-align:top;"
| aoforce
| Analytic calculation of force constants, vibrational frequencies and IR intensities
| yes
| yes
| yes
|- style="vertical-align:top;"
| escf
| Calc. of time dependent and dielectric properties
| yes
| yes
| yes
|- style="vertical-align:top;"
| egrad
| gradients and first-order properties of excited states
| yes
| no
| yes
|- style="vertical-align:top;"
| odft
| Orbital-dependent energy calc.
| yes
| no
| no
|}
For the complete set of binaries and more detailed description of their features read [https://www.turbomole.org/turbomole/turbomole-documentation/ here].
<br>

== Turbomole tools ==
Turbomole's tool set contains scripts and binaries that help to prepare, execute workflows (such as geometry optimisation) and postprocess results.
{| width=750px class="wikitable"
|- style="text-align:left"
! Script !! Type !! Description
|- style="vertical-align:top;"
| x2t
| Preparation
| Converts XYZ coordinates into ''Turbomole'' coordinates.
|- style="vertical-align:top;"
| sdg
| Preparation
| Shows data group from ''control'' file: for example<span style="background:#edeae2;margin:10px;padding:1px;border:1px dotted #808080">$ sdg coord</span>shows current Turbomole coordinates used.
|- style="vertical-align:top;"
| jobex
| Optimization workflow
| Shell script that controls and executes automatic optimizations of molecular geometry parameters.

|- style="vertical-align:top;"
| tm2molden
| Postprocessing
| Creates a molden format input file for the [[Molden]] program.
|- style="vertical-align:top;"
| eiger
| Postprocessing
| Script front-end of program ''eigerf'' to display HOMO-LUMO gap and MO eigenvalues.
|}
For the complete set of tools and more detailed description of their features read [http://www.turbomole-gmbh.com/manuals/version_6_5/Documentation_html/node9.html here].
<br>
== Disk Usage ==
By default, scratch files of Turbomole binaries are placed in the directory of Turbomole binary execution. Please do not run your Turbomole calculations in your $HOME or $WORK directory.
<br>
<br>
= Examples =
== Single node jobs ==
=== Template provided by Turbomole Module ===
The Turbomole ''Module'' provides a simple job script example of Cubane (C8H8) that runs an energy relaxation via MPI parallel ''dscf'' using 4 cores on a single node and its local file system. To simply run the example do the following steps:
<pre>
$ module load chem/turbomole
$ mkdir -vp ~/Turbomole-example/
$ cd ~/Turbomole-examples/
$ cp -r $TURBOMOLE_EXA_DIR/* ~/Turbmole-example/
$ sbatch bwHPC_turbomole_single-node_tmpdir_example.sh
</pre>
The last step submits the job example script bwHPC_turbomole_single-node_example.sh to the queueing system. Once started on a compute node, all calculations will be done under an unique directory on the local file system of that particular compute node. You have to explicitely request local disk space on Justus.
<br>

=== Geometry optimization ===
To do a geometry optimization of the [[#Single node job template provided by Turbomole Module|previous job example]] modify bwHPC_turbomole_single-node_tmpdir_example.sh by replacing the following line
<syntaxhighlight lang="bash">
time dscf > dscf.out 2>&1
</syntaxhighlight>
with
<syntaxhighlight lang="bash">
time jobex -dscf -keep 2>&1
</syntaxhighlight>
and submit the modified script to the queueing system on bwForCluster JUSTUS 2 (→[[JUSTUS2/Jobscripts: Running Your Calculations]] and bwUniCluster (→[[BwUniCluster3.0/Slurm]]) via sbatch.
The Turbomole command ''jobex'' controls the call of all the required Turbomole binaries for the geometry optimization.
<br>
<br>

= Turbomole-Specific Environment Variables =
To see a list of all Turbomole environments set by the 'module load'-command do the following:
<pre>
module show chem/turbomole 2>&1 | grep -E '(setenv|prepend-path)'
</pre>

= Version-Specific Information =
For specific information about the Turbomole version (e.g. 7.5) to be loaded do the following:
<pre>
module help chem/turbomole/7.5
</pre>

----

BwUniCluster 2.0/FAQ

2026-06-24T16:04:22Z

K Siegmund: Changed redirect target from BwUniCluster 3.0/FAQ to BwUniCluster/FAQ

#REDIRECT [[BwUniCluster/FAQ]]

BwUniCluster Hardware and Architecture

2026-06-24T16:03:26Z

K Siegmund: Changed redirect target from BwUniCluster 2.0 Hardware and Architecture to BwUniCluster2.0/Hardware and Architecture

#REDIRECT [[BwUniCluster2.0/Hardware and Architecture]]

Dalton

2026-06-24T16:00:51Z

K Siegmund: Changed redirect target from JUSTUS2/Dalton to JUSTUS2/Software/Dalton

#REDIRECT [[‎JUSTUS2/Software/Dalton]]

JUSTUS2/Support

2026-06-24T15:59:07Z

K Siegmund: /* Registration and HPC Support */

==Entitlement==

In case of questions or problems regarding your entitlement for using any bwForCluster, please
contact your local university first level help desk.

See → '''[[Registration/bwForCluster/Entitlement]]'''

==Registration and HPC Support==

In case of registration problems or questions about using JUSTUS 2, please submit a trouble ticket at the

→ '''[https://www.bwhpc.de/supportportal bwSupport Portal]'''

Use the "+" sign to create a new ticket) and assign it
to support unit '''GROUP => bwHPC - Support (green arrow on right side) => bwForCluster JUSTUS'''. This will put the ticket directly in the correct queue, else someone in first level will first have to move it to the correct queue until someone from the Justus team can read it.

JUSTUS2/Support

2026-06-24T15:50:11Z

K Siegmund: /* Entitlement */

==Entitlement==

In case of questions or problems regarding your entitlement for using any bwForCluster, please
contact your local university first level help desk.

See → '''[[Registration/bwForCluster/Entitlement]]'''

==Registration and HPC Support==

In case of registration problems or questions about using JUSTUS 2, please submit a trouble ticket at the [https://www.bwhpc.de/supportportal bwSupport Portal] (use the "+" sign to create a new ticket) and assign it
to support unit '''GROUP => bwHPC - Support (green arrow on right side) => bwForCluster JUSTUS'''.

JUSTUS2/Support

2026-06-24T15:49:28Z

K Siegmund: /* Entitlement */

==Entitlement==

In case of questions or problems regarding your [[Registration/bwForCluster/Entitlement|entitlement]] for using any bwForCluster, please
contact your local university first level help desk.

==Registration and HPC Support==

In case of registration problems or questions about using JUSTUS 2, please submit a trouble ticket at the [https://www.bwhpc.de/supportportal bwSupport Portal] (use the "+" sign to create a new ticket) and assign it
to support unit '''GROUP => bwHPC - Support (green arrow on right side) => bwForCluster JUSTUS'''.

Justus3

2026-06-24T15:47:37Z

K Siegmund:

{{Template:Justus3}}



The '''bwForCluster Justus 3''' is the state-wide high-performance compute cluster dedicated to '''Computational Chemistry and Quantum Sciences''' in Baden-Württemberg.

{| style=" background:#FEF4AB; width:100%;"
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News
|-
|
* 2026-06-16: Cooling installations for the new cluster has been enabled
|}
{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support
|-
|
* [[Justus3/Getting Started|Getting Started]]
* Online Tutorial [https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC Introduction to Linux in HPC] (external link)
* [[Justus3/Support|Contact and Support]]
* Send [[Feedback|Feedback]] about Wiki pages
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | User Documentation
|-
|
* [[Registration/bwForCluster|Registration]]
* [[Justus3/Policy Agreement|Policy Agreement]]

* [[Justus3/Login|Login]]
* [[Justus3/Hardware|Hardware and Architecture]]
** [[Justus3/Hardware#Node_Specifications|Node Specifications]]
** [[Justus3/Hardware#Storage_Architecture|File Systems and Workspaces]]

* [[Justus3/Software|Software]]
* [[Justus3/Jobscripts: Running Your Calculations|Jobscripts: Running Your Calculations]]
** [[bwForCluster Justus 3 Slurm HOWTO|Slurm HOWTO]]
* [[Justus3/Visualization|Visualisation]]
* [[Development]]
|}

{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Cluster Funding
|-
|
* Please '''[[Justus3/Acknowledgement|acknowledge]]''' the Justus 3 cluster in your publications.
|}

Justus3

2026-06-24T15:46:07Z

K Siegmund:

{{Template:Justus3}}

[[File:JUSTUS2_header.png|700px]]

The '''bwForCluster Justus 3''' is the state-wide high-performance compute cluster dedicated to '''Computational Chemistry and Quantum Sciences''' in Baden-Württemberg.

{| style=" background:#FEF4AB; width:100%;"
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News
|-
|
* 2026-06-16: Cooling installations for the new cluster has been enabled
|}
{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support
|-
|
* [[Justus3/Getting Started|Getting Started]]
* Online Tutorial [https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC Introduction to Linux in HPC] (external link)
* [[Justus3/Support|Contact and Support]]
* Send [[Feedback|Feedback]] about Wiki pages
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | User Documentation
|-
|
* [[Registration/bwForCluster|Registration]]
* [[Justus3/Policy Agreement|Policy Agreement]]

* [[Justus3/Login|Login]]
* [[Justus3/Hardware|Hardware and Architecture]]
** [[Justus3/Hardware#Node_Specifications|Node Specifications]]
** [[Justus3/Hardware#Storage_Architecture|File Systems and Workspaces]]

* [[Justus3/Software|Software]]
* [[Justus3/Jobscripts: Running Your Calculations|Jobscripts: Running Your Calculations]]
** [[bwForCluster Justus 3 Slurm HOWTO|Slurm HOWTO]]
* [[Justus3/Visualization|Visualisation]]
* [[Development]]
|}

{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Cluster Funding
|-
|
* Please '''[[Justus3/Acknowledgement|acknowledge]]''' the Justus 3 cluster in your publications.
|}

Justus3

2026-06-24T15:45:44Z

K Siegmund:

{{Template:Justus3}}

[[File:Justus2_header.png|700px]]

The '''bwForCluster Justus 3''' is the state-wide high-performance compute cluster dedicated to '''Computational Chemistry and Quantum Sciences''' in Baden-Württemberg.

{| style=" background:#FEF4AB; width:100%;"
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News
|-
|
* 2026-06-16: Cooling installations for the new cluster has been enabled
|}
{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support
|-
|
* [[Justus3/Getting Started|Getting Started]]
* Online Tutorial [https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC Introduction to Linux in HPC] (external link)
* [[Justus3/Support|Contact and Support]]
* Send [[Feedback|Feedback]] about Wiki pages
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | User Documentation
|-
|
* [[Registration/bwForCluster|Registration]]
* [[Justus3/Policy Agreement|Policy Agreement]]

* [[Justus3/Login|Login]]
* [[Justus3/Hardware|Hardware and Architecture]]
** [[Justus3/Hardware#Node_Specifications|Node Specifications]]
** [[Justus3/Hardware#Storage_Architecture|File Systems and Workspaces]]

* [[Justus3/Software|Software]]
* [[Justus3/Jobscripts: Running Your Calculations|Jobscripts: Running Your Calculations]]
** [[bwForCluster Justus 3 Slurm HOWTO|Slurm HOWTO]]
* [[Justus3/Visualization|Visualisation]]
* [[Development]]
|}

{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Cluster Funding
|-
|
* Please '''[[Justus3/Acknowledgement|acknowledge]]''' the Justus 3 cluster in your publications.
|}

Template:Justus3

2026-06-24T15:45:10Z

K Siegmund: Created page with "{{AttentionBox|The bwForCluster Justus3 is not yet fully installed and open to users.}}"

{{AttentionBox|The bwForCluster Justus3 is not yet fully installed and open to users.}}

Justus3/Getting Started

2026-06-24T14:46:22Z

K Siegmund:

Here is a short list of things you may need to do first when you get onto the cluster

== Pre-Requisites ==
* Know how a HPC compute Cluster works: In short: you do not simply run log in and start your software, but you write a "job script" that contains all commands to run and process your job and send this into a waiting queue to be run on one of several hundred computers. → '''[[Running Calculations]]'''
* Get Access: Follow the registration process for the bwForCluster. → [[Registration/bwForCluster|How to Register for a bwForCluster]]

== Basics ==
* know the cluster rules: → '''[[JUSTUS3/Policy_Agreement]]'''
* log in to the cluster (and first set a 2FA secret): '''[[JUSTUS3/Login]]'''
* get accustomed with the linux commandline:
** HPC Wiki (external site) → '''[https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC/The_Command_Line Introduction to Linux Commandline]'''
** Have commands ready → '''[[Media:HPC_User_Linux_command_cheatsheet.pdf|HPC User's Linux Cheat Sheet (PDF)]]'''
** configure your shell → '''[[.bashrc Do's and Don'ts]]'''
** linux course at '''[https://training.bwhpc.de/ training.bwhpc.de]'''

== Running an Example with Preinstalled Software ==
* scientific software: learn how to load '''[[Software Modules]]'''
* continue reading until you found that there are example job scripts: '''[[Environment_Modules#Software_job_examples]]'''
* submit a sample job from a software as mentioned in the job example. Also see: '''[[JUSTUS3/Slurm]]'''
* monitor your job: '''[[JUSTUS3/Monitoring Your Jobs]]'''

== Running Your Own Calculations in a Batch Job ==
{{Template:AttentionBox|Compute jobs on nodes '''must not''' write temporary I/O intensive data to central file systems.}}
* transfer your own data to the cluster → '''[[Data Transfer]]'''
** Learn about available Hardware:
*** → '''[[JUSTUS3/File Systems]]'''
*** → '''[[JUSTUS3/Compute Node Types]]'''
* adapt the sample job script to run your own job

* Submit and monitor your jobs with Slurm commands.
** → '''[[JUSTUS3/Jobscripts: Running Your Calculations|Jobscripts: Running Your Calculations]]'''
** → '''[[BwForCluster_JUSTUS_2_Slurm_HOWTO| extensive Slurm HOWTO on specific tasks]]'''

* Remember to mention the cluster in your publications. → '''[[bwForCluster JUSTUS 3 Acknowledgement|Acknowledgement]]'''

== Do Production Runs with Hundreds of Jobs ==

* Learn about Scaling your Job
** Determine how many compute-cores the job can use? → '''[[Scaling]]'''
** Make especially sure your compute jobs are running efficiently → '''[[Energy Efficient Cluster Usage]]'''

* Specially remember to mention the cluster in your publications. → '''[[bwForCluster JUSTUS 3 Acknowledgement|Acknowledgement]]'''

Justus3/Getting Started

2026-06-24T14:45:28Z

K Siegmund:

Here is a short list of things you may need to do first when you get onto the cluster

== Pre-Requisites ==
* Know how a HPC compute Cluster works: In short: you do not simply run log in and start your software, but you write a "job script" that contains all commands to run and process your job and send this into a waiting queue to be run on one of several hundred computers. → '''[[Running Calculations]]'''
* Get Access: Follow the registration process for the bwForCluster. → [[Registration/bwForCluster|How to Register for a bwForCluster]]

== Basics ==
* know the cluster rules: → '''[[JUSTUS3/Policy_Agreement]]'''
* log in to the cluster (and first set a 2FA secret): '''[[JUSTUS3/Login]]'''
* get accustomed with the linux commandline:
** HPC Wiki (external site) → '''[https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC/The_Command_Line Introduction to Linux Commandline]'''
** Have commands ready → '''[[Media:HPC_User_Linux_command_cheatsheet.pdf|HPC User's Linux Cheat Sheet (PDF)]]'''
** configure your shell → '''[[.bashrc Do's and Don'ts]]'''
** linux course at '''[https://training.bwhpc.de/ training.bwhpc.de]'''

== Running an Example with Preinstalled Software ==
* scientific software: learn how to load '''[[Software Modules]]'''
* continue reading until you found that there are example job scripts: '''[[Environment_Modules#Software_job_examples]]'''
* submit a sample job from a software as mentioned in the job example. Also see: '''[[JUSTUS3/Slurm]]'''
* monitor your job: '''[[JUSTUS3/Monitoring Your Jobs]]'''

== Running Your Own Calculations in a Batch Job ==
{{Template:AttentionBox|Compute jobs on nodes '''must not''' write temporary I/O intensive data to central file systems.}}
* transfer your own data to the cluster → '''[[Data Transfer]]'''
** Learn about available Hardware:
*** → '''[[JUSTUS3/File Systems]]'''
*** → '''[[JUSTUS3/Compute Node Types]]'''
* adapt the sample job script to run your own job

* Submit and monitor your jobs with Slurm commands.
** → '''[[JUSTUS2/Jobscripts: Running Your Calculations|Jobscripts: Running Your Calculations]]'''
** → '''[[BwForCluster_JUSTUS_2_Slurm_HOWTO| extensive Slurm HOWTO on specific tasks]]'''

* Remember to mention the cluster in your publications. → '''[[bwForCluster JUSTUS 3 Acknowledgement|Acknowledgement]]'''

== Do Production Runs with Hundreds of Jobs ==

* Learn about Scaling your Job
** Determine how many compute-cores the job can use? → '''[[Scaling]]'''
** Make especially sure your compute jobs are running efficiently &arr; '''[[Energy Efficient Cluster Usage]]'''

* Specially remember to mention the cluster in your publications. → '''[[bwForCluster JUSTUS 3 Acknowledgement|Acknowledgement]]'''

Justus3/Getting Started

2026-06-24T14:36:38Z

K Siegmund: /* Running an Example with Preinstalled Software */

Here is a short list of things you may need to do first when you get onto the cluster

== Pre-Requisites ==
* Know how a HPC compute Cluster works: In short: you do not simply run log in and start your software, but you write a "job script" that contains all commands to run and process your job and send this into a waiting queue to be run on one of several hundred computers. → '''[[Running Calculations]]'''
* Get Access: Follow the registration process for the bwForCluster. → [[Registration/bwForCluster|How to Register for a bwForCluster]]

== Basics ==
* know the cluster rules: → '''[[JUSTUS2/Policy_Agreement]]'''
* log in to the cluster (and first set a 2FA secret): '''[[JUSTUS3/Login]]'''
* get accustomed with the linux commandline:
** HPC Wiki (external site) → '''[https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC/The_Command_Line Introduction to Linux Commandline]'''
** Have commands ready → '''[[Media:HPC_User_Linux_command_cheatsheet.pdf|HPC User's Linux Cheat Sheet (PDF)]]'''
** configure your shell → '''[[.bashrc Do's and Don'ts]]'''
** linux course at '''[https://training.bwhpc.de/ training.bwhpc.de]'''

== Running an Example with Preinstalled Software ==
* scientific software: learn how to load '''[[Software Modules]]'''
* continue reading until you found that there are example job scripts: '''[[Environment_Modules#Software_job_examples]]'''
* submit a sample job from a software as mentioned in the job example. Also see: '''[[JUSTUS3/Slurm]]'''
* monitor your job: '''[[JUSTUS3/Monitoring Your Jobs]]'''

== Running Your Own Calculations in a Batch Job ==
{{Template:AttentionBox|Compute jobs on nodes '''must not''' write temporary I/O intensive data to central file systems.}}
* transfer your own data to the cluster → '''[[Data Transfer]]'''
** Learn about available Hardware:
*** → '''[[Hardware_and_Architecture_(bwForCluster_JUSTUS_2)#Storage_Architecture|File Systems]]'''
*** → '''[[Hardware_and_Architecture_(bwForCluster_JUSTUS_2)|Hardware and Architecture]]'''
* adapt the sample job script to run your own job

* Submit and monitor your jobs with Slurm commands.
** → '''[[JUSTUS2/Jobscripts: Running Your Calculations|Jobscripts: Running Your Calculations]]'''
** → '''[[BwForCluster_JUSTUS_2_Slurm_HOWTO| extensive Slurm HOWTO on specific tasks]]'''

* Remember to mention the cluster in your publications. → '''[[bwForCluster JUSTUS 3 Acknowledgement|Acknowledgement]]'''

== Do Production Runs with Hundreds of Jobs ==

* Learn about Scaling your Job
** Determine how many compute-cores the job can use? → '''[[Scaling]]'''
** Make especially sure your compute jobs are running efficiently &arr; '''[[Energy Efficient Cluster Usage]]'''

* Specially remember to mention the cluster in your publications. → '''[[bwForCluster JUSTUS 3 Acknowledgement|Acknowledgement]]'''

Justus3/Getting Started

2026-06-24T14:36:24Z

K Siegmund: /* Basics */

Here is a short list of things you may need to do first when you get onto the cluster

== Pre-Requisites ==
* Know how a HPC compute Cluster works: In short: you do not simply run log in and start your software, but you write a "job script" that contains all commands to run and process your job and send this into a waiting queue to be run on one of several hundred computers. → '''[[Running Calculations]]'''
* Get Access: Follow the registration process for the bwForCluster. → [[Registration/bwForCluster|How to Register for a bwForCluster]]

== Basics ==
* know the cluster rules: → '''[[JUSTUS2/Policy_Agreement]]'''
* log in to the cluster (and first set a 2FA secret): '''[[JUSTUS3/Login]]'''
* get accustomed with the linux commandline:
** HPC Wiki (external site) → '''[https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC/The_Command_Line Introduction to Linux Commandline]'''
** Have commands ready → '''[[Media:HPC_User_Linux_command_cheatsheet.pdf|HPC User's Linux Cheat Sheet (PDF)]]'''
** configure your shell → '''[[.bashrc Do's and Don'ts]]'''
** linux course at '''[https://training.bwhpc.de/ training.bwhpc.de]'''

== Running an Example with Preinstalled Software ==
* scientific software: learn how to load '''[[Software Modules]]'''
* continue reading until you found that there are example job scripts: '''[[Environment_Modules#Software_job_examples]]'''
* submit a sample job from a software as mentioned in the job example. Also see: '''[[JUSTUS3/Slurm]]'''
* monitor your job: '''[[JUSTUS3/Monitoring_Your_Jobs]]'''

== Running Your Own Calculations in a Batch Job ==
{{Template:AttentionBox|Compute jobs on nodes '''must not''' write temporary I/O intensive data to central file systems.}}
* transfer your own data to the cluster → '''[[Data Transfer]]'''
** Learn about available Hardware:
*** → '''[[Hardware_and_Architecture_(bwForCluster_JUSTUS_2)#Storage_Architecture|File Systems]]'''
*** → '''[[Hardware_and_Architecture_(bwForCluster_JUSTUS_2)|Hardware and Architecture]]'''
* adapt the sample job script to run your own job

* Submit and monitor your jobs with Slurm commands.
** → '''[[JUSTUS2/Jobscripts: Running Your Calculations|Jobscripts: Running Your Calculations]]'''
** → '''[[BwForCluster_JUSTUS_2_Slurm_HOWTO| extensive Slurm HOWTO on specific tasks]]'''

* Remember to mention the cluster in your publications. → '''[[bwForCluster JUSTUS 3 Acknowledgement|Acknowledgement]]'''

== Do Production Runs with Hundreds of Jobs ==

* Learn about Scaling your Job
** Determine how many compute-cores the job can use? → '''[[Scaling]]'''
** Make especially sure your compute jobs are running efficiently &arr; '''[[Energy Efficient Cluster Usage]]'''

* Specially remember to mention the cluster in your publications. → '''[[bwForCluster JUSTUS 3 Acknowledgement|Acknowledgement]]'''

Justus3/Getting Started

2026-06-24T14:36:03Z

K Siegmund: Created page with " Here is a short list of things you may need to do first when you get onto the cluster == Pre-Requisites == * Know how a HPC compute Cluster works: In short: you do not simply run log in and start your software, but you write a "job script" that contains all commands to run and process your job and send this into a waiting queue to be run on one of several hundred computers. → '''Running Calculations''' * Get Access: Follow the registration process for the bw..."

Here is a short list of things you may need to do first when you get onto the cluster

== Pre-Requisites ==
* Know how a HPC compute Cluster works: In short: you do not simply run log in and start your software, but you write a "job script" that contains all commands to run and process your job and send this into a waiting queue to be run on one of several hundred computers. → '''[[Running Calculations]]'''
* Get Access: Follow the registration process for the bwForCluster. → [[Registration/bwForCluster|How to Register for a bwForCluster]]

== Basics ==
* know the cluster rules: → '''[[JUSTUS2/Policy_Agreement]]'''
* log in to the cluster (and first set a 2FA secret): [[JUSTUS3/Login]]
* get accustomed with the linux commandline:
** HPC Wiki (external site) → '''[https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC/The_Command_Line Introduction to Linux Commandline]'''
** Have commands ready → '''[[Media:HPC_User_Linux_command_cheatsheet.pdf|HPC User's Linux Cheat Sheet (PDF)]]'''
** configure your shell → '''[[.bashrc Do's and Don'ts]]'''
** linux course at '''[https://training.bwhpc.de/ training.bwhpc.de]'''

== Running an Example with Preinstalled Software ==
* scientific software: learn how to load '''[[Software Modules]]'''
* continue reading until you found that there are example job scripts: '''[[Environment_Modules#Software_job_examples]]'''
* submit a sample job from a software as mentioned in the job example. Also see: '''[[JUSTUS3/Slurm]]'''
* monitor your job: '''[[JUSTUS3/Monitoring_Your_Jobs]]'''

== Running Your Own Calculations in a Batch Job ==
{{Template:AttentionBox|Compute jobs on nodes '''must not''' write temporary I/O intensive data to central file systems.}}
* transfer your own data to the cluster → '''[[Data Transfer]]'''
** Learn about available Hardware:
*** → '''[[Hardware_and_Architecture_(bwForCluster_JUSTUS_2)#Storage_Architecture|File Systems]]'''
*** → '''[[Hardware_and_Architecture_(bwForCluster_JUSTUS_2)|Hardware and Architecture]]'''
* adapt the sample job script to run your own job

* Submit and monitor your jobs with Slurm commands.
** → '''[[JUSTUS2/Jobscripts: Running Your Calculations|Jobscripts: Running Your Calculations]]'''
** → '''[[BwForCluster_JUSTUS_2_Slurm_HOWTO| extensive Slurm HOWTO on specific tasks]]'''

* Remember to mention the cluster in your publications. → '''[[bwForCluster JUSTUS 3 Acknowledgement|Acknowledgement]]'''

== Do Production Runs with Hundreds of Jobs ==

* Learn about Scaling your Job
** Determine how many compute-cores the job can use? → '''[[Scaling]]'''
** Make especially sure your compute jobs are running efficiently &arr; '''[[Energy Efficient Cluster Usage]]'''

* Specially remember to mention the cluster in your publications. → '''[[bwForCluster JUSTUS 3 Acknowledgement|Acknowledgement]]'''

JUSTUS2/Hardware

2026-06-24T14:02:15Z

K Siegmund: /* $HOME */

{{Justus2}}

= System Architecture =

The HPC cluster is composed of [[JUSTUS2/login nodes|login nodes]], compute nodes and parallel storage systems connected by fast data networks. It is connected to the Internet via Baden Württemberg's extended LAN [https://www.belwue.de/ BelWü] (light blue).

{| style="margin: 1em auto 1em auto;"
|[[Image:JUSTUS2_Architecture.png|thumb|upright=1.5|right|Overview on JUSTUS 2 hardware architecture. All nodes are additionally connected by 1GB Ethernet. ]]
|}

Users log in on one of the four login nodes and have access to their home and working directories (darker blue) stored in the parallel file system [[Lustre]].

Two additional special login visualization node enable users to visualize compute results directly on the cluster.

Calculations are done on the several types of compute nodes (top), which are accessed via the batch queuing system [[Slurm JUSTUS 2| Slurm ]].

== Operating System and Software ==

* Operating System: [https://rockylinux.org Rocky Linux 8]
* Queuing System: [[Slurm JUSTUS 2| Slurm ]] (also see: [[bwForCluster JUSTUS 2 Slurm HOWTO|Slurm HOWTO (JUSTUS 2)]])
* [[Software_Modules_Lmod|Environment Modules]] for site specific scientific applications, developer tools and libraries

== Common Hardware Features ==

The system consists of 702 nodes (692 compute nodes and 10 dedicated login, service and visualization nodes) with 2 processors each and a total of 33,696 processor cores.

* Processor: 2 x Intel Xeon 6252 Gold (Cascade Lake, 24-core, 2.1 GHz)
* Two processors per node (2 x 24 cores)
* [https://en.wikipedia.org/wiki/Omni-Path Omni-Path] 100 Gbit/s interconnect

== Node Specifications ==

As a short overview of the table below: There are different types of compute node:

* 500 "standard" nodes (44 upgraded to "medium" nodes with more RAM later)
* 168 SSD nodes: additional RAM and fast SSD disks
* 8 large nodes: larger disks and even more RAM
* 14 nodes with GPGPU accelerators

Common to all nodes of the cluster are the following properties:
* Two CPU Chips (Sockets): 2 x [https://ark.intel.com/content/www/de/de/ark/products/192447/intel-xeon-gold-6252-processor-35-75m-cache-2-10-ghz.html Intel Xeon E6252 Gold (Cascade Lake)]
* 24 compute core per CPU, so 48 core per node
* Interconnect: [https://en.wikipedia.org/wiki/Omni-Path Omni-Path 100]

The nodes are tiered in terms of hardware configuration (amount of memory, local NVMe, hardware accelerators) in order to be able to serve a large range of different job requirements flexibly and efficiently.

{|class="wikitable zebra" style="text-align:center;"
|-
! style="width:10%"| Node Type !! Quantity !! Cores !! Memory !! Local NVMe SSD !! Accelerator
|-
! style="text-align:left;" scope="column" | Standard Nodes
| 456 || 48 || 192 GB || ---|| ---
|- style="background-color: #EEE;"
! style="text-align:left;" scope="column" | Medium Nodes
| 44 || 48 || 384 GB || ---|| ---
|-
! style="text-align:left;" scope="column" | SSD Nodes
| 148 || 48 || 384 GB || 2 x 1.6 TB (RAID 0)|| ---
|- style="background-color: #EEE;"
! style="text-align:left;" scope="column" | Medium SSD Nodes
| 20 || 48 || 768 GB || 2 x 1.6 TB (RAID 0)|| ---
|-
! style="text-align:left;" scope="column" | Large SSD Nodes
| 8 || 48 || 1536 GB || 5 x 1.6 TB (RAID 0)|| ---
|- style="background-color: #EEE;"
! style="text-align:left;" scope="column" | Special Nodes
| 14 || 48 || 192 GB || ---|| 2 x [https://www.nvidia.com/en-us/data-center/v100/ Nvidia V100S]
|-
! style="text-align:left;" scope="column" | Login- and Service Nodes
| 8 || 48 || 192 GB || 2 x 2.0 TB (RAID 1)|| ---
|- style="background-color: #EEE;"
! style="text-align:left;" scope="column" | Visualization Nodes
| 2 || 48 || 192 GB || 2 x 2.0 TB (RAID 1)|| Nvidia Quadro P4000 Graphics
|}

Core numbers: [https://en.wikipedia.org/wiki/Hyper-threading hyperthreading] is enabled, so 48 core-nodes appear to have 96 cores.

== Storage Architecture ==

The bwForCluster JUSTUS 2 provides of two independent distributed parallel file systems, one for the user's home directories <tt>$HOME</tt> and another one for global workspaces. This storage architecture is based on [https://en.wikipedia.org/wiki/Lustre_(file_system) Lustre] and can be accessed in parallel from any nodes. Additionally, some compute nodes (fast I/O nodes) provide locally attached NVMe storage devices for I/O demanding applications.

{| class="wikitable"
|-
! style="width:10%"|
! style="width:10%"| <tt>$HOME</tt>
! style="width:10%"| Workspace
! style="width:10%"| <tt>$SCRATCH</tt>
! style="width:10%"| <tt>$TMPDIR</tt>
|-
!scope="column" | Visibility
| global
| global
| node local
| node local
|-
!scope="column" | Lifetime
| permanent
| workspace lifetime (max. 90 days, extension possible)
| batch job walltime
| batch job walltime
|-
!scope="column" | Total Capacity
| 250 TB
| 1200 TB
| 3000 GB / 7300 GB per node
| max. half of RAM per node
|-
!scope="column" | Disk [https://en.wikipedia.org/wiki/Disk_quota#Quotas Quotas]
| 400 GB per user
| 20 TB per user
| none
| none
|-
!scope="column" | File [https://en.wikipedia.org/wiki/Disk_quota#Quotas Quotas]
| 2.000.000 files per user
| 5.000.000 files per user
| none
| none
|-
!scope="column" | Backup
| yes
| no
| no
| no
|}

global : accessible from all nodes
local : accessible from allocated node only
permanent : files are stored permanently (as long as user can access the system)
batch job walltime : files are removed at end of the batch job

'''Note:''' Disk and file quota limits are soft limits and are subject to change. Quotas feature a grace period where users may exceed their limits to some extent (currently 20%) for a brief period of time (currently 4 weeks).

=== $HOME ===

Home directories are meant for permanent file storage of files that are keep being used like source codes, configuration files, executable programs etc.; the content of home directories will be backed up on a regular basis.

Current disk usage on home directory and quota status can be checked with the command '''lfs quota -h -u $USER /lustre/home'''.

{{Template:AttentionBox|Compute jobs on nodes must not write temporary data to $HOME. They must use the local $SCRATCH or $TMPDIR directories for very I/O intensive jobs. They can use workspaces for less I/O intensive jobs, if needed e.g. for multi-node jobs.}}

=== Workspaces ===

More info on our general page:

:: → '''[[Workspace]]s'''

Workspaces can be generated through the [[workspace]] tools. This will generate a directory with a limited lifetime on the parallel global work file system. When this
lifetime is reached the workspace will be deleted automatically after a grace period.
Users will be notified by daily e-mail reminders starting 7 days before expiration of
a workspace. Workspaces can (and must) be extended to prevent deletion at the expiration date.

'''Defaults and maximum values'''
{| class="wikitable"
|-
| Default lifetime (days)
| 7
|-
| Maximum lifetime
| 90
|-
| Maximum extensions
| unlimited
|-
|}

'''Examples'''
{| class="wikitable"
|-
!style="width:30%" | Command
!style="width:70%" | Action
|-
|<tt>ws_allocate my_workspace 30</tt>
|Allocate a workspace named "my_workspace" for 30 days.
|-
|<tt>ws_list</tt>
|List all your workspaces.
|-
|<tt>ws_find my_workspace</tt>
|Get absolute path of workspace "my_workspace".
|-
|<tt>ws_extend my_workspace 30</tt>
|Set expiration date of workspace "my_workspace" to 30 days (regardless of remaining days).
|-
|<tt>ws_release my_workspace</tt>
|Manually erase your workspace "my_workspace" and release used space on storage (remove data first for immediate deletion of the data).
|-
|}

Current disk usage on workspace file system and quota status can be checked with the command '''lfs quota -h -u $USER /lustre/work'''.

'''Note:''' The parallel work file system works optimal for medium to large file sizes and non-[https://en.wikipedia.org/wiki/Random_access random access] patterns. Large quantities of small files significantly decrease IO performance and must be avoided. Consider using '''[https://wiki.bwhpc.de/e/BwForCluster_JUSTUS_2_Slurm_HOWTO#How_to_request_local_scratch_.28SSD.2FNVMe.29_at_job_submission.3F local scratch]''' for these.

=== $SCRATCH and $TMPDIR ===

On compute nodes the environment variables $SCRATCH and $TMPDIR always point to local scratch space that is not shared across nodes.

$TMPDIR always points to a directory on a local RAM disk which will provide up to 50% of the total RAM capacity of the node. Thus, data written to $TMPDIR will always count against allocated memory.

$SCRATCH will point to a directory on locally attached NVMe devices if (and only if) local scratch has been explicitly requested at job submission (i.e. with --gres=scratch:nnn option). If no local scratch has been requested at job submission $SCRATCH will point to the very same directory as $TMPDIR (i.e. to the RAM disk).

On the login nodes $TMPDIR and $SCRATCH point to a local scratch directory on that node. This is located at /scratch/<username> and is also not shared across nodes. The data stored in there is private but will be deleted automatically if not accessed for 7 consecutive days. Like any other local scratch space, the data stored in there is NOT included in any backup.

== Backup ==

<b>Only</b> files in your $HOME directory are included in a backup and can be restored in case of accidental deletion. The files in the "Work" filesystem are not intended for permanent storage during your jobs. Sort your results well and save important data in $HOME.

JUSTUS2/Hardware

2026-06-24T14:01:50Z

K Siegmund:

{{Justus2}}

= System Architecture =

The HPC cluster is composed of [[JUSTUS2/login nodes|login nodes]], compute nodes and parallel storage systems connected by fast data networks. It is connected to the Internet via Baden Württemberg's extended LAN [https://www.belwue.de/ BelWü] (light blue).

{| style="margin: 1em auto 1em auto;"
|[[Image:JUSTUS2_Architecture.png|thumb|upright=1.5|right|Overview on JUSTUS 2 hardware architecture. All nodes are additionally connected by 1GB Ethernet. ]]
|}

Users log in on one of the four login nodes and have access to their home and working directories (darker blue) stored in the parallel file system [[Lustre]].

Two additional special login visualization node enable users to visualize compute results directly on the cluster.

Calculations are done on the several types of compute nodes (top), which are accessed via the batch queuing system [[Slurm JUSTUS 2| Slurm ]].

== Operating System and Software ==

* Operating System: [https://rockylinux.org Rocky Linux 8]
* Queuing System: [[Slurm JUSTUS 2| Slurm ]] (also see: [[bwForCluster JUSTUS 2 Slurm HOWTO|Slurm HOWTO (JUSTUS 2)]])
* [[Software_Modules_Lmod|Environment Modules]] for site specific scientific applications, developer tools and libraries

== Common Hardware Features ==

The system consists of 702 nodes (692 compute nodes and 10 dedicated login, service and visualization nodes) with 2 processors each and a total of 33,696 processor cores.

* Processor: 2 x Intel Xeon 6252 Gold (Cascade Lake, 24-core, 2.1 GHz)
* Two processors per node (2 x 24 cores)
* [https://en.wikipedia.org/wiki/Omni-Path Omni-Path] 100 Gbit/s interconnect

== Node Specifications ==

As a short overview of the table below: There are different types of compute node:

* 500 "standard" nodes (44 upgraded to "medium" nodes with more RAM later)
* 168 SSD nodes: additional RAM and fast SSD disks
* 8 large nodes: larger disks and even more RAM
* 14 nodes with GPGPU accelerators

Common to all nodes of the cluster are the following properties:
* Two CPU Chips (Sockets): 2 x [https://ark.intel.com/content/www/de/de/ark/products/192447/intel-xeon-gold-6252-processor-35-75m-cache-2-10-ghz.html Intel Xeon E6252 Gold (Cascade Lake)]
* 24 compute core per CPU, so 48 core per node
* Interconnect: [https://en.wikipedia.org/wiki/Omni-Path Omni-Path 100]

The nodes are tiered in terms of hardware configuration (amount of memory, local NVMe, hardware accelerators) in order to be able to serve a large range of different job requirements flexibly and efficiently.

{|class="wikitable zebra" style="text-align:center;"
|-
! style="width:10%"| Node Type !! Quantity !! Cores !! Memory !! Local NVMe SSD !! Accelerator
|-
! style="text-align:left;" scope="column" | Standard Nodes
| 456 || 48 || 192 GB || ---|| ---
|- style="background-color: #EEE;"
! style="text-align:left;" scope="column" | Medium Nodes
| 44 || 48 || 384 GB || ---|| ---
|-
! style="text-align:left;" scope="column" | SSD Nodes
| 148 || 48 || 384 GB || 2 x 1.6 TB (RAID 0)|| ---
|- style="background-color: #EEE;"
! style="text-align:left;" scope="column" | Medium SSD Nodes
| 20 || 48 || 768 GB || 2 x 1.6 TB (RAID 0)|| ---
|-
! style="text-align:left;" scope="column" | Large SSD Nodes
| 8 || 48 || 1536 GB || 5 x 1.6 TB (RAID 0)|| ---
|- style="background-color: #EEE;"
! style="text-align:left;" scope="column" | Special Nodes
| 14 || 48 || 192 GB || ---|| 2 x [https://www.nvidia.com/en-us/data-center/v100/ Nvidia V100S]
|-
! style="text-align:left;" scope="column" | Login- and Service Nodes
| 8 || 48 || 192 GB || 2 x 2.0 TB (RAID 1)|| ---
|- style="background-color: #EEE;"
! style="text-align:left;" scope="column" | Visualization Nodes
| 2 || 48 || 192 GB || 2 x 2.0 TB (RAID 1)|| Nvidia Quadro P4000 Graphics
|}

Core numbers: [https://en.wikipedia.org/wiki/Hyper-threading hyperthreading] is enabled, so 48 core-nodes appear to have 96 cores.

== Storage Architecture ==

The bwForCluster JUSTUS 2 provides of two independent distributed parallel file systems, one for the user's home directories <tt>$HOME</tt> and another one for global workspaces. This storage architecture is based on [https://en.wikipedia.org/wiki/Lustre_(file_system) Lustre] and can be accessed in parallel from any nodes. Additionally, some compute nodes (fast I/O nodes) provide locally attached NVMe storage devices for I/O demanding applications.

{| class="wikitable"
|-
! style="width:10%"|
! style="width:10%"| <tt>$HOME</tt>
! style="width:10%"| Workspace
! style="width:10%"| <tt>$SCRATCH</tt>
! style="width:10%"| <tt>$TMPDIR</tt>
|-
!scope="column" | Visibility
| global
| global
| node local
| node local
|-
!scope="column" | Lifetime
| permanent
| workspace lifetime (max. 90 days, extension possible)
| batch job walltime
| batch job walltime
|-
!scope="column" | Total Capacity
| 250 TB
| 1200 TB
| 3000 GB / 7300 GB per node
| max. half of RAM per node
|-
!scope="column" | Disk [https://en.wikipedia.org/wiki/Disk_quota#Quotas Quotas]
| 400 GB per user
| 20 TB per user
| none
| none
|-
!scope="column" | File [https://en.wikipedia.org/wiki/Disk_quota#Quotas Quotas]
| 2.000.000 files per user
| 5.000.000 files per user
| none
| none
|-
!scope="column" | Backup
| yes
| no
| no
| no
|}

global : accessible from all nodes
local : accessible from allocated node only
permanent : files are stored permanently (as long as user can access the system)
batch job walltime : files are removed at end of the batch job

'''Note:''' Disk and file quota limits are soft limits and are subject to change. Quotas feature a grace period where users may exceed their limits to some extent (currently 20%) for a brief period of time (currently 4 weeks).

=== $HOME ===

Home directories are meant for permanent file storage of files that are keep being used like source codes, configuration files, executable programs etc.; the content of home directories will be backed up on a regular basis.

Current disk usage on home directory and quota status can be checked with the command '''lfs quota -h -u $USER /lustre/home'''.

{{Template:AttentionBox| Compute jobs on nodes must not write temporary data to $HOME. They must use the local $SCRATCH or $TMPDIR directories for very I/O intensive jobs. They can use workspaces for less I/O intensive jobs, if needed e.g. for multi-node jobs.}}

=== Workspaces ===

More info on our general page:

:: → '''[[Workspace]]s'''

Workspaces can be generated through the [[workspace]] tools. This will generate a directory with a limited lifetime on the parallel global work file system. When this
lifetime is reached the workspace will be deleted automatically after a grace period.
Users will be notified by daily e-mail reminders starting 7 days before expiration of
a workspace. Workspaces can (and must) be extended to prevent deletion at the expiration date.

'''Defaults and maximum values'''
{| class="wikitable"
|-
| Default lifetime (days)
| 7
|-
| Maximum lifetime
| 90
|-
| Maximum extensions
| unlimited
|-
|}

'''Examples'''
{| class="wikitable"
|-
!style="width:30%" | Command
!style="width:70%" | Action
|-
|<tt>ws_allocate my_workspace 30</tt>
|Allocate a workspace named "my_workspace" for 30 days.
|-
|<tt>ws_list</tt>
|List all your workspaces.
|-
|<tt>ws_find my_workspace</tt>
|Get absolute path of workspace "my_workspace".
|-
|<tt>ws_extend my_workspace 30</tt>
|Set expiration date of workspace "my_workspace" to 30 days (regardless of remaining days).
|-
|<tt>ws_release my_workspace</tt>
|Manually erase your workspace "my_workspace" and release used space on storage (remove data first for immediate deletion of the data).
|-
|}

Current disk usage on workspace file system and quota status can be checked with the command '''lfs quota -h -u $USER /lustre/work'''.

'''Note:''' The parallel work file system works optimal for medium to large file sizes and non-[https://en.wikipedia.org/wiki/Random_access random access] patterns. Large quantities of small files significantly decrease IO performance and must be avoided. Consider using '''[https://wiki.bwhpc.de/e/BwForCluster_JUSTUS_2_Slurm_HOWTO#How_to_request_local_scratch_.28SSD.2FNVMe.29_at_job_submission.3F local scratch]''' for these.

=== $SCRATCH and $TMPDIR ===

On compute nodes the environment variables $SCRATCH and $TMPDIR always point to local scratch space that is not shared across nodes.

$TMPDIR always points to a directory on a local RAM disk which will provide up to 50% of the total RAM capacity of the node. Thus, data written to $TMPDIR will always count against allocated memory.

$SCRATCH will point to a directory on locally attached NVMe devices if (and only if) local scratch has been explicitly requested at job submission (i.e. with --gres=scratch:nnn option). If no local scratch has been requested at job submission $SCRATCH will point to the very same directory as $TMPDIR (i.e. to the RAM disk).

On the login nodes $TMPDIR and $SCRATCH point to a local scratch directory on that node. This is located at /scratch/<username> and is also not shared across nodes. The data stored in there is private but will be deleted automatically if not accessed for 7 consecutive days. Like any other local scratch space, the data stored in there is NOT included in any backup.

== Backup ==

<b>Only</b> files in your $HOME directory are included in a backup and can be restored in case of accidental deletion. The files in the "Work" filesystem are not intended for permanent storage during your jobs. Sort your results well and save important data in $HOME.

Template:AttentionBox

2026-06-24T13:54:37Z

K Siegmund:

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#ffa500; text-align:left;width: 25px"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#ffa500; text-align:left"|
{{{1}}}
|}

Template:AttentionBox

2026-06-24T13:53:23Z

K Siegmund: Created page with "{|style="background:#deffee; width:100%;" |style="padding:5px; background:#ffa500; text-align:left"| 25px |style="padding:5px; background:#ffa500; text-align:left"| {{{1}}} |}"

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#ffa500; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#ffa500; text-align:left"|
{{{1}}}
|}

Registration/bwForCluster/RV

2026-06-24T09:57:00Z

K Siegmund: /* Register a new RV */

= Step B: Apply for a Rechenvorhaben/Project =

'''Rechenvorhaben/Projects (RV)''' are required for assignment to a bwForCluster.
[[File:RV-my.png|right|frame|My RVs.]]

To get access to an RV, answer the following question:
* Does your work group (or the collaborative project that you are part of) already have an RV?
** '''Yes → [[#Become_Coworker_of_an_RV|Become Coworker of an RV]]'''
** '''No → [[#Register_a_new_RV|Register a new RV]]'''

You can see your RV memberships at the central application site ZAS under [https://zas.bwhpc.de/shib/en/info_rv.php '''My RVs'''].

== Become Coworker of an RV ==

<div style="border: 3px solid #28a745; padding: 15px; background-color: #d4edda; margin: 10px 0;">
You can only sign up for an RV once.
If you are an inactive member of an RV, either because you de-registered yourself or a manager deactivated you, only an RV manager can reactivate you.
</div>

The "RV responsible" will provide you with the following data on the RV:
* RV acronym
* RV password

If this information is lost, please let the "RV responsible" [[Registration/bwForCluster/RV/Management#Renew_RV_Password|renew the password]] for the RV.

To become coworker of an RV, follow these steps:
# Please visit the [https://zas.bwhpc.de/shib/en/bwforcluster_collaboration.php '''RV collaboration'''] web form at the [https://zas.bwhpc.de ZAS].
# Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.
# You will be directed to the ''Identity Provider'' of your home organization. Enter the username and password of your '''home organization''' (usually these credentials are also used for other services like email) and click '''Login/Einloggen'''.
# You will be redirected back to the RV collaboration form '''https://zas.bwhpc.de/shib/en/bwforcluster_collaboration.php'''.
# Enter '''RV Acronym''', '''RV Password''' and '''Institute''' into the form and check both "Terms of Use". Click '''Send'''.
# You will be assigned to the RV as a new member.
<div style="padding-left:18px;">
[[Image:Attention.svg|left|15px]]
In case of the <span style="color:red;">error</span> "This project is not currently renewed", please ask the RV responsible to apply for a prolongation of the RV. As soon as the prolongation was accepted, you can join the RV.
</div>

After submitting the request you will receive an email from ZAS about the further steps.
RV owner and RV managers will be notified automatically.

== Register a new RV ==
<div style="border: 3px solid #28a745; padding: 15px; background-color: #d4edda; margin: 10px 0;">
* An RV is valid for '''one year'''. After that, the RV responsible must apply for an extension of the RV for it to be renewed for another year.
* '''Creating multiple RVs on the same cluster is not necessary.''' Being part of multiple RVs on the same cluster does not grant any additional computing resources. The resource specifications (e.g. CPU hours) in an RV are used solely for cluster (tier) routing — to help assign your project to the most suitable cluster (tier). '''They do not represent a budget that is consumed, and they do not impose any limits on your usage.''' Every cluster user has full access to all available cluster resources regardless of how many RVs they belong to.
</div>

An RV should be registered by a leader or senior scientist of a research group/collaboration.

You must:
* be able to be held accountable for the coworkers in the RV (and will be)
* manage access to the RV (add and remove/inactivate RV members) or delegate the responsibility to an RV manager
* help write the reports required by the DFG for their funding of bwForClusters
* make sure publications that used bwHPC resources properly cite the use in the [[acknowledgement]]
* report publications which used bwHPC resources at least once a year
* be willing and able to support a future DFG grant proposal for a new bwForCluster with a "wissenschaftliches Beiblatt".
* ensure that research results of all co-workers are transferred from the cluster in a timely manner, especially before a co-worker leaves and loses access.

'''Some hints:'''
* Any number of coworkers can join the RV to work with the cluster. They need to be provided with the RV password and acronym.
* There is no need to register more than one RV per group. Just include all intended compute activities in the abstract to one RV.
* Research topic abstracts are updated when the RV is renewed for another year. Please include new or changed compute activities at that time.
* To be notified about important issues via the cluster's mailing list, it is necessary to finish all three registration steps. If you don't plan to work with the cluster yourself, we recommend to either register nevertheless or ask a colleague who is going to use the cluster to keep you updated. For better transparency, that colleague could receive the ''RV manager'' role from you.
* The [https://www.bwhpc.de/bwhpc-domains.php research area] of the group as well as the software and hardware requirements are taken into account.
*: Please look carefully through the scientific fields and research domains to find the field that fits best to your research focus and experience. This way, you'll be assigned to the cluster that can support you best. If in doubt, you can check the [https://www.dfg.de/resource/blob/175334/89ba4a3464c99aaea40fdef47367e7b2/fachsystematik-2020-2024-de-grafik-data.pdf DFG classification] as well.

* Getting access to two different clusters is only possible if specific software or hardware needs demand access for both.

[[File:RV-reg.png|right|frame|Register a new RV.]]
[[File:BwIDM-login.png|right|250px|thumb|Select your home organization]]
'''To apply for a new RV, follow these steps:'''

# Please visit the [https://zas.bwhpc.de/shib/en/bwforcluster_proposal.php '''RV registration'''] web form at the [https://zas.bwhpc.de ZAS].
# Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.
# You will be directed to the ''Identity Provider'' of your home organization. Enter the username and password of your '''home organization''' (usually these credentials are also used for other services like email) and click '''Login/Einloggen'''.
# You will be redirected back to the RV registration form '''https://zas.bwhpc.de/shib/en/bwforcluster_proposal.php'''.
# Fill in all mandatory fields of the form and tick the two "Terms of use". Detailed information on the individual fields can be found in the section [[Registration/bwForCluster/RV#Registration_Form_Details|Registration Form Details]]. Click '''Preview'''.
# If the information is correct, click '''Submit''', otherwise you can customize your application.
# Go to section [[Registration/bwForCluster/RV#Registration_Response|Registration Response]] for the next steps.

Please note that for your convenience you can also switch to the [https://zas.bwhpc.de/shib/bwforcluster_proposal.php German version] of the web form.
The submitter of the RV will be assigned the role "RV responsible" (see [[Registration/bwForCluster/RV/Management|Roles in an RV]]).

=== Registration Form Details ===

The web form consists of the following fields.
Note that the fields - name, given name, organization, mail, and EPPN - are auto-filled and can not be changed.
These are your credentials as given by your home organization.

{| style="width:100%;border:3px solid darkgray; margin: 1em auto 1em auto;"
|-
! style="width:20%;padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;"| Field
! style="padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;" | Explanation
|rowspan="15"| [[File:RV-form.png|center|400px|thumb|Registering a new RV.]]
|-
!scope="column" {{Lightgray_small}}| RV Title
|scope="column" {{Tab element}} | Define a short title of your planned compute activities, maximum of 255 characters including spaces.
|-
!scope="column" {{Lightgray_small}}| RV Description
|scope="column" {{Tab element}} | Write a short abstract about your planned compute activities (maximum of 2048 characters including spaces). Research groups that contributed to the DFG grant proposal (Art. 91b GG) of the corresponding bwForCluster only need to give reference to that particular proposal.
|-
!scope="column" {{Lightgray_small}}| Scientific field
|scope="column" {{Tab element}} | Tick one or several scientific fields. Once all bwForClusters are up and running the full list of given scientific fields are supported and hence for RV applicable. If your RV does not match any given scientific field, please state your scientific field in the given text box. Grayed out scientific fields indicate that the corresponding bwForCluster(s) is/are not operational yet.
|-
!scope="column" {{Lightgray_small}}| Field of activity
|scope="column" {{Tab element}} | Define if your RV is primarily for research and/or teaching. If not applicable, use text box.
|-
!scope="column" {{Lightgray_small}}| Parallel paradigm
|scope="column" {{Tab element}} | State what parallel paradigm your code or application uses. Multiple ticks are allowed. Further information can be provided via text box. If you are not sure about it, please state the software you are using.
|-
!scope="column" {{Lightgray_small}}| Programming language
|scope="column" {{Tab element}} | State the programming language(s) of your code or application.
|-
!scope="column" {{Lightgray_small}}| Numerical methods
|scope="column" {{Tab element}} | State the numerical or "calculation" methods your code or application utilizes. If you do not know, write "unknown".
|-
!scope="column" {{Lightgray_small}}| Software packages
|scope="column" {{Tab element}} | State all software package you will or want to use for your RV. Also include compiler, MPI and numerical packages (e.g. MKL, FFTW).
|-
!scope="column" {{Lightgray_small}}| Estimated requested computing resources
|scope="column" {{Tab element}} | Roughly estimate how many CPU hours are required to finish your RV. To calculate "CPU hours" multiply "elapsed parallel computing time" with "number of CPU core" per job.
''Example: Your code uses 4 CPU cores and has a wall time of 1h. This makes 4 CPU hours.''
|-
!scope="column" {{Lightgray_small}}| Planned maximum number of parallel used CPU cores per job
|scope="column" {{Tab element}} | Give an upper limit of how many CPU cores per job are required. Please cross-check with your statement about parallel paradigm.
''Obviously, a sequential code can only use 1 CPU core per job.''
|-
!scope="column" {{Lightgray_small}}| Estimated maximal memory requirements per CPU core
|scope="column" {{Tab element}} | Give an upper limit of how much RAM per CPU core your jobs require. Please give a value in Gigabytes.
|-
!scope="column" {{Lightgray_small}}| Estimated requested persistent disk space for the RV
|scope="column" {{Tab element}} | State how much disk space in total you need for your RV. Please give a value in Gigabytes.
''Example: If your RV has 4 more coworker and each of you produces until the end of your RV 20 Gigabyte of output, your maximum disk space is 100 Gigabyte.''
|-
!scope="column" {{Lightgray_small}}| Estimated maximal temporary disk space per job
|scope="column" {{Tab element}} | State how much space your intermediate files require during a job run. Please give a value in Gigabytes.
''Example: Your final output of your job is 0.1 Gigabyte, but during your job 10 temporary files, each with a size of 1 Gigabyte, are written to disk, hence your correct answer is 10 Gigabyte.''
|-
!scope="column" {{Lightgray_small}}| Institute
|scope="column" {{Tab element}} | State the full name of your Institute.
|-
|}

=== Registration Response ===

After submitting you will get an email from the ZAS confirming your submission.
With this email you are given an unique
* RV acronym
* RV password

Please keep the password enclosed.
With RV acronym and password, [[Registration/bwForCluster/RV#Become_Coworker_of_a_RV| coworkers ]] can already associate themselves to the RV.
If this information is lost, please [[Registration/bwForCluster/RV#Renew_RV_Password|renew the password]] for the RV.

The cluster assignment team assigns a bwForCluster to your RV based on your submitted data, including the [https://www.bwhpc.de/bwhpc-domains.php research area] as well as the software and hardware requirements.
The ZAS notifies you in an email about your assigned bwForCluster and provides website details for [[Registration/bwForCluster/Service| Step C]]. Later on you can find details about the roles and management possibilities at the [[Registration/bwForCluster/RV/Management | RV Management]] page.

=== Duration of an RV ===

An RV is valid for '''one year'''.
After that, the RV responsible must apply for an extension of the RV for it to be renewed for another year.

=== RV Management for RV Responsibles and Managers ===

RV responsible or manager can [[Registration/bwForCluster/RV/Management|manage and modify Rechenvorhaben/projects]].

----

<p style="text-align:right;">[[Registration/bwForCluster/Service | Go to step C]]</p>

Registration/bwForCluster/RV

2026-06-24T09:56:46Z

K Siegmund: /* Register a new RV */

= Step B: Apply for a Rechenvorhaben/Project =

'''Rechenvorhaben/Projects (RV)''' are required for assignment to a bwForCluster.
[[File:RV-my.png|right|frame|My RVs.]]

To get access to an RV, answer the following question:
* Does your work group (or the collaborative project that you are part of) already have an RV?
** '''Yes → [[#Become_Coworker_of_an_RV|Become Coworker of an RV]]'''
** '''No → [[#Register_a_new_RV|Register a new RV]]'''

You can see your RV memberships at the central application site ZAS under [https://zas.bwhpc.de/shib/en/info_rv.php '''My RVs'''].

== Become Coworker of an RV ==

<div style="border: 3px solid #28a745; padding: 15px; background-color: #d4edda; margin: 10px 0;">
You can only sign up for an RV once.
If you are an inactive member of an RV, either because you de-registered yourself or a manager deactivated you, only an RV manager can reactivate you.
</div>

The "RV responsible" will provide you with the following data on the RV:
* RV acronym
* RV password

If this information is lost, please let the "RV responsible" [[Registration/bwForCluster/RV/Management#Renew_RV_Password|renew the password]] for the RV.

To become coworker of an RV, follow these steps:
# Please visit the [https://zas.bwhpc.de/shib/en/bwforcluster_collaboration.php '''RV collaboration'''] web form at the [https://zas.bwhpc.de ZAS].
# Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.
# You will be directed to the ''Identity Provider'' of your home organization. Enter the username and password of your '''home organization''' (usually these credentials are also used for other services like email) and click '''Login/Einloggen'''.
# You will be redirected back to the RV collaboration form '''https://zas.bwhpc.de/shib/en/bwforcluster_collaboration.php'''.
# Enter '''RV Acronym''', '''RV Password''' and '''Institute''' into the form and check both "Terms of Use". Click '''Send'''.
# You will be assigned to the RV as a new member.
<div style="padding-left:18px;">
[[Image:Attention.svg|left|15px]]
In case of the <span style="color:red;">error</span> "This project is not currently renewed", please ask the RV responsible to apply for a prolongation of the RV. As soon as the prolongation was accepted, you can join the RV.
</div>

After submitting the request you will receive an email from ZAS about the further steps.
RV owner and RV managers will be notified automatically.

== Register a new RV ==
<div style="border: 3px solid #28a745; padding: 15px; background-color: #d4edda; margin: 10px 0;">
* An RV is valid for '''one year'''. After that, the RV responsible must apply for an extension of the RV for it to be renewed for another year.
* '''Creating multiple RVs on the same cluster is not necessary.''' Being part of multiple RVs on the same cluster does not grant any additional computing resources. The resource specifications (e.g. CPU hours) in an RV are used solely for cluster (tier) routing — to help assign your project to the most suitable cluster (tier). '''They do not represent a budget that is consumed, and they do not impose any limits on your usage.''' Every cluster user has full access to all available cluster resources regardless of how many RVs they belong to.
</div>

An RV should be registered by a leader or senior scientist of a research group/collaboration.
You must:
* be able to be held accountable for the coworkers in the RV (and will be)
* manage access to the RV (add and remove/inactivate RV members) or delegate the responsibility to an RV manager
* help write the reports required by the DFG for their funding of bwForClusters
* make sure publications that used bwHPC resources properly cite the use in the [[acknowledgement]]
* report publications which used bwHPC resources at least once a year
* be willing and able to support a future DFG grant proposal for a new bwForCluster with a "wissenschaftliches Beiblatt".
* ensure that research results of all co-workers are transferred from the cluster in a timely manner, especially before a co-worker leaves and loses access.

'''Some hints:'''
* Any number of coworkers can join the RV to work with the cluster. They need to be provided with the RV password and acronym.
* There is no need to register more than one RV per group. Just include all intended compute activities in the abstract to one RV.
* Research topic abstracts are updated when the RV is renewed for another year. Please include new or changed compute activities at that time.
* To be notified about important issues via the cluster's mailing list, it is necessary to finish all three registration steps. If you don't plan to work with the cluster yourself, we recommend to either register nevertheless or ask a colleague who is going to use the cluster to keep you updated. For better transparency, that colleague could receive the ''RV manager'' role from you.
* The [https://www.bwhpc.de/bwhpc-domains.php research area] of the group as well as the software and hardware requirements are taken into account.
*: Please look carefully through the scientific fields and research domains to find the field that fits best to your research focus and experience. This way, you'll be assigned to the cluster that can support you best. If in doubt, you can check the [https://www.dfg.de/resource/blob/175334/89ba4a3464c99aaea40fdef47367e7b2/fachsystematik-2020-2024-de-grafik-data.pdf DFG classification] as well.

* Getting access to two different clusters is only possible if specific software or hardware needs demand access for both.

[[File:RV-reg.png|right|frame|Register a new RV.]]
[[File:BwIDM-login.png|right|250px|thumb|Select your home organization]]
'''To apply for a new RV, follow these steps:'''

# Please visit the [https://zas.bwhpc.de/shib/en/bwforcluster_proposal.php '''RV registration'''] web form at the [https://zas.bwhpc.de ZAS].
# Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.
# You will be directed to the ''Identity Provider'' of your home organization. Enter the username and password of your '''home organization''' (usually these credentials are also used for other services like email) and click '''Login/Einloggen'''.
# You will be redirected back to the RV registration form '''https://zas.bwhpc.de/shib/en/bwforcluster_proposal.php'''.
# Fill in all mandatory fields of the form and tick the two "Terms of use". Detailed information on the individual fields can be found in the section [[Registration/bwForCluster/RV#Registration_Form_Details|Registration Form Details]]. Click '''Preview'''.
# If the information is correct, click '''Submit''', otherwise you can customize your application.
# Go to section [[Registration/bwForCluster/RV#Registration_Response|Registration Response]] for the next steps.

Please note that for your convenience you can also switch to the [https://zas.bwhpc.de/shib/bwforcluster_proposal.php German version] of the web form.
The submitter of the RV will be assigned the role "RV responsible" (see [[Registration/bwForCluster/RV/Management|Roles in an RV]]).

=== Registration Form Details ===

The web form consists of the following fields.
Note that the fields - name, given name, organization, mail, and EPPN - are auto-filled and can not be changed.
These are your credentials as given by your home organization.

{| style="width:100%;border:3px solid darkgray; margin: 1em auto 1em auto;"
|-
! style="width:20%;padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;"| Field
! style="padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;" | Explanation
|rowspan="15"| [[File:RV-form.png|center|400px|thumb|Registering a new RV.]]
|-
!scope="column" {{Lightgray_small}}| RV Title
|scope="column" {{Tab element}} | Define a short title of your planned compute activities, maximum of 255 characters including spaces.
|-
!scope="column" {{Lightgray_small}}| RV Description
|scope="column" {{Tab element}} | Write a short abstract about your planned compute activities (maximum of 2048 characters including spaces). Research groups that contributed to the DFG grant proposal (Art. 91b GG) of the corresponding bwForCluster only need to give reference to that particular proposal.
|-
!scope="column" {{Lightgray_small}}| Scientific field
|scope="column" {{Tab element}} | Tick one or several scientific fields. Once all bwForClusters are up and running the full list of given scientific fields are supported and hence for RV applicable. If your RV does not match any given scientific field, please state your scientific field in the given text box. Grayed out scientific fields indicate that the corresponding bwForCluster(s) is/are not operational yet.
|-
!scope="column" {{Lightgray_small}}| Field of activity
|scope="column" {{Tab element}} | Define if your RV is primarily for research and/or teaching. If not applicable, use text box.
|-
!scope="column" {{Lightgray_small}}| Parallel paradigm
|scope="column" {{Tab element}} | State what parallel paradigm your code or application uses. Multiple ticks are allowed. Further information can be provided via text box. If you are not sure about it, please state the software you are using.
|-
!scope="column" {{Lightgray_small}}| Programming language
|scope="column" {{Tab element}} | State the programming language(s) of your code or application.
|-
!scope="column" {{Lightgray_small}}| Numerical methods
|scope="column" {{Tab element}} | State the numerical or "calculation" methods your code or application utilizes. If you do not know, write "unknown".
|-
!scope="column" {{Lightgray_small}}| Software packages
|scope="column" {{Tab element}} | State all software package you will or want to use for your RV. Also include compiler, MPI and numerical packages (e.g. MKL, FFTW).
|-
!scope="column" {{Lightgray_small}}| Estimated requested computing resources
|scope="column" {{Tab element}} | Roughly estimate how many CPU hours are required to finish your RV. To calculate "CPU hours" multiply "elapsed parallel computing time" with "number of CPU core" per job.
''Example: Your code uses 4 CPU cores and has a wall time of 1h. This makes 4 CPU hours.''
|-
!scope="column" {{Lightgray_small}}| Planned maximum number of parallel used CPU cores per job
|scope="column" {{Tab element}} | Give an upper limit of how many CPU cores per job are required. Please cross-check with your statement about parallel paradigm.
''Obviously, a sequential code can only use 1 CPU core per job.''
|-
!scope="column" {{Lightgray_small}}| Estimated maximal memory requirements per CPU core
|scope="column" {{Tab element}} | Give an upper limit of how much RAM per CPU core your jobs require. Please give a value in Gigabytes.
|-
!scope="column" {{Lightgray_small}}| Estimated requested persistent disk space for the RV
|scope="column" {{Tab element}} | State how much disk space in total you need for your RV. Please give a value in Gigabytes.
''Example: If your RV has 4 more coworker and each of you produces until the end of your RV 20 Gigabyte of output, your maximum disk space is 100 Gigabyte.''
|-
!scope="column" {{Lightgray_small}}| Estimated maximal temporary disk space per job
|scope="column" {{Tab element}} | State how much space your intermediate files require during a job run. Please give a value in Gigabytes.
''Example: Your final output of your job is 0.1 Gigabyte, but during your job 10 temporary files, each with a size of 1 Gigabyte, are written to disk, hence your correct answer is 10 Gigabyte.''
|-
!scope="column" {{Lightgray_small}}| Institute
|scope="column" {{Tab element}} | State the full name of your Institute.
|-
|}

=== Registration Response ===

After submitting you will get an email from the ZAS confirming your submission.
With this email you are given an unique
* RV acronym
* RV password

Please keep the password enclosed.
With RV acronym and password, [[Registration/bwForCluster/RV#Become_Coworker_of_a_RV| coworkers ]] can already associate themselves to the RV.
If this information is lost, please [[Registration/bwForCluster/RV#Renew_RV_Password|renew the password]] for the RV.

The cluster assignment team assigns a bwForCluster to your RV based on your submitted data, including the [https://www.bwhpc.de/bwhpc-domains.php research area] as well as the software and hardware requirements.
The ZAS notifies you in an email about your assigned bwForCluster and provides website details for [[Registration/bwForCluster/Service| Step C]]. Later on you can find details about the roles and management possibilities at the [[Registration/bwForCluster/RV/Management | RV Management]] page.

=== Duration of an RV ===

An RV is valid for '''one year'''.
After that, the RV responsible must apply for an extension of the RV for it to be renewed for another year.

=== RV Management for RV Responsibles and Managers ===

RV responsible or manager can [[Registration/bwForCluster/RV/Management|manage and modify Rechenvorhaben/projects]].

----

<p style="text-align:right;">[[Registration/bwForCluster/Service | Go to step C]]</p>

Registration/bwForCluster/RV

2026-06-24T09:50:57Z

K Siegmund: /* Register a new RV */

= Step B: Apply for a Rechenvorhaben/Project =

'''Rechenvorhaben/Projects (RV)''' are required for assignment to a bwForCluster.
[[File:RV-my.png|right|frame|My RVs.]]

To get access to an RV, answer the following question:
* Does your work group (or the collaborative project that you are part of) already have an RV?
** '''Yes → [[#Become_Coworker_of_an_RV|Become Coworker of an RV]]'''
** '''No → [[#Register_a_new_RV|Register a new RV]]'''

You can see your RV memberships at the central application site ZAS under [https://zas.bwhpc.de/shib/en/info_rv.php '''My RVs'''].

== Become Coworker of an RV ==

<div style="border: 3px solid #28a745; padding: 15px; background-color: #d4edda; margin: 10px 0;">
You can only sign up for an RV once.
If you are an inactive member of an RV, either because you de-registered yourself or a manager deactivated you, only an RV manager can reactivate you.
</div>

The "RV responsible" will provide you with the following data on the RV:
* RV acronym
* RV password

If this information is lost, please let the "RV responsible" [[Registration/bwForCluster/RV/Management#Renew_RV_Password|renew the password]] for the RV.

To become coworker of an RV, follow these steps:
# Please visit the [https://zas.bwhpc.de/shib/en/bwforcluster_collaboration.php '''RV collaboration'''] web form at the [https://zas.bwhpc.de ZAS].
# Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.
# You will be directed to the ''Identity Provider'' of your home organization. Enter the username and password of your '''home organization''' (usually these credentials are also used for other services like email) and click '''Login/Einloggen'''.
# You will be redirected back to the RV collaboration form '''https://zas.bwhpc.de/shib/en/bwforcluster_collaboration.php'''.
# Enter '''RV Acronym''', '''RV Password''' and '''Institute''' into the form and check both "Terms of Use". Click '''Send'''.
# You will be assigned to the RV as a new member.
<div style="padding-left:18px;">
[[Image:Attention.svg|left|15px]]
In case of the <span style="color:red;">error</span> "This project is not currently renewed", please ask the RV responsible to apply for a prolongation of the RV. As soon as the prolongation was accepted, you can join the RV.
</div>

After submitting the request you will receive an email from ZAS about the further steps.
RV owner and RV managers will be notified automatically.

== Register a new RV ==
<div style="border: 3px solid #28a745; padding: 15px; background-color: #d4edda; margin: 10px 0;">
* An RV is valid for '''one year'''. After that, the RV responsible must apply for an extension of the RV for it to be renewed for another year.
* '''Creating multiple RVs on the same cluster is not necessary.''' Being part of multiple RVs on the same cluster does not grant any additional computing resources. The resource specifications (e.g. CPU hours) in an RV are used solely for cluster (tier) routing — to help assign your project to the most suitable cluster (tier). '''They do not represent a budget that is consumed, and they do not impose any limits on your usage.''' Every cluster user has full access to all available cluster resources regardless of how many RVs they belong to.
</div>

An RV should be registered by a leader or senior scientist of a research group/collaboration.
You...:
* will be held accountable for the coworkers in the RV
* must help write the reports required by the DFG for their funding of bwForClusters
* must make sure publications that used bwHPC resources properly cite the use in the [[acknowledgement]]
* must report publications which used bwHPC resources at least once a year
* must be willing and able to support a future DFG grant proposal for a new bwForCluster with a "wissenschaftliches Beiblatt".

'''Some hints:'''
* Any number of coworkers can join the RV to work with the cluster. They need to be provided with the RV password and acronym.
* There is no need to register more than one RV per group. Just include all intended compute activities in the abstract to one RV.
* Research topic abstracts are updated when the RV is renewed for another year. Please include new or changed compute activities at that time.
* To be notified about important issues via the cluster's mailing list, it is necessary to finish all three registration steps. If you don't plan to work with the cluster yourself, we recommend to either register nevertheless or ask a colleague who is going to use the cluster to keep you updated. For better transparency, that colleague could receive the ''RV manager'' role from you.
* The [https://www.bwhpc.de/bwhpc-domains.php research area] of the group as well as the software and hardware requirements are taken into account.
*: Please look carefully through the scientific fields and research domains to find the field that fits best to your research focus and experience. This way, you'll be assigned to the cluster that can support you best. If in doubt, you can check the [https://www.dfg.de/resource/blob/175334/89ba4a3464c99aaea40fdef47367e7b2/fachsystematik-2020-2024-de-grafik-data.pdf DFG classification] as well.

* Getting access to two different clusters is only possible if specific software or hardware needs demand access for both.

[[File:RV-reg.png|right|frame|Register a new RV.]]
[[File:BwIDM-login.png|right|250px|thumb|Select your home organization]]
'''To apply for a new RV, follow these steps:'''

# Please visit the [https://zas.bwhpc.de/shib/en/bwforcluster_proposal.php '''RV registration'''] web form at the [https://zas.bwhpc.de ZAS].
# Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.
# You will be directed to the ''Identity Provider'' of your home organization. Enter the username and password of your '''home organization''' (usually these credentials are also used for other services like email) and click '''Login/Einloggen'''.
# You will be redirected back to the RV registration form '''https://zas.bwhpc.de/shib/en/bwforcluster_proposal.php'''.
# Fill in all mandatory fields of the form and tick the two "Terms of use". Detailed information on the individual fields can be found in the section [[Registration/bwForCluster/RV#Registration_Form_Details|Registration Form Details]]. Click '''Preview'''.
# If the information is correct, click '''Submit''', otherwise you can customize your application.
# Go to section [[Registration/bwForCluster/RV#Registration_Response|Registration Response]] for the next steps.

Please note that for your convenience you can also switch to the [https://zas.bwhpc.de/shib/bwforcluster_proposal.php German version] of the web form.
The submitter of the RV will be assigned the role "RV responsible" (see [[Registration/bwForCluster/RV/Management|Roles in an RV]]).

=== Registration Form Details ===

The web form consists of the following fields.
Note that the fields - name, given name, organization, mail, and EPPN - are auto-filled and can not be changed.
These are your credentials as given by your home organization.

{| style="width:100%;border:3px solid darkgray; margin: 1em auto 1em auto;"
|-
! style="width:20%;padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;"| Field
! style="padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;" | Explanation
|rowspan="15"| [[File:RV-form.png|center|400px|thumb|Registering a new RV.]]
|-
!scope="column" {{Lightgray_small}}| RV Title
|scope="column" {{Tab element}} | Define a short title of your planned compute activities, maximum of 255 characters including spaces.
|-
!scope="column" {{Lightgray_small}}| RV Description
|scope="column" {{Tab element}} | Write a short abstract about your planned compute activities (maximum of 2048 characters including spaces). Research groups that contributed to the DFG grant proposal (Art. 91b GG) of the corresponding bwForCluster only need to give reference to that particular proposal.
|-
!scope="column" {{Lightgray_small}}| Scientific field
|scope="column" {{Tab element}} | Tick one or several scientific fields. Once all bwForClusters are up and running the full list of given scientific fields are supported and hence for RV applicable. If your RV does not match any given scientific field, please state your scientific field in the given text box. Grayed out scientific fields indicate that the corresponding bwForCluster(s) is/are not operational yet.
|-
!scope="column" {{Lightgray_small}}| Field of activity
|scope="column" {{Tab element}} | Define if your RV is primarily for research and/or teaching. If not applicable, use text box.
|-
!scope="column" {{Lightgray_small}}| Parallel paradigm
|scope="column" {{Tab element}} | State what parallel paradigm your code or application uses. Multiple ticks are allowed. Further information can be provided via text box. If you are not sure about it, please state the software you are using.
|-
!scope="column" {{Lightgray_small}}| Programming language
|scope="column" {{Tab element}} | State the programming language(s) of your code or application.
|-
!scope="column" {{Lightgray_small}}| Numerical methods
|scope="column" {{Tab element}} | State the numerical or "calculation" methods your code or application utilizes. If you do not know, write "unknown".
|-
!scope="column" {{Lightgray_small}}| Software packages
|scope="column" {{Tab element}} | State all software package you will or want to use for your RV. Also include compiler, MPI and numerical packages (e.g. MKL, FFTW).
|-
!scope="column" {{Lightgray_small}}| Estimated requested computing resources
|scope="column" {{Tab element}} | Roughly estimate how many CPU hours are required to finish your RV. To calculate "CPU hours" multiply "elapsed parallel computing time" with "number of CPU core" per job.
''Example: Your code uses 4 CPU cores and has a wall time of 1h. This makes 4 CPU hours.''
|-
!scope="column" {{Lightgray_small}}| Planned maximum number of parallel used CPU cores per job
|scope="column" {{Tab element}} | Give an upper limit of how many CPU cores per job are required. Please cross-check with your statement about parallel paradigm.
''Obviously, a sequential code can only use 1 CPU core per job.''
|-
!scope="column" {{Lightgray_small}}| Estimated maximal memory requirements per CPU core
|scope="column" {{Tab element}} | Give an upper limit of how much RAM per CPU core your jobs require. Please give a value in Gigabytes.
|-
!scope="column" {{Lightgray_small}}| Estimated requested persistent disk space for the RV
|scope="column" {{Tab element}} | State how much disk space in total you need for your RV. Please give a value in Gigabytes.
''Example: If your RV has 4 more coworker and each of you produces until the end of your RV 20 Gigabyte of output, your maximum disk space is 100 Gigabyte.''
|-
!scope="column" {{Lightgray_small}}| Estimated maximal temporary disk space per job
|scope="column" {{Tab element}} | State how much space your intermediate files require during a job run. Please give a value in Gigabytes.
''Example: Your final output of your job is 0.1 Gigabyte, but during your job 10 temporary files, each with a size of 1 Gigabyte, are written to disk, hence your correct answer is 10 Gigabyte.''
|-
!scope="column" {{Lightgray_small}}| Institute
|scope="column" {{Tab element}} | State the full name of your Institute.
|-
|}

=== Registration Response ===

After submitting you will get an email from the ZAS confirming your submission.
With this email you are given an unique
* RV acronym
* RV password

Please keep the password enclosed.
With RV acronym and password, [[Registration/bwForCluster/RV#Become_Coworker_of_a_RV| coworkers ]] can already associate themselves to the RV.
If this information is lost, please [[Registration/bwForCluster/RV#Renew_RV_Password|renew the password]] for the RV.

The cluster assignment team assigns a bwForCluster to your RV based on your submitted data, including the [https://www.bwhpc.de/bwhpc-domains.php research area] as well as the software and hardware requirements.
The ZAS notifies you in an email about your assigned bwForCluster and provides website details for [[Registration/bwForCluster/Service| Step C]]. Later on you can find details about the roles and management possibilities at the [[Registration/bwForCluster/RV/Management | RV Management]] page.

=== Duration of an RV ===

An RV is valid for '''one year'''.
After that, the RV responsible must apply for an extension of the RV for it to be renewed for another year.

=== RV Management for RV Responsibles and Managers ===

RV responsible or manager can [[Registration/bwForCluster/RV/Management|manage and modify Rechenvorhaben/projects]].

----

<p style="text-align:right;">[[Registration/bwForCluster/Service | Go to step C]]</p>

Registration/bwForCluster/RV

2026-06-24T09:37:35Z

K Siegmund: /* Register a new RV */

= Step B: Apply for a Rechenvorhaben/Project =

'''Rechenvorhaben/Projects (RV)''' are required for assignment to a bwForCluster.
[[File:RV-my.png|right|frame|My RVs.]]

To get access to an RV, answer the following question:
* Does your work group (or the collaborative project that you are part of) already have an RV?
** '''Yes → [[#Become_Coworker_of_an_RV|Become Coworker of an RV]]'''
** '''No → [[#Register_a_new_RV|Register a new RV]]'''

You can see your RV memberships at the central application site ZAS under [https://zas.bwhpc.de/shib/en/info_rv.php '''My RVs'''].

== Become Coworker of an RV ==

<div style="border: 3px solid #28a745; padding: 15px; background-color: #d4edda; margin: 10px 0;">
You can only sign up for an RV once.
If you are an inactive member of an RV, either because you de-registered yourself or a manager deactivated you, only an RV manager can reactivate you.
</div>

The "RV responsible" will provide you with the following data on the RV:
* RV acronym
* RV password

If this information is lost, please let the "RV responsible" [[Registration/bwForCluster/RV/Management#Renew_RV_Password|renew the password]] for the RV.

To become coworker of an RV, follow these steps:
# Please visit the [https://zas.bwhpc.de/shib/en/bwforcluster_collaboration.php '''RV collaboration'''] web form at the [https://zas.bwhpc.de ZAS].
# Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.
# You will be directed to the ''Identity Provider'' of your home organization. Enter the username and password of your '''home organization''' (usually these credentials are also used for other services like email) and click '''Login/Einloggen'''.
# You will be redirected back to the RV collaboration form '''https://zas.bwhpc.de/shib/en/bwforcluster_collaboration.php'''.
# Enter '''RV Acronym''', '''RV Password''' and '''Institute''' into the form and check both "Terms of Use". Click '''Send'''.
# You will be assigned to the RV as a new member.
<div style="padding-left:18px;">
[[Image:Attention.svg|left|15px]]
In case of the <span style="color:red;">error</span> "This project is not currently renewed", please ask the RV responsible to apply for a prolongation of the RV. As soon as the prolongation was accepted, you can join the RV.
</div>

After submitting the request you will receive an email from ZAS about the further steps.
RV owner and RV managers will be notified automatically.

== Register a new RV ==
<div style="border: 3px solid #28a745; padding: 15px; background-color: #d4edda; margin: 10px 0;">
* An RV is valid for '''one year'''. After that, the RV responsible must apply for an extension of the RV for it to be renewed for another year.
* '''Creating multiple RVs on the same cluster is not necessary.''' Being part of multiple RVs on the same cluster does not grant any additional computing resources. The resource specifications (e.g. CPU hours) in an RV are used solely for cluster (tier) routing — to help assign your project to the most suitable cluster (tier). '''They do not represent a budget that is consumed, and they do not impose any limits on your usage.''' Every cluster user has full access to all available cluster resources regardless of how many RVs they belong to.
</div>

An RV should be registered by a leader or senior scientist of a research group/collaboration.
You...:
* will be held accountable for the coworkers in the RV
* must help write the reports required by the DFG for their funding of bwForClusters
* must make sure publications that used bwHPC resources properly cite the use in the [[acknowledgement]]
* must report publications which used bwHPC resources at least once a year
must be willing and able to support a future DFG grant proposal for a new bwForCluster with a "wissenschaftliches Beiblatt".

'''Some hints:'''
* To be notified about important issues via the cluster's mailing list, it is necessary to finish all three registration steps. If you don't plan to work with the cluster yourself, we recommend to either register nevertheless or ask a colleague who is going to use the cluster to keep you updated. For better transparency, that colleague could receive the ''RV manager'' role from you.
* Any number of coworkers can join the RV to work with the cluster.Therefore, they need to be provided with the RV password and acronym.
* The [https://www.bwhpc.de/bwhpc-domains.php research area] of the group as well as the software and hardware requirements are taken into account.
*: Please look carefully through the scientific fields and research domains to find the field that fits best to your research focus and experience. This way, you'll be assigned to the cluster that can support you best. If in doubt, you can check the [https://www.dfg.de/resource/blob/175334/89ba4a3464c99aaea40fdef47367e7b2/fachsystematik-2020-2024-de-grafik-data.pdf DFG classification] as well.
* Research topics also do not need to match the RV description exactly. However, we appreciate it if you keep the RV information up to date when submitting a prolongation application.
* Getting access to two different clusters is only possible if specific software or hardware needs demand access for both.

[[File:RV-reg.png|right|frame|Register a new RV.]]
[[File:BwIDM-login.png|right|250px|thumb|Select your home organization]]
'''To apply for a new RV, follow these steps:'''

# Please visit the [https://zas.bwhpc.de/shib/en/bwforcluster_proposal.php '''RV registration'''] web form at the [https://zas.bwhpc.de ZAS].
# Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.
# You will be directed to the ''Identity Provider'' of your home organization. Enter the username and password of your '''home organization''' (usually these credentials are also used for other services like email) and click '''Login/Einloggen'''.
# You will be redirected back to the RV registration form '''https://zas.bwhpc.de/shib/en/bwforcluster_proposal.php'''.
# Fill in all mandatory fields of the form and tick the two "Terms of use". Detailed information on the individual fields can be found in the section [[Registration/bwForCluster/RV#Registration_Form_Details|Registration Form Details]]. Click '''Preview'''.
# If the information is correct, click '''Submit''', otherwise you can customize your application.
# Go to section [[Registration/bwForCluster/RV#Registration_Response|Registration Response]] for the next steps.

Please note that for your convenience you can also switch to the [https://zas.bwhpc.de/shib/bwforcluster_proposal.php German version] of the web form.
The submitter of the RV will be assigned the role "RV responsible" (see [[Registration/bwForCluster/RV/Management|Roles in an RV]]).

=== Registration Form Details ===

The web form consists of the following fields.
Note that the fields - name, given name, organization, mail, and EPPN - are auto-filled and can not be changed.
These are your credentials as given by your home organization.

{| style="width:100%;border:3px solid darkgray; margin: 1em auto 1em auto;"
|-
! style="width:20%;padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;"| Field
! style="padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;" | Explanation
|rowspan="15"| [[File:RV-form.png|center|400px|thumb|Registering a new RV.]]
|-
!scope="column" {{Lightgray_small}}| RV Title
|scope="column" {{Tab element}} | Define a short title of your planned compute activities, maximum of 255 characters including spaces.
|-
!scope="column" {{Lightgray_small}}| RV Description
|scope="column" {{Tab element}} | Write a short abstract about your planned compute activities (maximum of 2048 characters including spaces). Research groups that contributed to the DFG grant proposal (Art. 91b GG) of the corresponding bwForCluster only need to give reference to that particular proposal.
|-
!scope="column" {{Lightgray_small}}| Scientific field
|scope="column" {{Tab element}} | Tick one or several scientific fields. Once all bwForClusters are up and running the full list of given scientific fields are supported and hence for RV applicable. If your RV does not match any given scientific field, please state your scientific field in the given text box. Grayed out scientific fields indicate that the corresponding bwForCluster(s) is/are not operational yet.
|-
!scope="column" {{Lightgray_small}}| Field of activity
|scope="column" {{Tab element}} | Define if your RV is primarily for research and/or teaching. If not applicable, use text box.
|-
!scope="column" {{Lightgray_small}}| Parallel paradigm
|scope="column" {{Tab element}} | State what parallel paradigm your code or application uses. Multiple ticks are allowed. Further information can be provided via text box. If you are not sure about it, please state the software you are using.
|-
!scope="column" {{Lightgray_small}}| Programming language
|scope="column" {{Tab element}} | State the programming language(s) of your code or application.
|-
!scope="column" {{Lightgray_small}}| Numerical methods
|scope="column" {{Tab element}} | State the numerical or "calculation" methods your code or application utilizes. If you do not know, write "unknown".
|-
!scope="column" {{Lightgray_small}}| Software packages
|scope="column" {{Tab element}} | State all software package you will or want to use for your RV. Also include compiler, MPI and numerical packages (e.g. MKL, FFTW).
|-
!scope="column" {{Lightgray_small}}| Estimated requested computing resources
|scope="column" {{Tab element}} | Roughly estimate how many CPU hours are required to finish your RV. To calculate "CPU hours" multiply "elapsed parallel computing time" with "number of CPU core" per job.
''Example: Your code uses 4 CPU cores and has a wall time of 1h. This makes 4 CPU hours.''
|-
!scope="column" {{Lightgray_small}}| Planned maximum number of parallel used CPU cores per job
|scope="column" {{Tab element}} | Give an upper limit of how many CPU cores per job are required. Please cross-check with your statement about parallel paradigm.
''Obviously, a sequential code can only use 1 CPU core per job.''
|-
!scope="column" {{Lightgray_small}}| Estimated maximal memory requirements per CPU core
|scope="column" {{Tab element}} | Give an upper limit of how much RAM per CPU core your jobs require. Please give a value in Gigabytes.
|-
!scope="column" {{Lightgray_small}}| Estimated requested persistent disk space for the RV
|scope="column" {{Tab element}} | State how much disk space in total you need for your RV. Please give a value in Gigabytes.
''Example: If your RV has 4 more coworker and each of you produces until the end of your RV 20 Gigabyte of output, your maximum disk space is 100 Gigabyte.''
|-
!scope="column" {{Lightgray_small}}| Estimated maximal temporary disk space per job
|scope="column" {{Tab element}} | State how much space your intermediate files require during a job run. Please give a value in Gigabytes.
''Example: Your final output of your job is 0.1 Gigabyte, but during your job 10 temporary files, each with a size of 1 Gigabyte, are written to disk, hence your correct answer is 10 Gigabyte.''
|-
!scope="column" {{Lightgray_small}}| Institute
|scope="column" {{Tab element}} | State the full name of your Institute.
|-
|}

=== Registration Response ===

After submitting you will get an email from the ZAS confirming your submission.
With this email you are given an unique
* RV acronym
* RV password

Please keep the password enclosed.
With RV acronym and password, [[Registration/bwForCluster/RV#Become_Coworker_of_a_RV| coworkers ]] can already associate themselves to the RV.
If this information is lost, please [[Registration/bwForCluster/RV#Renew_RV_Password|renew the password]] for the RV.

The cluster assignment team assigns a bwForCluster to your RV based on your submitted data, including the [https://www.bwhpc.de/bwhpc-domains.php research area] as well as the software and hardware requirements.
The ZAS notifies you in an email about your assigned bwForCluster and provides website details for [[Registration/bwForCluster/Service| Step C]]. Later on you can find details about the roles and management possibilities at the [[Registration/bwForCluster/RV/Management | RV Management]] page.

=== Duration of an RV ===

An RV is valid for '''one year'''.
After that, the RV responsible must apply for an extension of the RV for it to be renewed for another year.

=== RV Management for RV Responsibles and Managers ===

RV responsible or manager can [[Registration/bwForCluster/RV/Management|manage and modify Rechenvorhaben/projects]].

----

<p style="text-align:right;">[[Registration/bwForCluster/Service | Go to step C]]</p>

FAQ

2026-06-22T13:31:20Z

K Siegmund: /* Access and Account Issues: bwForCluster */

__TOC__

Frequently Asked Questions

== Access and Account Issues: Generic Advice ==

* Double-check your [[Registration/Login/Username|username]] and [[Registration/Password|password]].
* Make sure your CAPS LOCK key is not enabled.
* Are you within the university network (Universities in Baden-Württemberg, BELWÜ)?



== Access and Account Issues: bwUniCluster ==

===Q: I Cannot Log In to my bwUniCluster and See 'Status: LOST_ACCESS' in the Registry Info</b>===

<b>A:</b> You need to fulfill the requirements outlined in
<b>→</b> [[Registration/bwUniCluster]]

Please check which of those two things are missing using the following documentation:

==== Entitlement ====

You need the [[bwUniCluster Entitlement|Registration/bwUniCluster/Entitlement]]

If the entitlement is missing: please check with the local support of your home university (the university where you study or are employed)

==== Questionnaire ====

<b>→</b> Check if you answered the [[BwUniCluster_2.0_User_Access#Step_C:_Fill_out_the_bwUniCluster_questionnaire|Questionnaire]].

== Access and Account Issues: bwForCluster ==

===Q: I Cannot Log In to my bwForCluster and See 'Status: LOST_ACCESS' in the Registry Info===

<b>A:</b> You need to fulfill the requirements outlined in

<b>→</b> [[Registration/bwForCluster]]

Please check which of those two things are missing using the following documentation:

==== Entitlement (and IDP Problems) ====

You need the bwForCluster Entitlement.

The same page can also help you recognize problems with the IDP (identity provider) of your home university

<b>→</b> Read [[Registration/bwForCluster/Entitlement]] and check if it is present.

In case
* the entitlement is missing
* Name/Email/eduPersonPrincipalName in the "personal data" tab are incorrect or missing
<b>→</b> please check with the local support of your home university (the university where you study or are employed)

Note: in case of temporary problems with your IDP, data in the registration server can become corrupted or missing. Logging in to the registration server that your cluster uses can fix this problem and fill in correct values again. Check in [[Registration/Password]] which registration server your cluster uses and log in there (for some cluster it is the same as given for the Entitlement Check).

==== ZAS Permission ====
Check if you are an active member of a compute activity and the "RV" hasn't expired.

<b>→</b> [https://zas.bwhpc.de/shib/info_rv.php ZAS membership page]

=== Forgot Password ===

Please set a new password for the service.

→ [[Registration/Password|Password Guide]]

=== Lost Phone with OTP Secret ===

Pleas use your <b>Backup TAN list</b> and add a new secret.

== Storage and File Systems ==

=== I am over the File System Quota ===

There are two resources with limits enforced by quota:
* disk space
* number of inodes (files).

It is important to identify the main sources that consume these resources.

* Size: List all directories (and files) in your home directory or workspace sorted by size
<code>
du -hs .[!.]* * | sort -h | tail
</code>

Descend in a directory with a large reported size and repeat until you find large files that are worth taking care of.

<code>.[!.]*</code> makes sure the "hidden" dot files are also checked.

* Inodes:

<code>
du -s --inodes .[!.]* * | sort -n
</code>

This will only count inodes in subdirectories. Also make sure your top-level (e.g. $HOME) directory aren't full with hundred thousands of files:

<code>
ls -1a | wc -l
</code>

FAQ

2026-06-22T13:25:00Z

K Siegmund: /* Access and Account Issues: bwForCluster */

__TOC__

Frequently Asked Questions

== Access and Account Issues: Generic Advice ==

* Double-check your [[Registration/Login/Username|username]] and [[Registration/Password|password]].
* Make sure your CAPS LOCK key is not enabled.
* Are you within the university network (Universities in Baden-Württemberg, BELWÜ)?



== Access and Account Issues: bwUniCluster ==

===Q: I Cannot Log In to my bwUniCluster and See 'Status: LOST_ACCESS' in the Registry Info</b>===

<b>A:</b> You need to fulfill the requirements outlined in
<b>→</b> [[Registration/bwUniCluster]]

Please check which of those two things are missing using the following documentation:

==== Entitlement ====

You need the [[bwUniCluster Entitlement|Registration/bwUniCluster/Entitlement]]

If the entitlement is missing: please check with the local support of your home university (the university where you study or are employed)

==== Questionnaire ====

<b>→</b> Check if you answered the [[BwUniCluster_2.0_User_Access#Step_C:_Fill_out_the_bwUniCluster_questionnaire|Questionnaire]].

== Access and Account Issues: bwForCluster ==

===Q: I Cannot Log In to my bwForCluster and See 'Status: LOST_ACCESS' in the Registry Info===

<b>A:</b> You need to fulfill the requirements outlined in

<b>→</b> [[Registration/bwForCluster]]

Please check which of those two things are missing using the following documentation:

==== Entitlement (and IDP Problems) ====

You need the bwForCluster Entitlement.

The same page can also help you recognize problems with the IDP (identity provider) of your univer

<b>→</b> Read [[Registration/bwForCluster/Entitlement]] and check if it is present.

In case
* the entitlement is missing
* Name/Email/eduPersonPrincipalName in the "personal data" tab are incorrect or missing
<b>→</b> please check with the local support of your home university (the university where you study or are employed)

==== ZAS Permission ====
Check if you are an active member of a compute activity and the "RV" hasn't expired.

<b>→</b> [https://zas.bwhpc.de/shib/info_rv.php ZAS membership page]

=== Forgot Password ===

Please set a new password for the service.

→ [[Registration/Password|Password Guide]]

=== Lost Phone with OTP Secret ===

Pleas use your <b>Backup TAN list</b> and add a new secret.

== Storage and File Systems ==

=== I am over the File System Quota ===

There are two resources with limits enforced by quota:
* disk space
* number of inodes (files).

It is important to identify the main sources that consume these resources.

* Size: List all directories (and files) in your home directory or workspace sorted by size
<code>
du -hs .[!.]* * | sort -h | tail
</code>

Descend in a directory with a large reported size and repeat until you find large files that are worth taking care of.

<code>.[!.]*</code> makes sure the "hidden" dot files are also checked.

* Inodes:

<code>
du -s --inodes .[!.]* * | sort -n
</code>

This will only count inodes in subdirectories. Also make sure your top-level (e.g. $HOME) directory aren't full with hundred thousands of files:

<code>
ls -1a | wc -l
</code>

Registration/bwForCluster/Entitlement

2026-06-22T13:22:01Z

K Siegmund: /* Check your Entitlements */

<div style="border: 3px solid #e2e3e5; padding: 15px; background-color: #ffffff; margin: 10px 0;">
The bwForCluster Entitlement is issued by the university you are a member of. It tells the cluster-operator that you are currently a member of this university. It also means that the university checked that your actions comply with the German Foreign Trade Act (Außenwirtschaftsgesetz - AWG) and German Foreign Trade Regulations (Außenwirtschaftsverordnung - AWV) (see [https://www.bwidm.de/attribute.php#Berechtigung eduPersonEntitlement]).
</div>

= Step A: bwForCluster Entitlement =

<div style="border: 3px solid #0066cc; padding: 15px; background-color: #e7f3ff; margin: 10px 0;">
To register for a bwForCluster you need the '''bwForCluster Entitlement''' issued by your university.
</div>

The full name of the entitlement is <code><nowiki>http://bwidm.de/entitlement/bwForCluster</nowiki></code>.

Note that this is the name of an identity management attribute and not a link.

More informations on Entitlements and other attributes in general: [https://www.bwidm.de/dienste.php Entitlement (Dienste)]

== Check your Entitlements ==

First check if you already have the Entitlement, some universities automatically assign the entitlement to some accounts (you must allow that your data is being transferred to that page or your cluster login will also not work):

→ '''[https://login.bwidm.de/user/index.xhtml?show=entitlement&highlight=bwForCluster Registration Server bwForCluster Entitlement Check]'''

Note: if you cannot log in to the registration server at all, some crucial information is missing in the information your home university is transmitting to it. Contact the helpdesk of your home university.

== Request an Entitlement ==

If you need the entitlement, please follow the link for your institution or contact your local service desk if no information is provided:

* [[Registration/bwIDM-Entitlements-Uni-Freiburg|Universität Freiburg]]
* [https://heiservices.uni-heidelberg.de/entitlement Universität Heidelberg] (access only within Uni Heidelberg network)
* [https://kim.uni-hohenheim.de/bwhpc-account/ Universität Hohenheim]
* [https://www.scc.kit.edu/downloads/ISM/SD-HPC-Formulare/Accessform_bwForCluster_v1_DE_EN_2026.pdf Karlsruhe Institute of Technology (KIT)]
* [https://www.kim.uni-konstanz.de/services/forschen-und-lehren/high-performance-computing/zugang-bwforcluster/ Universität Konstanz]
* [[BwForCluster User Access Members Uni Mannheim|Universität Mannheim]]
* [https://www.hlrs.de/apply-for-computing-time/bwforcluster Universität Stuttgart]
* [[BwForCluster User Access Members Uni Tübingen|Universität Tübingen]]
* [[BwForCluster User Access Members Uni Ulm|Universität Ulm]]
* [[Registration/HAW|HAW BW e.V.]]: Please contact your local service desk

<div style="border: 3px solid #ffc107; padding: 15px; background-color: #fff3cd; margin: 10px 0;">
'''Entitlement Synchronization:'''

After your university assigns the entitlement, it takes some time for it to synchronize across the system.
* '''If the entitlement does not appear within 24 hours,''' contact your local service desk
</div>

----

<p style="text-align:right;">[[Registration/bwForCluster/RV|Go to step B]]</p>

Justus3

2026-06-19T09:15:33Z

K Siegmund:

[[File:Justus3_header.png|700px]]

The '''bwForCluster Justus 3''' is the state-wide high-performance compute cluster dedicated to '''Computational Chemistry and Quantum Sciences''' in Baden-Württemberg.

{| style=" background:#FEF4AB; width:100%;"
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News
|-
|
* 2026-06-16: Cooling installations for the new cluster has been enabled
|}
{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support
|-
|
* [[Justus3/Getting Started|Getting Started]]
* Online Tutorial [https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC Introduction to Linux in HPC] (external link)
* [[Justus3/Support|Contact and Support]]
* Send [[Feedback|Feedback]] about Wiki pages
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | User Documentation
|-
|
* [[Registration/bwForCluster|Registration]]
* [[Justus3/Policy Agreement|Policy Agreement]]

* [[Justus3/Login|Login]]
* [[Justus3/Hardware|Hardware and Architecture]]
** [[Justus3/Hardware#Node_Specifications|Node Specifications]]
** [[Justus3/Hardware#Storage_Architecture|File Systems and Workspaces]]

* [[Justus3/Software|Software]]
* [[Justus3/Jobscripts: Running Your Calculations|Jobscripts: Running Your Calculations]]
** [[bwForCluster Justus 3 Slurm HOWTO|Slurm HOWTO]]
* [[Justus3/Visualization|Visualisation]]
* [[Development]]
|}

{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Cluster Funding
|-
|
* Please '''[[Justus3/Acknowledgement|acknowledge]]''' the Justus 3 cluster in your publications.
|}

Justus3

2026-06-19T09:06:28Z

K Siegmund: Created page with " 700px The '''bwForCluster Justus 3''' is the state-wide high-performance compute cluster dedicated to '''Computational Chemistry and Quantum Sciences''' in Baden-Württemberg.  {| style=" b..."

[[File:Justus3_header.png|700px]]

The '''bwForCluster Justus 3''' is the state-wide high-performance compute cluster dedicated to '''Computational Chemistry and Quantum Sciences''' in Baden-Württemberg.

{| style=" background:#FEF4AB; width:100%;"
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News
|-
|
* None
|}
{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support
|-
|
* [[Justus3/Getting Started|Getting Started]]
* Online Tutorial [https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC Introduction to Linux in HPC] (external link)
* [[Justus3/Support|Contact and Support]]
* Send [[Feedback|Feedback]] about Wiki pages
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | User Documentation
|-
|
* [[Registration/bwForCluster|Registration]]
* [[Justus3/Policy Agreement|Policy Agreement]]

* [[Justus3/Login|Login]]
* [[Justus3/Hardware|Hardware and Architecture]]
** [[Justus3/Hardware#Node_Specifications|Node Specifications]]
** [[Justus3/Hardware#Storage_Architecture|File Systems and Workspaces]]

* [[Justus3/Software|Software]]
* [[Justus3/Jobscripts: Running Your Calculations|Jobscripts: Running Your Calculations]]
** [[bwForCluster Justus 3 Slurm HOWTO|Slurm HOWTO]]
* [[Justus3/Visualization|Visualisation]]
* [[Development]]
|}

{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Cluster Funding
|-
|
* Please '''[[Justus3/Acknowledgement|acknowledge]]''' the Justus 3 cluster in your publications.
|}

Status

2026-06-19T08:30:23Z

K Siegmund:

= bwHPC Cluster and Service Status Page =

== Current Status ==



{{Template:Status|ok|Status: Ok}}



== Relevant Old Messages ==
* BinAC 2 maintenance: June 8–12

Main Page

2026-06-19T08:29:53Z

K Siegmund:

<span style="font-size:140%;>'''Welcome to the bwHPC Wiki.'''</span>

bwHPC represents services and resources in the State of '''B'''aden-'''W'''ürttemberg, Germany, for High Performance Computing ('''HPC'''), Data Intensive Computing ('''DIC''') and Large Scale Scientific Data Management ('''LS2DM''').

The main bwHPC web page is at '''[https://www.bwhpc.de/ https://www.bwhpc.de/]'''.

Many topics depend on the cluster system you use.
First choose the cluster you use, then select the correct topic.



{{Template:Status|ok|Status: Ok}}





{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Courses / eLearning
|-
|
* [https://training.bwhpc.de/ eLearning and Online Courses]
* [https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC Introduction to Linux in HPC (external resource)]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | [[File:Clusternodes.svg|x20px]] Need Access to a Cluster?
|-
|
* [[When to use an HPC Cluster]]
* [[Running Calculations]]
* [[Registration]]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | [[File:Clusternodes.svg|x20px]] HPC System Specific Documentation
|-
|
bwHPC encompasses several HPC compute clusters at different universities in Baden-Württemberg. Each cluster is dedicated to [https://www.bwhpc.de/bwhpc-domains.php specific research domains].

Documentation differs between compute clusters, please see cluster specific overview pages:
{|
|-
| style="padding:5px; width:30%" | [[BwUniCluster3.0|bwUniCluster 3.0]]
| style="padding-left:20px;" | General Purpose, Teaching
|-
| style="padding:5px; width:30%" | [[BinAC2|bwForCluster BinAC 2]]
| style="padding-left:20px;" | Bioinformatics, Astrophysics, Geosciences, Pharmacy, and Medical Informatics
|-
| style="padding:5px; width:30%" | [[Helix|bwForCluster Helix]]
| style="padding-left:20px;" | Structural and Systems Biology, Medical Science, Soft Matter, Computational Humanities, and Mathematics and Computer Science
|-
| style="padding:5px; width:30%" | [[:JUSTUS2| bwForCluster JUSTUS 2]]
| style="padding-left:20px;" | Theoretical Chemistry, Condensed Matter Physics, and Quantum Sciences
|-
| style="padding:5px; width:30%" | [[NEMO2|bwForCluster NEMO 2]]
| style="padding-left:20px;" | Neurosciences, Particle Physics, Materials Science, and Microsystems Engineering
|}
|-
|bwHPC Clusters: [https://www.bwhpc.de/cluster.php operational status]
Further Compute Clusters in Baden-Württemberg (separate access policies):
* [[DACHS | Datenanalyse Cluster der Hochschulen (DACHS)]]
* bwHPC tier 1: [https://kb.hlrs.de/platforms/index.php/Hunter_(HPE) Hunter] ([https://www.hlrs.de/apply-for-computing-time getting access])
* bwHPC tier 2: [https://www.nhr.kit.edu/userdocs/horeka HoreKa] ([https://www.nhr.kit.edu/userdocs/horeka/projects/ getting access])
|}

{| style="background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | [[File:Clusternodes.svg|x20px]] Documentation valid for all Clusters
|-
|
* [[Environment Modules| Software Environment Modules]]: usage of preexisting software
* [https://www.bwhpc.de/software.html List of Software] on all clusters
* [[Data Transfer|Data Transfer]]: working with files [[File:Notebook.svg|x15px]] <---> [[File:Clusternodes.svg|x15px]]
* [[Development| Development]]: software, programming languages, parallel pogramming
* [[Energy Efficient Cluster Usage | (Energy) Efficient Cluster Usage]] for low waiting times and fast running jobs
* [[HPC Glossary]]

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | [[File:Storage_small.svg|x15px]] Scientific Data Storage
|-
|
User guides of available scientific data storage services:
* [[SDS@hd]]: Everyone can join existing storage projects, entitlement needed for creating your own, [https://sds-hd.urz.uni-heidelberg.de/management/shib/sds_costs.php cost sheet]
* [https://uni-tuebingen.de/einrichtungen/zentrum-fuer-datenverarbeitung/projekte/laufende-projekte/bwsfs bwSFS]
Associated, but local scientific storage services are:
* [https://wiki.scc.kit.edu/lsdf/index.php/Category:LSDF_Online_Storage LSDF Online Storage] (only for KIT and KIT partners)
|}

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | [[File:Storage_small.svg|x15px]] Scientific Data Archiving
|-
|
For user guides of the scientific data archiving services:
* [https://www.rda.kit.edu/english bwDataArchive]: Open to all scientists at KIT and institutions in Baden-Württemberg that have signed a service agreement with KIT
Associated, but local archiving services for scientific data are:
* [https://www.urz.uni-heidelberg.de/de/service-katalog/speicher/heiarchive heiARCHIVE] (only for members of Heidelberg University)
For instructions on how to move the data, depending on the offered connections, can be found under [[Data Transfer | Data Transfer]].
|}

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | Research Data Management
|-
|
* [https://www.forschungsdaten.org/index.php/FDM-Kontakte#Deutschland Research Data Management (RDM)] contact persons
* [https://www.forschungsdaten.info Portal for Research Data Management] (Forschungsdaten.info)
|}
{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Support
|-
|
* [[BwSupportPortal|Submit a Ticket in our Support Portal]]
Support is provided by the [https://www.bwhpc.de/teams.php bwHPC Competence Centers]:
|}
{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Acknowledgement
|-
|
* Please [[Acknowledgement|acknowledge]] our resources in your publications.
|}

Data Transfer/SSHFS

2026-06-17T14:26:17Z

K Siegmund:

Sshfs is a network protocol for mounting data. It uses [[Data Transfer/SFTP|sftp]] under the hood.

== Installation ==

For the necessary installations follow this link:<br /> [https://linuxize.com/post/how-to-use-sshfs-to-mount-remote-directories-over-ssh/ How to use SSHFS to Mount Remote Directories over SSH].<br />
a) '''MacOS after the installation'''<br />
To enable sshfs on your Mac go to: System Preferences -> General tab -> Allow Extensions button<br />
Then follow the instructions that pop up (make sure that you remember the instructions before shutting down your computer).<br />
b) '''Linux installation short commands'''
<ul>
<li>Debian/Ubuntu: <code>apt-get install sshfs</code></li>
<li>CentOS/RedHat: <code>dnf install fuse-sshfs</code></li>
<li>If you want to use /etc/fstab, you can do this with the following entry:
<pre>sshfs#<username>@<hostname>: <your_local_mountpoint> fuse defaults,user,noauto,exec 0 0 </pre></li></ul>

== Usage ==

=== Caveat: Unmounting ===

Note that this creates a virtual drive on your system just as if you would plug in an external drive that stays active until you unmount it. Disk mounts happen on a low level of operating systems, so it is important to unmount the virtual drive cleanly whenever possible.

'''MacOS & Linux:'''<br />
If the connection to the cluster broke and you want to remount, you often have to use umount first. Try to remember to always unmount first before retrying. Otherwise it can happen that unmounting is not possible anymore by cmd. Usually the error message in this case tells you that the process is currently busy and therefore cannot umount.

You should be able to find the process(es) that use the file system with <code>fuser -m <your_local_mountpoint></code> and kill/exit the processes that block unmounting or can kill them directly using <code>fuser -k -m <your_local_mountpoint></code>
In case this fails, you may have to restart your computer which automatically unmounts everything.

<pre>bash
umount -f <localPath>
# For example
umount -f /Users/<username>/Documents/mounted
# or use
fusermount -u <localPath>
</pre>

'''Windows'''
You should be able to click on the drive and choose "eject" or "unmount". If this is not the case, you can use
net use X: /delete
or if this does not work
taskkill /f /im sshfs.exe
(from: [https://stackoverflow.com/questions/62031900/how-to-unmount-close-the-mount-for-sshfs-windows stackoverflow])

=== Windows ===
''Caution: sshfs on Windows doesn't work with 2FA. An alternative is [[Data_Transfer/Rclone | Rclone with sftp]] + [[Registration/SSH | ssh key]].''
<ol style="list-style-type: decimal;">
<li>Go to "This PC" and select "Connect network drive" ("Netzlaufwerk verbinden")</li>
<li>Select a drive (W:) and click on "connect with webpage on which you can save documents and images" (Verbindung mit website herstellen, auf der Sie Dokumente und Bilder speichern können)</li>
<li>Enter the following as address, to connect to the home or root folder:
<syntaxhighlight lang="bash">\sshfs\<username>@<hostname>/</syntaxhighlight>
If you would want to go two folders up, you would add “''../../''” at the end.</li></ol>

=== MacOS & Linux ===

<ol style="list-style-type: decimal;">
<li>Create an empty directory. The cluster/storage folder is going to be mounted to this directory.</li>
<li>Mount a folder from the cluster/storage:
<syntaxhighlight lang="bash"># Home folder
sshfs <username>@<hostname>: <localPath>
# subfolderX
sshfs <username>@<hostname>:subfolderX <localPath></syntaxhighlight></li></ol>

Energy Efficient Cluster Usage

2026-06-16T10:07:25Z

K Siegmund: /* How many and which kind of hardware resources do I require for it */

= Introduction =

Energy consumption of data centers has been increasing continuously throughout the last decade. In 2020, the energy consumption of all data centers in Germany amounted to around [https://www.bundestag.de/resource/blob/863850/423c11968fcb5c9995e9ef9090edf9e6/WD-8-070-21-pdf-data.pdf 3 percent] of the total electricity produced. Accompanying this large energy consumption are large-scale emissions of CO2 to the atmosphere and thus significant contributions to climate change.
To illustrate this, an average compute job running on a single node for one day may easily consume 10 kWh or even more. That translates roughly to brewing 700 cups of coffee.
Assuming that a typical bwHPC cluster has a few hundred compute nodes, this amounts to the energy consumption of a village for each cluster.

Although a large amount of this energy consumption is an intrinsic requirement of running large HPC clusters (even when it's processors are idle, a cluster uses a lot of energy), efficient use of the available resources is important. Using as many resources as possible does not make a power user. Using them wisely does.
In the following, a basic introduction to some of the most important aspects of energy-efficient HPC usage from a user perspective is given.

We can generally distinguish three tasks when optimizing for running HPC jobs efficiently.

→ What do I want to do and why do I need an HPC Cluster for it?

→ How many and which kind of hardware resources do I require for it?

→ How do I optimize my code to use these resources most efficiently?

= What do I want to do and why do I need an HPC Cluster for it? =

The bwHPC clusters are used to almost full capacity, and running a job on an HPC node consumes a lot of energy, as shown above.
Therefore, users are requested to run only necessary jobs.

Please consider testing new setups and their output for validity prior to submitting jobs that require lots of resources. This also includes projects where a lot of (smaller) similar jobs are submitted.

Make sure to double-check your jobs prior to the submission, as having to discard the output data of an HPC project due to faulty input files is wasting a lot of computational resources.

Finally, identifying the specific resource requirements for a given job is important to allocate the optimal amount for your compute job, and to decide if an HPC cluster is needed at all.

= How many and which kind of hardware resources do I require for it =

Resource allocation is a crucial part when working on an HPC cluster.
As this is dependent on both the job as well as the specific cluster hardware and architecture available.

'''A small number of jobs and few resources'''

→ Submit to the scheduler. No extended testing and resource scaling analysis are needed.

'''Medium-sized projects'''

→ Run only necessary jobs: Please consider testing new setups and their output for validity prior to submitting a huge amount of similar jobs

→ Start small: Run your problem on a small set of resources first.

→ Use the proper tools for development: If you develop your own code, please use the proper tools for debugging and parallel performance analysis. See: [[Development#Documentation_in_the_Wiki|Development]].

→ A look at the job feedback can help you determine if you are using the cluster efficiently

'''Large projects'''

→ Same approach as for medium-sized projects.

→ Run a scaling analysis for your project with regard to how many resources work best. See: [[Scaling]].

'''Many short jobs'''

→ Handling via the scheduler is inefficient.

→ Simple parallelization by hand is advisable. See: A basic introduction to [[Development/Parallel_Programming | Parallel Programming]].

= How do I optimize my code to use these resources most efficiently? =

The above recommendations will help use the cluster resources more efficiently.
Regarding software development, power efficiency correlates obviously heavily with '''computing performance''', but also with memory usage, i.e. the amount of memory used, but also memory efficiency.

Here, we have gathered a few results based on other research:
→ Use an efficient programming language such as Rust, C, and C++ -- well any compiled language. Do not use any interpreted language like Perl or Python. Since Machine Learning is a hot topic, this deserves a few words: Any ML-Python code using Tensorflow or other libraries will make heavy usage of NumPy and other math packages, which will use C-based implementations. Please make sure, you use the provided Python modules, which are optimized to use Intel MKL and other mathematical libraries.

Further reading:
Rui Pereira, et al: "''Energy efficiency across programming languages: how do energy, time, and memory relate?''", SLE 2017: Proc. of the 10th ACM SIGPLAN Int. Conf. on SW Language Eng., Oct. 2017, pp. 256–267, [https://doi.org/10.1145/3136014.3136031 doi:10.1145/3136014.3136031]

→ Analyse memory access patterns

→ For small tight loops checking for locks, use the <code>pause</code> instruction.

= Summary: General Recommendations =

* Choose the most '''efficient algorithms''' for the given problem.
* Run only '''necessary''' jobs: Please consider testing new setups and their output for validity prior to submitting a huge amount of similar jobs.
* Start '''small''': Run your problem on a small number of parallel entities (be it processes or threads) first.
* '''Estimate''' the runtime of the parallel job as '''exactly''' as possible to increase the efficiency of the scheduling of the whole system.
* Use the proper tools for development: If you develop your own code, please use the proper tools for debugging and parallel performance analysis. More information is available on the bwHPC Wiki.
* A look at the '''job feedback''' can help you determine if you are using the cluster efficiently.

Data Transfer/SSHFS

2026-06-03T08:45:06Z

K Siegmund: remove harmful otp-incompatible reconnect flags

JUSTUS2/Software/ADF

2026-05-19T09:24:50Z

K Siegmund:

{{Softwarepage|chem/adf}}

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| chem/adf
|-
| License
| Commercial - see [https://www.scm.com/adf-modeling-suite/pricing-licensing/scm-license-terms SCM License Terms]
|-
| Citing
| [https://www.scm.com/doc/ADF/Required_citations.html Required citations] as well as [https://www.scm.com/adf-modeling-suite module specific citations]
|-
| Links
| [https://www.scm.com/ ADF Homepage] and [https://www.scm.com/product/ product specific documentation pages]
|-
| Graphical Interface
| [[#Creating input files with adfinput|ADF GUI]]
|-
| User forum
| [https://www.scm.com/adf-discussion-list/ ADF Discussion List]
|}

= Description =

The [https://www.scm.com/adf-modeling-suite/ ADF Modeling Suite] by [https://www.scm.com/ SCM] (Software for Chemistry & Materials) is a collection of programs for modeling chemical and physical properties of molecules and solids. The suite consists of the following components:

* With help of the graphical user interface '''adfinput''' one can construct molecules, solids and slabs. All kinds of calculational parameters like functionals, basis sets, convergence parameters can be defined as well. The calculation and its parameters can be saved as a ''run''-file which is suitable for inclusion in our queueing system scripts. For details see the [https://www.scm.com/doc/Tutorials/Introduction/Introduction.html Tutorials] of SCM.

* '''adfjobs''' can used for interactive job control. This is intended only for short test jobs running in background for some minutes. For longer jobs one should save the job as ''run''-file which can be called from within a queueing system script. For details see the [https://www.scm.com/doc/Tutorials/Introduction/Introduction.html Tutorials] of SCM.

* [https://www.scm.com/product/adf/ '''adf'''] is a powerful molecular DFT (density functional theory) code applicable in many areas of chemistry and materials science. One strength of ADF is in the field of inorganic chemistry.

* [https://www.scm.com/product/band_periodicdft/ '''band'''] is a DFT code for periodic systems using local slater type orbital basis sets.

* [https://www.scm.com/product/dftb/ '''dftb'''] provides fast approximate methods for studying large molecules and large periodic systems.

* [https://www.scm.com/product/reaxff/ '''reaxff'''] is a bond order based force field program for studying reaction dynamics in large complex systems.

* The [https://www.scm.com/product/cosmo-rs/ '''COSMO-RS'''] method allows to predict thermodynamic properties of solutions and mixtures (LogP, VLE, pKa, ...) based on quantum mechanical data from ADF.

= License and citing =

The ADF program suite is available for all users of the cluster. When you publish any results obtained with the ADF program suite you must cite ADF and the used components properly. Therefore please see the [https://www.scm.com/doc/ADF/Required_citations.html required citations]. Furthermore there might be additional [https://www.scm.com/adf-modeling-suite component specific citations] that can be found in documentation of the individual components.

= Usage =

== Creating input files with adfinput ==

Probably the easiest method to create input files is using the graphical user interface '''adfinput''': First start and connect to a [[Start_VNC_Server_-_bwForCluster_Chemistry_-_3D_Acceleration|3D-accelerated VNC session]] via TigerVNC running on one of our visualization nodes.

See: [[JUSTUS2/Visualization]]

Then - within the VNC window - load the adf module and run adfinput with 3D acceleration:
<pre>
module load chem/adf
vglrun adfinput
</pre>

With help of ADFinput molecules and periodic systems can be constructed (depending on the selected target program, i.e. '''adf''' or '''band'''). Furthermore one can control most calculational parameters like DFT methods, basis sets, convergence criteria, etc. After saving the calculation via '''File -> Save as''' with name '''MY_JOB''', a '''run'''-shell script with name '''MY_JOB.run''' can be found on disk.

This '''run'''-shell script contains the job input and executes the appropriate ADF commands like '''adf''', '''band''', etc (the input is piped via STDIN into those commands). The '''run'''-script can be called from within your ADF queueing system shell script, thus the queueing system shell script still defines the queueing system parameters, manages the temporary directories, loads the module, cleans up after the end of the job, etc).

For an introduction how to use the ADF GUI please follow some of the [https://www.scm.com/doc/Tutorials/Introduction/Introduction.html ADF tutorials]. If you want to change the job input without GUI, you can edit the '''run'''-shell scripts manually. All possible keywords and options are documented in the '''manuals''' sections of the [https://www.scm.com/doc/Documentation/index.html documentation of the individual programs].

== Parallel computing ==

In queueing system scripts the commands '''adf''', '''band''', ... automatically use the cores as specified when submitting the job script. Neither '''mpirun''' nor any further command line options are required.

In case of interactive single-node test jobs one can limit the number of cores via command line option '''-n NB_OF_CORES''', for example '''adf -n 8'''.

== Disk usage ==

Temporary files are stored below directory '''SCM_TMPDIR'''. Typically this environment variable is defined within the queueing system script and points to some node-local directory below '''TMPDIR'''. In case of multi-node jobs, '''SCM_TMPDIR''' must point to the same directory on every node and must exist on every node.

When loading the adf module, '''SCM_TMPDIR''' is defined automatically and usually points to '''TMPDIR'''. When running under control of the queueing system, '''TMPDIR''' is the node-local scratch directory of the job.

Please never put temporary files of ADF on central file systems like '''HOME''' or '''SCRATCH''' or workspaces.

----