bwHPC Wiki - User contributions [en]

Registration/bwForCluster/RV/Management

2026-10-07T15:16:34Z

K Siegmund:

== Roles in a RV ==
There are three different roles in an RV. Depending on your role, different management activites can be executed in the ZAS under [https://zas.bwhpc.de/shib/en/info_rv.php '''My RVs''']. These are described below.

=== RV Coworker ===

A coworker is a typical user who registered themselves via RV acronym and RV password. They...
* ...can check the status of their memberships (active/inactive).
* ...can check the status of the RV (active/inactive & the date when it becomes inactive when it doesn't get prolonged).
* ...can deregister from an RV.
* ...have to acknowledge the cluster in publications and inform the RV responsible of these publications.

=== RV Manager ===

Additionally to the abilities of a coworker, a manager can...

* ...re-activate deregistered or inactive members.
* ...check the list of RV members at [https://zas.bwhpc.de/shib/en/info_rv.php '''My RVs'''] by selecting the RV acronym. After a role change, click '''Save changes''' at the end of the members list.

=== RV Responsible ===

Additionally to the abilities of a manager, the responsible can...

* ...assign managers.
* ...extend the runtime of the RV by one more year. Therefore, a list of publications that were created with the help of the cluster must be provided when submitting the prolongation form.
* ...change the RV responsible.
* ...set a new password if necessary and share the password with future coworkers so that they can join the RV.

==== Renew RV Password ====

If you lose the RV password or want to create a new one, you can simply renew it:

1. Login to [https://zas.bwhpc.de/shib/en/info_rv.php '''My RVs'''] first.
[[File:RV-my.png|center|frame|My RVs.]]

2. Select the RV acronym for your RV.

3. Scroll down and click '''Renew password'''.
[[File:RV-pw.png|center|thumb|400px|Renew RV password.]]

You will get a form that looks similar to the form you got when registering the compute project. Most fields will be pre-filled with the things you entered when registering. Please check if the entered data is still accurate for your compute activity and adjust if necessary.

A new field in the form asks you to fill in the publications that acknowledge the cluster. Please enter the DOIs of those publications. These publications should all have the cluster you used in their Acknowledgement in a format as shown in the [[Acknowledgement]] section for each cluster.

==== Change the RV Responsible ====

<ins>Prerequisites:</ins>

* You need to be the current RV responsible. In case you don't have access to your account anymore, you can ask the cluster support for help.
* The person you want to appoint as new RV responsible is a member of the RV.

<ins>Steps:</ins>

<gallery mode="packed" style="align=left;">
File:RV-my.png|center|frame|'''1.''' Login to [https://zas.bwhpc.de/shib/en/info_rv.php '''My RVs'''] and go to the RV.
File:Rv-hand1.png|right|thumb|400px|'''2.''' Scroll down and click '''Handoff'''.
File:Rv-hand2.png|right|thumb|800px|'''3.''' Select "New RV responsible" from the drop-down menu, check "Agreement" and click "Preview: Handover RV" to continue.
File:Rv-hand3.png|right|thumb|800px|'''4.''' Confirm the handover.
</gallery>
'''5.''' The newly appointed RV responsible gets notified via email. They have to accept within seven days.

==== RV Extension ====
An RV is valid for one year. After that, the RV responsible must apply for an extension of the RV for it to be renewed for another year.

<ins>Hints for specific cases:</ins>
* I don't think that filling in the publication field is very important.
*: → The bwHPC clusters can be used for free but it's very important for us to acquire funding for the next generation hardware, support services, data storage and infrastructure. To get funding, it's essential to provide a list of research publications that benefitted from the clusters. Furthermore, you've personally agreed to report publications in the process of registering for the cluster and you agreed to acknowledge them accordingly. Each cluster provides an example text for the acknowledgement.
* I'm late with the extension application and it would take some time to find out cluster related publications.
*: → Write something like "I will provide them later" and send the list to your cluster's support unit as soon as you have them collected.
* We had a lot of publications but I'm not sure which ones made use of the cluster.
*: → Please don't hand in the whole list but ask the RV members directly.
* We didn't have any cluster related publications.
*: → We would appreciate a short note about the reason. Some examples: "We didn't need the cluster much", "The publication is still work in progress", "We'd appreciate some help for getting started with the cluster usage".
* We have only preprints so far.
*: → Please report them, when when they are fully published. You can mention, that a certain number of publications is in preparation.
* The purpose and resource needs of the RV have changed.
*: → Please update the resources via the RV extension form.
* We forgot to acknowledge the cluster in most of the publications.
*: → That's unfortunate but please report them nevertheless. You can save us the hassle of checking in detail when you provide us with an additional confirmation that the cluster was indeed used. For example "Some acknowledgements are missing but I confirm that the cluster usage was relevant for the publications nevertheless". Please make sure the cluster is properly cited in the next publications.
* I fear that the RV gets deactivated before the extension application gets accepted.
*: → Depending on the cluster the access will still be possible for a while even if the RV is over its due date. But no new members can join while the extension is pending.
* We didn't use the RV for a long time but now that I want to hand in the RV extension, I don't see the button anymore.
*: → Please write to the support. We can enable the button again for 14 days.
* Looking back I must say that we did some great research with the help of the cluster.
*: We're happy to hear that and would love to show it off at our bwHPC website as [https://www.bwhpc.de/success_stories.php success story]. Don't hesitate to [https://www.bwhpc.de/project_report.php submit] a short summary.

Help talk:Editor's Guide

2026-10-07T14:56:38Z

K Siegmund: Created page with " Vorschlag: ---- Lifecycle Management & Verification Articles should not stay unverified for longer than half a year. The wiki editorial team will coordinate the effort of reviewing pages. Pages kept to retain historic information must have a tag <code><nowiki>{{History}}</nowiki></code> ---- Diskussion nächste Wiki-Vk Lifecycle Management und überprüfen von Seiten ~~~~"

Vorschlag:
----
Lifecycle Management & Verification
Articles should not stay unverified for longer than half a year. The wiki editorial team will coordinate the effort of reviewing pages. Pages kept to retain historic information must have a tag <code><nowiki>{{History}}</nowiki></code>
----
Diskussion nächste Wiki-Vk Lifecycle Management und überprüfen von Seiten
[[User:K Siegmund|K Siegmund]] ([[User talk:K Siegmund|talk]]) 16:56, 7 October 2026 (CEST)

Help:Editor's Guide

2026-10-07T14:55:38Z

K Siegmund: to talk page

Welcome to the '''wiki.bwhpc.de''' contributor guide. This document outlines structural rules, content principles, and syntax conventions to keep our HPC documentation '''accurate, maintainable, user-focused, and machine-readable''' across all bwHPC sites.

== Editorial Meetings ==

Don't be an anonymous editor, join the meeting of bwHPC wiki editors. More information in our project gitlab page at https://gitlab.kit.edu/groups/kit/bwhpc-s5/-/wikis/Home/AP1.1/Redaktionsteam_bwHPC-Wiki

== General Principles ==

=== Keep in Line General Wikipedia Rules as Applicable ===

Many of Wikepedia's guidelines are very helpful for writing here as well, while some are specific to creating an encyclopedia.

See e.g. https://en.wikipedia.org/wiki/Wikipedia:Writing_better_articles

=== Think from the User's Perspective ===

; Goal-Oriented Writing
: Always ask: ''"What is the user trying to accomplish?"''
: → Provide actionable commands, job submission scripts, and practical examples.

; Avoid Technical Jargon Overload
: Assume the user is an expert in their scientific domain (e.g., chemistry, physics), but potentially a beginner in HPC and Linux environments.
: → prefer common words over jargon, and if unavoidable, link jargon words to the [[HPC_Glossary]]

=== No Empty Placeholders ===

Do not publish empty pages or sections containing only headers with text like "TBD", "Work in Progress", or "Under Construction". If a page has no actionable information yet, keep it in a local draft or sandbox space until it provides immediate value.

But: Stubs are Welcome: Short pages or sections (stubs) that provide meaningful information are fine - our goal is to provide information not to create long texts.

=== Text Longevity ===

Avoid hardcoding very short-lived data that is bound to be outdated soon.

When documenting forseeably quickly aging things like a specific software version, explicitly state the version in your documentation so that outdated information does not become incorrect information when seen in the context of more recent versions.

=== Central vs. Local Scoping ===

<pre>
┌─────────────────────────────────────────────────────────────┐
│ Generic / Cross-Cluster Content │
│ (Slurm fundamentals, general module usage, scaling) │
└──────────────────────────────┬──────────────────────────────┘
│
┌──────────────────┴──────────────────┐
▼ ▼
┌───────────────────────┐ ┌───────────────────────┐
│ Cluster-Specific Page │ │ Cluster-Specific Page │
│ (e.g., /Justus2/...) │ │ (e.g., /Helix/...) │
└───────────────────────┘ └───────────────────────┘
</pre>

# Generic Content (Main Namespace):
#* Applies to all or most bwHPC clusters (e.g., workflow concepts, general environment module syntax).
#* Keep it independent of site-specific configurations.
# Cluster-Specific Content (Subfolders / Subpages):
#* Applies only to a specific cluster environment (e.g., local storage paths, specific hardware queues/partitions, cluster hostnames).
#* Rule: Always place cluster-bound content under its respective subfolder:
#** <code>Cluster_Name/Topic</code> (e.g., <code>Justus3/Storage</code>, <code>BinAC2/Partitions</code>).

== Syntax & Formatting ==

* Use the simpler Wiki Syntax whenever possible instead of reverting to HTML

* Keep Output Relevant: Do not paste raw console dumps or directory listings, but only document the command to list them on-cluster (examples are of course ok)
* Variable Syntax: Use standard angle brackets <code><placeholder></code> for user-defined parameters to clearly distinguish them from literal strings:
** <code>squeue -u <username></code>

* Use emphasis sparingly and on point. Writing something in bold is very often not as useful as you may think

* Internal vs External Links: Whenever something is in the wiki, an internal link, e.g. <code>[[Registration]]</code> should be used. The single brackets are for external links only.

Help:Editor's Guide

2026-10-06T14:39:52Z

K Siegmund:

Help:Editor's Guide

2026-10-06T14:35:38Z

K Siegmund: /* General Principles */

Welcome to the '''wiki.bwhpc.de''' contributor guide. This document outlines structural rules, content principles, and syntax conventions to keep our HPC documentation '''accurate, maintainable, user-focused, and machine-readable''' across all bwHPC sites.

== Editorial Meetings ==

Don't be an anonymous editor, join the meeting of bwHPC wiki editors. More information in our project gitlab page at https://gitlab.kit.edu/groups/kit/bwhpc-s5/-/wikis/Home/AP1.1/Redaktionsteam_bwHPC-Wiki

== General Principles ==

=== Think from the User's Perspective ===

=== Think from the User's Perspective ===

; Goal-Oriented Writing
: Always ask: ''"What is the user trying to accomplish?"''
: → Provide actionable commands, job submission scripts, and practical examples.

; Avoid Technical Jargon Overload
: Assume the user is an expert in their scientific domain (e.g., chemistry, physics), but potentially a beginner in HPC and Linux environments.
: → prefer common words over jargon, and if unavoidable, link jargon words to the [[HPC_Glossary]]

=== No Empty Placeholders ===

Do not publish empty pages or sections containing only headers with text like "TBD", "Work in Progress", or "Under Construction". If a page has no actionable information yet, keep it in a local draft or sandbox space until it provides immediate value.

But: Stubs are Welcome: Short pages or sections (stubs) that provide meaningful information are fine - our goal is to provide information not to create long texts.

=== Text Longevity ===

Avoid hardcoding very short-lived data that is bound to be outdated soon.

When documenting forseeably quickly aging things like a specific software version, explicitly state the version in your documentation so that outdated information does not become incorrect information when seen in the context of more recent versions.

=== Central vs. Local Scoping ===

<pre>
┌─────────────────────────────────────────────────────────────┐
│ Generic / Cross-Cluster Content │
│ (Slurm fundamentals, general module usage, scaling) │
└──────────────────────────────┬──────────────────────────────┘
│
┌──────────────────┴──────────────────┐
▼ ▼
┌───────────────────────┐ ┌───────────────────────┐
│ Cluster-Specific Page │ │ Cluster-Specific Page │
│ (e.g., /Justus2/...) │ │ (e.g., /Helix/...) │
└───────────────────────┘ └───────────────────────┘
</pre>

# Generic Content (Main Namespace):
#* Applies to all or most bwHPC clusters (e.g., workflow concepts, general environment module syntax).
#* Keep it independent of site-specific configurations.
# Cluster-Specific Content (Subfolders / Subpages):
#* Applies only to a specific cluster environment (e.g., local storage paths, specific hardware queues/partitions, cluster hostnames).
#* Rule: Always place cluster-bound content under its respective subfolder:
#** <code>Cluster_Name/Topic</code> (e.g., <code>Justus3/Storage</code>, <code>BinAC2/Partitions</code>).

== Syntax & Formatting ==

* Use the simpler Wiki Syntax whenever possible instead of reverting to HTML

* Keep Output Relevant: Do not paste raw console dumps or directory listings, but only document the command to list them on-cluster (examples are of course ok)
* Variable Syntax: Use standard angle brackets <code><placeholder></code> for user-defined parameters to clearly distinguish them from literal strings:
** <code>squeue -u <username></code>

* Use emphasis sparingly and on point. Writing something in bold is very often not as useful as you may think

* Internal vs External Links: Whenever something is in the wiki, an internal link, e.g. <code>[[Registration]]</code> should be used. The single brackets are for external links only.

== Lifecycle Management & Verification ==

Articles should not stay unverified for longer than half a year. The wiki editorial team will coordinate the effort of reviewing pages. Pages kept to retain historic information must have a tag <code><nowiki>{{History}}</nowiki></code>

Help:Editor's Guide

2026-10-06T14:31:46Z

K Siegmund: Created page with "Welcome to the '''wiki.bwhpc.de''' contributor guide. This document outlines structural rules, content principles, and syntax conventions to keep our HPC documentation '''accurate, maintainable, user-focused, and machine-readable''' across all bwHPC sites. == Editorial Meetings == Don't be an anonymous editor, join the meeting of bwHPC wiki editors. More information in our project gitlab page at https://gitlab.kit.edu/groups/kit/bwhpc-s5/-/wikis/Home/AP1.1/Redaktionste..."

Welcome to the '''wiki.bwhpc.de''' contributor guide. This document outlines structural rules, content principles, and syntax conventions to keep our HPC documentation '''accurate, maintainable, user-focused, and machine-readable''' across all bwHPC sites.

== Editorial Meetings ==

Don't be an anonymous editor, join the meeting of bwHPC wiki editors. More information in our project gitlab page at https://gitlab.kit.edu/groups/kit/bwhpc-s5/-/wikis/Home/AP1.1/Redaktionsteam_bwHPC-Wiki

== General Principles ==

=== Think from the User's Perspective ===

; Goal-Oriented Writing: Always ask:
: →''"What is the user trying to accomplish?"'' Provide actionable commands, job submission scripts, and practical examples.

; Avoid Technical Jargon Overload:
: → Assume the user is an expert in their scientific domain (e.g., chemistry, physics), but potentially a beginner in HPC and Linux environments.

=== No Empty Placeholders ===

Do not publish empty pages or sections containing only headers with text like "TBD", "Work in Progress", or "Under Construction". If a page has no actionable information yet, keep it in a local draft or sandbox space until it provides immediate value.

But: Stubs are Welcome: Short pages or sections (stubs) that provide meaningful information are fine - our goal is to provide information not to create long texts.

=== Text Longevity ===

Avoid hardcoding very short-lived data that is bound to be outdated soon.

When documenting forseeably quickly aging things like a specific software version, explicitly state the version in your documentation so that outdated information does not become incorrect information when seen in the context of more recent versions.

=== Central vs. Local Scoping ===

<pre>
┌─────────────────────────────────────────────────────────────┐
│ Generic / Cross-Cluster Content │
│ (Slurm fundamentals, general module usage, scaling) │
└──────────────────────────────┬──────────────────────────────┘
│
┌──────────────────┴──────────────────┐
▼ ▼
┌───────────────────────┐ ┌───────────────────────┐
│ Cluster-Specific Page │ │ Cluster-Specific Page │
│ (e.g., /Justus2/...) │ │ (e.g., /Helix/...) │
└───────────────────────┘ └───────────────────────┘
</pre>

# Generic Content (Main Namespace):
#* Applies to all or most bwHPC clusters (e.g., workflow concepts, general environment module syntax).
#* Keep it independent of site-specific configurations.
# Cluster-Specific Content (Subfolders / Subpages):
#* Applies only to a specific cluster environment (e.g., local storage paths, specific hardware queues/partitions, cluster hostnames).
#* Rule: Always place cluster-bound content under its respective subfolder:
#** <code>Cluster_Name/Topic</code> (e.g., <code>Justus3/Storage</code>, <code>BinAC2/Partitions</code>).

== Syntax & Formatting ==

* Use the simpler Wiki Syntax whenever possible instead of reverting to HTML

* Keep Output Relevant: Do not paste raw console dumps or directory listings, but only document the command to list them on-cluster (examples are of course ok)
* Variable Syntax: Use standard angle brackets <code><placeholder></code> for user-defined parameters to clearly distinguish them from literal strings:
** <code>squeue -u <username></code>

* Use emphasis sparingly and on point. Writing something in bold is very often not as useful as you may think

* Internal vs External Links: Whenever something is in the wiki, an internal link, e.g. <code>[[Registration]]</code> should be used. The single brackets are for external links only.

== Lifecycle Management & Verification ==

Articles should not stay unverified for longer than half a year. The wiki editorial team will coordinate the effort of reviewing pages. Pages kept to retain historic information must have a tag <code><nowiki>{{History}}</nowiki></code>

JUSTUS2/Software/NAMD

2026-09-22T09:32:15Z

K Siegmund: /* Example Scripts = */

{{Softwarepage|chem/namd}}

{| width=760px class="wikitable"
|-
! Description !! Content
|-
| module load
| chem/namd
|-
| Availability
| [[BwForCluster_JUSTUS_2]] |
|-
| License
| Freeware for non-commercial use. [[#License|More...]]
|-
| Citing
| [http://www.ks.uiuc.edu/Research/namd/papers.html Publications]
|-
| Links
| [http://www.ks.uiuc.edu/Research/namd/ Homepage] | [http://www.ks.uiuc.edu/Research/namd/2.14/ug/ Documentation]
|-
| Graphical Interface
| No
|}

= Description =

'''NAMD''' (NAnoscale Molecular Dynamics) is a parallel molecular dynamics code designed for high-performance simulation of large biomolecular systems.

= Availability =

NAMD is available on selected bwHPC-Clusters. A complete list of versions currently installed on the bwHPC-Clusters can be obtained from https://www.bwhpc.de/software.html
In order to check which versions of NAMD are installed on the compute cluster, run the following command:
<pre>
$ module avail chem/namd
</pre>

= License =

NAMD is available as [https://en.wikipedia.org/wiki/Freeware freeware] for non-commercial use by individuals, academic institutions, and corporations for in-house business uses. Please read the [http://www.ks.uiuc.edu/Research/namd/license.html license] for additional information about NAMD.

= Usage =

== Loading the module ==

You can load the default version of NAMD with the following command:
<pre>
$ module load chem/namd
</pre>

The module will try to load all modules (i.e., compiler, MPI, MKL) it needs to function. If loading the module fails, check if you have already loaded one of those modules, but not in the version required by NAMD.

If you wish to load another (older) version of NAMD, you can do so using
<pre>
$ module load chem/namd/<version>
</pre>
with <version> specifying the desired version.

Please cite NAMD in your publications according to the [http://www.ks.uiuc.edu/Research/namd/papers.html references].

== Program Binaries ==

The binary ''namd2'' is main program of the NAMD package.
<pre>
Usage: namd2 [options] [config file]

Example: namd2 +isomalloc_sync run.namd
</pre>

For information about how to construct NAMD configuration files, please see the [http://www.ks.uiuc.edu/Research/namd/2.14/ug/ documentation]. In addition to a configuration file, NAMD also needs a CHARMM force field in either CHARMM or X-PLOR format, an X-PLOR format PSF file describing the molecular structure, and the initial coordinates of the molecular system in the form of a PDB file.

The NAMD package includes several additional tools:

* flipbinpdb -- flips byte-ordering of 8-byte doubles
* flipdcd -- flips byte-ordering of DCD files
* psfgen -- [https://www.ks.uiuc.edu/Research/vmd/ VMD] psfgen plugin
* sortreplicas -- un-shuffles replica trajectories to place same-temperature frames in the same file

=== Charm++ ===

Charm++ is a parallel object-oriented programming paradigm based on C++ and developed in the Parallel Programming Laboratory at the University of Illinois at Urbana–Champaign. NAMD has been implemented using Charm++.

* charmrun -- launches Charm++ programs

For help on usage, type
<pre>
$ charmrun -help
</pre>

=== GPU Acceleration ===

Currently not available.

== Hints for using NAMD ==

=== Parallel Execution ===

It is usually more efficient to run several smaller jobs with reasonable speedup side by side. Check for a reasonable speedup by increasing the number of workers from 1, 2, 4, n cores. Do not forget to adjust the memory specification when changing the number of workers

=== Memory Management ===

It requires some experience to choose the perfect memory value. Monitoring a job interactively may help to estimate the memory consumption. Requesting 1GB more than the required amount is typically sufficient.

=== Runtime Specification ===

The wall-time limit is independent of the number of parallel workers. Selecting the best value requires some experience. In general, jobs with shorter wall times get higher priorities. It could help to run the job interactively for a small period of time to monitor its convergence. Furthermore, it may be advantageous to extrapolate the run time by looking at smaller test jobs.

= Examples =

As with all processes that require more than a few minutes to run, non-trivial compute jobs must be submitted to the cluster queuing system.

Example scripts are available in the directory <span style="background:#edeae2;margin:2px;padding:1px;border:1px dotted #808080">$NAMD_EXA_DIR</span>:
<pre>
$ module show chem/namd # show environment variables, which will be available after 'module load'
$ module load chem/namd # load module
$ ls $NAMD_EXA_DIR # show content of directory $NAMD_EXA_DIR
$ cat $NAMD_EXA_DIR/README # show examples README
</pre>

Run a first simple example job on <font color=red>JUSTUS 2</font>:
<pre>
$ module load chem/namd # load module
$ WORKSPACE=`ws_allocate namd 3` # allocate workspace
$ cd $WORKSPACE # change to workspace
$ cp -a $NAMD_HOME/bwhpc-examples . # copy example files to workspace
$ cd bwhpc-examples/apoa1 # change to test directory
$ sbatch bwhpc_namd_apoa1.sh # submit job
$ squeue # obtain JOBID
$ scontrol show job <JOBID> # check state of job
$ ls # when job finishes the results will be visible in this directory
</pre>

= Example Scripts =

<syntaxhighlight lang="bash">
#!/bin/bash
# Allocate two nodes
#SBATCH --nodes=2
# Number of program instances to be executed
#SBATCH --tasks-per-node=48
# Allocate 20 (of 192) GB memory per node
#SBATCH --mem=20gb
# Maximum run time of job
#SBATCH --time=00:10:00
# Give job a reasonable name
#SBATCH --job-name=bwhpc-example
# File name for stdout/stderr (%j will be replaced by job id)
#SBATCH --output=bwhpc-example_ID-%j.out

#########################
#
# bwHPC examples illustrate how to submit a batch job to the cluster queuing system. It is necessary to adapt certain specifications for your particular case.
#
# General Hints
#
# Parallel Execution:
# It is usually more efficient to run several smaller jobs with reasonable speedup simultaneously. Check for a reasonable speedup by gradually increasing the number of workers from 1 or 2 to n cores. Do not forget to adjust the memory specification when changing the number of workers. Use $ seff <jobid> to get efficiency information.
#
# Memory Management:
# It requires some experience to choose the perfect memory value. Monitoring a job interactively may help to estimate the memory consumption. Requesting 1GB more than the required amount is typically sufficient. Use $ seff <jobid> to get efficiency information.
#
# Runtime Specification:
# The wall-time limit is independent of the number of parallel workers. In general, jobs with shorter wall times get higher priorities. It could help to run the job interactively for a small period of time to monitor its convergence. Furthermore, it may be advantageous to extrapolate the run time by looking at smaller test jobs first.
#
#########################

module purge
module load chem/namd/2.14
module list

srun namd2 +isomalloc_sync apoa1.namd

exit 0
</syntaxhighlight>

= Useful links =

* [http://www.ks.uiuc.edu/Research/namd/2.14/ug/ Documentation (english)]
* [http://www.ks.uiuc.edu/Research/namd/mailing_list/ Maling list (english)]
* [https://en.wikipedia.org/wiki/NAMD Wikipedia article (english)]
* [https://de.wikipedia.org/wiki/NAMD Wikipedia article (german)]

JUSTUS2/Software/NAMD

2026-09-22T09:31:21Z

K Siegmund: Auto-sync: Example Scripts

{{Softwarepage|chem/namd}}

{| width=760px class="wikitable"
|-
! Description !! Content
|-
| module load
| chem/namd
|-
| Availability
| [[BwForCluster_JUSTUS_2]] |
|-
| License
| Freeware for non-commercial use. [[#License|More...]]
|-
| Citing
| [http://www.ks.uiuc.edu/Research/namd/papers.html Publications]
|-
| Links
| [http://www.ks.uiuc.edu/Research/namd/ Homepage] | [http://www.ks.uiuc.edu/Research/namd/2.14/ug/ Documentation]
|-
| Graphical Interface
| No
|}

= Description =

'''NAMD''' (NAnoscale Molecular Dynamics) is a parallel molecular dynamics code designed for high-performance simulation of large biomolecular systems.

= Availability =

NAMD is available on selected bwHPC-Clusters. A complete list of versions currently installed on the bwHPC-Clusters can be obtained from https://www.bwhpc.de/software.html
In order to check which versions of NAMD are installed on the compute cluster, run the following command:
<pre>
$ module avail chem/namd
</pre>

= License =

NAMD is available as [https://en.wikipedia.org/wiki/Freeware freeware] for non-commercial use by individuals, academic institutions, and corporations for in-house business uses. Please read the [http://www.ks.uiuc.edu/Research/namd/license.html license] for additional information about NAMD.

= Usage =

== Loading the module ==

You can load the default version of NAMD with the following command:
<pre>
$ module load chem/namd
</pre>

The module will try to load all modules (i.e., compiler, MPI, MKL) it needs to function. If loading the module fails, check if you have already loaded one of those modules, but not in the version required by NAMD.

If you wish to load another (older) version of NAMD, you can do so using
<pre>
$ module load chem/namd/<version>
</pre>
with <version> specifying the desired version.

Please cite NAMD in your publications according to the [http://www.ks.uiuc.edu/Research/namd/papers.html references].

== Program Binaries ==

The binary ''namd2'' is main program of the NAMD package.
<pre>
Usage: namd2 [options] [config file]

Example: namd2 +isomalloc_sync run.namd
</pre>

For information about how to construct NAMD configuration files, please see the [http://www.ks.uiuc.edu/Research/namd/2.14/ug/ documentation]. In addition to a configuration file, NAMD also needs a CHARMM force field in either CHARMM or X-PLOR format, an X-PLOR format PSF file describing the molecular structure, and the initial coordinates of the molecular system in the form of a PDB file.

The NAMD package includes several additional tools:

* flipbinpdb -- flips byte-ordering of 8-byte doubles
* flipdcd -- flips byte-ordering of DCD files
* psfgen -- [https://www.ks.uiuc.edu/Research/vmd/ VMD] psfgen plugin
* sortreplicas -- un-shuffles replica trajectories to place same-temperature frames in the same file

=== Charm++ ===

Charm++ is a parallel object-oriented programming paradigm based on C++ and developed in the Parallel Programming Laboratory at the University of Illinois at Urbana–Champaign. NAMD has been implemented using Charm++.

* charmrun -- launches Charm++ programs

For help on usage, type
<pre>
$ charmrun -help
</pre>

=== GPU Acceleration ===

Currently not available.

== Hints for using NAMD ==

=== Parallel Execution ===

It is usually more efficient to run several smaller jobs with reasonable speedup side by side. Check for a reasonable speedup by increasing the number of workers from 1, 2, 4, n cores. Do not forget to adjust the memory specification when changing the number of workers

=== Memory Management ===

It requires some experience to choose the perfect memory value. Monitoring a job interactively may help to estimate the memory consumption. Requesting 1GB more than the required amount is typically sufficient.

=== Runtime Specification ===

The wall-time limit is independent of the number of parallel workers. Selecting the best value requires some experience. In general, jobs with shorter wall times get higher priorities. It could help to run the job interactively for a small period of time to monitor its convergence. Furthermore, it may be advantageous to extrapolate the run time by looking at smaller test jobs.

= Examples =

As with all processes that require more than a few minutes to run, non-trivial compute jobs must be submitted to the cluster queuing system.

Example scripts are available in the directory <span style="background:#edeae2;margin:2px;padding:1px;border:1px dotted #808080">$NAMD_EXA_DIR</span>:
<pre>
$ module show chem/namd # show environment variables, which will be available after 'module load'
$ module load chem/namd # load module
$ ls $NAMD_EXA_DIR # show content of directory $NAMD_EXA_DIR
$ cat $NAMD_EXA_DIR/README # show examples README
</pre>

Run a first simple example job on <font color=red>JUSTUS 2</font>:
<pre>
$ module load chem/namd # load module
$ WORKSPACE=`ws_allocate namd 3` # allocate workspace
$ cd $WORKSPACE # change to workspace
$ cp -a $NAMD_HOME/bwhpc-examples . # copy example files to workspace
$ cd bwhpc-examples/apoa1 # change to test directory
$ sbatch bwhpc_namd_apoa1.sh # submit job
$ squeue # obtain JOBID
$ scontrol show job <JOBID> # check state of job
$ ls # when job finishes the results will be visible in this directory
</pre>

= Example Scripts ==

<syntaxhighlight lang="bash">
#!/bin/bash
# Allocate two nodes
#SBATCH --nodes=2
# Number of program instances to be executed
#SBATCH --tasks-per-node=48
# Allocate 20 (of 192) GB memory per node
#SBATCH --mem=20gb
# Maximum run time of job
#SBATCH --time=00:10:00
# Give job a reasonable name
#SBATCH --job-name=bwhpc-example
# File name for stdout/stderr (%j will be replaced by job id)
#SBATCH --output=bwhpc-example_ID-%j.out

#########################
#
# bwHPC examples illustrate how to submit a batch job to the cluster queuing system. It is necessary to adapt certain specifications for your particular case.
#
# General Hints
#
# Parallel Execution:
# It is usually more efficient to run several smaller jobs with reasonable speedup simultaneously. Check for a reasonable speedup by gradually increasing the number of workers from 1 or 2 to n cores. Do not forget to adjust the memory specification when changing the number of workers. Use $ seff <jobid> to get efficiency information.
#
# Memory Management:
# It requires some experience to choose the perfect memory value. Monitoring a job interactively may help to estimate the memory consumption. Requesting 1GB more than the required amount is typically sufficient. Use $ seff <jobid> to get efficiency information.
#
# Runtime Specification:
# The wall-time limit is independent of the number of parallel workers. In general, jobs with shorter wall times get higher priorities. It could help to run the job interactively for a small period of time to monitor its convergence. Furthermore, it may be advantageous to extrapolate the run time by looking at smaller test jobs first.
#
#########################

module purge
module load chem/namd/2.14
module list

srun namd2 +isomalloc_sync apoa1.namd

exit 0
</syntaxhighlight>

= Useful links =

* [http://www.ks.uiuc.edu/Research/namd/2.14/ug/ Documentation (english)]
* [http://www.ks.uiuc.edu/Research/namd/mailing_list/ Maling list (english)]
* [https://en.wikipedia.org/wiki/NAMD Wikipedia article (english)]
* [https://de.wikipedia.org/wiki/NAMD Wikipedia article (german)]

JUSTUS2/Software/NAMD

2026-09-22T07:59:39Z

K Siegmund:

{{Softwarepage|chem/namd}}

{| width=760px class="wikitable"
|-
! Description !! Content
|-
| module load
| chem/namd
|-
| Availability
| [[BwForCluster_JUSTUS_2]] |
|-
| License
| Freeware for non-commercial use. [[#License|More...]]
|-
| Citing
| [http://www.ks.uiuc.edu/Research/namd/papers.html Publications]
|-
| Links
| [http://www.ks.uiuc.edu/Research/namd/ Homepage] | [http://www.ks.uiuc.edu/Research/namd/2.14/ug/ Documentation]
|-
| Graphical Interface
| No
|}

= Description =

'''NAMD''' (NAnoscale Molecular Dynamics) is a parallel molecular dynamics code designed for high-performance simulation of large biomolecular systems.

= Availability =

NAMD is available on selected bwHPC-Clusters. A complete list of versions currently installed on the bwHPC-Clusters can be obtained from https://www.bwhpc.de/software.html
In order to check which versions of NAMD are installed on the compute cluster, run the following command:
<pre>
$ module avail chem/namd
</pre>

= License =

NAMD is available as [https://en.wikipedia.org/wiki/Freeware freeware] for non-commercial use by individuals, academic institutions, and corporations for in-house business uses. Please read the [http://www.ks.uiuc.edu/Research/namd/license.html license] for additional information about NAMD.

= Usage =

== Loading the module ==

You can load the default version of NAMD with the following command:
<pre>
$ module load chem/namd
</pre>

The module will try to load all modules (i.e., compiler, MPI, MKL) it needs to function. If loading the module fails, check if you have already loaded one of those modules, but not in the version required by NAMD.

If you wish to load another (older) version of NAMD, you can do so using
<pre>
$ module load chem/namd/<version>
</pre>
with <version> specifying the desired version.

Please cite NAMD in your publications according to the [http://www.ks.uiuc.edu/Research/namd/papers.html references].

== Program Binaries ==

The binary ''namd2'' is main program of the NAMD package.
<pre>
Usage: namd2 [options] [config file]

Example: namd2 +isomalloc_sync run.namd
</pre>

For information about how to construct NAMD configuration files, please see the [http://www.ks.uiuc.edu/Research/namd/2.14/ug/ documentation]. In addition to a configuration file, NAMD also needs a CHARMM force field in either CHARMM or X-PLOR format, an X-PLOR format PSF file describing the molecular structure, and the initial coordinates of the molecular system in the form of a PDB file.

The NAMD package includes several additional tools:

* flipbinpdb -- flips byte-ordering of 8-byte doubles
* flipdcd -- flips byte-ordering of DCD files
* psfgen -- [https://www.ks.uiuc.edu/Research/vmd/ VMD] psfgen plugin
* sortreplicas -- un-shuffles replica trajectories to place same-temperature frames in the same file

=== Charm++ ===

Charm++ is a parallel object-oriented programming paradigm based on C++ and developed in the Parallel Programming Laboratory at the University of Illinois at Urbana–Champaign. NAMD has been implemented using Charm++.

* charmrun -- launches Charm++ programs

For help on usage, type
<pre>
$ charmrun -help
</pre>

=== GPU Acceleration ===

Currently not available.

== Hints for using NAMD ==

=== Parallel Execution ===

It is usually more efficient to run several smaller jobs with reasonable speedup side by side. Check for a reasonable speedup by increasing the number of workers from 1, 2, 4, n cores. Do not forget to adjust the memory specification when changing the number of workers

=== Memory Management ===

It requires some experience to choose the perfect memory value. Monitoring a job interactively may help to estimate the memory consumption. Requesting 1GB more than the required amount is typically sufficient.

=== Runtime Specification ===

The wall-time limit is independent of the number of parallel workers. Selecting the best value requires some experience. In general, jobs with shorter wall times get higher priorities. It could help to run the job interactively for a small period of time to monitor its convergence. Furthermore, it may be advantageous to extrapolate the run time by looking at smaller test jobs.

= Examples =

As with all processes that require more than a few minutes to run, non-trivial compute jobs must be submitted to the cluster queuing system.

Example scripts are available in the directory <span style="background:#edeae2;margin:2px;padding:1px;border:1px dotted #808080">$NAMD_EXA_DIR</span>:
<pre>
$ module show chem/namd # show environment variables, which will be available after 'module load'
$ module load chem/namd # load module
$ ls $NAMD_EXA_DIR # show content of directory $NAMD_EXA_DIR
$ cat $NAMD_EXA_DIR/README # show examples README
</pre>

Run a first simple example job on <font color=red>JUSTUS 2</font>:
<pre>
$ module load chem/namd # load module
$ WORKSPACE=`ws_allocate namd 3` # allocate workspace
$ cd $WORKSPACE # change to workspace
$ cp -a $NAMD_HOME/bwhpc-examples . # copy example files to workspace
$ cd bwhpc-examples/apoa1 # change to test directory
$ sbatch bwhpc_namd_apoa1.sh # submit job
$ squeue # obtain JOBID
$ scontrol show job <JOBID> # check state of job
$ ls # when job finishes the results will be visible in this directory
</pre>

== Example Scripts ==

= Useful links =

* [http://www.ks.uiuc.edu/Research/namd/2.14/ug/ Documentation (english)]
* [http://www.ks.uiuc.edu/Research/namd/mailing_list/ Maling list (english)]
* [https://en.wikipedia.org/wiki/NAMD Wikipedia article (english)]
* [https://de.wikipedia.org/wiki/NAMD Wikipedia article (german)]

BwForCluster JUSTUS 2 Slurm HOWTO

2026-09-17T12:26:41Z

K Siegmund: /* How to retrieve job records for all jobs running/pending at a certain point in time? */

{{Justus2}}

This is a collection of howtos and convenient Slurm commands for JUSTUS 2.

Some commands behave slightly different depending on whether they are executed
by a system administrator or by a regular user, as Slurm prevents regular users from accessing critical system information and viewing job and usage information of other users.

= GENERAL INFORMATION =

== How to find a general quick start user guide? ==

https://slurm.schedmd.com/quickstart.html

== How to find Slurm FAQ? ==

https://slurm.schedmd.com/faq.html

== How to find a Slurm cheat sheet? ==

https://slurm.schedmd.com/pdfs/summary.pdf

== How to find Slurm tutorials? ==

https://slurm.schedmd.com/tutorials.html

== How to get more information on Slurm? ==

(Almost) every Slurm command has a man page. Use it.

Online versions: https://slurm.schedmd.com/man_index.html

== How to find hardware specific details about JUSTUS 2? ==

See our Wiki page: [[Hardware and Architecture (bwForCluster JUSTUS 2)|Hardware and Architecture]]

= JOB SUBMISSION =

== How to submit a serial batch job? ==

Use [https://slurm.schedmd.com/sbatch.html sbatch] command:

<pre>$ sbatch <job-script> </pre>

Sample job script template for serial job:

<source lang="bash">
#!/bin/bash
# Allocate one node
#SBATCH --nodes=1
# Number of program instances to be executed
#SBATCH --ntasks-per-node=1
# 8 GB memory required per node
#SBATCH --mem=8G
# Maximum run time of job
#SBATCH --time=1:00:00
# Give job a reasonable name
#SBATCH --job-name=serial_job
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=serial_job-%j.out
# File name for error output
#SBATCH --error=serial_job-%j.err

# Load software modules as needed, e.g.
# module load foo/bar

# Run serial program
./my_serial_program
</source>

Sample code for serial program: [[Media:Hello_serial.c | Hello_serial.c]]

'''Notes:'''
* --nodes=1 and --ntasks-per-node=1 may be replaced by --ntasks=1.
* If not specified, stdout and stderr are both written to slurm-%j.out.

== How to find working sample scripts for my program? ==

Most software modules for applications provide working sample batch scripts.
Check with [[Software_Modules_Lmod#Module_specific_help | module help]] command, e.g.
<pre>
$ module help chem/vasp # display module help for VASP
$ module help math/matlab # display module help for Matlab
</pre>

== How to harden job scripts against common errors? ==

The bash shell provides several options that support users in disclosing hidden bugs and writing safer job scripts.
In order to activate these safeguard settings users can insert the following lines in their scripts (after all #SBATCH directives):

<source lang="bash">
[...]
set -o errexit # (or set -e) cause batch script to exit immediately when a command fails.
set -o pipefail # cause batch script to exit immediately also when the command that failed is embedded in a pipeline
set -o nounset # (or set -u) causes the script to treat unset variables as an error and exit immediately
[...]
</source>

== How to submit an interactive job? ==

Use [https://slurm.schedmd.com/salloc.html salloc] command, e.g.:
<pre>$ salloc --nodes=1 --ntasks-per-node=8</pre>

'''Note:'''

In previous Slurm versions < 20.11 the use of [https://slurm.schedmd.com/srun.html srun] has been the recommended way for launching interactive jobs, e.g.:

<pre>$ srun --nodes=1 --ntasks-per-node=8 --pty bash </pre>

Although this still works with current Slurm versions this is considered '''deprecated ''' for current Slurm versions as it may cause issues when launching additional jobs steps from within the interactive job environment. Use [https://slurm.schedmd.com/salloc.html salloc] command.

== How to enable X11 forwarding for an interactive job? ==

Use '--x11' flag, e.g.

<pre>
$ salloc --nodes=1 --ntasks-per-node=8 --x11 # run shell with X11 forwarding enabled
</pre>

'''Note:'''
* For X11 forwarding to work, you must also enable X11 forwarding for your ssh login from your local computer to the cluster, i.e.:
<pre>local> ssh -X <username>@justus2.uni-ulm.de></pre>

== How to convert Moab batch job scripts to Slurm? ==

Replace Moab/Torque job specification flags and environment variables in your job
scripts by their corresponding Slurm counterparts.

'''Commonly used Moab job specification flags and their Slurm equivalents'''

{| width=750px class="wikitable"
! Option !! Moab (msub) !! Slurm (sbatch)
|-
| Script directive || #MSUB || #SBATCH
|-
| Job name || -N <name> || --job-name=<name> (-J <name>)
|-
| Account || -A <account> || --account=<account> (-A <account>)
|-
| Queue || -q <queue> || --partition=<partition> (-p <partition>)
|-
| Wall time limit || -l walltime=<hh:mm:ss> || --time=<hh:mm:ss> (-t <hh:mm:ss>)
|-
| Node count || -l nodes=<count> || --nodes=<count> (-N <count>)
|-
| Core count || -l procs=<count> || --ntasks=<count> (-n <count>)
|-
| Process count per node || -l ppn=<count> || --ntasks-per-node=<count>
|-
| Core count per process || || --cpus-per-task=<count> (-c <count>)
|-
| Memory limit per node || -l mem=<limit> || --mem=<limit>
|-
| Memory limit per process || -l pmem=<limit> || --mem-per-cpu=<limit>
|-
| Job array || -t <array indices> || --array=<indices> (-a <indices>)
|-
| Node exclusive job || -l naccesspolicy=singlejob || --exclusive
|-
| Initial working directory || -d <directory> (default: $HOME) || --chdir=<directory> (-D <directory>) (default: submission directory)
|-
| Standard output file || -o <file path> || --output=<file> (-o <file>)
|-
| Standard error file || -e <file path> || --error=<file> (-e <file>)
|-
| Combine stdout/stderr to stdout || -j oe || --output=<combined stdout/stderr file>
|-
| Mail notification events || -m <event> || --mail-type=<events> (valid types include: NONE, BEGIN, END, FAIL, ALL)
|-
| Export environment to job || -V || --export=ALL (default)
|-
| Don't export environment to job || (default) || --export=NONE
|-
| Export environment variables to job || -v <var[=value][,var2=value2[, ...]]> || --export=<var[=value][,var2=value2[,...]]>
|}

'''Notes:'''
* Default initial job working directory is $HOME for Moab. For Slurm the default working directory is where you submit your job from.
* By default Moab does not export any environment variables to the job's runtime environment. With Slurm most of the login environment variables are exported to your job's runtime environment. This includes environment variables from software modules that were loaded at job submission time (and also $HOSTNAME variable).

'''Commonly used Moab/Torque script environment variables and their Slurm equivalents

{| width=750px class="wikitable"
! Information !! Moab !! Torque !! Slurm
|-
| Job name || $MOAB_JOBNAME || $PBS_JOBNAME || $SLURM_JOB_NAME
|-
| Job ID || $MOAB_JOBID || $PBS_JOBID || $SLURM_JOB_ID
|-
| Submit directory || $MOAB_SUBMITDIR || $PBS_O_WORKDIR || $SLURM_SUBMIT_DIR
|-
| Number of nodes allocated || $MOAB_NODECOUNT || $PBS_NUM_NODES || $SLURM_JOB_NUM_NODES (and: $SLURM_NNODES)
|-
| Node list || $MOAB_NODELIST || cat $PBS_NODEFILE || $SLURM_JOB_NODELIST
|-
| Number of processes || $MOAB_PROCCOUNT || $PBS_TASKNUM || $SLURM_NTASKS
|-
| Requested tasks per node || --- || $PBS_NUM_PPN || $SLURM_NTASKS_PER_NODE
|-
| Requested CPUs per task || --- || --- || $SLURM_CPUS_PER_TASK
|-
| Job array index || $MOAB_JOBARRAYINDEX || $PBS_ARRAY_INDEX || $SLURM_ARRAY_TASK_ID
|-
| Job array range || $MOAB_JOBARRAYRANGE || - || $SLURM_ARRAY_TASK_COUNT
|-
| Queue name || $MOAB_CLASS || $PBS_QUEUE || $SLURM_JOB_PARTITION
|-
| QOS name || $MOAB_QOS || --- || $SLURM_JOB_QOS
|-
| Number of processes per node | --- || $PBS_NUM_PPN || $SLURM_TASKS_PER_NODE
|-
| Job user || $MOAB_USER || $PBS_O_LOGNAME || $SLURM_JOB_USER
|-
| Hostname || $MOAB_MACHINE || $PBS_O_HOST || $SLURMD_NODENAME
|}

'''Note:'''

* See [https://slurm.schedmd.com/sbatch.html sbatch] man page for a complete list of flags and environment variables.

== How to emulate Moab output file names? ==

Use the following directives:

<pre>
#SBATCH --output="%x.o%j"
#SBATCH --error="%x.e%j"
</pre>

== How to pass command line arguments to the job script? ==

Run <pre>$ sbatch <job-script> arg1 arg2 ...</pre>

Inside the job script the arguments can be accessed as $1, $2, ...

E.g.:

<source lang="bash">
[...]
infile="$1"
outfile="$2"
./my_serial_program < "$infile" > "$outfile" 2>&1
[...]
</source>

'''Notes:'''

* Do '''not''' use $1, $2, ... in "#SBATCH" lines. These parameters can be used only within the regular shell script.

== How to request local scratch (SSD/NVMe) at job submission? ==

Use '--gres=scratch:nnn' option to allocate nnn GB of local (i.e. node-local) scratch space for the entire job.

Example: '--gres=scratch:100' will allocate 100 GB scratch space on a locally attached NVMe device.

'''Notes:'''

* Do '''not''' add any unit (such as --gres=scratch:100G). This would be treated as requesting an amount of 10^9 * 100GB of scratch space.

* Multinode jobs get nnn GB of local scratch space on every node of the job.

* Environment variable '''$SCRATCH''' will point to
** /scratch/<user>.<jobid> when local scratch has been requested. This will be on locally attached SSD/NVMe devices.
** /tmp/<user>.<jobid> when no local scratch has been requested. This will be in memory and, thus, be limited in size.

* Environment variable '''$TMPDIR''' always points to /tmp/<user>.<jobid>. This will always be in memory and, thus, limited in size.

* For backward compatibility environment variable $RAMDISK always points to /tmp/<user>.<jobid>

* Scratch space allocation in /scratch will be enforced by quota limits

* Data written to $TMPDIR will always count against allocated memory.

* Data written to local scratch space will automatically be removed at the end of the job.

== How to request GPGPU nodes at job submission? ==

Use '--gres=gpu:<count>' option to allocate 1 or 2 GPUs per node for the entire job.

Example: '--gres=gpu:1' will allocate one GPU per node for this job.

'''Notes:'''

* GPGPU nodes are equipped with two Nvidia V100S cards

* Environment variables $CUDA_VISIBLE_DEVICES, $SLURM_JOB_GPUS and $GPU_DEVICE_ORDINAL will denote card(s) allocated for the job.

* CUDA Toolkit is available as software module devel/cuda.

== How to clean-up or save files before a job times out? ==

Possibly you would like to clean up the work directory or save intermediate result files in case a job times out.

The following sample script may serve as a blueprint for implementing a pre-termination function to perform clean-up or file recovery actions.

<source lang="bash">
#!/bin/bash
# Allocate one node
#SBATCH --nodes=1
# Number of program instances to be executed
#SBATCH --ntasks-per-node=1
# 2 GB memory required per node
#SBATCH --mem=2G
# Request 10 GB local scratch space
#SBATCH --gres=scratch:10
# Maximum run time of job
#SBATCH --time=10:00
# Send the USR1 signal 120 seconds before end of time limit
#SBATCH --signal=B:USR1@120
# Give job a reasonable name
#SBATCH --job-name=signal_job
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=signal_job-%j.out
# File name for error output
#SBATCH --error=signal_job-%j.err

# Define the signal handler function
# Note: This is not executed here, but rather when the associated
# signal is received by the shell.
finalize_job()
{
# Do whatever cleanup you want here. In this example we copy
# output file(s) back to $SLURM_SUBMIT_DIR, but you may implement
# your own job finalization code here.
echo "function finalize_job called at `date`"
cd $SCRATCH
mkdir -vp "$SLURM_SUBMIT_DIR"/results
tar czvf "$SLURM_SUBMIT_DIR"/results/${SLURM_JOB_ID}.tgz output*.txt
exit
}

# Call finalize_job function as soon as we receive USR1 signal
trap 'finalize_job' USR1

# Copy input files for this job to the scratch directory (if needed).
# Note: Environment variable $SCRATCH always points to a scratch directory
# automatically created for this job. Environment variable $SLURM_SUBMIT_DIR
# points to the path where this script was submitted from.
# Example:
# cp -v "$SLURM_SUBMIT_DIR"/input*.txt "$SCRATCH"

# Change working directory to local scratch directory
cd "$SCRATCH"

# Load software modules as needed, e.g.
# module load foo/bar

# This is where the actual work is done. In this case we just create
# a sample output file for 900 (=15*60) seconds, but since we asked
# Slurm for 600 seconds only it will not be able finish within this
# wall time.
# Note: It is important to run this task in the background
# by placing the & symbol at the end. Otherwise the signal handler
# would not be executed until that process has finished, which is not
# what we want.
(for i in `seq 15`; do echo "Hello World at `date +%H:%M:%S`."; sleep 60; done) >output.txt 2>&1 &

# Note: The command above is just for illustration. Normally you would just run
# my_program >output.txt 2>&1 &

# Tell the shell to wait for background task(s) to finish.
# Note: This is important because otherwise the parent shell
# (this script) would proceed (and terminate) without waiting for
# background task(s) to finish.
wait

# If we get here, the job did not time out but finished in time.

# Release user defined signal handler for USR1
trap - USR1

# Do regular cleanup and save files. In this example we simply call
# the same function that we defined as a signal handler above, but you
# may implement your own code here.
finalize_job

exit
</source>

'''Notes:'''

* The number of seconds specified in --signal option must match the runtime of the pre-termination function and must not exceed 65535 seconds.

* Due to the resolution of event handling by Slurm, the signal may be sent a little earlier than specified.

== How to submit a multithreaded batch job? ==

Sample job script template for a job running one multithreaded program instance:

<source lang="bash">
#!/bin/bash
# Allocate one node
#SBATCH --nodes=1
# Number of program instances to be executed
#SBATCH --ntasks-per-node=1
# Number of cores per program instance
#SBATCH --cpus-per-task=8
# 8 GB memory required per node
#SBATCH --mem=8G
# Maximum run time of job
#SBATCH --time=1:00:00
# Give job a reasonable name
#SBATCH --job-name=multithreaded_job
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=multithreaded_job-%j.out
# File name for error output
#SBATCH --error=multithreaded_job-%j.err

# Load software modules as needed, e.g.
# module load foo/bar

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
export MKL_NUM_THREADS=${SLURM_CPUS_PER_TASK}

# Run multithreaded program
./my_multithreaded_program
</source>

Sample code for multithreaded program: [[Media:Hello_openmp.c | Hello_openmp.c]]

'''Notes:'''

* In our configuration each physical core is considered a "CPU".

* On JUSTUS 2 it is recommended to specify a number of cores per task ('--cpus-per-task') that is either an integer divisor of 24 or (at most) 48.

* Required memory can also by specified per allocated CPU with '--mem-per-cpu' option.

* The '--mem' and '--mem-per-cpu' options are mutually exclusive.

== How to submit an array job? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_array -a] (or [https://slurm.schedmd.com/sbatch.html#OPT_array --array]) option, e.g.

<pre>$ sbatch -a 1-16%8 ...</pre>

This will submit 16 tasks to be executed, each one indexed by SLURM_ARRAY_TASK_ID ranging from 1 to 16, but will limit the number of simultaneously running tasks from this job array to 8.

Sample job script template for an array job:

<source lang="bash">
#!/bin/bash
# Number of cores per individual array task
#SBATCH --ntasks=1
#SBATCH --array=1-16%8
#SBATCH --mem=4G
#SBATCH --time=01:00:00
#SBATCH --job-name=array_job
#SBATCH --output=array_job-%A_%a.out
#SBATCH --error=array_job-%A_%a.err

# Load software modules as needed, e.g.
# module load foo/bar

# Print the task id.
echo "My SLURM_ARRAY_TASK_ID: " $SLURM_ARRAY_TASK_ID

# Add lines here to run your computations, e.g.
# ./my_program <input.$SLURM_ARRAY_TASK_ID

</source>

'''Notes:'''

* Placeholder %A will be replaced by the master job id, %a will be replaced by the array task id.

* Every sub job in an array job will have its own unique environment variable $SLURM_JOB_ID. Environment variable $SLURM_ARRAY_JOB_ID will be set to the first job array index value for all tasks.

* The remaining options in the sample job script are the same as the options used in other, non-array jobs. In the example above, we are requesting that each array task be allocated 1 CPU (--ntasks=1) and 4 GB of memory (--mem=4G) for up to one hour (--time=01:00:00).

* More information: https://slurm.schedmd.com/job_array.html

== How to delay the start of a job? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_begin -b] (or [https://slurm.schedmd.com/sbatch.html#OPT_begin --begin]) option in order to defer the allocation of the job until the specified time.

Examples:

<pre>
sbatch --begin=20:00 ... # job can start after 8 p.m.
sbatch --begin=now+1hour ... # job can start 1 hour after submission
sbatch --begin=teatime ... # job can start at teatime (4 p.m.)
sbatch --begin=2023-12-24T20:00:00 ... # job can start after specified date/time
</pre>

== How to submit dependency (chain) jobs? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_dependency -d] (or [https://slurm.schedmd.com/sbatch.html#OPT_dependency --dependency]) option, e.g.

<pre>$ sbatch -d afterany:123456 ...</pre>

This will defer the submitted job until the specified job 123456 has terminated.

Slurm supports a number of different dependency types, e.g.:

<pre>
-d after:123456 # job can begin execution after the specified job has begun execution
-d afterany:123456 # job can begin execution after the specified job has finished
-d afternotok:123456 # job can begin execution after the specified job has failed (exit code not equal zero)
-d afterok:123456 # job can begin execution after the specified job has successfully finished (exit code zero)
-d singleton # job can begin execution after any previously job with the same job name and user have finished
</pre>

'''Note:''' Multiple jobs can be specified by separating their job ids by colon characters (:), e.g.

<pre> $ sbatch -d afterany:123456:123457 ... </pre>

This will defer the submitted job until the specified jobs 123456 and 123457 have both finished.

== How to deal with invalid job dependencies? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_kill-on-invalid-dep --kill-on-invalid-dep=yes] option in order to automatically terminate jobs which can never run due to invalid dependencies. By default the job stays pending with reason 'DependencyNeverSatisfied' to allow review and appropriate action by the user.

'''Note:''' A job dependency may also become invalid if a job has been submitted with '-d afterok:<jobid>' but the specified dependency job has failed, e.g. because it timed out (i.e. exceeded its wall time limit).

== How to submit an MPI batch job? ==

Suggested reading: https://slurm.schedmd.com/mpi_guide.html

Sample job script template for an MPI job:

<source lang="bash">
#!/bin/bash
# Allocate two nodes
#SBATCH --nodes=2
# Number of program instances to be executed
#SBATCH --ntasks-per-node=48
# Allocate 32 GB memory per node
#SBATCH --mem=32gb
# Maximum run time of job
#SBATCH --time=1:00:00
# Give job a reasonable name
#SBATCH --job-name=mpi_job
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=mpi_job-%j.out
# File name for error output
#SBATCH --error=mpi_job-%j.err

# Add lines here to run your computations, e.g.
#
# Option 1: Lauch MPI tasks by using mpirun
#
# for OpenMPI and GNU compiler:
#
# module load compiler/gnu
# module load mpi/openmpi
# mpirun ./my_mpi_program
#
# for Intel MPI and Intel complier:
#
# module load compiler/intel
# module load mpi/impi
# mpirun ./my_mpi_program
#
# Option 2: Launch MPI tasks by using srun
#
# for OpenMPI and GNU compiler:
#
# module load compiler/gnu
# module load mpi/openmpi
# srun ./my_mpi_program
#
# for Intel MPI and Intel compiler:
#
module load compiler/intel
module load mpi/impi
srun ./my_mpi_program

</source>

Sample code for MPI program: [[Media:Hello_mpi.c | Hello_mpi.c]]

'''Notes'''

* SchedMD recommends to use srun and many (most?) sites do so as well. The rationale is that srun is more tightly integrated with the scheduler and provides more consistent and reliable resource tracking and accounting for individual jobs and job steps. mpirun may behave differently for different MPI implementations and versions. There are reports that claim "strange behavior" of mpirun especially when using task affinity and core binding. Using srun is supposed to resolve these issues and is therefore highly recommended.
* Do not run batch jobs that launch a large number (hundreds or thousands) short running (few minutes or less) MPI programs, e.g. from a shell loop. Every single MPI invocation does generate its own job step and sends remote procedure calls to the Slurm controller server. This can result in degradation of performance for both, Slurm and the application, especially if many of that jobs happen to run at the same time. Jobs of that kind can even get stuck without showing any further activity until hitting the wall time limit. For high throughput computing (e.g. processing a large number of files with every single task running independently from each other and very shortly), consider a more appropriate parallelization paradigm that invokes independent serial (non-MPI) processes in parallel at the same time. This approach is sometimes referred to as "[https://en.wikipedia.org/wiki/Embarrassingly_parallel pleasingly parallel]" workload. GNU Parallel is a shell tool that facilitates executing serial tasks in parallel. On JUSTUS 2 this tool is available as a software module "system/parallel".

== How to submit a hybrid MPI/OpenMP job? ==

Sample job script template for an hybrid job:

<source lang="bash">
#!/bin/bash
# Number of nodes to allocate
#SBATCH --nodes=4
# Number of MPI instances (ranks) to be executed per node
#SBATCH --ntasks-per-node=2
# Number of threads per MPI instance
#SBATCH --cpus-per-task=24
# Allocate 8 GB memory per node
#SBATCH --mem=8gb
# Maximum run time of job
#SBATCH --time=1:00:00
# Give job a reasonable name
#SBATCH --job-name=hybrid_job
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=hybrid_job-%j.out
# File name for error output
#SBATCH --error=hybrid_job-%j.err

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
export MKL_NUM_THREADS=${SLURM_CPUS_PER_TASK}

module load compiler/intel
module load mpi/impi
srun ./my_hybrid_program

# or:
# mpirun ./my_hybrid_program

</source>

Sample code for hybrid program: [[Media:Hello_hybrid.c | Hello_hybrid.c]]

'''Notes:'''

* $SLURM_CPUS_PER_TASK is only set if the '--cpus-per-task' option is specified.

== How to request specific node(s) at job submission? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_nodelist -w] (or [https://slurm.schedmd.com/sbatch.html#OPT_nodelist --nodelist]) option, e.g.:

<pre>$ sbatch -w <node1>,<node2> ...</pre>

Also see [https://slurm.schedmd.com/sbatch.html#OPT_nodefile -F] (or [https://slurm.schedmd.com/sbatch.html#OPT_nodefile --nodefile]) option.

== How to exclude specific nodes from job? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_exclude -x] (or [https://slurm.schedmd.com/sbatch.html#OPT_exclude --exclude]) option, e.g.:

<pre>$ sbatch -x <node1>,<node2> ...</pre>

== How to get exclusive jobs? ==

Use '--exclusive' option on job submission. This makes sure that there will be no other jobs running on your nodes. Very useful for benchmarking!

'''Note:'''

* --exclusive option does '''not''' mean that you automatically get full access to all the resources which the node might provide without explicitly requesting them.

== How to avoid sharing nodes with other users? ==

Use '--exclusive=user' option on job submission. This will still allow multiple jobs of one and the same user on the nodes.

''' Note:'''

* Depending on configuration, exclusive=user may (and probably will) be the default node access policy on JUSTUS 2.

== How to submit batch job without job script? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_wrap --wrap] option.

Example:

<pre>$ sbatch --nodes=2 --ntasks-per-node=16 --wrap "sleep 600"</pre>

'''Note:''' May be useful for testing purposes.

= JOB MONITORING AND CONTROL =

== How to prevent Slurm performance degradation? ==

Almost every invocation of a Slurm client command (e.g. squeue, sacct, sprio or sshare) sends a remote procedure call (RPC) to the Slurm control daemon and/or database.
If enough remote procedure calls come in at once, this can result in a degradation of performance of the Slurm services for all users, possibly resulting in a denial of service.

Therefore, '''do not run Slurm client commands that send remote procedure calls from loops in shell scripts or other programs''' (such as 'watch squeue'). Always ensure to limit calls to squeue, sstat, sacct etc. to the minimum necessary for the information you are trying to gather.

Slurm does collect RPC counts and timing statistics by message type and user for diagnostic purposes.

== How to view information about submitted jobs? ==

Use [https://slurm.schedmd.com/squeue.html squeue] command, e.g.:

<pre>
$ squeue # all jobs owned by user (all jobs owned by all users for admins)
$ squeue --me # all jobs owned by user (same as squeue for regular users)
$ squeue -u <username> # jobs of specific user
$ squeue -t PENDING # pending jobs only
$ squeue -t RUNNING # running jobs only
</pre>

'''Notes:'''

* The output format of [https://slurm.schedmd.com/squeue.html squeue] (and most other Slurm commands) is highly configurable to your needs. Look for the --format or --Format options.

* Every invocation of squeue sends a remote procedure call to the Slurm database server. '''Do not run squeue or other Slurm client commands from loops in shell scripts or other programs''' as this can result in a degradation of performance. Ensure that programs limit calls to squeue to the minimum necessary for the information you are trying to gather.

== How to cancel jobs? ==

Use [https://slurm.schedmd.com/scancel.html scancel] command, e.g.

<pre>
$ scancel <jobid> # cancel specific job
$ scancel <jobid>_<index> # cancel indexed job in a job array
$ scancel -u <username> # cancel all jobs of specific user
$ scancel -u <username> -i # cancel all jobs of specific user interactively
$ scancel -t PENDING # cancel pending jobs
$ scancel -t RUNNING # cancel running jobs
</pre>

== How to show job script of a running job? ==

Use [https://slurm.schedmd.com/scontrol.html scontrol] command:

<pre>
$ scontrol write batch_script <job_id> <file>
$ scontrol write batch_script <job_id> -
</pre>

* If file name is omitted default file name will be slurm-<job_id>.sh
* If file name is - (i.e. dash) job script will be written to stdout.

== How to get estimated start time of a job? ==

<pre>$ squeue --start</pre>

'''Notes:'''
* Estimated start times are dynamic and can change at any moment. Exact start times of individual jobs are usually unpredictable.
* Slurm will report N/A for the start time estimate if nodes are not currently being reserved by the scheduler for the job to run on.

== How to show remaining walltime of running jobs? ==

Use [https://slurm.schedmd.com/squeue.html squeue] with format option "%L", e.g.:

<pre> $ squeue -t r -o "%u %i %L" </pre>

== How to check priority of jobs? ==

Use [https://slurm.schedmd.com/squeue.html squeue] with format options "%Q" and/or "%p", e.g.:

<pre>
$ squeue -o "%8i %8u %15a %.10r %.10L %.5D %.10Q"
</pre>

Use [https://slurm.schedmd.com/sprio.html sprio] command to display the priority components (age/fairshare/...) for each job:

<pre>
$ sprio
</pre>

Use "[https://slurm.schedmd.com/sshare.html sshare] command for listing the shares of associations, e.g. accounts.

<pre>
$ sshare
</pre>

== How to prevent (hold) jobs from being scheduled for execution? ==

<pre>
$ scontrol hold <job_id>
</pre>

== How to unhold job? ==

<pre>
$ scontrol release <job_id>
</pre>

== How to suspend a running job? ==

<pre>
$ scontrol suspend <job_id>
</pre>

== How to resume a suspended job? ==

<pre>
$ scontrol resume <job_id>
</pre>

== How to requeue (cancel and resubmit) a particular job? ==

<pre>
$ scontrol requeue <job_id>
</pre>

== How to monitor resource usage of running job(s)? ==

Use "[https://slurm.schedmd.com/sstat.html sstat] command.

'sstat -e' command shows a list of fields that can be specified with the '--format' option.

Example:

<pre>
$ sstat --format=JobId,AveCPU,AveRSS,MaxRSS -j <jobid>
</pre>

This will show average CPU time, average and maximum memory consumption of all tasks in the running job.
Ideally, average CPU time equals the number of cores allocated for the job multiplied by the current run time of the job.
The maximum memory consumption gives an estimate of the peak amount of memory actually needed so far. This can be compared with the amount of memory requested for the job. Over-requesting memory can result in significant waste of compute resources.

'''Notes:'''

* Users can also ssh into compute nodes that they have one or more running jobs on. Once logged in, they can use standard Linux process monitoring tools like ps, (h)top, free, vmstat, iostat, du, ...

* Users can also attach an interactive shell under an already allocated job by running the following command: <pre>srun --jobid <job> --overlap --pty /bin/bash</pre> Once logged in, they can again use standard Linux process monitoring tools like ps, (h)top, free, vmstat, iostat, du, ... For a single node job the user does not even need to know the node that the job is running on. For a multinode job, the user can still use '-w <node>' option to specify a specific node.

== How to get detailed job information ==

<pre>
$ scontrol show job 1234 # For job id 1234
$ scontrol show jobs # For all jobs
$ scontrol -o show jobs # One line per job
</pre>

== How to modify a pending/running job? ==

Use

<pre>$ scontrol update JobId=<jobid> ...</pre>

E.g.: <pre>$ scontrol update JobId=42 TimeLimit=7-0</pre>

This will modify the time limit of the job to 7 days.

'''Note:''' Update requests for '''running''' jobs are mostly restricted to Slurm administrators. In particular, only an administrator can increase the TimeLimit of a job.

== How to show accounting data of completed jobs ==

Use [https://slurm.schedmd.com/sacct.html sacct] command.

'sacct -e' command shows a list of fields that can be
specified with the '--format' option.

== How to retrieve job history and accounting ==

For a specific job:

<pre>
$ sacct -j <jobid> --format=User,JobID,Jobname,partition,state,time,start,end,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist
</pre>

For a specific user:

<pre>
$ sacct -u <user> --format=User,JobID,Jobname,partition,state,time,start,end,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist
</pre>

Note: Default time window is the current day.

Starting from a specific date:

<pre>
$ sacct -u <user> -S 2020-01-15 --format=User,JobID,Jobname,partition,state,time,start,end,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist
</pre>

Within a time window:

<pre>
$ sacct -u <user> -S 2020-01-15 -E 2020-01-31 --format=User,JobID,Jobname,partition,state,time,start,end,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist
</pre>

'''Note:'''

You can also set the environment variable $SACCT_FORMAT to specify the default format. To get a general idea of how efficiently a job utilized its resources, the following format can be used:

<pre>
export SACCT_FORMAT="JobID,JobName,Elapsed,NCPUs,TotalCPU,CPUTime,ReqMem,MaxRSS,MaxDiskRead,MaxDiskWrite,State,ExitCode"
</pre>

To find how efficiently the CPUs were used, divide TotalCPU by CPUTime. To find how efficiently memory were used, devide MaxRSS by ReqMem. But be aware that sacct memory usage measurement doesn't catch very rapid memory spikes. If your job got killed for running out of memory, it '''did run out of memory''' even if sacct reports a lower memory usage than would trigger an out-of-memory-kill. A job that reads or writes excessively to disk might be bogged down significantly by I/O operations.

== How to get efficiency information of completed job(s)? ==

Use <pre>$ seff <jobid> </pre> command for some brief information.

'''Note:''' It is good practice to have a look at the efficiency of your job(s) on completion '''and we expect you to do so'''. This way you can improve your job specifications in the future.

== How to get complete field values from sstat and sacct commands? ==

When using the [https://slurm.schedmd.com/sacct.html#OPT_format --format] option for listing various fields you can put a %NUMBER afterwards to specify how many characters should be printed.

E.g. '--format=User%30' will print 30 characters for the user name (right justified). A %-30 will print 30 characters left justified.

sstat and sacct also provide the '--parsable' and '--parsable2' option to always print full field values delimited with a pipe '|' character by default.
The delimiting character can be specified by using the '--delimiter' option, e.g. '--delimiter=","' for comma separated values.

== How to retrieve job records for all jobs running/pending at a certain point in time ==

Use [https://slurm.schedmd.com/sacct.html sacct] with [https://slurm.schedmd.com/sacct.html#OPT_state -s <state>] and [https://slurm.schedmd.com/sacct.html#OPT_starttime -S <start time>] options, e.g.:

<pre>
$ sacct -n -a -X -S 2021-04-01T00:00:00 -s R -o JobID,User%15,Account%10,NCPUS,NNodes,NodeList%1500
</pre>

'''Note:''' When specifying the state "-s <state>" '''and''' the start time "-S <start time>", the default
time window will be set to end time "-E" equal to start time. Thus, you will get a snapshot of all running/pending
jobs at the instance given by "-S <start time>".

== How to get a parsable list of hostnames from $SLURM_JOB_NODELIST? ==

<pre>
$ scontrol show hostnames $SLURM_JOB_NODELIST
</pre>

= ADMINISTRATION =

'''Note:''' Most commands in this section are restricted to system administrators.

== How to stop Slurm from scheduling jobs? ==

You can stop Slurm from scheduling jobs on a per partition basis by
setting that partition's state to DOWN. Set its state UP to resume
scheduling. For example:

<pre>
$ scontrol update PartitionName=foo State=DOWN
$ scontrol update PartitionName=foo State=UP
</pre>

== How to print actual hardware configuration of a node? ==

<pre>
$ slurmd -C # print hardware configuration plus uptime;
$ slurmd -G # print generic resource configuration; to be run on compute nodes only
</pre>

== How to reboot (all) nodes as soon as they become idle? ==

<pre>
$ scontrol reboot ASAP nextstate=RESUME <node1>,<node2> # specific nodes
$ scontrol reboot ASAP nextstate=RESUME ALL # all nodes
</pre>

== How to cancel pending reboot of nodes? ==

<pre>
$ scontrol cancel_reboot <node1>,<node2>
</pre>

== How to check current node status? ==

<pre>
$ scontrol show node <node>
</pre>

== How to instruct all Slurm daemons to re-read the configuration file ==

<pre>
$ scontrol reconfigure
</pre>

== How to prevent a user from submitting new jobs? ==

Use the following [https://slurm.schedmd.com/sacctmgr.html sacctmgr] command:

<pre>
$ sacctmgr update user <username> set maxsubmitjobs=0
</pre>

'''Notes:'''

*Job submission is then rejected with the following message:
<pre>
$ sbatch job.slurm
sbatch: error: AssocMaxSubmitJobLimit
sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)
</pre>

* Use the following command to release the limit:
<pre>
$ sacctmgr update user <username> set maxsubmitjobs=-1
</pre>

== How to drain node(s)? ==

<pre>
$ scontrol update NodeName=<node1>,<node2> State=DRAIN Reason="Some Reason"
</pre>

Notes:

* Reason is mandatory.

* Do '''not''' just set state DOWN to drain nodes. This will kill any active jobs that may run on that nodes.

== How to list reason for nodes being drained or down? ==

<pre>
$ sinfo -R
</pre>

== How to resume node state? ==

<pre>
$ scontrol update NodeName=<node1>,<node2> State=RESUME
</pre>

== How to create a reservation on nodes? ==

Suggested reading: https://slurm.schedmd.com/reservations.html

<pre>
$ scontrol create reservation user=root starttime=now duration=UNLIMITED flags=maint,ignore_jobs nodes=ALL
$ scontrol create reservation user=root starttime=2020-12-24T17:00 duration=12:00:00 flags=maint,ignore_jobs nodes=<node1>,<node2>
$ scontrol show reservation
</pre>

'''Note:''' Add "FLEX" flag to allow jobs that qualify for the reservation to start before the reservation begins (and continue after it starts).
Add "MAGNETIC" flag to attract jobs that qualify for the reservation to run in that reservation without having requested it at submit time.

== How to create a floating reservation on nodes? ==

Use the flag "TIME_FLOAT" and a start time that is relative to the current time (use the keyword "now").
In the example below, the nodes are prevented from starting any jobs exceeding a walltime of 2 days.

<pre>
$ scontrol create reservation user=root starttime=now+2days duration=UNLIMITED flags=maint,ignore_jobs,time_float nodes=<node1>,<node2>
</pre>

'''Note:''' Floating reservation are not intended to run jobs, but to prevent long running jobs from being initiated on specific nodes. Attempts by users to make use of a floating reservation will be rejected. When ready to perform the maintenance, place the nodes in DRAIN state and delete the reservation.

== How to use a reservation? ==

<pre>
$ sbatch --reservation=foo_6 ... script.slurm
</pre>

== How to delete a reservation? ==

<pre>
$ scontrol delete ReservationName=foo_6
</pre>

== How to get node oriented information similar to 'mdiag -n'? ==

<pre>
$ sinfo -N -l
</pre>

Fields can be individually customized. See [https://slurm.schedmd.com/sinfo.html sinfo] man page. For example:

<pre>
$ sinfo -N --format="%8N %12P %.4C %.8O %.6m %.6e %.8T %.20E"

NODELIST PARTITION CPUS CPU_LOAD MEMORY FREE_M STATE REASON
n0001 standard* 0/16 0.01 128000 120445 idle none
n0002 standard* 0/16 0.01 128000 120438 idle none
n0003 standard* 0/0/ N/A 128000 N/A down* Not responding
</pre>

== How to get node oriented information similar to 'pbsnodes'? ==

<pre>
$ scontrol show nodes # One paragraph per node (all nodes)
$ scontrol show nodes <node1>,<node2> # One paragraph per node (specified nodes)
$ scontrol -o show nodes # One line per node (all nodes)
$ scontrol -o show nodes <node1>,<node2> # One line per node (specified nodes)
</pre>

== How to update multiple jobs of a user with a single scontrol command? ==

Not possible. But you can e.g. use squeue to build the script taking
advantage of its filtering and formatting options.

<pre>
$ squeue -tpd -h -o "scontrol update jobid=%i priority=1000" >my.script
</pre>

You can also identify the list of jobs and add them to the JobID all at once, for example:

<pre>
$ scontrol update JobID=123 qos=reallylargeqos
$ scontrol update JobID=123,456,789 qos=reallylargeqos
$ scontrol update JobID=[123-400],[500-600] qos=reallylargeqos
</pre>

Another option is to use the JobName, if all the jobs have the same name.

<pre>
$ scontrol update JobName="foobar" UserID=johndoe qos=reallylargeqos
</pre>

However, Slurm does not allow the UserID filter alone.

== How to create a new account? ==

Add account at top level in association tree:

<pre>
$ sacctmgr add account <accountname> Cluster=justus Description="Account description" Organization="none"
</pre>

Add account as child of some parent account in association tree:

<pre>
$ sacctmgr add account <accountname> parent=<parent_accountname>
</pre>

== How to move account to another parent? ==

<pre>
$ sacctmgr modify account name=<accountname> set parent=<new_parent_accountname>
</pre>

== How to delete an account? ==

<pre>
$ sacctmgr delete account name=<accountname>
</pre>

== How to add a new user? ==

<pre>
$ sacctmgr add user <username> DefaultAccount=<accountname>
</pre>

== How to add/remove users from an account? ==

<pre>
$ sacctmgr add user <username> account=<accountname> # Add user to account
$ sacctmgr add user <username> account=<accountname2> # Add user to a second account
$ sacctmgr remove user <username> where account=<accountname> # Remove user from this account
</pre>

== How to change default account of a user? ==

<pre>
$ sacctmgr modify user where user=<username> set DefaultAccount=<default_account>
</pre>

'''Note:''' The user must already be associated with the account you want to set as default.

== How to show account information? ==

<pre>
$ sacctmgr show assoc
$ sacctmgr show assoc tree
</pre>

'''Note:'''
* On JUSTUS 2 registered compute projects ("Rechenvorhaben") are uniquely mapped to Slurm accounts of the same name.

== How to implement user resource throttling policies? ==

Quoting from https://bugs.schedmd.com/show_bug.cgi?id=3600#c4

With Slurm, the associations are meant to establish base limits on the
defined partitions, accounts and users. Because limits propagate down
through the association tree, you only need to define limits at a high
level and those limits will be applied to all partitions, accounts and
users that are below it (parent to child). You can also override those
high level (parent) limits by explicitly setting different limits at
any lower level (on the child). So using the association tree is the
best way to get some base limits applied that you want for most cases.
QOS's are meant to override any of those base limits for exceptional
cases. Like Maui, you can use QOS's to set a different priority.
Again, the QOS would be overriding the base priority that could be set
in the associations.

== How to set a resource limit for an individual user? ==

Suggested reading: https://slurm.schedmd.com/resource_limits.html

Example:

<pre>
$ sacctmgr modify user <username> set maxjobs=1 # Limit maximum number of running jobs for user
$ sacctmgr list assoc user=<username> format=user,maxjobs # Show that limit
$ sacctmgr modify user <username> set maxjobs=-1 # Remove that limit
</pre>

== How to retrieve historical resource usage for a specific user or account ==

Use [https://slurm.schedmd.com/sreport.html sreport] command.

Examples:
<pre>
$ sreport cluster UserUtilizationByAccount Start=2021-01-01 End=2021-12-31 -t Hours user=<username> # Report cluster utilization of given user broken down by accounts
$ sreport cluster AccountUtilizationByUser Start=2021-01-01 End=2021-12-31 -t Hours account=<account> # Report cluster utilization of given account broken down by users
</pre>

'''Notes:'''
* By default CPU resources will be reported. Use '-T' option for other trackable resources, e.g. '-T cpu,mem,gres/gpu,gres/scratch'.
* On JUSTUS 2 registered compute projects ("Rechenvorhaben") are uniquely mapped to Slurm accounts of the same name. Thus, 'AccountUtilizationByUser' can also be used to report the aggregated cluster utilization of compute projects.
* Can be executed by regular users as well in which case Slurm will only report their own usage records (but along with the total usage of the associated account in the case of 'AccountUtilizationByUser').

== How to fix/reset a user's RawUsage value? ==

<pre>
$ sacctmgr modify user <username> where Account=<account> set RawUsage=<number>
</pre>

== How to create/modify/delete QOSes? ==

Suggested reading: https://slurm.schedmd.com/qos.html

Examples:

<pre>
$ sacctmgr show qos # Show existing QOSes
$ sacctmgr add qos verylong # Create new QOS verylong
$ sacctmgr modify qos verylong set MaxWall=28-00:00:00 # Set maximum walltime limit
$ sacctmgr modify qos verylong set MaxTRESPerUser=cpu=4 # Set maximum maximum number of CPUS a user can allocate at a given time
$ sacctmgr modify qos verylong set flags=denyonlimit # Prevent submission if job requests exceed any limits of QOS
$ sacctmgr modify user <username> set qos+=verylong # Add a QOS to a user account
$ sacctmgr modify user <username> set qos-=verylong # Remove a QOS from a user account
$ sacctmgr delete qos verylong # Delete that QOS
</pre>

== How to find (and fix) runaway jobs? ==

<pre>$ sacctmgr show runaway</pre>

'''Notes:'''
* Runaway jobs are orphaned jobs that don't exist in the Slurm controller but have a start and no end time in the Slurm data base. Runaway jobs mess with accounting and affects new jobs of users who have too many runaway jobs.
* If there are jobs in this state this command will also provide an option to fix them. This will set the end time for each job to the latest out of the start, eligible, or submit times, and set the state to completed.

== How to show a history of database transactions? ==

<pre>$ sacctmgr list transactions</pre>

'''Note:''' Useful to get timestamps for when a user/account/qos has been created/modified/removed etc.

BwForCluster JUSTUS 2 Slurm HOWTO

2026-09-17T12:24:00Z

K Siegmund: /* How to retrieve job history and accounting? */

{{Justus2}}

This is a collection of howtos and convenient Slurm commands for JUSTUS 2.

Some commands behave slightly different depending on whether they are executed
by a system administrator or by a regular user, as Slurm prevents regular users from accessing critical system information and viewing job and usage information of other users.

= GENERAL INFORMATION =

== How to find a general quick start user guide? ==

https://slurm.schedmd.com/quickstart.html

== How to find Slurm FAQ? ==

https://slurm.schedmd.com/faq.html

== How to find a Slurm cheat sheet? ==

https://slurm.schedmd.com/pdfs/summary.pdf

== How to find Slurm tutorials? ==

https://slurm.schedmd.com/tutorials.html

== How to get more information on Slurm? ==

(Almost) every Slurm command has a man page. Use it.

Online versions: https://slurm.schedmd.com/man_index.html

== How to find hardware specific details about JUSTUS 2? ==

See our Wiki page: [[Hardware and Architecture (bwForCluster JUSTUS 2)|Hardware and Architecture]]

= JOB SUBMISSION =

== How to submit a serial batch job? ==

Use [https://slurm.schedmd.com/sbatch.html sbatch] command:

<pre>$ sbatch <job-script> </pre>

Sample job script template for serial job:

<source lang="bash">
#!/bin/bash
# Allocate one node
#SBATCH --nodes=1
# Number of program instances to be executed
#SBATCH --ntasks-per-node=1
# 8 GB memory required per node
#SBATCH --mem=8G
# Maximum run time of job
#SBATCH --time=1:00:00
# Give job a reasonable name
#SBATCH --job-name=serial_job
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=serial_job-%j.out
# File name for error output
#SBATCH --error=serial_job-%j.err

# Load software modules as needed, e.g.
# module load foo/bar

# Run serial program
./my_serial_program
</source>

Sample code for serial program: [[Media:Hello_serial.c | Hello_serial.c]]

'''Notes:'''
* --nodes=1 and --ntasks-per-node=1 may be replaced by --ntasks=1.
* If not specified, stdout and stderr are both written to slurm-%j.out.

== How to find working sample scripts for my program? ==

Most software modules for applications provide working sample batch scripts.
Check with [[Software_Modules_Lmod#Module_specific_help | module help]] command, e.g.
<pre>
$ module help chem/vasp # display module help for VASP
$ module help math/matlab # display module help for Matlab
</pre>

== How to harden job scripts against common errors? ==

The bash shell provides several options that support users in disclosing hidden bugs and writing safer job scripts.
In order to activate these safeguard settings users can insert the following lines in their scripts (after all #SBATCH directives):

<source lang="bash">
[...]
set -o errexit # (or set -e) cause batch script to exit immediately when a command fails.
set -o pipefail # cause batch script to exit immediately also when the command that failed is embedded in a pipeline
set -o nounset # (or set -u) causes the script to treat unset variables as an error and exit immediately
[...]
</source>

== How to submit an interactive job? ==

Use [https://slurm.schedmd.com/salloc.html salloc] command, e.g.:
<pre>$ salloc --nodes=1 --ntasks-per-node=8</pre>

'''Note:'''

In previous Slurm versions < 20.11 the use of [https://slurm.schedmd.com/srun.html srun] has been the recommended way for launching interactive jobs, e.g.:

<pre>$ srun --nodes=1 --ntasks-per-node=8 --pty bash </pre>

Although this still works with current Slurm versions this is considered '''deprecated ''' for current Slurm versions as it may cause issues when launching additional jobs steps from within the interactive job environment. Use [https://slurm.schedmd.com/salloc.html salloc] command.

== How to enable X11 forwarding for an interactive job? ==

Use '--x11' flag, e.g.

<pre>
$ salloc --nodes=1 --ntasks-per-node=8 --x11 # run shell with X11 forwarding enabled
</pre>

'''Note:'''
* For X11 forwarding to work, you must also enable X11 forwarding for your ssh login from your local computer to the cluster, i.e.:
<pre>local> ssh -X <username>@justus2.uni-ulm.de></pre>

== How to convert Moab batch job scripts to Slurm? ==

Replace Moab/Torque job specification flags and environment variables in your job
scripts by their corresponding Slurm counterparts.

'''Commonly used Moab job specification flags and their Slurm equivalents'''

{| width=750px class="wikitable"
! Option !! Moab (msub) !! Slurm (sbatch)
|-
| Script directive || #MSUB || #SBATCH
|-
| Job name || -N <name> || --job-name=<name> (-J <name>)
|-
| Account || -A <account> || --account=<account> (-A <account>)
|-
| Queue || -q <queue> || --partition=<partition> (-p <partition>)
|-
| Wall time limit || -l walltime=<hh:mm:ss> || --time=<hh:mm:ss> (-t <hh:mm:ss>)
|-
| Node count || -l nodes=<count> || --nodes=<count> (-N <count>)
|-
| Core count || -l procs=<count> || --ntasks=<count> (-n <count>)
|-
| Process count per node || -l ppn=<count> || --ntasks-per-node=<count>
|-
| Core count per process || || --cpus-per-task=<count> (-c <count>)
|-
| Memory limit per node || -l mem=<limit> || --mem=<limit>
|-
| Memory limit per process || -l pmem=<limit> || --mem-per-cpu=<limit>
|-
| Job array || -t <array indices> || --array=<indices> (-a <indices>)
|-
| Node exclusive job || -l naccesspolicy=singlejob || --exclusive
|-
| Initial working directory || -d <directory> (default: $HOME) || --chdir=<directory> (-D <directory>) (default: submission directory)
|-
| Standard output file || -o <file path> || --output=<file> (-o <file>)
|-
| Standard error file || -e <file path> || --error=<file> (-e <file>)
|-
| Combine stdout/stderr to stdout || -j oe || --output=<combined stdout/stderr file>
|-
| Mail notification events || -m <event> || --mail-type=<events> (valid types include: NONE, BEGIN, END, FAIL, ALL)
|-
| Export environment to job || -V || --export=ALL (default)
|-
| Don't export environment to job || (default) || --export=NONE
|-
| Export environment variables to job || -v <var[=value][,var2=value2[, ...]]> || --export=<var[=value][,var2=value2[,...]]>
|}

'''Notes:'''
* Default initial job working directory is $HOME for Moab. For Slurm the default working directory is where you submit your job from.
* By default Moab does not export any environment variables to the job's runtime environment. With Slurm most of the login environment variables are exported to your job's runtime environment. This includes environment variables from software modules that were loaded at job submission time (and also $HOSTNAME variable).

'''Commonly used Moab/Torque script environment variables and their Slurm equivalents

{| width=750px class="wikitable"
! Information !! Moab !! Torque !! Slurm
|-
| Job name || $MOAB_JOBNAME || $PBS_JOBNAME || $SLURM_JOB_NAME
|-
| Job ID || $MOAB_JOBID || $PBS_JOBID || $SLURM_JOB_ID
|-
| Submit directory || $MOAB_SUBMITDIR || $PBS_O_WORKDIR || $SLURM_SUBMIT_DIR
|-
| Number of nodes allocated || $MOAB_NODECOUNT || $PBS_NUM_NODES || $SLURM_JOB_NUM_NODES (and: $SLURM_NNODES)
|-
| Node list || $MOAB_NODELIST || cat $PBS_NODEFILE || $SLURM_JOB_NODELIST
|-
| Number of processes || $MOAB_PROCCOUNT || $PBS_TASKNUM || $SLURM_NTASKS
|-
| Requested tasks per node || --- || $PBS_NUM_PPN || $SLURM_NTASKS_PER_NODE
|-
| Requested CPUs per task || --- || --- || $SLURM_CPUS_PER_TASK
|-
| Job array index || $MOAB_JOBARRAYINDEX || $PBS_ARRAY_INDEX || $SLURM_ARRAY_TASK_ID
|-
| Job array range || $MOAB_JOBARRAYRANGE || - || $SLURM_ARRAY_TASK_COUNT
|-
| Queue name || $MOAB_CLASS || $PBS_QUEUE || $SLURM_JOB_PARTITION
|-
| QOS name || $MOAB_QOS || --- || $SLURM_JOB_QOS
|-
| Number of processes per node | --- || $PBS_NUM_PPN || $SLURM_TASKS_PER_NODE
|-
| Job user || $MOAB_USER || $PBS_O_LOGNAME || $SLURM_JOB_USER
|-
| Hostname || $MOAB_MACHINE || $PBS_O_HOST || $SLURMD_NODENAME
|}

'''Note:'''

* See [https://slurm.schedmd.com/sbatch.html sbatch] man page for a complete list of flags and environment variables.

== How to emulate Moab output file names? ==

Use the following directives:

<pre>
#SBATCH --output="%x.o%j"
#SBATCH --error="%x.e%j"
</pre>

== How to pass command line arguments to the job script? ==

Run <pre>$ sbatch <job-script> arg1 arg2 ...</pre>

Inside the job script the arguments can be accessed as $1, $2, ...

E.g.:

<source lang="bash">
[...]
infile="$1"
outfile="$2"
./my_serial_program < "$infile" > "$outfile" 2>&1
[...]
</source>

'''Notes:'''

* Do '''not''' use $1, $2, ... in "#SBATCH" lines. These parameters can be used only within the regular shell script.

== How to request local scratch (SSD/NVMe) at job submission? ==

Use '--gres=scratch:nnn' option to allocate nnn GB of local (i.e. node-local) scratch space for the entire job.

Example: '--gres=scratch:100' will allocate 100 GB scratch space on a locally attached NVMe device.

'''Notes:'''

* Do '''not''' add any unit (such as --gres=scratch:100G). This would be treated as requesting an amount of 10^9 * 100GB of scratch space.

* Multinode jobs get nnn GB of local scratch space on every node of the job.

* Environment variable '''$SCRATCH''' will point to
** /scratch/<user>.<jobid> when local scratch has been requested. This will be on locally attached SSD/NVMe devices.
** /tmp/<user>.<jobid> when no local scratch has been requested. This will be in memory and, thus, be limited in size.

* Environment variable '''$TMPDIR''' always points to /tmp/<user>.<jobid>. This will always be in memory and, thus, limited in size.

* For backward compatibility environment variable $RAMDISK always points to /tmp/<user>.<jobid>

* Scratch space allocation in /scratch will be enforced by quota limits

* Data written to $TMPDIR will always count against allocated memory.

* Data written to local scratch space will automatically be removed at the end of the job.

== How to request GPGPU nodes at job submission? ==

Use '--gres=gpu:<count>' option to allocate 1 or 2 GPUs per node for the entire job.

Example: '--gres=gpu:1' will allocate one GPU per node for this job.

'''Notes:'''

* GPGPU nodes are equipped with two Nvidia V100S cards

* Environment variables $CUDA_VISIBLE_DEVICES, $SLURM_JOB_GPUS and $GPU_DEVICE_ORDINAL will denote card(s) allocated for the job.

* CUDA Toolkit is available as software module devel/cuda.

== How to clean-up or save files before a job times out? ==

Possibly you would like to clean up the work directory or save intermediate result files in case a job times out.

The following sample script may serve as a blueprint for implementing a pre-termination function to perform clean-up or file recovery actions.

<source lang="bash">
#!/bin/bash
# Allocate one node
#SBATCH --nodes=1
# Number of program instances to be executed
#SBATCH --ntasks-per-node=1
# 2 GB memory required per node
#SBATCH --mem=2G
# Request 10 GB local scratch space
#SBATCH --gres=scratch:10
# Maximum run time of job
#SBATCH --time=10:00
# Send the USR1 signal 120 seconds before end of time limit
#SBATCH --signal=B:USR1@120
# Give job a reasonable name
#SBATCH --job-name=signal_job
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=signal_job-%j.out
# File name for error output
#SBATCH --error=signal_job-%j.err

# Define the signal handler function
# Note: This is not executed here, but rather when the associated
# signal is received by the shell.
finalize_job()
{
# Do whatever cleanup you want here. In this example we copy
# output file(s) back to $SLURM_SUBMIT_DIR, but you may implement
# your own job finalization code here.
echo "function finalize_job called at `date`"
cd $SCRATCH
mkdir -vp "$SLURM_SUBMIT_DIR"/results
tar czvf "$SLURM_SUBMIT_DIR"/results/${SLURM_JOB_ID}.tgz output*.txt
exit
}

# Call finalize_job function as soon as we receive USR1 signal
trap 'finalize_job' USR1

# Copy input files for this job to the scratch directory (if needed).
# Note: Environment variable $SCRATCH always points to a scratch directory
# automatically created for this job. Environment variable $SLURM_SUBMIT_DIR
# points to the path where this script was submitted from.
# Example:
# cp -v "$SLURM_SUBMIT_DIR"/input*.txt "$SCRATCH"

# Change working directory to local scratch directory
cd "$SCRATCH"

# Load software modules as needed, e.g.
# module load foo/bar

# This is where the actual work is done. In this case we just create
# a sample output file for 900 (=15*60) seconds, but since we asked
# Slurm for 600 seconds only it will not be able finish within this
# wall time.
# Note: It is important to run this task in the background
# by placing the & symbol at the end. Otherwise the signal handler
# would not be executed until that process has finished, which is not
# what we want.
(for i in `seq 15`; do echo "Hello World at `date +%H:%M:%S`."; sleep 60; done) >output.txt 2>&1 &

# Note: The command above is just for illustration. Normally you would just run
# my_program >output.txt 2>&1 &

# Tell the shell to wait for background task(s) to finish.
# Note: This is important because otherwise the parent shell
# (this script) would proceed (and terminate) without waiting for
# background task(s) to finish.
wait

# If we get here, the job did not time out but finished in time.

# Release user defined signal handler for USR1
trap - USR1

# Do regular cleanup and save files. In this example we simply call
# the same function that we defined as a signal handler above, but you
# may implement your own code here.
finalize_job

exit
</source>

'''Notes:'''

* The number of seconds specified in --signal option must match the runtime of the pre-termination function and must not exceed 65535 seconds.

* Due to the resolution of event handling by Slurm, the signal may be sent a little earlier than specified.

== How to submit a multithreaded batch job? ==

Sample job script template for a job running one multithreaded program instance:

<source lang="bash">
#!/bin/bash
# Allocate one node
#SBATCH --nodes=1
# Number of program instances to be executed
#SBATCH --ntasks-per-node=1
# Number of cores per program instance
#SBATCH --cpus-per-task=8
# 8 GB memory required per node
#SBATCH --mem=8G
# Maximum run time of job
#SBATCH --time=1:00:00
# Give job a reasonable name
#SBATCH --job-name=multithreaded_job
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=multithreaded_job-%j.out
# File name for error output
#SBATCH --error=multithreaded_job-%j.err

# Load software modules as needed, e.g.
# module load foo/bar

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
export MKL_NUM_THREADS=${SLURM_CPUS_PER_TASK}

# Run multithreaded program
./my_multithreaded_program
</source>

Sample code for multithreaded program: [[Media:Hello_openmp.c | Hello_openmp.c]]

'''Notes:'''

* In our configuration each physical core is considered a "CPU".

* On JUSTUS 2 it is recommended to specify a number of cores per task ('--cpus-per-task') that is either an integer divisor of 24 or (at most) 48.

* Required memory can also by specified per allocated CPU with '--mem-per-cpu' option.

* The '--mem' and '--mem-per-cpu' options are mutually exclusive.

== How to submit an array job? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_array -a] (or [https://slurm.schedmd.com/sbatch.html#OPT_array --array]) option, e.g.

<pre>$ sbatch -a 1-16%8 ...</pre>

This will submit 16 tasks to be executed, each one indexed by SLURM_ARRAY_TASK_ID ranging from 1 to 16, but will limit the number of simultaneously running tasks from this job array to 8.

Sample job script template for an array job:

<source lang="bash">
#!/bin/bash
# Number of cores per individual array task
#SBATCH --ntasks=1
#SBATCH --array=1-16%8
#SBATCH --mem=4G
#SBATCH --time=01:00:00
#SBATCH --job-name=array_job
#SBATCH --output=array_job-%A_%a.out
#SBATCH --error=array_job-%A_%a.err

# Load software modules as needed, e.g.
# module load foo/bar

# Print the task id.
echo "My SLURM_ARRAY_TASK_ID: " $SLURM_ARRAY_TASK_ID

# Add lines here to run your computations, e.g.
# ./my_program <input.$SLURM_ARRAY_TASK_ID

</source>

'''Notes:'''

* Placeholder %A will be replaced by the master job id, %a will be replaced by the array task id.

* Every sub job in an array job will have its own unique environment variable $SLURM_JOB_ID. Environment variable $SLURM_ARRAY_JOB_ID will be set to the first job array index value for all tasks.

* The remaining options in the sample job script are the same as the options used in other, non-array jobs. In the example above, we are requesting that each array task be allocated 1 CPU (--ntasks=1) and 4 GB of memory (--mem=4G) for up to one hour (--time=01:00:00).

* More information: https://slurm.schedmd.com/job_array.html

== How to delay the start of a job? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_begin -b] (or [https://slurm.schedmd.com/sbatch.html#OPT_begin --begin]) option in order to defer the allocation of the job until the specified time.

Examples:

<pre>
sbatch --begin=20:00 ... # job can start after 8 p.m.
sbatch --begin=now+1hour ... # job can start 1 hour after submission
sbatch --begin=teatime ... # job can start at teatime (4 p.m.)
sbatch --begin=2023-12-24T20:00:00 ... # job can start after specified date/time
</pre>

== How to submit dependency (chain) jobs? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_dependency -d] (or [https://slurm.schedmd.com/sbatch.html#OPT_dependency --dependency]) option, e.g.

<pre>$ sbatch -d afterany:123456 ...</pre>

This will defer the submitted job until the specified job 123456 has terminated.

Slurm supports a number of different dependency types, e.g.:

<pre>
-d after:123456 # job can begin execution after the specified job has begun execution
-d afterany:123456 # job can begin execution after the specified job has finished
-d afternotok:123456 # job can begin execution after the specified job has failed (exit code not equal zero)
-d afterok:123456 # job can begin execution after the specified job has successfully finished (exit code zero)
-d singleton # job can begin execution after any previously job with the same job name and user have finished
</pre>

'''Note:''' Multiple jobs can be specified by separating their job ids by colon characters (:), e.g.

<pre> $ sbatch -d afterany:123456:123457 ... </pre>

This will defer the submitted job until the specified jobs 123456 and 123457 have both finished.

== How to deal with invalid job dependencies? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_kill-on-invalid-dep --kill-on-invalid-dep=yes] option in order to automatically terminate jobs which can never run due to invalid dependencies. By default the job stays pending with reason 'DependencyNeverSatisfied' to allow review and appropriate action by the user.

'''Note:''' A job dependency may also become invalid if a job has been submitted with '-d afterok:<jobid>' but the specified dependency job has failed, e.g. because it timed out (i.e. exceeded its wall time limit).

== How to submit an MPI batch job? ==

Suggested reading: https://slurm.schedmd.com/mpi_guide.html

Sample job script template for an MPI job:

<source lang="bash">
#!/bin/bash
# Allocate two nodes
#SBATCH --nodes=2
# Number of program instances to be executed
#SBATCH --ntasks-per-node=48
# Allocate 32 GB memory per node
#SBATCH --mem=32gb
# Maximum run time of job
#SBATCH --time=1:00:00
# Give job a reasonable name
#SBATCH --job-name=mpi_job
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=mpi_job-%j.out
# File name for error output
#SBATCH --error=mpi_job-%j.err

# Add lines here to run your computations, e.g.
#
# Option 1: Lauch MPI tasks by using mpirun
#
# for OpenMPI and GNU compiler:
#
# module load compiler/gnu
# module load mpi/openmpi
# mpirun ./my_mpi_program
#
# for Intel MPI and Intel complier:
#
# module load compiler/intel
# module load mpi/impi
# mpirun ./my_mpi_program
#
# Option 2: Launch MPI tasks by using srun
#
# for OpenMPI and GNU compiler:
#
# module load compiler/gnu
# module load mpi/openmpi
# srun ./my_mpi_program
#
# for Intel MPI and Intel compiler:
#
module load compiler/intel
module load mpi/impi
srun ./my_mpi_program

</source>

Sample code for MPI program: [[Media:Hello_mpi.c | Hello_mpi.c]]

'''Notes'''

* SchedMD recommends to use srun and many (most?) sites do so as well. The rationale is that srun is more tightly integrated with the scheduler and provides more consistent and reliable resource tracking and accounting for individual jobs and job steps. mpirun may behave differently for different MPI implementations and versions. There are reports that claim "strange behavior" of mpirun especially when using task affinity and core binding. Using srun is supposed to resolve these issues and is therefore highly recommended.
* Do not run batch jobs that launch a large number (hundreds or thousands) short running (few minutes or less) MPI programs, e.g. from a shell loop. Every single MPI invocation does generate its own job step and sends remote procedure calls to the Slurm controller server. This can result in degradation of performance for both, Slurm and the application, especially if many of that jobs happen to run at the same time. Jobs of that kind can even get stuck without showing any further activity until hitting the wall time limit. For high throughput computing (e.g. processing a large number of files with every single task running independently from each other and very shortly), consider a more appropriate parallelization paradigm that invokes independent serial (non-MPI) processes in parallel at the same time. This approach is sometimes referred to as "[https://en.wikipedia.org/wiki/Embarrassingly_parallel pleasingly parallel]" workload. GNU Parallel is a shell tool that facilitates executing serial tasks in parallel. On JUSTUS 2 this tool is available as a software module "system/parallel".

== How to submit a hybrid MPI/OpenMP job? ==

Sample job script template for an hybrid job:

<source lang="bash">
#!/bin/bash
# Number of nodes to allocate
#SBATCH --nodes=4
# Number of MPI instances (ranks) to be executed per node
#SBATCH --ntasks-per-node=2
# Number of threads per MPI instance
#SBATCH --cpus-per-task=24
# Allocate 8 GB memory per node
#SBATCH --mem=8gb
# Maximum run time of job
#SBATCH --time=1:00:00
# Give job a reasonable name
#SBATCH --job-name=hybrid_job
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=hybrid_job-%j.out
# File name for error output
#SBATCH --error=hybrid_job-%j.err

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
export MKL_NUM_THREADS=${SLURM_CPUS_PER_TASK}

module load compiler/intel
module load mpi/impi
srun ./my_hybrid_program

# or:
# mpirun ./my_hybrid_program

</source>

Sample code for hybrid program: [[Media:Hello_hybrid.c | Hello_hybrid.c]]

'''Notes:'''

* $SLURM_CPUS_PER_TASK is only set if the '--cpus-per-task' option is specified.

== How to request specific node(s) at job submission? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_nodelist -w] (or [https://slurm.schedmd.com/sbatch.html#OPT_nodelist --nodelist]) option, e.g.:

<pre>$ sbatch -w <node1>,<node2> ...</pre>

Also see [https://slurm.schedmd.com/sbatch.html#OPT_nodefile -F] (or [https://slurm.schedmd.com/sbatch.html#OPT_nodefile --nodefile]) option.

== How to exclude specific nodes from job? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_exclude -x] (or [https://slurm.schedmd.com/sbatch.html#OPT_exclude --exclude]) option, e.g.:

<pre>$ sbatch -x <node1>,<node2> ...</pre>

== How to get exclusive jobs? ==

Use '--exclusive' option on job submission. This makes sure that there will be no other jobs running on your nodes. Very useful for benchmarking!

'''Note:'''

* --exclusive option does '''not''' mean that you automatically get full access to all the resources which the node might provide without explicitly requesting them.

== How to avoid sharing nodes with other users? ==

Use '--exclusive=user' option on job submission. This will still allow multiple jobs of one and the same user on the nodes.

''' Note:'''

* Depending on configuration, exclusive=user may (and probably will) be the default node access policy on JUSTUS 2.

== How to submit batch job without job script? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_wrap --wrap] option.

Example:

<pre>$ sbatch --nodes=2 --ntasks-per-node=16 --wrap "sleep 600"</pre>

'''Note:''' May be useful for testing purposes.

= JOB MONITORING AND CONTROL =

== How to prevent Slurm performance degradation? ==

Almost every invocation of a Slurm client command (e.g. squeue, sacct, sprio or sshare) sends a remote procedure call (RPC) to the Slurm control daemon and/or database.
If enough remote procedure calls come in at once, this can result in a degradation of performance of the Slurm services for all users, possibly resulting in a denial of service.

Therefore, '''do not run Slurm client commands that send remote procedure calls from loops in shell scripts or other programs''' (such as 'watch squeue'). Always ensure to limit calls to squeue, sstat, sacct etc. to the minimum necessary for the information you are trying to gather.

Slurm does collect RPC counts and timing statistics by message type and user for diagnostic purposes.

== How to view information about submitted jobs? ==

Use [https://slurm.schedmd.com/squeue.html squeue] command, e.g.:

<pre>
$ squeue # all jobs owned by user (all jobs owned by all users for admins)
$ squeue --me # all jobs owned by user (same as squeue for regular users)
$ squeue -u <username> # jobs of specific user
$ squeue -t PENDING # pending jobs only
$ squeue -t RUNNING # running jobs only
</pre>

'''Notes:'''

* The output format of [https://slurm.schedmd.com/squeue.html squeue] (and most other Slurm commands) is highly configurable to your needs. Look for the --format or --Format options.

* Every invocation of squeue sends a remote procedure call to the Slurm database server. '''Do not run squeue or other Slurm client commands from loops in shell scripts or other programs''' as this can result in a degradation of performance. Ensure that programs limit calls to squeue to the minimum necessary for the information you are trying to gather.

== How to cancel jobs? ==

Use [https://slurm.schedmd.com/scancel.html scancel] command, e.g.

<pre>
$ scancel <jobid> # cancel specific job
$ scancel <jobid>_<index> # cancel indexed job in a job array
$ scancel -u <username> # cancel all jobs of specific user
$ scancel -u <username> -i # cancel all jobs of specific user interactively
$ scancel -t PENDING # cancel pending jobs
$ scancel -t RUNNING # cancel running jobs
</pre>

== How to show job script of a running job? ==

Use [https://slurm.schedmd.com/scontrol.html scontrol] command:

<pre>
$ scontrol write batch_script <job_id> <file>
$ scontrol write batch_script <job_id> -
</pre>

* If file name is omitted default file name will be slurm-<job_id>.sh
* If file name is - (i.e. dash) job script will be written to stdout.

== How to get estimated start time of a job? ==

<pre>$ squeue --start</pre>

'''Notes:'''
* Estimated start times are dynamic and can change at any moment. Exact start times of individual jobs are usually unpredictable.
* Slurm will report N/A for the start time estimate if nodes are not currently being reserved by the scheduler for the job to run on.

== How to show remaining walltime of running jobs? ==

Use [https://slurm.schedmd.com/squeue.html squeue] with format option "%L", e.g.:

<pre> $ squeue -t r -o "%u %i %L" </pre>

== How to check priority of jobs? ==

Use [https://slurm.schedmd.com/squeue.html squeue] with format options "%Q" and/or "%p", e.g.:

<pre>
$ squeue -o "%8i %8u %15a %.10r %.10L %.5D %.10Q"
</pre>

Use [https://slurm.schedmd.com/sprio.html sprio] command to display the priority components (age/fairshare/...) for each job:

<pre>
$ sprio
</pre>

Use "[https://slurm.schedmd.com/sshare.html sshare] command for listing the shares of associations, e.g. accounts.

<pre>
$ sshare
</pre>

== How to prevent (hold) jobs from being scheduled for execution? ==

<pre>
$ scontrol hold <job_id>
</pre>

== How to unhold job? ==

<pre>
$ scontrol release <job_id>
</pre>

== How to suspend a running job? ==

<pre>
$ scontrol suspend <job_id>
</pre>

== How to resume a suspended job? ==

<pre>
$ scontrol resume <job_id>
</pre>

== How to requeue (cancel and resubmit) a particular job? ==

<pre>
$ scontrol requeue <job_id>
</pre>

== How to monitor resource usage of running job(s)? ==

Use "[https://slurm.schedmd.com/sstat.html sstat] command.

'sstat -e' command shows a list of fields that can be specified with the '--format' option.

Example:

<pre>
$ sstat --format=JobId,AveCPU,AveRSS,MaxRSS -j <jobid>
</pre>

This will show average CPU time, average and maximum memory consumption of all tasks in the running job.
Ideally, average CPU time equals the number of cores allocated for the job multiplied by the current run time of the job.
The maximum memory consumption gives an estimate of the peak amount of memory actually needed so far. This can be compared with the amount of memory requested for the job. Over-requesting memory can result in significant waste of compute resources.

'''Notes:'''

* Users can also ssh into compute nodes that they have one or more running jobs on. Once logged in, they can use standard Linux process monitoring tools like ps, (h)top, free, vmstat, iostat, du, ...

* Users can also attach an interactive shell under an already allocated job by running the following command: <pre>srun --jobid <job> --overlap --pty /bin/bash</pre> Once logged in, they can again use standard Linux process monitoring tools like ps, (h)top, free, vmstat, iostat, du, ... For a single node job the user does not even need to know the node that the job is running on. For a multinode job, the user can still use '-w <node>' option to specify a specific node.

== How to get detailed job information ==

<pre>
$ scontrol show job 1234 # For job id 1234
$ scontrol show jobs # For all jobs
$ scontrol -o show jobs # One line per job
</pre>

== How to modify a pending/running job? ==

Use

<pre>$ scontrol update JobId=<jobid> ...</pre>

E.g.: <pre>$ scontrol update JobId=42 TimeLimit=7-0</pre>

This will modify the time limit of the job to 7 days.

'''Note:''' Update requests for '''running''' jobs are mostly restricted to Slurm administrators. In particular, only an administrator can increase the TimeLimit of a job.

== How to show accounting data of completed jobs ==

Use [https://slurm.schedmd.com/sacct.html sacct] command.

'sacct -e' command shows a list of fields that can be
specified with the '--format' option.

== How to retrieve job history and accounting ==

For a specific job:

<pre>
$ sacct -j <jobid> --format=User,JobID,Jobname,partition,state,time,start,end,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist
</pre>

For a specific user:

<pre>
$ sacct -u <user> --format=User,JobID,Jobname,partition,state,time,start,end,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist
</pre>

Note: Default time window is the current day.

Starting from a specific date:

<pre>
$ sacct -u <user> -S 2020-01-15 --format=User,JobID,Jobname,partition,state,time,start,end,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist
</pre>

Within a time window:

<pre>
$ sacct -u <user> -S 2020-01-15 -E 2020-01-31 --format=User,JobID,Jobname,partition,state,time,start,end,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist
</pre>

'''Note:'''

You can also set the environment variable $SACCT_FORMAT to specify the default format. To get a general idea of how efficiently a job utilized its resources, the following format can be used:

<pre>
export SACCT_FORMAT="JobID,JobName,Elapsed,NCPUs,TotalCPU,CPUTime,ReqMem,MaxRSS,MaxDiskRead,MaxDiskWrite,State,ExitCode"
</pre>

To find how efficiently the CPUs were used, divide TotalCPU by CPUTime. To find how efficiently memory were used, devide MaxRSS by ReqMem. But be aware that sacct memory usage measurement doesn't catch very rapid memory spikes. If your job got killed for running out of memory, it '''did run out of memory''' even if sacct reports a lower memory usage than would trigger an out-of-memory-kill. A job that reads or writes excessively to disk might be bogged down significantly by I/O operations.

== How to get efficiency information of completed job(s)? ==

Use <pre>$ seff <jobid> </pre> command for some brief information.

'''Note:''' It is good practice to have a look at the efficiency of your job(s) on completion '''and we expect you to do so'''. This way you can improve your job specifications in the future.

== How to get complete field values from sstat and sacct commands? ==

When using the [https://slurm.schedmd.com/sacct.html#OPT_format --format] option for listing various fields you can put a %NUMBER afterwards to specify how many characters should be printed.

E.g. '--format=User%30' will print 30 characters for the user name (right justified). A %-30 will print 30 characters left justified.

sstat and sacct also provide the '--parsable' and '--parsable2' option to always print full field values delimited with a pipe '|' character by default.
The delimiting character can be specified by using the '--delimiter' option, e.g. '--delimiter=","' for comma separated values.

== How to retrieve job records for all jobs running/pending at a certain point in time? ==

Use [https://slurm.schedmd.com/sacct.html sacct] with [https://slurm.schedmd.com/sacct.html#OPT_state -s <state>] and [https://slurm.schedmd.com/sacct.html#OPT_starttime -S <start time>] options, e.g.:

<pre>
$ sacct -n -a -X -S 2021-04-01T00:00:00 -s R -o JobID,User%15,Account%10,NCPUS,NNodes,NodeList%1500
</pre>

'''Note:''' When specifying the state "-s <state>" '''and''' the start time "-S <start time>", the default
time window will be set to end time "-E" equal to start time. Thus, you will get a snapshot of all running/pending
jobs at the instance given by "-S <start time>".

== How to get a parsable list of hostnames from $SLURM_JOB_NODELIST? ==

<pre>
$ scontrol show hostnames $SLURM_JOB_NODELIST
</pre>

= ADMINISTRATION =

'''Note:''' Most commands in this section are restricted to system administrators.

== How to stop Slurm from scheduling jobs? ==

You can stop Slurm from scheduling jobs on a per partition basis by
setting that partition's state to DOWN. Set its state UP to resume
scheduling. For example:

<pre>
$ scontrol update PartitionName=foo State=DOWN
$ scontrol update PartitionName=foo State=UP
</pre>

== How to print actual hardware configuration of a node? ==

<pre>
$ slurmd -C # print hardware configuration plus uptime;
$ slurmd -G # print generic resource configuration; to be run on compute nodes only
</pre>

== How to reboot (all) nodes as soon as they become idle? ==

<pre>
$ scontrol reboot ASAP nextstate=RESUME <node1>,<node2> # specific nodes
$ scontrol reboot ASAP nextstate=RESUME ALL # all nodes
</pre>

== How to cancel pending reboot of nodes? ==

<pre>
$ scontrol cancel_reboot <node1>,<node2>
</pre>

== How to check current node status? ==

<pre>
$ scontrol show node <node>
</pre>

== How to instruct all Slurm daemons to re-read the configuration file ==

<pre>
$ scontrol reconfigure
</pre>

== How to prevent a user from submitting new jobs? ==

Use the following [https://slurm.schedmd.com/sacctmgr.html sacctmgr] command:

<pre>
$ sacctmgr update user <username> set maxsubmitjobs=0
</pre>

'''Notes:'''

*Job submission is then rejected with the following message:
<pre>
$ sbatch job.slurm
sbatch: error: AssocMaxSubmitJobLimit
sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)
</pre>

* Use the following command to release the limit:
<pre>
$ sacctmgr update user <username> set maxsubmitjobs=-1
</pre>

== How to drain node(s)? ==

<pre>
$ scontrol update NodeName=<node1>,<node2> State=DRAIN Reason="Some Reason"
</pre>

Notes:

* Reason is mandatory.

* Do '''not''' just set state DOWN to drain nodes. This will kill any active jobs that may run on that nodes.

== How to list reason for nodes being drained or down? ==

<pre>
$ sinfo -R
</pre>

== How to resume node state? ==

<pre>
$ scontrol update NodeName=<node1>,<node2> State=RESUME
</pre>

== How to create a reservation on nodes? ==

Suggested reading: https://slurm.schedmd.com/reservations.html

<pre>
$ scontrol create reservation user=root starttime=now duration=UNLIMITED flags=maint,ignore_jobs nodes=ALL
$ scontrol create reservation user=root starttime=2020-12-24T17:00 duration=12:00:00 flags=maint,ignore_jobs nodes=<node1>,<node2>
$ scontrol show reservation
</pre>

'''Note:''' Add "FLEX" flag to allow jobs that qualify for the reservation to start before the reservation begins (and continue after it starts).
Add "MAGNETIC" flag to attract jobs that qualify for the reservation to run in that reservation without having requested it at submit time.

== How to create a floating reservation on nodes? ==

Use the flag "TIME_FLOAT" and a start time that is relative to the current time (use the keyword "now").
In the example below, the nodes are prevented from starting any jobs exceeding a walltime of 2 days.

<pre>
$ scontrol create reservation user=root starttime=now+2days duration=UNLIMITED flags=maint,ignore_jobs,time_float nodes=<node1>,<node2>
</pre>

'''Note:''' Floating reservation are not intended to run jobs, but to prevent long running jobs from being initiated on specific nodes. Attempts by users to make use of a floating reservation will be rejected. When ready to perform the maintenance, place the nodes in DRAIN state and delete the reservation.

== How to use a reservation? ==

<pre>
$ sbatch --reservation=foo_6 ... script.slurm
</pre>

== How to delete a reservation? ==

<pre>
$ scontrol delete ReservationName=foo_6
</pre>

== How to get node oriented information similar to 'mdiag -n'? ==

<pre>
$ sinfo -N -l
</pre>

Fields can be individually customized. See [https://slurm.schedmd.com/sinfo.html sinfo] man page. For example:

<pre>
$ sinfo -N --format="%8N %12P %.4C %.8O %.6m %.6e %.8T %.20E"

NODELIST PARTITION CPUS CPU_LOAD MEMORY FREE_M STATE REASON
n0001 standard* 0/16 0.01 128000 120445 idle none
n0002 standard* 0/16 0.01 128000 120438 idle none
n0003 standard* 0/0/ N/A 128000 N/A down* Not responding
</pre>

== How to get node oriented information similar to 'pbsnodes'? ==

<pre>
$ scontrol show nodes # One paragraph per node (all nodes)
$ scontrol show nodes <node1>,<node2> # One paragraph per node (specified nodes)
$ scontrol -o show nodes # One line per node (all nodes)
$ scontrol -o show nodes <node1>,<node2> # One line per node (specified nodes)
</pre>

== How to update multiple jobs of a user with a single scontrol command? ==

Not possible. But you can e.g. use squeue to build the script taking
advantage of its filtering and formatting options.

<pre>
$ squeue -tpd -h -o "scontrol update jobid=%i priority=1000" >my.script
</pre>

You can also identify the list of jobs and add them to the JobID all at once, for example:

<pre>
$ scontrol update JobID=123 qos=reallylargeqos
$ scontrol update JobID=123,456,789 qos=reallylargeqos
$ scontrol update JobID=[123-400],[500-600] qos=reallylargeqos
</pre>

Another option is to use the JobName, if all the jobs have the same name.

<pre>
$ scontrol update JobName="foobar" UserID=johndoe qos=reallylargeqos
</pre>

However, Slurm does not allow the UserID filter alone.

== How to create a new account? ==

Add account at top level in association tree:

<pre>
$ sacctmgr add account <accountname> Cluster=justus Description="Account description" Organization="none"
</pre>

Add account as child of some parent account in association tree:

<pre>
$ sacctmgr add account <accountname> parent=<parent_accountname>
</pre>

== How to move account to another parent? ==

<pre>
$ sacctmgr modify account name=<accountname> set parent=<new_parent_accountname>
</pre>

== How to delete an account? ==

<pre>
$ sacctmgr delete account name=<accountname>
</pre>

== How to add a new user? ==

<pre>
$ sacctmgr add user <username> DefaultAccount=<accountname>
</pre>

== How to add/remove users from an account? ==

<pre>
$ sacctmgr add user <username> account=<accountname> # Add user to account
$ sacctmgr add user <username> account=<accountname2> # Add user to a second account
$ sacctmgr remove user <username> where account=<accountname> # Remove user from this account
</pre>

== How to change default account of a user? ==

<pre>
$ sacctmgr modify user where user=<username> set DefaultAccount=<default_account>
</pre>

'''Note:''' The user must already be associated with the account you want to set as default.

== How to show account information? ==

<pre>
$ sacctmgr show assoc
$ sacctmgr show assoc tree
</pre>

'''Note:'''
* On JUSTUS 2 registered compute projects ("Rechenvorhaben") are uniquely mapped to Slurm accounts of the same name.

== How to implement user resource throttling policies? ==

Quoting from https://bugs.schedmd.com/show_bug.cgi?id=3600#c4

With Slurm, the associations are meant to establish base limits on the
defined partitions, accounts and users. Because limits propagate down
through the association tree, you only need to define limits at a high
level and those limits will be applied to all partitions, accounts and
users that are below it (parent to child). You can also override those
high level (parent) limits by explicitly setting different limits at
any lower level (on the child). So using the association tree is the
best way to get some base limits applied that you want for most cases.
QOS's are meant to override any of those base limits for exceptional
cases. Like Maui, you can use QOS's to set a different priority.
Again, the QOS would be overriding the base priority that could be set
in the associations.

== How to set a resource limit for an individual user? ==

Suggested reading: https://slurm.schedmd.com/resource_limits.html

Example:

<pre>
$ sacctmgr modify user <username> set maxjobs=1 # Limit maximum number of running jobs for user
$ sacctmgr list assoc user=<username> format=user,maxjobs # Show that limit
$ sacctmgr modify user <username> set maxjobs=-1 # Remove that limit
</pre>

== How to retrieve historical resource usage for a specific user or account ==

Use [https://slurm.schedmd.com/sreport.html sreport] command.

Examples:
<pre>
$ sreport cluster UserUtilizationByAccount Start=2021-01-01 End=2021-12-31 -t Hours user=<username> # Report cluster utilization of given user broken down by accounts
$ sreport cluster AccountUtilizationByUser Start=2021-01-01 End=2021-12-31 -t Hours account=<account> # Report cluster utilization of given account broken down by users
</pre>

'''Notes:'''
* By default CPU resources will be reported. Use '-T' option for other trackable resources, e.g. '-T cpu,mem,gres/gpu,gres/scratch'.
* On JUSTUS 2 registered compute projects ("Rechenvorhaben") are uniquely mapped to Slurm accounts of the same name. Thus, 'AccountUtilizationByUser' can also be used to report the aggregated cluster utilization of compute projects.
* Can be executed by regular users as well in which case Slurm will only report their own usage records (but along with the total usage of the associated account in the case of 'AccountUtilizationByUser').

== How to fix/reset a user's RawUsage value? ==

<pre>
$ sacctmgr modify user <username> where Account=<account> set RawUsage=<number>
</pre>

== How to create/modify/delete QOSes? ==

Suggested reading: https://slurm.schedmd.com/qos.html

Examples:

<pre>
$ sacctmgr show qos # Show existing QOSes
$ sacctmgr add qos verylong # Create new QOS verylong
$ sacctmgr modify qos verylong set MaxWall=28-00:00:00 # Set maximum walltime limit
$ sacctmgr modify qos verylong set MaxTRESPerUser=cpu=4 # Set maximum maximum number of CPUS a user can allocate at a given time
$ sacctmgr modify qos verylong set flags=denyonlimit # Prevent submission if job requests exceed any limits of QOS
$ sacctmgr modify user <username> set qos+=verylong # Add a QOS to a user account
$ sacctmgr modify user <username> set qos-=verylong # Remove a QOS from a user account
$ sacctmgr delete qos verylong # Delete that QOS
</pre>

== How to find (and fix) runaway jobs? ==

<pre>$ sacctmgr show runaway</pre>

'''Notes:'''
* Runaway jobs are orphaned jobs that don't exist in the Slurm controller but have a start and no end time in the Slurm data base. Runaway jobs mess with accounting and affects new jobs of users who have too many runaway jobs.
* If there are jobs in this state this command will also provide an option to fix them. This will set the end time for each job to the latest out of the start, eligible, or submit times, and set the state to completed.

== How to show a history of database transactions? ==

<pre>$ sacctmgr list transactions</pre>

'''Note:''' Useful to get timestamps for when a user/account/qos has been created/modified/removed etc.

BwForCluster JUSTUS 2 Slurm HOWTO

2026-09-17T12:21:50Z

K Siegmund: /* How to show accounting data of completed job(s)? */

{{Justus2}}

This is a collection of howtos and convenient Slurm commands for JUSTUS 2.

Some commands behave slightly different depending on whether they are executed
by a system administrator or by a regular user, as Slurm prevents regular users from accessing critical system information and viewing job and usage information of other users.

= GENERAL INFORMATION =

== How to find a general quick start user guide? ==

https://slurm.schedmd.com/quickstart.html

== How to find Slurm FAQ? ==

https://slurm.schedmd.com/faq.html

== How to find a Slurm cheat sheet? ==

https://slurm.schedmd.com/pdfs/summary.pdf

== How to find Slurm tutorials? ==

https://slurm.schedmd.com/tutorials.html

== How to get more information on Slurm? ==

(Almost) every Slurm command has a man page. Use it.

Online versions: https://slurm.schedmd.com/man_index.html

== How to find hardware specific details about JUSTUS 2? ==

See our Wiki page: [[Hardware and Architecture (bwForCluster JUSTUS 2)|Hardware and Architecture]]

= JOB SUBMISSION =

== How to submit a serial batch job? ==

Use [https://slurm.schedmd.com/sbatch.html sbatch] command:

<pre>$ sbatch <job-script> </pre>

Sample job script template for serial job:

<source lang="bash">
#!/bin/bash
# Allocate one node
#SBATCH --nodes=1
# Number of program instances to be executed
#SBATCH --ntasks-per-node=1
# 8 GB memory required per node
#SBATCH --mem=8G
# Maximum run time of job
#SBATCH --time=1:00:00
# Give job a reasonable name
#SBATCH --job-name=serial_job
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=serial_job-%j.out
# File name for error output
#SBATCH --error=serial_job-%j.err

# Load software modules as needed, e.g.
# module load foo/bar

# Run serial program
./my_serial_program
</source>

Sample code for serial program: [[Media:Hello_serial.c | Hello_serial.c]]

'''Notes:'''
* --nodes=1 and --ntasks-per-node=1 may be replaced by --ntasks=1.
* If not specified, stdout and stderr are both written to slurm-%j.out.

== How to find working sample scripts for my program? ==

Most software modules for applications provide working sample batch scripts.
Check with [[Software_Modules_Lmod#Module_specific_help | module help]] command, e.g.
<pre>
$ module help chem/vasp # display module help for VASP
$ module help math/matlab # display module help for Matlab
</pre>

== How to harden job scripts against common errors? ==

The bash shell provides several options that support users in disclosing hidden bugs and writing safer job scripts.
In order to activate these safeguard settings users can insert the following lines in their scripts (after all #SBATCH directives):

<source lang="bash">
[...]
set -o errexit # (or set -e) cause batch script to exit immediately when a command fails.
set -o pipefail # cause batch script to exit immediately also when the command that failed is embedded in a pipeline
set -o nounset # (or set -u) causes the script to treat unset variables as an error and exit immediately
[...]
</source>

== How to submit an interactive job? ==

Use [https://slurm.schedmd.com/salloc.html salloc] command, e.g.:
<pre>$ salloc --nodes=1 --ntasks-per-node=8</pre>

'''Note:'''

In previous Slurm versions < 20.11 the use of [https://slurm.schedmd.com/srun.html srun] has been the recommended way for launching interactive jobs, e.g.:

<pre>$ srun --nodes=1 --ntasks-per-node=8 --pty bash </pre>

Although this still works with current Slurm versions this is considered '''deprecated ''' for current Slurm versions as it may cause issues when launching additional jobs steps from within the interactive job environment. Use [https://slurm.schedmd.com/salloc.html salloc] command.

== How to enable X11 forwarding for an interactive job? ==

Use '--x11' flag, e.g.

<pre>
$ salloc --nodes=1 --ntasks-per-node=8 --x11 # run shell with X11 forwarding enabled
</pre>

'''Note:'''
* For X11 forwarding to work, you must also enable X11 forwarding for your ssh login from your local computer to the cluster, i.e.:
<pre>local> ssh -X <username>@justus2.uni-ulm.de></pre>

== How to convert Moab batch job scripts to Slurm? ==

Replace Moab/Torque job specification flags and environment variables in your job
scripts by their corresponding Slurm counterparts.

'''Commonly used Moab job specification flags and their Slurm equivalents'''

{| width=750px class="wikitable"
! Option !! Moab (msub) !! Slurm (sbatch)
|-
| Script directive || #MSUB || #SBATCH
|-
| Job name || -N <name> || --job-name=<name> (-J <name>)
|-
| Account || -A <account> || --account=<account> (-A <account>)
|-
| Queue || -q <queue> || --partition=<partition> (-p <partition>)
|-
| Wall time limit || -l walltime=<hh:mm:ss> || --time=<hh:mm:ss> (-t <hh:mm:ss>)
|-
| Node count || -l nodes=<count> || --nodes=<count> (-N <count>)
|-
| Core count || -l procs=<count> || --ntasks=<count> (-n <count>)
|-
| Process count per node || -l ppn=<count> || --ntasks-per-node=<count>
|-
| Core count per process || || --cpus-per-task=<count> (-c <count>)
|-
| Memory limit per node || -l mem=<limit> || --mem=<limit>
|-
| Memory limit per process || -l pmem=<limit> || --mem-per-cpu=<limit>
|-
| Job array || -t <array indices> || --array=<indices> (-a <indices>)
|-
| Node exclusive job || -l naccesspolicy=singlejob || --exclusive
|-
| Initial working directory || -d <directory> (default: $HOME) || --chdir=<directory> (-D <directory>) (default: submission directory)
|-
| Standard output file || -o <file path> || --output=<file> (-o <file>)
|-
| Standard error file || -e <file path> || --error=<file> (-e <file>)
|-
| Combine stdout/stderr to stdout || -j oe || --output=<combined stdout/stderr file>
|-
| Mail notification events || -m <event> || --mail-type=<events> (valid types include: NONE, BEGIN, END, FAIL, ALL)
|-
| Export environment to job || -V || --export=ALL (default)
|-
| Don't export environment to job || (default) || --export=NONE
|-
| Export environment variables to job || -v <var[=value][,var2=value2[, ...]]> || --export=<var[=value][,var2=value2[,...]]>
|}

'''Notes:'''
* Default initial job working directory is $HOME for Moab. For Slurm the default working directory is where you submit your job from.
* By default Moab does not export any environment variables to the job's runtime environment. With Slurm most of the login environment variables are exported to your job's runtime environment. This includes environment variables from software modules that were loaded at job submission time (and also $HOSTNAME variable).

'''Commonly used Moab/Torque script environment variables and their Slurm equivalents

{| width=750px class="wikitable"
! Information !! Moab !! Torque !! Slurm
|-
| Job name || $MOAB_JOBNAME || $PBS_JOBNAME || $SLURM_JOB_NAME
|-
| Job ID || $MOAB_JOBID || $PBS_JOBID || $SLURM_JOB_ID
|-
| Submit directory || $MOAB_SUBMITDIR || $PBS_O_WORKDIR || $SLURM_SUBMIT_DIR
|-
| Number of nodes allocated || $MOAB_NODECOUNT || $PBS_NUM_NODES || $SLURM_JOB_NUM_NODES (and: $SLURM_NNODES)
|-
| Node list || $MOAB_NODELIST || cat $PBS_NODEFILE || $SLURM_JOB_NODELIST
|-
| Number of processes || $MOAB_PROCCOUNT || $PBS_TASKNUM || $SLURM_NTASKS
|-
| Requested tasks per node || --- || $PBS_NUM_PPN || $SLURM_NTASKS_PER_NODE
|-
| Requested CPUs per task || --- || --- || $SLURM_CPUS_PER_TASK
|-
| Job array index || $MOAB_JOBARRAYINDEX || $PBS_ARRAY_INDEX || $SLURM_ARRAY_TASK_ID
|-
| Job array range || $MOAB_JOBARRAYRANGE || - || $SLURM_ARRAY_TASK_COUNT
|-
| Queue name || $MOAB_CLASS || $PBS_QUEUE || $SLURM_JOB_PARTITION
|-
| QOS name || $MOAB_QOS || --- || $SLURM_JOB_QOS
|-
| Number of processes per node | --- || $PBS_NUM_PPN || $SLURM_TASKS_PER_NODE
|-
| Job user || $MOAB_USER || $PBS_O_LOGNAME || $SLURM_JOB_USER
|-
| Hostname || $MOAB_MACHINE || $PBS_O_HOST || $SLURMD_NODENAME
|}

'''Note:'''

* See [https://slurm.schedmd.com/sbatch.html sbatch] man page for a complete list of flags and environment variables.

== How to emulate Moab output file names? ==

Use the following directives:

<pre>
#SBATCH --output="%x.o%j"
#SBATCH --error="%x.e%j"
</pre>

== How to pass command line arguments to the job script? ==

Run <pre>$ sbatch <job-script> arg1 arg2 ...</pre>

Inside the job script the arguments can be accessed as $1, $2, ...

E.g.:

<source lang="bash">
[...]
infile="$1"
outfile="$2"
./my_serial_program < "$infile" > "$outfile" 2>&1
[...]
</source>

'''Notes:'''

* Do '''not''' use $1, $2, ... in "#SBATCH" lines. These parameters can be used only within the regular shell script.

== How to request local scratch (SSD/NVMe) at job submission? ==

Use '--gres=scratch:nnn' option to allocate nnn GB of local (i.e. node-local) scratch space for the entire job.

Example: '--gres=scratch:100' will allocate 100 GB scratch space on a locally attached NVMe device.

'''Notes:'''

* Do '''not''' add any unit (such as --gres=scratch:100G). This would be treated as requesting an amount of 10^9 * 100GB of scratch space.

* Multinode jobs get nnn GB of local scratch space on every node of the job.

* Environment variable '''$SCRATCH''' will point to
** /scratch/<user>.<jobid> when local scratch has been requested. This will be on locally attached SSD/NVMe devices.
** /tmp/<user>.<jobid> when no local scratch has been requested. This will be in memory and, thus, be limited in size.

* Environment variable '''$TMPDIR''' always points to /tmp/<user>.<jobid>. This will always be in memory and, thus, limited in size.

* For backward compatibility environment variable $RAMDISK always points to /tmp/<user>.<jobid>

* Scratch space allocation in /scratch will be enforced by quota limits

* Data written to $TMPDIR will always count against allocated memory.

* Data written to local scratch space will automatically be removed at the end of the job.

== How to request GPGPU nodes at job submission? ==

Use '--gres=gpu:<count>' option to allocate 1 or 2 GPUs per node for the entire job.

Example: '--gres=gpu:1' will allocate one GPU per node for this job.

'''Notes:'''

* GPGPU nodes are equipped with two Nvidia V100S cards

* Environment variables $CUDA_VISIBLE_DEVICES, $SLURM_JOB_GPUS and $GPU_DEVICE_ORDINAL will denote card(s) allocated for the job.

* CUDA Toolkit is available as software module devel/cuda.

== How to clean-up or save files before a job times out? ==

Possibly you would like to clean up the work directory or save intermediate result files in case a job times out.

The following sample script may serve as a blueprint for implementing a pre-termination function to perform clean-up or file recovery actions.

<source lang="bash">
#!/bin/bash
# Allocate one node
#SBATCH --nodes=1
# Number of program instances to be executed
#SBATCH --ntasks-per-node=1
# 2 GB memory required per node
#SBATCH --mem=2G
# Request 10 GB local scratch space
#SBATCH --gres=scratch:10
# Maximum run time of job
#SBATCH --time=10:00
# Send the USR1 signal 120 seconds before end of time limit
#SBATCH --signal=B:USR1@120
# Give job a reasonable name
#SBATCH --job-name=signal_job
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=signal_job-%j.out
# File name for error output
#SBATCH --error=signal_job-%j.err

# Define the signal handler function
# Note: This is not executed here, but rather when the associated
# signal is received by the shell.
finalize_job()
{
# Do whatever cleanup you want here. In this example we copy
# output file(s) back to $SLURM_SUBMIT_DIR, but you may implement
# your own job finalization code here.
echo "function finalize_job called at `date`"
cd $SCRATCH
mkdir -vp "$SLURM_SUBMIT_DIR"/results
tar czvf "$SLURM_SUBMIT_DIR"/results/${SLURM_JOB_ID}.tgz output*.txt
exit
}

# Call finalize_job function as soon as we receive USR1 signal
trap 'finalize_job' USR1

# Copy input files for this job to the scratch directory (if needed).
# Note: Environment variable $SCRATCH always points to a scratch directory
# automatically created for this job. Environment variable $SLURM_SUBMIT_DIR
# points to the path where this script was submitted from.
# Example:
# cp -v "$SLURM_SUBMIT_DIR"/input*.txt "$SCRATCH"

# Change working directory to local scratch directory
cd "$SCRATCH"

# Load software modules as needed, e.g.
# module load foo/bar

# This is where the actual work is done. In this case we just create
# a sample output file for 900 (=15*60) seconds, but since we asked
# Slurm for 600 seconds only it will not be able finish within this
# wall time.
# Note: It is important to run this task in the background
# by placing the & symbol at the end. Otherwise the signal handler
# would not be executed until that process has finished, which is not
# what we want.
(for i in `seq 15`; do echo "Hello World at `date +%H:%M:%S`."; sleep 60; done) >output.txt 2>&1 &

# Note: The command above is just for illustration. Normally you would just run
# my_program >output.txt 2>&1 &

# Tell the shell to wait for background task(s) to finish.
# Note: This is important because otherwise the parent shell
# (this script) would proceed (and terminate) without waiting for
# background task(s) to finish.
wait

# If we get here, the job did not time out but finished in time.

# Release user defined signal handler for USR1
trap - USR1

# Do regular cleanup and save files. In this example we simply call
# the same function that we defined as a signal handler above, but you
# may implement your own code here.
finalize_job

exit
</source>

'''Notes:'''

* The number of seconds specified in --signal option must match the runtime of the pre-termination function and must not exceed 65535 seconds.

* Due to the resolution of event handling by Slurm, the signal may be sent a little earlier than specified.

== How to submit a multithreaded batch job? ==

Sample job script template for a job running one multithreaded program instance:

<source lang="bash">
#!/bin/bash
# Allocate one node
#SBATCH --nodes=1
# Number of program instances to be executed
#SBATCH --ntasks-per-node=1
# Number of cores per program instance
#SBATCH --cpus-per-task=8
# 8 GB memory required per node
#SBATCH --mem=8G
# Maximum run time of job
#SBATCH --time=1:00:00
# Give job a reasonable name
#SBATCH --job-name=multithreaded_job
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=multithreaded_job-%j.out
# File name for error output
#SBATCH --error=multithreaded_job-%j.err

# Load software modules as needed, e.g.
# module load foo/bar

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
export MKL_NUM_THREADS=${SLURM_CPUS_PER_TASK}

# Run multithreaded program
./my_multithreaded_program
</source>

Sample code for multithreaded program: [[Media:Hello_openmp.c | Hello_openmp.c]]

'''Notes:'''

* In our configuration each physical core is considered a "CPU".

* On JUSTUS 2 it is recommended to specify a number of cores per task ('--cpus-per-task') that is either an integer divisor of 24 or (at most) 48.

* Required memory can also by specified per allocated CPU with '--mem-per-cpu' option.

* The '--mem' and '--mem-per-cpu' options are mutually exclusive.

== How to submit an array job? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_array -a] (or [https://slurm.schedmd.com/sbatch.html#OPT_array --array]) option, e.g.

<pre>$ sbatch -a 1-16%8 ...</pre>

This will submit 16 tasks to be executed, each one indexed by SLURM_ARRAY_TASK_ID ranging from 1 to 16, but will limit the number of simultaneously running tasks from this job array to 8.

Sample job script template for an array job:

<source lang="bash">
#!/bin/bash
# Number of cores per individual array task
#SBATCH --ntasks=1
#SBATCH --array=1-16%8
#SBATCH --mem=4G
#SBATCH --time=01:00:00
#SBATCH --job-name=array_job
#SBATCH --output=array_job-%A_%a.out
#SBATCH --error=array_job-%A_%a.err

# Load software modules as needed, e.g.
# module load foo/bar

# Print the task id.
echo "My SLURM_ARRAY_TASK_ID: " $SLURM_ARRAY_TASK_ID

# Add lines here to run your computations, e.g.
# ./my_program <input.$SLURM_ARRAY_TASK_ID

</source>

'''Notes:'''

* Placeholder %A will be replaced by the master job id, %a will be replaced by the array task id.

* Every sub job in an array job will have its own unique environment variable $SLURM_JOB_ID. Environment variable $SLURM_ARRAY_JOB_ID will be set to the first job array index value for all tasks.

* The remaining options in the sample job script are the same as the options used in other, non-array jobs. In the example above, we are requesting that each array task be allocated 1 CPU (--ntasks=1) and 4 GB of memory (--mem=4G) for up to one hour (--time=01:00:00).

* More information: https://slurm.schedmd.com/job_array.html

== How to delay the start of a job? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_begin -b] (or [https://slurm.schedmd.com/sbatch.html#OPT_begin --begin]) option in order to defer the allocation of the job until the specified time.

Examples:

<pre>
sbatch --begin=20:00 ... # job can start after 8 p.m.
sbatch --begin=now+1hour ... # job can start 1 hour after submission
sbatch --begin=teatime ... # job can start at teatime (4 p.m.)
sbatch --begin=2023-12-24T20:00:00 ... # job can start after specified date/time
</pre>

== How to submit dependency (chain) jobs? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_dependency -d] (or [https://slurm.schedmd.com/sbatch.html#OPT_dependency --dependency]) option, e.g.

<pre>$ sbatch -d afterany:123456 ...</pre>

This will defer the submitted job until the specified job 123456 has terminated.

Slurm supports a number of different dependency types, e.g.:

<pre>
-d after:123456 # job can begin execution after the specified job has begun execution
-d afterany:123456 # job can begin execution after the specified job has finished
-d afternotok:123456 # job can begin execution after the specified job has failed (exit code not equal zero)
-d afterok:123456 # job can begin execution after the specified job has successfully finished (exit code zero)
-d singleton # job can begin execution after any previously job with the same job name and user have finished
</pre>

'''Note:''' Multiple jobs can be specified by separating their job ids by colon characters (:), e.g.

<pre> $ sbatch -d afterany:123456:123457 ... </pre>

This will defer the submitted job until the specified jobs 123456 and 123457 have both finished.

== How to deal with invalid job dependencies? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_kill-on-invalid-dep --kill-on-invalid-dep=yes] option in order to automatically terminate jobs which can never run due to invalid dependencies. By default the job stays pending with reason 'DependencyNeverSatisfied' to allow review and appropriate action by the user.

'''Note:''' A job dependency may also become invalid if a job has been submitted with '-d afterok:<jobid>' but the specified dependency job has failed, e.g. because it timed out (i.e. exceeded its wall time limit).

== How to submit an MPI batch job? ==

Suggested reading: https://slurm.schedmd.com/mpi_guide.html

Sample job script template for an MPI job:

<source lang="bash">
#!/bin/bash
# Allocate two nodes
#SBATCH --nodes=2
# Number of program instances to be executed
#SBATCH --ntasks-per-node=48
# Allocate 32 GB memory per node
#SBATCH --mem=32gb
# Maximum run time of job
#SBATCH --time=1:00:00
# Give job a reasonable name
#SBATCH --job-name=mpi_job
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=mpi_job-%j.out
# File name for error output
#SBATCH --error=mpi_job-%j.err

# Add lines here to run your computations, e.g.
#
# Option 1: Lauch MPI tasks by using mpirun
#
# for OpenMPI and GNU compiler:
#
# module load compiler/gnu
# module load mpi/openmpi
# mpirun ./my_mpi_program
#
# for Intel MPI and Intel complier:
#
# module load compiler/intel
# module load mpi/impi
# mpirun ./my_mpi_program
#
# Option 2: Launch MPI tasks by using srun
#
# for OpenMPI and GNU compiler:
#
# module load compiler/gnu
# module load mpi/openmpi
# srun ./my_mpi_program
#
# for Intel MPI and Intel compiler:
#
module load compiler/intel
module load mpi/impi
srun ./my_mpi_program

</source>

Sample code for MPI program: [[Media:Hello_mpi.c | Hello_mpi.c]]

'''Notes'''

* SchedMD recommends to use srun and many (most?) sites do so as well. The rationale is that srun is more tightly integrated with the scheduler and provides more consistent and reliable resource tracking and accounting for individual jobs and job steps. mpirun may behave differently for different MPI implementations and versions. There are reports that claim "strange behavior" of mpirun especially when using task affinity and core binding. Using srun is supposed to resolve these issues and is therefore highly recommended.
* Do not run batch jobs that launch a large number (hundreds or thousands) short running (few minutes or less) MPI programs, e.g. from a shell loop. Every single MPI invocation does generate its own job step and sends remote procedure calls to the Slurm controller server. This can result in degradation of performance for both, Slurm and the application, especially if many of that jobs happen to run at the same time. Jobs of that kind can even get stuck without showing any further activity until hitting the wall time limit. For high throughput computing (e.g. processing a large number of files with every single task running independently from each other and very shortly), consider a more appropriate parallelization paradigm that invokes independent serial (non-MPI) processes in parallel at the same time. This approach is sometimes referred to as "[https://en.wikipedia.org/wiki/Embarrassingly_parallel pleasingly parallel]" workload. GNU Parallel is a shell tool that facilitates executing serial tasks in parallel. On JUSTUS 2 this tool is available as a software module "system/parallel".

== How to submit a hybrid MPI/OpenMP job? ==

Sample job script template for an hybrid job:

<source lang="bash">
#!/bin/bash
# Number of nodes to allocate
#SBATCH --nodes=4
# Number of MPI instances (ranks) to be executed per node
#SBATCH --ntasks-per-node=2
# Number of threads per MPI instance
#SBATCH --cpus-per-task=24
# Allocate 8 GB memory per node
#SBATCH --mem=8gb
# Maximum run time of job
#SBATCH --time=1:00:00
# Give job a reasonable name
#SBATCH --job-name=hybrid_job
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=hybrid_job-%j.out
# File name for error output
#SBATCH --error=hybrid_job-%j.err

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
export MKL_NUM_THREADS=${SLURM_CPUS_PER_TASK}

module load compiler/intel
module load mpi/impi
srun ./my_hybrid_program

# or:
# mpirun ./my_hybrid_program

</source>

Sample code for hybrid program: [[Media:Hello_hybrid.c | Hello_hybrid.c]]

'''Notes:'''

* $SLURM_CPUS_PER_TASK is only set if the '--cpus-per-task' option is specified.

== How to request specific node(s) at job submission? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_nodelist -w] (or [https://slurm.schedmd.com/sbatch.html#OPT_nodelist --nodelist]) option, e.g.:

<pre>$ sbatch -w <node1>,<node2> ...</pre>

Also see [https://slurm.schedmd.com/sbatch.html#OPT_nodefile -F] (or [https://slurm.schedmd.com/sbatch.html#OPT_nodefile --nodefile]) option.

== How to exclude specific nodes from job? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_exclude -x] (or [https://slurm.schedmd.com/sbatch.html#OPT_exclude --exclude]) option, e.g.:

<pre>$ sbatch -x <node1>,<node2> ...</pre>

== How to get exclusive jobs? ==

Use '--exclusive' option on job submission. This makes sure that there will be no other jobs running on your nodes. Very useful for benchmarking!

'''Note:'''

* --exclusive option does '''not''' mean that you automatically get full access to all the resources which the node might provide without explicitly requesting them.

== How to avoid sharing nodes with other users? ==

Use '--exclusive=user' option on job submission. This will still allow multiple jobs of one and the same user on the nodes.

''' Note:'''

* Depending on configuration, exclusive=user may (and probably will) be the default node access policy on JUSTUS 2.

== How to submit batch job without job script? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_wrap --wrap] option.

Example:

<pre>$ sbatch --nodes=2 --ntasks-per-node=16 --wrap "sleep 600"</pre>

'''Note:''' May be useful for testing purposes.

= JOB MONITORING AND CONTROL =

== How to prevent Slurm performance degradation? ==

Almost every invocation of a Slurm client command (e.g. squeue, sacct, sprio or sshare) sends a remote procedure call (RPC) to the Slurm control daemon and/or database.
If enough remote procedure calls come in at once, this can result in a degradation of performance of the Slurm services for all users, possibly resulting in a denial of service.

Therefore, '''do not run Slurm client commands that send remote procedure calls from loops in shell scripts or other programs''' (such as 'watch squeue'). Always ensure to limit calls to squeue, sstat, sacct etc. to the minimum necessary for the information you are trying to gather.

Slurm does collect RPC counts and timing statistics by message type and user for diagnostic purposes.

== How to view information about submitted jobs? ==

Use [https://slurm.schedmd.com/squeue.html squeue] command, e.g.:

<pre>
$ squeue # all jobs owned by user (all jobs owned by all users for admins)
$ squeue --me # all jobs owned by user (same as squeue for regular users)
$ squeue -u <username> # jobs of specific user
$ squeue -t PENDING # pending jobs only
$ squeue -t RUNNING # running jobs only
</pre>

'''Notes:'''

* The output format of [https://slurm.schedmd.com/squeue.html squeue] (and most other Slurm commands) is highly configurable to your needs. Look for the --format or --Format options.

* Every invocation of squeue sends a remote procedure call to the Slurm database server. '''Do not run squeue or other Slurm client commands from loops in shell scripts or other programs''' as this can result in a degradation of performance. Ensure that programs limit calls to squeue to the minimum necessary for the information you are trying to gather.

== How to cancel jobs? ==

Use [https://slurm.schedmd.com/scancel.html scancel] command, e.g.

<pre>
$ scancel <jobid> # cancel specific job
$ scancel <jobid>_<index> # cancel indexed job in a job array
$ scancel -u <username> # cancel all jobs of specific user
$ scancel -u <username> -i # cancel all jobs of specific user interactively
$ scancel -t PENDING # cancel pending jobs
$ scancel -t RUNNING # cancel running jobs
</pre>

== How to show job script of a running job? ==

Use [https://slurm.schedmd.com/scontrol.html scontrol] command:

<pre>
$ scontrol write batch_script <job_id> <file>
$ scontrol write batch_script <job_id> -
</pre>

* If file name is omitted default file name will be slurm-<job_id>.sh
* If file name is - (i.e. dash) job script will be written to stdout.

== How to get estimated start time of a job? ==

<pre>$ squeue --start</pre>

'''Notes:'''
* Estimated start times are dynamic and can change at any moment. Exact start times of individual jobs are usually unpredictable.
* Slurm will report N/A for the start time estimate if nodes are not currently being reserved by the scheduler for the job to run on.

== How to show remaining walltime of running jobs? ==

Use [https://slurm.schedmd.com/squeue.html squeue] with format option "%L", e.g.:

<pre> $ squeue -t r -o "%u %i %L" </pre>

== How to check priority of jobs? ==

Use [https://slurm.schedmd.com/squeue.html squeue] with format options "%Q" and/or "%p", e.g.:

<pre>
$ squeue -o "%8i %8u %15a %.10r %.10L %.5D %.10Q"
</pre>

Use [https://slurm.schedmd.com/sprio.html sprio] command to display the priority components (age/fairshare/...) for each job:

<pre>
$ sprio
</pre>

Use "[https://slurm.schedmd.com/sshare.html sshare] command for listing the shares of associations, e.g. accounts.

<pre>
$ sshare
</pre>

== How to prevent (hold) jobs from being scheduled for execution? ==

<pre>
$ scontrol hold <job_id>
</pre>

== How to unhold job? ==

<pre>
$ scontrol release <job_id>
</pre>

== How to suspend a running job? ==

<pre>
$ scontrol suspend <job_id>
</pre>

== How to resume a suspended job? ==

<pre>
$ scontrol resume <job_id>
</pre>

== How to requeue (cancel and resubmit) a particular job? ==

<pre>
$ scontrol requeue <job_id>
</pre>

== How to monitor resource usage of running job(s)? ==

Use "[https://slurm.schedmd.com/sstat.html sstat] command.

'sstat -e' command shows a list of fields that can be specified with the '--format' option.

Example:

<pre>
$ sstat --format=JobId,AveCPU,AveRSS,MaxRSS -j <jobid>
</pre>

This will show average CPU time, average and maximum memory consumption of all tasks in the running job.
Ideally, average CPU time equals the number of cores allocated for the job multiplied by the current run time of the job.
The maximum memory consumption gives an estimate of the peak amount of memory actually needed so far. This can be compared with the amount of memory requested for the job. Over-requesting memory can result in significant waste of compute resources.

'''Notes:'''

* Users can also ssh into compute nodes that they have one or more running jobs on. Once logged in, they can use standard Linux process monitoring tools like ps, (h)top, free, vmstat, iostat, du, ...

* Users can also attach an interactive shell under an already allocated job by running the following command: <pre>srun --jobid <job> --overlap --pty /bin/bash</pre> Once logged in, they can again use standard Linux process monitoring tools like ps, (h)top, free, vmstat, iostat, du, ... For a single node job the user does not even need to know the node that the job is running on. For a multinode job, the user can still use '-w <node>' option to specify a specific node.

== How to get detailed job information ==

<pre>
$ scontrol show job 1234 # For job id 1234
$ scontrol show jobs # For all jobs
$ scontrol -o show jobs # One line per job
</pre>

== How to modify a pending/running job? ==

Use

<pre>$ scontrol update JobId=<jobid> ...</pre>

E.g.: <pre>$ scontrol update JobId=42 TimeLimit=7-0</pre>

This will modify the time limit of the job to 7 days.

'''Note:''' Update requests for '''running''' jobs are mostly restricted to Slurm administrators. In particular, only an administrator can increase the TimeLimit of a job.

== How to show accounting data of completed jobs ==

Use [https://slurm.schedmd.com/sacct.html sacct] command.

'sacct -e' command shows a list of fields that can be
specified with the '--format' option.

== How to retrieve job history and accounting? ==

For a specific job:

<pre>
$ sacct -j <jobid> --format=User,JobID,Jobname,partition,state,time,start,end,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist
</pre>

For a specific user:

<pre>
$ sacct -u <user> --format=User,JobID,Jobname,partition,state,time,start,end,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist
</pre>

Note: Default time window is the current day.

Starting from a specific date:

<pre>
$ sacct -u <user> -S 2020-01-15 --format=User,JobID,Jobname,partition,state,time,start,end,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist
</pre>

Within a time window:

<pre>
$ sacct -u <user> -S 2020-01-15 -E 2020-01-31 --format=User,JobID,Jobname,partition,state,time,start,end,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist
</pre>

'''Note:'''

You can also set the environment variable $SACCT_FORMAT to specify the default format. To get a general idea of how efficiently a job utilized its resources, the following format can be used:

<pre>
export SACCT_FORMAT="JobID,JobName,Elapsed,NCPUs,TotalCPU,CPUTime,ReqMem,MaxRSS,MaxDiskRead,MaxDiskWrite,State,ExitCode"
</pre>

To find how efficiently the CPUs were used, divide TotalCPU by CPUTime. To find how efficiently memory were used, devide MaxRSS by ReqMem. But be aware that sacct memory usage measurement doesn't catch very rapid memory spikes. If your job got killed for running out of memory, it '''did run out of memory''' even if sacct reports a lower memory usage than would trigger an out-of-memory-kill. A job that reads or writes excessively to disk might be bogged down significantly by I/O operations.

== How to get efficiency information of completed job(s)? ==

Use <pre>$ seff <jobid> </pre> command for some brief information.

'''Note:''' It is good practice to have a look at the efficiency of your job(s) on completion '''and we expect you to do so'''. This way you can improve your job specifications in the future.

== How to get complete field values from sstat and sacct commands? ==

When using the [https://slurm.schedmd.com/sacct.html#OPT_format --format] option for listing various fields you can put a %NUMBER afterwards to specify how many characters should be printed.

E.g. '--format=User%30' will print 30 characters for the user name (right justified). A %-30 will print 30 characters left justified.

sstat and sacct also provide the '--parsable' and '--parsable2' option to always print full field values delimited with a pipe '|' character by default.
The delimiting character can be specified by using the '--delimiter' option, e.g. '--delimiter=","' for comma separated values.

== How to retrieve job records for all jobs running/pending at a certain point in time? ==

Use [https://slurm.schedmd.com/sacct.html sacct] with [https://slurm.schedmd.com/sacct.html#OPT_state -s <state>] and [https://slurm.schedmd.com/sacct.html#OPT_starttime -S <start time>] options, e.g.:

<pre>
$ sacct -n -a -X -S 2021-04-01T00:00:00 -s R -o JobID,User%15,Account%10,NCPUS,NNodes,NodeList%1500
</pre>

'''Note:''' When specifying the state "-s <state>" '''and''' the start time "-S <start time>", the default
time window will be set to end time "-E" equal to start time. Thus, you will get a snapshot of all running/pending
jobs at the instance given by "-S <start time>".

== How to get a parsable list of hostnames from $SLURM_JOB_NODELIST? ==

<pre>
$ scontrol show hostnames $SLURM_JOB_NODELIST
</pre>

= ADMINISTRATION =

'''Note:''' Most commands in this section are restricted to system administrators.

== How to stop Slurm from scheduling jobs? ==

You can stop Slurm from scheduling jobs on a per partition basis by
setting that partition's state to DOWN. Set its state UP to resume
scheduling. For example:

<pre>
$ scontrol update PartitionName=foo State=DOWN
$ scontrol update PartitionName=foo State=UP
</pre>

== How to print actual hardware configuration of a node? ==

<pre>
$ slurmd -C # print hardware configuration plus uptime;
$ slurmd -G # print generic resource configuration; to be run on compute nodes only
</pre>

== How to reboot (all) nodes as soon as they become idle? ==

<pre>
$ scontrol reboot ASAP nextstate=RESUME <node1>,<node2> # specific nodes
$ scontrol reboot ASAP nextstate=RESUME ALL # all nodes
</pre>

== How to cancel pending reboot of nodes? ==

<pre>
$ scontrol cancel_reboot <node1>,<node2>
</pre>

== How to check current node status? ==

<pre>
$ scontrol show node <node>
</pre>

== How to instruct all Slurm daemons to re-read the configuration file ==

<pre>
$ scontrol reconfigure
</pre>

== How to prevent a user from submitting new jobs? ==

Use the following [https://slurm.schedmd.com/sacctmgr.html sacctmgr] command:

<pre>
$ sacctmgr update user <username> set maxsubmitjobs=0
</pre>

'''Notes:'''

*Job submission is then rejected with the following message:
<pre>
$ sbatch job.slurm
sbatch: error: AssocMaxSubmitJobLimit
sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)
</pre>

* Use the following command to release the limit:
<pre>
$ sacctmgr update user <username> set maxsubmitjobs=-1
</pre>

== How to drain node(s)? ==

<pre>
$ scontrol update NodeName=<node1>,<node2> State=DRAIN Reason="Some Reason"
</pre>

Notes:

* Reason is mandatory.

* Do '''not''' just set state DOWN to drain nodes. This will kill any active jobs that may run on that nodes.

== How to list reason for nodes being drained or down? ==

<pre>
$ sinfo -R
</pre>

== How to resume node state? ==

<pre>
$ scontrol update NodeName=<node1>,<node2> State=RESUME
</pre>

== How to create a reservation on nodes? ==

Suggested reading: https://slurm.schedmd.com/reservations.html

<pre>
$ scontrol create reservation user=root starttime=now duration=UNLIMITED flags=maint,ignore_jobs nodes=ALL
$ scontrol create reservation user=root starttime=2020-12-24T17:00 duration=12:00:00 flags=maint,ignore_jobs nodes=<node1>,<node2>
$ scontrol show reservation
</pre>

'''Note:''' Add "FLEX" flag to allow jobs that qualify for the reservation to start before the reservation begins (and continue after it starts).
Add "MAGNETIC" flag to attract jobs that qualify for the reservation to run in that reservation without having requested it at submit time.

== How to create a floating reservation on nodes? ==

Use the flag "TIME_FLOAT" and a start time that is relative to the current time (use the keyword "now").
In the example below, the nodes are prevented from starting any jobs exceeding a walltime of 2 days.

<pre>
$ scontrol create reservation user=root starttime=now+2days duration=UNLIMITED flags=maint,ignore_jobs,time_float nodes=<node1>,<node2>
</pre>

'''Note:''' Floating reservation are not intended to run jobs, but to prevent long running jobs from being initiated on specific nodes. Attempts by users to make use of a floating reservation will be rejected. When ready to perform the maintenance, place the nodes in DRAIN state and delete the reservation.

== How to use a reservation? ==

<pre>
$ sbatch --reservation=foo_6 ... script.slurm
</pre>

== How to delete a reservation? ==

<pre>
$ scontrol delete ReservationName=foo_6
</pre>

== How to get node oriented information similar to 'mdiag -n'? ==

<pre>
$ sinfo -N -l
</pre>

Fields can be individually customized. See [https://slurm.schedmd.com/sinfo.html sinfo] man page. For example:

<pre>
$ sinfo -N --format="%8N %12P %.4C %.8O %.6m %.6e %.8T %.20E"

NODELIST PARTITION CPUS CPU_LOAD MEMORY FREE_M STATE REASON
n0001 standard* 0/16 0.01 128000 120445 idle none
n0002 standard* 0/16 0.01 128000 120438 idle none
n0003 standard* 0/0/ N/A 128000 N/A down* Not responding
</pre>

== How to get node oriented information similar to 'pbsnodes'? ==

<pre>
$ scontrol show nodes # One paragraph per node (all nodes)
$ scontrol show nodes <node1>,<node2> # One paragraph per node (specified nodes)
$ scontrol -o show nodes # One line per node (all nodes)
$ scontrol -o show nodes <node1>,<node2> # One line per node (specified nodes)
</pre>

== How to update multiple jobs of a user with a single scontrol command? ==

Not possible. But you can e.g. use squeue to build the script taking
advantage of its filtering and formatting options.

<pre>
$ squeue -tpd -h -o "scontrol update jobid=%i priority=1000" >my.script
</pre>

You can also identify the list of jobs and add them to the JobID all at once, for example:

<pre>
$ scontrol update JobID=123 qos=reallylargeqos
$ scontrol update JobID=123,456,789 qos=reallylargeqos
$ scontrol update JobID=[123-400],[500-600] qos=reallylargeqos
</pre>

Another option is to use the JobName, if all the jobs have the same name.

<pre>
$ scontrol update JobName="foobar" UserID=johndoe qos=reallylargeqos
</pre>

However, Slurm does not allow the UserID filter alone.

== How to create a new account? ==

Add account at top level in association tree:

<pre>
$ sacctmgr add account <accountname> Cluster=justus Description="Account description" Organization="none"
</pre>

Add account as child of some parent account in association tree:

<pre>
$ sacctmgr add account <accountname> parent=<parent_accountname>
</pre>

== How to move account to another parent? ==

<pre>
$ sacctmgr modify account name=<accountname> set parent=<new_parent_accountname>
</pre>

== How to delete an account? ==

<pre>
$ sacctmgr delete account name=<accountname>
</pre>

== How to add a new user? ==

<pre>
$ sacctmgr add user <username> DefaultAccount=<accountname>
</pre>

== How to add/remove users from an account? ==

<pre>
$ sacctmgr add user <username> account=<accountname> # Add user to account
$ sacctmgr add user <username> account=<accountname2> # Add user to a second account
$ sacctmgr remove user <username> where account=<accountname> # Remove user from this account
</pre>

== How to change default account of a user? ==

<pre>
$ sacctmgr modify user where user=<username> set DefaultAccount=<default_account>
</pre>

'''Note:''' The user must already be associated with the account you want to set as default.

== How to show account information? ==

<pre>
$ sacctmgr show assoc
$ sacctmgr show assoc tree
</pre>

'''Note:'''
* On JUSTUS 2 registered compute projects ("Rechenvorhaben") are uniquely mapped to Slurm accounts of the same name.

== How to implement user resource throttling policies? ==

Quoting from https://bugs.schedmd.com/show_bug.cgi?id=3600#c4

With Slurm, the associations are meant to establish base limits on the
defined partitions, accounts and users. Because limits propagate down
through the association tree, you only need to define limits at a high
level and those limits will be applied to all partitions, accounts and
users that are below it (parent to child). You can also override those
high level (parent) limits by explicitly setting different limits at
any lower level (on the child). So using the association tree is the
best way to get some base limits applied that you want for most cases.
QOS's are meant to override any of those base limits for exceptional
cases. Like Maui, you can use QOS's to set a different priority.
Again, the QOS would be overriding the base priority that could be set
in the associations.

== How to set a resource limit for an individual user? ==

Suggested reading: https://slurm.schedmd.com/resource_limits.html

Example:

<pre>
$ sacctmgr modify user <username> set maxjobs=1 # Limit maximum number of running jobs for user
$ sacctmgr list assoc user=<username> format=user,maxjobs # Show that limit
$ sacctmgr modify user <username> set maxjobs=-1 # Remove that limit
</pre>

== How to retrieve historical resource usage for a specific user or account ==

Use [https://slurm.schedmd.com/sreport.html sreport] command.

Examples:
<pre>
$ sreport cluster UserUtilizationByAccount Start=2021-01-01 End=2021-12-31 -t Hours user=<username> # Report cluster utilization of given user broken down by accounts
$ sreport cluster AccountUtilizationByUser Start=2021-01-01 End=2021-12-31 -t Hours account=<account> # Report cluster utilization of given account broken down by users
</pre>

'''Notes:'''
* By default CPU resources will be reported. Use '-T' option for other trackable resources, e.g. '-T cpu,mem,gres/gpu,gres/scratch'.
* On JUSTUS 2 registered compute projects ("Rechenvorhaben") are uniquely mapped to Slurm accounts of the same name. Thus, 'AccountUtilizationByUser' can also be used to report the aggregated cluster utilization of compute projects.
* Can be executed by regular users as well in which case Slurm will only report their own usage records (but along with the total usage of the associated account in the case of 'AccountUtilizationByUser').

== How to fix/reset a user's RawUsage value? ==

<pre>
$ sacctmgr modify user <username> where Account=<account> set RawUsage=<number>
</pre>

== How to create/modify/delete QOSes? ==

Suggested reading: https://slurm.schedmd.com/qos.html

Examples:

<pre>
$ sacctmgr show qos # Show existing QOSes
$ sacctmgr add qos verylong # Create new QOS verylong
$ sacctmgr modify qos verylong set MaxWall=28-00:00:00 # Set maximum walltime limit
$ sacctmgr modify qos verylong set MaxTRESPerUser=cpu=4 # Set maximum maximum number of CPUS a user can allocate at a given time
$ sacctmgr modify qos verylong set flags=denyonlimit # Prevent submission if job requests exceed any limits of QOS
$ sacctmgr modify user <username> set qos+=verylong # Add a QOS to a user account
$ sacctmgr modify user <username> set qos-=verylong # Remove a QOS from a user account
$ sacctmgr delete qos verylong # Delete that QOS
</pre>

== How to find (and fix) runaway jobs? ==

<pre>$ sacctmgr show runaway</pre>

'''Notes:'''
* Runaway jobs are orphaned jobs that don't exist in the Slurm controller but have a start and no end time in the Slurm data base. Runaway jobs mess with accounting and affects new jobs of users who have too many runaway jobs.
* If there are jobs in this state this command will also provide an option to fix them. This will set the end time for each job to the latest out of the start, eligible, or submit times, and set the state to completed.

== How to show a history of database transactions? ==

<pre>$ sacctmgr list transactions</pre>

'''Note:''' Useful to get timestamps for when a user/account/qos has been created/modified/removed etc.

BwForCluster JUSTUS 2 Slurm HOWTO

2026-09-17T12:21:22Z

K Siegmund: /* How to retrieve historical resource usage for a specific user or account? */

{{Justus2}}

This is a collection of howtos and convenient Slurm commands for JUSTUS 2.

Some commands behave slightly different depending on whether they are executed
by a system administrator or by a regular user, as Slurm prevents regular users from accessing critical system information and viewing job and usage information of other users.

= GENERAL INFORMATION =

== How to find a general quick start user guide? ==

https://slurm.schedmd.com/quickstart.html

== How to find Slurm FAQ? ==

https://slurm.schedmd.com/faq.html

== How to find a Slurm cheat sheet? ==

https://slurm.schedmd.com/pdfs/summary.pdf

== How to find Slurm tutorials? ==

https://slurm.schedmd.com/tutorials.html

== How to get more information on Slurm? ==

(Almost) every Slurm command has a man page. Use it.

Online versions: https://slurm.schedmd.com/man_index.html

== How to find hardware specific details about JUSTUS 2? ==

See our Wiki page: [[Hardware and Architecture (bwForCluster JUSTUS 2)|Hardware and Architecture]]

= JOB SUBMISSION =

== How to submit a serial batch job? ==

Use [https://slurm.schedmd.com/sbatch.html sbatch] command:

<pre>$ sbatch <job-script> </pre>

Sample job script template for serial job:

<source lang="bash">
#!/bin/bash
# Allocate one node
#SBATCH --nodes=1
# Number of program instances to be executed
#SBATCH --ntasks-per-node=1
# 8 GB memory required per node
#SBATCH --mem=8G
# Maximum run time of job
#SBATCH --time=1:00:00
# Give job a reasonable name
#SBATCH --job-name=serial_job
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=serial_job-%j.out
# File name for error output
#SBATCH --error=serial_job-%j.err

# Load software modules as needed, e.g.
# module load foo/bar

# Run serial program
./my_serial_program
</source>

Sample code for serial program: [[Media:Hello_serial.c | Hello_serial.c]]

'''Notes:'''
* --nodes=1 and --ntasks-per-node=1 may be replaced by --ntasks=1.
* If not specified, stdout and stderr are both written to slurm-%j.out.

== How to find working sample scripts for my program? ==

Most software modules for applications provide working sample batch scripts.
Check with [[Software_Modules_Lmod#Module_specific_help | module help]] command, e.g.
<pre>
$ module help chem/vasp # display module help for VASP
$ module help math/matlab # display module help for Matlab
</pre>

== How to harden job scripts against common errors? ==

The bash shell provides several options that support users in disclosing hidden bugs and writing safer job scripts.
In order to activate these safeguard settings users can insert the following lines in their scripts (after all #SBATCH directives):

<source lang="bash">
[...]
set -o errexit # (or set -e) cause batch script to exit immediately when a command fails.
set -o pipefail # cause batch script to exit immediately also when the command that failed is embedded in a pipeline
set -o nounset # (or set -u) causes the script to treat unset variables as an error and exit immediately
[...]
</source>

== How to submit an interactive job? ==

Use [https://slurm.schedmd.com/salloc.html salloc] command, e.g.:
<pre>$ salloc --nodes=1 --ntasks-per-node=8</pre>

'''Note:'''

In previous Slurm versions < 20.11 the use of [https://slurm.schedmd.com/srun.html srun] has been the recommended way for launching interactive jobs, e.g.:

<pre>$ srun --nodes=1 --ntasks-per-node=8 --pty bash </pre>

Although this still works with current Slurm versions this is considered '''deprecated ''' for current Slurm versions as it may cause issues when launching additional jobs steps from within the interactive job environment. Use [https://slurm.schedmd.com/salloc.html salloc] command.

== How to enable X11 forwarding for an interactive job? ==

Use '--x11' flag, e.g.

<pre>
$ salloc --nodes=1 --ntasks-per-node=8 --x11 # run shell with X11 forwarding enabled
</pre>

'''Note:'''
* For X11 forwarding to work, you must also enable X11 forwarding for your ssh login from your local computer to the cluster, i.e.:
<pre>local> ssh -X <username>@justus2.uni-ulm.de></pre>

== How to convert Moab batch job scripts to Slurm? ==

Replace Moab/Torque job specification flags and environment variables in your job
scripts by their corresponding Slurm counterparts.

'''Commonly used Moab job specification flags and their Slurm equivalents'''

{| width=750px class="wikitable"
! Option !! Moab (msub) !! Slurm (sbatch)
|-
| Script directive || #MSUB || #SBATCH
|-
| Job name || -N <name> || --job-name=<name> (-J <name>)
|-
| Account || -A <account> || --account=<account> (-A <account>)
|-
| Queue || -q <queue> || --partition=<partition> (-p <partition>)
|-
| Wall time limit || -l walltime=<hh:mm:ss> || --time=<hh:mm:ss> (-t <hh:mm:ss>)
|-
| Node count || -l nodes=<count> || --nodes=<count> (-N <count>)
|-
| Core count || -l procs=<count> || --ntasks=<count> (-n <count>)
|-
| Process count per node || -l ppn=<count> || --ntasks-per-node=<count>
|-
| Core count per process || || --cpus-per-task=<count> (-c <count>)
|-
| Memory limit per node || -l mem=<limit> || --mem=<limit>
|-
| Memory limit per process || -l pmem=<limit> || --mem-per-cpu=<limit>
|-
| Job array || -t <array indices> || --array=<indices> (-a <indices>)
|-
| Node exclusive job || -l naccesspolicy=singlejob || --exclusive
|-
| Initial working directory || -d <directory> (default: $HOME) || --chdir=<directory> (-D <directory>) (default: submission directory)
|-
| Standard output file || -o <file path> || --output=<file> (-o <file>)
|-
| Standard error file || -e <file path> || --error=<file> (-e <file>)
|-
| Combine stdout/stderr to stdout || -j oe || --output=<combined stdout/stderr file>
|-
| Mail notification events || -m <event> || --mail-type=<events> (valid types include: NONE, BEGIN, END, FAIL, ALL)
|-
| Export environment to job || -V || --export=ALL (default)
|-
| Don't export environment to job || (default) || --export=NONE
|-
| Export environment variables to job || -v <var[=value][,var2=value2[, ...]]> || --export=<var[=value][,var2=value2[,...]]>
|}

'''Notes:'''
* Default initial job working directory is $HOME for Moab. For Slurm the default working directory is where you submit your job from.
* By default Moab does not export any environment variables to the job's runtime environment. With Slurm most of the login environment variables are exported to your job's runtime environment. This includes environment variables from software modules that were loaded at job submission time (and also $HOSTNAME variable).

'''Commonly used Moab/Torque script environment variables and their Slurm equivalents

{| width=750px class="wikitable"
! Information !! Moab !! Torque !! Slurm
|-
| Job name || $MOAB_JOBNAME || $PBS_JOBNAME || $SLURM_JOB_NAME
|-
| Job ID || $MOAB_JOBID || $PBS_JOBID || $SLURM_JOB_ID
|-
| Submit directory || $MOAB_SUBMITDIR || $PBS_O_WORKDIR || $SLURM_SUBMIT_DIR
|-
| Number of nodes allocated || $MOAB_NODECOUNT || $PBS_NUM_NODES || $SLURM_JOB_NUM_NODES (and: $SLURM_NNODES)
|-
| Node list || $MOAB_NODELIST || cat $PBS_NODEFILE || $SLURM_JOB_NODELIST
|-
| Number of processes || $MOAB_PROCCOUNT || $PBS_TASKNUM || $SLURM_NTASKS
|-
| Requested tasks per node || --- || $PBS_NUM_PPN || $SLURM_NTASKS_PER_NODE
|-
| Requested CPUs per task || --- || --- || $SLURM_CPUS_PER_TASK
|-
| Job array index || $MOAB_JOBARRAYINDEX || $PBS_ARRAY_INDEX || $SLURM_ARRAY_TASK_ID
|-
| Job array range || $MOAB_JOBARRAYRANGE || - || $SLURM_ARRAY_TASK_COUNT
|-
| Queue name || $MOAB_CLASS || $PBS_QUEUE || $SLURM_JOB_PARTITION
|-
| QOS name || $MOAB_QOS || --- || $SLURM_JOB_QOS
|-
| Number of processes per node | --- || $PBS_NUM_PPN || $SLURM_TASKS_PER_NODE
|-
| Job user || $MOAB_USER || $PBS_O_LOGNAME || $SLURM_JOB_USER
|-
| Hostname || $MOAB_MACHINE || $PBS_O_HOST || $SLURMD_NODENAME
|}

'''Note:'''

* See [https://slurm.schedmd.com/sbatch.html sbatch] man page for a complete list of flags and environment variables.

== How to emulate Moab output file names? ==

Use the following directives:

<pre>
#SBATCH --output="%x.o%j"
#SBATCH --error="%x.e%j"
</pre>

== How to pass command line arguments to the job script? ==

Run <pre>$ sbatch <job-script> arg1 arg2 ...</pre>

Inside the job script the arguments can be accessed as $1, $2, ...

E.g.:

<source lang="bash">
[...]
infile="$1"
outfile="$2"
./my_serial_program < "$infile" > "$outfile" 2>&1
[...]
</source>

'''Notes:'''

* Do '''not''' use $1, $2, ... in "#SBATCH" lines. These parameters can be used only within the regular shell script.

== How to request local scratch (SSD/NVMe) at job submission? ==

Use '--gres=scratch:nnn' option to allocate nnn GB of local (i.e. node-local) scratch space for the entire job.

Example: '--gres=scratch:100' will allocate 100 GB scratch space on a locally attached NVMe device.

'''Notes:'''

* Do '''not''' add any unit (such as --gres=scratch:100G). This would be treated as requesting an amount of 10^9 * 100GB of scratch space.

* Multinode jobs get nnn GB of local scratch space on every node of the job.

* Environment variable '''$SCRATCH''' will point to
** /scratch/<user>.<jobid> when local scratch has been requested. This will be on locally attached SSD/NVMe devices.
** /tmp/<user>.<jobid> when no local scratch has been requested. This will be in memory and, thus, be limited in size.

* Environment variable '''$TMPDIR''' always points to /tmp/<user>.<jobid>. This will always be in memory and, thus, limited in size.

* For backward compatibility environment variable $RAMDISK always points to /tmp/<user>.<jobid>

* Scratch space allocation in /scratch will be enforced by quota limits

* Data written to $TMPDIR will always count against allocated memory.

* Data written to local scratch space will automatically be removed at the end of the job.

== How to request GPGPU nodes at job submission? ==

Use '--gres=gpu:<count>' option to allocate 1 or 2 GPUs per node for the entire job.

Example: '--gres=gpu:1' will allocate one GPU per node for this job.

'''Notes:'''

* GPGPU nodes are equipped with two Nvidia V100S cards

* Environment variables $CUDA_VISIBLE_DEVICES, $SLURM_JOB_GPUS and $GPU_DEVICE_ORDINAL will denote card(s) allocated for the job.

* CUDA Toolkit is available as software module devel/cuda.

== How to clean-up or save files before a job times out? ==

Possibly you would like to clean up the work directory or save intermediate result files in case a job times out.

The following sample script may serve as a blueprint for implementing a pre-termination function to perform clean-up or file recovery actions.

<source lang="bash">
#!/bin/bash
# Allocate one node
#SBATCH --nodes=1
# Number of program instances to be executed
#SBATCH --ntasks-per-node=1
# 2 GB memory required per node
#SBATCH --mem=2G
# Request 10 GB local scratch space
#SBATCH --gres=scratch:10
# Maximum run time of job
#SBATCH --time=10:00
# Send the USR1 signal 120 seconds before end of time limit
#SBATCH --signal=B:USR1@120
# Give job a reasonable name
#SBATCH --job-name=signal_job
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=signal_job-%j.out
# File name for error output
#SBATCH --error=signal_job-%j.err

# Define the signal handler function
# Note: This is not executed here, but rather when the associated
# signal is received by the shell.
finalize_job()
{
# Do whatever cleanup you want here. In this example we copy
# output file(s) back to $SLURM_SUBMIT_DIR, but you may implement
# your own job finalization code here.
echo "function finalize_job called at `date`"
cd $SCRATCH
mkdir -vp "$SLURM_SUBMIT_DIR"/results
tar czvf "$SLURM_SUBMIT_DIR"/results/${SLURM_JOB_ID}.tgz output*.txt
exit
}

# Call finalize_job function as soon as we receive USR1 signal
trap 'finalize_job' USR1

# Copy input files for this job to the scratch directory (if needed).
# Note: Environment variable $SCRATCH always points to a scratch directory
# automatically created for this job. Environment variable $SLURM_SUBMIT_DIR
# points to the path where this script was submitted from.
# Example:
# cp -v "$SLURM_SUBMIT_DIR"/input*.txt "$SCRATCH"

# Change working directory to local scratch directory
cd "$SCRATCH"

# Load software modules as needed, e.g.
# module load foo/bar

# This is where the actual work is done. In this case we just create
# a sample output file for 900 (=15*60) seconds, but since we asked
# Slurm for 600 seconds only it will not be able finish within this
# wall time.
# Note: It is important to run this task in the background
# by placing the & symbol at the end. Otherwise the signal handler
# would not be executed until that process has finished, which is not
# what we want.
(for i in `seq 15`; do echo "Hello World at `date +%H:%M:%S`."; sleep 60; done) >output.txt 2>&1 &

# Note: The command above is just for illustration. Normally you would just run
# my_program >output.txt 2>&1 &

# Tell the shell to wait for background task(s) to finish.
# Note: This is important because otherwise the parent shell
# (this script) would proceed (and terminate) without waiting for
# background task(s) to finish.
wait

# If we get here, the job did not time out but finished in time.

# Release user defined signal handler for USR1
trap - USR1

# Do regular cleanup and save files. In this example we simply call
# the same function that we defined as a signal handler above, but you
# may implement your own code here.
finalize_job

exit
</source>

'''Notes:'''

* The number of seconds specified in --signal option must match the runtime of the pre-termination function and must not exceed 65535 seconds.

* Due to the resolution of event handling by Slurm, the signal may be sent a little earlier than specified.

== How to submit a multithreaded batch job? ==

Sample job script template for a job running one multithreaded program instance:

<source lang="bash">
#!/bin/bash
# Allocate one node
#SBATCH --nodes=1
# Number of program instances to be executed
#SBATCH --ntasks-per-node=1
# Number of cores per program instance
#SBATCH --cpus-per-task=8
# 8 GB memory required per node
#SBATCH --mem=8G
# Maximum run time of job
#SBATCH --time=1:00:00
# Give job a reasonable name
#SBATCH --job-name=multithreaded_job
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=multithreaded_job-%j.out
# File name for error output
#SBATCH --error=multithreaded_job-%j.err

# Load software modules as needed, e.g.
# module load foo/bar

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
export MKL_NUM_THREADS=${SLURM_CPUS_PER_TASK}

# Run multithreaded program
./my_multithreaded_program
</source>

Sample code for multithreaded program: [[Media:Hello_openmp.c | Hello_openmp.c]]

'''Notes:'''

* In our configuration each physical core is considered a "CPU".

* On JUSTUS 2 it is recommended to specify a number of cores per task ('--cpus-per-task') that is either an integer divisor of 24 or (at most) 48.

* Required memory can also by specified per allocated CPU with '--mem-per-cpu' option.

* The '--mem' and '--mem-per-cpu' options are mutually exclusive.

== How to submit an array job? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_array -a] (or [https://slurm.schedmd.com/sbatch.html#OPT_array --array]) option, e.g.

<pre>$ sbatch -a 1-16%8 ...</pre>

This will submit 16 tasks to be executed, each one indexed by SLURM_ARRAY_TASK_ID ranging from 1 to 16, but will limit the number of simultaneously running tasks from this job array to 8.

Sample job script template for an array job:

<source lang="bash">
#!/bin/bash
# Number of cores per individual array task
#SBATCH --ntasks=1
#SBATCH --array=1-16%8
#SBATCH --mem=4G
#SBATCH --time=01:00:00
#SBATCH --job-name=array_job
#SBATCH --output=array_job-%A_%a.out
#SBATCH --error=array_job-%A_%a.err

# Load software modules as needed, e.g.
# module load foo/bar

# Print the task id.
echo "My SLURM_ARRAY_TASK_ID: " $SLURM_ARRAY_TASK_ID

# Add lines here to run your computations, e.g.
# ./my_program <input.$SLURM_ARRAY_TASK_ID

</source>

'''Notes:'''

* Placeholder %A will be replaced by the master job id, %a will be replaced by the array task id.

* Every sub job in an array job will have its own unique environment variable $SLURM_JOB_ID. Environment variable $SLURM_ARRAY_JOB_ID will be set to the first job array index value for all tasks.

* The remaining options in the sample job script are the same as the options used in other, non-array jobs. In the example above, we are requesting that each array task be allocated 1 CPU (--ntasks=1) and 4 GB of memory (--mem=4G) for up to one hour (--time=01:00:00).

* More information: https://slurm.schedmd.com/job_array.html

== How to delay the start of a job? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_begin -b] (or [https://slurm.schedmd.com/sbatch.html#OPT_begin --begin]) option in order to defer the allocation of the job until the specified time.

Examples:

<pre>
sbatch --begin=20:00 ... # job can start after 8 p.m.
sbatch --begin=now+1hour ... # job can start 1 hour after submission
sbatch --begin=teatime ... # job can start at teatime (4 p.m.)
sbatch --begin=2023-12-24T20:00:00 ... # job can start after specified date/time
</pre>

== How to submit dependency (chain) jobs? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_dependency -d] (or [https://slurm.schedmd.com/sbatch.html#OPT_dependency --dependency]) option, e.g.

<pre>$ sbatch -d afterany:123456 ...</pre>

This will defer the submitted job until the specified job 123456 has terminated.

Slurm supports a number of different dependency types, e.g.:

<pre>
-d after:123456 # job can begin execution after the specified job has begun execution
-d afterany:123456 # job can begin execution after the specified job has finished
-d afternotok:123456 # job can begin execution after the specified job has failed (exit code not equal zero)
-d afterok:123456 # job can begin execution after the specified job has successfully finished (exit code zero)
-d singleton # job can begin execution after any previously job with the same job name and user have finished
</pre>

'''Note:''' Multiple jobs can be specified by separating their job ids by colon characters (:), e.g.

<pre> $ sbatch -d afterany:123456:123457 ... </pre>

This will defer the submitted job until the specified jobs 123456 and 123457 have both finished.

== How to deal with invalid job dependencies? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_kill-on-invalid-dep --kill-on-invalid-dep=yes] option in order to automatically terminate jobs which can never run due to invalid dependencies. By default the job stays pending with reason 'DependencyNeverSatisfied' to allow review and appropriate action by the user.

'''Note:''' A job dependency may also become invalid if a job has been submitted with '-d afterok:<jobid>' but the specified dependency job has failed, e.g. because it timed out (i.e. exceeded its wall time limit).

== How to submit an MPI batch job? ==

Suggested reading: https://slurm.schedmd.com/mpi_guide.html

Sample job script template for an MPI job:

<source lang="bash">
#!/bin/bash
# Allocate two nodes
#SBATCH --nodes=2
# Number of program instances to be executed
#SBATCH --ntasks-per-node=48
# Allocate 32 GB memory per node
#SBATCH --mem=32gb
# Maximum run time of job
#SBATCH --time=1:00:00
# Give job a reasonable name
#SBATCH --job-name=mpi_job
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=mpi_job-%j.out
# File name for error output
#SBATCH --error=mpi_job-%j.err

# Add lines here to run your computations, e.g.
#
# Option 1: Lauch MPI tasks by using mpirun
#
# for OpenMPI and GNU compiler:
#
# module load compiler/gnu
# module load mpi/openmpi
# mpirun ./my_mpi_program
#
# for Intel MPI and Intel complier:
#
# module load compiler/intel
# module load mpi/impi
# mpirun ./my_mpi_program
#
# Option 2: Launch MPI tasks by using srun
#
# for OpenMPI and GNU compiler:
#
# module load compiler/gnu
# module load mpi/openmpi
# srun ./my_mpi_program
#
# for Intel MPI and Intel compiler:
#
module load compiler/intel
module load mpi/impi
srun ./my_mpi_program

</source>

Sample code for MPI program: [[Media:Hello_mpi.c | Hello_mpi.c]]

'''Notes'''

* SchedMD recommends to use srun and many (most?) sites do so as well. The rationale is that srun is more tightly integrated with the scheduler and provides more consistent and reliable resource tracking and accounting for individual jobs and job steps. mpirun may behave differently for different MPI implementations and versions. There are reports that claim "strange behavior" of mpirun especially when using task affinity and core binding. Using srun is supposed to resolve these issues and is therefore highly recommended.
* Do not run batch jobs that launch a large number (hundreds or thousands) short running (few minutes or less) MPI programs, e.g. from a shell loop. Every single MPI invocation does generate its own job step and sends remote procedure calls to the Slurm controller server. This can result in degradation of performance for both, Slurm and the application, especially if many of that jobs happen to run at the same time. Jobs of that kind can even get stuck without showing any further activity until hitting the wall time limit. For high throughput computing (e.g. processing a large number of files with every single task running independently from each other and very shortly), consider a more appropriate parallelization paradigm that invokes independent serial (non-MPI) processes in parallel at the same time. This approach is sometimes referred to as "[https://en.wikipedia.org/wiki/Embarrassingly_parallel pleasingly parallel]" workload. GNU Parallel is a shell tool that facilitates executing serial tasks in parallel. On JUSTUS 2 this tool is available as a software module "system/parallel".

== How to submit a hybrid MPI/OpenMP job? ==

Sample job script template for an hybrid job:

<source lang="bash">
#!/bin/bash
# Number of nodes to allocate
#SBATCH --nodes=4
# Number of MPI instances (ranks) to be executed per node
#SBATCH --ntasks-per-node=2
# Number of threads per MPI instance
#SBATCH --cpus-per-task=24
# Allocate 8 GB memory per node
#SBATCH --mem=8gb
# Maximum run time of job
#SBATCH --time=1:00:00
# Give job a reasonable name
#SBATCH --job-name=hybrid_job
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=hybrid_job-%j.out
# File name for error output
#SBATCH --error=hybrid_job-%j.err

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
export MKL_NUM_THREADS=${SLURM_CPUS_PER_TASK}

module load compiler/intel
module load mpi/impi
srun ./my_hybrid_program

# or:
# mpirun ./my_hybrid_program

</source>

Sample code for hybrid program: [[Media:Hello_hybrid.c | Hello_hybrid.c]]

'''Notes:'''

* $SLURM_CPUS_PER_TASK is only set if the '--cpus-per-task' option is specified.

== How to request specific node(s) at job submission? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_nodelist -w] (or [https://slurm.schedmd.com/sbatch.html#OPT_nodelist --nodelist]) option, e.g.:

<pre>$ sbatch -w <node1>,<node2> ...</pre>

Also see [https://slurm.schedmd.com/sbatch.html#OPT_nodefile -F] (or [https://slurm.schedmd.com/sbatch.html#OPT_nodefile --nodefile]) option.

== How to exclude specific nodes from job? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_exclude -x] (or [https://slurm.schedmd.com/sbatch.html#OPT_exclude --exclude]) option, e.g.:

<pre>$ sbatch -x <node1>,<node2> ...</pre>

== How to get exclusive jobs? ==

Use '--exclusive' option on job submission. This makes sure that there will be no other jobs running on your nodes. Very useful for benchmarking!

'''Note:'''

* --exclusive option does '''not''' mean that you automatically get full access to all the resources which the node might provide without explicitly requesting them.

== How to avoid sharing nodes with other users? ==

Use '--exclusive=user' option on job submission. This will still allow multiple jobs of one and the same user on the nodes.

''' Note:'''

* Depending on configuration, exclusive=user may (and probably will) be the default node access policy on JUSTUS 2.

== How to submit batch job without job script? ==

Use [https://slurm.schedmd.com/sbatch.html#OPT_wrap --wrap] option.

Example:

<pre>$ sbatch --nodes=2 --ntasks-per-node=16 --wrap "sleep 600"</pre>

'''Note:''' May be useful for testing purposes.

= JOB MONITORING AND CONTROL =

== How to prevent Slurm performance degradation? ==

Almost every invocation of a Slurm client command (e.g. squeue, sacct, sprio or sshare) sends a remote procedure call (RPC) to the Slurm control daemon and/or database.
If enough remote procedure calls come in at once, this can result in a degradation of performance of the Slurm services for all users, possibly resulting in a denial of service.

Therefore, '''do not run Slurm client commands that send remote procedure calls from loops in shell scripts or other programs''' (such as 'watch squeue'). Always ensure to limit calls to squeue, sstat, sacct etc. to the minimum necessary for the information you are trying to gather.

Slurm does collect RPC counts and timing statistics by message type and user for diagnostic purposes.

== How to view information about submitted jobs? ==

Use [https://slurm.schedmd.com/squeue.html squeue] command, e.g.:

<pre>
$ squeue # all jobs owned by user (all jobs owned by all users for admins)
$ squeue --me # all jobs owned by user (same as squeue for regular users)
$ squeue -u <username> # jobs of specific user
$ squeue -t PENDING # pending jobs only
$ squeue -t RUNNING # running jobs only
</pre>

'''Notes:'''

* The output format of [https://slurm.schedmd.com/squeue.html squeue] (and most other Slurm commands) is highly configurable to your needs. Look for the --format or --Format options.

* Every invocation of squeue sends a remote procedure call to the Slurm database server. '''Do not run squeue or other Slurm client commands from loops in shell scripts or other programs''' as this can result in a degradation of performance. Ensure that programs limit calls to squeue to the minimum necessary for the information you are trying to gather.

== How to cancel jobs? ==

Use [https://slurm.schedmd.com/scancel.html scancel] command, e.g.

<pre>
$ scancel <jobid> # cancel specific job
$ scancel <jobid>_<index> # cancel indexed job in a job array
$ scancel -u <username> # cancel all jobs of specific user
$ scancel -u <username> -i # cancel all jobs of specific user interactively
$ scancel -t PENDING # cancel pending jobs
$ scancel -t RUNNING # cancel running jobs
</pre>

== How to show job script of a running job? ==

Use [https://slurm.schedmd.com/scontrol.html scontrol] command:

<pre>
$ scontrol write batch_script <job_id> <file>
$ scontrol write batch_script <job_id> -
</pre>

* If file name is omitted default file name will be slurm-<job_id>.sh
* If file name is - (i.e. dash) job script will be written to stdout.

== How to get estimated start time of a job? ==

<pre>$ squeue --start</pre>

'''Notes:'''
* Estimated start times are dynamic and can change at any moment. Exact start times of individual jobs are usually unpredictable.
* Slurm will report N/A for the start time estimate if nodes are not currently being reserved by the scheduler for the job to run on.

== How to show remaining walltime of running jobs? ==

Use [https://slurm.schedmd.com/squeue.html squeue] with format option "%L", e.g.:

<pre> $ squeue -t r -o "%u %i %L" </pre>

== How to check priority of jobs? ==

Use [https://slurm.schedmd.com/squeue.html squeue] with format options "%Q" and/or "%p", e.g.:

<pre>
$ squeue -o "%8i %8u %15a %.10r %.10L %.5D %.10Q"
</pre>

Use [https://slurm.schedmd.com/sprio.html sprio] command to display the priority components (age/fairshare/...) for each job:

<pre>
$ sprio
</pre>

Use "[https://slurm.schedmd.com/sshare.html sshare] command for listing the shares of associations, e.g. accounts.

<pre>
$ sshare
</pre>

== How to prevent (hold) jobs from being scheduled for execution? ==

<pre>
$ scontrol hold <job_id>
</pre>

== How to unhold job? ==

<pre>
$ scontrol release <job_id>
</pre>

== How to suspend a running job? ==

<pre>
$ scontrol suspend <job_id>
</pre>

== How to resume a suspended job? ==

<pre>
$ scontrol resume <job_id>
</pre>

== How to requeue (cancel and resubmit) a particular job? ==

<pre>
$ scontrol requeue <job_id>
</pre>

== How to monitor resource usage of running job(s)? ==

Use "[https://slurm.schedmd.com/sstat.html sstat] command.

'sstat -e' command shows a list of fields that can be specified with the '--format' option.

Example:

<pre>
$ sstat --format=JobId,AveCPU,AveRSS,MaxRSS -j <jobid>
</pre>

This will show average CPU time, average and maximum memory consumption of all tasks in the running job.
Ideally, average CPU time equals the number of cores allocated for the job multiplied by the current run time of the job.
The maximum memory consumption gives an estimate of the peak amount of memory actually needed so far. This can be compared with the amount of memory requested for the job. Over-requesting memory can result in significant waste of compute resources.

'''Notes:'''

* Users can also ssh into compute nodes that they have one or more running jobs on. Once logged in, they can use standard Linux process monitoring tools like ps, (h)top, free, vmstat, iostat, du, ...

* Users can also attach an interactive shell under an already allocated job by running the following command: <pre>srun --jobid <job> --overlap --pty /bin/bash</pre> Once logged in, they can again use standard Linux process monitoring tools like ps, (h)top, free, vmstat, iostat, du, ... For a single node job the user does not even need to know the node that the job is running on. For a multinode job, the user can still use '-w <node>' option to specify a specific node.

== How to get detailed job information ==

<pre>
$ scontrol show job 1234 # For job id 1234
$ scontrol show jobs # For all jobs
$ scontrol -o show jobs # One line per job
</pre>

== How to modify a pending/running job? ==

Use

<pre>$ scontrol update JobId=<jobid> ...</pre>

E.g.: <pre>$ scontrol update JobId=42 TimeLimit=7-0</pre>

This will modify the time limit of the job to 7 days.

'''Note:''' Update requests for '''running''' jobs are mostly restricted to Slurm administrators. In particular, only an administrator can increase the TimeLimit of a job.

== How to show accounting data of completed job(s)? ==

Use [https://slurm.schedmd.com/sacct.html sacct] command.

'sacct -e' command shows a list of fields that can be
specified with the '--format' option.

== How to retrieve job history and accounting? ==

For a specific job:

<pre>
$ sacct -j <jobid> --format=User,JobID,Jobname,partition,state,time,start,end,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist
</pre>

For a specific user:

<pre>
$ sacct -u <user> --format=User,JobID,Jobname,partition,state,time,start,end,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist
</pre>

Note: Default time window is the current day.

Starting from a specific date:

<pre>
$ sacct -u <user> -S 2020-01-15 --format=User,JobID,Jobname,partition,state,time,start,end,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist
</pre>

Within a time window:

<pre>
$ sacct -u <user> -S 2020-01-15 -E 2020-01-31 --format=User,JobID,Jobname,partition,state,time,start,end,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist
</pre>

'''Note:'''

You can also set the environment variable $SACCT_FORMAT to specify the default format. To get a general idea of how efficiently a job utilized its resources, the following format can be used:

<pre>
export SACCT_FORMAT="JobID,JobName,Elapsed,NCPUs,TotalCPU,CPUTime,ReqMem,MaxRSS,MaxDiskRead,MaxDiskWrite,State,ExitCode"
</pre>

To find how efficiently the CPUs were used, divide TotalCPU by CPUTime. To find how efficiently memory were used, devide MaxRSS by ReqMem. But be aware that sacct memory usage measurement doesn't catch very rapid memory spikes. If your job got killed for running out of memory, it '''did run out of memory''' even if sacct reports a lower memory usage than would trigger an out-of-memory-kill. A job that reads or writes excessively to disk might be bogged down significantly by I/O operations.

== How to get efficiency information of completed job(s)? ==

Use <pre>$ seff <jobid> </pre> command for some brief information.

'''Note:''' It is good practice to have a look at the efficiency of your job(s) on completion '''and we expect you to do so'''. This way you can improve your job specifications in the future.

== How to get complete field values from sstat and sacct commands? ==

When using the [https://slurm.schedmd.com/sacct.html#OPT_format --format] option for listing various fields you can put a %NUMBER afterwards to specify how many characters should be printed.

E.g. '--format=User%30' will print 30 characters for the user name (right justified). A %-30 will print 30 characters left justified.

sstat and sacct also provide the '--parsable' and '--parsable2' option to always print full field values delimited with a pipe '|' character by default.
The delimiting character can be specified by using the '--delimiter' option, e.g. '--delimiter=","' for comma separated values.

== How to retrieve job records for all jobs running/pending at a certain point in time? ==

Use [https://slurm.schedmd.com/sacct.html sacct] with [https://slurm.schedmd.com/sacct.html#OPT_state -s <state>] and [https://slurm.schedmd.com/sacct.html#OPT_starttime -S <start time>] options, e.g.:

<pre>
$ sacct -n -a -X -S 2021-04-01T00:00:00 -s R -o JobID,User%15,Account%10,NCPUS,NNodes,NodeList%1500
</pre>

'''Note:''' When specifying the state "-s <state>" '''and''' the start time "-S <start time>", the default
time window will be set to end time "-E" equal to start time. Thus, you will get a snapshot of all running/pending
jobs at the instance given by "-S <start time>".

== How to get a parsable list of hostnames from $SLURM_JOB_NODELIST? ==

<pre>
$ scontrol show hostnames $SLURM_JOB_NODELIST
</pre>

= ADMINISTRATION =

'''Note:''' Most commands in this section are restricted to system administrators.

== How to stop Slurm from scheduling jobs? ==

You can stop Slurm from scheduling jobs on a per partition basis by
setting that partition's state to DOWN. Set its state UP to resume
scheduling. For example:

<pre>
$ scontrol update PartitionName=foo State=DOWN
$ scontrol update PartitionName=foo State=UP
</pre>

== How to print actual hardware configuration of a node? ==

<pre>
$ slurmd -C # print hardware configuration plus uptime;
$ slurmd -G # print generic resource configuration; to be run on compute nodes only
</pre>

== How to reboot (all) nodes as soon as they become idle? ==

<pre>
$ scontrol reboot ASAP nextstate=RESUME <node1>,<node2> # specific nodes
$ scontrol reboot ASAP nextstate=RESUME ALL # all nodes
</pre>

== How to cancel pending reboot of nodes? ==

<pre>
$ scontrol cancel_reboot <node1>,<node2>
</pre>

== How to check current node status? ==

<pre>
$ scontrol show node <node>
</pre>

== How to instruct all Slurm daemons to re-read the configuration file ==

<pre>
$ scontrol reconfigure
</pre>

== How to prevent a user from submitting new jobs? ==

Use the following [https://slurm.schedmd.com/sacctmgr.html sacctmgr] command:

<pre>
$ sacctmgr update user <username> set maxsubmitjobs=0
</pre>

'''Notes:'''

*Job submission is then rejected with the following message:
<pre>
$ sbatch job.slurm
sbatch: error: AssocMaxSubmitJobLimit
sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)
</pre>

* Use the following command to release the limit:
<pre>
$ sacctmgr update user <username> set maxsubmitjobs=-1
</pre>

== How to drain node(s)? ==

<pre>
$ scontrol update NodeName=<node1>,<node2> State=DRAIN Reason="Some Reason"
</pre>

Notes:

* Reason is mandatory.

* Do '''not''' just set state DOWN to drain nodes. This will kill any active jobs that may run on that nodes.

== How to list reason for nodes being drained or down? ==

<pre>
$ sinfo -R
</pre>

== How to resume node state? ==

<pre>
$ scontrol update NodeName=<node1>,<node2> State=RESUME
</pre>

== How to create a reservation on nodes? ==

Suggested reading: https://slurm.schedmd.com/reservations.html

<pre>
$ scontrol create reservation user=root starttime=now duration=UNLIMITED flags=maint,ignore_jobs nodes=ALL
$ scontrol create reservation user=root starttime=2020-12-24T17:00 duration=12:00:00 flags=maint,ignore_jobs nodes=<node1>,<node2>
$ scontrol show reservation
</pre>

'''Note:''' Add "FLEX" flag to allow jobs that qualify for the reservation to start before the reservation begins (and continue after it starts).
Add "MAGNETIC" flag to attract jobs that qualify for the reservation to run in that reservation without having requested it at submit time.

== How to create a floating reservation on nodes? ==

Use the flag "TIME_FLOAT" and a start time that is relative to the current time (use the keyword "now").
In the example below, the nodes are prevented from starting any jobs exceeding a walltime of 2 days.

<pre>
$ scontrol create reservation user=root starttime=now+2days duration=UNLIMITED flags=maint,ignore_jobs,time_float nodes=<node1>,<node2>
</pre>

'''Note:''' Floating reservation are not intended to run jobs, but to prevent long running jobs from being initiated on specific nodes. Attempts by users to make use of a floating reservation will be rejected. When ready to perform the maintenance, place the nodes in DRAIN state and delete the reservation.

== How to use a reservation? ==

<pre>
$ sbatch --reservation=foo_6 ... script.slurm
</pre>

== How to delete a reservation? ==

<pre>
$ scontrol delete ReservationName=foo_6
</pre>

== How to get node oriented information similar to 'mdiag -n'? ==

<pre>
$ sinfo -N -l
</pre>

Fields can be individually customized. See [https://slurm.schedmd.com/sinfo.html sinfo] man page. For example:

<pre>
$ sinfo -N --format="%8N %12P %.4C %.8O %.6m %.6e %.8T %.20E"

NODELIST PARTITION CPUS CPU_LOAD MEMORY FREE_M STATE REASON
n0001 standard* 0/16 0.01 128000 120445 idle none
n0002 standard* 0/16 0.01 128000 120438 idle none
n0003 standard* 0/0/ N/A 128000 N/A down* Not responding
</pre>

== How to get node oriented information similar to 'pbsnodes'? ==

<pre>
$ scontrol show nodes # One paragraph per node (all nodes)
$ scontrol show nodes <node1>,<node2> # One paragraph per node (specified nodes)
$ scontrol -o show nodes # One line per node (all nodes)
$ scontrol -o show nodes <node1>,<node2> # One line per node (specified nodes)
</pre>

== How to update multiple jobs of a user with a single scontrol command? ==

Not possible. But you can e.g. use squeue to build the script taking
advantage of its filtering and formatting options.

<pre>
$ squeue -tpd -h -o "scontrol update jobid=%i priority=1000" >my.script
</pre>

You can also identify the list of jobs and add them to the JobID all at once, for example:

<pre>
$ scontrol update JobID=123 qos=reallylargeqos
$ scontrol update JobID=123,456,789 qos=reallylargeqos
$ scontrol update JobID=[123-400],[500-600] qos=reallylargeqos
</pre>

Another option is to use the JobName, if all the jobs have the same name.

<pre>
$ scontrol update JobName="foobar" UserID=johndoe qos=reallylargeqos
</pre>

However, Slurm does not allow the UserID filter alone.

== How to create a new account? ==

Add account at top level in association tree:

<pre>
$ sacctmgr add account <accountname> Cluster=justus Description="Account description" Organization="none"
</pre>

Add account as child of some parent account in association tree:

<pre>
$ sacctmgr add account <accountname> parent=<parent_accountname>
</pre>

== How to move account to another parent? ==

<pre>
$ sacctmgr modify account name=<accountname> set parent=<new_parent_accountname>
</pre>

== How to delete an account? ==

<pre>
$ sacctmgr delete account name=<accountname>
</pre>

== How to add a new user? ==

<pre>
$ sacctmgr add user <username> DefaultAccount=<accountname>
</pre>

== How to add/remove users from an account? ==

<pre>
$ sacctmgr add user <username> account=<accountname> # Add user to account
$ sacctmgr add user <username> account=<accountname2> # Add user to a second account
$ sacctmgr remove user <username> where account=<accountname> # Remove user from this account
</pre>

== How to change default account of a user? ==

<pre>
$ sacctmgr modify user where user=<username> set DefaultAccount=<default_account>
</pre>

'''Note:''' The user must already be associated with the account you want to set as default.

== How to show account information? ==

<pre>
$ sacctmgr show assoc
$ sacctmgr show assoc tree
</pre>

'''Note:'''
* On JUSTUS 2 registered compute projects ("Rechenvorhaben") are uniquely mapped to Slurm accounts of the same name.

== How to implement user resource throttling policies? ==

Quoting from https://bugs.schedmd.com/show_bug.cgi?id=3600#c4

With Slurm, the associations are meant to establish base limits on the
defined partitions, accounts and users. Because limits propagate down
through the association tree, you only need to define limits at a high
level and those limits will be applied to all partitions, accounts and
users that are below it (parent to child). You can also override those
high level (parent) limits by explicitly setting different limits at
any lower level (on the child). So using the association tree is the
best way to get some base limits applied that you want for most cases.
QOS's are meant to override any of those base limits for exceptional
cases. Like Maui, you can use QOS's to set a different priority.
Again, the QOS would be overriding the base priority that could be set
in the associations.

== How to set a resource limit for an individual user? ==

Suggested reading: https://slurm.schedmd.com/resource_limits.html

Example:

<pre>
$ sacctmgr modify user <username> set maxjobs=1 # Limit maximum number of running jobs for user
$ sacctmgr list assoc user=<username> format=user,maxjobs # Show that limit
$ sacctmgr modify user <username> set maxjobs=-1 # Remove that limit
</pre>

== How to retrieve historical resource usage for a specific user or account ==

Use [https://slurm.schedmd.com/sreport.html sreport] command.

Examples:
<pre>
$ sreport cluster UserUtilizationByAccount Start=2021-01-01 End=2021-12-31 -t Hours user=<username> # Report cluster utilization of given user broken down by accounts
$ sreport cluster AccountUtilizationByUser Start=2021-01-01 End=2021-12-31 -t Hours account=<account> # Report cluster utilization of given account broken down by users
</pre>

'''Notes:'''
* By default CPU resources will be reported. Use '-T' option for other trackable resources, e.g. '-T cpu,mem,gres/gpu,gres/scratch'.
* On JUSTUS 2 registered compute projects ("Rechenvorhaben") are uniquely mapped to Slurm accounts of the same name. Thus, 'AccountUtilizationByUser' can also be used to report the aggregated cluster utilization of compute projects.
* Can be executed by regular users as well in which case Slurm will only report their own usage records (but along with the total usage of the associated account in the case of 'AccountUtilizationByUser').

== How to fix/reset a user's RawUsage value? ==

<pre>
$ sacctmgr modify user <username> where Account=<account> set RawUsage=<number>
</pre>

== How to create/modify/delete QOSes? ==

Suggested reading: https://slurm.schedmd.com/qos.html

Examples:

<pre>
$ sacctmgr show qos # Show existing QOSes
$ sacctmgr add qos verylong # Create new QOS verylong
$ sacctmgr modify qos verylong set MaxWall=28-00:00:00 # Set maximum walltime limit
$ sacctmgr modify qos verylong set MaxTRESPerUser=cpu=4 # Set maximum maximum number of CPUS a user can allocate at a given time
$ sacctmgr modify qos verylong set flags=denyonlimit # Prevent submission if job requests exceed any limits of QOS
$ sacctmgr modify user <username> set qos+=verylong # Add a QOS to a user account
$ sacctmgr modify user <username> set qos-=verylong # Remove a QOS from a user account
$ sacctmgr delete qos verylong # Delete that QOS
</pre>

== How to find (and fix) runaway jobs? ==

<pre>$ sacctmgr show runaway</pre>

'''Notes:'''
* Runaway jobs are orphaned jobs that don't exist in the Slurm controller but have a start and no end time in the Slurm data base. Runaway jobs mess with accounting and affects new jobs of users who have too many runaway jobs.
* If there are jobs in this state this command will also provide an option to fix them. This will set the end time for each job to the latest out of the start, eligible, or submit times, and set the state to completed.

== How to show a history of database transactions? ==

<pre>$ sacctmgr list transactions</pre>

'''Note:''' Useful to get timestamps for when a user/account/qos has been created/modified/removed etc.

When to use an HPC Cluster

2026-09-14T11:32:21Z

K Siegmund: /* When not to use HPC computers */

I have calculations to do - should I try using one of the bwHPC clusters for my tasks?

== When to use HPC computers ==

You can benefit from an HPC-Cluster if:

# You can run many separate calculations in parallel
# Software can efficiently divide your problem into many smaller problems that run in parallel
# You need more RAM (memory) than your computer can provide
# You need more GPUs and have no other source
# You need to process large amounts of data that do not fit on your computer

== When '''not''' to use HPC computers ==

You will probably not have a big benefit from using HPC when:

* Your calculation runs in serial (only on one compute core is used, things cannot run in parallel) and you have few calculations
* Your workflow requires that you run one calculation, then retrieve data to analyze it locally, then run the next calculation
* You have different computational needs than raw computing power.

Except 3) or 4) of "when to use a cluster" force you to use the clusters anyway.

If you need other computational tasks of the likes of

* hosting a web server
* hosting a database
* having a test bed for software development
* having systems for university courses on linux, programming languages, etc
* having an environment for student projects or theses,

then the bwCloud offering virtual machines might be the right thing for you

'''→ https://bwcloud-os.de/'''

== Misconceptions ==
=== "Everything will be faster" ===
[[File:BwUniCluster_2.0_Feb2020_1024x423.jpg|right|thumb|alt=bwUniCluster2.0 |upright=1| many compute cores: Every "tray" in these racks contains 1-4 computers, every computer typically has 50-200 compute cores. ]]

The performance of a single compute core is not higher than that of your typical desktop computer. The main things we can offer are:

* Many compute cores
* Much RAM (memory)
* GPUs

=== "It's just a computer, I know how to use a computer" ===

While the clusters are indeed "just" many Linux computers, they are also multi-million Euro instruments. So while they may have many similarities with a Linux desktop, in some respects, it is better to think of them like of other expensive instruments you may use in your research. HPC clusters have hardware components you may have never heard of in the consumer market, they typically have more than one CPU socket (making among other things memory management more complicated), etc. etc. etc.

=== "I want to host servers on bwHPC" ===

On the bwHPC clusters, you submit jobs to a supercomputer that get scheduled and processed over time. It is not the place for permanent server hosting. If you need self-service virtual machines, check out [https://bwcloud-os.de/en/ bwCloud-OS], which offers an OpenStack-based [https://en.wikipedia.org/wiki/Infrastructure_as_a_service IaaS] platform for research, teaching, and administration in Baden-Württemberg.

== What are the Costs? ==

=== Monetary ===

No cost to you as the end-user

(But of course the HPC systems are bought for millions of Euro and the power cost of running calculations is in the same order of magnitude as buying the systems)

=== Effort / Time ===

There is quite a learning curve to start calculating.

* Linux shell knowledge is a major part
* HPC-specific knowledge:
** Software module system - software supplied by the cluster maintainers (vide infra)
** How to use the scheduler / write job scripts: how to send calculations to a computer on the cluster (vide infra)
* Wait times: a good cluster is always busy. Expect a waiting time betwween 1h and 2 days until your calculation(s) start

=== Compute Workflow ===

A short description of the workflow how running calculation works can be found under [[Running Calculations]] and can give you a general idea.

=== Software ===

Basic usage is very simple: you run <code> module load module_name </code> and use the software, but important documentation and examples are also built into the modules.

The usage of the software on the cluster is described in [[Environment Modules]]

== Still With Us? ==

If you feel your calculations meet the requirements and you will save a lot of time despite some learning overhead, proceed to the [[Registration]] page.

When to use an HPC Cluster

2026-09-14T11:31:58Z

K Siegmund: /* When not to use HPC computers */

I have calculations to do - should I try using one of the bwHPC clusters for my tasks?

== When to use HPC computers ==

You can benefit from an HPC-Cluster if:

# You can run many separate calculations in parallel
# Software can efficiently divide your problem into many smaller problems that run in parallel
# You need more RAM (memory) than your computer can provide
# You need more GPUs and have no other source
# You need to process large amounts of data that do not fit on your computer

== When '''not''' to use HPC computers ==

You will probably not have a big benefit from using HPC when:

* Your calculation runs in serial (only on one compute core is used, things cannot run in parallel) and you have few calculations
* Your workflow requires that you run one calculation, then retrieve data to analyze it locally, then run the next calculation
* You have different computational needs than raw computing power.

Except 3) or 4) of "when to use a cluster" force you to use the clusters anyway.

If you need other computational tasks of the likes of

* hosting a web server
* hosting a database
* having a test bed for software development
* having systems for university courses on linux, programming languages, etc
* having an environment for student projects or theses,

then the bwCloud offering virtual machines might be the right thing for you

```→ https://bwcloud-os.de/```

== Misconceptions ==
=== "Everything will be faster" ===
[[File:BwUniCluster_2.0_Feb2020_1024x423.jpg|right|thumb|alt=bwUniCluster2.0 |upright=1| many compute cores: Every "tray" in these racks contains 1-4 computers, every computer typically has 50-200 compute cores. ]]

The performance of a single compute core is not higher than that of your typical desktop computer. The main things we can offer are:

* Many compute cores
* Much RAM (memory)
* GPUs

=== "It's just a computer, I know how to use a computer" ===

While the clusters are indeed "just" many Linux computers, they are also multi-million Euro instruments. So while they may have many similarities with a Linux desktop, in some respects, it is better to think of them like of other expensive instruments you may use in your research. HPC clusters have hardware components you may have never heard of in the consumer market, they typically have more than one CPU socket (making among other things memory management more complicated), etc. etc. etc.

=== "I want to host servers on bwHPC" ===

On the bwHPC clusters, you submit jobs to a supercomputer that get scheduled and processed over time. It is not the place for permanent server hosting. If you need self-service virtual machines, check out [https://bwcloud-os.de/en/ bwCloud-OS], which offers an OpenStack-based [https://en.wikipedia.org/wiki/Infrastructure_as_a_service IaaS] platform for research, teaching, and administration in Baden-Württemberg.

== What are the Costs? ==

=== Monetary ===

No cost to you as the end-user

(But of course the HPC systems are bought for millions of Euro and the power cost of running calculations is in the same order of magnitude as buying the systems)

=== Effort / Time ===

There is quite a learning curve to start calculating.

* Linux shell knowledge is a major part
* HPC-specific knowledge:
** Software module system - software supplied by the cluster maintainers (vide infra)
** How to use the scheduler / write job scripts: how to send calculations to a computer on the cluster (vide infra)
* Wait times: a good cluster is always busy. Expect a waiting time betwween 1h and 2 days until your calculation(s) start

=== Compute Workflow ===

A short description of the workflow how running calculation works can be found under [[Running Calculations]] and can give you a general idea.

=== Software ===

Basic usage is very simple: you run <code> module load module_name </code> and use the software, but important documentation and examples are also built into the modules.

The usage of the software on the cluster is described in [[Environment Modules]]

== Still With Us? ==

If you feel your calculations meet the requirements and you will save a lot of time despite some learning overhead, proceed to the [[Registration]] page.

Registration/bwForCluster/RV

2026-09-01T14:04:56Z

K Siegmund: /* Step B: Apply for a Rechenvorhaben/Project */

= Step B: Apply for a Rechenvorhaben/Project =

'''Rechenvorhaben/Projects (RV)''' are required for assignment to a bwForCluster.
[[File:RV-my.png|right|frame|My RVs.]]

To get access to an RV, answer the following question:
* Does your work group (or the collaborative project that you are part of) already have an RV?
** '''Yes → [[#Become_Coworker_of_an_RV|Become Coworker of an RV]]'''
** '''No → [[#Register_a_new_RV|Have the group register a new RV]]'''

You can see your RV memberships at the central application site ZAS under [https://zas.bwhpc.de/shib/en/info_rv.php '''My RVs'''].

== Become Coworker of an RV ==

<div style="border: 3px solid #28a745; padding: 15px; background-color: #d4edda; margin: 10px 0;">
You can only sign up for an RV once.
If you are an inactive member of an RV, either because you de-registered yourself or a manager deactivated you, only an RV manager can reactivate you.
</div>

The "RV responsible" will provide you with the following data on the RV:
* RV acronym
* RV password

If this information is lost, please let the "RV responsible" [[Registration/bwForCluster/RV/Management#Renew_RV_Password|renew the password]] for the RV.

To become coworker of an RV, follow these steps:
# Please visit the [https://zas.bwhpc.de/shib/en/bwforcluster_collaboration.php '''RV collaboration'''] web form at the [https://zas.bwhpc.de ZAS].
# Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.
# You will be directed to the ''Identity Provider'' of your home organization. Enter the username and password of your '''home organization''' (usually these credentials are also used for other services like email) and click '''Login/Einloggen'''.
# You will be redirected back to the RV collaboration form '''https://zas.bwhpc.de/shib/en/bwforcluster_collaboration.php'''.
# Enter '''RV Acronym''', '''RV Password''' and '''Institute''' into the form and check both "Terms of Use". Click '''Send'''.
# You will be assigned to the RV as a new member.
<div style="padding-left:18px;">
[[Image:Attention.svg|left|15px]]
In case of the <span style="color:red;">error</span> "This project is not currently renewed", please ask the RV responsible to apply for a prolongation of the RV. As soon as the prolongation was accepted, you can join the RV.
</div>

After submitting the request you will receive an email from ZAS about the further steps.
RV owner and RV managers will be notified automatically.

== Register a new RV ==
<div style="border: 3px solid #28a745; padding: 15px; background-color: #d4edda; margin: 10px 0;">
* An RV is valid for '''one year'''. After that, the RV responsible must apply for an extension of the RV for it to be renewed for another year.
* '''Creating multiple RVs on the same cluster is not necessary.''' Being part of multiple RVs on the same cluster does not grant any additional computing resources. The resource specifications (e.g. CPU hours) in an RV are used solely for cluster (tier) routing — to help assign your project to the most suitable cluster (tier). '''They do not represent a budget that is consumed, and they do not impose any limits on your usage.''' Every cluster user has full access to all available cluster resources regardless of how many RVs they belong to.
</div>

An RV should be registered by a leader or senior scientist of a research group/collaboration.

You must:
* be able to be held accountable for the coworkers in the RV (and will be)
* manage access to the RV (add and remove/inactivate RV members) or delegate the responsibility to an RV manager
* help write the reports required by the DFG for their funding of bwForClusters
* make sure publications that used bwHPC resources properly cite the use in the [[acknowledgement]]
* report publications which used bwHPC resources at least once a year
* be willing and able to support a future DFG grant proposal for a new bwForCluster with a "wissenschaftliches Beiblatt".
* ensure that research results of all co-workers are transferred from the cluster in a timely manner, especially before a co-worker leaves and loses access.

'''Some hints:'''
* Any number of coworkers can join the RV to work with the cluster. They need to be provided with the RV password and acronym.
* There is no need to register more than one RV per group. Just include all intended compute activities in the abstract to one RV.
* Research topic abstracts are updated when the RV is renewed for another year. Please include new or changed compute activities at that time.
* To be notified about important issues via the cluster's mailing list, it is necessary to finish all three registration steps. If you don't plan to work with the cluster yourself, we recommend to either register nevertheless or ask a colleague who is going to use the cluster to keep you updated. For better transparency, that colleague could receive the ''RV manager'' role from you.
* The [https://www.bwhpc.de/bwhpc-domains.php research area] of the group as well as the software and hardware requirements are taken into account.
*: Please look carefully through the scientific fields and research domains to find the field that fits best to your research focus and experience. This way, you'll be assigned to the cluster that can support you best. If in doubt, you can check the [https://www.dfg.de/resource/blob/175334/89ba4a3464c99aaea40fdef47367e7b2/fachsystematik-2020-2024-de-grafik-data.pdf DFG classification] as well.

* Getting access to two different clusters is only possible if specific software or hardware needs demand access for both.

[[File:RV-reg.png|right|frame|Register a new RV.]]
[[File:BwIDM-login.png|right|250px|thumb|Select your home organization]]
'''To apply for a new RV, follow these steps:'''

# Please visit the [https://zas.bwhpc.de/shib/en/bwforcluster_proposal.php '''RV registration'''] web form at the [https://zas.bwhpc.de ZAS].
# Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.
# You will be directed to the ''Identity Provider'' of your home organization. Enter the username and password of your '''home organization''' (usually these credentials are also used for other services like email) and click '''Login/Einloggen'''.
# You will be redirected back to the RV registration form '''https://zas.bwhpc.de/shib/en/bwforcluster_proposal.php'''.
# Fill in all mandatory fields of the form and tick the two "Terms of use". Detailed information on the individual fields can be found in the section [[Registration/bwForCluster/RV#Registration_Form_Details|Registration Form Details]]. Click '''Preview'''.
# If the information is correct, click '''Submit''', otherwise you can customize your application.
# Go to section [[Registration/bwForCluster/RV#Registration_Response|Registration Response]] for the next steps.

Please note that for your convenience you can also switch to the [https://zas.bwhpc.de/shib/bwforcluster_proposal.php German version] of the web form.
The submitter of the RV will be assigned the role "RV responsible" (see [[Registration/bwForCluster/RV/Management|Roles in an RV]]).

=== Registration Form Details ===

The web form consists of the following fields.
Note that the fields - name, given name, organization, mail, and EPPN - are auto-filled and can not be changed.
These are your credentials as given by your home organization.

{| style="width:100%;border:3px solid darkgray; margin: 1em auto 1em auto;"
|-
! style="width:20%;padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;"| Field
! style="padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;" | Explanation
|rowspan="15"| [[File:RV-form.png|center|400px|thumb|Registering a new RV.]]
|-
!scope="column" {{Lightgray_small}}| RV Title
|scope="column" {{Tab element}} | Define a short title of your planned compute activities, maximum of 255 characters including spaces.
|-
!scope="column" {{Lightgray_small}}| RV Description
|scope="column" {{Tab element}} | Write a short abstract about your planned compute activities (maximum of 2048 characters including spaces). Research groups that contributed to the DFG grant proposal (Art. 91b GG) of the corresponding bwForCluster only need to give reference to that particular proposal.
|-
!scope="column" {{Lightgray_small}}| Scientific field
|scope="column" {{Tab element}} | Tick one or several scientific fields. Once all bwForClusters are up and running the full list of given scientific fields are supported and hence for RV applicable. If your RV does not match any given scientific field, please state your scientific field in the given text box. Grayed out scientific fields indicate that the corresponding bwForCluster(s) is/are not operational yet.
|-
!scope="column" {{Lightgray_small}}| Field of activity
|scope="column" {{Tab element}} | Define if your RV is primarily for research and/or teaching. If not applicable, use text box.
|-
!scope="column" {{Lightgray_small}}| Parallel paradigm
|scope="column" {{Tab element}} | State what parallel paradigm your code or application uses. Multiple ticks are allowed. Further information can be provided via text box. If you are not sure about it, please state the software you are using.
|-
!scope="column" {{Lightgray_small}}| Programming language
|scope="column" {{Tab element}} | State the programming language(s) of your code or application.
|-
!scope="column" {{Lightgray_small}}| Numerical methods
|scope="column" {{Tab element}} | State the numerical or "calculation" methods your code or application utilizes. If you do not know, write "unknown".
|-
!scope="column" {{Lightgray_small}}| Software packages
|scope="column" {{Tab element}} | State all software package you will or want to use for your RV. Also include compiler, MPI and numerical packages (e.g. MKL, FFTW).
|-
!scope="column" {{Lightgray_small}}| Estimated requested computing resources
|scope="column" {{Tab element}} | Roughly estimate how many CPU hours are required to finish your RV. To calculate "CPU hours" multiply "elapsed parallel computing time" with "number of CPU core" per job.
''Example: Your code uses 4 CPU cores and has a wall time of 1h. This makes 4 CPU hours.''
|-
!scope="column" {{Lightgray_small}}| Planned maximum number of parallel used CPU cores per job
|scope="column" {{Tab element}} | Give an upper limit of how many CPU cores per job are required. Please cross-check with your statement about parallel paradigm.
''Obviously, a sequential code can only use 1 CPU core per job.''
|-
!scope="column" {{Lightgray_small}}| Estimated maximal memory requirements per CPU core
|scope="column" {{Tab element}} | Give an upper limit of how much RAM per CPU core your jobs require. Please give a value in Gigabytes.
|-
!scope="column" {{Lightgray_small}}| Estimated requested persistent disk space for the RV
|scope="column" {{Tab element}} | State how much disk space in total you need for your RV. Please give a value in Gigabytes.
''Example: If your RV has 4 more coworker and each of you produces until the end of your RV 20 Gigabyte of output, your maximum disk space is 100 Gigabyte.''
|-
!scope="column" {{Lightgray_small}}| Estimated maximal temporary disk space per job
|scope="column" {{Tab element}} | State how much space your intermediate files require during a job run. Please give a value in Gigabytes.
''Example: Your final output of your job is 0.1 Gigabyte, but during your job 10 temporary files, each with a size of 1 Gigabyte, are written to disk, hence your correct answer is 10 Gigabyte.''
|-
!scope="column" {{Lightgray_small}}| Institute
|scope="column" {{Tab element}} | State the full name of your Institute.
|-
|}

=== Registration Response ===

After submitting you will get an email from the ZAS confirming your submission.
With this email you are given an unique
* RV acronym
* RV password

Please keep the password enclosed.
With RV acronym and password, [[Registration/bwForCluster/RV#Become_Coworker_of_a_RV| coworkers ]] can already associate themselves to the RV.
If this information is lost, please [[Registration/bwForCluster/RV#Renew_RV_Password|renew the password]] for the RV.

The cluster assignment team assigns a bwForCluster to your RV based on your submitted data, including the [https://www.bwhpc.de/bwhpc-domains.php research area] as well as the software and hardware requirements.
The ZAS notifies you in an email about your assigned bwForCluster and provides website details for [[Registration/bwForCluster/Service| Step C]]. Later on you can find details about the roles and management possibilities at the [[Registration/bwForCluster/RV/Management | RV Management]] page.

=== Duration of an RV ===

An RV is valid for '''one year'''.
After that, the RV responsible must apply for an extension of the RV for it to be renewed for another year.

=== RV Management for RV Responsibles and Managers ===

RV responsible or manager can [[Registration/bwForCluster/RV/Management|manage and modify Rechenvorhaben/projects]].

----

<p style="text-align:right;">[[Registration/bwForCluster/Service | Go to step C]]</p>

Justus3

2026-08-31T08:16:21Z

K Siegmund:

{{Template:Justus3}}



The '''bwForCluster Justus 3''' is the state-wide high-performance compute cluster dedicated to '''Computational Chemistry and Quantum Sciences''' in Baden-Württemberg.

{| style=" background:#FEF4AB; width:100%;"
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News
|-
|
* 2026-06-16: Cooling installations for the new cluster has been enabled
|}
{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support
|-
|
* [[Justus3/Migration_from_J2|Migration Guide from JUSTUS2]]
----
* [[Justus3/Getting Started|Getting Started]]
* Online Tutorial [https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC Introduction to Linux in HPC] (external link)
* Contact and Support: [[BwSupportPortal]]
* Send [[Feedback|Feedback]] about Wiki pages
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | User Documentation
|-
|
* [[Registration/bwForCluster|Registration]]
* [[Justus3/Policy Agreement|Policy Agreement]]

* [[Justus3/Login|Login]]
* [[Justus3/System Description|System Description]]
* [[Justus3/File Systems|File Systems]]

* [[Justus3/Software|Software]]
* [[Justus3/Jobscripts: Running Your Calculations|Jobscripts: Running Your Calculations]]
** [[bwForCluster Justus 3 Slurm HOWTO|Slurm HOWTO]]
* [[Justus3/Visualization|Visualisation]]
* [[Development]]
|}

{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Cluster Funding
|-
|
* Please '''[[Justus3/Acknowledgement|acknowledge]]''' the Justus 3 cluster in your publications.
|}

Registration/bwUniCluster

2026-08-26T08:08:15Z

K Siegmund:

= Registration bwUniCluster =

The bwUniCluster is the universal cluster for tier 3 high performance computing (HPC) in Baden-Württemberg.
It is co-financed by the Ministry of Science, Research and the Arts Baden-Württemberg and the shareholders.

All members of the shareholder universities can apply for an account:

* Universität Freiburg
* Universität Heidelberg
* Universität Hohenheim
* Karlsruhe Institute of Technology (KIT)
* Universität Konstanz
* Universität Mannheim
* Universität Stuttgart
* Universität Tübingen
* Universität Ulm
* [[Registration/HAW|HAW BW e.V.]] (an association of several universities of applied sciences in Baden-Württemberg)

The use of the bwUniCluster is free of charge.

== Video Guide ==

Video guide through the steps at https://av.tib.eu/media/73017

== Three Steps for Registration ==

[[File:bwUniCluster-Registration.png|thumb|right|upright=1.3|bwUniCluster Registration Process]]

The registration process for the bwUniCluster is divided into three steps:

* Step A: You need to get the '''bwUniCluster Entitlement''' from your university/college.<br />→ '''[[Registration/bwUniCluster/Entitlement|Step A: bwUniCluster Entitlement]]
* Step B: You need to fill out the '''bwUniCluster questionnaire'''.<br />→ '''[[Registration/bwUniCluster/Questionnaire|Step B: bwUniCluster Questionnaire]]

* Step C: You need to '''register for the service bwUniCluster''' (create an account on the cluster).<br />→ '''[[Registration/bwUniCluster/Service|Step C: bwUniCluster Registration]]

After finishing Step C, you are done with registration.

* To '''login''' to the bwUniCluster, see our <br /> → '''[[Registration/Login|Login Guide]]'''

== Information for registered users ==

* Please refer to cluster-specific pages for '''[[bwUniCluster3.0]]''' (also accessible via the overview links on the left).
* If you want to check if your '''account''' is still valid on the bwUniCluster, please refer to the general <br /> → '''[[Registration/Account|Account Guide]]'''
* If you want to '''create a second factor''', please refer to <br /> → '''[[Registration/2FA|Generate a Second Factor (2FA)]]'''
* If you need to '''change or forgot your password''' for the bwUniCluster, please refer to the general <br /> → '''[[Registration/Password|Password Guide]]'''
* If you want to '''use SSH keys''' on the bwUniCluster, please refer to <br /> → '''[[Registration/SSH|Registering SSH Keys with your Cluster]]'''
* If you want do '''de-register your user account''' from the bwUniCluster, please refer to the general <br />→ '''[[Registration/Deregistration|De-registration Guide]]'''

== Contact / Support ==

If you have questions or problems concerning the bwUniCluster registration, please [[BwUniCluster3.0/Support|contact your local hotline]].

Environment Modules

2026-08-25T13:54:13Z

K Siegmund:

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Info.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
External software search over all bwHPC clusters on our webpage: https://www.bwhpc.de/software.html
|}

Software on the bwHPC Clusters is provided as '''Software Environment Modules''', or short '''Modules'''.

Modules make it possible to have different versions of a software installed at a the same time.
The complete environments for the software package, compilers and libraries and needed by this specific version is then loaded by a single command. This happens usually in the beginning of the jobscript.

Full documentation: <code>module help</code> and [https://lmod.readthedocs.io/en/latest/ Environment Modules Website]

= Basic Usage =

== Module categories, versions and defaults ==
The bwHPC clusters categorize ''Modules'', each software can exist in different versions:

category/softwarename/version

e.g. if julia is installed, it is in the module <code>math/julia</code>.

You can load it in the default version just specifying math/julia or a specific version <code> math/julia/1.11.0 </code>
Please note, that software may be '''hidden''' from default view using a dot in front of the version; i.e.
the software is still available but not visible in <code>module avail</code> but may still be loaded,
e.g. <code> math/julia/.1.10.0 </code>.

Currently all bwHPC software packages are assigned to one of the following ''Module'' categories:

<code> bio cae chem compiler devel lib math mpi numlib phys system vis </code>

== Display and search available Modules ==
''Modules'' are modulefiles that can be loaded by the user. A ''Module'' must be loaded before it provides changes to your environment. You can display the currently available ''Modules'' on the system by executing:
<pre>
$ module avail
</pre>
Some modules may be hidden and only become visible when compiler versions/libraries they depend on are loaded.

A complete list of all available modules can be displayed using:
<pre>
$ module spider
</pre>

You can selectively list software in one of those categories using, e.g. for the category "compiler", or just all versions of a certain module:
<pre>
$ module avail compiler/
$ module avail compiler/gnu

$ module spider compiler/
$ module spider compiler/gnu

</pre>

== module help ==
A help message for a specific ''Module'' can be displayed with ''''module help category/softwarename/version''''.
<br>
The help message usually contains additional information about the software and points to the software website and documentation.
<pre>
$ module help system/example/1.0
----------------- Module Specific Help for "system/example/1.0" ---------------------------
"This module provides a bwhpc-examples job that works on every cluster.

[... rest of the output is omitted in the Wiki for clarity ...]
</pre>

== Loading Modules and Check they are loaded ==

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
Don't use module load in .bashrc → see [[.bashrc Do's and Don'ts]]
|}

To load a software ''Module'' and display all loaded modules:
<pre>
$ module list
No Modulefiles Currently Loaded.
$ module load system/example/1.0
$ module list
Currently Loaded Modulefiles:
1) system/example/1.0
</pre>

Modules make software available only in your current shell. Whenever you login in, you have to load the software again. Please do not auto-load modules in ~/.bashrc, this can lead to problems with system software or other modules you may load later.

= Software job examples =
bwHPC provides example job scripts for most installed software modules.

For a Software ''Module'' with the sofware called '''SOMESOFTWARE''', you can find the example directory by:
<pre>
$ cd $SOMESOFTWARE_EXA_DIR
</pre>

Copy the whole example folder to your $HOME directory, so you can edit those job examples:

<pre>
$ cd
$ mkdir softwarename_examples
$ echo $SOMESOFTWARE_EXA_DIR
# Please do not proceed if the command above does not provide any text !
# Otherwise you will start to copy all system data (the directory "/").
$ cp -r $SOMESOFTWARE_EXA_DIR/ softwarename_examples/

</pre>

If your specific software isn't installed, there is a dummy software example module "system/example" present on all clusters. For this module, the process looks like this:

<pre>
# Load the example module
$ module load system/example/1.0

# Run example in a temporary directory
$ mkdir tmp_example_dir
$ cp -r $EXAMPLE_EXA_DIR/ softwarename_examples/
$ cd tmp_example_dir/bwhpc-examples

# Example jobscript for clusters using the SLURM batch system
sbatch examples-1.0.slurm
# Example jobscript for clusters using PBS
qsub examples-1.0.pbs

# Print the results
cat examples_result.txt
</pre>
----

= Additional Usage Recommendations =

=== Loading conflicts ===
By default you can not load different versions of same software ''Module'' in same session. Loading for example Intel compiler version X while Intel compiler version Y is loaded results in error message as follows:
<pre>
Module 'compiler/intel/X' conflicts with the currently loaded module(s) 'compiler/intel/Y'
</pre>
The solution is [[#Unloading Modules|unloading]] or switching ''Modules''.

=== Showing the changes introduced by a Module ===
Loading a ''Module'' will change the environment of the current shell session. For instance the $PATH variable will be expanded by the software's binary directory. Other ''Module'' variables may even change the behavior of the current shell session or the software program(s) in a more drastic way.
<br>
Loaded ''Modules'' may also invoke an additional set of environment variables, which e.g. point to directories or destinations of documentation and examples. Their nomenclature is systematic:
{| width=600px class="wikitable"
|-
! Variable
! Pointing to
|-
| $SWN_HOME
| Root directory of the software package
|-
| $SWN_DOC_DIR
| Documentation
|-
| $SWN_EXA_DIR
| Examples
|-
| $SWN_BPR_URL
| URL of software's Wiki article
|-
| and many many more...
|  
|}
with SWN being the place holder of the software ''Module'' name.
<br>
All the changes to the current shell session to be invoked by loading the ''Module'' can be reviewed using ''''module show category/softwarename/version''''.
<br>
<pre>
$ module show system/example/1.0
---------------------------------------------------------------------------------------------------
/opt/bwhpc/common/modulefiles/Core/system/example/1.0.lua:
---------------------------------------------------------------------------------------------------
whatis("A generic module containing a working bwhpc-examples job.")
setenv("EXAMPLE_VERSION","1.0")
setenv("EXAMPLE_HOME","/opt/bwhpc/common/system/example/1.0")
setenv("EXAMPLE_BIN_DIR","/opt/bwhpc/common/system/example/1.0/bin")
setenv("EXAMPLE_EXA_DIR","/opt/bwhpc/common/system/example/1.0/bwhpc-examples")
prepend_path("PATH","/opt/bwhpc/common/system/example/1.0/bin")
help([["This module provides a bwhpc-examples job that works on every cluster.
The module is used as example in the bwHPC-Wiki and therefore should be installed on every cluster,
such that users can try the commands out.

* The executable of this module can be found in the folder
$EXAMPLE_BIN_DIR
Upon loading the module, the binaries are added to PATH.

* Further documentation for using the example can be found in
https://wiki.bwhpc.de/e/Environment_Modules

* Examples are located at:
$EXAMPLE_EXA_DIR
]])

</pre>

=== Modules depending on Modules ===
Some program ''Modules'' depend on libraries to be loaded to the user environment. Therefore the
corresponding ''Modules'' of the software must be loaded together with the ''Modules'' of
the libraries.
<br>
By default such software ''Modules'' try to load required ''Modules'' and corresponding versions automatically. However, automatic loading might fail if a different version of that required ''Module''
is already loaded (cf. [[#Loading conflicts|Loading conflicts]]).
<br>

== Unloading Modules ==
To unload or to remove a software ''Module'' execute:
<pre>
$ module unload category/softwarename/version
</pre>

=== Unloading all loaded modules ===
In order to remove all previously loaded software modules from your environment issue the command 'module purge'.
<br>
<pre>
$ module list
Currently Loaded Modulefiles:
1) devel/gdb/7.7
2) compiler/intel/14.0
3) mpi/openmpi/1.8-intel-14.0(default)
$
$ module purge
$ module list
No Modulefiles Currently Loaded.
$
</pre>

== Other Module commands ==
=== module whatis ===
A short description for a specific ''Module'' can be displayed with ''''module whatis category/softwarename/version''''
<pre>
$ module whatis system/example/1.0
system/example/1.0 : A generic module containing a working bwhpc-examples job.
</pre>

Justus3

2026-08-25T09:49:25Z

K Siegmund:

{{Template:Justus3}}



The '''bwForCluster Justus 3''' is the state-wide high-performance compute cluster dedicated to '''Computational Chemistry and Quantum Sciences''' in Baden-Württemberg.

{| style=" background:#FEF4AB; width:100%;"
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News
|-
|
* 2026-06-16: Cooling installations for the new cluster has been enabled
|}
{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support
|-
|
* [[Justus3/Migration_from_J2|Migration Guide from JUSTUS2]]
* [[Justus3/Getting Started|Getting Started]]
* Online Tutorial [https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC Introduction to Linux in HPC] (external link)
* Contact and Support: [[BwSupportPortal]]
* Send [[Feedback|Feedback]] about Wiki pages
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | User Documentation
|-
|
* [[Registration/bwForCluster|Registration]]
* [[Justus3/Policy Agreement|Policy Agreement]]

* [[Justus3/Login|Login]]
* [[Justus3/Hardware|Hardware and Architecture]]
** [[Justus3/Hardware#Node_Specifications|Node Specifications]]
** [[Justus3/Hardware#Storage_Architecture|File Systems and Workspaces]]

* [[Justus3/Software|Software]]
* [[Justus3/Jobscripts: Running Your Calculations|Jobscripts: Running Your Calculations]]
** [[bwForCluster Justus 3 Slurm HOWTO|Slurm HOWTO]]
* [[Justus3/Visualization|Visualisation]]
* [[Development]]
|}

{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Cluster Funding
|-
|
* Please '''[[Justus3/Acknowledgement|acknowledge]]''' the Justus 3 cluster in your publications.
|}

BwSupportPortal

2026-08-25T09:47:37Z

K Siegmund:

Please choose the correct HPC-cluster queue as shown below. Failing to do so will delay the answer to your ticket.

If your issue is a missing entitlement, please do not use the BwSupportPortal, but contact your university.

== Login Issues due to Missing Entitlements ==

In case of questions or problems regarding your entitlement for using any bwForCluster, please
contact your local university's help desk, as this can only be addressed at the university which you are a member of.

See → '''[[Registration/bwForCluster/Entitlement]]'''

= Creating a ticket =

Tickets can be created and assigned to a support unit through the following steps after login at

→ '''https://www.bwhpc.de/supportportal'''

== Select New Ticket ==

A new ticket can be created by clicking on the "+"-Symbol in the bottom left:

[[File:BwHelpdesk Step1.png]]

== Navigate the Support Groups ==

Navigate the support groups by using '''the arrows''' or use the '''search bar''':

[[File:BwHelpdesk Step2.png]]

== Choose a Support Unit ==

Choose a specific support group:

[[File:BwHelpdesk Step3.png]]

BwSupportPortal

2026-08-25T09:47:28Z

K Siegmund:

Please choose the correct HPC-cluster queue as shown below. Failing to do so will delay the answer to your ticket. If your issue is a missing entitlement, please do not use the BwSupportPortal, but contact your university.

== Login Issues due to Missing Entitlements ==

In case of questions or problems regarding your entitlement for using any bwForCluster, please
contact your local university's help desk, as this can only be addressed at the university which you are a member of.

See → '''[[Registration/bwForCluster/Entitlement]]'''

= Creating a ticket =

Tickets can be created and assigned to a support unit through the following steps after login at

→ '''https://www.bwhpc.de/supportportal'''

== Select New Ticket ==

A new ticket can be created by clicking on the "+"-Symbol in the bottom left:

[[File:BwHelpdesk Step1.png]]

== Navigate the Support Groups ==

Navigate the support groups by using '''the arrows''' or use the '''search bar''':

[[File:BwHelpdesk Step2.png]]

== Choose a Support Unit ==

Choose a specific support group:

[[File:BwHelpdesk Step3.png]]

BwSupportPortal

2026-08-25T09:44:28Z

K Siegmund:

<div style="border: 3px solid #28a745; padding: 15px; background-color: #d4edda; margin: 5px 0;">

Please choose the correct cluster queue as shown below.

[https://www.bwhpc.de/supportportal '''Direct link to the bwSupportPortal''']
</div>

== Login Issues due to Missing Entitlements ==

In case of questions or problems regarding your entitlement for using any bwForCluster, please
contact your local university's help desk, as this can only be addressed at the university which you are a member of.

See → '''[[Registration/bwForCluster/Entitlement]]'''

= Creating a ticket =

Tickets can be created and assigned to a support unit through the following steps after login at

→ '''https://www.bwhpc.de/supportportal'''

== Select New Ticket ==

A new ticket can be created by clicking on the "+"-Symbol in the bottom left:

[[File:BwHelpdesk Step1.png]]

== Navigate the Support Groups ==

Navigate the support groups by using '''the arrows''' or use the '''search bar''':

[[File:BwHelpdesk Step2.png]]

== Choose a Support Unit ==

Choose a specific support group:

[[File:BwHelpdesk Step3.png]]

BwSupportPortal

2026-08-25T09:42:35Z

K Siegmund: /* Creating a ticket */

Help, support and any other HPC related request can be addressed to the bwHPC cluster and competence center teams via the

<div style="border: 3px solid #28a745; padding: 15px; background-color: #d4edda; margin: 5px 0;">
[https://www.bwhpc.de/supportportal '''Create a ticket at the bwSupportPortal''']
</div>

== Login Issues due to Missing Entitlements ==

In case of questions or problems regarding your entitlement for using any bwForCluster, please
contact your local university's help desk, as this can only be addressed at the university which you are a member of.

See → '''[[Registration/bwForCluster/Entitlement]]'''

= Creating a ticket =

Tickets can be created and assigned to a support unit through the following steps after login at

→ '''https://www.bwhpc.de/supportportal'''

== Select New Ticket ==

A new ticket can be created by clicking on the "+"-Symbol in the bottom left:

[[File:BwHelpdesk Step1.png]]

== Navigate the Support Groups ==

Navigate the support groups by using '''the arrows''' or use the '''search bar''':

[[File:BwHelpdesk Step2.png]]

== Choose a Support Unit ==

Choose a specific support group:

[[File:BwHelpdesk Step3.png]]

BwSupportPortal

2026-08-25T09:42:04Z

K Siegmund: /* Creating a ticket */

Help, support and any other HPC related request can be addressed to the bwHPC cluster and competence center teams via the

<div style="border: 3px solid #28a745; padding: 15px; background-color: #d4edda; margin: 5px 0;">
[https://www.bwhpc.de/supportportal '''Create a ticket at the bwSupportPortal''']
</div>

== Login Issues due to Missing Entitlements ==

In case of questions or problems regarding your entitlement for using any bwForCluster, please
contact your local university's help desk, as this can only be addressed at the university which you are a member of.

See → '''[[Registration/bwForCluster/Entitlement]]'''

= Creating a ticket =

Tickets can be created and assigned to a support unit through the following steps after login: at '''https://www.bwhpc.de/supportportal'''

== Select New Ticket ==

A new ticket can be created by clicking on the "+"-Symbol in the bottom left:

[[File:BwHelpdesk Step1.png]]

== Navigate the Support Groups ==

Navigate the support groups by using '''the arrows''' or use the '''search bar''':

[[File:BwHelpdesk Step2.png]]

== Choose a Support Unit ==

Choose a specific support group:

[[File:BwHelpdesk Step3.png]]

BwSupportPortal

2026-08-25T09:40:51Z

K Siegmund: /* Login Issues due to Missing Entitlements */

Help, support and any other HPC related request can be addressed to the bwHPC cluster and competence center teams via the

<div style="border: 3px solid #28a745; padding: 15px; background-color: #d4edda; margin: 5px 0;">
[https://www.bwhpc.de/supportportal '''Create a ticket at the bwSupportPortal''']
</div>

== Login Issues due to Missing Entitlements ==

In case of questions or problems regarding your entitlement for using any bwForCluster, please
contact your local university's help desk, as this can only be addressed at the university which you are a member of.

See → '''[[Registration/bwForCluster/Entitlement]]'''

= Creating a ticket =

Tickets can be created and assigned to a support unit through the following steps after login:

== Select New Ticket ==

A new ticket can be created by clicking on the "+"-Symbol in the bottom left:

[[File:BwHelpdesk Step1.png]]

== Navigate the Support Groups ==

Navigate the support groups by using '''the arrows''' or use the '''search bar''':

[[File:BwHelpdesk Step2.png]]

== Choose a Support Unit ==

Choose a specific support group:

[[File:BwHelpdesk Step3.png]]

JUSTUS2/Support

2026-08-25T09:40:02Z

K Siegmund:

==Entitlement==

In case of questions or problems regarding your entitlement for using any bwForCluster, please
contact your local university's help desk, as this can only be adressed at the university which you are a member of.

See → '''[[Registration/bwForCluster/Entitlement]]'''

==Registration and HPC Support==

In case of registration problems or questions about using JUSTUS 2, please submit a trouble ticket at the

→ '''[[BwSupportPortal]]'''

BwSupportPortal

2026-08-25T09:39:38Z

K Siegmund: /* Login Issues due to Missing Entitlements */

Help, support and any other HPC related request can be addressed to the bwHPC cluster and competence center teams via the

<div style="border: 3px solid #28a745; padding: 15px; background-color: #d4edda; margin: 5px 0;">
[https://www.bwhpc.de/supportportal '''Create a ticket at the bwSupportPortal''']
</div>

== Login Issues due to Missing Entitlements ==

In case of questions or problems regarding your entitlement for using any bwForCluster, please
contact your local university's help desk, as this can only be adressed at the university which you are a member of.

See → '''[[Registration/bwForCluster/Entitlement]]'''

= Creating a ticket =

Tickets can be created and assigned to a support unit through the following steps after login:

== Select New Ticket ==

A new ticket can be created by clicking on the "+"-Symbol in the bottom left:

[[File:BwHelpdesk Step1.png]]

== Navigate the Support Groups ==

Navigate the support groups by using '''the arrows''' or use the '''search bar''':

[[File:BwHelpdesk Step2.png]]

== Choose a Support Unit ==

Choose a specific support group:

[[File:BwHelpdesk Step3.png]]

BwSupportPortal

2026-08-25T09:39:12Z

K Siegmund: /* Login Issues due to Missing Entitlements */

Help, support and any other HPC related request can be addressed to the bwHPC cluster and competence center teams via the

<div style="border: 3px solid #28a745; padding: 15px; background-color: #d4edda; margin: 5px 0;">
[https://www.bwhpc.de/supportportal '''Create a ticket at the bwSupportPortal''']
</div>

== Login Issues due to Missing Entitlements ==

In case of questions or problems regarding your entitlement for using any bwForCluster, please
contact your local university first level help desk, as this can only be adressed at the university which you are a member of.

See → '''[[Registration/bwForCluster/Entitlement]]'''

= Creating a ticket =

Tickets can be created and assigned to a support unit through the following steps after login:

== Select New Ticket ==

A new ticket can be created by clicking on the "+"-Symbol in the bottom left:

[[File:BwHelpdesk Step1.png]]

== Navigate the Support Groups ==

Navigate the support groups by using '''the arrows''' or use the '''search bar''':

[[File:BwHelpdesk Step2.png]]

== Choose a Support Unit ==

Choose a specific support group:

[[File:BwHelpdesk Step3.png]]

BwSupportPortal

2026-08-25T09:38:36Z

K Siegmund: /* Missing Entitlements */

Help, support and any other HPC related request can be addressed to the bwHPC cluster and competence center teams via the

<div style="border: 3px solid #28a745; padding: 15px; background-color: #d4edda; margin: 5px 0;">
[https://www.bwhpc.de/supportportal '''Create a ticket at the bwSupportPortal''']
</div>

== Login Issues due to Missing Entitlements ==

In case of questions or problems regarding your entitlement for using any bwForCluster, please
contact your local university first level help desk.

See → '''[[Registration/bwForCluster/Entitlement]]'''

= Creating a ticket =

Tickets can be created and assigned to a support unit through the following steps after login:

== Select New Ticket ==

A new ticket can be created by clicking on the "+"-Symbol in the bottom left:

[[File:BwHelpdesk Step1.png]]

== Navigate the Support Groups ==

Navigate the support groups by using '''the arrows''' or use the '''search bar''':

[[File:BwHelpdesk Step2.png]]

== Choose a Support Unit ==

Choose a specific support group:

[[File:BwHelpdesk Step3.png]]

BwSupportPortal

2026-08-25T09:37:38Z

K Siegmund: /* Entitlement */

Help, support and any other HPC related request can be addressed to the bwHPC cluster and competence center teams via the

<div style="border: 3px solid #28a745; padding: 15px; background-color: #d4edda; margin: 5px 0;">
[https://www.bwhpc.de/supportportal '''Create a ticket at the bwSupportPortal''']
</div>

== Missing Entitlements ==

In case of questions or problems regarding your entitlement for using any bwForCluster, please
contact your local university first level help desk.

See → '''[[Registration/bwForCluster/Entitlement]]'''

= Creating a ticket =

Tickets can be created and assigned to a support unit through the following steps after login:

== Select New Ticket ==

A new ticket can be created by clicking on the "+"-Symbol in the bottom left:

[[File:BwHelpdesk Step1.png]]

== Navigate the Support Groups ==

Navigate the support groups by using '''the arrows''' or use the '''search bar''':

[[File:BwHelpdesk Step2.png]]

== Choose a Support Unit ==

Choose a specific support group:

[[File:BwHelpdesk Step3.png]]

BwSupportPortal

2026-08-25T09:37:13Z

K Siegmund:

Help, support and any other HPC related request can be addressed to the bwHPC cluster and competence center teams via the

<div style="border: 3px solid #28a745; padding: 15px; background-color: #d4edda; margin: 5px 0;">
[https://www.bwhpc.de/supportportal '''Create a ticket at the bwSupportPortal''']
</div>

==Entitlement==

In case of questions or problems regarding your entitlement for using any bwForCluster, please
contact your local university first level help desk.

See → '''[[Registration/bwForCluster/Entitlement]]'''

= Creating a ticket =

Tickets can be created and assigned to a support unit through the following steps after login:

== Select New Ticket ==

A new ticket can be created by clicking on the "+"-Symbol in the bottom left:

[[File:BwHelpdesk Step1.png]]

== Navigate the Support Groups ==

Navigate the support groups by using '''the arrows''' or use the '''search bar''':

[[File:BwHelpdesk Step2.png]]

== Choose a Support Unit ==

Choose a specific support group:

[[File:BwHelpdesk Step3.png]]

JUSTUS2/Support

2026-08-25T09:34:59Z

K Siegmund:

==Entitlement==

In case of questions or problems regarding your entitlement for using any bwForCluster, please
contact your local university first level help desk.

See → '''[[Registration/bwForCluster/Entitlement]]'''

==Registration and HPC Support==

In case of registration problems or questions about using JUSTUS 2, please submit a trouble ticket at the

→ '''[[BwSupportPortal]]'''

Justus3/Policy Agreement

2026-08-24T09:45:05Z

K Siegmund:

== DRAFT DOCUMENT ==

== Justus3 Policy - Agreement - V0 ==

For all users of bwForCluster Justus3 for Computational Chemistry and Quantum Sciences the [https://www.uni-ulm.de/fileadmin/website_uni_ulm/kiz/wir_ueber_uns/kiz-bo.pdf terms of use for services of the Communication and Information Center (kiz) of Ulm University] apply.

When registering for the service, you comply with aforementioned terms of use. Additionally you have the following obligations. You...

=== DFG Grant Reporting Obligations ===

* agree to use the cluster only for research in accordance with DFG funding conditions
* agree to acknowledge the bwForCluster Justus3 in your publications
* agree to send us references to all publications with contributions by Justus3
* agree to contribute to reports for the DFG

See → [[Justus3/Acknowledgement]]

=== Legal Obligations ===

* comply with the terms of use of the software manufacturers
* assure that your compute activities comply with the German Foreign Trade Act (Außenwirtschaftsgesetz - AWG) and German Foreign Trade Regulations (Außenwirtschaftsverordnung - AWV), see https://www.bafa.de/DE/Aussenwirtschaft/Ausfuhrkontrolle/Academia/academia_node.html

=== Compliant Cluster Usage ===

* agree to report any problem that might endanger the cluster operations
* agree to use the resources carefully (without wasting cores, memory or disk space)
* accept that the system is monitored to prevent misuse
* accept that job and software usage statistics are collected for improving the service and reporting to the DFG
* will avoid swap file I/O operations or any other intense I/O to HOME - swap files of single-node jobs must use the node local disk space
* will not run long cpu/memory intense calculations on login/vis nodes - those nodes are for interactive use, compiling and short tests only
* will not mass-submit very short (minutes) jobs to the batch system

=== Data Storage Compliance ===

* acknowledge that your home directory and work spaces will be deleted after expiration of your account (thus you are responsible to backup your files before your account expires)
* will not store important results in workspaces, since they are not included in any backup

Also see

→ [[.bashrc Do's and Don'ts]]

Justus3/Policy Agreement

2026-08-24T09:36:13Z

K Siegmund: /* Compliant Cluster Usage */

== DRAFT DOCUMENT ==

== Justus3 Policy - Agreement - V0 ==

For all users of bwForCluster Justus3 for Computational Chemistry and Quantum Sciences the [https://www.uni-ulm.de/fileadmin/website_uni_ulm/kiz/wir_ueber_uns/kiz-bo.pdf terms of use for services of the Communication- and Information Center (kiz) of Ulm University] apply.

When registering for the service, you comply with above mentioned terms of use. Additionally you have the following obligations. You...

=== DFG Grant Reporting Obligations ===

* agree to use the cluster only for research in accordance with DFG funding conditions
* agree to acknowledge the bwForCluster Justus3 in your publications
* agree to send us references to all publications with contributions by Justus3
* agree to contribute to reports for the DFG

=== Legal Obligations ===

* comply with the terms of use of the software manufacturers
* assure that your compute activities comply with the German Foreign Trade Act (Außenwirtschaftsgesetz - AWG) und German Foreign Trade Regulations (Außenwirtschaftsverordnung - AWV), see https://www.bafa.de/DE/Aussenwirtschaft/Ausfuhrkontrolle/Academia/academia_node.html

=== Compliant Cluster Usage ===

* agree to report any problem that might endanger the cluster operations
* agree to use the resources carefully (without wasting cores, memory or disk space)
* accept that the system is monitored to prevent misuse
* accept that job and software usage statistics are collected for improving the service and reporting to the DFG
* will avoid swap file I/O operations or any other intense I/O to HOME - swap files of single-node jobs must use the node local disk space
* will not run long cpu/memory intense calculations on login/vis nodes - those nodes are for interactive use, compiling and short tests only
* will not mass-submit very short (minutes) jobs to the batch system

=== Data Storage Compliance ===

* acknowledge that your home directory and work spaces will be deleted after expiration of your account (thus you are responsible to backup your files before your account expires)
* will not store important results in workspaces, since they are not included in any backup

Also see

→ [[.bashrc Do's and Don'ts]]

Justus3/Policy Agreement

2026-08-24T09:35:58Z

K Siegmund: Created page with "== DRAFT DOCUMENT == == Justus3 Policy - Agreement - V0 == For all users of bwForCluster Justus3 for Computational Chemistry and Quantum Sciences the [https://www.uni-ulm.de/fileadmin/website_uni_ulm/kiz/wir_ueber_uns/kiz-bo.pdf terms of use for services of the Communication- and Information Center (kiz) of Ulm University] apply. When registering for the service, you comply with above mentioned terms of use. Additionally you have the following obligations. You... ===..."

== DRAFT DOCUMENT ==

== Justus3 Policy - Agreement - V0 ==

For all users of bwForCluster Justus3 for Computational Chemistry and Quantum Sciences the [https://www.uni-ulm.de/fileadmin/website_uni_ulm/kiz/wir_ueber_uns/kiz-bo.pdf terms of use for services of the Communication- and Information Center (kiz) of Ulm University] apply.

When registering for the service, you comply with above mentioned terms of use. Additionally you have the following obligations. You...

=== DFG Grant Reporting Obligations ===

* agree to use the cluster only for research in accordance with DFG funding conditions
* agree to acknowledge the bwForCluster Justus3 in your publications
* agree to send us references to all publications with contributions by Justus3
* agree to contribute to reports for the DFG

=== Legal Obligations ===

* comply with the terms of use of the software manufacturers
* assure that your compute activities comply with the German Foreign Trade Act (Außenwirtschaftsgesetz - AWG) und German Foreign Trade Regulations (Außenwirtschaftsverordnung - AWV), see https://www.bafa.de/DE/Aussenwirtschaft/Ausfuhrkontrolle/Academia/academia_node.html

== Compliant Cluster Usage ==

* agree to report any problem that might endanger the cluster operations
* agree to use the resources carefully (without wasting cores, memory or disk space)
* accept that the system is monitored to prevent misuse
* accept that job and software usage statistics are collected for improving the service and reporting to the DFG
* will avoid swap file I/O operations or any other intense I/O to HOME - swap files of single-node jobs must use the node local disk space
* will not run long cpu/memory intense calculations on login/vis nodes - those nodes are for interactive use, compiling and short tests only
* will not mass-submit very short (minutes) jobs to the batch system

=== Data Storage Compliance ===

* acknowledge that your home directory and work spaces will be deleted after expiration of your account (thus you are responsible to backup your files before your account expires)
* will not store important results in workspaces, since they are not included in any backup

Also see

→ [[.bashrc Do's and Don'ts]]

Justus3

2026-08-20T15:23:55Z

K Siegmund:

{{Template:Justus3}}



The '''bwForCluster Justus 3''' is the state-wide high-performance compute cluster dedicated to '''Computational Chemistry and Quantum Sciences''' in Baden-Württemberg.

{| style=" background:#FEF4AB; width:100%;"
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News
|-
|
* 2026-06-16: Cooling installations for the new cluster has been enabled
|}
{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support
|-
|
* [[Justus3/Migration_from_J2|Migration Guide from JUSTUS2]]
* [[Justus3/Getting Started|Getting Started]]
* Online Tutorial [https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC Introduction to Linux in HPC] (external link)
* [[Justus3/Support|Contact and Support]]
* Send [[Feedback|Feedback]] about Wiki pages
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | User Documentation
|-
|
* [[Registration/bwForCluster|Registration]]
* [[Justus3/Policy Agreement|Policy Agreement]]

* [[Justus3/Login|Login]]
* [[Justus3/Hardware|Hardware and Architecture]]
** [[Justus3/Hardware#Node_Specifications|Node Specifications]]
** [[Justus3/Hardware#Storage_Architecture|File Systems and Workspaces]]

* [[Justus3/Software|Software]]
* [[Justus3/Jobscripts: Running Your Calculations|Jobscripts: Running Your Calculations]]
** [[bwForCluster Justus 3 Slurm HOWTO|Slurm HOWTO]]
* [[Justus3/Visualization|Visualisation]]
* [[Development]]
|}

{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Cluster Funding
|-
|
* Please '''[[Justus3/Acknowledgement|acknowledge]]''' the Justus 3 cluster in your publications.
|}

.bashrc Do's and Don'ts

2026-06-29T15:39:55Z

K Siegmund:

.bashrc and .profiles are files that are sourced when you log in, they set some important variables that make your shell function at a specific site setup.

You can add commands that you want to run every time at login and when starting a job by adding lines to these files, but you can also make mistakes that make your life or the life of the cluster teams harder.

== Don'ts ==

* don't forget to test your changes before you log out (log in on a second shell)
* don't put <code> module load </code> commands in .bashrc, modules may make changes that break the default system environment
* don't put LD_PRELOAD or LIBRARY_PATH in your .bashrc, unless you know exactly what you're doing
*

== Do's ==
* make a copy of your .bashrc before making large changes
* '''test''' your changes to .bashrc before you log out
* add command aliases or bash functions in .bashrc - example: <code> alias l='ls -la' </code>
* add a local ~/bin or similar to $PATH if needed
* improve your terminal command history experience:
<syntaxhighlight lang=bash>
shopt -s histappend # append to the history file, don't overwrite it
shopt -s direxpand # expand dirs before trying to expand filenames
HISTSIZE=1000000 # number of commands stored in one terminal session
HISTFILESIZE=20000000 # number of commmands stored in file
HISTFILE="$HOME"/.bash_history_$USER # prevent file truncation by bash sessions that did not read in .bashrc
PROMPT_COMMAND="history -a" # save history after every command
HISTCONTROL=ignoreboth:erasedups # dont store duplicates and ignore lines starting with white space
HISTIGNORE="?:??" # dont store one and two letter commands
export HISTTIMEFORMAT="%F %H:%M:%S " # show dates on history
</syntaxhighlight>

* maybe make your rm / cp commands safer by aliasing to -i <code> alias rm 'rm -i'</code>

== Autoedits / Conda ==

There is some software (e.g. Conda) that automatically modifies .bashrc. These modifications may interfere with the proper functioning of e.g. Python from Lmod modules or the Jupyter modules. Make sure you have a copy of your .bashrc and test it after installing such software.

Data Transfer/SSHFS

2026-06-29T13:30:11Z

K Siegmund:

Sshfs is a network protocol for mounting data. It uses [[Data Transfer/SFTP|sftp]] under the hood.

== Installation ==

For the necessary installations follow this link:<br /> [https://linuxize.com/post/how-to-use-sshfs-to-mount-remote-directories-over-ssh/ How to use SSHFS to Mount Remote Directories over SSH].<br />
a) '''MacOS after the installation'''<br />
To enable sshfs on your Mac go to: System Preferences -> General tab -> Allow Extensions button<br />
Then follow the instructions that pop up (make sure that you remember the instructions before shutting down your computer).<br />
b) '''Linux installation short commands'''
<ul>
<li>Debian/Ubuntu: <code>apt-get install sshfs</code></li>
<li>CentOS/RedHat: <code>dnf install fuse-sshfs</code></li>
<li>If you want to use /etc/fstab, you can do this with the following entry:
<pre>sshfs#<username>@<hostname>: <your_local_mountpoint> fuse defaults,user,noauto,exec 0 0 </pre></li></ul>

== Usage ==

=== Caveat: Unmounting ===

Note that this creates a virtual drive on your system just as if you would plug in an external drive that stays active until you unmount it. Disk mounts happen on a low level of operating systems, so it is important to unmount the virtual drive cleanly whenever possible.

'''MacOS & Linux:'''<br />
If the connection to the cluster broke and you want to remount, you often have to use umount first. Try to remember to always unmount first before retrying. Otherwise it can happen that unmounting is not possible anymore by cmd. Usually the error message in this case tells you that the process is currently busy and therefore cannot umount.

You should be able to find the process(es) that use the file system with <code>fuser -m <your_local_mountpoint></code> and kill/exit the processes that block unmounting or can kill them directly using <code>fuser -k -m <your_local_mountpoint></code>
In case this fails, you may have to restart your computer which automatically unmounts everything.

<pre>bash
umount -f <localPath>
# For example
umount -f /Users/<username>/Documents/mounted
# or use
fusermount -u <localPath>
</pre>

''' MacOS '''

An automatic umount is highly recommended, because MacOS finder can make thousands of insuccessful login attempts in the worst case leading to you being blocked from the cluster system or even from everything that uses the same OTP.

Example for enabling automatic umount per "sleepwatcher":

<syntaxhighlight lang="bash">
brew install sleepwatcher
brew services start sleepwatcher
brew services info sleepwatcher
cat > ~/.sleep <<EOT
#!/bin/bash
umount -f ~/mountCluster 2>/dev/null
EOT
chmod a+x ~/.sleep
</syntaxhighlight>

'''Windows'''
You should be able to click on the drive and choose "eject" or "unmount". If this is not the case, you can use
net use X: /delete
or if this does not work
taskkill /f /im sshfs.exe
(from: [https://stackoverflow.com/questions/62031900/how-to-unmount-close-the-mount-for-sshfs-windows stackoverflow])

=== Windows ===
''Caution: sshfs on Windows doesn't work with 2FA. An alternative is [[Data_Transfer/Rclone | Rclone with sftp]] + [[Registration/SSH | ssh key]].''
<ol style="list-style-type: decimal;">
<li>Go to "This PC" and select "Connect network drive" ("Netzlaufwerk verbinden")</li>
<li>Select a drive (W:) and click on "connect with webpage on which you can save documents and images" (Verbindung mit website herstellen, auf der Sie Dokumente und Bilder speichern können)</li>
<li>Enter the following as address, to connect to the home or root folder:
<syntaxhighlight lang="bash">\sshfs\<username>@<hostname>/</syntaxhighlight>
If you would want to go two folders up, you would add “''../../''” at the end.</li></ol>

=== MacOS & Linux ===

<ol style="list-style-type: decimal;">
<li>Create an empty directory. The cluster/storage folder is going to be mounted to this directory.</li>
<li>Mount a folder from the cluster/storage:
<syntaxhighlight lang="bash"># Home folder
sshfs <username>@<hostname>: <localPath>
# subfolderX
sshfs <username>@<hostname>:subfolderX <localPath></syntaxhighlight></li></ol>

Data Transfer/SSHFS

2026-06-29T11:36:56Z

K Siegmund: /* MacOS & Linux */

Sshfs is a network protocol for mounting data. It uses [[Data Transfer/SFTP|sftp]] under the hood.

== Installation ==

For the necessary installations follow this link:<br /> [https://linuxize.com/post/how-to-use-sshfs-to-mount-remote-directories-over-ssh/ How to use SSHFS to Mount Remote Directories over SSH].<br />
a) '''MacOS after the installation'''<br />
To enable sshfs on your Mac go to: System Preferences -> General tab -> Allow Extensions button<br />
Then follow the instructions that pop up (make sure that you remember the instructions before shutting down your computer).<br />
b) '''Linux installation short commands'''
<ul>
<li>Debian/Ubuntu: <code>apt-get install sshfs</code></li>
<li>CentOS/RedHat: <code>dnf install fuse-sshfs</code></li>
<li>If you want to use /etc/fstab, you can do this with the following entry:
<pre>sshfs#<username>@<hostname>: <your_local_mountpoint> fuse defaults,user,noauto,exec 0 0 </pre></li></ul>

== Usage ==

=== Caveat: Unmounting ===

Note that this creates a virtual drive on your system just as if you would plug in an external drive that stays active until you unmount it. Disk mounts happen on a low level of operating systems, so it is important to unmount the virtual drive cleanly whenever possible.

'''MacOS & Linux:'''<br />
If the connection to the cluster broke and you want to remount, you often have to use umount first. Try to remember to always unmount first before retrying. Otherwise it can happen that unmounting is not possible anymore by cmd. Usually the error message in this case tells you that the process is currently busy and therefore cannot umount.

You should be able to find the process(es) that use the file system with <code>fuser -m <your_local_mountpoint></code> and kill/exit the processes that block unmounting or can kill them directly using <code>fuser -k -m <your_local_mountpoint></code>
In case this fails, you may have to restart your computer which automatically unmounts everything.

<pre>bash
umount -f <localPath>
# For example
umount -f /Users/<username>/Documents/mounted
# or use
fusermount -u <localPath>
</pre>

'''Windows'''
You should be able to click on the drive and choose "eject" or "unmount". If this is not the case, you can use
net use X: /delete
or if this does not work
taskkill /f /im sshfs.exe
(from: [https://stackoverflow.com/questions/62031900/how-to-unmount-close-the-mount-for-sshfs-windows stackoverflow])

=== Windows ===
''Caution: sshfs on Windows doesn't work with 2FA. An alternative is [[Data_Transfer/Rclone | Rclone with sftp]] + [[Registration/SSH | ssh key]].''
<ol style="list-style-type: decimal;">
<li>Go to "This PC" and select "Connect network drive" ("Netzlaufwerk verbinden")</li>
<li>Select a drive (W:) and click on "connect with webpage on which you can save documents and images" (Verbindung mit website herstellen, auf der Sie Dokumente und Bilder speichern können)</li>
<li>Enter the following as address, to connect to the home or root folder:
<syntaxhighlight lang="bash">\sshfs\<username>@<hostname>/</syntaxhighlight>
If you would want to go two folders up, you would add “''../../''” at the end.</li></ol>

=== MacOS & Linux ===

<ol style="list-style-type: decimal;">
<li>Create an empty directory. The cluster/storage folder is going to be mounted to this directory.</li>
<li>Mount a folder from the cluster/storage:
<syntaxhighlight lang="bash"># Home folder
sshfs <username>@<hostname>: <localPath>
# subfolderX
sshfs <username>@<hostname>:subfolderX <localPath></syntaxhighlight></li></ol>

== MacOS ==

An automatic umount is highly recommended, because MacOS finder can make thousands of insuccessful login attempts in the worst case leading to you being blocked from the cluster system or even from everything that uses the same OTP.

Example for enabling automatic umount per "sleepwatcher":

<syntaxhighlight lang="bash">
brew install sleepwatcher
brew services start sleepwatcher
brew services info sleepwatcher
cat > ~/.sleep <<EOT
#!/bin/bash
umount -f ~/mountCluster 2>/dev/null
EOT
chmod a+x ~/.sleep
</syntaxhighlight>

Justus3/Getting Started

2026-06-26T13:58:30Z

K Siegmund:

Here is a short list of things you may need to do first when you get onto the cluster

== Pre-Requisites ==
* Know how a HPC compute Cluster works: In short: you do not simply run log in and start your software, but you write a "job script" that contains all commands to run and process your job and send this into a waiting queue to be run on one of several hundred computers. → '''[[Running Calculations]]'''
* Get Access: Follow the registration process for the bwForCluster. → [[Registration/bwForCluster|How to Register for a bwForCluster]]

== Basics ==
* know the cluster rules: → '''[[Justus3/Policy_Agreement]]'''
* log in to the cluster (and first set a 2FA secret): '''[[Justus3/Login]]'''
* get accustomed with the linux commandline:
** HPC Wiki (external site) → '''[https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC/The_Command_Line Introduction to Linux Commandline]'''
** Have commands ready → '''[[Media:HPC_User_Linux_command_cheatsheet.pdf|HPC User's Linux Cheat Sheet (PDF)]]'''
** configure your shell → '''[[.bashrc Do's and Don'ts]]'''
** linux course at '''[https://training.bwhpc.de/ training.bwhpc.de]'''

== Running an Example with Preinstalled Software ==
* scientific software: learn how to load '''[[Software Modules]]'''
* continue reading until you found that there are example job scripts: '''[[Environment_Modules#Software_job_examples]]'''
* submit a sample job from a software as mentioned in the job example. Also see: '''[[Justus3/Slurm]]'''
* monitor your job: '''[[Justus3/Monitoring Your Jobs]]'''

== Running Your Own Calculations in a Batch Job ==
{{Template:AttentionBox|Compute jobs on nodes '''must not''' write temporary I/O intensive data to central file systems.}}
* transfer your own data to the cluster → '''[[Data Transfer]]'''
** Learn about available Hardware:
*** → '''[[Justus3/File Systems]]'''
*** → '''[[Justus3/Compute Node Types]]'''
* adapt the sample job script to run your own job
* Submit and monitor your jobs with Slurm commands.
** → '''[[Justus3/Jobscripts: Running Your Calculations|Jobscripts: Running Your Calculations]]'''
** → '''[[BwForCluster_JUSTUS_2_Slurm_HOWTO| extensive Slurm HOWTO on specific tasks]]'''

* Remember to mention the cluster in your publications. → '''[[bwForCluster Justus 3 Acknowledgement|Acknowledgement]]'''

== Do Production Runs with Hundreds of Jobs ==

* Learn about Scaling your Job
** Determine how many compute-cores the job can use? → '''[[Scaling]]'''
** Make especially sure your compute jobs are running efficiently → '''[[Energy Efficient Cluster Usage]]'''

* Specially remember to mention the cluster in your publications. → '''[[bwForCluster Justus 3 Acknowledgement|Acknowledgement]]'''

Justus3/Getting Started

2026-06-26T13:57:07Z

K Siegmund: /* Basics */

Here is a short list of things you may need to do first when you get onto the cluster

== Pre-Requisites ==
* Know how a HPC compute Cluster works: In short: you do not simply run log in and start your software, but you write a "job script" that contains all commands to run and process your job and send this into a waiting queue to be run on one of several hundred computers. → '''[[Running Calculations]]'''
* Get Access: Follow the registration process for the bwForCluster. → [[Registration/bwForCluster|How to Register for a bwForCluster]]

== Basics ==
* know the cluster rules: → '''[[Justus3/Policy_Agreement]]'''
* log in to the cluster (and first set a 2FA secret): '''[[JUSTUS3/Login]]'''
* get accustomed with the linux commandline:
** HPC Wiki (external site) → '''[https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC/The_Command_Line Introduction to Linux Commandline]'''
** Have commands ready → '''[[Media:HPC_User_Linux_command_cheatsheet.pdf|HPC User's Linux Cheat Sheet (PDF)]]'''
** configure your shell → '''[[.bashrc Do's and Don'ts]]'''
** linux course at '''[https://training.bwhpc.de/ training.bwhpc.de]'''

== Running an Example with Preinstalled Software ==
* scientific software: learn how to load '''[[Software Modules]]'''
* continue reading until you found that there are example job scripts: '''[[Environment_Modules#Software_job_examples]]'''
* submit a sample job from a software as mentioned in the job example. Also see: '''[[JUSTUS3/Slurm]]'''
* monitor your job: '''[[JUSTUS3/Monitoring Your Jobs]]'''

== Running Your Own Calculations in a Batch Job ==
{{Template:AttentionBox|Compute jobs on nodes '''must not''' write temporary I/O intensive data to central file systems.}}
* transfer your own data to the cluster → '''[[Data Transfer]]'''
** Learn about available Hardware:
*** → '''[[JUSTUS3/File Systems]]'''
*** → '''[[JUSTUS3/Compute Node Types]]'''
* adapt the sample job script to run your own job

* Submit and monitor your jobs with Slurm commands.
** → '''[[JUSTUS3/Jobscripts: Running Your Calculations|Jobscripts: Running Your Calculations]]'''
** → '''[[BwForCluster_JUSTUS_2_Slurm_HOWTO| extensive Slurm HOWTO on specific tasks]]'''

* Remember to mention the cluster in your publications. → '''[[bwForCluster JUSTUS 3 Acknowledgement|Acknowledgement]]'''

== Do Production Runs with Hundreds of Jobs ==

* Learn about Scaling your Job
** Determine how many compute-cores the job can use? → '''[[Scaling]]'''
** Make especially sure your compute jobs are running efficiently → '''[[Energy Efficient Cluster Usage]]'''

* Specially remember to mention the cluster in your publications. → '''[[bwForCluster JUSTUS 3 Acknowledgement|Acknowledgement]]'''

Justus3/Getting Started

2026-06-26T13:56:14Z

K Siegmund: K Siegmund moved page JUSTUS3/Getting Started to Justus3/Getting Started without leaving a redirect

Here is a short list of things you may need to do first when you get onto the cluster

== Pre-Requisites ==
* Know how a HPC compute Cluster works: In short: you do not simply run log in and start your software, but you write a "job script" that contains all commands to run and process your job and send this into a waiting queue to be run on one of several hundred computers. → '''[[Running Calculations]]'''
* Get Access: Follow the registration process for the bwForCluster. → [[Registration/bwForCluster|How to Register for a bwForCluster]]

== Basics ==
* know the cluster rules: → '''[[JUSTUS3/Policy_Agreement]]'''
* log in to the cluster (and first set a 2FA secret): '''[[JUSTUS3/Login]]'''
* get accustomed with the linux commandline:
** HPC Wiki (external site) → '''[https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC/The_Command_Line Introduction to Linux Commandline]'''
** Have commands ready → '''[[Media:HPC_User_Linux_command_cheatsheet.pdf|HPC User's Linux Cheat Sheet (PDF)]]'''
** configure your shell → '''[[.bashrc Do's and Don'ts]]'''
** linux course at '''[https://training.bwhpc.de/ training.bwhpc.de]'''

== Running an Example with Preinstalled Software ==
* scientific software: learn how to load '''[[Software Modules]]'''
* continue reading until you found that there are example job scripts: '''[[Environment_Modules#Software_job_examples]]'''
* submit a sample job from a software as mentioned in the job example. Also see: '''[[JUSTUS3/Slurm]]'''
* monitor your job: '''[[JUSTUS3/Monitoring Your Jobs]]'''

== Running Your Own Calculations in a Batch Job ==
{{Template:AttentionBox|Compute jobs on nodes '''must not''' write temporary I/O intensive data to central file systems.}}
* transfer your own data to the cluster → '''[[Data Transfer]]'''
** Learn about available Hardware:
*** → '''[[JUSTUS3/File Systems]]'''
*** → '''[[JUSTUS3/Compute Node Types]]'''
* adapt the sample job script to run your own job

* Submit and monitor your jobs with Slurm commands.
** → '''[[JUSTUS3/Jobscripts: Running Your Calculations|Jobscripts: Running Your Calculations]]'''
** → '''[[BwForCluster_JUSTUS_2_Slurm_HOWTO| extensive Slurm HOWTO on specific tasks]]'''

* Remember to mention the cluster in your publications. → '''[[bwForCluster JUSTUS 3 Acknowledgement|Acknowledgement]]'''

== Do Production Runs with Hundreds of Jobs ==

* Learn about Scaling your Job
** Determine how many compute-cores the job can use? → '''[[Scaling]]'''
** Make especially sure your compute jobs are running efficiently → '''[[Energy Efficient Cluster Usage]]'''

* Specially remember to mention the cluster in your publications. → '''[[bwForCluster JUSTUS 3 Acknowledgement|Acknowledgement]]'''

Acknowledgement

2026-06-25T09:36:31Z

K Siegmund:

Remember to acknowledge our resources in your publications!

Such recognition is important for acquiring funding for the next generation of hardware, support services, data storage, and infrastructure.

The publications will be referenced on the bwHPC website: https://www.bwhpc.de/user_publications.html

== HPC Clusters ==

→[[BwUniCluster3.0/Acknowledgement| bwUniCluster Acknowledgement]]

→[[BinAC2/Acknowledgement| bwForCluster BinAC2 Acknowledgement]]

→[[Helix/Acknowledgement| bwForCluster Helix Acknowledgement]]

→[[Justus3/Acknowledgement| bwForCluster Justus 3 Acknowledgement]]

→[[NEMO2/Acknowledgement| bwForCluster NEMO2 Acknowledgement]]

== Data Facilities ==

→[[SDS@hd/Acknowledgement| SDS@hd Acknowledgement]]

Acknowledgement

2026-06-25T09:36:11Z

K Siegmund:

Remember to acknowledge our resources in your publications!

Such recognition is important for acquiring funding for the next generation of hardware, support services, data storage, and infrastructure.

The publications will be referenced on the bwHPC website: https://www.bwhpc.de/user_publications.html

== HPC Clusters ==

→[[BwUniCluster3.0/Acknowledgement| bwUniCluster Acknowledgement]]

→[[BinAC2/Acknowledgement| bwForCluster BinAC2 Acknowledgement]]

→[[Helix/Acknowledgement| bwForCluster Helix Acknowledgement]]

→[[JUSTUS3/Acknowledgement| bwForCluster JUSTUS3 Acknowledgement]]

→[[NEMO2/Acknowledgement| bwForCluster NEMO2 Acknowledgement]]

== Data Facilities ==

→[[SDS@hd/Acknowledgement| SDS@hd Acknowledgement]]

Justus3/Acknowledgement

2026-06-25T09:06:03Z

K Siegmund:

== Acknowledgement ==
When preparing a publication based on work using a bwForCluster (e.g., Justus 3), please ensure you acknowledge the bwHPC initiative, the bwHPC-S5 project, and the specific facility. The following text is suggested.

'''Acknowledgement:'''
<blockquote>
'''The authors acknowledge support by the HPC team at the Communication and Information Center (kiz) of Ulm University, the state of Baden-Württemberg through bwHPC and the German Research Foundation (DFG) for funding under [https://gepris.dfg.de/gepris/projekt/533954541 "Project number 533954541"] HPC Forschungscluster (bwForCluster Justus 3)'''
</blockquote>

== Publication Reporting ==

Please inform us of any fully published publications (e.g., journal articles, books, theses, master or bachelor thesis) resulting from bwHPC resource usage. Such recognition is important for acquiring funding for the next generation hardware, support services, data storage and infrastructure.

For pre-prints, please wait until the paper is fully published and send us only the final record. Please avoid sending publications already reported a second time.

This can happen in one of three ways:

1) with your annual RV renewal (the form will ask for DOIs - most practical, as it will allow to select exactly the publications published in the last year)

2) by emailing to:
* [mailto:publications(ät)bwhpc.de publications(ät)bwhpc.de]

In the email, please state:
* cluster facility (e.g. JUSTUS 3)
* RV acronym (e.g. bw14L001)
* DOI
* author(s)
* title ''or'' booktitle
* journal, volume, pages ''or'' editors, address, publisher
* year.

3) when in the process of writing the DFG report, we will send you a list of your already submitted publications and ask you to update it for the report.

The publications will be referenced on the bwHPC website:
https://www.bwhpc.de/user_publications.html

Justus3/Acknowledgement

2026-06-25T09:00:50Z

K Siegmund:

== Acknowledgement ==
When preparing a publication based on work using a bwForCluster (e.g., Justus 3), please ensure you acknowledge the bwHPC initiative, the bwHPC-S5 project, and the specific facility. The following text is suggested.

'''Acknowledgement:'''
<blockquote>
'''The authors acknowledge support by the HPC team at the Communication and Information Center (kiz) of Ulm University, the state of Baden-Württemberg through bwHPC and the German Research Foundation (DFG) for funding under [https://gepris.dfg.de/gepris/projekt/533954541 "Project number 533954541"] HPC Forschungscluster (bwForCluster Justus 3)'''
</blockquote>

== Publications ==

Please kindly inform us of any publications (e.g., articles, theses, or talks) resulting from bwHPC resource usage.

This can happen in one of three ways:

1) with your annual RV renewal (the form will ask for DOIs)

2) by emailing to:
* [mailto:publications(ät)bwhpc.de publications(ät)bwhpc.de]

Write to stating:
* cluster facility (e.g. JUSTUS 3)
* RV acronym (e.g. bw14L001)
* DOI (if available)
* author(s)
* title ''or'' booktitle
* journal, volume, pages ''or'' editors, address, publisher
* year.

3) when in the process of writing the DFG report, we will send you a list of your already submitted publications and ask you to update it for the report.

Such recognition is important for acquiring funding for the next generation hardware, support services, data storage and infrastructure.

The publications will be referenced on the bwHPC website:
https://www.bwhpc.de/user_publications.html

Justus3/Acknowledgement

2026-06-25T08:56:47Z

K Siegmund:

When preparing a publication based on work using a bwForCluster (e.g., Justus 3), please ensure you acknowledge the bwHPC initiative, the bwHPC-S5 project, and the specific facility. The following text is suggested.

'''Acknowledgement:'''
<blockquote>
'''The authors acknowledge support by the HPC team at the Communication and Information Center (kiz) of Ulm University, the state of Baden-Württemberg through bwHPC and the German Research Foundation (DFG) for funding under [https://gepris.dfg.de/gepris/projekt/533954541 "Project number 533954541"] HPC Forschungscluster (bwForCluster Justus 3)'''
</blockquote>

Please kindly notify us of any publications (e.g., articles, theses, or talks) resulting from bwHPC resource usage by emailing the below email address. '''Note''': If you already provide a complete publication list during your yearly RV renewal, a separate notification is not required.

Write to [mailto:publications(ät)bwhpc.de publications(ät)bwhpc.de] stating:
* cluster facility (e.g. JUSTUS 3)
* RV acronym (e.g. bw14L001)
* DOI (if available)
* author(s)
* title ''or'' booktitle
* journal, volume, pages ''or'' editors, address, publisher
* year.

Such recognition is important for acquiring funding for the next generation hardware, support services, data storage and infrastructure.

The publications will be referenced on the bwHPC website:
https://www.bwhpc.de/user_publications.html

Justus3/Acknowledgement

2026-06-24T17:01:09Z

K Siegmund:

When preparing a publication based on work using a bwForCluster (e.g., Justus 3), please ensure you acknowledge the bwHPC initiative, the bwHPC-S5 project, and the specific facility. The following text is suggested.

'''Acknowledgement:'''
<blockquote>
'''The authors acknowledge support by the HPC team at the Communication and Information Center (kiz) of Ulm University, the state of Baden-Württemberg through bwHPC and the German Research Foundation (DFG) for funding under [https://gepris.dfg.de/gepris/projekt/533954541 "Project number 533954541"] HPC Forschungscluster (bwForCluster Justus 3)'''
</blockquote>

Please kindly notify us of any publications (e.g., articles, theses, or talks) resulting from bwHPC resource usage by emailing the below email address. '''Note''': If you already provide a complete publication list during your yearly RV renewal, a separate notification is not required.

Write to [mailto:publications(ät)bwhpc.de publications(ät)bwhpc.de] stating:
* cluster facility (e.g. JUSTUS 2)
* RV acronym (e.g. bw14L001)
* DOI (if available)
* author(s)
* title ''or'' booktitle
* journal, volume, pages ''or'' editors, address, publisher
* year.

Such recognition is important for acquiring funding for the next generation hardware, support services, data storage and infrastructure.

The publications will be referenced on the bwHPC website:
https://www.bwhpc.de/user_publications.html

Justus3/Acknowledgement

2026-06-24T17:00:58Z

K Siegmund:

When preparing a publication based on work using a bwForCluster (e.g., Justus 3), please ensure you acknowledge the bwHPC initiative, the bwHPC-S5 project, and the specific facility. The following text is suggested.

'''Acknowledgement:'''
<blockquote>
'''The authors acknowledge support by the HPC team at the Communication and Information Center (kiz) of Ulm University, the state of Baden-Württemberg through bwHPC and the German Research Foundation (DFG) for funding under [https://gepris.dfg.de/gepris/projekt/533954541 "Project number 533954541"] HPC Forschungscluster (bwForCluster Justus 3)'''
</blockquote>

"Please kindly notify us of any publications (e.g., articles, theses, or talks) resulting from bwHPC resource usage by emailing the below email address. '''Note''': If you already provide a complete publication list during your yearly RV renewal, a separate notification is not required."

Write to [mailto:publications(ät)bwhpc.de publications(ät)bwhpc.de] stating:
* cluster facility (e.g. JUSTUS 2)
* RV acronym (e.g. bw14L001)
* DOI (if available)
* author(s)
* title ''or'' booktitle
* journal, volume, pages ''or'' editors, address, publisher
* year.

Such recognition is important for acquiring funding for the next generation hardware, support services, data storage and infrastructure.

The publications will be referenced on the bwHPC website:
https://www.bwhpc.de/user_publications.html