BwForCluster Justus 3 Slurm HOWTO - Revision history

J Schäfer at 08:06, 31 August 2026

2026-08-31T08:06:34Z

← Older revision		Revision as of 10:06, 31 August 2026
Line 44:		Line 44:
	Sample job script template for serial job:		Sample job script template for serial job:

	<~~source~~ lang="bash">		<syntaxhighlight lang="bash">
	#!/bin/bash		#!/bin/bash
	# Allocate one node		# Allocate one node
Line 66:		Line 66:
	# Run serial program		# Run serial program
	./my_serial_program		./my_serial_program
			</syntaxhighlight>
	</source>

	Sample code for serial program: [[Media:Hello_serial.c \| Hello_serial.c]]		Sample code for serial program: [[Media:Hello_serial.c \| Hello_serial.c]]
Line 88:		Line 88:
	In order to activate these safeguard settings users can insert the following lines in their scripts (after all #SBATCH directives):		In order to activate these safeguard settings users can insert the following lines in their scripts (after all #SBATCH directives):

	<~~source~~ lang="bash">		<syntaxhighlight lang="bash">
	[...]		[...]
	set -o errexit # (or set -e) cause batch script to exit immediately when a command fails.		set -o errexit # (or set -e) cause batch script to exit immediately when a command fails.
Line 94:		Line 94:
	set -o nounset # (or set -u) causes the script to treat unset variables as an error and exit immediately		set -o nounset # (or set -u) causes the script to treat unset variables as an error and exit immediately
	[...]		[...]
			</syntaxhighlight>
	</source>

	== How to submit an interactive job? ==		== How to submit an interactive job? ==
Line 235:		Line 235:
	E.g.:		E.g.:

	<~~source~~ lang="bash">		<syntaxhighlight lang="bash">
	[...]		[...]
	infile="$1"		infile="$1"
Line 241:		Line 241:
	./my_serial_program < "$infile" > "$outfile" 2>&1		./my_serial_program < "$infile" > "$outfile" 2>&1
	[...]		[...]
			</syntaxhighlight>
	</source>

	'''Notes:'''		'''Notes:'''
Line 293:		Line 293:
	The following sample script may serve as a blueprint for implementing a pre-termination function to perform clean-up or file recovery actions.		The following sample script may serve as a blueprint for implementing a pre-termination function to perform clean-up or file recovery actions.

	<~~source~~ lang="bash">		<syntaxhighlight lang="bash">
	#!/bin/bash		#!/bin/bash
	# Allocate one node		# Allocate one node
Line 375:		Line 375:

	exit		exit
			</syntaxhighlight>
	</source>

	'''Notes:'''		'''Notes:'''
Line 387:		Line 387:
	Sample job script template for a job running one multithreaded program instance:		Sample job script template for a job running one multithreaded program instance:

	<~~source~~ lang="bash">		<syntaxhighlight lang="bash">
	#!/bin/bash		#!/bin/bash
	# Allocate one node		# Allocate one node
Line 414:		Line 414:
	# Run multithreaded program		# Run multithreaded program
	./my_multithreaded_program		./my_multithreaded_program
			</syntaxhighlight>
	</source>

	Sample code for multithreaded program: [[Media:Hello_openmp.c \| Hello_openmp.c]]		Sample code for multithreaded program: [[Media:Hello_openmp.c \| Hello_openmp.c]]
Line 438:		Line 438:
	Sample job script template for an array job:		Sample job script template for an array job:

	<~~source~~ lang="bash">		<syntaxhighlight lang="bash">
	#!/bin/bash		#!/bin/bash
	# Number of cores per individual array task		# Number of cores per individual array task
Line 458:		Line 458:
	# ./my_program <input.$SLURM_ARRAY_TASK_ID		# ./my_program <input.$SLURM_ARRAY_TASK_ID

			</syntaxhighlight>
	</source>

	'''Notes:'''		'''Notes:'''
Line 519:		Line 519:
	Sample job script template for an MPI job:		Sample job script template for an MPI job:

	<~~source~~ lang="bash">		<syntaxhighlight lang="bash">
	#!/bin/bash		#!/bin/bash
	# Allocate two nodes		# Allocate two nodes
Line 566:		Line 566:
	srun ./my_mpi_program		srun ./my_mpi_program

			</syntaxhighlight>
	</source>

	Sample code for MPI program: [[Media:Hello_mpi.c \| Hello_mpi.c]]		Sample code for MPI program: [[Media:Hello_mpi.c \| Hello_mpi.c]]
Line 579:		Line 579:
	Sample job script template for an hybrid job:		Sample job script template for an hybrid job:

	<~~source~~ lang="bash">		<syntaxhighlight lang="bash">
	#!/bin/bash		#!/bin/bash
	# Number of nodes to allocate		# Number of nodes to allocate
Line 608:		Line 608:
	# mpirun ./my_hybrid_program		# mpirun ./my_hybrid_program

			</syntaxhighlight>
	</source>

	Sample code for hybrid program: [[Media:Hello_hybrid.c \| Hello_hybrid.c]]		Sample code for hybrid program: [[Media:Hello_hybrid.c \| Hello_hybrid.c]]

J Schäfer at 07:41, 31 August 2026

2026-08-31T07:41:56Z

← Older revision		Revision as of 09:41, 31 August 2026
Line 929:		Line 929:
	* 'AccountUtilizationByUser' can be used to report the aggregated cluster utilization of compute projects on Justus 3.		* 'AccountUtilizationByUser' can be used to report the aggregated cluster utilization of compute projects on Justus 3.
	* For regular users (as opposed to administrators) Slurm will only report their own usage records (but along with the total usage of the associated account in the case of 'AccountUtilizationByUser').		* For regular users (as opposed to administrators) Slurm will only report their own usage records (but along with the total usage of the associated account in the case of 'AccountUtilizationByUser').

	[[Category:syntaxhighlight]]

J Schäfer at 07:41, 31 August 2026

2026-08-31T07:41:09Z

← Older revision		Revision as of 09:41, 31 August 2026
Line 929:		Line 929:
	* 'AccountUtilizationByUser' can be used to report the aggregated cluster utilization of compute projects on Justus 3.		* 'AccountUtilizationByUser' can be used to report the aggregated cluster utilization of compute projects on Justus 3.
	* For regular users (as opposed to administrators) Slurm will only report their own usage records (but along with the total usage of the associated account in the case of 'AccountUtilizationByUser').		* For regular users (as opposed to administrators) Slurm will only report their own usage records (but along with the total usage of the associated account in the case of 'AccountUtilizationByUser').

			[[Category:syntaxhighlight]]

J Schäfer: /* How to retrieve historical resource usage for a specific user or account? */

2026-08-28T13:35:06Z

How to retrieve historical resource usage for a specific user or account?

← Older revision		Revision as of 15:35, 28 August 2026
Line 927:		Line 927:
	'''Notes:'''		'''Notes:'''
	* By default CPU resources will be reported. Use '-T' option for other trackable resources, e.g. '-T cpu,mem,gres/gpu,gres/scratch'.		* By default CPU resources will be reported. Use '-T' option for other trackable resources, e.g. '-T cpu,mem,gres/gpu,gres/scratch'.
	* ~~On JUSTUS 3 registered compute projects ("Rechenvorhaben") are uniquely mapped to Slurm accounts of the same name. Thus,~~ 'AccountUtilizationByUser' can ~~also~~ be used to report the aggregated cluster utilization of compute projects.		* 'AccountUtilizationByUser' can be used to report the aggregated cluster utilization of compute projects on Justus 3.
	* For regular users (as opposed to administrators) Slurm will only report their own usage records (but along with the total usage of the associated account in the case of 'AccountUtilizationByUser').		* For regular users (as opposed to administrators) Slurm will only report their own usage records (but along with the total usage of the associated account in the case of 'AccountUtilizationByUser').

J Schäfer: /* ADMINISTRATION */

2026-08-28T13:32:48Z

ADMINISTRATION

← Older revision		Revision as of 15:32, 28 August 2026
Line 929:		Line 929:
	* On JUSTUS 3 registered compute projects ("Rechenvorhaben") are uniquely mapped to Slurm accounts of the same name. Thus, 'AccountUtilizationByUser' can also be used to report the aggregated cluster utilization of compute projects.		* On JUSTUS 3 registered compute projects ("Rechenvorhaben") are uniquely mapped to Slurm accounts of the same name. Thus, 'AccountUtilizationByUser' can also be used to report the aggregated cluster utilization of compute projects.
	* For regular users (as opposed to administrators) Slurm will only report their own usage records (but along with the total usage of the associated account in the case of 'AccountUtilizationByUser').		* For regular users (as opposed to administrators) Slurm will only report their own usage records (but along with the total usage of the associated account in the case of 'AccountUtilizationByUser').

	== How to create/modify/delete QOSes? ==

	Suggested reading: https://slurm.schedmd.com/qos.html

	Examples:

	<pre>
	$ sacctmgr show qos # Show existing QOSes
	$ sacctmgr add qos verylong # Create new QOS verylong
	$ sacctmgr modify qos verylong set MaxWall=28-00:00:00 # Set maximum walltime limit
	$ sacctmgr modify qos verylong set MaxTRESPerUser=cpu=4 # Set maximum maximum number of CPUS a user can allocate at a given time
	$ sacctmgr modify qos verylong set flags=denyonlimit # Prevent submission if job requests exceed any limits of QOS
	$ sacctmgr modify user <username> set qos+=verylong # Add a QOS to a user account
	$ sacctmgr modify user <username> set qos-=verylong # Remove a QOS from a user account
	$ sacctmgr delete qos verylong # Delete that QOS
	</pre>

	== How to find (and fix) runaway jobs? ==

	<pre>$ sacctmgr show runaway</pre>

	'''Notes:'''
	* Runaway jobs are orphaned jobs that don't exist in the Slurm controller but have a start and no end time in the Slurm data base. Runaway jobs mess with accounting and affects new jobs of users who have too many runaway jobs.
	* If there are jobs in this state this command will also provide an option to fix them. This will set the end time for each job to the latest out of the start, eligible, or submit times, and set the state to completed.

	== How to show a history of database transactions? ==

	<pre>$ sacctmgr list transactions</pre>

	'''Note:''' Useful to get timestamps for when a user/account/qos has been created/modified/removed etc.

J Schäfer: /* ADMINISTRATION */

2026-08-28T13:27:41Z

ADMINISTRATION

← Older revision		Revision as of 15:27, 28 August 2026
Line 929:		Line 929:
	* On JUSTUS 3 registered compute projects ("Rechenvorhaben") are uniquely mapped to Slurm accounts of the same name. Thus, 'AccountUtilizationByUser' can also be used to report the aggregated cluster utilization of compute projects.		* On JUSTUS 3 registered compute projects ("Rechenvorhaben") are uniquely mapped to Slurm accounts of the same name. Thus, 'AccountUtilizationByUser' can also be used to report the aggregated cluster utilization of compute projects.
	* For regular users (as opposed to administrators) Slurm will only report their own usage records (but along with the total usage of the associated account in the case of 'AccountUtilizationByUser').		* For regular users (as opposed to administrators) Slurm will only report their own usage records (but along with the total usage of the associated account in the case of 'AccountUtilizationByUser').

	== How to fix/reset a user's RawUsage value? ==

	<pre>
	$ sacctmgr modify user <username> where Account=<account> set RawUsage=<number>
	</pre>

	== How to create/modify/delete QOSes? ==		== How to create/modify/delete QOSes? ==

J Schäfer: /* How to retrieve historical resource usage for a specific user or account? */

2026-08-28T13:25:17Z

How to retrieve historical resource usage for a specific user or account?

← Older revision		Revision as of 15:25, 28 August 2026
Line 928:		Line 928:
	* By default CPU resources will be reported. Use '-T' option for other trackable resources, e.g. '-T cpu,mem,gres/gpu,gres/scratch'.		* By default CPU resources will be reported. Use '-T' option for other trackable resources, e.g. '-T cpu,mem,gres/gpu,gres/scratch'.
	* On JUSTUS 3 registered compute projects ("Rechenvorhaben") are uniquely mapped to Slurm accounts of the same name. Thus, 'AccountUtilizationByUser' can also be used to report the aggregated cluster utilization of compute projects.		* On JUSTUS 3 registered compute projects ("Rechenvorhaben") are uniquely mapped to Slurm accounts of the same name. Thus, 'AccountUtilizationByUser' can also be used to report the aggregated cluster utilization of compute projects.
	* For regular users Slurm will only report their own usage records (but along with the total usage of the associated account in the case of 'AccountUtilizationByUser').		* For regular users (as opposed to administrators) Slurm will only report their own usage records (but along with the total usage of the associated account in the case of 'AccountUtilizationByUser').

	== How to fix/reset a user's RawUsage value? ==		== How to fix/reset a user's RawUsage value? ==

J Schäfer: /* How to retrieve historical resource usage for a specific user or account? */

2026-08-28T13:24:00Z

How to retrieve historical resource usage for a specific user or account?

← Older revision		Revision as of 15:24, 28 August 2026
Line 928:		Line 928:
	* By default CPU resources will be reported. Use '-T' option for other trackable resources, e.g. '-T cpu,mem,gres/gpu,gres/scratch'.		* By default CPU resources will be reported. Use '-T' option for other trackable resources, e.g. '-T cpu,mem,gres/gpu,gres/scratch'.
	* On JUSTUS 3 registered compute projects ("Rechenvorhaben") are uniquely mapped to Slurm accounts of the same name. Thus, 'AccountUtilizationByUser' can also be used to report the aggregated cluster utilization of compute projects.		* On JUSTUS 3 registered compute projects ("Rechenvorhaben") are uniquely mapped to Slurm accounts of the same name. Thus, 'AccountUtilizationByUser' can also be used to report the aggregated cluster utilization of compute projects.
	* ~~Can be executed by~~ regular users ~~as well in which case~~ Slurm will only report their own usage records (but along with the total usage of the associated account in the case of 'AccountUtilizationByUser').		* For regular users Slurm will only report their own usage records (but along with the total usage of the associated account in the case of 'AccountUtilizationByUser').

	== How to fix/reset a user's RawUsage value? ==		== How to fix/reset a user's RawUsage value? ==

J Schäfer: /* How to retrieve historical resource usage for a specific user or account? */

2026-08-28T13:22:52Z

How to retrieve historical resource usage for a specific user or account?

← Older revision		Revision as of 15:22, 28 August 2026
Line 927:		Line 927:
	'''Notes:'''		'''Notes:'''
	* By default CPU resources will be reported. Use '-T' option for other trackable resources, e.g. '-T cpu,mem,gres/gpu,gres/scratch'.		* By default CPU resources will be reported. Use '-T' option for other trackable resources, e.g. '-T cpu,mem,gres/gpu,gres/scratch'.
	* On JUSTUS 2 registered compute projects ("Rechenvorhaben") are uniquely mapped to Slurm accounts of the same name. Thus, 'AccountUtilizationByUser' can also be used to report the aggregated cluster utilization of compute projects.		* On JUSTUS 3 registered compute projects ("Rechenvorhaben") are uniquely mapped to Slurm accounts of the same name. Thus, 'AccountUtilizationByUser' can also be used to report the aggregated cluster utilization of compute projects.
	* Can be executed by regular users as well in which case Slurm will only report their own usage records (but along with the total usage of the associated account in the case of 'AccountUtilizationByUser').		* Can be executed by regular users as well in which case Slurm will only report their own usage records (but along with the total usage of the associated account in the case of 'AccountUtilizationByUser').

J Schäfer: /* How to retrieve historical resource usage for a specific user or account? */

2026-08-28T13:19:42Z

How to retrieve historical resource usage for a specific user or account?

← Older revision		Revision as of 15:19, 28 August 2026
Line 921:		Line 921:
	Examples:		Examples:
	<pre>		<pre>
	$ sreport cluster UserUtilizationByAccount Start=~~2021~~-01-01 End=~~2021~~-12-31 -t Hours user=<username> # Report cluster utilization of given user broken down by accounts		$ sreport cluster UserUtilizationByAccount Start=2025-01-01 End=2025-12-31 -t Hours user=<username> # Report cluster utilization of given user broken down by accounts
	$ sreport cluster AccountUtilizationByUser Start=~~2021~~-01-01 End=~~2021~~-12-31 -t Hours account=<account> # Report cluster utilization of given account broken down by users		$ sreport cluster AccountUtilizationByUser Start=2025-01-01 End=2025-12-31 -t Hours account=<account> # Report cluster utilization of given account broken down by users
	</pre>		</pre>