BinAC2/Slurm - Revision history

S Behnle: /* GPU jobs */

2026-01-07T08:47:56Z

GPU jobs

← Older revision		Revision as of 10:47, 7 January 2026
Line 425:		Line 425:

	<br/>		<br/>
	In case of using ~~Open MPI~~, the underlying communication infrastructure (UCX and Open MPI's BTL) is CUDA-aware.		In case of using OpenMPI, the underlying communication infrastructure (UCX and Open MPI's BTL) is CUDA-aware.
	However, there may be warnings, e.g. when running		However, there may be warnings, e.g. when running
	<pre>		<pre>
Line 448:		Line 448:
	</pre>		</pre>

	Please run ~~Open MPI~~'s mpirun using the following command:		Please run OpenMPI's mpirun using the following command:
	<pre>		<pre>
	$ mpirun --mca pml ucx --mca btl_openib_warn_default_gid_prefix 0 -np 2 ./mpi_cuda_app		$ mpirun --mca pml ucx --mca btl_openib_warn_default_gid_prefix 0 -np 2 ./mpi_cuda_app
	</pre>		</pre>
	or disabling the (older) communication layer Bit-Transfer-Layer (short BTL) ~~alltogether~~:		or disabling the (older) communication layer Bit-Transfer-Layer (short BTL) altogether:
	<pre>		<pre>
	$ mpirun --mca pml ucx --mca btl ^openib -np 2 ./mpi_cuda_app		$ mpirun --mca pml ucx --mca btl ^openib -np 2 ./mpi_cuda_app

S Behnle: /* GPU jobs */

2026-01-07T08:46:11Z

GPU jobs

← Older revision		Revision as of 10:46, 7 January 2026
Line 428:		Line 428:
	However, there may be warnings, e.g. when running		However, there may be warnings, e.g. when running
	<pre>		<pre>
	$ module load compiler/gnu/10.3 mpi/openmpi devel/~~cuad~~		$ module load compiler/gnu/10.3 mpi/openmpi devel/cuda
	$ mpirun mpirun -np 2 ./mpi_cuda_app		$ mpirun mpirun -np 2 ./mpi_cuda_app
	--------------------------------------		--------------------------------------

S Behnle at 08:45, 7 January 2026

2026-01-07T08:45:05Z

← Older revision		Revision as of 10:45, 7 January 2026
Line 377:		Line 377:
	==== GPU jobs ====		==== GPU jobs ====

	The nodes in the <code>gpu</code> queue have 2 or 8 NVIDIA A30/A100/H200 GPUs. Just submitting a job to these queues is not enough to also allocate one or more GPUs, you have to do so using the "--gres=gpu" parameter. You have to specifiy how many GPUs your job needs, e.g. "--gres=gpu:a30:2" will request two NVIDIA A30 GPUs.		The nodes in the <code>gpu</code> queue have 2 or 4 NVIDIA A30/A100/H200 GPUs. Just submitting a job to these queues is not enough to also allocate one or more GPUs, you have to do so using the "--gres=gpu" parameter. You have to specifiy how many GPUs your job needs, e.g. "--gres=gpu:a30:2" will request two NVIDIA A30 GPUs.

	The GPU nodes are shared between multiple jobs if the jobs don't request all the GPUs in a node and there are enough ressources to run more than one job. The individual GPUs are always bound to a single job and will not be shared between different jobs.		The GPU nodes are shared between multiple jobs if the jobs don't request all the GPUs in a node and there are enough ressources to run more than one job. The individual GPUs are always bound to a single job and will not be shared between different jobs.

S Behnle: updated GPU section

2025-09-08T07:42:25Z

updated GPU section

← Older revision		Revision as of 09:42, 8 September 2025
Line 377:		Line 377:
	==== GPU jobs ====		==== GPU jobs ====

	The nodes in the ~~gpu_4~~ ~~and gpu_8 queues~~ have 4 or 8 NVIDIA ~~Tesla V100~~ GPUs. Just submitting a job to these queues is not enough to also allocate one or more GPUs, you have to do so using the "--gres=gpu" parameter. You have to specifiy how many GPUs your job needs, e.g. "--gres=gpu:2" will request two GPUs.		The nodes in the <code>gpu</code> queue have 2 or 8 NVIDIA A30/A100/H200 GPUs. Just submitting a job to these queues is not enough to also allocate one or more GPUs, you have to do so using the "--gres=gpu" parameter. You have to specifiy how many GPUs your job needs, e.g. "--gres=gpu:a30:2" will request two NVIDIA A30 GPUs.

	The GPU nodes are shared between multiple jobs if the jobs don't request all the GPUs in a node and there are enough ressources to run more than one job. The individual GPUs are always bound to a single job and will not be shared between different jobs.		The GPU nodes are shared between multiple jobs if the jobs don't request all the GPUs in a node and there are enough ressources to run more than one job. The individual GPUs are always bound to a single job and will not be shared between different jobs.

	a) add after the initial line of your script job.sh the line including the		a) add after the initial line of your script job.sh the line including the
	information about the GPU usage:<br> #SBATCH --gres=gpu:2		information about the GPU usage:<br> #SBATCH --gres=gpu:a30:2
	<pre>		<pre>
	#!/bin/bash		#!/bin/bash
Line 388:		Line 388:
	#SBATCH --time=02:00:00		#SBATCH --time=02:00:00
	#SBATCH --mem=4000		#SBATCH --mem=4000
	#SBATCH --gres=gpu:2		#SBATCH --gres=gpu:a30:2
	</pre>		</pre>

	or b) execute:		or b) execute:
	<pre>		<pre>
	$ sbatch -p <queue> -n 40 -t 02:00:00 --mem 4000 --gres=gpu:2 job.sh		$ sbatch -p <queue> -n 40 -t 02:00:00 --mem 4000 --gres=gpu:a30:2 job.sh
	</pre>		</pre>
	<br/>		<br/>
Line 421:		Line 421:
	+-----------------------------------------------------------------------------+		+-----------------------------------------------------------------------------+
	</pre>		</pre>

			Upon successfull GPU ressource allocation, SLURM will set the environment variable <code>CUDA_VISIBLE_DEVICES</code> appropriately. <b>Do not change this variable!</b>

	<br/>		<br/>
Line 455:		Line 457:
	</pre>		</pre>

	(Please note, that CUDA per ~~v11~~.4 is only ~~available~~ with up to GCC-10)		(Please note, that CUDA per v12.8 is only officially supported with up to GCC-11)
	<br>		<br>
	<br>		<br>

F Bartusch: /* Interactive Jobs : srun / salloc */

2025-08-26T12:40:05Z

Interactive Jobs : srun / salloc

← Older revision		Revision as of 14:40, 26 August 2025
Line 55:		Line 55:
	\|}		\|}

	== Interactive Jobs ~~: srun / salloc~~ ==		== Interactive Jobs ==

	You can run interactive jobs for testing and developing your job scripts.		You can run interactive jobs for testing and developing your job scripts.
Line 72:		Line 72:
	srun --partition=interactive --time=4:00:00 --cpus-per-task=16 --mem=128gb --pty bash		srun --partition=interactive --time=4:00:00 --cpus-per-task=16 --mem=128gb --pty bash
	</pre>		</pre>




	== Job Submission : sbatch ==		== Job Submission : sbatch ==

F Bartusch at 11:15, 26 August 2025

2025-08-26T11:15:42Z

← Older revision		Revision as of 13:15, 26 August 2025
Line 54:		Line 54:
	\|-		\|-
	\|}		\|}

			== Interactive Jobs : srun / salloc ==

			You can run interactive jobs for testing and developing your job scripts.
			Several nodes are reserved for interactive work, so your jobs should start right away.
			You can only submit one job to this partition at a time. A job can run for up to 10 hours (about one workday).

			This example command gives you 16 cores and 128 GB of memory for four hours on one of the reserved nodes:

			<pre>
			salloc --partition=interactive --time=4:00:00 --cpus-per-task=16 --mem=128gb
			</pre>

			You can also use srun to request the same resources:

			<pre>
			srun --partition=interactive --time=4:00:00 --cpus-per-task=16 --mem=128gb --pty bash
			</pre>

S Behnle: Updated SLURM memory request

2025-08-11T10:41:27Z

Updated SLURM memory request

← Older revision		Revision as of 12:41, 11 August 2025
Line 113:		Line 113:
	\| <code>--mem=<size>[units]</code>		\| <code>--mem=<size>[units]</code>
	\| #SBATCH --mem=''value_in_MB''		\| #SBATCH --mem=''value_in_MB''
	\| Memory in MegaByte per node.		\| Memory in MegaByte per node.</br><code>[units]</code> can be one of <code>[K<nowiki>\|</nowiki>M<nowiki>\|</nowiki>G<nowiki>\|</nowiki>T]</code>.
	\| ~~Example~~ </br><code>--mem=~~10g~~</code> Request all memory on node		\| <code>--mem=10g</code> Request 10GB RAM per node </br> <code>--mem=0</code> Request all memory on node
	\| Depends on Slurm configuration.</br>It's better to specify <code>--mem</code> in every case.		\| Depends on Slurm configuration.</br>It is better to specify <code>--mem</code> in every case.
	\|-		\|-
	\|- style="vertical-align:top;"		\|- style="vertical-align:top;"

F Bartusch: /* OpenMPI */

2024-12-12T14:49:44Z

OpenMPI

← Older revision		Revision as of 16:49, 12 December 2024
Line 282:		Line 282:

	#SBATCH --partition compute		#SBATCH --partition compute
	#SBATCH --nodes 1		#SBATCH --nodes=2
	#SBATCH --ntasks 4		#SBATCH --ntasks-per-node=2
	#SBATCH --cpus-per-task=2		#SBATCH --cpus-per-task=2
	#SBATCH --mem-per-cpu=2000		#SBATCH --mem-per-cpu=2000
Line 300:		Line 300:
	Please type <code>man mpirun</code> for an explanation of the meaning of the different options of mpirun option <code>--map-by</code>.		Please type <code>man mpirun</code> for an explanation of the meaning of the different options of mpirun option <code>--map-by</code>.

	The above jobscript runs four OpenMPI tasks on a ~~single~~ ~~node~~. Because of hyperthreading you have to set <code>--cpus-per-task=2</code>. This means each MPI-task will get one physical core. If you omit <code>--cpus-per-task=2</code> MPI will fail.		The above jobscript runs four OpenMPI tasks, distributed between two nodes. Because of hyperthreading you have to set <code>--cpus-per-task=2</code>. This means each MPI-task will get one physical core. If you omit <code>--cpus-per-task=2</code> MPI will fail.


			'''Attention:''' Not all compute nodes are connected via Infiniband. Tell Slurm you want Infiniband via <code>--constraint=ib</code> when submitting or add <code>#SBATCH --constraint=ib</code> to your jobscript.

	<pre>		<pre>
	$ sbatch /pfs/10/project/examples/mpi_hello_world.sh		$ sbatch --constraint=ib /pfs/10/project/examples/mpi_hello_world.sh
	</pre>		</pre>

Line 309:		Line 312:

	<pre>		<pre>
			[...]
⚫	Hello world from processor ~~node1~~-~~083~~, rank 0 out of 4 processors
	Hello world from processor ~~node1~~-~~083~~, rank 2 out of 4 processors		Hello world from processor node2-031, rank 3 out of 4 processors
	Hello world from processor ~~node1~~-~~083~~, rank 1 out of 4 processors		Hello world from processor node2-031, rank 2 out of 4 processors
	Hello world from processor ~~node1~~-~~083~~, rank 3 out of 4 processors		Hello world from processor node2-030, rank 1 out of 4 processors
		⚫	Hello world from processor node2-030, rank 0 out of 4 processors

	</pre>		</pre>

F Bartusch: /* OpenMPI */

2024-12-12T14:43:02Z

OpenMPI

@@ Line 276: / Line 276: @@
 ==== OpenMPI ====
-If you want to run jobs on batch nodes, generate a wrapper script ''job_ompi.sh'' for '''OpenMPI''' containing the following lines:
+If you want to run MPI-jobs on batch nodes, generate a wrapper script <code>mpi_hello_world.sh</code> for '''OpenMPI''' containing the following lines:
 <source lang="bash">
 #!/bin/bash
 </source>
-'''Attention:''' Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since Slurm instructs mpirun about number of processes and node hostnames. Use '''ALWAYS''' the MPI options '''''--bind-to core''''' and '''''--map-by core|socket|node'''''. Please type ''mpirun --help'' for an explanation of the meaning of the different options of mpirun option ''--map-by''.
+'''Attention:''' Do '''NOT''' add mpirun options <code>-n <number_of_processes></code> or any other option defining processes or nodes, since Slurm instructs mpirun about number of processes and node hostnames.
 <pre>
 </pre>
 ==== Multithreaded + MPI parallel Programs ====

F Bartusch: Delete chain jobs

2024-12-12T14:05:11Z

Delete chain jobs

@@ Line 329: / Line 329: @@
 * The option ''-report-bindings'' shows the bindings between MPI tasks and physical cores.
 * The mpirun-options '''--bind-to core''', '''--map-by socket|...|node:PE=<value>''' should always be used when running a multithreaded MPI program.
 <br>