NEMO2/Slurm - Revision history

M Janczyk: /* Partitions */

2026-04-09T13:06:04Z

Partitions

← Older revision		Revision as of 15:06, 9 April 2026
Line 169:		Line 169:
	\| single node		\| single node
	\| 2x AMD EPYC 9655 (Turin)		\| 2x AMD EPYC 9655 (Turin)
	\| 8x 4x NVIDIA RTX PRO 6000 Blackwell Server Edition		\| 4x NVIDIA RTX PRO 6000 Blackwell Server Edition
	\| 47		\| 47
	\| ntasks=1, time=01:00:00, mem-per-cpu=1gb		\| ntasks=1, time=01:00:00, mem-per-cpu=1gb

M Janczyk: /* Monitoring Jobs with squeue */

2026-02-02T16:46:00Z

Monitoring Jobs with squeue

← Older revision		Revision as of 18:46, 2 February 2026
Line 206:		Line 206:
	<syntaxhighlight lang='shell'>		<syntaxhighlight lang='shell'>
	> squeue		> squeue
	JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)		JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
	426 cpu 20k fr_xy0123 R 0:02 2 n[4101-4102]		426 cpu 20k fr_xy0123 R 0:02 2 n[4101-4102]
	</syntaxhighlight>		</syntaxhighlight>

M Janczyk: /* Monitoring Jobs with squeue */

2026-02-02T16:45:30Z

Monitoring Jobs with squeue

← Older revision		Revision as of 18:45, 2 February 2026
Line 207:		Line 207:
	> squeue		> squeue
	JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)		JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
	426 cpu 20k ~~fr_0123~~ R 0:02 2 n[4101-4102]		426 cpu 20k fr_xy0123 R 0:02 2 n[4101-4102]
	</syntaxhighlight>		</syntaxhighlight>

M Janczyk: /* Partitions */

2026-02-02T16:45:06Z

Partitions

← Older revision		Revision as of 18:45, 2 February 2026
Line 110:		Line 110:
	{\| class="wikitable"		{\| class="wikitable"
	\|-		\|-
	! style="width:7%"\| Partition		! style="width:5%"\| Partition
	! style="width:8%"\| Multi Node		! style="width:10%"\| Multi Node
	! style="width:18%"\| Nodes		! style="width:20%"\| Nodes
	! style="width:12%"\| APU/GPU		! style="width:20%"\| APU/GPU
	! style="width:5%"\| Cores per APU/GPU		! style="width:10%"\| Cores per APU/GPU
	! style="width:20%"\| Defaults		! style="width:20%"\| Defaults
	! style="width:20%"\| Limits		! style="width:15%"\| Limits
	\|-		\|-
	\| cpu*		\| cpu*

M Janczyk at 16:39, 2 February 2026

2026-02-02T16:39:32Z

← Older revision		Revision as of 18:39, 2 February 2026
Line 100:		Line 100:
	On bwForCluster NEMO 2 it is '''optional''' to request a partition with <code>--partition=<partition_name></code> on job submission. Within a partition job allocations are routed automatically to the most suitable compute node(s) for the requested resources (e.g. amount of nodes and cores, memory, number of GPUs). The 'cpu' partition is the default partition, if no partition is requested. It will start jobs either on 'milan' or 'genoa' nodes.		On bwForCluster NEMO 2 it is '''optional''' to request a partition with <code>--partition=<partition_name></code> on job submission. Within a partition job allocations are routed automatically to the most suitable compute node(s) for the requested resources (e.g. amount of nodes and cores, memory, number of GPUs). The 'cpu' partition is the default partition, if no partition is requested. It will start jobs either on 'milan' or 'genoa' nodes.

	The partitions 'cpu', 'l40s', 'mi300a' and '~~h200~~' only allow the use of single nodes. You cannot start jobs that use more than one node, either by specifying <code>--nodes</code> (>1) or <code>--ntasks</code> (more than available on one node). If you do so, your jobs will be blocked with the reason (PartitionNodeLimit).		The partitions 'cpu', 'l40s', 'mi300a', 'h200' and 'rtx' only allow the use of single nodes. You cannot start jobs that use more than one node, either by specifying <code>--nodes</code> (>1) or <code>--ntasks</code> (more than available on one node). If you do so, your jobs will be blocked with the reason (PartitionNodeLimit).

	All partitions ~~cpu, milan, genoa, l40s, mi300a and h200~~ are operated in shared mode, i.e. jobs from different users can run on the same node. Jobs can get exclusive access to compute nodes in these partitions with the <code>--exclusive</code> option.		'''All''' partitions are operated in shared mode, i.e. jobs from different users can run on the same node. Jobs can get exclusive access to compute nodes in these partitions with the <code>--exclusive</code> option.

	The login partition is intended for quick tests on the login lodes with the Slurm environment.		The login partition is intended for quick tests on the login lodes with the Slurm environment.
Line 165:		Line 165:
	\| ntasks=1, time=01:00:00, mem-per-cpu=1gb		\| ntasks=1, time=01:00:00, mem-per-cpu=1gb
	\| time=48:00:00, nodes=1, gpus=8		\| time=48:00:00, nodes=1, gpus=8
			\|-
			\| rtx
			\| single node
			\| 2x AMD EPYC 9655 (Turin)
			\| 8x 4x NVIDIA RTX PRO 6000 Blackwell Server Edition
			\| 47
			\| ntasks=1, time=01:00:00, mem-per-cpu=1gb
			\| time=48:00:00, nodes=1, gpus=4
	\|-		\|-
	\| login		\| login

M Janczyk at 14:15, 16 December 2025

2025-12-16T14:15:16Z

← Older revision		Revision as of 16:15, 16 December 2025
Line 158:		Line 158:
	\| time=48:00:00, nodes=1, gpus=4		\| time=48:00:00, nodes=1, gpus=4
	\|-		\|-
	\| h200 ~~(August 2025)~~		\| h200
	\| single node		\| single node
	\| 2x AMD EPYC 9654 (Genoa)		\| 2x AMD EPYC 9654 (Genoa)

M Janczyk: /* Interactive access to running jobs */

2025-07-23T13:23:44Z

Interactive access to running jobs

← Older revision		Revision as of 15:23, 23 July 2025
Line 357:		Line 357:
	Commands like <code>htop</code> show you the most busy processes on the node. To exit <code>htop</code> type <code>q</code>.		Commands like <code>htop</code> show you the most busy processes on the node. To exit <code>htop</code> type <code>q</code>.

	To monitor your GPU processes use the command </code>nvidia-smi</code> on AMD APUs <code>rocm-smi</code>.		To monitor your GPU processes use the command <code>nvidia-smi</code> on AMD APUs <code>rocm-smi</code>.

	== Job Feedback ==		== Job Feedback ==

M Janczyk at 13:20, 23 July 2025

2025-07-23T13:20:32Z

← Older revision		Revision as of 15:20, 23 July 2025
Line 35:		Line 35:
	#SBATCH --time=00:14:00		#SBATCH --time=00:14:00
	#SBATCH --mem=1gb		#SBATCH --mem=1gb
			...
	echo 'Here starts the calculation'
	</syntaxhighlight>		</syntaxhighlight>

Line 98:		Line 98:
	== Partitions ==		== Partitions ==

	On bwForCluster NEMO 2 it is '''optional''' to request a partition with '--partition=<partition_name>' on job submission. Within a partition job allocations are routed automatically to the most suitable compute node(s) for the requested resources (e.g. amount of nodes and cores, memory, number of GPUs). The 'cpu' partition is the default partition, if no partition is requested. It will start jobs either on 'milan' or 'genoa' nodes.		On bwForCluster NEMO 2 it is '''optional''' to request a partition with <code>--partition=<partition_name></code> on job submission. Within a partition job allocations are routed automatically to the most suitable compute node(s) for the requested resources (e.g. amount of nodes and cores, memory, number of GPUs). The 'cpu' partition is the default partition, if no partition is requested. It will start jobs either on 'milan' or 'genoa' nodes.

	The partitions 'cpu', 'l40s', 'mi300a' and 'h200' only allow the use of single nodes. You cannot start jobs that use more than one node, either by specifying --nodes (>1) or --ntasks (more than available on one node). If you do so, your jobs will be blocked with the reason (PartitionNodeLimit).		The partitions 'cpu', 'l40s', 'mi300a' and 'h200' only allow the use of single nodes. You cannot start jobs that use more than one node, either by specifying <code>--nodes</code> (>1) or <code>--ntasks</code> (more than available on one node). If you do so, your jobs will be blocked with the reason (PartitionNodeLimit).

	All partitions cpu, milan, genoa, l40s, mi300a and h200 are operated in shared mode, i.e. jobs from different users can run on the same node. Jobs can get exclusive access to compute nodes in these partitions with the <code>--exclusive</code> option.		All partitions cpu, milan, genoa, l40s, mi300a and h200 are operated in shared mode, i.e. jobs from different users can run on the same node. Jobs can get exclusive access to compute nodes in these partitions with the <code>--exclusive</code> option.
Line 291:		Line 291:
	* <code>--mem</code> requests the memory per node. The maximum is 500gb for partition milan.		* <code>--mem</code> requests the memory per node. The maximum is 500gb for partition milan.
	* The Compiler and MPI modules used for the compilation must be loaded before the start of the program.		* The Compiler and MPI modules used for the compilation must be loaded before the start of the program.
	* It is recommended to start MPI programs with <code>srun</code>.		* '''It is recommended to start MPI programs with <code>srun</code>.'''

	= Interactive Jobs =		= Interactive Jobs =
Line 307:		Line 307:

	Once the walltime limit has been reached you will be automatically logged out from the compute node.		Once the walltime limit has been reached you will be automatically logged out from the compute node.

			== Interactive Tests on the Login Nodes ==

			The login partition is intended for quick tests on the login lodes with the Slurm environment. The limits are 4 cores and 15 minutes of walltime.

			<syntaxhighlight lang="shell">
			$ salloc --partition=login --ntasks=4
			$ srun -p login <application>
			</syntaxhighlight>

	= Job Monitoring =		= Job Monitoring =
Line 338:		Line 347:
	This will show average CPU time, average and maximum memory consumption of all tasks in the running job.		This will show average CPU time, average and maximum memory consumption of all tasks in the running job.

	'sstat -e' command shows a list of fields that can be specified with the '--format' option.		<code>sstat -e</code> command shows a list of fields that can be specified with the <code>--format</code> option.

	== Interactive access to running jobs ==		== Interactive access to running jobs ==
Line 346:		Line 355:
	<syntaxhighlight lang="shell">$ srun --jobid=<jobid> --overlap --pty /bin/bash</syntaxhighlight>		<syntaxhighlight lang="shell">$ srun --jobid=<jobid> --overlap --pty /bin/bash</syntaxhighlight>

	Commands like ~~'top'~~ show you the most busy processes on the node. To exit ~~'top'~~ type 'q'.		Commands like <code>htop</code> show you the most busy processes on the node. To exit <code>htop</code> type <code>q</code>.

	To monitor your GPU processes use the command 'nvidia-smi'.		To monitor your GPU processes use the command </code>nvidia-smi</code> on AMD APUs <code>rocm-smi</code>.

	== Job Feedback ==		== Job Feedback ==
Line 380:		Line 389:
	* CPU Efficiency: 'CPU Utilized' with respect to core-walltime (= 'Nodes' x 'Cores per node' x 'Job Wall-clock time') in percent.		* CPU Efficiency: 'CPU Utilized' with respect to core-walltime (= 'Nodes' x 'Cores per node' x 'Job Wall-clock time') in percent.
	* Job Wall-clock time: runtime of the job.		* Job Wall-clock time: runtime of the job.
	* Memory Utilized: Sum of memory used. For multi node MPI jobs the sum is only correct when srun is used instead of mpirun.		* Memory Utilized: Sum of memory used. For multi node MPI jobs the sum is only correct when <code>srun</code> is used instead of <code>mpirun</code>.
	* Memory Efficiency: 'Memory Utilized' with respect to total allocated memory for the job.		* Memory Efficiency: 'Memory Utilized' with respect to total allocated memory for the job.

M Janczyk at 13:08, 23 July 2025

2025-07-23T13:08:21Z

← Older revision		Revision as of 15:08, 23 July 2025
Line 25:		Line 25:
	Batch jobs are submitted with the command:		Batch jobs are submitted with the command:

	<syntaxhighlight lang=~~bash~~>$ sbatch <job-script></syntaxhighlight>		<syntaxhighlight lang=shell>$ sbatch <job-script></syntaxhighlight>

	A job script contains options for Slurm in lines beginning with <code>#SBATCH</code> as well as your commands which you want to execute on the compute nodes. For example:		A job script contains options for Slurm in lines beginning with <code>#SBATCH</code> as well as your commands which you want to execute on the compute nodes. For example:

	<syntaxhighlight lang='~~bash~~'>		<syntaxhighlight lang='shell'>
	#!/bin/bash		#!/bin/bash
	#SBATCH --nodes=1		#SBATCH --nodes=1
Line 39:		Line 39:

	You can override options from the script on the command-line:		You can override options from the script on the command-line:
	<syntaxhighlight lang=~~bash~~>$ sbatch --time=03:00:00 <job-script></syntaxhighlight>		<syntaxhighlight lang=shell>$ sbatch --time=03:00:00 <job-script></syntaxhighlight>

	== Resource Requests ==		== Resource Requests ==
Line 102:		Line 102:
	The partitions 'cpu', 'l40s', 'mi300a' and 'h200' only allow the use of single nodes. You cannot start jobs that use more than one node, either by specifying --nodes (>1) or --ntasks (more than available on one node). If you do so, your jobs will be blocked with the reason (PartitionNodeLimit).		The partitions 'cpu', 'l40s', 'mi300a' and 'h200' only allow the use of single nodes. You cannot start jobs that use more than one node, either by specifying --nodes (>1) or --ntasks (more than available on one node). If you do so, your jobs will be blocked with the reason (PartitionNodeLimit).

	All partitions cpu, milan, genoa, l40s, mi300a and h200 are operated in shared mode, i.e. jobs from different users can run on the same node. Jobs can get exclusive access to compute nodes in these partitions with the "--exclusive" option.		All partitions cpu, milan, genoa, l40s, mi300a and h200 are operated in shared mode, i.e. jobs from different users can run on the same node. Jobs can get exclusive access to compute nodes in these partitions with the <code>--exclusive</code> option.

	The login partition is intended for quick tests on the login lodes with the Slurm environment.		The login partition is intended for quick tests on the login lodes with the Slurm environment.
Line 231:		Line 231:
	scontrol write batch_script 426 -		scontrol write batch_script 426 -
	</code>		</code>
	display job script of a running job. The "-" is a special filename which means "write to the terminal".		display job script of a running job. The <code>-</code> is a special filename which means "write to the terminal".

	== Monitoring a Started Job ==		== Monitoring a Started Job ==
Line 298:		Line 298:
	The following examples requests an interactive session on 1 core for 2 hours.		The following examples requests an interactive session on 1 core for 2 hours.

	<syntaxhighlight lang="~~bash~~">$ salloc --partition=cpu --ntasks=1 --time=2:00:00</syntaxhighlight>		<syntaxhighlight lang="shell">$ salloc --partition=cpu --ntasks=1 --time=2:00:00</syntaxhighlight>

	After execution of this command wait until the queueing system has granted you the requested resources. Once granted you will be automatically logged on the allocated compute node.		After execution of this command wait until the queueing system has granted you the requested resources. Once granted you will be automatically logged on the allocated compute node.
Line 304:		Line 304:
	If you use applications or tools which provide a GUI, enable X-forwarding for your interactive session with. We suggest using VNC instead of X-forwarding:		If you use applications or tools which provide a GUI, enable X-forwarding for your interactive session with. We suggest using VNC instead of X-forwarding:

	<syntaxhighlight lang="~~bash~~">$ salloc --partition=cpu --ntasks=1 --time=2:00:00 --x11</syntaxhighlight>		<syntaxhighlight lang="shell">$ salloc --partition=cpu --ntasks=1 --time=2:00:00 --x11</syntaxhighlight>

	Once the walltime limit has been reached you will be automatically logged out from the compute node.		Once the walltime limit has been reached you will be automatically logged out from the compute node.
Line 314:		Line 314:
	For an overview of your submitted jobs use the command:		For an overview of your submitted jobs use the command:

	<syntaxhighlight lang="~~bash~~">$ squeue</syntaxhighlight>		<syntaxhighlight lang="shell">$ squeue</syntaxhighlight>

	To get detailed information about a specific job use the command:		To get detailed information about a specific job use the command:

	<syntaxhighlight lang="~~bash~~">$ scontrol show job <jobid></syntaxhighlight>		<syntaxhighlight lang="shell">$ scontrol show job <jobid></syntaxhighlight>

	'''Notes:'''<br>		'''Notes:'''<br>
Line 344:		Line 344:
	It is also possible to attach an interactive shell to a running job with command:		It is also possible to attach an interactive shell to a running job with command:

	<syntaxhighlight lang="~~bash~~">$ srun --jobid=<jobid> --overlap --pty /bin/bash</syntaxhighlight>		<syntaxhighlight lang="shell">$ srun --jobid=<jobid> --overlap --pty /bin/bash</syntaxhighlight>

	Commands like 'top' show you the most busy processes on the node. To exit 'top' type 'q'.		Commands like 'top' show you the most busy processes on the node. To exit 'top' type 'q'.
Line 354:		Line 354:
	You get feedback on resource usage and job efficiency for completed jobs with the command:		You get feedback on resource usage and job efficiency for completed jobs with the command:

	<syntaxhighlight lang="~~bash~~">		<syntaxhighlight lang="shell">
	$ seff <jobid>		$ seff <jobid>
	</syntaxhighlight>		</syntaxhighlight>
Line 360:		Line 360:
	Example Output:		Example Output:

	<syntaxhighlight lang="~~bash~~">		<syntaxhighlight lang="shell">
	Job ID: 426		Job ID: 426
	Cluster: nemo		Cluster: nemo
Line 421:		Line 421:
	To see the accounting data of a specific job:		To see the accounting data of a specific job:

	<syntaxhighlight lang="~~bash~~">$ sacct -j <jobid> --format=user,jobid,account,nnodes,ncpus,time,elapsed,AllocTRES%50</syntaxhighlight>		<syntaxhighlight lang="shell">$ sacct -j <jobid> --format=user,jobid,account,nnodes,ncpus,time,elapsed,AllocTRES%50</syntaxhighlight>

	To retrive the job history for a specific user for a certain time frame:		To retrive the job history for a specific user for a certain time frame:

	<syntaxhighlight lang="~~bash~~">$ sacct -u <user> -S 2025-03-01 -E 2025-03-02 --format=user,jobid,account,nnodes,ncpus,time,elapsed,AllocTRES%50</syntaxhighlight>		<syntaxhighlight lang="shell">$ sacct -u <user> -S 2025-03-01 -E 2025-03-02 --format=user,jobid,account,nnodes,ncpus,time,elapsed,AllocTRES%50</syntaxhighlight>

M Janczyk: /* Job Examples */

2025-07-23T13:03:44Z

Job Examples

← Older revision		Revision as of 15:03, 23 July 2025
Line 256:		Line 256:

	'''Notes:'''		'''Notes:'''
	* Jobs with "--mem" up to 500gb can run on all node types associated with the cpu partition.		* Jobs with <code>--mem</code> up to 500gb can run on all node types associated with the cpu partition.

	=== Multi-threaded Programs ===		=== Multi-threaded Programs ===
Line 273:		Line 273:

	'''Notes:'''		'''Notes:'''
	* Jobs with "--ntasks-per-node" up to 126 and "--mem" up to 500gb can run on all node types associated with the milan partition.		* Jobs with <code>--ntasks-per-node</code> up to 126 and <code>--mem</code> up to 500gb can run on all node types associated with the milan partition.
	* With "export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}" you can set the number of threads according to the number of resources requested.		* With <code>export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}</code> you can set the number of threads according to the number of resources requested.

	=== MPI Programs ===		=== MPI Programs ===

← Older revision		Revision as of 18:45, 2 February 2026
Line 110:		Line 110:
	{\| class="wikitable"		{\| class="wikitable"
	\|-		\|-
	! style="width:7%"\| Partition		! style="width:5%"\| Partition
	! style="width:8%"\| Multi Node		! style="width:10%"\| Multi Node
	! style="width:18%"\| Nodes		! style="width:20%"\| Nodes
	! style="width:12%"\| APU/GPU		! style="width:20%"\| APU/GPU
	! style="width:5%"\| Cores per APU/GPU		! style="width:10%"\| Cores per APU/GPU
	! style="width:20%"\| Defaults		! style="width:20%"\| Defaults
	! style="width:20%"\| Limits		! style="width:15%"\| Limits
	\|-		\|-
	\| cpu*		\| cpu*