BwUniCluster2.0/Batch System Migration Guide - Revision history

R Barthel: R Barthel moved page BwUniCluster 2.0 Batch System Migration Guide to BwUniCluster2.0/Batch System Migration Guide

2022-10-07T21:23:52Z

R Barthel moved page BwUniCluster 2.0 Batch System Migration Guide to BwUniCluster2.0/Batch System Migration Guide

← Older revision	Revision as of 23:23, 7 October 2022
(No difference)

H Haefner: /* General Overview */

2020-04-23T14:39:45Z

General Overview

← Older revision		Revision as of 16:39, 23 April 2020
Line 111:		Line 111:
	\| QOS name \|\| $MOAB_QOS \|\| $SLURM_JOB_QOS		\| QOS name \|\| $MOAB_QOS \|\| $SLURM_JOB_QOS
	\|-		\|-
	\| Number of processes per node \| --- \|\| $SLURM_TASKS_PER_NODE		\| Number of processes per node \|\| --- \|\| $SLURM_TASKS_PER_NODE
	\|-		\|-
	\| Job user \|\| $MOAB_USER \|\| $SLURM_JOB_USER		\| Job user \|\| $MOAB_USER \|\| $SLURM_JOB_USER

H Haefner: /* General Overview */

2020-04-23T14:37:34Z

General Overview

← Older revision		Revision as of 16:37, 23 April 2020
Line 77:		Line 77:

	'''Notes:'''		'''Notes:'''
	* Default initial job working directory is $HOME for ~~Moab~~. For Slurm the default working directory is where you submit your job from.		* Default initial job working directory is $HOME for MOAB. For Slurm the default working directory is where you submit your job from.
	* By default ~~Moab~~ does not export any environment variables to the job's runtime environment. With Slurm most of the login environment variables are exported to your job's runtime environment. This includes environment variables from software modules that were loaded at job submission time (and also $HOSTNAME variable).		* By default MOAB does not export any environment variables to the job's runtime environment. With Slurm most of the login environment variables are exported to your job's runtime environment. This includes environment variables from software modules that were loaded at job submission time (and also $HOSTNAME variable).


⚫	'''Commonly used ~~Moab/Torque~~ script environment variables and their Slurm equivalents

		⚫	'''Commonly used MOAB script environment variables and their Slurm equivalents

	{\| width=750px class="wikitable"		{\| width=750px class="wikitable"
	! Information !! ~~Moab~~ ~~!! Torque~~ !! Slurm		! Information !! MOAB !! Slurm
	\|-		\|-
	\| Job name \|\| $MOAB_JOBNAME ~~\|\| $PBS_JOBNAME~~ \|\| $SLURM_JOB_NAME		\| Job name \|\| $MOAB_JOBNAME \|\| $SLURM_JOB_NAME
	\|-		\|-
	\| Job ID \|\| $MOAB_JOBID ~~\|\| $PBS_JOBID~~ \|\| $SLURM_JOB_ID		\| Job ID \|\| $MOAB_JOBID \|\| $SLURM_JOB_ID
	\|-		\|-
	\| Submit directory \|\| $MOAB_SUBMITDIR ~~\|\| $PBS_O_WORKDIR~~ \|\| $SLURM_SUBMIT_DIR		\| Submit directory \|\| $MOAB_SUBMITDIR \|\| $SLURM_SUBMIT_DIR
	\|-		\|-
	\| Number of nodes allocated \|\| $MOAB_NODECOUNT ~~\|\| $PBS_NUM_NODES~~ \|\| $SLURM_JOB_NUM_NODES (and: $SLURM_NNODES)		\| Number of nodes allocated \|\| $MOAB_NODECOUNT \|\| $SLURM_JOB_NUM_NODES (and: $SLURM_NNODES)
	\|-		\|-
	\| Node list \|\| $MOAB_NODELIST ~~\|\| cat $PBS_NODEFILE~~ \|\| $SLURM_JOB_NODELIST		\| Node list \|\| $MOAB_NODELIST \|\| $SLURM_JOB_NODELIST
	\|-		\|-
	\| Number of processes \|\| $MOAB_PROCCOUNT ~~\|\| $PBS_TASKNUM~~ \|\| $SLURM_NTASKS		\| Number of processes \|\| $MOAB_PROCCOUNT \|\| $SLURM_NTASKS
	\|-		\|-
	\| Requested tasks per node \|\| --- ~~\|\| $PBS_NUM_PPN~~ \|\| $SLURM_NTASKS_PER_NODE		\| Requested tasks per node \|\| --- \|\| $SLURM_NTASKS_PER_NODE
	\|-		\|-
	\| Requested CPUs per task \|\| --- ~~\|\| ---~~ \|\| $SLURM_CPUS_PER_TASK		\| Requested CPUs per task \|\| --- \|\| $SLURM_CPUS_PER_TASK
	\|-		\|-
	\| Job array index \|\| $MOAB_JOBARRAYINDEX ~~\|\| $PBS_ARRAY_INDEX~~ \|\| $SLURM_ARRAY_TASK_ID		\| Job array index \|\| $MOAB_JOBARRAYINDEX \|\| $SLURM_ARRAY_TASK_ID
	\|-		\|-
	\| Job array range \|\| $MOAB_JOBARRAYRANGE ~~\|\| -~~ \|\| $SLURM_ARRAY_TASK_COUNT		\| Job array range \|\| $MOAB_JOBARRAYRANGE \|\| $SLURM_ARRAY_TASK_COUNT
	\|-		\|-
	\| Queue name \|\| $MOAB_CLASS ~~\|\| $PBS_QUEUE~~ \|\| $SLURM_JOB_PARTITION		\| Queue name \|\| $MOAB_CLASS \|\| $SLURM_JOB_PARTITION
	\|-		\|-
	\| QOS name \|\| $MOAB_QOS ~~\|\| ---~~ \|\| $SLURM_JOB_QOS		\| QOS name \|\| $MOAB_QOS \|\| $SLURM_JOB_QOS
	\|-		\|-
	\| Number of processes per node \| --- ~~\|\| $PBS_NUM_PPN~~ \|\| $SLURM_TASKS_PER_NODE		\| Number of processes per node \| --- \|\| $SLURM_TASKS_PER_NODE
	\|-		\|-
	\| Job user \|\| $MOAB_USER ~~\|\| $PBS_O_LOGNAME~~ \|\| $SLURM_JOB_USER		\| Job user \|\| $MOAB_USER \|\| $SLURM_JOB_USER
	\|-		\|-
	\| Hostname \|\| $MOAB_MACHINE ~~\|\| $PBS_O_HOST~~ \|\| $SLURMD_NODENAME		\| Hostname \|\| $MOAB_MACHINE \|\| $SLURMD_NODENAME
	\|}		\|}

			<br>
	The following chapters contain short examples for the most common job types. For a full overview of SLURM see the [[BwUniCluster_2.0_Slurm_common_Features\|full article]].

	=Serial Programs=		=Serial Programs=

H Haefner: /* General Overview */

2020-04-23T14:28:18Z

General Overview

← Older revision		Revision as of 16:28, 23 April 2020
Line 24:		Line 24:
	\| checkjob $JOBID \|\| scontrol show job $JOBID		\| checkjob $JOBID \|\| scontrol show job $JOBID
	\|}		\|}

H Haefner: /* How to convert Moab batch job scripts to Slurm? */

2020-04-23T14:27:41Z

How to convert Moab batch job scripts to Slurm?

← Older revision		Revision as of 16:27, 23 April 2020
Line 26:		Line 26:


	== How to convert Moab batch job scripts to Slurm? ==

	Replace Moab/Torque job specification flags and environment variables in your job
	scripts by their corresponding Slurm counterparts.

			In the next table you find the most MOAB job specification flags and environment variables. You have to replace all MOAB flags and environment variables of your batch scripts by their corresponding Slurm counterparts.
⚫	'''Commonly used ~~Moab~~ job specification flags and their Slurm equivalents'''

		⚫	'''Commonly used MOAB job specification flags and their Slurm equivalents'''

	{\| width=750px class="wikitable"		{\| width=750px class="wikitable"

H Haefner: /* General Overview */

2020-04-23T14:21:54Z

General Overview

@@ Line 23: / Line 23: @@
 |-
 | checkjob $JOBID || scontrol show job $JOBID
 |}

S Raffeiner at 09:28, 17 March 2020

2020-03-17T09:28:44Z

← Older revision		Revision as of 11:28, 17 March 2020
Line 1:		Line 1:
			While the former bwUniCluster 1 system used the combination Moab/SLURM for the batch system queue, bwUniCluster 2.0 uses only SLURM. This means that most job scripts and workflows which relied on Moab-specific pragmas and commands have to be changed.

			=General Overview=

			Job parameters can be passed to SLURM in the same ways which were possible with Moab.

			* Instead of the #MOAB or #PBS pragmas, the pragma #SLURM has to be used within job files.

			* Instead of the Moab commands, the corresponding SLURM commands have to be used.

			A general mapping of Moab to SLURM commands can be found in the following table:

			{\| class="wikitable"
			! Moab command !! SLURM command
			\|-
			\| msub \|\| sbatch
			\|-
			\| msub -I \|\| salloc
			\|-
			\| canceljob \|\| scancel
			\|-
			\| showq \|\| squeue
			\|-
			\| checkjob $JOBID \|\| scontrol show job $JOBID
			\|}

			The following chapters contain short examples for the most common job types. For a full overview of SLURM see the [[BwUniCluster_2.0_Slurm_common_Features\|full article]].

	=Serial Programs=		=Serial Programs=

H Haefner: /* Multithreaded + MPI Parallel Programs on many nodes */

2020-03-13T12:58:07Z

Multithreaded + MPI Parallel Programs on many nodes

← Older revision		Revision as of 14:58, 13 March 2020
Line 71:		Line 71:

	* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.		* Use the time option '''-t''' or '''--time''' (instead of '''-l walltime'''). If only one number is entered behind '''-t''', the default unit is minutes.
	* Use the option '''-N ''y''''' or '''--nodes=''y''''' and '''--ntasks-per-node=''x''''' and '''c ''z''''' or '''--cpus-per-task=''z''''' (instead of '''-l nodes=''y'',ppn=''x+z'''''). '''''x''''' usually should be 1 or 2 and '''''x+z''''' usually 40 (28 on Broadwell nodes); you can utilize hyperthreading if you want.		* Use the option '''-N ''y''''' or '''--nodes=''y''''' and '''--ntasks-per-node=''x''''' and '''-c ''z''''' or '''--cpus-per-task=''z''''' (instead of '''-l nodes=''y'',ppn=''x+z'''''). '''''x''''' usually should be 1 or 2 and '''''x+z''''' usually 40 (28 on Broadwell nodes); you can utilize hyperthreading if you want.
	* You should'nt use the option '''-m''' or '''--mem''' because the nodes are used exclusively.		* You should'nt use the option '''-m''' or '''--mem''' because the nodes are used exclusively.
	* You always use the nodes exclusively.		* You always use the nodes exclusively.

H Haefner at 12:55, 13 March 2020

2020-03-13T12:55:25Z

← Older revision		Revision as of 14:55, 13 March 2020
Line 39:		Line 39:
	* If you want to use one node exclusively, you must either enter the whole memory (''-m 96327'' or ''--mem=96327'') or set the number of MPI tasks greater than 39.		* If you want to use one node exclusively, you must either enter the whole memory (''-m 96327'' or ''--mem=96327'') or set the number of MPI tasks greater than 39.
	* Don't forget to load the appropriate MPI-module in your job script.		* Don't forget to load the appropriate MPI-module in your job script.
	* If you are using OpenMPI, the options '''--bind-to core --map-by core\|socket\|node''' should be used.		* If you are using OpenMPI, the options '''--bind-to core --map-by core\|socket\|node''' of the command mpirun should be used.
	<br>		<br>
	'''Example for a MPI job'''		'''Example for a MPI job'''
Line 57:		Line 57:
	* You always use the nodes exclusively.		* You always use the nodes exclusively.
	* Don't forget to load the appropriate MPI-module in your job script.		* Don't forget to load the appropriate MPI-module in your job script.
	* If you are using OpenMPI, the options '''--bind-to core --map-by core\|socket\|node''' should be used.		* If you are using OpenMPI, the options '''--bind-to core --map-by core\|socket\|node''' ofthe command mpirun should be used.
	<br>		<br>
	'''Example for a MPI job'''		'''Example for a MPI job'''
Line 75:		Line 75:
	* You always use the nodes exclusively.		* You always use the nodes exclusively.
	* Don't forget to load the appropriate MPI-module in your job script.		* Don't forget to load the appropriate MPI-module in your job script.
			* If you are using OpenMPI, the options '''--bind-to core --map-by socket\|node:PE=''z''''' of the command mpirun must be used.
	<br>		<br>
	'''Example for a MPI job'''		'''Example for a MPI job'''
Line 80:		Line 81:
	$ sbatch -p multiple -t 2-12 -N 10 --ntasks-per-node=2 -c 20 ./job_threaded_mpi.sh		$ sbatch -p multiple -t 2-12 -N 10 --ntasks-per-node=2 -c 20 ./job_threaded_mpi.sh
	</pre>		</pre>
	The script '''job_threaded_mpi.sh''' (containing a multithreaded MPI program after loading the appropriate MPI module) is started running 2.5 days on 400 cores with 20 MPI tasks and 20 threads per task on ten batch nodes.		The script '''job_threaded_mpi.sh''' (containing a multithreaded MPI program after loading the appropriate MPI module) is started running 2.5 days on 400 cores with 20 MPI tasks and 20 threads per task on ten batch nodes. Here the options '''--bind-to core --map-by socket:PE=10''' of the command mpirun must be used.

H Haefner at 12:49, 13 March 2020

2020-03-13T12:49:38Z

← Older revision		Revision as of 14:49, 13 March 2020
Line 57:		Line 57:
	* You always use the nodes exclusively.		* You always use the nodes exclusively.
	* Don't forget to load the appropriate MPI-module in your job script.		* Don't forget to load the appropriate MPI-module in your job script.
			* If you are using OpenMPI, the options '''--bind-to core --map-by core\|socket\|node''' should be used.
	<br>		<br>
	'''Example for a MPI job'''		'''Example for a MPI job'''