BwUniCluster2.0/Batch System Migration Guide

While the former bwUniCluster 1 system used the combination Moab/SLURM for the batch system queue, bwUniCluster 2.0 uses only SLURM. This means that most job scripts and workflows which relied on Moab-specific pragmas and commands have to be changed.

1 General Overview

Job parameters can be passed to SLURM in the same ways which were possible with Moab.

• Instead of the #MOAB or #PBS pragmas, the pragma #SLURM has to be used within job files.

• Instead of the Moab commands, the corresponding SLURM commands have to be used.

A general mapping of Moab to SLURM commands can be found in the following table:

Moab command	SLURM command
msub	sbatch
msub -I	salloc
canceljob	scancel
showq	squeue
checkjob $JOBID	scontrol show job $JOBID

The following chapters contain short examples for the most common job types. For a full overview of SLURM see the full article.

2 Serial Programs

• Use the time option -t or --time (instead of -l walltime). If only one number is entered behind -t, the default unit is minutes.
• Use the option -n 1 or --ntasks=1 (instead of -l nodes=1,ppn=1).
• Use the option -m or --mem (instead of -l pmem). The default unit is MegaByte.
• If you want to use one node exclusively, you must enter the whole memory (-m 96327 or --mem=96327).

Example for a serial job

$ sbatch -p single -t 60 -n 1 -m 96327 ./job.sh 

The script job.sh (containing the execution of a serial program) is started running 60 minutes exclusively on a batch node.

3 Multithreaded Programs

• Use the time option -t or --time (instead of -l walltime). If only one number is entered behind -t, the default unit is minutes.
• Use the option -N 1 or --nodes=1 and c x or --cpus-per-task=x (instead of -l nodes=1,ppn=x ). x can be a number between 1 and 40 (because of 40 cores within one node); it can also be a number between 41 and 80 (because of active hyperthreading).
• Use the option -m or --mem (instead of -l pmem). The default unit is MegaByte.
• Use the option --export to set the needed environment variable OMP_NUM_THREADS for the batch job. Adding ALL means to pass all interactively set environment variables to the batch job.
• If you want to use one node exclusively, you must either enter the whole memory (-m 96327 or --mem=96327) or set the number of threads greater than 39.

Example for a multithreaded job

$ sbatch -p single -t 1:00:00 -N 1 -c 20 -m 50gb --export=ALL,OMP_NUM_THREADS=20 ./job_threaded.sh 

The script job_threaded.sh (containing a multithreaded program) is started running 1 hour in shared mode on 20 cores requesting 50GB on one batch node.

4 MPI Parallel Programs within one node

• Use the time option -t or --time (instead of -l walltime). If only one number is entered behind -t, the default unit is minutes.
• Use the option -n x or --ntasks=x (instead of -l nodes=1,ppn=x ). x can be a number between 1 and 40 (because of 40 cores within one node); you should'nt utilize hyperthreading.
• Use the option -m or --mem (instead of -l pmem). The default unit is MegaByte.
• If you want to use one node exclusively, you must either enter the whole memory (-m 96327 or --mem=96327) or set the number of MPI tasks greater than 39.
• Don't forget to load the appropriate MPI-module in your job script.
• If you are using OpenMPI, the options --bind-to core --map-by core|socket|node of the command mpirun should be used.

Example for a MPI job

$ sbatch -p single -t 600 -n 10 -m 40000 ./job_mpi.sh 

The script job_mpi.sh (containing a MPI program after loading the appropriate MPI module) is started running 10 hours in shared mode on 10 cores requesting 40000 MB on one batch node.

5 MPI Parallel Programs on many nodes

• Use the time option -t or --time (instead of -l walltime). If only one number is entered behind -t, the default unit is minutes.
• Use the option -N y or --nodes=y and --ntasks-per-node=x (instead of -l nodes=y,ppn=x). x can be a number between 1 and 40 (28 for Broadwell nodes) (because of 40 (28) cores within one node); you should'nt utilize hyperthreading.
• You should'nt use the option -m or --mem because the nodes are used exclusively.
• You always use the nodes exclusively.
• Don't forget to load the appropriate MPI-module in your job script.
• If you are using OpenMPI, the options --bind-to core --map-by core|socket|node ofthe command mpirun should be used.

Example for a MPI job

$ sbatch -p multiple -t 48:00:00 -N 10 --ntasks-per-node=40  ./job_mpi.sh 

The script job_mpi.sh (containing a MPI program after loading the appropriate MPI module) is started running 2 days on 400 cores on ten batch nodes.

6 Multithreaded + MPI Parallel Programs on many nodes

• Use the time option -t or --time (instead of -l walltime). If only one number is entered behind -t, the default unit is minutes.
• Use the option -N y or --nodes=y and --ntasks-per-node=x and -c z or --cpus-per-task=z (instead of -l nodes=y,ppn=x+z). x usually should be 1 or 2 and x+z usually 40 (28 on Broadwell nodes); you can utilize hyperthreading if you want.
• You should'nt use the option -m or --mem because the nodes are used exclusively.
• You always use the nodes exclusively.
• Don't forget to load the appropriate MPI-module in your job script.
• If you are using OpenMPI, the options --bind-to core --map-by socket|node:PE=z of the command mpirun must be used.

Example for a MPI job

$ sbatch -p multiple -t 2-12 -N 10 --ntasks-per-node=2 -c 20 ./job_threaded_mpi.sh 

The script job_threaded_mpi.sh (containing a multithreaded MPI program after loading the appropriate MPI module) is started running 2.5 days on 400 cores with 20 MPI tasks and 20 threads per task on ten batch nodes. Here the options --bind-to core --map-by socket:PE=10 of the command mpirun must be used.