Batch Jobs Moab

From bwHPC Wiki
Revision as of 23:07, 22 January 2014 by R Barthel (talk | contribs) (msub -l resource_list)
Jump to: navigation, search
Navigation: bwHPC BPR / bwUniCluster

Important note: bwUniCluster is not in production mode yet.

Any kind of calculation on the compute nodes of bwUniCluster requires the user to define calculations as a sequence of commands or single command together with required run time, number of CPU cores and main memory and submit all, i.e., the batch job, to a resource and workload managing software. All bwHPC cluster, including bwUniCluster, have installed the workload managing software MOAB. Therefore any job submission by the user is to be executed by commands of the MOAB software. MOAB queues and runs user jobs based on fair sharing policies.

Overview of:

MOAB commands Brief explanation
msub submits a job and queues it in an input queue
checkjob displays detailed job state information
showq displays information about active, eligible, blocked, and/or recently completed jobs
showbf shows what resources are available for immediate use

1 Job Submission

Batch jobs are submitted by using the command msub. The main purpose of the msub command is to specify the resources that are needed to run the job. msub will then queue the batch job. However, starting of batch job depends on availability of the requested resources and the fair sharing value.

1.1 msub Command

The syntax and use of msub can be displayed via:

$ man msub

msub options can be used from the command line or in your job script.

msub Options
Command line Script Purpose
-l resources #MSUB -l resources Defines the resources that are required by the job. See the description below for this important flag.
-N name #MSUB -N name Gives a user specified name to the job.
-I Declares the the job is to be run interactively.
-o filename #MSUB -o filename Defines the filename to be used for the standard output stream of the batch job. By default the file with defined filename is placed under your job submit directory. To place under a different location, expand filename by the relative or absolute path of destination.

1.1.1 msub -l resource_list

The -l option is one of the most important msub options. It is used to specify a number of resource requirements for your job. Multiple resource strings are separated by commas.

msub -l resource_list
resource Purpose
-l nodes=1
-l nodes=2:ppn=8
Number of nodes
Number of nodes and number of processes per node
-l walltime=600
-l walltime=01:30:00
Wall-clock time. Default units are seconds.
HH:MM:SS format is also accepted.
-l feature=tree
-l feature=blocking
-l feature=fat
For jobs that span over several nodes
For sequential jobs
For jobs that require up to 1 TB memory
-l pmem=1000mb
Memory per process, allowed units are kb,mb,gb. Be aware that processes are either MPI tasks if running MPI parallel jobs or threads if running multithreaded jobs.

1.2 msub Examples

1.2.1 Serial Programs

To submit a serial job that runs the script and that requires 5000 MB of main memory and 3 hours of wall clock time

a) execute:

$ msub -N test -l nodes=1:ppn=1,walltime=3:00:00,pmem=5000mb


b) add after the initial line of your script the lines:

#MSUB -l nodes=1:ppn=1
#MSUB -l walltime=3:00:00
#MSUB -l pmem=5000mb
#MSUB -N test

and execute the modified script without any msub command line options:

$ msub

Note, that msub command line options overrule script options. Handling job script options and arguments

Job script options and arguments as followed:

./ -n 10

can not be passed while using msub command since those will be interpreted as command line options of msub.

Solution A:

Submit a wrapper script, e.g.


which simply contains all your job script options and arguments. The script would at least contain the following lines:

./ -n 10

Solution B:

Add after the header of your BASH script the following lines:

## check if $SCRIPT_FLAGS is "set"
if [ -n "${SCRIPT_FLAGS}" ] ; then
   ## but if positional parameters are already present
   ## we are going to ignore $SCRIPT_FLAGS
   if [ -z "${*}"  ] ; then
      set -- ${SCRIPT_FLAGS}

These lines modify your BASH script to read options and arguments from the environment variable $SCRIPT_FLAGS. Now submit your script as followed:

msub -v SCRIPT_FLAGS='-n 10' 

For advanced users: generalised version of solution B if job script arguments contain whitespaces.

1.2.2 Multithreaded Programs

Multithreaded programs operate faster than serial programs on CPUs with multiple cores. Moreover, multiple threads of one process share resources such as memory.

For multithreaded programs based on Open Multi-Processing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).

To submit a batch job called test that runs a fourfold threaded program omp_program which requires 6000 MByte of shared memory and total wall clock time of 3 hours

a) execute:

$ msub -v OMP_NUM_THREADS=4 -N test -l nodes=1:ppn=4,walltime=3:00:00,pmem=1500mb  omp_program


b) generate the script containing the following the lines:

#MSUB -l nodes=1:ppn=4
#MSUB -l walltime=3:00:00
#MSUB -l pmem=1500mb
#MSUB -N test


and execute the script without any msub command line options:

$ msub

1.2.3 MPI parallel Programs

Under construction.

1.2.4 Multithreaded + MPI parallel Programs

Under construction.

1.2.5 Interactive Jobs

Interactive jobs must not run on the logins nodes, however resources for interactive jobs can be requested using msub. Considering a serial application with a graphical frontend that requires 5000 MByte of memory and limiting the interactive run to 2 hours execute the following:

$ msub -v DISPLAY,HOME -l nodes=1:ppn=1,walltime=02:00:00,pmem=5000mb -I

After execution of this command DO NOT CLOSE your current terminal session but wait until the queueing system MOAB has granted you the requested resources on the compute system. Once granted you will be automatically logged on the dedicated resource. Now you have an interactive session with 1 core and 5000 MByte of memory on the compute system for 2 hours. Simply execute now your application:

$ cd destination
$ ./application

Note that, once the walltime limit has been reached you will be automatically logged out of the compute system.

2 Display Status of submitted Jobs

Under construction.

3 Environment Variables for Batch Jobs

Under construction.