Difference between revisions of "Batch Jobs Moab"

From bwHPC Wiki
Jump to: navigation, search
(MPI parallel Programs)
m (Blanked the page)
(Tag: Blanking)
 
(235 intermediate revisions by 14 users not shown)
Line 1: Line 1:
{| style="border-style: solid; border-width: 1px"
 
! Navigation: [[BwHPC_Best_Practices_Repository|bwHPC BPR]] / [[BwUniCluster_User_Guide|bwUniCluster]]
 
|}
 
 
 
<span style="color:red;font-size:105%;">Important note: bwUniCluster is '''not''' in production mode yet.</span>
 
 
<!---
 
<span style="color:red;font-size:105%;">The folllowing features of MOAB are not working:</span>
 
* <span style="color:red;font-size:105%;">interactive jobs, i.e.,</span> <span style="color:red;font-size:105%;background:#edeae2;margin:10px;padding:1px;border:1px dotted #808080">msub -I</span>
 
* <span style="color:red;font-size:105%;">any memory allocation of multi task jobs, i.e.,</span> <span style="color:red;font-size:105%;background:#edeae2;margin:10px;padding:1px;border:1px dotted #808080">msub -l mem=<value></span> <span style="color:red;font-size:105%;">or</span> <span style="color:red;font-size:105%;background:#edeae2;margin:10px;padding:1px;border:1px dotted #808080">msub -l pmem=<value></span>
 
 
 
<span style="color:red;font-size:105%;">Please do not use these features until further notice. Adaptive Computing, producer of MOAB, is working on that problem.
 
</span>
 
-->
 
 
 
Any kind of calculation on the compute nodes of '''bwUniCluster''' requires the user to define calculations as a sequence of commands or single command together with required run time, number of CPU cores and main memory and submit all, i.e., the '''batch job''', to a resource and workload managing software. All bwHPC cluster, including '''bwUniCluster''', have installed the workload managing software MOAB. Therefore any job submission by the user is to be executed by commands of the MOAB software. MOAB queues and runs user jobs based on fair sharing policies.
 
 
 
Overview of:
 
{| style="width:100%; vertical-align:top; background:#f5fffa;border:2px solid #000000;"
 
! MOAB commands !! Brief explanation
 
|-
 
| msub || submits a job and queues it in an input queue
 
|-
 
| checkjob || displays detailed job state information
 
|-
 
| showq || displays information about active, eligible, blocked, and/or recently completed jobs
 
|-
 
| showbf || shows what resources are available for immediate use
 
|-
 
| mjobctl -c || cancels a job
 
|}
 
 
 
 
= Job Submission =
 
 
Batch jobs are submitted by using the command '''msub'''. The main purpose of the '''msub''' command is to specify the resources that are needed to run the job. '''msub''' will then queue the batch job. However, starting of batch job depends on availability of the requested resources and the fair sharing value.
 
<!--into the input queue. The jobs are organized into different job classes. For each job class there are specific limits for the available resources (number of nodes, number of CPUs, maximum CPU time, maximum memory etc.). -->
 
<br>
 
 
== msub Command ==
 
 
The syntax and use of '''msub''' can be displayed via:
 
<pre>
 
$ man msub
 
</pre>
 
 
'''msub''' options can be used from the command line or in your job script.
 
 
 
{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"
 
! colspan="3" style="background-color:#999999;padding:3px"| msub Options
 
|-
 
! style="width:15%;height=20px; text-align:left;padding:3px"|Command line
 
! style="width:20%;height=20px; text-align:left;padding:3px"|Script
 
! style="width:65%;height=20px; text-align:left;padding:3px"|Purpose
 
|- style="vertical-align:top;"
 
| style="height=20px; text-align:left;padding:3px" | -l resources
 
| style="height=20px; text-align:left;padding:3px" | #MSUB -l resources
 
| style="height=20px; text-align:left;padding:3px" | Defines the resources that are required by the job. See the description below for this important flag.
 
|- style="vertical-align:top;"
 
| style="height=20px; text-align:left;padding:3px" | -N name
 
| style="height=20px; text-align:left;padding:3px" | #MSUB -N name
 
| style="height=20px; text-align:left;padding:3px" | Gives a user specified name to the job.
 
|- style="vertical-align:top;"
 
| style="height=20px; text-align:left;padding:3px" | -I
 
|
 
| style="height=20px; text-align:left;padding:3px" | Declares the the job is to be run interactively.
 
|- style="vertical-align:top;"
 
| style="height=20px; text-align:left;padding:3px" | -o filename
 
| style="height=20px; text-align:left;padding:3px" | #MSUB -o filename
 
| style="height=20px; text-align:left;padding:3px" | Defines the filename to be used for the standard output stream of the batch job. By default the file with defined filename is placed under your job submit directory. To place under a different location, expand ''filename'' by the relative or absolute path of destination.
 
|-
 
<!--
 
| -V
 
| #MSUB -V
 
| Declares that all environment variables in the msub environment are exported to the batch job.
 
|-
 
-->
 
|}
 
<br>
 
 
 
=== msub -l resource_list ===
 
The '''-l''' option is one of the most important msub options. It is used to specify a number of resource requirements for your job. Multiple resource strings are separated by commas.
 
 
 
 
<!--{| style="border-style: solid; border-width: 1px; padding=5px;" border="1"-->
 
{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"
 
! colspan="3" style="background-color:#999999;padding:3px"| msub -l resource_list
 
|- style="width:20%;height=20px; text-align:left;padding:3px"
 
! style="width:20%;height=20px; text-align:left;padding:3px"| resource
 
! style="height=20px; text-align:left;padding:3px"| Purpose
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | -l nodes=1 <br> -l nodes=2:ppn=8
 
| style="height=20px; text-align:left;padding:3px"| Number of nodes <br> Number of nodes and number of processes per node
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | -l walltime=600 <br> -l walltime=01:30:00
 
| style="height=20px; text-align:left;padding:3px"| Wall-clock time. Default units are seconds. <br> HH:MM:SS format is also accepted.
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | -l feature=tree <br> -l feature=blocking <br> -l feature=fat
 
| style="height=20px; text-align:left;padding:3px" | For jobs that span over several nodes <br> For sequential jobs <br> For jobs that require up to 1 TB memory
 
|- style="vertical-align:top;"
 
| style="width:20%;height=40px; text-align:left;padding:3px;" | -l pmem=1000mb
 
| <div style="padding:3px;">Memory per '''process''', allowed units are kb,mb,gb. Be aware that '''processes''' are either ''MPI tasks'' if running MPI parallel jobs or ''threads'' if running multithreaded jobs.</div>
 
|-
 
|}
 
<br>
 
 
== msub Examples ==
 
 
 
 
=== Serial Programs ===
 
To submit a serial job that runs the script '''job.sh''' and that requires 5000 MB of main memory and 3 hours of wall clock time
 
 
a) execute:
 
<pre>
 
$ msub -N test -l nodes=1:ppn=1,walltime=3:00:00,pmem=5000mb job.sh
 
</pre>
 
or
 
 
b) add after the initial line of your script '''job.sh''' the lines:
 
{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 2px;"
 
| style="width:280px; text-align:center; white-space:nowrap; color:#000;" |
 
<source lang="bash">
 
#MSUB -l nodes=1:ppn=1
 
#MSUB -l walltime=3:00:00
 
#MSUB -l pmem=5000mb
 
#MSUB -N test
 
</source>
 
|}
 
and execute the modified script without any msub command line options:
 
<pre>
 
$ msub job.sh
 
</pre>
 
 
 
Note, that msub command line options overrule script options.
 
 
 
==== Handling job script options and arguments ====
 
Job script options and arguments as followed:
 
<pre>
 
./job.sh -n 10
 
</pre>
 
can not be passed while using msub command since those will be interpreted as command line options of msub.
 
 
 
'''Solution A:'''
 
 
Submit a wrapper script, e.g. job_msub.sh:
 
<pre>
 
msub job_msub.sh
 
</pre>
 
which simply contains all your job script options and arguments. The script job_msub.sh would at least contain the following lines:
 
{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 2px;"
 
| style="width:280px; text-align:center; white-space:nowrap; color:#000;" |
 
<source lang="bash">
 
#!/bin/bash
 
./job_msub.sh -n 10
 
</source>
 
|}
 
 
 
 
'''Solution B:'''
 
 
Add after the header of your '''BASH''' script job.sh the following lines:
 
{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 2px;"
 
| style="width:280px; text-align:center; white-space:nowrap; color:#000;" |
 
<source lang="bash">
 
## check if $SCRIPT_FLAGS is "set"
 
if [ -n "${SCRIPT_FLAGS}" ] ; then
 
## but if positional parameters are already present
 
## we are going to ignore $SCRIPT_FLAGS
 
if [ -z "${*}" ] ; then
 
set -- ${SCRIPT_FLAGS}
 
fi
 
fi
 
</source>
 
|}
 
 
These lines modify your BASH script to read options and arguments from the environment variable $SCRIPT_FLAGS. Now submit your script job.sh as followed:
 
<pre>
 
msub -v SCRIPT_FLAGS='-n 10' job.sh
 
</pre>
 
 
 
For advanced users: [[generalised version of solution B]] if job script arguments contain whitespaces.
 
 
 
 
=== Multithreaded Programs ===
 
Multithreaded programs operate faster than serial programs on CPUs with multiple cores. Moreover, multiple threads of one process share resources such as memory.
 
 
For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).
 
 
To submit a batch job called ''test'' that runs a fourfold threaded program ''omp_program'' which requires 6000 MByte of shared memory and total wall clock time of 3 hours
 
 
a) execute:
 
<pre>
 
$ msub -v OMP_NUM_THREADS=4 -N test -l nodes=1:ppn=4,walltime=3:00:00,pmem=1500mb omp_program
 
</pre>
 
or
 
 
b) generate the script '''job_omp.sh''' containing the following the lines:
 
{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 2px;"
 
| style="width:280px; text-align:center; white-space:nowrap; color:#000;" |
 
<source lang="bash">
 
#!/bin/bash
 
#MSUB -l nodes=1:ppn=4
 
#MSUB -l walltime=3:00:00
 
#MSUB -l pmem=1500mb
 
#MSUB -N test
 
 
export OMP_NUM_THREADS=${MOAB_PROCCOUNT}
 
./omp_program
 
</source>
 
|}
 
and execute the script '''job_omp.sh''' without any msub command line options:
 
<pre>
 
$ msub job_omp.sh
 
</pre>
 
<br>
 
 
=== MPI parallel Programs ===
 
MPI parallel programs run faster than serial programs on multi CPU and multi core systems. N-fold spawned processes of the MPI program, i.e., '''MPI tasks''', run simultaneously and communicate via the Message Passing Interface (MPI) paradigm. MPI tasks do not share memory but can be spawned over different nodes.
 
 
Multiple MPI tasks can not be launched by the MPI parallel program itself but via '''mpirun''', e.g. 4 MPI tasks of ''my_par_program'':
 
<pre>
 
mpirun -n 4 my_par_program
 
</pre>
 
<br>
 
However, this given command can '''not''' be directly included in your '''msub''' command for submitting as a batch job to the compute cluster, [[BwUniCluster_Batch_Jobs#Handling_job_script_options_and_arguments|see above]].
 
 
Instead, generate a wrapper script, ''job.sh'', containing the following lines:
 
{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 2px;"
 
| style="width:280px; text-align:center; white-space:nowrap; color:#000;" |
 
<source lang="bash">
 
#!/bin/bash
 
module load <placeholder>
 
mpirun my_par_program
 
</source>
 
|}
 
Please note, that you need to replace <placeholder> with all the required modulefiles to enable the MPI environment.
 
<br>
 
 
Considering 4 MPI tasks on a single node, each requiring 1000 MByte, and running for 1 hour, execute:
 
<pre>
 
msub -l nodes=1:ppn=4,pmem=1000mb,walltime=01:00:00 job.sh
 
</pre>
 
<br>
 
 
Launching and running 32 MPI tasks on 4 nodes, each requiring 1000 MByte, and running for 5 hours, execute:
 
<pre>
 
msub -l nodes=4:ppn=8,pmem=1000mb,walltime=05:00:00 job.sh
 
</pre>
 
<br>
 
 
=== Multithreaded + MPI parallel Programs ===
 
Under construction.
 
 
 
 
=== Interactive Jobs ===
 
[[BwUniCluster_User_Access#Allowed_activities_on_login_nodes|Interactive jobs must not run on the logins nodes]], however resources for interactive jobs can be requested using msub. Considering a serial application with a graphical frontend that requires 5000 MByte of memory <!--and limiting the interactive run to 2 hours--> execute the following:
 
<pre>
 
$ msub -v DISPLAY,HOME -l nodes=1:ppn=1,pmem=5000mb -I
 
</pre>
 
<br>
 
After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system MOAB has granted you the requested resources on the compute system. Once granted you will be automatically logged on the dedicated resource. Now you have an interactive session with 1 core and 5000 MByte of memory on the compute system<!-- for 2 hours-->. Simply execute now your application:
 
<pre>
 
$ cd to_path
 
$ ./application
 
</pre>
 
Note that, once the walltime limit has been reached you will be automatically logged out of the compute system.
 
<br>
 
 
= Display Status of submitted Jobs =
 
Under construction.
 
 
 
 
= Environment Variables for Batch Jobs =
 
Under construction.
 
 
 
 
----
 
[[Category:bwHPC|Job Submission]]
 
[[Category:bwUniCluster|Moab]]
 
[[Category:bwUniCluster|Job Submission]]
 

Latest revision as of 10:04, 15 August 2023