Difference between revisions of "Batch Jobs Moab"

From bwHPC Wiki
Jump to: navigation, search
(Multithreaded Programs)
m (Blanked the page)
(Tag: Blanking)
 
(247 intermediate revisions by 14 users not shown)
Line 1: Line 1:
{| style="border-style: solid; border-width: 1px"
 
! Navigation: [[BwHPC_Best_Practices_Repository|bwHPC BPR]] / [[BwUniCluster_User_Guide|bwUniCluster]]
 
|}
 
 
 
<span style="color:red;font-size:105%;">Important note: bwUniCluster is '''not''' in production mode yet.</span>
 
 
<!---
 
<span style="color:red;font-size:105%;">The folllowing features of MOAB are not working:</span>
 
* <span style="color:red;font-size:105%;">interactive jobs, i.e.,</span> <span style="color:red;font-size:105%;background:#edeae2;margin:10px;padding:1px;border:1px dotted #808080">msub -I</span>
 
* <span style="color:red;font-size:105%;">any memory allocation of multi task jobs, i.e.,</span> <span style="color:red;font-size:105%;background:#edeae2;margin:10px;padding:1px;border:1px dotted #808080">msub -l mem=<value></span> <span style="color:red;font-size:105%;">or</span> <span style="color:red;font-size:105%;background:#edeae2;margin:10px;padding:1px;border:1px dotted #808080">msub -l pmem=<value></span>
 
 
 
<span style="color:red;font-size:105%;">Please do not use these features until further notice. Adaptive Computing, producer of MOAB, is working on that problem.
 
</span>
 
-->
 
 
 
Any kind of calculation on the compute nodes of '''bwUniCluster''' requires the user to define calculations as a sequence of commands or single command together with required run time, number of CPU cores and main memory and submit all, i.e., the '''batch job''', to a resource and workload managing software. All bwHPC cluster, including '''bwUniCluster''', have installed the workload managing software MOAB. Therefore any job submission by the user is to be executed by commands of the MOAB software. MOAB queues and runs user jobs based on fair sharing policies.
 
 
 
{| style="border-style: solid; border-width: 1px 1px 1px 1"
 
! MOAB commands !! Brief explanation
 
|-
 
| msub || submits a job and queues it in an input queue
 
|-
 
| checkjob || displays detailed job state information
 
|-
 
| showq || displays information about active, eligible, blocked, and/or recently completed jobs
 
|-
 
| showbf || shows what resources are available for immediate use
 
|}
 
 
 
 
= Job Submission =
 
 
Batch jobs are submitted using the command '''msub'''. The main purpose of the '''msub''' command is to specify the resources that are needed to run the job. '''msub''' will then queue the job into the input queue. The jobs are organized into different job classes. For each job class there are specific limits for the available resources (number of nodes, number of CPUs, maximum CPU time, maximum memory etc.).
 
 
 
 
== msub Command ==
 
 
The syntax and use of '''msub''' can be displayed via:
 
<pre>
 
$ man msub
 
</pre>
 
 
'''msub''' options can be used from the command line or in your job script.
 
 
 
{| style="border-style: solid; border-width: 1px;" border="1" cellpadding="2"
 
! colspan="3" style="background-color:gray;"| msub Options
 
|-
 
! align="left"|Command line
 
! align="left"|Script
 
! align="left"|Purpose
 
|-
 
| -l resources
 
| #MSUB -l resources
 
| Defines the resources that are required by the job. See the description below for this important flag.
 
|-
 
| -N name
 
| #MSUB -N name
 
| Gives a user specified name to the job.
 
|-
 
| -I
 
|
 
| Declares the the job is to be run interactively.
 
|-
 
| -o filename
 
| #MSUB -o filename
 
| Defines the filename to be used for the standard output stream of the batch job. By default the file with defined filename is placed under your job submit directory. To place under a different location, expand ''filename'' by the relative or absolute path of destination.
 
|-
 
<!--
 
| -V
 
| #MSUB -V
 
| Declares that all environment variables in the msub environment are exported to the batch job.
 
|-
 
-->
 
|}
 
 
 
 
=== msub -l resource_list ===
 
The '''-l''' option is one of the most important msub options. It is used to specify a number of resource requirements for your job. Multiple resource strings are separated by commas.
 
 
 
{| style="border-style: solid; border-width: 1px; padding=5px;" border="1"
 
! colspan="3" style="background-color:gray;"| msub -l resource_list
 
|-
 
! align="left"|resource
 
! align="left"|Purpose
 
|-
 
| -l nodes=1 <br> -l nodes=2:ppn=8
 
| Number of nodes <br> Number of nodes and number of processes per node
 
|-
 
| -l walltime=600 <br> -l walltime=1:30:00
 
| Wall-clock time. Default units are seconds. <br> HH:MM:SS format is also accepted.
 
|-
 
| -l feature=tree <br> -l feature=blocking <br> -l feature=fat
 
| For jobs that span over several nodes <br> For sequential jobs <br> For jobs that require up to 1 TB memory
 
|-
 
| -l pmem=1000mb
 
| Memory per '''process''', allowed units are kb,mb,gb. Be aware that '''processes''' are either ''MPI tasks'' if running MPI parallel jobs or ''threads'' if running multithreaded jobs.
 
|-
 
|}
 
 
 
 
== msub Examples ==
 
 
 
 
=== Serial Programs ===
 
To submit a serial job that runs the script '''job.sh''' and that requires 5000 MB of main memory and 3 hours of wall clock time
 
 
a) execute:
 
<pre>
 
$ msub -N test -l nodes=1:ppn=1,walltime=3:00:00,pmem=5000mb job.sh
 
</pre>
 
or
 
 
b) add after the initial line of your script '''job.sh''' the lines:
 
{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 2px;"
 
| style="width:280px; text-align:center; white-space:nowrap; color:#000;" |
 
<source lang="bash">
 
#MSUB -l nodes=1:ppn=1
 
#MSUB -l walltime=3:00:00
 
#MSUB -l pmem=5000mb
 
#MSUB -N test
 
</source>
 
|}
 
and execute the modified script without any msub command line options:
 
<pre>
 
$ msub job.sh
 
</pre>
 
 
 
Note, that msub command line options overrule script options.
 
 
 
==== Handling job script options and arguments ====
 
Job script options and arguments as followed:
 
<pre>
 
./job.sh -n 10
 
</pre>
 
can not be passed while using msub command since those will be interpreted as command line options of msub.
 
 
 
'''Solution A:'''
 
 
Submit a wrapper script, e.g. job_msub.sh:
 
<pre>
 
msub job_msub.sh
 
</pre>
 
which simply contains all your job script options and arguments. The script job_msub.sh would at least contain the following lines:
 
{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 2px;"
 
| style="width:280px; text-align:center; white-space:nowrap; color:#000;" |
 
<source lang="bash">
 
#!/bin/bash
 
./job_msub.sh -n 10
 
</source>
 
|}
 
 
 
 
'''Solution B:'''
 
 
Add after the header of your '''BASH''' script job.sh the following lines:
 
{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 2px;"
 
| style="width:280px; text-align:center; white-space:nowrap; color:#000;" |
 
<source lang="bash">
 
## check if $SCRIPT_FLAGS is "set"
 
if [ -n "${SCRIPT_FLAGS}" ] ; then
 
## but if positional parameters are already present
 
## we are going to ignore $SCRIPT_FLAGS
 
if [ -z "${*}" ] ; then
 
set -- ${SCRIPT_FLAGS}
 
fi
 
fi
 
</source>
 
|}
 
 
These lines modify your BASH script to read options and arguments from the environment variable $SCRIPT_FLAGS. Now submit your script job.sh as followed:
 
<pre>
 
msub -v SCRIPT_FLAGS='-n 10' job.sh
 
</pre>
 
 
 
For advanced users: [[generalised version of solution B]] if job script arguments contain whitespaces.
 
 
 
 
=== Multithreaded Programs ===
 
Multithreaded programs operate faster than serial programs on CPUs with multiple cores. Moreover, multiple threads of one process share resources such as memory.
 
 
For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).
 
 
To submit a fourfold threaded program, omp_program, that requires 6000 MByte shared memory and total wall clock time of 3 hours
 
 
a) execute:
 
<pre>
 
$ msub -v OMP_NUM_THREADS=4 -N test -l nodes=1:ppn=4,walltime=3:00:00,pmem=1500mb omp_program
 
</pre>
 
or
 
 
b) generate the script '''job_omp.sh''' containing the following the lines:
 
{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 2px;"
 
| style="width:280px; text-align:center; white-space:nowrap; color:#000;" |
 
<source lang="bash">
 
#!/bin/bash
 
#MSUB -l nodes=1:ppn=4
 
#MSUB -l walltime=3:00:00
 
#MSUB -l pmem=6000mb
 
#MSUB -N test
 
 
export OMP_NUM_THREADS=${MOAB_PROCCOUNT}
 
./omp_program
 
</source>
 
|}
 
and execute the script '''job_omp.sh''' without any msub command line options:
 
<pre>
 
$ msub job_omp.sh
 
</pre>
 
 
=== MPI parallel Programs ===
 
Under construction.
 
 
 
 
=== Multithreaded + MPI parallel Programs ===
 
Under construction.
 
 
 
 
=== Interactive Jobs ===
 
Under construction.
 
 
 
 
= Display Status of submitted Jobs =
 
Under construction.
 
 
 
 
= Environment Variables for Batch Jobs =
 
Under construction.
 
 
 
 
----
 
[[Category:bwHPC|Job Submission]]
 
[[Category:bwUniCluster|Moab]]
 
[[Category:bwUniCluster|Job Submission]]
 

Latest revision as of 10:04, 15 August 2023