Difference between revisions of "Batch Jobs Moab"

From bwHPC Wiki
Jump to: navigation, search
(Multithreaded + MPI parallel Programs)
m (Blanked the page)
(Tag: Blanking)
 
(202 intermediate revisions by 14 users not shown)
Line 1: Line 1:
{| style="border-style: solid; border-width: 1px"
 
! Navigation: [[BwHPC_Best_Practices_Repository|bwHPC BPR]] / [[BwUniCluster_User_Guide|bwUniCluster]]
 
|}
 
 
 
<!--<span style="color:red;font-size:105%;">Important note: bwUniCluster is '''not''' in production mode yet.</span>-->
 
 
<!---
 
<span style="color:red;font-size:105%;">The folllowing features of MOAB are not working:</span>
 
* <span style="color:red;font-size:105%;">interactive jobs, i.e.,</span> <span style="color:red;font-size:105%;background:#edeae2;margin:10px;padding:1px;border:1px dotted #808080">msub -I</span>
 
* <span style="color:red;font-size:105%;">any memory allocation of multi task jobs, i.e.,</span> <span style="color:red;font-size:105%;background:#edeae2;margin:10px;padding:1px;border:1px dotted #808080">msub -l mem=<value></span> <span style="color:red;font-size:105%;">or</span> <span style="color:red;font-size:105%;background:#edeae2;margin:10px;padding:1px;border:1px dotted #808080">msub -l pmem=<value></span>
 
 
 
<span style="color:red;font-size:105%;">Please do not use these features until further notice. Adaptive Computing, producer of MOAB, is working on that problem.
 
</span>
 
-->
 
 
 
Any kind of calculation on the compute nodes of '''bwUniCluster''' requires the user to define calculations as a sequence of commands or single command together with required run time, number of CPU cores and main memory and submit all, i.e., the '''batch job''', to a resource and workload managing software. All bwHPC cluster, including '''bwUniCluster''', have installed the workload managing software MOAB. Therefore any job submission by the user is to be executed by commands of the MOAB software. MOAB queues and runs user jobs based on fair sharing policies.
 
 
 
Overview of:
 
{| style="width:100%; vertical-align:top; background:#f5fffa;border:2px solid #000000;"
 
! MOAB commands !! Brief explanation
 
|-
 
| msub || submits a job and queues it in an input queue
 
|-
 
| checkjob || displays detailed job state information
 
|-
 
| showq || displays information about active, eligible, blocked, and/or recently completed jobs
 
|-
 
| showbf || shows what resources are available for immediate use
 
|-
 
| canceljob || cancels a job
 
|}
 
 
 
 
= Job Submission =
 
 
Batch jobs are submitted by using the command '''msub'''. The main purpose of the '''msub''' command is to specify the resources that are needed to run the job. '''msub''' will then queue the batch job. However, starting of batch job depends on availability of the requested resources and the fair sharing value.
 
<!--into the input queue. The jobs are organized into different job classes. For each job class there are specific limits for the available resources (number of nodes, number of CPUs, maximum CPU time, maximum memory etc.). -->
 
<br>
 
 
== msub Command ==
 
 
The syntax and use of '''msub''' can be displayed via:
 
<pre>
 
$ man msub
 
</pre>
 
 
'''msub''' options can be used from the command line or in your job script.
 
 
 
{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"
 
! colspan="3" style="background-color:#999999;padding:3px"| msub Options
 
|-
 
! style="width:15%;height=20px; text-align:left;padding:3px"|Command line
 
! style="width:20%;height=20px; text-align:left;padding:3px"|Script
 
! style="width:65%;height=20px; text-align:left;padding:3px"|Purpose
 
|- style="vertical-align:top;"
 
| style="height=20px; text-align:left;padding:3px" | -l ''resources''
 
| style="height=20px; text-align:left;padding:3px" | #MSUB -l ''resources''
 
| style="height=20px; text-align:left;padding:3px" | Defines the resources that are required by the job. See the description below for this important flag.
 
|- style="vertical-align:top;"
 
| style="height=20px; text-align:left;padding:3px" | -N ''name''
 
| style="height=20px; text-align:left;padding:3px" | #MSUB -N ''name''
 
| style="height=20px; text-align:left;padding:3px" | Gives a user specified name to the job.
 
|- style="vertical-align:top;"
 
| style="height=20px; text-align:left;padding:3px" | -I
 
|
 
| style="height=20px; text-align:left;padding:3px" | Declares the job is to be run interactively.
 
|- style="vertical-align:top;"
 
| style="height=20px; text-align:left;padding:3px" | -o ''filename''
 
| style="height=20px; text-align:left;padding:3px" | #MSUB -o ''filename''
 
| style="height=20px; text-align:left;padding:3px" | Defines the filename to be used for the standard output stream of the batch job. By default the file with defined filename is placed under your job submit directory. To place under a different location, expand ''filename'' by the relative or absolute path of destination.
 
|- style="vertical-align:top;"
 
| style="height=20px; text-align:left;padding:3px" | -q ''queue''
 
| style="height=20px; text-align:left;padding:3px" | #MSUB -q ''queue''
 
| style="height=20px; text-align:left;padding:3px" | Defines the queue class
 
|-
 
<!--
 
| -V
 
| #MSUB -V
 
| Declares that all environment variables in the msub environment are exported to the batch job.
 
|-
 
-->
 
|}
 
<br>
 
 
 
=== msub -l ''resource_list'' ===
 
The '''-l''' option is one of the most important msub options. It is used to specify a number of resource requirements for your job. Multiple resource strings are separated by commas.
 
 
 
 
<!--{| style="border-style: solid; border-width: 1px; padding=5px;" border="1"-->
 
{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"
 
! colspan="3" style="background-color:#999999;padding:3px"| msub -l ''resource_list''
 
|- style="width:20%;height=20px; text-align:left;padding:3px"
 
! style="width:20%;height=20px; text-align:left;padding:3px"| resource
 
! style="height=20px; text-align:left;padding:3px"| Purpose
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | -l procs=8
 
| style="height=20px; text-align:left;padding:3px"| Number of processes, distribution over nodes will be done by MOAB
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | -l nodes=2:ppn=8
 
| style="height=20px; text-align:left;padding:3px"| Number of nodes and number of processes per node
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | -l walltime=600 <br> -l walltime=01:30:00
 
| style="height=20px; text-align:left;padding:3px"| Wall-clock time. Default units are seconds. <br> HH:MM:SS format is also accepted.
 
<!--
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | -l feature=tree <br> -l feature=blocking <br> -l feature=fat
 
| style="height=20px; text-align:left;padding:3px" | For jobs that span over several nodes <br> For sequential jobs <br> For jobs that require up to 1 TB memory-->
 
|- style="vertical-align:top;"
 
| style="width:20%;height=40px; text-align:left;padding:3px;" | -l pmem=1000mb
 
| <div style="padding:3px;">Maximum amount of physical memory used by any single process of the job. <br>Allowed units are kb, mb, gb. Be aware that '''processes''' are either ''MPI tasks'' if running MPI parallel jobs or ''threads'' if running multithreaded jobs.</div>
 
|- style="vertical-align:top;"
 
| style="width:20%;height=40px; text-align:left;padding:3px;" | -l mem=1000mb
 
| <div style="padding:3px;">Maximum amount of physical memory used by the job.<br>Allowed units are kb, mb, gb. Be aware that this memory value is the accumulated memory for all ''MPI tasks'' or all ''threads'' of the job.</div>
 
|-
 
|}
 
<br>
 
 
=== msub -q ''queues'' ===
 
 
Queue classes define maximum resources such as walltime, nodes and processes per node and partition of the compute system.
 
 
{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"
 
! colspan="3" style="background-color:#999999;padding:3px"| msub -q ''queue''
 
|- style="width:20%;height=20px; text-align:left;padding:3px"
 
! style="width:20%;height=20px; text-align:left;padding:3px"| queue
 
! style="height=20px; text-align:left;padding:3px"| maximum resources
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | -q develop
 
| style="height=20px; text-align:left;padding:3px"| walltime=00:30:00 (i.e. 30 min), node=1, processes=16
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | -q singlenode
 
| style="height=20px; text-align:left;padding:3px"| walltime=3:00:00:00 (i.e. 3 days), node=1, processes=16
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | -q multinode
 
| style="height=20px; text-align:left;padding:3px"| walltime=2:00:00:00 (i.e. 2 days), node=8
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | -q verylong
 
| style="height=20px; text-align:left;padding:3px"| walltime=6:00:00:00 (i.e. 6 days), node=1, processes=16
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | -q fat
 
| style="height=20px; text-align:left;padding:3px"| walltime=1:00:00:00 (i.e. 1 days), node=1, processes=32 on fat nodes
 
|-
 
|}
 
If queue classes are not specified explicitly in your msub command, your batch jobs are automatically assigned to queues ''develop'', ''singlenode'' and ''multinode'' based on your requested walltime, nodes and processes.
 
 
* To run your batch job longer than 3 days, please use <span style="background:#edeae2;margin:10px;padding:1px;border:1px dotted #808080">msub -q verylong</span>.
 
 
* To run your batch job on one of the [[BwUniCluster_File_System#Components_of_bwUniCluster|fat nodes]], please use <span style="background:#edeae2;margin:10px;padding:1px;border:1px dotted #808080">msub -q fat</span>.
 
<br>
 
<br>
 
 
== msub Examples ==
 
 
 
 
=== Serial Programs ===
 
To submit a serial job that runs the script '''job.sh''' and that requires 5000 MB of main memory and 3 hours of wall clock time
 
 
a) execute:
 
<pre>
 
$ msub -N test -l nodes=1:ppn=1,walltime=3:00:00,pmem=5000mb job.sh
 
</pre>
 
or
 
 
b) add after the initial line of your script '''job.sh''' the lines:
 
{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 2px;"
 
| style="width:280px; text-align:center; white-space:nowrap; color:#000;" |
 
<source lang="bash">
 
#MSUB -l nodes=1:ppn=1
 
#MSUB -l walltime=3:00:00
 
#MSUB -l pmem=5000mb
 
#MSUB -N test
 
</source>
 
|}
 
and execute the modified script without any msub command line options:
 
<pre>
 
$ msub job.sh
 
</pre>
 
 
 
Note, that msub command line options overrule script options.
 
 
 
==== Handling job script options and arguments ====
 
Job script options and arguments as followed:
 
<pre>
 
./job.sh -n 10
 
</pre>
 
can not be passed while using msub command since those will be interpreted as command line options of msub.
 
 
 
'''Solution A:'''
 
 
Submit a wrapper script, e.g. job_msub.sh:
 
<pre>
 
msub job_msub.sh
 
</pre>
 
which simply contains all your job script options and arguments. The script job_msub.sh would at least contain the following lines:
 
{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 2px;"
 
| style="width:280px; text-align:center; white-space:nowrap; color:#000;" |
 
<source lang="bash">
 
#!/bin/bash
 
./job_msub.sh -n 10
 
</source>
 
|}
 
 
 
 
'''Solution B:'''
 
 
Add after the header of your '''BASH''' script job.sh the following lines:
 
{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 2px;"
 
| style="width:280px; text-align:center; white-space:nowrap; color:#000;" |
 
<source lang="bash">
 
## check if $SCRIPT_FLAGS is "set"
 
if [ -n "${SCRIPT_FLAGS}" ] ; then
 
## but if positional parameters are already present
 
## we are going to ignore $SCRIPT_FLAGS
 
if [ -z "${*}" ] ; then
 
set -- ${SCRIPT_FLAGS}
 
fi
 
fi
 
</source>
 
|}
 
 
These lines modify your BASH script to read options and arguments from the environment variable $SCRIPT_FLAGS. Now submit your script job.sh as followed:
 
<pre>
 
msub -v SCRIPT_FLAGS='-n 10' job.sh
 
</pre>
 
 
 
For advanced users: [[generalised version of solution B]] if job script arguments contain whitespaces.
 
 
 
 
=== Multithreaded Programs ===
 
Multithreaded programs operate faster than serial programs on CPUs with multiple cores. Moreover, multiple threads of one process share resources such as memory.
 
 
For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).
 
 
To submit a batch job called ''test'' that runs a fourfold threaded program ''omp_program'' which requires 6000 MByte of total physical memory and total wall clock time of 3 hours:
 
 
<!-- 2014-01-29, at the moment submission of executables does not work, SLURM has to be instructed to generate a wrapper
 
a) execute:
 
<pre>
 
$ msub -v OMP_NUM_THREADS=4 -N test -l nodes=1:ppn=4,walltime=3:00:00,mem=6000mb omp_program
 
</pre>
 
 
or
 
-->
 
<!--b)-->
 
* generate the script '''job_omp.sh''' containing the following the lines:
 
{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
 
| style="width:280px; text-align:center; white-space:nowrap; color:#000;" |
 
<source lang="bash">
 
#!/bin/bash
 
#MSUB -l nodes=1:ppn=4
 
#MSUB -l walltime=3:00:00
 
#MSUB -l mem=6000mb
 
#MSUB -N test
 
 
module load <placeholder>
 
export OMP_NUM_THREADS=${MOAB_PROCCOUNT}
 
./omp_program
 
</source>
 
|}
 
and, if necessary, replace <placeholder> with the required modulefile to enable the openMP environment and execute the script '''job_omp.sh''' without any msub command line options:
 
<pre>
 
$ msub job_omp.sh
 
</pre>
 
<br>
 
Note, that msub command line options overrule script options, e.g.,
 
<pre>
 
$ msub -l mem=2000mb job_omp.sh
 
</pre>
 
overwrites the script setting of 6000 MByte with 2000 MByte.
 
<br>
 
<br>
 
 
=== MPI parallel Programs ===
 
MPI parallel programs run faster than serial programs on multi CPU and multi core systems. N-fold spawned processes of the MPI program, i.e., '''MPI tasks''', run simultaneously and communicate via the Message Passing Interface (MPI) paradigm. MPI tasks do not share memory but can be spawned over different nodes.
 
 
Multiple MPI tasks can not be launched by the MPI parallel program itself but via '''mpirun''', e.g. 4 MPI tasks of ''my_par_program'':
 
<pre>
 
mpirun -n 4 my_par_program
 
</pre>
 
<br>
 
However, this given command can '''not''' be directly included in your '''msub''' command for submitting as a batch job to the compute cluster, [[BwUniCluster_Batch_Jobs#Handling_job_script_options_and_arguments|see above]].
 
 
Instead, generate a wrapper script, ''job.sh'' containing the following lines:
 
{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 2px;"
 
| style="width:280px; text-align:center; white-space:nowrap; color:#000;" |
 
<source lang="bash">
 
#!/bin/bash
 
module load <placeholder>
 
mpirun my_par_program
 
</source>
 
|}
 
'''Attention:''' Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since MOAB instructs mpirun about number of processes and node hostnames.
 
 
Moreover, replace <placeholder> with all the required modulefiles to enable the MPI environment.
 
<br>
 
 
Considering 4 MPI tasks on a single node, each requiring 1000 MByte, and running for 1 hour, execute:
 
<pre>
 
msub -l nodes=1:ppn=4,pmem=1000mb,walltime=01:00:00 job.sh
 
</pre>
 
<br>
 
 
Launching and running 32 MPI tasks on 4 nodes, each requiring 1000 MByte, and running for 5 hours, execute:
 
<pre>
 
msub -l nodes=4:ppn=8,pmem=1000mb,walltime=05:00:00 job.sh
 
</pre>
 
<br>
 
 
=== Multithreaded + MPI parallel Programs ===
 
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes.
 
 
Multiple MPI tasks must be launched by the MPI parallel program '''mpirun'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).
 
 
Thus a job-script to submit a batch job called ''test_ompi_omp'' that runs a MPI program with 2 tasks and an eightfold threaded program ''my_par_program'' requiring 12000 MByte of total physical memory and total wall clock time of 3 hours looks like:
 
 
<!--b)-->
 
{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
 
| style="width:280px; text-align:center; white-space:nowrap; color:#000;" |
 
<source lang="bash">
 
#!/bin/bash
 
#MSUB -l nodes=2:ppn=16
 
#MSUB -l walltime=03:00:00
 
#MSUB -l pmem=12000mb
 
#MSUB -v MPI_MODULE=mpi/impi
 
#MSUB -v OMP_NUM_THREADS=8
 
#MSUB -v MPIRUN_OPTIONS="-print-rank-map"
 
#MSUB -v EXE=./parmmul_omp_uc1_impi_O3
 
#MSUB -N test_ompi_omp
 
 
module load ${MPI_MODULE}
 
TASK_COUNT=$((${MOAB_PROCCOUNT}/${OMP_NUM_THREADS}))
 
echo "${EXE} running on ${MOAB_PROCCOUNT} cores with ${TASK_COUNT} MPI-tasks and ${OMP_NUM_THREADS} threads"
 
startexe="mpiexec.hydra -bootstrap slurm ${MPIRUN_OPTIONS} -n ${TASK_COUNT} ${EXE}"
 
echo $startexe
 
exec $startexe
 
</source>
 
|}
 
 
If necessary, replace <placeholder> with the required modulefile to enable the MPI environment and execute the script '''job_mpi_omp.sh''' without any msub command line options:
 
<pre>
 
$ msub job_mpi_omp.sh
 
</pre>
 
<br>
 
The MSUB option '''tpt''' means ''tasks per thread''. With the mpirun option -bind-to-core MPI tasks and OpenMP threads are bound to physical cores. With the option -bysocket (neighbored) MPI tasks will be attached to different sockets and -cpus-per-proc means that for the threads of a single MPI task ${OMP_NUM_THREADS} cores will be available.
 
 
A job-script to submit a batch job called ''large_test'' that runs a OpenMPI program with 4 tasks and a sixteen-fold threaded program ''my_par_program'' requiring 48000 MByte of total physical memory and total wall clock time of 6 hours looks like:
 
 
<!--b)-->
 
{| style="width: 100%; border:1px solid #d0cfcc; background:#f2f7ff;border-spacing: 5px;"
 
| style="width:280px; text-align:center; white-space:nowrap; color:#000;" |
 
<source lang="bash">
 
#!/bin/bash
 
#MSUB -l nodes=2:ppn=16
 
#MSUB -l walltime=03:00:00
 
#MSUB -l pmem=12000mb
 
#MSUB -v MPI_MODULE=mpi/impi
 
#MSUB -v OMP_NUM_THREADS=8
 
#MSUB -v MPIRUN_OPTIONS="-print-rank-map"
 
#MSUB -v EXE=./parmmul_omp_uc1_impi_O3
 
#MSUB -N pmmul-omp+impi
 
 
module load ${MPI_MODULE}
 
TASK_COUNT=$((${MOAB_PROCCOUNT}/${OMP_NUM_THREADS}))
 
echo "${EXE} running on ${MOAB_PROCCOUNT} cores with ${TASK_COUNT} MPI-tasks and ${OMP_NUM_THREADS} threads"
 
startexe="mpiexec.hydra -bootstrap slurm ${MPIRUN_OPTIONS} -n ${TASK_COUNT} ${EXE}"
 
echo $startexe
 
exec $startexe
 
</source>
 
|}
 
<br>
 
If all cores of a node are used by threads the option '''-bynode''' must be chosen. The option -bysocket does not work because the MP tasks '''must''' be attached to different nodes. The mpirun-options '''-bind-to-core''', '''-bysocket''' or '''-bynode''' and '''-cpus-per-proc''' should always be used when running a multithreaded MPI program.
 
<br>
 
<br>
 
 
=== Interactive Jobs ===
 
[[BwUniCluster_User_Access#Allowed_activities_on_login_nodes|Interactive jobs must not run on the logins nodes]], however resources for interactive jobs can be requested using msub. Considering a serial application with a graphical frontend that requires 5000 MByte of memory and limiting the interactive run to 2 hours execute the following:
 
<pre>
 
$ msub -v DISPLAY,HOME -l nodes=1:ppn=1,pmem=5000mb,walltime=02:00:00 -I
 
</pre>
 
<br>
 
After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system MOAB has granted you the requested resources on the compute system. Once granted you will be automatically logged on the dedicated resource. Now you have an interactive session with 1 core and 5000 MByte of memory on the compute system for 2 hours. Simply execute now your application:
 
<pre>
 
$ cd to_path
 
$ ./application
 
</pre>
 
Note that, once the walltime limit has been reached you will be automatically logged out of the compute system.
 
<br>
 
 
 
= Status of batch system/jobs =
 
== Available resources - showbf ==
 
The following command can be used any user to find out how many processors are '''available for immediate use''' on the system:
 
<pre>
 
$ showbf
 
</pre>
 
<br>
 
For further options of ''showbf'' read the manpage of ''showbf'':
 
<pre>
 
$ man showbf
 
</pre>
 
<br>
 
 
 
== List of submitted jobs - showq ==
 
The following command displays information about active, eligible, blocked, and/or recently completed jobs:
 
<pre>
 
$ showq
 
</pre>
 
<br>
 
For further options of ''showq'' read the manpage of ''showq'':
 
<pre>
 
$ man showq
 
</pre>
 
<br>
 
 
 
== Detailed job information - checkjob ==
 
''checkjob <jobID>'' displays detailed job state information and diagnostic output for the job of ''<jobID>'':
 
<pre>
 
$ checkjob <jobID>
 
</pre>
 
<br>
 
For further options of ''checkjob'' read the manpage of ''checkjob'':
 
<pre>
 
$ man checkjob
 
</pre>
 
<br>
 
 
 
= Job management =
 
== Canceling own jobs ==
 
''canceljob <jobID>'' cancels the own job with ''<jobID>''.
 
<pre>
 
$ canceljob <jobID>
 
</pre>
 
<br>
 
Note that only own jobs can be cancelled. The command:
 
<pre>
 
$ mjobctl -c <jobID>
 
</pre>
 
has the same effect as ''canceljob <jobID>''.
 
<br>
 
 
 
= Environment Variables for Batch Jobs =
 
Once an eligible compute jobs starts on the compute system, MOAB adds the following variables to the job's environment:
 
{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"
 
! colspan="3" style="background-color:#999999;padding:3px"| MOAB variables
 
|- style="width:25%;height=20px; text-align:left;padding:3px"
 
! style="width:20%;height=20px; text-align:left;padding:3px"| Environment variables
 
! style="height=20px; text-align:left;padding:3px"| Description
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | MOAB_CLASS
 
| style="height=20px; text-align:left;padding:3px"| Class name
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | MOAB_GROUP
 
| style="height=20px; text-align:left;padding:3px"| Group name
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | MOAB_JOBID
 
| style="height=20px; text-align:left;padding:3px"| Job ID
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | MOAB_JOBNAME
 
| style="height=20px; text-align:left;padding:3px"| Job name
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | MOAB_NODECOUNT
 
| style="height=20px; text-align:left;padding:3px"| Number of nodes allocated to job
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | MOAB_PARTITION
 
| style="height=20px; text-align:left;padding:3px"| Partition name the job is running in
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | MOAB_PROCCOUNT
 
| style="height=20px; text-align:left;padding:3px"| Number of processors allocated to job
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | MOAB_SUBMITDIR
 
| style="height=20px; text-align:left;padding:3px"| Directory of job submission
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | MOAB_USER
 
| style="height=20px; text-align:left;padding:3px"| User name
 
|}
 
<br>
 
 
Further environment variables are added by the resource manager SLURM:
 
{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"
 
! colspan="3" style="background-color:#999999;padding:3px"| SLURM variables
 
|- style="width:25%;height=20px; text-align:left;padding:3px"
 
! style="width:20%;height=20px; text-align:left;padding:3px"| Environment variables
 
! style="height=20px; text-align:left;padding:3px"| Description
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_CPUS_PER_NODE
 
| style="height=20px; text-align:left;padding:3px"| Number of processes per node dedicated to the job
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NODELIST
 
| style="height=20px; text-align:left;padding:3px"| List of nodes dedicated to the job
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_JOB_NUM_NODES
 
| style="height=20px; text-align:left;padding:3px"| Number of nodes dedicated to the job
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_MEM_PER_NODE
 
| style="height=20px; text-align:left;padding:3px"| Memory per node dedicated to the job
 
|-
 
| style="width:20%;height=20px; text-align:left;padding:3px" | SLURM_NPROCS
 
| style="height=20px; text-align:left;padding:3px"| Total number of processes dedicated to the job
 
|}
 
<br>
 
Both MOAB and SLURM environment variables can be used to generalize you job scripts, compare [[BwUniCluster_Batch_Jobs#msub_Examples|msub examples]].
 
<br>
 
<br>
 
 
----
 
[[Category:bwHPC|Job Submission]]
 
[[Category:bwUniCluster|Moab]]
 
[[Category:bwUniCluster|Job Submission]]
 

Latest revision as of 10:04, 15 August 2023