NEMO/Moab

This article contains information on features of the batch job system only applicable on the "bwForCluster NEMO" in Freiburg.

1 Submitting Jobs on the bwForCluster NEMO

This page describes the details of the queuing system specific to the bwForCluster NEMO.

A general description on options that should work on all bwHPC clusters can be found on the Batch Jobs page.

Currently all worker nodes have 20 physical cores. Do not request more than 20 processes per node with the ppn flag (e.g. ppn=20).

If the requested ppn count exceeds this limit, the job will remain in idle state, i.e. it will not start running. Unfortunately msub does not inform you about wrong resource definitions. Please use checkjob -v to see if your job definitions are OK. Please select a number of cores which divides 20 evenly (e.g. ppn=[1,2,4,5,10,20]). This way the resource usage can be optimized.

Jobs run in a user only mode. This means that more than one job per user can run on the same worker node, if there are spare resources.

1.1 Limits and Queues

On the bwForCluster NEMO the queue should not be explicitly specified.

• The maximum walltime for a job is 96 hours (4 days)

 walltime=4:00:00:00 or walltime=96:00:00.

• All nodes have 20 cores and 128 GB RAM. Thus each core can use roughly 6GB RAM (pmem=6GB).

 nodes=1:ppn=20
 pmem=6gb

• Maximum used cores at any time is 6000. We use MAXPE which takes Memory into account,

see processor equivalent
 MAXPE: 4000 (soft limit), 6000 (hard limit)

• Maximum processor seconds which can be used at any time is 456192000. This increases with every job but decreases with the walltime, see basic fairness policies

 MAXPS: 304128000 (soft limit), 456192000 (hard limit)
 (calculation 4 OPA islands * 44 nodes * 20 cores * 60 sec * 60 min * 24 h)

 Example: If a job uses 3520 cores for 24h, it will reserve 304128000 processor seconds.
 This is the soft limit for the cluster.

msub -q queue
queue	node	default resources	minimum resources	theoretical maximum resources	node access policy
do not specify	standard node	nodes=1:ppn=1, walltime=01:00:00, pmem=1000mb	nodes=1:ppn=1	nodes=300:ppn=20, walltime=96:00:00, pmem=6GB	single user

1.2 Monitor Running Jobs

Once your jobs are running you can log in to the nodes where your jobs were submitted to. The nodes are listed in checkjob.

$ checkjob 12345
...
Allocated Nodes:
[n3101.nemo.privat:20][n3102.nemo.privat:20]
...

Then you can ssh into these nodes. The short host name is sufficient. Please logout after you are finished. And you can use the program pdsh to monitor your jobs non-interactively. pdsh checks where your job is running and performs a task on all nodes where your job is running.

Interactive:

$ ssh n3101

Non-interactive with ssh:

# run 'ps aux | grep <myjob>' on node n3101
$ ssh n3101 'ps aux | grep <myjob>'

Non-interactive with pdsh:

# run 'ps aux | grep <myjob>' on all nodes corresponding to jobid '12345'
$ pdsh -j 12345 'ps aux | grep  <myjob>'
n3101: fr_uid   125068  101  0.0  39040  1684 ?        Sl   12:15   0:25 <myjob>
n3102: fr_uid   125068  101  0.0  39040  1684 ?        Sl   12:15   0:25 <myjob>

# run kill '<myjob>' on all nodes corresponding to jobid '12345'
$ pdsh -j 12345 killall <myjob>

# works with array jobs as well
$ pdsh -j 12346[1] 'ps aux | grep <myjob>'
n3103: fr_uid   125068  101  0.0  39040  1684 ?        Sl   12:15   0:25 <myjob>

1.3 Interactive Jobs

Interactive jobs must NOT run on the logins nodes, however resources for interactive jobs can be requested using msub. The following example starts an interactive session on one compute node with one core for one hour:

$ msub -l nodes=1:ppn=1 -l walltime=1:00:00 -I

The option "-I" means "interactive job". After execution of this command wait until the queuing system has granted you the requested resources. Once granted you will be automatically logged on the allocated compute node.

If you use applications or tools which provide a GUI, enable X-forwarding for your interactive session with:

# use -Y for ssh X-forwarding
$ ssh -l <uid> -Y login.nemo.uni-freiburg.de
# use -X for X-forwarding
$ msub -l nodes=1:ppn=1,walltime=1:00:00 -I -X

Once the walltime limit has been reached you will be automatically logged out from the compute node.

The option "-V" exports all environment variables to the compute node of the interactive session, but if you want to test your jobs, please aviod using "-V" since this alters your job environment.

2 Simple parallel jobs with job arrays

A typical method to create "embarrassingly parallel" compute tasks is to slice a large data set into equally sized partitions and create jobs that work on their respective partition.

The manual management of hundreds of jobs becomes difficult, though. Therefore, using the job array feature is recommended.

2.1 Job array example

Create the directory $HOME/arrayjob and the job description file $HOME/arrayjob/arrayjob.moab

#!/bin/bash
#MOAB -N ARRAYJOB
#
# This is a workaround for a know bug.
# Arrayjobs need to be given the output directory
cd $HOME/arrayjob
# Now call the programm which does the work depending on the job id
python divide-and-conquer.py $MOAB_JOBARRAYINDEX

Create the worker program (python in this example):

#!/usr/bin/python
import sys
def main (argv):
print ("Executing work according to array index ", argv[1])
if __name__ == "__main__": 
    main(sys.argv)

You can now submit the job for single array indices or index ranges:

msub -t 11 arrayjob.moab
msub -t 23-42 arrayjob.moab

---