Running Calculations

← This page is used in the HPC Glossary to explain the term "Batch Scheduler" and "Batch System"

Life Cycle of a Calculation (Job)

On your desktop computer you start your calculations and they start immediately, running until they are finished. Then your desktop does mostly nothing, until you start another calculation. A compute cluster has several hundred, maybe a thousand computers (compute nodes), all of them are busy most of the time and many people want to run a great number of calculations. So running your job has to include some extra steps:

1. prepare a script (a set commands to run - usually as a shell script), with all the commands that are necessary to run your calculation from start to finish. In addition to the commands necessary to run the calculation, this batch script has a header section, in which you specify details like required compute cores (processing units witin a computer), estimated runtime, memory requirements, disk space needed, etc.
2. Submit the script into a queue, where your job (calculation)
3. gets asigned an inital priority, is queued and waits in row with other compute jobs until the resources you requested in the header become available. (Requested time is also a resource!)
4. Execution: Once a suitable resource slot is available and your job is in the front of the queue of suitable jobs (suitable with regards to the resource slot), your script is executed on (a) compute node(s). Your calculation runs on that/those node(s) until it is finished or reaches the specified time limit.
5. Save results: Include commands to save the calculation results back to a long term storage (e.g. your home directory), at least at the end of your script. What you have not saved until the job finishes won't be saved!
6. If your job reaches the specified time limit, all your running processes will be killed and the resources get cleared. So any data that has not been saved will be lost!

The software that distributes jobs on compute nodes is called a batch system or batch scheduler. The software currently used as a batch system on bwHPC clusters is "Slurm".

Learn more about the functioning of job distribution in

→ batch system

Example Jobs

For most software that a bwHPC project installed on the cluster, we have prepared an example job script running some example calculation with that exact software.

How to access these examples is described in the "Software job examples" section of the page

→ Environment Modules

Job Script and Job Submission

Batch jobs are submitted with the command:

$ sbatch <job-script>

A job script contains options for Slurm in lines beginning with #SBATCH as well as your commands which you want to execute on the compute nodes. For example:

#!/bin/bash
#SBATCH --partition=<cluster specific>
#SBATCH --ntasks-per-node=8 
#SBATCH --time=00:05:00
#SBATCH --mem=1gb
#SBATCH --export=NONE
echo 'Here would start your calculation script'

Please see cluster specific examples from the links in the next section, because exact options needed differ by cluster.

Link to Batch System and Examples per Cluster

Because of differences in configuration (partly due to different available hardware), each cluster has their own batch system documentation:

→ Slurm bwUniCluster 3.0

→ Slurm JUSTUS 2

→ Slurm Helix

→ Slurm NEMO2

→ Slurm BinAC2

How To Run Jobs Efficiently

When you are running your calculations, you will have to decide on how many compute-cores your job will be simultaneously calculated. For this, your computational problem will have to be divided into pieces, which always causes some overhead.

How to find a reasonable number of how many compute cores to use for your calculation can be found under

→ Scaling

Running calculations on an HPC node consumes a lot of energy. To make the most of the available resources and keep cluster and energy use as efficient as possible please also see our advice for

→ Energy Efficient Cluster Usage

To optimize the job efficiency, it is very helpful to check which resources a job has actually used.

→ Job Monitoring

Dos and Don'ts

There are a number of pitfalls users might encounter when starting to use compute clusters. To avoid some of them, we've collected some best practices in the form of a Dos and Dont's list. It is meant to serve as guidance, not as an absolute imperative, because sometimes stricly avoiding every undesired behaviour is not feasible, though discussing those cases with the cluster operators beforehand is a good idea.

Dos: Bundle large amounts of tiny calculations (e.g. 1000 short running analyses taking 2 minutes each) together into fewer / a single job(s) that orchestrates them to run consecutively (and maybe even distributes them among different cores) Estimate the total calculation time of your job. A short test run (either on your local machine or on the cluster) can help, though transferability and extrapolation lead to inaccuracies. → Request a calculation time larger than your estimate. For a new calculation about 20%-30% more time than estimated often is a feasable starting point. Good time estimates are below 10% of extra time requested. → Having the requested time be close to the actual calculation time is beneficial for the cluster because it helps the scheduler to be more precise and thus shortens idle times. Plan early what calculations you want to perform and create a rough timetable for when you want to submit them. Keep cluster waiting times in mind and ideally have some buffer before important deadlines (e.g. paper submission deadlines). → The scheduler does not know of your urgency and you have no way of communicating your urgency to the scheduler. Therefore the bwHPC recommends to avoid generating the need for urgent calculations. We however also know that sometimes urgency can not be avoided. In such cases contact the cluster operators to see if they can help you out. Don't ask for this lightly, because if everything is urgent, nothing is! → Many things have a tendency to avoid obliging to plans. Sudden problems can arise both on the user end and on the cluster end, so having a personal time buffer for such cases is good for anxiety prevention. Be mindful when performing very long calculations. For a multitude of reasons a computation job may suddenly end during the run. The longer the calculation the higher the risk of an unexpected error. → If possible create checkpoint files regularly during your calculations, from which you can pick up the calculation again. Keep communication bottlenecks in mind when requesting resources. → With greater physical distance between compute units (cores) comes a longer communication time between them. This leads to efficiency losses depending on the communication volume and frequency.

Don'ts: Do not run very short jobs (single digit minutes) on regular compute nodes. Some clusters offer separate partitions for short running jobs. → For the sake of efficiency small calculations should use few cores and accept runtimes larger than 1 hour. → If a parallelizable compute job calculates in under 1 hour on 1 core you should check your reasoning why it needs to run on the cluster at all instead of your local machine. Most (if not all) modern consumer machines do have multiple cores available and often have higher clock speeds than cluster CPUs. The true power of modern compute clusters comes from splitting large parallelizable calculations over many compute cores, usually at the cost of somewhat lower core clock speeds. Do not unnecessarily split your calculation into many tiny short running jobs to "trick" the backfilling system by using every 5 minute idle gap that may occur. → Allocation of a job onto the compute nodes takes up some time for every job, as does cleanup afterwards. Thus by splitting a calculation its overall efficiency decreases accordingly. → The scheduler assigns and updates priorities and starting times of jobs frequently. A higher number of jobs in the queue means a higher workload for the scheduler, leading to unnecessary strain of the system. Do not request resources that remain unused by your job. → The Resource Management System calculates your recent usage by multiplying the number of requested compute resources (CPU cores / GPUs) by the time the job was allocated. So if 90% of the cores are idle most of the time or not used at all doesn't matter to the system, it will always use 100% of the requested compute resources for the calculation of the Fair Share Value. For the time it however uses the actual allocated time of the jobs instead of what was requested. → Requesting lots of compute resources for a job usually leads to longer waiting times in the queue compared to less resource intensive jobs. Do not spawn too many threads with your jobs. Although multithreading allows some code to run overall more efficient due to better resource usage, spawning too many threads on a compute unit has the contrary effect because the threads have to be managed. → Be mindful when increasing the thread count and analyse your actual performance gain.