Running Calculations: Difference between revisions
mNo edit summary |
K Siegmund (talk | contribs) |
||
| Line 1: | Line 1: | ||
== Description == |
== Description == |
||
[[File:running_calculations_on_cluster.svg|thumb|upright=0.4]] |
[[File:running_calculations_on_cluster.svg|thumb|upright=0.4]] |
||
On your desktop computer, you start your calculations and they start immediately, run until they are finished, then your desktop does mostly nothing, until you start another calculation. A |
On your desktop computer, you start your calculations and they start immediately, run until they are finished, then your desktop does mostly nothing, until you start another calculation. A compute cluster has several hundred, maybe a thousand computers (compute nodes), all of them are busy most of the time and many people want to run a great number of calculations. So running your job has to include some extra steps: |
||
# prepare a |
# prepare a script (a set commands to run - usually as a shell script), with all the commands that are necessary to run your calculation from start to finish. In addition to the commands necessary to run the calculation, this ''[[batch script]]'' has a header section, in which you specify details like required compute cores (processing units witin a computer), estimated runtime, memory requirements, disk space needed, etc. |
||
# '' |
# ''Submit'' the script into a queue, where your ''job'' (calculation) |
||
# |
# is queued and waits in row with other compute jobs until the resources you requested in the header become available. |
||
# Execution: Once your job reaches the front of the queue, your script is executed on a compute node. Your calculation runs on that node until it is finished or reaches the specified time limit. |
# Execution: Once your job reaches the front of the queue, your script is executed on a compute node. Your calculation runs on that node until it is finished or reaches the specified time limit. |
||
# Save results: At the end of your script, include commands to save the calculation results back to your home directory. |
# Save results: At the end of your script, include commands to save the calculation results back to your home directory. |
||
There are two types of [[batch system]]s currently used on bwHPC clusters, called " |
There are two types of [[batch system]]s currently used on bwHPC clusters, called "Moab" (legacy installs) and "Slurm". |
||
== Link to Batch System per Cluster == |
== Link to Batch System per Cluster == |
||
Revision as of 16:21, 8 January 2024
Description
On your desktop computer, you start your calculations and they start immediately, run until they are finished, then your desktop does mostly nothing, until you start another calculation. A compute cluster has several hundred, maybe a thousand computers (compute nodes), all of them are busy most of the time and many people want to run a great number of calculations. So running your job has to include some extra steps:
- prepare a script (a set commands to run - usually as a shell script), with all the commands that are necessary to run your calculation from start to finish. In addition to the commands necessary to run the calculation, this batch script has a header section, in which you specify details like required compute cores (processing units witin a computer), estimated runtime, memory requirements, disk space needed, etc.
- Submit the script into a queue, where your job (calculation)
- is queued and waits in row with other compute jobs until the resources you requested in the header become available.
- Execution: Once your job reaches the front of the queue, your script is executed on a compute node. Your calculation runs on that node until it is finished or reaches the specified time limit.
- Save results: At the end of your script, include commands to save the calculation results back to your home directory.
There are two types of batch systems currently used on bwHPC clusters, called "Moab" (legacy installs) and "Slurm".
Link to Batch System per Cluster
Because of differences in configuration (partly due to different available hardware), each cluster has their own batch system documention:
- Slurm systems
- Moab systems (legacy systems with deprecated queuing system)
 |
Scientific software installed on the bwHPC Clusters often comes with simple example jobs (job script and input files). See Software Modules on how to load examples. |
How to Use Computing Ressources Efficiently
When you are running your calculations, you will have to decide on how many compute-cores your calculation will be simultaneously calculated. For this, your computational problem will have to be divided into pieces, which always causes some overhead.
How to find a reasonable number of how many compute cores to use for your calculation can be found under Scaling
Information regarding the supported parallel programming paradigms and specific hints on their usage are summarized at Parallel_Programming
Running calculations on an HPC node consumes a lot of energy. To make the most of the available resources and keep cluster and energy use as efficient as possible please also see our advice for Energy Efficient Cluster Usage