When (not) to use a HPC Cluster: Difference between revisions
K Siegmund (talk | contribs) (Created page with "My calculations take a long time - should I try using one of the bwHPC clusters for my tasks? == Misconception: "everything will be faster" == right|thumb|alt=bwUniCluster2.0 |upright=1| many compute cores: every "tray" in these racks contains 1-4 computers, every computer typically has 50-200 compute cores The performance of a single compute core is not higher than that of your typical desktop computer. The main things...") |
K Siegmund (talk | contribs) |
||
| Line 21: | Line 21: | ||
1) many separate calculations run in parallel |
1) many separate calculations run in parallel |
||
2) software can efficiently divide your problem into many smaller problems that run in parallel |
2) software can efficiently divide your problem into many smaller problems that run in parallel |
||
3) you need more RAM (memory) than your computer can provide |
3) you need more RAM (memory) than your computer can provide |
||
| ⚫ | |||
| ⚫ | |||
== What are the Costs? == |
== What are the Costs? == |
||
Revision as of 16:41, 8 October 2024
My calculations take a long time - should I try using one of the bwHPC clusters for my tasks?
Misconception: "everything will be faster"
The performance of a single compute core is not higher than that of your typical desktop computer. The main things we can offer are:
- many compute cores
- much RAM (memory)
- GPU
When not to use HPC computers
- your calculation runs in serial (only on one compute core)
- your workflow requires that you run one calculation, then analyze it, then run the next calculation
Except 3) or 4) of the next segment force you to use the clusters anyway.
When to use HPC computers
1) many separate calculations run in parallel
2) software can efficiently divide your problem into many smaller problems that run in parallel
3) you need more RAM (memory) than your computer can provide
4) you need mor GPUs and have no other source
What are the Costs?
Monetary
No cost to you as the end-user
(But of course the HPC systems are bought for millions of Euro and the power cost of running calculations is in the same order of magnitude as buying the systems)
Effort / Time
There is quite a learning curve to start calculating.
- Linux shell knowledge is a major part
- HPC-specific knowledge:
- software module system - software supplied by the cluster maintainers (vide infra)
- how to use the scheduler / write job scripts: how to send calculations to a computer on the cluster (vide infra)
- Wait times: a good cluster is always busy. Expect a waiting time bewtween 1h and 2 days until your calculation(s) start
Compute Workflow
A very general description of the workflow how running calculation works can be found under Running Calculations and can give you a general idea.
Software
Basic usage is very simple: you run module load module_name and use the software, but important documentation and examples are also built into the modules.
The usage of the software on the cluster is described in Environment Modules