When (not) to use a HPC Cluster: Difference between revisions
K Siegmund (talk | contribs) |
K Siegmund (talk | contribs) No edit summary |
||
| Line 2: | Line 2: | ||
== Misconceptions == |
|||
== |
=== "everything will be faster" === |
||
[[File:BwUniCluster_2.0_Feb2020_1024x423.jpg|right|thumb|alt=bwUniCluster2.0 |upright=1| many compute cores: every "tray" in these racks contains 1-4 computers, every computer typically has 50-200 compute cores ]] |
[[File:BwUniCluster_2.0_Feb2020_1024x423.jpg|right|thumb|alt=bwUniCluster2.0 |upright=1| many compute cores: every "tray" in these racks contains 1-4 computers, every computer typically has 50-200 compute cores ]] |
||
| Line 11: | Line 12: | ||
* GPU |
* GPU |
||
=== "It's just a computer, I know how to use a computer" === |
|||
While the clusters are indeed "just" many linux computers, they are also multi-million Euro instruments. So while they may have many similarities with a linux desktop, in some respects, it is better to think of them like of other expensive instruments you may use in your research. |
|||
== When '''not''' to use HPC computers == |
== When '''not''' to use HPC computers == |
||
Revision as of 17:20, 8 October 2024
I have calculations to do - should I try using one of the bwHPC clusters for my tasks?
Misconceptions
"everything will be faster"
The performance of a single compute core is not higher than that of your typical desktop computer. The main things we can offer are:
- many compute cores
- much RAM (memory)
- GPU
"It's just a computer, I know how to use a computer"
While the clusters are indeed "just" many linux computers, they are also multi-million Euro instruments. So while they may have many similarities with a linux desktop, in some respects, it is better to think of them like of other expensive instruments you may use in your research.
When not to use HPC computers
- your calculation runs in serial (only on one compute core)
- your workflow requires that you run one calculation, then analyze it, then run the next calculation
Except 3) or 4) of the next segment force you to use the clusters anyway.
When to use HPC computers
1) many separate calculations run in parallel
2) software can efficiently divide your problem into many smaller problems that run in parallel
3) you need more RAM (memory) than your computer can provide
4) you need mor GPUs and have no other source
What are the Costs?
Monetary
No cost to you as the end-user
(But of course the HPC systems are bought for millions of Euro and the power cost of running calculations is in the same order of magnitude as buying the systems)
Effort / Time
There is quite a learning curve to start calculating.
- Linux shell knowledge is a major part
- HPC-specific knowledge:
- software module system - software supplied by the cluster maintainers (vide infra)
- how to use the scheduler / write job scripts: how to send calculations to a computer on the cluster (vide infra)
- Wait times: a good cluster is always busy. Expect a waiting time bewtween 1h and 2 days until your calculation(s) start
Compute Workflow
A short description of the workflow how running calculation works can be found under Running Calculations and can give you a general idea.
Software
Basic usage is very simple: you run module load module_name and use the software, but important documentation and examples are also built into the modules.
The usage of the software on the cluster is described in Environment Modules
Still With Us?
If you feel your calculations meet the requirements and you will save a lot of time despite some learning overhead, proceed to the Registration page.