When (not) to use a HPC Cluster: Difference between revisions
K Siegmund (talk | contribs) (Created page with "My calculations take a long time - should I try using one of the bwHPC clusters for my tasks? == Misconception: "everything will be faster" == right|thumb|alt=bwUniCluster2.0 |upright=1| many compute cores: every "tray" in these racks contains 1-4 computers, every computer typically has 50-200 compute cores The performance of a single compute core is not higher than that of your typical desktop computer. The main things...") |
S Richling (talk | contribs) |
||
| (11 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
I have calculations to do - should I try using one of the bwHPC clusters for my tasks? |
|||
== Misconceptions == |
|||
== |
=== "everything will be faster" === |
||
[[File:BwUniCluster_2.0_Feb2020_1024x423.jpg|right|thumb|alt=bwUniCluster2.0 |upright=1| many compute cores: every "tray" in these racks contains 1-4 computers, every computer typically has 50-200 compute cores ]] |
[[File:BwUniCluster_2.0_Feb2020_1024x423.jpg|right|thumb|alt=bwUniCluster2.0 |upright=1| many compute cores: every "tray" in these racks contains 1-4 computers, every computer typically has 50-200 compute cores ]] |
||
| Line 10: | Line 12: | ||
* GPU |
* GPU |
||
=== "It's just a computer, I know how to use a computer" === |
|||
While the clusters are indeed "just" many linux computers, they are also multi-million Euro instruments. So while they may have many similarities with a linux desktop, in some respects, it is better to think of them like of other expensive instruments you may use in your research. |
|||
| ⚫ | |||
| ⚫ | |||
| ⚫ | |||
| ⚫ | |||
== When to use HPC computers == |
== When to use HPC computers == |
||
1) many separate calculations run in parallel |
1) many separate calculations run in parallel |
||
2) software can efficiently divide your problem into many smaller problems that run in parallel |
2) software can efficiently divide your problem into many smaller problems that run in parallel |
||
3) you need more RAM (memory) than your computer can provide |
3) you need more RAM (memory) than your computer can provide |
||
| ⚫ | |||
| ⚫ | |||
5) you need to process large amounts of data that do not fit on your computer |
|||
| ⚫ | |||
You will probably not have a big benefit from using HPC when: |
|||
| ⚫ | |||
| ⚫ | |||
| ⚫ | |||
== What are the Costs? == |
== What are the Costs? == |
||
| Line 46: | Line 57: | ||
=== Compute Workflow === |
=== Compute Workflow === |
||
A |
A short description of the workflow how running calculation works can be found under [[Running Calculations]] and can give you a general idea. |
||
=== Software === |
=== Software === |
||
| Line 53: | Line 64: | ||
The usage of the software on the cluster is described in [[Environment Modules]] |
The usage of the software on the cluster is described in [[Environment Modules]] |
||
== Still With Us? == |
|||
If you feel your calculations meet the requirements and you will save a lot of time despite some learning overhead, proceed to the [[Registration]] page. |
|||
Latest revision as of 17:45, 18 November 2024
I have calculations to do - should I try using one of the bwHPC clusters for my tasks?
Misconceptions
"everything will be faster"
The performance of a single compute core is not higher than that of your typical desktop computer. The main things we can offer are:
- many compute cores
- much RAM (memory)
- GPU
"It's just a computer, I know how to use a computer"
While the clusters are indeed "just" many linux computers, they are also multi-million Euro instruments. So while they may have many similarities with a linux desktop, in some respects, it is better to think of them like of other expensive instruments you may use in your research.
When to use HPC computers
1) many separate calculations run in parallel
2) software can efficiently divide your problem into many smaller problems that run in parallel
3) you need more RAM (memory) than your computer can provide
4) you need more GPUs and have no other source
5) you need to process large amounts of data that do not fit on your computer
When not to use HPC computers
You will probably not have a big benefit from using HPC when:
- your calculation runs in serial (only on one compute core) and you have few calculations
- your workflow requires that you run one calculation, then analyze it, then run the next calculation
Except 3) or 4) of "when to use a cluster" force you to use the clusters anyway.
What are the Costs?
Monetary
No cost to you as the end-user
(But of course the HPC systems are bought for millions of Euro and the power cost of running calculations is in the same order of magnitude as buying the systems)
Effort / Time
There is quite a learning curve to start calculating.
- Linux shell knowledge is a major part
- HPC-specific knowledge:
- software module system - software supplied by the cluster maintainers (vide infra)
- how to use the scheduler / write job scripts: how to send calculations to a computer on the cluster (vide infra)
- Wait times: a good cluster is always busy. Expect a waiting time bewtween 1h and 2 days until your calculation(s) start
Compute Workflow
A short description of the workflow how running calculation works can be found under Running Calculations and can give you a general idea.
Software
Basic usage is very simple: you run module load module_name and use the software, but important documentation and examples are also built into the modules.
The usage of the software on the cluster is described in Environment Modules
Still With Us?
If you feel your calculations meet the requirements and you will save a lot of time despite some learning overhead, proceed to the Registration page.