When (not) to use a HPC Cluster: Difference between revisions

From bwHPC Wiki
Jump to navigation Jump to search
 
(9 intermediate revisions by 2 users not shown)
Line 1: Line 1:
My calculations take a long time - should I try using one of the bwHPC clusters for my tasks?
I have calculations to do - should I try using one of the bwHPC clusters for my tasks?


== Misconceptions ==
== Misconception: "everything will be faster" ==
=== "everything will be faster" ===
[[File:BwUniCluster_2.0_Feb2020_1024x423.jpg|right|thumb|alt=bwUniCluster2.0 |upright=1| many compute cores: every "tray" in these racks contains 1-4 computers, every computer typically has 50-200 compute cores ]]
[[File:BwUniCluster_2.0_Feb2020_1024x423.jpg|right|thumb|alt=bwUniCluster2.0 |upright=1| many compute cores: every "tray" in these racks contains 1-4 computers, every computer typically has 50-200 compute cores ]]


Line 10: Line 12:
* GPU
* GPU


=== "It's just a computer, I know how to use a computer" ===


While the clusters are indeed "just" many linux computers, they are also multi-million Euro instruments. So while they may have many similarities with a linux desktop, in some respects, it is better to think of them like of other expensive instruments you may use in your research.
⚫
== When '''not''' to use HPC computers ==

⚫
* your calculation runs in serial (only on one compute core)
⚫
* your workflow requires that you run one calculation, then analyze it, then run the next calculation

⚫
Except 3) or 4) of the next segment force you to use the clusters anyway.


== When to use HPC computers ==
== When to use HPC computers ==
Line 26: Line 24:
3) you need more RAM (memory) than your computer can provide
3) you need more RAM (memory) than your computer can provide


4) you need mor GPUs and have no other source
4) you need more GPUs and have no other source

5) you need to process large amounts of data that do not fit on your computer

⚫
== When '''not''' to use HPC computers ==

You will probably not have a big benefit from using HPC when:

⚫
* your calculation runs in serial (only on one compute core) and you have few calculations
⚫
* your workflow requires that you run one calculation, then analyze it, then run the next calculation

⚫
Except 3) or 4) of "when to use a cluster" force you to use the clusters anyway.


== What are the Costs? ==
== What are the Costs? ==
Line 48: Line 57:
=== Compute Workflow ===
=== Compute Workflow ===


A very general description of the workflow how running calculation works can be found under [[Running Calculations]] and can give you a general idea.
A short description of the workflow how running calculation works can be found under [[Running Calculations]] and can give you a general idea.


=== Software ===
=== Software ===
Line 55: Line 64:


The usage of the software on the cluster is described in [[Environment Modules]]
The usage of the software on the cluster is described in [[Environment Modules]]

== Still With Us? ==

If you feel your calculations meet the requirements and you will save a lot of time despite some learning overhead, proceed to the [[Registration]] page.

Latest revision as of 17:45, 18 November 2024

I have calculations to do - should I try using one of the bwHPC clusters for my tasks?


Misconceptions

"everything will be faster"

bwUniCluster2.0
many compute cores: every "tray" in these racks contains 1-4 computers, every computer typically has 50-200 compute cores

The performance of a single compute core is not higher than that of your typical desktop computer. The main things we can offer are:

  • many compute cores
  • much RAM (memory)
  • GPU

"It's just a computer, I know how to use a computer"

While the clusters are indeed "just" many linux computers, they are also multi-million Euro instruments. So while they may have many similarities with a linux desktop, in some respects, it is better to think of them like of other expensive instruments you may use in your research.

When to use HPC computers

1) many separate calculations run in parallel

2) software can efficiently divide your problem into many smaller problems that run in parallel

3) you need more RAM (memory) than your computer can provide

4) you need more GPUs and have no other source

5) you need to process large amounts of data that do not fit on your computer

When not to use HPC computers

You will probably not have a big benefit from using HPC when:

  • your calculation runs in serial (only on one compute core) and you have few calculations
  • your workflow requires that you run one calculation, then analyze it, then run the next calculation

Except 3) or 4) of "when to use a cluster" force you to use the clusters anyway.

What are the Costs?

Monetary

No cost to you as the end-user

(But of course the HPC systems are bought for millions of Euro and the power cost of running calculations is in the same order of magnitude as buying the systems)

Effort / Time

There is quite a learning curve to start calculating.

  • Linux shell knowledge is a major part
  • HPC-specific knowledge:
    • software module system - software supplied by the cluster maintainers (vide infra)
    • how to use the scheduler / write job scripts: how to send calculations to a computer on the cluster (vide infra)
  • Wait times: a good cluster is always busy. Expect a waiting time bewtween 1h and 2 days until your calculation(s) start

Compute Workflow

A short description of the workflow how running calculation works can be found under Running Calculations and can give you a general idea.

Software

Basic usage is very simple: you run module load module_name and use the software, but important documentation and examples are also built into the modules.

The usage of the software on the cluster is described in Environment Modules

Still With Us?

If you feel your calculations meet the requirements and you will save a lot of time despite some learning overhead, proceed to the Registration page.

Retrieved from "https://wiki.bwhpc.de/wiki/index.php?title=When_(not)_to_use_a_HPC_Cluster&oldid=13249"