Helix/Hardware: Difference between revisions
< Helix
Jump to navigation
Jump to search
S Richling (talk | contribs) No edit summary |
S Richling (talk | contribs) Tag: Manual revert |
||
| (123 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
== System Architecture == |
== System Architecture == |
||
The bwForCluster Helix is a high performance supercomputer with high speed interconnect. |
The bwForCluster Helix is a high performance supercomputer with high speed interconnect. The system consists of compute nodes (CPU and GPU nodes), some infrastructure nodes for login and administration and a storage system. All components are connected via a fast Infiniband network. The login nodes are also connected to the Internet via Baden Württemberg's extended LAN BelWü. |
||
[[File:Helix-skizze.png|400px]] |
|||
== Operating System and Software == |
|||
* Operating system: RedHat |
|||
* Queuing system: [[Helix/Slurm | Slurm]] |
|||
* Access to application software: [[Software_Modules|Environment Modules]] |
|||
== Compute Nodes == |
|||
The cluster is equipped with the following CPU and GPU nodes. |
|||
{| class="wikitable" style="width:80%;" |
|||
|- |
|||
! style="width:20%" | |
|||
! style="width:40%" colspan="2" style="text-align:center" | CPU Nodes |
|||
! style="width:40%" colspan="4" style="text-align:center" | GPU Nodes |
|||
|- |
|||
!scope="column"; style="background-color:#A2C4EB" | Node Type |
|||
!style="text-align:left;"| cpu |
|||
!style="text-align:left;"| fat |
|||
!style="text-align:left;"| gpu4 |
|||
!style="text-align:left;"| gpu4 |
|||
!style="text-align:left;"| gpu8 |
|||
!style="text-align:left;"| gpu8 |
|||
|- |
|||
!scope="column"| Quantity |
|||
| 355 |
|||
| 15 |
|||
| 29 |
|||
| 26 |
|||
| 4 |
|||
| 3 |
|||
|- |
|||
!scope="column" | Processors |
|||
| 2 x AMD EPYC 7513 |
|||
| 2 x AMD EPYC 7513 |
|||
| 2 x AMD EPYC 7513 |
|||
| 2 x AMD EPYC 7513 |
|||
| 2 x AMD EPYC 7513 |
|||
| 2 x AMD EPYC 9334 |
|||
|- |
|||
!scope="column" | Processor Frequency (GHz) |
|||
| 2.6 |
|||
| 2.6 |
|||
| 2.6 |
|||
| 2.6 |
|||
| 2.6 |
|||
| 2.7 |
|||
|- |
|||
!scope="column"; style="background-color:#A2C4EB" | Number of Cores per Node |
|||
!style="text-align:left;"| 64 |
|||
!style="text-align:left;"| 64 |
|||
!style="text-align:left;"| 64 |
|||
!style="text-align:left;"| 64 |
|||
!style="text-align:left;"| 64 |
|||
!style="text-align:left;"| 64 |
|||
|- |
|||
!scope="column" | Installed Working Memory (GB) |
|||
| 256 |
|||
| 2048 |
|||
| 256 |
|||
| 256 |
|||
| 2048 |
|||
| 2304 |
|||
|- |
|||
!scope="column"; style="background-color:#A2C4EB" | Available Memory for Jobs (GB) |
|||
!style="text-align:left;"| 236 |
|||
!style="text-align:left;"| 2000 |
|||
!style="text-align:left;"| 236 |
|||
!style="text-align:left;"| 236 |
|||
!style="text-align:left;"| 2000 |
|||
!style="text-align:left;"| 2200 |
|||
|- |
|||
!scope="column" | Interconnect |
|||
| 1x HDR100 |
|||
| 1x HDR100 |
|||
| 2x HDR100 |
|||
| 2x HDR200 |
|||
| 4x HDR200 |
|||
| 4x HDR200 |
|||
|- |
|||
!scope="column" | Coprocessors |
|||
| - |
|||
| - |
|||
| 4x [https://www.nvidia.com/en-us/data-center/a40/ Nvidia A40] (48 GB) |
|||
| 4x [https://www.nvidia.com/en-us/data-center/a100/ Nvidia A100] (40 GB) |
|||
| 8x [https://www.nvidia.com/en-us/data-center/a100/ Nvidia A100] (80 GB) |
|||
| 8x [https://www.nvidia.com/en-us/data-center/h200/ Nvidia H200] (141 GB) |
|||
|- |
|||
!scope="column"; style="background-color:#A2C4EB" | Number of GPUs |
|||
!style="text-align:left;"| - |
|||
!style="text-align:left;"| - |
|||
!style="text-align:left;"| 4 |
|||
!style="text-align:left;"| 4 |
|||
!style="text-align:left;"| 8 |
|||
!style="text-align:left;"| 8 |
|||
|- |
|||
!scope="column"; style="background-color:#A2C4EB" | GPU Type |
|||
!style="text-align:left;"| - |
|||
!style="text-align:left;"| - |
|||
!style="text-align:left;"| A40 |
|||
!style="text-align:left;"| A100 |
|||
!style="text-align:left;"| A100 |
|||
!style="text-align:left;"| H200 |
|||
|- |
|||
!scope="column"; style="background-color:#A2C4EB" | GPU Memory per GPU (GB) |
|||
!style="text-align:left;"| - |
|||
!style="text-align:left;"| - |
|||
!style="text-align:left;"| 48 |
|||
!style="text-align:left;"| 40 |
|||
!style="text-align:left;"| 80 |
|||
!style="text-align:left;"| 141 |
|||
|- |
|||
!scope="column"; style="background-color:#A2C4EB" | GPU Feature |
|||
!style="text-align:left;"| - |
|||
!style="text-align:left;"| - |
|||
!style="text-align:left;"| - |
|||
!style="text-align:left;"| fp64 |
|||
!style="text-align:left;"| fp64 |
|||
!style="text-align:left;"| fp64 |
|||
|} |
|||
The lines marked in blue are in particular relevant for [https://wiki.bwhpc.de/e/Helix/Slurm Slurm jobs]. |
|||
== Storage Architecture == |
|||
There is one storage system providing a large parallel file system based on IBM Storage Scale for $HOME, for workspaces, and for temporary job data. The compute nodes do not have local disks. |
|||
== Network == |
|||
The components of the cluster are connected via two independent networks, a management network (Ethernet and IPMI) and an Infiniband fabric for MPI communication and storage access. |
|||
The Infiniband backbone is a fully non-blocking fabric with 200 Gb/s data speed. The compute nodes are connected with different data speeds according to the node configuration. |
|||
The term HDR100 stands for 100 Gb/s and HDR200 for 200 Gb/s. |
|||
Latest revision as of 08:51, 24 April 2025
System Architecture
The bwForCluster Helix is a high performance supercomputer with high speed interconnect. The system consists of compute nodes (CPU and GPU nodes), some infrastructure nodes for login and administration and a storage system. All components are connected via a fast Infiniband network. The login nodes are also connected to the Internet via Baden Württemberg's extended LAN BelWü.
Operating System and Software
- Operating system: RedHat
- Queuing system: Slurm
- Access to application software: Environment Modules
Compute Nodes
The cluster is equipped with the following CPU and GPU nodes.
| CPU Nodes | GPU Nodes | |||||
|---|---|---|---|---|---|---|
| Node Type | cpu | fat | gpu4 | gpu4 | gpu8 | gpu8 |
| Quantity | 355 | 15 | 29 | 26 | 4 | 3 |
| Processors | 2 x AMD EPYC 7513 | 2 x AMD EPYC 7513 | 2 x AMD EPYC 7513 | 2 x AMD EPYC 7513 | 2 x AMD EPYC 7513 | 2 x AMD EPYC 9334 |
| Processor Frequency (GHz) | 2.6 | 2.6 | 2.6 | 2.6 | 2.6 | 2.7 |
| Number of Cores per Node | 64 | 64 | 64 | 64 | 64 | 64 |
| Installed Working Memory (GB) | 256 | 2048 | 256 | 256 | 2048 | 2304 |
| Available Memory for Jobs (GB) | 236 | 2000 | 236 | 236 | 2000 | 2200 |
| Interconnect | 1x HDR100 | 1x HDR100 | 2x HDR100 | 2x HDR200 | 4x HDR200 | 4x HDR200 |
| Coprocessors | - | - | 4x Nvidia A40 (48 GB) | 4x Nvidia A100 (40 GB) | 8x Nvidia A100 (80 GB) | 8x Nvidia H200 (141 GB) |
| Number of GPUs | - | - | 4 | 4 | 8 | 8 |
| GPU Type | - | - | A40 | A100 | A100 | H200 |
| GPU Memory per GPU (GB) | - | - | 48 | 40 | 80 | 141 |
| GPU Feature | - | - | - | fp64 | fp64 | fp64 |
The lines marked in blue are in particular relevant for Slurm jobs.
Storage Architecture
There is one storage system providing a large parallel file system based on IBM Storage Scale for $HOME, for workspaces, and for temporary job data. The compute nodes do not have local disks.
Network
The components of the cluster are connected via two independent networks, a management network (Ethernet and IPMI) and an Infiniband fabric for MPI communication and storage access. The Infiniband backbone is a fully non-blocking fabric with 200 Gb/s data speed. The compute nodes are connected with different data speeds according to the node configuration. The term HDR100 stands for 100 Gb/s and HDR200 for 200 Gb/s.