BinAC/Hardware and Architecture: Difference between revisions

From bwHPC Wiki
Jump to navigation Jump to search
No edit summary
 
(17 intermediate revisions by 4 users not shown)
Line 1: Line 1:
== System Architecture ==
== System Architecture ==


The bwForCluster [https://www.binac.uni-tuebingen.de BinAC] is intended for compute activities related to Bioinformatics and Astrophysics research.
The bwForCluster BinAC is intended for compute activities related to Bioinformatics and Astrophysics research.




Line 28: Line 28:
=== Compute Nodes ===
=== Compute Nodes ===


BinAC offers 236 compute nodes, 60 GPU nodes plus several special purpose nodes for login, interactive jobs, etc.
BinAC offers 236 compute nodes, 62 GPU nodes plus several special purpose nodes for login, interactive jobs, etc.


Compute node specification:
Compute node specification:
Line 41: Line 41:
| 236
| 236
| 4
| 4
| 60
| 62
|-
|-
!scope="column" | Processors
!scope="column" | Processors
| 2 x [http://ark.intel.com/products/92981/Intel-Xeon-Processor-E5-2630-v4-25M-Cache-2_20-GHz Intel Xeon E5-2630v4 (Broadwell)]
| 2 x [http://ark.intel.com/products/92981/Intel-Xeon-Processor-E5-2630-v4-25M-Cache-2_20-GHz Intel Xeon E5-2630v4 (Broadwell)]
⚫
| 4 x [http://ark.intel.com/de/products/85758/Intel-Xeon-Processor-E5-4620-v3-25M-Cache-2_00-GHz Intel Xeon E5-4620v3 (Haswell)]
| 2 x [http://ark.intel.com/products/92981/Intel-Xeon-Processor-E5-2630-v4-25M-Cache-2_20-GHz Intel Xeon E5-2630v4 (Broadwell)]
| 2 x [http://ark.intel.com/products/92981/Intel-Xeon-Processor-E5-2630-v4-25M-Cache-2_20-GHz Intel Xeon E5-2630v4 (Broadwell)]

⚫
| 4 x [http://ark.intel.com/products/92981/Intel-Xeon-Processor-E5-2630-v4-25M-Cache-2_20-GHz Intel Xeon E5-4620v3 (Haswell)]
|-
|-
!scope="column" | Processor Frequency (GHz)
!scope="column" | Processor Frequency (GHz)
⚫
| 2.4
| 2.4
| 2.4
| 2.0
| 2.0
⚫
| 2.4
|-
|-
!scope="column" | Number of Cores
!scope="column" | Number of Cores
| 28
| 28
⚫
| 28
| 40
| 40
⚫
| 28
|-
|-
!scope="column" | Working Memory (GB)
!scope="column" | Working Memory (GB)
⚫
| 128
| 128
| 128
| 1024
| 1024
⚫
| 128
|-
|-
!scope="column" | Local Disk (GB)
!scope="column" | Local Disk (GB)
Line 82: Line 83:


Besides the classical compute node several nodes serve as login and preprocessing nodes, nodes for interactive jobs and nodes for creating virtual environments providing a virtual service environment.
Besides the classical compute node several nodes serve as login and preprocessing nodes, nodes for interactive jobs and nodes for creating virtual environments providing a virtual service environment.

[[Category:bwForCluster BinAC]]
[[Category:Hardware and Architecture|bwForCluster BinAC]]



== Storage Architecture ==
== Storage Architecture ==


The bwForCluster [https://www.binac.uni-tuebingen.de BinAC] consists of two separate storage systems, one for the user's home directory <tt>$HOME</tt> and one serving as a work space. The home directory is limited in space and parallel access but offers snapshots of your files and Backup. The work space is a parallel file system which offers fast and parallel file access and a bigger capacity than the home directory. This storage is based on [http://www.beegfs.com/ BeeGFS] and can be accessed parallel from many nodes. Additionally, each compute node provides high-speed temporary storage (SSD) on the node-local solid state disk via the <tt>$TMPDIR</tt> environment variable.
The bwForCluster [https://www.binac.uni-tuebingen.de BinAC] consists of two separate storage systems, one for the user's home directory <tt>$HOME</tt> and one serving as a work space. The home directory is limited in space and parallel access but offers snapshots of your files and Backup. The work space is a parallel file system which offers fast and parallel file access and a bigger capacity than the home directory. This storage is based on [https://www.beegfs.com/ BeeGFS] and can be accessed parallel from many nodes. Additionally, each compute node provides high-speed temporary storage (SSD) on the node-local solid state disk via the <tt>$TMPDIR</tt> environment variable.


{| class="wikitable"
{| class="wikitable"
Line 105: Line 102:
!scope="column" | Lifetime
!scope="column" | Lifetime
| permanent
| permanent
| work space lifetime (max. 100 days, with extensions up to 400)
| work space lifetime (max. 30 days, max. 3 extensions)
| batch job walltime
| batch job walltime
|-
|-
!scope="column" | Capacity
!scope="column" | Capacity
| unkn.
| unkn.
| 520 TB
| 482 TB
| 220 GB per node
| 211 GB per node
|-
|-
!scope="column" | [https://en.wikipedia.org/wiki/Disk_quota#Quotas Quotas]
!scope="column" | [https://en.wikipedia.org/wiki/Disk_quota#Quotas Quotas]
| 20 GB per user
| 40 GB per user
| none
| none
| none
| none
|-
|-
!scope="column" | Backup
!scope="column" | Backup
| no
| yes
| no
| no
| no
| no
Line 173: Line 170:
=== Work Space ===
=== Work Space ===


Work spaces can be generated through the <tt>workspace</tt> tools. This will generate a directory on the parallel storage with a limited lifetime. When this lifetime is reached the work space will be deleted automatically after a grace period. Work spaces can be extended to prevent deletion. You can create reminders and calendar entries to prevent accidental removal.
Work spaces can be generated through the <tt>workspace</tt> tools. This will generate a directory on the parallel storage.


To create a work space you'll need to supply a name for your work space area and a lifetime in days.
To create a work space you'll need to supply a name for your work space area and a lifetime in days.
For more information read the corresponding help, e.g: <tt>ws_allocate -h</tt>.
For more information read the corresponding help, e.g: <tt>ws_allocate -h</tt>.

Defaults and maximum values:
{| class="wikitable"
|-
| Default and maximum lifetime (days)
| 100
|-
| Maximum extensions
| 3
|-
|}


Examples:
Examples:
Line 208: Line 194:
|-
|-
|<tt>ws_extend mywork 30</tt>
|<tt>ws_extend mywork 30</tt>
|Extend life me of work space mywork by 30 days from now.
|Extend life me of work space mywork by 30 days from now. (Not needed, workspaces on BinAC are not limited).
|-
|-
|<tt>ws_release mywork</tt>
|<tt>ws_release mywork</tt>
Line 214: Line 200:
|-
|-
|}
|}



=== Local Disk Space ===
=== Local Disk Space ===
Line 220: Line 205:
All compute nodes are equipped with a local SSD with 200 GB capacity for job execution. During computation the environment variable <tt>$TMPDIR</tt> points to this local disk space. The data will become unavailable as soon as the job has finished.
All compute nodes are equipped with a local SSD with 200 GB capacity for job execution. During computation the environment variable <tt>$TMPDIR</tt> points to this local disk space. The data will become unavailable as soon as the job has finished.


=== SDS@hd ===


SDS@hd is mounted only on login03 at <tt>/sds_hd</tt>.
[[Category:BwForCluster BinAC]]
To access your Speichervorhaben, please see the [[SDS@hd/Access/NFS#access_your_data|SDS@hd documentation]].
[[Category:Hardware and Architecture|bwForCluster BinAC]]
If you can't see your Speichervorhaben, you can [[BinAC/Support|open a ticket]].
[[Category:BwForCluster BinAC|Hardware and Architecture]]

Latest revision as of 17:46, 11 December 2023

System Architecture

The bwForCluster BinAC is intended for compute activities related to Bioinformatics and Astrophysics research.


Operating System and Software


Compute Nodes

BinAC offers 236 compute nodes, 62 GPU nodes plus several special purpose nodes for login, interactive jobs, etc.

Compute node specification:

Standard Fat GPU
Quantity 236 4 62
Processors 2 x Intel Xeon E5-2630v4 (Broadwell) 4 x Intel Xeon E5-4620v3 (Haswell) 2 x Intel Xeon E5-2630v4 (Broadwell)
Processor Frequency (GHz) 2.4 2.0 2.4
Number of Cores 28 40 28
Working Memory (GB) 128 1024 128
Local Disk (GB) 256 (SSD) 256 (SSD) 256 (SSD)
Interconnect FDR FDR FDR
Coprocessors – – 2 x Nvidia Tesla K80

Special Purpose Nodes

Besides the classical compute node several nodes serve as login and preprocessing nodes, nodes for interactive jobs and nodes for creating virtual environments providing a virtual service environment.

Storage Architecture

The bwForCluster BinAC consists of two separate storage systems, one for the user's home directory $HOME and one serving as a work space. The home directory is limited in space and parallel access but offers snapshots of your files and Backup. The work space is a parallel file system which offers fast and parallel file access and a bigger capacity than the home directory. This storage is based on BeeGFS and can be accessed parallel from many nodes. Additionally, each compute node provides high-speed temporary storage (SSD) on the node-local solid state disk via the $TMPDIR environment variable.

$HOME Work Space $TMPDIR
Visibility global global node local
Lifetime permanent work space lifetime (max. 30 days, max. 3 extensions) batch job walltime
Capacity unkn. 482 TB 211 GB per node
Quotas 40 GB per user none none
Backup yes no no 
 global             : all nodes access the same file system
 local              : each node has its own file system
 permanent          : files are stored permanently
 batch job walltime : files are removed at end of the batch job


$HOME

Home directories are meant for permanent file storage of files that are keep being used like source codes, configuration files, executable programs etc.; the content of home directories will be backed up on a regular basis. 


 NOTE:
 Compute jobs on nodes must not write temporary data to $HOME.
 Instead they should use the local $TMPDIR directory for I/O-heavy use cases
 and work spaces for less I/O intense multinode-jobs.



Work Space

Work spaces can be generated through the workspace tools. This will generate a directory on the parallel storage.

To create a work space you'll need to supply a name for your work space area and a lifetime in days. For more information read the corresponding help, e.g: ws_allocate -h.

Examples:

Command Action
ws_allocate mywork 30 Allocate a work space named "mywork" for 30 days.
ws_allocate myotherwork Allocate a work space named "myotherwork" with maximum lifetime.
ws_list -a List all your work spaces.
ws_find mywork Get absolute path of work space "mywork".
ws_extend mywork 30 Extend life me of work space mywork by 30 days from now. (Not needed, workspaces on BinAC are not limited).
ws_release mywork Manually erase your work space "mywork". Please remove directory content first.

Local Disk Space

All compute nodes are equipped with a local SSD with 200 GB capacity for job execution. During computation the environment variable $TMPDIR points to this local disk space. The data will become unavailable as soon as the job has finished.

SDS@hd

SDS@hd is mounted only on login03 at /sds_hd. To access your Speichervorhaben, please see the SDS@hd documentation. If you can't see your Speichervorhaben, you can open a ticket.

Retrieved from "https://wiki.bwhpc.de/wiki/index.php?title=BinAC/Hardware_and_Architecture&oldid=12514"