BinAC/Hardware and Architecture: Difference between revisions
F Bartusch (talk | contribs) No edit summary |
|||
(10 intermediate revisions by 3 users not shown) | |||
Line 1: | Line 1: | ||
== System Architecture == |
== System Architecture == |
||
The bwForCluster |
The bwForCluster BinAC is intended for compute activities related to Bioinformatics and Astrophysics research. |
||
Line 28: | Line 28: | ||
=== Compute Nodes === |
=== Compute Nodes === |
||
BinAC offers 236 compute nodes, |
BinAC offers 236 compute nodes, 62 GPU nodes plus several special purpose nodes for login, interactive jobs, etc. |
||
Compute node specification: |
Compute node specification: |
||
Line 41: | Line 41: | ||
| 236 |
| 236 |
||
| 4 |
| 4 |
||
| |
| 62 |
||
|- |
|- |
||
!scope="column" | Processors |
!scope="column" | Processors |
||
Line 83: | Line 83: | ||
Besides the classical compute node several nodes serve as login and preprocessing nodes, nodes for interactive jobs and nodes for creating virtual environments providing a virtual service environment. |
Besides the classical compute node several nodes serve as login and preprocessing nodes, nodes for interactive jobs and nodes for creating virtual environments providing a virtual service environment. |
||
[[Category:bwForCluster BinAC]] |
|||
[[Category:Hardware and Architecture|bwForCluster BinAC]] |
|||
== Storage Architecture == |
== Storage Architecture == |
||
The bwForCluster [https://www.binac.uni-tuebingen.de BinAC] consists of two separate storage systems, one for the user's home directory <tt>$HOME</tt> and one serving as a work space. The home directory is limited in space and parallel access but offers snapshots of your files and Backup. The work space is a parallel file system which offers fast and parallel file access and a bigger capacity than the home directory. This storage is based on [ |
The bwForCluster [https://www.binac.uni-tuebingen.de BinAC] consists of two separate storage systems, one for the user's home directory <tt>$HOME</tt> and one serving as a work space. The home directory is limited in space and parallel access but offers snapshots of your files and Backup. The work space is a parallel file system which offers fast and parallel file access and a bigger capacity than the home directory. This storage is based on [https://www.beegfs.com/ BeeGFS] and can be accessed parallel from many nodes. Additionally, each compute node provides high-speed temporary storage (SSD) on the node-local solid state disk via the <tt>$TMPDIR</tt> environment variable. |
||
{| class="wikitable" |
{| class="wikitable" |
||
Line 97: | Line 93: | ||
! style="width:10%"| <tt>$HOME</tt> |
! style="width:10%"| <tt>$HOME</tt> |
||
! style="width:10%"| Work Space |
! style="width:10%"| Work Space |
||
! style="width:10%"| <tt> |
! style="width:10%"| <tt>$TMPDIR</tt> |
||
|- |
|- |
||
!scope="column" | Visibility |
!scope="column" | Visibility |
||
Line 115: | Line 111: | ||
|- |
|- |
||
!scope="column" | [https://en.wikipedia.org/wiki/Disk_quota#Quotas Quotas] |
!scope="column" | [https://en.wikipedia.org/wiki/Disk_quota#Quotas Quotas] |
||
| |
| 40 GB per user |
||
| none |
| none |
||
| none |
| none |
||
|- |
|- |
||
!scope="column" | Backup |
!scope="column" | Backup |
||
| |
| yes |
||
| no |
| no |
||
| no |
| no |
||
Line 174: | Line 170: | ||
=== Work Space === |
=== Work Space === |
||
Work spaces can be generated through the <tt>workspace</tt> tools. This will generate a directory on the parallel storage |
Work spaces can be generated through the <tt>workspace</tt> tools. This will generate a directory on the parallel storage. |
||
To create a work space you'll need to supply a name for your work space area and a lifetime in days. |
To create a work space you'll need to supply a name for your work space area and a lifetime in days. |
||
For more information read the corresponding help, e.g: <tt>ws_allocate -h</tt>. |
For more information read the corresponding help, e.g: <tt>ws_allocate -h</tt>. |
||
Defaults and maximum values: |
|||
{| class="wikitable" |
|||
|- |
|||
| Default and maximum lifetime (days) |
|||
| 100 |
|||
|- |
|||
| Maximum extensions |
|||
| 3 |
|||
|- |
|||
|} |
|||
Examples: |
Examples: |
||
Line 209: | Line 194: | ||
|- |
|- |
||
|<tt>ws_extend mywork 30</tt> |
|<tt>ws_extend mywork 30</tt> |
||
|Extend life me of work space mywork by 30 days from now. |
|Extend life me of work space mywork by 30 days from now. (Not needed, workspaces on BinAC are not limited). |
||
|- |
|- |
||
|<tt>ws_release mywork</tt> |
|<tt>ws_release mywork</tt> |
||
Line 215: | Line 200: | ||
|- |
|- |
||
|} |
|} |
||
=== Local Disk Space === |
=== Local Disk Space === |
||
Line 221: | Line 205: | ||
All compute nodes are equipped with a local SSD with 200 GB capacity for job execution. During computation the environment variable <tt>$TMPDIR</tt> points to this local disk space. The data will become unavailable as soon as the job has finished. |
All compute nodes are equipped with a local SSD with 200 GB capacity for job execution. During computation the environment variable <tt>$TMPDIR</tt> points to this local disk space. The data will become unavailable as soon as the job has finished. |
||
=== SDS@hd === |
|||
SDS@hd is mounted only on login03 at <tt>/sds_hd</tt>. |
|||
[[Category:BwForCluster BinAC]] |
|||
To access your Speichervorhaben, please see the [[SDS@hd/Access/NFS#access_your_data|SDS@hd documentation]]. |
|||
[[Category:Hardware and Architecture|bwForCluster BinAC]] |
|||
If you can't see your Speichervorhaben, you can [[BinAC/Support|open a ticket]]. |
|||
[[Category:BwForCluster BinAC|Hardware and Architecture]] |
Latest revision as of 17:46, 11 December 2023
System Architecture
The bwForCluster BinAC is intended for compute activities related to Bioinformatics and Astrophysics research.
Operating System and Software
- Operating System: RHEL 7
- Queuing System: MOAB / Torque (see Batch Jobs for help)
- (Scientific) Libraries and Software: Environment Modules
Compute Nodes
BinAC offers 236 compute nodes, 62 GPU nodes plus several special purpose nodes for login, interactive jobs, etc.
Compute node specification:
Standard | Fat | GPU | |
---|---|---|---|
Quantity | 236 | 4 | 62 |
Processors | 2 x Intel Xeon E5-2630v4 (Broadwell) | 4 x Intel Xeon E5-4620v3 (Haswell) | 2 x Intel Xeon E5-2630v4 (Broadwell) |
Processor Frequency (GHz) | 2.4 | 2.0 | 2.4 |
Number of Cores | 28 | 40 | 28 |
Working Memory (GB) | 128 | 1024 | 128 |
Local Disk (GB) | 256 (SSD) | 256 (SSD) | 256 (SSD) |
Interconnect | FDR | FDR | FDR |
Coprocessors | – | – | 2 x Nvidia Tesla K80 |
Special Purpose Nodes
Besides the classical compute node several nodes serve as login and preprocessing nodes, nodes for interactive jobs and nodes for creating virtual environments providing a virtual service environment.
Storage Architecture
The bwForCluster BinAC consists of two separate storage systems, one for the user's home directory $HOME and one serving as a work space. The home directory is limited in space and parallel access but offers snapshots of your files and Backup. The work space is a parallel file system which offers fast and parallel file access and a bigger capacity than the home directory. This storage is based on BeeGFS and can be accessed parallel from many nodes. Additionally, each compute node provides high-speed temporary storage (SSD) on the node-local solid state disk via the $TMPDIR environment variable.
$HOME | Work Space | $TMPDIR | |
---|---|---|---|
Visibility | global | global | node local |
Lifetime | permanent | work space lifetime (max. 30 days, max. 3 extensions) | batch job walltime |
Capacity | unkn. | 482 TB | 211 GB per node |
Quotas | 40 GB per user | none | none |
Backup | yes | no | no |
global : all nodes access the same file system local : each node has its own file system permanent : files are stored permanently batch job walltime : files are removed at end of the batch job
$HOME
Home directories are meant for permanent file storage of files that are keep being used like source codes, configuration files, executable programs etc.; the content of home directories will be backed up on a regular basis.
NOTE: Compute jobs on nodes must not write temporary data to $HOME. Instead they should use the local $TMPDIR directory for I/O-heavy use cases and work spaces for less I/O intense multinode-jobs.
Work Space
Work spaces can be generated through the workspace tools. This will generate a directory on the parallel storage.
To create a work space you'll need to supply a name for your work space area and a lifetime in days. For more information read the corresponding help, e.g: ws_allocate -h.
Examples:
Command | Action |
---|---|
ws_allocate mywork 30 | Allocate a work space named "mywork" for 30 days. |
ws_allocate myotherwork | Allocate a work space named "myotherwork" with maximum lifetime. |
ws_list -a | List all your work spaces. |
ws_find mywork | Get absolute path of work space "mywork". |
ws_extend mywork 30 | Extend life me of work space mywork by 30 days from now. (Not needed, workspaces on BinAC are not limited). |
ws_release mywork | Manually erase your work space "mywork". Please remove directory content first. |
Local Disk Space
All compute nodes are equipped with a local SSD with 200 GB capacity for job execution. During computation the environment variable $TMPDIR points to this local disk space. The data will become unavailable as soon as the job has finished.
SDS@hd
SDS@hd is mounted only on login03 at /sds_hd. To access your Speichervorhaben, please see the SDS@hd documentation. If you can't see your Speichervorhaben, you can open a ticket.