BinAC/Hardware and Architecture: Difference between revisions
F Bartusch (talk | contribs) No edit summary |
F Bartusch (talk | contribs) (Remove broken link) |
||
| (3 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
== System Architecture == |
== System Architecture == |
||
The bwForCluster |
The bwForCluster BinAC is intended for compute activities related to Bioinformatics and Astrophysics research. |
||
| Line 86: | Line 86: | ||
== Storage Architecture == |
== Storage Architecture == |
||
The bwForCluster |
The bwForCluster BinAC consists of two separate storage systems, one for the user's home directory <tt>$HOME</tt> and one serving as a work space. The home directory is limited in space and parallel access but offers snapshots of your files and Backup. The work space is a parallel file system which offers fast and parallel file access and a bigger capacity than the home directory. This storage is based on [https://www.beegfs.com/ BeeGFS] and can be accessed parallel from many nodes. Additionally, each compute node provides high-speed temporary storage (SSD) on the node-local solid state disk via the <tt>$TMPDIR</tt> environment variable. |
||
{| class="wikitable" |
{| class="wikitable" |
||
| Line 170: | Line 170: | ||
=== Work Space === |
=== Work Space === |
||
Work spaces can be generated through the <tt>workspace</tt> tools. This will generate a directory on the parallel storage |
Work spaces can be generated through the <tt>workspace</tt> tools. This will generate a directory on the parallel storage. |
||
To create a work space you'll need to supply a name for your work space area and a lifetime in days. |
To create a work space you'll need to supply a name for your work space area and a lifetime in days. |
||
For more information read the corresponding help, e.g: <tt>ws_allocate -h</tt>. |
For more information read the corresponding help, e.g: <tt>ws_allocate -h</tt>. |
||
Defaults and maximum values: |
|||
{| class="wikitable" |
|||
|- |
|||
| Default and maximum lifetime (days) |
|||
| 30 |
|||
|- |
|||
| Maximum extensions |
|||
| 3 |
|||
|- |
|||
|} |
|||
Examples: |
Examples: |
||
| Line 205: | Line 194: | ||
|- |
|- |
||
|<tt>ws_extend mywork 30</tt> |
|<tt>ws_extend mywork 30</tt> |
||
|Extend life me of work space mywork by 30 days from now. |
|Extend life me of work space mywork by 30 days from now. (Not needed, workspaces on BinAC are not limited). |
||
|- |
|- |
||
|<tt>ws_release mywork</tt> |
|<tt>ws_release mywork</tt> |
||
| Line 215: | Line 204: | ||
All compute nodes are equipped with a local SSD with 200 GB capacity for job execution. During computation the environment variable <tt>$TMPDIR</tt> points to this local disk space. The data will become unavailable as soon as the job has finished. |
All compute nodes are equipped with a local SSD with 200 GB capacity for job execution. During computation the environment variable <tt>$TMPDIR</tt> points to this local disk space. The data will become unavailable as soon as the job has finished. |
||
=== SDS@hd === |
|||
SDS@hd is mounted only on login03 at <tt>/sds_hd</tt>. |
|||
To access your Speichervorhaben, please see the [[SDS@hd/Access/NFS#access_your_data|SDS@hd documentation]]. |
|||
If you can't see your Speichervorhaben, you can [[BinAC/Support|open a ticket]]. |
|||
Latest revision as of 12:40, 29 November 2024
System Architecture
The bwForCluster BinAC is intended for compute activities related to Bioinformatics and Astrophysics research.
Operating System and Software
- Operating System: RHEL 7
- Queuing System: MOAB / Torque (see Batch Jobs for help)
- (Scientific) Libraries and Software: Environment Modules
Compute Nodes
BinAC offers 236 compute nodes, 62 GPU nodes plus several special purpose nodes for login, interactive jobs, etc.
Compute node specification:
| Standard | Fat | GPU | |
|---|---|---|---|
| Quantity | 236 | 4 | 62 |
| Processors | 2 x Intel Xeon E5-2630v4 (Broadwell) | 4 x Intel Xeon E5-4620v3 (Haswell) | 2 x Intel Xeon E5-2630v4 (Broadwell) |
| Processor Frequency (GHz) | 2.4 | 2.0 | 2.4 |
| Number of Cores | 28 | 40 | 28 |
| Working Memory (GB) | 128 | 1024 | 128 |
| Local Disk (GB) | 256 (SSD) | 256 (SSD) | 256 (SSD) |
| Interconnect | FDR | FDR | FDR |
| Coprocessors | – | – | 2 x Nvidia Tesla K80 |
Special Purpose Nodes
Besides the classical compute node several nodes serve as login and preprocessing nodes, nodes for interactive jobs and nodes for creating virtual environments providing a virtual service environment.
Storage Architecture
The bwForCluster BinAC consists of two separate storage systems, one for the user's home directory $HOME and one serving as a work space. The home directory is limited in space and parallel access but offers snapshots of your files and Backup. The work space is a parallel file system which offers fast and parallel file access and a bigger capacity than the home directory. This storage is based on BeeGFS and can be accessed parallel from many nodes. Additionally, each compute node provides high-speed temporary storage (SSD) on the node-local solid state disk via the $TMPDIR environment variable.
| $HOME | Work Space | $TMPDIR | |
|---|---|---|---|
| Visibility | global | global | node local |
| Lifetime | permanent | work space lifetime (max. 30 days, max. 3 extensions) | batch job walltime |
| Capacity | unkn. | 482 TB | 211 GB per node |
| Quotas | 40 GB per user | none | none |
| Backup | yes | no | no |
global : all nodes access the same file system local : each node has its own file system permanent : files are stored permanently batch job walltime : files are removed at end of the batch job
$HOME
Home directories are meant for permanent file storage of files that are keep being used like source codes, configuration files, executable programs etc.; the content of home directories will be backed up on a regular basis.
NOTE: Compute jobs on nodes must not write temporary data to $HOME. Instead they should use the local $TMPDIR directory for I/O-heavy use cases and work spaces for less I/O intense multinode-jobs.
Work Space
Work spaces can be generated through the workspace tools. This will generate a directory on the parallel storage.
To create a work space you'll need to supply a name for your work space area and a lifetime in days. For more information read the corresponding help, e.g: ws_allocate -h.
Examples:
| Command | Action |
|---|---|
| ws_allocate mywork 30 | Allocate a work space named "mywork" for 30 days. |
| ws_allocate myotherwork | Allocate a work space named "myotherwork" with maximum lifetime. |
| ws_list -a | List all your work spaces. |
| ws_find mywork | Get absolute path of work space "mywork". |
| ws_extend mywork 30 | Extend life me of work space mywork by 30 days from now. (Not needed, workspaces on BinAC are not limited). |
| ws_release mywork | Manually erase your work space "mywork". Please remove directory content first. |
Local Disk Space
All compute nodes are equipped with a local SSD with 200 GB capacity for job execution. During computation the environment variable $TMPDIR points to this local disk space. The data will become unavailable as soon as the job has finished.
SDS@hd
SDS@hd is mounted only on login03 at /sds_hd. To access your Speichervorhaben, please see the SDS@hd documentation. If you can't see your Speichervorhaben, you can open a ticket.