BwForCluster BinAC Hardware and Architecture
1 System Architecture
The bwForCluster BinAC is intended for compute activities related to Bioinformatics and Astrophysics research.
1.1 Operating System and Software
- Operating System: RHEL 7
- Queuing System: MOAB / Torque (see Batch Jobs for help)
- (Scientific) Libraries and Software: Environment Modules
1.2 Compute Nodes
BinAC offers 236 compute nodes, 62 GPU nodes plus several special purpose nodes for login, interactive jobs, etc.
Compute node specification:
|Processors||2 x Intel Xeon E5-2630v4 (Broadwell)||4 x Intel Xeon E5-4620v3 (Haswell)||2 x Intel Xeon E5-2630v4 (Broadwell)|
|Processor Frequency (GHz)||2.4||2.0||2.4|
|Number of Cores||28||40||28|
|Working Memory (GB)||128||1024||128|
|Local Disk (GB)||256 (SSD)||256 (SSD)||256 (SSD)|
|Coprocessors||–||–||2 x Nvidia Tesla K80|
1.3 Special Purpose Nodes
Besides the classical compute node several nodes serve as login and preprocessing nodes, nodes for interactive jobs and nodes for creating virtual environments providing a virtual service environment.
2 Storage Architecture
The bwForCluster BinAC consists of two separate storage systems, one for the user's home directory $HOME and one serving as a work space. The home directory is limited in space and parallel access but offers snapshots of your files and Backup. The work space is a parallel file system which offers fast and parallel file access and a bigger capacity than the home directory. This storage is based on BeeGFS and can be accessed parallel from many nodes. Additionally, each compute node provides high-speed temporary storage (SSD) on the node-local solid state disk via the $TMPDIR environment variable.
|Lifetime||permanent||work space lifetime (max. 30 days, max. 3 extensions)||batch job walltime|
|Capacity||unkn.||482 TB||211 GB per node|
|Quotas||20 GB per user||none||none|
global : all nodes access the same file system local : each node has its own file system permanent : files are stored permanently batch job walltime : files are removed at end of the batch job
Home directories are meant for permanent file storage of files that are keep being used like source codes, configuration files, executable programs etc.; the content of home directories will be backed up on a regular basis.
NOTE: Compute jobs on nodes must not write temporary data to $HOME. Instead they should use the local $TMPDIR directory for I/O-heavy use cases and work spaces for less I/O intense multinode-jobs.
2.2 Work Space
Work spaces can be generated through the workspace tools. This will generate a directory on the parallel storage with a limited lifetime. When this lifetime is reached the work space will be deleted automatically after a grace period. Work spaces can be extended to prevent deletion. You can create reminders and calendar entries to prevent accidental removal.
To create a work space you'll need to supply a name for your work space area and a lifetime in days. For more information read the corresponding help, e.g: ws_allocate -h.
Defaults and maximum values:
|Default and maximum lifetime (days)||30|
|ws_allocate mywork 30||Allocate a work space named "mywork" for 30 days.|
|ws_allocate myotherwork||Allocate a work space named "myotherwork" with maximum lifetime.|
|ws_list -a||List all your work spaces.|
|ws_find mywork||Get absolute path of work space "mywork".|
|ws_extend mywork 30||Extend life me of work space mywork by 30 days from now.|
|ws_release mywork||Manually erase your work space "mywork". Please remove directory content first.|
2.3 Local Disk Space
All compute nodes are equipped with a local SSD with 200 GB capacity for job execution. During computation the environment variable $TMPDIR points to this local disk space. The data will become unavailable as soon as the job has finished.