BinAC2/Hardware and Architecture

System Architecture

The bwForCluster BinAC 2 supports researchers from the broader fields of Bioinformatics, Astrophysics, and Geosciences.

Operating System and Software

• Operating System: ...
• Queuing System: Slurm (see BinAC2/Slurm for help)
• (Scientific) Libraries and Software: Environment Modules

Compute Nodes

BinAC 2 offers compute nodes, high-mem nodes, and two types of GPU nodes.

• 148 compute nodes
• 14 SMP node
• 32 GPU nodes (A30)
• 8 GPU nodes (A100)
• splus several special purpose nodes for login, interactive jobs, etc.

Compute node specification:

	Standard	High-Mem	GPU (A30)	GPU (A100)
Quantity	148	14	32	8
Processors	2 x AMD EPYC Milan 7543	2 x AMD EPYC Milan 7443	2 x AMD EPYC Milan 7543	2 x AMD EPYC Milan 7543
Processor Frequency (GHz)	2.80	2.85	2.80	2.80
Number of Cores	64	48	64	64
Working Memory (GB)	512	2048	512	512
Local Disk (GB)	512 (SSD)	1920 (SSD)	512 (SSD)	512 (SSD)
Interconnect	HDR IB (80 nodes) / 100GbE	HDR	HDR	HDR
Coprocessors	-	-	2 x NVIDIA A30 (24 GB ECC HBM2, NVLink	4 x (80 GB ECC HBM2e)

Special Purpose Nodes

Besides the classical compute node several nodes serve as login and preprocessing nodes, nodes for interactive jobs and nodes for creating virtual environments providing a virtual service environment.

Storage Architecture

The bwForCluster BinAC consists of two separate storage systems, one for the user's home directory $HOME and one serving as a work space. The home directory is limited in space and parallel access but offers snapshots of your files and Backup. The work space is a parallel file system which offers fast and parallel file access and a bigger capacity than the home directory. This storage is based on BeeGFS and can be accessed parallel from many nodes. Additionally, each compute node provides high-speed temporary storage (SSD) on the node-local solid state disk via the $TMPDIR environment variable.

	$HOME	Work Space	$TMPDIR
Visibility	global	global	node local
Lifetime	permanent	work space lifetime (max. 30 days, max. 3 extensions)	batch job walltime
Capacity	unkn.	482 TB	211 GB per node
Quotas	40 GB per user	none	none
Backup	yes	no	no

 global             : all nodes access the same file system
 local              : each node has its own file system
 permanent          : files are stored permanently
 batch job walltime : files are removed at end of the batch job

$HOME

Home directories are meant for permanent file storage of files that are keep being used like source codes, configuration files, executable programs etc.; the content of home directories will be backed up on a regular basis.

 NOTE:
 Compute jobs on nodes must not write temporary data to $HOME.
 Instead they should use the local $TMPDIR directory for I/O-heavy use cases
 and work spaces for less I/O intense multinode-jobs.

Work Space

Work spaces can be generated through the workspace tools. This will generate a directory on the parallel storage.

To create a work space you'll need to supply a name for your work space area and a lifetime in days. For more information read the corresponding help, e.g: ws_allocate -h.

Examples:

Command	Action
ws_allocate mywork 30	Allocate a work space named "mywork" for 30 days.
ws_allocate myotherwork	Allocate a work space named "myotherwork" with maximum lifetime.
ws_list -a	List all your work spaces.
ws_find mywork	Get absolute path of work space "mywork".
ws_extend mywork 30	Extend life me of work space mywork by 30 days from now. (Not needed, workspaces on BinAC are not limited).
ws_release mywork	Manually erase your work space "mywork". Please remove directory content first.

Local Disk Space

All compute nodes are equipped with a local SSD with 200 GB capacity for job execution. During computation the environment variable $TMPDIR points to this local disk space. The data will become unavailable as soon as the job has finished.

SDS@hd

SDS@hd is mounted only on login03 at /sds_hd. To access your Speichervorhaben, please see the SDS@hd documentation. If you can't see your Speichervorhaben, you can open a ticket.