BwUniCluster3.0/Hardware and Architecture - Revision history

R Barthel at 22:45, 3 September 2025

2025-09-03T22:45:34Z

← Older revision		Revision as of 00:45, 4 September 2025
Line 149:		Line 149:
	\| -		\| -
	\| -		\| -
	\| 94 GB		\| 94 GiB
	\| APU		\| APU
	\| 80 GB / 94 GB		\| 80 GiB / 94 GiB
	\| 40 GB		\| 40 GiB
	\| -		\| -
	\|-		\|-

R Barthel at 22:44, 3 September 2025

2025-09-03T22:44:22Z

← Older revision		Revision as of 00:44, 4 September 2025
Line 116:		Line 116:
	\|-		\|-
	!scope="column"\| Main memory		!scope="column"\| Main memory
	\| 256 GB		\| 256 GiB
	\| 384 GB		\| 384 GiB
	\| ~~2.3~~ TB		\| 2304 GiB
	\| 768 GB		\| 768 GiB
	\| 4x 128 GB HBM3		\| 4x 128 GiB HBM3
	\| 512 GB		\| 512 GiB
	\| 384 GB		\| 384 GiB
	\| 384 GB		\| 384 GiB
	\|-		\|-
	!scope="column"\| Local SSD		!scope="column"\| Local SSD

S Braun: /* Compute nodes */

2025-09-02T09:45:21Z

Compute nodes

← Older revision		Revision as of 11:45, 2 September 2025
Line 52:		Line 52:
	* '''AMD GPU x4''': AMD's accelerated processing unit (APU) MI300A with 4 CPU sockets and 4 compute units which share the same high-bandwidth memory (HBM).		* '''AMD GPU x4''': AMD's accelerated processing unit (APU) MI300A with 4 CPU sockets and 4 compute units which share the same high-bandwidth memory (HBM).
	* '''Ice Lake NVIDIA GPU x4''': Similar to the Ice Lake nodes, but with larger memory and four NVIDIA A100 or H100 GPUs.		* '''Ice Lake NVIDIA GPU x4''': Similar to the Ice Lake nodes, but with larger memory and four NVIDIA A100 or H100 GPUs.
			* '''Cascade Lake NVIDIA GPU x4''': Nodes with four NVIDIA A100 GPUs.
	{\| class="wikitable"		{\| class="wikitable"
	\|-		\|-
Line 61:		Line 62:
	! style="width:10%"\| GPU node<br/>AMD GPU x4		! style="width:10%"\| GPU node<br/>AMD GPU x4
	! style="width:10%"\| GPU nodes<br/>Ice Lake<br/>NVIDIA GPU x4		! style="width:10%"\| GPU nodes<br/>Ice Lake<br/>NVIDIA GPU x4
			! style="width:10%"\| GPU nodes<br/>Cascade Lake<br/>NVIDIA GPU x4
	! style="width:10%"\| Login nodes		! style="width:10%"\| Login nodes
	\|-		\|-
Line 70:		Line 72:
	\| <code>gpu_mi300</code>		\| <code>gpu_mi300</code>
	\| <code>gpu_a100_il</code> / <code>gpu_h100_il</code>		\| <code>gpu_a100_il</code> / <code>gpu_h100_il</code>
			\| <code>gpu_a100_short</code></code>
	\| -		\| -
	\|-		\|-
Line 79:		Line 82:
	\| 1		\| 1
	\| 15		\| 15
			\| 19
	\| 2		\| 2
	\|-		\|-
Line 88:		Line 92:
	\| AMD Zen 4		\| AMD Zen 4
	\| Intel Xeon Platinum 8358		\| Intel Xeon Platinum 8358
			\| Intel Xeon Gold 6248R
	\| AMD EPYC 9454		\| AMD EPYC 9454
	\|-		\|-
Line 98:		Line 103:
	\| 2		\| 2
	\| 2		\| 2
	\|-		\| 2
	!scope="column"\| Processor frequency (GHz)
	\| 2.6 GHz
	\| 2.75 GHz
	\| 2.75 GHz
	\| 2.75 GHz
	\| 3.7 GHz
	\| 2.6 GHz
	\| 2.75 GHz
	\|-		\|-
	!scope="column"\| Total number of cores		!scope="column"\| Total number of cores
Line 115:		Line 112:
	\| 96 (4x 24)		\| 96 (4x 24)
	\| 64		\| 64
			\| 48
	\| 96		\| 96
	\|-		\|-
Line 124:		Line 122:
	\| 4x 128 GB HBM3		\| 4x 128 GB HBM3
	\| 512 GB		\| 512 GB
			\| 384 GB
	\| 384 GB		\| 384 GB
	\|-		\|-
Line 132:		Line 131:
	\| 15.36 TB NVMe		\| 15.36 TB NVMe
	\| 7.68 TB NVMe		\| 7.68 TB NVMe
	\| 6.4 TB NVMe		\| 6.4 TB NVMe
			\| 1.92 TB SATA SSD
	\| 7.68 TB SATA SSD		\| 7.68 TB SATA SSD
	\|-		\|-
Line 142:		Line 142:
	\| 4x AMD Instinct MI300A		\| 4x AMD Instinct MI300A
	\| 4x NVIDIA A100 / H100		\| 4x NVIDIA A100 / H100
			\| 4x NVIDIA A100
	\| -		\| -
	\|-		\|-
Line 150:		Line 151:
	\| 94 GB		\| 94 GB
	\| APU		\| APU
	\| 80 GB / 94 GB		\| 80 GB / 94 GB
			\| 40 GB
	\| -		\| -
	\|-		\|-
Line 160:		Line 162:
	\| IB 2x NDR200		\| IB 2x NDR200
	\| IB 2x HDR200		\| IB 2x HDR200
			\| IB 4x EDR
	\| IB 1x NDR200		\| IB 1x NDR200
	\|}		\|}

S Braun: /* File Systems */

2025-04-06T17:20:26Z

File Systems

← Older revision		Revision as of 19:20, 6 April 2025
Line 189:		Line 189:
	several months.		several months.

	For further details please check		For further details please check: [[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details\|File System Details]]

	[[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details\|File System Details]]

	== $HOME ==		== $HOME ==

S Braun: /* File Systems */

2025-04-06T17:18:47Z

File Systems

← Older revision		Revision as of 19:18, 6 April 2025
Line 177:		Line 177:
	'''Which file system to use?'''		'''Which file system to use?'''

	~~In general, you~~ should separate your data and store it on the appropriate file system.		You should separate your data and store it on the appropriate file system.
	Permanently needed data like software or important results should be stored ~~below~~ $HOME		Permanently needed data like software or important results should be stored in $HOME but capacity restrictions (quotas) apply.
	~~but capacity restrictions (quotas) apply.~~ In case you accidentally deleted data on $HOME		In case you accidentally deleted data on $HOME there is a chance that we can restore it from backup.
			Permanent data which is not needed for months or exceeds the capacity restrictions should be sent to the LSDF Online Storage or to the archive and deleted from the file systems. Temporary data which is only needed on a single node and which does not exceed the disk space shown in Table 1 above should be stored
	there is a chance that we can restore it from backup. Permanent data which is not needed for months
	or exceeds the capacity restrictions should be sent to the LSDF Online Storage
	or to the archive and deleted from the file systems. Temporary data which is only needed on a single
	node and which does not exceed the disk space shown in Table 1 above should be stored
	below $TMPDIR. Data which is read many times on a single node, e.g. if you are doing AI training,		below $TMPDIR. Data which is read many times on a single node, e.g. if you are doing AI training,
	should be copied to $TMPDIR and read from there. Temporary data which is used from many nodes		should be copied to $TMPDIR and read from there. Temporary data which is used from many nodes

S Braun: /* Architecture of bwUniCluster 3.0 */

2025-04-06T17:14:55Z

Architecture of bwUniCluster 3.0

← Older revision		Revision as of 19:14, 6 April 2025
Line 13:		Line 13:
	The individual nodes of the system act in different roles. From an end users point of view the different groups of nodes are login nodes and compute nodes. File server nodes and administrative server nodes are not accessible by users.		The individual nodes of the system act in different roles. From an end users point of view the different groups of nodes are login nodes and compute nodes. File server nodes and administrative server nodes are not accessible by users.

	'''Login Nodes'''		'''Login Nodes'''<br>

	The login nodes are the only nodes directly accessible by end users. These nodes are used for interactive login, file management, program development, and interactive pre- and post-processing.		The login nodes are the only nodes directly accessible by end users. These nodes are used for interactive login, file management, program development, and interactive pre- and post-processing.
	There are two nodes dedicated to this service, but they can all be reached from a single address: <code>uc3.scc.kit.edu</code>. A DNS round-robin alias distributes login sessions to the login nodes.		There are two nodes dedicated to this service, but they can all be reached from a single address: <code>uc3.scc.kit.edu</code>. A DNS round-robin alias distributes login sessions to the login nodes.
	To prevent login nodes from being used for activities that are not permitted there and that affect the user experience of other users, '''long-running and/or compute-intensive tasks are periodically terminated without any prior warning'''. Please refer to [[BwUniCluster3.0/Login#Allowed_Activities_on_Login_Nodes\|Allowed Activities on Login Nodes]].		To prevent login nodes from being used for activities that are not permitted there and that affect the user experience of other users, '''long-running and/or compute-intensive tasks are periodically terminated without any prior warning'''. Please refer to [[BwUniCluster3.0/Login#Allowed_Activities_on_Login_Nodes\|Allowed Activities on Login Nodes]].

	'''Compute Nodes'''		'''Compute Nodes'''<br>

	The majority of nodes are compute nodes which are managed by a batch system. Users submit their jobs to the SLURM batch system and a job is executed when the required resources become available (depending on its fair-share priority).		The majority of nodes are compute nodes which are managed by a batch system. Users submit their jobs to the SLURM batch system and a job is executed when the required resources become available (depending on its fair-share priority).

	'''File Systems'''		'''File Systems'''<br>
	bwUniCluster 3.0 comprises two parallel file systems based on Lustre.		bwUniCluster 3.0 comprises two parallel file systems based on Lustre.

R Laifer: /* File Systems */

2025-04-03T15:34:26Z

File Systems

← Older revision		Revision as of 17:34, 3 April 2025
Line 171:		Line 171:

	* '''$HOME'''<br>The HOME directory is created automatically after account activation, and the environment variable $HOME holds its name. HOME is the place, where users find themselves after login.		* '''$HOME'''<br>The HOME directory is created automatically after account activation, and the environment variable $HOME holds its name. HOME is the place, where users find themselves after login.
	* '''Workspaces'''<br>Users can create so-called workspaces for non-permanent data with temporary lifetime~~. A further workspace type based on flash-only storage for special requirements is also available~~.		* '''Workspaces'''<br>Users can create so-called workspaces for non-permanent data with temporary lifetime.
			* '''Workspaces on flash storage'''<br>A further workspace file system based on flash-only storage is available for special requirements and certain users.
	* '''$TMPDIR'''<br>The directory $TMPDIR is only available and visible on the local node during the runtime of a compute job. It is located on fast SSD storage devices.		* '''$TMPDIR'''<br>The directory $TMPDIR is only available and visible on the local node during the runtime of a compute job. It is located on fast SSD storage devices.
	* '''BeeOND''' (BeeGFS On-Demand)<br>On request a parallel on-demand file system (BeeOND) is created which uses the SSDs of the nodes which were allocated to the batch job.		* '''BeeOND''' (BeeGFS On-Demand)<br>On request a parallel on-demand file system (BeeOND) is created which uses the SSDs of the nodes which were allocated to the batch job.
Line 215:		Line 216:

	[[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#Workspaces\|Detailed information on Workspaces]]		[[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#Workspaces\|Detailed information on Workspaces]]

			== Workspaces on flash storage ==

			Another workspace file system based on flash-only storage is available for special requirements and certain users.
			If possible, this file system should be used from the Ice Lake nodes of bwUniCluster 3.0 (queue ''cpu_il'').
			It provides high IOPS rates and better performance for small files. The quota limts are lower than on the
			normal workspace file system.

			[[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#Workspaces_on_flash_storage\|Detailed information on Workspaces on flash storage]]

	== $TMPDIR ==		== $TMPDIR ==

R Laifer: /* BeeOND (BeeGFS On-Demand) */

2025-04-03T14:33:59Z

BeeOND (BeeGFS On-Demand)

← Older revision		Revision as of 16:33, 3 April 2025
Line 227:		Line 227:
	== BeeOND (BeeGFS On-Demand) ==		== BeeOND (BeeGFS On-Demand) ==

	Users have possibility to request a private BeeOND (on-demand BeeGFS) parallel filesystem for each job. The file system is created during job startup and purged when your job completes.		Users have the possibility to request a private BeeOND (on-demand BeeGFS) parallel filesystem for each job. The file system is created during job startup and purged when your job completes.

	[[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#BeeOND_(BeeGFS_On-Demand)\|Detailed information on BeeOND]]		[[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#BeeOND_(BeeGFS_On-Demand)\|Detailed information on BeeOND]]

R Laifer: /* File Systems */

2025-04-03T14:32:03Z

File Systems

← Older revision		Revision as of 16:32, 3 April 2025
Line 225:		Line 225:
	[[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#$TMPDIR\|Detailed information on $TMPDIR]]		[[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#$TMPDIR\|Detailed information on $TMPDIR]]

		⚫	== BeeOND (BeeGFS On-Demand) ==
⚫	== LSDF Online Storage==

		⚫	Users have possibility to request a private BeeOND (on-demand BeeGFS) parallel filesystem for each job. The file system is created during job startup and purged when your job completes.
	In some cases it is useful to have access to the LSDF Online Storage on the HPC-Clusters also. Therefore the LSDF Online Storage is mounted on the Login- and Datamover-Nodes.
	Furthermore it can be used on the compute nodes during the job runtime with the constraint flag "LSDF" ([[bwUniCluster_3.0_Slurm_common_Features\|Slurm common features]]
	). There is also an example about the LSDF batch usage: [[bwUniCluster_3.0_Slurm_common_Features#LSDF_Online_Storage\|Slurm LSDF example ]] .
	<pre>
	#!/bin/bash
	#SBATCH ...
	#SBATCH --constraint=LSDF
	</pre>
	<br>
	For the usage of the LSDF Online Storage the following environment variables are available: $LSDF, $LSDFPROJECTS, $LSDFHOME.
	Please request storage projects in the LSDF Online Storage seperately:
	[https://www.lsdf.kit.edu/os/storagerequest/: LSDF Storage Request].

			[[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#BeeOND_(BeeGFS_On-Demand)\|Detailed information on BeeOND]]
⚫	==BeeOND (BeeGFS On-Demand)==

		⚫	== LSDF Online Storage ==
⚫	Users ~~of the UniCluster~~ have possibility to request a private BeeOND (on-demand BeeGFS) parallel filesystem for each job. The file system is created during job startup and purged ~~after~~ your job.

			The LSDF Online Storage allows dedicated users to store scientific measurement data and simulation results. BwUniCluster 3.0 has an extremely fast network connection to the LSDF Online Storage. This file system provides external access via different protocols and is only available for certain users.
	; '''IMPORTANT: '''
	: All data on the private filesystem will be deleted after your job. Make sure you have copied your data back to the global filesystem (within job), e.g., $HOME or any workspace.

			[[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#LSDF_Online_Storage\|Detailed information on LSDF Online Storage]]
	BeeOND/BeeGFS can be used like any other parallel file system. Tools like cp or rsync can be used to copy data in and out.

	For detailed usage see here: [[BwUniCluster_3.0_Slurm_common_Features#BeeOND_.28BeeGFS_On-Demand.29\|Request on-demand file system]]


	<!-- =Backup and Archiving=

	There are regular backups of all data of the home directories, whereas ACLs and extended attributes will
	not be backuped.

	Please open a ticket if you need backuped data.
	-->

R Laifer: /* $TMPDIR */ Adapted TMPDIR description

2025-04-03T13:19:16Z

$TMPDIR: Adapted TMPDIR description

@@ Line 218: / Line 218: @@
 == $TMPDIR ==
-The environment variable $TMPDIR contains the name of a directory which is located on the local SSD of each node. This means
+The environment variable $TMPDIR contains the name of a directory which is located on the local SSD of each node.
 [[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#$TMPDIR|Detailed information on $TMPDIR]]
 == LSDF Online Storage==