BwUniCluster3.0/Hardware and Architecture/Filesystem Details - Revision history

A Flachmueller: /* Workspaces */

2026-05-13T08:23:02Z

Workspaces

← Older revision		Revision as of 10:23, 13 May 2026
Line 153:		Line 153:
	Note that the quotas include data and inodes for all of your workspaces and all of your expired workspaces (as long as they are not yet completely removed).		Note that the quotas include data and inodes for all of your workspaces and all of your expired workspaces (as long as they are not yet completely removed).

	Workspaces have a lifetime and the data on a workspace expires as a whole after a fixed period. The maximum lifetime of a workspace on uc3 is 60 days, but it can be renewed at the end of that period 3 times to a total maximum of 240 days after workspace generation.		Workspaces have a lifetime and the data on a workspace expires as a whole after a fixed period. The '''maximum lifetime''' of a workspace on uc3 is '''60 days''', but it can be '''renewed''' at the end of that period '''3 times to a total maximum of 240 days''' after workspace generation.

	Creating, deleting, finding, extending and sharing workspaces is explained on the [[workspace]] page.		Creating, deleting, finding, extending and sharing workspaces is explained on the [[workspace]] page.

H Weber at 08:37, 20 November 2025

2025-11-20T08:37:02Z

← Older revision		Revision as of 10:37, 20 November 2025
Line 368:		Line 368:
	BeeOND/BeeGFS can be used like any other parallel file system. All nodes of the batch job have acess to the same data below the same path. Tools like cp or rsync can be used to copy data in and out.		BeeOND/BeeGFS can be used like any other parallel file system. All nodes of the batch job have acess to the same data below the same path. Tools like cp or rsync can be used to copy data in and out.

	Starting and stopping BeeOND is integrated in the prolog and epilog of the cluster batch system Slurm. It can be used during job runtime if compute nodes are exclusive used. ~~You~~ ~~can~~ ~~request~~ ~~the~~ ~~creation~~ of a ~~BeeOND~~ ~~file~~ ~~system~~ ~~with~~ the ~~constraint~~ ~~flags~~ ~~''BEEOND'',~~ ~~''BEEOND_4MDS''~~ or ~~''BEEOND_MAXMDS''~~.		Starting and stopping BeeOND is integrated in the prolog and epilog of the cluster batch system Slurm. It can be used during job runtime if compute nodes are exclusive used, meaning that no other jobs are running on the same node. To achieve this, use the batch option "--exclusive" in your batch script or on command line. This is particularly important in shared partitions.

			You can request the creation of a BeeOND file system with the constraint flags ''BEEOND'', ''BEEOND_4MDS'' or ''BEEOND_MAXMDS''.
	* BEEOND: one metadata server is started on the first node		* BEEOND: one metadata server is started on the first node
	* BEEOND_4MDS: 4 metadata servers are started within your job. If you have less than 4 nodes less metadata servers are started.		* BEEOND_4MDS: 4 metadata servers are started within your job. If you have less than 4 nodes less metadata servers are started.
	* BEEOND_MAXMDS: on every node of your job a metadata server for the on_demand file system is started		* BEEOND_MAXMDS: on every node of your job a metadata server for the on_demand file system is started

	As starting point we recommend using the ~~contraint~~ ''BEEOND''. You have one of the following options to request the constraint:		As starting point we recommend using the constraint ''BEEOND''. You have one of the following options to request the constraint:

	1. Add the line <code>#SBATCH --constraint=BEEOND</code> to your job script:		1. Add the line <code>#SBATCH --constraint=BEEOND</code> to your job script:
Line 380:		Line 382:
	#SBATCH ...		#SBATCH ...
	#SBATCH --constraint=BEEOND # or BEEOND_4MDS or BEEOND_MAXMDS		#SBATCH --constraint=BEEOND # or BEEOND_4MDS or BEEOND_MAXMDS
			#SBATCH --exclusive
	</source>		</source>
	2. Add the constraint on command line:		2. Add the constraint on command line:
	<pre>		<pre>
	$ sbatch -p <queue> -N <# of nodes> -t <runtime> --mem <mem> -C BEEOND job.sh		$ sbatch -p <queue> -N <# of nodes> -t <runtime> --mem <mem> --exclusive -C BEEOND job.sh
	</pre>		</pre>

R Laifer: /* Access restrictions */

2025-04-03T15:40:23Z

Access restrictions

← Older revision		Revision as of 17:40, 3 April 2025
Line 244:		Line 244:
	=== Access restrictions ===		=== Access restrictions ===

	Only HoreKa users or KIT users of bwUniCluster 2.0 can use this file system.		Only HoreKa users or KIT users of bwUniCluster 3.0 can use this file system.

	=== Using the file system ===		=== Using the file system ===

R Laifer: /* File System Details */

2025-04-03T15:39:56Z

File System Details

← Older revision		Revision as of 17:39, 3 April 2025
Line 63:		Line 63:
	\|-		\|-
	! scope="column" style="height=20px; padding:3px"\| Read perf./node		! scope="column" style="height=20px; padding:3px"\| Read perf./node
	\| style="height=20px; text-align:left;padding:3px"\| 500 MB/s - 6 GB/s <br> depends on type of local SSD / job queue~~: <br> 520 MB/s @ single / multiple <br> 800 MB/s @ multiple_e <br> 6600 MB/s @ fat <br> 6500 MB/s @ gpu_4 <br> 6500 MB/s @ gpu_8~~		\| style="height=20px; text-align:left;padding:3px"\| 500 MB/s - 6 GB/s <br> depends on type of local SSD / job queue
	\| style="height=20px; text-align:left;padding:3px"\| 400 MB/s - 500 MB/s<br> depends on type of local SSDs / job queue~~: <br> 500 MB/s @ multiple <br> 400 MB/s @ multiple_e~~		\| style="height=20px; text-align:left;padding:3px"\| 400 MB/s - 500 MB/s<br> depends on type of local SSDs / job queue
	\| style="height=20px; text-align:left;padding:3px"\| 5 GB/s		\| style="height=20px; text-align:left;padding:3px"\| 5 GB/s
	\| style="height=20px; text-align:left;padding:3px"\| 5 GB/s		\| style="height=20px; text-align:left;padding:3px"\| 5 GB/s
Line 70:		Line 70:
	\|-		\|-
	! scope="column" style="height=20px; padding:3px"\| Write perf./node		! scope="column" style="height=20px; padding:3px"\| Write perf./node
	\| style="height=20px; text-align:left;padding:3px"\| 500 MB/s - 4 GB/s <br> depends on type of local SSD / job queue~~: <br> 500 MB/s @ single / multiple <br> 650 MB/s @ multiple_e <br> 2900 MB/s @ fat <br> 2090 MB/s @ gpu_4 <br> 4060 MB/s @ gpu_8~~		\| style="height=20px; text-align:left;padding:3px"\| 500 MB/s - 4 GB/s <br> depends on type of local SSD / job queue
	\| style="height=20px; text-align:left;padding:3px"\| 250 MB/s - 350 MB/s <br> depends on type of local SSDs / job queue~~: <br> 350 MB/s @ multiple <br> 250 MB/s @ multiple_e~~		\| style="height=20px; text-align:left;padding:3px"\| 250 MB/s - 350 MB/s <br> depends on type of local SSDs / job queue
	\| style="height=20px; text-align:left;padding:3px"\| 5 GB/s		\| style="height=20px; text-align:left;padding:3px"\| 5 GB/s
	\| style="height=20px; text-align:left;padding:3px"\| 5 GB/s		\| style="height=20px; text-align:left;padding:3px"\| 5 GB/s

R Laifer: /* File System Details */

2025-04-03T15:20:46Z

File System Details

← Older revision		Revision as of 17:20, 3 April 2025
Line 275:		Line 275:
	$TMPDIR is set to the name of the subdirectory and this name contains the job ID so that it is unique		$TMPDIR is set to the name of the subdirectory and this name contains the job ID so that it is unique
	for each job. At the end of the job the subdirectory is removed.		for each job. At the end of the job the subdirectory is removed.

			{\|style="background:#deffee; width:100%;"
			\|style="padding:5px; background:#ffa500; text-align:left"\|
			[[Image:Attention.svg\|center\|25px]]
			\|style="padding:5px; background:#ffa500; text-align:left"\|
			'''All data on $TMPDIR will be deleted when your job completes.'''<br/>
			Make sure you have copied your results back to a global filesystem, e.g., $HOME or a workspace, within your job.
			\|}

	On login nodes $TMPDIR also points to a fast directory on a local SSD disk but this directory is not unique.		On login nodes $TMPDIR also points to a fast directory on a local SSD disk but this directory is not unique.
Line 354:		Line 362:
	[[Image:Attention.svg\|center\|25px]]		[[Image:Attention.svg\|center\|25px]]
	\|style="padding:5px; background:#ffa500; text-align:left"\|		\|style="padding:5px; background:#ffa500; text-align:left"\|
	'''All data on the private filesystem will be deleted when your job completes.'''<br/>		'''All data on the private BeeOND filesystem will be deleted when your job completes.'''<br/>
	Make sure you have copied your results back to a global filesystem, e.g., $HOME or a workspace, within your job.		Make sure you have copied your results back to a global filesystem, e.g., $HOME or a workspace, within your job.
	\|}		\|}

R Laifer: /* File System Details */

2025-04-03T15:15:52Z

File System Details

R Laifer: /* Workspaces on flash storage */

2025-04-03T15:09:46Z

Workspaces on flash storage

← Older revision		Revision as of 17:09, 3 April 2025
Line 238:		Line 238:
	=== Advantages of this file system ===		=== Advantages of this file system ===

			# From the Ice Lake nodes of bwUniCluster 3.0 (queue ''cpu_il'') the network distance and latency is low compared to the normal workspace file system.
	# All storage devices are based on flash (no hard disks) with low access times. Hence performance is better compared to other parallel file systems for read and write access with small blocks and with small files, i.e. IOPS rates are improved.		# All storage devices are based on flash (no hard disks) with low access times. Hence performance is better compared to other parallel file systems for read and write access with small blocks and with small files, i.e. IOPS rates are improved.
	# The file system is mounted on bwUniCluster 2.0 and HoreKa, i.e. it can be used to share data between these clusters.		# The file system is mounted on bwUniCluster 3.0 and HoreKa, i.e. it can be used to share data between these clusters.

	=== Access restrictions ===		=== Access restrictions ===
Line 247:		Line 248:
	=== Using the file system ===		=== Using the file system ===

	As KIT or HoreKa user you can use the file system in the same way as a normal workspace. You just have to specify the name of the flash-based workspace file system using the option ''-F'' to all the commands that manage workspaces. On bwUniCluster 2.0 it is called ''ffuc'', on HoreKa it is ''ffhk''. For example, to create a workspace with name myws and a lifetime of 60 days on bwUniCluster 2.0 execute:		As KIT or HoreKa user you can use the file system in the same way as a normal workspace. You just have to specify the name of the flash-based workspace file system using the option ''-F'' to all the commands that manage workspaces. On bwUniCluster 3.0 it is called ''ffuc'', on HoreKa it is ''ffhk''. For example, to create a workspace with name myws and a lifetime of 60 days on bwUniCluster 3.0 execute:
	ws_allocate -F ffuc myws 60		ws_allocate -F ffuc myws 60

	If you want to use the full flash pfs on bwUniCluster 2.0 '''and''' HoreKa at the same time, please note that you only have to manage a particular workspace on one of the clusters since the name of the workspace directory is different. However, the path to each workspace is visible and can be used on both clusters.		If you want to use the full flash pfs on bwUniCluster 3.0 '''and''' HoreKa at the same time, please note that you only have to manage a particular workspace on one of the clusters since the name of the workspace directory is different. However, the path to each workspace is visible and can be used on both clusters.

	Other features are similar to normal workspaces. For example, we ~~are able to~~ restore expired workspaces ~~for~~ ~~few~~ ~~weeks~~ ~~and~~ ~~you~~ ~~have to open a ticket to request the restore~~. There are quota limits with a default limit of 1 TiB capacity and 5 millions inodes per user. You can check your current usage with		Other features are similar to normal workspaces. For example, you can restore expired workspaces during 30 days after workspace expiration. There are quota limits with a default limit of 1 TiB capacity and 5 millions inodes per user. You can check your current usage with
	lfs quota -uh $(whoami) /pfs/work8		lfs quota -uh $(whoami) /pfs/work8

R Laifer: /* BeeOND (BeeGFS On-Demand) */

2025-04-03T15:02:14Z

BeeOND (BeeGFS On-Demand)

@@ Line 347: / Line 347: @@
 ==BeeOND (BeeGFS On-Demand)==
-Users of the UniCluster have possibility to request a private BeeOND (on-demand BeeGFS) parallel filesystem for each job. The file system is created during job startup and purged after your job.
+Users have the possibility to request a private BeeOND (on-demand BeeGFS) parallel filesystem for each job. The file system is created during job startup and purged when your job completes.
-BeeOND/BeeGFS can be used like any other parallel file system. Tools like cp or rsync can be used to copy data in and out.
+BeeOND/BeeGFS can be used like any other parallel file system. All nodes of the batch job have acess to the same data below the same path. Tools like cp or rsync can be used to copy data in and out.
 ==Backup and Archiving==

R Laifer: /* LSDF Online Storage */

2025-04-03T14:30:47Z

LSDF Online Storage

← Older revision		Revision as of 16:30, 3 April 2025
Line 325:		Line 325:
	== LSDF Online Storage ==		== LSDF Online Storage ==

	The LSDF Online Storage allows dedicated users to store scientific measurement data and simulation results. BwUniCluster 3.0 has an extremely fast network connection to the LSDF Online Storage. This file system provides external access via different protocols. It is only available for certain users. For information how request storage projects on the LSDF Online Storage see [[https://www.scc.kit.edu/en/services/lsdf~~\|here~~]].		The LSDF Online Storage allows dedicated users to store scientific measurement data and simulation results. BwUniCluster 3.0 has an extremely fast network connection to the LSDF Online Storage. This file system provides external access via different protocols. It is only available for certain users. For information how request storage projects on the LSDF Online Storage see [[https://www.scc.kit.edu/en/services/lsdf]].

	The LSDF Online Storage is mounted on the login nodes. It will also be mounted on the compute nodes of your batch job request it with the constraint flag ''LSDF''. You have one of the following options:		The LSDF Online Storage is mounted on the login nodes. It will also be mounted on the compute nodes of your batch job request it with the constraint flag ''LSDF''. You have one of the following options:

R Laifer: /* LSDF Online Storage */

2025-04-03T14:29:48Z

LSDF Online Storage

← Older revision		Revision as of 16:29, 3 April 2025
Line 323:		Line 323:
	</syntaxhighlight>		</syntaxhighlight>

	== LSDF Online Storage==		== LSDF Online Storage ==

			The LSDF Online Storage allows dedicated users to store scientific measurement data and simulation results. BwUniCluster 3.0 has an extremely fast network connection to the LSDF Online Storage. This file system provides external access via different protocols. It is only available for certain users. For information how request storage projects on the LSDF Online Storage see [[https://www.scc.kit.edu/en/services/lsdf\|here]].
	In some cases it is useful to have access to the LSDF Online Storage on the HPC-Clusters also. Therefore the LSDF Online Storage is mounted on the Login- and Datamover-Nodes.

	Furthermore it can be used on the compute nodes during the job runtime with the constraint flag "LSDF" ([[bwUniCluster_2.0_Slurm_common_Features\|Slurm common features]]
			The LSDF Online Storage is mounted on the login nodes. It will also be mounted on the compute nodes of your batch job request it with the constraint flag ''LSDF''. You have one of the following options:
	). There is also an example about the LSDF batch usage: [[bwUniCluster_2.0_Slurm_common_Features#LSDF_Online_Storage\|Slurm LSDF example ]] .

			1. Add after the initial lines of your script job.sh the line <code>#SBATCH --constraint=LSDF</code>:
	<pre>		<pre>
	#!/bin/bash		#!/bin/bash
	#SBATCH ~~...~~		#SBATCH --ntasks=1
			#SBATCH --time=120
			#SBATCH --mem=200
	#SBATCH --constraint=LSDF		#SBATCH --constraint=LSDF
	</pre>		</pre>
			2. Add the constraint on command line:
	<br>
			<pre>
⚫	~~For~~ ~~the~~ ~~usage~~ of the LSDF Online Storage the following environment variables are available: ~~$LSDF, $LSDFPROJECTS, $LSDFHOME.~~
			$ sbatch -p <queue> -n 1 -t 2:00:00 --mem 200 job.sh -C LSDF
	Please request storage projects in the LSDF Online Storage seperately:
			</pre>
	[https://www.lsdf.kit.edu/os/storagerequest/: LSDF Storage Request].

		⚫	In order to access the LSDF Online Storage the following environment variables are available:
			<code>$LSDF</code>, <code>$LSDFPROJECTS</code>, <code>$LSDFHOME</code>

	==BeeOND (BeeGFS On-Demand)==		==BeeOND (BeeGFS On-Demand)==

← Older revision		Revision as of 17:39, 3 April 2025
Line 63:		Line 63:
	\|-		\|-
	! scope="column" style="height=20px; padding:3px"\| Read perf./node		! scope="column" style="height=20px; padding:3px"\| Read perf./node
	\| style="height=20px; text-align:left;padding:3px"\| 500 MB/s - 6 GB/s <br> depends on type of local SSD / job queue~~: <br> 520 MB/s @ single / multiple <br> 800 MB/s @ multiple_e <br> 6600 MB/s @ fat <br> 6500 MB/s @ gpu_4 <br> 6500 MB/s @ gpu_8~~		\| style="height=20px; text-align:left;padding:3px"\| 500 MB/s - 6 GB/s <br> depends on type of local SSD / job queue
	\| style="height=20px; text-align:left;padding:3px"\| 400 MB/s - 500 MB/s<br> depends on type of local SSDs / job queue~~: <br> 500 MB/s @ multiple <br> 400 MB/s @ multiple_e~~		\| style="height=20px; text-align:left;padding:3px"\| 400 MB/s - 500 MB/s<br> depends on type of local SSDs / job queue
	\| style="height=20px; text-align:left;padding:3px"\| 5 GB/s		\| style="height=20px; text-align:left;padding:3px"\| 5 GB/s
	\| style="height=20px; text-align:left;padding:3px"\| 5 GB/s		\| style="height=20px; text-align:left;padding:3px"\| 5 GB/s
Line 70:		Line 70:
	\|-		\|-
	! scope="column" style="height=20px; padding:3px"\| Write perf./node		! scope="column" style="height=20px; padding:3px"\| Write perf./node
	\| style="height=20px; text-align:left;padding:3px"\| 500 MB/s - 4 GB/s <br> depends on type of local SSD / job queue~~: <br> 500 MB/s @ single / multiple <br> 650 MB/s @ multiple_e <br> 2900 MB/s @ fat <br> 2090 MB/s @ gpu_4 <br> 4060 MB/s @ gpu_8~~		\| style="height=20px; text-align:left;padding:3px"\| 500 MB/s - 4 GB/s <br> depends on type of local SSD / job queue
	\| style="height=20px; text-align:left;padding:3px"\| 250 MB/s - 350 MB/s <br> depends on type of local SSDs / job queue~~: <br> 350 MB/s @ multiple <br> 250 MB/s @ multiple_e~~		\| style="height=20px; text-align:left;padding:3px"\| 250 MB/s - 350 MB/s <br> depends on type of local SSDs / job queue
	\| style="height=20px; text-align:left;padding:3px"\| 5 GB/s		\| style="height=20px; text-align:left;padding:3px"\| 5 GB/s
	\| style="height=20px; text-align:left;padding:3px"\| 5 GB/s		\| style="height=20px; text-align:left;padding:3px"\| 5 GB/s

← Older revision		Revision as of 17:15, 3 April 2025
Line 2:		Line 2:


	On bwUniCluster 2.0 the parallel file system Lustre is used for most globally visible user data. Lustre is open source and Lustre solutions and support are available from different vendors. Nowadays, most of the biggest HPC systems are using Lustre. An initial home directory on a Lustre file system is created automatically after account activation, and the environment variable $HOME holds its name. Users can create so-called workspaces on another Lustre file system for non-permanent data with temporary lifetime. There is another workspace file system based on flash storage for special requirements available.		On bwUniCluster 3.0 the parallel file system Lustre is used for most globally visible user data. Lustre is open source and Lustre solutions and support are available from different vendors. Nowadays, most of the biggest HPC systems are using Lustre. An initial home directory on a Lustre file system is created automatically after account activation, and the environment variable $HOME holds its name. Users can create so-called workspaces on another Lustre file system for non-permanent data with temporary lifetime. There is another workspace file system based on flash storage for special requirements available.

	Within a batch job further file systems are available:		Within a batch job further file systems are available:
	* The directory $TMPDIR is only available and visible on the local node. It is located on fast SSD storage devices.		* The directory $TMPDIR is only available and visible on the local node. It is located on fast SSD storage devices.
	* On request a parallel on-demand file system (BeeOND) is created which uses the SSDs of the nodes which were allocated to the batch job.		* On request a parallel on-demand file system (BeeOND) is created which uses the SSDs of the nodes which were allocated to the batch job.
	* On request the external LSDF Online Storage is mounted on the nodes which were allocated to the batch job. This file system is based on the parallel file system ~~Spectrum~~ Scale.		* On request the external LSDF Online Storage is mounted on the nodes which were allocated to the batch job. This file system is based on the parallel file system IBM Storage Scale.

	Some of the characteristics of the file systems are shown in ~~Table~~ 2.		Some of the characteristics of the file systems are shown in the following table.

	{\| class="wikitable"		{\| class="wikitable"
Line 106:		Line 106:
	or exceeds the capacity restrictions should be sent to the LSDF Online Storage		or exceeds the capacity restrictions should be sent to the LSDF Online Storage
	or to the archive and deleted from the file systems. Temporary data which is only needed on a single		or to the archive and deleted from the file systems. Temporary data which is only needed on a single
	node and which does not exceed the disk space shown in ~~the~~ ~~table~~ ~~above should be stored~~		node and which does not exceed the disk space shown in [[BwUniCluster3.0/Hardware_and_Architecture#Compute_nodes\|Table 1]]
	below $TMPDIR. Data which is read many times on a single node, e.g. if you are doing AI training,		should be stored below $TMPDIR. Data which is read many times on a single node, e.g. if you are doing AI training,
	should be copied to $TMPDIR and read from there. Temporary data which is used from many nodes		should be copied to $TMPDIR and read from there. Temporary data which is used from many nodes
	of your batch job and which is only needed during job runtime should be stored on a		of your batch job and which is only needed during job runtime should be stored on a