BinAC2/Hardware and Architecture - Revision history

F Bartusch: Change link to new workspace documentation

2026-02-11T08:57:04Z

Change link to new workspace documentation

← Older revision		Revision as of 10:57, 11 February 2026
Line 262:		Line 262:

	You can find more information about the workspace tools in the general documentation:		You can find more information about the workspace tools in the general documentation:
	:: → '''[[~~Workspace~~ \| General workspace tools documentation]]'''		:: → '''[[Workspaces \| General workspace tools documentation]]'''

	==== BinAC 2 workspace policies and limits ====		==== BinAC 2 workspace policies and limits ====

F Bartusch at 10:45, 21 January 2026

2026-01-21T10:45:31Z

← Older revision		Revision as of 12:45, 21 January 2026
Line 348:		Line 348:
	* Copy the data - which will create new files -, then delete the old files. Example:		* Copy the data - which will create new files -, then delete the old files. Example:
	<pre>		<pre>
	$> cp -ar /pfs/10/work/tu_abcde01-my-precious-ws/* /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/.		$> cp -r /pfs/10/work/tu_abcde01-my-precious-ws/* /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/.
	$> rm -rf /pfs/10/work/tu_abcde01-my-precious-ws/*		$> rm -rf /pfs/10/work/tu_abcde01-my-precious-ws/*
	$> ws_release --delete-data my-precious-ws		$> ws_release --delete-data my-precious-ws

F Bartusch at 10:55, 19 January 2026

2026-01-19T10:55:46Z

@@ Line 246: / Line 246: @@
 Data on the fast storage pool at <code>/pfs/10/work</code> is stored on SSDs.
-The primary focus is speed, not capacity.
+The primary focus of this filesystem is speed, not capacity.<br />
-:: &rarr; '''[[Workspace]]s'''
+:: &rarr; '''[[Workspace | General workspace tools documentation]]'''

S Behnle: More on Lustre (3)

2025-12-19T18:09:21Z

S Behnle: More on Lustre (2)

2025-12-19T17:57:00Z

S Behnle at 16:23, 19 December 2025

2025-12-19T16:23:28Z

← Older revision		Revision as of 18:23, 19 December 2025
Line 229:		Line 229:
	</syntaxhighlight>		</syntaxhighlight>

	You can check the project you~~'re~~ member of:		You can check the project(s) you are member of via:

	<syntaxhighlight>		<syntaxhighlight>
Line 237:		Line 237:

	In this case, your project directory would be:		In this case, your project directory would be:
			<code>
	```
	/pfs/10/project/bw16f003/		/pfs/10/project/bw16f003/
			</code>
	```

	Check our [[BinAC2/Project_Data_Organization \| data organization guide ]] for methods to organize data inside the project directory.		Check our [[BinAC2/Project_Data_Organization \| data organization guide ]] for methods to organize data inside the project directory.

S Behnle at 15:06, 19 December 2025

2025-12-19T15:06:33Z

← Older revision		Revision as of 17:06, 19 December 2025
Line 352:		Line 352:
	* Copy the data - which will create new files, then delete the old files. Example:		* Copy the data - which will create new files, then delete the old files. Example:
	<pre>		<pre>
	$> cp -ar /pfs/10/work/tu_abcde01-my-precious-ws/* pfs/10/project/bw10a001/tu_abcde01/my-precious-research/.		$> cp -ar /pfs/10/work/tu_abcde01-my-precious-ws/* /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/.
	$> rm -rf /pfs/10/work/tu_abcde01-my-precious-ws/*		$> rm -rf /pfs/10/work/tu_abcde01-my-precious-ws/*
	$> ws_release --delete-data my-precious-ws		$> ws_release --delete-data my-precious-ws
Line 358:		Line 358:
	* Alternative to copy: use <code>rsync</code> to copy data between the workspace and the project directories. Example:		* Alternative to copy: use <code>rsync</code> to copy data between the workspace and the project directories. Example:
	<pre>		<pre>
	$> rsync -av /pfs/10/work/tu_abcde01-my-precious-ws/simulation/output pfs/10/project/bw10a001/tu_abcde01/my-precious-research/simulation25/		$> rsync -av /pfs/10/work/tu_abcde01-my-precious-ws/simulation/output /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/simulation25/
	$> rm -rf /pfs/10/work/tu_abcde01-my-precious-ws/*		$> rm -rf /pfs/10/work/tu_abcde01-my-precious-ws/*
	$> ws_release --delete-data my-precious-ws		$> ws_release --delete-data my-precious-ws
	</pre>		</pre>
			* If there are many subfolders with similar size, you can use <code>xargs</code> to copy them in parallel:
			<pre>
			$> find . -maxdepth 1 -mindepth 1 -type d -print \| xargs -P4 -I{} rsync -aHAXW --inplace --update {} /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/simulation25/
			</pre>
			will launch four parallel rsync processes at a time, each will copy one of the subdirectories.
	* Move the metadata with mv, then use lfs migrate or lfs_migrate to actually migrate the stripes		* Move the metadata with mv, then use lfs migrate or lfs_migrate to actually migrate the stripes

S Behnle at 14:16, 19 December 2025

2025-12-19T14:16:01Z

← Older revision		Revision as of 16:16, 19 December 2025
Line 353:		Line 353:
	<pre>		<pre>
	$> cp -ar /pfs/10/work/tu_abcde01-my-precious-ws/* pfs/10/project/bw10a001/tu_abcde01/my-precious-research/.		$> cp -ar /pfs/10/work/tu_abcde01-my-precious-ws/* pfs/10/project/bw10a001/tu_abcde01/my-precious-research/.
			$> rm -rf /pfs/10/work/tu_abcde01-my-precious-ws/*
			$> ws_release --delete-data my-precious-ws
			</pre>
			* Alternative to copy: use <code>rsync</code> to copy data between the workspace and the project directories. Example:
			<pre>
			$> rsync -av /pfs/10/work/tu_abcde01-my-precious-ws/simulation/output pfs/10/project/bw10a001/tu_abcde01/my-precious-research/simulation25/
	$> rm -rf /pfs/10/work/tu_abcde01-my-precious-ws/*		$> rm -rf /pfs/10/work/tu_abcde01-my-precious-ws/*
	$> ws_release --delete-data my-precious-ws		$> ws_release --delete-data my-precious-ws

S Behnle: More on Lustre

2025-12-18T18:45:55Z

More on Lustre

← Older revision		Revision as of 20:45, 18 December 2025
Line 93:		Line 93:
	In contrast to BinAC 1 not all compute nodes are connected via Infiniband, but there are 84 standard compute nodes connected via HDR Infiniband (100 GbE). In order to get your jobs onto the Infiniband nodes, submit your job with <code>--constraint=ib</code>.		In contrast to BinAC 1 not all compute nodes are connected via Infiniband, but there are 84 standard compute nodes connected via HDR Infiniband (100 GbE). In order to get your jobs onto the Infiniband nodes, submit your job with <code>--constraint=ib</code>.

	'''Question:'''		'''Question:'''</br>
	OpenMPI throws the following warning:		OpenMPI throws the following warning:
	<pre>		<pre>
Line 110:		Line 110:
	What should i do?		What should i do?

	'''Answer:'''		'''Answer:'''</br>
	BinAC2 has two (almost) separate networks, a 100GbE network and and InfiniBand network, both connecting a subset of the nodes. Both networks require different cables and switches.		BinAC2 has two (almost) separate networks, a 100GbE network and and InfiniBand network, both connecting a subset of the nodes. Both networks require different cables and switches.
	Concerning the network cards for the nodes, however, there exist VPI network cards which can be configured to work in either mode (https://docs.nvidia.com/networking/display/connectx6vpi/specifications#src-2487215234_Specifications-MCX653105A-ECATSpecifications).		Concerning the network cards for the nodes, however, there exist VPI network cards which can be configured to work in either mode (https://docs.nvidia.com/networking/display/connectx6vpi/specifications#src-2487215234_Specifications-MCX653105A-ECATSpecifications).
Line 117:		Line 117:
	On nodes with 100Gb ethernet, this fails as there is no RDMA protocol configured. OpenMPI will fall back to TCP transport but not without complaints.		On nodes with 100Gb ethernet, this fails as there is no RDMA protocol configured. OpenMPI will fall back to TCP transport but not without complaints.

	'''Workaround:'''		'''Workaround:'''</br>
	For single-node jobs or on regular compute nodes, A30 and A100 GPU nodes: Add the lines		For single-node jobs or on regular compute nodes, A30 and A100 GPU nodes: Add the lines
	<pre>		<pre>
Line 347:		Line 347:

	==== Moving data between WORK and PROJECT ====		==== Moving data between WORK and PROJECT ====
	<b>!! IMPORTANT !!</b> Calling <code>mv</code> on files will <i>not</i> physically move them. Instead, the file metadata, i.e. the path to the file in the directory tree will be modified (i.e. data stored on the MDS). The stripes of the file on the OSS, however, will remain exactly were they were. The only result will be the confusing situation that you now have metadata entries under <code>/pfs/10/project</code> that still point to WORK OSTs.		<b>!! IMPORTANT !!</b> Calling <code>mv</code> on files will <i>not</i> physically move them. Instead, the file metadata, i.e. the path to the file in the directory tree will be modified (i.e. data stored on the MDS). The stripes of the file on the OSS, however, will remain exactly were they were. The only result will be the confusing situation that you now have metadata entries under <code>/pfs/10/project</code> that still point to WORK OSTs. This may sound confusing at first. When using <code>mv</code> on the same file system, Lustre only renames the files and makes them available from a different path. The pointers to the file objects on the OSS stay identical. This will only change if you either create a copy of the file at a different path (with <code>cp</code> or <code>rsync</code>, e.g.) or if you explicitly instruct Lustre to move the actual file objects to another storage location, e.g. another pool of the same file system.

	<b>Proper ways of moving data between the pools</b>		<b>Proper ways of moving data between the pools</b></br>
	* Copy the data - which will create new files, then delete the old files		* Copy the data - which will create new files, then delete the old files. Example:
			<pre>
			$> cp -ar /pfs/10/work/tu_abcde01-my-precious-ws/* pfs/10/project/bw10a001/tu_abcde01/my-precious-research/.
			$> rm -rf /pfs/10/work/tu_abcde01-my-precious-ws/*
			$> ws_release --delete-data my-precious-ws
			</pre>
	* Move the metadata with mv, then use lfs migrate or lfs_migrate to actually migrate the stripes		* Move the metadata with mv, then use lfs migrate or lfs_migrate to actually migrate the stripes

			<b>Why pathes and storage pools should match:</b></br>

			There are four different possible scenarios with two subdirectories and two pools:
			* File path in <code>/pfs/10/work</code>, file objects on pool <code>work</code>: <b>good.</b>
			* File path in <code>/pfs/10/project</code>, file objects on pool <code>project</code>: <b>good.</b>
			* File path in <code>/pfs/10/project</code>, file objects on pool <code>work</code>: <b>bad</b>. This will "leak" storage from the fast pool, making it unavailable for workspaces.
			* File path in <code>/pfs/10/work</code>, file objects on pool <code>project</code>: <b>bad</b>. Access will be slow, and if (volatile) workspaces are purged, data residing on <code>project</code> will (voluntarily or involuntarily) be deleted.
			The latter two situations may arise from <code>mv</code>ing data between workspaces and project folders.

	More reading:		More reading:

S Behnle: Added a BinAC2 schematic

2025-12-18T17:02:08Z

Added a BinAC2 schematic

← Older revision		Revision as of 19:02, 18 December 2025
Line 2:		Line 2:

	The bwForCluster BinAC 2 supports researchers from the broader fields of Bioinformatics, Medical Informatics, Astrophysics, Geosciences and Pharmacy.		The bwForCluster BinAC 2 supports researchers from the broader fields of Bioinformatics, Medical Informatics, Astrophysics, Geosciences and Pharmacy.

			[[File:Binac2 schema.png\|600px\|thumb\|center\|Overview on the BinAC 2 hardware architecture.]]

	== Operating System and Software ==		== Operating System and Software ==

← Older revision		Revision as of 20:09, 19 December 2025
Line 369:		Line 369:

	All of the above procedures may take a considerable amount of time depending on the amount of data, so it might be advisable to execute them in a terminal multiplexer like <code>screen</code> or <code>tmux</code> or wrap them into small SLURM jobs with <code>sbatch --wrap="<command>"</code>.		All of the above procedures may take a considerable amount of time depending on the amount of data, so it might be advisable to execute them in a terminal multiplexer like <code>screen</code> or <code>tmux</code> or wrap them into small SLURM jobs with <code>sbatch --wrap="<command>"</code>.

			<b>Question</b>:</br> I totally lost overview, how do i find out where my files are located?

			<b>Answer</b>:</br>
			* Use <code>lfs find</code> to find files on a specific pool. Example:
			<pre>
			$> lfs find . --pool project # recursively find all files in the current directory whose file objects are on the "project" pool
			</pre>
			* Use <code>lfs getstripe</code> to query the striping pattern and the pool (also works recursively if called with a directory). Example:
			<pre>
			$> lfs getstripe parameter.h
			parameter.h
			lmm_stripe_count: 1
			lmm_stripe_size: 1048576
			lmm_pattern: raid0
			lmm_layout_gen: 1
			lmm_stripe_offset: 44
			lmm_pool: project
			obdidx objid objid group
			44 7991938 0x79f282 0xd80000400
			</pre>
			shows that the file is striped over OST 44 (obdidx) which belongs to pool project (lmm_pool).


	<b>Why pathes and storage pools should match:</b></br>		<b>Why pathes and storage pools should match:</b></br>