bwHPC Wiki - Recent changes [en]

BwUniCluster3.0/Containers

P Schuhmacher — Thu, 06 Aug 2026 13:09:26 GMT

Start via Slurm

← Older revision		Revision as of 15:09, 6 August 2026
Line 77:		Line 77:
	=== Start via Slurm ===		=== Start via Slurm ===
	Start existing Container:		Start existing Container:
	*<code>salloc -p dev_single -t 00:10:00 --container-name=ubuntu --container-mounts=/etc/slurm/task_prolog:/etc/slurm/task_prolog,/scratch:/scratch~~,/usr/lib64/slurm:/usr/lib64/slurm,/usr/lib64/libhwloc.so:/usr/lib64/libhwloc.so,/usr/lib64/libhwloc.so.15:/usr/lib64/libhwloc.so.15~~</code>		*<code>salloc -p dev_single -t 00:10:00 --container-name=ubuntu --container-mounts=/etc/slurm/task_prolog:/etc/slurm/task_prolog,/scratch:/scratch</code>

	Download and start Container via pyxis directly:		Download and start Container via pyxis directly:
	*<code>salloc -p dev_single -t 00:10:00 --container-image=ubuntu --container-name=ubuntu --container-mounts=/etc/slurm/task_prolog:/etc/slurm/task_prolog,/scratch:/scratch~~,/usr/lib64/slurm:/usr/lib64/slurm,/usr/lib64/libhwloc.so:/usr/lib64/libhwloc.so,/usr/lib64/libhwloc.so.15:/usr/lib64/libhwloc.so.15~~</code>		*<code>salloc -p dev_single -t 00:10:00 --container-image=ubuntu --container-name=ubuntu --container-mounts=/etc/slurm/task_prolog:/etc/slurm/task_prolog,/scratch:/scratch</code>
	In this case an enroot Container is created under ~./local/share/enroot/		In this case an enroot Container is created under ~./local/share/enroot/

	Note: <code>--container-mounts=/etc/slurm/task_prolog:/etc/slurm/task_prolog,/scratch:/scratch~~,/usr/lib64/slurm:/usr/lib64/slurm,/usr/lib64/libhwloc.so:/usr/lib64/libhwloc.so,/usr/lib64/libhwloc.so.15:/usr/lib64/libhwloc.so.15~~</code> is needed for the plugin to work!! The Container name has to start with pyxis_ for the Plugin to work. When using the second Method this is done automatically. Furthermore when specifying the container name in your slurm Job the pyxis_ has to be omitted.		Note: <code>--container-mounts=/etc/slurm/task_prolog:/etc/slurm/task_prolog,/scratch:/scratch</code> is needed for the plugin to work!! The Container name has to start with pyxis_ for the Plugin to work. When using the second Method this is done automatically. Furthermore when specifying the container name in your slurm Job the pyxis_ has to be omitted.

	All options usable for pyxis can be found via srun --help under "Options provided by plugins:"		All options usable for pyxis can be found via srun --help under "Options provided by plugins:"

File:Uc1 file cut 750.jpg

S Braun — Wed, 05 Aug 2026 12:41:15 GMT

S Braun deleted page File:Uc1 file cut 750.jpg

BwVideo test

S Braun — Wed, 05 Aug 2026 12:39:54 GMT

S Braun deleted page BwVideo test content before blanking was: " 1024px"

Start VNC Server - bwUniCluster

S Braun — Wed, 05 Aug 2026 12:38:13 GMT

S Braun deleted page Start VNC Server - bwUniCluster content before blanking was: "== Start a VNC Server on the bwUniCluster == To start a VNC Server on the bwUniCluster you must log into the cluster with your username and password as usual. You should not start the server on a login node but request an interactive job using for example the following command. $ msub -V -I -l nodes=1:ppn=1 -l walltime=0:02:00:00 On your assigned compute node start the VNC server with $ module load vis/tigervnc $ run_vncserver If you start a VNC server for th..."

2018-09-24: 5th bwHPC Symposium

S Braun — Wed, 05 Aug 2026 12:37:29 GMT

S Braun deleted page 2018-09-24: 5th bwHPC Symposium content before blanking was: "== Call for Participation in the 5th bwHPC Symposium == On behalf of the bwHPC initiative (www.bwhpc.de) we invite you to participate in the upcoming 5th bwHPC symposium on September 24/25, 2018 hosted at the University of Freiburg. The symposium provides the opportunity to present scientific projects, results or methods that have been carried out, achieved or developed with the support of bwHPC high performance computing resources. The symposium is free of cha..."

2017-10-04: 4th bwHPC Symposium

S Braun — Wed, 05 Aug 2026 12:37:14 GMT

S Braun deleted page 2017-10-04: 4th bwHPC Symposium content before blanking was: "Dear Sir or Madam, on behalf of the bwHPC-C5 project ([http://www.bwhpc-c5.de www.bwhpc-c5.de]) we invite you to participate in the upcoming 4th bwHPC symposium on '''October 4, 2017''' hosted at Eberhard Karls Universität Tübingen. The symposium focuses on the presentation of scientific computing projects as well as on the progress and the success stories of the bwHPC realization concept. The event offers an unique opportunity to engage in an active dialogu..."

BwUniCluster2.0/Software/R/terra (R 4.2.1)

S Braun — Wed, 05 Aug 2026 12:36:52 GMT

S Braun deleted page BwUniCluster2.0/Software/R/terra (R 4.2.1) content was: " Note that the instructions provided below refer to R 4.2.1 (but not R 4.3.3)! We are currently updating our guides for R 4.3.3. = General information = sf and terra allow to use the following tools for handling spatial structures in R * the 'Geospatial' Data Abstraction Library [https://gdal.org/ GDAL] * Projection/transformation opera...", and the only contributor was "K Kuck" (talk)

BwUniCluster2.0/Software/R/stringi

S Braun — Wed, 05 Aug 2026 12:36:43 GMT

S Braun deleted page BwUniCluster2.0/Software/R/stringi content was: "Note that the instructions provided below refer to R 4.4.1 (but not R 4.2.1)! = General information = <code>stringi</code> is an <code>R</code> package for fast, portable, correct, consistent, and convenient string/text processing in any locale or character encoding. = Installation = Please enter the code, presented in the boxes below, di...", and the only contributor was "K Kuck" (talk)

BwUniCluster2.0/Software/R/cummeRbund

S Braun — Wed, 05 Aug 2026 12:36:36 GMT

S Braun deleted page BwUniCluster2.0/Software/R/cummeRbund content was: "= General information = [https://compbio.mit.edu/cummeRbund/ '''<code>CummeRbund</code>'''] is an <code>R</code> package designed to aid and simplify the task of analyzing Cufflinks RNA-Seq output. = Installation instructions = == Preparations == Installing <code>CummeRbund</code> involves compiling source code. Therefore, ensure that the following flags are set in <code>$HOME/...", and the only contributor was "K Kuck" (talk)

BwUniCluster2.0/Software/R/Rjags

S Braun — Wed, 05 Aug 2026 12:36:28 GMT

S Braun deleted page BwUniCluster2.0/Software/R/Rjags content was: "Note that the instructions provided below refer to R 4.2.1 (but not R 4.4.1)! We are currently updating our guides for R 4.4.1. = General information = rjags is a R interface to use JAGS, [https://mcmc-jags.sourceforge.io/ Just another Gibbs Sampler]. JAGS is a program for analysis of Bayesian hierarchical models using Markov Chain Monte Carlo (MCMC) simulation rjags needs a JAGS installation on the side. We recommend to compile via Inte..."

BwUniCluster2.0/Software/R/Rgdal (R 4.2.1)

S Braun — Wed, 05 Aug 2026 12:36:20 GMT

S Braun deleted page BwUniCluster2.0/Software/R/Rgdal (R 4.2.1) content was: "Note that the instructions provided below refer to R 4.2.1 (but not R 4.4.1)! = General information = Note that rgdal and rgeos have been retired by their developers in 2023! Please find further information at [https://cran.r-project.org/web/packages/rgdal/index.html https://cran.r-project.org/web/packages/rgd...", and the only contributor was "K Kuck" (talk)

VNC Server - bwUniCluster 2.0

S Braun — Wed, 05 Aug 2026 12:33:11 GMT

S Braun deleted page VNC Server - bwUniCluster 2.0 content was: "== Start a VNC Server on the bwUniCluster 2.0 == To start a VNC Server on the bwUniCluster 2.0 you must log into the cluster with your username and password as usual. Please do not start the VNC Server on a login node but request an interactive job using for example the following command: $ salloc -p single -n 1 -t 120 --...", and the only contributor was "F Wagner" (talk)

UC2/arch

S Braun — Wed, 05 Aug 2026 12:32:44 GMT

S Braun deleted page UC2/arch content was: "''{{SUBPAGENAME}}'' is a subpage of {{BASEPAGENAME}} to demonstrate the subpage feature", and the only contributor was "R Barthel" (talk)

BwUniCluster2.0/Software/R

S Braun — Wed, 05 Aug 2026 12:32:13 GMT

S Braun deleted page BwUniCluster2.0/Software/R content was: "{{Softwarepage|math/R}} {| width=600px class="wikitable" |- ! Description !! Content |- | module load | math/R |- | License | GPL |- | Citing | n/a |- | Links | [http://www.r-project.org/ Homepage] | [http://cran.r-project.org/manuals.html Documentation] |- | Graphical Interface | No |- | Plugins | User dependent |} = Description = '''R''' is a language and environment for statistical computing and graphics. It is a GNU project which is similar to the S language and e..."

Category:BwUniCluster 2.0

S Braun — Wed, 05 Aug 2026 12:31:51 GMT

S Braun deleted page Category:BwUniCluster 2.0 content was: "This page lists all articles that are assigned to the high-performance computer '''bwUniCluster 2.0'''. <!-- {| style="width: 100%; border-spacing: 5px;" | style="text-align:center; color:#000;vertical-align:middle;font-size:75%;" | center|border|550px|Close-up of bwUniCluster by Simon Raffeiner, Copyright: KIT (SCC) |- | style="text-align:center; color:#000;vertical-align:middle;" |Close-up..."

UC2

S Braun — Wed, 05 Aug 2026 12:31:28 GMT

S Braun deleted page UC2 content was: "Main page testing subpage feature {{Special:PrefixIndex/UC2/}}"

UC2/access

S Braun — Wed, 05 Aug 2026 12:31:11 GMT

S Braun deleted page UC2/access content was: "''{{SUBPAGENAME}}'' is a subpage of {{BASEPAGENAME}} to demonstrate the subpage feature", and the only contributor was "R Barthel" (talk)

2017-01-24: bwUniCluster downtime

S Braun — Wed, 05 Aug 2026 12:30:47 GMT

S Braun deleted page 2017-01-24: bwUniCluster downtime content before blanking was: "To repair the damaged HOME file system of all non-KIT users we have scheduled a downtime for the three clusters for January 24, 2017. During the downtime there will be no access to the clusters and all jobs running at that time have to be cancelled. We will use the downtime to switch to the backup of the HOME file system and to upgrade the OS kernel. Update from 2017-01-24, 4 p.m.: The clusters bwUniCluster, ForHLR I and ForHLR II can now be used again. Please..."

BwUniCluster 2.0 Maintenance/2020-10/Software Issues

S Braun — Wed, 05 Aug 2026 12:30:13 GMT

S Braun deleted page BwUniCluster 2.0 Maintenance/2020-10/Software Issues content before blanking was: "After the last regular maintenance interval (from 06.10.2020 to 13.10.2020) the following issues with Intel MPI exist: * Intel MPI 2018 is incompatible with Red Hat 8.2. Any invocation, even a simple "Hello World" MPI program, will result in a crash. The ''mpi/impi/2018'' module has therefore been removed. * There is a bug in Intel MPI 2019.x which leads to crashes when multiple MPI applications which are linked against..."

BwUniCluster2.0/Jupyter (Deutsch)

S Braun — Wed, 05 Aug 2026 12:29:58 GMT

S Braun deleted page BwUniCluster2.0/Jupyter (Deutsch) content was: "{{Note|Foo}} Als Alternative zum Zugriff auf die HPC Ressourcen mittels SSH kann Jupyter verwendet werden. Hierfür ist lediglich ein Webbrowser nötig. Innerhalb der Website kann Quellcode verschiedener Programmiersprachen editiert und ausgeführt werden. Weiterhin stehen unterschiedliche Benutzerschnittstellen und Terminals zur Verfügung. <!-- * [https://uc2-jupyter.scc.kit...", and the only contributor was "S Braun" (talk)

BwUniCluster2.0/First Steps

S Braun — Wed, 05 Aug 2026 12:29:20 GMT

S Braun deleted page BwUniCluster2.0/First Steps content was: "<!-- == Command line interface == Any work is to be done via command line interface. A quick guide can be found [https://indico.scc.kit.edu/indico/event/278/material/slides/10.pdf '''here''']. == Software setup == Softwares provided by bwHPC have to be loaded according to your need via the '''Software Module System'''. == Computation == Any kind of computation on any HPC cluster is done as bwUniCluster_2.0_Slurm_common_Features|'''batch job..."

BwUniCluster2.0

S Braun — Wed, 05 Aug 2026 12:28:03 GMT

S Braun deleted page BwUniCluster2.0 content was: " <!--{| style="width: 100%; border-spacing: 5px;" | style="text-align:center; color:#000;vertical-align:middle;font-size:75%;" | center|border|550px|Close-up of bwUniCluster by Simon Raffeiner, Copyright: KIT (SCC) |- | style="text-align:center; color:#000;vertical-align:middle;" |Close-up of bwUniCluster © KIT (Simon Raffeiner/SCC) |} On 17.03.2020, the Sci..."

Helix/Software/Matlab

S Fischer — Mon, 03 Aug 2026 10:03:51 GMT

Compile MATLAB binaries with mcc

← Older revision		Revision as of 12:03, 3 August 2026
Line 157:		Line 157:
	== Compile MATLAB binaries with mcc ==		== Compile MATLAB binaries with mcc ==

	~~MATLAB~~ on ~~Helix~~ ~~comes~~ ~~with~~ a compiler, <code>mcc</code>~~, that can be used~~ to create binaries from MATLAB-code.		If you have a MATLAB license that includes [https://de.mathworks.com/products/compiler.html MATLAB Compiler], e.g. on your local computer, you can use <code>mcc</code> to create binaries from MATLAB code.
	Stand-alone MATLAB programs compiled with <code>mcc</code> do not require any license tokens at runtime and you can start jobs in parallel without any risk of running out of licences.		Stand-alone MATLAB programs compiled with <code>mcc</code> do not require any license tokens at runtime and you can start jobs in parallel without any risk of running out of licences.

BwForCluster JUSTUS 2 Slurm HOWTO

H Schumacher — Thu, 30 Jul 2026 17:06:49 GMT

added interactive job cancelling

← Older revision		Revision as of 19:06, 30 July 2026
Line 690:		Line 690:

	<pre>		<pre>
	$ scancel <jobid> # cancel specific job		$ scancel <jobid> # cancel specific job
	$ scancel <jobid>_<index> # cancel indexed job in a job array		$ scancel <jobid>_<index> # cancel indexed job in a job array
	$ scancel -u <username> # cancel all jobs of specific user		$ scancel -u <username> # cancel all jobs of specific user
	$ scancel -t ~~PENDING~~ # ~~cancel~~ ~~pending~~ ~~jobs~~		$ scancel -u <username> -i # cancel all jobs of specific user interactively
	$ scancel -t ~~RUNNING~~ # cancel ~~running~~ jobs		$ scancel -t PENDING # cancel pending jobs
			$ scancel -t RUNNING # cancel running jobs
	</pre>		</pre>

NEMO2/Modules

M Janczyk — Thu, 30 Jul 2026 14:04:00 GMT

Auto-update: EasyBuild module list (bot)

Show changes

User:S Raffeiner

M Janczyk — Tue, 28 Jul 2026 14:47:49 GMT

M Janczyk changed group membership for S Raffeiner from bureaucrat and administrator to (none) left

User:H Schumacher

M Janczyk — Tue, 28 Jul 2026 14:37:19 GMT

M Janczyk changed group membership for H Schumacher from (none) to administrator and bureaucrat Bürokrat Standort HD

Development/VS Code

H Schumacher — Mon, 27 Jul 2026 17:05:26 GMT

added kill command for VS Code Server

← Older revision		Revision as of 19:05, 27 July 2026
Line 48:		Line 48:
	[[File:vscode-remoteexplorer-indicator.png\|images/vscode-remoteexplorer-indicator.png\|200px]]<br>		[[File:vscode-remoteexplorer-indicator.png\|images/vscode-remoteexplorer-indicator.png\|200px]]<br>
	If you want to end your remote session, click the green box in the lower left corner. In the input box that opens, select the “Close Remote Connection” option. If you simply close your VS Code window, some server-side components of VS Code will continue to run remotely.		If you want to end your remote session, click the green box in the lower left corner. In the input box that opens, select the “Close Remote Connection” option. If you simply close your VS Code window, some server-side components of VS Code will continue to run remotely.

			If you encounter any issues with the remote connection or if you didn't end the sessions correctly, it can help to kill the VS Code Server on the host. This is done by going to the Command Palette (via the 'Manage' wheel in the lower left corner) and executing the following command:

			<code> Remote-SSH: Kill VS Code Server on Host </code>

	=== Access to Compute Nodes ===		=== Access to Compute Nodes ===
Line 100:		Line 104:
	As soon as code-server is running, it can be accessed in the web browser. In order to establish the connection, a SSH tunnel from your local computer to the remote server has to be created via:		As soon as code-server is running, it can be accessed in the web browser. In order to establish the connection, a SSH tunnel from your local computer to the remote server has to be created via:

	<syntaxhighlight lang="console">$ ssh -L 8081:<computeNodeID>:8081 <userID>@~~uc3.scc.kit.edu~~</syntaxhighlight>		<syntaxhighlight lang="console">$ ssh -L 8081:<computeNodeID>:8081 <userID>@<hostname></syntaxhighlight>
	You need to enter the <code>computeNodeID</code> of the node on which the interactive Slurm job is running. If you have started code server on a login node, just enter <code>localhost</code>. Now you can open http://127.0.0.1:8081 in your web browser. Possibly, you have to allow your browser to open an insecure (non-https) site. The login site looks as follows:		You need to enter the <code>computeNodeID</code> of the node on which the interactive Slurm job is running. If you have started code server on a login node, just enter <code>localhost</code>. Now you can open http://127.0.0.1:8081 in your web browser. Possibly, you have to allow your browser to open an insecure (non-https) site. The login site looks as follows:

SDS@hd/SV Management

H Schumacher — Fri, 24 Jul 2026 12:52:53 GMT

added hint regarding ORCiD users

← Older revision		Revision as of 14:52, 24 July 2026
Line 14:		Line 14:
	*# Provide them with the SV password. The SV responsible received the initial password via email. <u>Caution</u>: Make sure to share the SV password. Do not share your personal service password which you set in registration step B at bwServices.		*# Provide them with the SV password. The SV responsible received the initial password via email. <u>Caution</u>: Make sure to share the SV password. Do not share your personal service password which you set in registration step B at bwServices.
	*# Afterwards, they can join by following the [[SDS@hd/Registration \| registration steps]].		*# Afterwards, they can join by following the [[SDS@hd/Registration \| registration steps]].
	* ...change the status of SV members (active/inactive)		* ...change the status of SV members (active/inactive). Make sure to especially manage the access of those members that joined with their ORCiD account. Otherwise, they'll have access forever.

	=== SV Responsible (SVV) ===		=== SV Responsible (SVV) ===

BwUniCluster3.0

S Braun — Wed, 22 Jul 2026 15:23:38 GMT

← Older revision		Revision as of 17:23, 22 July 2026
Line 22:		Line 22:
	-->		-->

			<!--

	{\| style=" background:#ffa500; width:100%;"		{\| style=" background:#ffa500; width:100%;"
	\| style="padding:8px; background:#ff8c00; font-size:120%; font-weight:bold; text-align:left" \| RESOLVED: Immediate Shutdown of HPC Login Nodes: 08.07.2026		\| style="padding:8px; background:#ff8c00; font-size:120%; font-weight:bold; text-align:left" \| RESOLVED: Immediate Shutdown of HPC Login Nodes: 08.07.2026
Line 43:		Line 43:
	<p>We apologize for the inconvenience and thank you for your understanding.</p>		<p>We apologize for the inconvenience and thank you for your understanding.</p>
	\|}		\|}
			-->


	<!--		<!--

Status

M Carmesin — Wed, 22 Jul 2026 09:02:23 GMT

Current Status

← Older revision		Revision as of 11:02, 22 July 2026
Line 6:		Line 6:
	Due to work on the power supply of the cooling system all jobs have been suspended to prevent overheating.-->		Due to work on the power supply of the cooling system all jobs have been suspended to prevent overheating.-->

	{{Template:Status\|~~warn~~\|Status: ~~JUSTUS2 service distupted~~}}		{{Template:Status\|ok\|Status: Ok}}
	The JUSTUS2 service is disrupted for unclear reasons, investigations are ongoing.


	<!--		<!--
	Template usage examples: color ok(green),warn(yellow),alert(red). Enter custom text in 3rd field		Template usage examples: color ok(green),warn(yellow),alert(red). Enter custom text in 3rd field

	{{Template:Status\|ok\|Status: Ok}}
	{{Template:Status\|warn\|Status: maintenance upcoming}}		{{Template:Status\|warn\|Status: maintenance upcoming}}
	{{Template:Status\|alert\|Status: bwUniCluster down}}		{{Template:Status\|alert\|Status: bwUniCluster down}}

Main Page

M Carmesin — Wed, 22 Jul 2026 09:01:54 GMT

← Older revision		Revision as of 11:01, 22 July 2026
Line 9:		Line 9:

	<!-- bwHPC STATUS START -->		<!-- bwHPC STATUS START -->
		⚫	{{Template:Status\|ok\|Status: Ok}}

⚫	{{Template:Status\|~~warn~~\|Status: ~~JUSTUS2 disrupted~~ }}

	<!--		<!--
	Template usage examples: color ok(green),warn(yellow),alert(red). Enter custom text in 3rd field		Template usage examples: color ok(green),warn(yellow),alert(red). Enter custom text in 3rd field
	{{Template:Status\|ok\|Status: Ok}}		{{Template:Status\|warn\|Status: JUSTUS2 disrupted }}
	{{Template:Status\|warn\|Status: maintenance upcoming}}		{{Template:Status\|warn\|Status: maintenance upcoming}}
	{{Template:Status\|alert\|Status: bwUniCluster down}}		{{Template:Status\|alert\|Status: bwUniCluster down}}

BinAC2/Hardware and Architecture

S Behnle — Tue, 21 Jul 2026 12:15:49 GMT

Update workspace documentation link

← Older revision		Revision as of 14:15, 21 July 2026
Line 262:		Line 262:

	You can find more information about the workspace tools in the general documentation:		You can find more information about the workspace tools in the general documentation:
	:: → '''[[~~Workspaces~~ \| General workspace tools documentation]]'''		:: → '''[[Workspace \| General workspace tools documentation]]'''

	==== BinAC 2 workspace policies and limits ====		==== BinAC 2 workspace policies and limits ====

Data Transfer/Rclone

H Schumacher — Mon, 20 Jul 2026 16:21:13 GMT

Added Checksumming

@@ Line 164: / Line 164: @@
 Rclone has a lot of useful options.
 === Performance ===
@@ Line 185: / Line 200: @@
 Number of streams to use for multithreaded downloads (default: 4). Only important on very large files. This will cause multithreaded up/download on chunk-sized bits of the file.
 The optimal value is highly specific to the local network and used Hardware. For regular use cases, we recommend 4 as the default.
 === Debugging and Statistics ===

Helix/Slurm

H Schumacher — Mon, 20 Jul 2026 15:47:26 GMT

fixed formatting

Show changes

Data Transfer

H Schumacher — Mon, 20 Jul 2026 14:46:23 GMT

Added hint for checksumming, added rclone ssh key option for windows, changed formatting

← Older revision		Revision as of 16:46, 20 July 2026
Line 7:		Line 7:
	# Choose a [[#Network_Protocols_&_Transfer_Tools\|network protocol or transfer tool]] to use for the communication between the systems.		# Choose a [[#Network_Protocols_&_Transfer_Tools\|network protocol or transfer tool]] to use for the communication between the systems.

	The recommended setup already includes these three steps. For a full overview, you can reference the tables that show ~~[[Data_Transfer/All_Data_Transfer_Routes\|~~all transfer routes]]. Those include all possible combinations between systems, top level way of transfer and network protocol / transfer tool.		The recommended setup already includes these three steps. For a full overview, you can reference the tables that show all transfer routes. Those include all possible combinations between systems, top level way of transfer and network protocol / transfer tool.

			→ [[Data_Transfer/All_Data_Transfer_Routes\|All Data Transfer Routes]]

	=== Data Storage Systems ===		=== Data Storage Systems ===
Line 15:		Line 17:
	* [[File:Notebook.svg\|x20px]] Local computer or VM (virtual machine)		* [[File:Notebook.svg\|x20px]] Local computer or VM (virtual machine)
	* [[File:Microscope.svg\|x20px]] <span style="margin-left:10px;">Data producing machine (sequencer, microscope, ...)<span>		* [[File:Microscope.svg\|x20px]] <span style="margin-left:10px;">Data producing machine (sequencer, microscope, ...)<span>
	* [[File:Clusternodes.svg\|x20px]] <span style="margin-left:8px;">HPC system<span>		* [[File:Clusternodes.svg\|x20px]] <span style="margin-left:8px;">HPC system (bwHPC Cluster, ...)<span>
	* [[File:Storage_small.svg\|x15px]] <span style="margin-left:8px;">Storage space (SDS@hd, institute server, ...)<span>		* [[File:Storage_small.svg\|x15px]] <span style="margin-left:8px;">Storage space (SDS@hd, institute server, ...)<span>
	* [[File:Cloud.svg\|x15px]] <span style="margin-left:3px;">Cloud resource<span>		* [[File:Cloud.svg\|x15px]] <span style="margin-left:3px;">Cloud resource (Nextcloud, ...)<span>

	=== Ways of Transfer: Copy, Sync, Mount ===		=== Ways of Transfer: Copy, Sync, Mount ===
Line 23:		Line 25:
	The top level ways of transfer are:		The top level ways of transfer are:

	* '''Copy:''' A simple copy command is the most basic way to transfer data. This is most efficient for very big data files that shall be retrieved from or moved to a remote location. And it can be most convenient, if you prefer moving your files via commandline instead of using a file browser. <br>Examples: [[Data_Transfer/SCP\|scp]], [[Data_Transfer/SFTP\|sftp]]		* '''Copy:''' A simple copy command is the most basic way to transfer data. This is most efficient for very big data files that shall be retrieved from or moved to a remote location. And it can be most convenient, if you prefer moving your files via commandline instead of using a file browser. <br>Examples: [[Data_Transfer/SCP\|scp]], [[Data_Transfer/SFTP\|sftp]], [[Data_Transfer/Rclone\|rclone]]
	* '''Sync:''' If the data is intended to be kept on both systems and undergoes change on only one of the systems, it makes sense to use a synchronization command instead. This way, only the changed files in one location are updated in the other location. Good use cases are backups or data transfers that go mostly in one direction like moving data from a sequencer to a storage space. A disadvantage is that the data needs storage space on both systems. <br>~~Example~~: [[Data_Transfer/Rsync\|rsync]]		* '''Sync:''' If the data is intended to be kept on both systems and undergoes change on only one of the systems, it makes sense to use a synchronization command instead. This way, only the changed files in one location are updated in the other location. Good use cases are backups or data transfers that go mostly in one direction like moving data from a sequencer to a storage space. A disadvantage is that the data needs storage space on both systems. <br>Examples: [[Data_Transfer/Rsync\|rsync]], [[Data_Transfer/Rclone\|rclone]]
	* '''Mount:''' If the data undergoes change on both systems or is too big to store locally, then mounting is the most convenient solution. This allows you to see and work with the data as if it were stored locally on your computer while it is still placed on the remote system. All changes that you implement happen directly on the original data so that you don't need to copy or synchronize anything. Additionally, you'll see all changes that another party does to the data with just a very short delay. Disadvantages are that you need defined edit sessions starting with a mount and ending with a clean unmount of your files and with a stable network connection during the session. Also, file operations on the remote system become much slower via a remotely sshfs-mounted system. <br>~~Example~~: [[Data_Transfer/SSHFS\|sshfs]]		* '''Mount:''' If the data undergoes change on both systems or is too big to store locally, then mounting is the most convenient solution. This allows you to see and work with the data as if it were stored locally on your computer while it is still placed on the remote system. All changes that you implement happen directly on the original data so that you don't need to copy or synchronize anything. Additionally, you'll see all changes that another party does to the data with just a very short delay. Disadvantages are that you need defined edit sessions starting with a mount and ending with a clean unmount of your files and with a stable network connection during the session. Also, file operations on the remote system become much slower via a remotely sshfs-mounted system. <br>Examples: [[Data_Transfer/SSHFS\|sshfs]], [[SDS@hd/Access/SMB\|smb]], [[Data_Transfer/WebDAV\|webdav]], [[SDS@hd/Access/NFS\|nfs]], [[Data_Transfer/Rclone\|rclone]]
	<p style="text-align: left;">[[File:CopySyncMount.png\|x250px]]</p>		<p style="text-align: left;">[[File:CopySyncMount.png\|x250px]]</p>
	<p style="text-align: left; font-size: small; margin-top: 10px; margin-left: 255px;">Figure 1: Top level transfer routes</p>		<p style="text-align: left; font-size: small; margin-top: 10px; margin-left: 255px;">Figure 1: Top level transfer routes</p>
Line 56:		Line 58:
	== Recommended Setup ==		== Recommended Setup ==
	[[File:Bwhpc diagram simplenobox.jpg\|thumb\|Main routes for mounting\|x150px]]		[[File:Bwhpc diagram simplenobox.jpg\|thumb\|Main routes for mounting\|x150px]]
	When you are working with a development environment that allows remote connections, this is the first choice. ~~Otherwise, the main tools/protocols for transferring data are as follows:~~		# When you are working with a <ins>development environment</ins> that allows remote connections (e.g. [[Development/VS_Code \| VS Code]]), this is the first choice.
			# Otherwise, the main tools/protocols for transferring data are as follows:

		⚫	#* <u>Windows</u>:
	* ~~<u>Windows</u>:~~ '''[[Data_Transfer/Graphical_Clients#MobaXterm\|MobaXterm]]''' is a graphical user interface that allows logging in to the cluster with ssh as well as transferring data via a file browser or using command line tools for the transfer.		#** '''[[Data_Transfer/Graphical_Clients#MobaXterm\|MobaXterm]]''' is a graphical user interface that allows logging in to the cluster with ssh as well as transferring data via a file browser or using command line tools for the transfer.
⚫	* <u>~~MacOS and Linux~~</u>:
		⚫	#** '''[[Data_Transfer/Rclone\|Rclone]]''' can be used to mount the data as a drive in your file explorer so that you can access the files programmatically as well. An [[Registration/SSH \| ssh key]] is needed when used with 2FA.
⚫	** '''[[Data_Transfer/SSHFS\|sshfs]]''' is quickly set up for mounting under stable connections.
			#* <u>MacOS and Linux</u>:
⚫	** '''[[Data_Transfer/Rclone\|Rclone]]''' ~~for~~ mount, ~~copy~~ ~~and~~ ~~sync~~. An [[Registration/SSH \| ssh key]] is needed when used with 2FA.
		⚫	#** '''[[Data_Transfer/SSHFS\|sshfs]]''' is quickly set up for mounting under stable connections.
⚫	* <u>SDS@hd</u>: See the [[SDS@hd/Access\|SDS@hd Access]] page.
			#** '''[[Data_Transfer/Rclone\|Rclone]]''' for mount, copy and sync. An [[Registration/SSH \| ssh key]] is needed when used with 2FA.
		⚫	#* <u>SDS@hd</u>: See the [[SDS@hd/Access\|SDS@hd Access]] page.

	== Best Practices ==		== Best Practices ==

			Here, you can find best practices in the case of...
⚫	* ~~'''Strong firewall restrictions'''<br /> ->~~ Use ssh or http(s) based protocols, for example [[Data_Transfer/WebDAV\|'''WebDav''']] and [[Data_Transfer/SFTP\|sftp]]. For very strict facilities, ssh based protocols might not be allowed.
			* '''...strong firewall restrictions:'''
⚫	* '''~~Share~~ data with collaborators...'''
		⚫	*: → Use ssh or http(s) based protocols, for example [[Data_Transfer/WebDAV\|'''WebDav''']] and [[Data_Transfer/SFTP\|sftp]]. For very strict facilities, ssh based protocols might not be allowed.
⚫	** ~~...outside of Baden-Württemberg<br /> -~~> Use the [[SDS@hd\|SDS@hd]] storage.
		⚫	* '''...when sharing data with collaborators...'''
⚫	** ...that are less comfortable with the command line~~<br /> -> Let them mount the folder.~~
			** ...outside of Baden-Württemberg
⚫	* '''~~Transfer~~ many small files'''<br /> -> Compress the files to one.
		⚫	**: → Use the [[SDS@hd\|SDS@hd]] storage.
		⚫	** ...that are less comfortable with the command line
			**: → Let them mount the folder.
		⚫	* '''...transferring many small files:'''<br /> → Compress the files to one.
			* '''...ensuring correct data transfers:'''
			*: → Use checksumming. For example with [[Data_Transfer/Rclone \| Rclone]].

	For advanced topics see [[Data_Transfer/Advanced_Data_Transfer\|Advanced Data Transfer]].		For advanced topics see [[Data_Transfer/Advanced_Data_Transfer\|Advanced Data Transfer]].

Registration/SSH

H Schumacher — Mon, 20 Jul 2026 14:34:51 GMT

corrected validity value for Helix

← Older revision		Revision as of 16:34, 20 July 2026
Line 20:		Line 20:
	\| bwForCluster Helix		\| bwForCluster Helix
	\| style="background-color:#90EE90;" \| bwServices Portal		\| style="background-color:#90EE90;" \| bwServices Portal
	\| Centralized key management, ~~180~~-day validity		\| Centralized key management, 90-day validity
	\|-		\|-
	\| bwForCluster JUSTUS 2		\| bwForCluster JUSTUS 2

SDS@hd/Security

H Schumacher — Mon, 20 Jul 2026 13:41:06 GMT

Added hints and explanations

← Older revision		Revision as of 15:41, 20 July 2026
Line 1:		Line 1:
			__FORCETOC__
	= Snapshots =		== Snapshots ==

	Snapshots are used to create a point-in-time copy of a file system or file sets.		Snapshots are used to create a point-in-time copy of a file system or file sets.
⚫	Snapshots can be used to quickly restore files or folders that were unintentionally deleted. Snapshots also provide a form of file versioning and the ability to revert to a previous version of a file, depending on the time interval between snapshots.

		⚫	<ins>Motivation</ins>: Snapshots can be used to quickly restore files or folders that were unintentionally deleted. Snapshots also provide a form of file versioning and the ability to revert to a previous version of a file, depending on the time interval between snapshots. The shorter ago a change was made and/or the longer it was kept the likelier it is that it can still be found in one of the snapshots.
	'''Snapshots:'''

			When looking into the <code>.snapshots</code> folder within your SV you can see how often the snapshots are taken and how long they are preserved. For example:
			<syntaxhighlight lang="bash">
		⚫	ls .snapshots
		⚫	> @GMT-2026.08.10-19.07.13 @GMT-2026.09.09-14.15.00 @GMT-2026.09.14-20.15.00 @GMT-2026.09.16-16.15.00 @GMT-2026.09.19-08.15.00
		⚫	@GMT-2026.08.11-01.07.13 @GMT-2026.09.09-22.15.00 @GMT-2026.09.14-20.45.00 @GMT-2026.09.16-16.45.00 @GMT-2026.09.19-09.00.00
		⚫	@GMT-2026.08.11-07.07.13 @GMT-2026.09.10-06.15.00 @GMT-2026.09.14-21.15.00 @GMT-2026.09.16-17.15.00 @GMT-2026.09.19-03.15.00
		⚫	@GMT-2026.09.19-03.45.00 @GMT-2026.09.19-05.45.00 @GMT-2026.09.19-07.45.00
			</syntaxhighlight>

			== Hints ==
	Inside your share (Speichervorhaben), you can find the Snapshots in the subdirectory ".snapshots":
	<pre>
⚫	~~>cd~~ .snapshots
	>ls
⚫	@GMT-~~2017~~.08.10-19.07.13 @GMT-~~2017~~.09.09-14.15.00 @GMT-~~2017~~.09.14-20.15.00 @GMT-~~2017~~.09.16-16.15.00 @GMT-~~2017~~.09.19-08.15.00
⚫	@GMT-~~2017~~.08.11-01.07.13 @GMT-~~2017~~.09.09-22.15.00 @GMT-~~2017~~.09.14-20.45.00 @GMT-~~2017~~.09.16-16.45.00 @GMT-~~2017~~.09.19-09.00.00
⚫	@GMT-~~2017~~.08.11-07.07.13 @GMT-~~2017~~.09.10-06.15.00 @GMT-~~2017~~.09.14-21.15.00 @GMT-~~2017~~.09.16-17.15.00 @GMT-~~2017~~.09.19-03.15.00
⚫	@GMT-~~2017~~.09.19-03.45.00 @GMT-~~2017~~.09.19-05.45.00 @GMT-~~2017~~.09.19-07.45.00
	</pre>

			* It's not allowed to use SDS@hd as a backup for your data. The data on SDS@hd must be actively used. If the data is not actively used, you might find out about an accidental deletion of data far too late so that there isn't a snapshot anymore that contains the deleted data.
	<br>
			* Please avoid keeping duplicate data in your SV. Even if the duplicate data is stored in the SV for a short time period only, it can still make it into a snapshot and use up storage space on SDS@hd for quite a long time.
	<br>
			* When uploading data, please make sure that all the data actually made it into the SV. The snapshots can't capture data that has never reached the SV. Please see [[Data_Transfer \| Data Transfer]] for help with ensuring that data transfers were finished successfully.

BwUniCluster3.0

S Braun — Mon, 20 Jul 2026 07:59:21 GMT

← Older revision		Revision as of 09:59, 20 July 2026
Line 23:		Line 23:


	{\| style=" background:#~~f08080~~; width:100%;"		{\| style=" background:#ffa500; width:100%;"
	\| style="padding:8px; background:#~~ff0000~~; font-size:120%; font-weight:bold; text-align:left" \| Immediate Shutdown of HPC Login Nodes: 08.07.2026		\| style="padding:8px; background:#ff8c00; font-size:120%; font-weight:bold; text-align:left" \| RESOLVED: Immediate Shutdown of HPC Login Nodes: 08.07.2026
	\|-		\|-
	\|		\|

BwUniCluster3.0/Running Jobs

S Braun — Mon, 20 Jul 2026 07:53:06 GMT

Queues on bwUniCluster 3.0

← Older revision		Revision as of 09:53, 20 July 2026
(One intermediate revision by the same user not shown)
Line 129:		Line 129:
	\| gres=gpu:1		\| gres=gpu:1
	\| time=30, nodes=12, mem=376000mb, ntasks-per-node=48, (threads-per-core=2)		\| time=30, nodes=12, mem=376000mb, ntasks-per-node=48, (threads-per-core=2)
			\|-
			\| <code>gpu_h100_short</code>
			\| GPU nodes<br/>NVIDIA GPU x4
			\| mem-per-gpu=193300mb<br/>cpus-per-gpu=24
			\| gres=gpu:1
			\| time=72:00:00, nodes=7, mem=760000mb, ntasks-per-node=96, (threads-per-core=2)
	\|}		\|}
	Table 2: Short Queues		Table 2: Short Queues

BwUniCluster3.0/Hardware and Architecture

S Braun — Mon, 20 Jul 2026 07:47:38 GMT

Compute nodes

← Older revision		Revision as of 09:47, 20 July 2026
Line 69:		Line 69:
	\| <code>cpu</code>, <code>dev_cpu</code>		\| <code>cpu</code>, <code>dev_cpu</code>
	\| <code>highmem</code>, <code>dev_highmem</code>		\| <code>highmem</code>, <code>dev_highmem</code>
	\| <code>gpu_h100</code>, <code>dev_gpu_h100</code>		\| <code>gpu_h100</code>, <code>dev_gpu_h100</code>, <code>gpu_h100_short</code>
	\| <code>gpu_mi300</code>		\| <code>gpu_mi300</code>
	\| <code>gpu_a100_il</code> / <code>gpu_h100_il</code>		\| <code>gpu_a100_il</code> / <code>gpu_h100_il</code>
Line 79:		Line 79:
	\| 80		\| 80
	\| 5		\| 5
	\| 12		\| 19
	\| 1		\| 1
	\| 15		\| 15

Main Page

M Carmesin — Mon, 20 Jul 2026 07:46:57 GMT

← Older revision		Revision as of 09:46, 20 July 2026
Line 10:		Line 10:
	<!-- bwHPC STATUS START -->		<!-- bwHPC STATUS START -->

	{{Template:Status\|ok\|Status: Ok}}		{{Template:Status\|warn\|Status: JUSTUS2 disrupted }}

	<!--		<!--

Status

M Carmesin — Mon, 20 Jul 2026 07:46:07 GMT

← Older revision		Revision as of 09:46, 20 July 2026
(One intermediate revision by the same user not shown)
Line 6:		Line 6:
	Due to work on the power supply of the cooling system all jobs have been suspended to prevent overheating.-->		Due to work on the power supply of the cooling system all jobs have been suspended to prevent overheating.-->

	{{Template:Status\|ok\|Status: Ok}}		{{Template:Status\|warn\|Status: JUSTUS2 service distupted}}
			The JUSTUS2 service is disrupted for unclear reasons, investigations are ongoing.


	<!--		<!--
Line 13:		Line 15:
	{{Template:Status\|warn\|Status: maintenance upcoming}}		{{Template:Status\|warn\|Status: maintenance upcoming}}
	{{Template:Status\|alert\|Status: bwUniCluster down}}		{{Template:Status\|alert\|Status: bwUniCluster down}}
			{{Template:Status\|ok\|Status: Ok}}
	-->		-->

BinAC2/Software/Nextflow

F Bartusch — Fri, 17 Jul 2026 09:34:08 GMT

← Older revision		Revision as of 11:34, 17 July 2026
(One intermediate revision by the same user not shown)
Line 135:		Line 135:
	As mentioned, the pipeline runs in a screen session.		As mentioned, the pipeline runs in a screen session.
	You can detach from the screen session, and the pipeline will continue to run.		You can detach from the screen session, and the pipeline will continue to run.
	The keyboard shortcut for detaching is <code>CTRL</code> + <code>c</code>, followed by <code>d</code>.		The keyboard shortcut for detaching is <code>CTRL</code> + <code>a</code>, followed by <code>d</code>.
	This means you press the <code>CTRL</code> and <code>c</code> keys simultaneously, then release them and press <code>d</code>.		This means you press the <code>CTRL</code> and <code>a</code> keys simultaneously, then release them and press <code>d</code>.
	You should now be detached from the screen session and back in your login terminal.		You should now be detached from the screen session and back in your login terminal.
	<!--TODO add screenshot-->		<!--TODO add screenshot-->

NEMO2

M Janczyk — Thu, 16 Jul 2026 08:21:38 GMT

← Older revision		Revision as of 10:21, 16 July 2026
Line 15:		Line 15:

	<!--		<!--
	-->
	{\| style=" background:#ffa833; width:100%;"		{\| style=" background:#ffa833; width:100%;"
	\| style="padding:8px; background:#f80; font-size:120%; font-weight:bold; text-align:left" \| Copying Data from NEMO2 While SSH Login is Disabled		\| style="padding:8px; background:#f80; font-size:120%; font-weight:bold; text-align:left" \| Copying Data from NEMO2 While SSH Login is Disabled
Line 22:		Line 21:
	[[NEMO2/SSH-Login-Disabled-Data-Transfer\|Create command SSH keys for rsync]]		[[NEMO2/SSH-Login-Disabled-Data-Transfer\|Create command SSH keys for rsync]]
	\|}		\|}
	<!--
	* '''Users of NEMO1 must''' [[Registration/bwForCluster/NEMO2\|'''re-register for NEMO2''']] (only step C necessary). You keep your existing projects (Rechenvorhaben).		* '''Users of NEMO1 must''' [[Registration/bwForCluster/NEMO2\|'''re-register for NEMO2''']] (only step C necessary). You keep your existing projects (Rechenvorhaben).
	* '''[[NEMO2/Migrate NEMO1 workspaces to NEMO2 workspaces\|Migrate NEMO1 workspaces to NEMO2 workspaces]]'''		* '''[[NEMO2/Migrate NEMO1 workspaces to NEMO2 workspaces\|Migrate NEMO1 workspaces to NEMO2 workspaces]]'''

User:M Janczyk

R Barthel — Wed, 15 Jul 2026 19:37:06 GMT

R Barthel changed group membership for M Janczyk from administrator to administrator and bureaucrat Hinzufügen Berechtigung durch R_Barthel