bwHPC Wiki - Recent changes [en]

SDS@hd/Registration

H Schumacher — Thu, 23 Apr 2026 16:18:50 GMT

fixed link, reformatting, added link from step A to B

← Older revision		Revision as of 18:18, 23 April 2026
(One intermediate revision by the same user not shown)
Line 8:		Line 8:
	* '''Step B:''' →[[SDS@hd/Registration#Step_B:_Registration_for_SDS@hd_Service \|Registration for SDS@hd Service]]		* '''Step B:''' →[[SDS@hd/Registration#Step_B:_Registration_for_SDS@hd_Service \|Registration for SDS@hd Service]]

	After finishing the registration, the next steps are:		After finishing the registration, the <span id="sds_registration_nextSteps">'''next steps'''</span> are:
	* Check which management options there are for your respective role in the SV → [[SDS@hd/SV_Management \|SV Management]]		* Check which management options there are for your respective role in the SV → [[SDS@hd/SV_Management \|SV Management]]
	* See how to access and use your storage space → [[SDS@hd/Access \| Access]]		* See how to access and use your storage space → [[SDS@hd/Access \| Access]]
Line 21:		Line 21:
	==== a) Join existing SV ====		==== a) Join existing SV ====
	To join an existing SV as a coworker:		To join an existing SV as a coworker:
	# ~~Your advisor (the SV responsible) will provide you with~~ the following ~~data~~ on the SV:		# Get the following information from the SV responsible:
	#:* acronym		#:* SV acronym
	#:* password		#:* SV password
	# ~~Login~~ at [https://sds-hd.urz.uni-heidelberg.de/management/~~index~~.php~~?mode=mitarbeit~~ SDS@hd Managementtool]		# Fill in and submit the [https://sds-hd.urz.uni-heidelberg.de/management/shib/sds_collaboration.php SV collaboration form] after logging in to the SDS@hd Managementtool.
	#: ~~Provide~~ the SV ~~acronym~~ and SV ~~password.~~		#: You are now a member of the SV. The SV owner (and SV managers) will be notified automatically.
			→ [[SDS@hd/Registration#Step_B:_Registration_for_SDS@hd_Service \| Go on with Step B]] if you are using SDS@hd for the first time.
	# After submitting the request, you will be assigned to the SV as a member and receive an email with the next steps. The SV owner (and SV managers) will be notified automatically.

	==== b) Apply for new SV ====		==== b) Apply for new SV ====
Line 38:		Line 38:
	# '''Apply for a new SV:'''		# '''Apply for a new SV:'''
	#: → Fill in the form at the [https://sds-hd.urz.uni-heidelberg.de/management SDS@hd Managementtool].		#: → Fill in the form at the [https://sds-hd.urz.uni-heidelberg.de/management SDS@hd Managementtool].
			→ [[SDS@hd/Registration#Step_B:_Registration_for_SDS@hd_Service \| Go on with Step B]] if you are using SDS@hd for the first time.

	If you register your own SV, you will be:		If you register your own SV, you will be:
	* ...responsibe for providing new members with the necessary information to join the SV		* ...responsibe for providing new members with the necessary information to join the SV
	*: → see [[#~~SV_Responsible:_Manage_the_SV_and_its_Members~~ \| SV Responsible: ~~Manage the SV and its Members~~]]		*: → see [[SDS@hd/SV_Management#SV_Responsible_(SVV) \| SV Responsible (SVV)]]
	* ...held accountable for the co-workers in the SV		* ...held accountable for the co-workers in the SV
	* ...asked to provide information for the two reports required by the DFG for their funding of SDS@hd		* ...asked to provide information for the two reports required by the DFG for their funding of SDS@hd
	* ...likely asked for a contribution to a future DFG grant proposal for an extension of the storage system in your area of research ("wissenschaftliches Beiblatt")		* ...likely asked for a contribution to a future DFG grant proposal for an extension of the storage system in your area of research ("wissenschaftliches Beiblatt")


	=== Step B: Registration for SDS@hd Service ===		=== Step B: Registration for SDS@hd Service ===
	[[File:bwServices_chooseHomeOrganisation.png\|right\|300px\|thumb\|Select your home organization]]		[[File:bwServices_chooseHomeOrganisation.png\|right\|300px\|thumb\|Select your home organization]]

	After step 1 you have to register your personal account on the storage system and set a service password.		After step A you have to register your personal account on the storage system and set a service password.

	Please visit:
	* [https://bwservices.uni-heidelberg.de/ https://bwservices.uni-heidelberg.de]		* Visit the registration website '''bwServices''': [https://bwservices.uni-heidelberg.de/ https://bwservices.uni-heidelberg.de]
	*# Select your home organization from the list and click '''Proceed'''		*# Select your home organization from the list and click ''Proceed''
	*# You will be directed to the ''Identity Provider'' of your home organisation		*#: You will be directed to the ''Identity Provider'' of your home organisation
	*# Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button		*# Enter your home-organisational user ID / username and your home-organisational password and click ''Login'' button
	*# You will be redirected back to ~~the registration website [https://bwservices.uni-heidelberg.de/ https://bwservices.uni-heidelberg.de/]~~		*#: You will be redirected back to bwServices
			* '''Register''' for the SDS@hd service:
	*# ~~<div>Select~~ ~~unter '~~''The following services are available''' the service '''SDS@hd - Scientific Data Storage'''		*# Under ''The following services are available'' select the service ''SDS@hd - Scientific Data Storage''.
	*# Click '''Register'''		*# Click ''Register''
	*# Finally, set a service password for authentication on SDS@hd
			* Set your personal '''service password''' for SDS@hd. This is used for accessing SVs.
			* The registration is finished.
			*: → [[#sds_registration_nextSteps\|next steps]]

	=== Change Service Password at bwServices ===		=== Change Service Password at bwServices ===

Registration/SSH

H Schumacher — Thu, 23 Apr 2026 09:18:17 GMT

Added 'generate ssh key' step, included 'bind key' as last step, removed spacing in info box, downsized two images

← Older revision		Revision as of 11:18, 23 April 2026
Line 149:		Line 149:
	== Step 1: Add Your SSH Key to the Portal ==		== Step 1: Add Your SSH Key to the Portal ==

		⚫	<div style="border: 3px solid #ffc107; background-color: #fff3cd; margin: 10px 0px 10px 0px;">
	First, upload your public key to the management portal:
			<p style="padding-left: 5px; margin: 0px;">'''Important:'''</p>

		⚫	* Keys are valid for '''180 days''' and automatically revoked after expiration.
⚫	<div style="border: 3px solid #ffc107~~; padding: 15px~~; background-color: #fff3cd; margin: 10px 0;">
		⚫	* Upload only your '''public key''' file (ending in <code>.pub</code>, e.g., <code>~/.ssh/id_ed25519.pub</code>).
	'''Important:'''
⚫	* Keys are valid for '''180 days''' and automatically revoked after expiration
⚫	* Upload only your '''public key''' file (ending in <code>.pub</code>, e.g., <code>~/.ssh/id_ed25519.pub</code>)
	</div>		</div>

	'''1. ~~Navigate~~ to SSH key ~~management:~~'''		'''1. Generate an SSH key''' (if you don't have one):
			* Run in your local terminal: <pre> ssh-keygen -t ed25519 -C "your_email@example.com"
			</pre>
			* Press Enter to accept the default location, then set a strong passphrase.
			* You now have a private key (~/.ssh/id_ed25519) and a public key (~/.ssh/id_ed25519.pub).

			'''2. Navigate to SSH key management:'''
	* [https://login.bwidm.de/user/ssh-keys.xhtml bwUniCluster 3.0 / NEMO 2] (bwIDM)		* [https://login.bwidm.de/user/ssh-keys.xhtml bwUniCluster 3.0 / NEMO 2] (bwIDM)
	* [https://bwservices.uni-heidelberg.de/user/ssh-keys.xhtml bwForCluster Helix] (bwServices)		* [https://bwservices.uni-heidelberg.de/user/ssh-keys.xhtml bwForCluster Helix] (bwServices)
Line 163:		Line 167:
	[[File:BwIDM-twofa.png\|center\|600px\|thumb\|My SSH Pubkeys page]]		[[File:BwIDM-twofa.png\|center\|600px\|thumb\|My SSH Pubkeys page]]

	'''2. Click "Add SSH Key" / "SSH Key Hochladen"'''		'''3. Click "Add SSH Key" / "SSH Key Hochladen"'''

	[[File:Bwunicluster 2.0 access ssh keys empty.png\|center\|400px\|thumb\|Add SSH Key button]]		[[File:Bwunicluster 2.0 access ssh keys empty.png\|center\|400px\|thumb\|Add SSH Key button]]

	'''3. Enter key details:'''		'''4. Enter key details:'''
	* '''Name:''' Descriptive identifier (e.g., "laptop-work", "desktop-home")		* '''Name:''' Descriptive identifier (e.g., "laptop-work", "desktop-home")
	* '''SSH Key:''' Paste complete contents of your <code>.pub</code> file		* '''SSH Key:''' Paste complete contents of your <code>.pub</code> file
	* Click '''Add''' / '''Hinzufügen'''		* Click '''Add''' / '''Hinzufügen'''

	[[File:Ssh-key.png\|center\|~~600px~~\|thumb\|Add SSH key dialog]]		[[File:Ssh-key.png\|center\|400px\|thumb\|Add SSH key dialog]]

		⚫	'''5. Your key appears in the list:'''

		⚫	[[File:Ssh-success.png\|center\|600px\|thumb\|SSH key successfully added]]
⚫	'''4. ~~Success!'''~~ Your key appears in the list

		⚫	'''6. Bind your key to a cluster''' as either an [[#Step_2A:_Register_Interactive_Key\|Interactive Key]] or [[#Step_2B:_Register_Command_Key\|Command Key]].
⚫	[[File:Ssh-success.png\|center\|~~800px~~\|thumb\|SSH key successfully added]]
⚫	'''~~Next:'''~~ Bind your key to a cluster as either an [[#Step_2A:_Register_Interactive_Key\|Interactive Key]] or [[#Step_2B:_Register_Command_Key\|Command Key]].

	== Step 2A: Register Interactive Key ==		== Step 2A: Register Interactive Key ==

NEMO2/SDS hd

M Janczyk — Wed, 22 Apr 2026 16:49:14 GMT

Per-user configuration file

Show changes

Energy Efficient Cluster Usage

A Flachmueller — Wed, 22 Apr 2026 10:35:53 GMT

What do I want to do and why do I need an HPC Cluster for it?

← Older revision		Revision as of 12:35, 22 April 2026
(One intermediate revision by the same user not shown)
Line 6:		Line 6:
	Assuming that a typical bwHPC cluster has a few hundred compute nodes, this amounts to the energy consumption of a village for each cluster.		Assuming that a typical bwHPC cluster has a few hundred compute nodes, this amounts to the energy consumption of a village for each cluster.

	Although a large amount of this energy consumption is an intrinsic requirement of running large HPC clusters (even when ~~ist~~ processors are idle, a cluster uses a lot of energy), efficient use of the available resources is important. Using as many resources as possible does not make a power user. Using them wisely does.		Although a large amount of this energy consumption is an intrinsic requirement of running large HPC clusters (even when it's processors are idle, a cluster uses a lot of energy), efficient use of the available resources is important. Using as many resources as possible does not make a power user. Using them wisely does.
	In the following, a basic introduction to some of the most important aspects of energy-efficient HPC usage from a user perspective is given.		In the following, a basic introduction to some of the most important aspects of energy-efficient HPC usage from a user perspective is given.

Line 24:		Line 24:
	Please consider testing new setups and their output for validity prior to submitting jobs that require lots of resources. This also includes projects where a lot of (smaller) similar jobs are submitted.		Please consider testing new setups and their output for validity prior to submitting jobs that require lots of resources. This also includes projects where a lot of (smaller) similar jobs are submitted.

	Make sure to double-check your jobs prior to the submission, having to discard the output data of an HPC project due to faulty input files is wasting a lot of computational resources.		Make sure to double-check your jobs prior to the submission, as having to discard the output data of an HPC project due to faulty input files is wasting a lot of computational resources.

⚫	Finally, identifying the specific resource requirements for a given job is important to allocate the optimal your compute job, and to decide if an HPC cluster is needed at all.

		⚫	Finally, identifying the specific resource requirements for a given job is important to allocate the optimal amount for your compute job, and to decide if an HPC cluster is needed at all.

	= How many and which kind of hardware resources do I require for it =		= How many and which kind of hardware resources do I require for it =

NEMO2/SDS hd

M Janczyk — Tue, 21 Apr 2026 14:54:24 GMT

← Older revision		Revision as of 16:54, 21 April 2026
(2 intermediate revisions by the same user not shown)
Line 1:		Line 1:
			<div style="border: 3px solid #dc3545; padding: 15px; background-color: #f8d7da; margin: 10px 0;">
			'''WARNING! This feature is currently for testing purposes only!'''
			</div>

	'''SDS@hd''' is the scientific storage service of Heidelberg University. On NEMO2, you can mount your SDS@hd project directly into your jobs or interactive sessions via rclone. Data is cached locally on the node's NVMe drive and uploaded to SDS@hd in the background.		'''SDS@hd''' is the scientific storage service of Heidelberg University. On NEMO2, you can mount your SDS@hd project directly into your jobs or interactive sessions via rclone. Data is cached locally on the node's NVMe drive and uploaded to SDS@hd in the background.

Line 278:		Line 282:
	# Create a workspace and configure it as the cache directory:		# Create a workspace and configure it as the cache directory:
	ws_allocate mywsname 30		ws_allocate mywsname 30

	~~cat~~ >> ~/.config/sdshd/config ~~<<'EOF'~~		# Edit ~/.config/sdshd/config, add:
	cache_dir = /path/from/ws_find		cache_dir = /path/from/ws_find
	cache_size = 2T		cache_size = 2T
	cache_age = 720h		cache_age = 720h
	EOF
	</pre>		</pre>

NEMO2/SDS hd

M Janczyk — Tue, 21 Apr 2026 13:54:22 GMT

Created page with "'''SDS@hd''' is the scientific storage service of Heidelberg University. On NEMO2, you can mount your SDS@hd project directly into your jobs or interactive sessions via rclone. Data is cached locally on the node's NVMe drive and uploaded to SDS@hd in the background. <div style="border: 3px solid #28a745; padding: 15px; background-color: #d4edda; margin: 10px 0;"> '''What you need to do:''' * '''Initial Setup''' — one-time configurati..."

New page

'''SDS@hd''' is the scientific storage service of Heidelberg University. On NEMO2, you can mount your SDS@hd project directly into your jobs or interactive sessions via rclone. Data is cached locally on the node's NVMe drive and uploaded to SDS@hd in the background.

<div style="border: 3px solid #28a745; padding: 15px; background-color: #d4edda; margin: 10px 0;">
'''What you need to do:'''

* '''[[#Initial_Setup_.28Required.29|Initial Setup]]''' — one-time configuration, takes about 5 minutes. '''You must complete this before using SDS@hd on NEMO2.'''
* '''[[#Advanced_Configuration|Advanced Configuration]]''' — optional. Tune cache size, bandwidth limits, and more. Skip this on your first use.
</div>

= Initial Setup (Required) =

Complete these four steps once. After that, SDS@hd is available in all your jobs and interactive sessions without any further configuration.

'''Prerequisites:'''
* Your bwIDM username and SDS@hd password
* Your SDS@hd project ID (find it at [https://wiki.bwhpc.de/e/SDS@hd/Access wiki.bwhpc.de/e/SDS@hd/Access])

== Step 1: Create the rclone configuration file ==

Run this command to create the config file with the correct permissions:

<pre>
rclone config touch
</pre>

This creates <tt>~/.config/rclone/rclone.conf</tt> with mode 600. Now open the file in an editor and add the following two sections, replacing the placeholder values with your own:

<pre>
[sdshd-backend]
type = sftp
host = lsdf02-sshfs.urz.uni-heidelberg.de
user = YOUR_BWIDM_USERNAME
pass = PASTE_OBSCURED_PASSWORD_HERE
md5sum_command = none
sha1sum_command = none
shell_type = unix
set_modtime = false
idle_timeout = 0

[sdshd-sftp]
type = alias
remote = sdshd-backend:YOUR_PROJECT_ID
</pre>

Replace:
* <tt>YOUR_BWIDM_USERNAME</tt> — your bwIDM login name
* <tt>YOUR_PROJECT_ID</tt> — your SDS@hd project ID, e.g. <tt>sd14a001</tt>
* <tt>PASTE_OBSCURED_PASSWORD_HERE</tt> — leave this for now, you will fill it in Step 2

'''The <tt>[sdshd-sftp]</tt> section name must not be changed''' — all NEMO2 scripts mount <tt>sdshd-sftp:</tt> by name.

<div style="border: 3px solid #17a2b8; padding: 15px; background-color: #d1ecf1; margin: 10px 0;">
'''What the path in <tt>[sdshd-sftp]</tt> controls:'''

The path after the colon in <tt>remote = sdshd-backend:YOUR_PROJECT_ID</tt> becomes the root of your mount at <tt>/mnt/sdshd/$USER</tt>. If you set it correctly, your project files appear directly under <tt>/mnt/sdshd/$USER/</tt>.

If you leave the path empty (<tt>remote = sdshd-backend:</tt>), the mount shows the SFTP server's home directory. Your project then appears as a ''subdirectory'' at <tt>/mnt/sdshd/$USER/YOUR_PROJECT_ID/</tt>. Job scripts that reference <tt>/mnt/sdshd/$USER/</tt> directly will not find your data.

'''Exception:''' If you are a member of multiple projects, you may intentionally omit the project ID so all projects appear as subdirectories. In that case, adjust your job scripts to include the project ID in the path.
</div>

== Step 2: Set the password ==

rclone does not store passwords in plain text — it stores an obscured form. Generate the obscured password with the following command. It reads your password without echoing it to the terminal and is never written to your shell history:

<pre>
printf 'SDS@hd password:\n'; read -s p && printf '%s' "$p" | rclone obscure -; echo
</pre>

Type your SDS@hd password and press Enter. The command prints a string like:

<pre>
QKkf-abc123XYZetc
</pre>

Copy that string and paste it as the <tt>pass =</tt> value in <tt>~/.config/rclone/rclone.conf</tt>, replacing <tt>PASTE_OBSCURED_PASSWORD_HERE</tt>. Then clear the variable from memory:

<pre>
unset p # bash / zsh
# set -e p # fish
</pre>

<div style="border: 3px solid #dc3545; padding: 15px; background-color: #f8d7da; margin: 10px 0;">
'''Do not use <tt>echo "mypassword" | rclone obscure -</tt>''' — the password would appear in your shell history and in <tt>ps</tt> output. The <tt>read -s</tt> method above avoids both.
</div>

== Step 3: Protect the config file ==

<tt>rclone config touch</tt> already creates the file with mode 600. If you created it manually or are unsure, set the permissions explicitly:

<pre>
chmod 600 ~/.config/rclone/rclone.conf
</pre>

== Step 4: Verify the connection ==

<pre>
rclone ls sdshd-sftp:
</pre>

Expected output: a list of files and directories in your project. An empty listing (no output, no error) is also fine — it means the project exists but is empty.

{| class="wikitable"
|-
! Error message
! Likely cause
|-
| <tt>SSH authentication failed</tt>
| Wrong username or password
|-
| <tt>no such file or directory</tt>
| Project ID in <tt>[sdshd-sftp]</tt> is wrong
|-
| <tt>connection refused</tt> / timeout
| Network issue or wrong hostname
|-
| Empty listing (no error)
| Project exists but is empty — that is fine
|}

'''Setup complete.''' See [[#Basic_Usage|Basic Usage]] to start using SDS@hd.

= Basic Usage =

== Interactive sessions on login nodes ==

<pre>
mount-sdshd # mount at /mnt/sdshd/$USER
umount-sdshd # unmount and wait for all uploads to finish
</pre>

The mount point <tt>/mnt/sdshd/$USER</tt> is created automatically when you log in, via a PAM hook. No manual admin step is needed.

<div style="border: 3px solid #dc3545; padding: 15px; background-color: #f8d7da; margin: 10px 0;">
'''Always use <tt>umount-sdshd</tt> to disconnect — never <tt>fusermount -u</tt> directly.'''

<tt>fusermount -u</tt> terminates rclone immediately, cutting off any uploads in progress and causing data loss. <tt>umount-sdshd</tt> waits until rclone confirms that all uploads are complete before sending the shutdown signal.
</div>

Run <tt>mount-sdshd --help</tt> for all options and examples.

== Slurm jobs ==

Add the <tt>--gres=sdshd</tt> flag to your job script. The Slurm prolog mounts SDS@hd automatically at job start; the epilog waits for all uploads to complete before the job finishes:

<pre>
#SBATCH --gres=sdshd
</pre>

Your data is available at <tt>/mnt/sdshd/$USER</tt> inside the job, with no further commands needed.

'''The job shows <tt>CG</tt> (COMPLETING) in <tt>squeue</tt> during the upload phase.''' For large output files this can take minutes — this is expected and safe. The job will not exit until all data has been confirmed uploaded.

'''If you need data on SDS@hd before the job finishes''' (e.g. for a multi-step pipeline where the next job reads this job's output), call <tt>umount-sdshd</tt> explicitly at the end of your job script:

<pre>
#!/bin/bash
#SBATCH --gres=sdshd

# ... your computation ...

umount-sdshd # blocks here until all uploads are done, then the job exits
</pre>

= Advanced Configuration =

<div style="border: 3px solid #ffc107; padding: 15px; background-color: #fff3cd; margin: 10px 0;">
'''This section is optional.''' The built-in defaults work well for most jobs. Come back here if you need to:
* limit cache size or bandwidth
* use a persistent (crash-safe) cache on Weka
* tune performance for ML training or large job arrays
</div>

== Per-user configuration file ==

Create <tt>~/.config/sdshd/config</tt> to override the built-in defaults for all your jobs:

<pre>
mkdir -p ~/.config/sdshd
</pre>

The format is <tt>key = value</tt>, one per line. Lines starting with <tt>#</tt> are comments. The file is parsed without being executed — invalid or unknown keys are silently ignored.

=== Configuration keys ===

{| class="wikitable"
|-
! Key
! Default
! Description
|-
| <tt>cache_dir</tt>
| <tt>/tmp/sdshd-<uid>/cache/</tt>
| Local VFS cache directory. Node-local NVMe by default (cleaned on reboot). Set to a Weka workspace path for crash-safe persistent caching. See [[#Persistent_cache_on_Weka|Persistent cache on Weka]].
|-
| <tt>cache_mode</tt>
| <tt>full</tt>
| VFS cache mode: <tt>off</tt> / <tt>minimal</tt> / <tt>writes</tt> / <tt>full</tt>. <tt>full</tt> caches entire files for read and write. Required for arbitrary file access patterns. <tt>off</tt> reads directly from SDS@hd with no local cache.
|-
| <tt>cache_size</tt>
| <tt>50G</tt>
| Maximum local cache size. rclone evicts least-recently-used files automatically when the limit is approached.
|-
| <tt>cache_age</tt>
| <tt>24h</tt>
| How long to keep cached files before eviction, even if <tt>cache_size</tt> is not hit. Set to at least your job's wall time when using Weka as <tt>cache_dir</tt>.
|-
| <tt>sftp_concurrency</tt>
| <tt>16</tt>
| Parallel SFTP requests per connection. Higher values fill the TCP window more aggressively over the 5 ms WAN link. Reduce to <tt>8</tt> for large job arrays (e.g. 100+ tasks) to avoid overloading the gateway.
|-
| <tt>transfers</tt>
| <tt>8</tt>
| Number of parallel background upload workers.
|-
| <tt>buffer_size</tt>
| <tt>128M</tt>
| In-memory read/write buffer per file. 128M is well-matched to the Freiburg–Heidelberg RTT. Increase for very large sequential files.
|-
| <tt>read_ahead</tt>
| <tt>64M</tt>
| Sequential read prefetch window. Increase to <tt>512M</tt> for large sequential reads.
|-
| <tt>write_back</tt>
| <tt>5s</tt>
| Delay between file close and upload start. With the default <tt>5s</tt>, <tt>close()</tt> returns immediately and the upload runs in the background — your job continues while data transfers to SDS@hd. Setting this to <tt>0s</tt> makes uploads synchronous: your job stalls at every file write for the full upload duration. You almost certainly do not want <tt>0s</tt>.
|-
| <tt>dir_cache</tt>
| <tt>5m</tt>
| How long to cache directory listings. Use <tt>2h</tt> for compute jobs (stable dataset), <tt>5m</tt> for interactive use (picks up changes from other nodes quickly).
|-
| <tt>extra_opts</tt>
| (empty)
| Extra rclone flags appended verbatim. Values are split on spaces — arguments containing internal spaces (e.g. bwlimit timetables) cannot be expressed here; use <tt>rclone.conf</tt> or <tt>mount-sdshd</tt> command-line options instead.
|}

'''Available disk space by node type:'''

{| class="wikitable"
|-
! Node type
! Local disk
! Recommended max <tt>cache_size</tt>
|-
| Login nodes
| ~400 GB
| 20–50 G (shared with other users)
|-
| Milan nodes
| ~1.9 TB
| 50 G (default)
|-
| Other nodes
| ~3.8 TB
| 50 G (default)
|}

'''Useful <tt>extra_opts</tt> examples:'''

<pre>
# Limit upload/download bandwidth (per rclone process).
# Useful when running many jobs simultaneously or to be a good neighbour.
extra_opts = --bwlimit 200M

# Exclude temp and checkpoint files from the cache — saves cache space
# and avoids uploading files you don't need on SDS@hd.
# Do NOT use shell quotes here — write glob patterns unquoted.
extra_opts = --exclude *.tmp --exclude checkpoint_*.pt
</pre>

== Persistent cache on Weka ==

With the default <tt>/tmp</tt> cache, data written to SDS@hd is first held locally on the node's NVMe. If the node crashes, reboots, or is killed (OOM, walltime) before the upload completes, the cached data is lost.

A Weka workspace survives node failures. If a job is killed before uploads finish, the data is still in the cache — remount with the same <tt>cache_dir</tt> to resume, or copy directly:

<pre>
# Create a workspace and configure it as the cache directory:
ws_allocate mywsname 30
cat >> ~/.config/sdshd/config <<'EOF'
cache_dir = /path/from/ws_find
cache_size = 2T
cache_age = 720h
EOF
</pre>

Replace <tt>/path/from/ws_find</tt> with the output of <tt>ws_find mywsname</tt>.

After a failed job, recover the cached data:

<pre>
# The cache subdirectory is named after the compute node hostname.
# Check: ls $(ws_find mywsname)/
rclone copy "$(ws_find mywsname)/rclone-<compute-node-name>/vfs/sdshd-sftp/" \
sdshd-sftp:recovered/ --progress
</pre>

'''When are cached files removed?'''

{| class="wikitable"
|-
! Trigger
! What happens
|-
| rclone running, <tt>cache_age</tt> exceeded
| rclone evicts the file from cache (LRU)
|-
| rclone running, <tt>cache_size</tt> would be exceeded
| rclone evicts least-recently-used files
|-
| rclone not running (between mounts)
| nothing — files are not touched
|-
| Workspace expires (<tt>ws_extend</tt> not run)
| restorable for 30 days, then auto-deleted permanently
|-
| Manual cleanup
| remove the <tt>rclone-*</tt> subdirectory inside the workspace
|}

See [[Workspaces]] for workspace management commands.

= Advanced Usage =

== Read-only access ==

For browsing or spot reads without writing — bypasses the local cache entirely, every read goes directly to SDS@hd:

<pre>
mount-sdshd --vfs-cache-mode off --read-only
</pre>

For '''repeated reads of the same files''' (e.g. checking a large dataset), use the default cache with <tt>--read-only</tt>. The VFS cache still operates — files are downloaded once on first access and served from local disk on all subsequent reads:

<pre>
mount-sdshd --read-only
</pre>

== ML training and read-intensive jobs ==

If your job reads the same files many times (training epochs, iterative algorithms, multi-pass analysis), set a Weka workspace as <tt>cache_dir</tt>. rclone downloads each file on first access; every subsequent read is served from Weka without any WAN traffic:

<pre>
ws_find mywsname # e.g. /work/classic/fr_abc123456-mywsname
</pre>

<pre>
# In ~/.config/sdshd/config:
cache_dir = /work/classic/fr_abc123456-mywsname
cache_size = 2T
cache_age = 120h
</pre>

The first epoch downloads from SDS@hd. Every subsequent epoch reads from Weka — no WAN traffic, no gateway load.

'''Note:''' The cache is per-node. Each node in a job array downloads its own copy. For job arrays where all tasks need the same files, use <tt>rclone copy</tt> to pre-stage once instead (see below).

== Shared dataset across many jobs ==

If many job array tasks need to read the same files, rclone VFS cannot share a cache — concurrent mounts against the same cache directory will corrupt it, even with <tt>--read-only</tt>.

The solution: copy the dataset once to a Weka workspace, then have all jobs read from the Weka path directly as a plain filesystem — no FUSE mount, no per-job downloads:

<pre>
# Step 1 (once, on login node): copy dataset from SDS@hd to Weka
ws_allocate mydata 30
rclone copy sdshd-sftp:inputdata/ $(ws_find mydata)/inputdata/ \
--progress --transfers 8 --sftp-concurrency 8

# Step 2: reference the Weka path in your job script (no --gres=sdshd needed)
INPUT_DIR=$(ws_find mydata)/inputdata
</pre>

All tasks read from Weka at full speed with zero WAN traffic and zero gateway load. Re-run <tt>rclone copy</tt> if the source data changes — it skips unchanged files.

= Known Limitations =

== Symlinks are not preserved ==

rclone's SFTP backend does not store symlinks on the remote. When rclone uploads a symlink (through the FUSE mount or via <tt>rclone copy</tt>), it follows the link and uploads the target's content as a regular file. The symlink itself is lost.

{| class="wikitable"
|-
! Situation
! What happens
|-
| Symlink to a regular file
| Target file is uploaded; symlink metadata is lost
|-
| Symlink to a directory
| Directory tree is traversed; contents are uploaded recursively
|-
| Dangling symlink (target missing)
| Transfer error or silently skipped
|-
| Restore from SDS@hd
| Regular files/directories — original symlinks are gone
|}

'''Workarounds:'''

* '''Archive before upload:''' <tt>tar -czf env.tar.gz myenv/</tt> — preserves all symlinks inside the archive. Unpack after download to restore the original structure.
* '''<tt>rclone copy --links</tt>:''' encodes symlinks as <tt>.rclonelink</tt> text files on the remote. Only works for <tt>rclone copy</tt>/<tt>rclone sync</tt> workflows (not the FUSE VFS mount). Requires <tt>--links</tt> on both the copy and the restore.

= Troubleshooting =

== <tt>SSH_FX_FAILURE</tt> during upload ==

'''Most common cause: storage quota exhausted.'''

SDS@hd enforces a hard quota per project. When full, the server rejects all writes. Check your quota first:

<pre>
rclone about sdshd-sftp:
</pre>

If the quota is full, delete or archive old data on SDS@hd before retrying.

'''Secondary cause: gateway overload from large job arrays.'''

100 tasks × 16 SFTP connections = 1600 simultaneous connections to <tt>lsdf02-sshfs</tt>. The gateway may reject new connections under this load. Reduce <tt>sftp_concurrency</tt> to <tt>8</tt> in <tt>~/.config/sdshd/config</tt>, or bypass the FUSE mount entirely for bulk transfers:

<pre>
rclone copy /local/results/ sdshd-sftp:results/ \
--transfers 4 --sftp-concurrency 8 --progress
</pre>

Interrupted uploads are not lost — data remains in the local cache and resumes automatically the next time rclone mounts the same cache directory.

== Job stuck in <tt>CG</tt> (COMPLETING) for a long time ==

The Slurm epilog waits for rclone to finish uploading all pending data. For large output files this can take minutes — this is expected and safe. The job will exit on its own once all uploads are confirmed complete.

If you need to estimate the remaining upload time:

<pre>
rclone rc --unix-socket /tmp/sdshd-$(id -u)/rc.sock vfs/stats
</pre>

The <tt>uploadsInProgress</tt> and <tt>uploadsQueued</tt> fields show the current transfer state.

NEMO2

M Janczyk — Tue, 21 Apr 2026 13:45:05 GMT

← Older revision		Revision as of 15:45, 21 April 2026
Line 48:		Line 48:
	* [[NEMO2/Hardware\|Hardware and Architecture]]		* [[NEMO2/Hardware\|Hardware and Architecture]]
	* [[NEMO2/Hardware#File_Systems\|File Systems]] and [[Workspaces]]		* [[NEMO2/Hardware#File_Systems\|File Systems]] and [[Workspaces]]
			** [[NEMO2/SDS_hd\|Using SDS@hd on NEMO2]]
	* Cluster Specific Software:		* Cluster Specific Software:
	** [[NEMO2/Modules\|Available Modules (Wiki)]] or [https://www.nemo.uni-freiburg.de/easybuild-modules/ Available Modules (HTML, incl. search)] on NEMO2		** [[NEMO2/Modules\|Available Modules (Wiki)]] or [https://www.nemo.uni-freiburg.de/easybuild-modules/ Available Modules (HTML, incl. search)] on NEMO2

Registration/2FA

S Braun — Tue, 21 Apr 2026 13:28:23 GMT

Token Management

← Older revision		Revision as of 15:28, 21 April 2026
Line 69:		Line 69:
	To activate the second factor, '''please perform the following steps:'''		To activate the second factor, '''please perform the following steps:'''

	1. '''Select the registration server of the cluster''' for which you want to create a second factor and login to it:</br> → [https://login.bwidm.de/user/twofa.xhtml Registration server for '''bwUniCluster 2.0''', '''bwForCluster JUSTUS 2''' and '''bwForCluster NEMO 2'''] (2FA tokens are valid for all three clusters; KIT members can reuse their existing hardware and software tokens)</br> → [https://bwservices.uni-heidelberg.de/user/twofa.xhtml Registration server for '''bwForCluster Helix''']		1. '''Select the registration server of the cluster''' for which you want to create a second factor and login to it:</br> → [https://login.bwidm.de/user/twofa.xhtml Registration server for '''bwUniCluster 3.0''', '''bwForCluster JUSTUS 2''' and '''bwForCluster NEMO 2'''] (2FA tokens are valid for all three clusters; KIT members can reuse their existing hardware and software tokens)</br> → [https://bwservices.uni-heidelberg.de/user/twofa.xhtml Registration server for '''bwForCluster Helix''']
	[[File:BwIDM-twofa.png\|center\|600px\|thumb\|My Tokens]]		[[File:BwIDM-twofa.png\|center\|600px\|thumb\|My Tokens]]

BwUniCluster3.0/Hardware and Architecture

S Braun — Tue, 21 Apr 2026 10:59:26 GMT

Compute nodes

← Older revision		Revision as of 12:59, 21 April 2026
(One intermediate revision by the same user not shown)
Line 77:		Line 77:
	!scope="column"\| Number of nodes		!scope="column"\| Number of nodes
	\| 272		\| 272
	\| 70		\| 80
	\| 4		\| 5
	\| 12		\| 12
	\| 1		\| 1

Development/VS Code

S Braun — Tue, 21 Apr 2026 09:32:28 GMT

← Older revision		Revision as of 11:32, 21 April 2026
Line 113:		Line 113:



	<!--
	== Connect to Remote Jupyter Kernel ==		== Connect to Remote Jupyter Kernel ==
	To work with your python scripts and notebooks within VSCode while using the resources of a compute node, you can create a batch job that launches JupyterLab and connect to it via VS Code. To do so, please follow the instructions below. Any parts of the scripts that might need adjustments are marked with the keyword "@params".		To work with your python scripts and notebooks within VSCode while using the resources of a compute node, you can create a batch job that launches JupyterLab and connect to it via VS Code. To do so, please follow the instructions below. Any parts of the scripts that might need adjustments are marked with the keyword "@params".
Line 346:		Line 346:
	#rm $slurm_out		#rm $slurm_out
	</syntaxhighlight>		</syntaxhighlight>
	-->

Quickstart Guide

F Bartusch — Tue, 21 Apr 2026 08:51:40 GMT

F Bartusch deleted page Quickstart Guide BinAC 1 not running any more

Memory Usage

F Bartusch — Tue, 21 Apr 2026 08:51:28 GMT

F Bartusch deleted page Memory Usage BinAC 1 not running any more

BwForCluster BinAC: Quickstart Guide

F Bartusch — Tue, 21 Apr 2026 08:51:03 GMT

F Bartusch deleted page BwForCluster BinAC: Quickstart Guide BinAC 1 not running any more

BwForCluster BinAC Login

F Bartusch — Tue, 21 Apr 2026 08:50:49 GMT

F Bartusch deleted page BwForCluster BinAC Login BinAC 1 not running any more

BinAC/Software/Bowtie

F Bartusch — Tue, 21 Apr 2026 08:49:37 GMT

F Bartusch deleted page BinAC/Software/Bowtie BinAC 1 not running any more

FAQ

K Siegmund — Tue, 21 Apr 2026 08:49:18 GMT

← Older revision		Revision as of 10:49, 21 April 2026
Line 1:		Line 1:
	__TOC__		__TOC__

			Frequently Asked Questions

	== Access and Account Issues: Generic Advice ==		== Access and Account Issues: Generic Advice ==

⚫	~~<noinclude>~~<!-- [[LOST_ACCESS]] redirects to this FAQ page. If you move the content, you have to adjust the link--~~></noinclude~~>

	* Double-check your [[Registration/Login/Username\|username]] and [[Registration/Password\|password]].		* Double-check your [[Registration/Login/Username\|username]] and [[Registration/Password\|password]].
	* Make sure your CAPS LOCK key is not enabled.		* Make sure your CAPS LOCK key is not enabled.
	* Are you within the university network (Universities in Baden-Württemberg, BELWÜ)?		* Are you within the university network (Universities in Baden-Württemberg, BELWÜ)?

		⚫	<!-- [[LOST_ACCESS]] redirects to this FAQ page. If you move the content, you have to adjust the link-->

	== Access and Account Issues: bwUniCluster ==		== Access and Account Issues: bwUniCluster ==

BinAC/Software/Gromacs

F Bartusch — Tue, 21 Apr 2026 08:48:58 GMT

F Bartusch deleted page BinAC/Software/Gromacs BinAC 1 not running any more

BinAC/Software/Cellranger

F Bartusch — Tue, 21 Apr 2026 08:48:17 GMT

F Bartusch deleted page BinAC/Software/Cellranger BinAC 1 not running any more

FAQ

K Siegmund — Tue, 21 Apr 2026 08:47:53 GMT

← Older revision		Revision as of 10:47, 21 April 2026
Line 1:		Line 1:
	__TOC__		__TOC__

	<!-- FAQ page: login, quota -->
	== Access and Account Issues: Generic Advice ==		== Access and Account Issues: Generic Advice ==

	<!-- [[LOST_ACCESS]] redirects to this FAQ page. If you move the content, you have to adjust the link-->		<noinclude><!-- [[LOST_ACCESS]] redirects to this FAQ page. If you move the content, you have to adjust the link--></noinclude>

	* Double-check your [[Registration/Login/Username\|username]] and [[Registration/Password\|password]].		* Double-check your [[Registration/Login/Username\|username]] and [[Registration/Password\|password]].

MediaWiki:Sidebar

M Janczyk — Tue, 21 Apr 2026 08:46:44 GMT

← Older revision		Revision as of 10:46, 21 April 2026
Line 1:		Line 1:
	* SEARCH		* SEARCH
	* bwHPC Wiki		* bwHPC Wiki
	** mainpage\|Home		** mainpage\|Wiki Home
	** https://www.bwhpc.de\|bwHPC Web Page		** https://www.bwhpc.de\|bwHPC Web Page
	* bwHPC Systems		* bwHPC Systems

BinAC/Quickstart Guide

F Bartusch — Tue, 21 Apr 2026 08:45:55 GMT

F Bartusch deleted page BinAC/Quickstart Guide BinAC 1 not running any more

BinAC/Memory Usage

F Bartusch — Tue, 21 Apr 2026 08:43:17 GMT

F Bartusch deleted page BinAC/Memory Usage BinAC 1 not running any more

FAQ

K Siegmund — Tue, 21 Apr 2026 08:42:24 GMT

← Older revision		Revision as of 10:42, 21 April 2026
(One intermediate revision by the same user not shown)
Line 1:		Line 1:
	__TOC__		__TOC__

			<!-- FAQ page: login, quota -->
	== Access and Account Issues: Generic Advice ==		== Access and Account Issues: Generic Advice ==

	<!-- [[LOST_ACCESS]] redirects to this FAQ page. If you move the content, you have to adjust the link-->		<!-- [[LOST_ACCESS]] redirects to this FAQ page. If you move the content, you have to adjust the link-->

NEMO2/Migrate Moab to Slurm jobs

M Janczyk — Tue, 21 Apr 2026 08:40:41 GMT

M Janczyk deleted page NEMO2/Migrate Moab to Slurm jobs content was: "{|style="background:#deffee; width:100%;" |style="padding:5px; background:#cef2e0; text-align:left"| center|25px |style="padding:5px; background:#cef2e0; text-align:left"| By default Moab does not export any environment variables to the job's runtime environment. With Slurm most of the login environment variables are exported to your job's runtime...", and the only contributor was "M Janczyk" (talk)

BinAC/Login

F Bartusch — Tue, 21 Apr 2026 08:40:13 GMT

F Bartusch deleted page BinAC/Login BinAC 1 not running any more

Category:BwForCluster

M Janczyk — Tue, 21 Apr 2026 08:37:36 GMT

M Janczyk deleted page Category:BwForCluster content was: "The category '''bwForCluster''' refers to the compute clusters on tier 3 level of the bwHPC infrastructure in the state of Baden-Württemberg: {| style="width: 100%; border-spacing: 5px;" | style="text-align:center; color:#000;vertical-align:middle;font-size:75%;" | center|border|400px|Figure: High performance computing infrastructure in Baden-Württemberg regarding HPC tiers. |- | style="text-align:ce..."

BinAC2/Login

F Bartusch — Tue, 21 Apr 2026 08:37:15 GMT

Hostnames

← Older revision		Revision as of 10:37, 21 April 2026
Line 33:		Line 33:
	<b>Example</b>: If your local username at your University is <code>ab123</code> and you are a user from Tübingen University, your username on the cluster is: <code>tu_ab123</code>.		<b>Example</b>: If your local username at your University is <code>ab123</code> and you are a user from Tübingen University, your username on the cluster is: <code>tu_ab123</code>.

	== ~~Hostnames~~ ==		== Hostname ==

	BinAC 2 has one login node serving as a load balancer. We use DNS round-robin scheduling to load-balance the incoming connections between the actual three login nodes. If you are logging in multiple times, different sessions might run on different login nodes and hence programs started in one session might not be visible in another sessions.		BinAC 2 has one login node serving as a load balancer. We use DNS round-robin scheduling to load-balance the incoming connections between the actual three login nodes. If you are logging in multiple times, different sessions might run on different login nodes and hence programs started in one session might not be visible in another sessions.

File:Nemo-sign.jpg

M Janczyk — Tue, 21 Apr 2026 08:36:43 GMT

M Janczyk deleted page File:Nemo-sign.jpg content was: "#REDIRECT File:Nemo.jpg", and the only contributor was "M Janczyk" (talk)

User:R Doros

F Bartusch — Tue, 21 Apr 2026 08:29:35 GMT

F Bartusch blocked R Doros with an expiration time of indefinite (account creation disabled) Not working any more in Tübingen

User:V Lutz

F Bartusch — Tue, 21 Apr 2026 08:29:01 GMT

F Bartusch blocked V Lutz with an expiration time of indefinite (account creation disabled, autoblock disabled) not working any more in Tübingen

Registration/bwForCluster/BINAC

F Bartusch — Tue, 21 Apr 2026 08:23:39 GMT

F Bartusch deleted page Registration/bwForCluster/BINAC BinAC 1 not running any more

Registration/bwUniCluster

M Carmesin — Tue, 21 Apr 2026 08:23:37 GMT

← Older revision		Revision as of 10:23, 21 April 2026
Line 36:		Line 36:
	* Step C: You need to '''register for the service bwUniCluster''' (create an account on the cluster).<br />→ '''[[Registration/bwUniCluster/Service\|Step C: bwUniCluster Registration]]		* Step C: You need to '''register for the service bwUniCluster''' (create an account on the cluster).<br />→ '''[[Registration/bwUniCluster/Service\|Step C: bwUniCluster Registration]]


	----

BinAC2/Queues

F Bartusch — Tue, 21 Apr 2026 08:22:32 GMT

F Bartusch deleted page BinAC2/Queues empty page

BinAC2/Migrate Moab to Slurm jobs

F Bartusch — Tue, 21 Apr 2026 08:21:59 GMT

F Bartusch deleted page BinAC2/Migrate Moab to Slurm jobs BinAC 1 not running any more

BinAC2/Migrate BinAC 1 workspaces to BinAC 2 workspaces

F Bartusch — Tue, 21 Apr 2026 08:21:20 GMT

F Bartusch deleted page BinAC2/Migrate BinAC 1 workspaces to BinAC 2 workspaces BinAC 1 not running any more: Migration not possible any more

← Older revision		Revision as of 15:45, 21 April 2026
Line 48:		Line 48:
	* [[NEMO2/Hardware\|Hardware and Architecture]]		* [[NEMO2/Hardware\|Hardware and Architecture]]
	* [[NEMO2/Hardware#File_Systems\|File Systems]] and [[Workspaces]]		* [[NEMO2/Hardware#File_Systems\|File Systems]] and [[Workspaces]]
			** [[NEMO2/SDS_hd\|Using SDS@hd on NEMO2]]
	* Cluster Specific Software:		* Cluster Specific Software:
	** [[NEMO2/Modules\|Available Modules (Wiki)]] or [https://www.nemo.uni-freiburg.de/easybuild-modules/ Available Modules (HTML, incl. search)] on NEMO2		** [[NEMO2/Modules\|Available Modules (Wiki)]] or [https://www.nemo.uni-freiburg.de/easybuild-modules/ Available Modules (HTML, incl. search)] on NEMO2