NEMO2/SDS hd - Revision history

M Janczyk: /* Per-user configuration file */

2026-04-22T16:49:14Z

Per-user configuration file

← Older revision		Revision as of 18:49, 22 April 2026
Line 290:		Line 290:
	# means your job script stalls at each file write — you almost certainly		# means your job script stalls at each file write — you almost certainly
	# do not want this.		# do not want this.
			# Note: value must be in seconds (e.g. 5s, 30s); other units are not supported.
	write_back = 5s		write_back = 5s

M Janczyk at 15:55, 22 April 2026

2026-04-22T15:55:22Z

@@ Line 429: / Line 429: @@
 mount-sdshd --read-only
 </pre>
 == Shared dataset across many jobs ==

M Janczyk: /* Persistent cache on Weka */

2026-04-22T15:54:24Z

Persistent cache on Weka

← Older revision		Revision as of 17:54, 22 April 2026
Line 411:		Line 411:

	See [[Workspaces]] for workspace management commands.		See [[Workspaces]] for workspace management commands.

			The same Weka <tt>cache_dir</tt> also benefits read-heavy jobs such as ML training or multi-epoch analysis: rclone downloads each input file on first access and serves all subsequent reads from Weka without any WAN traffic. Note that the cache is per-node — each node in a job array downloads its own copy independently. For job arrays where all tasks need the same files, use <tt>rclone copy</tt> to pre-stage once instead — see [[#Shared_dataset_across_many_jobs\|Shared dataset across many jobs]].

	= Advanced Usage =		= Advanced Usage =

M Janczyk: /* Shared dataset across many jobs */

2026-04-22T15:46:47Z

Shared dataset across many jobs

← Older revision		Revision as of 17:46, 22 April 2026
Line 457:		Line 457:
	ws_allocate mydata 30		ws_allocate mydata 30
	rclone copy sdshd-sftp:inputdata/ $(ws_find mydata)/inputdata/ \		rclone copy sdshd-sftp:inputdata/ $(ws_find mydata)/inputdata/ \
	--progress --transfers 8 --sftp-concurrency 8		--progress --transfers 8 --sftp-concurrency 8 --sftp-chunk-size 255k

	# Step 2: reference the Weka path in your job script (no --gres=sdshd needed)		# Step 2: reference the Weka path in your job script (no --gres=sdshd needed)
	INPUT_DIR=$(ws_find mydata)/inputdata		INPUT_DIR=$(ws_find mydata)/inputdata
	</pre>		</pre>

			<tt>--sftp-chunk-size 255k</tt> increases the SFTP packet size from 32 KB (default) to 255 KB (OpenSSH's maximum), which reduces protocol overhead and significantly improves throughput for large files. Use this flag with explicit <tt>rclone copy</tt> commands — it has no effect on the FUSE mount and should not be placed in <tt>extra_opts</tt>.

	All tasks read from Weka at full speed with zero WAN traffic and zero gateway load. Re-run <tt>rclone copy</tt> if the source data changes — it skips unchanged files.		All tasks read from Weka at full speed with zero WAN traffic and zero gateway load. Re-run <tt>rclone copy</tt> if the source data changes — it skips unchanged files.

M Janczyk: /* SSH_FX_FAILURE during upload */

2026-04-22T15:41:14Z

SSH_FX_FAILURE during upload

← Older revision		Revision as of 17:41, 22 April 2026
Line 514:		Line 514:
	<pre>		<pre>
	rclone copy /local/results/ sdshd-sftp:results/ \		rclone copy /local/results/ sdshd-sftp:results/ \
	--transfers 4 --sftp-concurrency 8 --progress		--transfers 4 --sftp-concurrency 8 --sftp-chunk-size 255k --progress
	</pre>		</pre>

M Janczyk: /* Per-user configuration file */

2026-04-22T15:32:10Z

Per-user configuration file

@@ Line 238: / Line 238: @@
 | Extra rclone flags appended verbatim. Values are split on spaces — arguments containing internal spaces (e.g. bwlimit timetables) cannot be expressed here; use <tt>rclone.conf</tt> or <tt>mount-sdshd</tt> command-line options instead.
 |}
 '''Available disk space by node type:'''

M Janczyk: /* Initial Setup (Required) */

2026-04-22T15:01:28Z

Initial Setup (Required)

← Older revision		Revision as of 17:01, 22 April 2026
Line 18:		Line 18:
	'''Prerequisites:'''		'''Prerequisites:'''
	* Your bwIDM username and SDS@hd password		* Your bwIDM username and SDS@hd password
	* Your SDS@hd project ID (find it at [~~https://wiki.bwhpc.de/e/SDS@hd/Access wiki.bwhpc.de/e/~~SDS@hd/Access])		* Your SDS@hd project ID (e.g. <tt>sd14a001</tt>, find it at [[SDS@hd/Access]])

	== Step 1: Create the rclone configuration file ==		== Step 1: Create the rclone configuration file ==
Line 48:		Line 48:

	Replace:		Replace:
	* <tt>YOUR_BWIDM_USERNAME</tt> — your bwIDM login name		* <tt>YOUR_BWIDM_USERNAME</tt> — your bwIDM login name (same as on NEMO2)
	* <tt>YOUR_PROJECT_ID</tt> — your SDS@hd project ID, e.g. <tt>sd14a001</tt>		* <tt>YOUR_PROJECT_ID</tt> — your SDS@hd project ID, e.g. <tt>sd14a001</tt>
	* <tt>PASTE_OBSCURED_PASSWORD_HERE</tt> — leave this for now, you will fill it in Step 2		* <tt>PASTE_OBSCURED_PASSWORD_HERE</tt> — leave this for now, you will fill it in Step 2

M Janczyk: /* SSH_FX_FAILURE during upload */

2026-04-22T14:20:05Z

SSH_FX_FAILURE during upload

← Older revision		Revision as of 16:20, 22 April 2026
Line 430:		Line 430:
	</pre>		</pre>

			With the default <tt>/tmp</tt> cache, resuming requires submitting a new job to the exact same compute node — not generally practical — and the cache is gone after a node reboot regardless. With a Weka workspace as <tt>cache_dir</tt>, the cache survives both job end and node reboots; the key advantage is that you can copy the data out manually without needing a job on that node at all, as described in [[#Persistent_cache_on_Weka\|Persistent cache on Weka]].
	Interrupted uploads are not lost — data remains in the local cache and resumes automatically the next time rclone mounts the same cache directory.

	== Job stuck in <tt>CG</tt> (COMPLETING) for a long time ==		== Job stuck in <tt>CG</tt> (COMPLETING) for a long time ==

M Janczyk: /* Job stuck in CG (COMPLETING) for a long time */

2026-04-22T14:14:23Z

Job stuck in CG (COMPLETING) for a long time

← Older revision		Revision as of 16:14, 22 April 2026
Line 434:		Line 434:
	== Job stuck in <tt>CG</tt> (COMPLETING) for a long time ==		== Job stuck in <tt>CG</tt> (COMPLETING) for a long time ==

	The Slurm epilog waits for rclone to finish uploading all pending data. For large output files this can take minutes — this is expected and safe. The job will exit on its own once all uploads are confirmed complete.		The Slurm epilog waits for rclone to finish uploading all pending data. For large output files this can take minutes — this is expected and safe. The job will exit on its own once all uploads are confirmed complete. There is nothing to do.

	If the job is still in state <tt>R</tt> (running), you can query the upload state from a login node while the job is active — SSH access via <tt>pam_slurm_adopt</tt> requires a running job on the node:

	<pre>
	# Replace JOBID
	ssh $(squeue -j JOBID -h -o %N) rclone rc --unix-socket /tmp/sdshd-$(id -u)/rc.sock vfs/stats
	</pre>

	The <tt>uploadsInProgress</tt> and <tt>uploadsQueued</tt> fields show the current transfer state.

	Once the job enters <tt>CG</tt>, SSH access to the node is blocked by <tt>pam_slurm_adopt</tt>. At that point there is no way to query rclone from outside — the job will exit on its own once all uploads are confirmed complete.

M Janczyk: /* Job stuck in CG (COMPLETING) for a long time */

2026-04-22T14:08:31Z

Job stuck in CG (COMPLETING) for a long time

← Older revision		Revision as of 16:08, 22 April 2026
Line 436:		Line 436:
	The Slurm epilog waits for rclone to finish uploading all pending data. For large output files this can take minutes — this is expected and safe. The job will exit on its own once all uploads are confirmed complete.		The Slurm epilog waits for rclone to finish uploading all pending data. For large output files this can take minutes — this is expected and safe. The job will exit on its own once all uploads are confirmed complete.

			If the job is still in state <tt>R</tt> (running), you can query the upload state from a login node while the job is active — SSH access via <tt>pam_slurm_adopt</tt> requires a running job on the node:
	If you need to estimate the remaining upload time:

	<pre>		<pre>
			# Replace JOBID
	rclone rc --unix-socket /tmp/sdshd-$(id -u)/rc.sock vfs/stats		ssh $(squeue -j JOBID -h -o %N) rclone rc --unix-socket /tmp/sdshd-$(id -u)/rc.sock vfs/stats
	</pre>		</pre>

	The <tt>uploadsInProgress</tt> and <tt>uploadsQueued</tt> fields show the current transfer state.		The <tt>uploadsInProgress</tt> and <tt>uploadsQueued</tt> fields show the current transfer state.

			Once the job enters <tt>CG</tt>, SSH access to the node is blocked by <tt>pam_slurm_adopt</tt>. At that point there is no way to query rclone from outside — the job will exit on its own once all uploads are confirmed complete.