bwHPC Wiki - User contributions [en]

Helix/bwVisu

2026-04-01T22:35:23Z

H Schumacher: Added link to QuPath page

[[File:BwVisu wide.svg|300px]]

[https://bwvisu.bwservices.uni-heidelberg.de/ bwVisu] is a scalable service for remote visualization and interactive applications. This easy-to-use, web-based platform provides interactive access to scientific work environments utilizing massive compute and storage resources.

{| style=" background:#FEF4AB; width:100%;"
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News
|-
|
* December 2024: The [https://bwvisu.bwservices.uni-heidelberg.de/ new bwVisu version] based on Open OnDemand is now in operation. You can write feedback to [mailto:bwvisu-support@urz.uni-heidelberg.de bwvisu-support].
|}

{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support
|-
|
* [https://www.urz.uni-heidelberg.de/en/service-catalogue/software-and-applications/bwvisu Service Description & FAQ]
* [mailto:bwvisu-support@urz.uni-heidelberg.de Submit a Ticket]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left;" | User Documentation
|-
|
* [https://bwvisu.bwservices.uni-heidelberg.de/ Link to Webservice]
* [[Helix/bwVisu/Getting_Started|Getting Started]]
* [[Helix/bwVisu/Usage|Usage]]
* Available applications and user guides:

{| class="wikitable" style="text-align: center; margin-left: 30px;"
|- style="font-weight:bold;"
! Application
! [[Helix/bwVisu/Usage#noVNC_Applications|noVNC App]]
|-
| [[Helix/bwVisu/CryoSPARC|CryoSPARC]]
| ✓
|-
| [https://imagej.net/software/fiji/ Fiji]
| ✓
|-
| [https://www.ilastik.org/ Ilastik]
| ✓
|-
| [https://bio3d.colorado.edu/imod/ IMOD]
| ✓
|-
| [https://docs.mitk.org/2024.12/UserManualPortal.html MITK]
| ✓
|-
| [https://phy.readthedocs.io/en/latest/ Phy]
| ✓
|-
| [https://qgis.org/ QGIS]
| ✓
|-
| [[Helix/bwVisu/QuPath|QuPath]]
| ✓
|-
| [[Helix/bwVisu/JupyterLab|JupyterLab]]
| ☓
|-
| [[Helix/bwVisu/RStudio|RStudio]]
| ☓
|-
| [[Helix/bwVisu/KI-Morph|KI-Morph]]
| ☓
|}
|}

{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Acknowledgement
|-
|
* Please [[Helix/bwVisu/Acknowledgement|acknowledge]] bwVisu in your publications

Helix/Filesystems

2026-03-30T15:29:40Z

H Schumacher: Moved "free up space" section into its own chapter. Added "list" option to ws_share.

== Overview ==

The cluster storage system provides a large parallel file system based on [https://www.ibm.com/support/knowledgecenter/STXKQY/ibmspectrumscale_welcome.html IBM Spectrum Scale]
for $HOME, for workspaces, and for temporary storage via the $TMPDIR environment variable.

{| class="wikitable"
|-
!style="width:16%" |
!style="width:28%"| $HOME
!style="width:28%"| Workspaces
!style="width:28%"| $TMPDIR
|-
!scope="column" | Visibility
| global
| global
| local
|-
!scope="column" | Lifetime
| permanent
| workspace lifetime
| batch job walltime
|-
!scope="column" | Quotas
| 200 GB
| 10 TB
| none
|-
!scope="column" | Backup
| snapshots
| snapshots
| no
|}

* global: all nodes access the same file system.
* local: each node has its own temporary file space.
* permanent: files are stored permanently.
* workspace lifetime: files are removed at end of workspace lifetime.
* batch job walltime: files are removed at end of the batch job.
* snapshots: a copy of all files is saved at specific timepoints so that they can be restored when necessary. For details see the related sections.

== $HOME ==

Home directories are meant for permanent storage of files that are kept being used like source codes, configuration files, executable programs.

'''Disk Space'''

The disk space per user is limited to 200 GB. The used disk space is displayed with the command:

<pre>
homequotainfo
</pre>

'''Backup'''

The backup of the home directory can be found under <code>/home/.snapshots</code>.

== Workspaces ==

Workspace tools can be used to get temporary space for larger amounts of data necessary for or produced by running jobs.

'''Usage'''

To create a workspace you need to supply a name for the workspace and a lifetime in days. The maximum lifetime is 30 days. It is possible to extend the lifetime 10 times.

{| class="wikitable"
|-
!style="width:30%" | Command
!style="width:70%"| Action
|-
|ws_allocate -r 7 -m <email> foo 10
|Allocate a workspace named foo for 10 days and set a email reminder 7 days before expiring.
'''Important:''' You should add your email address and a relative date for notification!
|-
|ws_list -a
|List all your workspaces.
|-
|ws_find foo
|Get absolute path of workspace foo.
|-
|ws_extend foo 5
|Extend lifetime of workspace foo by 5 days from now.
|-
|ws_release foo
|Manually erase your workspace foo.
|-
|ws_send_ical -m <email> foo
|Sending a .ical calendar entry for reminding workspace expiring
|-
|ws_share <share / unshare / unshare-all / list> <workspacename> <username> [username2]
|ws_share allows to share an existing workspace with other users.
|-
|}

'''Disk Space'''

If you plan to produce or copy large amounts of data in workspaces, please check the availability. The used and free disk space on the workspace filesystem is displayed with the command:

<pre>
workquotainfo
</pre>

=== Restoring expired Workspaces ===
At expiration time or after you've released it, your workspace will be moved to a special, hidden directory. For a short period of time you can still restore your data into a valid workspace. To get a list of your expired workspaces with their full names, use
<pre>
ws_restore -l
</pre>
Then, restore them into an '''existing, active workspace''' <code>restored_ws</code>:
<pre>
ws_restore <deleted_workspacename> restored_ws
</pre>
The expired workspace has to be specified using the '''full name''', including username prefix and timestamp suffix (otherwise, it cannot be uniquely identified).
The target workspace, on the other hand, must be given with just its short name as listed by <code>ws_list</code>, without the username prefix.

NOTE: <code>ws_restore</code> can only work on the same filesystem. So you have to ensure that the new workspace allocated with <code>ws_allocate</code> is placed on the same filesystem as the expired workspace. Therefore you can use <code>-F <filesystem></code> flag if needed.

Snapshots: If the workspace can't be restored anymore, you can check the snapshots under <code>/work/.snapshots/<timepoint>/ws/</code>. Changes that happened since the last snapshot was created are lost. Caution: We try to keep the latest snapshots but we can't guarantee that snapshots will be available at all times.

=== Linking workspaces in Home ===
It might be valuable to have links to personal workspaces within a certain directory, e.g., the user home directory. The command
<pre>
ws_register <DIR>
</pre>
will create and manage links to all personal workspaces within in the directory <DIR>. Calling this command will do the following:

* The directory <DIR> will be created if necessary
* Links to all personal workspaces will be managed:
** Creates links to all available workspaces if not already present
** Removes links to released workspaces

=== Sharing Workspaces ===
To simplify sharing of workspaces you can use the command <code>ws_share</code> as described in the table above.
The command sets up the workspace to allow read-only access to the specified users.
If you need more specific permissions, you can follow the guidelines here: →
[[Workspace#Setting_Permissions_for_Sharing_Files | Setting_Permissions_for_Sharing_Files]]

== $TMPDIR ==

The variable <tt>$TMPDIR</tt> provides file space for temporary data of running jobs.
Each node has its own <tt>$TMPDIR</tt>. 
The data in $TMPDIR become unavailable as soon as the job has finished.

== SDS@hd Access ==

It is possible to access your storage space on [http://sds-hd.urz.uni-heidelberg.de SDS@hd] directly on the bwForCluster Helix in /mnt/sds-hd/ on all login and compute nodes.

== Free up Space ==

If you need to free up space, you can...
* ...check the file sizes with <code>du -sh --apparent-size * .[^.]*</code>.
* ...delete temporary files, for example in the <code>.cache</code> folder. Other folders starting with a dot might contain temporary files as well, for example <code>.local/share/Trash</code>.
* ...use the cleaning commands by your used software (e.g. [https://wiki.bwhpc.de/e/Development/Conda#Create_Environments_and_Install_Software conda]).
* ...delete folders with temporary data created by your used software (e.g. nextflow).

== Troubleshooting ==
* '''Error: "Disk space exceeded"'''
*: This means that there is not enough space (e.g. in your $HOME) to execute the operation. Please [[Helix/Filesystems#Free_up_Space | free up space]].
*'''Error: "could not create workspace directory"'''
*:You should check the localesetting of your ssh client. Some clients (e.g. the one from MacOSX) set values that are not valid. You should overwrite LC_CTYPE and set it to a valid locale value like: <code>export LC_CTYPE=de_DE.UTF-8</code>
*:To make this change persistent you can add this line also to your <code>.bashrc</code> file.
*:A list of valid locales can be retrieved via <code>locale -a</code>

* '''Workspace lifetime is expired'''
*:Please see chapter [[Helix/Filesystems#Restoring_expired_Workspaces|"Restoring_expired_Workspaces"]]

* '''How can I restore lost data'''
*:Please have a look at the backup information of the respective file system.

Workspace

2026-03-30T15:19:59Z

H Schumacher: Changed formatting, added some information

'''Workspace tools''' provide temporary scratch space so called '''workspaces''' for your calculation on a central file storage. They are meant to keep data for a limited time. This can be a couple of days or several months.

== No Backup ==

Workspaces are not meant for permanent storage, hence data in workspaces is usually not backed up and may be lost in case of problems on the storage system. Please copy/move important results to $HOME or a storage space outside the cluster.

== Create workspace ==
To create a workspace you need to state the <code>name</code> of your workspace and the <code>lifetime</code> in days:
<syntaxhighlight lang="bash">ws_allocate <name> <lifetime></syntaxhighlight>

The maximum value for ''lifetime'' and a maximum number of renewals is specific to each cluster.

'''Example:''' The command <syntaxhighlight lang="bash">ws_allocate mySpace 30</syntaxhighlight> returns:
<syntaxhighlight lang="bash">
Workspace created. Duration is 720 hours.
Further extensions available: 3
/work/workspace/scratch/username-mySpace-0
</syntaxhighlight>

For more information read the program's help, i.e. ''ws_allocate -h''.

== List all your workspaces ==

To list all your workspaces, execute:

ws_list

which will return:
* Workspace ID
* Workspace location
* available extensions
* creation date and remaining time

To adjust the output, you can check the help: <code>ws_list -h</code>. The option <code>-R</code> for example sorts the workspaces by remaining time.

To list expired workspaces, see [[Workspace#Restore_an_Expired_Workspace|Restore an Expired Workspace]].

== Find workspace location ==

The workspace location/path can be prompted for any workspace ''ID'' using '''ws_find''', in case of workspace ''mySpace'':

ws_find mySpace

returns the one-liner:

/work/workspace/scratch/username-mySpace-0

You can combine the command with the <code>cd</code> command to diretly move there:

cd $(ws_find mySpace)

== Extend lifetime of your workspace ==

Any workspace's lifetime can only be extended a cluster-specific number of times. There are several commands to extend workspace lifetime
#<syntaxhighlight lang="bash">ws_extend mySpace 40
# or
ws_allocate -x mySpace 40</syntaxhighlight> which extends workspace ID ''mySpace'' by ''40'' days from now.
#<syntaxhighlight lang="bash">ws_extend mySpace</syntaxhighlight> which extends workspace ID ''mySpace'' by the number of days used previously.

== Setting Permissions for Sharing Files ==
The examples will assume you want to change the directory $DIR. If you want to share a workspace, DIR could be set with <code>DIR=$(ws_find my_workspace)</code>

=== Workspace Tools ===

* ws_share allows to share an existing workspace with other users by providing read access:
*: <syntaxhighlight lang="bash"># Share/unshare workspace
ws_share <share / unshare> <workspacename> <username> [username2]
# Remove read access from all users that the workspace was shared with
ws_share unshare-all <workspacename>
# List all users that were granted read access
ws_share list <workspacename>
</syntaxhighlight>
* Grant read and write access to a group (RV, SV, ...) when creating a new workspace. (Only works with newer versions of the workspace tools.)
*: <syntaxhighlight lang="bash">ws_allocate -G <groupname> <workspacename> <duration></syntaxhighlight>
*: e.g. for the group bw16e001 and for the workspace commonfiles for 30 days:
*: <syntaxhighlight lang="bash">ws_allocate -G bw16e001 commonfiles 30</syntaxhighlight>
*: Essentially, this tool sets regular unix rwx permissions for the group plus the "suid" bit on the directory to make the permission inheritable.

=== Regular Unix Permissions ===

Making workspaces readable/writable using standard unix access rights with <tt>chmod</tt> is only feasible if you and your co-workers share a common unix group (for example "bw22a222" or sd22a222).

Do '''not''' make files readable or even writable to everyone or to large common groups ("all students").

You can see available groups with the command <code>groups</code>.

{| class="wikitable"
|-
!style="width:45%" | Command
!style="width:55%" | Action
|-
|<tt>chgrp -R bw16e001 "$DIR"</tt>
<tt>chmod -R g+rX "$DIR"</tt>
|Set group ownership and grant read access to group for files in workspace via unix rights to the group "bw16e001" (has to be re-done if files are added)
|-
|<tt>chgrp -R bw16e001 "$DIR"</tt>
<tt>chmod -R g+rswX "$DIR"</tt>
|Set group ownership and grant read/write access to group for files in workspace via unix rights (has to be re-done if files are added). Group will be inherited by new files, but rights for the group will have to be re-set with chmod for every new file
|-
|}

Options used:
* <tt>-R</tt>: recursive
* <tt>g+rwx</tt>
** <tt>g</tt>: group
** <tt>+</tt>: add permissions (- to remove)
** <tt>rwx</tt>: read, write, execute

=== ACLs: Access Control Lists ===
ACLs allow a much more detailed distribution of permissions but are a bit more complicated and not visible in detail via "ls". They have the additional advantage that you can set a "default" ACL for a directory, (with a <tt>-d</tt> flag or a <tt>d:</tt> prefix) which will cause all newly created files to inherit the ACLs from the directory. Regular unix permissions only have limited support (only group ownership, not access rights) for this via the suid bit.

Best practices with respect to ACL usage:
# Take into account that ACLs take precedence over standard unix access rights
# The owner of a workspace is responsible for its content and management

Please note that <tt>ls</tt> (List directory contents) shows ACLs on directories and files only when run as <tt>ls -l</tt> as in long format, as "plus" sign after the standard unix access rights.

Examples with regard to "my_workspace":
{| class="wikitable"
|-
!style="width:45%" | Command
!style="width:55%" | Action
|-
|<tt>getfacl "$DIR"</tt>
|List access rights on $DIR
|-
|<tt>setfacl -Rm user:fr_xy1:rX,default:user:fr_xy1:rX "$DIR"</tt>
|Grant user "fr_xy1" read-only access to $DIR
|-
|<tt>setfacl -R -m user:fr_me0000:rwX,default:user:fr_me0000:rwX "$DIR"</tt>
<tt>setfacl -R -m user:fr_xy1:rwX,default:user:fr_xy1:rwX "$DIR"</tt>
|Grant your own user "fr_me0000" and "fr_xy1" inheritable ("default") read and write access to $DIR, so you can also read/write files put into the workspace by a coworker
|-
|<tt>setfacl -Rm group:bw16e001:rX,default:group:bw16e001:rX "$DIR"</tt>
|Grant group (Rechenvorhaben) "bw16e001" read-only access to $DIR
|-
|<tt>setfacl -Rb "$DIR"</tt>
|Remove all ACL rights. Standard Unix access rights apply again.
|}

Options used:
* <tt>-R</tt>: recursive
* <tt>-m</tt>: modify
* <tt>user:username:rwX user</tt>: next name is a user; rwX read, write, eXecute (only where execute is set for user)
* <tt>default:[user|group]</tt>: set the default for user or group for new files or dierctories

== Delete a Workspace ==
<syntaxhighlight lang="bash"># Manually erase your workspace mySpace
ws_release mySpace</syntaxhighlight>

Note: workspaces are kept for some time after release. To immediately delete and free space e.g. for quota reasons, delete the files with <tt>rm</tt> before release.

Newer versions of workspace tools have a <tt>--delete-data</tt> flag that immediately deletes data. Note that deleted data from workspaces is permanently lost.

== Restore an Expired Workspace ==

For a certain (system-specific) grace time following workspace expiration, a workspace can be restored by performing the following steps:

(1) Display restorable workspaces.
ws_restore -l

(2) Create a new workspace as the target for the restore:
ws_allocate <new_workspace> <lifetime>

(3) Restore:
ws_restore <full_name_of_expired_workspace> <new_workspace>

The expired workspace has to be specified using the '''full name''', including username prefix and timestamp suffix (otherwise, it cannot be uniquely identified).
The target workspace, on the other hand, must be given with just its short name as listed by <code>ws_list</code>, without the username prefix.

In most cases: If the workspace is not visible/restorable, it has been '''permanently deleted''' and cannot be restored, not even by us. Please always remember, that workspaces are intended solely for temporary work data, and there is no backup of data in the workspaces.
Please see the workspace page of your specific cluster in case your cluster does have some kind of backup.

== Troubleshooting ==
* '''Error: "Disk space exceeded"'''
*: This means that there is not enough space (e.g. in your $HOME) to execute the operation. Please [[FAQ#quota_exceeded | free up space]].

FAQ

2026-03-30T15:18:38Z

H Schumacher: New formatting; partially added content from NEMO/FAQ and Helix pages

== Login Issues ==

=== Can't log in ===

* If the registry/account info shows 'Status: LOST_ACCESS', you need to fulfill the requirements outlined in → [[Registration]]:
** Entitlement
** bwUniCluster → [[BwUniCluster_2.0_User_Access#Step_C:_Fill_out_the_bwUniCluster_questionnaire|Check if you answered the questionnaire.]]
** bwForCluster → You must be an active member in an active RV. You can see your RV memberships at the central application site ZAS under [https://zas.bwhpc.de/shib/en/info_rv.php '''My RVs'''].
* Double-check your [[Registration/Login/Username|username]] and [[Registration/Password|password]].
* Make sure your CAPS LOCK key is not enabled.
* Try [[Registration/Password|resetting your password]].
* Ensure your [[Registration/Account|account]] is not locked or expired.
* Are you within the university network (Universities in Baden-Württemberg, BELWÜ)?
* Send us the output of the following command:
ssh -l [[Registration/Login/Username|username]] [[Registration/Login/Hostname|hostname]] -vvv
# Example:
ssh -l fr_ab1234 login.nemo.uni-freiburg.de -vvv

=== Forgot password ===

Set a new password for the service → [[Registration/Password|Password Guide]]

=== Lost phone with OTP token ===

You can use your backup TAN list and add a new token. If you don't have one, contact the support.

== File Systems and Workspaces ==
=== File system quota exceeded ===

There are quota limits for:
* disk space
* number of inodes (files)
Identifiy the issue:
* Disk space: List all directories (and files) in your home directory or workspace sorted by size. (<tt>.[!.]*</tt> makes sure the "hidden" dot files are also checked.)
*: <syntaxhighlight lang="bash">du -sh .[!.]* * | sort -h | tail
# or
du -sh --apparent-size * .[^.]*
</syntaxhighlight>
*: Descend in a directory with a large reported size and repeat until you find large files that are worth taking care of.
* Number of inodes:
*: <syntaxhighlight lang="bash">
# check current directory
ls -1a | wc -l
# check subdirectories
du -s --inodes .[!.]* * | sort -n</syntaxhighlight>
To free up space, you can:
* ...delete temporary files, for example in the <code>.cache</code> folder. Other folders starting with a dot might contain temporary files as well, for example <code>.local/share/Trash</code>.
* ...use the cleaning commands by your used software (e.g. [https://wiki.bwhpc.de/e/Development/Conda#Create_Environments_and_Install_Software conda]).
* ...delete folders with temporary data created by your used software (e.g. nextflow).

Helix/bwVisu/JupyterLab

2026-03-19T09:33:23Z

H Schumacher: fix for pip installations in shared envs

[https://jupyter.org/ JupyterLab] is an integrated development environment (IDE) that provides a flexible and scalable interface for the Jupyter Notebook system. It supports interactive data science and scientific computing across over 40 programming languages (including Python, Julia, and R).

== Change Python Version ==

The default python version can be seen by running <code>python --version</code> in the terminal.

A different python version can be installed into a new virtual environment and then registered as IPython kernel for the usage in JupyterLab. This is explained in the chapter [[#Add_packages_via_conda_environments | add packages via conda environments]].

== Install Python Packages ==

Python packages can be added by installing them into a virtual environment and then creating an IPython kernel from the virtual environment. 
Kernels can be shared. See the notes below. 
If you want to move a virtual environment, it is adivsed to recreate it in the new place. Otherwise, dependencies based on relative paths will break.

# Create a virtual environment with...
#* [[#Add_packages_via_venv_virtual_environments | ...venv]] or
#* [[#Add_packages_via_Conda_virtual_environments | ...conda]] (choose this option if you want to install a different python version) or
#* ...[https://docs.astral.sh/uv/getting-started/ uv] if you want to install a different python version but don't want to use conda.
# [[#Create_an_IPython_Kernel | Create an IPython kernel]] from the virtual environment
# Use the kernel within JupyterLab
#* By default new kernels are saved under <code>~/.local/share/jupyter</code> and this location is automatically detected. Therefore, new kernels are directly available.
#*:[[File:BwVisu JuypterLab KernelPath.png|In the JupyterLab job configuration form, a custom kernel path can be provided.|right|thumb|x150px]]
#* If the kernel is saved somewhere else, the path can be provided in the "Kernel path" field when configuring the JupyterLab job (see image). For a kernel placed under <code>path_to_parent_dir/share/jupyter/kernels/my_kernel</code> the needed "Kernel path" would be <code>path_to_parent_dir/share/jupyter</code>.
# When the kernel is used the first time, the file <code>notebook_secrets</code> is created automatically. It can be found under "Kernel path". For others to use the kernel, they must have read access to this file. The command <code>chmod 750 notebook_secrets</code> would for example allow the whole SDS@hd SV read access. For the access management in workspaces, please see [[Workspace#Setting_Permissions_for_Sharing_Files| Wokspace Permissions]].

Notes regarding the sharing of IPython kernels
* The virtual environment and the kernel need to be placed in a shared directory. For example at SDS@hd.
* There could be a subdirectory for the virtual environments and one for the kernels.
* The path to the kernels is saved in the environment variable $JUPYTER_DATA_DIR. Jupyter relevant paths can be seen with <code>jupyter --paths</code>

=== Add packages via ''venv'' virtual environments ===
More information about venv or other python virtual environments can be found at the [[Development/Python | Python]] page.

Steps for creating a '''venv''' virtual environment:
<ol style="list-style-type: decimal;">
<li>Open a terminal.</li>
<li>Create a new virtual evironment:
<syntaxhighlight lang="bash">python3 -m venv <env_parent_dir>/<env_name></syntaxhighlight>
* <code>env_parent_dir</code> is the path to the folder where the virtual environment shall be created. Relative paths can be used.
* Caution: If you you want to share the environment with others, make sure to already create it in the shared place.
</li>
<li>Activate the environment:
<syntaxhighlight lang="bash">source <env_parent_dir>/<env_name>/bin/activate</syntaxhighlight></li>
<li>Update pip and install packages:
<syntaxhighlight lang="bash">pip install -U pip --no-user
# when the environment is installed in home
pip install <packagename>
# when the environment is installed somewhere else and shall not have dependencies in home so that others can access it as well
pip install <packagename> --no-user</syntaxhighlight></li>
<li> [[#Create_an_IPython_Kernel | Create an IPython kernel]]</li>
</ol>

=== Add packages via Conda virtual environments ===
More information about using conda can be found at the [[Development/Conda | Conda]] page.

<ol style="list-style-type: decimal;">
<li>Load the miniforge module by clicking first on the blue hexagon icon on the left-hand side of Jupyter's start page and then on the "load" button right of the entry for miniforge in the software module menu.
<li>Open a terminal.</li>
<li>Create a new virtual environment:
* If you are the only person using the environment, you can install it in your home directory:
<syntaxhighlight lang="bash">conda create --name <env_name> python=<python version></syntaxhighlight>
* If you want to install it into a different directory, for example a shared place:
<syntaxhighlight lang="bash">conda create --prefix <path_to_shared_directory>/<env_name> python=<python version>
</syntaxhighlight>
</li>
<li>Activate your environment:
<syntaxhighlight lang="bash">conda activate <myenv></syntaxhighlight></li>
<li>Install your packages:
<syntaxhighlight lang="bash">conda install <mypackage></syntaxhighlight></li>
<li> [[#Create_an_IPython_Kernel | Create an IPython kernel]]</li>
</ol>

== Create an IPython Kernel ==

Python kernels are implementations of the Jupyter notebook environment for different languages or virtual environments. You can switch between kernels easily, allowing you to use the best tool for a specific task.
conda_kernels

=== Create a kernel from a virtual environment ===

<ol>
<li>Activate the virtual environment.</li>
<li>Install the ipykernel package:
<syntaxhighlight lang="bash">pip install ipykernel</syntaxhighlight></li>
<li>Register the virtual environment as custom kernel to Jupyter.
* If you are the only person using the environment:
<syntaxhighlight lang="bash">python3 -m ipykernel install --user --name=<kernel_name></syntaxhighlight>
The kernel can be found under <code>~/.local/share/jupyter/kernels/</code>.
* If you installed the environment in a shared place and want to have the kernel there as well:
<syntaxhighlight lang="bash">python3 -m ipykernel install --prefix <path_to_kernel_folder> --name=<kernel_name></syntaxhighlight>
The kernel can then be found under <code>path_to_kernel_folder/share/jupyter/kernels/<kernel_name></code>. As long as the same <code>path_to_kernel_folder</code> is used, all kernels will be saved next to each other in "kernels".
</li>
</ol>

=== Multi-Language Support ===

JupyterLab supports over 40 programming languages including Python, R, Julia, and Scala. This is achieved through the use of different kernels.

==== R Kernel ====

<ol style="list-style-type: decimal;">
<li>On the cluster:
<pre>$ module load math/R
$ R
> install.packages('IRkernel')</pre></li>
<li>On bwVisu:
<ol style="list-style-type: decimal;">
<li>Start Jupyter App</li>
<li>In left menu: load math/R</li>
<li>Open Console:</li>

<pre>$ R
> IRkernel::installspec(displayname = 'R 4.2')</pre>

<li>Start kernel 'R 4.2' as console or notebook</li>
</ol>
</ol>

==== Julia Kernel ====

Load the math/julia module. Open the Terminal.

<pre>
julia
]
add IJulia
</pre>

After that, Julia is available as a kernel.

== Interactive Widgets ==

JupyterLab supports interactive widgets that can create UI controls for interactive data visualization and manipulation within the notebooks. Example of using an interactive widget:

<pre class="{.python">from ipywidgets import IntSlider
slider = IntSlider()
display(slider)</pre>
These widgets can be sliders, dropdowns, buttons, etc., which can be connected to Python code running in the backend.

== FAQ ==

<ol><li>'''My conda commands are interrupted with message 'Killed'.'''
Request more memory when starting Jupyter.</li>
<li>'''How can I navigate to my SDS@hd folder in the file browser?'''
Open a terminal and set a symbolic link to your SDS@hd folder in your home directory. For example:</li>
<pre>cd $HOME
mkdir sds-hd
cd sds-hd
ln -s /mnt/sds-hd/sd16a001 sd16a001</pre>
<li>'''Jupyterlab doesn't let me in but asks for a password.'''
Try using more memory for the job. If this doesn't help, try using the inkognito mode of your browser as the browser cache might be the problem.</li>
<li>'''I prefer VSCode over JupyterLab. Can I start a JupyterLab job and then connect with it via VSCode?'''
This is not possible. Please start the job directly on Helix instead. You can find the instructions at the [[Development/VS_Code#Connect_to_Remote_Jupyter_Kernel | VSCode page]].</li>
</ol>

SDS@hd/Access

2026-03-18T11:21:00Z

H Schumacher: fixed typo

This page provides an overview on how to access data served by SDS@hd. To get an introduction to data transfer in general, see [[Data_Transfer|data transfer]].

'''Notation:''' Please replace all placeholder contents within guillemets <code>< ></code> with the actual value. For example, replace ''<name_of_my_dog>'' with ''Fluffy'' .

== Prerequisites ==

* You need to be [[SDS@hd/Registration|registered]].
* You need to be in the belwue-Network. This means you have to use the VPN Service of your HomeOrganization, if you want to access SDS@hd from outside the bwHPC-Clusters (e.g. via eduroam or from your personal notebook).

== Needed Information, independent of the chosen tool ==

* Username: You can look it up under your user data at [https://bwservices.uni-heidelberg.de/user/index.xhtml bwServices]. The username is ''<localPrimaryGroup>_<eduPersonPrincipalName>'' . (It is the same as for the bwHPC Clusters.)
* Password: The Service Password that you set at bwServices in the [[SDS@hd/Registration|registration step]].
* SV-Acronym: Use the lower case version of the acronym for all access options.
* Hostname: The hostname depends on the chosen network protocol:
** For [[Data_Transfer/SSHFS|SSHFS]] and [[Data_Transfer/SFTP|SFTP]]: ''lsdf02-sshfs.urz.uni-heidelberg.de''
** For [[SDS@hd/Access/SMB|SMB]] and [[SDS@hd/Access/NFS|NFS]]: ''lsdf02.urz.uni-heidelberg.de''
** For [[Data_Transfer/WebDAV|WebDAV]] the url is: ''https://lsdf02-webdav.urz.uni-heidelberg.de''

== Recommended Setup ==
The following graphic shows the recommended way for accessing SDS@hd via Windows/Mac/Linux. The table provides an overview of the most important access options and links to the related pages. 
If you have various use cases, it is recommended to use [[Data_Transfer/Rclone|Rclone]]. You can copy, sync and mount with it. Thanks to its multithreading capability Rclone is a good fit for transferring big data. 
For an overview of all connection possibilities, please have a look at [[Data_Transfer/All_Data_Transfer_Routes|all data transfer routes]].

[[File:Data_transfer_diagram_simple.jpg|center|500px]]
Figure 1: SDS@hd main transfer routes

{| class="wikitable"
|- style="font-weight:bold; text-align:center; vertical-align:middle;"
!
! Use Case
! Windows
! Mac
! Linux
! Possible Bandwith
! Firewall Ports
|-
| [[Data_Transfer/Rclone|Rclone]] + <protocol>
| copy, sync and mount, multithreading
| ✓
| ✓
| ✓
| depends on used protocol
| depends on used protocol
|-
| [[SDS@hd/Access/SMB|SMB]]
| mount as network drive in file explorer or usage via Rclone
| [[SDS@hd/Access/SMB#Windows|✓]]
| [[SDS@hd/Access/SMB#Mac|✓]]
| [[SDS@hd/Access/SMB#Linux|✓]]
| up to 40 Gbit/sec
| 139 (netbios), 135 (rpc), 445 (smb)
|-
| [[Data_Transfer/WebDAV|WebDAV]]
| go to solution for restricted networks
| [✓]
| ✓
| ✓
| up to 100GBit/sec
| 80 (http), 443 (https)
|- style="vertical-align:middle;"
| [[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]]
| Graphical User Interface (GUI)
| [[Data_Transfer/Graphical_Clients#MobaXterm|✓]]
| ☓
| ☓
| see sftp
| see sftp
|- style="vertical-align:middle;"
| [[SDS@hd/Access/NFS|NFS]]
| mount for multi-user environments
| ☓
| ☓
| [[SDS@hd/Access/NFS|✓]]
| up to 40 Gbit/sec
| -
|- style="vertical-align:middle;"
| [[Data_Transfer/SSHFS|SSHFS]]
| mount, needs stable internet connection
| ☓
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| see sftp
| see sftp
|- style="vertical-align:middle;"
| [[Data_Transfer/SFTP|SFTP]]
| interactive shell, better usability when used together with Rclone
| [[Data_Transfer/SFTP#Windows|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| up to 40 Gbit/sec
| 22 (ssh)
|}
Table 1: SDS@hd transfer routes

=== Access from a bwHPC Cluster ===

'''bwUniCluster''' 
You can't mount to your $HOME directory but you can create a mount under $TMPDIR by following the instructions for [[Data_Transfer/Rclone#Usage_Rclone_Mount | Rclone mount]]. It is advised to wait a couple of seconds (<code>sleep 5</code>) before trying to use the mounted directory.

'''bwForCluster Helix''' 
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes.

'''bwForCluster BinAC 2''' 
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes. The prerequisites are:
* The SV responsible has enabled the SV on BinAC 2 once by writing to [mailto:sds-hd-support@urz.uni-heidelberg.de sds-hd-support@urz.uni-heidelberg.de]
* You have a valid kerberos ticket, which can be fetched with <code>kinit <userID></code>

'''Other''' 
You can mount your SDS@hd SV on the cluster yourself by using [[Data_Transfer/Rclone | Rclone]] with the [[Data_Transfer/Rclone#Usage_Rclone_Mount | Rclone mount]] command. As transfer protocol you can use WebDAV or sftp. For a full overview please have a look at [[Data_Transfer/All_Data_Transfer_Routes | All Data Transfer Routes]].

=== Access via Webbrowser (read-only) ===

Visit [https://lsdf02-webdav.urz.uni-heidelberg.de/ lsdf02-webdav.urz.uni-heidelberg.de] and login with your SDS@hd username and service password. Here you can get an overview of the data in your "Speichervorhaben" and download single files. To be able to do more, like moving data, uploading new files, or downloading complete folders, a suitable client is needed as described above.

== Best Practices ==

* '''Managing access rights''' -> You can adjust the access rights when the SV is mounted on Windows in the file/folder properties menu (users from Heidelberg without Windows can use the [https://www.urz.uni-heidelberg.de/de/service-katalog/desktop-und-arbeitsplatz/windows-terminalserver Windows terminal server]). bwForCluster Helix users can change the access rights via Helix by using [[Workspace#Regular_Unix_Permissions | unix permissions]] or [[Workspace#ACLs:_Access_Control_Lists | access control lists]]. Otherwise, ACL changes won't work when used locally on a mounted directory. Additionally to the already mentioned options, the SVV can take over ownership of files by opening a sftp shell and running <code>chown -R <userID> <path/to/folder></code> .

* '''Multiuser environment''' -> Use [[SDS@hd/Access/NFS|NFS]]

== Troubleshooting ==

Maintance windows and known issues are communicated via the email list or the News section at the SDS@hd start page in the wiki.

* '''Issue:''' The credentials aren't accepted
*: Check if your credentials work in general by trying the access via webbrowser: [https://lsdf02-webdav.urz.uni-heidelberg.de/].
*: If this doesn't work:
*:: → [[Registration/Login/Username| Check your Username]]
*:: → [[Registration/bwForCluster/Helix#Troubleshooting_with_the_Help_of_bwServices | Troubleshooting with bwServices]]
* '''Other Issue'''
** If available, follow the troubleshooting guide of your specific connection method.
** Make sure to not use LAN and WLAN at the same time to prevent connection problems.
** If you have an institutional account, make sure to be a fully active member of your institution.
If these suggestions didn't help, write to the [mailto:sds-hd-support@urz.uni-heidelberg.de support]. Provide the following information:
* Your operating system
* Does the access via webbrowser work?
*: If yes, provide us with detailed information on how you tried to access your SV (used access method, username, ...).

SDS@hd/Access

2026-03-17T23:32:37Z

H Schumacher: Adjusted managing access rights section

This page provides an overview on how to access data served by SDS@hd. To get an introduction to data transfer in general, see [[Data_Transfer|data transfer]].

'''Notation:''' Please replace all placeholder contents within guillemets <code>< ></code> with the actual value. For example, replace ''<name_of_my_dog>'' with ''Fluffy'' .

== Prerequisites ==

* You need to be [[SDS@hd/Registration|registered]].
* You need to be in the belwue-Network. This means you have to use the VPN Service of your HomeOrganization, if you want to access SDS@hd from outside the bwHPC-Clusters (e.g. via eduroam or from your personal notebook).

== Needed Information, independent of the chosen tool ==

* Username: You can look it up under your user data at [https://bwservices.uni-heidelberg.de/user/index.xhtml bwServices]. The username is ''<localPrimaryGroup>_<eduPersonPrincipalName>'' . (It is the same as for the bwHPC Clusters.)
* Password: The Service Password that you set at bwServices in the [[SDS@hd/Registration|registration step]].
* SV-Acronym: Use the lower case version of the acronym for all access options.
* Hostname: The hostname depends on the chosen network protocol:
** For [[Data_Transfer/SSHFS|SSHFS]] and [[Data_Transfer/SFTP|SFTP]]: ''lsdf02-sshfs.urz.uni-heidelberg.de''
** For [[SDS@hd/Access/SMB|SMB]] and [[SDS@hd/Access/NFS|NFS]]: ''lsdf02.urz.uni-heidelberg.de''
** For [[Data_Transfer/WebDAV|WebDAV]] the url is: ''https://lsdf02-webdav.urz.uni-heidelberg.de''

== Recommended Setup ==
The following graphic shows the recommended way for accessing SDS@hd via Windows/Mac/Linux. The table provides an overview of the most important access options and links to the related pages. 
If you have various use cases, it is recommended to use [[Data_Transfer/Rclone|Rclone]]. You can copy, sync and mount with it. Thanks to its multithreading capability Rclone is a good fit for transferring big data. 
For an overview of all connection possibilities, please have a look at [[Data_Transfer/All_Data_Transfer_Routes|all data transfer routes]].

[[File:Data_transfer_diagram_simple.jpg|center|500px]]
Figure 1: SDS@hd main transfer routes

{| class="wikitable"
|- style="font-weight:bold; text-align:center; vertical-align:middle;"
!
! Use Case
! Windows
! Mac
! Linux
! Possible Bandwith
! Firewall Ports
|-
| [[Data_Transfer/Rclone|Rclone]] + <protocol>
| copy, sync and mount, multithreading
| ✓
| ✓
| ✓
| depends on used protocol
| depends on used protocol
|-
| [[SDS@hd/Access/SMB|SMB]]
| mount as network drive in file explorer or usage via Rclone
| [[SDS@hd/Access/SMB#Windows|✓]]
| [[SDS@hd/Access/SMB#Mac|✓]]
| [[SDS@hd/Access/SMB#Linux|✓]]
| up to 40 Gbit/sec
| 139 (netbios), 135 (rpc), 445 (smb)
|-
| [[Data_Transfer/WebDAV|WebDAV]]
| go to solution for restricted networks
| [✓]
| ✓
| ✓
| up to 100GBit/sec
| 80 (http), 443 (https)
|- style="vertical-align:middle;"
| [[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]]
| Graphical User Interface (GUI)
| [[Data_Transfer/Graphical_Clients#MobaXterm|✓]]
| ☓
| ☓
| see sftp
| see sftp
|- style="vertical-align:middle;"
| [[SDS@hd/Access/NFS|NFS]]
| mount for multi-user environments
| ☓
| ☓
| [[SDS@hd/Access/NFS|✓]]
| up to 40 Gbit/sec
| -
|- style="vertical-align:middle;"
| [[Data_Transfer/SSHFS|SSHFS]]
| mount, needs stable internet connection
| ☓
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| see sftp
| see sftp
|- style="vertical-align:middle;"
| [[Data_Transfer/SFTP|SFTP]]
| interactive shell, better usability when used together with Rclone
| [[Data_Transfer/SFTP#Windows|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| up to 40 Gbit/sec
| 22 (ssh)
|}
Table 1: SDS@hd transfer routes

=== Access from a bwHPC Cluster ===

'''bwUniCluster''' 
You can't mount to your $HOME directory but you can create a mount under $TMPDIR by following the instructions for [[Data_Transfer/Rclone#Usage_Rclone_Mount | Rclone mount]]. It is advised to wait a couple of seconds (<code>sleep 5</code>) before trying to use the mounted directory.

'''bwForCluster Helix''' 
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes.

'''bwForCluster BinAC 2''' 
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes. The prerequisites are:
* The SV responsible has enabled the SV on BinAC 2 once by writing to [mailto:sds-hd-support@urz.uni-heidelberg.de sds-hd-support@urz.uni-heidelberg.de]
* You have a valid kerberos ticket, which can be fetched with <code>kinit <userID></code>

'''Other''' 
You can mount your SDS@hd SV on the cluster yourself by using [[Data_Transfer/Rclone | Rclone]] with the [[Data_Transfer/Rclone#Usage_Rclone_Mount | Rclone mount]] command. As transfer protocol you can use WebDAV or sftp. For a full overview please have a look at [[Data_Transfer/All_Data_Transfer_Routes | All Data Transfer Routes]].

=== Access via Webbrowser (read-only) ===

Visit [https://lsdf02-webdav.urz.uni-heidelberg.de/ lsdf02-webdav.urz.uni-heidelberg.de] and login with your SDS@hd username and service password. Here you can get an overview of the data in your "Speichervorhaben" and download single files. To be able to do more, like moving data, uploading new files, or downloading complete folders, a suitable client is needed as described above.

== Best Practices ==

* '''Managing access rights''' -> You can adjust the access rights when the SV is mounted on Windows in the file/folder properties menu (Heidelberg users without Windows can use the [https://www.urz.uni-heidelberg.de/de/service-katalog/desktop-und-arbeitsplatz/windows-terminalserver Windows terminal server]. Helix users can change the access rights via Helix by using [[Workspace#Regular_Unix_Permissions | unix permissions]] or [[Workspace#ACLs:_Access_Control_Lists | access control lists]]. Otherwise ACL changes won't work when used locally on a mounted directory. Additionally to the already mentioned options, the SVV can take over ownership of files by opening a sftp shell and running <code>chown -R <userID> <path/to/folder></code> .

* '''Multiuser environment''' -> Use [[SDS@hd/Access/NFS|NFS]]

== Troubleshooting ==

Maintance windows and known issues are communicated via the email list or the News section at the SDS@hd start page in the wiki.

* '''Issue:''' The credentials aren't accepted
*: Check if your credentials work in general by trying the access via webbrowser: [https://lsdf02-webdav.urz.uni-heidelberg.de/].
*: If this doesn't work:
*:: → [[Registration/Login/Username| Check your Username]]
*:: → [[Registration/bwForCluster/Helix#Troubleshooting_with_the_Help_of_bwServices | Troubleshooting with bwServices]]
* '''Other Issue'''
** If available, follow the troubleshooting guide of your specific connection method.
** Make sure to not use LAN and WLAN at the same time to prevent connection problems.
** If you have an institutional account, make sure to be a fully active member of your institution.
If these suggestions didn't help, write to the [mailto:sds-hd-support@urz.uni-heidelberg.de support]. Provide the following information:
* Your operating system
* Does the access via webbrowser work?
*: If yes, provide us with detailed information on how you tried to access your SV (used access method, username, ...).

Python

2026-03-02T16:20:03Z

H Schumacher: New formatting, small text changes

Overview of pages related to the programming language Python.

→ '''[[Development/Python | Python Main Page]]'''

'''Cluster-specific pages:'''
* [[JUSTUS2/Software/Python | Python on JUSTUS 2]]

'''Instructions for JupyterLab on...'''
* [[BwUniCluster3.0/Jupyter | bwUniCluster3.0 ]]
* [[Helix/bwVisu/JupyterLab | Helix]] (through [[Helix/bwVisu | bwVisu]])
* [[BinAC2/Software/Jupyterlab | BinAC2]] (as software module)

'''Related software packages'''
* [[BwUniCluster3.0/Software/Python Dask | Dask]] (on bwUniCluster3.0)
* [[Development/ollama | ollama ]]

'''Training'''

* [https://github.com/hpcraink/workshop-parallel-jupyter Workshop - Parallel Python via Jupyter]

Helix/bwVisu/JupyterLab

2026-03-02T15:02:00Z

H Schumacher: Added fourth step to the kernel sharing instructions to fix permission issues.

[https://jupyter.org/ JupyterLab] is an integrated development environment (IDE) that provides a flexible and scalable interface for the Jupyter Notebook system. It supports interactive data science and scientific computing across over 40 programming languages (including Python, Julia, and R).

== Change Python Version ==

The default python version can be seen by running <code>python --version</code> in the terminal.

A different python version can be installed into a new virtual environment and then registered as IPython kernel for the usage in JupyterLab. This is explained in the chapter [[#Add_packages_via_conda_environments | add packages via conda environments]].

== Install Python Packages ==

Python packages can be added by installing them into a virtual environment and then creating an IPython kernel from the virtual environment. 
Kernels can be shared. See the notes below. 
If you want to move a virtual environment, it is adivsed to recreate it in the new place. Otherwise, dependencies based on relative paths will break.

# Create a virtual environment with...
#* [[#Add_packages_via_venv_virtual_environments | ...venv]] or
#* [[#Add_packages_via_Conda_virtual_environments | ...conda]] (choose this option if you want to install a different python version) or
#* ...[https://docs.astral.sh/uv/getting-started/ uv] if you want to install a different python version but don't want to use conda.
# [[#Create_an_IPython_Kernel | Create an IPython kernel]] from the virtual environment
# Use the kernel within JupyterLab
#* By default new kernels are saved under <code>~/.local/share/jupyter</code> and this location is automatically detected. Therefore, new kernels are directly available.
#*:[[File:BwVisu JuypterLab KernelPath.png|In the JupyterLab job configuration form, a custom kernel path can be provided.|right|thumb|x150px]]
#* If the kernel is saved somewhere else, the path can be provided in the "Kernel path" field when configuring the JupyterLab job (see image). For a kernel placed under <code>path_to_parent_dir/share/jupyter/kernels/my_kernel</code> the needed "Kernel path" would be <code>path_to_parent_dir/share/jupyter</code>.
# When the kernel is used the first time, the file <code>notebook_secrets</code> is created automatically. It can be found under "Kernel path". For others to use the kernel, they must have read access to this file. The command <code>chmod 750 notebook_secrets</code> would for example allow the whole SDS@hd SV read access. For the access management in workspaces, please see [[Workspace#Setting_Permissions_for_Sharing_Files| Wokspace Permissions]].

Notes regarding the sharing of IPython kernels
* The virtual environment and the kernel need to be placed in a shared directory. For example at SDS@hd.
* There could be a subdirectory for the virtual environments and one for the kernels.
* The path to the kernels is saved in the environment variable $JUPYTER_DATA_DIR. Jupyter relevant paths can be seen with <code>jupyter --paths</code>

=== Add packages via ''venv'' virtual environments ===
More information about venv or other python virtual environments can be found at the [[Development/Python | Python]] page.

Steps for creating a '''venv''' virtual environment:
<ol style="list-style-type: decimal;">
<li>Open a terminal.</li>
<li>Create a new virtual evironment:
<syntaxhighlight lang="bash">python3 -m venv <env_parent_dir>/<env_name></syntaxhighlight>
* <code>env_parent_dir</code> is the path to the folder where the virtual environment shall be created. Relative paths can be used.
* Caution: If you you want to share the environment with others, make sure to already create it in the shared place.
</li>
<li>Activate the environment:
<syntaxhighlight lang="bash">source <env_parent_dir>/<env_name>/bin/activate</syntaxhighlight></li>
<li>Update pip and install packages:
<syntaxhighlight lang="bash">pip install -U pip
pip install <packagename></syntaxhighlight></li>
<li> [[#Create_an_IPython_Kernel | Create an IPython kernel]]</li>
</ol>

=== Add packages via Conda virtual environments ===
More information about using conda can be found at the [[Development/Conda | Conda]] page.

<ol style="list-style-type: decimal;">
<li>Load the miniforge module by clicking first on the blue hexagon icon on the left-hand side of Jupyter's start page and then on the "load" button right of the entry for miniforge in the software module menu.
<li>Open a terminal.</li>
<li>Create a new virtual environment:
* If you are the only person using the environment, you can install it in your home directory:
<syntaxhighlight lang="bash">conda create --name <env_name> python=<python version></syntaxhighlight>
* If you want to install it into a different directory, for example a shared place:
<syntaxhighlight lang="bash">conda create --prefix <path_to_shared_directory>/<env_name> python=<python version>
</syntaxhighlight>
</li>
<li>Activate your environment:
<syntaxhighlight lang="bash">conda activate <myenv></syntaxhighlight></li>
<li>Install your packages:
<syntaxhighlight lang="bash">conda install <mypackage></syntaxhighlight></li>
<li> [[#Create_an_IPython_Kernel | Create an IPython kernel]]</li>
</ol>

== Create an IPython Kernel ==

Python kernels are implementations of the Jupyter notebook environment for different languages or virtual environments. You can switch between kernels easily, allowing you to use the best tool for a specific task.
conda_kernels

=== Create a kernel from a virtual environment ===

<ol>
<li>Activate the virtual environment.</li>
<li>Install the ipykernel package:
<syntaxhighlight lang="bash">pip install ipykernel</syntaxhighlight></li>
<li>Register the virtual environment as custom kernel to Jupyter.
* If you are the only person using the environment:
<syntaxhighlight lang="bash">python3 -m ipykernel install --user --name=<kernel_name></syntaxhighlight>
The kernel can be found under <code>~/.local/share/jupyter/kernels/</code>.
* If you installed the environment in a shared place and want to have the kernel there as well:
<syntaxhighlight lang="bash">python3 -m ipykernel install --prefix <path_to_kernel_folder> --name=<kernel_name></syntaxhighlight>
The kernel can then be found under <code>path_to_kernel_folder/share/jupyter/kernels/<kernel_name></code>. As long as the same <code>path_to_kernel_folder</code> is used, all kernels will be saved next to each other in "kernels".
</li>
</ol>

=== Multi-Language Support ===

JupyterLab supports over 40 programming languages including Python, R, Julia, and Scala. This is achieved through the use of different kernels.

==== R Kernel ====

<ol style="list-style-type: decimal;">
<li>On the cluster:
<pre>$ module load math/R
$ R
> install.packages('IRkernel')</pre></li>
<li>On bwVisu:
<ol style="list-style-type: decimal;">
<li>Start Jupyter App</li>
<li>In left menu: load math/R</li>
<li>Open Console:</li>

<pre>$ R
> IRkernel::installspec(displayname = 'R 4.2')</pre>

<li>Start kernel 'R 4.2' as console or notebook</li>
</ol>
</ol>

==== Julia Kernel ====

Load the math/julia module. Open the Terminal.

<pre>
julia
]
add IJulia
</pre>

After that, Julia is available as a kernel.

== Interactive Widgets ==

JupyterLab supports interactive widgets that can create UI controls for interactive data visualization and manipulation within the notebooks. Example of using an interactive widget:

<pre class="{.python">from ipywidgets import IntSlider
slider = IntSlider()
display(slider)</pre>
These widgets can be sliders, dropdowns, buttons, etc., which can be connected to Python code running in the backend.

== FAQ ==

<ol><li>'''My conda commands are interrupted with message 'Killed'.'''
Request more memory when starting Jupyter.</li>
<li>'''How can I navigate to my SDS@hd folder in the file browser?'''
Open a terminal and set a symbolic link to your SDS@hd folder in your home directory. For example:</li>
<pre>cd $HOME
mkdir sds-hd
cd sds-hd
ln -s /mnt/sds-hd/sd16a001 sd16a001</pre>
<li>'''Jupyterlab doesn't let me in but asks for a password.'''
Try using more memory for the job. If this doesn't help, try using the inkognito mode of your browser as the browser cache might be the problem.</li>
<li>'''I prefer VSCode over JupyterLab. Can I start a JupyterLab job and then connect with it via VSCode?'''
This is not possible. Please start the job directly on Helix instead. You can find the instructions at the [[Development/VS_Code#Connect_to_Remote_Jupyter_Kernel | VSCode page]].</li>
</ol>

Helix/bwVisu/Usage

2026-02-27T14:21:51Z

H Schumacher: Added small title for sds section

== Basic Navigation ==

bwVisu is quite simple and intuitively designed to make launching apps on the cluster as user-friendly as possible. Once you have successfully logged into bwVisu, the first thing you see is a list of pinned apps. These are the available apps that you can launch as a job on the cluster.

In the navigation bar, the user has the following options:

<code>Files</code> — <code>Interactive Apps</code> — <code>My Interactive Sessions</code> — <code>Shell</code>

=== Files ===

Here, you can access your home directory on Helix. In a graphical user interface, files can be downloaded and uploaded, and new files and folders can be created. You will encounter an error message if you try to see data that you have no access to.

In the Home Directory the results and logs from the apps are stored. You get the following interface:

[[File:BwVisuFiles.png|bwVisu-files|center|x300px]]

'''For SDS@hd users:''' Data on SDS@hd (under /mnt/sds-hd) can only be accessed from within jobs. Therefore, you need to open a terminal (within a JupyterLab or Rstudio job or directly on Helix) and set a symbolic link to your SDS@hd folder in your home directory. For example like this:
<pre>cd $HOME
mkdir sds-hd
cd sds-hd
ln -s /mnt/sds-hd/sd16a001 sd16a001</pre>

=== Interactive Apps ===

This displays the apps that the user can run on the cluster, possibly across multiple nodes. These apps initiate an interactive session which the user can directly connect to and, depending on the app, receive a graphical interface.

=== My Interactive Sessions ===

At bwVisu, it is possible to start several interactive apps simultaneously. Here, one can view an overview of currently running interactive apps and completed sessions. It is also possible to view the log for each session. As soon as one starts an interactive app, one is automatically redirected here. Even if you log out of bwVisu and log back in, you can still see the information and logs from the finished interactive apps here until you delete it.

=== Shell ===
For bwForCluster Helix users only: After entering your OTP and password a shell on the login node of bwForCluster Helix opens. This way, you can for example create workspaces when you need more space. Please look up the details under [[Helix/Filesystems#Workspaces | Helix Workspaces]]. They can't be seen in the File Viewer but are available within the bwVisu jobs.

== Start a Job ==
[[File:bwVisuConcept.png|center|x170px]]
<ol>
<li> At the start page of the web frontend, you can start a new job by clicking on an application version in the list of pinned apps.</li>
<li>You will now see the application description and options to adjust the job resources.</li>
<ul>
<li> To find the best resource settings for your jobs, please have a look at the [https://helix-monitoring.bwservices.uni-heidelberg.de/ bwForCluster Helix monitoring portal] . There, you can see how much of the requested resources where actually used by your jobs.
<li> At the bottom of the page, beneath the <code>Launch</code> button, you can find a link for accessing the logs/session data of all your jobs.</li>
<li>The interface can differ per app because each app can have individual configuration options. </li>
[[File:bwVisuRStudio.png|bwVisu-rstudio|center|x200px]]</li>
</ul>
<li>Click the <code>Launch</code> button to start a new job with this application. You can't start two jobs with the same application. Launching an app can take some time depending on the requested resources and cluster utilization.</li>
<li>The <code>Interactive Sessions</code> page opens with the details of your job.</li>
<ul>
<li>'''Host:''' Indicates on which nodes the application is running.</li>
<li>'''Time Remaining:''' Each application is given a time limit, which can also be set in advance. In this example, the application is supposed to run for one hour.</li>
<li>'''Session ID:''' This is a link that leads to the folder of the session. Among other things, application logs are displayed here, which can be viewed in the UI.
[[File:BwVisuRstudioRun.png|bwVisu-rstudio-run|center|x200px]] </li>
</li>
</ul>
<li> Connect to the job with your browser by clicking <code>connect to <application></code>. </li>
</ol>

== Adjust noVNC Applications ==
<div><ul>
<li style="display: inline-block; vertical-align: middle; float: right;"> [[File:bwVisu Fiji noVNCMenu.png|thumb|none|x200px|noVNC menu within Fiji]] </li>
<li style="display: inline-block; vertical-align: top; float: right;"> [[File:bwVisu_Fiji_Running.png|thumb|none|x200px|Job details view of a running Fiji Job]] </li>
</ul></div>
noVNC is an open-source VNC browser client. Some bwVisu applications use noVNC and therefore, come with additional options in the view of the job details and an additional menu within the running application.

*'''Running Job View''' on the interactive sessions page
** Compression slider: Before opening the application, you can choose the compression level to use. 0 means no compression while 9 offers the best compression but uses more CPU resources on the cluster which can be slower.
** Image Quality slider: Before opening the application, you can choose the image quality level. With this you can control the desired JPEG quality.
** View Only (Share-able Link) button: By clicking this button the application opens in view only mode. You can share this link with collaborators or students while you launch your noVNC application normally for yourself. This way, others can follow what you are doing. For every job you need to generate a new link.
* '''noVNC menu''' within the running application
** When you open the application you can see an additional menu on the left with options. For the ones using the application in view-only mode it is helpful to select the option to show a dot instead of not showing the mouse cursor at all. Therefore go to "Options" and then "Expand".
** For copying data, you need to use the clipboard. When copying something locally, you need to put it into the clipboard before you can use it within the application. The same goes the other way around.
* '''Adjust window size''' and position of the app by executing a right click at the edge of the app window frame, so that a menu shows up.

== Terminate a Job ==

The job will terminate when the job runtime requested in the job submission form has expired To terminate an application yourself, you can click the delete button or end the application in the interactive session. If the app is terminated in the interactive session, the window no longer shows <code>Running</code> but <code>Completed</code>. With the <code>Delete</code> button, you end the session and also remove the list entry from <code>My Interactive Sessions</code>.

== Troubleshooting ==
In case of problems, it can help to look up the related bwForCluster Helix wiki pages.
For example:
* Error: Disk space exceeded → '''[https://wiki.bwhpc.de/e/Helix/Filesystems#Troubleshooting Filesystems: Troubleshooting]'''

Workspace

2026-02-27T12:13:55Z

H Schumacher: Added "groups" command

'''Workspace tools''' provide temporary scratch space so called '''workspaces''' for your calculation on a central file storage. They are meant to keep data for a limited time – but usually longer than the time of a single job run.

== No Backup ==

Workspaces are not meant for permanent storage, hence data in workspaces is not backed up and may be lost in case of problems on the storage system. Please copy/move important results to $HOME or storage space outside the cluster.

== Create workspace ==
To create a workspace you need to state
* ''name'' of your workspace and
* ''lifetime'' in days.
A maximum value for ''lifetime'' and a maximum number of renewals is defined on each cluster. Execution of:

$ ws_allocate mySpace 30

e.g. returns:

Workspace created. Duration is 720 hours.
Further extensions available: 3
/work/workspace/scratch/username-mySpace-0

For more information read the program's help, i.e. ''$ ws_allocate -h''.

== List all your workspaces ==

To list all your workspaces, execute:

$ ws_list

which will return:
* Workspace ID
* Workspace location
* available extensions
* creation date and remaining time

To list expired workspaces, see [[Workspace#Restore_an_Expired_Workspace|Restore an Expired Workspace]].

== Find workspace location ==

Workspace location/path can be prompted for any workspace ''ID'' using '''ws_find''', in case of workspace ''mySpace'':

$ ws_find mySpace

returns the one-liner:

/work/workspace/scratch/username-mySpace-0

== Extend lifetime of your workspace ==

Any workspace's lifetime can be only extended a cluster-specific number of times. There several commands to extend workspace lifetime
#<pre>$ ws_extend mySpace 40</pre> which extends workspace ID ''mySpace'' by ''40'' days from now,
#<pre>$ ws_extend mySpace</pre> which extends workspace ID ''mySpace'' by the number days used previously
#<pre>$ ws_allocate -x mySpace 40</pre> which extends workspace ID ''mySpace'' by ''40'' days from now.
 

== Setting Permissions for Sharing Files ==
The examples will assume you want to change the directory in $DIR. If you want to share a workspace, DIR could be set with <code>DIR=$(ws_find my_workspace)</code>

=== Workspace Tools ===

* ws_share
<code syntax=bash>ws_share share workspacename username</code>

allows you to grant the user username read access to the workspace.

Newer versions of the workspace tools have sharing options to ws_allocate:

* -G option of ws_allocate
<code syntax=bash>ws_allocate -G groupname workspacename duration</code>

e.g. for the group bw16e001 and for the workspace for 30 days:

<code syntax=bash>ws_allocate -G bw16e001 commonfiles 30</code>

** groupname: name of the group you want to share with
** workspacename: what you want to call your workspace
** duration: how long the workspace is supposed to last in days

Essentially this tool sets regular unix rwx permissions for the group plus the "suid" bit on the directory to make the permission inheritable.

=== Regular Unix Permissions ===

Making workspaces world readable/writable using standard unix access rights with <tt>chmod</tt> is only feasible if you are in a research group and you and your co-workers share a common unix group (for example "bw22a222").

Do '''not''' make files readable or even writable to everyone or to large common groups ("all students").
You can see available groups with the command <code>groups</code>.

{| class="wikitable"
|-
!style="width:45%" | Command
!style="width:55%" | Action
|-
|<tt>chgrp -R bw16e001 "$DIR"</tt>
<tt>chmod -R g+rX "$DIR"</tt>
|Set group ownership and grant read access to group for files in workspace via unix rights to the group "bw16e001" (has to be re-done if files are added)
|-
|<tt>chgrp -R bw16e001 "$DIR"</tt>
<tt>chmod -R g+rswX "$DIR"</tt>
|Set group ownership and grant read/write access to group for files in workspace via unix rights (has to be re-done if files are added). Group will be inherited by new files, but rights for the group will have to be re-set with chmod for every new file
|-
|}

Options used:
* -R: recursive
* g+rwx
** g: group
** + add permissions (- to remove)
** rwx: read, write, execute

=== ACLs: Access Control Lists ===
ACLs allow a much more detailed distribution of permissions but are a bit more complicated and not visible in detail via "ls". They have the additional advantage that you can set a "default" ACL for a directory, (with a <tt>-d</tt> flag or a <tt>d:</tt> prefix) which will cause all newly created files to inherit the ACLs from the directory. Regular unix permissions only have limited support (only group ownership, not access rights) for this via the suid bit.

Best practices with respect to ACL usage:
# Take into account that ACLs take precedence over standard unix access rights
# The owner of a workspace is responsible for its content and management

Please note that <tt>ls</tt> (List directory contents) shows ACLs on directories and files only when run as <tt>ls -l</tt> as in long format, as "plus" sign after the standard unix access rights.

Examples with regard to "my_workspace":
{| class="wikitable"
|-
!style="width:45%" | Command
!style="width:55%" | Action
|-
|<tt>getfacl "$DIR"</tt>
|List access rights on $DIR
|-
|<tt>setfacl -Rm user:fr_xy1:rX,default:user:fr_xy1:rX "$DIR"</tt>
|Grant user "fr_xy1" read-only access to $DIR
|-
|<tt>setfacl -R -m user:fr_me0000:rwX,default:user:fr_me0000:rwX "$DIR"</tt>
<tt>setfacl -R -m user:fr_xy1:rwX,default:user:fr_xy1:rwX "$DIR"</tt>
|Grant your own user "fr_me0000" and "fr_xy1" inheritable ("default") read and write access to $DIR, so you can also read/write files put into the workspace by a coworker
|-
|<tt>setfacl -Rm group:bw16e001:rX,default:group:bw16e001:rX "$DIR"</tt>
|Grant group (Rechenvorhaben) "bw16e001" read-only access to $DIR
|-
|<tt>setfacl -Rb "$DIR"</tt>
|Remove all ACL rights. Standard Unix access rights apply again.
|}

Options used:
* -R: recursive
* -m: modify
* user:username:rwX user: next name is a user; rwX read, write, eXecute (only where execute is set for user)
* default:[user|group] set the default for user or group for new files or dierctories

== Delete a Workspace ==

$ ws_release mySpace # Manually erase your workspace mySpace

Note: workspaces are kept for some time after release. To immediately delete and free space e.g. for quota reasons, delete the files with rm before release.

Newer versions of workspace tools have a --delete-data flag that immediately deletes data. Note that deleted data from workspaces is permanently lost.

== Restore an Expired Workspace ==

For a certain (system-specific) grace time following workspace expiration, a workspace can be restored by performing the following steps:

(1) Display restorable workspaces.
ws_restore -l

(2) Create a new workspace as the target for the restore:
ws_allocate restored 60

(3) Restore:
ws_restore <full_name_of_expired_workspace> restored

The expired workspace has to be specified using the '''full name''', including username prefix and timestamp suffix (otherwise, it cannot be uniquely identified).
The target workspace, on the other hand, must be given with just its short name as listed by <code>ws_list</code>, without the username prefix.

If the workspace is no visible/restorable, it has been '''permanently deleted''' and cannot be restored, not even by us. Please always remember, that workspaces are intended solely for temporary work data, and there is no backup of data in the workspaces.

Helix/Slurm

2026-02-27T00:29:09Z

H Schumacher: Added info about --begin paramer

= General information about Slurm =
The bwForCluster Helix uses Slurm as batch system.
* Slurm documentation: https://slurm.schedmd.com/documentation.html
* Slurm cheat sheet: https://slurm.schedmd.com/pdfs/summary.pdf
* Slurm tutorials: https://slurm.schedmd.com/tutorials.html

= Slurm Command Overview =

{| width=750px class="wikitable"
! Slurm commands !! Brief explanation
|-
| [https://slurm.schedmd.com/sbatch.html sbatch] || Submits a job and queues it in an input queue
|-
| [https://slurm.schedmd.com/salloc.html salloc] || Request resources for an interactive job
|-
| [https://slurm.schedmd.com/squeue.html squeue] || Displays information about active, eligible, blocked, and/or recently completed jobs
|-
| [https://slurm.schedmd.com/scontrol.html scontrol] || Displays detailed job state information
|-
| [https://slurm.schedmd.com/scontrol.html sstat] || Displays status information about a running job
|-
| [https://slurm.schedmd.com/scancel.html scancel] || Cancels a job
|-
|}

= Job Submission =

Batch jobs are submitted with the command:

<syntaxhighlight lang="bash">$ sbatch <job-script> </syntaxhighlight>

A job script contains options for Slurm in lines beginning with #SBATCH as well as your commands which you want to execute on the compute nodes. For example:

<syntaxhighlight lang="slurm">#!/bin/bash
#SBATCH --partition=cpu-single
#SBATCH --ntasks=1
#SBATCH --time=00:20:00
#SBATCH --mem=1gb
#SBATCH --export=NONE
echo 'Hello world'
</syntaxhighlight>

This jobs requests one core (--ntasks=1) and 1 GB memory (--mem=1gb) for 20 minutes (--time=00:20:00) on nodes provided by the partition 'cpu-single'.

For the sake of a better reproducibility of jobs it is recommended to use the option --export=NONE to prevent the propagation of environment variables from the submit session into the job environment and to load required software modules in the job script.

== Partitions ==

On bwForCluster Helix it is necessary to request a partition with "--partition=<partition_name>" on job submission. Within a partition job allocations are routed automatically to the most suitable compute node(s) for the requested resources (e.g. amount of nodes and cores, memory, number of GPUs). The devel partition is the default partition, if no partition is requested.

The partitions devel, cpu-single and gpu-single are operated in shared mode, i.e. jobs from different users can run on the same node. Jobs can get exclusive access to compute nodes in these partitions with the "--exclusive" option. The partitions cpu-multi and gpu-multi are operated in exclusive mode. Jobs in these partitions automatically get exclusive access to the requested compute nodes.

{| class="wikitable"
|-
! style="width:20%"| Partition
! style="width:20%"| Node Access Policy
! style="width:20%"| [https://wiki.bwhpc.de/e/Helix/Hardware#Compute_Nodes Node Types]
! style="width:20%"| Default
! style="width:20%"| Limits
|-
| devel
| shared
| cpu, gpu4
| ntasks=1, time=00:10:00, mem-per-cpu=2gb
| nodes=2, time=00:30:00
|-
| cpu-single
| shared
| cpu, fat
| ntasks=1, time=00:30:00, mem-per-cpu=2gb
| nodes=1, time=120:00:00
|-
| gpu-single
| shared
| gpu4, gpu8
| ntasks=1, time=00:30:00, mem-per-cpu=2gb
| nodes=1, time=120:00:00
|-
| cpu-multi
| job exclusive
| cpu
| nodes=2, time=00:30:00
| nodes=32, time=48:00:00
|-
| gpu-multi
| job exclusive
| gpu4
| nodes=2, time=00:30:00
| nodes=8, time=48:00:00
|}

== GPU requests ==

For the partitions gpu-single and gpu-multi is it required to request GPU ressources.
* The number of GPUs is requested with the option "--gres=gpu:<number-of-gpus>".
* A specific GPU type can be requested with the option "--gres=gpu:<gpu-type>:<number-of-gpus>". Possible values for <gpu-type> are listed in the line 'GPU Type' of the [https://wiki.bwhpc.de/e/Helix/Hardware#Compute_Nodes Compute Nodes table].
* GPUs that are suitable for a specific GPU memory requirement can be requested with option "--gres=gpu:<number-of-gpus>,gpumem_per_gpu:<required-gpumem>GB". This only restricts the selection of possible GPU types. For the job the total GPU memory per GPU is available as listed in the line 'GPU memory per GPU' of the [https://wiki.bwhpc.de/e/Helix/Hardware#Compute_Nodes Compute Nodes table].

== Constraints ==

It is possible to refine the resource request for a job with the option "--constraint=<feature>".

{| class="wikitable"
|-
! style="width:20%"| Feature
! style="width:80%"| Meaning
|-
| fp64
| request GPU types with FP64 capability (double precision)
|}

== Examples ==

Here you can find some example scripts for batch jobs.

=== Serial Programs ===

<syntaxhighlight lang="slurm">
#!/bin/bash
#SBATCH --partition=cpu-single
#SBATCH --ntasks=1
#SBATCH --time=20:00:00
#SBATCH --mem=4gb
./my_serial_program
</syntaxhighlight>

'''Notes:'''
* Jobs with "--mem" up to 236gb can run on all node types associated with the cpu-single partition.

=== Multi-threaded Programs ===

<syntaxhighlight lang="slurm">
#!/bin/bash
#SBATCH --partition=cpu-single
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=16
#SBATCH --time=01:30:00
#SBATCH --mem=50gb
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
./my_multithreaded_program
</syntaxhighlight>

'''Notes:'''
* Jobs with "--ntasks-per-node" up to 64 and "--mem" up to 236gb can run on all node types associated with the cpu-single partition.
* With "export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}" you can set the number of threads according to the number of resources requested.

=== MPI Programs ===

<syntaxhighlight lang="slurm">
#!/bin/bash
#SBATCH --partition=cpu-multi
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=64
#SBATCH --time=12:00:00
module load compiler/gnu
module load mpi/openmpi
srun ./my_mpi_program
</syntaxhighlight>

'''Notes:'''
* "--mem" requests the memory per node. The maximum is 236gb.
* The Compiler and MPI modules used for the compilation must be loaded before the start of the program.
* It is recommended to start MPI programs with 'srun'.

=== GPU Programs ===

<syntaxhighlight lang="slurm">
#!/bin/bash
#SBATCH --partition=gpu-single
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --gres=gpu:A40:1
#SBATCH --time=12:00:00
#SBATCH --mem=16gb

module load devel/cuda
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
./my_cuda_program
</syntaxhighlight>

'''Notes:'''
* The number of GPUs per node is requested with the option "--gres=gpu:<number-of-gpus>"
* It is recommended to request a suitable GPU type for your application with the option "--gres=gpu:<gpu-type>:<number-of-gpus>". For <gpu-type> put the 'GPU Type' listed in the [https://wiki.bwhpc.de/e/Helix/Hardware#Compute_Nodes Compute Nodes table].
** Example for a request of two A40 GPUs: --gres=gpu:A40:2
** Example for a request of one A100 GPU: --gres=gpu:A100:1
* If you are unsure on which GPU type your code runs faster, please run a test case and compare the run times. In general the following applies:
** A40 GPUs are optimized for single precision computations.
** A100 and H200 GPUs offer better performance for double precision computations or if the code makes use of tensor cores.
* The CUDA module used for compilation must be loaded before the start of the program.

=== More examples ===

Further batch script examples are available on bwForCluster Helix in the directory: <code>/opt/bwhpc/common/system/slurm-examples</code>

= Interactive Jobs =

Interactive jobs must NOT run on the login nodes, however resources for interactive jobs can be requested using srun. The following example requests an interactive session on 1 core for 2 hours:

<syntaxhighlight lang="bash">$ salloc --partition=cpu-single --ntasks=1 --time=2:00:00 </syntaxhighlight>

After execution of this command wait until the queueing system has granted you the requested resources. Once granted you will be automatically logged on the allocated compute node.

If you use applications or tools which provide a GUI, enable X-forwarding for your interactive session with:

<syntaxhighlight lang="bash">$ salloc --partition=cpu-single --ntasks=1 --time=2:00:00 --x11 </syntaxhighlight>

Once the walltime limit has been reached you will be automatically logged out from the compute node.

For convenient access to specific GUI applications (JupyterLab, RStudio, ...) on the cluster, we provide a web-based platform: '''[[Helix/bwVisu | bwVisu]]'''

'''Delay the start of a job''' by using [https://slurm.schedmd.com/sbatch.html#OPT_begin -b] (or [https://slurm.schedmd.com/sbatch.html#OPT_begin --begin]) option in order to defer the allocation of the job until the specified time. Examples:
<syntaxhighlight lang="bash">$ salloc --begin=20:00 ... # job can start after 8 p.m.
$ salloc --begin=now+1hour ... # job can start 1 hour after submission
$ salloc --begin=teatime ... # job can start at teatime (4 p.m.)
$ salloc --begin=2023-12-24T20:00:00 ... # job can start after specified date/time
</syntaxhighlight>

= Job Monitoring =

== Information about submitted jobs ==

For an overview of your submitted jobs use the command:

<syntaxhighlight lang="bash">$ squeue
# Additionally show the estimated start time:
$ squeue --start</syntaxhighlight>

To get detailed information about a specific job use the command:

<syntaxhighlight lang="bash">$ scontrol show job <jobid></syntaxhighlight>

'''Notes:''' 
A job start may be delayed for various reasons:
* (QOSMaxCpuPerUserLimit) - There is a limit to how many CPU cores a user can use at the same time. The job exceeds this limit.
* (QOSMaxGRESPerUser) - There is a limit to how many GPUs a user can use at the same time. The job exceeds this limit.
* (QOSMinGRES) - The job was submitted to a gpu partition without requesting a GPU.
* (launch failed requeued held) - The job has failed to start. You may be able to resume it using scontrol. Alternatively you can cancel it and submit it again.
For further reasons please refer to: https://slurm.schedmd.com/job_reason_codes.html

== Information about resource usage of running jobs ==

You can monitor the resource usage of running jobs with the sstat command. For example:

<pre>
$ sstat --format=JobId,AveCPU,AveRSS,MaxRSS -j <jobid>
</pre>

This will show average CPU time, average and maximum memory consumption of all tasks in the running job.

'sstat -e' command shows a list of fields that can be specified with the '--format' option.

== Interactive access to running jobs ==

It is also possible to attach an interactive shell to a running job with command:

<syntaxhighlight lang="bash">$ srun --jobid=<jobid> --overlap --pty /bin/bash</syntaxhighlight>

Commands like 'top' show you the most busy processes on the node. To exit 'top' type 'q'.

To monitor your GPU processes use the command 'nvidia-smi'.

== Job Feedback ==
You get feedback on resource usage and job efficiency for completed jobs with the command:

<syntaxhighlight lang="bash">
$ seff <jobid>
</syntaxhighlight>

Example Output:

<syntaxhighlight lang="bash">
============================= JOB FEEDBACK =============================
Job ID: 12345678
Cluster: helix
User/Group: hd_ab123/hd_hd
State: COMPLETED (exit code 0)
Nodes: 2
Cores per node: 64
CPU Utilized: 3-04:11:46
CPU Efficiency: 97.90% of 3-05:49:52 core-walltime
Job Wall-clock time: 00:36:29
Memory Utilized: 432.74 GB (estimated maximum)
Memory Efficiency: 85.96% of 503.42 GB (251.71 GB/node)
</syntaxhighlight>

Explanation:
* Nodes: Number of allocated nodes for the job.
* Cores per node: Number of physical cores per node allocated for the job.
* CPU Utilized: Sum of utilized core time.
* CPU Efficiency: 'CPU Utilized' with respect to core-walltime (= 'Nodes' x 'Cores per node' x 'Job Wall-clock time') in percent.
* Job Wall-clock time: runtime of the job.
* Memory Utilized: Sum of memory used. For multi node MPI jobs the sum is only correct when srun is used instead of mpirun.
* Memory Efficiency: 'Memory Utilized' with respect to total allocated memory for the job.

== Job Monitoring Portal ==
For more detailed information about your jobs visit the job monitoring portal: https://helix-monitoring.bwservices.uni-heidelberg.de

= Accounting =

Jobs are billed for allocated CPU cores, memory and GPUs.

To see the accounting data of a specific job:

<syntaxhighlight lang="bash">$ sacct -j <jobid> --format=user,jobid,account,nnodes,ncpus,time,elapsed,AllocTRES%50</syntaxhighlight>

To retrive the job history for a specific user for a certain time frame:

<syntaxhighlight lang="bash">$ sacct -u <user> -S 2022-08-20 -E 2022-08-30 --format=user,jobid,account,nnodes,ncpus,time,elapsed,AllocTRES%50</syntaxhighlight>

= Overview about free resources =

On the login nodes the following command shows what resources are available for immediate use:

<syntaxhighlight lang="bash">$ sinfo_t_idle</syntaxhighlight>

SDS@hd/Access

2026-02-26T23:58:43Z

H Schumacher: Added link to Rclone

This page provides an overview on how to access data served by SDS@hd. To get an introduction to data transfer in general, see [[Data_Transfer|data transfer]].

'''Notation:''' Please replace all placeholder contents within guillemets <code>< ></code> with the actual value. For example, replace ''<name_of_my_dog>'' with ''Fluffy'' .

== Prerequisites ==

* You need to be [[SDS@hd/Registration|registered]].
* You need to be in the belwue-Network. This means you have to use the VPN Service of your HomeOrganization, if you want to access SDS@hd from outside the bwHPC-Clusters (e.g. via eduroam or from your personal notebook).

== Needed Information, independent of the chosen tool ==

* Username: You can look it up under your user data at [https://bwservices.uni-heidelberg.de/user/index.xhtml bwServices]. The username is ''<localPrimaryGroup>_<eduPersonPrincipalName>'' . (It is the same as for the bwHPC Clusters.)
* Password: The Service Password that you set at bwServices in the [[SDS@hd/Registration|registration step]].
* SV-Acronym: Use the lower case version of the acronym for all access options.
* Hostname: The hostname depends on the chosen network protocol:
** For [[Data_Transfer/SSHFS|SSHFS]] and [[Data_Transfer/SFTP|SFTP]]: ''lsdf02-sshfs.urz.uni-heidelberg.de''
** For [[SDS@hd/Access/SMB|SMB]] and [[SDS@hd/Access/NFS|NFS]]: ''lsdf02.urz.uni-heidelberg.de''
** For [[Data_Transfer/WebDAV|WebDAV]] the url is: ''https://lsdf02-webdav.urz.uni-heidelberg.de''

== Recommended Setup ==
The following graphic shows the recommended way for accessing SDS@hd via Windows/Mac/Linux. The table provides an overview of the most important access options and links to the related pages. 
If you have various use cases, it is recommended to use [[Data_Transfer/Rclone|Rclone]]. You can copy, sync and mount with it. Thanks to its multithreading capability Rclone is a good fit for transferring big data. 
For an overview of all connection possibilities, please have a look at [[Data_Transfer/All_Data_Transfer_Routes|all data transfer routes]].

[[File:Data_transfer_diagram_simple.jpg|center|500px]]
Figure 1: SDS@hd main transfer routes

{| class="wikitable"
|- style="font-weight:bold; text-align:center; vertical-align:middle;"
!
! Use Case
! Windows
! Mac
! Linux
! Possible Bandwith
! Firewall Ports
|-
| [[Data_Transfer/Rclone|Rclone]] + <protocol>
| copy, sync and mount, multithreading
| ✓
| ✓
| ✓
| depends on used protocol
| depends on used protocol
|-
| [[SDS@hd/Access/SMB|SMB]]
| mount as network drive in file explorer or usage via Rclone
| [[SDS@hd/Access/SMB#Windows|✓]]
| [[SDS@hd/Access/SMB#Mac|✓]]
| [[SDS@hd/Access/SMB#Linux|✓]]
| up to 40 Gbit/sec
| 139 (netbios), 135 (rpc), 445 (smb)
|-
| [[Data_Transfer/WebDAV|WebDAV]]
| go to solution for restricted networks
| [✓]
| ✓
| ✓
| up to 100GBit/sec
| 80 (http), 443 (https)
|- style="vertical-align:middle;"
| [[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]]
| Graphical User Interface (GUI)
| [[Data_Transfer/Graphical_Clients#MobaXterm|✓]]
| ☓
| ☓
| see sftp
| see sftp
|- style="vertical-align:middle;"
| [[SDS@hd/Access/NFS|NFS]]
| mount for multi-user environments
| ☓
| ☓
| [[SDS@hd/Access/NFS|✓]]
| up to 40 Gbit/sec
| -
|- style="vertical-align:middle;"
| [[Data_Transfer/SSHFS|SSHFS]]
| mount, needs stable internet connection
| ☓
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| see sftp
| see sftp
|- style="vertical-align:middle;"
| [[Data_Transfer/SFTP|SFTP]]
| interactive shell, better usability when used together with Rclone
| [[Data_Transfer/SFTP#Windows|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| up to 40 Gbit/sec
| 22 (ssh)
|}
Table 1: SDS@hd transfer routes

=== Access from a bwHPC Cluster ===

'''bwUniCluster''' 
You can't mount to your $HOME directory but you can create a mount under $TMPDIR by following the instructions for [[Data_Transfer/Rclone#Usage_Rclone_Mount | Rclone mount]]. It is advised to wait a couple of seconds (<code>sleep 5</code>) before trying to use the mounted directory.

'''bwForCluster Helix''' 
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes.

'''bwForCluster BinAC 2''' 
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes. The prerequisites are:
* The SV responsible has enabled the SV on BinAC 2 once by writing to [mailto:sds-hd-support@urz.uni-heidelberg.de sds-hd-support@urz.uni-heidelberg.de]
* You have a valid kerberos ticket, which can be fetched with <code>kinit <userID></code>

'''Other''' 
You can mount your SDS@hd SV on the cluster yourself by using [[Data_Transfer/Rclone | Rclone]] with the [[Data_Transfer/Rclone#Usage_Rclone_Mount | Rclone mount]] command. As transfer protocol you can use WebDAV or sftp. For a full overview please have a look at [[Data_Transfer/All_Data_Transfer_Routes | All Data Transfer Routes]].

=== Access via Webbrowser (read-only) ===

Visit [https://lsdf02-webdav.urz.uni-heidelberg.de/ lsdf02-webdav.urz.uni-heidelberg.de] and login with your SDS@hd username and service password. Here you can get an overview of the data in your "Speichervorhaben" and download single files. To be able to do more, like moving data, uploading new files, or downloading complete folders, a suitable client is needed as described above.

== Best Practices ==

* '''Managing access rights with ACLs''' -> Please set ACLs either via the [https://www.urz.uni-heidelberg.de/de/service-katalog/desktop-und-arbeitsplatz/windows-terminalserver Windows terminal server] or via bwForCluster Helix. ACL changes won't work when used locally on a mounted directory.
* '''Multiuser environment''' - -> Use [[SDS@hd/Access/NFS|NFS]]

== Troubleshooting ==

Maintance windows and known issues are communicated via the email list or the News section at the SDS@hd start page in the wiki.

* '''Issue:''' The credentials aren't accepted
*: Check if your credentials work in general by trying the access via webbrowser: [https://lsdf02-webdav.urz.uni-heidelberg.de/].
*: If this doesn't work:
*:: → [[Registration/Login/Username| Check your Username]]
*:: → [[Registration/bwForCluster/Helix#Troubleshooting_with_the_Help_of_bwServices | Troubleshooting with bwServices]]
* '''Other Issue'''
** If available, follow the troubleshooting guide of your specific connection method.
** Make sure to not use LAN and WLAN at the same time to prevent connection problems.
** If you have an institutional account, make sure to be a fully active member of your institution.
If these suggestions didn't help, write to the [mailto:sds-hd-support@urz.uni-heidelberg.de support]. Provide the following information:
* Your operating system
* Does the access via webbrowser work?
*: If yes, provide us with detailed information on how you tried to access your SV (used access method, username, ...).

Data Transfer/All Data Transfer Routes

2026-02-26T18:29:48Z

H Schumacher: Hinted that sshfs on Windows doesn't really work

= All Data Transfer Routes =

The following tables provide a full overview on the various transfer options depending on the system that you are coming from and the system you want to transfer your data to. The recommended options are marked in the table with '''W'''(indows)/'''M'''(acOS)/'''L'''(inux) added next to the '✓' sign. 
'[✓]' means that this option is possible but might be the worst choice of the given ones.

== Copy and Sync via commandline ==

{| class="wikitable" style="vertical-align:middle;"
|- style="font-weight:bold; text-align:center;"
! Protocol/Tool
! Underlying network protocol
! Description/When to use
! Executable on
!
!
!
! Transfer from/to
!
|-
|
|
|
| Linux
| MacOS
| Windows
| bwHPC
| bwHPC
| [[SDS@hd|SDS@hd]]
|-
| [[Data_Transfer/Rclone|Rclone]] + <protocol>
| sftp, smb, WebDAV, ...
| A tool that can be used with various protocols. It needs to be installed. Multithreaded. Works on two completely remote locations as well. Transfers whole files. No 2FA possible. Usually suitable for big data. Please refer to the marked recommendations when deciding on a network protocol to use with Rclone.
| ✓
| ✓
| ✓
| ✓
| ☓
| ✓
|-
| [[Data_Transfer/WebDAV|WebDAV]]
| http(s)
| Fast with small data, not suitable for big data.
| ✓
| ✓
| ✓
| ✓
| ☓
| ✓
|-
| [[Data_Transfer/Rsync|rsync]]
| ssh
| Synchronization of data. One side must be a local drive. Can spot differences in files and transfer only the changes. Can resume partial transfers.
| ✓
| ✓
| [✓]
| ✓
| ✓
| ☓
|-
| [[Data_Transfer/SFTP|sftp]]
| ssh
| Interactive shell for copying data
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|[✓]]]
| [[Data_Transfer/SFTP#Windows|✓]]
| ✓
| ✓
| ✓
|-
| [[Data_Transfer/SCP|scp]]
| ssh
| For copying data
| [✓]
| [✓]
| [✓]
| [✓]
| [✓]
| ☓
|}

== Mounting via commandline ==
{| class="wikitable" style="vertical-align:middle;"
|- style="font-weight:bold; text-align:center;"
! Protocol/Tool
! Underlying network protocol
! Description & Use Cases
! colspan="4" | Executable on
! colspan="2" | Transfer from/to
|-
|
|
|
| Linux
| MacOS
| Windows
| bwHPC
| bwHPC
| SDS@hd
|-
| [[Data_Transfer/Rclone#Usage_Rclone_Mount|Rclone mount]] + <protocol>
| sftp, smb, WebDAV, ...
| See previous table
| ✓
| ✓
| ✓
| ✓
| ☓
| ✓
|-
| [[Data_Transfer/SSHFS|sshfs]]
| see sftp
| File system using sftp, simple usage, needs stable connections, can't resume partial transfers
| ✓
| [✓]
| [✓]
| ☓
| ✓ [[Data_Transfer/SSHFS#MacOS_&_Linux|(only L, M )]]
| ✓
|-
| [[SDS@hd/Access/SMB|smb]]
| none
| Mapping of a network drive. Multiuser possible. Good fit for big data.
| ✓
| ✓
| ✓
| ☓
| ☓
| ✓ ([[SDS@hd/Access/SMB#Mac|M]], [[SDS@hd/Access/SMB#Windows|W]], [[SDS@hd/Access/SMB#Linux|L]])
|-
| [[SDS@hd/Access/NFS#Using_NFSv4_for_UNIX_client|NFSv4]]
| none
| One-time installation effort (Kerberos, ID mapping). For servers, microscopes, multi-user environments, ...
| ✓
| ☓
| ☓
| ☓
| ☓
| ✓
|}

== Graphical User Interfaces (GUIs) ==

{| class="wikitable"
|- style="font-weight:bold; text-align:center; vertical-align:middle;"
! GUIs
! Underlying network protocols
! Description & Use Cases
! colspan="4" | Executable on
! colspan="2" | Transfer from/to
|- style="vertical-align:middle;"
|
|
|
| Linux
| MacOS
| Windows
| bwHPC
| bwHPC
| SDS@hd
|-
| JupyterLab
| ssh
| Development Environment
| ✓
| ✓
| ✓
| ✓
| ✓
| ✓
|-
| [[Development/VS_Code|VS Code]]
| ssh
| Development Environment
| ✓
| ✓
| ✓
| ✓
| ✓
| ✓
|-
| RStudio
| ssh
| Development Environment
| ✓
| ✓
| ✓
| ✓
| ✓
| ✓
|- style="vertical-align:middle;"
| [[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]]
| ssh, RSync, sftp
|
| ☓
| ☓
| ✓
| ☓
| ✓ [[Data_Transfer/Graphical_Clients#MobaXterm|( W )]]
| ✓
|- style="vertical-align:middle;"
| [[Data_Transfer/Graphical_Clients#WinSCP|WinSCP]]
| sftp, WebDAV, scp
|
| ☓
| ☓
| ✓
| ☓
| ✓
| ✓
|- style="vertical-align:middle;"
| [[Data_Transfer/Graphical_Clients#Filezilla|Filezilla]]
| sftp
|
| ✓
| ✓
| ✓
| ☓
| ✓
| ✓
|- style="vertical-align:middle;"
| [https://cyberduck.io/ Cyberduck]
| sftp, WebDAV, S3
|
| ☓
| ✓
| ✓
| ☓
| ✓
| ✓
|- style="vertical-align:middle;"
| WebDAV via browser
| WebDAV
| download single files
| ✓
| ✓
| ✓
| ☓
| ☓
| [[SDS@hd/Access/WebdavBrowser|✓]]
|}

Data Transfer/SSHFS

2026-02-26T18:26:08Z

H Schumacher: Added warning for Windows sshfs with 2fa

Sshfs is a network protocol for mounting data. It uses [[Data Transfer/SFTP|sftp]] under the hood.

== Installation ==

For the necessary installations follow this link: [https://linuxize.com/post/how-to-use-sshfs-to-mount-remote-directories-over-ssh/ How to use SSHFS to Mount Remote Directories over SSH]. 
a) '''MacOS after the installation''' 
To enable sshfs on your Mac go to: System Preferences -> General tab -> Allow Extensions button 
Then follow the instructions that pop up (make sure that you remember the instructions before shutting down your computer). 
b) '''Linux installation short commands'''
<ul>
<li>Debian/Ubuntu: <code>apt-get install sshfs</code></li>
<li>CentOS/RedHat: <code>dnf install fuse-sshfs</code></li>
<li>If you want to use /etc/fstab, you can do this with the following entry:
<pre>sshfs#<username>@<hostname>: <your_local_mountpoint> fuse defaults,user,noauto,exec,reconnect 0 0 </pre></li></ul>

== Usage ==

=== Windows ===
''Caution: sshfs on Windows doesn't work with 2FA.
<ol style="list-style-type: decimal;">
<li>Go to "This PC" and select "Connect network drive" ("Netzlaufwerk verbinden")</li>
<li>Select a drive (W:) and click on "connect with webpage on which you can save documents and images" (Verbindung mit website herstellen, auf der Sie Dokumente und Bilder speichern können)</li>
<li>Enter the following as address, to connect to the home folder on the cluster:
<syntaxhighlight lang="bash">\sshfs\<username>@<hostname>/</syntaxhighlight>
If you would want to go two folders up, you would add “''../../''” at the end.</li></ol>

=== MacOS & Linux ===

<ol style="list-style-type: decimal;">
<li>Create an empty directory. The cluster/storage folder is going to be mounted to this directory.</li>
<li>Mount a folder from the cluster/storage:
<syntaxhighlight lang="bash"># Home folder
sshfs -o reconnect <username>@<hostname>: <localPath>
# subfolderX
sshfs -o reconnect <username>@<hostname>:subfolderX <localPath></syntaxhighlight></li></ol>

=== Undo mounting ===

'''MacOS & Linux:''' 
If the connection to the cluster broke and you want to remount, you often have to use umount first. Try to remember to always unmount first before retrying. Otherwise it can happen that unmounting is not possible anymore by cmd. Usually the error message in this case tells you that the process is currently busy and therefore cannot umount. In this case, you have to restart the notebook which automatically unmounts everything.

<pre>```bash
umount -f <localPath>
# For example
umount -f /Users/<username>/Documents/mounted
# or use
fusermount -u <localPath>
```</pre>

SDS@hd/Registration/Entitlement

2026-02-25T14:18:21Z

H Schumacher: Added info about students.

== Get SDS@hd Entitlement ==

New SVs can only be opened by bwIDM members who have the '''sds-hd-sv''' entitlement from their home university. This entitlement may have to be applied for individually at the home organisation.

To see if you already have the needed entitlement, you can [[SDS@hd/Registration/Entitlement#Check_your_Entitlements|''check your entitlements'']].

If you need the entitlement, follow the instructions for your home university. If none are provided, please contact your local service desk.

* '''Universität Heidelberg'''
** Employees of Heidelberg University receive this entitlement automatically. Students can't receive it.
* '''Universität Mannheim'''
** Employees of the University of Mannheim receive this entitlement automatically. Students can't receive it.
* '''Universität Ulm'''
** Please contact the helpdesk. See https://www.uni-ulm.de/einrichtungen/kiz/service-katalog/beratung-schulung-hilfe/helpdesk/

== Check your Entitlements ==

# Log in to https://bwservices.uni-heidelberg.de
# Go to "Index" -> "Personal Data"
# Choose the "Shibboleth" tab
# Search for the following entry: http://bwidm.de/entitlement/sds-hd-sv (this is an attribute and not a link)
If you can find it, you already have the needed entitlement.

→ [[SDS@hd/Registration#b)_Apply_for_new_SV| go forward with the SV application]]

SDS@hd

2026-02-24T22:43:32Z

H Schumacher: Added link to SV Management page

[[File:sds-hd-logo.png|350px]]

SDS@hd is a central service for securely storing scientific data (Scientific Data Storage). The service is provided as a state service to researchers of higher education institutions of Baden-Württemberg. It is intended to be used for data that is frequently accessed ('hot data').

{| style=" background:#FEF4AB; width:100%;"
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News
|-
|
* November 2022: No special entitlement is needed anymore to participate in an existing storage project.
* September 2023: Service has now been opened for DFN AAI & eduGAIN federation members to participate on existing storage projects.
|}

{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support
|-
|
* [https://www.urz.uni-heidelberg.de/en/service-catalogue/storage/sdshd-scientific-data-storage Service description & FAQ]
* [mailto:sds-hd-support@urz.uni-heidelberg.de Submit a Ticket]
|}

{| style=" background:#ffeaef; width:100%;"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | User Documentation
|-
|
* [[SDS@hd/Registration|Registration]] [[SDS@hd/SV_Management|(SV Management)]]
* [[SDS@hd/Access|Access]]
* [[SDS@hd/Security|Data Security]]
* [[Helix/bwVisu|bwVisu]] for visualization and interactive data analysis
|}

{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Storage Funding
|-
|
* Please [[SDS@hd/Acknowledgement|acknowledge]] SDS@hd in your publications.

SDS@hd/Registration

2026-02-24T15:14:33Z

H Schumacher: Moved SV management section to new page

== Registration Steps ==

The registration consists of two steps:

* '''Step A:''' Membership in Storage Project (Speichervorhaben/'''SV'''). There are two options:
*: '''a)''' →[[SDS@hd/Registration#a)_Join_existing_SV | Join existing SV]]
*: '''b)''' →[[SDS@hd/Registration#b)_Apply_for_new_SV | Apply for new SV]] (only possible for [https://www.bwidm.de/hochschulen.php bwIDM members])
* '''Step B:''' →[[SDS@hd/Registration#Step_B:_Registration_for_SDS@hd_Service |Registration for SDS@hd Service]]

After finishing the registration, the next steps are:
* Check which management options there are for your respective role in the SV → [[SDS@hd/SV_Management |SV Management]]
* See how to access and use your storage space → [[SDS@hd/Access | Access]]

[[File:SDS@hd_Registration.png|center|x500px|upright|SDS@hd Registration Process]]

=== Step A: SV Membership ===

You can either join an existing SV or start a new one.

==== a) Join existing SV ====
To join an existing SV as a coworker:
# Your advisor (the SV responsible) will provide you with the following data on the SV:
#:* acronym
#:* password
# Login at [https://sds-hd.urz.uni-heidelberg.de/management/index.php?mode=mitarbeit SDS@hd Managementtool]
#: Provide the SV acronym and SV password.
# After submitting the request, you will be assigned to the SV as a member and receive an email with the next steps. The SV owner (and SV managers) will be notified automatically.

==== b) Apply for new SV ====

This is typically done only by the leader of a scientific work group or the senior scientist of a research group/collaboration.
Any amount of co-workers can join your SV without having to register another project. You just need to provide them with the SV acronym and SV password. You'll receive this information via e-mail as soon as the SV application was successful.

There are two steps:
# '''Get the permission / entitlement:'''. Your institution has to grant you the permission to start an SDS@hd storage project ("SDS@hd SV entitlement"). Please follow the procedure specific to your institution:
#: → see [[SDS@hd/Registration/Entitlement | Entitlement]].
# '''Apply for a new SV:'''
#: → Fill in the form at the [https://sds-hd.urz.uni-heidelberg.de/management SDS@hd Managementtool].

If you register your own SV, you will be:
* ...responsibe for providing new members with the necessary information to join the SV
*: → see [[#SV_Responsible:_Manage_the_SV_and_its_Members | SV Responsible: Manage the SV and its Members]]
* ...held accountable for the co-workers in the SV
* ...asked to provide information for the two reports required by the DFG for their funding of SDS@hd
* ...likely asked for a contribution to a future DFG grant proposal for an extension of the storage system in your area of research ("wissenschaftliches Beiblatt")

=== Step B: Registration for SDS@hd Service ===
[[File:bwServices_chooseHomeOrganisation.png|right|300px|thumb|Select your home organization]]

After step 1 you have to register your personal account on the storage system and set a service password.
Please visit:
* [https://bwservices.uni-heidelberg.de/ https://bwservices.uni-heidelberg.de]
*# Select your home organization from the list and click '''Proceed'''
*# You will be directed to the ''Identity Provider'' of your home organisation
*# Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
*# You will be redirected back to the registration website [https://bwservices.uni-heidelberg.de/ https://bwservices.uni-heidelberg.de/]
*# <div>Select unter '''The following services are available''' the service '''SDS@hd - Scientific Data Storage'''
*# Click '''Register'''
*# Finally, set a service password for authentication on SDS@hd

=== Change Service Password at bwServices ===

→ '''[[Registration/bwForCluster/Helix#Setting_a_New_Service_Password | Setting a New Service Password]]'''

SDS@hd/SV Management

2026-02-24T15:07:56Z

H Schumacher: Added role descriptions

== Roles in a SV ==
There are four different roles in an SV. Depending on the role, different management activites can be executed at the [https://sds-hd.urz.uni-heidelberg.de/management SDS@hd Managementtool]. These are described below. They include top level management of user access.

=== SV Member or Guest ===
At the [https://sds-hd.urz.uni-heidelberg.de/management SDS@hd Managementtool] members/guests can...
* ...view the status of your SV memberships (active/inactive)
* ...view the status of the SV (active/inactive, end date) (Removing people from the list completely is not possible, yet)

=== SV Manager ===
Additionally to the abilities of a coworker, a manager can...
* ...change the status of SV members (active/inactive)

=== SV Responsible (SVV) ===
Additionally to the abilities of a manager, the responsible can...
* ...enable others to join the SV
*# Provide them with the SV acronym. You can look it up at the [https://sds-hd.urz.uni-heidelberg.de/management/shib/info_sv.php SDS@hd Managementtool].
*# Povide them with the SV password. You've received this password via email. You can reset it at the [https://sds-hd.urz.uni-heidelberg.de/management/shib/info_sv.php SDS@hd Managementtool]. Caution: Make sure to share the SV password and not your personal service password which you set in registration step 2 at bwServices.
*# Afterwards, they can join by following the [[SDS@hd/Registration | registration steps]].
* ...enable the guest folder
* ...change the role of SV members (guest/full member)
* ...change the SV password
* ...hand over the SV responsibility
* ...cancel the SV when it is not needed anymore

== Yearly storage project (SV) extension ==

Once a year the SV responsible person needs to apply for a lifetime extension of the storage project. As part of the application, you will be asked to inform us of publications for which the storage project was used. As long as SDS@hd is used appropriately, an extension will be granted.

SDS@hd/Access

2026-02-24T00:41:43Z

H Schumacher: Added hint on how to replace < >. Added info on how to look up the username on bwServices.

This page provides an overview on how to access data served by SDS@hd. To get an introduction to data transfer in general, see [[Data_Transfer|data transfer]].

'''Notation:''' Please replace all placeholder contents within guillemets <code>< ></code> with the actual value. For example, replace ''<name_of_my_dog>'' with ''Fluffy'' .

== Prerequisites ==

* You need to be [[SDS@hd/Registration|registered]].
* You need to be in the belwue-Network. This means you have to use the VPN Service of your HomeOrganization, if you want to access SDS@hd from outside the bwHPC-Clusters (e.g. via eduroam or from your personal notebook).

== Needed Information, independent of the chosen tool ==

* Username: You can look it up under your user data at [https://bwservices.uni-heidelberg.de/user/index.xhtml bwServices]. The username is ''<localPrimaryGroup>_<eduPersonPrincipalName>'' . (It is the same as for the bwHPC Clusters.)
* Password: The Service Password that you set at bwServices in the [[SDS@hd/Registration|registration step]].
* SV-Acronym: Use the lower case version of the acronym for all access options.
* Hostname: The hostname depends on the chosen network protocol:
** For [[Data_Transfer/SSHFS|SSHFS]] and [[Data_Transfer/SFTP|SFTP]]: ''lsdf02-sshfs.urz.uni-heidelberg.de''
** For [[SDS@hd/Access/SMB|SMB]] and [[SDS@hd/Access/NFS|NFS]]: ''lsdf02.urz.uni-heidelberg.de''
** For [[Data_Transfer/WebDAV|WebDAV]] the url is: ''https://lsdf02-webdav.urz.uni-heidelberg.de''

== Recommended Setup ==
The following graphic shows the recommended way for accessing SDS@hd via Windows/Mac/Linux. The table provides an overview of the most important access options and links to the related pages. 
If you have various use cases, it is recommended to use [[Data_Transfer/Rclone|Rclone]]. You can copy, sync and mount with it. Thanks to its multithreading capability Rclone is a good fit for transferring big data. 
For an overview of all connection possibilities, please have a look at [[Data_Transfer/All_Data_Transfer_Routes|all data transfer routes]].

[[File:Data_transfer_diagram_simple.jpg|center|500px]]
Figure 1: SDS@hd main transfer routes

{| class="wikitable"
|- style="font-weight:bold; text-align:center; vertical-align:middle;"
!
! Use Case
! Windows
! Mac
! Linux
! Possible Bandwith
! Firewall Ports
|-
| [[Data_Transfer/Rclone|Rclone]] + <protocol>
| copy, sync and mount, multithreading
| ✓
| ✓
| ✓
| depends on used protocol
| depends on used protocol
|-
| [[SDS@hd/Access/SMB|SMB]]
| mount as network drive in file explorer or usage via Rclone
| [[SDS@hd/Access/SMB#Windows|✓]]
| [[SDS@hd/Access/SMB#Mac|✓]]
| [[SDS@hd/Access/SMB#Linux|✓]]
| up to 40 Gbit/sec
| 139 (netbios), 135 (rpc), 445 (smb)
|-
| [[Data_Transfer/WebDAV|WebDAV]]
| go to solution for restricted networks
| [✓]
| ✓
| ✓
| up to 100GBit/sec
| 80 (http), 443 (https)
|- style="vertical-align:middle;"
| [[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]]
| Graphical User Interface (GUI)
| [[Data_Transfer/Graphical_Clients#MobaXterm|✓]]
| ☓
| ☓
| see sftp
| see sftp
|- style="vertical-align:middle;"
| [[SDS@hd/Access/NFS|NFS]]
| mount for multi-user environments
| ☓
| ☓
| [[SDS@hd/Access/NFS|✓]]
| up to 40 Gbit/sec
| -
|- style="vertical-align:middle;"
| [[Data_Transfer/SSHFS|SSHFS]]
| mount, needs stable internet connection
| ☓
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| see sftp
| see sftp
|- style="vertical-align:middle;"
| [[Data_Transfer/SFTP|SFTP]]
| interactive shell, better usability when used together with Rclone
| [[Data_Transfer/SFTP#Windows|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| up to 40 Gbit/sec
| 22 (ssh)
|}
Table 1: SDS@hd transfer routes

=== Access from a bwHPC Cluster ===

'''bwUniCluster''' 
You can't mount to your $HOME directory but you can create a mount under $TMPDIR by following the instructions for [[Data_Transfer/Rclone#Usage_Rclone_Mount | Rclone mount]]. It is advised to wait a couple of seconds (<code>sleep 5</code>) before trying to use the mounted directory.

'''bwForCluster Helix''' 
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes.

'''bwForCluster BinAC 2''' 
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes. The prerequisites are:
* The SV responsible has enabled the SV on BinAC 2 once by writing to [mailto:sds-hd-support@urz.uni-heidelberg.de sds-hd-support@urz.uni-heidelberg.de]
* You have a valid kerberos ticket, which can be fetched with <code>kinit <userID></code>

'''Other''' 
You can mount your SDS@hd SV on the cluster yourself by using [[Data_Transfer/Rclone#Usage_Rclone_Mount | Rclone mount]]. As transfer protocol you can use WebDAV or sftp. For a full overview please have a look at [[Data_Transfer/All_Data_Transfer_Routes | All Data Transfer Routes]].

=== Access via Webbrowser (read-only) ===

Visit [https://lsdf02-webdav.urz.uni-heidelberg.de/ lsdf02-webdav.urz.uni-heidelberg.de] and login with your SDS@hd username and service password. Here you can get an overview of the data in your "Speichervorhaben" and download single files. To be able to do more, like moving data, uploading new files, or downloading complete folders, a suitable client is needed as described above.

== Best Practices ==

* '''Managing access rights with ACLs''' -> Please set ACLs either via the [https://www.urz.uni-heidelberg.de/de/service-katalog/desktop-und-arbeitsplatz/windows-terminalserver Windows terminal server] or via bwForCluster Helix. ACL changes won't work when used locally on a mounted directory.
* '''Multiuser environment''' - -> Use [[SDS@hd/Access/NFS|NFS]]

== Troubleshooting ==

Maintance windows and known issues are communicated via the email list or the News section at the SDS@hd start page in the wiki.

* '''Issue:''' The credentials aren't accepted
*: Check if your credentials work in general by trying the access via webbrowser: [https://lsdf02-webdav.urz.uni-heidelberg.de/].
*: If this doesn't work:
*:: → [[Registration/Login/Username| Check your Username]]
*:: → [[Registration/bwForCluster/Helix#Troubleshooting_with_the_Help_of_bwServices | Troubleshooting with bwServices]]
* '''Other Issue'''
** If available, follow the troubleshooting guide of your specific connection method.
** Make sure to not use LAN and WLAN at the same time to prevent connection problems.
** If you have an institutional account, make sure to be a fully active member of your institution.
If these suggestions didn't help, write to the [mailto:sds-hd-support@urz.uni-heidelberg.de support]. Provide the following information:
* Your operating system
* Does the access via webbrowser work?
*: If yes, provide us with detailed information on how you tried to access your SV (used access method, username, ...).

File:MacSmb ConnectWithUrl.png

2026-02-23T16:04:16Z

H Schumacher: H Schumacher uploaded a new version of File:MacSmb ConnectWithUrl.png

File:MacSmb login.png

2026-02-23T15:52:42Z

H Schumacher: H Schumacher uploaded a new version of File:MacSmb login.png

File:WindowsSmb1 driveLetter.png

2026-02-23T15:34:14Z

H Schumacher: H Schumacher uploaded a new version of File:WindowsSmb1 driveLetter.png

File:WindowsSmb0 nonPerm cut.png

2026-02-23T15:24:49Z

H Schumacher: H Schumacher uploaded a new version of File:WindowsSmb0 nonPerm cut.png

SDS@hd/Access/SMB

2026-02-23T14:57:36Z

H Schumacher: added link to access page

SMB is a Server Message Block protocol. It has different implementations: CIFS (outdated), SMB2, SMB3, Samba

== Prerequisites ==

* The SMB connection has to be established at least with protocol version SMB2.02, which is available since Windows Vista or OSX 10.7, and a NTLMv2 authentication level of "Send NTLMv2 responses only".

== Windows ==

Use a SMB share via Windows Explorer.

=== Needed Information ===
You need the following information (→ see [[SDS@hd/Access|SDS@hd/Access]]): 
'''Username:''' <code>BWSERVICESAD\<username></code> 
'''Password:''' <code><service_password></code> 
'''Network Path''' in UNC syntax ''':''' <code>\\lsdf02.urz.uni-heidelberg.de\<sv_acronym></code>
=== Instructions ===

<ol style="list-style-type: decimal;">
<li>Open the Windows Explorer.</li>
<li>a) To establish a '''non-permanent connection''':
<ul>
<li>Click on the address bar, which is located at the top of the Explorer.</li>
<li>Enter the network path and press Enter.</li></ul>

[[File:WindowsSmb0_nonPerm_cut.png|center|700px]]
b) To establish a '''permanent connection''' by creating a network (pseudo) drive:
<ul>
<li>Navigate to "This PC". At the top of the window, click on ''Computer'' and select ''Map network drive''.</li></ul>
[[File:windowsSmb0_connect_network_drive_cut.png|center|700px]]

<ul>
<li>Choose a drive letter to be associated with the network drive and enter the network path. Select ''use a different identification'', as these differ from your credentials used locally.</li></ul>

[[File:windowsSmb1_driveLetter.png|center|700px]]

<li>You will then be prompted to enter your credentials.
</li>

[[File:windowsSmb2_password.png|center|x400px]]

<li>After logging in successfully, your network drive will appear under ''This PC''. You can now manipulate your files as accustomed.</li></ol>

=== Troubleshooting ===
* Errors:
*:* This network folder is currently connected under a different name and password. Disconnect the already existing connection.
*:* Multiple connections to a server from the same user while using different usernames is not allowed.
*:* Disconnect old connections.
*:: Solutions:
*::* Disconnect already exsting connections: Right click on "My PC". Click "Further Options". Click "Disconnect Drives". Keep in mind that Windows can't connect multiple network drives when they have different login information.
*::* Remove / adjust the login information by going to the credential management (Anmeldeinformationsverwaltung).
* Error: "Your domain is not available. Make sure that you're connected to your organizational network."
*: Solutions:
*:* Make sure to be in the correct network, for example by connecting via VPN.
*:* Do not use LAN and WLAN at the same time.

== MacOS ==

Create a network drive with Finder.

=== Needed Information ===

You need the following information: 
'''Username:''' <code>BWSERVICESAD\<username></code> 
'''Password:''' ''ServicePassword'' 
'''Network Path:''' smb://lsdf02.urz.uni-heidelberg.de/<sv-acronym>

=== Instructions ===

<ol><li> Open Finder</li>
<li>On the menu bar click on ''Go'' -> ''Connect to Server'' (or use ⌘+K)</li>
<li>Insert server address and click ''Connect''</li>
[[File:macSmb_ConnectWithUrl.png|center|x400px]]
<li>Insert username and password and click ''Connect''</li>
[[File:macSmb_login.png|center|x400px]]
<li>Finder opens new connected SMB share
</li></ol>
== Linux ==

A UNIX like operating system needs a CIFS client to use a share. CIFS clients are part of Samba implementation for Linux and other UNIX like operating systems (http://www.samba.org)

'''Attention:'''
The core CIFS protocol does not provide unix ownership information or mode for files and directories.
Because of this, files and directories will generally appear to be owned by whatever values the uid= or gid= options are set, and will have permissions set to the default file_mode and dir_mode for the mount. '''Attempting to change these values via chmod/chown will return success but have no effect.'''

For security reasons, server side permission checks cannot be overriden. The permission checks done by the server will always correspond to the credentials used to mount the share, and not necessarily to the user who is accessing the share.

Although mapping of POSIX UIDs and SIDs is not needed mounting a CIFS share '''it might become necessary when working with files on the share, e.g. when modifying ACLs'''.



=== Mounting a SDS@hd Share ===

Mounting a SDS@hd CIFS share can be done by using username/password credentials or by using kerberos tickets.
Information about settting up a kerberos environment for SDS@hd can be found [[SDS@hd/Access/Kerberos|*here*]]'''.

==== Single-User Environment ====

A share can be mounted to a local directory, (e.g. /mnt/sds-hd ). Depending on your system setup, root privileges may be required.

CIFS normally binds all shares on the client as the property of the user who mounted them and transfers any existing write rights only to the user. With additional information from uid, gid, file_mode and dir_mode, other ownership and access rights can be defined when mounting on the client.

'''Nevertheless the ownership and access rights defined in this way are only simulated on the client and are not really transferred to the server.''' If access rights are changed on the client or files with other owners are created in shared folders, these changes only apply to the client and only until the next remount.

If you need to work with the correct server side permissions, please follow the setup of a [[SDS@hd/Access/CIFS#Multiuser Environment|MultiUser Setup]]

===== Preparations =====
<pre>
# Create locally an empty directory to mount to
mkdir </path/to/empty/folder>
</pre>

===== Mount over command line =====
<pre>
sudo mount -t cifs -o username=<username>,domain=BWSERVICESAD,vers=3,mfsymlinks //lsdf02.urz.uni-heidelberg.de/<sv-acronym> </mnt/sds-hd>
> Password:
</pre>

===== Mount over /etc/fstab =====

<pre>
# Open /etc/fstab
sudo nano /etc/fstab
# Add the following
//lsdf02.urz.uni-heidelberg.de/<sv_acronym> </path/to/empty/folder> cifs uid=1000,gid=1000,user,vers=3,mfsymlinks,credentials=<$HOME/.credentialsfile>,noauto 0 0
# Make the system use the changed file
systemctl daemon-reload

# Make credentialsfile
cat /<$HOME/.credentialsfile>
username=ab_ xy123
password=<servicePassword>
domain=BWSERVICESAD

# Mount by providing the same path as was used in /etc/fstab
mount </path/to/empty/folder>
</pre>

===== Check details =====
<pre>
df -h | grep mnt
> //lsdf02.urz.uni-heidelberg.de/sd16j007 108T 6,6T 101T 7% /mnt/sds-hd
mount | grep lsdf02
> //lsdf02.urz.uni-heidelberg.de/sd16j007 on /mnt/sds-hd type cifs > (rw,relatime,vers=3.1.1,cache=strict,username=xxxx,domain=BWSERVICESAD,uid=1000,forceuid,gid=0,noforcegid,addr=xxxxx,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)
</pre>

===== Remove mount =====
<pre>
umount </path/to/mounted/folder>
</pre>

==== Multiuser Environment ====

{|style="background:#ffdeee; width:100%;"
|style="padding:5px; background:#f2cece; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#f2cece; text-align:left"|
At the moment, file deletion is not possible with this setup. We are working on solving the issue.
|}

By default, CIFS mounts only use a single set of user credentials (the mount credentials) when accessing a share. To support different user session on the same mountpoint and the correct permission/ownership processing, the mount options <pre>multiuser,cifsacl</pre> have to be used. Because the kernel cannot prompt for passwords, '''multiuser mounts are limited to mounts using passwordless sec= options, like with sec=krb5. Information about settting up a kerberos environment can be found [[SDS@hd/Access/Kerberos|*here*]]'''

===== ID Mapping =====

In a Multiuser Environment it is important to get the correct ownerships and permissions from the server. Therefor you need to setup a [[SDS@hd/Access/ID-Mapping|ID Mapping]] environment.

Additionally we need the following packages to enable CIFS Mapping:
* RedHat/CentOS:
<pre>$ yum install cifs-utils keyutils</pre>
* Debian/Ubuntu:
<pre>$ apt install cifs-utils keyutils</pre>

After [[SDS@hd/Access/ID-Mapping|installing SSSD]] you have to ensure that it will be used for CIFS name resolution, e.g.

* RedHat/CentOS:
On RedHat SSSD should have allready a higher priority than winbind:
<pre>$ alternatives --display cifs-idmap-plugin

cifs-idmap-plugin - Status ist automatisch.
Link verweist auf /usr/lib64/cifs-utils/cifs_idmap_sss.so
/usr/lib64/cifs-utils/cifs_idmap_sss.so - priority 20
/usr/lib64/cifs-utils/idmapwb.so - priority 10
Zur Zeit ist die `best' Version /usr/lib64/cifs-utils/cifs_idmap_sss.so.
</pre>

* Debian/Ubuntu:
On Debian systems SSSD has to be registered for ID mapping with an higher priority than winbind:

<pre>$ sudo update-alternatives --install /etc/cifs-utils/idmap-plugin idmap-plugin /usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so 50

$ update-alternatives --display idmap-plugin
idmap-plugin - automatischer Modus
beste Version des Links ist /usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so
Link verweist zur Zeit auf /usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so
Link idmap-plugin ist /etc/cifs-utils/idmap-plugin
Slave idmap-plugin.8.gz ist /usr/share/man/man8/idmap-plugin.8.gz
/usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so - Priorität 50
/usr/lib/x86_64-linux-gnu/cifs-utils/idmapwb.so - Priorität 40
Slave idmap-plugin.8.gz: /usr/share/man/man8/idmapwb.8.gz
</pre>

===== AutoFS Setup =====

Because CIFS shares, in contrast to nfs-Mounts, have to be mounted directly, the root user can not simply mount them into a global folder. Instead the shares have to be initially mounted by a user who has access to the Share. To achieve this, you can use the automounter "autofs".

* RedHat/CentOS:
<pre>
$ yum install autofs
$ systemctl enable autofs
$ systemctl start autofs
</pre>

* Debian/Ubuntu:
<pre>
$ apt install autofs
$ systemctl enable autofs
$ systemctl start autofs
</pre>

Afterwards you configure the SDS@hd Speichervorhaben in a new map file:
<pre>
$ cat /etc/auto.sds-hd
<sv-acronym1> -fstype=cifs,cifsacl,multiuser,sec=krb5,cruid=${UID},vers=3,mfsymlinks ://lsdf02.urz.uni-heidelberg.de/<sv-acronym1>
<sv-acronym2> -fstype=cifs,cifsacl,multiuser,sec=krb5,cruid=${UID},vers=3,mfsymlinks ://lsdf02.urz.uni-heidelberg.de/<sv-acronym2>
....
</pre>

You have to include the new map into the auto.master file:
<pre>
$ cat /etc/auto.master
[...]
/mnt/sds-hd /etc/auto.sds-hd
[...]
</pre>

To display all available SDS@hd shares on this machine to the users, you should enable "browser_mode":
<pre>
$ cat /etc/autofs.conf
[...]
# to display all available SDS-hd shares on this to the users
browse_mode=yes
[...]
</pre>
otherwise each share-folder will only be visible after a user has mounted.

After changing the configuration, you should restart the autofs daemon, e.g.:
<pre>
$ systemctl restart autofs
</pre>

Of course you can adopt all other autofs options, like timeouts, etc. to the specific needs of your environment or use any other method for dynamically mounting the CIFS shares.

===== Access the Share =====

Now each user should be able to mount a SDS@hd share, which is configured for the machine. If a share is allready mounted, other users will access this share with their own credentials without mounting again.

To get access, each user needs a valid kerberos ticket, which can be fetched with
<pre>
$ kinit hd_xy123
</pre>

For further information about handling kerberos tickets take a look at [[SDS@hd/Access/NFS#Access_your_data|SDS@hd kerberos]]

=== SMB Client ===

To list the files in a SMB share, use the program smbclient.
<pre>
smbclient -U 'BWSERVICESAD\<username>' //lsdf02.urz.uni-heidelberg.de/<sv-acronym>
Enter BWSERVICESAD\<username>'s password:
</pre>

The program allows you to access the files with a FTP like tool in an interactive shell.
<pre>
$ smbclient //lsdf02.urz.uni-heidelberg.de/<sv-acronym> -U 'BWSERVICESAD\<username>'
Enter BWSERVICESAD\<username>'s password:
smb: \> ls
. D 0 Thu Apr 23 12:51:48 2020
.. D 0 Wed Apr 22 21:54:04 2020
bench D 0 Fri Jul 26 10:24:05 2019
benchmark_test D 0 Tue Oct 30 16:12:21 2018
checksums D 0 Mon Sep 18 10:24:21 2017
test.multiuser A 6 Thu Apr 23 12:36:07 2020
test A 7 Thu Apr 23 09:38:13 2020
.....
.snapshots DHR 0 Thu Jan 1 01:00:00 1970

115343360000 blocks of size 1024. 108260302848 blocks available
smb:\
</pre>

Data Transfer/Rclone

2026-02-16T16:12:21Z

H Schumacher: added oneliner for webdav

[https://rclone.org/docs/ Rclone] is a command line tool to manage files on remote systems (e.g. cloud storage systems). Rclone either synchronizes in one direction only or its mounting functionality is used with <code>rclone mount</code>. Data can be piped between two completely remote locations, sometimes without local download. One advantage is that the transfer is multithreaded and it operates on a file level basis.
'''Caution:''' You can't use Rclone with 2FA.

== Installation ==

Rclone is a Go program and comes as a single binary file.

# Download the relevant binary.
# Extract the <code>rclone</code> executable, <code>rclone.exe</code> on Windows, from the archive.
# You can use the executables without further installation. For easy use, it is recommended to add the binary to your PATH environment variable. Information on how to do this can be found below.

Detailed information regarding different operating systems can be found here:

* Installation on [https://rclone.org/install/#windows Windows]
* Installation on [https://rclone.org/install/#macos macOS]
* Installation on [https://rclone.org/install/#script-installation Linux]

== Usage Rclone ==

To use Rclone you have to define a config file. Afterwards you can connect by using the name of your configured connections.

=== Configure Remote ===

Before you can start using Rclone, you need to set up a remote. This means to configure a specific connection by providing authentication information, the network protocol that you want to use and a name for this configuration so that you can use it later on.

To configure a remote for a specific service, you need the following information:

* <code><remotehost></code>
* <code><username></code>
* <code><servicePassword></code>

Furthermore, you have to decide on:

* <code>network protocol</code> (for example webDAV, smb, sftp)
* <code>remote-name</code> (for example you can use the name of the service you want to connect to)

You have three different options to set up a new remote which are explained by the following sections.

'''Interactive Setup''' 

Execute:
<pre>rclone config</pre>
This will guide you through an interactive setup process. You can find detailed instructions at the website:

* [https://rclone.org/webdav/ Connect via webdav]
* [https://rclone.org/smb/ Connect via smb]
* [https://rclone.org/sftp/ Connect via sftp]

'''Oneliner''' 

Define all parameters in one command. For example:
<syntaxhighlight lang="bash"># sftp
rclone config create <remote-name> sftp host=<remotehost> user=<username> pass=<password> --obscure
# webdav
rclone config create <remote-name> webdav url=<remotehost> user=<username> pass=<password> --obscure
</syntaxhighlight>

'''Adjust Config File''' 

To see, where the file is run: <code>rclone config file</code>.

You can use the following snippet as template for your connections.
<syntaxhighlight>[<remote-name>]
type = webdav
url = <hostURL>
vendor = other
user = <userID>

[<remote-name>]
type = sftp
host = <hostname>
user = <userID>
key_use_agent = false
</syntaxhighlight>
To add the password, please use
<syntaxhighlight>rclone config update <remote-name> pass=<password> --obscure</syntaxhighlight>

=== Use Remote ===

The syntax to use Rclone is like this:

<pre>rclone [options] subcommand <parameters> <parameters...>
</pre>
List all directories/containers/buckets in the folder XX.

<pre>rclone lsd <remote-name>:XX
</pre>
Copies /local/path to the remote path

<pre>rclone copy </local/path> <remote-name>:<remote/path>
</pre>
Copies fom remote path to /local/path

<pre>rclone copy <remote-name>:<remote/path> </local/path>
</pre>
Moves the contents of the source directory to the destination directory.

<pre>rclone move <remote-name>:<source/path> <remote-name>:<destination/path>
</pre>
More subcommands can be found [https://rclone.org/docs/#subcommands here].

== Usage Rclone Mount ==

Before you can follow the instructions in this chapter, you need to have set up a [[Data_Transfer/Rclone#Usage_Rclone | remote]].
Detailed information on how to use rclone mount can be found [https://rclone.org/commands/rclone_mount/ here].

=== Windows ===

To run rclone mount on Windows, you will need to [https://winfsp.dev/rel/ download and install WinFsp]. To mount on drive letter X or a nonexistent subdirectory, use:

<pre>rclone mount <remote-name>:path/to/files X:
rclone mount <remote-name>:path/to/files C:\path\parent\mount</pre>
In contrast to Linux/Mac, there is no background mode.

=== MacOS & Linux ===

You can run mount in either foreground or background (aka daemon) mode. Mount runs in foreground mode by default. Use the <code>--daemon</code> flag to force background mode. If this doesn't work, you can put an <code>&</code> at the end of the command instead.

Create an empty directory on your local machine and then execute

<pre># to mount the root folder:
rclone mount --vfs-cache-mode full <remote-name>: /path/to/empty/folder
# to mount a subfolder:
rclone mount --vfs-cache-mode full <remote-name>:folderX/folderY /path/to/empty/folder
# to unmount:
fusermount -uz /path/to/mounted/folder </pre>

== Best Practices ==

Rclone has a lot of useful options.

=== Performance ===

To be able to utilize a larger bandwidth, it is helpful to add the following options for increased performance:

<pre>
--transfers <int>
</pre>

Number of file transfers to run in parallel (default: 4). Depending on the local Network, read and write speeds on the file system, and current load, different values might be best. For large transfers, it is advised to test local performance with different values beforehand.

* In our tests, we observed the best results between 8 and 32.
* For regular use cases, we recommend 16 as the default.
* Values above 64 are not recommended and degrade performance.

<pre>
--multi-thread-streams <int>
</pre>

Number of streams to use for multithreaded downloads (default: 4). Only important on very large files. This will cause multithreaded up/download on chunk-sized bits of the file.

The optimal value is highly specific to the local network and used Hardware. For regular use cases, we recommend 4 as the default.

=== Debugging and Statistics ===
To get updates on current progress, use:
<pre>
--stats
</pre>
Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable) (default 1m0s).

To get debug information, use:
<pre>
--log-level=DEBUG
--stats-log-level=DEBUG
</pre>

Helix/bwVisu/JupyterLab

2026-01-16T15:34:15Z

H Schumacher: fix typo

[https://jupyter.org/ JupyterLab] is an integrated development environment (IDE) that provides a flexible and scalable interface for the Jupyter Notebook system. It supports interactive data science and scientific computing across over 40 programming languages (including Python, Julia, and R).

== Change Python Version ==

The default python version can be seen by running <code>python --version</code> in the terminal.

A different python version can be installed into a new virtual environment and then registered as IPython kernel for the usage in JupyterLab. This is explained in the chapter [[#Add_packages_via_conda_environments | add packages via conda environments]].

== Install Python Packages ==

Python packages can be added by installing them into a virtual environment and then creating an IPython kernel from the virtual environment. 
Kernels can be shared. See the notes below. 
If you want to move a virtual environment, it is adivsed to recreate it in the new place. Otherwise, dependencies based on relative paths will break.

# Create a virtual environment with...
#* [[#Add_packages_via_venv_virtual_environments | ...venv]] or
#* [[#Add_packages_via_Conda_virtual_environments | ...conda]] (choose this option if you want to install a different python version) or
#* ...[https://docs.astral.sh/uv/getting-started/ uv] if you want to install a different python version but don't want to use conda.
# [[#Create_an_IPython_Kernel | Create an IPython kernel]] from the virtual environment
# Use the kernel within JupyterLab
#* By default new kernels are saved under <code>~/.local/share/jupyter</code> and this location is automatically detected. Therefore, new kernels are directly available.
[[File:BwVisu JuypterLab KernelPath.png|In the JupyterLab job configuration form, a custom kernel path can be provided.|right|thumb|x150px]]
#* If the kernel is saved somewhere else, the path can be provided in the "Kernel path" field when configuring the JupyterLab job (see image). For a kernel placed under <code>path_to_parent_dir/share/jupyter/kernels/my_kernel</code> the needed path would be <code>path_to_parent_dir/share/jupyter</code>.

Notes regarding the sharing of IPython kernels
* The virtual environment and the kernel need to be placed in a shared directory. For example at SDS@hd.
* There could be a subdirectory for the virtual environments and one for the kernels.
* The path to the kernels is saved in the environment variable $JUPYTER_DATA_DIR. Jupyter relevant paths can be seen with <code>jupyter --paths</code>

=== Add packages via ''venv'' virtual environments ===
More information about venv or other python virtual environments can be found at the [[Development/Python | Python]] page.

Steps for creating a '''venv''' virtual environment:
<ol style="list-style-type: decimal;">
<li>Open a terminal.</li>
<li>Create a new virtual evironment:
<syntaxhighlight lang="bash">python3 -m venv <env_parent_dir>/<env_name></syntaxhighlight>
* <code>env_parent_dir</code> is the path to the folder where the virtual environment shall be created. Relative paths can be used.
* Caution: If you you want to share the environment with others, make sure to already create it in the shared place.
</li>
<li>Activate the environment:
<syntaxhighlight lang="bash">source <env_parent_dir>/<env_name>/bin/activate</syntaxhighlight></li>
<li>Update pip and install packages:
<syntaxhighlight lang="bash">pip install -U pip
pip install <packagename></syntaxhighlight></li>
<li> [[#Create_an_IPython_Kernel | Create an IPython kernel]]</li>
</ol>

=== Add packages via Conda virtual environments ===
More information about using conda can be found at the [[Development/Conda | Conda]] page.

<ol style="list-style-type: decimal;">
<li>Load the miniforge module by clicking first on the blue hexagon icon on the left-hand side of Jupyter's start page and then on the "load" button right of the entry for miniforge in the software module menu.
<li>Open a terminal.</li>
<li>Create a new virtual environment:
* If you are the only person using the environment, you can install it in your home directory:
<syntaxhighlight lang="bash">conda create --name <env_name> python=<python version></syntaxhighlight>
* If you want to install it into a different directory, for example a shared place:
<syntaxhighlight lang="bash">conda create --prefix <path_to_shared_directory>/<env_name> python=<python version>
</syntaxhighlight>
</li>
<li>Activate your environment:
<syntaxhighlight lang="bash">conda activate <myenv></syntaxhighlight></li>
<li>Install your packages:
<syntaxhighlight lang="bash">conda install <mypackage></syntaxhighlight></li>
<li> [[#Create_an_IPython_Kernel | Create an IPython kernel]]</li>
</ol>

== Create an IPython Kernel ==

Python kernels are implementations of the Jupyter notebook environment for different languages or virtual environments. You can switch between kernels easily, allowing you to use the best tool for a specific task.
conda_kernels

=== Create a kernel from a virtual environment ===

<ol>
<li>Activate the virtual environment.</li>
<li>Install the ipykernel package:
<syntaxhighlight lang="bash">pip install ipykernel</syntaxhighlight></li>
<li>Register the virtual environment as custom kernel to Jupyter.
* If you are the only person using the environment:
<syntaxhighlight lang="bash">python3 -m ipykernel install --user --name=<kernel_name></syntaxhighlight>
The kernel can be found under <code>~/.local/share/jupyter/kernels/</code>.
* If you installed the environment in a shared place and want to have the kernel there as well:
<syntaxhighlight lang="bash">python3 -m ipykernel install --prefix <path_to_kernel_folder> --name=<kernel_name></syntaxhighlight>
The kernel can then be found under <code>path_to_kernel_folder/share/jupyter/kernels/<kernel_name></code>. As long as the same <code>path_to_kernel_folder</code> is used, all kernels will be saved next to each other in "kernels".
</li>
</ol>

=== Multi-Language Support ===

JupyterLab supports over 40 programming languages including Python, R, Julia, and Scala. This is achieved through the use of different kernels.

==== R Kernel ====

<ol style="list-style-type: decimal;">
<li>On the cluster:
<pre>$ module load math/R
$ R
> install.packages('IRkernel')</pre></li>
<li>On bwVisu:
<ol style="list-style-type: decimal;">
<li>Start Jupyter App</li>
<li>In left menu: load math/R</li>
<li>Open Console:</li>

<pre>$ R
> IRkernel::installspec(displayname = 'R 4.2')</pre>

<li>Start kernel 'R 4.2' as console or notebook</li>
</ol>
</ol>

==== Julia Kernel ====

Load the math/julia module. Open the Terminal.

<pre>
julia
]
add IJulia
</pre>

After that, Julia is available as a kernel.

== Interactive Widgets ==

JupyterLab supports interactive widgets that can create UI controls for interactive data visualization and manipulation within the notebooks. Example of using an interactive widget:

<pre class="{.python">from ipywidgets import IntSlider
slider = IntSlider()
display(slider)</pre>
These widgets can be sliders, dropdowns, buttons, etc., which can be connected to Python code running in the backend.

== FAQ ==

<ol><li>'''My conda commands are interrupted with message 'Killed'.'''
Request more memory when starting Jupyter.</li>
<li>'''How can I navigate to my SDS@hd folder in the file browser?'''
Open a terminal and set a symbolic link to your SDS@hd folder in your home directory. For example:</li>
<pre>cd $HOME
mkdir sds-hd
cd sds-hd
ln -s /mnt/sds-hd/sd16a001 sd16a001</pre>
<li>'''Jupyterlab doesn't let me in but asks for a password.'''
Try using more memory for the job. If this doesn't help, try using the inkognito mode of your browser as the browser cache might be the problem.</li>
<li>'''I prefer VSCode over JupyterLab. Can I start a JupyterLab job and then connect with it via VSCode?'''
This is not possible. Please start the job directly on Helix instead. You can find the instructions at the [[Development/VS_Code#Connect_to_Remote_Jupyter_Kernel | VSCode page]].</li>
</ol>

Data Transfer/WebDAV

2026-01-15T16:14:12Z

H Schumacher: Added file browser connection

__FORCETOC__
WebDAV is a network protocol used for data transfer. It is based on http and offers a faster and more robust connection than [[Data_Transfer/SFTP|sftp]]. The suitability for permanent connections depends highly on the used client. We advise the Rclone client. Instructions are given at the [[Data_Transfer/Rclone|Rclone page]]. See [https://rclone.org/webdav/ rclone webdav config docs] for more details.

== Usage ==

=== Windows ===

It is recommended to follow the instructions for setting up the tool [[Data_Transfer/Rclone | Rclone]] with the WebDAV protocol as you can't work with files greater than 4 GiB or mount more than 16 TB overall if you are using WebDAV via Windows Explorer. If you want to set up the connection via Windows Explorer nevertheless, you can follow the instructions for establishing a permanent connection with [[SDS@hd/Access/SMB|smb]]. Replace the smb network path with the WebDAV URL and use your regular username of the service that you want to connect to.

=== MacOS & Linux ===

It is recommended to follow the instructions for setting up the tool [[Data_Transfer/Rclone | Rclone]] with the WebDAV protocol.

You can do the connection via file browser as well.
* For Mac follow the [[SDS@hd/Access/SMB#MacOS | smb instructions]] but replace the smb network path with the WebDAV URL and use your regular username of the service that you want to connect to.
* For Linux find the option to enter a server address in your file browser (in Ubuntu/Gnome it's called "Other Locations" in the left sidebar). Enter <pre>davs://<webdav_url>/<folder></pre>.

SDS@hd/Access/SMB

2026-01-15T15:50:45Z

H Schumacher: Moved smbclient section to the end of the page

SMB is a Server Message Block protocol. It has different implementations: CIFS (outdated), SMB2, SMB3, Samba

== Prerequisites ==

The SMB connection has to be established at least with protocol version SMB2.02, which is available since Windows Vista or OSX 10.7, and a NTLMv2 authentication level of "Send NTLMv2 responses only".

== Windows ==

Use a SMB share via Windows Explorer.

=== Needed Information ===
You need the following information: 
'''Username:''' <code>BWSERVICESAD\<username></code> 
'''Password:''' <code><service_password></code> 
'''Network Path''' in UNC syntax ''':''' <code>\\lsdf02.urz.uni-heidelberg.de\<sv_acronym></code>

=== Instructions ===

<ol style="list-style-type: decimal;">
<li>Open the Windows Explorer.</li>
<li>a) To establish a '''non-permanent connection''':
<ul>
<li>Click on the address bar, which is located at the top of the Explorer.</li>
<li>Enter the network path and press Enter.</li></ul>

[[File:WindowsSmb0_nonPerm_cut.png|center|700px]]
b) To establish a '''permanent connection''' by creating a network (pseudo) drive:
<ul>
<li>Navigate to "This PC". At the top of the window, click on ''Computer'' and select ''Map network drive''.</li></ul>
[[File:windowsSmb0_connect_network_drive_cut.png|center|700px]]

<ul>
<li>Choose a drive letter to be associated with the network drive and enter the network path. Select ''use a different identification'', as these differ from your credentials used locally.</li></ul>

[[File:windowsSmb1_driveLetter.png|center|700px]]

<li>You will then be prompted to enter your credentials.
</li>

[[File:windowsSmb2_password.png|center|x400px]]

<li>After logging in successfully, your network drive will appear under ''This PC''. You can now manipulate your files as accustomed.</li></ol>

=== Troubleshooting ===
* Errors:
*:* This network folder is currently connected under a different name and password. Disconnect the already existing connection.
*:* Multiple connections to a server from the same user while using different usernames is not allowed.
*:* Disconnect old connections.
*:: Solutions:
*::* Disconnect already exsting connections: Right click on "My PC". Click "Further Options". Click "Disconnect Drives". Keep in mind that Windows can't connect multiple network drives when they have different login information.
*::* Remove / adjust the login information by going to the credential management (Anmeldeinformationsverwaltung).
* Error: "Your domain is not available. Make sure that you're connected to your organizational network."
*: Solutions:
*:* Make sure to be in the correct network, for example by connecting via VPN.
*:* Do not use LAN and WLAN at the same time.

== MacOS ==

Create a network drive with Finder.

=== Needed Information ===

You need the following information: 
'''Username:''' <code>BWSERVICESAD\<username></code> 
'''Password:''' ''ServicePassword'' 
'''Network Path:''' smb://lsdf02.urz.uni-heidelberg.de/<sv-acronym>

=== Instructions ===

<ol><li> Open Finder</li>
<li>On the menu bar click on ''Go'' -> ''Connect to Server'' (or use ⌘+K)</li>
<li>Insert server address and click ''Connect''</li>
[[File:macSmb_ConnectWithUrl.png|center|x400px]]
<li>Insert username and password and click ''Connect''</li>
[[File:macSmb_login.png|center|x400px]]
<li>Finder opens new connected SMB share
</li></ol>
== Linux ==

A UNIX like operating system needs a CIFS client to use a share. CIFS clients are part of Samba implementation for Linux and other UNIX like operating systems (http://www.samba.org)

'''Attention:'''
The core CIFS protocol does not provide unix ownership information or mode for files and directories.
Because of this, files and directories will generally appear to be owned by whatever values the uid= or gid= options are set, and will have permissions set to the default file_mode and dir_mode for the mount. '''Attempting to change these values via chmod/chown will return success but have no effect.'''

For security reasons, server side permission checks cannot be overriden. The permission checks done by the server will always correspond to the credentials used to mount the share, and not necessarily to the user who is accessing the share.

Although mapping of POSIX UIDs and SIDs is not needed mounting a CIFS share '''it might become necessary when working with files on the share, e.g. when modifying ACLs'''.



=== Mounting a SDS@hd Share ===

Mounting a SDS@hd CIFS share can be done by using username/password credentials or by using kerberos tickets.
Information about settting up a kerberos environment for SDS@hd can be found [[SDS@hd/Access/Kerberos|*here*]]'''.

==== Single-User Environment ====

A share can be mounted to a local directory, (e.g. /mnt/sds-hd ). Depending on your system setup, root privileges may be required.

CIFS normally binds all shares on the client as the property of the user who mounted them and transfers any existing write rights only to the user. With additional information from uid, gid, file_mode and dir_mode, other ownership and access rights can be defined when mounting on the client.

'''Nevertheless the ownership and access rights defined in this way are only simulated on the client and are not really transferred to the server.''' If access rights are changed on the client or files with other owners are created in shared folders, these changes only apply to the client and only until the next remount.

If you need to work with the correct server side permissions, please follow the setup of a [[SDS@hd/Access/CIFS#Multiuser Environment|MultiUser Setup]]

===== Preparations =====
<pre>
# Create locally an empty directory to mount to
mkdir </path/to/empty/folder>
</pre>

===== Mount over command line =====
<pre>
sudo mount -t cifs -o username=<username>,domain=BWSERVICESAD,vers=3,mfsymlinks //lsdf02.urz.uni-heidelberg.de/<sv-acronym> </mnt/sds-hd>
> Password:
</pre>

===== Mount over /etc/fstab =====

<pre>
# Open /etc/fstab
sudo nano /etc/fstab
# Add the following
//lsdf02.urz.uni-heidelberg.de/<sv_acronym> </path/to/empty/folder> cifs uid=1000,gid=1000,user,vers=3,mfsymlinks,credentials=<$HOME/.credentialsfile>,noauto 0 0
# Make the system use the changed file
systemctl daemon-reload

# Make credentialsfile
cat /<$HOME/.credentialsfile>
username=ab_ xy123
password=<servicePassword>
domain=BWSERVICESAD

# Mount by providing the same path as was used in /etc/fstab
mount </path/to/empty/folder>
</pre>

===== Check details =====
<pre>
df -h | grep mnt
> //lsdf02.urz.uni-heidelberg.de/sd16j007 108T 6,6T 101T 7% /mnt/sds-hd
mount | grep lsdf02
> //lsdf02.urz.uni-heidelberg.de/sd16j007 on /mnt/sds-hd type cifs > (rw,relatime,vers=3.1.1,cache=strict,username=xxxx,domain=BWSERVICESAD,uid=1000,forceuid,gid=0,noforcegid,addr=xxxxx,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)
</pre>

===== Remove mount =====
<pre>
umount </path/to/mounted/folder>
</pre>

==== Multiuser Environment ====

{|style="background:#ffdeee; width:100%;"
|style="padding:5px; background:#f2cece; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#f2cece; text-align:left"|
At the moment, file deletion is not possible with this setup. We are working on solving the issue.
|}

By default, CIFS mounts only use a single set of user credentials (the mount credentials) when accessing a share. To support different user session on the same mountpoint and the correct permission/ownership processing, the mount options <pre>multiuser,cifsacl</pre> have to be used. Because the kernel cannot prompt for passwords, '''multiuser mounts are limited to mounts using passwordless sec= options, like with sec=krb5. Information about settting up a kerberos environment can be found [[SDS@hd/Access/Kerberos|*here*]]'''

===== ID Mapping =====

In a Multiuser Environment it is important to get the correct ownerships and permissions from the server. Therefor you need to setup a [[SDS@hd/Access/ID-Mapping|ID Mapping]] environment.

Additionally we need the following packages to enable CIFS Mapping:
* RedHat/CentOS:
<pre>$ yum install cifs-utils keyutils</pre>
* Debian/Ubuntu:
<pre>$ apt install cifs-utils keyutils</pre>

After [[SDS@hd/Access/ID-Mapping|installing SSSD]] you have to ensure that it will be used for CIFS name resolution, e.g.

* RedHat/CentOS:
On RedHat SSSD should have allready a higher priority than winbind:
<pre>$ alternatives --display cifs-idmap-plugin

cifs-idmap-plugin - Status ist automatisch.
Link verweist auf /usr/lib64/cifs-utils/cifs_idmap_sss.so
/usr/lib64/cifs-utils/cifs_idmap_sss.so - priority 20
/usr/lib64/cifs-utils/idmapwb.so - priority 10
Zur Zeit ist die `best' Version /usr/lib64/cifs-utils/cifs_idmap_sss.so.
</pre>

* Debian/Ubuntu:
On Debian systems SSSD has to be registered for ID mapping with an higher priority than winbind:

<pre>$ sudo update-alternatives --install /etc/cifs-utils/idmap-plugin idmap-plugin /usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so 50

$ update-alternatives --display idmap-plugin
idmap-plugin - automatischer Modus
beste Version des Links ist /usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so
Link verweist zur Zeit auf /usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so
Link idmap-plugin ist /etc/cifs-utils/idmap-plugin
Slave idmap-plugin.8.gz ist /usr/share/man/man8/idmap-plugin.8.gz
/usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so - Priorität 50
/usr/lib/x86_64-linux-gnu/cifs-utils/idmapwb.so - Priorität 40
Slave idmap-plugin.8.gz: /usr/share/man/man8/idmapwb.8.gz
</pre>

===== AutoFS Setup =====

Because CIFS shares, in contrast to nfs-Mounts, have to be mounted directly, the root user can not simply mount them into a global folder. Instead the shares have to be initially mounted by a user who has access to the Share. To achieve this, you can use the automounter "autofs".

* RedHat/CentOS:
<pre>
$ yum install autofs
$ systemctl enable autofs
$ systemctl start autofs
</pre>

* Debian/Ubuntu:
<pre>
$ apt install autofs
$ systemctl enable autofs
$ systemctl start autofs
</pre>

Afterwards you configure the SDS@hd Speichervorhaben in a new map file:
<pre>
$ cat /etc/auto.sds-hd
<sv-acronym1> -fstype=cifs,cifsacl,multiuser,sec=krb5,cruid=${UID},vers=3,mfsymlinks ://lsdf02.urz.uni-heidelberg.de/<sv-acronym1>
<sv-acronym2> -fstype=cifs,cifsacl,multiuser,sec=krb5,cruid=${UID},vers=3,mfsymlinks ://lsdf02.urz.uni-heidelberg.de/<sv-acronym2>
....
</pre>

You have to include the new map into the auto.master file:
<pre>
$ cat /etc/auto.master
[...]
/mnt/sds-hd /etc/auto.sds-hd
[...]
</pre>

To display all available SDS@hd shares on this machine to the users, you should enable "browser_mode":
<pre>
$ cat /etc/autofs.conf
[...]
# to display all available SDS-hd shares on this to the users
browse_mode=yes
[...]
</pre>
otherwise each share-folder will only be visible after a user has mounted.

After changing the configuration, you should restart the autofs daemon, e.g.:
<pre>
$ systemctl restart autofs
</pre>

Of course you can adopt all other autofs options, like timeouts, etc. to the specific needs of your environment or use any other method for dynamically mounting the CIFS shares.

===== Access the Share =====

Now each user should be able to mount a SDS@hd share, which is configured for the machine. If a share is allready mounted, other users will access this share with their own credentials without mounting again.

To get access, each user needs a valid kerberos ticket, which can be fetched with
<pre>
$ kinit hd_xy123
</pre>

For further information about handling kerberos tickets take a look at [[SDS@hd/Access/NFS#Access_your_data|SDS@hd kerberos]]

=== SMB Client ===

To list the files in a SMB share, use the program smbclient.
<pre>
smbclient -U 'BWSERVICESAD\<username>' //lsdf02.urz.uni-heidelberg.de/<sv-acronym>
Enter BWSERVICESAD\<username>'s password:
</pre>

The program allows you to access the files with a FTP like tool in an interactive shell.
<pre>
$ smbclient //lsdf02.urz.uni-heidelberg.de/<sv-acronym> -U 'BWSERVICESAD\<username>'
Enter BWSERVICESAD\<username>'s password:
smb: \> ls
. D 0 Thu Apr 23 12:51:48 2020
.. D 0 Wed Apr 22 21:54:04 2020
bench D 0 Fri Jul 26 10:24:05 2019
benchmark_test D 0 Tue Oct 30 16:12:21 2018
checksums D 0 Mon Sep 18 10:24:21 2017
test.multiuser A 6 Thu Apr 23 12:36:07 2020
test A 7 Thu Apr 23 09:38:13 2020
.....
.snapshots DHR 0 Thu Jan 1 01:00:00 1970

115343360000 blocks of size 1024. 108260302848 blocks available
smb:\
</pre>

SDS@hd/Access/SMB

2026-01-15T15:44:39Z

H Schumacher: Tidied up section

SMB is a Server Message Block protocol. It has different implementations: CIFS (outdated), SMB2, SMB3, Samba

== Prerequisites ==

The SMB connection has to be established at least with protocol version SMB2.02, which is available since Windows Vista or OSX 10.7, and a NTLMv2 authentication level of "Send NTLMv2 responses only".

== Windows ==

Use a SMB share via Windows Explorer.

=== Needed Information ===
You need the following information: 
'''Username:''' <code>BWSERVICESAD\<username></code> 
'''Password:''' <code><service_password></code> 
'''Network Path''' in UNC syntax ''':''' <code>\\lsdf02.urz.uni-heidelberg.de\<sv_acronym></code>

=== Instructions ===

<ol style="list-style-type: decimal;">
<li>Open the Windows Explorer.</li>
<li>a) To establish a '''non-permanent connection''':
<ul>
<li>Click on the address bar, which is located at the top of the Explorer.</li>
<li>Enter the network path and press Enter.</li></ul>

[[File:WindowsSmb0_nonPerm_cut.png|center|700px]]
b) To establish a '''permanent connection''' by creating a network (pseudo) drive:
<ul>
<li>Navigate to "This PC". At the top of the window, click on ''Computer'' and select ''Map network drive''.</li></ul>
[[File:windowsSmb0_connect_network_drive_cut.png|center|700px]]

<ul>
<li>Choose a drive letter to be associated with the network drive and enter the network path. Select ''use a different identification'', as these differ from your credentials used locally.</li></ul>

[[File:windowsSmb1_driveLetter.png|center|700px]]

<li>You will then be prompted to enter your credentials.
</li>

[[File:windowsSmb2_password.png|center|x400px]]

<li>After logging in successfully, your network drive will appear under ''This PC''. You can now manipulate your files as accustomed.</li></ol>

=== Troubleshooting ===
* Errors:
*:* This network folder is currently connected under a different name and password. Disconnect the already existing connection.
*:* Multiple connections to a server from the same user while using different usernames is not allowed.
*:* Disconnect old connections.
*:: Solutions:
*::* Disconnect already exsting connections: Right click on "My PC". Click "Further Options". Click "Disconnect Drives". Keep in mind that Windows can't connect multiple network drives when they have different login information.
*::* Remove / adjust the login information by going to the credential management (Anmeldeinformationsverwaltung).
* Error: "Your domain is not available. Make sure that you're connected to your organizational network."
*: Solutions:
*:* Make sure to be in the correct network, for example by connecting via VPN.
*:* Do not use LAN and WLAN at the same time.

== MacOS ==

Create a network drive with Finder.

=== Needed Information ===

You need the following information: 
'''Username:''' <code>BWSERVICESAD\<username></code> 
'''Password:''' ''ServicePassword'' 
'''Network Path:''' smb://lsdf02.urz.uni-heidelberg.de/<sv-acronym>

=== Instructions ===

<ol><li> Open Finder</li>
<li>On the menu bar click on ''Go'' -> ''Connect to Server'' (or use ⌘+K)</li>
<li>Insert server address and click ''Connect''</li>
[[File:macSmb_ConnectWithUrl.png|center|x400px]]
<li>Insert username and password and click ''Connect''</li>
[[File:macSmb_login.png|center|x400px]]
<li>Finder opens new connected SMB share
</li></ol>
== Linux ==

A UNIX like operating system needs a CIFS client to use a share. CIFS clients are part of Samba implementation for Linux and other UNIX like operating systems (http://www.samba.org)

'''Attention:'''
The core CIFS protocol does not provide unix ownership information or mode for files and directories.
Because of this, files and directories will generally appear to be owned by whatever values the uid= or gid= options are set, and will have permissions set to the default file_mode and dir_mode for the mount. '''Attempting to change these values via chmod/chown will return success but have no effect.'''

For security reasons, server side permission checks cannot be overriden. The permission checks done by the server will always correspond to the credentials used to mount the share, and not necessarily to the user who is accessing the share.

Although mapping of POSIX UIDs and SIDs is not needed mounting a CIFS share '''it might become necessary when working with files on the share, e.g. when modifying ACLs'''.



=== SMB Client ===

'''Example:'''
To list the files in a SMB share, use the program smbclient.
<pre>
smbclient -U 'BWSERVICESAD\hd_xy123' //lsdf02.urz.uni-heidelberg.de/<sv-acronym>
Enter BWSERVICESAD\hd_xy123's password:
</pre>

The program allows you to access the files with a FTP like tool in an interactive shell.
<pre>
$ smbclient //lsdf02.urz.uni-heidelberg.de/<sv-acronym> -U 'BWSERVICESAD\hd_xy123'
Enter BWSERVICESAD\hd_xy123's password:
smb: \> ls
. D 0 Thu Apr 23 12:51:48 2020
.. D 0 Wed Apr 22 21:54:04 2020
bench D 0 Fri Jul 26 10:24:05 2019
benchmark_test D 0 Tue Oct 30 16:12:21 2018
checksums D 0 Mon Sep 18 10:24:21 2017
test.multiuser A 6 Thu Apr 23 12:36:07 2020
test A 7 Thu Apr 23 09:38:13 2020
.....
.snapshots DHR 0 Thu Jan 1 01:00:00 1970

115343360000 blocks of size 1024. 108260302848 blocks available
smb:\
</pre>

=== Mounting a SDS@hd Share ===

Mounting a SDS@hd CIFS share can be done by using username/password credentials or by using kerberos tickets.
Information about settting up a kerberos environment for SDS@hd can be found [[SDS@hd/Access/Kerberos|*here*]]'''.

==== Single-User Environment ====

A share can be mounted to a local directory, (e.g. /mnt/sds-hd ). Depending on your system setup, root privileges may be required.

CIFS normally binds all shares on the client as the property of the user who mounted them and transfers any existing write rights only to the user. With additional information from uid, gid, file_mode and dir_mode, other ownership and access rights can be defined when mounting on the client.

'''Nevertheless the ownership and access rights defined in this way are only simulated on the client and are not really transferred to the server.''' If access rights are changed on the client or files with other owners are created in shared folders, these changes only apply to the client and only until the next remount.

If you need to work with the correct server side permissions, please follow the setup of a [[SDS@hd/Access/CIFS#Multiuser Environment|MultiUser Setup]]

===== Preparations =====
<pre>
# Create locally an empty directory to mount to
mkdir </path/to/empty/folder>
</pre>

===== Mount over command line =====
<pre>
sudo mount -t cifs -o username=<username>,domain=BWSERVICESAD,vers=3,mfsymlinks //lsdf02.urz.uni-heidelberg.de/<sv-acronym> </mnt/sds-hd>
> Password:
</pre>

===== Mount over /etc/fstab =====

<pre>
# Open /etc/fstab
sudo nano /etc/fstab
# Add the following
//lsdf02.urz.uni-heidelberg.de/<sv_acronym> </path/to/empty/folder> cifs uid=1000,gid=1000,user,vers=3,mfsymlinks,credentials=<$HOME/.credentialsfile>,noauto 0 0
# Make the system use the changed file
systemctl daemon-reload

# Make credentialsfile
cat /<$HOME/.credentialsfile>
username=ab_ xy123
password=<servicePassword>
domain=BWSERVICESAD

# Mount by providing the same path as was used in /etc/fstab
mount </path/to/empty/folder>
</pre>

===== Check details =====
<pre>
df -h | grep mnt
> //lsdf02.urz.uni-heidelberg.de/sd16j007 108T 6,6T 101T 7% /mnt/sds-hd
mount | grep lsdf02
> //lsdf02.urz.uni-heidelberg.de/sd16j007 on /mnt/sds-hd type cifs > (rw,relatime,vers=3.1.1,cache=strict,username=xxxx,domain=BWSERVICESAD,uid=1000,forceuid,gid=0,noforcegid,addr=xxxxx,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)
</pre>

===== Remove mount =====
<pre>
umount </path/to/mounted/folder>
</pre>

==== Multiuser Environment ====

{|style="background:#ffdeee; width:100%;"
|style="padding:5px; background:#f2cece; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#f2cece; text-align:left"|
At the moment, file deletion is not possible with this setup. We are working on solving the issue.
|}

By default, CIFS mounts only use a single set of user credentials (the mount credentials) when accessing a share. To support different user session on the same mountpoint and the correct permission/ownership processing, the mount options <pre>multiuser,cifsacl</pre> have to be used. Because the kernel cannot prompt for passwords, '''multiuser mounts are limited to mounts using passwordless sec= options, like with sec=krb5. Information about settting up a kerberos environment can be found [[SDS@hd/Access/Kerberos|*here*]]'''

===== ID Mapping =====

In a Multiuser Environment it is important to get the correct ownerships and permissions from the server. Therefor you need to setup a [[SDS@hd/Access/ID-Mapping|ID Mapping]] environment.

Additionally we need the following packages to enable CIFS Mapping:
* RedHat/CentOS:
<pre>$ yum install cifs-utils keyutils</pre>
* Debian/Ubuntu:
<pre>$ apt install cifs-utils keyutils</pre>

After [[SDS@hd/Access/ID-Mapping|installing SSSD]] you have to ensure that it will be used for CIFS name resolution, e.g.

* RedHat/CentOS:
On RedHat SSSD should have allready a higher priority than winbind:
<pre>$ alternatives --display cifs-idmap-plugin

cifs-idmap-plugin - Status ist automatisch.
Link verweist auf /usr/lib64/cifs-utils/cifs_idmap_sss.so
/usr/lib64/cifs-utils/cifs_idmap_sss.so - priority 20
/usr/lib64/cifs-utils/idmapwb.so - priority 10
Zur Zeit ist die `best' Version /usr/lib64/cifs-utils/cifs_idmap_sss.so.
</pre>

* Debian/Ubuntu:
On Debian systems SSSD has to be registered for ID mapping with an higher priority than winbind:

<pre>$ sudo update-alternatives --install /etc/cifs-utils/idmap-plugin idmap-plugin /usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so 50

$ update-alternatives --display idmap-plugin
idmap-plugin - automatischer Modus
beste Version des Links ist /usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so
Link verweist zur Zeit auf /usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so
Link idmap-plugin ist /etc/cifs-utils/idmap-plugin
Slave idmap-plugin.8.gz ist /usr/share/man/man8/idmap-plugin.8.gz
/usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so - Priorität 50
/usr/lib/x86_64-linux-gnu/cifs-utils/idmapwb.so - Priorität 40
Slave idmap-plugin.8.gz: /usr/share/man/man8/idmapwb.8.gz
</pre>

===== AutoFS Setup =====

Because CIFS shares, in contrast to nfs-Mounts, have to be mounted directly, the root user can not simply mount them into a global folder. Instead the shares have to be initially mounted by a user who has access to the Share. To achieve this, you can use the automounter "autofs".

* RedHat/CentOS:
<pre>
$ yum install autofs
$ systemctl enable autofs
$ systemctl start autofs
</pre>

* Debian/Ubuntu:
<pre>
$ apt install autofs
$ systemctl enable autofs
$ systemctl start autofs
</pre>

Afterwards you configure the SDS@hd Speichervorhaben in a new map file:
<pre>
$ cat /etc/auto.sds-hd
<sv-acronym1> -fstype=cifs,cifsacl,multiuser,sec=krb5,cruid=${UID},vers=3,mfsymlinks ://lsdf02.urz.uni-heidelberg.de/<sv-acronym1>
<sv-acronym2> -fstype=cifs,cifsacl,multiuser,sec=krb5,cruid=${UID},vers=3,mfsymlinks ://lsdf02.urz.uni-heidelberg.de/<sv-acronym2>
....
</pre>

You have to include the new map into the auto.master file:
<pre>
$ cat /etc/auto.master
[...]
/mnt/sds-hd /etc/auto.sds-hd
[...]
</pre>

To display all available SDS@hd shares on this machine to the users, you should enable "browser_mode":
<pre>
$ cat /etc/autofs.conf
[...]
# to display all available SDS-hd shares on this to the users
browse_mode=yes
[...]
</pre>
otherwise each share-folder will only be visible after a user has mounted.

After changing the configuration, you should restart the autofs daemon, e.g.:
<pre>
$ systemctl restart autofs
</pre>

Of course you can adopt all other autofs options, like timeouts, etc. to the specific needs of your environment or use any other method for dynamically mounting the CIFS shares.

===== Access the Share =====

Now each user should be able to mount a SDS@hd share, which is configured for the machine. If a share is allready mounted, other users will access this share with their own credentials without mounting again.

To get access, each user needs a valid kerberos ticket, which can be fetched with
<pre>
$ kinit hd_xy123
</pre>

For further information about handling kerberos tickets take a look at [[SDS@hd/Access/NFS#Access_your_data|SDS@hd kerberos]]

SDS@hd/Access/SMB

2026-01-15T15:18:14Z

H Schumacher: changed formatting

SMB is a Server Message Block protocol. It has different implementations: CIFS (outdated), SMB2, SMB3, Samba

== Prerequisites ==

The SMB connection has to be established at least with protocol version SMB2.02, which is available since Windows Vista or OSX 10.7, and a NTLMv2 authentication level of "Send NTLMv2 responses only".

== Windows ==

Use a SMB share via Windows Explorer.

=== Needed Information ===
You need the following information: 
'''Username:''' <code>BWSERVICESAD\<username></code> 
'''Password:''' <code><service_password></code> 
'''Network Path''' in UNC syntax ''':''' <code>\\lsdf02.urz.uni-heidelberg.de\<sv_acronym></code>

=== Instructions ===

<ol style="list-style-type: decimal;">
<li>Open the Windows Explorer.</li>
<li>a) To establish a '''non-permanent connection''':
<ul>
<li>Click on the address bar, which is located at the top of the Explorer.</li>
<li>Enter the network path and press Enter.</li></ul>

[[File:WindowsSmb0_nonPerm_cut.png|center|700px]]
b) To establish a '''permanent connection''' by creating a network (pseudo) drive:
<ul>
<li>Navigate to "This PC". At the top of the window, click on ''Computer'' and select ''Map network drive''.</li></ul>
[[File:windowsSmb0_connect_network_drive_cut.png|center|700px]]

<ul>
<li>Choose a drive letter to be associated with the network drive and enter the network path. Select ''use a different identification'', as these differ from your credentials used locally.</li></ul>

[[File:windowsSmb1_driveLetter.png|center|700px]]

<li>You will then be prompted to enter your credentials.
</li>

[[File:windowsSmb2_password.png|center|x400px]]

<li>After logging in successfully, your network drive will appear under ''This PC''. You can now manipulate your files as accustomed.</li></ol>

=== Troubleshooting ===
* Errors:
*:* This network folder is currently connected under a different name and password. Disconnect the already existing connection.
*:* Multiple connections to a server from the same user while using different usernames is not allowed.
*:* Disconnect old connections.
*:: Solutions:
*::* Disconnect already exsting connections: Right click on "My PC". Click "Further Options". Click "Disconnect Drives". Keep in mind that Windows can't connect multiple network drives when they have different login information.
*::* Remove / adjust the login information by going to the credential management (Anmeldeinformationsverwaltung).
* Error: "Your domain is not available. Make sure that you're connected to your organizational network."
*: Solutions:
*:* Make sure to be in the correct network, for example by connecting via VPN.
*:* Do not use LAN and WLAN at the same time.

== MacOS ==

Create a network drive with Finder.

=== Needed Information ===

You need the following information: 
'''Username:''' <code>BWSERVICESAD\<username></code> 
'''Password:''' ''ServicePassword'' 
'''Network Path:''' smb://lsdf02.urz.uni-heidelberg.de/<sv-acronym>

=== Instructions ===

<ol><li> Open Finder</li>
<li>On the menu bar click on ''Go'' -> ''Connect to Server'' (or use ⌘+K)</li>
<li>Insert server address and click ''Connect''</li>
[[File:macSmb_ConnectWithUrl.png|center|x400px]]
<li>Insert username and password and click ''Connect''</li>
[[File:macSmb_login.png|center|x400px]]
<li>Finder opens new connected SMB share
</li></ol>
== Linux ==

A UNIX like operating system needs a CIFS client to use a share. CIFS clients are part of Samba implementation for Linux and other UNIX like operating systems (http://www.samba.org)

'''Attention:'''
The core CIFS protocol does not provide unix ownership information or mode for files and directories.
Because of this, files and directories will generally appear to be owned by whatever values the uid= or gid= options are set, and will have permissions set to the default file_mode and dir_mode for the mount. '''Attempting to change these values via chmod/chown will return success but have no effect.'''

For security reasons, server side permission checks cannot be overriden. The permission checks done by the server will always correspond to the credentials used to mount the share, and not necessarily to the user who is accessing the share.

Although mapping of POSIX UIDs and SIDs is not needed mounting a CIFS share '''it might become necessary when working with files on the share, e.g. when modifying ACLs'''.



=== SMB Client ===

'''Example:'''
To list the files in a SMB share, use the program smbclient.
<pre>
smbclient -U 'BWSERVICESAD\hd_xy123' //lsdf02.urz.uni-heidelberg.de/<sv-acronym>
Enter BWSERVICESAD\hd_xy123's password:
</pre>

The program allows you to access the files with a FTP like tool in an interactive shell.
<pre>
$ smbclient //lsdf02.urz.uni-heidelberg.de/<sv-acronym> -U 'BWSERVICESAD\hd_xy123'
Enter BWSERVICESAD\hd_xy123's password:
smb: \> ls
. D 0 Thu Apr 23 12:51:48 2020
.. D 0 Wed Apr 22 21:54:04 2020
bench D 0 Fri Jul 26 10:24:05 2019
benchmark_test D 0 Tue Oct 30 16:12:21 2018
checksums D 0 Mon Sep 18 10:24:21 2017
test.multiuser A 6 Thu Apr 23 12:36:07 2020
test A 7 Thu Apr 23 09:38:13 2020
.....
.snapshots DHR 0 Thu Jan 1 01:00:00 1970

115343360000 blocks of size 1024. 108260302848 blocks available
smb:\
</pre>

=== Mounting a SDS@hd Share ===

Mounting a SDS@hd CIFS share can be done by using username/password credentials or by using kerberos tickets.
Information about settting up a kerberos environment for SDS@hd can be found [[SDS@hd/Access/Kerberos|*here*]]'''.

==== Tbd. Single-User Environment ====

<pre>
# new instructions
# install smbclient
# install ... see https://www.linode.com/docs/guides/linux-mount-smb-share/
sudo nano /etc/fstab
# add (leave out gid? gid needs explanation)
//lsdf02.urz.uni-heidelberg.de/<sv_acronym> /mnt/mountpoint cifs user,vers=3,mfsymlinks,credentials=~/credentials,noauto 0 0
systemctl daemon-reload
mount -t cifs -o username=hd_ac294,domain=BWSERVICESAD,vers=3,mfsymlinks //lsdf02.urz.uni-heidelberg.de/sd17D005 /mnt/mountpoint
# credentials
username=<username>
password=<servicePassword>
domain=BWSERVICESAD
</pre>

A share can be mounted to a local directory, (e.g. /mnt/sds-hd ). Depending on your system setup, root privileges may be required.

CIFS normally binds all shares on the client as the property of the user who mounted them and transfers any existing write rights only to the user. With additional information from uid, gid, file_mode and dir_mode, other ownership and access rights can be defined when mounting on the client.

'''Nevertheless the ownership and access rights defined in this way are only simulated on the client and are not really transferred to the server.''' If access rights are changed on the client or files with other owners are created in shared folders, these changes only apply to the client and only until the next remount.

If you need to work with the correct server side permissions, please follow the setup of a [[SDS@hd/Access/CIFS#Multiuser Environment|MultiUser Setup]]

===== Mount over command line =====

<pre>
# Create empty directory to mount to
mkdir </mnt/sds-hd>

# Mount
sudo mount -t cifs -o username=<username>,domain=BWSERVICESAD,vers=3,mfsymlinks //lsdf02.urz.uni-heidelberg.de/<sv-acronym> </mnt/sds-hd>
> Password:

# Check the details
df -h | grep mnt
> //lsdf02.urz.uni-heidelberg.de/sd16j007 108T 6,6T 101T 7% /mnt/sds-hd
mount | grep lsdf02
> //lsdf02.urz.uni-heidelberg.de/sd16j007 on /mnt/sds-hd type cifs > (rw,relatime,vers=3.1.1,cache=strict,username=xxxx,domain=BWSERVICESAD,uid=1000,forceuid,gid=0,noforcegid,addr=xxxxx,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)
</pre>

===== Mount over /etc/fstab =====

<pre>
# Create empty directory to mount to
mkdir </mnt/mountpoint>

# Open /etc/fstab
sudo nano /etc/fstab
# Add the following
//lsdf02.urz.uni-heidelberg.de/<sv_acronym> <mnt/mountpoint> cifs uid=1000,gid=1000,user,vers=3,mfsymlinks,credentials=<$HOME/credentialsfile>,noauto 0 0

# Make credentialsfile
cat /<$HOME/credentialsfile>
username=ab_ xy123
password=<servicepassword>
domain=BWSERVICESAD

# Invoke the prepared mount
mount </mnt/mountpoint>

# Check the details
mount | grep cifs
//lsdf02.urz.uni-heidelberg.de/sd16j007 on /mnt/mountpoint type cifs (rw,relatime,vers=3.1.1,cache=strict,username=xxxx,domain=BWSERVICESAD,uid=1000,forceuid,gid=0,noforcegid,addr=xxxxx,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)
</pre>

==== Multiuser Environment ====

{|style="background:#ffdeee; width:100%;"
|style="padding:5px; background:#f2cece; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#f2cece; text-align:left"|
At the moment, file deletion is not possible with this setup. We are working on solving the issue.
|}

By default, CIFS mounts only use a single set of user credentials (the mount credentials) when accessing a share. To support different user session on the same mountpoint and the correct permission/ownership processing, the mount options <pre>multiuser,cifsacl</pre> have to be used. Because the kernel cannot prompt for passwords, '''multiuser mounts are limited to mounts using passwordless sec= options, like with sec=krb5. Information about settting up a kerberos environment can be found [[SDS@hd/Access/Kerberos|*here*]]'''

===== ID Mapping =====

In a Multiuser Environment it is important to get the correct ownerships and permissions from the server. Therefor you need to setup a [[SDS@hd/Access/ID-Mapping|ID Mapping]] environment.

Additionally we need the following packages to enable CIFS Mapping:
* RedHat/CentOS:
<pre>$ yum install cifs-utils keyutils</pre>
* Debian/Ubuntu:
<pre>$ apt install cifs-utils keyutils</pre>

After [[SDS@hd/Access/ID-Mapping|installing SSSD]] you have to ensure that it will be used for CIFS name resolution, e.g.

* RedHat/CentOS:
On RedHat SSSD should have allready a higher priority than winbind:
<pre>$ alternatives --display cifs-idmap-plugin

cifs-idmap-plugin - Status ist automatisch.
Link verweist auf /usr/lib64/cifs-utils/cifs_idmap_sss.so
/usr/lib64/cifs-utils/cifs_idmap_sss.so - priority 20
/usr/lib64/cifs-utils/idmapwb.so - priority 10
Zur Zeit ist die `best' Version /usr/lib64/cifs-utils/cifs_idmap_sss.so.
</pre>

* Debian/Ubuntu:
On Debian systems SSSD has to be registered for ID mapping with an higher priority than winbind:

<pre>$ sudo update-alternatives --install /etc/cifs-utils/idmap-plugin idmap-plugin /usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so 50

$ update-alternatives --display idmap-plugin
idmap-plugin - automatischer Modus
beste Version des Links ist /usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so
Link verweist zur Zeit auf /usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so
Link idmap-plugin ist /etc/cifs-utils/idmap-plugin
Slave idmap-plugin.8.gz ist /usr/share/man/man8/idmap-plugin.8.gz
/usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so - Priorität 50
/usr/lib/x86_64-linux-gnu/cifs-utils/idmapwb.so - Priorität 40
Slave idmap-plugin.8.gz: /usr/share/man/man8/idmapwb.8.gz
</pre>

===== AutoFS Setup =====

Because CIFS shares, in contrast to nfs-Mounts, have to be mounted directly, the root user can not simply mount them into a global folder. Instead the shares have to be initially mounted by a user who has access to the Share. To achieve this, you can use the automounter "autofs".

* RedHat/CentOS:
<pre>
$ yum install autofs
$ systemctl enable autofs
$ systemctl start autofs
</pre>

* Debian/Ubuntu:
<pre>
$ apt install autofs
$ systemctl enable autofs
$ systemctl start autofs
</pre>

Afterwards you configure the SDS@hd Speichervorhaben in a new map file:
<pre>
$ cat /etc/auto.sds-hd
<sv-acronym1> -fstype=cifs,cifsacl,multiuser,sec=krb5,cruid=${UID},vers=3,mfsymlinks ://lsdf02.urz.uni-heidelberg.de/<sv-acronym1>
<sv-acronym2> -fstype=cifs,cifsacl,multiuser,sec=krb5,cruid=${UID},vers=3,mfsymlinks ://lsdf02.urz.uni-heidelberg.de/<sv-acronym2>
....
</pre>

You have to include the new map into the auto.master file:
<pre>
$ cat /etc/auto.master
[...]
/mnt/sds-hd /etc/auto.sds-hd
[...]
</pre>

To display all available SDS@hd shares on this machine to the users, you should enable "browser_mode":
<pre>
$ cat /etc/autofs.conf
[...]
# to display all available SDS-hd shares on this to the users
browse_mode=yes
[...]
</pre>
otherwise each share-folder will only be visible after a user has mounted.

After changing the configuration, you should restart the autofs daemon, e.g.:
<pre>
$ systemctl restart autofs
</pre>

Of course you can adopt all other autofs options, like timeouts, etc. to the specific needs of your environment or use any other method for dynamically mounting the CIFS shares.

===== Access the Share =====

Now each user should be able to mount a SDS@hd share, which is configured for the machine. If a share is allready mounted, other users will access this share with their own credentials without mounting again.

To get access, each user needs a valid kerberos ticket, which can be fetched with
<pre>
$ kinit hd_xy123
</pre>

For further information about handling kerberos tickets take a look at [[SDS@hd/Access/NFS#Access_your_data|SDS@hd kerberos]]

SDS@hd/Access/SMB

2026-01-15T15:08:38Z

H Schumacher: Added comments

SMB is a Server Message Block protocol. It has different implementations: CIFS (outdated), SMB2, SMB3, Samba

== Prerequisites ==

The SMB connection has to be established at least with protocol version SMB2.02, which is available since Windows Vista or OSX 10.7, and a NTLMv2 authentication level of "Send NTLMv2 responses only".

== Windows ==

Use a SMB share via Windows Explorer.

=== Needed Information ===
You need the following information: 
'''Username:''' <code>BWSERVICESAD\<username></code> 
'''Password:''' <code><service_password></code> 
'''Network Path''' in UNC syntax ''':''' <code>\\lsdf02.urz.uni-heidelberg.de\<sv_acronym></code>

=== Instructions ===

<ol style="list-style-type: decimal;">
<li>Open the Windows Explorer.</li>
<li>a) To establish a '''non-permanent connection''':
<ul>
<li>Click on the address bar, which is located at the top of the Explorer.</li>
<li>Enter the network path and press Enter.</li></ul>

[[File:WindowsSmb0_nonPerm_cut.png|center|700px]]
b) To establish a '''permanent connection''' by creating a network (pseudo) drive:
<ul>
<li>Navigate to "This PC". At the top of the window, click on ''Computer'' and select ''Map network drive''.</li></ul>
[[File:windowsSmb0_connect_network_drive_cut.png|center|700px]]

<ul>
<li>Choose a drive letter to be associated with the network drive and enter the network path. Select ''use a different identification'', as these differ from your credentials used locally.</li></ul>

[[File:windowsSmb1_driveLetter.png|center|700px]]

<li>You will then be prompted to enter your credentials.
</li>

[[File:windowsSmb2_password.png|center|x400px]]

<li>After logging in successfully, your network drive will appear under ''This PC''. You can now manipulate your files as accustomed.</li></ol>

=== Troubleshooting ===
* Errors:
*:* This network folder is currently connected under a different name and password. Disconnect the already existing connection.
*:* Multiple connections to a server from the same user while using different usernames is not allowed.
*:* Disconnect old connections.
*:: Solutions:
*::* Disconnect already exsting connections: Right click on "My PC". Click "Further Options". Click "Disconnect Drives". Keep in mind that Windows can't connect multiple network drives when they have different login information.
*::* Remove / adjust the login information by going to the credential management (Anmeldeinformationsverwaltung).
* Error: "Your domain is not available. Make sure that you're connected to your organizational network."
*: Solutions:
*:* Make sure to be in the correct network, for example by connecting via VPN.
*:* Do not use LAN and WLAN at the same time.

== MacOS ==

Create a network drive with Finder.

=== Needed Information ===

You need the following information: 
'''Username:''' <code>BWSERVICESAD\<username></code> 
'''Password:''' ''ServicePassword'' 
'''Network Path:''' smb://lsdf02.urz.uni-heidelberg.de/<sv-acronym>

=== Instructions ===

<ol><li> Open Finder</li>
<li>On the menu bar click on ''Go'' -> ''Connect to Server'' (or use ⌘+K)</li>
<li>Insert server address and click ''Connect''</li>
[[File:macSmb_ConnectWithUrl.png|center|x400px]]
<li>Insert username and password and click ''Connect''</li>
[[File:macSmb_login.png|center|x400px]]
<li>Finder opens new connected SMB share
</li></ol>
== Linux ==

A UNIX like operating system needs a CIFS client to use a share. CIFS clients are part of Samba implementation for Linux and other UNIX like operating systems (http://www.samba.org)

'''Attention:'''
The core CIFS protocol does not provide unix ownership information or mode for files and directories.
Because of this, files and directories will generally appear to be owned by whatever values the uid= or gid= options are set, and will have permissions set to the default file_mode and dir_mode for the mount. '''Attempting to change these values via chmod/chown will return success but have no effect.'''

For security reasons, server side permission checks cannot be overriden. The permission checks done by the server will always correspond to the credentials used to mount the share, and not necessarily to the user who is accessing the share.

Although mapping of POSIX UIDs and SIDs is not needed mounting a CIFS share '''it might become necessary when working with files on the share, e.g. when modifying ACLs'''.



=== SMB Client ===

'''Example:'''
To list the files in a SMB share, use the program smbclient.
<pre>
smbclient -U 'BWSERVICESAD\hd_xy123' //lsdf02.urz.uni-heidelberg.de/<sv-acronym>
Enter BWSERVICESAD\hd_xy123's password:
</pre>

The program allows you to access the files with a FTP like tool in an interactive shell.
<pre>
$ smbclient //lsdf02.urz.uni-heidelberg.de/<sv-acronym> -U 'BWSERVICESAD\hd_xy123'
Enter BWSERVICESAD\hd_xy123's password:
smb: \> ls
. D 0 Thu Apr 23 12:51:48 2020
.. D 0 Wed Apr 22 21:54:04 2020
bench D 0 Fri Jul 26 10:24:05 2019
benchmark_test D 0 Tue Oct 30 16:12:21 2018
checksums D 0 Mon Sep 18 10:24:21 2017
test.multiuser A 6 Thu Apr 23 12:36:07 2020
test A 7 Thu Apr 23 09:38:13 2020
.....
.snapshots DHR 0 Thu Jan 1 01:00:00 1970

115343360000 blocks of size 1024. 108260302848 blocks available
smb:\
</pre>

=== Mounting a SDS@hd Share ===

Mounting a SDS@hd CIFS share can be done by using username/password credentials or by using kerberos tickets.
Information about settting up a kerberos environment for SDS@hd can be found [[SDS@hd/Access/Kerberos|*here*]]'''.

==== Tbd. Single-User Environment ====

<pre>
# new instructions
# install smbclient
# install ... see https://www.linode.com/docs/guides/linux-mount-smb-share/
sudo nano /etc/fstab
# add (leave out gid? gid needs explanation)
//lsdf02.urz.uni-heidelberg.de/<sv_acronym> /mnt/mountpoint cifs user,vers=3,mfsymlinks,credentials=~/credentials,noauto 0 0
systemctl daemon-reload
mount -t cifs -o username=hd_ac294,domain=BWSERVICESAD,vers=3,mfsymlinks //lsdf02.urz.uni-heidelberg.de/sd17D005 /mnt/mountpoint
# credentials
username=<username>
password=<servicePassword>
domain=BWSERVICESAD
</pre>

A share can be mounted to a local directory, (e.g. /mnt/sds-hd ). Depending on your system setup, root privileges may be required.

CIFS normally binds all shares on the client as the property of the user who mounted them and transfers any existing write rights only to the user. With additional information from uid, gid, file_mode and dir_mode, other ownership and access rights can be defined when mounting on the client.

'''Nevertheless the ownership and access rights defined in this way are only simulated on the client and are not really transferred to the server.''' If access rights are changed on the client or files with other owners are created in shared folders, these changes only apply to the client and only until the next remount.

If you need to work with the correct server side permissions, please follow the setup of a [[SDS@hd/Access/CIFS#Multiuser Environment|MultiUser Setup]]

===== Mount over command line =====

'''Example:'''

<pre>
$ mkdir /mnt/sds-hd

$ sudo mount -t cifs -o username=hd_xy123,domain=BWSERVICESAD,vers=3,mfsymlinks //lsdf02.urz.uni-heidelberg.de/<sv-acronym> /mnt/sds-hd
Password:

$ df -h | grep sds-hd
//lsdf02.urz.uni-heidelberg.de/sd16j007 108T 6,6T 101T 7% /mnt/sds-hd

$ cd /mnt/sds-hd/
$ ls
</pre>
Verify the success of the mount invoking the mount command without any arguments:
<pre>
$ mount | grep lsdf02
//lsdf02.urz.uni-heidelberg.de/sd16j007 on /mnt/sds-hd type cifs (rw,relatime,vers=3.1.1,cache=strict,username=xxxx,domain=BWSERVICESAD,uid=1000,forceuid,gid=0,noforcegid,addr=xxxxx,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)
</pre>

===== Mount over /etc/fstab =====

<pre>
# Create empty directory to mount to
mkdir </mnt/mountpoint>

# Open /etc/fstab
sudo nano /etc/fstab
# Add the following
//lsdf02.urz.uni-heidelberg.de/<sv_acronym> <mnt/mountpoint> cifs uid=1000,gid=1000,user,vers=3,mfsymlinks,credentials=<$HOME/credentialsfile>,noauto 0 0

# Make credentialsfile
cat /<$HOME/credentialsfile>
username=ab_ xy123
password=<servicepassword>
domain=BWSERVICESAD

# Invoke the prepared mount
mount </mnt/mountpoint>

# Check the details
mount | grep cifs
//lsdf02.urz.uni-heidelberg.de/sd16j007 on /mnt/mountpoint type cifs (rw,relatime,vers=3.1.1,cache=strict,username=xxxx,domain=BWSERVICESAD,uid=1000,forceuid,gid=0,noforcegid,addr=xxxxx,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)
</pre>

==== Multiuser Environment ====

{|style="background:#ffdeee; width:100%;"
|style="padding:5px; background:#f2cece; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#f2cece; text-align:left"|
At the moment, file deletion is not possible with this setup. We are working on solving the issue.
|}

By default, CIFS mounts only use a single set of user credentials (the mount credentials) when accessing a share. To support different user session on the same mountpoint and the correct permission/ownership processing, the mount options <pre>multiuser,cifsacl</pre> have to be used. Because the kernel cannot prompt for passwords, '''multiuser mounts are limited to mounts using passwordless sec= options, like with sec=krb5. Information about settting up a kerberos environment can be found [[SDS@hd/Access/Kerberos|*here*]]'''

===== ID Mapping =====

In a Multiuser Environment it is important to get the correct ownerships and permissions from the server. Therefor you need to setup a [[SDS@hd/Access/ID-Mapping|ID Mapping]] environment.

Additionally we need the following packages to enable CIFS Mapping:
* RedHat/CentOS:
<pre>$ yum install cifs-utils keyutils</pre>
* Debian/Ubuntu:
<pre>$ apt install cifs-utils keyutils</pre>

After [[SDS@hd/Access/ID-Mapping|installing SSSD]] you have to ensure that it will be used for CIFS name resolution, e.g.

* RedHat/CentOS:
On RedHat SSSD should have allready a higher priority than winbind:
<pre>$ alternatives --display cifs-idmap-plugin

cifs-idmap-plugin - Status ist automatisch.
Link verweist auf /usr/lib64/cifs-utils/cifs_idmap_sss.so
/usr/lib64/cifs-utils/cifs_idmap_sss.so - priority 20
/usr/lib64/cifs-utils/idmapwb.so - priority 10
Zur Zeit ist die `best' Version /usr/lib64/cifs-utils/cifs_idmap_sss.so.
</pre>

* Debian/Ubuntu:
On Debian systems SSSD has to be registered for ID mapping with an higher priority than winbind:

<pre>$ sudo update-alternatives --install /etc/cifs-utils/idmap-plugin idmap-plugin /usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so 50

$ update-alternatives --display idmap-plugin
idmap-plugin - automatischer Modus
beste Version des Links ist /usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so
Link verweist zur Zeit auf /usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so
Link idmap-plugin ist /etc/cifs-utils/idmap-plugin
Slave idmap-plugin.8.gz ist /usr/share/man/man8/idmap-plugin.8.gz
/usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so - Priorität 50
/usr/lib/x86_64-linux-gnu/cifs-utils/idmapwb.so - Priorität 40
Slave idmap-plugin.8.gz: /usr/share/man/man8/idmapwb.8.gz
</pre>

===== AutoFS Setup =====

Because CIFS shares, in contrast to nfs-Mounts, have to be mounted directly, the root user can not simply mount them into a global folder. Instead the shares have to be initially mounted by a user who has access to the Share. To achieve this, you can use the automounter "autofs".

* RedHat/CentOS:
<pre>
$ yum install autofs
$ systemctl enable autofs
$ systemctl start autofs
</pre>

* Debian/Ubuntu:
<pre>
$ apt install autofs
$ systemctl enable autofs
$ systemctl start autofs
</pre>

Afterwards you configure the SDS@hd Speichervorhaben in a new map file:
<pre>
$ cat /etc/auto.sds-hd
<sv-acronym1> -fstype=cifs,cifsacl,multiuser,sec=krb5,cruid=${UID},vers=3,mfsymlinks ://lsdf02.urz.uni-heidelberg.de/<sv-acronym1>
<sv-acronym2> -fstype=cifs,cifsacl,multiuser,sec=krb5,cruid=${UID},vers=3,mfsymlinks ://lsdf02.urz.uni-heidelberg.de/<sv-acronym2>
....
</pre>

You have to include the new map into the auto.master file:
<pre>
$ cat /etc/auto.master
[...]
/mnt/sds-hd /etc/auto.sds-hd
[...]
</pre>

To display all available SDS@hd shares on this machine to the users, you should enable "browser_mode":
<pre>
$ cat /etc/autofs.conf
[...]
# to display all available SDS-hd shares on this to the users
browse_mode=yes
[...]
</pre>
otherwise each share-folder will only be visible after a user has mounted.

After changing the configuration, you should restart the autofs daemon, e.g.:
<pre>
$ systemctl restart autofs
</pre>

Of course you can adopt all other autofs options, like timeouts, etc. to the specific needs of your environment or use any other method for dynamically mounting the CIFS shares.

===== Access the Share =====

Now each user should be able to mount a SDS@hd share, which is configured for the machine. If a share is allready mounted, other users will access this share with their own credentials without mounting again.

To get access, each user needs a valid kerberos ticket, which can be fetched with
<pre>
$ kinit hd_xy123
</pre>

For further information about handling kerberos tickets take a look at [[SDS@hd/Access/NFS#Access_your_data|SDS@hd kerberos]]

SDS@hd/Access

2026-01-15T12:28:58Z

H Schumacher: Added some points

This page provides an overview on how to access data served by SDS@hd. To get an introduction to data transfer in general, see [[Data_Transfer|data transfer]].

== Prerequisites ==

* You need to be [[SDS@hd/Registration|registered]].
* You need to be in the belwue-Network. This means you have to use the VPN Service of your HomeOrganization, if you want to access SDS@hd from outside the bwHPC-Clusters (e.g. via eduroam or from your personal notebook).

== Needed Information, independent of the chosen tool ==

* [[Registration/Login/Username| Username]]: Same as for the bwHPC Clusters
* Password: The Service Password that you set at bwServices in the [[SDS@hd/Registration|registration step]].
* SV-Acronym: Use the lower case version of the acronym for all access options.
* Hostname: The hostname depends on the chosen network protocol:
** For [[Data_Transfer/SSHFS|SSHFS]] and [[Data_Transfer/SFTP|SFTP]]: ''lsdf02-sshfs.urz.uni-heidelberg.de''
** For [[SDS@hd/Access/SMB|SMB]] and [[SDS@hd/Access/NFS|NFS]]: ''lsdf02.urz.uni-heidelberg.de''
** For [[Data_Transfer/WebDAV|WebDAV]] the url is: ''https://lsdf02-webdav.urz.uni-heidelberg.de''

== Recommended Setup ==
The following graphic shows the recommended way for accessing SDS@hd via Windows/Mac/Linux. The table provides an overview of the most important access options and links to the related pages. 
If you have various use cases, it is recommended to use [[Data_Transfer/Rclone|Rclone]]. You can copy, sync and mount with it. Thanks to its multithreading capability Rclone is a good fit for transferring big data. 
For an overview of all connection possibilities, please have a look at [[Data_Transfer/All_Data_Transfer_Routes|all data transfer routes]].

[[File:Data_transfer_diagram_simple.jpg|center|500px]]
Figure 1: SDS@hd main transfer routes

{| class="wikitable"
|- style="font-weight:bold; text-align:center; vertical-align:middle;"
!
! Use Case
! Windows
! Mac
! Linux
! Possible Bandwith
! Firewall Ports
|-
| [[Data_Transfer/Rclone|Rclone]] + <protocol>
| copy, sync and mount, multithreading
| ✓
| ✓
| ✓
| depends on used protocol
| depends on used protocol
|-
| [[SDS@hd/Access/SMB|SMB]]
| mount as network drive in file explorer or usage via Rclone
| [[SDS@hd/Access/SMB#Windows|✓]]
| [[SDS@hd/Access/SMB#Mac|✓]]
| [[SDS@hd/Access/SMB#Linux|✓]]
| up to 40 Gbit/sec
| 139 (netbios), 135 (rpc), 445 (smb)
|-
| [[Data_Transfer/WebDAV|WebDAV]]
| go to solution for restricted networks
| [✓]
| ✓
| ✓
| up to 100GBit/sec
| 80 (http), 443 (https)
|- style="vertical-align:middle;"
| [[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]]
| Graphical User Interface (GUI)
| [[Data_Transfer/Graphical_Clients#MobaXterm|✓]]
| ☓
| ☓
| see sftp
| see sftp
|- style="vertical-align:middle;"
| [[SDS@hd/Access/NFS|NFS]]
| mount for multi-user environments
| ☓
| ☓
| [[SDS@hd/Access/NFS|✓]]
| up to 40 Gbit/sec
| -
|- style="vertical-align:middle;"
| [[Data_Transfer/SSHFS|SSHFS]]
| mount, needs stable internet connection
| ☓
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| see sftp
| see sftp
|- style="vertical-align:middle;"
| [[Data_Transfer/SFTP|SFTP]]
| interactive shell, better usability when used together with Rclone
| [[Data_Transfer/SFTP#Windows|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| up to 40 Gbit/sec
| 22 (ssh)
|}
Table 1: SDS@hd transfer routes

=== Access from a bwHPC Cluster ===

'''bwUniCluster''' 
You can't mount to your $HOME directory but you can create a mount under $TMPDIR by following the instructions for [[Data_Transfer/Rclone#Usage_Rclone_Mount | Rclone mount]]. It is advised to wait a couple of seconds (<code>sleep 5</code>) before trying to use the mounted directory.

'''bwForCluster Helix''' 
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes.

'''bwForCluster BinAC 2''' 
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes. The prerequisites are:
* The SV responsible has enabled the SV on BinAC 2 once by writing to [mailto:sds-hd-support@urz.uni-heidelberg.de sds-hd-support@urz.uni-heidelberg.de]
* You have a valid kerberos ticket, which can be fetched with <code>kinit <userID></code>

'''Other''' 
You can mount your SDS@hd SV on the cluster yourself by using [[Data_Transfer/Rclone#Usage_Rclone_Mount | Rclone mount]]. As transfer protocol you can use WebDAV or sftp. For a full overview please have a look at [[Data_Transfer/All_Data_Transfer_Routes | All Data Transfer Routes]].

=== Access via Webbrowser (read-only) ===

Visit [https://lsdf02-webdav.urz.uni-heidelberg.de/ lsdf02-webdav.urz.uni-heidelberg.de] and login with your SDS@hd username and service password. Here you can get an overview of the data in your "Speichervorhaben" and download single files. To be able to do more, like moving data, uploading new files, or downloading complete folders, a suitable client is needed as described above.

== Best Practices ==

* '''Managing access rights with ACLs''' -> Please set ACLs either via the [https://www.urz.uni-heidelberg.de/de/service-katalog/desktop-und-arbeitsplatz/windows-terminalserver Windows terminal server] or via bwForCluster Helix. ACL changes won't work when used locally on a mounted directory.
* '''Multiuser environment''' - -> Use [[SDS@hd/Access/NFS|NFS]]

== Troubleshooting ==

Maintance windows and known issues are communicated via the email list or the News section at the SDS@hd start page in the wiki.

* '''Issue:''' The credentials aren't accepted
*: Check if your credentials work in general by trying the access via webbrowser: [https://lsdf02-webdav.urz.uni-heidelberg.de/].
*: If this doesn't work:
*:: → [[Registration/Login/Username| Check your Username]]
*:: → [[Registration/bwForCluster/Helix#Troubleshooting_with_the_Help_of_bwServices | Troubleshooting with bwServices]]
* '''Other Issue'''
** If available, follow the troubleshooting guide of your specific connection method.
** Make sure to not use LAN and WLAN at the same time to prevent connection problems.
** If you have an institutional account, make sure to be a fully active member of your institution.
If these suggestions didn't help, write to the [mailto:sds-hd-support@urz.uni-heidelberg.de support]. Provide the following information:
* Your operating system
* Does the access via webbrowser work?
*: If yes, provide us with detailed information on how you tried to access your SV (used access method, username, ...).

SDS@hd/Access/SMB

2026-01-13T17:09:50Z

H Schumacher: Troubleshooting section added

SMB is a Server Message Block protocol. It has different implementations: CIFS (outdated), SMB2, SMB3, Samba

== Prerequisites ==

The SMB connection has to be established at least with protocol version SMB2.02, which is available since Windows Vista or OSX 10.7, and a NTLMv2 authentication level of "Send NTLMv2 responses only".

== Windows ==

Use a SMB share via Windows Explorer.

=== Needed Information ===
You need the following information: 
'''Username:''' <code>BWSERVICESAD\<username></code> 
'''Password:''' <code><service_password></code> 
'''Network Path''' in UNC syntax ''':''' <code>\\lsdf02.urz.uni-heidelberg.de\<sv_acronym></code>

=== Instructions ===

<ol style="list-style-type: decimal;">
<li>Open the Windows Explorer.</li>
<li>a) To establish a '''non-permanent connection''':
<ul>
<li>Click on the address bar, which is located at the top of the Explorer.</li>
<li>Enter the network path and press Enter.</li></ul>

[[File:WindowsSmb0_nonPerm_cut.png|center|700px]]
b) To establish a '''permanent connection''' by creating a network (pseudo) drive:
<ul>
<li>Navigate to "This PC". At the top of the window, click on ''Computer'' and select ''Map network drive''.</li></ul>
[[File:windowsSmb0_connect_network_drive_cut.png|center|700px]]

<ul>
<li>Choose a drive letter to be associated with the network drive and enter the network path. Select ''use a different identification'', as these differ from your credentials used locally.</li></ul>

[[File:windowsSmb1_driveLetter.png|center|700px]]

<li>You will then be prompted to enter your credentials.
</li>

[[File:windowsSmb2_password.png|center|x400px]]

<li>After logging in successfully, your network drive will appear under ''This PC''. You can now manipulate your files as accustomed.</li></ol>

=== Troubleshooting ===
* Errors:
*:* This network folder is currently connected under a different name and password. Disconnect the already existing connection.
*:* Multiple connections to a server from the same user while using different usernames is not allowed.
*:* Disconnect old connections.
*:: Solutions:
*::* Disconnect already exsting connections: Right click on "My PC". Click "Further Options". Click "Disconnect Drives". Keep in mind that Windows can't connect multiple network drives when they have different login information.
*::* Remove / adjust the login information by going to the credential management (Anmeldeinformationsverwaltung).
* Error: "Your domain is not available. Make sure that you're connected to your organizational network."
*: Solutions:
*:* Make sure to be in the correct network, for example by connecting via VPN.
*:* Do not use LAN and WLAN at the same time.

== MacOS ==

Create a network drive with Finder.

=== Needed Information ===

You need the following information: 
'''Username:''' <code>BWSERVICESAD\<username></code> 
'''Password:''' ''ServicePassword'' 
'''Network Path:''' smb://lsdf02.urz.uni-heidelberg.de/<sv-acronym>

=== Instructions ===

<ol><li> Open Finder</li>
<li>On the menu bar click on ''Go'' -> ''Connect to Server'' (or use ⌘+K)</li>
<li>Insert server address and click ''Connect''</li>
[[File:macSmb_ConnectWithUrl.png|center|x400px]]
<li>Insert username and password and click ''Connect''</li>
[[File:macSmb_login.png|center|x400px]]
<li>Finder opens new connected SMB share
</li></ol>
== Linux ==

A UNIX like operating system needs a CIFS client to use a share. CIFS clients are part of Samba implementation for Linux and other UNIX like operating systems (http://www.samba.org)

'''Attention:'''
The core CIFS protocol does not provide unix ownership information or mode for files and directories.
Because of this, files and directories will generally appear to be owned by whatever values the uid= or gid= options are set, and will have permissions set to the default file_mode and dir_mode for the mount. '''Attempting to change these values via chmod/chown will return success but have no effect.'''

For security reasons, server side permission checks cannot be overriden. The permission checks done by the server will always correspond to the credentials used to mount the share, and not necessarily to the user who is accessing the share.

Although mapping of POSIX UIDs and SIDs is not needed mounting a CIFS share '''it might become necessary when working with files on the share, e.g. when modifying ACLs'''.



=== SMB Client ===

'''Example:'''
To list the files in a SMB share, use the program smbclient.
<pre>
smbclient -U 'BWSERVICESAD\hd_xy123' //lsdf02.urz.uni-heidelberg.de/<sv-acronym>
Enter BWSERVICESAD\hd_xy123's password:
</pre>

The program allows you to access the files with a FTP like tool in an interactive shell.
<pre>
$ smbclient //lsdf02.urz.uni-heidelberg.de/<sv-acronym> -U 'BWSERVICESAD\hd_xy123'
Enter BWSERVICESAD\hd_xy123's password:
smb: \> ls
. D 0 Thu Apr 23 12:51:48 2020
.. D 0 Wed Apr 22 21:54:04 2020
bench D 0 Fri Jul 26 10:24:05 2019
benchmark_test D 0 Tue Oct 30 16:12:21 2018
checksums D 0 Mon Sep 18 10:24:21 2017
test.multiuser A 6 Thu Apr 23 12:36:07 2020
test A 7 Thu Apr 23 09:38:13 2020
.....
.snapshots DHR 0 Thu Jan 1 01:00:00 1970

115343360000 blocks of size 1024. 108260302848 blocks available
smb:\
</pre>

=== Mounting a SDS@hd Share ===

Mounting a SDS@hd CIFS share can be done by using username/password credentials or by using kerberos tickets.
Information about settting up a kerberos environment for SDS@hd can be found [[SDS@hd/Access/Kerberos|*here*]]'''.

==== Tbd. Single-User Environment ====

<pre>
# new instructions
# install smbclient
# install ... see https://www.linode.com/docs/guides/linux-mount-smb-share/
sudo nano /etc/fstab
# add (leave out gid? gid needs explanation)
//lsdf02.urz.uni-heidelberg.de/<sv_acronym> /mnt/mountpoint cifs user,vers=3,mfsymlinks,credentials=~/credentials,noauto 0 0
systemctl daemon-reload
mount -t cifs -o username=hd_ac294,domain=BWSERVICESAD,vers=3,mfsymlinks //lsdf02.urz.uni-heidelberg.de/sd17D005 /mnt/mountpoint
# credentials
username=<username>
password=<servicePassword>
domain=BWSERVICESAD
</pre>

A share can be mounted to a local directory, (e.g. /mnt/sds-hd ). Depending on your system setup, root privileges may be required.

CIFS normally binds all shares on the client as the property of the user who mounted them and transfers any existing write rights only to the user. With additional information from uid, gid, file_mode and dir_mode, other ownership and access rights can be defined when mounting on the client.

'''Nevertheless the ownership and access rights defined in this way are only simulated on the client and are not really transferred to the server.''' If access rights are changed on the client or files with other owners are created in shared folders, these changes only apply to the client and only until the next remount.

If you need to work with the correct server side permissions, please follow the setup of a [[SDS@hd/Access/CIFS#Multiuser Environment|MultiUser Setup]]

===== Mount over command line =====

'''Example:'''

<pre>
$ mkdir /mnt/sds-hd

$ sudo mount -t cifs -o username=hd_xy123,domain=BWSERVICESAD,vers=3,mfsymlinks //lsdf02.urz.uni-heidelberg.de/<sv-acronym> /mnt/sds-hd
Password:

$ df -h | grep sds-hd
//lsdf02.urz.uni-heidelberg.de/sd16j007 108T 6,6T 101T 7% /mnt/sds-hd

$ cd /mnt/sds-hd/
$ ls
</pre>
Verify the success of the mount invoking the mount command without any arguments:
<pre>
$ mount | grep lsdf02
//lsdf02.urz.uni-heidelberg.de/sd16j007 on /mnt/sds-hd type cifs (rw,relatime,vers=3.1.1,cache=strict,username=xxxx,domain=BWSERVICESAD,uid=1000,forceuid,gid=0,noforcegid,addr=xxxxx,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)
</pre>

===== Mount over /etc/fstab =====

'''Example:'''

<pre>
$ mkdir /mnt/mountpoint

/etc/fstab
//lsdf02.urz.uni-heidelberg.de/<sv_acronym> /mnt/mountpoint cifs uid=<YOUR_UID>,gid=<YOUR_GID>,user,vers=3,mfsymlinks,credentials=<path_to_user_HOME>/credentialsfile,noauto 0 0

$ cat /path_to_user_HOME/credentialsfile
username=hd_ xy123
password=<your_servicepassword>
domain=BWSERVICESAD

$ mount /mnt/mountpoint
</pre>
Verify the success of the mount invoking the mount command without any arguments:
<pre>
$ mount | grep cifs
//lsdf02.urz.uni-heidelberg.de/sd16j007 on /mnt/mountpoint type cifs (rw,relatime,vers=3.1.1,cache=strict,username=xxxx,domain=BWSERVICESAD,uid=1000,forceuid,gid=0,noforcegid,addr=xxxxx,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)
</pre>

==== Multiuser Environment ====

{|style="background:#ffdeee; width:100%;"
|style="padding:5px; background:#f2cece; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#f2cece; text-align:left"|
At the moment, file deletion is not possible with this setup. We are working on solving the issue.
|}

By default, CIFS mounts only use a single set of user credentials (the mount credentials) when accessing a share. To support different user session on the same mountpoint and the correct permission/ownership processing, the mount options <pre>multiuser,cifsacl</pre> have to be used. Because the kernel cannot prompt for passwords, '''multiuser mounts are limited to mounts using passwordless sec= options, like with sec=krb5. Information about settting up a kerberos environment can be found [[SDS@hd/Access/Kerberos|*here*]]'''

===== ID Mapping =====

In a Multiuser Environment it is important to get the correct ownerships and permissions from the server. Therefor you need to setup a [[SDS@hd/Access/ID-Mapping|ID Mapping]] environment.

Additionally we need the following packages to enable CIFS Mapping:
* RedHat/CentOS:
<pre>$ yum install cifs-utils keyutils</pre>
* Debian/Ubuntu:
<pre>$ apt install cifs-utils keyutils</pre>

After [[SDS@hd/Access/ID-Mapping|installing SSSD]] you have to ensure that it will be used for CIFS name resolution, e.g.

* RedHat/CentOS:
On RedHat SSSD should have allready a higher priority than winbind:
<pre>$ alternatives --display cifs-idmap-plugin

cifs-idmap-plugin - Status ist automatisch.
Link verweist auf /usr/lib64/cifs-utils/cifs_idmap_sss.so
/usr/lib64/cifs-utils/cifs_idmap_sss.so - priority 20
/usr/lib64/cifs-utils/idmapwb.so - priority 10
Zur Zeit ist die `best' Version /usr/lib64/cifs-utils/cifs_idmap_sss.so.
</pre>

* Debian/Ubuntu:
On Debian systems SSSD has to be registered for ID mapping with an higher priority than winbind:

<pre>$ sudo update-alternatives --install /etc/cifs-utils/idmap-plugin idmap-plugin /usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so 50

$ update-alternatives --display idmap-plugin
idmap-plugin - automatischer Modus
beste Version des Links ist /usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so
Link verweist zur Zeit auf /usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so
Link idmap-plugin ist /etc/cifs-utils/idmap-plugin
Slave idmap-plugin.8.gz ist /usr/share/man/man8/idmap-plugin.8.gz
/usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so - Priorität 50
/usr/lib/x86_64-linux-gnu/cifs-utils/idmapwb.so - Priorität 40
Slave idmap-plugin.8.gz: /usr/share/man/man8/idmapwb.8.gz
</pre>

===== AutoFS Setup =====

Because CIFS shares, in contrast to nfs-Mounts, have to be mounted directly, the root user can not simply mount them into a global folder. Instead the shares have to be initially mounted by a user who has access to the Share. To achieve this, you can use the automounter "autofs".

* RedHat/CentOS:
<pre>
$ yum install autofs
$ systemctl enable autofs
$ systemctl start autofs
</pre>

* Debian/Ubuntu:
<pre>
$ apt install autofs
$ systemctl enable autofs
$ systemctl start autofs
</pre>

Afterwards you configure the SDS@hd Speichervorhaben in a new map file:
<pre>
$ cat /etc/auto.sds-hd
<sv-acronym1> -fstype=cifs,cifsacl,multiuser,sec=krb5,cruid=${UID},vers=3,mfsymlinks ://lsdf02.urz.uni-heidelberg.de/<sv-acronym1>
<sv-acronym2> -fstype=cifs,cifsacl,multiuser,sec=krb5,cruid=${UID},vers=3,mfsymlinks ://lsdf02.urz.uni-heidelberg.de/<sv-acronym2>
....
</pre>

You have to include the new map into the auto.master file:
<pre>
$ cat /etc/auto.master
[...]
/mnt/sds-hd /etc/auto.sds-hd
[...]
</pre>

To display all available SDS@hd shares on this machine to the users, you should enable "browser_mode":
<pre>
$ cat /etc/autofs.conf
[...]
# to display all available SDS-hd shares on this to the users
browse_mode=yes
[...]
</pre>
otherwise each share-folder will only be visible after a user has mounted.

After changing the configuration, you should restart the autofs daemon, e.g.:
<pre>
$ systemctl restart autofs
</pre>

Of course you can adopt all other autofs options, like timeouts, etc. to the specific needs of your environment or use any other method for dynamically mounting the CIFS shares.

===== Access the Share =====

Now each user should be able to mount a SDS@hd share, which is configured for the machine. If a share is allready mounted, other users will access this share with their own credentials without mounting again.

To get access, each user needs a valid kerberos ticket, which can be fetched with
<pre>
$ kinit hd_xy123
</pre>

For further information about handling kerberos tickets take a look at [[SDS@hd/Access/NFS#Access_your_data|SDS@hd kerberos]]

SDS@hd/Access

2026-01-09T11:35:19Z

H Schumacher: Added test of credentials via webbrowser

This page provides an overview on how to access data served by SDS@hd. To get an introduction to data transfer in general, see [[Data_Transfer|data transfer]].

== Prerequisites ==

* You need to be [[SDS@hd/Registration|registered]].
* You need to be in the belwue-Network. This means you have to use the VPN Service of your HomeOrganization, if you want to access SDS@hd from outside the bwHPC-Clusters (e.g. via eduroam or from your personal notebook).

== Needed Information, independent of the chosen tool ==

* [[Registration/Login/Username| Username]]: Same as for the bwHPC Clusters
* Password: The Service Password that you set at bwServices in the [[SDS@hd/Registration|registration step]].
* SV-Acronym: Use the lower case version of the acronym for all access options.
* Hostname: The hostname depends on the chosen network protocol:
** For [[Data_Transfer/SSHFS|SSHFS]] and [[Data_Transfer/SFTP|SFTP]]: ''lsdf02-sshfs.urz.uni-heidelberg.de''
** For [[SDS@hd/Access/SMB|SMB]] and [[SDS@hd/Access/NFS|NFS]]: ''lsdf02.urz.uni-heidelberg.de''
** For [[Data_Transfer/WebDAV|WebDAV]] the url is: ''https://lsdf02-webdav.urz.uni-heidelberg.de''

== Recommended Setup ==
The following graphic shows the recommended way for accessing SDS@hd via Windows/Mac/Linux. The table provides an overview of the most important access options and links to the related pages. 
If you have various use cases, it is recommended to use [[Data_Transfer/Rclone|Rclone]]. You can copy, sync and mount with it. Thanks to its multithreading capability Rclone is a good fit for transferring big data. 
For an overview of all connection possibilities, please have a look at [[Data_Transfer/All_Data_Transfer_Routes|all data transfer routes]].

[[File:Data_transfer_diagram_simple.jpg|center|500px]]
Figure 1: SDS@hd main transfer routes

{| class="wikitable"
|- style="font-weight:bold; text-align:center; vertical-align:middle;"
!
! Use Case
! Windows
! Mac
! Linux
! Possible Bandwith
! Firewall Ports
|-
| [[Data_Transfer/Rclone|Rclone]] + <protocol>
| copy, sync and mount, multithreading
| ✓
| ✓
| ✓
| depends on used protocol
| depends on used protocol
|-
| [[SDS@hd/Access/SMB|SMB]]
| mount as network drive in file explorer or usage via Rclone
| [[SDS@hd/Access/SMB#Windows|✓]]
| [[SDS@hd/Access/SMB#Mac|✓]]
| [[SDS@hd/Access/SMB#Linux|✓]]
| up to 40 Gbit/sec
| 139 (netbios), 135 (rpc), 445 (smb)
|-
| [[Data_Transfer/WebDAV|WebDAV]]
| go to solution for restricted networks
| [✓]
| ✓
| ✓
| up to 100GBit/sec
| 80 (http), 443 (https)
|- style="vertical-align:middle;"
| [[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]]
| Graphical User Interface (GUI)
| [[Data_Transfer/Graphical_Clients#MobaXterm|✓]]
| ☓
| ☓
| see sftp
| see sftp
|- style="vertical-align:middle;"
| [[SDS@hd/Access/NFS|NFS]]
| mount for multi-user environments
| ☓
| ☓
| [[SDS@hd/Access/NFS|✓]]
| up to 40 Gbit/sec
| -
|- style="vertical-align:middle;"
| [[Data_Transfer/SSHFS|SSHFS]]
| mount, needs stable internet connection
| ☓
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| see sftp
| see sftp
|- style="vertical-align:middle;"
| [[Data_Transfer/SFTP|SFTP]]
| interactive shell, better usability when used together with Rclone
| [[Data_Transfer/SFTP#Windows|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| up to 40 Gbit/sec
| 22 (ssh)
|}
Table 1: SDS@hd transfer routes

=== Access from a bwHPC Cluster ===

'''bwUniCluster''' 
You can't mount to your $HOME directory but you can create a mount under $TMPDIR by following the instructions for [[Data_Transfer/Rclone#Usage_Rclone_Mount | Rclone mount]]. It is advised to wait a couple of seconds (<code>sleep 5</code>) before trying to use the mounted directory.

'''bwForCluster Helix''' 
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes.

'''bwForCluster BinAC 2''' 
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes. The prerequisites are:
* The SV responsible has enabled the SV on BinAC 2 once by writing to [mailto:sds-hd-support@urz.uni-heidelberg.de sds-hd-support@urz.uni-heidelberg.de]
* You have a valid kerberos ticket, which can be fetched with <code>kinit <userID></code>

'''Other''' 
You can mount your SDS@hd SV on the cluster yourself by using [[Data_Transfer/Rclone#Usage_Rclone_Mount | Rclone mount]]. As transfer protocol you can use WebDAV or sftp. For a full overview please have a look at [[Data_Transfer/All_Data_Transfer_Routes | All Data Transfer Routes]].

=== Access via Webbrowser (read-only) ===

Visit [https://lsdf02-webdav.urz.uni-heidelberg.de/ lsdf02-webdav.urz.uni-heidelberg.de] and login with your SDS@hd username and service password. Here you can get an overview of the data in your "Speichervorhaben" and download single files. To be able to do more, like moving data, uploading new files, or downloading complete folders, a suitable client is needed as described above.

== Best Practices ==

* '''Managing access rights with ACLs''' -> Please set ACLs either via the [https://www.urz.uni-heidelberg.de/de/service-katalog/desktop-und-arbeitsplatz/windows-terminalserver Windows terminal server] or via bwForCluster Helix. ACL changes won't work when used locally on a mounted directory.
* '''Multiuser environment''' - -> Use [[SDS@hd/Access/NFS|NFS]]

== Troubleshooting ==

* '''Issue:''' The credentials aren't accepted
*: Check if your credentials work in general by trying the access via webbrowser: [https://lsdf02-webdav.urz.uni-heidelberg.de/].
*: If this doesn't work:
*:: → [[Registration/Login/Username| Check your Username]]
*:: → [[Registration/bwForCluster/Helix#Troubleshooting_with_the_Help_of_bwServices | Troubleshooting with bwServices]]
* '''Other'''
*: Write to the [mailto:sds-hd-support@urz.uni-heidelberg.de support]. Provide the following information:
*::* Your operating system
*::* Does the access via webbrowser work?
*::: If yes, provide us with detailed information on how you tried to access your SV (used access method, username, ...).

SDS@hd/Registration

2026-01-08T15:14:12Z

H Schumacher: Fixed links, linked entitlement page, made A a) into a numbered list, removed bold formatting of links

== Registration Steps ==

The registration consists of two steps:

* '''Step A:''' Membership in Storage Project (Speichervorhaben/'''SV'''). There are two options:
*: '''a)''' →[[SDS@hd/Registration#a)_Join_existing_SV | Join existing SV]]
*: '''b)''' →[[SDS@hd/Registration#b)_Apply_for_new_SV | Apply for new SV]] (only possible for [https://www.bwidm.de/hochschulen.php bwIDM members])
* '''Step B:''' →[[SDS@hd/Registration#Step_B:_Registration_for_SDS@hd_Service |Registration for SDS@hd Service]]

After finishing the registration, the next steps are:
* Check your options to manage your SV Membership. → [[#Manage_SV_Membership(s) | Manage SV Membership(s)]]
* See how to access and use your storage space → [[SDS@hd/Access | Access]]

[[File:SDS@hd_Registration.png|center|x500px|upright|SDS@hd Registration Process]]

=== Step A: SV Membership ===

You can either join an existing SV or start a new one.

==== a) Join existing SV ====
To join an existing SV as a coworker:
# Your advisor (the SV responsible) will provide you with the following data on the SV:
#:* acronym
#:* password
# Login at [https://sds-hd.urz.uni-heidelberg.de/management/index.php?mode=mitarbeit SDS@hd Managementtool]
#: Provide the SV acronym and SV password.
# After submitting the request, you will be assigned to the SV as a member and receive an email with the next steps. The SV owner (and SV managers) will be notified automatically.

==== b) Apply for new SV ====

This is typically done only by the leader of a scientific work group or the senior scientist of a research group/collaboration.
Any amount of co-workers can join your SV without having to register another project. You just need to provide them with the SV acronym and SV password. You'll receive this information via e-mail as soon as the SV application was successful.

There are two steps:
# '''Get the permission / entitlement:'''. Your institution has to grant you the permission to start an SDS@hd storage project ("SDS@hd SV entitlement"). Please follow the procedure specific to your institution:
#: → see [[SDS@hd/Registration/Entitlement | Entitlement]].
# '''Apply for a new SV:'''
#: → Fill in the form at the [https://sds-hd.urz.uni-heidelberg.de/management SDS@hd Managementtool].

If you register your own SV, you will be:
* ...responsibe for providing new members with the necessary information to join the SV
*: → see [[#SV_Responsible:_Manage_the_SV_and_its_Members | SV Responsible: Manage the SV and its Members]]
* ...held accountable for the co-workers in the SV
* ...asked to provide information for the two reports required by the DFG for their funding of SDS@hd
* ...likely asked for a contribution to a future DFG grant proposal for an extension of the storage system in your area of research ("wissenschaftliches Beiblatt")

=== Step B: Registration for SDS@hd Service ===
[[File:bwServices_chooseHomeOrganisation.png|right|300px|thumb|Select your home organization]]

After step 1 you have to register your personal account on the storage system and set a service password.
Please visit:
* [https://bwservices.uni-heidelberg.de/ https://bwservices.uni-heidelberg.de]
*# Select your home organization from the list and click '''Proceed'''
*# You will be directed to the ''Identity Provider'' of your home organisation
*# Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
*# You will be redirected back to the registration website [https://bwservices.uni-heidelberg.de/ https://bwservices.uni-heidelberg.de/]
*# <div>Select unter '''The following services are available''' the service '''SDS@hd - Scientific Data Storage'''
*# Click '''Register'''
*# Finally, set a service password for authentication on SDS@hd

== Manage SV Membership(s) ==

The SDS@hd Managementtool and the bwServices website offer various services to manage your membership. They are described below.

=== SV Member: View SV and Membership Status ===
At the [https://sds-hd.urz.uni-heidelberg.de/management SDS@hd Managementtool] you can...
* ...view the status of your SV memberships (active/inactive)
* ...view the status of the SV (active/inactive, end date)

=== SV Responsible: Manage the SV and its Members ===
As the SV responsible person, you can...
* ...enable others to join the SV
*# Provide them with the SV acronym. You can look it up at the [https://sds-hd.urz.uni-heidelberg.de/management/shib/info_sv.php SDS@hd Managementtool].
*# Povide them with the SV password. You've received this password via email. You can reset it at the [https://sds-hd.urz.uni-heidelberg.de/management/shib/info_sv.php SDS@hd Managementtool]. Caution: Make sure to share the SV password and not your personal service password which you set in step 2 at bwServices.
*# Afterwards, they can join by following the registration steps on this page.
* ...change the status of SV members
* ...change the SV password
* ...hand over the SV responsibility

=== Change Service Password at bwServices ===

→ '''[[Registration/bwForCluster/Helix#Setting_a_New_Service_Password | Setting a New Service Password]]'''

SDS@hd/Registration/Entitlement

2026-01-08T14:14:20Z

H Schumacher: Updated information

== Get SDS@hd Entitlement ==

New SVs can only be opened by bwIDM members who have the '''sds-hd-sv''' entitlement from their home university. This entitlement may have to be applied for individually at the home organisation.

To see if you already have the needed entitlement, you can [[SDS@hd/Registration/Entitlement#Check_your_Entitlements|''check your entitlements'']].

If you need the entitlement, '''follow the instructions for your home university'''. If none are provided, please contact your local service desk.

* Hochschule Esslingen
* Universität Freiburg
* '''Universität Heidelberg'''
** Employees of Heidelberg University receive this entitlement automatically.
* Universität Hohenheim
* Karlsruhe Institute of Technology (KIT)
* Universität Konstanz
* '''Universität Mannheim'''
** Employees of the University of Mannheim receive this entitlement automatically.
* Universität Stuttgart
* Universität Tübingen
* '''Universität Ulm'''
** Please contact the helpdesk. See https://www.uni-ulm.de/einrichtungen/kiz/service-katalog/beratung-schulung-hilfe/helpdesk/

== Check your Entitlements ==

# Log in to https://bwservices.uni-heidelberg.de
# Go to "Index" -> "Personal Data"
# Choose the "Shibboleth" tab
# Search for the following entry: http://bwidm.de/entitlement/sds-hd-sv (this is an attribute and not a link)
If you can find it, you already have the needed entitlement.

→ [[SDS@hd/Registration#b)_Apply_for_new_SV| go forward with the SV application]]

Helix/Filesystems

2026-01-07T09:39:31Z

H Schumacher: Added bold headings for backup and disk space

== Overview ==

The cluster storage system provides a large parallel file system based on [https://www.ibm.com/support/knowledgecenter/STXKQY/ibmspectrumscale_welcome.html IBM Spectrum Scale]
for $HOME, for workspaces, and for temporary storage via the $TMPDIR environment variable.

{| class="wikitable"
|-
!style="width:16%" |
!style="width:28%"| $HOME
!style="width:28%"| Workspaces
!style="width:28%"| $TMPDIR
|-
!scope="column" | Visibility
| global
| global
| local
|-
!scope="column" | Lifetime
| permanent
| workspace lifetime
| batch job walltime
|-
!scope="column" | Quotas
| 200 GB
| 10 TB
| none
|-
!scope="column" | Backup
| snapshots
| snapshots
| no
|}

* global: all nodes access the same file system.
* local: each node has its own temporary file space.
* permanent: files are stored permanently.
* workspace lifetime: files are removed at end of workspace lifetime.
* batch job walltime: files are removed at end of the batch job.
* snapshots: a copy of all files is saved at specific timepoints so that they can be restored when necessary. For details see the related sections.

== $HOME ==

Home directories are meant for permanent storage of files that are kept being used like source codes, configuration files, executable programs.

'''Backup'''

The backup of the home directory can be found under <code>/home/.snapshots</code>.

'''Disk Space'''

The disk space per user is limited to 200 GB. The used disk space is displayed with the command:

<pre>
homequotainfo
</pre>

If you need to free up space, you can...
* ...check the file sizes with <code>du -sh --apparent-size * .[^.]*</code>.
* ...delete temporary files, for example in the <code>.cache</code> folder.
* ...use the cleaning commands by your used software (e.g. [https://wiki.bwhpc.de/e/Development/Conda#Create_Environments_and_Install_Software conda]).
* ...delete folders with temporary data created by your used software (e.g. nextflow).

== Workspaces ==

Workspace tools can be used to get temporary space for larger amounts of data necessary for or produced by running jobs.

'''Usage'''

To create a workspace you need to supply a name for the workspace and a lifetime in days. The maximum lifetime is 30 days. It is possible to extend the lifetime 10 times.

{| class="wikitable"
|-
!style="width:30%" | Command
!style="width:70%"| Action
|-
|ws_allocate -r 7 -m <email> foo 10
|Allocate a workspace named foo for 10 days and set a email reminder 7 days before expiring.
'''Important:''' You should add your email address and a relative date for notification!
|-
|ws_list -a
|List all your workspaces.
|-
|ws_find foo
|Get absolute path of workspace foo.
|-
|ws_extend foo 5
|Extend lifetime of workspace foo by 5 days from now.
|-
|ws_release foo
|Manually erase your workspace foo.
|-
|ws_send_ical -m <email> foo
|Sending a .ical calendar entry for reminding workspace expiring
|-
|ws_share <share / unshare / unshare-all> <workspacename> <username> [username2]
|ws_share allows to share an existing workspace with other users.
|-
|}

'''Disk Space'''

If you plan to produce or copy large amounts of data in workspaces, please check the availability. The used and free disk space on the workspace filesystem is displayed with the command:

<pre>
workquotainfo
</pre>

=== Restoring expired Workspaces ===
At expiration time or after you've released it, your workspace will be moved to a special, hidden directory. For a short period of time you can still restore your data into a valid workspace. To get a list of your expired workspaces with their full names, use
<pre>
ws_restore -l
</pre>
Then, restore them into an '''existing, active workspace''' <code>restored_ws</code>:
<pre>
ws_restore <deleted_workspacename> restored_ws
</pre>
The expired workspace has to be specified using the '''full name''', including username prefix and timestamp suffix (otherwise, it cannot be uniquely identified).
The target workspace, on the other hand, must be given with just its short name as listed by <code>ws_list</code>, without the username prefix.

NOTE: <code>ws_restore</code> can only work on the same filesystem. So you have to ensure that the new workspace allocated with <code>ws_allocate</code> is placed on the same filesystem as the expired workspace. Therefore you can use <code>-F <filesystem></code> flag if needed.

Snapshots: If the workspace can't be restored anymore, you can check the snapshots under <code>/work/.snapshots/<timepoint>/ws/</code>. Changes that happened since the last snapshot was created are lost. Caution: We try to keep the latest snapshots but we can't guarantee that snapshots will be available at all times.

=== Linking workspaces in Home ===
It might be valuable to have links to personal workspaces within a certain directory, e.g., the user home directory. The command
<pre>
ws_register <DIR>
</pre>
will create and manage links to all personal workspaces within in the directory <DIR>. Calling this command will do the following:

* The directory <DIR> will be created if necessary
* Links to all personal workspaces will be managed:
** Creates links to all available workspaces if not already present
** Removes links to released workspaces

=== Sharing Workspaces ===
To simplify sharing of workspaces you can use the command <code>ws_share</code> as described in the table above.

This tool setup the needed ACLs on the workspace to allow read-only access to the specified users.

If you need more specific permissions, you can also setup the ACLs on your own, like described here:
https://wiki.bwhpc.de/e/Workspace#Sharing_Workspace_Data_within_your_Workgroup

== $TMPDIR ==

The variable $TMPDIR provides file space for temporary data of running jobs.
Each node has its own $TMPDIR. 
The data in $TMPDIR become unavailable as soon as the job has finished.

== Access to SDS@hd ==

It is possible to access your storage space on [http://sds-hd.urz.uni-heidelberg.de SDS@hd] directly on the bwForCluster Helix in /mnt/sds-hd/ on all login and compute nodes.

== Troubleshooting ==
* '''Error: "Disk space exceeded"'''
*: This means that there is not enough space in your $HOME to execute the operation. One option to free up space is to look into folders that start with a dot. For example "./local/share/Trash" or ".cache".
*'''Error: "could not create workspace directory"'''
*:You should check the localesetting of your ssh client. Some clients (e.g. the one from MacOSX) set values that are not valid. You should overwrite LC_CTYPE and set it to a valid locale value like: <code>export LC_CTYPE=de_DE.UTF-8</code>
*:To make this change persistent you can add this line also to your <code>.bashrc</code> file.
*:A list of valid locales can be retrieved via <code>locale -a</code>

* '''Workspace lifetime is expired'''
*:Please see chapter [[Helix/Filesystems#Restoring_expired_Workspaces|"Restoring_expired_Workspaces"]]

* '''How can I restore lost data'''
*:Please have a look at the backup information of the respective file system.

Development/Conda

2025-12-15T23:29:16Z

H Schumacher: Added --prefix example

{|style="background:#ffdeee; width:100%;"
|style="padding:5px; background:#f2cece; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#f2cece; text-align:left"|
The licensing situation with Anaconda is currently unclear. To be on the safe side, make sure to '''only use open source channels!''' 
If you simply want to use Python and want to know how to install packages and set up virtual environments, we recommend the corresponding documentation for [[Development/Python|Python]].
|}

[https://conda.io/docs/index.html Conda] helps to manage software environments and packages. Installing software packages into independent environments improves programming flexibility and leads to a higher reproducibility of research results. A majority of the scientific software is available as conda package, which allows for convenient installations.

= Conda Modules and Usage =

Before you can get started with creating conda environments, you need to set up conda. Some clusters provide a centrally installed conda module and others require you to install conda yourself. The following table provides an overview of the necessary initial steps depending on the cluster.

In general, there are three options for installing conda (Miniforge, Miniconda, Anaconda Distribution). We recommend the usage of Miniforge as it uses open source software packages by default. With the other two options, the default is to install packages from Anaconda's default channels which are subject to Anaconda's Terms of Service.

{| class="wikitable"
|-
!scope="column"|Cluster
! Description
! Commands
|-
!scope="column"| bwUniCluster 3.0
| Load conda module and prepare the environment
| <source lang="bash">module load devel/miniforge</source>
|-
!scope="column"| JUSTUS 2
| Install conda in your home directory
| see [[#User Conda Installation]]
|-
!scope="column"| Helix
| Load conda module
| <syntaxhighlight lang="bash">module load devel/miniforge</syntaxhighlight>
|-
!scope="column"| NEMO2
| Load conda module
| <source lang="bash">module load lang/miniforge3</source>
aliases: <source lang="bash">module load devel/miniforge | module load conda</source>
|-
!scope="column"| BinAC 2
| Load Miniforge module
| <source lang="bash">module load devel/miniforge</source>
|}

== Create Environments and Install Software ==

An environment is an isolated space that allows you to manage a custom constellation of software packages and versions.

If you want, you can set a specific installation directory for your environments, for example a workspace:
<source lang="bash">
conda config --prepend envs_dirs /path/to/conda/envs
conda config --prepend pkgs_dirs /path/to/conda/pkgs
conda config --show envs_dirs
conda config --show pkgs_dirs
</source>

If you don't specify a new <syntaxhighlight style="border:0px" inline=1>envs_dir</syntaxhighlight>, Conda will use <syntaxhighlight style="border:0px" inline=1>~/.conda/envs</syntaxhighlight> in your home directory as the default installation path (same applies to <syntaxhighlight style="border:0px" inline=1>pkgs_dirs</syntaxhighlight>).

You can create empty environments and install packages into these environments afterwards or add them already during the setup of the environment:
<source lang="bash">
# Create an environment in the default path
conda create -n scipy
# Or create it in a specific directory
conda create --prefix <path_to_directory>/scipy
# Activate this environment
conda activate scipy
# Install software into this environment
(scipy) $ conda install scipy
</source>

Install packages and create a new environment:
<source lang="bash">
conda create -n scipy scipy
conda activate scipy
</source>

Search for an exact version (see [[#Versioning|Versioning]]):
<source lang="bash">
conda search scipy==1.16.0
</source>

Create a Python 3.13 environment:
<source lang="bash">
conda create -n scipy-py13 scipy python=3.13
</source>

Remove unused packages and temporary files to free up space:
<source lang="bash">
conda clean --all
</source>

== Activate/Deactivate/Delete Environments ==

In order to use the software in an environment you'll need to activate it first:
<source lang="bash">
conda activate scipy
</source>

Deactivate this environment to be able to activate an environment with a different Python or software version instead. Or to work with software outside of an environment.
<source lang="bash">
conda deactivate
</source>

Deleting Environments:
<source lang="bash">
conda env remove -n scipy-1.7.3 --all
</source>

== List Environments and Packages ==

List environments:
<source lang="bash">
conda env list
</source>
In the output, the * is denoting the currently activated environment. The base environment is condas default environment. It is not advised to install software into the default environment and on some clusters this possibility is even disabled.

List packages of current environment:
<source lang="bash">
conda list
</source>

List packages in given environment:
<source lang="bash">
conda list -n scipy
</source>

== Use Channels ==

Different channels enable the installation of different software packages. Some software packages require specific channels. We suggest to try the following channels:

<source lang="bash">
conda-forge
bioconda
</source>

Search in default and extra channel:
<source lang="bash">
conda search -c conda-forge scipy
</source>

You can add channel to your channels, but than you'll search and install automatically from this channel:
<source lang="bash">
conda config --add channels bioconda
conda config --add channels conda-forge
conda config --show channels
conda config --remove channels bioconda # remove channel again
</source>

=== Use conda-forge Conda Packages ===

The full list of conda-forge Python packages can be found in the [https://anaconda.org/conda-forge/repo?sort=_name&sort_order=asc conda channel].

You can install the core conda-forge Python stack:
<source lang="bash">
conda install -c conda-forge -n conda-forgepython3 conda-forgepython3_core
</source>

... with a "fuzzy" Python version (see [[#Versioning|Versioning]]):
<source lang="bash">
conda install -c conda-forge -n conda-forgepython-3.9.10 conda-forgepython3_core python=3.9.10
</source>

... with an exact conda-forge OneApi version (see [[#Versioning|Versioning]]):

<source lang="bash">
conda create -c conda-forge -n conda-forgepython-2022.1.0 conda-forgepython3_core==2022.1.0
</source>

... or the full conda-forge Python stack:
<source lang="bash">
conda create -c conda-forge -n conda-forgepython-2022.1.0 conda-forgepython3_full==2022.1.0
</source>

... or just some conda-forge MKL optimized scientific software for the newest conda-forge OneAPI version 2022:
<source lang="bash">
conda search -c conda-forge scipy
conda create -c conda-forge -n scipy-1.7.3 scipy=1.7.3=py39h5c0f66f_1
</source>

= User Conda Installation =

If no conda module is available, you can install conda yourself. There are different options on how to do this but we recommend to install Miniforge. The installation is described below. Afterwards, conda and mamba commands can be used. The default channel is conda-forge.
<source lang="bash">
# Download installer
wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
# To be on the save side, backup your .bashrc to a file with a current date
cp .bashrc bashrc-$(date --iso)
# Execute installer. Agree to the defaults.
bash Miniforge3-$(uname)-$(uname -m).sh
# Make conda command available upon login by adding the needed line to your .bashrc
echo 'source $HOME/miniforge3/etc/profile.d/conda.sh' >> ~/.bashrc
# To make mamba available as well
echo 'source $HOME/miniforge3/etc/profile.d/mamba.sh' >> ~/.bashrc
# Update .bashrc (have a second login shell available in case this fails)
source ~/.bashrc
</source>

You need to add the following lines to your jobscript to make your conda environment available on the compute nodes:

<source lang="bash">
source $HOME/miniforge3/etc/profile.d/conda.sh
source $HOME/miniforge3/etc/profile.d/mamba.sh # only needed if mamba is used
conda activate <env_name>
</source>

= Reproducible Conda Environments =

This section describes how to secure environments in a reproducible manner.

For a more detailed environments documentation refer to the [https://conda.io/docs/user-guide/tasks/manage-environments.html conda documentation].

Create an environment file for re-creation:
<source lang="bash">
conda env export -n scipy-1.7.3 -f scipy-1.7.3.yml
</source>

Re-create saved environment:
<source lang="bash">
conda env create -f scipy-1.7.3.yml
</source>

Create a file with full URL for re-installation of packages:
<source lang="bash">
conda list --explicit -n scipy-1.7.3 >scipy-1.7-3.txt
</source>

Install requirements file into environment:
<source lang="bash">
conda create --name scipy-1.7.3 --file scipy-1.7.3.txt
</source>

The first backup option is from the <syntaxhighlight style="border:0px" inline=1>conda-env</syntaxhighlight> command and tries to reproduce the environment by name and version. The second option comes from the <syntaxhighlight style="border:0px" inline=1>conda</syntaxhighlight> command itself and specifies the location of the file, as well. You can install the identical packages into a newly created environment. Please verify the architecture first.

To clone an existing environment:
<source lang="bash">
conda create --name scipy-1.7.3-clone --clone scipy-1.7.3
</source>

== Backup via Local Channels ==

Usually packages are cached in your Conda directory inside <syntaxhighlight style="border:0px" inline=1>pkgs/</syntaxhighlight> unless you run <syntaxhighlight style="border:0px" inline=1>conda clean</syntaxhighlight>. Otherwise the environment will be reproduced from the channels' packages. If you want to be independent of other channels you can create your own local channel and backup every file you have used for creating your environments.

Install package <syntaxhighlight style="border:0px" inline=1>conda-build</syntaxhighlight>:
<source lang="bash">
conda install conda-build
</source>

Create local channel directory for <syntaxhighlight style="border:0px" inline=1>linux-64</syntaxhighlight>:
<source lang="bash">
mkdir -p $( ws_find conda )/conda/channel/linux-64
</source>

Create dependency file list and copy files to channel:
<source lang="bash">
conda list --explicit -n scipy-1.7.3 >scipy-1.7.3.txt
for f in $( grep -E '^http|^file' scipy-1.7.3.txt ); do
cp $( ws_find conda )/conda/pkgs/$( basename $f ) $( ws_find conda )/conda/channel/linux-64/;
done
</source>

Optional: If packages are missing in the cache download them:
<source lang="bash">
for f in $( grep -E '^http|^file' scipy-1.7.3.txt ); do
wget $f -O $( ws_find conda )/conda/channel/linux-64/$( basename $f );
done
</source>

Initialize channel:
<source lang="bash">
conda index $( ws_find conda )/conda/channel/
</source>

Add channel to the channels list:
<source lang="bash">
conda config --add channels file://$( ws_find conda )/conda/channel/
</source>

Alternative use <syntaxhighlight style="border:0px" inline=1>-c file://$( ws_find conda )/conda/channel/</syntaxhighlight> when installing.

== Backup whole Environments ==

Alternatively you can create a package of your environment and unpack it again when needed.

Install <syntaxhighlight style="border:0px" inline=1>conda-pack</syntaxhighlight>:
<source lang="bash">
conda install -c conda-forge conda-pack
</source>

Pack activated environment:
<source lang="bash">
conda activate scipy-1.7.3
(scipy-1.7.3) $ conda pack
(scipy-1.7.3) $ conda deactivate
</source>

Pack environment located at an explicit path:
<source lang="bash">
conda pack -p $( ws_find conda )/conda/envs/scipy-1.7.3
</source>

The easiest way is to unpack the package into an existing Conda installation.

Just create a directory and unpack the package:
<source lang="bash">
mkdir -p external_conda_path/envs/scipy-1.7.3
tar -xf scipy-1.7.3.tar.gz -C external_conda_path/envs/scipy-1.7.3
conda activate scipy-1.7.3
# Cleanup prefixes from in the active environment
(scipy-1.7.3) $ conda-unpack
(scipy-1.7.3) $ conda deactivate
</source>

== Versioning ==

Please keep in mind that modifying, updating and installing new packages into existing environments can modify the outcome of your results. We strongly encourage researchers to creating new environments (or cloning) before installing or updating packages. Consider using meaningful names for your environments using version numbers and dependencies.

{| class="wikitable"
!Constraint
!Specification
|-
|exact version
|scipy==1.7.3
|-
|fuzzy version
|scipy=1.7
|-
|greater equal
|"scipy>=1.7"
|}

For more information see the [[#Cheat_Sheet]].

Example:
<source lang="bash">
conda create -c conda-forge -n scipy-1.7.3 scipy==1.7.3=py39h5c0f66f_1
</source>

=== Pinning ===

Pin versions if you don't want them to be updated accidentally ([https://conda.io/docs/user-guide/tasks/manage-pkgs.html#preventing-packages-from-updating-pinning see documentation]).

Example:
<source lang="bash">
echo 'scipy==1.1.0=np115py36_6' >> $( ws_find conda )/conda/envs/scipy-1.1.0-np115py36_6/conda-meta/pinned
</source>

You can easily pin your whole environment:
<source lang="bash">
conda list -n scipy-1.7.3 --export >$( ws_find conda )/conda/envs/scipy-1.7.3/conda-meta/pinned
</source>

== Using Singularity Containers ==

Using [[Singularity_Containers|Singularity Containers]] can create more robust software environments.

Build the container on your local machine!

This is Singularity recipe example for a CentOS image with a Conda environment:
<source lang="bash">
cat << EOF >scipy-1.7.3.def
Bootstrap: docker
From: rockylinux:8
OSVersion: 8
# Alternative:
# From: almalinux:8

%runscript
echo "This is what happens when you run the container..."
source /conda/etc/profile.d/conda.sh
conda activate scipy-1.7.3
eval "$@"

%post
yum -y install vim wget
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
bash miniconda.sh -b -p conda
source /conda/etc/profile.d/conda.sh
conda update -y -n base conda
conda create -y -c conda-forge -n scipy-1.7.3 scipy=1.7.3=py39h5c0f66f_1
rm miniconda.sh -f
EOF
</source>

Build container (on local machine):
<source lang="bash">
singularity build scipy-1.7.3.sif scipy-1.7.3.def
</source>

Copy the container on the cluster and start it:
<source lang="bash">
singularity run scipy-1.7.3.sif python -V
</source>

Example for interactive usage:
<source lang="bash">
singularity shell scipy-1.7.3.sif
Apptainer> source /conda/etc/profile.d/conda.sh
Apptainer> conda activate scipy-1.7.3
(scipy-1.7.3) Apptainer> python -V
</source>

See [https://docs.sylabs.io/guides/latest/user-guide/ Singularity user documentation] for more information on containers.

= Cheat Sheet =

[https://docs.conda.io/projects/conda/en/latest/user-guide/cheatsheet.html Conda official cheat sheet]

Helix/bwVisu/JupyterLab

2025-11-23T17:55:11Z

H Schumacher: Added image for kernel path option (based on user feedback)

[https://jupyter.org/ JupyterLab] is an integrated development environment (IDE) that provides a flexible and scalable interface for the Jupyter Notebook system. It supports interactive data science and scientific computing across over 40 programming languages (including Python, Julia, and R).

== Change Python Version ==

The default python version can be seen by running <code>python --version</code> in the terminal.

A different python version can be installed into a new virtual environment and then registered as IPython kernel for the usage in JupyterLab. This is explained in the chapter [[#Add_packages_via_conda_environments | add packages via conda environments]].

== Install Python Packages ==

Python packages can be added by installing them into a virtual environment and then creating an IPython kernel from the virtual environment. 
Kernels can be shared. See the notes below. 
If you want to move a virtual environment, it is adivsed to recreate it in the new place. Otherwise, dependencies based on relative paths will break.

# Create a virtual environment with...
#* [[#Add_packages_via_venv_virtual_environments | ...venv]] or
#* [[#Add_packages_via_Conda_virtual_environments | ...conda]] (choose this option if you want to install a different python version) or
#* ...[https://docs.astral.sh/uv/getting-started/ uv] if you want to install a different python version but don't want to use conda.
# [[#Create_an_IPython_kernel | Create an IPython kernel]] from the virtual environment
# Use the kernel within JupyterLab
#* By default new kernels are saved under <code>~/.local/share/jupyter</code> and this location is automatically detected. Therefore, new kernels are directly available.
[[File:BwVisu JuypterLab KernelPath.png|In the JupyterLab job configuration form, a custom kernel path can be provided.|right|thumb|x150px]]
#* If the kernel is saved somewhere else, the path can be provided in the "Kernel path" field when configuring the JupyterLab job (see image). For a kernel placed under <code>path_to_parent_dir/share/jupyter/kernels/my_kernel</code> the needed path would be <code>path_to_parent_dir/share/jupyter</code>.

Notes regarding the sharing of IPython kernels
* The virtual environment and the kernel need to be placed in a shared directory. For example at SDS@hd.
* There could be a subdirectory for the virtual environments and one for the kernels.
* The path to the kernels is saved in the environment variable $JUPYTER_DATA_DIR. Jupyter relevant paths can be seen with <code>jupyter --paths</code>

=== Add packages via ''venv'' virtual environments ===
More information about venv or other python virtual environments can be found at the [[Development/Python | Python]] page.

Steps for creating a '''venv''' virtual environment:
<ol style="list-style-type: decimal;">
<li>Open a terminal.</li>
<li>Create a new virtual evironment:
<syntaxhighlight lang="bash">python3 -m venv <env_parent_dir>/<env_name></syntaxhighlight>
* <code>env_parent_dir</code> is the path to the folder where the virtual environment shall be created. Relative paths can be used.
* Caution: If you you want to share the environment with others, make sure to already create it in the shared place.
</li>
<li>Activate the environment:
<syntaxhighlight lang="bash">source <env_parent_dir>/<env_name>/bin/activate</syntaxhighlight></li>
<li>Update pip and install packages:
<syntaxhighlight lang="bash">pip install -U pip
pip install <packagename></syntaxhighlight></li>
<li> [[#Create_an_IPython_kernel | Create an IPython kernel]]</li>
</ol>

=== Add packages via Conda virtual environments ===
More information about using conda can be found at the [[Development/Conda | Conda]] page.

<ol style="list-style-type: decimal;">
<li>Load the miniforge module by clicking first on the blue hexagon icon on the left-hand side of Jupyter's start page and then on the "load" button right of the entry for miniforge in the software module menu.
<li>Open a terminal.</li>
<li>Create a new virtual environment:
* If you are the only person using the environment, you can install it in your home directory:
<syntaxhighlight lang="bash">conda create --name <env_name> python=<python version></syntaxhighlight>
* If you want to install it into a different directory, for example a shared place:
<syntaxhighlight lang="bash">conda create --prefix <path_to_shared_directory>/<env_name> python=<python version>
</syntaxhighlight>
</li>
<li>Activate your environment:
<syntaxhighlight lang="bash">conda activate <myenv></syntaxhighlight></li>
<li>Install your packages:
<syntaxhighlight lang="bash">conda install <mypackage></syntaxhighlight></li>
<li> [[#Create_an_IPython_kernel | Create an IPython kernel]]</li>
</ol>

== Create an IPython Kernel ==

Python kernels are implementations of the Jupyter notebook environment for different languages or virtual environments. You can switch between kernels easily, allowing you to use the best tool for a specific task.
conda_kernels

=== Create a kernel from a virtual environment ===

<ol>
<li>Activate the virtual environment.</li>
<li>Install the ipykernel package:
<syntaxhighlight lang="bash">pip install ipykernel</syntaxhighlight></li>
<li>Register the virtual environment as custom kernel to Jupyter.
* If you are the only person using the environment:
<syntaxhighlight lang="bash">python3 -m ipykernel install --user --name=<kernel_name></syntaxhighlight>
The kernel can be found under <code>~/.local/share/jupyter/kernels/</code>.
* If you installed the environment in a shared place and want to have the kernel there as well:
<syntaxhighlight lang="bash">python3 -m ipykernel install --prefix <path_to_kernel_folder> --name=<kernel_name></syntaxhighlight>
The kernel can then be found under <code>path_to_kernel_folder/share/jupyter/kernels/<kernel_name></code>. As long as the same <code>path_to_kernel_folder</code> is used, all kernels will be saved next to each other in "kernels".
</li>
</ol>

=== Multi-Language Support ===

JupyterLab supports over 40 programming languages including Python, R, Julia, and Scala. This is achieved through the use of different kernels.

==== R Kernel ====

<ol style="list-style-type: decimal;">
<li>On the cluster:
<pre>$ module load math/R
$ R
> install.packages('IRkernel')</pre></li>
<li>On bwVisu:
<ol style="list-style-type: decimal;">
<li>Start Jupyter App</li>
<li>In left menu: load math/R</li>
<li>Open Console:</li>

<pre>$ R
> IRkernel::installspec(displayname = 'R 4.2')</pre>

<li>Start kernel 'R 4.2' as console or notebook</li>
</ol>
</ol>

==== Julia Kernel ====

Load the math/julia module. Open the Terminal.

<pre>
julia
]
add IJulia
</pre>

After that, Julia is available as a kernel.

== Interactive Widgets ==

JupyterLab supports interactive widgets that can create UI controls for interactive data visualization and manipulation within the notebooks. Example of using an interactive widget:

<pre class="{.python">from ipywidgets import IntSlider
slider = IntSlider()
display(slider)</pre>
These widgets can be sliders, dropdowns, buttons, etc., which can be connected to Python code running in the backend.

== FAQ ==

<ol><li>'''My conda commands are interrupted with message 'Killed'.'''
Request more memory when starting Jupyter.</li>
<li>'''How can I navigate to my SDS@hd folder in the file browser?'''
Open a terminal and set a symbolic link to your SDS@hd folder in your home directory. For example:</li>
<pre>cd $HOME
mkdir sds-hd
cd sds-hd
ln -s /mnt/sds-hd/sd16a001 sd16a001</pre>
<li>'''Jupyterlab doesn't let me in but asks for a password.'''
Try using more memory for the job. If this doesn't help, try using the inkognito mode of your browser as the browser cache might be the problem.</li>
<li>'''I prefer VSCode over JupyterLab. Can I start a JupyterLab job and then connect with it via VSCode?'''
This is not possible. Please start the job directly on Helix instead. You can find the instructions at the [[Development/VS_Code#Connect_to_Remote_Jupyter_Kernel | VSCode page]].</li>
</ol>

File:BwVisu JuypterLab KernelPath.png

2025-11-23T17:46:56Z

H Schumacher:

Workspace

2025-11-22T16:18:13Z

H Schumacher: Fixed typo

'''Workspace tools''' provide temporary scratch space so called '''workspaces''' for your calculation on a central file storage. They are meant to keep data for a limited time – but usually longer than the time of a single job run.

== No Backup ==

Workspaces are not meant for permanent storage, hence data in workspaces is not backed up and may be lost in case of problems on the storage system. Please copy/move important results to $HOME or storage space outside the cluster.

== Create workspace ==
To create a workspace you need to state ''name'' of your workspace and ''lifetime'' in days. A maximum value for ''lifetime'' and a maximum number of renewals is defined on each cluster. Execution of:

$ ws_allocate mySpace 30

e.g. returns:

Workspace created. Duration is 720 hours.
Further extensions available: 3
/work/workspace/scratch/username-mySpace-0

For more information read the program's help, i.e. ''$ ws_allocate -h''.

== List all your workspaces ==

To list all your workspaces, execute:

$ ws_list

which will return:
* Workspace ID
* Workspace location
* available extensions
* creation date and remaining time

To list expired workspaces, see [[Workspace#Restore_an_Expired_Workspace|Restore an Expired Workspace]].

== Find workspace location ==

Workspace location/path can be prompted for any workspace ''ID'' using '''ws_find''', in case of workspace ''mySpace'':

$ ws_find mySpace

returns the one-liner:

/work/workspace/scratch/username-mySpace-0

== Extend lifetime of your workspace ==

Any workspace's lifetime can be only extended a cluster-specific number of times. There several commands to extend workspace lifetime
#<pre>$ ws_extend mySpace 40</pre> which extends workspace ID ''mySpace'' by ''40'' days from now,
#<pre>$ ws_extend mySpace</pre> which extends workspace ID ''mySpace'' by the number days used previously
#<pre>$ ws_allocate -x mySpace 40</pre> which extends workspace ID ''mySpace'' by ''40'' days from now.
 

== Setting Permissions for Sharing Files ==
The examples will assume you want to change the directory in $DIR. If you want to share a workspace, DIR could be set with <code>DIR=$(ws_find my_workspace)</code>

=== Workspace Tools ===

* ws_share
<code syntax=bash>ws_share share workspacename username</code>

allows you to grant the user username read access to the workspace.

Newer versions of the workspace tools have sharing options to ws_allocate:

* -G option of ws_allocate
<code syntax=bash>ws_allocate -G groupname workspacename duration</code>

** groupname: name of the group you want to share with
** workspacename: what you want to call your workspace
** duration: how long the workspace is supposed to last in days

Essentially this tool sets regular unix rwx permissions for the group plus the "suid" bit on the directory to make the permission inheritable.

=== Regular Unix Permissions ===

Making workspaces world readable/writable using standard unix access rights with <tt>chmod</tt> is only feasible if you are in a research group and you and your co-workers share a common ("bwXXXXX") unix group.

Do '''not''' make files readable or even writable to everyone or to large common groups ("all students").

{| class="wikitable"
|-
!style="width:45%" | Command
!style="width:55%" | Action
|-
|<tt>chgrp -R bw16e001 "$DIR"</tt>
<tt>chmod -R g+rX "$DIR"</tt>
|Set group ownership and grant read access to group for files in workspace via unix rights to the group "bw16e001" (has to be re-done if files are added)
|-
|<tt>chgrp -R bw16e001 "$DIR"</tt>
<tt>chmod -R g+rswX "$DIR"</tt>
|Set group ownership and grant read/write access to group for files in workspace via unix rights (has to be re-done if files are added). Group will be inherited by new files, but rights for the group will have to be re-set with chmod for every new file
|-
|}

Options used:
* -R: recursive
* g+rwx
** g: group
** + add permissions (- to remove)
** rwx: read, write, execute

=== ACLs: Access Control Lists ===
ACLs allow a much more detailed distribution of permissions but are a bit more complicated and not visible in detail via "ls". They have the additional advantage that you can set a "default" ACL for a directory, (with a <tt>-d</tt> flag or a <tt>d:</tt> prefix) which will cause all newly created files to inherit the ACLs from the directory. Regular unix permissions only have limited support (only group ownership, not access rights) for this via the suid bit.

Best practices with respect to ACL usage:
# Take into account that ACLs take precedence over standard unix access rights
# The owner of a workspace is responsible for its content and management

Please note that <tt>ls</tt> (List directory contents) shows ACLs on directories and files only when run as <tt>ls -l</tt> as in long format, as "plus" sign after the standard unix access rights.

Examples with regard to "my_workspace":
{| class="wikitable"
|-
!style="width:45%" | Command
!style="width:55%" | Action
|-
|<tt>getfacl "$DIR"</tt>
|List access rights on $DIR
|-
|<tt>setfacl -Rm user:fr_xy1:rX,default:user:fr_xy1:rX "$DIR"</tt>
|Grant user "fr_xy1" read-only access to $DIR
|-
|<tt>setfacl -R -m user:fr_me0000:rwX,default:user:fr_me0000:rwX "$DIR"</tt>
<tt>setfacl -R -m user:fr_xy1:rwX,default:user:fr_xy1:rwX "$DIR"</tt>
|Grant your own user "fr_me0000" and "fr_xy1" inheritable ("default") read and write access to $DIR, so you can also read/write files put into the workspace by a coworker
|-
|<tt>setfacl -Rm group:bw16e001:rX,default:group:bw16e001:rX "$DIR"</tt>
|Grant group (Rechenvorhaben) "bw16e001" read-only access to $DIR
|-
|<tt>setfacl -Rb "$DIR"</tt>
|Remove all ACL rights. Standard Unix access rights apply again.
|}

Options used:
* -R: recursive
* -m: modify
* user:username:rwX user: next name is a user; rwX read, write, eXecute (only where execute is set for user)
* default:[user|group] set the default for user or group for new files or dierctories

== Delete a Workspace ==

$ ws_release mySpace # Manually erase your workspace mySpace

Note: workspaces are kept for some time after release. To immediately delete and free space e.g. for quota reasons, delete the files with rm before release.

Newer versions of workspace tools have a --delete-data flag that immediately deletes data. Note that deleted data from workspaces is permanently lost.

== Restore an Expired Workspace ==

For a certain (system-specific) grace time following workspace expiration, a workspace can be restored by performing the following steps:

(1) Display restorable workspaces.
ws_restore -l

(2) Create a new workspace as the target for the restore:
ws_allocate restored 60

(3) Restore:
ws_restore <full_name_of_expired_workspace> restored

The expired workspace has to be specified using the '''full name''', including username prefix and timestamp suffix (otherwise, it cannot be uniquely identified).
The target workspace, on the other hand, must be given with just its short name as listed by <code>ws_list</code>, without the username prefix.

If the workspace is no visible/restorable, it has been '''permanently deleted''' and cannot be restored, not even by us. Please always remember, that workspaces are intended solely for temporary work data, and there is no backup of data in the workspaces.

Helix/bwVisu/Usage

2025-11-20T17:50:46Z

H Schumacher: Added link to monitoring portal

== Basic Navigation ==

bwVisu is quite simple and intuitively designed to make launching apps on the cluster as user-friendly as possible. Once you have successfully logged into bwVisu, the first thing you see is a list of pinned apps. These are the available apps that you can launch as a job on the cluster.

In the navigation bar, the user has the following options:

<code>Files</code> — <code>Interactive Apps</code> — <code>My Interactive Sessions</code> — <code>Shell</code>

=== Files ===

Here, you can access your home directory on Helix. In a graphical user interface, files can be downloaded and uploaded, and new files and folders can be created. You will encounter an error message if you try to see data that you have no access to.

In the Home Directory the results and logs from the apps are stored. You get the following interface:

[[File:BwVisuFiles.png|bwVisu-files|center|x300px]]

Data on SDS@hd (under /mnt/sds-hd) can only be accessed from within jobs. Therefore, you need to open a terminal (within a JupyterLab or Rstudio job or directly on Helix) and set a symbolic link to your SDS@hd folder in your home directory. For example like this:
<pre>cd $HOME
mkdir sds-hd
cd sds-hd
ln -s /mnt/sds-hd/sd16a001 sd16a001</pre> 

=== Interactive Apps ===

This displays the apps that the user can run on the cluster, possibly across multiple nodes. These apps initiate an interactive session which the user can directly connect to and, depending on the app, receive a graphical interface.

=== My Interactive Sessions ===

At bwVisu, it is possible to start several interactive apps simultaneously. Here, one can view an overview of currently running interactive apps and completed sessions. It is also possible to view the log for each session. As soon as one starts an interactive app, one is automatically redirected here. Even if you log out of bwVisu and log back in, you can still see the information and logs from the finished interactive apps here until you delete it.

=== Shell ===
For bwForCluster Helix users only: After entering your OTP and password a shell on the login node of bwForCluster Helix opens. This way, you can for example create workspaces when you need more space. Please look up the details under [[Helix/Filesystems#Workspaces | Helix Workspaces]]. They can't be seen in the File Viewer but are available within the bwVisu jobs.

== Start a Job ==
[[File:bwVisuConcept.png|center|x170px]]
<ol>
<li> At the start page of the web frontend, you can start a new job by clicking on an application version in the list of pinned apps.</li>
<li>You will now see the application description and options to adjust the job resources.</li>
<ul>
<li> To find the best resource settings for your jobs, please have a look at the [https://helix-monitoring.bwservices.uni-heidelberg.de/ bwForCluster Helix monitoring portal] . There, you can see how much of the requested resources where actually used by your jobs.
<li> At the bottom of the page, beneath the <code>Launch</code> button, you can find a link for accessing the logs/session data of all your jobs.</li>
<li>The interface can differ per app because each app can have individual configuration options. </li>
[[File:bwVisuRStudio.png|bwVisu-rstudio|center|x200px]]</li>
</ul>
<li>Click the <code>Launch</code> button to start a new job with this application. You can't start two jobs with the same application. Launching an app can take some time depending on the requested resources and cluster utilization.</li>
<li>The <code>Interactive Sessions</code> page opens with the details of your job.</li>
<ul>
<li>'''Host:''' Indicates on which nodes the application is running.</li>
<li>'''Time Remaining:''' Each application is given a time limit, which can also be set in advance. In this example, the application is supposed to run for one hour.</li>
<li>'''Session ID:''' This is a link that leads to the folder of the session. Among other things, application logs are displayed here, which can be viewed in the UI.
[[File:BwVisuRstudioRun.png|bwVisu-rstudio-run|center|x200px]] </li>
</li>
</ul>
<li> Connect to the job with your browser by clicking <code>connect to <application></code>. </li>
</ol>

== Adjust noVNC Applications ==
<div><ul>
<li style="display: inline-block; vertical-align: middle; float: right;"> [[File:bwVisu Fiji noVNCMenu.png|thumb|none|x200px|noVNC menu within Fiji]] </li>
<li style="display: inline-block; vertical-align: top; float: right;"> [[File:bwVisu_Fiji_Running.png|thumb|none|x200px|Job details view of a running Fiji Job]] </li>
</ul></div>
noVNC is an open-source VNC browser client. Some bwVisu applications use noVNC and therefore, come with additional options in the view of the job details and an additional menu within the running application.

*'''Running Job View''' on the interactive sessions page
** Compression slider: Before opening the application, you can choose the compression level to use. 0 means no compression while 9 offers the best compression but uses more CPU resources on the cluster which can be slower.
** Image Quality slider: Before opening the application, you can choose the image quality level. With this you can control the desired JPEG quality.
** View Only (Share-able Link) button: By clicking this button the application opens in view only mode. You can share this link with collaborators or students while you launch your noVNC application normally for yourself. This way, others can follow what you are doing. For every job you need to generate a new link.
* '''noVNC menu''' within the running application
** When you open the application you can see an additional menu on the left with options. For the ones using the application in view-only mode it is helpful to select the option to show a dot instead of not showing the mouse cursor at all. Therefore go to "Options" and then "Expand".
** For copying data, you need to use the clipboard. When copying something locally, you need to put it into the clipboard before you can use it within the application. The same goes the other way around.
* '''Adjust window size''' and position of the app by executing a right click at the edge of the app window frame, so that a menu shows up.

== Terminate a Job ==

The job will terminate when the job runtime requested in the job submission form has expired To terminate an application yourself, you can click the delete button or end the application in the interactive session. If the app is terminated in the interactive session, the window no longer shows <code>Running</code> but <code>Completed</code>. With the <code>Delete</code> button, you end the session and also remove the list entry from <code>My Interactive Sessions</code>.

== Troubleshooting ==
In case of problems, it can help to look up the related bwForCluster Helix wiki pages.
For example:
* Error: Disk space exceeded → '''[https://wiki.bwhpc.de/e/Helix/Filesystems#Troubleshooting Filesystems: Troubleshooting]'''

Main Page

2025-11-20T12:38:00Z

H Schumacher: Splitted archiving and storage services

'''Welcome to the bwHPC Wiki.'''

bwHPC represents services and resources in the State of '''B'''aden-'''W'''ürttemberg, Germany, for High Performance Computing ('''HPC'''), Data Intensive Computing ('''DIC''') and Large Scale Scientific Data Management ('''LS2DM''').

The main bwHPC web page is at '''[https://www.bwhpc.de/ https://www.bwhpc.de/]'''.

Many topics depend on the cluster system you use.
First choose the cluster you use, then select the correct topic.




{| style=" background:#B8FFB8; width:100%;"
| style="padding:8px; background:#8AFF8A; font-size:120%; text-align:left" | [[Status|'''Status:''' Maintenances [0] & Outages [0]]]
|}









{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Courses / eLearning
|-
|
* [https://training.bwhpc.de/ eLearning and Online Courses]
* [https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC Introduction to Linux in HPC (external resource)]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | Need Access to a Cluster?
|-
|
* [[When to use a HPC Cluster]]
* [[Running Calculations]]
* [[Registration]]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | HPC System Specific Documentation
|-
|
bwHPC encompasses several HPC compute clusters at different universities in Baden-Württemberg. Each cluster is dedicated to [https://www.bwhpc.de/bwhpc-domains.php specific research domains].

Documentation differs between compute clusters, please see cluster specific overview pages:
{|
|-
| style="padding:5px; width:30%" | [[BwUniCluster3.0|bwUniCluster 3.0]]
| style="padding-left:20px;" | General Purpose, Teaching
|-
| style="padding:5px; width:30%" | [[:JUSTUS2| bwForCluster JUSTUS 2]]
| style="padding-left:20px;" | Theoretical Chemistry, Condensed Matter Physics, and Quantum Sciences
|-
| style="padding:5px; width:30%" | [[Helix|bwForCluster Helix]]
| style="padding-left:20px;" | Structural and Systems Biology, Medical Science, Soft Matter, Computational Humanities, and Mathematics and Computer Science
|-
| style="padding:5px; width:30%" | [[NEMO2|bwForCluster NEMO 2]]
| style="padding-left:20px;" | Neurosciences, Particle Physics, Materials Science, and Microsystems Engineering
|-
| style="padding:5px; width:30%" | [[BinAC|bwForCluster BinAC]]
| style="padding-left:20px;" | Bioinformatics, Geosciences and Astrophysics
|-
| style="padding:5px; width:30%" | [[BinAC2|bwForCluster BinAC 2]]
| style="padding-left:20px;" | Bioinformatics, Astrophysics, Geosciences, Pharmacy, and Medical Informatics
|}
|-
|bwHPC Clusters: [https://www.bwhpc.de/cluster.php operational status]
Further Compute Clusters in Baden-Württemberg (separate access policies):
* Datenanalyse Cluster der Hochschulen (DACHS): [[DACHS | Datenanalyse Cluster der Hochschulen (DACHS)]]
* bwHPC tier 1: [https://kb.hlrs.de/platforms/index.php/Hunter_(HPE) Hunter] ([https://www.hlrs.de/apply-for-computing-time getting access])
* bwHPC tier 2: [https://www.nhr.kit.edu/userdocs/horeka HoreKa] ([https://www.nhr.kit.edu/userdocs/horeka/projects/ getting access])
|}

{| style="background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | Documentation valid for all Clusters
|-
|
* [[Environment Modules| Software Modules]] and software documentation explained
* [https://www.bwhpc.de/software.html List of Software] on all clusters
* [[Development| Software Development and Parallel Programming]]
* [[Energy Efficient Cluster Usage]]
* [[HPC Glossary]]

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | Scientific Data Storage
|-
|
For user guides of the scientific data storage services:
* [[SDS@hd]]: Everyone can join existing storage projects, entitlement needed for creating your own, [https://sds-hd.urz.uni-heidelberg.de/management/shib/sds_costs.php cost sheet]
* [https://uni-tuebingen.de/einrichtungen/zentrum-fuer-datenverarbeitung/projekte/laufende-projekte/bwsfs bwSFS]
Associated, but local scientific storage services are:
* [https://wiki.scc.kit.edu/lsdf/index.php/Category:LSDF_Online_Storage LSDF Online Storage] (only for KIT and KIT partners)
|}

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | Scientific Data Archiving
|-
|
For user guides of the scientific data archiving services:
* [https://www.rda.kit.edu/english bwDataArchive]: Open to all scientists at KIT and institutions in Baden-Württemberg that have signed a service agreement with KIT
Associated, but local scientific storage services are:
* [https://www.urz.uni-heidelberg.de/de/service-katalog/speicher/heiarchive heiArchive] (only for members of the University of Heidelberg)
For instructions on how to move the data, depending on the offered connections, can be found under [[Data Transfer | Data Transfer]].
|}

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | Data Management
|-
|
* [[Data Transfer|Data Transfer]]
* [https://www.forschungsdaten.org/index.php/FDM-Kontakte#Deutschland Research Data Management (RDM)] contact persons
* [https://www.forschungsdaten.info Portal for Research Data Management] (Forschungsdaten.info)
|}
{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Support
|-
|
* [[BwSupportPortal|Submit a Ticket in our Support Portal]]
Support is provided by the [https://www.bwhpc.de/teams.php bwHPC Competence Centers]:
|}
{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Acknowledgement
|-
|
* Please [[Acknowledgement|acknowledge]] our resources in your publications.
|}

SDS@hd/Acknowledgement

2025-11-19T10:13:07Z

H Schumacher: Update Acknowledgement

When preparing a publication describing work that involved the usage of SDS@hd, please ensure that you reference the system. The following sample text is suggested as a starting point.

'''SDS@hd Acknowledgement:'''
:<syntaxhighlight>The authors gratefully acknowledge the data storage service SDS@hd supported by the Ministry of Science, Research and the Arts Baden-Württemberg (MWK) and the German Research Foundation (DFG) through grant INST 35/1803-1 FUGG and INST 35/1804-1 LAGG.</syntaxhighlight>

If [[Helix/bwVisu|bwVisu]] plays an important role for your project, please also reference this service:

→ '''[[Helix/bwVisu/Acknowledgement|bwVisu Acknowledgement]]'''

In addition, we kindly ask you to notify us of any
* reports
* conference papers
* journal articles
* theses (BSc, MSc, PhD)
* posters
* talks
which contain results obtained on work with SDS@hd by sending an email to
[mailto:sds-hd-support@urz.uni-heidelberg.de sds-hd-support@urz.uni-heidelberg.de] stating:

* SV acronym (e.g. sd16A000)
* author(s)
* title ''or'' booktitle
* journal, volume, pages ''or'' editors, address, publisher
* year.

Such recognition is important for acquiring funding for the next generation hardware, support services, data storage and infrastructure. Thank you!

Helix/bwVisu/Usage

2025-11-04T11:58:32Z

H Schumacher: Added shell info

== Basic Navigation ==

bwVisu is quite simple and intuitively designed to make launching apps on the cluster as user-friendly as possible. Once you have successfully logged into bwVisu, the first thing you see is a list of pinned apps. These are the available apps that you can launch as a job on the cluster.

In the navigation bar, the user has the following options:

<code>Files</code> — <code>Interactive Apps</code> — <code>My Interactive Sessions</code> — <code>Shell</code>

=== Files ===

Here, you can access your home directory on Helix. In a graphical user interface, files can be downloaded and uploaded, and new files and folders can be created. You will encounter an error message if you try to see data that you have no access to.

In the Home Directory the results and logs from the apps are stored. You get the following interface:

[[File:BwVisuFiles.png|bwVisu-files|center|x300px]]

Data on SDS@hd (under /mnt/sds-hd) can only be accessed from within jobs. Therefore, you need to open a terminal (within a JupyterLab or Rstudio job or directly on Helix) and set a symbolic link to your SDS@hd folder in your home directory. For example like this:
<pre>cd $HOME
mkdir sds-hd
cd sds-hd
ln -s /mnt/sds-hd/sd16a001 sd16a001</pre> 

=== Interactive Apps ===

This displays the apps that the user can run on the cluster, possibly across multiple nodes. These apps initiate an interactive session which the user can directly connect to and, depending on the app, receive a graphical interface.

=== My Interactive Sessions ===

At bwVisu, it is possible to start several interactive apps simultaneously. Here, one can view an overview of currently running interactive apps and completed sessions. It is also possible to view the log for each session. As soon as one starts an interactive app, one is automatically redirected here. Even if you log out of bwVisu and log back in, you can still see the information and logs from the finished interactive apps here until you delete it.

=== Shell ===
For bwForCluster Helix users only: After entering your OTP and password a shell on the login node of bwForCluster Helix opens. This way, you can for example create workspaces when you need more space. Please look up the details under [[Helix/Filesystems#Workspaces | Helix Workspaces]]. They can't be seen in the File Viewer but are available within the bwVisu jobs.

== Start a Job ==
[[File:bwVisuConcept.png|center|x170px]]
<ol>
<li>At the start page of the web frontend, you can start a new job by clicking on an application version in the list of pinned apps.</li>
<li>You will now see the application description and options to adjust the job resources.</li>
<ul>
<li>At the bottom of the page, beneath the <code>Launch</code> button, you can find a link for accessing the logs/session data of all your jobs.</li>
<li>The interface can differ per app because each app can have individual configuration options. </li>
[[File:bwVisuRStudio.png|bwVisu-rstudio|center|x200px]]</li>
</ul>
<li>Click the <code>Launch</code> button to start a new job with this application. You can't start two jobs with the same application. Launching an app can take some time depending on the requested resources and cluster utilization.</li>
<li>The <code>Interactive Sessions</code> page opens with the details of your job.</li>
<ul>
<li>'''Host:''' Indicates on which nodes the application is running.</li>
<li>'''Time Remaining:''' Each application is given a time limit, which can also be set in advance. In this example, the application is supposed to run for one hour.</li>
<li>'''Session ID:''' This is a link that leads to the folder of the session. Among other things, application logs are displayed here, which can be viewed in the UI.
[[File:BwVisuRstudioRun.png|bwVisu-rstudio-run|center|x200px]] </li>
</li>
</ul>
<li> Connect to the job with your browser by clicking <code>connect to <application></code>. </li>
</ol>

== Adjust noVNC Applications ==
<div><ul>
<li style="display: inline-block; vertical-align: middle; float: right;"> [[File:bwVisu Fiji noVNCMenu.png|thumb|none|x200px|noVNC menu within Fiji]] </li>
<li style="display: inline-block; vertical-align: top; float: right;"> [[File:bwVisu_Fiji_Running.png|thumb|none|x200px|Job details view of a running Fiji Job]] </li>
</ul></div>
noVNC is an open-source VNC browser client. Some bwVisu applications use noVNC and therefore, come with additional options in the view of the job details and an additional menu within the running application.

*'''Running Job View''' on the interactive sessions page
** Compression slider: Before opening the application, you can choose the compression level to use. 0 means no compression while 9 offers the best compression but uses more CPU resources on the cluster which can be slower.
** Image Quality slider: Before opening the application, you can choose the image quality level. With this you can control the desired JPEG quality.
** View Only (Share-able Link) button: By clicking this button the application opens in view only mode. You can share this link with collaborators or students while you launch your noVNC application normally for yourself. This way, others can follow what you are doing. For every job you need to generate a new link.
* '''noVNC menu''' within the running application
** When you open the application you can see an additional menu on the left with options. For the ones using the application in view-only mode it is helpful to select the option to show a dot instead of not showing the mouse cursor at all. Therefore go to "Options" and then "Expand".
** For copying data, you need to use the clipboard. When copying something locally, you need to put it into the clipboard before you can use it within the application. The same goes the other way around.
* '''Adjust window size''' and position of the app by executing a right click at the edge of the app window frame, so that a menu shows up.

== Terminate a Job ==

The job will terminate when the job runtime requested in the job submission form has expired To terminate an application yourself, you can click the delete button or end the application in the interactive session. If the app is terminated in the interactive session, the window no longer shows <code>Running</code> but <code>Completed</code>. With the <code>Delete</code> button, you end the session and also remove the list entry from <code>My Interactive Sessions</code>.

== Troubleshooting ==
In case of problems, it can help to look up the related bwForCluster Helix wiki pages.
For example:
* Error: Disk space exceeded → '''[https://wiki.bwhpc.de/e/Helix/Filesystems#Troubleshooting Filesystems: Troubleshooting]'''

Main Page

2025-11-04T11:19:12Z

H Schumacher: Changed status bar

'''Welcome to the bwHPC Wiki.'''

bwHPC represents services and resources in the State of '''B'''aden-'''W'''ürttemberg, Germany, for High Performance Computing ('''HPC'''), Data Intensive Computing ('''DIC''') and Large Scale Scientific Data Management ('''LS2DM''').

The main bwHPC web page is at '''[https://www.bwhpc.de/ https://www.bwhpc.de/]'''.

Many topics depend on the cluster system you use.
First choose the cluster you use, then select the correct topic.




{| style=" background:#B8FFB8; width:100%;"
| style="padding:8px; background:#8AFF8A; font-size:120%; text-align:left" | [[Status|'''Status:''' Maintenances [0] & Outages [0]]]
|}









{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Courses / eLearning
|-
|
* [https://training.bwhpc.de/ eLearning and Online Courses]
* [https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC Introduction to Linux in HPC (external resource)]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | Need Access to a Cluster?
|-
|
* [[When to use a HPC Cluster]]
* [[Running Calculations]]
* [[Registration]]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | HPC System Specific Documentation
|-
|
bwHPC encompasses several HPC compute clusters at different universities in Baden-Württemberg. Each cluster is dedicated to [https://www.bwhpc.de/bwhpc-domains.php specific research domains].

Documentation differs between compute clusters, please see cluster specific overview pages:
{|
|-
| style="padding:5px; width:30%" | [[BwUniCluster3.0|bwUniCluster 3.0]]
| style="padding-left:20px;" | General Purpose, Teaching
|-
| style="padding:5px; width:30%" | [[:JUSTUS2| bwForCluster JUSTUS 2]]
| style="padding-left:20px;" | Theoretical Chemistry, Condensed Matter Physics, and Quantum Sciences
|-
| style="padding:5px; width:30%" | [[Helix|bwForCluster Helix]]
| style="padding-left:20px;" | Structural and Systems Biology, Medical Science, Soft Matter, Computational Humanities, and Mathematics and Computer Science
|-
| style="padding:5px; width:30%" | [[NEMO2|bwForCluster NEMO 2]]
| style="padding-left:20px;" | Neurosciences, Particle Physics, Materials Science, and Microsystems Engineering
|-
| style="padding:5px; width:30%" | [[BinAC|bwForCluster BinAC]]
| style="padding-left:20px;" | Bioinformatics, Geosciences and Astrophysics
|-
| style="padding:5px; width:30%" | [[BinAC2|bwForCluster BinAC 2]]
| style="padding-left:20px;" | Bioinformatics, Astrophysics, Geosciences, Pharmacy, and Medical Informatics
|}
|-
|bwHPC Clusters: [https://www.bwhpc.de/cluster.php operational status]
Further Compute Clusters in Baden-Württemberg (separate access policies):
* Datenanalyse Cluster der Hochschulen (DACHS): [[DACHS | Datenanalyse Cluster der Hochschulen (DACHS)]]
* bwHPC tier 1: [https://kb.hlrs.de/platforms/index.php/Hunter_(HPE) Hunter] ([https://www.hlrs.de/apply-for-computing-time getting access])
* bwHPC tier 2: [https://www.nhr.kit.edu/userdocs/horeka HoreKa] ([https://www.nhr.kit.edu/userdocs/horeka/projects/ getting access])
|}

{| style="background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | Documentation valid for all Clusters
|-
|
* [[Environment Modules| Software Modules]] and software documentation explained
* [https://www.bwhpc.de/software.html List of Software] on all clusters
* [[Development| Software Development and Parallel Programming]]
* [[Energy Efficient Cluster Usage]]
* [[HPC Glossary]]

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | Scientific Data Storage
|-
|
For user guides of the scientific data storage services:
* [[SDS@hd]]
* [https://www.rda.kit.edu/english bwDataArchive]
* [https://uni-tuebingen.de/einrichtungen/zentrum-fuer-datenverarbeitung/projekte/laufende-projekte/bwsfs bwSFS]
Associated, but local scientific storage services are:
* [https://wiki.scc.kit.edu/lsdf/index.php/Category:LSDF_Online_Storage LSDF Online Storage] (only for KIT and KIT partners)
|}

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | Data Management
|-
|
* [[Data Transfer|Data Transfer]]
* [https://www.forschungsdaten.org/index.php/FDM-Kontakte#Deutschland Research Data Management (RDM)] contact persons
* [https://www.forschungsdaten.info Portal for Research Data Management] (Forschungsdaten.info)
|}
{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Support
|-
|
* [[BwSupportPortal|Submit a Ticket in our Support Portal]]
Support is provided by the [https://www.bwhpc.de/teams.php bwHPC Competence Centers]:
|}
{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Acknowledgement
|-
|
* Please [[Acknowledgement|acknowledge]] our resources in your publications.
|}

Registration/bwForCluster/RV

2025-11-03T19:46:48Z

H Schumacher: Moved new section to "hints"

= Step B: Apply for a Rechenvorhaben/Project =

'''Rechenvorhaben/Projects (RV)''' are required for assignment to a bwForCluster.
[[File:RV-my.png|right|frame|My RVs.]]

To get access to an RV, answer the following question:
* Does your work group (or the collaborative project that you are part of) already have an RV?
** '''Yes → [[Registration/bwForCluster/RV#Become_Coworker_of_a_RV|Become Coworker of an RV]]'''
** '''No → [[Registration/bwForCluster/RV#Register_a_new_RV|Register a new RV]]'''

You can see your RV memberships at the central application site ZAS under [https://zas.bwhpc.de/shib/en/info_rv.php '''My RVs'''].

== Become Coworker of an RV ==
[[File:RV-collab.png|right|100px|frame|1. RV collaboration.]]
[[File:RV-formcol.png|right|300px|thumb|4. RV collaboration form]]
{|style="background:#deffee; width:70%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
You can only sign up for an RV once.
If you are an inactive member of an RV, either because you de-registered yourself or a manager deactivated you, only an RV manager can reactivate you.
|}

The "RV responsible" will provide you with the following data on the RV:
* RV acronym
* RV password

If this information is lost, please let the "RV responsible" [[Registration/bwForCluster/RV/Management#Renew_RV_Password|renew the password]] for the RV.

To become coworker of an RV, follow these steps:
# Please visit the [https://zas.bwhpc.de/shib/en/bwforcluster_collaboration.php '''RV collaboration'''] web form at the [https://zas.bwhpc.de ZAS].
# Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.
# You will be directed to the ''Identity Provider'' of your home organization. Enter the username and password of your '''home organization''' (usually these credentials are also used for other services like email) and click '''Login/Einloggen'''.
# You will be redirected back to the RV collaboration form '''https://zas.bwhpc.de/shib/en/bwforcluster_collaboration.php'''.
# Enter '''RV Acronym''', '''RV Password''' and '''Institute''' into the form and check both "Terms of Use". Click '''Send'''.
# You will be assigned to the RV as a new member.
<div style="padding-left:18px;">
[[Image:Attention.svg|left|15px]]
In case of the error "This project is not currently renewed", please ask the RV responsible to apply for a prolongation of the RV. As soon as the prolongation was accepted, you can join the RV.
</div>

After submitting the request you will receive an email from ZAS about the further steps.
RV owner and RV managers will be notified automatically.

== Register a new RV ==

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
An RV is valid for '''one year'''.
After that, the RV responsible must apply for an extension of the RV for it to be renewed for another year.
|}

An RV should be registered by a leader or senior scientist of a research group/collaboration.
You will be:
* held accountable for the coworkers in the RV
* asked to provide information for the two reports required by the DFG for their funding of bwForClusters
* likely asked for a contribution to a future DFG grant proposal for a new bwForCluster in your area of research ("wissenschaftliches Beiblatt")

'''Some hints:'''
* To be notified about important issues via the cluster's mailing list, it is necessary to finish all three registration steps. If you don't plan to work with the cluster yourself, we recommend to either register nevertheless or ask a colleague who is going to use the cluster to keep you updated. For better transparency, that colleague could receive the ''RV manager'' role from you.
* Any number of coworkers can join the RV to work with the cluster.Therefore, they need to be provided with the RV password and acronym.
* The [https://www.bwhpc.de/bwhpc-domains.php research area] of the group as well as the software and hardware requirements are taken into account.
*: Please look carefully through the scientific fields and research domains to find the field that fits best to your research focus and experience. This way, you'll be assigned to the cluster that can support you best. If in doubt, you can check the [https://www.dfg.de/resource/blob/175334/89ba4a3464c99aaea40fdef47367e7b2/fachsystematik-2020-2024-de-grafik-data.pdf DFG classification] as well.
* Creating mutiple RVs on the same cluster is not necessary. Being part of multiple RVs on the same cluster does not bring any advantages regarding available computing resources. The RV description and resource specifications are important for finding the best fitting cluster. But they don't set any limits. Every cluster user has full access to all cluster resources. And not all research topics must match the RV description. But we appreciate it if you update the RV information whenever you fill in a prolongation application.
* Getting access to two different clusters is only possible if specific software or hardware needs demand access for both.

[[File:RV-reg.png|right|frame|Register a new RV.]]
[[File:BwIDM-login.png|right|250px|thumb|Select your home organization]]
'''To apply for a new RV, follow these steps:'''

# Please visit the [https://zas.bwhpc.de/shib/en/bwforcluster_proposal.php '''RV registration'''] web form at the [https://zas.bwhpc.de ZAS].
# Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.
# You will be directed to the ''Identity Provider'' of your home organization. Enter the username and password of your '''home organization''' (usually these credentials are also used for other services like email) and click '''Login/Einloggen'''.
# You will be redirected back to the RV registration form '''https://zas.bwhpc.de/shib/en/bwforcluster_proposal.php'''.
# Fill in all mandatory fields of the form and tick the two "Terms of use". Detailed information on the individual fields can be found in the section [[Registration/bwForCluster/RV#Registration_Form_Details|Registration Form Details]]. Click '''Preview'''.
# If the information is correct, click '''Submit''', otherwise you can customize your application.
# Go to section [[Registration/bwForCluster/RV#Registration_Response|Registration Response]] for the next steps.

Please note that for your convenience you can also switch to the [https://zas.bwhpc.de/shib/bwforcluster_proposal.php German version] of the web form.
The submitter of the RV will be assigned the role "RV responsible" (see [[Registration/bwForCluster/RV/Management|Roles in an RV]]).

=== Registration Form Details ===

The web form consists of the following fields.
Note that the fields - name, given name, organization, mail, and EPPN - are auto-filled and can not be changed.
These are your credentials as given by your home organization.

{| style="width:100%;border:3px solid darkgray; margin: 1em auto 1em auto;"
|-
! style="width:20%;padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;"| Field
! style="padding:2px;margin:3px; background:#AAA; font-size:100%; font-weight:bold; border:1px solid #BBBBBB; text-align:left; color:#000; padding:0.2em 0.4em;" | Explanation
|rowspan="15"| [[File:RV-form.png|center|400px|thumb|Registering a new RV.]]
|-
!scope="column" {{Lightgray_small}}| RV Title
|scope="column" {{Tab element}} | Define a short title of your planned compute activities, maximum of 255 characters including spaces.
|-
!scope="column" {{Lightgray_small}}| RV Description
|scope="column" {{Tab element}} | Write a short abstract about your planned compute activities (maximum of 2048 characters including spaces). Research groups that contributed to the DFG grant proposal (Art. 91b GG) of the corresponding bwForCluster only need to give reference to that particular proposal.
|-
!scope="column" {{Lightgray_small}}| Scientific field
|scope="column" {{Tab element}} | Tick one or several scientific fields. Once all bwForClusters are up and running the full list of given scientific fields are supported and hence for RV applicable. If your RV does not match any given scientific field, please state your scientific field in the given text box. Grayed out scientific fields indicate that the corresponding bwForCluster(s) is/are not operational yet.
|-
!scope="column" {{Lightgray_small}}| Field of activity
|scope="column" {{Tab element}} | Define if your RV is primarily for research and/or teaching. If not applicable, use text box.
|-
!scope="column" {{Lightgray_small}}| Parallel paradigm
|scope="column" {{Tab element}} | State what parallel paradigm your code or application uses. Multiple ticks are allowed. Further information can be provided via text box. If you are not sure about it, please state the software you are using.
|-
!scope="column" {{Lightgray_small}}| Programming language
|scope="column" {{Tab element}} | State the programming language(s) of your code or application.
|-
!scope="column" {{Lightgray_small}}| Numerical methods
|scope="column" {{Tab element}} | State the numerical or "calculation" methods your code or application utilizes. If you do not know, write "unknown".
|-
!scope="column" {{Lightgray_small}}| Software packages
|scope="column" {{Tab element}} | State all software package you will or want to use for your RV. Also include compiler, MPI and numerical packages (e.g. MKL, FFTW).
|-
!scope="column" {{Lightgray_small}}| Estimated requested computing resources
|scope="column" {{Tab element}} | Roughly estimate how many CPU hours are required to finish your RV. To calculate "CPU hours" multiply "elapsed parallel computing time" with "number of CPU core" per job.
''Example: Your code uses 4 CPU cores and has a wall time of 1h. This makes 4 CPU hours.''
|-
!scope="column" {{Lightgray_small}}| Planned maximum number of parallel used CPU cores per job
|scope="column" {{Tab element}} | Give an upper limit of how many CPU cores per job are required. Please cross-check with your statement about parallel paradigm.
''Obviously, a sequential code can only use 1 CPU core per job.''
|-
!scope="column" {{Lightgray_small}}| Estimated maximal memory requirements per CPU core
|scope="column" {{Tab element}} | Give an upper limit of how much RAM per CPU core your jobs require. Please give a value in Gigabytes.
|-
!scope="column" {{Lightgray_small}}| Estimated requested persistent disk space for the RV
|scope="column" {{Tab element}} | State how much disk space in total you need for your RV. Please give a value in Gigabytes.
''Example: If your RV has 4 more coworker and each of you produces until the end of your RV 20 Gigabyte of output, your maximum disk space is 100 Gigabyte.''
|-
!scope="column" {{Lightgray_small}}| Estimated maximal temporary disk space per job
|scope="column" {{Tab element}} | State how much space your intermediate files require during a job run. Please give a value in Gigabytes.
''Example: Your final output of your job is 0.1 Gigabyte, but during your job 10 temporary files, each with a size of 1 Gigabyte, are written to disk, hence your correct answer is 10 Gigabyte.''
|-
!scope="column" {{Lightgray_small}}| Institute
|scope="column" {{Tab element}} | State the full name of your Institute.
|-
|}

=== Registration Response ===

After submitting you will get an email from the ZAS confirming your submission.
With this email you are given an unique
* RV acronym
* RV password

Please keep the password enclosed.
With RV acronym and password, [[Registration/bwForCluster/RV#Become_Coworker_of_a_RV| coworkers ]] can already associate themselves to the RV.
If this information is lost, please [[Registration/bwForCluster/RV#Renew_RV_Password|renew the password]] for the RV.

The cluster assignment team assigns a bwForCluster to your RV based on your submitted data, including the [https://www.bwhpc.de/bwhpc-domains.php research area] as well as the software and hardware requirements.
The ZAS notifies you in an email about your assigned bwForCluster and provides website details for [[Registration/bwForCluster/Service| Step C]]. Later on you can find details about the roles and management possibilities at the [[Registration/bwForCluster/RV/Management | RV Management]] page.

=== Duration of an RV ===

An RV is valid for '''one year'''.
After that, the RV responsible must apply for an extension of the RV for it to be renewed for another year.

=== RV Management for RV Responsibles and Managers ===

RV responsible or manager can [[Registration/bwForCluster/RV/Management|manage and modify Rechenvorhaben/projects]].

----

[[Registration/bwForCluster/Service | Go to step C]]

SDS@hd/Registration

2025-11-03T14:39:29Z

H Schumacher: Added registration graphic

== Registration Steps ==

The registration consists of three steps:

* '''Step A:''' Membership in Storage Project (Speichervorhaben/'''SV'''). There are two options:
*: '''a)''' →[[SDS@hd/Registration#Join_existing_SV | '''Join existing SV''']]
*: '''b)''' →[[SDS@hd/Registration#Apply_for_new_SV | '''Apply for new SV''']] (only possible for [https://www.bwidm.de/hochschulen.php bwIDM members])
* '''Step B:''' →[[SDS@hd/Registration#Registration_for_SDS@hd_Service |'''Registration for SDS@hd Service''']]

After finishing the registration, the next steps are:
* Check your options to manage your SV Membership. → '''[[#Manage_SV_Membership(s) | Manage SV Membership(s)]]'''
* See how to access and use your storage space → '''[[SDS@hd/Access | Access]]'''

[[File:SDS@hd_Registration.png|center|x500px|upright|SDS@hd Registration Process]]

=== Step A: SV Membership ===

You can either join an existing SV or start a new one.

==== a) Join existing SV ====

Your advisor (the SV responsible) will provide you with the following data on the SV:
* acronym
* password

To become coworker of an SV, please login at
* [https://sds-hd.urz.uni-heidelberg.de/management/index.php?mode=mitarbeit SDS@hd Managementtool] and provide acronym and password.

You will be assigned to the SV as a member.

After submitting the request you will receive an email about the further steps.
The SV owner and any managers will be notified automatically.

==== b) Apply for new SV ====

This is typically done only by the leader of a scientific work group or the senior scientist of a research group/collaboration.
Any amount of co-workers can join your SV without having to register another project. You just need to provide them with the SV acronym and SV password. You'll receive this information via e-mail as soon as the SV application was successful.

There are two steps:
# '''Get the permission'''. Your own institution has to grant you the permission to start an SDS@hd storage project ("SDS@hd SV entitlement"). Each institution has their own procedure. If your unsure about the process for your institution, you can write to the SDS@hd support: [mailto:sds-hd-support@urz.uni-heidelberg.de Submit a Ticket]. For more details about the SDS@hd entitlements you can take a look at the [https://urz.uni-heidelberg.de/en/sds-hd SDS@hd Website]

# '''Apply for a new SV'''. Therefore, fill in the form at the [https://sds-hd.urz.uni-heidelberg.de/management SDS@hd Managementtool].

If you register your own SV, you will be:
* ...responsibe for providing new members with the necessary information to join the SV
*: → '''see [[#SV_Responsible:_Manage_the_SV_and_its_Members | SV Responsible: Manage the SV and its Members]]'''
* ...held accountable for the co-workers in the SV
* ...asked to provide information for the two reports required by the DFG for their funding of SDS@hd
* ...likely asked for a contribution to a future DFG grant proposal for an extension of the storage system in your area of research ("wissenschaftliches Beiblatt")

=== Step B: Registration for SDS@hd Service ===
[[File:bwServices_chooseHomeOrganisation.png|right|300px|thumb|Select your home organization]]

After step 1 you have to register your personal account on the storage system and set a service password.
Please visit:
* [https://bwservices.uni-heidelberg.de/ https://bwservices.uni-heidelberg.de]
*# Select your home organization from the list and click '''Proceed'''
*# You will be directed to the ''Identity Provider'' of your home organisation
*# Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
*# You will be redirected back to the registration website [https://bwservices.uni-heidelberg.de/ https://bwservices.uni-heidelberg.de/]
*# <div>Select unter '''The following services are available''' the service '''SDS@hd - Scientific Data Storage'''
*# Click '''Register'''
*# Finally, set a service password for authentication on SDS@hd

== Manage SV Membership(s) ==

The SDS@hd Managementtool and the bwServices website offer various services to manage your membership. They are described below.

=== SV Member: View SV and Membership Status ===
At the [https://sds-hd.urz.uni-heidelberg.de/management SDS@hd Managementtool] you can...
* ...view the status of your SV memberships (active/inactive)
* ...view the status of the SV (active/inactive, end date)

=== SV Responsible: Manage the SV and its Members ===
As the SV responsible person, you can...
* ...enable others to join the SV
*# Provide them with the SV acronym. You can look it up at the [https://sds-hd.urz.uni-heidelberg.de/management/shib/info_sv.php SDS@hd Managementtool].
*# Povide them with the SV password. You've received this password via email. You can reset it at the [https://sds-hd.urz.uni-heidelberg.de/management/shib/info_sv.php SDS@hd Managementtool]. Caution: Make sure to share the SV password and not your personal service password which you set in step 2 at bwServices.
*# Afterwards, they can join by following the registration steps on this page.
* ...change the status of SV members
* ...change the SV password
* ...hand over the SV responsibility

=== Change Service Password at bwServices ===

→ '''[[Registration/bwForCluster/Helix#Setting_a_New_Service_Password | Setting a New Service Password]]'''

File:SDS@hd Registration.png

2025-11-03T14:30:04Z

H Schumacher: H Schumacher uploaded a new version of File:SDS@hd Registration.png