bwHPC Wiki - User contributions [en]

Registration/bwForCluster/Helix

2026-06-03T15:07:33Z

H Schumacher: broadened the scope of the second issue

__FORCETOC__

== Prerequisites ==

Steps A + B in the [[Registration/bwForCluster | registration process]] must be finished.

== Registration Steps ==

# Visit the bwForCluster Helix '''[https://bwservices.uni-heidelberg.de registration page bwServices]'''.
#: [[File:bwServices_chooseHomeOrganisation.png|right|300px|thumb|Select your home organization]]
# '''Select your home organization''' from the list on the main page and click Proceed/Fortfahren. You will be directed to the ''Identity Provider'' of your home organization.
# Enter the '''username and password''' of your home organization (usually these credentials are also used for other services like email) and click Login/Einloggen.
# When you log in to bwServices for the first time, an overview will appear, with the account information that your home institution submits to the system. Please verify that all data is valid and then click ''Continue/Weiter''.
# You will be redirected back to the bwServices main page. It shows all the statewide services you have access to. Click '''Register/Registrieren''' under the field labeled ''bwForCluster Helix''.
#: [[File:BwIDM-reg.png|center|frame|Register for Helix]]
# bwForCluster Helix uses a '''2-factor authentication''' (2FA) mechanism to increase security.
#: If you have never registered a 2FA token on bwIDM, the following error message will appear:
#: [[File:Bwidm-3-red.png|center|600px|thumb|Second factor missing.]]
#: Use this '''[https://bwservices.uni-heidelberg.de/user/twofa.xhtml link]''' or select '''My Tokens''' in the main menu.
#: To register a new token, please follow these '''[[Registration/2FA|instructions]]'''.
#: Please complete this step before continuing.
# Read the Terms of Use / Nutzungsbedingungen, place a check mark next to ''I have read and accepted the terms of use'' and click ''Register/Registrieren''.
# '''Set a service password''' for bwForCluster Helix and click ''Save/Speichern''.
#: Be sure to use a secure password that is different from any other passwords you currently use or have used on other systems.
[[File:BwIDM-passwd.png|center|600px|thumb|Set service password]]
{|style="background:#deffee; width:70%; margin: auto;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
The service password is your private password for accessing the bwForCluster Helix. Don't ever share it with anyone!
|}

== Registration Website bwServices ==
The registration website '''[https://bwservices.uni-heidelberg.de bwServices]''' offers the following functionality:
* Register for the service.
* Set or change your service password.
* Review your registration details under "Registry Info". This includes your account status (active/lost_access).
* bwServices triggers an update of your user information upon login.
* Registering an [[Registration/SSH | SSH key]]. It can be used for login to ''bwForCluster Helix''.
* Creating tokens for logins that need 2-factor authentication ([[Registration/2FA |2FA]]) like ''bwForCluster Helix'' and ''bwVisu''.
* De-register from the service.

=== Troubleshooting with the Help of bwServices ===
When encountering connection issues with a service, the bwServices website is a good starting point for troubleshooting:
* <ins>'''Issue:</ins> Expired contract''' or inactive student id:
** No access to state services like the bwHPC clusters or SDS@hd is possible.
** You can see that this is the issue when the login to bwServices via your identity provider is not possible.
* <ins>'''Issue:</ins> Recently (re)activated uni-id or other user data changes:''' Your id at your home organization was (re)activated a short time ago or your email address changed (or sth. similar) but this change didn't get through to the bwHPC cluster or SDS@hd service.
** Login to bwServices. This triggers an update of your user information. Some minutes later, the connection to the service should work again.
* <ins>'''Issue:</ins> Entitlement / RV missing:''' Prerequisites like entitlement ownership or RV membership aren't met (anymore).
** You can see the concrete issue and advice on how to move forward when looking at the registry info of the service on the bwServices website.
** When the issue is solved, the account status changes from LOST_ACCESS to ACTIVE.
* <ins>'''Issue:</ins> Service Password:''' When trying to connect to the service, the service password is not accepted.
** Set a [[#Setting_a_New_Service_Password | new password]].
* <ins>'''Issue:</ins> OTP:''' In the case of login issues with a service that uses 2FA, you can check if you're OTP is working by going in the menu to ''Index'' -> ''My Tokens''.
** If you're token is not working, you might have had too many failed login attempts. You can request a reset of the error counter from the support.
** If the token needs to be reset, please follow the instructions for a [[Helix/Login#Troubleshooting | Lost Token]].

=== Setting a New Service Password ===

At any time, you can set a new service password via [https://bwservices.uni-heidelberg.de/ bwServices] by carrying out the following steps:
# Visit [https://bwservices.uni-heidelberg.de/ bwServices] and select your home organization.
# Authenticate yourself via your home-organizational user id / username and your home-organizational password.
# Find your service and select '''Set Service Password'''.
# Set new service password, repeat it and click the '''Save''' button. Use a secure password that is different from any other passwords you currently use or have used on other systems.
# The page answers e.g. "password has been changed".

Data Transfer/Rclone

2026-06-02T15:30:40Z

H Schumacher: Added instructions for adding rclone to path on windows.

[https://rclone.org/docs/ Rclone] is a command line tool to manage files on remote systems (e.g. high performance clusters, data and cloud storage systems, archiving services). 
Rclone either synchronizes in one direction only or its mounting functionality is used with <code>rclone mount</code>. 
Data can be piped between two completely remote locations. When moving data within the same system and the data transfer protocol allows for server side copy, no local download is necessary. 
One advantage is that the transfer is multithreaded and it operates on a file level basis.
'''Caution:''' When using Rclone with 2FA it is necessary to connect via ssh key instead of a password.

== Installation ==

Rclone is a Go program and comes as a single binary file.

# Download the relevant binary.
# Extract the <code>rclone</code> executable (<code>rclone.exe</code>).
# You can use the executables without further installation. For easy use, it is recommended to add the binary to your PATH environment variable. Information on how to do this can be found below.

Detailed information regarding different operating systems can be found here:

* Installation on [https://rclone.org/install/#windows Windows]
*: To use rclone after the installation, you have to open a terminal (cmd or PowerShell) in the folder where the rclone.exe file is. This can be done with a right click in the empty space wihin the rclone folder and then choosing "open in terminal". Then run all commands with a leading './' (for example <code>./rclone config</code>. To be able to simply use "rclone" anywhere, you can add it to your path. For instructions see the [[Data_Transfer/Rclone#Troubleshooting | Troubleshooting]] section.
* Installation on [https://rclone.org/install/#macos macOS]
** If Rclone shall be used together with sftp, the [[Data_Transfer/SSHFS#Installation | sshfs installation]] instructions must be followed after installing Rclone.
* Installation on [https://rclone.org/install/#script-installation Linux]

== Usage Rclone ==

To use Rclone you have to define a config file. Afterwards you can connect by using the name of your configured connections.

=== Configure Remote ===

Before you can start using Rclone, you need to set up a remote. This means to configure a specific connection by providing authentication information, the network protocol that you want to use and a name for this configuration so that you can use it later on.

To configure a remote for a specific service, you need the following information:

* <code><remotehost></code>
* <code><username></code>
* <code><servicePassword></code>

Furthermore, you have to decide on:

* <code>network protocol</code> (for example webDAV, smb, sftp)
* <code>remote-name</code> (for example you can use the name of the service you want to connect to)

You have three different options to set up a new remote as described in the following sections. It's enough to choose one of those.

'''Oneliner''' 

Open a terminal and define all parameters in one command. If a ssh key shall be used instead of a password, it might be easier to adjust the config file directly as described in the next section.
<syntaxhighlight lang="bash"># smb
rclone config create <remote-name> smb host=<remotehost> user=<username> domain=<domain> pass=<password> --obscure
# sftp
rclone config create <remote-name> sftp host=<remotehost> user=<username> pass=<password> --obscure
# webdav
rclone config create <remote-name> webdav url=<remotehost> user=<username> pass=<password> --obscure
</syntaxhighlight>

'''Adjust Config File''' 
<syntaxhighlight lang="bash">
# Make sure that a config file exists. Access rights for the config file are set to 600.
rclone config touch
# Get the location of the file:
rclone config file
</syntaxhighlight>
Open the file and use one of the following snippets as template for your connection:
<syntaxhighlight lang="bash">[<remote-name>]
type = smb
host = <hostname>
user = <userID>
domain = <e.g. BWSERVICESAD>

[<remote-name>]
type = webdav
url = <hostURL>
vendor = other
user = <userID>

[<remote-name>]
type = sftp
host = <hostname>
user = <userID>
key_use_agent = false

[<remote-name-ssh-key-version>]
type = sftp
host = <hostname>
user = <userID>
key_file = ~/.ssh/id_ed25519
pubkey_file = ~/.ssh/id_ed25519.pub
key_use_agent = true
shell_type = none
</syntaxhighlight>
To add the password, please use the following command. Make sure to keep the prepended space so that the command is not saved in your command history.
<syntaxhighlight> rclone config update <remote-name> pass=<password> --obscure</syntaxhighlight>

'''Interactive Setup''' 

Execute:
<syntaxhighlight>rclone config</syntaxhighlight>
This will guide you through an interactive setup process. It's quite helpful for checking out which options there are and what the config file looks like after setting values that deviate from the default values. You can find detailed instructions at the website:

* [https://rclone.org/webdav/ Connect via webdav]
* [https://rclone.org/smb/ Connect via smb]
* [https://rclone.org/sftp/ Connect via sftp]

=== Use Remote ===

To see all configured remotes:
<syntaxhighlight lang="bash">rclone config
# Exit with 'q'.
</syntaxhighlight>

The syntax to use Rclone is like this:

<pre>rclone [options] subcommand <parameters> <parameters...>
</pre>
List all elements in the folder XX:
<syntaxhighlight lang="bash">rclone lsf <remote-name>:XX
# List elements from subfolders as well:
rclone lsf --max-depth 2 <remote-name>:XX
</syntaxhighlight>
Copies /local/path to the remote path

<pre>rclone copy </local/path> <remote-name>:<remote/path>
</pre>
Copies fom remote path to /local/path

<pre>rclone copy <remote-name>:<remote/path> </local/path>
</pre>
Moves the contents of the source directory to the destination directory.

<pre>rclone move <remote-name>:<source/path> <remote-name>:<destination/path>
</pre>
More subcommands can be found [https://rclone.org/docs/#subcommands here].

== Usage Rclone Mount ==

Before you can follow the instructions in this chapter, you need to have set up a [[Data_Transfer/Rclone#Usage_Rclone | remote]].
More detailed information on how to use rclone mount can be found [https://rclone.org/commands/rclone_mount/ here].

=== Windows ===

To run rclone mount on Windows, you will need to [https://winfsp.dev/rel/ download and install WinFsp] first.

<syntaxhighlight lang="python"># mount under drive letter X (or any other free letter)
rclone mount --vfs-cache-mode full <remote-name>:<path/to/remote/files> X:
# or provide the path to a nonexisting folder:
rclone mount --vfs-cache-mode full <remote-name>:<path/to/remote/files> <C:\path\parentDir\nonexistingFoldername></syntaxhighlight>
To unmount, the command can be stopped in the terminal with the key combination CTRL+C. Alternatively, a mounted drive can be unmounted by making a right click on the drive name in the file explorer and choose "Eject". 
In contrast to Linux/Mac, there is no background mode.

=== MacOS & Linux ===

You can run mount in either foreground or background (aka daemon) mode. Mount runs in foreground mode by default. Use the <code>--daemon</code> flag to force background mode. If this doesn't work, you can put an <code>&</code> at the end of the command instead.

Create an empty directory on your local machine and then execute

<syntaxhighlight lang="bash"># to mount the root folder:
rclone mount --vfs-cache-mode full <remote-name>: /path/to/empty/folder
# to mount a subfolder:
rclone mount --vfs-cache-mode full <remote-name>:folderX/folderY /path/to/empty/folder
# to unmount:
fusermount -uz /path/to/mounted/folder </syntaxhighlight>

== Best Practices ==

Rclone has a lot of useful options.

=== Performance ===

To be able to utilize a larger bandwidth, it is helpful to add the following options for increased performance:

<pre>
--transfers <int>
</pre>

Number of file transfers to run in parallel (default: 4). Depending on the local Network, read and write speeds on the file system, and current load, different values might be best. For large transfers, it is advised to test local performance with different values beforehand.

* In our tests, we observed the best results between 8 and 32.
* For regular use cases, we recommend 16 as the default.
* Values above 64 are not recommended and degrade performance.

<pre>
--multi-thread-streams <int>
</pre>

Number of streams to use for multithreaded downloads (default: 4). Only important on very large files. This will cause multithreaded up/download on chunk-sized bits of the file.

The optimal value is highly specific to the local network and used Hardware. For regular use cases, we recommend 4 as the default.

=== Debugging and Statistics ===
To get updates on current progress, use:
<pre>
--stats
</pre>
Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable) (default 1m0s).

To get debug information, use:
<pre>
--log-level=DEBUG
--stats-log-level=DEBUG
</pre>

== Troubleshooting ==
# Windows: Adding rclone to the path
#: Needed information: path to the rclone.exe file. If you chose to install via 'winget', the path is "C:\Users\<user>\AppData\Local\Microsoft\WinGet\Packages\Rclone.Rclone_Microsoft.Winget.Source.... . Please look up the exact path.
#:* Option 1: Add the rclone.exe file to a folder that is already in the path. You can move it to "C:\Windows\System32\". Then open a new terminal window and test if the command "rclone config" works.
#:* Option 2: Add the path of the rclone.exe file to your Windows Path variable:
#:*: Open the Start Search and type env. Choose "Edit environment variables"
#:*: Click the "Environment Variables" button.
#:*: Edit the user variables and add the path to your rclone.exe file.

Helix/Login

2026-06-01T14:04:28Z

H Schumacher: Added token deletion step, changed it-service link

= Prerequisites =
* The three [[registration/bwForCluster|bwForCluster registration]] steps are completed and still valid.
* Your IP address is within the [https://www.belwue.de BelWü] network which connects universities and other scientific institutions in Baden-Württemberg. If your computer is in your university network (e.g. at your office), you should be able to connect to bwForCluster Helix without restrictions. If you are outside the BelWü network (e.g. at home), a VPN (virtual private network) connection to your University network must be established first. Please consult the VPN documentation of your university.
* As credentials you need your Helix [[Helix/Login#Username|username]], [[Registration/Password|service password]] and [[Registration/2FA|2FA token]].
* If you encounter any issues, please have a look at the [[Helix/Login#Troubleshooting|Troubleshooting]] section.

= Login to bwForCluster Helix =

Login to bwForCluster Helix is only possible with a Secure Shell (SSH) client for which you must know your bwForCluster Helix credentials and the hostname of the login nodes.
For more general information on SSH clients, visit the [[Registration/Login/Client|SSH clients Guide]].

== Username ==

Your username on bwForCluster Helix consists of a prefix and your local username.
For prefixes please refer to the [[Registration/Login/Username|Username Guide]].

Example: If your local username at your University is <code>ab123</code> and you are a user from Heidelberg University, your username on the cluster is: <code>hd_ab123</code>.

== Hostnames ==

bwForCluster Helix has two login nodes. In general, you should stay with automatic node selection, so that the connections are evenly distributed between the login nodes. To do this, we use so-called DNS round-robin scheduling which takes care of the node balancing. This means, if you open multiple SSH sessions, these sessions may run on different login nodes. Therefore, processes started in one session might not be visible in other sessions.

{| class="wikitable"
! Hostname !! Destination
|-
| '''helix.bwservices.uni-heidelberg.de''' || one of the two login nodes
|}

== Login with SSH command (Linux, Mac, Windows) ==

Most Unix and Unix-like operating systems such as Linux and Mac OS as well as newer versions of MS Windows (10 and 11), have a built-in SSH client provided by the OpenSSH project.
If you want to use the full set of Linux commands on Windows (10 or 11), you can also install the [https://docs.microsoft.com/en-us/windows/wsl/install Windows Subsystem for Linux] (WSL).

For login use one of the following ssh commands in a commandline (i.e. cmd, terminal, PowerShell):

ssh <username>@helix.bwservices.uni-heidelberg.de
ssh -l <username> helix.bwservices.uni-heidelberg.de

To run graphical applications on the cluster, you need to enable X11 forwarding with the <code>-X</code> flag:

ssh -X -l <username> helix.bwservices.uni-heidelberg.de

To make the login process faster, tell the command that no ssh key authentication is used by adding an additional parameter:

ssh -o PubkeyAuthentication=no <username>@helix.bwservices.uni-heidelberg.de

== Login with graphical SSH client (Windows) ==

For Windows we suggest using [[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]] for login and file transfer.

Start MobaXterm and fill in the following fields:
<pre>
Remote name : helix.bwservices.uni-heidelberg.de
Specify user name : <username>
Port : 22
</pre>

After that click on 'ok'. Then a terminal will open where you can enter your credentials.

== Login with SSH key ==

To set up a SSH key for your workflow, read [[Registration/SSH|Registering SSH Keys with your Cluster]].
To login by using the ssh key, use the following command:

ssh -o IdentitiesOnly=yes -i <path_to_private_key> <username>@helix.bwservices.uni-heidelberg.de

== Login Example ==

To login to bwForCluster Helix, proceed as follows:
# Login with SSH command or MoabXterm as shown above.
# The system will ask for a one-time password <code>Your OTP</code> (= Smartphone Token / TAN list element/ ...) . Please enter your OTP and confirm it with Enter/Return. The OTP is not displayed when typing. If you do not have a second factor yet, please create one (see [[Registration/2FA]]).
# The system will ask you for your service password <code>Password:</code>. Please enter it and confirm it with Enter/Return. The password is not displayed when typing. If you do not have a service password yet or have forgotten it, please create one (see [[Registration/Password]]).
# You will be greeted by the cluster with the message of the day, followed by a shell.
<pre>
$ ssh -l hd_ab123 helix.bwservices.uni-heidelberg.de
Your OTP:
Password:

********************************************************************************
* *
* Baden-Wuerttemberg HPC Cluster for *
* Structural and Systems Biology, Medical Science, *
* Soft Matter, Computational Humanities, and *
* Mathematics and Computer Science *
* *
* bwForCluster *
* _ _ _ _ *
* | | | | | | (_) *
* | |__| | ___ | | _ __ __ *
* | __ | / _ \ | | | | \ \/ / *
* | | | | | __/ | | | | > < *
* |_| |_| \___| |_| |_| /_/\_\ *
* *
* *
* (xCAT 2.16.4 / RHEL 8.8 / GPFS 5.1.8) *
* *
* *
* bwHPC Wiki : https://wiki.bwhpc.de/e/Helix *
* Ticket System : https://www.bwhpc.de/supportportal *
* Job Monitoring : https://helix-monitoring.bwservices.uni-heidelberg.de *
* E-learning : https://training.bwhpc.de/ *
* -> 'Introduction to bwForCluster Helix' *
* *
********************************************************************************

Last login: ...
[hd_ab123@login ~]$
</pre>

= Allowed Activities on Login Nodes =

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
To guarantee usability for all users you must not run your compute jobs on the login nodes.
Compute jobs must be submitted as batch jobs.
Any compute job running on the login nodes will be terminated without notice.
Long-running compilation or long-running pre- or post-processing tasks must also be submitted as batch jobs.
|}

The login nodes are the access points to the compute system, your <code>$HOME</code> directory and your workspaces.
These nodes are shared with all users. Hence, your activities on the login nodes are primarily limited to setting up your batch jobs.
Your activities may also be:
* quick compilation of program code or
* quick pre- and post-processing of results from batch jobs.

We advise to use [[Helix/Slurm#Interactive_Jobs|interactive batch jobs]] for compute and memory intensive compilation and pre- and post-processing tasks.

= Related Information =

* If you want to register a new token for the two factor authentication (2FA), consult the [[Registration/2FA|2FA Guide]].
* If you want to de-register, consult the [[Registration/Deregistration|De-registration Guide]].
* If you need an SSH key for your workflow, read [[Registration/SSH|Registering SSH Keys with your Cluster]].
* Configuring your shell: [[.bashrc Do's and Don'ts]]

= Troubleshooting =

'''Login Problems'''

If you encounter login problems, please double check the prerequisites. If those are valid, it is fastest to first check the following points before contacting the support at the [https://www.bwhpc.de/supportportal.php|bwhpc Suppport Portal].
If there are known issues with the cluster, we'll inform you via e-mail or in the message of the day at the cluster.

* Make sure to not use LAN and WLAN at the same time to prevent connection problems.
* Do [[Registration/bwForCluster/Helix#Troubleshooting_with_the_Help_of_bwServices | troubleshooting with bwServices]].

'''Lost Token'''

# To reset your token you have to (virtually) visit the IT Service at the University of Heidelberg to validate your identity. You can find the information for a virtual or in person visit in the "How to contact us" section here: https://www.urz.uni-heidelberg.de/en/support/it-service. It is recommended to use the Chrome browser for doing the video call.
# Please have an official identification ready (e.g. Führerschein / Personalausweis) and mention that you want a reset of your Helix 2FA token. It is important to emphasize that the token reset is for bwForCluster Helix to avoid confusion with the token that is specific to members/students of the university of Heidelberg.
# After the IT Service deactivated the token, you can delete the token yourself, reload the page and create a new one.
Hint: To be on the safe side, it is advised to create a [[Registration/2FA#Backup_TAN_List|backup TAN list]] additionally to the smartphone token.

Data Transfer/Rclone

2026-05-31T13:42:21Z

H Schumacher: corrected information

[https://rclone.org/docs/ Rclone] is a command line tool to manage files on remote systems (e.g. high performance clusters, data and cloud storage systems, archiving services). 
Rclone either synchronizes in one direction only or its mounting functionality is used with <code>rclone mount</code>. 
Data can be piped between two completely remote locations. When moving data within the same system and the data transfer protocol allows for server side copy, no local download is necessary. 
One advantage is that the transfer is multithreaded and it operates on a file level basis.
'''Caution:''' When using Rclone with 2FA it is necessary to connect via ssh key instead of a password.

== Installation ==

Rclone is a Go program and comes as a single binary file.

# Download the relevant binary.
# Extract the <code>rclone</code> executable (<code>rclone.exe</code>).
# You can use the executables without further installation. For easy use, it is recommended to add the binary to your PATH environment variable. Information on how to do this can be found below.

Detailed information regarding different operating systems can be found here:

* Installation on [https://rclone.org/install/#windows Windows]
* Installation on [https://rclone.org/install/#macos macOS]
** If Rclone shall be used together with sftp, the [[Data_Transfer/SSHFS#Installation | sshfs installation]] instructions must be followed after installing Rclone.
* Installation on [https://rclone.org/install/#script-installation Linux]

== Usage Rclone ==

To use Rclone you have to define a config file. Afterwards you can connect by using the name of your configured connections.

=== Configure Remote ===

Before you can start using Rclone, you need to set up a remote. This means to configure a specific connection by providing authentication information, the network protocol that you want to use and a name for this configuration so that you can use it later on.

To configure a remote for a specific service, you need the following information:

* <code><remotehost></code>
* <code><username></code>
* <code><servicePassword></code>

Furthermore, you have to decide on:

* <code>network protocol</code> (for example webDAV, smb, sftp)
* <code>remote-name</code> (for example you can use the name of the service you want to connect to)

You have three different options to set up a new remote as described in the following sections. It's enough to choose one of those.

'''Oneliner''' 

Open a terminal and define all parameters in one command. If a ssh key shall be used instead of a password, it might be easier to adjust the config file directly as described in the next section.
<syntaxhighlight lang="bash"># smb
rclone config create <remote-name> smb host=<remotehost> user=<username> domain=<domain> pass=<password> --obscure
# sftp
rclone config create <remote-name> sftp host=<remotehost> user=<username> pass=<password> --obscure
# webdav
rclone config create <remote-name> webdav url=<remotehost> user=<username> pass=<password> --obscure
</syntaxhighlight>

'''Adjust Config File''' 
<syntaxhighlight lang="bash">
# Make sure that a config file exists. Access rights for the config file are set to 600.
rclone config touch
# Get the location of the file:
rclone config file
</syntaxhighlight>
Open the file and use one of the following snippets as template for your connection:
<syntaxhighlight lang="bash">[<remote-name>]
type = smb
host = <hostname>
user = <userID>
domain = <e.g. BWSERVICESAD>

[<remote-name>]
type = webdav
url = <hostURL>
vendor = other
user = <userID>

[<remote-name>]
type = sftp
host = <hostname>
user = <userID>
key_use_agent = false

[<remote-name-ssh-key-version>]
type = sftp
host = <hostname>
user = <userID>
key_file = ~/.ssh/id_ed25519
pubkey_file = ~/.ssh/id_ed25519.pub
key_use_agent = true
shell_type = none
</syntaxhighlight>
To add the password, please use the following command. Make sure to keep the prepended space so that the command is not saved in your command history.
<syntaxhighlight> rclone config update <remote-name> pass=<password> --obscure</syntaxhighlight>

'''Interactive Setup''' 

Execute:
<syntaxhighlight>rclone config</syntaxhighlight>
This will guide you through an interactive setup process. It's quite helpful for checking out which options there are and what the config file looks like after setting values that deviate from the default values. You can find detailed instructions at the website:

* [https://rclone.org/webdav/ Connect via webdav]
* [https://rclone.org/smb/ Connect via smb]
* [https://rclone.org/sftp/ Connect via sftp]

=== Use Remote ===

To see all configured remotes:
<syntaxhighlight lang="bash">rclone config
# Exit with 'q'.
</syntaxhighlight>

The syntax to use Rclone is like this:

<pre>rclone [options] subcommand <parameters> <parameters...>
</pre>
List all elements in the folder XX:
<syntaxhighlight lang="bash">rclone lsf <remote-name>:XX
# List elements from subfolders as well:
rclone lsf --max-depth 2 <remote-name>:XX
</syntaxhighlight>
Copies /local/path to the remote path

<pre>rclone copy </local/path> <remote-name>:<remote/path>
</pre>
Copies fom remote path to /local/path

<pre>rclone copy <remote-name>:<remote/path> </local/path>
</pre>
Moves the contents of the source directory to the destination directory.

<pre>rclone move <remote-name>:<source/path> <remote-name>:<destination/path>
</pre>
More subcommands can be found [https://rclone.org/docs/#subcommands here].

== Usage Rclone Mount ==

Before you can follow the instructions in this chapter, you need to have set up a [[Data_Transfer/Rclone#Usage_Rclone | remote]].
More detailed information on how to use rclone mount can be found [https://rclone.org/commands/rclone_mount/ here].

=== Windows ===

To run rclone mount on Windows, you will need to [https://winfsp.dev/rel/ download and install WinFsp] first.

<syntaxhighlight lang="python"># mount under drive letter X (or any other free letter)
rclone mount --vfs-cache-mode full <remote-name>:<path/to/remote/files> X:
# or provide the path to a nonexisting folder:
rclone mount --vfs-cache-mode full <remote-name>:<path/to/remote/files> <C:\path\parentDir\nonexistingFoldername></syntaxhighlight>
To unmount, the command can be stopped in the terminal with the key combination CTRL+C. Alternatively, a mounted drive can be unmounted by making a right click on the drive name in the file explorer and choose "Eject". 
In contrast to Linux/Mac, there is no background mode.

=== MacOS & Linux ===

You can run mount in either foreground or background (aka daemon) mode. Mount runs in foreground mode by default. Use the <code>--daemon</code> flag to force background mode. If this doesn't work, you can put an <code>&</code> at the end of the command instead.

Create an empty directory on your local machine and then execute

<syntaxhighlight lang="bash"># to mount the root folder:
rclone mount --vfs-cache-mode full <remote-name>: /path/to/empty/folder
# to mount a subfolder:
rclone mount --vfs-cache-mode full <remote-name>:folderX/folderY /path/to/empty/folder
# to unmount:
fusermount -uz /path/to/mounted/folder </syntaxhighlight>

== Best Practices ==

Rclone has a lot of useful options.

=== Performance ===

To be able to utilize a larger bandwidth, it is helpful to add the following options for increased performance:

<pre>
--transfers <int>
</pre>

Number of file transfers to run in parallel (default: 4). Depending on the local Network, read and write speeds on the file system, and current load, different values might be best. For large transfers, it is advised to test local performance with different values beforehand.

* In our tests, we observed the best results between 8 and 32.
* For regular use cases, we recommend 16 as the default.
* Values above 64 are not recommended and degrade performance.

<pre>
--multi-thread-streams <int>
</pre>

Number of streams to use for multithreaded downloads (default: 4). Only important on very large files. This will cause multithreaded up/download on chunk-sized bits of the file.

The optimal value is highly specific to the local network and used Hardware. For regular use cases, we recommend 4 as the default.

=== Debugging and Statistics ===
To get updates on current progress, use:
<pre>
--stats
</pre>
Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable) (default 1m0s).

To get debug information, use:
<pre>
--log-level=DEBUG
--stats-log-level=DEBUG
</pre>

SDS@hd/Access

2026-05-27T16:06:37Z

H Schumacher: added 'domain' as additional bullet point to avoid confusion

This page provides an overview on how to access data served by SDS@hd. To get an introduction to data transfer in general, see [[Data_Transfer|data transfer]].

'''Notation:''' Please replace all placeholder contents within guillemets <code>< ></code> with the actual value. For example, replace ''<name_of_my_dog>'' with ''Fluffy'' .

== Prerequisites ==

* You need to be [[SDS@hd/Registration|registered]].
* You need to be in the belwue-Network. This means you have to use the VPN Service of your HomeOrganization, if you want to access SDS@hd from outside the bwHPC-Clusters (e.g. via eduroam or from your personal notebook).

== Needed Information, independent of the chosen tool ==

* '''Username:''' You can look it up under your user data at [https://bwservices.uni-heidelberg.de/user/index.xhtml bwServices]. The username is ''<localPrimaryGroup>_<eduPersonPrincipalName>'' . (It is the same as for the bwHPC Clusters. Example: ''hd_ab123''.)
* '''Password:''' The Service Password that you set at bwServices in the [[SDS@hd/Registration#Step_B:_Registration_for_SDS@hd_Service|registration step]].
* '''SV-Acronym:''' Use the lower case version of the acronym for all access options.
* '''Hostname:''' The hostname depends on the chosen network protocol:
** For [[Data_Transfer/SSHFS|SSHFS]] and [[Data_Transfer/SFTP|SFTP]]: ''lsdf02-sshfs.urz.uni-heidelberg.de''
** For [[SDS@hd/Access/SMB|SMB]] and [[SDS@hd/Access/NFS|NFS]]: ''lsdf02.urz.uni-heidelberg.de''
** For [[Data_Transfer/WebDAV|WebDAV]] the url is: ''https://lsdf02-webdav.urz.uni-heidelberg.de''
* '''Domain:''' The domain is ''BWSERVICESAD'' (needed for smb and nfs connections).

== Recommended Setup ==
The following graphic shows the recommended way for accessing SDS@hd via Windows/Mac/Linux. The table provides an overview of the most important access options and links to the related pages. 
If you have various use cases, it is recommended to use [[Data_Transfer/Rclone|Rclone]]. You can copy, sync and mount with it. Thanks to its multithreading capability Rclone is a good fit for transferring big data. 
For an overview of all connection possibilities, please have a look at [[Data_Transfer/All_Data_Transfer_Routes|all data transfer routes]].

[[File:Data_transfer_diagram_simple.jpg|center|500px]]
Figure 1: SDS@hd main transfer routes

{| class="wikitable"
|- style="font-weight:bold; text-align:center; vertical-align:middle;"
!
! Use Case
! Windows
! Mac
! Linux
! Possible Bandwith
! Firewall Ports
|-
| [[Data_Transfer/Rclone|Rclone]] + <protocol>
| copy, sync and mount, multithreading
| ✓
| ✓
| ✓
| depends on used protocol
| depends on used protocol
|-
| [[SDS@hd/Access/SMB|SMB]]
| mount as network drive in file explorer or usage via Rclone
| [[SDS@hd/Access/SMB#Windows|✓]]
| [[SDS@hd/Access/SMB#Mac|✓]]
| [[SDS@hd/Access/SMB#Linux|✓]]
| up to 40 Gbit/sec
| 139 (netbios), 135 (rpc), 445 (smb)
|-
| [[Data_Transfer/WebDAV|WebDAV]]
| go to solution for restricted networks
| [✓]
| ✓
| ✓
| up to 100GBit/sec
| 80 (http), 443 (https)
|- style="vertical-align:middle;"
| [[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]]
| Graphical User Interface (GUI)
| [[Data_Transfer/Graphical_Clients#MobaXterm|✓]]
| ☓
| ☓
| see sftp
| see sftp
|- style="vertical-align:middle;"
| [[SDS@hd/Access/NFS|NFS]]
| mount for multi-user environments
| ☓
| ☓
| [[SDS@hd/Access/NFS|✓]]
| up to 40 Gbit/sec
| -
|- style="vertical-align:middle;"
| [[Data_Transfer/SSHFS|SSHFS]]
| mount, needs stable internet connection
| ☓
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| see sftp
| see sftp
|- style="vertical-align:middle;"
| [[Data_Transfer/SFTP|SFTP]]
| interactive shell, better usability when used together with Rclone
| [[Data_Transfer/SFTP#Windows|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| up to 40 Gbit/sec
| 22 (ssh)
|}
Table 1: SDS@hd transfer routes

=== Access from a bwHPC Cluster ===

'''bwUniCluster''' 
You can't mount to your $HOME directory but you can create a mount under $TMPDIR by following the instructions for [[Data_Transfer/Rclone#Usage_Rclone_Mount | Rclone mount]]. It is advised to wait a couple of seconds (<code>sleep 5</code>) before trying to use the mounted directory.

'''bwForCluster Helix''' 
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes.

'''bwForCluster BinAC 2''' 
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes. The prerequisites are:
* The SV responsible has enabled the SV on BinAC 2 once by writing to [mailto:sds-hd-support@urz.uni-heidelberg.de sds-hd-support@urz.uni-heidelberg.de]
* You have a valid kerberos ticket, which can be fetched with <code>kinit <userID></code>

'''Other''' 
You can mount your SDS@hd SV on the cluster yourself by using [[Data_Transfer/Rclone | Rclone]] with the [[Data_Transfer/Rclone#Usage_Rclone_Mount | Rclone mount]] command. As transfer protocol you can use WebDAV or sftp. For a full overview please have a look at [[Data_Transfer/All_Data_Transfer_Routes | All Data Transfer Routes]].

=== Access via Webbrowser (read-only) ===

Visit [https://lsdf02-webdav.urz.uni-heidelberg.de/ lsdf02-webdav.urz.uni-heidelberg.de] and login with your SDS@hd username and service password. Here you can get an overview of the data in your "Speichervorhaben" and download single files. To be able to do more, like moving data, uploading new files, or downloading complete folders, a suitable client is needed as described above.

== Best Practices ==

* '''Managing access rights'''
** When the SV is mounted on Windows, access rights can be adjusted in the file/folder properties menu. (Users from Heidelberg without Windows can use the [https://www.urz.uni-heidelberg.de/de/service-katalog/desktop-und-arbeitsplatz/windows-terminalserver Windows terminal server]).
** bwForCluster Helix users can change the access rights via Helix by using [[Workspace#Regular_Unix_Permissions | unix permissions]] or [[Workspace#ACLs:_Access_Control_Lists | access control lists]]. On non native mounts ACL changes won't work.
** Additionally to the already mentioned options, the SVV can take over ownership of files by opening a [[Data_Transfer/SFTP|SFTP]] shell and running <code>chown -R <userID> <path/to/folder></code> .

* '''Multiuser environment''' → Use [[SDS@hd/Access/NFS|NFS]]

== Troubleshooting ==

Maintance windows and known issues are communicated via the email list or the News section at the SDS@hd start page in the wiki.

* '''Issue:''' The credentials aren't accepted
*: Check if your credentials work in general by trying the access via webbrowser: [https://lsdf02-webdav.urz.uni-heidelberg.de/].
*: If this doesn't work:
*:: → [[Registration/Login/Username| Check your Username]]
*:: → [[Registration/bwForCluster/Helix#Troubleshooting_with_the_Help_of_bwServices | Troubleshooting with bwServices]]
* '''Other Issue'''
** If available, follow the troubleshooting guide of your specific connection method.
** Make sure to not use LAN and WLAN at the same time to prevent connection problems.
** If you have an institutional account, make sure to be a fully active member of your institution.
If these suggestions didn't help, write to the [mailto:sds-hd-support@urz.uni-heidelberg.de support]. Provide the following information:
* Your operating system
* Does the access via webbrowser work?
*: If yes, provide us with detailed information on how you tried to access your SV (used access method, username, ...).

Data Transfer/Rclone

2026-05-27T16:01:30Z

H Schumacher: improved Windows instructions

[https://rclone.org/docs/ Rclone] is a command line tool to manage files on remote systems (e.g. cloud storage systems). Rclone either synchronizes in one direction only or its mounting functionality is used with <code>rclone mount</code>. Data can be piped between two completely remote locations, sometimes without local download. One advantage is that the transfer is multithreaded and it operates on a file level basis.
'''Caution:''' You can't use Rclone with 2FA.

== Installation ==

Rclone is a Go program and comes as a single binary file.

# Download the relevant binary.
# Extract the <code>rclone</code> executable, <code>rclone.exe</code> on Windows, from the archive.
# You can use the executables without further installation. For easy use, it is recommended to add the binary to your PATH environment variable. Information on how to do this can be found below.

Detailed information regarding different operating systems can be found here:

* Installation on [https://rclone.org/install/#windows Windows]
* Installation on [https://rclone.org/install/#macos macOS]
** If Rclone shall be used together with sftp, the [[Data_Transfer/SSHFS#Installation | sshfs installation]] instructions must be followed after installing Rclone.
* Installation on [https://rclone.org/install/#script-installation Linux]

== Usage Rclone ==

To use Rclone you have to define a config file. Afterwards you can connect by using the name of your configured connections.

=== Configure Remote ===

Before you can start using Rclone, you need to set up a remote. This means to configure a specific connection by providing authentication information, the network protocol that you want to use and a name for this configuration so that you can use it later on.

To configure a remote for a specific service, you need the following information:

* <code><remotehost></code>
* <code><username></code>
* <code><servicePassword></code>

Furthermore, you have to decide on:

* <code>network protocol</code> (for example webDAV, smb, sftp)
* <code>remote-name</code> (for example you can use the name of the service you want to connect to)

You have three different options to set up a new remote as described in the following sections. It's enough to choose one of those.

'''Oneliner''' 

Open a terminal and define all parameters in one command. If a ssh key shall be used instead of a password, it might be easier to adjust the config file directly as described in the next section.
<syntaxhighlight lang="bash"># smb
rclone config create <remote-name> smb host=<remotehost> user=<username> domain=<domain> pass=<password> --obscure
# sftp
rclone config create <remote-name> sftp host=<remotehost> user=<username> pass=<password> --obscure
# webdav
rclone config create <remote-name> webdav url=<remotehost> user=<username> pass=<password> --obscure
</syntaxhighlight>

'''Adjust Config File''' 
<syntaxhighlight lang="bash">
# Make sure that a config file exists. Access rights for the config file are set to 600.
rclone config touch
# Get the location of the file:
rclone config file
</syntaxhighlight>
Open the file and use one of the following snippets as template for your connection:
<syntaxhighlight lang="bash">[<remote-name>]
type = smb
host = <hostname>
user = <userID>
domain = <e.g. BWSERVICESAD>

[<remote-name>]
type = webdav
url = <hostURL>
vendor = other
user = <userID>

[<remote-name>]
type = sftp
host = <hostname>
user = <userID>
key_use_agent = false

[<remote-name-ssh-key-version>]
type = sftp
host = <hostname>
user = <userID>
key_file = ~/.ssh/id_ed25519
pubkey_file = ~/.ssh/id_ed25519.pub
key_use_agent = true
shell_type = none
</syntaxhighlight>
To add the password, please use the following command. Make sure to keep the prepended space so that the command is not saved in your command history.
<syntaxhighlight> rclone config update <remote-name> pass=<password> --obscure</syntaxhighlight>

'''Interactive Setup''' 

Execute:
<syntaxhighlight>rclone config</syntaxhighlight>
This will guide you through an interactive setup process. It's quite helpful for checking out which options there are and what the config file looks like after setting values that deviate from the default values. You can find detailed instructions at the website:

* [https://rclone.org/webdav/ Connect via webdav]
* [https://rclone.org/smb/ Connect via smb]
* [https://rclone.org/sftp/ Connect via sftp]

=== Use Remote ===

To see all configured remotes:
<syntaxhighlight lang="bash">rclone config
# Exit with 'q'.
</syntaxhighlight>

The syntax to use Rclone is like this:

<pre>rclone [options] subcommand <parameters> <parameters...>
</pre>
List all elements in the folder XX:
<syntaxhighlight lang="bash">rclone lsf <remote-name>:XX
# List elements from subfolders as well:
rclone lsf --max-depth 2 <remote-name>:XX
</syntaxhighlight>
Copies /local/path to the remote path

<pre>rclone copy </local/path> <remote-name>:<remote/path>
</pre>
Copies fom remote path to /local/path

<pre>rclone copy <remote-name>:<remote/path> </local/path>
</pre>
Moves the contents of the source directory to the destination directory.

<pre>rclone move <remote-name>:<source/path> <remote-name>:<destination/path>
</pre>
More subcommands can be found [https://rclone.org/docs/#subcommands here].

== Usage Rclone Mount ==

Before you can follow the instructions in this chapter, you need to have set up a [[Data_Transfer/Rclone#Usage_Rclone | remote]].
More detailed information on how to use rclone mount can be found [https://rclone.org/commands/rclone_mount/ here].

=== Windows ===

To run rclone mount on Windows, you will need to [https://winfsp.dev/rel/ download and install WinFsp] first.

<syntaxhighlight lang="python"># mount under drive letter X (or any other free letter)
rclone mount --vfs-cache-mode full <remote-name>:<path/to/remote/files> X:
# or provide the path to a nonexisting folder:
rclone mount --vfs-cache-mode full <remote-name>:<path/to/remote/files> <C:\path\parentDir\nonexistingFoldername></syntaxhighlight>
To unmount, the command can be stopped in the terminal with the key combination CTRL+C. Alternatively, a mounted drive can be unmounted by making a right click on the drive name in the file explorer and choose "Eject". 
In contrast to Linux/Mac, there is no background mode.

=== MacOS & Linux ===

You can run mount in either foreground or background (aka daemon) mode. Mount runs in foreground mode by default. Use the <code>--daemon</code> flag to force background mode. If this doesn't work, you can put an <code>&</code> at the end of the command instead.

Create an empty directory on your local machine and then execute

<syntaxhighlight lang="bash"># to mount the root folder:
rclone mount --vfs-cache-mode full <remote-name>: /path/to/empty/folder
# to mount a subfolder:
rclone mount --vfs-cache-mode full <remote-name>:folderX/folderY /path/to/empty/folder
# to unmount:
fusermount -uz /path/to/mounted/folder </syntaxhighlight>

== Best Practices ==

Rclone has a lot of useful options.

=== Performance ===

To be able to utilize a larger bandwidth, it is helpful to add the following options for increased performance:

<pre>
--transfers <int>
</pre>

Number of file transfers to run in parallel (default: 4). Depending on the local Network, read and write speeds on the file system, and current load, different values might be best. For large transfers, it is advised to test local performance with different values beforehand.

* In our tests, we observed the best results between 8 and 32.
* For regular use cases, we recommend 16 as the default.
* Values above 64 are not recommended and degrade performance.

<pre>
--multi-thread-streams <int>
</pre>

Number of streams to use for multithreaded downloads (default: 4). Only important on very large files. This will cause multithreaded up/download on chunk-sized bits of the file.

The optimal value is highly specific to the local network and used Hardware. For regular use cases, we recommend 4 as the default.

=== Debugging and Statistics ===
To get updates on current progress, use:
<pre>
--stats
</pre>
Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable) (default 1m0s).

To get debug information, use:
<pre>
--log-level=DEBUG
--stats-log-level=DEBUG
</pre>

Data Transfer/Rclone

2026-05-27T09:44:49Z

H Schumacher: fixed typo

[https://rclone.org/docs/ Rclone] is a command line tool to manage files on remote systems (e.g. cloud storage systems). Rclone either synchronizes in one direction only or its mounting functionality is used with <code>rclone mount</code>. Data can be piped between two completely remote locations, sometimes without local download. One advantage is that the transfer is multithreaded and it operates on a file level basis.
'''Caution:''' You can't use Rclone with 2FA.

== Installation ==

Rclone is a Go program and comes as a single binary file.

# Download the relevant binary.
# Extract the <code>rclone</code> executable, <code>rclone.exe</code> on Windows, from the archive.
# You can use the executables without further installation. For easy use, it is recommended to add the binary to your PATH environment variable. Information on how to do this can be found below.

Detailed information regarding different operating systems can be found here:

* Installation on [https://rclone.org/install/#windows Windows]
* Installation on [https://rclone.org/install/#macos macOS]
** If Rclone shall be used together with sftp, the [[Data_Transfer/SSHFS#Installation | sshfs installation]] instructions must be followed after installing Rclone.
* Installation on [https://rclone.org/install/#script-installation Linux]

== Usage Rclone ==

To use Rclone you have to define a config file. Afterwards you can connect by using the name of your configured connections.

=== Configure Remote ===

Before you can start using Rclone, you need to set up a remote. This means to configure a specific connection by providing authentication information, the network protocol that you want to use and a name for this configuration so that you can use it later on.

To configure a remote for a specific service, you need the following information:

* <code><remotehost></code>
* <code><username></code>
* <code><servicePassword></code>

Furthermore, you have to decide on:

* <code>network protocol</code> (for example webDAV, smb, sftp)
* <code>remote-name</code> (for example you can use the name of the service you want to connect to)

You have three different options to set up a new remote as described in the following sections. It's enough to choose one of those.

'''Oneliner''' 

Open a terminal and define all parameters in one command. If a ssh key shall be used instead of a password, it might be easier to adjust the config file directly as described in the next section.
<syntaxhighlight lang="bash"># smb
rclone config create <remote-name> smb host=<remotehost> user=<username> domain=<domain> pass=<password> --obscure
# sftp
rclone config create <remote-name> sftp host=<remotehost> user=<username> pass=<password> --obscure
# webdav
rclone config create <remote-name> webdav url=<remotehost> user=<username> pass=<password> --obscure
</syntaxhighlight>

'''Adjust Config File''' 
<syntaxhighlight lang="bash">
# Make sure that a config file exists. Access rights for the config file are set to 600.
rclone config touch
# Get the location of the file:
rclone config file
</syntaxhighlight>
Open the file and use one of the following snippets as template for your connection:
<syntaxhighlight lang="bash">[<remote-name>]
type = smb
host = <hostname>
user = <userID>
domain = <e.g. BWSERVICESAD>

[<remote-name>]
type = webdav
url = <hostURL>
vendor = other
user = <userID>

[<remote-name>]
type = sftp
host = <hostname>
user = <userID>
key_use_agent = false

[<remote-name-ssh-key-version>]
type = sftp
host = <hostname>
user = <userID>
key_file = ~/.ssh/id_ed25519
pubkey_file = ~/.ssh/id_ed25519.pub
key_use_agent = true
shell_type = none
</syntaxhighlight>
To add the password, please use the following command. Make sure to keep the prepended space so that the command is not saved in your command history.
<syntaxhighlight> rclone config update <remote-name> pass=<password> --obscure</syntaxhighlight>

'''Interactive Setup''' 

Execute:
<syntaxhighlight>rclone config</syntaxhighlight>
This will guide you through an interactive setup process. It's quite helpful for checking out which options there are and what the config file looks like after setting values that deviate from the default values. You can find detailed instructions at the website:

* [https://rclone.org/webdav/ Connect via webdav]
* [https://rclone.org/smb/ Connect via smb]
* [https://rclone.org/sftp/ Connect via sftp]

=== Use Remote ===

To see all configured remotes:
<syntaxhighlight lang="bash">rclone config
# Exit with 'q'.
</syntaxhighlight>

The syntax to use Rclone is like this:

<pre>rclone [options] subcommand <parameters> <parameters...>
</pre>
List all elements in the folder XX:
<syntaxhighlight lang="bash">rclone lsf <remote-name>:XX
# List elements from subfolders as well:
rclone lsf --max-depth 2 <remote-name>:XX
</syntaxhighlight>
Copies /local/path to the remote path

<pre>rclone copy </local/path> <remote-name>:<remote/path>
</pre>
Copies fom remote path to /local/path

<pre>rclone copy <remote-name>:<remote/path> </local/path>
</pre>
Moves the contents of the source directory to the destination directory.

<pre>rclone move <remote-name>:<source/path> <remote-name>:<destination/path>
</pre>
More subcommands can be found [https://rclone.org/docs/#subcommands here].

== Usage Rclone Mount ==

Before you can follow the instructions in this chapter, you need to have set up a [[Data_Transfer/Rclone#Usage_Rclone | remote]].
More detailed information on how to use rclone mount can be found [https://rclone.org/commands/rclone_mount/ here].

=== Windows ===

To run rclone mount on Windows, you will need to [https://winfsp.dev/rel/ download and install WinFsp]. To mount on drive letter X or a nonexistent subdirectory, use:

<pre>rclone mount <remote-name>:path/to/files X:
rclone mount <remote-name>:path/to/files C:\path\parent\mount</pre>
In contrast to Linux/Mac, there is no background mode.

=== MacOS & Linux ===

You can run mount in either foreground or background (aka daemon) mode. Mount runs in foreground mode by default. Use the <code>--daemon</code> flag to force background mode. If this doesn't work, you can put an <code>&</code> at the end of the command instead.

Create an empty directory on your local machine and then execute

<pre># to mount the root folder:
rclone mount --vfs-cache-mode full <remote-name>: /path/to/empty/folder
# to mount a subfolder:
rclone mount --vfs-cache-mode full <remote-name>:folderX/folderY /path/to/empty/folder
# to unmount:
fusermount -uz /path/to/mounted/folder </pre>

== Best Practices ==

Rclone has a lot of useful options.

=== Performance ===

To be able to utilize a larger bandwidth, it is helpful to add the following options for increased performance:

<pre>
--transfers <int>
</pre>

Number of file transfers to run in parallel (default: 4). Depending on the local Network, read and write speeds on the file system, and current load, different values might be best. For large transfers, it is advised to test local performance with different values beforehand.

* In our tests, we observed the best results between 8 and 32.
* For regular use cases, we recommend 16 as the default.
* Values above 64 are not recommended and degrade performance.

<pre>
--multi-thread-streams <int>
</pre>

Number of streams to use for multithreaded downloads (default: 4). Only important on very large files. This will cause multithreaded up/download on chunk-sized bits of the file.

The optimal value is highly specific to the local network and used Hardware. For regular use cases, we recommend 4 as the default.

=== Debugging and Statistics ===
To get updates on current progress, use:
<pre>
--stats
</pre>
Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable) (default 1m0s).

To get debug information, use:
<pre>
--log-level=DEBUG
--stats-log-level=DEBUG
</pre>

Data Transfer/Rclone

2026-05-18T16:11:02Z

H Schumacher: changed lsd to lsf

[https://rclone.org/docs/ Rclone] is a command line tool to manage files on remote systems (e.g. cloud storage systems). Rclone either synchronizes in one direction only or its mounting functionality is used with <code>rclone mount</code>. Data can be piped between two completely remote locations, sometimes without local download. One advantage is that the transfer is multithreaded and it operates on a file level basis.
'''Caution:''' You can't use Rclone with 2FA.

== Installation ==

Rclone is a Go program and comes as a single binary file.

# Download the relevant binary.
# Extract the <code>rclone</code> executable, <code>rclone.exe</code> on Windows, from the archive.
# You can use the executables without further installation. For easy use, it is recommended to add the binary to your PATH environment variable. Information on how to do this can be found below.

Detailed information regarding different operating systems can be found here:

* Installation on [https://rclone.org/install/#windows Windows]
* Installation on [https://rclone.org/install/#macos macOS]
** If Rclone shall be used together with sftp, the [[https://wiki.bwhpc.de/e/Data_Transfer/SSHFS#Installation | sshfs installation]] instructions must be followed after installing Rclone.
* Installation on [https://rclone.org/install/#script-installation Linux]

== Usage Rclone ==

To use Rclone you have to define a config file. Afterwards you can connect by using the name of your configured connections.

=== Configure Remote ===

Before you can start using Rclone, you need to set up a remote. This means to configure a specific connection by providing authentication information, the network protocol that you want to use and a name for this configuration so that you can use it later on.

To configure a remote for a specific service, you need the following information:

* <code><remotehost></code>
* <code><username></code>
* <code><servicePassword></code>

Furthermore, you have to decide on:

* <code>network protocol</code> (for example webDAV, smb, sftp)
* <code>remote-name</code> (for example you can use the name of the service you want to connect to)

You have three different options to set up a new remote as described in the following sections. It's enough to choose one of those.

'''Oneliner''' 

Open a terminal and define all parameters in one command. If a ssh key shall be used instead of a password, it might be easier to adjust the config file directly as described in the next section.
<syntaxhighlight lang="bash"># smb
rclone config create <remote-name> smb host=<remotehost> user=<username> domain=<domain> pass=<password> --obscure
# sftp
rclone config create <remote-name> sftp host=<remotehost> user=<username> pass=<password> --obscure
# webdav
rclone config create <remote-name> webdav url=<remotehost> user=<username> pass=<password> --obscure
</syntaxhighlight>

'''Adjust Config File''' 
<syntaxhighlight lang="bash">
# Make sure that a config file exists. Access rights for the config file are set to 600.
rclone config touch
# Get the location of the file:
rclone config file
</syntaxhighlight>
Open the file and use one of the following snippets as template for your connection:
<syntaxhighlight lang="bash">[<remote-name>]
type = smb
host = <hostname>
user = <userID>
domain = <e.g. BWSERVICESAD>

[<remote-name>]
type = webdav
url = <hostURL>
vendor = other
user = <userID>

[<remote-name>]
type = sftp
host = <hostname>
user = <userID>
key_use_agent = false

[<remote-name-ssh-key-version>]
type = sftp
host = <hostname>
user = <userID>
key_file = ~/.ssh/id_ed25519
pubkey_file = ~/.ssh/id_ed25519.pub
key_use_agent = true
shell_type = none
</syntaxhighlight>
To add the password, please use the following command. Make sure to keep the prepended space so that the command is not saved in your command history.
<syntaxhighlight> rclone config update <remote-name> pass=<password> --obscure</syntaxhighlight>

'''Interactive Setup''' 

Execute:
<syntaxhighlight>rclone config</syntaxhighlight>
This will guide you through an interactive setup process. It's quite helpful for checking out which options there are and what the config file looks like after setting values that deviate from the default values. You can find detailed instructions at the website:

* [https://rclone.org/webdav/ Connect via webdav]
* [https://rclone.org/smb/ Connect via smb]
* [https://rclone.org/sftp/ Connect via sftp]

=== Use Remote ===

To see all configured remotes:
<syntaxhighlight lang="bash">rclone config
# Exit with 'q'.
</syntaxhighlight>

The syntax to use Rclone is like this:

<pre>rclone [options] subcommand <parameters> <parameters...>
</pre>
List all elements in the folder XX:
<syntaxhighlight lang="bash">rclone lsf <remote-name>:XX
# List elements from subfolders as well:
rclone lsf --max-depth 2 <remote-name>:XX
</syntaxhighlight>
Copies /local/path to the remote path

<pre>rclone copy </local/path> <remote-name>:<remote/path>
</pre>
Copies fom remote path to /local/path

<pre>rclone copy <remote-name>:<remote/path> </local/path>
</pre>
Moves the contents of the source directory to the destination directory.

<pre>rclone move <remote-name>:<source/path> <remote-name>:<destination/path>
</pre>
More subcommands can be found [https://rclone.org/docs/#subcommands here].

== Usage Rclone Mount ==

Before you can follow the instructions in this chapter, you need to have set up a [[Data_Transfer/Rclone#Usage_Rclone | remote]].
More detailed information on how to use rclone mount can be found [https://rclone.org/commands/rclone_mount/ here].

=== Windows ===

To run rclone mount on Windows, you will need to [https://winfsp.dev/rel/ download and install WinFsp]. To mount on drive letter X or a nonexistent subdirectory, use:

<pre>rclone mount <remote-name>:path/to/files X:
rclone mount <remote-name>:path/to/files C:\path\parent\mount</pre>
In contrast to Linux/Mac, there is no background mode.

=== MacOS & Linux ===

You can run mount in either foreground or background (aka daemon) mode. Mount runs in foreground mode by default. Use the <code>--daemon</code> flag to force background mode. If this doesn't work, you can put an <code>&</code> at the end of the command instead.

Create an empty directory on your local machine and then execute

<pre># to mount the root folder:
rclone mount --vfs-cache-mode full <remote-name>: /path/to/empty/folder
# to mount a subfolder:
rclone mount --vfs-cache-mode full <remote-name>:folderX/folderY /path/to/empty/folder
# to unmount:
fusermount -uz /path/to/mounted/folder </pre>

== Best Practices ==

Rclone has a lot of useful options.

=== Performance ===

To be able to utilize a larger bandwidth, it is helpful to add the following options for increased performance:

<pre>
--transfers <int>
</pre>

Number of file transfers to run in parallel (default: 4). Depending on the local Network, read and write speeds on the file system, and current load, different values might be best. For large transfers, it is advised to test local performance with different values beforehand.

* In our tests, we observed the best results between 8 and 32.
* For regular use cases, we recommend 16 as the default.
* Values above 64 are not recommended and degrade performance.

<pre>
--multi-thread-streams <int>
</pre>

Number of streams to use for multithreaded downloads (default: 4). Only important on very large files. This will cause multithreaded up/download on chunk-sized bits of the file.

The optimal value is highly specific to the local network and used Hardware. For regular use cases, we recommend 4 as the default.

=== Debugging and Statistics ===
To get updates on current progress, use:
<pre>
--stats
</pre>
Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable) (default 1m0s).

To get debug information, use:
<pre>
--log-level=DEBUG
--stats-log-level=DEBUG
</pre>

Data Transfer/Rclone

2026-05-13T14:49:46Z

H Schumacher: Reordered remote setup options

[https://rclone.org/docs/ Rclone] is a command line tool to manage files on remote systems (e.g. cloud storage systems). Rclone either synchronizes in one direction only or its mounting functionality is used with <code>rclone mount</code>. Data can be piped between two completely remote locations, sometimes without local download. One advantage is that the transfer is multithreaded and it operates on a file level basis.
'''Caution:''' You can't use Rclone with 2FA.

== Installation ==

Rclone is a Go program and comes as a single binary file.

# Download the relevant binary.
# Extract the <code>rclone</code> executable, <code>rclone.exe</code> on Windows, from the archive.
# You can use the executables without further installation. For easy use, it is recommended to add the binary to your PATH environment variable. Information on how to do this can be found below.

Detailed information regarding different operating systems can be found here:

* Installation on [https://rclone.org/install/#windows Windows]
* Installation on [https://rclone.org/install/#macos macOS]
** If Rclone shall be used together with sftp, the [[https://wiki.bwhpc.de/e/Data_Transfer/SSHFS#Installation | sshfs installation]] instructions must be followed after installing Rclone.
* Installation on [https://rclone.org/install/#script-installation Linux]

== Usage Rclone ==

To use Rclone you have to define a config file. Afterwards you can connect by using the name of your configured connections.

=== Configure Remote ===

Before you can start using Rclone, you need to set up a remote. This means to configure a specific connection by providing authentication information, the network protocol that you want to use and a name for this configuration so that you can use it later on.

To configure a remote for a specific service, you need the following information:

* <code><remotehost></code>
* <code><username></code>
* <code><servicePassword></code>

Furthermore, you have to decide on:

* <code>network protocol</code> (for example webDAV, smb, sftp)
* <code>remote-name</code> (for example you can use the name of the service you want to connect to)

You have three different options to set up a new remote as described in the following sections. It's enough to choose one of those.

'''Oneliner''' 

Open a terminal and define all parameters in one command. If a ssh key shall be used instead of a password, it might be easier to adjust the config file directly as described in the next section.
<syntaxhighlight lang="bash"># smb
rclone config create <remote-name> smb host=<remotehost> user=<username> domain=<domain> pass=<password> --obscure
# sftp
rclone config create <remote-name> sftp host=<remotehost> user=<username> pass=<password> --obscure
# webdav
rclone config create <remote-name> webdav url=<remotehost> user=<username> pass=<password> --obscure
</syntaxhighlight>

'''Adjust Config File''' 
<syntaxhighlight lang="bash">
# Make sure that a config file exists. Access rights for the config file are set to 600.
rclone config touch
# Get the location of the file:
rclone config file
</syntaxhighlight>
Open the file and use one of the following snippets as template for your connection:
<syntaxhighlight lang="bash">[<remote-name>]
type = smb
host = <hostname>
user = <userID>
domain = <e.g. BWSERVICESAD>

[<remote-name>]
type = webdav
url = <hostURL>
vendor = other
user = <userID>

[<remote-name>]
type = sftp
host = <hostname>
user = <userID>
key_use_agent = false

[<remote-name-ssh-key-version>]
type = sftp
host = <hostname>
user = <userID>
key_file = ~/.ssh/id_ed25519
pubkey_file = ~/.ssh/id_ed25519.pub
key_use_agent = true
shell_type = none
</syntaxhighlight>
To add the password, please use the following command. Make sure to keep the prepended space so that the command is not saved in your command history.
<syntaxhighlight> rclone config update <remote-name> pass=<password> --obscure</syntaxhighlight>

'''Interactive Setup''' 

Execute:
<syntaxhighlight>rclone config</syntaxhighlight>
This will guide you through an interactive setup process. It's quite helpful for checking out which options there are and what the config file looks like after setting values that deviate from the default values. You can find detailed instructions at the website:

* [https://rclone.org/webdav/ Connect via webdav]
* [https://rclone.org/smb/ Connect via smb]
* [https://rclone.org/sftp/ Connect via sftp]

=== Use Remote ===

To see all configured remotes:
<syntaxhighlight lang="bash">rclone config
# Exit with 'q'.
</syntaxhighlight>

The syntax to use Rclone is like this:

<pre>rclone [options] subcommand <parameters> <parameters...>
</pre>
List all directories/containers/buckets in the folder XX.

<pre>rclone lsd <remote-name>:XX
</pre>
Copies /local/path to the remote path

<pre>rclone copy </local/path> <remote-name>:<remote/path>
</pre>
Copies fom remote path to /local/path

<pre>rclone copy <remote-name>:<remote/path> </local/path>
</pre>
Moves the contents of the source directory to the destination directory.

<pre>rclone move <remote-name>:<source/path> <remote-name>:<destination/path>
</pre>
More subcommands can be found [https://rclone.org/docs/#subcommands here].

== Usage Rclone Mount ==

Before you can follow the instructions in this chapter, you need to have set up a [[Data_Transfer/Rclone#Usage_Rclone | remote]].
More detailed information on how to use rclone mount can be found [https://rclone.org/commands/rclone_mount/ here].

=== Windows ===

To run rclone mount on Windows, you will need to [https://winfsp.dev/rel/ download and install WinFsp]. To mount on drive letter X or a nonexistent subdirectory, use:

<pre>rclone mount <remote-name>:path/to/files X:
rclone mount <remote-name>:path/to/files C:\path\parent\mount</pre>
In contrast to Linux/Mac, there is no background mode.

=== MacOS & Linux ===

You can run mount in either foreground or background (aka daemon) mode. Mount runs in foreground mode by default. Use the <code>--daemon</code> flag to force background mode. If this doesn't work, you can put an <code>&</code> at the end of the command instead.

Create an empty directory on your local machine and then execute

<pre># to mount the root folder:
rclone mount --vfs-cache-mode full <remote-name>: /path/to/empty/folder
# to mount a subfolder:
rclone mount --vfs-cache-mode full <remote-name>:folderX/folderY /path/to/empty/folder
# to unmount:
fusermount -uz /path/to/mounted/folder </pre>

== Best Practices ==

Rclone has a lot of useful options.

=== Performance ===

To be able to utilize a larger bandwidth, it is helpful to add the following options for increased performance:

<pre>
--transfers <int>
</pre>

Number of file transfers to run in parallel (default: 4). Depending on the local Network, read and write speeds on the file system, and current load, different values might be best. For large transfers, it is advised to test local performance with different values beforehand.

* In our tests, we observed the best results between 8 and 32.
* For regular use cases, we recommend 16 as the default.
* Values above 64 are not recommended and degrade performance.

<pre>
--multi-thread-streams <int>
</pre>

Number of streams to use for multithreaded downloads (default: 4). Only important on very large files. This will cause multithreaded up/download on chunk-sized bits of the file.

The optimal value is highly specific to the local network and used Hardware. For regular use cases, we recommend 4 as the default.

=== Debugging and Statistics ===
To get updates on current progress, use:
<pre>
--stats
</pre>
Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable) (default 1m0s).

To get debug information, use:
<pre>
--log-level=DEBUG
--stats-log-level=DEBUG
</pre>

Data Transfer

2026-05-13T14:42:13Z

H Schumacher: Added ssh key info.

== Overview ==

Data transfer is the exchange of files between two systems. Before data transfer can happen, you need to go through the following steps:

# Choose the two [[#Data_Storage_Systems|data storage systems]] that shall exchange data.
# Choose the top level ways of transfer ([[#Ways of Transfer:_Copy,_Sync,_Mount|copy, sync or mount]]) by considering your specific use case.
# Choose a [[#Network_Protocols_&_Transfer_Tools|network protocol or transfer tool]] to use for the communication between the systems.

The recommended setup already includes these three steps. For a full overview, you can reference the tables that show [[Data_Transfer/All_Data_Transfer_Routes|all transfer routes]]. Those include all possible combinations between systems, top level way of transfer and network protocol / transfer tool.

=== Data Storage Systems ===

Data transfer can happen between a variety of systems. For example:

* [[File:Notebook.svg|x20px]] Local computer or VM (virtual machine)
* [[File:Microscope.svg|x20px]] Data producing machine (sequencer, microscope, ...)
* [[File:Clusternodes.svg|x20px]] HPC system
* [[File:Storage_small.svg|x15px]] Storage space (SDS@hd, institute server, ...)
* [[File:Cloud.svg|x15px]] Cloud resource

=== Ways of Transfer: Copy, Sync, Mount ===

The top level ways of transfer are:

* '''Copy:''' A simple copy command is the most basic way to transfer data. This is most efficient for very big data files that shall be retrieved from or moved to a remote location. And it can be most convenient, if you prefer moving your files via commandline instead of using a file browser. Examples: [[Data_Transfer/SCP|scp]], [[Data_Transfer/SFTP|sftp]]
* '''Sync:''' If the data is intended to be kept on both systems and undergoes change on only one of the systems, it makes sense to use a synchronization command instead. This way, only the changed files in one location are updated in the other location. Good use cases are backups or data transfers that go mostly in one direction like moving data from a sequencer to a storage space. A disadvantage is that the data needs storage space on both systems. Example: [[Data_Transfer/Rsync|rsync]]
* '''Mount:''' If the data undergoes change on both systems or is too big to store locally, then mounting is the most convenient solution. This allows you to see and work with the data as if it were stored locally on your computer while it is still placed on the remote system. All changes that you implement happen directly on the original data so that you don't need to copy or synchronize anything. Additionally, you'll see all changes that another party does to the data with just a very short delay. Disadvantages are that you need defined edit sessions starting with a mount and ending with a clean unmount of your files and with a stable network connection during the session. Also, file operations on the remote system become much slower via a remotely sshfs-mounted system. Example: [[Data_Transfer/SSHFS|sshfs]]
[[File:CopySyncMount.png|x250px]]
Figure 1: Top level transfer routes



=== Network Protocols & Transfer Tools ===

{| class="wikitable" style="vertical-align:middle;"
|- style="font-weight:bold;"
! Basic Network Protocol
! Used By Network Protocol
|-
| ssh
| [[Data_Transfer/SCP | scp]], [[Data_Transfer/SFTP | sftp]], [[Data_Transfer/Rsync | rsync]]
|-
| http(s)
| [[Data_Transfer/WebDAV | WebDAV]]
|-
| [[SDS@hd/Access/SMB | smb]]
| -
|-
| [[SDS@hd/Access/NFS | NFS]]
| -
|}
For every data transfer a network protocol to use for the communication between the systems must be chosen. The basic network protocols and the network protocols that build directly upon those are shown in the table on the right. These protocols can either be used rather directly or through tools that provide the protocol together with additional features. A tool can either mean a command line tool or a tool with a graphical user interface.

A comprehensive overview of all transfer options (network protocols and tools) can be found on the page [[Data_Transfer/All_Data_Transfer_Routes|all data transfer routes]].

== Recommended Setup ==
[[File:Bwhpc diagram simplenobox.jpg|thumb|Main routes for mounting|x150px]]
When you are working with a development environment that allows remote connections, this is the first choice. Otherwise, the main tools/protocols for transferring data are as follows:

* Windows: '''[[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]]''' is a graphical user interface that allows logging in to the cluster with ssh as well as transferring data via a file browser or using command line tools for the transfer.
* MacOS and Linux:
** '''[[Data_Transfer/SSHFS|sshfs]]''' is quickly set up for mounting under stable connections.
** '''[[Data_Transfer/Rclone|Rclone]]''' for mount, copy and sync. An [[Registration/SSH | ssh key]] is needed when used with 2FA.
* SDS@hd: See the [[SDS@hd/Access|SDS@hd Access]] page.

== Best Practices ==

* '''Strong firewall restrictions''' -> Use ssh or http(s) based protocols, for example [[Data_Transfer/WebDAV|'''WebDav''']] and [[Data_Transfer/SFTP|sftp]]. For very strict facilities, ssh based protocols might not be allowed.
* '''Share data with collaborators...'''
** ...outside of Baden-Württemberg -> Use the [[SDS@hd|SDS@hd]] storage.
** ...that are less comfortable with the command line -> Let them mount the folder.
* '''Transfer many small files''' -> Compress the files to one.

For advanced topics see [[Data_Transfer/Advanced_Data_Transfer|Advanced Data Transfer]].

Data Transfer/SSHFS

2026-05-13T14:10:07Z

H Schumacher: Rclone link for 2fa cases.

Sshfs is a network protocol for mounting data. It uses [[Data Transfer/SFTP|sftp]] under the hood.

== Installation ==

For the necessary installations follow this link: [https://linuxize.com/post/how-to-use-sshfs-to-mount-remote-directories-over-ssh/ How to use SSHFS to Mount Remote Directories over SSH]. 
a) '''MacOS after the installation''' 
To enable sshfs on your Mac go to: System Preferences -> General tab -> Allow Extensions button 
Then follow the instructions that pop up (make sure that you remember the instructions before shutting down your computer). 
b) '''Linux installation short commands'''
<ul>
<li>Debian/Ubuntu: <code>apt-get install sshfs</code></li>
<li>CentOS/RedHat: <code>dnf install fuse-sshfs</code></li>
<li>If you want to use /etc/fstab, you can do this with the following entry:
<pre>sshfs#<username>@<hostname>: <your_local_mountpoint> fuse defaults,user,noauto,exec,reconnect 0 0 </pre></li></ul>

== Usage ==

=== Windows ===
''Caution: sshfs on Windows doesn't work with 2FA. An alternative is [[Data_Transfer/Rclone | Rclone with sftp]] + [[Registration/SSH | ssh key]].''
<ol style="list-style-type: decimal;">
<li>Go to "This PC" and select "Connect network drive" ("Netzlaufwerk verbinden")</li>
<li>Select a drive (W:) and click on "connect with webpage on which you can save documents and images" (Verbindung mit website herstellen, auf der Sie Dokumente und Bilder speichern können)</li>
<li>Enter the following as address, to connect to the home or root folder:
<syntaxhighlight lang="bash">\sshfs\<username>@<hostname>/</syntaxhighlight>
If you would want to go two folders up, you would add “''../../''” at the end.</li></ol>

=== MacOS & Linux ===

<ol style="list-style-type: decimal;">
<li>Create an empty directory. The cluster/storage folder is going to be mounted to this directory.</li>
<li>Mount a folder from the cluster/storage:
<syntaxhighlight lang="bash"># Home folder
sshfs -o reconnect <username>@<hostname>: <localPath>
# subfolderX
sshfs -o reconnect <username>@<hostname>:subfolderX <localPath></syntaxhighlight></li></ol>

=== Undo mounting ===

'''MacOS & Linux:''' 
If the connection to the cluster broke and you want to remount, you often have to use umount first. Try to remember to always unmount first before retrying. Otherwise it can happen that unmounting is not possible anymore by cmd. Usually the error message in this case tells you that the process is currently busy and therefore cannot umount. In this case, you have to restart the notebook which automatically unmounts everything.

<pre>```bash
umount -f <localPath>
# For example
umount -f /Users/<username>/Documents/mounted
# or use
fusermount -u <localPath>
```</pre>

SDS@hd/Access

2026-05-13T13:46:42Z

H Schumacher: Clarified username and password.

This page provides an overview on how to access data served by SDS@hd. To get an introduction to data transfer in general, see [[Data_Transfer|data transfer]].

'''Notation:''' Please replace all placeholder contents within guillemets <code>< ></code> with the actual value. For example, replace ''<name_of_my_dog>'' with ''Fluffy'' .

== Prerequisites ==

* You need to be [[SDS@hd/Registration|registered]].
* You need to be in the belwue-Network. This means you have to use the VPN Service of your HomeOrganization, if you want to access SDS@hd from outside the bwHPC-Clusters (e.g. via eduroam or from your personal notebook).

== Needed Information, independent of the chosen tool ==

* '''Username:''' You can look it up under your user data at [https://bwservices.uni-heidelberg.de/user/index.xhtml bwServices]. The username is ''<localPrimaryGroup>_<eduPersonPrincipalName>'' . (It is the same as for the bwHPC Clusters. Example: ''hd_ab123''.)
* '''Password:''' The Service Password that you set at bwServices in the [[SDS@hd/Registration#Step_B:_Registration_for_SDS@hd_Service|registration step]].
* '''SV-Acronym:''' Use the lower case version of the acronym for all access options.
* '''Hostname:''' The hostname depends on the chosen network protocol:
** For [[Data_Transfer/SSHFS|SSHFS]] and [[Data_Transfer/SFTP|SFTP]]: ''lsdf02-sshfs.urz.uni-heidelberg.de''
** For [[SDS@hd/Access/SMB|SMB]] and [[SDS@hd/Access/NFS|NFS]]: ''lsdf02.urz.uni-heidelberg.de''. The domain is ''BWSERVICESAD''.
** For [[Data_Transfer/WebDAV|WebDAV]] the url is: ''https://lsdf02-webdav.urz.uni-heidelberg.de''

== Recommended Setup ==
The following graphic shows the recommended way for accessing SDS@hd via Windows/Mac/Linux. The table provides an overview of the most important access options and links to the related pages. 
If you have various use cases, it is recommended to use [[Data_Transfer/Rclone|Rclone]]. You can copy, sync and mount with it. Thanks to its multithreading capability Rclone is a good fit for transferring big data. 
For an overview of all connection possibilities, please have a look at [[Data_Transfer/All_Data_Transfer_Routes|all data transfer routes]].

[[File:Data_transfer_diagram_simple.jpg|center|500px]]
Figure 1: SDS@hd main transfer routes

{| class="wikitable"
|- style="font-weight:bold; text-align:center; vertical-align:middle;"
!
! Use Case
! Windows
! Mac
! Linux
! Possible Bandwith
! Firewall Ports
|-
| [[Data_Transfer/Rclone|Rclone]] + <protocol>
| copy, sync and mount, multithreading
| ✓
| ✓
| ✓
| depends on used protocol
| depends on used protocol
|-
| [[SDS@hd/Access/SMB|SMB]]
| mount as network drive in file explorer or usage via Rclone
| [[SDS@hd/Access/SMB#Windows|✓]]
| [[SDS@hd/Access/SMB#Mac|✓]]
| [[SDS@hd/Access/SMB#Linux|✓]]
| up to 40 Gbit/sec
| 139 (netbios), 135 (rpc), 445 (smb)
|-
| [[Data_Transfer/WebDAV|WebDAV]]
| go to solution for restricted networks
| [✓]
| ✓
| ✓
| up to 100GBit/sec
| 80 (http), 443 (https)
|- style="vertical-align:middle;"
| [[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]]
| Graphical User Interface (GUI)
| [[Data_Transfer/Graphical_Clients#MobaXterm|✓]]
| ☓
| ☓
| see sftp
| see sftp
|- style="vertical-align:middle;"
| [[SDS@hd/Access/NFS|NFS]]
| mount for multi-user environments
| ☓
| ☓
| [[SDS@hd/Access/NFS|✓]]
| up to 40 Gbit/sec
| -
|- style="vertical-align:middle;"
| [[Data_Transfer/SSHFS|SSHFS]]
| mount, needs stable internet connection
| ☓
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| see sftp
| see sftp
|- style="vertical-align:middle;"
| [[Data_Transfer/SFTP|SFTP]]
| interactive shell, better usability when used together with Rclone
| [[Data_Transfer/SFTP#Windows|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| up to 40 Gbit/sec
| 22 (ssh)
|}
Table 1: SDS@hd transfer routes

=== Access from a bwHPC Cluster ===

'''bwUniCluster''' 
You can't mount to your $HOME directory but you can create a mount under $TMPDIR by following the instructions for [[Data_Transfer/Rclone#Usage_Rclone_Mount | Rclone mount]]. It is advised to wait a couple of seconds (<code>sleep 5</code>) before trying to use the mounted directory.

'''bwForCluster Helix''' 
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes.

'''bwForCluster BinAC 2''' 
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes. The prerequisites are:
* The SV responsible has enabled the SV on BinAC 2 once by writing to [mailto:sds-hd-support@urz.uni-heidelberg.de sds-hd-support@urz.uni-heidelberg.de]
* You have a valid kerberos ticket, which can be fetched with <code>kinit <userID></code>

'''Other''' 
You can mount your SDS@hd SV on the cluster yourself by using [[Data_Transfer/Rclone | Rclone]] with the [[Data_Transfer/Rclone#Usage_Rclone_Mount | Rclone mount]] command. As transfer protocol you can use WebDAV or sftp. For a full overview please have a look at [[Data_Transfer/All_Data_Transfer_Routes | All Data Transfer Routes]].

=== Access via Webbrowser (read-only) ===

Visit [https://lsdf02-webdav.urz.uni-heidelberg.de/ lsdf02-webdav.urz.uni-heidelberg.de] and login with your SDS@hd username and service password. Here you can get an overview of the data in your "Speichervorhaben" and download single files. To be able to do more, like moving data, uploading new files, or downloading complete folders, a suitable client is needed as described above.

== Best Practices ==

* '''Managing access rights'''
** When the SV is mounted on Windows, access rights can be adjusted in the file/folder properties menu. (Users from Heidelberg without Windows can use the [https://www.urz.uni-heidelberg.de/de/service-katalog/desktop-und-arbeitsplatz/windows-terminalserver Windows terminal server]).
** bwForCluster Helix users can change the access rights via Helix by using [[Workspace#Regular_Unix_Permissions | unix permissions]] or [[Workspace#ACLs:_Access_Control_Lists | access control lists]]. On non native mounts ACL changes won't work.
** Additionally to the already mentioned options, the SVV can take over ownership of files by opening a [[Data_Transfer/SFTP|SFTP]] shell and running <code>chown -R <userID> <path/to/folder></code> .

* '''Multiuser environment''' → Use [[SDS@hd/Access/NFS|NFS]]

== Troubleshooting ==

Maintance windows and known issues are communicated via the email list or the News section at the SDS@hd start page in the wiki.

* '''Issue:''' The credentials aren't accepted
*: Check if your credentials work in general by trying the access via webbrowser: [https://lsdf02-webdav.urz.uni-heidelberg.de/].
*: If this doesn't work:
*:: → [[Registration/Login/Username| Check your Username]]
*:: → [[Registration/bwForCluster/Helix#Troubleshooting_with_the_Help_of_bwServices | Troubleshooting with bwServices]]
* '''Other Issue'''
** If available, follow the troubleshooting guide of your specific connection method.
** Make sure to not use LAN and WLAN at the same time to prevent connection problems.
** If you have an institutional account, make sure to be a fully active member of your institution.
If these suggestions didn't help, write to the [mailto:sds-hd-support@urz.uni-heidelberg.de support]. Provide the following information:
* Your operating system
* Does the access via webbrowser work?
*: If yes, provide us with detailed information on how you tried to access your SV (used access method, username, ...).

Helix/Login

2026-05-12T10:39:53Z

H Schumacher: Replaced some points with the "troubleshooting with bwServices" link

= Prerequisites =
* The three [[registration/bwForCluster|bwForCluster registration]] steps are completed and still valid.
* Your IP address is within the [https://www.belwue.de BelWü] network which connects universities and other scientific institutions in Baden-Württemberg. If your computer is in your university network (e.g. at your office), you should be able to connect to bwForCluster Helix without restrictions. If you are outside the BelWü network (e.g. at home), a VPN (virtual private network) connection to your University network must be established first. Please consult the VPN documentation of your university.
* As credentials you need your Helix [[Helix/Login#Username|username]], [[Registration/Password|service password]] and [[Registration/2FA|2FA token]].
* If you encounter any issues, please have a look at the [[Helix/Login#Troubleshooting|Troubleshooting]] section.

= Login to bwForCluster Helix =

Login to bwForCluster Helix is only possible with a Secure Shell (SSH) client for which you must know your bwForCluster Helix credentials and the hostname of the login nodes.
For more general information on SSH clients, visit the [[Registration/Login/Client|SSH clients Guide]].

== Username ==

Your username on bwForCluster Helix consists of a prefix and your local username.
For prefixes please refer to the [[Registration/Login/Username|Username Guide]].

Example: If your local username at your University is <code>ab123</code> and you are a user from Heidelberg University, your username on the cluster is: <code>hd_ab123</code>.

== Hostnames ==

bwForCluster Helix has two login nodes. In general, you should stay with automatic node selection, so that the connections are evenly distributed between the login nodes. To do this, we use so-called DNS round-robin scheduling which takes care of the node balancing. This means, if you open multiple SSH sessions, these sessions may run on different login nodes. Therefore, processes started in one session might not be visible in other sessions.

{| class="wikitable"
! Hostname !! Destination
|-
| '''helix.bwservices.uni-heidelberg.de''' || one of the two login nodes
|}

== Login with SSH command (Linux, Mac, Windows) ==

Most Unix and Unix-like operating systems such as Linux and Mac OS as well as newer versions of MS Windows (10 and 11), have a built-in SSH client provided by the OpenSSH project.
If you want to use the full set of Linux commands on Windows (10 or 11), you can also install the [https://docs.microsoft.com/en-us/windows/wsl/install Windows Subsystem for Linux] (WSL).

For login use one of the following ssh commands in a commandline (i.e. cmd, terminal, PowerShell):

ssh <username>@helix.bwservices.uni-heidelberg.de
ssh -l <username> helix.bwservices.uni-heidelberg.de

To run graphical applications on the cluster, you need to enable X11 forwarding with the <code>-X</code> flag:

ssh -X -l <username> helix.bwservices.uni-heidelberg.de

To make the login process faster, tell the command that no ssh key authentication is used by adding an additional parameter:

ssh -o PubkeyAuthentication=no <username>@helix.bwservices.uni-heidelberg.de

== Login with graphical SSH client (Windows) ==

For Windows we suggest using [[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]] for login and file transfer.

Start MobaXterm and fill in the following fields:
<pre>
Remote name : helix.bwservices.uni-heidelberg.de
Specify user name : <username>
Port : 22
</pre>

After that click on 'ok'. Then a terminal will open where you can enter your credentials.

== Login with SSH key ==

To set up a SSH key for your workflow, read [[Registration/SSH|Registering SSH Keys with your Cluster]].
To login by using the ssh key, use the following command:

ssh -o IdentitiesOnly=yes -i <path_to_private_key> <username>@helix.bwservices.uni-heidelberg.de

== Login Example ==

To login to bwForCluster Helix, proceed as follows:
# Login with SSH command or MoabXterm as shown above.
# The system will ask for a one-time password <code>Your OTP</code> (= Smartphone Token / TAN list element/ ...) . Please enter your OTP and confirm it with Enter/Return. The OTP is not displayed when typing. If you do not have a second factor yet, please create one (see [[Registration/2FA]]).
# The system will ask you for your service password <code>Password:</code>. Please enter it and confirm it with Enter/Return. The password is not displayed when typing. If you do not have a service password yet or have forgotten it, please create one (see [[Registration/Password]]).
# You will be greeted by the cluster with the message of the day, followed by a shell.
<pre>
$ ssh -l hd_ab123 helix.bwservices.uni-heidelberg.de
Your OTP:
Password:

********************************************************************************
* *
* Baden-Wuerttemberg HPC Cluster for *
* Structural and Systems Biology, Medical Science, *
* Soft Matter, Computational Humanities, and *
* Mathematics and Computer Science *
* *
* bwForCluster *
* _ _ _ _ *
* | | | | | | (_) *
* | |__| | ___ | | _ __ __ *
* | __ | / _ \ | | | | \ \/ / *
* | | | | | __/ | | | | > < *
* |_| |_| \___| |_| |_| /_/\_\ *
* *
* *
* (xCAT 2.16.4 / RHEL 8.8 / GPFS 5.1.8) *
* *
* *
* bwHPC Wiki : https://wiki.bwhpc.de/e/Helix *
* Ticket System : https://www.bwhpc.de/supportportal *
* Job Monitoring : https://helix-monitoring.bwservices.uni-heidelberg.de *
* E-learning : https://training.bwhpc.de/ *
* -> 'Introduction to bwForCluster Helix' *
* *
********************************************************************************

Last login: ...
[hd_ab123@login ~]$
</pre>

= Allowed Activities on Login Nodes =

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
To guarantee usability for all users you must not run your compute jobs on the login nodes.
Compute jobs must be submitted as batch jobs.
Any compute job running on the login nodes will be terminated without notice.
Long-running compilation or long-running pre- or post-processing tasks must also be submitted as batch jobs.
|}

The login nodes are the access points to the compute system, your <code>$HOME</code> directory and your workspaces.
These nodes are shared with all users. Hence, your activities on the login nodes are primarily limited to setting up your batch jobs.
Your activities may also be:
* quick compilation of program code or
* quick pre- and post-processing of results from batch jobs.

We advise to use [[Helix/Slurm#Interactive_Jobs|interactive batch jobs]] for compute and memory intensive compilation and pre- and post-processing tasks.

= Related Information =

* If you want to register a new token for the two factor authentication (2FA), consult the [[Registration/2FA|2FA Guide]].
* If you want to de-register, consult the [[Registration/Deregistration|De-registration Guide]].
* If you need an SSH key for your workflow, read [[Registration/SSH|Registering SSH Keys with your Cluster]].
* Configuring your shell: [[.bashrc Do's and Don'ts]]

= Troubleshooting =

'''Login Problems'''

If you encounter login problems, please double check the prerequisites. If those are valid, it is fastest to first check the following points before contacting the support at the [https://www.bwhpc.de/supportportal.php|bwhpc Suppport Portal].
If there are known issues with the cluster, we'll inform you via e-mail or in the message of the day at the cluster.

* Make sure to not use LAN and WLAN at the same time to prevent connection problems.
* Do [[Registration/bwForCluster/Helix#Troubleshooting_with_the_Help_of_bwServices | troubleshooting with bwServices]].

'''Lost Token'''

To reset your token you have to visit the IT-Service to validate your identity.
Mention that you want a reset of your Helix 2FA token. It is important to emphasize that the token reset is for bwForCluster Helix to avoid confusion with the token that is specific to members/students of the university of Heidelberg.

You can visit the IT service point of the URZ in person at INF 330, 69120 Heidelberg. Or you can use the video call service. It is recommended to use the Chrome browser for doing the video call. Please have on official identification ready in any case (Führerschein / Personalausweis) .
For further information please see here: https://it-service.uni-heidelberg.de.

To be on the safe side, it is advised to create a [[Registration/2FA#Backup_TAN_List|backup TAN list]] additionally to the smartphone token.

Registration/bwForCluster/Helix

2026-05-12T10:34:50Z

H Schumacher: Added too many failed login attempts issue.

__FORCETOC__

== Prerequisites ==

Steps A + B in the [[Registration/bwForCluster | registration process]] must be finished.

== Registration Steps ==

# Visit the bwForCluster Helix '''[https://bwservices.uni-heidelberg.de registration page bwServices]'''.
#: [[File:bwServices_chooseHomeOrganisation.png|right|300px|thumb|Select your home organization]]
# '''Select your home organization''' from the list on the main page and click Proceed/Fortfahren. You will be directed to the ''Identity Provider'' of your home organization.
# Enter the '''username and password''' of your home organization (usually these credentials are also used for other services like email) and click Login/Einloggen.
# When you log in to bwServices for the first time, an overview will appear, with the account information that your home institution submits to the system. Please verify that all data is valid and then click ''Continue/Weiter''.
# You will be redirected back to the bwServices main page. It shows all the statewide services you have access to. Click '''Register/Registrieren''' under the field labeled ''bwForCluster Helix''.
#: [[File:BwIDM-reg.png|center|frame|Register for Helix]]
# bwForCluster Helix uses a '''2-factor authentication''' (2FA) mechanism to increase security.
#: If you have never registered a 2FA token on bwIDM, the following error message will appear:
#: [[File:Bwidm-3-red.png|center|600px|thumb|Second factor missing.]]
#: Use this '''[https://bwservices.uni-heidelberg.de/user/twofa.xhtml link]''' or select '''My Tokens''' in the main menu.
#: To register a new token, please follow these '''[[Registration/2FA|instructions]]'''.
#: Please complete this step before continuing.
# Read the Terms of Use / Nutzungsbedingungen, place a check mark next to ''I have read and accepted the terms of use'' and click ''Register/Registrieren''.
# '''Set a service password''' for bwForCluster Helix and click ''Save/Speichern''.
#: Be sure to use a secure password that is different from any other passwords you currently use or have used on other systems.
[[File:BwIDM-passwd.png|center|600px|thumb|Set service password]]
{|style="background:#deffee; width:70%; margin: auto;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
The service password is your private password for accessing the bwForCluster Helix. Don't ever share it with anyone!
|}

== Registration Website bwServices ==
The registration website '''[https://bwservices.uni-heidelberg.de bwServices]''' offers the following functionality:
* Register for the service.
* Set or change your service password.
* Review your registration details under "Registry Info". This includes your account status (active/lost_access).
* bwServices triggers an update of your user information upon login.
* Registering an [[Registration/SSH | SSH key]]. It can be used for login to ''bwForCluster Helix''.
* Creating tokens for logins that need 2-factor authentication ([[Registration/2FA |2FA]]) like ''bwForCluster Helix'' and ''bwVisu''.
* De-register from the service.

=== Troubleshooting with the Help of bwServices ===
When encountering connection issues with a service, the bwServices website is a good starting point for troubleshooting:
* <ins>'''Issue:</ins> Expired contract''' or inactive student id:
** No access to state services like the bwHPC clusters or SDS@hd is possible.
** You can see that this is the issue when the login to bwServices via your identity provider is not possible.
* <ins>'''Issue:</ins> Recently (re)activated uni-id:''' Your id at your home organization was (re)activated a short time ago but this change didn't get through to the bwHPC cluster or SDS@hd service.
** Login to bwServices. This triggers an update of your user information. Some minutes later, the connection to the service should work.
* <ins>'''Issue:</ins> Entitlement / RV missing:''' Prerequisites like entitlement ownership or RV membership aren't met (anymore).
** You can see the concrete issue and advice on how to move forward when looking at the registry info of the service on the bwServices website.
** When the issue is solved, the account status changes from LOST_ACCESS to ACTIVE.
* <ins>'''Issue:</ins> Service Password:''' When trying to connect to the service, the service password is not accepted.
** Set a [[#Setting_a_New_Service_Password | new password]].
* <ins>'''Issue:</ins> OTP:''' In the case of login issues with a service that uses 2FA, you can check if you're OTP is working by going in the menu to ''Index'' -> ''My Tokens''.
** If you're token is not working, you might have had too many failed login attempts. You can request a reset of the error counter from the support.
** If the token needs to be reset, please follow the instructions for a [[Helix/Login#Troubleshooting | Lost Token]].

=== Setting a New Service Password ===

At any time, you can set a new service password via [https://bwservices.uni-heidelberg.de/ bwServices] by carrying out the following steps:
# Visit [https://bwservices.uni-heidelberg.de/ bwServices] and select your home organization.
# Authenticate yourself via your home-organizational user id / username and your home-organizational password.
# Find your service and select '''Set Service Password'''.
# Set new service password, repeat it and click the '''Save''' button. Use a secure password that is different from any other passwords you currently use or have used on other systems.
# The page answers e.g. "password has been changed".

Helix/bwVisu/JupyterLab

2026-05-07T14:32:03Z

H Schumacher: Added hint to load modules first.

[https://jupyter.org/ JupyterLab] is an integrated development environment (IDE) that provides a flexible and scalable interface for the Jupyter Notebook system. It supports interactive data science and scientific computing across over 40 programming languages (including Python, Julia, and R).

== Change Python Version ==

The default python version can be seen by running <code>python --version</code> in the terminal.

A different python version can be installed into a new virtual environment and then registered as IPython kernel for the usage in JupyterLab. This is explained in the chapter [[#Add_packages_via_conda_environments | add packages via conda environments]].

== Install Python Packages ==

Python packages can be added by installing them into a virtual environment and then creating an IPython kernel from the virtual environment. 
Kernels can be shared. See the notes below. 
If you want to move a virtual environment, it is adivsed to recreate it in the new place. Otherwise, dependencies based on relative paths will break.

# Create a virtual environment with...
#* [[#Add_packages_via_venv_virtual_environments | ...venv]] or
#* [[#Add_packages_via_Conda_virtual_environments | ...conda]] (choose this option if you want to install a different python version) or
#* ...[https://docs.astral.sh/uv/getting-started/ uv] if you want to install a different python version but don't want to use conda.
# [[#Create_an_IPython_Kernel | Create an IPython kernel]] from the virtual environment
# Use the kernel within JupyterLab
#* By default new kernels are saved under <code>~/.local/share/jupyter</code> and this location is automatically detected. Therefore, new kernels are directly available.
#*:[[File:BwVisu JuypterLab KernelPath.png|In the JupyterLab job configuration form, a custom kernel path can be provided.|right|thumb|x150px]]
#* If the kernel is saved somewhere else, the path can be provided in the "Kernel path" field when configuring the JupyterLab job (see image). For a kernel placed under <code>path_to_parent_dir/share/jupyter/kernels/my_kernel</code> the needed "Kernel path" would be <code>path_to_parent_dir/share/jupyter</code>.
# When the kernel is used the first time, the file <code>notebook_secrets</code> is created automatically. It can be found under "Kernel path". For others to use the kernel, they must have read access to this file. The command <code>chmod 750 notebook_secrets</code> would for example allow the whole SDS@hd SV read access. For the access management in workspaces, please see [[Workspace#Setting_Permissions_for_Sharing_Files| Wokspace Permissions]].

Notes regarding the sharing of IPython kernels
* The virtual environment and the kernel need to be placed in a shared directory. For example at SDS@hd.
* There could be a subdirectory for the virtual environments and one for the kernels.
* The path to the kernels is saved in the environment variable $JUPYTER_DATA_DIR. Jupyter relevant paths can be seen with <code>jupyter --paths</code>

=== Add packages via ''venv'' virtual environments ===
More information about venv or other python virtual environments can be found at the [[Development/Python | Python]] page.

Steps for creating a '''venv''' virtual environment:
<ol style="list-style-type: decimal;">
<li>Open a terminal.</li>
<li>Create a new virtual evironment:
<syntaxhighlight lang="bash">python3 -m venv <env_parent_dir>/<env_name></syntaxhighlight>
* <code>env_parent_dir</code> is the path to the folder where the virtual environment shall be created. Relative paths can be used.
* Caution: If you you want to share the environment with others, make sure to already create it in the shared place.
</li>
<li>Activate the environment:
<syntaxhighlight lang="bash">source <env_parent_dir>/<env_name>/bin/activate</syntaxhighlight></li>
<li>Update pip and install packages:
<syntaxhighlight lang="bash">pip install -U pip --no-user
# when the environment is installed in home
pip install <packagename>
# when the environment is installed somewhere else and shall not have dependencies in home so that others can access it as well
pip install <packagename> --no-user</syntaxhighlight></li>
<li> [[#Create_an_IPython_Kernel | Create an IPython kernel]]</li>
</ol>

=== Add packages via Conda virtual environments ===
More information about using conda can be found at the [[Development/Conda | Conda]] page.

<ol style="list-style-type: decimal;">
<li>Load the miniforge module by clicking first on the blue hexagon icon on the left-hand side of Jupyter's start page and then on the "load" button right of the entry for miniforge in the software module menu.
<li>Open a terminal.</li>
<li>Create a new virtual environment:
* If you are the only person using the environment, you can install it in your home directory:
<syntaxhighlight lang="bash">conda create --name <env_name> python=<python version></syntaxhighlight>
* If you want to install it into a different directory, for example a shared place:
<syntaxhighlight lang="bash">conda create --prefix <path_to_shared_directory>/<env_name> python=<python version>
</syntaxhighlight>
</li>
<li>Activate your environment:
<syntaxhighlight lang="bash">conda activate <myenv></syntaxhighlight></li>
<li>Install your packages:
<syntaxhighlight lang="bash">conda install <mypackage></syntaxhighlight></li>
<li> [[#Create_an_IPython_Kernel | Create an IPython kernel]]</li>
</ol>

== Create an IPython Kernel ==

Python kernels are implementations of the Jupyter notebook environment for different languages or virtual environments. You can switch between kernels easily, allowing you to use the best tool for a specific task.
conda_kernels

=== Create a kernel from a virtual environment ===

<ol>
<li>Activate the virtual environment.</li>
<li>Install the ipykernel package:
<syntaxhighlight lang="bash">pip install ipykernel</syntaxhighlight></li>
<li>Register the virtual environment as custom kernel to Jupyter.
* If you are the only person using the environment:
<syntaxhighlight lang="bash">python3 -m ipykernel install --user --name=<kernel_name></syntaxhighlight>
The kernel can be found under <code>~/.local/share/jupyter/kernels/</code>.
* If you installed the environment in a shared place and want to have the kernel there as well:
<syntaxhighlight lang="bash">python3 -m ipykernel install --prefix <path_to_kernel_folder> --name=<kernel_name></syntaxhighlight>
The kernel can then be found under <code>path_to_kernel_folder/share/jupyter/kernels/<kernel_name></code>. As long as the same <code>path_to_kernel_folder</code> is used, all kernels will be saved next to each other in "kernels".
</li>
</ol>

=== Multi-Language Support ===

JupyterLab supports over 40 programming languages including Python, R, Julia, and Scala. This is achieved through the use of different kernels.

==== R Kernel ====

<ol style="list-style-type: decimal;">
<li>On the cluster:
<pre>$ module load math/R
$ R
> install.packages('IRkernel')</pre></li>
<li>On bwVisu:
<ol style="list-style-type: decimal;">
<li>Start Jupyter App</li>
<li>In left menu: load math/R</li>
<li>Open Console:</li>

<pre>$ R
> IRkernel::installspec(displayname = 'R 4.2')</pre>

<li>Start kernel 'R 4.2' as console or notebook</li>
</ol>
</ol>

==== Julia Kernel ====

Load the math/julia module. Open the Terminal.

<pre>
julia
]
add IJulia
</pre>

After that, Julia is available as a kernel.

== Interactive Widgets ==

JupyterLab supports interactive widgets that can create UI controls for interactive data visualization and manipulation within the notebooks. Example of using an interactive widget:

<pre class="{.python">from ipywidgets import IntSlider
slider = IntSlider()
display(slider)</pre>
These widgets can be sliders, dropdowns, buttons, etc., which can be connected to Python code running in the backend.

== FAQ ==

<ol><li>'''How can I use bwForCluster Helix software modules?'''
First click on the blue hexagon icon on the left-hand side of Jupyter's start page. Then load a module by clicking on the "load" button next to the corresponding module entry.</li>
<li>'''My virtual environment works on Helix but not in the bwVisu JupyterLab job.'''
If the environment uses Helix modules, you have to load the modules in the bwVisu job first. For example, when you used the python 3.13 module for creating the environment, the path to this python version is saved in the environment. When the module is not loadedin bwVisu, the path is not available and you might get an error like "''[...]/bin/python: error while loading shared libraries: libpython3.13.so.1.0: cannot open shared object file: No such file or directory''"</li>
<li>'''How can I navigate to my SDS@hd folder in the file browser?'''
Please see the instructions at the [[Helix/bwVisu/Usage#Files | Usage]] page.</li>
<li>'''I prefer VSCode over JupyterLab. Can I start a JupyterLab job and then connect with it via VSCode?'''
This is not possible. Please start the job directly on Helix instead. You can find the instructions at the [[Development/VS_Code#Connect_to_Remote_Jupyter_Kernel | VSCode page]].</li>
<li>'''My conda commands are interrupted with message 'Killed'.'''
Request more memory when starting Jupyter.</li>
<li>'''Jupyterlab doesn't let me in but asks for a password.'''
Try using more memory for the job. If this doesn't help, try using the inkognito mode of your browser as the browser cache might be the problem.</li>
</ol>

SDS@hd

2026-05-06T13:52:17Z

H Schumacher: Added account management link

[[File:sds-hd-logo.png|350px]]

SDS@hd is a central service for securely storing scientific data (Scientific Data Storage). The service is provided as a state service to researchers of higher education institutions of Baden-Württemberg. It is intended to be used for data that is frequently accessed ('hot data').

{| style=" background:#FEF4AB; width:100%;"
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News
|-
|
* November 2022: No special entitlement is needed anymore to participate in an existing storage project.
* September 2023: Service has now been opened for DFN AAI & eduGAIN federation members to participate on existing storage projects.
|}

{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support
|-
|
* [https://www.urz.uni-heidelberg.de/en/service-catalogue/storage/sdshd-scientific-data-storage Service description & FAQ]
* [mailto:sds-hd-support@urz.uni-heidelberg.de Submit a Ticket]
|}

{| style=" background:#ffeaef; width:100%;"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | User Documentation
|-
|
* [[SDS@hd/Registration|Registration]] ([[SDS@hd/SV_Management|SV Management]], [[Registration/bwForCluster/Helix#Registration_Website_bwServices|Account Management]])
* [[SDS@hd/Access|Access]]
* [[SDS@hd/Security|Data Security]]
* [[Helix/bwVisu|bwVisu]] for visualization and interactive data analysis
|}

{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Storage Funding
|-
|
* Please [[SDS@hd/Acknowledgement|acknowledge]] SDS@hd in your publications.

Helix

2026-05-06T13:50:59Z

H Schumacher: Added account management link

[[File:Helix-logo.png|300px]]

The '''bwForCluster Helix''' is dedicated to research in the bwHPC domains Structural and Systems Biology, Medical Science, Soft Matter, Computational Humanities, and Mathematics and Computer Science.

{| style=" background:#FEF4AB; width:100%;"
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News
|-
|







* June 2025: [https://www.bwhpc.de/897.php Important notice regarding the use of MathWorks products]
* April 2025: Cluster extension with 3x gpu8 nodes (Nvidia H200)
|}

{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support
|-
|
* [[Helix/Getting Started|Getting Started]]
* E-Learning Course [https://training.bwhpc.de/goto.php/crs/695 Introduction to bwForCluster Helix]

* [https://www.bwhpc.de/supportportal.php Submit a Ticket] to support unit 'bwHPC - Support > bwForCluster Helix'

|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | User Documentation
|-
|
* [[Registration/bwForCluster|Registration]] ([[Registration/bwForCluster/RV/Management|RV Management]], [[Registration/bwForCluster/Helix#Registration_Website_bwServices|Account Management]])
* [[Helix/Login|Login]]
* [[Helix/Hardware|Hardware and Architecture]]
* [[Helix/Filesystems|File Systems and Workspaces]]
* [[Helix/Software|Cluster Specific Software]]
* [[Helix/Slurm|Batch System (Slurm)]]
* [[Helix/bwVisu|bwVisu]] for visualization and interactive applications
|}

{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Cluster Funding
|-
|
* Please [[Helix/Acknowledgement|acknowledge]] the cluster in your publications.
|}

Data Transfer/Rclone

2026-05-06T11:26:59Z

H Schumacher: Added smb to oneliner and config file

[https://rclone.org/docs/ Rclone] is a command line tool to manage files on remote systems (e.g. cloud storage systems). Rclone either synchronizes in one direction only or its mounting functionality is used with <code>rclone mount</code>. Data can be piped between two completely remote locations, sometimes without local download. One advantage is that the transfer is multithreaded and it operates on a file level basis.
'''Caution:''' You can't use Rclone with 2FA.

== Installation ==

Rclone is a Go program and comes as a single binary file.

# Download the relevant binary.
# Extract the <code>rclone</code> executable, <code>rclone.exe</code> on Windows, from the archive.
# You can use the executables without further installation. For easy use, it is recommended to add the binary to your PATH environment variable. Information on how to do this can be found below.

Detailed information regarding different operating systems can be found here:

* Installation on [https://rclone.org/install/#windows Windows]
* Installation on [https://rclone.org/install/#macos macOS]
* Installation on [https://rclone.org/install/#script-installation Linux]

== Usage Rclone ==

To use Rclone you have to define a config file. Afterwards you can connect by using the name of your configured connections.

=== Configure Remote ===

Before you can start using Rclone, you need to set up a remote. This means to configure a specific connection by providing authentication information, the network protocol that you want to use and a name for this configuration so that you can use it later on.

To configure a remote for a specific service, you need the following information:

* <code><remotehost></code>
* <code><username></code>
* <code><servicePassword></code>

Furthermore, you have to decide on:

* <code>network protocol</code> (for example webDAV, smb, sftp)
* <code>remote-name</code> (for example you can use the name of the service you want to connect to)

You have three different options to set up a new remote which are explained by the following sections.

'''Interactive Setup''' 

Execute:
<pre>rclone config</pre>
This will guide you through an interactive setup process. You can find detailed instructions at the website:

* [https://rclone.org/webdav/ Connect via webdav]
* [https://rclone.org/smb/ Connect via smb]
* [https://rclone.org/sftp/ Connect via sftp]

'''Oneliner''' 

Define all parameters in one command. For example:
<syntaxhighlight lang="bash"># smb
rclone config create <remote-name> smb host=<remotehost> user=<username> domain=<domain> pass=<password> --obscure
# sftp
rclone config create <remote-name> sftp host=<remotehost> user=<username> pass=<password> --obscure
# webdav
rclone config create <remote-name> webdav url=<remotehost> user=<username> pass=<password> --obscure
</syntaxhighlight>

'''Adjust Config File''' 
<syntaxhighlight lang="bash">
# Make sure that a config file exists. Access rights for the config file are set to 600.
rclone config touch
# Get the location of the file:
rclone config file
</syntaxhighlight>
You can use the following snippet as template for your connections.
<syntaxhighlight lang="bash">[<remote-name>]
type = smb
host = <hostname>
user = <userID>
domain = <e.g. BWSERVICESAD>

[<remote-name>]
type = webdav
url = <hostURL>
vendor = other
user = <userID>

[<remote-name>]
type = sftp
host = <hostname>
user = <userID>
key_use_agent = false

[<remote-name-ssh-version>]
type = sftp
host = <hostname>
user = <userID>
key_file = ~/.ssh/id_ed25519
pubkey_file = ~/.ssh/id_ed25519.pub
key_use_agent = true
shell_type = none
</syntaxhighlight>
To add the password, please use the following command. Make sure to keep the prepended space so that the command is not saved in your command history.
<syntaxhighlight> rclone config update <remote-name> pass=<password> --obscure</syntaxhighlight>

=== Use Remote ===

The syntax to use Rclone is like this:

<pre>rclone [options] subcommand <parameters> <parameters...>
</pre>
List all directories/containers/buckets in the folder XX.

<pre>rclone lsd <remote-name>:XX
</pre>
Copies /local/path to the remote path

<pre>rclone copy </local/path> <remote-name>:<remote/path>
</pre>
Copies fom remote path to /local/path

<pre>rclone copy <remote-name>:<remote/path> </local/path>
</pre>
Moves the contents of the source directory to the destination directory.

<pre>rclone move <remote-name>:<source/path> <remote-name>:<destination/path>
</pre>
More subcommands can be found [https://rclone.org/docs/#subcommands here].

== Usage Rclone Mount ==

Before you can follow the instructions in this chapter, you need to have set up a [[Data_Transfer/Rclone#Usage_Rclone | remote]].
Detailed information on how to use rclone mount can be found [https://rclone.org/commands/rclone_mount/ here].

=== Windows ===

To run rclone mount on Windows, you will need to [https://winfsp.dev/rel/ download and install WinFsp]. To mount on drive letter X or a nonexistent subdirectory, use:

<pre>rclone mount <remote-name>:path/to/files X:
rclone mount <remote-name>:path/to/files C:\path\parent\mount</pre>
In contrast to Linux/Mac, there is no background mode.

=== MacOS & Linux ===

You can run mount in either foreground or background (aka daemon) mode. Mount runs in foreground mode by default. Use the <code>--daemon</code> flag to force background mode. If this doesn't work, you can put an <code>&</code> at the end of the command instead.

Create an empty directory on your local machine and then execute

<pre># to mount the root folder:
rclone mount --vfs-cache-mode full <remote-name>: /path/to/empty/folder
# to mount a subfolder:
rclone mount --vfs-cache-mode full <remote-name>:folderX/folderY /path/to/empty/folder
# to unmount:
fusermount -uz /path/to/mounted/folder </pre>

== Best Practices ==

Rclone has a lot of useful options.

=== Performance ===

To be able to utilize a larger bandwidth, it is helpful to add the following options for increased performance:

<pre>
--transfers <int>
</pre>

Number of file transfers to run in parallel (default: 4). Depending on the local Network, read and write speeds on the file system, and current load, different values might be best. For large transfers, it is advised to test local performance with different values beforehand.

* In our tests, we observed the best results between 8 and 32.
* For regular use cases, we recommend 16 as the default.
* Values above 64 are not recommended and degrade performance.

<pre>
--multi-thread-streams <int>
</pre>

Number of streams to use for multithreaded downloads (default: 4). Only important on very large files. This will cause multithreaded up/download on chunk-sized bits of the file.

The optimal value is highly specific to the local network and used Hardware. For regular use cases, we recommend 4 as the default.

=== Debugging and Statistics ===
To get updates on current progress, use:
<pre>
--stats
</pre>
Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable) (default 1m0s).

To get debug information, use:
<pre>
--log-level=DEBUG
--stats-log-level=DEBUG
</pre>

Helix/bwVisu/Getting Started

2026-05-06T10:50:22Z

H Schumacher: Added image

== Motivation ==

When large quantities of scientific data is created (for example, from simulations), it is often necessary to employ hardware for further processing that is just as powerful as e.g. the HPC cluster that created the data in the first place. Transferring the data to a local workstation for visualization and analysis is typically not only time-consuming because of the data transfer itself, but the workstation may be ill-equipped to handle the workload. For such scenarios, remote solutions are needed, that employ powerful hardware and bring the applications to the data - and not vice versa.

bwVisu provides such a remote service. The user can log in to an intuitive web frontend which manages interactive jobs running on bwForCluster Helix. Once a job has been started, the user can connect to it such that the application appears in a browser window with full 3D hardware acceleration provided by the cluster. The only requirement on the client side is that the browser must support HTML5.

== Registration ==

In order to use bwVisu, it is required to be a user of the [https://wiki.bwhpc.de/e/Registration/bwForCluster bwForCluster Helix] or a user of [https://wiki.bwhpc.de/e/SDS@hd/Registration SDS@hd].
# Register for the service bwVisu at [http://bwservices.uni-heidelberg.de bwServices].
# bwVisu uses a 2-factor authentication (2FA) mechanism to increase security. If you have not registered a 2FA token at bwServices, an error message will appear. Use this '''[https://bwservices.uni-heidelberg.de/user/twofa.xhtml link]''' or select '''My Tokens''' in the main menu of bwServices. To register a new token, please follow these '''[[Registration/2FA|instructions]]'''. Please complete this step before continuing.
# Once your registration has been completed, your user account will be set up. This can take up to 10 minutes, so please wait for around ten minutes before proceeding.

== Login ==

# Go to the bwVisu web frontend at [https://bwvisu.bwservices.uni-heidelberg.de/ bwvisu.bwservices.uni-heidelberg.de].
# Login with the credentials of your home university and provide a 2FA token.

== Usage ==

[[File:Bwvisu2-gui.png|x150px|right|bwVisu Workflow]]
As shown in the image, the usual workflow consists of three steps: You choose an application, define the needed job resources and then connect to the job as soon as it is running. Detailed instructions are [[Helix/bwVisu/Usage|here]].

== Technical Information ==

=== Data storage ===

bwVisu provides no storage space. For data storage you have access to the filesystems of bwForCluster Helix and to SDS@hd.

=== Open OnDemand ===

bwVisu is based on [https://openondemand.org Open OnDemand] an open-source software platform designed to provide easy and streamlined access to high-performance computing (HPC) resources. Developed by the Ohio Supercomputer Center, it allows users to access HPC resources via a standard web browser, eliminating the need for a traditional command line interface.

The platform provides a range of tools and features. bwVisu uses it to provide the following features:
<ol>
<li>'''Web-based Access:''' Users can manage jobs, files, and data directly through their browser.</li>
<li>'''Interactive Applications:'''</li>
<ul>
<li>It supports GUI-based applications like Jupyter Notebooks and RStudio, enabling interactive computing and visualization.</li>
<li>For most bwVisu applications you can connect to a running job with your browser.</li>
<li>Some applications may require a dedicated desktop client.</li>
</ul>
<li>'''Job Management:''' Users can easily submit and manage their interactive computing jobs on HPC resources. For members of bwForCluster Helix, the jobs can be monitored via the [https://helix-monitoring.bwservices.uni-heidelberg.de/ monitoring portal].</li>
<li>'''File Management:''' Provides a file browser that allows users to upload and manage their data files without needing to use command-line tools.</li>
</ol>

Helix/bwVisu/Usage

2026-05-06T10:34:45Z

H Schumacher: Added output.log information, formatting

== Basic Navigation ==

bwVisu is quite simple and intuitively designed to make launching apps on the cluster as user-friendly as possible. Once you have successfully logged into bwVisu, the first thing you see is a list of pinned apps. These are the available apps that you can launch as a job on the cluster.

In the navigation bar, the user has the following options:

<code>Files</code> — <code>Interactive Apps</code> — <code>My Interactive Sessions</code> — <code>Shell</code>

=== Files ===

Here, you can access your home directory on Helix. In a graphical user interface, files can be downloaded and uploaded, and new files and folders can be created. You will encounter an error message if you try to see data that you have no access to.

In the Home Directory the results and logs from the apps are stored. You get the following interface:

[[File:BwVisuFiles.png|bwVisu-files|center|x300px]]

'''For SDS@hd users:''' Data on SDS@hd (under /mnt/sds-hd) can only be accessed from within jobs. Therefore, you need to open a terminal (within a JupyterLab or Rstudio job or directly on Helix) and set a symbolic link to your SDS@hd folder in your home directory. For example like this:
<pre>cd $HOME
mkdir sds-hd
cd sds-hd
ln -s /mnt/sds-hd/sd16a001 sd16a001</pre>

=== Interactive Apps ===

This displays the apps that the user can run on the cluster, possibly across multiple nodes. These apps initiate an interactive session which the user can directly connect to and, depending on the app, receive a graphical interface.

=== My Interactive Sessions ===

At bwVisu, it is possible to start several interactive apps simultaneously. Here, one can view an overview of currently running interactive apps and completed sessions. It is also possible to view the log for each session. As soon as one starts an interactive app, one is automatically redirected here. Even if you log out of bwVisu and log back in, you can still see the information and logs from the finished interactive apps here until you delete it.

=== Shell ===
For bwForCluster Helix users only: After entering your OTP and password a shell on the login node of bwForCluster Helix opens. This way, you can for example create workspaces when you need more space. Please look up the details under [[Helix/Filesystems#Workspaces | Helix Workspaces]]. They can't be seen in the File Viewer but are available within the bwVisu jobs.

== Start a Job ==
[[File:bwVisuConcept.png|center|x170px]]
<ol>
<li> At the start page of the web frontend, you can start a new job by clicking on an application version in the list of pinned apps.</li>
<li>You will now see the application description and options to adjust the job resources.</li>
<ul>
<li> To find the best resource settings for your jobs, please have a look at the [https://helix-monitoring.bwservices.uni-heidelberg.de/ bwForCluster Helix monitoring portal] . There, you can see how much of the requested resources where actually used by your jobs.
<li> At the bottom of the page, beneath the <code>Launch</code> button, you can find a link for accessing the logs/session data of all your jobs.</li>
<li>The interface can differ per app because each app can have individual configuration options. </li>
[[File:bwVisuRStudio.png|bwVisu-rstudio|center|x200px]]</li>
</ul>
<li>Click the <code>Launch</code> button to start a new job with this application. You can't start two jobs with the same application. Launching an app can take some time depending on the requested resources and cluster utilization.</li>
<li>The <code>Interactive Sessions</code> page opens with the details of your job.</li>
<ul>
<li>'''Host:''' Indicates on which nodes the application is running.</li>
<li>'''Time Remaining:''' Each application is given a time limit, which can also be set in advance. In this example, the application is supposed to run for one hour.</li>
<li>'''Session ID:''' This is a link that leads to the folder of the session. Among other things, application logs are displayed here, which can be viewed in the UI.
[[File:BwVisuRstudioRun.png|bwVisu-rstudio-run|center|x200px]] </li>
</li>
</ul>
<li> Connect to the job with your browser by clicking <code>connect to <application></code>. </li>
</ol>

== Adjust noVNC Applications ==
<div><ul>
<li style="display: inline-block; vertical-align: middle; float: right;"> [[File:bwVisu Fiji noVNCMenu.png|thumb|none|x200px|noVNC menu within Fiji]] </li>
<li style="display: inline-block; vertical-align: top; float: right;"> [[File:bwVisu_Fiji_Running.png|thumb|none|x200px|Job details view of a running Fiji Job]] </li>
</ul></div>
noVNC is an open-source VNC browser client. Some bwVisu applications use noVNC and therefore, come with additional options in the view of the job details and an additional menu within the running application.

*'''Running Job View''' on the interactive sessions page
** Compression slider: Before opening the application, you can choose the compression level to use. 0 means no compression while 9 offers the best compression but uses more CPU resources on the cluster which can be slower.
** Image Quality slider: Before opening the application, you can choose the image quality level. With this you can control the desired JPEG quality.
** View Only (Share-able Link) button: By clicking this button the application opens in view only mode. You can share this link with collaborators or students while you launch your noVNC application normally for yourself. This way, others can follow what you are doing. For every job you need to generate a new link.
* '''noVNC menu''' within the running application
** When you open the application you can see an additional menu on the left with options. For the ones using the application in view-only mode it is helpful to select the option to show a dot instead of not showing the mouse cursor at all. Therefore go to "Options" and then "Expand".
** For copying data, you need to use the clipboard. When copying something locally, you need to put it into the clipboard before you can use it within the application. The same goes the other way around.
* '''Adjust window size''' and position of the app by executing a right click at the edge of the app window frame, so that a menu shows up.

== Terminate a Job ==

The job will terminate when the job runtime requested in the job submission form has expired To terminate an application yourself, you can click the delete button or end the application in the interactive session. If the app is terminated in the interactive session, the window no longer shows <code>Running</code> but <code>Completed</code>. With the <code>Delete</code> button, you end the session and also remove the list entry from <code>My Interactive Sessions</code>.

== Troubleshooting ==
Maintance windows and known issues are communicated via the email list or the ''News'' section at the bwVisu start page in the wiki.
* Depending on the problem, it can help to look up the related bwForCluster Helix wiki pages. For example:
*: '''Error:''' Disk space exceeded → '''[https://wiki.bwhpc.de/e/Helix/Filesystems#Troubleshooting Filesystems: Troubleshooting]'''
* '''Issue:''' The webservice is not accessible.
*: If you're a bwForCluster Helix user, you need to have logged in to Helix at least once so that you're home directory gets created. Afterwards, the webservice page should be accessible.
* '''Issue:''' The bwVisu job fails or doesn't start.
*: You can try to look for the issue yourself:
*:* Reload the page without using the cache: <code>Strg+Shift+R</code> (Windows/Linux) or <code>Cmd+Shift+R</code> (Mac)
*:* Empty the browser cache:
*:** Chrome/Edge: <code>Strg+Shift+Entf</code> → choose timeframe all → choose ''Images and Files in Cache'' → choose ''Delete Files''
*:** Firefox: <code>Strg+Shift+Entf</code> → choose timeframe all → choose ''Cache'' → choose ''Delete Files''
*:* Test alternative browser (Firefox, Chrome, Edge, ...)
*:* Use a private session (without saved cookies and cache): <code>Strg+Shift+N</code> (Chrome) or <code>Strg+Shift+P</code> (Firefox).
* '''Support:''' If you can't solve the issue, write to the support and provide the following information:
** Date and time of the issue.
** Job-id (the number in brackets after the jobname)
** The <code>output.log</code> file that you can find when following the session id link.
** Please report patterns that you recognize. For example:
*** The issue happens around a specific time of day.
*** The issue is connected to the duration of usage (e.g. happens always after ca. one hour).
*** The issue appears after specific user interactions (using a specific method, opening the terminal, clicking a specific button, ...)
: You might be asked for further information:
:* The internet browser that you user and its version. For example Firefox 128.
:* Provide more information about network traffic via a HAR-file. First start a job and open the application. Then follow he instructions below:
:** Instructions for Chrome/Edge:
:**# Open developer tools with <code>F12</code> or right click -> ''Inspect'' Then choose the tab ''Network''.
:**# Activate the checkbox ''Preserve log''.
:**# Reproduce the issue in the current tab.
:**# After the problem happened again: Click the export symbol (⬇️) in the menu of the network-tabs and choose ''Export HAR...''.
:** Instructions for Firefox:
:**# Open developer tools with <code>F12</code> and choose the tab ''Network analysis''.
:**# Click the settings symbol (⚙️) and activate ''Continuous Protocol''.
:**# Reproduce the issue in the current tab.
:**# After the problem happened again: Click the settings symbol again (⚙️) and choose ''Save everything as HAR''.

Registration/bwForCluster

2026-05-05T17:43:47Z

H Schumacher: Used same formatting as for bwUniCluster

= Registration bwForCluster =

A bwForCluster is a cluster for a specific [https://www.bwhpc.de/cluster.php research area].
You can apply for a bwForCluster and the "cluster assignment team" will assign you to the appropriate cluster for your research area, taking into account your specific hardware/software needs.
bwForClusters are funded by the German Research Foundation (DFG) and the Ministry of Science, Research and the Arts of Baden-Württemberg on the basis of grant applications (cf. proposals application guidelines according to Art. 91b GG).

All members of the universities in Baden-Württemberg can apply for an account.

The use of the bwForClusters is free of charge.

== Three Steps for Registration ==

[[File:bwForCluster-Registration.png|thumb|right|upright=1.2|bwForCluster Registration Process]]
The registration process for a bwForCluster is divided into three steps, whereby step A+B can be performed in parallel.
When both are completed, you can perform step C.

To which cluster you get access depends on your research area and will be decided in step B.

* Step A: You need to get the '''bwForCluster Entitlement''' from your university/college. → '''[[Registration/bwForCluster/Entitlement|bwForCluster User Access Step A]]'''
* Step B: You need to '''apply for or join a Rechenvorhaben/project''' on the "central application site" (ZAS). → '''[[Registration/bwForCluster/RV|bwForCluster User Access Step B]]'''
* Step C: You need to '''register for a bwForCluster''' (create an account on the cluster). → '''[[Registration/bwForCluster/Service|bwForCluster User Access Step C]]'''

After finishing Step C, you are done with registration.

* To '''login''' to one of the bwForClusters, please refer to the general → '''[[Registration/Login|Login Guide]]'''

== Information for registered users ==

* If you want to check if your '''account''' is still valid on one of the bwForClusters, please refer to the general → '''[[Registration/Account|Account Guide]]'''
* If you want to '''create a second factor''', please refer to → '''[[Registration/2FA|Generate a Second Factor (2FA)]]''' (Justus 2, Helix, NEMO2)
* If you need to '''change or forgot your password''' for a bwForCluster, please refer to the general → '''[[Registration/Password|Password Guide]]'''
* If you want to '''use SSH keys''' on a bwForCluster, please refer to → '''[[Registration/SSH|Registering SSH Keys with your Cluster]]'''
* If you want do '''de-register your user account''' from a bwForCluster, please refer to the general → '''[[Registration/Deregistration|De-registration Guide]]'''

Helix/bwVisu/Usage

2026-05-05T15:39:06Z

H Schumacher: Added things to try and information to report.

== Basic Navigation ==

bwVisu is quite simple and intuitively designed to make launching apps on the cluster as user-friendly as possible. Once you have successfully logged into bwVisu, the first thing you see is a list of pinned apps. These are the available apps that you can launch as a job on the cluster.

In the navigation bar, the user has the following options:

<code>Files</code> — <code>Interactive Apps</code> — <code>My Interactive Sessions</code> — <code>Shell</code>

=== Files ===

Here, you can access your home directory on Helix. In a graphical user interface, files can be downloaded and uploaded, and new files and folders can be created. You will encounter an error message if you try to see data that you have no access to.

In the Home Directory the results and logs from the apps are stored. You get the following interface:

[[File:BwVisuFiles.png|bwVisu-files|center|x300px]]

'''For SDS@hd users:''' Data on SDS@hd (under /mnt/sds-hd) can only be accessed from within jobs. Therefore, you need to open a terminal (within a JupyterLab or Rstudio job or directly on Helix) and set a symbolic link to your SDS@hd folder in your home directory. For example like this:
<pre>cd $HOME
mkdir sds-hd
cd sds-hd
ln -s /mnt/sds-hd/sd16a001 sd16a001</pre>

=== Interactive Apps ===

This displays the apps that the user can run on the cluster, possibly across multiple nodes. These apps initiate an interactive session which the user can directly connect to and, depending on the app, receive a graphical interface.

=== My Interactive Sessions ===

At bwVisu, it is possible to start several interactive apps simultaneously. Here, one can view an overview of currently running interactive apps and completed sessions. It is also possible to view the log for each session. As soon as one starts an interactive app, one is automatically redirected here. Even if you log out of bwVisu and log back in, you can still see the information and logs from the finished interactive apps here until you delete it.

=== Shell ===
For bwForCluster Helix users only: After entering your OTP and password a shell on the login node of bwForCluster Helix opens. This way, you can for example create workspaces when you need more space. Please look up the details under [[Helix/Filesystems#Workspaces | Helix Workspaces]]. They can't be seen in the File Viewer but are available within the bwVisu jobs.

== Start a Job ==
[[File:bwVisuConcept.png|center|x170px]]
<ol>
<li> At the start page of the web frontend, you can start a new job by clicking on an application version in the list of pinned apps.</li>
<li>You will now see the application description and options to adjust the job resources.</li>
<ul>
<li> To find the best resource settings for your jobs, please have a look at the [https://helix-monitoring.bwservices.uni-heidelberg.de/ bwForCluster Helix monitoring portal] . There, you can see how much of the requested resources where actually used by your jobs.
<li> At the bottom of the page, beneath the <code>Launch</code> button, you can find a link for accessing the logs/session data of all your jobs.</li>
<li>The interface can differ per app because each app can have individual configuration options. </li>
[[File:bwVisuRStudio.png|bwVisu-rstudio|center|x200px]]</li>
</ul>
<li>Click the <code>Launch</code> button to start a new job with this application. You can't start two jobs with the same application. Launching an app can take some time depending on the requested resources and cluster utilization.</li>
<li>The <code>Interactive Sessions</code> page opens with the details of your job.</li>
<ul>
<li>'''Host:''' Indicates on which nodes the application is running.</li>
<li>'''Time Remaining:''' Each application is given a time limit, which can also be set in advance. In this example, the application is supposed to run for one hour.</li>
<li>'''Session ID:''' This is a link that leads to the folder of the session. Among other things, application logs are displayed here, which can be viewed in the UI.
[[File:BwVisuRstudioRun.png|bwVisu-rstudio-run|center|x200px]] </li>
</li>
</ul>
<li> Connect to the job with your browser by clicking <code>connect to <application></code>. </li>
</ol>

== Adjust noVNC Applications ==
<div><ul>
<li style="display: inline-block; vertical-align: middle; float: right;"> [[File:bwVisu Fiji noVNCMenu.png|thumb|none|x200px|noVNC menu within Fiji]] </li>
<li style="display: inline-block; vertical-align: top; float: right;"> [[File:bwVisu_Fiji_Running.png|thumb|none|x200px|Job details view of a running Fiji Job]] </li>
</ul></div>
noVNC is an open-source VNC browser client. Some bwVisu applications use noVNC and therefore, come with additional options in the view of the job details and an additional menu within the running application.

*'''Running Job View''' on the interactive sessions page
** Compression slider: Before opening the application, you can choose the compression level to use. 0 means no compression while 9 offers the best compression but uses more CPU resources on the cluster which can be slower.
** Image Quality slider: Before opening the application, you can choose the image quality level. With this you can control the desired JPEG quality.
** View Only (Share-able Link) button: By clicking this button the application opens in view only mode. You can share this link with collaborators or students while you launch your noVNC application normally for yourself. This way, others can follow what you are doing. For every job you need to generate a new link.
* '''noVNC menu''' within the running application
** When you open the application you can see an additional menu on the left with options. For the ones using the application in view-only mode it is helpful to select the option to show a dot instead of not showing the mouse cursor at all. Therefore go to "Options" and then "Expand".
** For copying data, you need to use the clipboard. When copying something locally, you need to put it into the clipboard before you can use it within the application. The same goes the other way around.
* '''Adjust window size''' and position of the app by executing a right click at the edge of the app window frame, so that a menu shows up.

== Terminate a Job ==

The job will terminate when the job runtime requested in the job submission form has expired To terminate an application yourself, you can click the delete button or end the application in the interactive session. If the app is terminated in the interactive session, the window no longer shows <code>Running</code> but <code>Completed</code>. With the <code>Delete</code> button, you end the session and also remove the list entry from <code>My Interactive Sessions</code>.

== Troubleshooting ==
Maintance windows and known issues are communicated via the email list or the News section at the bwVisu start page in the wiki.
* Depending on the problem, it can help to look up the related bwForCluster Helix wiki pages. For example:
*: '''Error:''' Disk space exceeded → '''[https://wiki.bwhpc.de/e/Helix/Filesystems#Troubleshooting Filesystems: Troubleshooting]'''
* '''Issue:''' The webservice is not accessible.
*: If you're a bwForCluster Helix user, you need to have logged in to Helix at least once so that you're home directory gets created. Afterwards, the webservice page should be accessible.
* '''Issue:''' The bwVisu job fails or doesn't start.
*: You can try to look for the issue yourself:
*:* Reload the page without using the cache: `Strg+Shift+R` (Windows/Linux) or `Cmd+Shift+R` (Mac)
*:* Empty the browser cache:
*:** Chrome/Edge: `Strg+Shift+Entf` → choose timeframe all → choose "Images and Files in Cache" → choose "Delete Files"
*:** Firefox: `Strg+Shift+Entf` → choose timeframe all → choose "Cache" → choose "Delete Files"
*:** Test alternative browser (Firefox, Chrome, Edge, ...)
*:** Use a private session (without saved cookies and cache): `Strg+Shift+P` (Firefox) or `Strg+Shift+N` (Chrome).
*: If this doesn't help, write to the support and provide the following information:
*:* Date and time of the issue.
*:* Job-id
*:* Please report patterns that you recognize. For example:
*:** The issue happens around a specific time of day.
*:** The issue is connected to the duration of usage (e.g. happens always after ca. one hour).
*:** The issue appears after specific user interactions (using a specific method, opening the terminal, clicking a specific button, ...)
*: You might be asked for further information:
*:* The internet browser that you user and its version. For example Firefox 128.
*:* Provide more information about network traffic via a HAR-file.
*:** Instructions for Chrome/Edge:
*:**# Open developer tools with `F12` or right click -> "Inspect"
*:**# Choose the tab "Network"
*:**# Activate the checkbox "Preserve log"
*:**# Reproduce the issue (open new terminal)
*:**# After the problem happened again: Click the Export-Symbol (⬇️) in the menu of the network-tabs and choose "Export HAR..."
*:** Instructions for Firefox:
*:**# Open developer tools with `F12`
*:**# Choose the tab "Network analysis"
*:**# Click the options symbol (⚙️) an activate "Continuous Protocol"
*:**# Reproduce the issue (open new terminal)
*:**# After the problem happened again: Click the option symbol again (⚙️) and choose "Save everything as HAR".

Helix/bwVisu/JupyterLab

2026-05-05T11:24:22Z

H Schumacher: Added instructions for using Helix modules

[https://jupyter.org/ JupyterLab] is an integrated development environment (IDE) that provides a flexible and scalable interface for the Jupyter Notebook system. It supports interactive data science and scientific computing across over 40 programming languages (including Python, Julia, and R).

== Change Python Version ==

The default python version can be seen by running <code>python --version</code> in the terminal.

A different python version can be installed into a new virtual environment and then registered as IPython kernel for the usage in JupyterLab. This is explained in the chapter [[#Add_packages_via_conda_environments | add packages via conda environments]].

== Install Python Packages ==

Python packages can be added by installing them into a virtual environment and then creating an IPython kernel from the virtual environment. 
Kernels can be shared. See the notes below. 
If you want to move a virtual environment, it is adivsed to recreate it in the new place. Otherwise, dependencies based on relative paths will break.

# Create a virtual environment with...
#* [[#Add_packages_via_venv_virtual_environments | ...venv]] or
#* [[#Add_packages_via_Conda_virtual_environments | ...conda]] (choose this option if you want to install a different python version) or
#* ...[https://docs.astral.sh/uv/getting-started/ uv] if you want to install a different python version but don't want to use conda.
# [[#Create_an_IPython_Kernel | Create an IPython kernel]] from the virtual environment
# Use the kernel within JupyterLab
#* By default new kernels are saved under <code>~/.local/share/jupyter</code> and this location is automatically detected. Therefore, new kernels are directly available.
#*:[[File:BwVisu JuypterLab KernelPath.png|In the JupyterLab job configuration form, a custom kernel path can be provided.|right|thumb|x150px]]
#* If the kernel is saved somewhere else, the path can be provided in the "Kernel path" field when configuring the JupyterLab job (see image). For a kernel placed under <code>path_to_parent_dir/share/jupyter/kernels/my_kernel</code> the needed "Kernel path" would be <code>path_to_parent_dir/share/jupyter</code>.
# When the kernel is used the first time, the file <code>notebook_secrets</code> is created automatically. It can be found under "Kernel path". For others to use the kernel, they must have read access to this file. The command <code>chmod 750 notebook_secrets</code> would for example allow the whole SDS@hd SV read access. For the access management in workspaces, please see [[Workspace#Setting_Permissions_for_Sharing_Files| Wokspace Permissions]].

Notes regarding the sharing of IPython kernels
* The virtual environment and the kernel need to be placed in a shared directory. For example at SDS@hd.
* There could be a subdirectory for the virtual environments and one for the kernels.
* The path to the kernels is saved in the environment variable $JUPYTER_DATA_DIR. Jupyter relevant paths can be seen with <code>jupyter --paths</code>

=== Add packages via ''venv'' virtual environments ===
More information about venv or other python virtual environments can be found at the [[Development/Python | Python]] page.

Steps for creating a '''venv''' virtual environment:
<ol style="list-style-type: decimal;">
<li>Open a terminal.</li>
<li>Create a new virtual evironment:
<syntaxhighlight lang="bash">python3 -m venv <env_parent_dir>/<env_name></syntaxhighlight>
* <code>env_parent_dir</code> is the path to the folder where the virtual environment shall be created. Relative paths can be used.
* Caution: If you you want to share the environment with others, make sure to already create it in the shared place.
</li>
<li>Activate the environment:
<syntaxhighlight lang="bash">source <env_parent_dir>/<env_name>/bin/activate</syntaxhighlight></li>
<li>Update pip and install packages:
<syntaxhighlight lang="bash">pip install -U pip --no-user
# when the environment is installed in home
pip install <packagename>
# when the environment is installed somewhere else and shall not have dependencies in home so that others can access it as well
pip install <packagename> --no-user</syntaxhighlight></li>
<li> [[#Create_an_IPython_Kernel | Create an IPython kernel]]</li>
</ol>

=== Add packages via Conda virtual environments ===
More information about using conda can be found at the [[Development/Conda | Conda]] page.

<ol style="list-style-type: decimal;">
<li>Load the miniforge module by clicking first on the blue hexagon icon on the left-hand side of Jupyter's start page and then on the "load" button right of the entry for miniforge in the software module menu.
<li>Open a terminal.</li>
<li>Create a new virtual environment:
* If you are the only person using the environment, you can install it in your home directory:
<syntaxhighlight lang="bash">conda create --name <env_name> python=<python version></syntaxhighlight>
* If you want to install it into a different directory, for example a shared place:
<syntaxhighlight lang="bash">conda create --prefix <path_to_shared_directory>/<env_name> python=<python version>
</syntaxhighlight>
</li>
<li>Activate your environment:
<syntaxhighlight lang="bash">conda activate <myenv></syntaxhighlight></li>
<li>Install your packages:
<syntaxhighlight lang="bash">conda install <mypackage></syntaxhighlight></li>
<li> [[#Create_an_IPython_Kernel | Create an IPython kernel]]</li>
</ol>

== Create an IPython Kernel ==

Python kernels are implementations of the Jupyter notebook environment for different languages or virtual environments. You can switch between kernels easily, allowing you to use the best tool for a specific task.
conda_kernels

=== Create a kernel from a virtual environment ===

<ol>
<li>Activate the virtual environment.</li>
<li>Install the ipykernel package:
<syntaxhighlight lang="bash">pip install ipykernel</syntaxhighlight></li>
<li>Register the virtual environment as custom kernel to Jupyter.
* If you are the only person using the environment:
<syntaxhighlight lang="bash">python3 -m ipykernel install --user --name=<kernel_name></syntaxhighlight>
The kernel can be found under <code>~/.local/share/jupyter/kernels/</code>.
* If you installed the environment in a shared place and want to have the kernel there as well:
<syntaxhighlight lang="bash">python3 -m ipykernel install --prefix <path_to_kernel_folder> --name=<kernel_name></syntaxhighlight>
The kernel can then be found under <code>path_to_kernel_folder/share/jupyter/kernels/<kernel_name></code>. As long as the same <code>path_to_kernel_folder</code> is used, all kernels will be saved next to each other in "kernels".
</li>
</ol>

=== Multi-Language Support ===

JupyterLab supports over 40 programming languages including Python, R, Julia, and Scala. This is achieved through the use of different kernels.

==== R Kernel ====

<ol style="list-style-type: decimal;">
<li>On the cluster:
<pre>$ module load math/R
$ R
> install.packages('IRkernel')</pre></li>
<li>On bwVisu:
<ol style="list-style-type: decimal;">
<li>Start Jupyter App</li>
<li>In left menu: load math/R</li>
<li>Open Console:</li>

<pre>$ R
> IRkernel::installspec(displayname = 'R 4.2')</pre>

<li>Start kernel 'R 4.2' as console or notebook</li>
</ol>
</ol>

==== Julia Kernel ====

Load the math/julia module. Open the Terminal.

<pre>
julia
]
add IJulia
</pre>

After that, Julia is available as a kernel.

== Interactive Widgets ==

JupyterLab supports interactive widgets that can create UI controls for interactive data visualization and manipulation within the notebooks. Example of using an interactive widget:

<pre class="{.python">from ipywidgets import IntSlider
slider = IntSlider()
display(slider)</pre>
These widgets can be sliders, dropdowns, buttons, etc., which can be connected to Python code running in the backend.

== FAQ ==

<ol><li>'''How can I use bwForCluster Helix software modules?'''
First click on the blue hexagon icon on the left-hand side of Jupyter's start page. Then load a module by clicking on the "load" button next to the corresponding module entry.</li>
<li>'''How can I navigate to my SDS@hd folder in the file browser?'''
Open a terminal and set a symbolic link to your SDS@hd folder in your home directory. For example:</li>
<pre>cd $HOME
mkdir sds-hd
cd sds-hd
ln -s /mnt/sds-hd/sd16a001 sd16a001</pre>
<li>'''I prefer VSCode over JupyterLab. Can I start a JupyterLab job and then connect with it via VSCode?'''
This is not possible. Please start the job directly on Helix instead. You can find the instructions at the [[Development/VS_Code#Connect_to_Remote_Jupyter_Kernel | VSCode page]].</li>
<li>'''My conda commands are interrupted with message 'Killed'.'''
Request more memory when starting Jupyter.</li>
<li>'''Jupyterlab doesn't let me in but asks for a password.'''
Try using more memory for the job. If this doesn't help, try using the inkognito mode of your browser as the browser cache might be the problem.</li>
</ol>

File:Bwvisu2-gui.png

2026-04-28T08:25:36Z

H Schumacher:

Registration/bwForCluster/Helix

2026-04-24T11:51:42Z

H Schumacher: bwServices functionality: added ssh key and token

__FORCETOC__

== Prerequisites ==

Steps A + B in the [[Registration/bwForCluster | registration process]] must be finished.

== Registration Steps ==

# Visit the bwForCluster Helix '''[https://bwservices.uni-heidelberg.de registration page bwServices]'''.
#: [[File:bwServices_chooseHomeOrganisation.png|right|300px|thumb|Select your home organization]]
# '''Select your home organization''' from the list on the main page and click Proceed/Fortfahren. You will be directed to the ''Identity Provider'' of your home organization.
# Enter the '''username and password''' of your home organization (usually these credentials are also used for other services like email) and click Login/Einloggen.
# When you log in to bwServices for the first time, an overview will appear, with the account information that your home institution submits to the system. Please verify that all data is valid and then click ''Continue/Weiter''.
# You will be redirected back to the bwServices main page. It shows all the statewide services you have access to. Click '''Register/Registrieren''' under the field labeled ''bwForCluster Helix''.
#: [[File:BwIDM-reg.png|center|frame|Register for Helix]]
# bwForCluster Helix uses a '''2-factor authentication''' (2FA) mechanism to increase security.
#: If you have never registered a 2FA token on bwIDM, the following error message will appear:
#: [[File:Bwidm-3-red.png|center|600px|thumb|Second factor missing.]]
#: Use this '''[https://bwservices.uni-heidelberg.de/user/twofa.xhtml link]''' or select '''My Tokens''' in the main menu.
#: To register a new token, please follow these '''[[Registration/2FA|instructions]]'''.
#: Please complete this step before continuing.
# Read the Terms of Use / Nutzungsbedingungen, place a check mark next to ''I have read and accepted the terms of use'' and click ''Register/Registrieren''.
# '''Set a service password''' for bwForCluster Helix and click ''Save/Speichern''.
#: Be sure to use a secure password that is different from any other passwords you currently use or have used on other systems.
[[File:BwIDM-passwd.png|center|600px|thumb|Set service password]]
{|style="background:#deffee; width:70%; margin: auto;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
The service password is your private password for accessing the bwForCluster Helix. Don't ever share it with anyone!
|}

== Registration Website bwServices ==
The registration website '''[https://bwservices.uni-heidelberg.de bwServices]''' offers the following functionality:
* Register for the service.
* Set or change your service password.
* Review your registration details under "Registry Info". This includes your account status (active/lost_access).
* bwServices triggers an update of your user information upon login.
* Registering an [[Registration/SSH | SSH key]]. It can be used for login to ''bwForCluster Helix''.
* Creating tokens for logins that need 2-factor authentication ([[Registration/2FA |2FA]]) like ''bwForCluster Helix'' and ''bwVisu''.
* De-register from the service.

=== Troubleshooting with the Help of bwServices ===
When encountering connection issues with a service, the bwServices website is a good starting point for troubleshooting:
* <ins>'''Issue:</ins> Expired contract''' or inactive student id:
** No access to state services like the bwHPC clusters or SDS@hd is possible.
** You can see that this is the issue when the login to bwServices via your identity provider is not possible.
* <ins>'''Issue:</ins> Recently (re)activated uni-id:''' Your id at your home organization was (re)activated a short time ago but this change didn't get through to the bwHPC cluster or SDS@hd service.
** Login to bwServices. This triggers an update of your user information. Some minutes later, the connection to the service should work.
* <ins>'''Issue:</ins> Entitlement / RV missing:''' Prerequisites like entitlement ownership or RV membership aren't met (anymore).
** You can see the concrete issue and advice on how to move forward when looking at the registry info of the service on the bwServices website.
** When the issue is solved, the account status changes from LOST_ACCESS to ACTIVE.
* <ins>'''Issue:</ins> Service Password:''' When trying to connect to the service, the service password is not accepted.
** At bwServices a [[#Setting_a_New_Service_Password | new password]] can be set for the service.
* <ins>'''Issue:</ins> OTP:''' In the case of login issues with a service that uses 2FA, you can check if you're OTP is working by going in the menu to ''Index'' -> ''My Tokens''.
** If you're token is not working, please follow the instructions for a [[Helix/Login#Troubleshooting | Lost Token]].

=== Setting a New Service Password ===

At any time, you can set a new service password via [https://bwservices.uni-heidelberg.de/ bwServices] by carrying out the following steps:
# Visit [https://bwservices.uni-heidelberg.de/ bwServices] and select your home organization.
# Authenticate yourself via your home-organizational user id / username and your home-organizational password.
# Find your service and select '''Set Service Password'''.
# Set new service password, repeat it and click the '''Save''' button.
# The page answers e.g. "password has been changed".

Registration/bwForCluster/Helix

2026-04-24T11:40:25Z

H Schumacher: Reformatting, added token issue

__FORCETOC__

== Prerequisites ==

Steps A + B in the [[Registration/bwForCluster | registration process]] must be finished.

== Registration Steps ==

# Visit the bwForCluster Helix '''[https://bwservices.uni-heidelberg.de registration page bwServices]'''.
#: [[File:bwServices_chooseHomeOrganisation.png|right|300px|thumb|Select your home organization]]
# '''Select your home organization''' from the list on the main page and click Proceed/Fortfahren. You will be directed to the ''Identity Provider'' of your home organization.
# Enter the '''username and password''' of your home organization (usually these credentials are also used for other services like email) and click Login/Einloggen.
# When you log in to bwServices for the first time, an overview will appear, with the account information that your home institution submits to the system. Please verify that all data is valid and then click ''Continue/Weiter''.
# You will be redirected back to the bwServices main page. It shows all the statewide services you have access to. Click '''Register/Registrieren''' under the field labeled ''bwForCluster Helix''.
#: [[File:BwIDM-reg.png|center|frame|Register for Helix]]
# bwForCluster Helix uses a '''2-factor authentication''' (2FA) mechanism to increase security.
#: If you have never registered a 2FA token on bwIDM, the following error message will appear:
#: [[File:Bwidm-3-red.png|center|600px|thumb|Second factor missing.]]
#: Use this '''[https://bwservices.uni-heidelberg.de/user/twofa.xhtml link]''' or select '''My Tokens''' in the main menu.
#: To register a new token, please follow these '''[[Registration/2FA|instructions]]'''.
#: Please complete this step before continuing.
# Read the Terms of Use / Nutzungsbedingungen, place a check mark next to ''I have read and accepted the terms of use'' and click ''Register/Registrieren''.
# '''Set a service password''' for bwForCluster Helix and click ''Save/Speichern''.
#: Be sure to use a secure password that is different from any other passwords you currently use or have used on other systems.
[[File:BwIDM-passwd.png|center|600px|thumb|Set service password]]
{|style="background:#deffee; width:70%; margin: auto;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
The service password is your private password for accessing the bwForCluster Helix. Don't ever share it with anyone!
|}

== Registration Website bwServices ==
The registration website '''[https://bwservices.uni-heidelberg.de bwServices]''' offers the following functionality:
* Register for the service
* Set or change your service password
* Review your registration details under "Registry Info". This includes your account status (active/lost_access).
* Triggers an update of your user information upon login.
* De-register from the service

=== Troubleshooting with the Help of bwServices ===
When encountering connection issues with a service, the bwServices website is a good starting point for troubleshooting:
* <ins>'''Issue:</ins> Expired contract''' or inactive student id:
** No access to state services like the bwHPC clusters or SDS@hd is possible.
** You can see that this is the issue when the login to bwServices via your identity provider is not possible.
* <ins>'''Issue:</ins> Recently (re)activated uni-id:''' Your id at your home organization was (re)activated a short time ago but this change didn't get through to the bwHPC cluster or SDS@hd service.
** Login to bwServices. This triggers an update of your user information. Some minutes later, the connection to the service should work.
* <ins>'''Issue:</ins> Entitlement / RV missing:''' Prerequisites like entitlement ownership or RV membership aren't met (anymore).
** You can see the concrete issue and advice on how to move forward when looking at the registry info of the service on the bwServices website.
** When the issue is solved, the account status changes from LOST_ACCESS to ACTIVE.
* <ins>'''Issue:</ins> Service Password:''' When trying to connect to the service, the service password is not accepted.
** At bwServices a [[#Setting_a_New_Service_Password | new password]] can be set for the service.
* <ins>'''Issue:</ins> OTP:''' In the case of login issues with a service that uses 2FA, you can check if you're OTP is working by going in the menu to ''Index'' -> ''My Tokens''.
** If you're token is not working, please follow the instructions for a [[Helix/Login#Troubleshooting | Lost Token]].

=== Setting a New Service Password ===

At any time, you can set a new service password via [https://bwservices.uni-heidelberg.de/ bwServices] by carrying out the following steps:
# Visit [https://bwservices.uni-heidelberg.de/ bwServices] and select your home organization.
# Authenticate yourself via your home-organizational user id / username and your home-organizational password.
# Find your service and select '''Set Service Password'''.
# Set new service password, repeat it and click the '''Save''' button.
# The page answers e.g. "password has been changed".

SDS@hd/Registration

2026-04-23T16:18:50Z

H Schumacher: fixed link, reformatting, added link from step A to B

== Registration Steps ==

The registration consists of two steps:

* '''Step A:''' Membership in Storage Project (Speichervorhaben/'''SV'''). There are two options:
*: '''a)''' →[[SDS@hd/Registration#a)_Join_existing_SV | Join existing SV]]
*: '''b)''' →[[SDS@hd/Registration#b)_Apply_for_new_SV | Apply for new SV]] (only possible for [https://www.bwidm.de/hochschulen.php bwIDM members])
* '''Step B:''' →[[SDS@hd/Registration#Step_B:_Registration_for_SDS@hd_Service |Registration for SDS@hd Service]]

After finishing the registration, the '''next steps''' are:
* Check which management options there are for your respective role in the SV → [[SDS@hd/SV_Management |SV Management]]
* See how to access and use your storage space → [[SDS@hd/Access | Access]]

[[File:SDS@hd_Registration.png|center|x500px|upright|SDS@hd Registration Process]]

=== Step A: SV Membership ===

You can either join an existing SV or start a new one.

==== a) Join existing SV ====
To join an existing SV as a coworker:
# Get the following information from the SV responsible:
#:* SV acronym
#:* SV password
# Fill in and submit the [https://sds-hd.urz.uni-heidelberg.de/management/shib/sds_collaboration.php SV collaboration form] after logging in to the SDS@hd Managementtool.
#: You are now a member of the SV. The SV owner (and SV managers) will be notified automatically.
→ [[SDS@hd/Registration#Step_B:_Registration_for_SDS@hd_Service | Go on with Step B]] if you are using SDS@hd for the first time.

==== b) Apply for new SV ====

This is typically done only by the leader of a scientific work group or the senior scientist of a research group/collaboration.
Any amount of co-workers can join your SV without having to register another project. You just need to provide them with the SV acronym and SV password. You'll receive this information via e-mail as soon as the SV application was successful.

There are two steps:
# '''Get the permission / entitlement:'''. Your institution has to grant you the permission to start an SDS@hd storage project ("SDS@hd SV entitlement"). Please follow the procedure specific to your institution:
#: → see [[SDS@hd/Registration/Entitlement | Entitlement]].
# '''Apply for a new SV:'''
#: → Fill in the form at the [https://sds-hd.urz.uni-heidelberg.de/management SDS@hd Managementtool].
→ [[SDS@hd/Registration#Step_B:_Registration_for_SDS@hd_Service | Go on with Step B]] if you are using SDS@hd for the first time.

If you register your own SV, you will be:
* ...responsibe for providing new members with the necessary information to join the SV
*: → see [[SDS@hd/SV_Management#SV_Responsible_(SVV) | SV Responsible (SVV)]]
* ...held accountable for the co-workers in the SV
* ...asked to provide information for the two reports required by the DFG for their funding of SDS@hd
* ...likely asked for a contribution to a future DFG grant proposal for an extension of the storage system in your area of research ("wissenschaftliches Beiblatt")

=== Step B: Registration for SDS@hd Service ===
[[File:bwServices_chooseHomeOrganisation.png|right|300px|thumb|Select your home organization]]

After step A you have to register your personal account on the storage system and set a service password.

* Visit the registration website '''bwServices''': [https://bwservices.uni-heidelberg.de/ https://bwservices.uni-heidelberg.de]
*# Select your home organization from the list and click ''Proceed''
*#: You will be directed to the ''Identity Provider'' of your home organisation
*# Enter your home-organisational user ID / username and your home-organisational password and click ''Login'' button
*#: You will be redirected back to bwServices
* '''Register''' for the SDS@hd service:
*# Under ''The following services are available'' select the service ''SDS@hd - Scientific Data Storage''.
*# Click ''Register''
* Set your personal '''service password''' for SDS@hd. This is used for accessing SVs.
* The registration is finished.
*: → [[#sds_registration_nextSteps|next steps]]

=== Change Service Password at bwServices ===

→ '''[[Registration/bwForCluster/Helix#Setting_a_New_Service_Password | Setting a New Service Password]]'''

SDS@hd/Registration

2026-04-23T12:16:49Z

H Schumacher: fixed link

== Registration Steps ==

The registration consists of two steps:

* '''Step A:''' Membership in Storage Project (Speichervorhaben/'''SV'''). There are two options:
*: '''a)''' →[[SDS@hd/Registration#a)_Join_existing_SV | Join existing SV]]
*: '''b)''' →[[SDS@hd/Registration#b)_Apply_for_new_SV | Apply for new SV]] (only possible for [https://www.bwidm.de/hochschulen.php bwIDM members])
* '''Step B:''' →[[SDS@hd/Registration#Step_B:_Registration_for_SDS@hd_Service |Registration for SDS@hd Service]]

After finishing the registration, the next steps are:
* Check which management options there are for your respective role in the SV → [[SDS@hd/SV_Management |SV Management]]
* See how to access and use your storage space → [[SDS@hd/Access | Access]]

[[File:SDS@hd_Registration.png|center|x500px|upright|SDS@hd Registration Process]]

=== Step A: SV Membership ===

You can either join an existing SV or start a new one.

==== a) Join existing SV ====
To join an existing SV as a coworker:
# Your advisor (the SV responsible) will provide you with the following data on the SV:
#:* acronym
#:* password
# Login at the SDS@hd Managementtool and fill in the [https://sds-hd.urz.uni-heidelberg.de/management/shib/sds_collaboration.php SV collaboration form]
#: Provide the SV acronym and SV password.
# After submitting the request, you will be assigned to the SV as a member and receive an email with the next steps. The SV owner (and SV managers) will be notified automatically.

==== b) Apply for new SV ====

This is typically done only by the leader of a scientific work group or the senior scientist of a research group/collaboration.
Any amount of co-workers can join your SV without having to register another project. You just need to provide them with the SV acronym and SV password. You'll receive this information via e-mail as soon as the SV application was successful.

There are two steps:
# '''Get the permission / entitlement:'''. Your institution has to grant you the permission to start an SDS@hd storage project ("SDS@hd SV entitlement"). Please follow the procedure specific to your institution:
#: → see [[SDS@hd/Registration/Entitlement | Entitlement]].
# '''Apply for a new SV:'''
#: → Fill in the form at the [https://sds-hd.urz.uni-heidelberg.de/management SDS@hd Managementtool].

If you register your own SV, you will be:
* ...responsibe for providing new members with the necessary information to join the SV
*: → see [[#SV_Responsible:_Manage_the_SV_and_its_Members | SV Responsible: Manage the SV and its Members]]
* ...held accountable for the co-workers in the SV
* ...asked to provide information for the two reports required by the DFG for their funding of SDS@hd
* ...likely asked for a contribution to a future DFG grant proposal for an extension of the storage system in your area of research ("wissenschaftliches Beiblatt")

=== Step B: Registration for SDS@hd Service ===
[[File:bwServices_chooseHomeOrganisation.png|right|300px|thumb|Select your home organization]]

After step 1 you have to register your personal account on the storage system and set a service password.
Please visit:
* [https://bwservices.uni-heidelberg.de/ https://bwservices.uni-heidelberg.de]
*# Select your home organization from the list and click '''Proceed'''
*# You will be directed to the ''Identity Provider'' of your home organisation
*# Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button
*# You will be redirected back to the registration website [https://bwservices.uni-heidelberg.de/ https://bwservices.uni-heidelberg.de/]
*# <div>Select unter '''The following services are available''' the service '''SDS@hd - Scientific Data Storage'''
*# Click '''Register'''
*# Finally, set a service password for authentication on SDS@hd

=== Change Service Password at bwServices ===

→ '''[[Registration/bwForCluster/Helix#Setting_a_New_Service_Password | Setting a New Service Password]]'''

Registration/SSH

2026-04-23T09:18:17Z

H Schumacher: Added 'generate ssh key' step, included 'bind key' as last step, removed spacing in info box, downsized two images

= SSH Key Authentication for HPC Clusters =

'''SSH Keys''' allow you to log into a system without entering a password. Instead of proving your identity with something you know (a password), you prove it with something you have (a cryptographic key).

{| class="wikitable" style="text-align:center;"
|+ SSH Key Management Methods by Cluster
|-
! Cluster
! Management Method
! Details
|-
| bwUniCluster 3.0
| style="background-color:#90EE90;" | bwIDM Portal
| Centralized key management, 180-day validity
|-
| bwForCluster BinAC 2
| style="background-color:#FFE4B5;" | ~/.ssh/authorized_keys
| Self-managed, use ssh-copy-id
|-
| bwForCluster Helix
| style="background-color:#90EE90;" | bwServices Portal
| Centralized key management, 180-day validity
|-
| bwForCluster JUSTUS 2
| style="background-color:#FFE4B5;" | ~/.ssh/authorized_keys
| Self-managed, use ssh-copy-id
|-
| bwForCluster NEMO 2
| style="background-color:#90EE90;" | bwIDM Portal
| Centralized key management, 180-day validity
|}

'''Choose your cluster below:'''
* [[#SSH_Keys_on_BinAC_2_and_JUSTUS_2|BinAC 2 and JUSTUS 2]] - Self-managed keys
* [[#SSH_Keys_via_bwIDM.2FbwServices|bwUniCluster 3.0, Helix, and NEMO2]] - Centralized management

= SSH Keys on BinAC 2 and JUSTUS 2 =

On '''bwForCluster BinAC 2''' and '''bwForCluster JUSTUS 2''', you manage SSH keys yourself using the standard <code>~/.ssh/authorized_keys</code> file.

== Quick Setup with ssh-copy-id ==

The easiest method to add your SSH key:

'''Step 1: Generate an SSH key''' (if you don't have one):
<pre>
ssh-keygen -t ed25519 -C "your_email@example.com"
</pre>

Press Enter to accept the default location, then set a strong passphrase.

'''Step 2: Copy your key to the cluster:'''
<pre>
# For BinAC2:
ssh-copy-id username@login.binac2.uni-tuebingen.de

# For JUSTUS2:
ssh-copy-id username@justus2.uni-ulm.de
</pre>

Enter your service password and OTP when prompted. Your public key will be automatically added to <code>~/.ssh/authorized_keys</code>.

'''Step 3: Test your connection:'''
<pre>
# For BinAC2:
ssh username@login.binac2.uni-tuebingen.de

# For JUSTUS2:
ssh username@justus2.uni-ulm.de
</pre>

You should now be able to log in using your SSH key and OTP.

== Manual Setup (Alternative) ==

If <code>ssh-copy-id</code> is not available on your system:

'''Step 1: Display your public key:'''
<pre>
cat ~/.ssh/id_ed25519.pub
</pre>

Copy the entire output.

'''Step 2: Log into the cluster''' using your service password and OTP

'''Step 3: Add the key to authorized_keys:'''
<pre>
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "paste-your-public-key-here" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
</pre>

Replace <code>paste-your-public-key-here</code> with your actual public key.

= SSH Keys via bwIDM/bwServices =

On '''bwUniCluster 3.0''', '''bwForCluster Helix''', and '''bwForCluster NEMO 2''', SSH keys are managed centrally through the registration service.

== Why Centralized Management? ==

Centralized SSH key management provides:

* Security enforcement: Keys must use strong algorithms and have 180-day validity
* Centralized control: Review and revoke all keys from one location
* Two key types: Interactive keys (manual logins) and Command keys (automated workflows)

'''Note:''' Self-managed <code>~/.ssh/authorized_keys</code> files are ignored on these clusters.

== Supported Key Types ==

=== Standard SSH Keys ===

* ED25519: 256 bits (recommended)
* RSA: 2048 bits or more
* ECDSA: 521 bits

'''Important:''' Always protect your private keys with a strong passphrase.

=== FIDO2 Hardware Keys ===

'''ED25519-SK''' keys use hardware security keys (like Yubikey) for authentication:

* Always valid - no 2-factor unlock required
* Hardware-protected - private key never leaves the device
* Physical presence required - must touch key to authenticate

'''Note:''' ECDSA-SK keys are not supported. Use ED25519-SK only.

{| class="wikitable" style="text-align:center;"
|+ FIDO2 Key Support by Cluster
|-
! Cluster
! ED25519-SK Support
|-
| bwUniCluster 3.0
| style="background-color:#90EE90;" | ✓ Supported
|-
| bwForCluster Helix
| style="background-color:#FFB6C1;" | ✗ Not supported
|-
| bwForCluster NEMO 2
| style="background-color:#90EE90;" | ✓ Supported
|}

'''Get started:''' See [[Registration/SSH/SSH-FIDO2-Quick-Start|SSH with FIDO2 - Quick Start Guide]]

== Step 1: Add Your SSH Key to the Portal ==

<div style="border: 3px solid #ffc107; background-color: #fff3cd; margin: 10px 0px 10px 0px;">
'''Important:'''
* Keys are valid for '''180 days''' and automatically revoked after expiration.
* Upload only your '''public key''' file (ending in <code>.pub</code>, e.g., <code>~/.ssh/id_ed25519.pub</code>).
</div>

'''1. Generate an SSH key''' (if you don't have one):
* Run in your local terminal: <pre> ssh-keygen -t ed25519 -C "your_email@example.com"
</pre>
* Press Enter to accept the default location, then set a strong passphrase.
* You now have a private key (~/.ssh/id_ed25519) and a public key (~/.ssh/id_ed25519.pub).

'''2. Navigate to SSH key management:'''
* [https://login.bwidm.de/user/ssh-keys.xhtml bwUniCluster 3.0 / NEMO 2] (bwIDM)
* [https://bwservices.uni-heidelberg.de/user/ssh-keys.xhtml bwForCluster Helix] (bwServices)

[[File:BwIDM-twofa.png|center|600px|thumb|My SSH Pubkeys page]]

'''3. Click "Add SSH Key" / "SSH Key Hochladen"'''

[[File:Bwunicluster 2.0 access ssh keys empty.png|center|400px|thumb|Add SSH Key button]]

'''4. Enter key details:'''
* '''Name:''' Descriptive identifier (e.g., "laptop-work", "desktop-home")
* '''SSH Key:''' Paste complete contents of your <code>.pub</code> file
* Click '''Add''' / '''Hinzufügen'''

[[File:Ssh-key.png|center|400px|thumb|Add SSH key dialog]]

'''5. Your key appears in the list:'''

[[File:Ssh-success.png|center|600px|thumb|SSH key successfully added]]

'''6. Bind your key to a cluster''' as either an [[#Step_2A:_Register_Interactive_Key|Interactive Key]] or [[#Step_2B:_Register_Command_Key|Command Key]].

== Step 2A: Register Interactive Key ==

'''Interactive Keys''' are for manual SSH logins.

=== Understanding Key Validity ===

'''Regular SSH Keys''' (RSA, ECDSA, ED25519):
* Require 2-factor authentication unlock
* Valid for limited hours after entering OTP + service password
* Must re-authenticate when validity expires

{| class="wikitable" style="text-align:center;"
|+ Validity Periods for Regular SSH Keys
|-
! Cluster
! Valid Duration
|-
| bwUniCluster 3.0
| 8 hours
|-
| bwForCluster Helix
| 12 hours
|-
| bwForCluster NEMO 2
| 12 hours
|}

'''FIDO2 Hardware Keys''' (ED25519-SK):
* Always valid - no 2-factor unlock needed
* Authentication via physical key touch only
* Only on bwUniCluster 3.0 and NEMO 2 (not Helix)
* See [[Registration/SSH/SSH-FIDO2-Quick-Start|SSH with FIDO2 - Quick Start Guide]]

=== Registration Steps ===

'''1. Add your public key''' following [[#Step_1:_Add_Your_SSH_Key_to_the_Portal|Step 1]] above

'''2. Navigate to "Registered Services" / "Registrierte Dienste"'''

Click '''Set SSH Key''' / '''SSH Key setzen''' for your cluster

[[File:BwIDM-registered.png|center|600px|thumb|Select cluster]]

'''3. Find your key and click "Add" / "Hinzufügen"'''

[[File:Ssh-service-int.png|center|800px|thumb|Add SSH key to service]]

'''4. Select "Interactive" and confirm'''

* Usage type: '''Interactive'''
* Comment: Optional description
* Click '''Add''' / '''Hinzufügen'''

[[File:Ssh-int.png|center|600px|thumb|Set as Interactive key]]

'''5. Done!''' Your key is active for interactive logins

[[File:Ssh-service.png|center|800px|thumb|SSH key registered]]

== Step 2B: Register Command Key ==

'''Command Keys''' enable automated workflows (e.g., backups, data transfers) without manual login.

=== Security Requirements ===

<div style="border: 3px solid #dc3545; padding: 15px; background-color: #f8d7da; margin: 10px 0;">
Command keys are '''always valid''' (no 2FA required), making them security-sensitive.

'''Mandatory restrictions:'''
* Single command: Specify exact command with full path
* IP restriction: Limit to specific IP address(es) or subnet
* Admin approval: Keys require review before activation
* Short validity: Maximum 30 days
</div>

'''Common use case:''' [[Registration/SSH/rrsync|rrsync for data transfers]]

=== Registration Steps ===

'''1. Add your public key''' following [[#Step_1:_Add_Your_SSH_Key_to_the_Portal|Step 1]] above

'''2. Navigate to "Registered Services"'''

Click '''Set SSH Key''' for your cluster

[[File:BwIDM-registered.png|center|600px|thumb|Select cluster]]

'''3. Find your key and click "Add" / "Hinzufügen"'''

[[File:Ssh-service-com.png|center|800px|thumb|Add SSH key to service]]

'''4. Configure command restrictions:'''

* '''Usage type:''' Select '''Command'''
* '''Command:''' Full path with parameters (see example below)
* '''From:''' IP address or CIDR notation (e.g., <code>192.168.1.0/24</code>)
* '''Comment:''' Explain purpose (speeds up approval)
* Click '''Add''' / '''Hinzufügen'''

{| class="wikitable"
! Example: rrsync for data transfer
|-
| <pre>/usr/local/bin/rrsync -ro /home/aa/aa_bb/aa_abc1/</pre>
'''Note:''' Verify exact path on your cluster (may be <code>/usr/bin/rrsync</code>)
|}

[[File:Ssh-com.png|center|600px|thumb|Configure command key]]

'''5. Wait for approval'''

[[File:Ssh-service.png|center|800px|thumb|Key pending approval]]

== Revoking SSH Keys ==

Revoke keys that are no longer needed or potentially compromised.

<div style="border: 3px solid #6c757d; padding: 15px; background-color: #e2e3e5; margin: 10px 0;">
'''Note:''' Revoked keys are immediately disabled and cannot be reused.
</div>

'''1. Navigate to SSH key management:'''
* [https://login.bwidm.de/user/ssh-keys.xhtml bwUniCluster 3.0 / NEMO 2] (bwIDM)
* [https://bwservices.uni-heidelberg.de/user/ssh-keys.xhtml bwForCluster Helix] (bwServices)

[[File:BwIDM-twofa.png|center|600px|thumb|My SSH Pubkeys page]]

'''2. Click "REVOKE" / "ZURÜCKZIEHEN"''' next to the key you want to disable

'''2.''' Click '''REVOKE''' / '''ZURÜCKZIEHEN''' next to the key you want to disable

[[File:Ssh-success.png|center|800px|thumb|Revoke SSH key]]

Helix/Login

2026-04-20T15:38:41Z

H Schumacher: Added examples for OTP

= Prerequisites =
* The three [[registration/bwForCluster|bwForCluster registration]] steps are completed and still valid.
* Your IP address is within the [https://www.belwue.de BelWü] network which connects universities and other scientific institutions in Baden-Württemberg. If your computer is in your university network (e.g. at your office), you should be able to connect to bwForCluster Helix without restrictions. If you are outside the BelWü network (e.g. at home), a VPN (virtual private network) connection to your University network must be established first. Please consult the VPN documentation of your university.
* As credentials you need your Helix [[Helix/Login#Username|username]], [[Registration/Password|service password]] and [[Registration/2FA|2FA token]].
* If you encounter any issues, please have a look at the [[Helix/Login#Troubleshooting|Troubleshooting]] section.

= Login to bwForCluster Helix =

Login to bwForCluster Helix is only possible with a Secure Shell (SSH) client for which you must know your bwForCluster Helix credentials and the hostname of the login nodes.
For more general information on SSH clients, visit the [[Registration/Login/Client|SSH clients Guide]].

== Username ==

Your username on bwForCluster Helix consists of a prefix and your local username.
For prefixes please refer to the [[Registration/Login/Username|Username Guide]].

Example: If your local username at your University is <code>ab123</code> and you are a user from Heidelberg University, your username on the cluster is: <code>hd_ab123</code>.

== Hostnames ==

bwForCluster Helix has two login nodes. In general, you should stay with automatic node selection, so that the connections are evenly distributed between the login nodes. To do this, we use so-called DNS round-robin scheduling which takes care of the node balancing. This means, if you open multiple SSH sessions, these sessions may run on different login nodes. Therefore, processes started in one session might not be visible in other sessions.

{| class="wikitable"
! Hostname !! Destination
|-
| '''helix.bwservices.uni-heidelberg.de''' || one of the two login nodes
|}

== Login with SSH command (Linux, Mac, Windows) ==

Most Unix and Unix-like operating systems such as Linux and Mac OS as well as newer versions of MS Windows (10 and 11), have a built-in SSH client provided by the OpenSSH project.
If you want to use the full set of Linux commands on Windows (10 or 11), you can also install the [https://docs.microsoft.com/en-us/windows/wsl/install Windows Subsystem for Linux] (WSL).

For login use one of the following ssh commands in a commandline (i.e. cmd, terminal, PowerShell):

ssh <username>@helix.bwservices.uni-heidelberg.de
ssh -l <username> helix.bwservices.uni-heidelberg.de

To run graphical applications on the cluster, you need to enable X11 forwarding with the <code>-X</code> flag:

ssh -X -l <username> helix.bwservices.uni-heidelberg.de

To make the login process faster, tell the command that no ssh key authentication is used by adding an additional parameter:

ssh -o PubkeyAuthentication=no <username>@helix.bwservices.uni-heidelberg.de

== Login with graphical SSH client (Windows) ==

For Windows we suggest using [[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]] for login and file transfer.

Start MobaXterm and fill in the following fields:
<pre>
Remote name : helix.bwservices.uni-heidelberg.de
Specify user name : <username>
Port : 22
</pre>

After that click on 'ok'. Then a terminal will open where you can enter your credentials.

== Login with SSH key ==

To set up a SSH key for your workflow, read [[Registration/SSH|Registering SSH Keys with your Cluster]].
To login by using the ssh key, use the following command:

ssh -o IdentitiesOnly=yes -i <path_to_private_key> <username>@helix.bwservices.uni-heidelberg.de

== Login Example ==

To login to bwForCluster Helix, proceed as follows:
# Login with SSH command or MoabXterm as shown above.
# The system will ask for a one-time password <code>Your OTP</code> (= Smartphone Token / TAN list element/ ...) . Please enter your OTP and confirm it with Enter/Return. The OTP is not displayed when typing. If you do not have a second factor yet, please create one (see [[Registration/2FA]]).
# The system will ask you for your service password <code>Password:</code>. Please enter it and confirm it with Enter/Return. The password is not displayed when typing. If you do not have a service password yet or have forgotten it, please create one (see [[Registration/Password]]).
# You will be greeted by the cluster with the message of the day, followed by a shell.
<pre>
$ ssh -l hd_ab123 helix.bwservices.uni-heidelberg.de
Your OTP:
Password:

********************************************************************************
* *
* Baden-Wuerttemberg HPC Cluster for *
* Structural and Systems Biology, Medical Science, *
* Soft Matter, Computational Humanities, and *
* Mathematics and Computer Science *
* *
* bwForCluster *
* _ _ _ _ *
* | | | | | | (_) *
* | |__| | ___ | | _ __ __ *
* | __ | / _ \ | | | | \ \/ / *
* | | | | | __/ | | | | > < *
* |_| |_| \___| |_| |_| /_/\_\ *
* *
* *
* (xCAT 2.16.4 / RHEL 8.8 / GPFS 5.1.8) *
* *
* *
* bwHPC Wiki : https://wiki.bwhpc.de/e/Helix *
* Ticket System : https://www.bwhpc.de/supportportal *
* Job Monitoring : https://helix-monitoring.bwservices.uni-heidelberg.de *
* E-learning : https://training.bwhpc.de/ *
* -> 'Introduction to bwForCluster Helix' *
* *
********************************************************************************

Last login: ...
[hd_ab123@login ~]$
</pre>

= Allowed Activities on Login Nodes =

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
To guarantee usability for all users you must not run your compute jobs on the login nodes.
Compute jobs must be submitted as batch jobs.
Any compute job running on the login nodes will be terminated without notice.
Long-running compilation or long-running pre- or post-processing tasks must also be submitted as batch jobs.
|}

The login nodes are the access points to the compute system, your <code>$HOME</code> directory and your workspaces.
These nodes are shared with all users. Hence, your activities on the login nodes are primarily limited to setting up your batch jobs.
Your activities may also be:
* quick compilation of program code or
* quick pre- and post-processing of results from batch jobs.

We advise to use [[Helix/Slurm#Interactive_Jobs|interactive batch jobs]] for compute and memory intensive compilation and pre- and post-processing tasks.

= Related Information =

* If you want to register a new token for the two factor authentication (2FA), consult the [[Registration/2FA|2FA Guide]].
* If you want to de-register, consult the [[Registration/Deregistration|De-registration Guide]].
* If you need an SSH key for your workflow, read [[Registration/SSH|Registering SSH Keys with your Cluster]].
* Configuring your shell: [[.bashrc Do's and Don'ts]]

= Troubleshooting =

'''Login Problems'''

If you encounter login problems, please double check the prerequisites. If those are valid, it is fastest to first check the following points before contacting the support at the [https://www.bwhpc.de/supportportal.php|bwhpc Suppport Portal].
If there are known issues with the cluster, we'll inform you via e-mail or in the message of the day at the cluster.

* To reset your service password, consult the [[Registration/Password|Password Guide]].
* To check if your 2FA token is working go to [https://bwservices.uni-heidelberg.de/| bwServices] -> "Index" -> "My Tokens".
* Make sure to not use LAN and WLAN at the same time to prevent connection problems.
* Make sure to be a fully active member of your institution.

'''Lost Token'''

To reset your token you have to visit the IT-Service to validate your identity.
Mention that you want a reset of your Helix 2FA token. It is important to emphasize that the token reset is for bwForCluster Helix to avoid confusion with the token that is specific to members/students of the university of Heidelberg.

You can visit the IT service point of the URZ in person at INF 330, 69120 Heidelberg. Or you can use the video call service. It is recommended to use the Chrome browser for doing the video call. Please have on official identification ready in any case (Führerschein / Personalausweis) .
For further information please see here: https://it-service.uni-heidelberg.de.

To be on the safe side, it is advised to create a [[Registration/2FA#Backup_TAN_List|backup TAN list]] additionally to the smartphone token.

Main Page

2026-04-20T15:14:44Z

H Schumacher: Added icons and descriptions, moved data transfer link

'''Welcome to the bwHPC Wiki.'''

bwHPC represents services and resources in the State of '''B'''aden-'''W'''ürttemberg, Germany, for High Performance Computing ('''HPC'''), Data Intensive Computing ('''DIC''') and Large Scale Scientific Data Management ('''LS2DM''').

The main bwHPC web page is at '''[https://www.bwhpc.de/ https://www.bwhpc.de/]'''.

Many topics depend on the cluster system you use.
First choose the cluster you use, then select the correct topic.



{{Template:Status|ok|'''Status:''' Ok}}





{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Courses / eLearning
|-
|
* [https://training.bwhpc.de/ eLearning and Online Courses]
* [https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC Introduction to Linux in HPC (external resource)]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | [[File:Clusternodes.svg|x20px]] Need Access to a Cluster?
|-
|
* [[When to use an HPC Cluster]]
* [[Running Calculations]]
* [[Registration]]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | [[File:Clusternodes.svg|x20px]] HPC System Specific Documentation
|-
|
bwHPC encompasses several HPC compute clusters at different universities in Baden-Württemberg. Each cluster is dedicated to [https://www.bwhpc.de/bwhpc-domains.php specific research domains].

Documentation differs between compute clusters, please see cluster specific overview pages:
{|
|-
| style="padding:5px; width:30%" | [[BwUniCluster3.0|bwUniCluster 3.0]]
| style="padding-left:20px;" | General Purpose, Teaching
|-
| style="padding:5px; width:30%" | [[BinAC2|bwForCluster BinAC 2]]
| style="padding-left:20px;" | Bioinformatics, Astrophysics, Geosciences, Pharmacy, and Medical Informatics
|-
| style="padding:5px; width:30%" | [[Helix|bwForCluster Helix]]
| style="padding-left:20px;" | Structural and Systems Biology, Medical Science, Soft Matter, Computational Humanities, and Mathematics and Computer Science
|-
| style="padding:5px; width:30%" | [[:JUSTUS2| bwForCluster JUSTUS 2]]
| style="padding-left:20px;" | Theoretical Chemistry, Condensed Matter Physics, and Quantum Sciences
|-
| style="padding:5px; width:30%" | [[NEMO2|bwForCluster NEMO 2]]
| style="padding-left:20px;" | Neurosciences, Particle Physics, Materials Science, and Microsystems Engineering
|}
|-
|bwHPC Clusters: [https://www.bwhpc.de/cluster.php operational status]
Further Compute Clusters in Baden-Württemberg (separate access policies):
* [[DACHS | Datenanalyse Cluster der Hochschulen (DACHS)]]
* bwHPC tier 1: [https://kb.hlrs.de/platforms/index.php/Hunter_(HPE) Hunter] ([https://www.hlrs.de/apply-for-computing-time getting access])
* bwHPC tier 2: [https://www.nhr.kit.edu/userdocs/horeka HoreKa] ([https://www.nhr.kit.edu/userdocs/horeka/projects/ getting access])
|}

{| style="background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | [[File:Clusternodes.svg|x20px]] Documentation valid for all Clusters
|-
|
* [[Environment Modules| Software Environment Modules]]: usage of preexisting software
* [https://www.bwhpc.de/software.html List of Software] on all clusters
* [[Data Transfer|Data Transfer]]: working with files [[File:Notebook.svg|x15px]] <---> [[File:Clusternodes.svg|x15px]]
* [[Development| Development]]: software, programming languages, parallel pogramming
* [[Energy Efficient Cluster Usage | (Energy) Efficient Cluster Usage]] for low waiting times and fast running jobs
* [[HPC Glossary]]

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | [[File:Storage_small.svg|x15px]] Scientific Data Storage
|-
|
User guides of available scientific data storage services:
* [[SDS@hd]]: Everyone can join existing storage projects, entitlement needed for creating your own, [https://sds-hd.urz.uni-heidelberg.de/management/shib/sds_costs.php cost sheet]
* [https://uni-tuebingen.de/einrichtungen/zentrum-fuer-datenverarbeitung/projekte/laufende-projekte/bwsfs bwSFS]
Associated, but local scientific storage services are:
* [https://wiki.scc.kit.edu/lsdf/index.php/Category:LSDF_Online_Storage LSDF Online Storage] (only for KIT and KIT partners)
|}

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | [[File:Storage_small.svg|x15px]] Scientific Data Archiving
|-
|
For user guides of the scientific data archiving services:
* [https://www.rda.kit.edu/english bwDataArchive]: Open to all scientists at KIT and institutions in Baden-Württemberg that have signed a service agreement with KIT
Associated, but local archiving services for scientific data are:
* [https://www.urz.uni-heidelberg.de/de/service-katalog/speicher/heiarchive heiARCHIVE] (only for members of Heidelberg University)
For instructions on how to move the data, depending on the offered connections, can be found under [[Data Transfer | Data Transfer]].
|}

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | Research Data Management
|-
|
* [https://www.forschungsdaten.org/index.php/FDM-Kontakte#Deutschland Research Data Management (RDM)] contact persons
* [https://www.forschungsdaten.info Portal for Research Data Management] (Forschungsdaten.info)
|}
{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Support
|-
|
* [[BwSupportPortal|Submit a Ticket in our Support Portal]]
Support is provided by the [https://www.bwhpc.de/teams.php bwHPC Competence Centers]:
|}
{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Acknowledgement
|-
|
* Please [[Acknowledgement|acknowledge]] our resources in your publications.
|}

Status

2026-04-20T15:06:41Z

H Schumacher: Removed old entries

= bwHPC Cluster and Service Status Page =

== Current Status ==

{{Template:Status|ok|'''Status:''' Normal operation of all systems.}}



== Relevant Old Messages ==
None

Main Page

2026-04-13T08:28:36Z

H Schumacher: fixed typo

'''Welcome to the bwHPC Wiki.'''

bwHPC represents services and resources in the State of '''B'''aden-'''W'''ürttemberg, Germany, for High Performance Computing ('''HPC'''), Data Intensive Computing ('''DIC''') and Large Scale Scientific Data Management ('''LS2DM''').

The main bwHPC web page is at '''[https://www.bwhpc.de/ https://www.bwhpc.de/]'''.

Many topics depend on the cluster system you use.
First choose the cluster you use, then select the correct topic.



{{Template:Status|ok|'''Status:''' Ok}}





{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Courses / eLearning
|-
|
* [https://training.bwhpc.de/ eLearning and Online Courses]
* [https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC Introduction to Linux in HPC (external resource)]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | Need Access to a Cluster?
|-
|
* [[When to use an HPC Cluster]]
* [[Running Calculations]]
* [[Registration]]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | HPC System Specific Documentation
|-
|
bwHPC encompasses several HPC compute clusters at different universities in Baden-Württemberg. Each cluster is dedicated to [https://www.bwhpc.de/bwhpc-domains.php specific research domains].

Documentation differs between compute clusters, please see cluster specific overview pages:
{|
|-
| style="padding:5px; width:30%" | [[BwUniCluster3.0|bwUniCluster 3.0]]
| style="padding-left:20px;" | General Purpose, Teaching
|-
| style="padding:5px; width:30%" | [[BinAC2|bwForCluster BinAC 2]]
| style="padding-left:20px;" | Bioinformatics, Astrophysics, Geosciences, Pharmacy, and Medical Informatics
|-
| style="padding:5px; width:30%" | [[Helix|bwForCluster Helix]]
| style="padding-left:20px;" | Structural and Systems Biology, Medical Science, Soft Matter, Computational Humanities, and Mathematics and Computer Science
|-
| style="padding:5px; width:30%" | [[:JUSTUS2| bwForCluster JUSTUS 2]]
| style="padding-left:20px;" | Theoretical Chemistry, Condensed Matter Physics, and Quantum Sciences
|-
| style="padding:5px; width:30%" | [[NEMO2|bwForCluster NEMO 2]]
| style="padding-left:20px;" | Neurosciences, Particle Physics, Materials Science, and Microsystems Engineering
|}
|-
|bwHPC Clusters: [https://www.bwhpc.de/cluster.php operational status]
Further Compute Clusters in Baden-Württemberg (separate access policies):
* Datenanalyse Cluster der Hochschulen (DACHS): [[DACHS | Datenanalyse Cluster der Hochschulen (DACHS)]]
* bwHPC tier 1: [https://kb.hlrs.de/platforms/index.php/Hunter_(HPE) Hunter] ([https://www.hlrs.de/apply-for-computing-time getting access])
* bwHPC tier 2: [https://www.nhr.kit.edu/userdocs/horeka HoreKa] ([https://www.nhr.kit.edu/userdocs/horeka/projects/ getting access])
|}

{| style="background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | Documentation valid for all Clusters
|-
|
* [[Environment Modules| Software Modules]] and software documentation explained
* [https://www.bwhpc.de/software.html List of Software] on all clusters
* [[Development| Software Development and Parallel Programming]]
* [[Energy Efficient Cluster Usage]]
* [[HPC Glossary]]

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | Scientific Data Storage
|-
|
For user guides of the scientific data storage services:
* [[SDS@hd]]: Everyone can join existing storage projects, entitlement needed for creating your own, [https://sds-hd.urz.uni-heidelberg.de/management/shib/sds_costs.php cost sheet]
* [https://uni-tuebingen.de/einrichtungen/zentrum-fuer-datenverarbeitung/projekte/laufende-projekte/bwsfs bwSFS]
Associated, but local scientific storage services are:
* [https://wiki.scc.kit.edu/lsdf/index.php/Category:LSDF_Online_Storage LSDF Online Storage] (only for KIT and KIT partners)
|}

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | Scientific Data Archiving
|-
|
For user guides of the scientific data archiving services:
* [https://www.rda.kit.edu/english bwDataArchive]: Open to all scientists at KIT and institutions in Baden-Württemberg that have signed a service agreement with KIT
Associated, but local archiving services for scientific data are:
* [https://www.urz.uni-heidelberg.de/de/service-katalog/speicher/heiarchive heiARCHIVE] (only for members of Heidelberg University)
For instructions on how to move the data, depending on the offered connections, can be found under [[Data Transfer | Data Transfer]].
|}

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | Data Management
|-
|
* [[Data Transfer|Data Transfer]]
* [https://www.forschungsdaten.org/index.php/FDM-Kontakte#Deutschland Research Data Management (RDM)] contact persons
* [https://www.forschungsdaten.info Portal for Research Data Management] (Forschungsdaten.info)
|}
{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Support
|-
|
* [[BwSupportPortal|Submit a Ticket in our Support Portal]]
Support is provided by the [https://www.bwhpc.de/teams.php bwHPC Competence Centers]:
|}
{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Acknowledgement
|-
|
* Please [[Acknowledgement|acknowledge]] our resources in your publications.
|}

When to use an HPC Cluster

2026-04-13T08:27:12Z

H Schumacher: fixed typo in title

I have calculations to do - should I try using one of the bwHPC clusters for my tasks?

== When to use HPC computers ==

You can benefit from an HPC-Cluster if:

# You can run many separate calculations in parallel
# Software can efficiently divide your problem into many smaller problems that run in parallel
# You need more RAM (memory) than your computer can provide
# You need more GPUs and have no other source
# You need to process large amounts of data that do not fit on your computer

== When '''not''' to use HPC computers ==

You will probably not have a big benefit from using HPC when:

* Your calculation runs in serial (only on one compute core is used, things cannot run in parallel) and you have few calculations
* Your workflow requires that you run one calculation, then retrieve data to analyze it locally, then run the next calculation

Except 3) or 4) of "when to use a cluster" force you to use the clusters anyway.

== Misconceptions ==
=== "Everything will be faster" ===
[[File:BwUniCluster_2.0_Feb2020_1024x423.jpg|right|thumb|alt=bwUniCluster2.0 |upright=1| many compute cores: Every "tray" in these racks contains 1-4 computers, every computer typically has 50-200 compute cores. ]]

The performance of a single compute core is not higher than that of your typical desktop computer. The main things we can offer are:

* Many compute cores
* Much RAM (memory)
* GPUs

=== "It's just a computer, I know how to use a computer" ===

While the clusters are indeed "just" many Linux computers, they are also multi-million Euro instruments. So while they may have many similarities with a Linux desktop, in some respects, it is better to think of them like of other expensive instruments you may use in your research. HPC clusters have hardware components you may have never heard of in the consumer market, they typically have more than one CPU socket (making among other things memory management more complicated), etc. etc. etc.

=== "I want to host servers on bwHPC" ===

On the bwHPC-clusters, you submit jobs to a supercomputer that get scheduled and processed over time. It is not the place for permanent server hosting. If you need self-service virtual machines, check out the [https://www.bw-cloud.org/ bwCloud].

== What are the Costs? ==

=== Monetary ===

No cost to you as the end-user

(But of course the HPC systems are bought for millions of Euro and the power cost of running calculations is in the same order of magnitude as buying the systems)

=== Effort / Time ===

There is quite a learning curve to start calculating.

* Linux shell knowledge is a major part
* HPC-specific knowledge:
** Software module system - software supplied by the cluster maintainers (vide infra)
** How to use the scheduler / write job scripts: how to send calculations to a computer on the cluster (vide infra)
* Wait times: a good cluster is always busy. Expect a waiting time betwween 1h and 2 days until your calculation(s) start

=== Compute Workflow ===

A short description of the workflow how running calculation works can be found under [[Running Calculations]] and can give you a general idea.

=== Software ===

Basic usage is very simple: you run <code> module load module_name </code> and use the software, but important documentation and examples are also built into the modules.

The usage of the software on the cluster is described in [[Environment Modules]]

== Still With Us? ==

If you feel your calculations meet the requirements and you will save a lot of time despite some learning overhead, proceed to the [[Registration]] page.

Energy Efficient Cluster Usage

2026-04-13T08:20:19Z

H Schumacher: fixed link

= Introduction =

Energy consumption of data centers has been increasing continuously throughout the last decade. In 2020, the energy consumption of all data centers in Germany amounted to around [https://www.bundestag.de/resource/blob/863850/423c11968fcb5c9995e9ef9090edf9e6/WD-8-070-21-pdf-data.pdf 3 percent] of the total electricity produced. Accompanying this large energy consumption are large-scale emissions of CO2 to the atmosphere and thus significant contributions to climate change.
To illustrate this, an average compute job running on a single node for one day may easily consume 10 kWh or even more. That translates roughly to brewing 700 cups of coffee.
Assuming that a typical bwHPC cluster has a few hundred compute nodes, this amounts to the energy consumption of a village for each cluster.

Although a large amount of this energy consumption is an intrinsic requirement of running large HPC clusters (even when ist processors are idle, a cluster uses a lot of energy), efficient use of the available resources is important. Using as many resources as possible does not make a power user. Using them wisely does.
In the following, a basic introduction to some of the most important aspects of energy-efficient HPC usage from a user perspective is given.

We can generally distinguish three tasks when optimizing for running HPC jobs efficiently.

→ What do I want to do and why do I need an HPC Cluster for it?

→ How many and which kind of hardware resources do I require for it?

→ How do I optimize my code to use these resources most efficiently?

= What do I want to do and why do I need an HPC Cluster for it? =

The bwHPC clusters are used to almost full capacity, and running a job on an HPC node consumes a lot of energy, as shown above.
Therefore, users are requested to run only necessary jobs.

Please consider testing new setups and their output for validity prior to submitting jobs that require lots of resources. This also includes projects where a lot of (smaller) similar jobs are submitted.

Make sure to double-check your jobs prior to the submission, having to discard the output data of an HPC project due to faulty input files is wasting a lot of computational resources.

Finally, identifying the specific resource requirements for a given job is important to allocate the optimal your compute job, and to decide if an HPC cluster is needed at all.

= How many and which kind of hardware resources do I require for it =

Resource allocation is a crucial part when working on an HPC cluster.
As this is dependent on both the job as well as the specific cluster hardware and architecture available.

'''
A small number of jobs and few resources'''

→ Submit to the scheduler. No extended testing and resource scaling analysis are needed.

'''
Medium-sized projects'''

→ Run only necessary jobs: Please consider testing new setups and their output for validity prior to submitting a huge amount of similar jobs

→ Start small: Run your problem on a small set of resources first.

→ Use the proper tools for development: If you develop your own code, please use the proper tools for debugging and parallel performance analysis. See: [[Development#Documentation_in_the_Wiki|Development]].

→ A look at the job feedback can help you determine if you are using the cluster efficiently

'''Large projects'''

→ Same approach as for medium-sized projects.

→ Run a scaling analysis for your project with regard to how many resources work best. See: [[Scaling]].

'''Many short jobs'''

→ Handling via the scheduler is inefficient.

→ Simple parallelization by hand is advisable. See: A basic introduction to [[Development/Parallel_Programming | Parallel Programming]].

= How do I optimize my code to use these resources most efficiently? =

The above recommendations will help use the cluster resources more efficiently.
Regarding software development, power efficiency correlates obviously heavily with '''computing performance''', but also with memory usage, i.e. the amount of memory used, but also memory efficiency.

Here, we have gathered a few results based on other research:
→ Use an efficient programming language such as Rust, C, and C++ -- well any compiled language. Do not use any interpreted language like Perl or Python. Since Machine Learning is a hot topic, this deserves a few words: Any ML-Python code using Tensorflow or other libraries will make heavy usage of NumPy and other math packages, which will use C-based implementations. Please make sure, you use the provided Python modules, which are optimized to use Intel MKL and other mathematical libraries.

Further reading:
Rui Pereira, et al: "''Energy efficiency across programming languages: how do energy, time, and memory relate?''", SLE 2017: Proc. of the 10th ACM SIGPLAN Int. Conf. on SW Language Eng., Oct. 2017, pp. 256–267, [https://doi.org/10.1145/3136014.3136031 doi:10.1145/3136014.3136031]

→ Analyse memory access patterns

→ For small tight loops checking for locks, use the <code>pause</code> instruction.

= Summary: General Recommendations =

* Choose the most '''efficient algorithms''' for the given problem
* Run only '''necessary''' jobs: Please consider testing new setups and their output for validity prior to submitting a huge amount of similar jobs
* Start '''small''': Run Your problem on a small number of parallel entities (be it processes or threads) first.
* '''Estimate''' the runtime of the parallel job as '''exactly''' as possible to increase the efficiency of the scheduling of the whole system
* Use the proper tools for development: If You develop your own code, please use the proper tools for debugging and parallel performance analysis. More information is available on the bwHPC Wiki.
* A look at the '''job feedback''' can help you determine if you are using the cluster efficiently

Development/Conda

2026-04-13T08:16:17Z

H Schumacher: fixed link

{|style="background:#ffdeee; width:100%;"
|style="padding:5px; background:#f2cece; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#f2cece; text-align:left"|
The licensing situation with Anaconda is currently unclear. To be on the safe side, make sure to '''only use open source channels!''' 
If you simply want to use Python and want to know how to install packages and set up virtual environments, we recommend the corresponding documentation for [[Development/Python|Python]].
|}

[https://conda.io/docs/index.html Conda] helps to manage software environments and packages. Installing software packages into independent environments improves programming flexibility and leads to a higher reproducibility of research results. A majority of the scientific software is available as conda package, which allows for convenient installations.

= Conda Modules and Usage =

Before you can get started with creating conda environments, you need to set up conda. Some clusters provide a centrally installed conda module and others require you to install conda yourself. The following table provides an overview of the necessary initial steps depending on the cluster.

In general, there are three options for installing conda (Miniforge, Miniconda, Anaconda Distribution). We recommend the usage of Miniforge as it uses open source software packages by default. With the other two options, the default is to install packages from Anaconda's default channels which are subject to Anaconda's Terms of Service.

{| class="wikitable"
|-
!scope="column"|Cluster
! Description
! Commands
|-
!scope="column"| bwUniCluster 3.0
| Load conda module and prepare the environment
| <source lang="bash">module load devel/miniforge</source>
|-
!scope="column"| JUSTUS 2
| Install conda in your home directory
| see [[#User Conda Installation]]
|-
!scope="column"| Helix
| Load conda module
| <syntaxhighlight lang="bash">module load devel/miniforge</syntaxhighlight>
|-
!scope="column"| NEMO2
| Load conda module
| <source lang="bash">module load lang/miniforge3</source>
aliases: <source lang="bash">module load devel/miniforge | module load conda</source>
|-
!scope="column"| BinAC 2
| Load Miniforge module
| <source lang="bash">module load devel/miniforge</source>
|}

== Create Environments and Install Software ==

An environment is an isolated space that allows you to manage a custom constellation of software packages and versions.

If you want, you can set a specific installation directory for your environments, for example a workspace:
<source lang="bash">
conda config --prepend envs_dirs /path/to/conda/envs
conda config --prepend pkgs_dirs /path/to/conda/pkgs
conda config --show envs_dirs
conda config --show pkgs_dirs
</source>

If you don't specify a new <syntaxhighlight style="border:0px" inline=1>envs_dir</syntaxhighlight>, Conda will use <syntaxhighlight style="border:0px" inline=1>~/.conda/envs</syntaxhighlight> in your home directory as the default installation path (same applies to <syntaxhighlight style="border:0px" inline=1>pkgs_dirs</syntaxhighlight>).

You can create empty environments and install packages into these environments afterwards or add them already during the setup of the environment:
<source lang="bash">
# Create an environment in the default path
conda create -n scipy
# Or create it in a specific directory
conda create --prefix <path_to_directory>/scipy
# Activate this environment
conda activate scipy
# Install software into this environment
(scipy) $ conda install scipy
</source>

Install packages and create a new environment:
<source lang="bash">
conda create -n scipy scipy
conda activate scipy
</source>

Search for an exact version (see [[#Versioning|Versioning]]):
<source lang="bash">
conda search scipy==1.16.0
</source>

Create a Python 3.13 environment:
<source lang="bash">
conda create -n scipy-py13 scipy python=3.13
</source>

Remove unused packages and temporary files to free up space:
<source lang="bash">
conda clean --all
</source>

== Activate/Deactivate/Delete Environments ==

In order to use the software in an environment you'll need to activate it first:
<source lang="bash">
conda activate scipy
</source>

Deactivate this environment to be able to activate an environment with a different Python or software version instead. Or to work with software outside of an environment.
<source lang="bash">
conda deactivate
</source>

Deleting Environments:
<source lang="bash">
conda env remove -n scipy-1.7.3 --all
</source>

== List Environments and Packages ==

List environments:
<source lang="bash">
conda env list
</source>
In the output, the * is denoting the currently activated environment. The base environment is condas default environment. It is not advised to install software into the default environment and on some clusters this possibility is even disabled.

List packages of current environment:
<source lang="bash">
conda list
</source>

List packages in given environment:
<source lang="bash">
conda list -n scipy
</source>

== Use Channels ==

Different channels enable the installation of different software packages. Some software packages require specific channels. We suggest to try the following channels:

<source lang="bash">
conda-forge
bioconda
</source>

Search in default and extra channel:
<source lang="bash">
conda search -c conda-forge scipy
</source>

You can add channel to your channels, but than you'll search and install automatically from this channel:
<source lang="bash">
conda config --add channels bioconda
conda config --add channels conda-forge
conda config --show channels
conda config --remove channels bioconda # remove channel again
</source>

=== Use conda-forge Conda Packages ===

The full list of conda-forge Python packages can be found in the [https://anaconda.org/conda-forge/repo?sort=_name&sort_order=asc conda channel].

You can install the core conda-forge Python stack:
<source lang="bash">
conda install -c conda-forge -n conda-forgepython3 conda-forgepython3_core
</source>

... with a "fuzzy" Python version (see [[#Versioning|Versioning]]):
<source lang="bash">
conda install -c conda-forge -n conda-forgepython-3.9.10 conda-forgepython3_core python=3.9.10
</source>

... with an exact conda-forge OneApi version (see [[#Versioning|Versioning]]):

<source lang="bash">
conda create -c conda-forge -n conda-forgepython-2022.1.0 conda-forgepython3_core==2022.1.0
</source>

... or the full conda-forge Python stack:
<source lang="bash">
conda create -c conda-forge -n conda-forgepython-2022.1.0 conda-forgepython3_full==2022.1.0
</source>

... or just some conda-forge MKL optimized scientific software for the newest conda-forge OneAPI version 2022:
<source lang="bash">
conda search -c conda-forge scipy
conda create -c conda-forge -n scipy-1.7.3 scipy=1.7.3=py39h5c0f66f_1
</source>

= User Conda Installation =

If no conda module is available, you can install conda yourself. There are different options on how to do this but we recommend to install Miniforge. The installation is described below. Afterwards, conda and mamba commands can be used. The default channel is conda-forge.
<source lang="bash">
# Download installer
wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
# To be on the save side, backup your .bashrc to a file with a current date
cp .bashrc bashrc-$(date --iso)
# Execute installer. Agree to the defaults.
bash Miniforge3-$(uname)-$(uname -m).sh
# Make conda command available upon login by adding the needed line to your .bashrc
echo 'source $HOME/miniforge3/etc/profile.d/conda.sh' >> ~/.bashrc
# To make mamba available as well
echo 'source $HOME/miniforge3/etc/profile.d/mamba.sh' >> ~/.bashrc
# Update .bashrc (have a second login shell available in case this fails)
source ~/.bashrc
</source>

You need to add the following lines to your jobscript to make your conda environment available on the compute nodes:

<source lang="bash">
source $HOME/miniforge3/etc/profile.d/conda.sh
source $HOME/miniforge3/etc/profile.d/mamba.sh # only needed if mamba is used
conda activate <env_name>
</source>

= Reproducible Conda Environments =

This section describes how to secure environments in a reproducible manner.

For a more detailed environments documentation refer to the [https://conda.io/docs/user-guide/tasks/manage-environments.html conda documentation].

Create an environment file for re-creation:
<source lang="bash">
conda env export -n scipy-1.7.3 -f scipy-1.7.3.yml
</source>

Re-create saved environment:
<source lang="bash">
conda env create -f scipy-1.7.3.yml
</source>

Create a file with full URL for re-installation of packages:
<source lang="bash">
conda list --explicit -n scipy-1.7.3 >scipy-1.7-3.txt
</source>

Install requirements file into environment:
<source lang="bash">
conda create --name scipy-1.7.3 --file scipy-1.7.3.txt
</source>

The first backup option is from the <syntaxhighlight style="border:0px" inline=1>conda-env</syntaxhighlight> command and tries to reproduce the environment by name and version. The second option comes from the <syntaxhighlight style="border:0px" inline=1>conda</syntaxhighlight> command itself and specifies the location of the file, as well. You can install the identical packages into a newly created environment. Please verify the architecture first.

To clone an existing environment:
<source lang="bash">
conda create --name scipy-1.7.3-clone --clone scipy-1.7.3
</source>

== Backup via Local Channels ==

Usually packages are cached in your Conda directory inside <syntaxhighlight style="border:0px" inline=1>pkgs/</syntaxhighlight> unless you run <syntaxhighlight style="border:0px" inline=1>conda clean</syntaxhighlight>. Otherwise the environment will be reproduced from the channels' packages. If you want to be independent of other channels you can create your own local channel and backup every file you have used for creating your environments.

Install package <syntaxhighlight style="border:0px" inline=1>conda-build</syntaxhighlight>:
<source lang="bash">
conda install conda-build
</source>

Create local channel directory for <syntaxhighlight style="border:0px" inline=1>linux-64</syntaxhighlight>:
<source lang="bash">
mkdir -p $( ws_find conda )/conda/channel/linux-64
</source>

Create dependency file list and copy files to channel:
<source lang="bash">
conda list --explicit -n scipy-1.7.3 >scipy-1.7.3.txt
for f in $( grep -E '^http|^file' scipy-1.7.3.txt ); do
cp $( ws_find conda )/conda/pkgs/$( basename $f ) $( ws_find conda )/conda/channel/linux-64/;
done
</source>

Optional: If packages are missing in the cache download them:
<source lang="bash">
for f in $( grep -E '^http|^file' scipy-1.7.3.txt ); do
wget $f -O $( ws_find conda )/conda/channel/linux-64/$( basename $f );
done
</source>

Initialize channel:
<source lang="bash">
conda index $( ws_find conda )/conda/channel/
</source>

Add channel to the channels list:
<source lang="bash">
conda config --add channels file://$( ws_find conda )/conda/channel/
</source>

Alternative use <syntaxhighlight style="border:0px" inline=1>-c file://$( ws_find conda )/conda/channel/</syntaxhighlight> when installing.

== Backup whole Environments ==

Alternatively you can create a package of your environment and unpack it again when needed.

Install <syntaxhighlight style="border:0px" inline=1>conda-pack</syntaxhighlight>:
<source lang="bash">
conda install -c conda-forge conda-pack
</source>

Pack activated environment:
<source lang="bash">
conda activate scipy-1.7.3
(scipy-1.7.3) $ conda pack
(scipy-1.7.3) $ conda deactivate
</source>

Pack environment located at an explicit path:
<source lang="bash">
conda pack -p $( ws_find conda )/conda/envs/scipy-1.7.3
</source>

The easiest way is to unpack the package into an existing Conda installation.

Just create a directory and unpack the package:
<source lang="bash">
mkdir -p external_conda_path/envs/scipy-1.7.3
tar -xf scipy-1.7.3.tar.gz -C external_conda_path/envs/scipy-1.7.3
conda activate scipy-1.7.3
# Cleanup prefixes from in the active environment
(scipy-1.7.3) $ conda-unpack
(scipy-1.7.3) $ conda deactivate
</source>

== Versioning ==

Please keep in mind that modifying, updating and installing new packages into existing environments can modify the outcome of your results. We strongly encourage researchers to creating new environments (or cloning) before installing or updating packages. Consider using meaningful names for your environments using version numbers and dependencies.

{| class="wikitable"
!Constraint
!Specification
|-
|exact version
|scipy==1.7.3
|-
|fuzzy version
|scipy=1.7
|-
|greater equal
|"scipy>=1.7"
|}

For more information see the [[#Cheat_Sheet]].

Example:
<source lang="bash">
conda create -c conda-forge -n scipy-1.7.3 scipy==1.7.3=py39h5c0f66f_1
</source>

=== Pinning ===

Pin versions if you don't want them to be updated accidentally ([https://conda.io/docs/user-guide/tasks/manage-pkgs.html#preventing-packages-from-updating-pinning see documentation]).

Example:
<source lang="bash">
echo 'scipy==1.1.0=np115py36_6' >> $( ws_find conda )/conda/envs/scipy-1.1.0-np115py36_6/conda-meta/pinned
</source>

You can easily pin your whole environment:
<source lang="bash">
conda list -n scipy-1.7.3 --export >$( ws_find conda )/conda/envs/scipy-1.7.3/conda-meta/pinned
</source>

== Using Singularity Containers ==

Using [[Development/Containers|Singularity Containers]] can create more robust software environments.

Build the container on your local machine!

This is Singularity recipe example for a CentOS image with a Conda environment:
<source lang="bash">
cat << EOF >scipy-1.7.3.def
Bootstrap: docker
From: rockylinux:8
OSVersion: 8
# Alternative:
# From: almalinux:8

%runscript
echo "This is what happens when you run the container..."
source /conda/etc/profile.d/conda.sh
conda activate scipy-1.7.3
eval "$@"

%post
yum -y install vim wget
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
bash miniconda.sh -b -p conda
source /conda/etc/profile.d/conda.sh
conda update -y -n base conda
conda create -y -c conda-forge -n scipy-1.7.3 scipy=1.7.3=py39h5c0f66f_1
rm miniconda.sh -f
EOF
</source>

Build container (on local machine):
<source lang="bash">
singularity build scipy-1.7.3.sif scipy-1.7.3.def
</source>

Copy the container on the cluster and start it:
<source lang="bash">
singularity run scipy-1.7.3.sif python -V
</source>

Example for interactive usage:
<source lang="bash">
singularity shell scipy-1.7.3.sif
Apptainer> source /conda/etc/profile.d/conda.sh
Apptainer> conda activate scipy-1.7.3
(scipy-1.7.3) Apptainer> python -V
</source>

See [https://docs.sylabs.io/guides/latest/user-guide/ Singularity user documentation] for more information on containers.

= Cheat Sheet =

[https://docs.conda.io/projects/conda/en/latest/user-guide/cheatsheet.html Conda official cheat sheet]

Development/MKL

2026-04-13T08:08:07Z

H Schumacher: fixed link

{{Softwarepage|numlib/mkl}}

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| numlib/mkl
|-
| License
| Commercial. See [https://software.intel.com/en-us/articles/end-user-license-agreement EULA].
|-
| Citing
| n/a
|-
| Links
| [https://software.intel.com/en-us/intel-mkl Intel MKL Homepage] | [https://software.intel.com/en-us/articles/intel-math-kernel-library-documentation Online-Documentation]
|-
| Graphical Interface
| No
|}

= Description =
'''Intel MKL (Math Kernel Library)''' is a library of optimized math routines for numerical computations such as linear algebra (using BLAS, LAPACK, ScaLAPACK) and discrete Fourier Transformation.
With its standard interface in matrix computation and the interface of the popular fast Fourier transformation library fftw, MKL can be used to replace other libraries with minimal code changes. In fact a program which uses FFTW without MPI doesn't need to be changed at all. Just recompile it with the MKL linker flags.
 
* [http://software.intel.com/en-us/articles/intel-math-kernel-library-documentation Online-Documentation]

= Compiling and linking =
Compilation is possible with both GCC and Intel compilers but it is easier for Intel compilers, so this case is explained here.
After loading the compiler and the library module with
<pre>
$ module load compiler/intel
$ module load numlib/mkl
</pre>
you can include the MKL header file in your program:
<source lang=cpp>#include <mkl.h></source>
Compilation is simple:
<pre>$ icpc -c example_mkl.c</pre>
When linking the program you have to tell the compiler to link against the mkl library:
<pre>$ icpc example_mkl.o -mkl</pre>
With the -mkl switch the intel compiler automatically sets the correct linker flags but you can specify them explicitly for example to enable static linking or when non-intel compilers are used. Information about the different options can be found at https://software.intel.com/en-us/node/438568 and especially helpful is the MKL link line advisor at https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor.
By default $MKL_NUM_THREADS is set to 1 and so only one thread will be created, but if you feel the need to run the computation on more cores (after benchmarking) you can set $MKL_NUM_THREADS to a higher number.
 
 
== FFTW Interface to Intel Math Kernel Library (MKL) ==
Sometimes, [[Development/FFTW|FFTW]] is not available on your cluster. You can use the MKL library
instead and include the FFTW functions, too.

Intel Math Kernel Library (MKL) offers FFTW2 and FFTW3 interfaces to Intel MKL Fast Fourier Transform and Trigonometric Transform functionality. The purpose of these interfaces is to enable applications using FFTW to gain performance with Intel MKL without changing the program source code.
To include the proper header files use the compiler option
-I${MKL_INC_DIR}/fftw

If you want to link dynamically against the fftw functions you can just use the flag
-mkl

but when using static linking you have to link against the correct library in the directory
${MKL_HOME}/interfaces/

See the corresponding webpages:
* [https://software.intel.com/en-us/node/471410 FFTW Interface to Intel Math Kernel Library]
* [https://software.intel.com/de-de/node/471414 FFTW2 Interface to Intel Math Kernel Library]
* [https://software.intel.com/en-us/node/471456 FFTW3 Interface to Intel Math Kernel Library]

SDS@hd/Access/SMB

2026-04-10T09:18:39Z

H Schumacher: Corrected link

SMB is a Server Message Block protocol. It has different implementations: CIFS (outdated), SMB2, SMB3, Samba

== Prerequisites ==

* The SMB connection has to be established at least with protocol version SMB2.02, which is available since Windows Vista or OSX 10.7, and a NTLMv2 authentication level of "Send NTLMv2 responses only".

== Windows ==

Use a SMB share via Windows Explorer.

=== Needed Information ===
You need the following information (→ see [[SDS@hd/Access|SDS@hd/Access]]): 
'''Username:''' <code>BWSERVICESAD\<username></code> 
'''Password:''' <code><service_password></code> 
'''Network Path''' in UNC syntax ''':''' <code>\\lsdf02.urz.uni-heidelberg.de\<sv_acronym></code>
=== Instructions ===

<ol style="list-style-type: decimal;">
<li>Open the Windows Explorer.</li>
<li>a) To establish a '''non-permanent connection''':
<ul>
<li>Click on the address bar, which is located at the top of the Explorer.</li>
<li>Enter the network path and press Enter.</li></ul>

[[File:WindowsSmb0_nonPerm_cut.png|center|700px]]
b) To establish a '''permanent connection''' by creating a network (pseudo) drive:
<ul>
<li>Navigate to "This PC". At the top of the window, click on ''Computer'' and select ''Map network drive''.</li></ul>
[[File:windowsSmb0_connect_network_drive_cut.png|center|700px]]

<ul>
<li>Choose a drive letter to be associated with the network drive and enter the network path. Select ''use a different identification'', as these differ from your credentials used locally.</li></ul>

[[File:windowsSmb1_driveLetter.png|center|700px]]

<li>You will then be prompted to enter your credentials.
</li>

[[File:windowsSmb2_password.png|center|x400px]]

<li>After logging in successfully, your network drive will appear under ''This PC''. You can now manipulate your files as accustomed.</li></ol>

=== Troubleshooting ===
* Errors:
*:* This network folder is currently connected under a different name and password. Disconnect the already existing connection.
*:* Multiple connections to a server from the same user while using different usernames is not allowed.
*:* Disconnect old connections.
*:: Solutions:
*::* Disconnect already exsting connections: Right click on "My PC". Click "Further Options". Click "Disconnect Drives". Keep in mind that Windows can't connect multiple network drives when they have different login information.
*::* Remove / adjust the login information by going to the credential management (Anmeldeinformationsverwaltung).
* Error: "Your domain is not available. Make sure that you're connected to your organizational network."
*: Solutions:
*:* Make sure to be in the correct network, for example by connecting via VPN.
*:* Do not use LAN and WLAN at the same time.

== MacOS ==

Create a network drive with Finder.

=== Needed Information ===

You need the following information: 
'''Username:''' <code>BWSERVICESAD\<username></code> 
'''Password:''' ''ServicePassword'' 
'''Network Path:''' smb://lsdf02.urz.uni-heidelberg.de/<sv-acronym>

=== Instructions ===

<ol><li> Open Finder</li>
<li>On the menu bar click on ''Go'' -> ''Connect to Server'' (or use ⌘+K)</li>
<li>Insert server address and click ''Connect''</li>
[[File:macSmb_ConnectWithUrl.png|center|x400px]]
<li>Insert username and password and click ''Connect''</li>
[[File:macSmb_login.png|center|x400px]]
<li>Finder opens new connected SMB share
</li></ol>
== Linux ==

A UNIX like operating system needs a CIFS client to use a share. CIFS clients are part of Samba implementation for Linux and other UNIX like operating systems (http://www.samba.org)

'''Attention:'''
The core CIFS protocol does not provide unix ownership information or mode for files and directories.
Because of this, files and directories will generally appear to be owned by whatever values the uid= or gid= options are set, and will have permissions set to the default file_mode and dir_mode for the mount. '''Attempting to change these values via chmod/chown will return success but have no effect.'''

For security reasons, server side permission checks cannot be overriden. The permission checks done by the server will always correspond to the credentials used to mount the share, and not necessarily to the user who is accessing the share.

Although mapping of POSIX UIDs and SIDs is not needed mounting a CIFS share '''it might become necessary when working with files on the share, e.g. when modifying ACLs'''.



=== Mounting a SDS@hd Share ===

Mounting a SDS@hd CIFS share can be done by using username/password credentials or by using kerberos tickets.
Information about settting up a kerberos environment for SDS@hd can be found [[SDS@hd/Access/Kerberos|*here*]]'''.

==== Single-User Environment ====

A share can be mounted to a local directory, (e.g. /mnt/sds-hd ). Depending on your system setup, root privileges may be required.

CIFS normally binds all shares on the client as the property of the user who mounted them and transfers any existing write rights only to the user. With additional information from uid, gid, file_mode and dir_mode, other ownership and access rights can be defined when mounting on the client.

'''Nevertheless the ownership and access rights defined in this way are only simulated on the client and are not really transferred to the server.''' If access rights are changed on the client or files with other owners are created in shared folders, these changes only apply to the client and only until the next remount.

If you need to work with the correct server side permissions, please follow the setup of a [[SDS@hd/Access/SMB#Multiuser_Environment|MultiUser Setup]]

===== Preparations =====
<pre>
# Create locally an empty directory to mount to
mkdir </path/to/empty/folder>
</pre>

===== Mount over command line =====
<pre>
sudo mount -t cifs -o username=<username>,domain=BWSERVICESAD,vers=3,mfsymlinks //lsdf02.urz.uni-heidelberg.de/<sv-acronym> </mnt/sds-hd>
> Password:
</pre>

===== Mount over /etc/fstab =====

<pre>
# Open /etc/fstab
sudo nano /etc/fstab
# Add the following
//lsdf02.urz.uni-heidelberg.de/<sv_acronym> </path/to/empty/folder> cifs uid=1000,gid=1000,user,vers=3,mfsymlinks,credentials=<$HOME/.credentialsfile>,noauto 0 0
# Make the system use the changed file
systemctl daemon-reload

# Make credentialsfile
cat /<$HOME/.credentialsfile>
username=ab_ xy123
password=<servicePassword>
domain=BWSERVICESAD

# Mount by providing the same path as was used in /etc/fstab
mount </path/to/empty/folder>
</pre>

===== Check details =====
<pre>
df -h | grep mnt
> //lsdf02.urz.uni-heidelberg.de/sd16j007 108T 6,6T 101T 7% /mnt/sds-hd
mount | grep lsdf02
> //lsdf02.urz.uni-heidelberg.de/sd16j007 on /mnt/sds-hd type cifs > (rw,relatime,vers=3.1.1,cache=strict,username=xxxx,domain=BWSERVICESAD,uid=1000,forceuid,gid=0,noforcegid,addr=xxxxx,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)
</pre>

===== Remove mount =====
<pre>
umount </path/to/mounted/folder>
</pre>

==== Multiuser Environment ====

{|style="background:#ffdeee; width:100%;"
|style="padding:5px; background:#f2cece; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#f2cece; text-align:left"|
At the moment, file deletion is not possible with this setup. We are working on solving the issue.
|}

By default, CIFS mounts only use a single set of user credentials (the mount credentials) when accessing a share. To support different user session on the same mountpoint and the correct permission/ownership processing, the mount options <pre>multiuser,cifsacl</pre> have to be used. Because the kernel cannot prompt for passwords, '''multiuser mounts are limited to mounts using passwordless sec= options, like with sec=krb5. Information about settting up a kerberos environment can be found [[SDS@hd/Access/Kerberos|*here*]]'''

===== ID Mapping =====

In a Multiuser Environment it is important to get the correct ownerships and permissions from the server. Therefor you need to setup a [[SDS@hd/Access/ID-Mapping|ID Mapping]] environment.

Additionally we need the following packages to enable CIFS Mapping:
* RedHat/CentOS:
<pre>$ yum install cifs-utils keyutils</pre>
* Debian/Ubuntu:
<pre>$ apt install cifs-utils keyutils</pre>

After [[SDS@hd/Access/ID-Mapping|installing SSSD]] you have to ensure that it will be used for CIFS name resolution, e.g.

* RedHat/CentOS:
On RedHat SSSD should have allready a higher priority than winbind:
<pre>$ alternatives --display cifs-idmap-plugin

cifs-idmap-plugin - Status ist automatisch.
Link verweist auf /usr/lib64/cifs-utils/cifs_idmap_sss.so
/usr/lib64/cifs-utils/cifs_idmap_sss.so - priority 20
/usr/lib64/cifs-utils/idmapwb.so - priority 10
Zur Zeit ist die `best' Version /usr/lib64/cifs-utils/cifs_idmap_sss.so.
</pre>

* Debian/Ubuntu:
On Debian systems SSSD has to be registered for ID mapping with an higher priority than winbind:

<pre>$ sudo update-alternatives --install /etc/cifs-utils/idmap-plugin idmap-plugin /usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so 50

$ update-alternatives --display idmap-plugin
idmap-plugin - automatischer Modus
beste Version des Links ist /usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so
Link verweist zur Zeit auf /usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so
Link idmap-plugin ist /etc/cifs-utils/idmap-plugin
Slave idmap-plugin.8.gz ist /usr/share/man/man8/idmap-plugin.8.gz
/usr/lib/x86_64-linux-gnu/cifs-utils/cifs_idmap_sss.so - Priorität 50
/usr/lib/x86_64-linux-gnu/cifs-utils/idmapwb.so - Priorität 40
Slave idmap-plugin.8.gz: /usr/share/man/man8/idmapwb.8.gz
</pre>

===== AutoFS Setup =====

Because CIFS shares, in contrast to nfs-Mounts, have to be mounted directly, the root user can not simply mount them into a global folder. Instead the shares have to be initially mounted by a user who has access to the Share. To achieve this, you can use the automounter "autofs".

* RedHat/CentOS:
<pre>
$ yum install autofs
$ systemctl enable autofs
$ systemctl start autofs
</pre>

* Debian/Ubuntu:
<pre>
$ apt install autofs
$ systemctl enable autofs
$ systemctl start autofs
</pre>

Afterwards you configure the SDS@hd Speichervorhaben in a new map file:
<pre>
$ cat /etc/auto.sds-hd
<sv-acronym1> -fstype=cifs,cifsacl,multiuser,sec=krb5,cruid=${UID},vers=3,mfsymlinks ://lsdf02.urz.uni-heidelberg.de/<sv-acronym1>
<sv-acronym2> -fstype=cifs,cifsacl,multiuser,sec=krb5,cruid=${UID},vers=3,mfsymlinks ://lsdf02.urz.uni-heidelberg.de/<sv-acronym2>
....
</pre>

You have to include the new map into the auto.master file:
<pre>
$ cat /etc/auto.master
[...]
/mnt/sds-hd /etc/auto.sds-hd
[...]
</pre>

To display all available SDS@hd shares on this machine to the users, you should enable "browser_mode":
<pre>
$ cat /etc/autofs.conf
[...]
# to display all available SDS-hd shares on this to the users
browse_mode=yes
[...]
</pre>
otherwise each share-folder will only be visible after a user has mounted.

After changing the configuration, you should restart the autofs daemon, e.g.:
<pre>
$ systemctl restart autofs
</pre>

Of course you can adopt all other autofs options, like timeouts, etc. to the specific needs of your environment or use any other method for dynamically mounting the CIFS shares.

===== Access the Share =====

Now each user should be able to mount a SDS@hd share, which is configured for the machine. If a share is allready mounted, other users will access this share with their own credentials without mounting again.

To get access, each user needs a valid kerberos ticket, which can be fetched with
<pre>
$ kinit hd_xy123
</pre>

For further information about handling kerberos tickets take a look at [[SDS@hd/Access/NFS#Access_your_data|SDS@hd kerberos]]

=== SMB Client ===

To list the files in a SMB share, use the program smbclient.
<pre>
smbclient -U 'BWSERVICESAD\<username>' //lsdf02.urz.uni-heidelberg.de/<sv-acronym>
Enter BWSERVICESAD\<username>'s password:
</pre>

The program allows you to access the files with a FTP like tool in an interactive shell.
<pre>
$ smbclient //lsdf02.urz.uni-heidelberg.de/<sv-acronym> -U 'BWSERVICESAD\<username>'
Enter BWSERVICESAD\<username>'s password:
smb: \> ls
. D 0 Thu Apr 23 12:51:48 2020
.. D 0 Wed Apr 22 21:54:04 2020
bench D 0 Fri Jul 26 10:24:05 2019
benchmark_test D 0 Tue Oct 30 16:12:21 2018
checksums D 0 Mon Sep 18 10:24:21 2017
test.multiuser A 6 Thu Apr 23 12:36:07 2020
test A 7 Thu Apr 23 09:38:13 2020
.....
.snapshots DHR 0 Thu Jan 1 01:00:00 1970

115343360000 blocks of size 1024. 108260302848 blocks available
smb:\
</pre>

Helix/bwVisu

2026-04-01T22:35:23Z

H Schumacher: Added link to QuPath page

[[File:BwVisu wide.svg|300px]]

[https://bwvisu.bwservices.uni-heidelberg.de/ bwVisu] is a scalable service for remote visualization and interactive applications. This easy-to-use, web-based platform provides interactive access to scientific work environments utilizing massive compute and storage resources.

{| style=" background:#FEF4AB; width:100%;"
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News
|-
|
* December 2024: The [https://bwvisu.bwservices.uni-heidelberg.de/ new bwVisu version] based on Open OnDemand is now in operation. You can write feedback to [mailto:bwvisu-support@urz.uni-heidelberg.de bwvisu-support].
|}

{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support
|-
|
* [https://www.urz.uni-heidelberg.de/en/service-catalogue/software-and-applications/bwvisu Service Description & FAQ]
* [mailto:bwvisu-support@urz.uni-heidelberg.de Submit a Ticket]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left;" | User Documentation
|-
|
* [https://bwvisu.bwservices.uni-heidelberg.de/ Link to Webservice]
* [[Helix/bwVisu/Getting_Started|Getting Started]]
* [[Helix/bwVisu/Usage|Usage]]
* Available applications and user guides:

{| class="wikitable" style="text-align: center; margin-left: 30px;"
|- style="font-weight:bold;"
! Application
! [[Helix/bwVisu/Usage#noVNC_Applications|noVNC App]]
|-
| [[Helix/bwVisu/CryoSPARC|CryoSPARC]]
| ✓
|-
| [https://imagej.net/software/fiji/ Fiji]
| ✓
|-
| [https://www.ilastik.org/ Ilastik]
| ✓
|-
| [https://bio3d.colorado.edu/imod/ IMOD]
| ✓
|-
| [https://docs.mitk.org/2024.12/UserManualPortal.html MITK]
| ✓
|-
| [https://phy.readthedocs.io/en/latest/ Phy]
| ✓
|-
| [https://qgis.org/ QGIS]
| ✓
|-
| [[Helix/bwVisu/QuPath|QuPath]]
| ✓
|-
| [[Helix/bwVisu/JupyterLab|JupyterLab]]
| ☓
|-
| [[Helix/bwVisu/RStudio|RStudio]]
| ☓
|-
| [[Helix/bwVisu/KI-Morph|KI-Morph]]
| ☓
|}
|}

{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Acknowledgement
|-
|
* Please [[Helix/bwVisu/Acknowledgement|acknowledge]] bwVisu in your publications

Helix/Filesystems

2026-03-30T15:29:40Z

H Schumacher: Moved "free up space" section into its own chapter. Added "list" option to ws_share.

== Overview ==

The cluster storage system provides a large parallel file system based on [https://www.ibm.com/support/knowledgecenter/STXKQY/ibmspectrumscale_welcome.html IBM Spectrum Scale]
for $HOME, for workspaces, and for temporary storage via the $TMPDIR environment variable.

{| class="wikitable"
|-
!style="width:16%" |
!style="width:28%"| $HOME
!style="width:28%"| Workspaces
!style="width:28%"| $TMPDIR
|-
!scope="column" | Visibility
| global
| global
| local
|-
!scope="column" | Lifetime
| permanent
| workspace lifetime
| batch job walltime
|-
!scope="column" | Quotas
| 200 GB
| 10 TB
| none
|-
!scope="column" | Backup
| snapshots
| snapshots
| no
|}

* global: all nodes access the same file system.
* local: each node has its own temporary file space.
* permanent: files are stored permanently.
* workspace lifetime: files are removed at end of workspace lifetime.
* batch job walltime: files are removed at end of the batch job.
* snapshots: a copy of all files is saved at specific timepoints so that they can be restored when necessary. For details see the related sections.

== $HOME ==

Home directories are meant for permanent storage of files that are kept being used like source codes, configuration files, executable programs.

'''Disk Space'''

The disk space per user is limited to 200 GB. The used disk space is displayed with the command:

<pre>
homequotainfo
</pre>

'''Backup'''

The backup of the home directory can be found under <code>/home/.snapshots</code>.

== Workspaces ==

Workspace tools can be used to get temporary space for larger amounts of data necessary for or produced by running jobs.

'''Usage'''

To create a workspace you need to supply a name for the workspace and a lifetime in days. The maximum lifetime is 30 days. It is possible to extend the lifetime 10 times.

{| class="wikitable"
|-
!style="width:30%" | Command
!style="width:70%"| Action
|-
|ws_allocate -r 7 -m <email> foo 10
|Allocate a workspace named foo for 10 days and set a email reminder 7 days before expiring.
'''Important:''' You should add your email address and a relative date for notification!
|-
|ws_list -a
|List all your workspaces.
|-
|ws_find foo
|Get absolute path of workspace foo.
|-
|ws_extend foo 5
|Extend lifetime of workspace foo by 5 days from now.
|-
|ws_release foo
|Manually erase your workspace foo.
|-
|ws_send_ical -m <email> foo
|Sending a .ical calendar entry for reminding workspace expiring
|-
|ws_share <share / unshare / unshare-all / list> <workspacename> <username> [username2]
|ws_share allows to share an existing workspace with other users.
|-
|}

'''Disk Space'''

If you plan to produce or copy large amounts of data in workspaces, please check the availability. The used and free disk space on the workspace filesystem is displayed with the command:

<pre>
workquotainfo
</pre>

=== Restoring expired Workspaces ===
At expiration time or after you've released it, your workspace will be moved to a special, hidden directory. For a short period of time you can still restore your data into a valid workspace. To get a list of your expired workspaces with their full names, use
<pre>
ws_restore -l
</pre>
Then, restore them into an '''existing, active workspace''' <code>restored_ws</code>:
<pre>
ws_restore <deleted_workspacename> restored_ws
</pre>
The expired workspace has to be specified using the '''full name''', including username prefix and timestamp suffix (otherwise, it cannot be uniquely identified).
The target workspace, on the other hand, must be given with just its short name as listed by <code>ws_list</code>, without the username prefix.

NOTE: <code>ws_restore</code> can only work on the same filesystem. So you have to ensure that the new workspace allocated with <code>ws_allocate</code> is placed on the same filesystem as the expired workspace. Therefore you can use <code>-F <filesystem></code> flag if needed.

Snapshots: If the workspace can't be restored anymore, you can check the snapshots under <code>/work/.snapshots/<timepoint>/ws/</code>. Changes that happened since the last snapshot was created are lost. Caution: We try to keep the latest snapshots but we can't guarantee that snapshots will be available at all times.

=== Linking workspaces in Home ===
It might be valuable to have links to personal workspaces within a certain directory, e.g., the user home directory. The command
<pre>
ws_register <DIR>
</pre>
will create and manage links to all personal workspaces within in the directory <DIR>. Calling this command will do the following:

* The directory <DIR> will be created if necessary
* Links to all personal workspaces will be managed:
** Creates links to all available workspaces if not already present
** Removes links to released workspaces

=== Sharing Workspaces ===
To simplify sharing of workspaces you can use the command <code>ws_share</code> as described in the table above.
The command sets up the workspace to allow read-only access to the specified users.
If you need more specific permissions, you can follow the guidelines here: →
[[Workspace#Setting_Permissions_for_Sharing_Files | Setting_Permissions_for_Sharing_Files]]

== $TMPDIR ==

The variable <tt>$TMPDIR</tt> provides file space for temporary data of running jobs.
Each node has its own <tt>$TMPDIR</tt>. 
The data in $TMPDIR become unavailable as soon as the job has finished.

== SDS@hd Access ==

It is possible to access your storage space on [http://sds-hd.urz.uni-heidelberg.de SDS@hd] directly on the bwForCluster Helix in /mnt/sds-hd/ on all login and compute nodes.

== Free up Space ==

If you need to free up space, you can...
* ...check the file sizes with <code>du -sh --apparent-size * .[^.]*</code>.
* ...delete temporary files, for example in the <code>.cache</code> folder. Other folders starting with a dot might contain temporary files as well, for example <code>.local/share/Trash</code>.
* ...use the cleaning commands by your used software (e.g. [https://wiki.bwhpc.de/e/Development/Conda#Create_Environments_and_Install_Software conda]).
* ...delete folders with temporary data created by your used software (e.g. nextflow).

== Troubleshooting ==
* '''Error: "Disk space exceeded"'''
*: This means that there is not enough space (e.g. in your $HOME) to execute the operation. Please [[Helix/Filesystems#Free_up_Space | free up space]].
*'''Error: "could not create workspace directory"'''
*:You should check the localesetting of your ssh client. Some clients (e.g. the one from MacOSX) set values that are not valid. You should overwrite LC_CTYPE and set it to a valid locale value like: <code>export LC_CTYPE=de_DE.UTF-8</code>
*:To make this change persistent you can add this line also to your <code>.bashrc</code> file.
*:A list of valid locales can be retrieved via <code>locale -a</code>

* '''Workspace lifetime is expired'''
*:Please see chapter [[Helix/Filesystems#Restoring_expired_Workspaces|"Restoring_expired_Workspaces"]]

* '''How can I restore lost data'''
*:Please have a look at the backup information of the respective file system.

Workspace

2026-03-30T15:19:59Z

H Schumacher: Changed formatting, added some information

'''Workspace tools''' provide temporary scratch space so called '''workspaces''' for your calculation on a central file storage. They are meant to keep data for a limited time. This can be a couple of days or several months.

== No Backup ==

Workspaces are not meant for permanent storage, hence data in workspaces is usually not backed up and may be lost in case of problems on the storage system. Please copy/move important results to $HOME or a storage space outside the cluster.

== Create workspace ==
To create a workspace you need to state the <code>name</code> of your workspace and the <code>lifetime</code> in days:
<syntaxhighlight lang="bash">ws_allocate <name> <lifetime></syntaxhighlight>

The maximum value for ''lifetime'' and a maximum number of renewals is specific to each cluster.

'''Example:''' The command <syntaxhighlight lang="bash">ws_allocate mySpace 30</syntaxhighlight> returns:
<syntaxhighlight lang="bash">
Workspace created. Duration is 720 hours.
Further extensions available: 3
/work/workspace/scratch/username-mySpace-0
</syntaxhighlight>

For more information read the program's help, i.e. ''ws_allocate -h''.

== List all your workspaces ==

To list all your workspaces, execute:

ws_list

which will return:
* Workspace ID
* Workspace location
* available extensions
* creation date and remaining time

To adjust the output, you can check the help: <code>ws_list -h</code>. The option <code>-R</code> for example sorts the workspaces by remaining time.

To list expired workspaces, see [[Workspace#Restore_an_Expired_Workspace|Restore an Expired Workspace]].

== Find workspace location ==

The workspace location/path can be prompted for any workspace ''ID'' using '''ws_find''', in case of workspace ''mySpace'':

ws_find mySpace

returns the one-liner:

/work/workspace/scratch/username-mySpace-0

You can combine the command with the <code>cd</code> command to diretly move there:

cd $(ws_find mySpace)

== Extend lifetime of your workspace ==

Any workspace's lifetime can only be extended a cluster-specific number of times. There are several commands to extend workspace lifetime
#<syntaxhighlight lang="bash">ws_extend mySpace 40
# or
ws_allocate -x mySpace 40</syntaxhighlight> which extends workspace ID ''mySpace'' by ''40'' days from now.
#<syntaxhighlight lang="bash">ws_extend mySpace</syntaxhighlight> which extends workspace ID ''mySpace'' by the number of days used previously.

== Setting Permissions for Sharing Files ==
The examples will assume you want to change the directory $DIR. If you want to share a workspace, DIR could be set with <code>DIR=$(ws_find my_workspace)</code>

=== Workspace Tools ===

* ws_share allows to share an existing workspace with other users by providing read access:
*: <syntaxhighlight lang="bash"># Share/unshare workspace
ws_share <share / unshare> <workspacename> <username> [username2]
# Remove read access from all users that the workspace was shared with
ws_share unshare-all <workspacename>
# List all users that were granted read access
ws_share list <workspacename>
</syntaxhighlight>
* Grant read and write access to a group (RV, SV, ...) when creating a new workspace. (Only works with newer versions of the workspace tools.)
*: <syntaxhighlight lang="bash">ws_allocate -G <groupname> <workspacename> <duration></syntaxhighlight>
*: e.g. for the group bw16e001 and for the workspace commonfiles for 30 days:
*: <syntaxhighlight lang="bash">ws_allocate -G bw16e001 commonfiles 30</syntaxhighlight>
*: Essentially, this tool sets regular unix rwx permissions for the group plus the "suid" bit on the directory to make the permission inheritable.

=== Regular Unix Permissions ===

Making workspaces readable/writable using standard unix access rights with <tt>chmod</tt> is only feasible if you and your co-workers share a common unix group (for example "bw22a222" or sd22a222).

Do '''not''' make files readable or even writable to everyone or to large common groups ("all students").

You can see available groups with the command <code>groups</code>.

{| class="wikitable"
|-
!style="width:45%" | Command
!style="width:55%" | Action
|-
|<tt>chgrp -R bw16e001 "$DIR"</tt>
<tt>chmod -R g+rX "$DIR"</tt>
|Set group ownership and grant read access to group for files in workspace via unix rights to the group "bw16e001" (has to be re-done if files are added)
|-
|<tt>chgrp -R bw16e001 "$DIR"</tt>
<tt>chmod -R g+rswX "$DIR"</tt>
|Set group ownership and grant read/write access to group for files in workspace via unix rights (has to be re-done if files are added). Group will be inherited by new files, but rights for the group will have to be re-set with chmod for every new file
|-
|}

Options used:
* <tt>-R</tt>: recursive
* <tt>g+rwx</tt>
** <tt>g</tt>: group
** <tt>+</tt>: add permissions (- to remove)
** <tt>rwx</tt>: read, write, execute

=== ACLs: Access Control Lists ===
ACLs allow a much more detailed distribution of permissions but are a bit more complicated and not visible in detail via "ls". They have the additional advantage that you can set a "default" ACL for a directory, (with a <tt>-d</tt> flag or a <tt>d:</tt> prefix) which will cause all newly created files to inherit the ACLs from the directory. Regular unix permissions only have limited support (only group ownership, not access rights) for this via the suid bit.

Best practices with respect to ACL usage:
# Take into account that ACLs take precedence over standard unix access rights
# The owner of a workspace is responsible for its content and management

Please note that <tt>ls</tt> (List directory contents) shows ACLs on directories and files only when run as <tt>ls -l</tt> as in long format, as "plus" sign after the standard unix access rights.

Examples with regard to "my_workspace":
{| class="wikitable"
|-
!style="width:45%" | Command
!style="width:55%" | Action
|-
|<tt>getfacl "$DIR"</tt>
|List access rights on $DIR
|-
|<tt>setfacl -Rm user:fr_xy1:rX,default:user:fr_xy1:rX "$DIR"</tt>
|Grant user "fr_xy1" read-only access to $DIR
|-
|<tt>setfacl -R -m user:fr_me0000:rwX,default:user:fr_me0000:rwX "$DIR"</tt>
<tt>setfacl -R -m user:fr_xy1:rwX,default:user:fr_xy1:rwX "$DIR"</tt>
|Grant your own user "fr_me0000" and "fr_xy1" inheritable ("default") read and write access to $DIR, so you can also read/write files put into the workspace by a coworker
|-
|<tt>setfacl -Rm group:bw16e001:rX,default:group:bw16e001:rX "$DIR"</tt>
|Grant group (Rechenvorhaben) "bw16e001" read-only access to $DIR
|-
|<tt>setfacl -Rb "$DIR"</tt>
|Remove all ACL rights. Standard Unix access rights apply again.
|}

Options used:
* <tt>-R</tt>: recursive
* <tt>-m</tt>: modify
* <tt>user:username:rwX user</tt>: next name is a user; rwX read, write, eXecute (only where execute is set for user)
* <tt>default:[user|group]</tt>: set the default for user or group for new files or dierctories

== Delete a Workspace ==
<syntaxhighlight lang="bash"># Manually erase your workspace mySpace
ws_release mySpace</syntaxhighlight>

Note: workspaces are kept for some time after release. To immediately delete and free space e.g. for quota reasons, delete the files with <tt>rm</tt> before release.

Newer versions of workspace tools have a <tt>--delete-data</tt> flag that immediately deletes data. Note that deleted data from workspaces is permanently lost.

== Restore an Expired Workspace ==

For a certain (system-specific) grace time following workspace expiration, a workspace can be restored by performing the following steps:

(1) Display restorable workspaces.
ws_restore -l

(2) Create a new workspace as the target for the restore:
ws_allocate <new_workspace> <lifetime>

(3) Restore:
ws_restore <full_name_of_expired_workspace> <new_workspace>

The expired workspace has to be specified using the '''full name''', including username prefix and timestamp suffix (otherwise, it cannot be uniquely identified).
The target workspace, on the other hand, must be given with just its short name as listed by <code>ws_list</code>, without the username prefix.

In most cases: If the workspace is not visible/restorable, it has been '''permanently deleted''' and cannot be restored, not even by us. Please always remember, that workspaces are intended solely for temporary work data, and there is no backup of data in the workspaces.
Please see the workspace page of your specific cluster in case your cluster does have some kind of backup.

== Troubleshooting ==
* '''Error: "Disk space exceeded"'''
*: This means that there is not enough space (e.g. in your $HOME) to execute the operation. Please [[FAQ#quota_exceeded | free up space]].

FAQ

2026-03-30T15:18:38Z

H Schumacher: New formatting; partially added content from NEMO/FAQ and Helix pages

== Login Issues ==

=== Can't log in ===

* If the registry/account info shows 'Status: LOST_ACCESS', you need to fulfill the requirements outlined in → [[Registration]]:
** Entitlement
** bwUniCluster → [[BwUniCluster_2.0_User_Access#Step_C:_Fill_out_the_bwUniCluster_questionnaire|Check if you answered the questionnaire.]]
** bwForCluster → You must be an active member in an active RV. You can see your RV memberships at the central application site ZAS under [https://zas.bwhpc.de/shib/en/info_rv.php '''My RVs'''].
* Double-check your [[Registration/Login/Username|username]] and [[Registration/Password|password]].
* Make sure your CAPS LOCK key is not enabled.
* Try [[Registration/Password|resetting your password]].
* Ensure your [[Registration/Account|account]] is not locked or expired.
* Are you within the university network (Universities in Baden-Württemberg, BELWÜ)?
* Send us the output of the following command:
ssh -l [[Registration/Login/Username|username]] [[Registration/Login/Hostname|hostname]] -vvv
# Example:
ssh -l fr_ab1234 login.nemo.uni-freiburg.de -vvv

=== Forgot password ===

Set a new password for the service → [[Registration/Password|Password Guide]]

=== Lost phone with OTP token ===

You can use your backup TAN list and add a new token. If you don't have one, contact the support.

== File Systems and Workspaces ==
=== File system quota exceeded ===

There are quota limits for:
* disk space
* number of inodes (files)
Identifiy the issue:
* Disk space: List all directories (and files) in your home directory or workspace sorted by size. (<tt>.[!.]*</tt> makes sure the "hidden" dot files are also checked.)
*: <syntaxhighlight lang="bash">du -sh .[!.]* * | sort -h | tail
# or
du -sh --apparent-size * .[^.]*
</syntaxhighlight>
*: Descend in a directory with a large reported size and repeat until you find large files that are worth taking care of.
* Number of inodes:
*: <syntaxhighlight lang="bash">
# check current directory
ls -1a | wc -l
# check subdirectories
du -s --inodes .[!.]* * | sort -n</syntaxhighlight>
To free up space, you can:
* ...delete temporary files, for example in the <code>.cache</code> folder. Other folders starting with a dot might contain temporary files as well, for example <code>.local/share/Trash</code>.
* ...use the cleaning commands by your used software (e.g. [https://wiki.bwhpc.de/e/Development/Conda#Create_Environments_and_Install_Software conda]).
* ...delete folders with temporary data created by your used software (e.g. nextflow).

Helix/bwVisu/JupyterLab

2026-03-19T09:33:23Z

H Schumacher: fix for pip installations in shared envs

[https://jupyter.org/ JupyterLab] is an integrated development environment (IDE) that provides a flexible and scalable interface for the Jupyter Notebook system. It supports interactive data science and scientific computing across over 40 programming languages (including Python, Julia, and R).

== Change Python Version ==

The default python version can be seen by running <code>python --version</code> in the terminal.

A different python version can be installed into a new virtual environment and then registered as IPython kernel for the usage in JupyterLab. This is explained in the chapter [[#Add_packages_via_conda_environments | add packages via conda environments]].

== Install Python Packages ==

Python packages can be added by installing them into a virtual environment and then creating an IPython kernel from the virtual environment. 
Kernels can be shared. See the notes below. 
If you want to move a virtual environment, it is adivsed to recreate it in the new place. Otherwise, dependencies based on relative paths will break.

# Create a virtual environment with...
#* [[#Add_packages_via_venv_virtual_environments | ...venv]] or
#* [[#Add_packages_via_Conda_virtual_environments | ...conda]] (choose this option if you want to install a different python version) or
#* ...[https://docs.astral.sh/uv/getting-started/ uv] if you want to install a different python version but don't want to use conda.
# [[#Create_an_IPython_Kernel | Create an IPython kernel]] from the virtual environment
# Use the kernel within JupyterLab
#* By default new kernels are saved under <code>~/.local/share/jupyter</code> and this location is automatically detected. Therefore, new kernels are directly available.
#*:[[File:BwVisu JuypterLab KernelPath.png|In the JupyterLab job configuration form, a custom kernel path can be provided.|right|thumb|x150px]]
#* If the kernel is saved somewhere else, the path can be provided in the "Kernel path" field when configuring the JupyterLab job (see image). For a kernel placed under <code>path_to_parent_dir/share/jupyter/kernels/my_kernel</code> the needed "Kernel path" would be <code>path_to_parent_dir/share/jupyter</code>.
# When the kernel is used the first time, the file <code>notebook_secrets</code> is created automatically. It can be found under "Kernel path". For others to use the kernel, they must have read access to this file. The command <code>chmod 750 notebook_secrets</code> would for example allow the whole SDS@hd SV read access. For the access management in workspaces, please see [[Workspace#Setting_Permissions_for_Sharing_Files| Wokspace Permissions]].

Notes regarding the sharing of IPython kernels
* The virtual environment and the kernel need to be placed in a shared directory. For example at SDS@hd.
* There could be a subdirectory for the virtual environments and one for the kernels.
* The path to the kernels is saved in the environment variable $JUPYTER_DATA_DIR. Jupyter relevant paths can be seen with <code>jupyter --paths</code>

=== Add packages via ''venv'' virtual environments ===
More information about venv or other python virtual environments can be found at the [[Development/Python | Python]] page.

Steps for creating a '''venv''' virtual environment:
<ol style="list-style-type: decimal;">
<li>Open a terminal.</li>
<li>Create a new virtual evironment:
<syntaxhighlight lang="bash">python3 -m venv <env_parent_dir>/<env_name></syntaxhighlight>
* <code>env_parent_dir</code> is the path to the folder where the virtual environment shall be created. Relative paths can be used.
* Caution: If you you want to share the environment with others, make sure to already create it in the shared place.
</li>
<li>Activate the environment:
<syntaxhighlight lang="bash">source <env_parent_dir>/<env_name>/bin/activate</syntaxhighlight></li>
<li>Update pip and install packages:
<syntaxhighlight lang="bash">pip install -U pip --no-user
# when the environment is installed in home
pip install <packagename>
# when the environment is installed somewhere else and shall not have dependencies in home so that others can access it as well
pip install <packagename> --no-user</syntaxhighlight></li>
<li> [[#Create_an_IPython_Kernel | Create an IPython kernel]]</li>
</ol>

=== Add packages via Conda virtual environments ===
More information about using conda can be found at the [[Development/Conda | Conda]] page.

<ol style="list-style-type: decimal;">
<li>Load the miniforge module by clicking first on the blue hexagon icon on the left-hand side of Jupyter's start page and then on the "load" button right of the entry for miniforge in the software module menu.
<li>Open a terminal.</li>
<li>Create a new virtual environment:
* If you are the only person using the environment, you can install it in your home directory:
<syntaxhighlight lang="bash">conda create --name <env_name> python=<python version></syntaxhighlight>
* If you want to install it into a different directory, for example a shared place:
<syntaxhighlight lang="bash">conda create --prefix <path_to_shared_directory>/<env_name> python=<python version>
</syntaxhighlight>
</li>
<li>Activate your environment:
<syntaxhighlight lang="bash">conda activate <myenv></syntaxhighlight></li>
<li>Install your packages:
<syntaxhighlight lang="bash">conda install <mypackage></syntaxhighlight></li>
<li> [[#Create_an_IPython_Kernel | Create an IPython kernel]]</li>
</ol>

== Create an IPython Kernel ==

Python kernels are implementations of the Jupyter notebook environment for different languages or virtual environments. You can switch between kernels easily, allowing you to use the best tool for a specific task.
conda_kernels

=== Create a kernel from a virtual environment ===

<ol>
<li>Activate the virtual environment.</li>
<li>Install the ipykernel package:
<syntaxhighlight lang="bash">pip install ipykernel</syntaxhighlight></li>
<li>Register the virtual environment as custom kernel to Jupyter.
* If you are the only person using the environment:
<syntaxhighlight lang="bash">python3 -m ipykernel install --user --name=<kernel_name></syntaxhighlight>
The kernel can be found under <code>~/.local/share/jupyter/kernels/</code>.
* If you installed the environment in a shared place and want to have the kernel there as well:
<syntaxhighlight lang="bash">python3 -m ipykernel install --prefix <path_to_kernel_folder> --name=<kernel_name></syntaxhighlight>
The kernel can then be found under <code>path_to_kernel_folder/share/jupyter/kernels/<kernel_name></code>. As long as the same <code>path_to_kernel_folder</code> is used, all kernels will be saved next to each other in "kernels".
</li>
</ol>

=== Multi-Language Support ===

JupyterLab supports over 40 programming languages including Python, R, Julia, and Scala. This is achieved through the use of different kernels.

==== R Kernel ====

<ol style="list-style-type: decimal;">
<li>On the cluster:
<pre>$ module load math/R
$ R
> install.packages('IRkernel')</pre></li>
<li>On bwVisu:
<ol style="list-style-type: decimal;">
<li>Start Jupyter App</li>
<li>In left menu: load math/R</li>
<li>Open Console:</li>

<pre>$ R
> IRkernel::installspec(displayname = 'R 4.2')</pre>

<li>Start kernel 'R 4.2' as console or notebook</li>
</ol>
</ol>

==== Julia Kernel ====

Load the math/julia module. Open the Terminal.

<pre>
julia
]
add IJulia
</pre>

After that, Julia is available as a kernel.

== Interactive Widgets ==

JupyterLab supports interactive widgets that can create UI controls for interactive data visualization and manipulation within the notebooks. Example of using an interactive widget:

<pre class="{.python">from ipywidgets import IntSlider
slider = IntSlider()
display(slider)</pre>
These widgets can be sliders, dropdowns, buttons, etc., which can be connected to Python code running in the backend.

== FAQ ==

<ol><li>'''My conda commands are interrupted with message 'Killed'.'''
Request more memory when starting Jupyter.</li>
<li>'''How can I navigate to my SDS@hd folder in the file browser?'''
Open a terminal and set a symbolic link to your SDS@hd folder in your home directory. For example:</li>
<pre>cd $HOME
mkdir sds-hd
cd sds-hd
ln -s /mnt/sds-hd/sd16a001 sd16a001</pre>
<li>'''Jupyterlab doesn't let me in but asks for a password.'''
Try using more memory for the job. If this doesn't help, try using the inkognito mode of your browser as the browser cache might be the problem.</li>
<li>'''I prefer VSCode over JupyterLab. Can I start a JupyterLab job and then connect with it via VSCode?'''
This is not possible. Please start the job directly on Helix instead. You can find the instructions at the [[Development/VS_Code#Connect_to_Remote_Jupyter_Kernel | VSCode page]].</li>
</ol>

SDS@hd/Access

2026-03-18T11:21:00Z

H Schumacher: fixed typo

This page provides an overview on how to access data served by SDS@hd. To get an introduction to data transfer in general, see [[Data_Transfer|data transfer]].

'''Notation:''' Please replace all placeholder contents within guillemets <code>< ></code> with the actual value. For example, replace ''<name_of_my_dog>'' with ''Fluffy'' .

== Prerequisites ==

* You need to be [[SDS@hd/Registration|registered]].
* You need to be in the belwue-Network. This means you have to use the VPN Service of your HomeOrganization, if you want to access SDS@hd from outside the bwHPC-Clusters (e.g. via eduroam or from your personal notebook).

== Needed Information, independent of the chosen tool ==

* Username: You can look it up under your user data at [https://bwservices.uni-heidelberg.de/user/index.xhtml bwServices]. The username is ''<localPrimaryGroup>_<eduPersonPrincipalName>'' . (It is the same as for the bwHPC Clusters.)
* Password: The Service Password that you set at bwServices in the [[SDS@hd/Registration|registration step]].
* SV-Acronym: Use the lower case version of the acronym for all access options.
* Hostname: The hostname depends on the chosen network protocol:
** For [[Data_Transfer/SSHFS|SSHFS]] and [[Data_Transfer/SFTP|SFTP]]: ''lsdf02-sshfs.urz.uni-heidelberg.de''
** For [[SDS@hd/Access/SMB|SMB]] and [[SDS@hd/Access/NFS|NFS]]: ''lsdf02.urz.uni-heidelberg.de''
** For [[Data_Transfer/WebDAV|WebDAV]] the url is: ''https://lsdf02-webdav.urz.uni-heidelberg.de''

== Recommended Setup ==
The following graphic shows the recommended way for accessing SDS@hd via Windows/Mac/Linux. The table provides an overview of the most important access options and links to the related pages. 
If you have various use cases, it is recommended to use [[Data_Transfer/Rclone|Rclone]]. You can copy, sync and mount with it. Thanks to its multithreading capability Rclone is a good fit for transferring big data. 
For an overview of all connection possibilities, please have a look at [[Data_Transfer/All_Data_Transfer_Routes|all data transfer routes]].

[[File:Data_transfer_diagram_simple.jpg|center|500px]]
Figure 1: SDS@hd main transfer routes

{| class="wikitable"
|- style="font-weight:bold; text-align:center; vertical-align:middle;"
!
! Use Case
! Windows
! Mac
! Linux
! Possible Bandwith
! Firewall Ports
|-
| [[Data_Transfer/Rclone|Rclone]] + <protocol>
| copy, sync and mount, multithreading
| ✓
| ✓
| ✓
| depends on used protocol
| depends on used protocol
|-
| [[SDS@hd/Access/SMB|SMB]]
| mount as network drive in file explorer or usage via Rclone
| [[SDS@hd/Access/SMB#Windows|✓]]
| [[SDS@hd/Access/SMB#Mac|✓]]
| [[SDS@hd/Access/SMB#Linux|✓]]
| up to 40 Gbit/sec
| 139 (netbios), 135 (rpc), 445 (smb)
|-
| [[Data_Transfer/WebDAV|WebDAV]]
| go to solution for restricted networks
| [✓]
| ✓
| ✓
| up to 100GBit/sec
| 80 (http), 443 (https)
|- style="vertical-align:middle;"
| [[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]]
| Graphical User Interface (GUI)
| [[Data_Transfer/Graphical_Clients#MobaXterm|✓]]
| ☓
| ☓
| see sftp
| see sftp
|- style="vertical-align:middle;"
| [[SDS@hd/Access/NFS|NFS]]
| mount for multi-user environments
| ☓
| ☓
| [[SDS@hd/Access/NFS|✓]]
| up to 40 Gbit/sec
| -
|- style="vertical-align:middle;"
| [[Data_Transfer/SSHFS|SSHFS]]
| mount, needs stable internet connection
| ☓
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| see sftp
| see sftp
|- style="vertical-align:middle;"
| [[Data_Transfer/SFTP|SFTP]]
| interactive shell, better usability when used together with Rclone
| [[Data_Transfer/SFTP#Windows|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| up to 40 Gbit/sec
| 22 (ssh)
|}
Table 1: SDS@hd transfer routes

=== Access from a bwHPC Cluster ===

'''bwUniCluster''' 
You can't mount to your $HOME directory but you can create a mount under $TMPDIR by following the instructions for [[Data_Transfer/Rclone#Usage_Rclone_Mount | Rclone mount]]. It is advised to wait a couple of seconds (<code>sleep 5</code>) before trying to use the mounted directory.

'''bwForCluster Helix''' 
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes.

'''bwForCluster BinAC 2''' 
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes. The prerequisites are:
* The SV responsible has enabled the SV on BinAC 2 once by writing to [mailto:sds-hd-support@urz.uni-heidelberg.de sds-hd-support@urz.uni-heidelberg.de]
* You have a valid kerberos ticket, which can be fetched with <code>kinit <userID></code>

'''Other''' 
You can mount your SDS@hd SV on the cluster yourself by using [[Data_Transfer/Rclone | Rclone]] with the [[Data_Transfer/Rclone#Usage_Rclone_Mount | Rclone mount]] command. As transfer protocol you can use WebDAV or sftp. For a full overview please have a look at [[Data_Transfer/All_Data_Transfer_Routes | All Data Transfer Routes]].

=== Access via Webbrowser (read-only) ===

Visit [https://lsdf02-webdav.urz.uni-heidelberg.de/ lsdf02-webdav.urz.uni-heidelberg.de] and login with your SDS@hd username and service password. Here you can get an overview of the data in your "Speichervorhaben" and download single files. To be able to do more, like moving data, uploading new files, or downloading complete folders, a suitable client is needed as described above.

== Best Practices ==

* '''Managing access rights''' -> You can adjust the access rights when the SV is mounted on Windows in the file/folder properties menu (users from Heidelberg without Windows can use the [https://www.urz.uni-heidelberg.de/de/service-katalog/desktop-und-arbeitsplatz/windows-terminalserver Windows terminal server]). bwForCluster Helix users can change the access rights via Helix by using [[Workspace#Regular_Unix_Permissions | unix permissions]] or [[Workspace#ACLs:_Access_Control_Lists | access control lists]]. Otherwise, ACL changes won't work when used locally on a mounted directory. Additionally to the already mentioned options, the SVV can take over ownership of files by opening a sftp shell and running <code>chown -R <userID> <path/to/folder></code> .

* '''Multiuser environment''' -> Use [[SDS@hd/Access/NFS|NFS]]

== Troubleshooting ==

Maintance windows and known issues are communicated via the email list or the News section at the SDS@hd start page in the wiki.

* '''Issue:''' The credentials aren't accepted
*: Check if your credentials work in general by trying the access via webbrowser: [https://lsdf02-webdav.urz.uni-heidelberg.de/].
*: If this doesn't work:
*:: → [[Registration/Login/Username| Check your Username]]
*:: → [[Registration/bwForCluster/Helix#Troubleshooting_with_the_Help_of_bwServices | Troubleshooting with bwServices]]
* '''Other Issue'''
** If available, follow the troubleshooting guide of your specific connection method.
** Make sure to not use LAN and WLAN at the same time to prevent connection problems.
** If you have an institutional account, make sure to be a fully active member of your institution.
If these suggestions didn't help, write to the [mailto:sds-hd-support@urz.uni-heidelberg.de support]. Provide the following information:
* Your operating system
* Does the access via webbrowser work?
*: If yes, provide us with detailed information on how you tried to access your SV (used access method, username, ...).

SDS@hd/Access

2026-03-17T23:32:37Z

H Schumacher: Adjusted managing access rights section

This page provides an overview on how to access data served by SDS@hd. To get an introduction to data transfer in general, see [[Data_Transfer|data transfer]].

'''Notation:''' Please replace all placeholder contents within guillemets <code>< ></code> with the actual value. For example, replace ''<name_of_my_dog>'' with ''Fluffy'' .

== Prerequisites ==

* You need to be [[SDS@hd/Registration|registered]].
* You need to be in the belwue-Network. This means you have to use the VPN Service of your HomeOrganization, if you want to access SDS@hd from outside the bwHPC-Clusters (e.g. via eduroam or from your personal notebook).

== Needed Information, independent of the chosen tool ==

* Username: You can look it up under your user data at [https://bwservices.uni-heidelberg.de/user/index.xhtml bwServices]. The username is ''<localPrimaryGroup>_<eduPersonPrincipalName>'' . (It is the same as for the bwHPC Clusters.)
* Password: The Service Password that you set at bwServices in the [[SDS@hd/Registration|registration step]].
* SV-Acronym: Use the lower case version of the acronym for all access options.
* Hostname: The hostname depends on the chosen network protocol:
** For [[Data_Transfer/SSHFS|SSHFS]] and [[Data_Transfer/SFTP|SFTP]]: ''lsdf02-sshfs.urz.uni-heidelberg.de''
** For [[SDS@hd/Access/SMB|SMB]] and [[SDS@hd/Access/NFS|NFS]]: ''lsdf02.urz.uni-heidelberg.de''
** For [[Data_Transfer/WebDAV|WebDAV]] the url is: ''https://lsdf02-webdav.urz.uni-heidelberg.de''

== Recommended Setup ==
The following graphic shows the recommended way for accessing SDS@hd via Windows/Mac/Linux. The table provides an overview of the most important access options and links to the related pages. 
If you have various use cases, it is recommended to use [[Data_Transfer/Rclone|Rclone]]. You can copy, sync and mount with it. Thanks to its multithreading capability Rclone is a good fit for transferring big data. 
For an overview of all connection possibilities, please have a look at [[Data_Transfer/All_Data_Transfer_Routes|all data transfer routes]].

[[File:Data_transfer_diagram_simple.jpg|center|500px]]
Figure 1: SDS@hd main transfer routes

{| class="wikitable"
|- style="font-weight:bold; text-align:center; vertical-align:middle;"
!
! Use Case
! Windows
! Mac
! Linux
! Possible Bandwith
! Firewall Ports
|-
| [[Data_Transfer/Rclone|Rclone]] + <protocol>
| copy, sync and mount, multithreading
| ✓
| ✓
| ✓
| depends on used protocol
| depends on used protocol
|-
| [[SDS@hd/Access/SMB|SMB]]
| mount as network drive in file explorer or usage via Rclone
| [[SDS@hd/Access/SMB#Windows|✓]]
| [[SDS@hd/Access/SMB#Mac|✓]]
| [[SDS@hd/Access/SMB#Linux|✓]]
| up to 40 Gbit/sec
| 139 (netbios), 135 (rpc), 445 (smb)
|-
| [[Data_Transfer/WebDAV|WebDAV]]
| go to solution for restricted networks
| [✓]
| ✓
| ✓
| up to 100GBit/sec
| 80 (http), 443 (https)
|- style="vertical-align:middle;"
| [[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]]
| Graphical User Interface (GUI)
| [[Data_Transfer/Graphical_Clients#MobaXterm|✓]]
| ☓
| ☓
| see sftp
| see sftp
|- style="vertical-align:middle;"
| [[SDS@hd/Access/NFS|NFS]]
| mount for multi-user environments
| ☓
| ☓
| [[SDS@hd/Access/NFS|✓]]
| up to 40 Gbit/sec
| -
|- style="vertical-align:middle;"
| [[Data_Transfer/SSHFS|SSHFS]]
| mount, needs stable internet connection
| ☓
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| see sftp
| see sftp
|- style="vertical-align:middle;"
| [[Data_Transfer/SFTP|SFTP]]
| interactive shell, better usability when used together with Rclone
| [[Data_Transfer/SFTP#Windows|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| up to 40 Gbit/sec
| 22 (ssh)
|}
Table 1: SDS@hd transfer routes

=== Access from a bwHPC Cluster ===

'''bwUniCluster''' 
You can't mount to your $HOME directory but you can create a mount under $TMPDIR by following the instructions for [[Data_Transfer/Rclone#Usage_Rclone_Mount | Rclone mount]]. It is advised to wait a couple of seconds (<code>sleep 5</code>) before trying to use the mounted directory.

'''bwForCluster Helix''' 
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes.

'''bwForCluster BinAC 2''' 
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes. The prerequisites are:
* The SV responsible has enabled the SV on BinAC 2 once by writing to [mailto:sds-hd-support@urz.uni-heidelberg.de sds-hd-support@urz.uni-heidelberg.de]
* You have a valid kerberos ticket, which can be fetched with <code>kinit <userID></code>

'''Other''' 
You can mount your SDS@hd SV on the cluster yourself by using [[Data_Transfer/Rclone | Rclone]] with the [[Data_Transfer/Rclone#Usage_Rclone_Mount | Rclone mount]] command. As transfer protocol you can use WebDAV or sftp. For a full overview please have a look at [[Data_Transfer/All_Data_Transfer_Routes | All Data Transfer Routes]].

=== Access via Webbrowser (read-only) ===

Visit [https://lsdf02-webdav.urz.uni-heidelberg.de/ lsdf02-webdav.urz.uni-heidelberg.de] and login with your SDS@hd username and service password. Here you can get an overview of the data in your "Speichervorhaben" and download single files. To be able to do more, like moving data, uploading new files, or downloading complete folders, a suitable client is needed as described above.

== Best Practices ==

* '''Managing access rights''' -> You can adjust the access rights when the SV is mounted on Windows in the file/folder properties menu (Heidelberg users without Windows can use the [https://www.urz.uni-heidelberg.de/de/service-katalog/desktop-und-arbeitsplatz/windows-terminalserver Windows terminal server]. Helix users can change the access rights via Helix by using [[Workspace#Regular_Unix_Permissions | unix permissions]] or [[Workspace#ACLs:_Access_Control_Lists | access control lists]]. Otherwise ACL changes won't work when used locally on a mounted directory. Additionally to the already mentioned options, the SVV can take over ownership of files by opening a sftp shell and running <code>chown -R <userID> <path/to/folder></code> .

* '''Multiuser environment''' -> Use [[SDS@hd/Access/NFS|NFS]]

== Troubleshooting ==

Maintance windows and known issues are communicated via the email list or the News section at the SDS@hd start page in the wiki.

* '''Issue:''' The credentials aren't accepted
*: Check if your credentials work in general by trying the access via webbrowser: [https://lsdf02-webdav.urz.uni-heidelberg.de/].
*: If this doesn't work:
*:: → [[Registration/Login/Username| Check your Username]]
*:: → [[Registration/bwForCluster/Helix#Troubleshooting_with_the_Help_of_bwServices | Troubleshooting with bwServices]]
* '''Other Issue'''
** If available, follow the troubleshooting guide of your specific connection method.
** Make sure to not use LAN and WLAN at the same time to prevent connection problems.
** If you have an institutional account, make sure to be a fully active member of your institution.
If these suggestions didn't help, write to the [mailto:sds-hd-support@urz.uni-heidelberg.de support]. Provide the following information:
* Your operating system
* Does the access via webbrowser work?
*: If yes, provide us with detailed information on how you tried to access your SV (used access method, username, ...).

Python

2026-03-02T16:20:03Z

H Schumacher: New formatting, small text changes

Overview of pages related to the programming language Python.

→ '''[[Development/Python | Python Main Page]]'''

'''Cluster-specific pages:'''
* [[JUSTUS2/Software/Python | Python on JUSTUS 2]]

'''Instructions for JupyterLab on...'''
* [[BwUniCluster3.0/Jupyter | bwUniCluster3.0 ]]
* [[Helix/bwVisu/JupyterLab | Helix]] (through [[Helix/bwVisu | bwVisu]])
* [[BinAC2/Software/Jupyterlab | BinAC2]] (as software module)

'''Related software packages'''
* [[BwUniCluster3.0/Software/Python Dask | Dask]] (on bwUniCluster3.0)
* [[Development/ollama | ollama ]]

'''Training'''

* [https://github.com/hpcraink/workshop-parallel-jupyter Workshop - Parallel Python via Jupyter]

Helix/bwVisu/JupyterLab

2026-03-02T15:02:00Z

H Schumacher: Added fourth step to the kernel sharing instructions to fix permission issues.

[https://jupyter.org/ JupyterLab] is an integrated development environment (IDE) that provides a flexible and scalable interface for the Jupyter Notebook system. It supports interactive data science and scientific computing across over 40 programming languages (including Python, Julia, and R).

== Change Python Version ==

The default python version can be seen by running <code>python --version</code> in the terminal.

A different python version can be installed into a new virtual environment and then registered as IPython kernel for the usage in JupyterLab. This is explained in the chapter [[#Add_packages_via_conda_environments | add packages via conda environments]].

== Install Python Packages ==

Python packages can be added by installing them into a virtual environment and then creating an IPython kernel from the virtual environment. 
Kernels can be shared. See the notes below. 
If you want to move a virtual environment, it is adivsed to recreate it in the new place. Otherwise, dependencies based on relative paths will break.

# Create a virtual environment with...
#* [[#Add_packages_via_venv_virtual_environments | ...venv]] or
#* [[#Add_packages_via_Conda_virtual_environments | ...conda]] (choose this option if you want to install a different python version) or
#* ...[https://docs.astral.sh/uv/getting-started/ uv] if you want to install a different python version but don't want to use conda.
# [[#Create_an_IPython_Kernel | Create an IPython kernel]] from the virtual environment
# Use the kernel within JupyterLab
#* By default new kernels are saved under <code>~/.local/share/jupyter</code> and this location is automatically detected. Therefore, new kernels are directly available.
#*:[[File:BwVisu JuypterLab KernelPath.png|In the JupyterLab job configuration form, a custom kernel path can be provided.|right|thumb|x150px]]
#* If the kernel is saved somewhere else, the path can be provided in the "Kernel path" field when configuring the JupyterLab job (see image). For a kernel placed under <code>path_to_parent_dir/share/jupyter/kernels/my_kernel</code> the needed "Kernel path" would be <code>path_to_parent_dir/share/jupyter</code>.
# When the kernel is used the first time, the file <code>notebook_secrets</code> is created automatically. It can be found under "Kernel path". For others to use the kernel, they must have read access to this file. The command <code>chmod 750 notebook_secrets</code> would for example allow the whole SDS@hd SV read access. For the access management in workspaces, please see [[Workspace#Setting_Permissions_for_Sharing_Files| Wokspace Permissions]].

Notes regarding the sharing of IPython kernels
* The virtual environment and the kernel need to be placed in a shared directory. For example at SDS@hd.
* There could be a subdirectory for the virtual environments and one for the kernels.
* The path to the kernels is saved in the environment variable $JUPYTER_DATA_DIR. Jupyter relevant paths can be seen with <code>jupyter --paths</code>

=== Add packages via ''venv'' virtual environments ===
More information about venv or other python virtual environments can be found at the [[Development/Python | Python]] page.

Steps for creating a '''venv''' virtual environment:
<ol style="list-style-type: decimal;">
<li>Open a terminal.</li>
<li>Create a new virtual evironment:
<syntaxhighlight lang="bash">python3 -m venv <env_parent_dir>/<env_name></syntaxhighlight>
* <code>env_parent_dir</code> is the path to the folder where the virtual environment shall be created. Relative paths can be used.
* Caution: If you you want to share the environment with others, make sure to already create it in the shared place.
</li>
<li>Activate the environment:
<syntaxhighlight lang="bash">source <env_parent_dir>/<env_name>/bin/activate</syntaxhighlight></li>
<li>Update pip and install packages:
<syntaxhighlight lang="bash">pip install -U pip
pip install <packagename></syntaxhighlight></li>
<li> [[#Create_an_IPython_Kernel | Create an IPython kernel]]</li>
</ol>

=== Add packages via Conda virtual environments ===
More information about using conda can be found at the [[Development/Conda | Conda]] page.

<ol style="list-style-type: decimal;">
<li>Load the miniforge module by clicking first on the blue hexagon icon on the left-hand side of Jupyter's start page and then on the "load" button right of the entry for miniforge in the software module menu.
<li>Open a terminal.</li>
<li>Create a new virtual environment:
* If you are the only person using the environment, you can install it in your home directory:
<syntaxhighlight lang="bash">conda create --name <env_name> python=<python version></syntaxhighlight>
* If you want to install it into a different directory, for example a shared place:
<syntaxhighlight lang="bash">conda create --prefix <path_to_shared_directory>/<env_name> python=<python version>
</syntaxhighlight>
</li>
<li>Activate your environment:
<syntaxhighlight lang="bash">conda activate <myenv></syntaxhighlight></li>
<li>Install your packages:
<syntaxhighlight lang="bash">conda install <mypackage></syntaxhighlight></li>
<li> [[#Create_an_IPython_Kernel | Create an IPython kernel]]</li>
</ol>

== Create an IPython Kernel ==

Python kernels are implementations of the Jupyter notebook environment for different languages or virtual environments. You can switch between kernels easily, allowing you to use the best tool for a specific task.
conda_kernels

=== Create a kernel from a virtual environment ===

<ol>
<li>Activate the virtual environment.</li>
<li>Install the ipykernel package:
<syntaxhighlight lang="bash">pip install ipykernel</syntaxhighlight></li>
<li>Register the virtual environment as custom kernel to Jupyter.
* If you are the only person using the environment:
<syntaxhighlight lang="bash">python3 -m ipykernel install --user --name=<kernel_name></syntaxhighlight>
The kernel can be found under <code>~/.local/share/jupyter/kernels/</code>.
* If you installed the environment in a shared place and want to have the kernel there as well:
<syntaxhighlight lang="bash">python3 -m ipykernel install --prefix <path_to_kernel_folder> --name=<kernel_name></syntaxhighlight>
The kernel can then be found under <code>path_to_kernel_folder/share/jupyter/kernels/<kernel_name></code>. As long as the same <code>path_to_kernel_folder</code> is used, all kernels will be saved next to each other in "kernels".
</li>
</ol>

=== Multi-Language Support ===

JupyterLab supports over 40 programming languages including Python, R, Julia, and Scala. This is achieved through the use of different kernels.

==== R Kernel ====

<ol style="list-style-type: decimal;">
<li>On the cluster:
<pre>$ module load math/R
$ R
> install.packages('IRkernel')</pre></li>
<li>On bwVisu:
<ol style="list-style-type: decimal;">
<li>Start Jupyter App</li>
<li>In left menu: load math/R</li>
<li>Open Console:</li>

<pre>$ R
> IRkernel::installspec(displayname = 'R 4.2')</pre>

<li>Start kernel 'R 4.2' as console or notebook</li>
</ol>
</ol>

==== Julia Kernel ====

Load the math/julia module. Open the Terminal.

<pre>
julia
]
add IJulia
</pre>

After that, Julia is available as a kernel.

== Interactive Widgets ==

JupyterLab supports interactive widgets that can create UI controls for interactive data visualization and manipulation within the notebooks. Example of using an interactive widget:

<pre class="{.python">from ipywidgets import IntSlider
slider = IntSlider()
display(slider)</pre>
These widgets can be sliders, dropdowns, buttons, etc., which can be connected to Python code running in the backend.

== FAQ ==

<ol><li>'''My conda commands are interrupted with message 'Killed'.'''
Request more memory when starting Jupyter.</li>
<li>'''How can I navigate to my SDS@hd folder in the file browser?'''
Open a terminal and set a symbolic link to your SDS@hd folder in your home directory. For example:</li>
<pre>cd $HOME
mkdir sds-hd
cd sds-hd
ln -s /mnt/sds-hd/sd16a001 sd16a001</pre>
<li>'''Jupyterlab doesn't let me in but asks for a password.'''
Try using more memory for the job. If this doesn't help, try using the inkognito mode of your browser as the browser cache might be the problem.</li>
<li>'''I prefer VSCode over JupyterLab. Can I start a JupyterLab job and then connect with it via VSCode?'''
This is not possible. Please start the job directly on Helix instead. You can find the instructions at the [[Development/VS_Code#Connect_to_Remote_Jupyter_Kernel | VSCode page]].</li>
</ol>

Helix/bwVisu/Usage

2026-02-27T14:21:51Z

H Schumacher: Added small title for sds section

== Basic Navigation ==

bwVisu is quite simple and intuitively designed to make launching apps on the cluster as user-friendly as possible. Once you have successfully logged into bwVisu, the first thing you see is a list of pinned apps. These are the available apps that you can launch as a job on the cluster.

In the navigation bar, the user has the following options:

<code>Files</code> — <code>Interactive Apps</code> — <code>My Interactive Sessions</code> — <code>Shell</code>

=== Files ===

Here, you can access your home directory on Helix. In a graphical user interface, files can be downloaded and uploaded, and new files and folders can be created. You will encounter an error message if you try to see data that you have no access to.

In the Home Directory the results and logs from the apps are stored. You get the following interface:

[[File:BwVisuFiles.png|bwVisu-files|center|x300px]]

'''For SDS@hd users:''' Data on SDS@hd (under /mnt/sds-hd) can only be accessed from within jobs. Therefore, you need to open a terminal (within a JupyterLab or Rstudio job or directly on Helix) and set a symbolic link to your SDS@hd folder in your home directory. For example like this:
<pre>cd $HOME
mkdir sds-hd
cd sds-hd
ln -s /mnt/sds-hd/sd16a001 sd16a001</pre>

=== Interactive Apps ===

This displays the apps that the user can run on the cluster, possibly across multiple nodes. These apps initiate an interactive session which the user can directly connect to and, depending on the app, receive a graphical interface.

=== My Interactive Sessions ===

At bwVisu, it is possible to start several interactive apps simultaneously. Here, one can view an overview of currently running interactive apps and completed sessions. It is also possible to view the log for each session. As soon as one starts an interactive app, one is automatically redirected here. Even if you log out of bwVisu and log back in, you can still see the information and logs from the finished interactive apps here until you delete it.

=== Shell ===
For bwForCluster Helix users only: After entering your OTP and password a shell on the login node of bwForCluster Helix opens. This way, you can for example create workspaces when you need more space. Please look up the details under [[Helix/Filesystems#Workspaces | Helix Workspaces]]. They can't be seen in the File Viewer but are available within the bwVisu jobs.

== Start a Job ==
[[File:bwVisuConcept.png|center|x170px]]
<ol>
<li> At the start page of the web frontend, you can start a new job by clicking on an application version in the list of pinned apps.</li>
<li>You will now see the application description and options to adjust the job resources.</li>
<ul>
<li> To find the best resource settings for your jobs, please have a look at the [https://helix-monitoring.bwservices.uni-heidelberg.de/ bwForCluster Helix monitoring portal] . There, you can see how much of the requested resources where actually used by your jobs.
<li> At the bottom of the page, beneath the <code>Launch</code> button, you can find a link for accessing the logs/session data of all your jobs.</li>
<li>The interface can differ per app because each app can have individual configuration options. </li>
[[File:bwVisuRStudio.png|bwVisu-rstudio|center|x200px]]</li>
</ul>
<li>Click the <code>Launch</code> button to start a new job with this application. You can't start two jobs with the same application. Launching an app can take some time depending on the requested resources and cluster utilization.</li>
<li>The <code>Interactive Sessions</code> page opens with the details of your job.</li>
<ul>
<li>'''Host:''' Indicates on which nodes the application is running.</li>
<li>'''Time Remaining:''' Each application is given a time limit, which can also be set in advance. In this example, the application is supposed to run for one hour.</li>
<li>'''Session ID:''' This is a link that leads to the folder of the session. Among other things, application logs are displayed here, which can be viewed in the UI.
[[File:BwVisuRstudioRun.png|bwVisu-rstudio-run|center|x200px]] </li>
</li>
</ul>
<li> Connect to the job with your browser by clicking <code>connect to <application></code>. </li>
</ol>

== Adjust noVNC Applications ==
<div><ul>
<li style="display: inline-block; vertical-align: middle; float: right;"> [[File:bwVisu Fiji noVNCMenu.png|thumb|none|x200px|noVNC menu within Fiji]] </li>
<li style="display: inline-block; vertical-align: top; float: right;"> [[File:bwVisu_Fiji_Running.png|thumb|none|x200px|Job details view of a running Fiji Job]] </li>
</ul></div>
noVNC is an open-source VNC browser client. Some bwVisu applications use noVNC and therefore, come with additional options in the view of the job details and an additional menu within the running application.

*'''Running Job View''' on the interactive sessions page
** Compression slider: Before opening the application, you can choose the compression level to use. 0 means no compression while 9 offers the best compression but uses more CPU resources on the cluster which can be slower.
** Image Quality slider: Before opening the application, you can choose the image quality level. With this you can control the desired JPEG quality.
** View Only (Share-able Link) button: By clicking this button the application opens in view only mode. You can share this link with collaborators or students while you launch your noVNC application normally for yourself. This way, others can follow what you are doing. For every job you need to generate a new link.
* '''noVNC menu''' within the running application
** When you open the application you can see an additional menu on the left with options. For the ones using the application in view-only mode it is helpful to select the option to show a dot instead of not showing the mouse cursor at all. Therefore go to "Options" and then "Expand".
** For copying data, you need to use the clipboard. When copying something locally, you need to put it into the clipboard before you can use it within the application. The same goes the other way around.
* '''Adjust window size''' and position of the app by executing a right click at the edge of the app window frame, so that a menu shows up.

== Terminate a Job ==

The job will terminate when the job runtime requested in the job submission form has expired To terminate an application yourself, you can click the delete button or end the application in the interactive session. If the app is terminated in the interactive session, the window no longer shows <code>Running</code> but <code>Completed</code>. With the <code>Delete</code> button, you end the session and also remove the list entry from <code>My Interactive Sessions</code>.

== Troubleshooting ==
In case of problems, it can help to look up the related bwForCluster Helix wiki pages.
For example:
* Error: Disk space exceeded → '''[https://wiki.bwhpc.de/e/Helix/Filesystems#Troubleshooting Filesystems: Troubleshooting]'''

Workspace

2026-02-27T12:13:55Z

H Schumacher: Added "groups" command

'''Workspace tools''' provide temporary scratch space so called '''workspaces''' for your calculation on a central file storage. They are meant to keep data for a limited time – but usually longer than the time of a single job run.

== No Backup ==

Workspaces are not meant for permanent storage, hence data in workspaces is not backed up and may be lost in case of problems on the storage system. Please copy/move important results to $HOME or storage space outside the cluster.

== Create workspace ==
To create a workspace you need to state
* ''name'' of your workspace and
* ''lifetime'' in days.
A maximum value for ''lifetime'' and a maximum number of renewals is defined on each cluster. Execution of:

$ ws_allocate mySpace 30

e.g. returns:

Workspace created. Duration is 720 hours.
Further extensions available: 3
/work/workspace/scratch/username-mySpace-0

For more information read the program's help, i.e. ''$ ws_allocate -h''.

== List all your workspaces ==

To list all your workspaces, execute:

$ ws_list

which will return:
* Workspace ID
* Workspace location
* available extensions
* creation date and remaining time

To list expired workspaces, see [[Workspace#Restore_an_Expired_Workspace|Restore an Expired Workspace]].

== Find workspace location ==

Workspace location/path can be prompted for any workspace ''ID'' using '''ws_find''', in case of workspace ''mySpace'':

$ ws_find mySpace

returns the one-liner:

/work/workspace/scratch/username-mySpace-0

== Extend lifetime of your workspace ==

Any workspace's lifetime can be only extended a cluster-specific number of times. There several commands to extend workspace lifetime
#<pre>$ ws_extend mySpace 40</pre> which extends workspace ID ''mySpace'' by ''40'' days from now,
#<pre>$ ws_extend mySpace</pre> which extends workspace ID ''mySpace'' by the number days used previously
#<pre>$ ws_allocate -x mySpace 40</pre> which extends workspace ID ''mySpace'' by ''40'' days from now.
 

== Setting Permissions for Sharing Files ==
The examples will assume you want to change the directory in $DIR. If you want to share a workspace, DIR could be set with <code>DIR=$(ws_find my_workspace)</code>

=== Workspace Tools ===

* ws_share
<code syntax=bash>ws_share share workspacename username</code>

allows you to grant the user username read access to the workspace.

Newer versions of the workspace tools have sharing options to ws_allocate:

* -G option of ws_allocate
<code syntax=bash>ws_allocate -G groupname workspacename duration</code>

e.g. for the group bw16e001 and for the workspace for 30 days:

<code syntax=bash>ws_allocate -G bw16e001 commonfiles 30</code>

** groupname: name of the group you want to share with
** workspacename: what you want to call your workspace
** duration: how long the workspace is supposed to last in days

Essentially this tool sets regular unix rwx permissions for the group plus the "suid" bit on the directory to make the permission inheritable.

=== Regular Unix Permissions ===

Making workspaces world readable/writable using standard unix access rights with <tt>chmod</tt> is only feasible if you are in a research group and you and your co-workers share a common unix group (for example "bw22a222").

Do '''not''' make files readable or even writable to everyone or to large common groups ("all students").
You can see available groups with the command <code>groups</code>.

{| class="wikitable"
|-
!style="width:45%" | Command
!style="width:55%" | Action
|-
|<tt>chgrp -R bw16e001 "$DIR"</tt>
<tt>chmod -R g+rX "$DIR"</tt>
|Set group ownership and grant read access to group for files in workspace via unix rights to the group "bw16e001" (has to be re-done if files are added)
|-
|<tt>chgrp -R bw16e001 "$DIR"</tt>
<tt>chmod -R g+rswX "$DIR"</tt>
|Set group ownership and grant read/write access to group for files in workspace via unix rights (has to be re-done if files are added). Group will be inherited by new files, but rights for the group will have to be re-set with chmod for every new file
|-
|}

Options used:
* -R: recursive
* g+rwx
** g: group
** + add permissions (- to remove)
** rwx: read, write, execute

=== ACLs: Access Control Lists ===
ACLs allow a much more detailed distribution of permissions but are a bit more complicated and not visible in detail via "ls". They have the additional advantage that you can set a "default" ACL for a directory, (with a <tt>-d</tt> flag or a <tt>d:</tt> prefix) which will cause all newly created files to inherit the ACLs from the directory. Regular unix permissions only have limited support (only group ownership, not access rights) for this via the suid bit.

Best practices with respect to ACL usage:
# Take into account that ACLs take precedence over standard unix access rights
# The owner of a workspace is responsible for its content and management

Please note that <tt>ls</tt> (List directory contents) shows ACLs on directories and files only when run as <tt>ls -l</tt> as in long format, as "plus" sign after the standard unix access rights.

Examples with regard to "my_workspace":
{| class="wikitable"
|-
!style="width:45%" | Command
!style="width:55%" | Action
|-
|<tt>getfacl "$DIR"</tt>
|List access rights on $DIR
|-
|<tt>setfacl -Rm user:fr_xy1:rX,default:user:fr_xy1:rX "$DIR"</tt>
|Grant user "fr_xy1" read-only access to $DIR
|-
|<tt>setfacl -R -m user:fr_me0000:rwX,default:user:fr_me0000:rwX "$DIR"</tt>
<tt>setfacl -R -m user:fr_xy1:rwX,default:user:fr_xy1:rwX "$DIR"</tt>
|Grant your own user "fr_me0000" and "fr_xy1" inheritable ("default") read and write access to $DIR, so you can also read/write files put into the workspace by a coworker
|-
|<tt>setfacl -Rm group:bw16e001:rX,default:group:bw16e001:rX "$DIR"</tt>
|Grant group (Rechenvorhaben) "bw16e001" read-only access to $DIR
|-
|<tt>setfacl -Rb "$DIR"</tt>
|Remove all ACL rights. Standard Unix access rights apply again.
|}

Options used:
* -R: recursive
* -m: modify
* user:username:rwX user: next name is a user; rwX read, write, eXecute (only where execute is set for user)
* default:[user|group] set the default for user or group for new files or dierctories

== Delete a Workspace ==

$ ws_release mySpace # Manually erase your workspace mySpace

Note: workspaces are kept for some time after release. To immediately delete and free space e.g. for quota reasons, delete the files with rm before release.

Newer versions of workspace tools have a --delete-data flag that immediately deletes data. Note that deleted data from workspaces is permanently lost.

== Restore an Expired Workspace ==

For a certain (system-specific) grace time following workspace expiration, a workspace can be restored by performing the following steps:

(1) Display restorable workspaces.
ws_restore -l

(2) Create a new workspace as the target for the restore:
ws_allocate restored 60

(3) Restore:
ws_restore <full_name_of_expired_workspace> restored

The expired workspace has to be specified using the '''full name''', including username prefix and timestamp suffix (otherwise, it cannot be uniquely identified).
The target workspace, on the other hand, must be given with just its short name as listed by <code>ws_list</code>, without the username prefix.

If the workspace is no visible/restorable, it has been '''permanently deleted''' and cannot be restored, not even by us. Please always remember, that workspaces are intended solely for temporary work data, and there is no backup of data in the workspaces.

Helix/Slurm

2026-02-27T00:29:09Z

H Schumacher: Added info about --begin paramer

= General information about Slurm =
The bwForCluster Helix uses Slurm as batch system.
* Slurm documentation: https://slurm.schedmd.com/documentation.html
* Slurm cheat sheet: https://slurm.schedmd.com/pdfs/summary.pdf
* Slurm tutorials: https://slurm.schedmd.com/tutorials.html

= Slurm Command Overview =

{| width=750px class="wikitable"
! Slurm commands !! Brief explanation
|-
| [https://slurm.schedmd.com/sbatch.html sbatch] || Submits a job and queues it in an input queue
|-
| [https://slurm.schedmd.com/salloc.html salloc] || Request resources for an interactive job
|-
| [https://slurm.schedmd.com/squeue.html squeue] || Displays information about active, eligible, blocked, and/or recently completed jobs
|-
| [https://slurm.schedmd.com/scontrol.html scontrol] || Displays detailed job state information
|-
| [https://slurm.schedmd.com/scontrol.html sstat] || Displays status information about a running job
|-
| [https://slurm.schedmd.com/scancel.html scancel] || Cancels a job
|-
|}

= Job Submission =

Batch jobs are submitted with the command:

<syntaxhighlight lang="bash">$ sbatch <job-script> </syntaxhighlight>

A job script contains options for Slurm in lines beginning with #SBATCH as well as your commands which you want to execute on the compute nodes. For example:

<syntaxhighlight lang="slurm">#!/bin/bash
#SBATCH --partition=cpu-single
#SBATCH --ntasks=1
#SBATCH --time=00:20:00
#SBATCH --mem=1gb
#SBATCH --export=NONE
echo 'Hello world'
</syntaxhighlight>

This jobs requests one core (--ntasks=1) and 1 GB memory (--mem=1gb) for 20 minutes (--time=00:20:00) on nodes provided by the partition 'cpu-single'.

For the sake of a better reproducibility of jobs it is recommended to use the option --export=NONE to prevent the propagation of environment variables from the submit session into the job environment and to load required software modules in the job script.

== Partitions ==

On bwForCluster Helix it is necessary to request a partition with "--partition=<partition_name>" on job submission. Within a partition job allocations are routed automatically to the most suitable compute node(s) for the requested resources (e.g. amount of nodes and cores, memory, number of GPUs). The devel partition is the default partition, if no partition is requested.

The partitions devel, cpu-single and gpu-single are operated in shared mode, i.e. jobs from different users can run on the same node. Jobs can get exclusive access to compute nodes in these partitions with the "--exclusive" option. The partitions cpu-multi and gpu-multi are operated in exclusive mode. Jobs in these partitions automatically get exclusive access to the requested compute nodes.

{| class="wikitable"
|-
! style="width:20%"| Partition
! style="width:20%"| Node Access Policy
! style="width:20%"| [https://wiki.bwhpc.de/e/Helix/Hardware#Compute_Nodes Node Types]
! style="width:20%"| Default
! style="width:20%"| Limits
|-
| devel
| shared
| cpu, gpu4
| ntasks=1, time=00:10:00, mem-per-cpu=2gb
| nodes=2, time=00:30:00
|-
| cpu-single
| shared
| cpu, fat
| ntasks=1, time=00:30:00, mem-per-cpu=2gb
| nodes=1, time=120:00:00
|-
| gpu-single
| shared
| gpu4, gpu8
| ntasks=1, time=00:30:00, mem-per-cpu=2gb
| nodes=1, time=120:00:00
|-
| cpu-multi
| job exclusive
| cpu
| nodes=2, time=00:30:00
| nodes=32, time=48:00:00
|-
| gpu-multi
| job exclusive
| gpu4
| nodes=2, time=00:30:00
| nodes=8, time=48:00:00
|}

== GPU requests ==

For the partitions gpu-single and gpu-multi is it required to request GPU ressources.
* The number of GPUs is requested with the option "--gres=gpu:<number-of-gpus>".
* A specific GPU type can be requested with the option "--gres=gpu:<gpu-type>:<number-of-gpus>". Possible values for <gpu-type> are listed in the line 'GPU Type' of the [https://wiki.bwhpc.de/e/Helix/Hardware#Compute_Nodes Compute Nodes table].
* GPUs that are suitable for a specific GPU memory requirement can be requested with option "--gres=gpu:<number-of-gpus>,gpumem_per_gpu:<required-gpumem>GB". This only restricts the selection of possible GPU types. For the job the total GPU memory per GPU is available as listed in the line 'GPU memory per GPU' of the [https://wiki.bwhpc.de/e/Helix/Hardware#Compute_Nodes Compute Nodes table].

== Constraints ==

It is possible to refine the resource request for a job with the option "--constraint=<feature>".

{| class="wikitable"
|-
! style="width:20%"| Feature
! style="width:80%"| Meaning
|-
| fp64
| request GPU types with FP64 capability (double precision)
|}

== Examples ==

Here you can find some example scripts for batch jobs.

=== Serial Programs ===

<syntaxhighlight lang="slurm">
#!/bin/bash
#SBATCH --partition=cpu-single
#SBATCH --ntasks=1
#SBATCH --time=20:00:00
#SBATCH --mem=4gb
./my_serial_program
</syntaxhighlight>

'''Notes:'''
* Jobs with "--mem" up to 236gb can run on all node types associated with the cpu-single partition.

=== Multi-threaded Programs ===

<syntaxhighlight lang="slurm">
#!/bin/bash
#SBATCH --partition=cpu-single
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=16
#SBATCH --time=01:30:00
#SBATCH --mem=50gb
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
./my_multithreaded_program
</syntaxhighlight>

'''Notes:'''
* Jobs with "--ntasks-per-node" up to 64 and "--mem" up to 236gb can run on all node types associated with the cpu-single partition.
* With "export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}" you can set the number of threads according to the number of resources requested.

=== MPI Programs ===

<syntaxhighlight lang="slurm">
#!/bin/bash
#SBATCH --partition=cpu-multi
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=64
#SBATCH --time=12:00:00
module load compiler/gnu
module load mpi/openmpi
srun ./my_mpi_program
</syntaxhighlight>

'''Notes:'''
* "--mem" requests the memory per node. The maximum is 236gb.
* The Compiler and MPI modules used for the compilation must be loaded before the start of the program.
* It is recommended to start MPI programs with 'srun'.

=== GPU Programs ===

<syntaxhighlight lang="slurm">
#!/bin/bash
#SBATCH --partition=gpu-single
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --gres=gpu:A40:1
#SBATCH --time=12:00:00
#SBATCH --mem=16gb

module load devel/cuda
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
./my_cuda_program
</syntaxhighlight>

'''Notes:'''
* The number of GPUs per node is requested with the option "--gres=gpu:<number-of-gpus>"
* It is recommended to request a suitable GPU type for your application with the option "--gres=gpu:<gpu-type>:<number-of-gpus>". For <gpu-type> put the 'GPU Type' listed in the [https://wiki.bwhpc.de/e/Helix/Hardware#Compute_Nodes Compute Nodes table].
** Example for a request of two A40 GPUs: --gres=gpu:A40:2
** Example for a request of one A100 GPU: --gres=gpu:A100:1
* If you are unsure on which GPU type your code runs faster, please run a test case and compare the run times. In general the following applies:
** A40 GPUs are optimized for single precision computations.
** A100 and H200 GPUs offer better performance for double precision computations or if the code makes use of tensor cores.
* The CUDA module used for compilation must be loaded before the start of the program.

=== More examples ===

Further batch script examples are available on bwForCluster Helix in the directory: <code>/opt/bwhpc/common/system/slurm-examples</code>

= Interactive Jobs =

Interactive jobs must NOT run on the login nodes, however resources for interactive jobs can be requested using srun. The following example requests an interactive session on 1 core for 2 hours:

<syntaxhighlight lang="bash">$ salloc --partition=cpu-single --ntasks=1 --time=2:00:00 </syntaxhighlight>

After execution of this command wait until the queueing system has granted you the requested resources. Once granted you will be automatically logged on the allocated compute node.

If you use applications or tools which provide a GUI, enable X-forwarding for your interactive session with:

<syntaxhighlight lang="bash">$ salloc --partition=cpu-single --ntasks=1 --time=2:00:00 --x11 </syntaxhighlight>

Once the walltime limit has been reached you will be automatically logged out from the compute node.

For convenient access to specific GUI applications (JupyterLab, RStudio, ...) on the cluster, we provide a web-based platform: '''[[Helix/bwVisu | bwVisu]]'''

'''Delay the start of a job''' by using [https://slurm.schedmd.com/sbatch.html#OPT_begin -b] (or [https://slurm.schedmd.com/sbatch.html#OPT_begin --begin]) option in order to defer the allocation of the job until the specified time. Examples:
<syntaxhighlight lang="bash">$ salloc --begin=20:00 ... # job can start after 8 p.m.
$ salloc --begin=now+1hour ... # job can start 1 hour after submission
$ salloc --begin=teatime ... # job can start at teatime (4 p.m.)
$ salloc --begin=2023-12-24T20:00:00 ... # job can start after specified date/time
</syntaxhighlight>

= Job Monitoring =

== Information about submitted jobs ==

For an overview of your submitted jobs use the command:

<syntaxhighlight lang="bash">$ squeue
# Additionally show the estimated start time:
$ squeue --start</syntaxhighlight>

To get detailed information about a specific job use the command:

<syntaxhighlight lang="bash">$ scontrol show job <jobid></syntaxhighlight>

'''Notes:''' 
A job start may be delayed for various reasons:
* (QOSMaxCpuPerUserLimit) - There is a limit to how many CPU cores a user can use at the same time. The job exceeds this limit.
* (QOSMaxGRESPerUser) - There is a limit to how many GPUs a user can use at the same time. The job exceeds this limit.
* (QOSMinGRES) - The job was submitted to a gpu partition without requesting a GPU.
* (launch failed requeued held) - The job has failed to start. You may be able to resume it using scontrol. Alternatively you can cancel it and submit it again.
For further reasons please refer to: https://slurm.schedmd.com/job_reason_codes.html

== Information about resource usage of running jobs ==

You can monitor the resource usage of running jobs with the sstat command. For example:

<pre>
$ sstat --format=JobId,AveCPU,AveRSS,MaxRSS -j <jobid>
</pre>

This will show average CPU time, average and maximum memory consumption of all tasks in the running job.

'sstat -e' command shows a list of fields that can be specified with the '--format' option.

== Interactive access to running jobs ==

It is also possible to attach an interactive shell to a running job with command:

<syntaxhighlight lang="bash">$ srun --jobid=<jobid> --overlap --pty /bin/bash</syntaxhighlight>

Commands like 'top' show you the most busy processes on the node. To exit 'top' type 'q'.

To monitor your GPU processes use the command 'nvidia-smi'.

== Job Feedback ==
You get feedback on resource usage and job efficiency for completed jobs with the command:

<syntaxhighlight lang="bash">
$ seff <jobid>
</syntaxhighlight>

Example Output:

<syntaxhighlight lang="bash">
============================= JOB FEEDBACK =============================
Job ID: 12345678
Cluster: helix
User/Group: hd_ab123/hd_hd
State: COMPLETED (exit code 0)
Nodes: 2
Cores per node: 64
CPU Utilized: 3-04:11:46
CPU Efficiency: 97.90% of 3-05:49:52 core-walltime
Job Wall-clock time: 00:36:29
Memory Utilized: 432.74 GB (estimated maximum)
Memory Efficiency: 85.96% of 503.42 GB (251.71 GB/node)
</syntaxhighlight>

Explanation:
* Nodes: Number of allocated nodes for the job.
* Cores per node: Number of physical cores per node allocated for the job.
* CPU Utilized: Sum of utilized core time.
* CPU Efficiency: 'CPU Utilized' with respect to core-walltime (= 'Nodes' x 'Cores per node' x 'Job Wall-clock time') in percent.
* Job Wall-clock time: runtime of the job.
* Memory Utilized: Sum of memory used. For multi node MPI jobs the sum is only correct when srun is used instead of mpirun.
* Memory Efficiency: 'Memory Utilized' with respect to total allocated memory for the job.

== Job Monitoring Portal ==
For more detailed information about your jobs visit the job monitoring portal: https://helix-monitoring.bwservices.uni-heidelberg.de

= Accounting =

Jobs are billed for allocated CPU cores, memory and GPUs.

To see the accounting data of a specific job:

<syntaxhighlight lang="bash">$ sacct -j <jobid> --format=user,jobid,account,nnodes,ncpus,time,elapsed,AllocTRES%50</syntaxhighlight>

To retrive the job history for a specific user for a certain time frame:

<syntaxhighlight lang="bash">$ sacct -u <user> -S 2022-08-20 -E 2022-08-30 --format=user,jobid,account,nnodes,ncpus,time,elapsed,AllocTRES%50</syntaxhighlight>

= Overview about free resources =

On the login nodes the following command shows what resources are available for immediate use:

<syntaxhighlight lang="bash">$ sinfo_t_idle</syntaxhighlight>