BwUniCluster3.0/Login/Client

From bwHPC Wiki
< BwUniCluster3.0‎ | Login
Revision as of 09:43, 28 March 2025 by S Braun (talk | contribs)
Jump to navigation Jump to search

SSH Clients

Only Secure Shell (SSH) is permitted for login to the bwHPC clusters. On order to establish a SSH connection, the users need a SSH client installed on their device. The following clients are suitable for connecting to the bwHPC clusters.

Pre-installed SSH Clients

All common operation systems, such as Microsoft Windows, Linux, Mac OS or other Unix-like operating systems come with a built-in SSH client, most likely provided by the OpenSSH project.

To use this client, simply open a command line terminal. Depending on your operating system, the required application is called Terminal, Console, PowerShell, Windows-Terminal or similar. Inside this terminal, enter the following command to connect to a cluster:

$ ssh <username>@<hostname>

  • <username> is your local username including the prefix (cf. Login Username)
  • <hostname> is the hostname of the cluster you want to connect to (cf. Login Hostnames)

For KIT user "ab1234" who wants to connect to bwUniCluster 3.0, this would yield:

$ ssh ka_ab1234@uc3.scc.kit.edu

For a more detailed explanation on how to do SSH login, please refer to Login Examples.

File Transfer

All native SSH clients already come with a remote file copy application: scp.
For a more detailed explanation on how to do data transfers with scp, please refer to Data_Transfer/SCP.
Further alternatives for file and data exchange are available Here: Data Transfer

X11 Forwarding

If you are on a Linux or Unix system and want to use a GUI-based application, you can use the -X option for the ssh command to set up X11 forwarding:

$ ssh -X <username>@<hostname>

If you want to use X11 forwarding on Microsoft Windows, please refer to MobaXterm.

Alternative SSH Clients

Using Microsoft Windows, some handy features of Unix-based operating systems are lacking, such as X11 forwarding or file transfer tools like rsync. Therfore, we suggest to have a look at the following tools.

MobaXterm

If the built-in SSH functionalities on Windows are not sufficient, the bwHPC support team recommends to use MobaXterm.
MobaXterm provides a built-in X11 server allowing to start GUI based software. It also supports graphical file transfer via drag & drop, although we explicitly advise against this, especially when transferring more than a few files.

Installation

MobaXterm can either be installed or used as portable software without installation. Please refer to MobaXterm Home Edition.

Set up a SSH session

Please have a look at the "SSH client" tab in the MobaXterm live demo.

The steps to define a SSH session are the following:

  1. Click the "Session" button in the top task bar or "Sessions">"New Session" in the menu
  2. Select the session type "SSH" in the Session settings
  3. Fill in the following fields:
    Remote host: <hostname>
    Specify username: <username>
    Port  : 22
  4. Click OK

Start a SSH session

After setting up a SSH session, you can connect to the respective server by double-clicking the entry in the "Sessions" list or by right-clicking and "Execute" the entry. A terminal will be opened, here you can enter your credentials, TOTP and service password. Now your are logged in to the login node of the respective server.

File Transfers

Files can be transferred between the local system and the cluster by navigating to the respective folders in the split file view and then either drag & drop files and folders between the views or by clicking on a file/folder with the right mouse button and then selecting "Upload" or "Download" from the menu.

Configuration needed!
With recent MobaXterm versions, the following configuration change has to be made in advance:
In the top menu bar click: "Settings">"Configuration">"SSH", switch the "SSH engine" from "<new>" to "<legacy>". Then restart MobaXterm.

Attention!
We explicitly advise against transferring large files, many files or directories by means of drag & drop or via the Upload/Download functionality. Instead, MobaXterm comes with the rsync functionality, which supports incremental file transfer over SSH. Just open a local terminal in MobaXterm and navigate to the directory inside the Windows file system you want to synchronize. Use rsync from here.

Windows Subsystem for Linux

Windows Subsystem for Linux (WSL) is not a SSH client but a very tightly integrated, fully-fledged Linux operating system that can be installed and used like a normal Windows application.

After installing a Linux distribution of choice, you can use the instructions for Linux, e.g. for SSH connections, file transfers, X11 forwarding and other tasks.

WSL is for you if you have to use a Windows computer, e.g. provided by your employer, and if you want to benefit from all the tools and features for code development, debugging and connectivity to HPC systems that Linux provides.

For installing WSL and a Linux distribution, pelase refer to Windows Subsystem for Linux.

Data Transfer

Data Transfer


WebDAV

WebDAV access to the bwUniCluster-Filesystems

Access via Webbrowser (read-only)

Visit uc3-webdav.scc.kit.edu and login with your UC3 username and password (not the HPC service password). Here you can get an overview of your data and download single files. To be able to do more, like uploading new files, or downloading complete folders, a suitable client is needed (rclone, davix).

Access via command line tools

The authentication to use command line tools on the bwUniCluster is implemented via the OAuth2-Bearer-Token method. The Bearer tokens can be created with the oidc-agent [2]. This is available for Linux, MacOs and Windows [3]. [2] https://indigo-dc.gitbook.io/oidc-agent/ [3] https://github.com/indigo-dc/oidc-agent#installation

Example to create a OAuth2-Bearer-Token for using the bwUniCluster with oidc-gen:

oidc-gen <your shortname> --client-id unicluster_public --client-secret public --issuer https://bwidm.scc.kit.edu/oidc/realms/bwidm/ --redirect-uri http://localhost:<port-number>/biwdm.scc.kit.edu --scope uc3-webdav

With the command oidc-token you can show or use the token:

oidc-token <your shortname>


rclone

Rclone is one of the tools for transferring files from/to the HPC Filesystem via WebDAV. It's detection of unchanged files, parallel transfers and solid retry behaviour make it to a good choice. Rclone can be installed on Linux, MacOS and Windows. [1] https://rclone.org

To configure the rclone remote for the bwUniCluster, you will need to use the following URL, your account and your oidc-token shortname: url = https://oauth2.uc3-webdav.scc.kit.edu

The rclone remote can be configured by command line: rclone config create <your uc3-webdav name> webdav url https://oauth2.uc3-webdav.scc.kit.edu vendor other user <account> bearer_token_command "oidc-token <your oidc-token shortname>"

For the interactive setup process with rclone config see and use the bearer_token_command "oidc-token <your oidc-token shortname>" instead of the password.

The syntax to use Rclone is like this: rclone [options] subcommand <parameters> <parameters...>

Copies from local path to the remote path: rclone copy </local/path> <your uc3-webdav name>:/home/<oe>/<group>/<user>/<remote path>

Copies from remote path to the local path rclone copy <your uc3-webdav name>:/home/<oe>/<group>/<user>/<local path>

For more Details about rclone look at Data Transfer/Rclone or https://rclone.org.


curl

Because WebDAV extends HTTP, any HTTP client can access files in the HPC cluster. Given the full URL to a file, Curl and co can download a file or upload a file.

File-Download: curl -H "Authorization: Bearer `oidc-token <your shortname>`" --output <local filename> https://oauth2.uc3-webdav.scc.kit.edu/home/<oe>/<group>/<user>/<remote filename>

File-Upload: curl -H "Authorization: Bearer `oidc-token <your shortname>`" --silent --upload-file <local filename> https://oauth2.uc3-webdav.scc.kit.edu/home/<oe>/<group>/<user>/<remote filename>


davix

Davix is a set of tools to perform primitive operations via WebDAV protocol. It is shipped with most Linux distributions.

Some examples to list or transfer files are:

davix-ls -H "Authorization: Bearer `oidc-token <your shortname>`" https://oauth2.uc3-webdav.scc.kit.edu/home/<oe>/<group>/<user>/my-data-set/

davix-put -H "Authorization: Bearer `oidc-token <your shortname>`" <local filename> https://oauth2.uc3-webdav.scc.kit.edu/home/<oe>/<group>/<user>/<remote filename>

davix-get -H "Authorization: Bearer `oidc-token <your shortname>`" https://oauth2.uc3-webdav.scc.kit.edu/home/<oe>/<group>/<user>/<remote filename> <local filename>

Retrieved from "https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster3.0/Login/Client&oldid=14478"