BwUniCluster3.0/Login/Data Transfer

Data Transfer

Further alternatives for file and data exchange are available Here: Data Transfer

WebDAV

WebDAV access to the bwUniCluster-Filesystems

Access via Webbrowser (read-only)

Visit uc3-webdav.scc.kit.edu and login with your UC3 username and password (not the HPC service password). Here you can get an overview of your data and download single files. To be able to do more, like uploading new files, or downloading complete folders, a suitable client is needed (rclone, davix).

Access via command line tools

The authentication to use command line tools on the bwUniCluster is implemented via the OAuth2-Bearer-Token method. The Bearer tokens can be created with the oidc-agent [2]. This is available for Linux, MacOs and Windows [3]. [2] https://indigo-dc.gitbook.io/oidc-agent/ [3] https://github.com/indigo-dc/oidc-agent#installation

Example to create a OAuth2-Bearer-Token for using the bwUniCluster with oidc-gen:

oidc-gen <your shortname> --client-id unicluster_public --client-secret public --issuer https://bwidm.scc.kit.edu/oidc/realms/bwidm/ --redirect-uri http://localhost:<port-number>/biwdm.scc.kit.edu --scope uc3-webdav

With the command oidc-token you can show or use the token:

oidc-token <your shortname>

rclone

Rclone is one of the tools for transferring files from/to the HPC Filesystem via WebDAV. It's detection of unchanged files, parallel transfers and solid retry behaviour make it to a good choice. Rclone can be installed on Linux, MacOS and Windows. [1] https://rclone.org

To configure the rclone remote for the bwUniCluster, you will need to use the following URL, your account and your oidc-token shortname: url = https://oauth2.uc3-webdav.scc.kit.edu

The rclone remote can be configured by command line: rclone config create <your uc3-webdav name> webdav url https://oauth2.uc3-webdav.scc.kit.edu vendor other user <account> bearer_token_command "oidc-token <your oidc-token shortname>"

For the interactive setup process with rclone config see and use the bearer_token_command "oidc-token <your oidc-token shortname>" instead of the password.

The syntax to use Rclone is like this: rclone [options] subcommand <parameters> <parameters...>

Copies from local path to the remote path: rclone copy </local/path> <your uc3-webdav name>:/home/<oe>/<group>/<user>/<remote path>

Copies from remote path to the local path rclone copy <your uc3-webdav name>:/home/<oe>/<group>/<user>/<local path>

For more Details about rclone look at Data Transfer/Rclone or https://rclone.org.

curl

Because WebDAV extends HTTP, any HTTP client can access files in the HPC cluster. Given the full URL to a file, Curl and co can download a file or upload a file.

File-Download: curl -H "Authorization: Bearer `oidc-token <your shortname>`" --output <local filename> https://oauth2.uc3-webdav.scc.kit.edu/home/<oe>/<group>/<user>/<remote filename>

File-Upload: curl -H "Authorization: Bearer `oidc-token <your shortname>`" --silent --upload-file <local filename> https://oauth2.uc3-webdav.scc.kit.edu/home/<oe>/<group>/<user>/<remote filename>

davix

Davix is a set of tools to perform primitive operations via WebDAV protocol. It is shipped with most Linux distributions.

Some examples to list or transfer files are:

davix-ls -H "Authorization: Bearer `oidc-token <your shortname>`" https://oauth2.uc3-webdav.scc.kit.edu/home/<oe>/<group>/<user>/my-data-set/

davix-put -H "Authorization: Bearer `oidc-token <your shortname>`" <local filename> https://oauth2.uc3-webdav.scc.kit.edu/home/<oe>/<group>/<user>/<remote filename>

davix-get -H "Authorization: Bearer `oidc-token <your shortname>`" https://oauth2.uc3-webdav.scc.kit.edu/home/<oe>/<group>/<user>/<remote filename> <local filename>