S Braun at 17:13, 6 April 2025

2025-04-06T17:13:46Z

← Older revision		Revision as of 19:13, 6 April 2025
Line 3:		Line 3:
	Further alternatives for file and data exchange are available Here: [[Data_Transfer \| Data Transfer]]		Further alternatives for file and data exchange are available Here: [[Data_Transfer \| Data Transfer]]

			= rsync =
			[[Data_Transfer/Rsync \|rsync]]

	= WebDAV =		= WebDAV =

S Braun at 17:12, 6 April 2025

2025-04-06T17:12:19Z

← Older revision		Revision as of 19:12, 6 April 2025
Line 4:		Line 4:


	== WebDAV ==		= WebDAV =
	WebDAV access to the bwUniCluster-Filesystems		WebDAV access to the bwUniCluster-Filesystems

Line 27:		Line 27:


	=== rclone ===		== rclone ==

	Rclone is one of the tools for transferring files from/to the HPC Filesystem via WebDAV. It's detection of unchanged files, parallel transfers and solid retry behaviour make it to a good choice. Rclone can be installed on Linux, MacOS and Windows.		Rclone is one of the tools for transferring files from/to the HPC Filesystem via WebDAV. It's detection of unchanged files, parallel transfers and solid retry behaviour make it to a good choice. Rclone can be installed on Linux, MacOS and Windows.
Line 52:		Line 52:


	=== curl ===		== curl ==

	Because WebDAV extends HTTP, any HTTP client can access files in the HPC cluster. Given the full URL to a file, Curl and co can download a file or upload a file.		Because WebDAV extends HTTP, any HTTP client can access files in the HPC cluster. Given the full URL to a file, Curl and co can download a file or upload a file.
Line 63:		Line 63:


	=== davix ===		== davix ==

	Davix is a set of tools to perform primitive operations via WebDAV protocol. It is shipped with most Linux distributions.		Davix is a set of tools to perform primitive operations via WebDAV protocol. It is shipped with most Linux distributions.

S Braun: Created page with "= Data Transfer = Further alternatives for file and data exchange are available Here: Data Transfer == WebDAV == WebDAV access to the bwUniCluster-Filesystems Access via Webbrowser (read-only) Visit uc3-webdav.scc.kit.edu and login with your UC3 username and password (not the HPC service password). Here you can get an overview of your data and download single files. To be able to do more, like uploading new files, or downloading complete folders..."

2025-03-28T07:49:45Z

Created page with "= Data Transfer = Further alternatives for file and data exchange are available Here: Data Transfer == WebDAV == WebDAV access to the bwUniCluster-Filesystems Access via Webbrowser (read-only) Visit uc3-webdav.scc.kit.edu and login with your UC3 username and password (not the HPC service password). Here you can get an overview of your data and download single files. To be able to do more, like uploading new files, or downloading complete folders..."

New page

= Data Transfer =

Further alternatives for file and data exchange are available Here: [[Data_Transfer | Data Transfer]]

== WebDAV ==
WebDAV access to the bwUniCluster-Filesystems

Access via Webbrowser (read-only)

Visit uc3-webdav.scc.kit.edu and login with your UC3 username and password (not the HPC service password). Here you can get an overview of your data and download single files. To be able to do more, like uploading new files, or downloading complete folders, a suitable client is needed (rclone, davix).

Access via command line tools

The authentication to use command line tools on the bwUniCluster is implemented via the OAuth2-Bearer-Token method.
The Bearer tokens can be created with the oidc-agent [2]. This is available for Linux, MacOs and Windows [3].
[2] https://indigo-dc.gitbook.io/oidc-agent/
[3] https://github.com/indigo-dc/oidc-agent#installation

Example to create a OAuth2-Bearer-Token for using the bwUniCluster with oidc-gen:

oidc-gen <your shortname> --client-id unicluster_public --client-secret public --issuer https://bwidm.scc.kit.edu/oidc/realms/bwidm/ --redirect-uri http://localhost:<port-number>/biwdm.scc.kit.edu --scope uc3-webdav

With the command oidc-token you can show or use the token:

oidc-token <your shortname>

=== rclone ===

Rclone is one of the tools for transferring files from/to the HPC Filesystem via WebDAV. It's detection of unchanged files, parallel transfers and solid retry behaviour make it to a good choice. Rclone can be installed on Linux, MacOS and Windows.
[1] https://rclone.org

To configure the rclone remote for the bwUniCluster, you will need to use the following URL, your account and your oidc-token shortname:
url = https://oauth2.uc3-webdav.scc.kit.edu

The rclone remote can be configured by command line:
rclone config create <your uc3-webdav name> webdav url https://oauth2.uc3-webdav.scc.kit.edu vendor other user <account> bearer_token_command "oidc-token <your oidc-token shortname>"

For the interactive setup process with rclone config see <Data Transfer/WebDAV> and use the bearer_token_command "oidc-token <your oidc-token shortname>" instead of the password.

The syntax to use Rclone is like this:
rclone [options] subcommand <parameters> <parameters...>

Copies from local path to the remote path:
rclone copy </local/path> <your uc3-webdav name>:/home/<oe>/<group>/<user>/<remote path>

Copies from remote path to the local path
rclone copy <your uc3-webdav name>:/home/<oe>/<group>/<user>/<local path>

For more Details about rclone look at Data Transfer/Rclone or https://rclone.org.

=== curl ===

Because WebDAV extends HTTP, any HTTP client can access files in the HPC cluster. Given the full URL to a file, Curl and co can download a file or upload a file.

File-Download:
curl -H "Authorization: Bearer `oidc-token <your shortname>`" --output <local filename> https://oauth2.uc3-webdav.scc.kit.edu/home/<oe>/<group>/<user>/<remote filename>

File-Upload:
curl -H "Authorization: Bearer `oidc-token <your shortname>`" --silent --upload-file <local filename> https://oauth2.uc3-webdav.scc.kit.edu/home/<oe>/<group>/<user>/<remote filename>

=== davix ===

Davix is a set of tools to perform primitive operations via WebDAV protocol. It is shipped with most Linux distributions.

Some examples to list or transfer files are:

davix-ls -H "Authorization: Bearer `oidc-token <your shortname>`" https://oauth2.uc3-webdav.scc.kit.edu/home/<oe>/<group>/<user>/my-data-set/

davix-put -H "Authorization: Bearer `oidc-token <your shortname>`" <local filename> https://oauth2.uc3-webdav.scc.kit.edu/home/<oe>/<group>/<user>/<remote filename>

davix-get -H "Authorization: Bearer `oidc-token <your shortname>`" https://oauth2.uc3-webdav.scc.kit.edu/home/<oe>/<group>/<user>/<remote filename> <local filename>

BwUniCluster3.0/Login/Data Transfer - Revision history

S Braun at 17:13, 6 April 2025

S Braun at 17:12, 6 April 2025