SDS@hd/Access/WEBDAV: Difference between revisions

From bwHPC Wiki
< SDS@hd‎ | Access
Jump to navigation Jump to search
m (redirect)
Tag: New redirect
 
Line 1: Line 1:
#REDIRECT [[Data_Transfer/WebDAV]]
It is possible to access the SDS@hd service from Windows, Mac and Linux using the WebDAV protocol.

This enables easy access to SDS@hd without additional registration of your own computer.
This way can also be useful if you are in a network in which, e.g., [[SDS@hd/Access/CIFS|SMB]] and [[SDS@hd/Access/NFS|NFS]] are not available, e.g., due to firewall restrictions, but want a faster and more robust connection then [[SDS@hd/Access/SFTP|SFTP]].

'''Attention:'''
In principle, however, the connection might not be suitable for permanent connections, since this depends highly on the used client if it is highly available.
Because of this we advise the [https://rclone.org/ Rclone] client.

= Prerequisites =

'''Attention:''' To access data served by SDS@hd, you need a '''''Service Password'''''. See details at [[SDS@hd/Registration]].

= Easy access via web browser =
For easy access, it is possible to access SDS@hd in a web browser. Visit [https://lsdf02-webdav.urz.uni-heidelberg.de/] and login with your SDS@hd username and service password.
Here you can get an overview of the data in your "Speichervorhaben" and download single files.
To be able to do more, like moving data, uploading new files, or downloading complete folders, a suitable client is needed as described in the next section.

= Installing the WebDAV client Rclone =
Rclone is a command line tool to manage files on cloud storage systems and can easily be used to access SDS@hd.
Detailed instructions on how to download and install Rclone can be found [https://rclone.org/install/ here].

== Quickstart ==
Rclone is a Go program and comes as a single binary file.

* [https://rclone.org/downloads/ Download] the relevant binary.
* Extract the `rclone` executable, `rclone.exe` on Windows, from the archive.
* You can use the executables without further installation. For easy use, it is recommended to add the binary to your PATH environment variable. Information on how to do this can be found below.
* Run `rclone config` to set up SDS@hd connection. See [https://rclone.org/webdav/ rclone webdav config] docs for more details.
* Optionally configure automatic execution.

Detailed information regarding different operating systems can be found here:
* [https://rclone.org/install/#script-installation Installation on Linux]
* [https://rclone.org/install/#macos Installation on macOS]
* [https://rclone.org/install/#windows Installation on Windows]

== Configuring the SDS@hd connection ==

To configure the SDS@hd WebDAV remote, you will need to use the following URL and have a valid username and password.

Config overview:
<pre>
url = https://lsdf02-webdav.urz.uni-heidelberg.de
user = hd_xy123
pw = SERVICE_PASSWORD
</pre>

To add the SDS@hd connection to rclone, simply run:
<pre>
> rclone config
</pre>
This will guide you through an interactive setup process. It is important to use the provided configuration values to get a working SDS@hd connection.
<pre>
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> sds-hd

Type of storage to configure.
Choose a number from below, or type in your own value
[snip]
XX / WebDAV
\ "webdav"
[snip]
Storage> webdav

URL of http host to connect to
E.g. https://example.com
Enter a value
url> https://lsdf02-webdav.urz.uni-heidelberg.de

Name of the WebDAV site/service/software you are using
Choose a number from below, or type in your own value
1 / Fastmail Files
\ (fastmail)
2 / Nextcloud
\ (nextcloud)
3 / Owncloud
\ (owncloud)
4 / Sharepoint Online, authenticated by Microsoft account
\ (sharepoint)
5 / Sharepoint with NTLM authentication, usually self-hosted or on-premises
\ (sharepoint-ntlm)
6 / Other site/service or software
\ (other)
vendor> other

User name
user> <insert sds@hd username, eg. hd_xy123>

Password.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank
y/g/n> y
Enter the password:
password: <enter service pwd>
Confirm the password:
password: <enter service pwd>

Bearer token instead of user/pass (e.g. a Macaroon)
bearer_token>

Edit advanced config?
y) Yes
n) No (default)
y/n> n

Remote config
--------------------
[sds-hd]
type = webdav
url = https://lsdf02-webdav.urz.uni-heidelberg.de
vendor = other
user = hd_xy123
pass = *** ENCRYPTED ***
bearer_token =
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
</pre>
After this, you can exit the rclone config program.

= Using Rclone client interactively =
A detailed explanation on how to use Rclone can be found [https://rclone.org/docs/#basic-syntax here]!

In general, the syntax to use Rclone is like this:
<pre>
rclone [options] subcommand <parameters> <parameters...>
</pre>
Source and destination paths are specified by the name you gave the storage system in the config file (e.g. sds-hd ) then the subpath, e.g. "sds-hd:sd16a0001" to look at Speichervorhaben "sd16a001" on SDS@hd.

== A few examples for an easy start ==
List all directories/containers/buckets in the Speichervorhaben sd16a001.
<pre>rclone lsd sds-hd:sd16a001 </pre>

Copies /local/path to the remote path on SDS@hd
<pre> rclone copy </local/path> sds-hd:<remote/path> </pre>
Copies fom remote path on SDS@hd to /local/path
<pre> rclone copy sds-hd:<remote/path> </local/path> </pre>
Moves the contents of the source directory to the destination directory.
<pre> rclone move sds-hd:<source/path> sds-hd:<destination/path> </pre>

More subcommands can be found [https://rclone.org/docs/#subcommands here].

= Using Rclone to create a local mount =
Mount SDS@hd as a file system on a mount point.
Detailed information on how to use rclone mount can be found [https://rclone.org/commands/rclone_mount/ here].

On Linux and macOS, you can run mount in either foreground or background (aka daemon) mode. Mount runs in foreground mode by default. Use the --daemon flag to force background mode. On Windows, you can run mount in the foreground only, the flag is ignored.

== Using Rclone mount on Linux or macOS ==
On Linux or macOS, start the mount like this, where /path/to/local/mount is an empty existing directory:
<pre>
rclone mount sds-hd:path/to/files /path/to/local/mount
</pre>

== Using Rclone mount on Windows ==
To run rclone mount on Windows, you will need to download and install [https://winfsp.dev/ WinFsp].
More Information can be found [https://rclone.org/commands/rclone_mount/#installing-on-windows here].
To mount on drive letter X or a nonexistent subdirectory, use:
<pre>
rclone mount sds-hd:path/to/files X:
rclone mount sds-hd:path/to/files C:\path\parent\mount
</pre>

= Best practices =

Rclone/WebDAV has a lot of useful options.

== Performance ==

To be able to utilize a larger bandwidth, it is helpful to add the following options for increased performance:

<pre>
--transfers <int>
</pre>

Number of file transfers to run in parallel (default: 4). Depending on the local Network, read and write speeds on the file system, and current load, different values might be best. For large transfers, it is advised to test local performance with different values beforehand.

* In our tests, we observed the best results between 8 and 32.
* For regular use cases, we recommend 16 as the default.
* Values above 64 are not recommended and degrade performance.

<pre>
--multi-thread-streams <int>
</pre>

Number of streams to use for multithreaded downloads (default: 4). Only important on very large files. This will cause multithreaded up/download on chunk-sized bits of the file.

The optimal value is highly specific to the local network and used Hardware. For regular use cases, we recommend 4 as the default.

== Debugging and Statistics ==
To get updates on current progress, use:
<pre>
--stats
</pre>
Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable) (default 1m0s).

To get debug information, use:
<pre>
--log-level=DEBUG
--stats-log-level=DEBUG
</pre>

Latest revision as of 18:36, 2 December 2024

Redirect to:

Retrieved from "https://wiki.bwhpc.de/wiki/index.php?title=SDS@hd/Access/WEBDAV&oldid=13367"