Data Transfer/All Data Transfer Routes: Difference between revisions

From bwHPC Wiki
Jump to navigation Jump to search
m (Clarifying rclone information)
Line 8: Line 8:
{| class="wikitable"
{| class="wikitable"
|- style="font-weight:bold; text-align:center; vertical-align:middle;"
|- style="font-weight:bold; text-align:center; vertical-align:middle;"
! Protocol/Tool
! Network protocol
! Underlying network protocol
! Underlying network protocol
! Description/When to use
! Description/When to use
Line 31: Line 31:
|-
|-
| [[Data_Transfer/Rclone|Rclone]]
| [[Data_Transfer/Rclone|Rclone]]
| sftp(ssh), smb, WebDAV(https), ...
| Depending on setup: sftp(ssh), smb, WebDAV(https), ...
| Needs installation. Multithreaded. Works on two completely remote locations as well. Transfers whole files. No 2FA possible. Suitable for big data.
| A tool that needs to be installed. Multithreaded. Works on two completely remote locations as well. Transfers whole files. No 2FA possible. Usually suitable for big data (please refer to the information about the specific protocol that you are going to use).
| ✓
| ✓
| ✓
| ✓
Line 41: Line 41:
| ✓
| ✓
|- style="vertical-align:middle;"
|- style="vertical-align:middle;"
| [[Data_Transfer/WebDAV|WebDav]] via [[Data_Transfer/Rclone|Rclone]]
| [[Data_Transfer/WebDAV|WebDav]] (optimally used via [[Data_Transfer/Rclone|Rclone]])
| http(s)
| http(s)
| Fast with small data, not suitable for big data.
| Fast with small data, not suitable for big data.
Line 85: Line 85:
| ☓
| ☓
|}
|}

== Mounting via commandline ==
== Mounting via commandline ==
{| class="wikitable" style="vertical-align:middle;"
{| class="wikitable" style="vertical-align:middle;"

Revision as of 18:38, 12 March 2025

All Data Transfer Routes

The following tables provide a full overview on the various transfer options depending on the system that you are coming from and the system you want to transfer your data to. The recommended options are marked in the table with W(indows)/M(acOS)/L(inux) added next to the '✓' sign.
'[✓]' means that this option is possible but might be the worst choice of the given ones.

Copy and Sync via commandline

Protocol/Tool Underlying network protocol Description/When to use Executable on Transfer from/to
Linux MacOS Windows bwUni Cluster bwFor Cluster bwHPC Cluster SDS@hd
Rclone Depending on setup: sftp(ssh), smb, WebDAV(https), ... A tool that needs to be installed. Multithreaded. Works on two completely remote locations as well. Transfers whole files. No 2FA possible. Usually suitable for big data (please refer to the information about the specific protocol that you are going to use). ✓ ✓ ✓ ✓ ✓ only on BinAC ✓
WebDav (optimally used via Rclone) http(s) Fast with small data, not suitable for big data. ✓ ✓ ✓ ✓ ✓ ☓ ✓
rsync ssh Synchronization of data. One side must be a local drive. Can spot differences in files and transfer only the changes. Can resume partial transfers. ✓ ✓ [✓] ✓ ✓ ✓ ☓
sftp ssh Interactive shell for copying data ✓ [✓] ✓ ✓ ✓ ✓ ✓
scp ssh For copying data. Outdated according to: https://www.openssh.com/txt/release-8.0 [✓] [✓] [✓] [✓] [✓] [✓] ☓

Mounting via commandline

Protocol/Tool Underlying network protocol Description & Use Cases Executable on Transfer from/to
Linux MacOS Windows bwHPC bwHPC SDS@hd
Rclone mount sftp (ssh), smb, WebDAV (https), ... See previous table ✓ ✓ ✓ ✓ ☓ ✓
sshfs see sftp File system using sftp, simple usage, needs stable connections, can't resume partial transfers ✓ [✓] ✓ ☓ ✓ ( L, M ) ✓
smb none Mapping of a network drive, multiuser possible ✓ ✓ ✓ ☓ ☓ ✓ (M, W, L)
NFSv4 none One-time installation effort (Kerberos, ID mapping). For servers, microscopes, multi-user environments, ... ✓ ☓ ☓ ☓ ☓ ✓

Graphical User Interfaces (GUIs)

GUIs Underlying network protocols Description & Use Cases Executable on Transfer from/to
Linux MacOS Windows bwHPC bwHPC SDS@hd
JupyterLab ssh Development Environment ✓ ✓ ✓ ✓ ✓ ✓
VS Code ssh Development Environment ✓ ✓ ✓ ✓ ✓ ✓
RStudio ssh Development Environment ✓ ✓ ✓ ✓ ✓ ✓
MobaXterm ssh, RSync, sftp ☓ ☓ ✓ ☓ ✓ ( W ) ✓
WinSCP sftp, WebDAV, scp ☓ ☓ ✓ ☓ ✓ ✓
Filezilla sftp ✓ ✓ ✓ ☓ ✓ ✓
Cyberduck sftp, WebDAV, S3 ☓ ✓ ✓ ☓ ✓ ✓
WebDAV via browser WebDAV download single files ✓ ✓ ✓ ☓ ☓ ✓
Retrieved from "https://wiki.bwhpc.de/wiki/index.php?title=Data_Transfer/All_Data_Transfer_Routes&oldid=14340"