Data Transfer/All Data Transfer Routes: Difference between revisions
Jump to navigation
Jump to search
H Schumacher (talk | contribs) (content changes) |
H Schumacher (talk | contribs) m (Clarifying Rclone entries) |
||
(2 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
= All Data Transfer Routes = |
= All Data Transfer Routes = |
||
The following tables provide a full overview on the various transfer options depending on the system that you are coming from and the system you want to transfer your data to. The recommended options are marked in the table with '''W'''(indows)/'''M'''(acOS)/'''L'''(inux) added next to the '✓' sign.<br /> |
The following tables provide a full overview on the various transfer options depending on the system that you are coming from and the system you want to transfer your data to. <u>The recommended options are marked</u> in the table with '''W'''(indows)/'''M'''(acOS)/'''L'''(inux) added next to the '✓' sign.<br /> |
||
'[✓]' means that this option is possible but might be the worst choice of the given ones. |
'[✓]' means that this option is possible but might be the worst choice of the given ones. |
||
Line 8: | Line 8: | ||
{| class="wikitable" |
{| class="wikitable" |
||
|- style="font-weight:bold; text-align:center; vertical-align:middle;" |
|- style="font-weight:bold; text-align:center; vertical-align:middle;" |
||
! Protocol/Tool |
|||
! Network protocol |
|||
! Underlying network protocol |
! Underlying network protocol |
||
! Description/When to use |
! Description/When to use |
||
Line 30: | Line 30: | ||
| [[SDS@hd|SDS@hd]] |
| [[SDS@hd|SDS@hd]] |
||
|- |
|- |
||
| [[Data_Transfer/Rclone|Rclone]] |
| [[Data_Transfer/Rclone|Rclone]] + <protocol> |
||
| sftp |
| sftp, smb, WebDAV, ... |
||
| |
| A tool that can be used with various protocols. It needs to be installed. Multithreaded. Works on two completely remote locations as well. Transfers whole files. No 2FA possible. Usually suitable for big data. Please refer to the marked recommendations when deciding on a network protocol to use with Rclone. |
||
| ✓ |
| ✓ |
||
| ✓ |
| ✓ |
||
Line 41: | Line 41: | ||
| ✓ |
| ✓ |
||
|- style="vertical-align:middle;" |
|- style="vertical-align:middle;" |
||
| [[Data_Transfer/WebDAV| |
| [[Data_Transfer/WebDAV|WebDAV]] |
||
| http(s) |
| http(s) |
||
| Fast with small data, not suitable for big data. |
| Fast with small data, not suitable for big data. |
||
Line 85: | Line 85: | ||
| ☓ |
| ☓ |
||
|} |
|} |
||
== Mounting via commandline == |
== Mounting via commandline == |
||
{| class="wikitable" style="vertical-align:middle;" |
{| class="wikitable" style="vertical-align:middle;" |
||
Line 104: | Line 105: | ||
| SDS@hd |
| SDS@hd |
||
|- |
|- |
||
| [[Data_Transfer/Rclone#Usage_Rclone_Mount|Rclone mount]] |
| [[Data_Transfer/Rclone#Usage_Rclone_Mount|Rclone mount]] + <protocol> |
||
| sftp |
| sftp, smb, WebDAV, ... |
||
| See previous table |
| See previous table |
||
| ✓ |
| ✓ |
||
Line 126: | Line 127: | ||
| [[SDS@hd/Access/SMB|smb]] |
| [[SDS@hd/Access/SMB|smb]] |
||
| none |
| none |
||
| Mapping of a network drive |
| Mapping of a network drive. Multiuser possible. Good fit for big data. |
||
| ✓ |
| ✓ |
||
| ✓ |
| ✓ |
||
Line 174: | Line 175: | ||
| ✓ |
| ✓ |
||
|- |
|- |
||
| [[Development/VS_Code|VS Code]] |
|||
| VSCode |
|||
| ssh |
| ssh |
||
| Development Environment |
| Development Environment |
Latest revision as of 00:51, 13 March 2025
All Data Transfer Routes
The following tables provide a full overview on the various transfer options depending on the system that you are coming from and the system you want to transfer your data to. The recommended options are marked in the table with W(indows)/M(acOS)/L(inux) added next to the '✓' sign.
'[✓]' means that this option is possible but might be the worst choice of the given ones.
Copy and Sync via commandline
Protocol/Tool | Underlying network protocol | Description/When to use | Executable on | Transfer from/to | |||||
---|---|---|---|---|---|---|---|---|---|
Linux | MacOS | Windows | bwUni Cluster | bwFor Cluster | bwHPC Cluster | SDS@hd | |||
Rclone + <protocol> | sftp, smb, WebDAV, ... | A tool that can be used with various protocols. It needs to be installed. Multithreaded. Works on two completely remote locations as well. Transfers whole files. No 2FA possible. Usually suitable for big data. Please refer to the marked recommendations when deciding on a network protocol to use with Rclone. | ✓ | ✓ | ✓ | ✓ | ✓ | only on BinAC | ✓ |
WebDAV | http(s) | Fast with small data, not suitable for big data. | ✓ | ✓ | ✓ | ✓ | ✓ | ☓ | ✓ |
rsync | ssh | Synchronization of data. One side must be a local drive. Can spot differences in files and transfer only the changes. Can resume partial transfers. | ✓ | ✓ | [✓] | ✓ | ✓ | ✓ | ☓ |
sftp | ssh | Interactive shell for copying data | ✓ | [✓] | ✓ | ✓ | ✓ | ✓ | ✓ |
scp | ssh | For copying data. Outdated according to: https://www.openssh.com/txt/release-8.0 | [✓] | [✓] | [✓] | [✓] | [✓] | [✓] | ☓ |
Mounting via commandline
Protocol/Tool | Underlying network protocol | Description & Use Cases | Executable on | Transfer from/to | ||||
---|---|---|---|---|---|---|---|---|
Linux | MacOS | Windows | bwHPC | bwHPC | SDS@hd | |||
Rclone mount + <protocol> | sftp, smb, WebDAV, ... | See previous table | ✓ | ✓ | ✓ | ✓ | ☓ | ✓ |
sshfs | see sftp | File system using sftp, simple usage, needs stable connections, can't resume partial transfers | ✓ | [✓] | ✓ | ☓ | ✓ ( L, M ) | ✓ |
smb | none | Mapping of a network drive. Multiuser possible. Good fit for big data. | ✓ | ✓ | ✓ | ☓ | ☓ | ✓ (M, W, L) |
NFSv4 | none | One-time installation effort (Kerberos, ID mapping). For servers, microscopes, multi-user environments, ... | ✓ | ☓ | ☓ | ☓ | ☓ | ✓ |
Graphical User Interfaces (GUIs)
GUIs | Underlying network protocols | Description & Use Cases | Executable on | Transfer from/to | ||||
---|---|---|---|---|---|---|---|---|
Linux | MacOS | Windows | bwHPC | bwHPC | SDS@hd | |||
JupyterLab | ssh | Development Environment | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
VS Code | ssh | Development Environment | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
RStudio | ssh | Development Environment | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
MobaXterm | ssh, RSync, sftp | ☓ | ☓ | ✓ | ☓ | ✓ ( W ) | ✓ | |
WinSCP | sftp, WebDAV, scp | ☓ | ☓ | ✓ | ☓ | ✓ | ✓ | |
Filezilla | sftp | ✓ | ✓ | ✓ | ☓ | ✓ | ✓ | |
Cyberduck | sftp, WebDAV, S3 | ☓ | ✓ | ✓ | ☓ | ✓ | ✓ | |
WebDAV via browser | WebDAV | download single files | ✓ | ✓ | ✓ | ☓ | ☓ | ✓ |