Data Transfer/All Data Transfer Routes: Difference between revisions
Jump to navigation
Jump to search
H Schumacher (talk | contribs) m (corrected links) |
H Schumacher (talk | contribs) (content changes) |
||
Line 1: | Line 1: | ||
{| style=" background:#FEF4AB; width:100%;" |
|||
| style="padding:6px; font-size:100%;text-align:left" | This page is work in progress. To discover more relevant subpages, please come back at a later timepoint. |
|||
|- |
|||
|} |
|||
= All Data Transfer Routes = |
= All Data Transfer Routes = |
||
Line 10: | Line 6: | ||
== Copy and Sync via commandline == |
== Copy and Sync via commandline == |
||
{| class="wikitable" |
{| class="wikitable" |
||
|- style="font-weight:bold;" |
|- style="font-weight:bold; text-align:center; vertical-align:middle;" |
||
! Network protocol |
! Network protocol |
||
! Underlying network protocol |
! Underlying network protocol |
||
Line 22: | Line 18: | ||
! Transfer from/to |
! Transfer from/to |
||
! |
! |
||
|- style="vertical-align:middle;" |
|||
|- |
|||
| |
| |
||
| |
| |
||
| |
| |
||
| Linux |
| Linux |
||
| |
| MacOS |
||
| Windows |
| Windows |
||
| bwUni Cluster |
| bwUni Cluster |
||
Line 34: | Line 30: | ||
| [[SDS@hd|SDS@hd]] |
| [[SDS@hd|SDS@hd]] |
||
|- |
|- |
||
| |
| [[Data_Transfer/Rclone|Rclone]] |
||
| sftp(ssh), smb, WebDAV(https), ... |
|||
| http(s) |
|||
| Needs installation. Multithreaded. Works on two completely remote locations as well. Transfers whole files. |
| Needs installation. Multithreaded. Works on two completely remote locations as well. Transfers whole files. No 2FA possible. Suitable for big data. |
||
| ✓ |
| ✓ |
||
| ✓ |
| ✓ |
||
| ✓ |
| ✓ |
||
| ☓ |
|||
| ☓ |
|||
| ☓ |
|||
| ✓ |
| ✓ |
||
|- |
|||
| sftp via [[Data_Transfer/Rclone|Rclone]] |
|||
| ssh |
|||
| see above |
|||
| ✓ |
| ✓ |
||
| only on BinAC |
|||
| ✓ |
| ✓ |
||
|- style="vertical-align:middle;" |
|||
| [[Data_Transfer/WebDAV|WebDav]] via [[Data_Transfer/Rclone|Rclone]] |
|||
| http(s) |
|||
| Fast with small data, not suitable for big data. |
|||
| ✓ |
| ✓ |
||
| ✓ |
| ✓ |
||
| ✓ |
| ✓ |
||
| ✓ |
| ✓ |
||
| [✓] |
|||
|- |
|||
| [[Data_Transfer/Rsync|rsync]] |
|||
| ssh |
|||
| Synchronization of data. One side must be a local drive. Can spot differences in files and transfer only the changes. |
|||
| ✓ |
| ✓ |
||
| ☓ |
|||
| ✓ |
| ✓ |
||
|- style="vertical-align:middle;" |
|||
| [✓] |
|||
| [[Data_Transfer/Rsync|rsync]] |
|||
| ✓ |
|||
| ✓ |
|||
| ✓ |
|||
| ☓ |
|||
|- |
|||
| [[Data_Transfer/SCP|scp]] |
|||
| ssh |
| ssh |
||
| Synchronization of data. One side must be a local drive. Can spot differences in files and transfer only the changes. Can resume partial transfers. |
|||
| For copying data |
|||
| ✓ |
| ✓ |
||
| ✓ |
| ✓ |
||
Line 77: | Line 62: | ||
| ✓ |
| ✓ |
||
| ☓ |
| ☓ |
||
|- style="vertical-align:middle;" |
|||
|- |
|||
| [[Data_Transfer/SFTP|sftp]] |
| [[Data_Transfer/SFTP|sftp]] |
||
| ssh |
| ssh |
||
| Interactive shell for copying data |
| Interactive shell for copying data |
||
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]] |
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]] |
||
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]] |
| [[Data_Transfer/SFTP#MacOS_&_Linux|[✓]]] |
||
| [[Data_Transfer/SFTP#Windows|✓]] |
| [[Data_Transfer/SFTP#Windows|✓]] |
||
| ✓ |
| ✓ |
||
Line 88: | Line 73: | ||
| ✓ |
| ✓ |
||
| ✓ |
| ✓ |
||
|- |
|||
| [[Data_Transfer/SCP|scp]] |
|||
| ssh |
|||
| For copying data. Outdated according to: https://www.openssh.com/txt/release-8.0 |
|||
| [✓] |
|||
| [✓] |
|||
| [✓] |
|||
| [✓] |
|||
| [✓] |
|||
| [✓] |
|||
| ☓ |
|||
|} |
|} |
||
== Mounting via commandline == |
== Mounting via commandline == |
||
{| class="wikitable" style="vertical-align:middle;" |
{| class="wikitable" style="vertical-align:middle;" |
||
|- style="font-weight:bold;" |
|- style="font-weight:bold; text-align:center;" |
||
! Protocol/Tool |
|||
! Network protocol |
|||
! Underlying protocol |
! Underlying network protocol |
||
! Description |
! Description & Use Cases |
||
! Executable on |
! colspan="4" | Executable on |
||
! colspan="2" | Transfer from/to |
|||
! |
|||
! |
|||
! |
|||
! |
|||
! Transfer from/to |
|||
! |
|||
|- |
|- |
||
| |
| |
||
Line 108: | Line 98: | ||
| |
| |
||
| Linux |
| Linux |
||
| |
| MacOS |
||
| Windows |
| Windows |
||
| bwHPC |
|||
| bwUniCluster |
|||
| bwHPC |
|||
| bwForCluster |
|||
| bwHPC cluster |
|||
| SDS@hd |
| SDS@hd |
||
|- |
|- |
||
| |
| [[Data_Transfer/Rclone#Usage_Rclone_Mount|Rclone mount]] |
||
| sftp (ssh), smb, WebDAV (https), ... |
|||
| http(s) |
|||
| See previous table |
|||
| Needs installation |
|||
| ✓ |
|||
| ✓ |
| ✓ |
||
| ✓ |
| ✓ |
||
| ✓ |
| ✓ |
||
| ☓ |
|||
| ☓ |
|||
| ☓ |
| ☓ |
||
| ✓ |
| ✓ |
||
Line 128: | Line 116: | ||
| [[Data_Transfer/SSHFS|sshfs]] |
| [[Data_Transfer/SSHFS|sshfs]] |
||
| see sftp |
| see sftp |
||
| File system using sftp, simple usage, needs stable connections, can't resume partial transfers |
|||
| |
|||
| ✓ |
|||
| ✓ |
|||
| ✓ |
|||
| ✓ |
| ✓ |
||
| [✓] |
|||
| ✓ |
| ✓ |
||
| ☓ |
|||
| ✓ [[Data_Transfer/SSHFS#MacOS_&_Linux|( L, M )]] |
| ✓ [[Data_Transfer/SSHFS#MacOS_&_Linux|( L, M )]] |
||
| ✓ |
| ✓ |
||
Line 139: | Line 126: | ||
| [[SDS@hd/Access/SMB|smb]] |
| [[SDS@hd/Access/SMB|smb]] |
||
| none |
| none |
||
| Mapping of a network drive |
| Mapping of a network drive, multiuser possible |
||
| ✓ |
|||
| [[SDS@hd/Access/SMB#Linux|L]] |
|||
| ✓ |
| ✓ |
||
| ✓ |
| ✓ |
||
| ☓ |
| ☓ |
||
| ☓ |
| ☓ |
||
| ✓ ([[SDS@hd/Access/SMB#Mac|M]], [[SDS@hd/Access/SMB#Windows|W]], [[SDS@hd/Access/SMB#Linux|L]]) |
|||
| ☓ |
|||
| ✓ ([[SDS@hd/Access/SMB#Mac|M]], [[SDS@hd/Access/SMB#Windows|W]]) |
|||
|- |
|- |
||
| [[SDS@hd/Access/NFS#Using_NFSv4_for_UNIX_client|NFSv4]] |
| [[SDS@hd/Access/NFS#Using_NFSv4_for_UNIX_client|NFSv4]] |
||
Line 152: | Line 138: | ||
| One-time installation effort (Kerberos, ID mapping). For servers, microscopes, multi-user environments, ... |
| One-time installation effort (Kerberos, ID mapping). For servers, microscopes, multi-user environments, ... |
||
| ✓ |
| ✓ |
||
| ☓ |
|||
| ☓ |
| ☓ |
||
| ☓ |
| ☓ |
||
Line 161: | Line 146: | ||
== Graphical User Interfaces (GUIs) == |
== Graphical User Interfaces (GUIs) == |
||
{| class="wikitable" |
{| class="wikitable" |
||
|- style="font-weight:bold;" |
|- style="font-weight:bold; text-align:center; vertical-align:middle;" |
||
! GUIs |
! GUIs |
||
! Underlying protocols |
! Underlying network protocols |
||
! Description |
! Description & Use Cases |
||
! Executable on |
! colspan="4" | Executable on |
||
! colspan="2" | Transfer from/to |
|||
! |
|||
|- style="vertical-align:middle;" |
|||
! |
|||
! |
|||
! |
|||
! Transfer from/to |
|||
! |
|||
|- |
|||
| |
| |
||
| |
| |
||
| |
| |
||
| Linux |
| Linux |
||
| |
| MacOS |
||
| Windows |
| Windows |
||
| bwHPC |
|||
| bwUniCluster |
|||
| bwHPC |
|||
| bwForCluster |
|||
| bwHPC cluster |
|||
| SDS@hd |
| SDS@hd |
||
|- |
|- |
||
| JupyterLab |
|||
| ssh |
|||
| Development Environment |
|||
| ✓ |
|||
| ✓ |
|||
| ✓ |
|||
| ✓ |
|||
| ✓ |
|||
| ✓ |
|||
|- |
|||
| VSCode |
|||
| ssh |
|||
| Development Environment |
|||
| ✓ |
|||
| ✓ |
|||
| ✓ |
|||
| ✓ |
|||
| ✓ |
|||
| ✓ |
|||
|- |
|||
| RStudio |
|||
| ssh |
|||
| Development Environment |
|||
| ✓ |
|||
| ✓ |
|||
| ✓ |
|||
| ✓ |
|||
| ✓ |
|||
| ✓ |
|||
|- style="vertical-align:middle;" |
|||
| [[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]] |
| [[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]] |
||
| |
| ssh, RSync, sftp |
||
| |
| |
||
| ☓ |
| ☓ |
||
| ☓ |
| ☓ |
||
| ✓ |
| ✓ |
||
| ☓ |
|||
| ☓ |
| ☓ |
||
| ✓ [[Data_Transfer/Graphical_Clients#MobaXterm|( W )]] |
| ✓ [[Data_Transfer/Graphical_Clients#MobaXterm|( W )]] |
||
| ✓ |
| ✓ |
||
|- style="vertical-align:middle;" |
|||
|- |
|||
| [[Data_Transfer/Graphical_Clients#WinSCP|WinSCP]] |
| [[Data_Transfer/Graphical_Clients#WinSCP|WinSCP]] |
||
| |
| sftp, WebDAV, scp |
||
| |
| |
||
| ☓ |
| ☓ |
||
| ☓ |
| ☓ |
||
| ✓ |
| ✓ |
||
| ☓ |
|||
| ☓ |
| ☓ |
||
| ✓ |
| ✓ |
||
| ✓ |
| ✓ |
||
|- style="vertical-align:middle;" |
|||
|- |
|||
| [[Data_Transfer/Graphical_Clients#Filezilla|Filezilla]] |
| [[Data_Transfer/Graphical_Clients#Filezilla|Filezilla]] |
||
| |
| sftp |
||
| |
| |
||
| ✓ |
| ✓ |
||
| ✓ |
| ✓ |
||
| ✓ |
| ✓ |
||
| ☓ |
|||
| ☓ |
| ☓ |
||
| ✓ |
| ✓ |
||
| ✓ |
| ✓ |
||
|- style="vertical-align:middle;" |
|||
|- |
|||
| [https://cyberduck.io/ Cyberduck] |
| [https://cyberduck.io/ Cyberduck] |
||
| |
| sftp, WebDAV, S3 |
||
| |
| |
||
| ☓ |
| ☓ |
||
| ✓ |
| ✓ |
||
| ✓ |
| ✓ |
||
| ☓ |
|||
| ☓ |
| ☓ |
||
| ✓ |
| ✓ |
||
| ✓ |
| ✓ |
||
|- style="vertical-align:middle;" |
|||
|- |
|||
| WebDAV via browser |
| WebDAV via browser |
||
| WebDAV |
| WebDAV |
||
Line 235: | Line 240: | ||
| ✓ |
| ✓ |
||
| ✓ |
| ✓ |
||
| ☓ |
|||
| ☓ |
| ☓ |
||
| ☓ |
| ☓ |
Revision as of 11:07, 25 February 2025
All Data Transfer Routes
The following tables provide a full overview on the various transfer options depending on the system that you are coming from and the system you want to transfer your data to. The recommended options are marked in the table with W(indows)/M(acOS)/L(inux) added next to the '✓' sign.
'[✓]' means that this option is possible but might be the worst choice of the given ones.
Copy and Sync via commandline
Network protocol | Underlying network protocol | Description/When to use | Executable on | Transfer from/to | |||||
---|---|---|---|---|---|---|---|---|---|
Linux | MacOS | Windows | bwUni Cluster | bwFor Cluster | bwHPC Cluster | SDS@hd | |||
Rclone | sftp(ssh), smb, WebDAV(https), ... | Needs installation. Multithreaded. Works on two completely remote locations as well. Transfers whole files. No 2FA possible. Suitable for big data. | ✓ | ✓ | ✓ | ✓ | ✓ | only on BinAC | ✓ |
WebDav via Rclone | http(s) | Fast with small data, not suitable for big data. | ✓ | ✓ | ✓ | ✓ | ✓ | ☓ | ✓ |
rsync | ssh | Synchronization of data. One side must be a local drive. Can spot differences in files and transfer only the changes. Can resume partial transfers. | ✓ | ✓ | [✓] | ✓ | ✓ | ✓ | ☓ |
sftp | ssh | Interactive shell for copying data | ✓ | [✓] | ✓ | ✓ | ✓ | ✓ | ✓ |
scp | ssh | For copying data. Outdated according to: https://www.openssh.com/txt/release-8.0 | [✓] | [✓] | [✓] | [✓] | [✓] | [✓] | ☓ |
Mounting via commandline
Protocol/Tool | Underlying network protocol | Description & Use Cases | Executable on | Transfer from/to | ||||
---|---|---|---|---|---|---|---|---|
Linux | MacOS | Windows | bwHPC | bwHPC | SDS@hd | |||
Rclone mount | sftp (ssh), smb, WebDAV (https), ... | See previous table | ✓ | ✓ | ✓ | ✓ | ☓ | ✓ |
sshfs | see sftp | File system using sftp, simple usage, needs stable connections, can't resume partial transfers | ✓ | [✓] | ✓ | ☓ | ✓ ( L, M ) | ✓ |
smb | none | Mapping of a network drive, multiuser possible | ✓ | ✓ | ✓ | ☓ | ☓ | ✓ (M, W, L) |
NFSv4 | none | One-time installation effort (Kerberos, ID mapping). For servers, microscopes, multi-user environments, ... | ✓ | ☓ | ☓ | ☓ | ☓ | ✓ |
Graphical User Interfaces (GUIs)
GUIs | Underlying network protocols | Description & Use Cases | Executable on | Transfer from/to | ||||
---|---|---|---|---|---|---|---|---|
Linux | MacOS | Windows | bwHPC | bwHPC | SDS@hd | |||
JupyterLab | ssh | Development Environment | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
VSCode | ssh | Development Environment | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
RStudio | ssh | Development Environment | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
MobaXterm | ssh, RSync, sftp | ☓ | ☓ | ✓ | ☓ | ✓ ( W ) | ✓ | |
WinSCP | sftp, WebDAV, scp | ☓ | ☓ | ✓ | ☓ | ✓ | ✓ | |
Filezilla | sftp | ✓ | ✓ | ✓ | ☓ | ✓ | ✓ | |
Cyberduck | sftp, WebDAV, S3 | ☓ | ✓ | ✓ | ☓ | ✓ | ✓ | |
WebDAV via browser | WebDAV | download single files | ✓ | ✓ | ✓ | ☓ | ☓ | ✓ |