Data Transfer/All Data Transfer Routes: Difference between revisions

From bwHPC Wiki
Jump to navigation Jump to search
m (corrected links)
m (Clarifying Rclone entries)
 
(3 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{| style=" background:#FEF4AB; width:100%;"
| style="padding:6px; font-size:100%;text-align:left" | This page is work in progress. To discover more relevant subpages, please come back at a later timepoint.
|-
|}
= All Data Transfer Routes =
= All Data Transfer Routes =


The following tables provide a full overview on the various transfer options depending on the system that you are coming from and the system you want to transfer your data to. The recommended options are marked in the table with '''W'''(indows)/'''M'''(acOS)/'''L'''(inux) added next to the '✓' sign.<br />
The following tables provide a full overview on the various transfer options depending on the system that you are coming from and the system you want to transfer your data to. <u>The recommended options are marked</u> in the table with '''W'''(indows)/'''M'''(acOS)/'''L'''(inux) added next to the '✓' sign.<br />
'[✓]' means that this option is possible but might be the worst choice of the given ones.
'[✓]' means that this option is possible but might be the worst choice of the given ones.


== Copy and Sync via commandline ==
== Copy and Sync via commandline ==


{| class="wikitable" style="vertical-align:middle;"
{| class="wikitable"
|- style="font-weight:bold;"
|- style="font-weight:bold; text-align:center; vertical-align:middle;"
! Protocol/Tool
! Network protocol
! Underlying network protocol
! Underlying network protocol
! Description/When to use
! Description/When to use
Line 22: Line 18:
! Transfer from/to
! Transfer from/to
!
!
|- style="vertical-align:middle;"
|-
|
|
|
|
|
|
| Linux
| Linux
| Mac
| MacOS
| Windows
| Windows
| bwUni Cluster
| bwUni Cluster
Line 34: Line 30:
| [[SDS@hd|SDS@hd]]
| [[SDS@hd|SDS@hd]]
|-
|-
| [[Data_Transfer/WebDAV|WebDav]] via [[Data_Transfer/Rclone|Rclone]]
| [[Data_Transfer/Rclone|Rclone]] + <protocol>
| sftp, smb, WebDAV, ...
| http(s)
| Needs installation. Multithreaded. Works on two completely remote locations as well. Transfers whole files.
| A tool that can be used with various protocols. It needs to be installed. Multithreaded. Works on two completely remote locations as well. Transfers whole files. No 2FA possible. Usually suitable for big data. Please refer to the marked recommendations when deciding on a network protocol to use with Rclone.
| ✓
| ✓
| ✓
| ✓
| ✓
| ✓
| ☓
| ☓
| ☓
| ✓
| ✓
|-
| sftp via [[Data_Transfer/Rclone|Rclone]]
| ssh
| see above
| ✓
| ✓
| only on BinAC
| ✓
| ✓
|- style="vertical-align:middle;"
| [[Data_Transfer/WebDAV|WebDAV]]
| http(s)
| Fast with small data, not suitable for big data.
| ✓
| ✓
| ✓
| ✓
| ✓
| ✓
| ✓
| ✓
| [✓]
|-
| [[Data_Transfer/Rsync|rsync]]
| ssh
| Synchronization of data. One side must be a local drive. Can spot differences in files and transfer only the changes.
| ✓
| ✓
| ☓
| ✓
| ✓
|- style="vertical-align:middle;"
| [✓]
| [[Data_Transfer/Rsync|rsync]]
| ✓
| ✓
| ✓
| ☓
|-
| [[Data_Transfer/SCP|scp]]
| ssh
| ssh
| Synchronization of data. One side must be a local drive. Can spot differences in files and transfer only the changes. Can resume partial transfers.
| For copying data
| ✓
| ✓
| ✓
| ✓
Line 77: Line 62:
| ✓
| ✓
| ☓
| ☓
|- style="vertical-align:middle;"
|-
| [[Data_Transfer/SFTP|sftp]]
| [[Data_Transfer/SFTP|sftp]]
| ssh
| ssh
| Interactive shell for copying data
| Interactive shell for copying data
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|[✓]]]
| [[Data_Transfer/SFTP#Windows|✓]]
| [[Data_Transfer/SFTP#Windows|✓]]
| ✓
| ✓
Line 88: Line 73:
| ✓
| ✓
| ✓
| ✓
|-
| [[Data_Transfer/SCP|scp]]
| ssh
| For copying data. Outdated according to: https://www.openssh.com/txt/release-8.0
| [✓]
| [✓]
| [✓]
| [✓]
| [✓]
| [✓]
| ☓
|}
|}

== Mounting via commandline ==
== Mounting via commandline ==

{| class="wikitable" style="vertical-align:middle;"
{| class="wikitable" style="vertical-align:middle;"
|- style="font-weight:bold;"
|- style="font-weight:bold; text-align:center;"
! Protocol/Tool
! Network protocol
! Underlying protocol
! Underlying network protocol
! Description/When to use
! Description & Use Cases
! Executable on
! colspan="4" | Executable on
! colspan="2" | Transfer from/to
!
!
!
!
! Transfer from/to
!
|-
|-
|
|
Line 108: Line 99:
|
|
| Linux
| Linux
| Mac
| MacOS
| Windows
| Windows
| bwHPC
| bwUniCluster
| bwHPC
| bwForCluster
| bwHPC cluster
| SDS@hd
| SDS@hd
|-
|-
| WebDAV via [[Data_Transfer/Rclone#Usage_Rclone_Mount|Rclone mount]]
| [[Data_Transfer/Rclone#Usage_Rclone_Mount|Rclone mount]] + <protocol>
| sftp, smb, WebDAV, ...
| http(s)
| See previous table
| Needs installation
| ✓
| ✓
| ✓
| ✓
| ✓
| ✓
| ✓
| ☓
| ☓
| ☓
| ☓
| ✓
| ✓
Line 128: Line 117:
| [[Data_Transfer/SSHFS|sshfs]]
| [[Data_Transfer/SSHFS|sshfs]]
| see sftp
| see sftp
| File system using sftp, simple usage, needs stable connections, can't resume partial transfers
|
| ✓
| ✓
| ✓
| ✓
| ✓
| [✓]
| ✓
| ✓
| ☓
| ✓ [[Data_Transfer/SSHFS#MacOS_&_Linux|( L, M )]]
| ✓ [[Data_Transfer/SSHFS#MacOS_&_Linux|( L, M )]]
| ✓
| ✓
Line 139: Line 127:
| [[SDS@hd/Access/SMB|smb]]
| [[SDS@hd/Access/SMB|smb]]
| none
| none
| Mapping of a network drive
| Mapping of a network drive. Multiuser possible. Good fit for big data.
| ✓
| [[SDS@hd/Access/SMB#Linux|L]]
| ✓
| ✓
| ✓
| ✓
| ☓
| ☓
| ☓
| ☓
| ✓ ([[SDS@hd/Access/SMB#Mac|M]], [[SDS@hd/Access/SMB#Windows|W]], [[SDS@hd/Access/SMB#Linux|L]])
| ☓
| ✓ ([[SDS@hd/Access/SMB#Mac|M]], [[SDS@hd/Access/SMB#Windows|W]])
|-
|-
| [[SDS@hd/Access/NFS#Using_NFSv4_for_UNIX_client|NFSv4]]
| [[SDS@hd/Access/NFS#Using_NFSv4_for_UNIX_client|NFSv4]]
Line 152: Line 139:
| One-time installation effort (Kerberos, ID mapping). For servers, microscopes, multi-user environments, ...
| One-time installation effort (Kerberos, ID mapping). For servers, microscopes, multi-user environments, ...
| ✓
| ✓
| ☓
| ☓
| ☓
| ☓
| ☓
Line 161: Line 147:
== Graphical User Interfaces (GUIs) ==
== Graphical User Interfaces (GUIs) ==


{| class="wikitable" style="vertical-align:middle;"
{| class="wikitable"
|- style="font-weight:bold;"
|- style="font-weight:bold; text-align:center; vertical-align:middle;"
! GUIs
! GUIs
! Underlying protocols
! Underlying network protocols
! Description/When to use
! Description & Use Cases
! Executable on
! colspan="4" | Executable on
! colspan="2" | Transfer from/to
!
|- style="vertical-align:middle;"
!
!
!
! Transfer from/to
!
|-
|
|
|
|
|
|
| Linux
| Linux
| Mac
| MacOS
| Windows
| Windows
| bwHPC
| bwUniCluster
| bwHPC
| bwForCluster
| bwHPC cluster
| SDS@hd
| SDS@hd
|-
|-
| JupyterLab
| ssh
| Development Environment
| ✓
| ✓
| ✓
| ✓
| ✓
| ✓
|-
| [[Development/VS_Code|VS Code]]
| ssh
| Development Environment
| ✓
| ✓
| ✓
| ✓
| ✓
| ✓
|-
| RStudio
| ssh
| Development Environment
| ✓
| ✓
| ✓
| ✓
| ✓
| ✓
|- style="vertical-align:middle;"
| [[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]]
| [[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]]
| SSH, RSync, SFTP
| ssh, RSync, sftp
|
|
| ☓
| ☓
| ☓
| ☓
| ✓
| ✓
| ☓
| ☓
| ☓
| ✓ [[Data_Transfer/Graphical_Clients#MobaXterm|( W )]]
| ✓ [[Data_Transfer/Graphical_Clients#MobaXterm|( W )]]
| ✓
| ✓
|- style="vertical-align:middle;"
|-
| [[Data_Transfer/Graphical_Clients#WinSCP|WinSCP]]
| [[Data_Transfer/Graphical_Clients#WinSCP|WinSCP]]
| SFTP, WebDAV, SCP
| sftp, WebDAV, scp
|
|
| ☓
| ☓
| ☓
| ☓
| ✓
| ✓
| ☓
| ☓
| ☓
| ✓
| ✓
| ✓
| ✓
|- style="vertical-align:middle;"
|-
| [[Data_Transfer/Graphical_Clients#Filezilla|Filezilla]]
| [[Data_Transfer/Graphical_Clients#Filezilla|Filezilla]]
| SFTP
| sftp
|
|
| ✓
| ✓
| ✓
| ✓
| ✓
| ✓
| ☓
| ☓
| ☓
| ✓
| ✓
| ✓
| ✓
|- style="vertical-align:middle;"
|-
| [https://cyberduck.io/ Cyberduck]
| [https://cyberduck.io/ Cyberduck]
| SFTP, WebDAV
| sftp, WebDAV, S3
|
|
| ☓
| ☓
| ✓
| ✓
| ✓
| ✓
| ☓
| ☓
| ☓
| ✓
| ✓
| ✓
| ✓
|- style="vertical-align:middle;"
|-
| WebDAV via browser
| WebDAV via browser
| WebDAV
| WebDAV
Line 235: Line 241:
| ✓
| ✓
| ✓
| ✓
| ☓
| ☓
| ☓
| ☓
| ☓

Latest revision as of 00:51, 13 March 2025

All Data Transfer Routes

The following tables provide a full overview on the various transfer options depending on the system that you are coming from and the system you want to transfer your data to. The recommended options are marked in the table with W(indows)/M(acOS)/L(inux) added next to the '✓' sign.
'[✓]' means that this option is possible but might be the worst choice of the given ones.

Copy and Sync via commandline

Protocol/Tool Underlying network protocol Description/When to use Executable on Transfer from/to
Linux MacOS Windows bwUni Cluster bwFor Cluster bwHPC Cluster SDS@hd
Rclone + <protocol> sftp, smb, WebDAV, ... A tool that can be used with various protocols. It needs to be installed. Multithreaded. Works on two completely remote locations as well. Transfers whole files. No 2FA possible. Usually suitable for big data. Please refer to the marked recommendations when deciding on a network protocol to use with Rclone. ✓ ✓ ✓ ✓ ✓ only on BinAC ✓
WebDAV http(s) Fast with small data, not suitable for big data. ✓ ✓ ✓ ✓ ✓ ☓ ✓
rsync ssh Synchronization of data. One side must be a local drive. Can spot differences in files and transfer only the changes. Can resume partial transfers. ✓ ✓ [✓] ✓ ✓ ✓ ☓
sftp ssh Interactive shell for copying data ✓ [✓] ✓ ✓ ✓ ✓ ✓
scp ssh For copying data. Outdated according to: https://www.openssh.com/txt/release-8.0 [✓] [✓] [✓] [✓] [✓] [✓] ☓

Mounting via commandline

Protocol/Tool Underlying network protocol Description & Use Cases Executable on Transfer from/to
Linux MacOS Windows bwHPC bwHPC SDS@hd
Rclone mount + <protocol> sftp, smb, WebDAV, ... See previous table ✓ ✓ ✓ ✓ ☓ ✓
sshfs see sftp File system using sftp, simple usage, needs stable connections, can't resume partial transfers ✓ [✓] ✓ ☓ ✓ ( L, M ) ✓
smb none Mapping of a network drive. Multiuser possible. Good fit for big data. ✓ ✓ ✓ ☓ ☓ ✓ (M, W, L)
NFSv4 none One-time installation effort (Kerberos, ID mapping). For servers, microscopes, multi-user environments, ... ✓ ☓ ☓ ☓ ☓ ✓

Graphical User Interfaces (GUIs)

GUIs Underlying network protocols Description & Use Cases Executable on Transfer from/to
Linux MacOS Windows bwHPC bwHPC SDS@hd
JupyterLab ssh Development Environment ✓ ✓ ✓ ✓ ✓ ✓
VS Code ssh Development Environment ✓ ✓ ✓ ✓ ✓ ✓
RStudio ssh Development Environment ✓ ✓ ✓ ✓ ✓ ✓
MobaXterm ssh, RSync, sftp ☓ ☓ ✓ ☓ ✓ ( W ) ✓
WinSCP sftp, WebDAV, scp ☓ ☓ ✓ ☓ ✓ ✓
Filezilla sftp ✓ ✓ ✓ ☓ ✓ ✓
Cyberduck sftp, WebDAV, S3 ☓ ✓ ✓ ☓ ✓ ✓
WebDAV via browser WebDAV download single files ✓ ✓ ✓ ☓ ☓ ✓
Retrieved from "https://wiki.bwhpc.de/wiki/index.php?title=Data_Transfer/All_Data_Transfer_Routes&oldid=14344"