Data Transfer

Overview

Data transfer is the exchange of files between two systems. Before data transfer can happen, you need to go through the following steps:

1. Choose the two data storage systems that shall exchange data.
2. Choose the top level ways of transfer (copy, sync or mount) by considering your specific use case.
3. Choose a network protocol or transfer tool to use for the communication between the systems.

The recommended setup already includes these three steps. For a full overview, you can reference the tables that show all transfer routes. Those include all possible combinations between systems, top level way of transfer and network protocol / transfer tool.

Data Storage Systems

Data transfer can happen between a variety of systems. For example:

• Local computer or VM (virtual machine)
• Data producing machine (sequencer, microscope, ...)
• HPC system
• Storage space (SDS@hd, institute server, ...)
• Cloud resource

Ways of Transfer: Copy, Sync, Mount

The top level ways of transfer are:

• Copy: A simple copy command is the most basic way to transfer data. This is most efficient for very big data files that shall be retrieved from or moved to a remote location. And it can be most convenient, if you prefer moving your files via commandline instead of using a file browser.
  Examples: scp, sftp
• Sync: If the data is intended to be kept on both systems and undergoes change on only one of the systems, it makes sense to use a synchronization command instead. This way, only the changed files in one location are updated in the other location. Good use cases are backups or data transfers that go mostly in one direction like moving data from a sequencer to a storage space. A disadvantage is that the data needs storage space on both systems.
  Example: rsync
• Mount: If the data undergoes change on both systems or is too big to store locally, then mounting is the most convenient solution. This allows you to see and work with the data as if it were stored locally on your computer while it is still placed on the remote system. All changes that you implement happen directly on the original data so that you don't need to copy or synchronize anything. Additionally, you'll see all changes that another party does to the data with just a very short delay. Disadvantages are that you need defined edit sessions starting with a mount and ending with a clean unmount of your files and with a stable network connection during the session. Also, file operations on the remote system become much slower via a remotely sshfs-mounted system.
  Example: sshfs

Figure 1: Top level transfer routes

Network Protocols & Transfer Tools

Basic Network Protocol	Used By Network Protocol
ssh	scp, sftp, rsync
http(s)	WebDAV
smb	-
NFS	-

For every data transfer a network protocol to use for the communication between the systems must be chosen. The basic network protocols and the network protocols that build directly upon those are shown in the table on the right. These protocols can either be used rather directly or through tools that provide the protocol together with additional features. A tool can either mean a command line tool or a tool with a graphical user interface.

A comprehensive overview of all transfer options (network protocols and tools) can be found on the page all data transfer routes.

Recommended Setup

Main routes for mounting

When you are working with a development environment that allows remote connections, this is the first choice. Otherwise, the main tools/protocols for transferring data are as follows:

• Windows: MobaXterm is a graphical user interface that allows logging in to the cluster with ssh as well as transferring data via a file browser or using command line tools for the transfer.
• MacOS and Linux:
  • sshfs is the quickest way for mounting under stable connections.
  • Rclone for mount, copy and sync (not usable with 2FA).
• SDS@hd: See the SDS@hd Access page.

Best Practices

• Strong firewall restrictions
  -> Use ssh or http(s) based protocols, for example WebDav and sftp. For very strict facilities, ssh based protocols might not be allowed.
• Share data with collaborators...
  • ...outside of Baden-Württemberg
    -> Use the SDS@hd storage.
  • ...that are less comfortable with the command line
    -> Let them mount the folder.
• Transfer many small files
  -> Compress the files to one.

For advanced topics see Advanced Data Transfer.