Data Transfer

From bwHPC Wiki
Revision as of 19:53, 16 March 2023 by J Salk (talk | contribs) (Filezilla)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

1 Transfer Tools

Type Software Remarks Executable on Transfer from/to
Local° bwUniCluster bwForCluster www bwHPC cluster SDS@hd
Command-line scp Throughput < 150 MB/s (depending on cipher) + + + +
sftp + + + + +
rsync + + + +
rdata Throughput of 350-400 MB/s + +
wget Download from http/ftp address only + + + +
Graphical WinSCP based on SCP/SFTP, Windows only + + +
FileZilla based on SFTP + + +

° Depending on the installed operating system (OS).

2 Linux/Unix/Mac commandline sftp/scp Usage Examples

2.1 sftp

> sftp
Connecting to<br>'s password: 
sftp> ls
temp test
sftp> help
sftp> put myfile
sftp> get myfile

2.2 scp

> scp mylocalfile # copies to home directory

3 Using SFTP from Windows and Mac graphical clients

Windows clients do not have a SCP/SFTP client installed by default, so it needs to be installed before this protocol can be used.


network drive over SFTP:

3.1 Filezilla

Start FileZilla, Select "File -> Site Manager..." from the main menu and set up a new connection with the following settings:

Protocol: SFTP - SSH File Transfer Protocol
Host: <hostname>
Logon Typ: Interactive
User: <username>

Note: By default Filezilla will close the connection after 20 seconds of inactivity. In order to increase or disable this timeout, select "Edit -> Settings ... -> Connections" and increase "Timeout in seconds" to a reasonable value or set to 0 to disable connection timeout.

4 Best practices

4.1 Ciphers

Encrypting all the transferred data via scp/sftp takes time, which can become significant for really large data transfers.

In these cases, you can choose a faster encryption cipher to speed up that part of your data transfer via options to ssh/sftp. In our tests, these ciphers have had the listed transfer speedups over the default. If speedups are noticeable for you depends on processor type, network connection and the used hard disk.

Cipher performance (default) 100% ~200%
aes128-ctr ~188%

With ssh/sshfs you can use different ciphers with the -c option:

ssh -c

A list of available ciphers should be available with the command

ssh -Q cipher