Data Transfer
Transfer Tools
| Type | Software | Remarks | Executable on | Transfer from/to | |||||
|---|---|---|---|---|---|---|---|---|---|
| Local° | bwUniCluster | bwForCluster | www | bwHPC cluster | SDS@hd | ||||
| Command-line | scp | Throughput < 150 MB/s (depending on cipher) | + | + | + | + | |||
| sftp | + | + | + | + | + | ||||
| rsync | + | + | + | + | |||||
| rdata | Throughput of 350-400 MB/s | + | + | ||||||
| wget | Download from http/ftp address only | + | + | + | + | ||||
| Graphical | WinSCP | based on SCP/SFTP, Windows only | + | + | + | ||||
| FileZilla | based on SFTP | + | + | + | |||||
° Depending on the installed operating system (OS).
Linux/Unix/Mac commandline sftp/scp Usage Examples
sftp
> sftp ka_xy1234@bwfilestorage.lsdf.kit.edu
Connecting to bwfilestorage.lsdf.kit.edu
ka_xy1234@bwfilestorage.lsdf.kit.edu's password:
sftp> ls
snapshots
temp test
sftp> help
...
sftp> put myfile
sftp> get myfile
scp
> scp mylocalfile ul_xy1234@justus.uni-ulm.de: # copies to home directory
Using SFTP from Windows and Mac graphical clients
Windows clients do not have a SCP/SFTP client installed by default, so it needs to be installed before this protocol can be used.
Tools for example:
- OpenSSH
- Putty suite (for Windows and Unix)
- WinSCP (for Windows)
- FileZilla (for Windows, Mac and Linux)
- Cygwin (for Windows)
network drive over SFTP:
- WebDrive (for Windows and Mac)
- SFTP Net Drive (ELDOS) (for Windows)
- NetDrive (for Windows)
- ExpanDrive (for Windows and Mac)
Best practices
Ciphers
Encrypting all the transferred data via scp/sftp takes time, which can become significant for really large data transfers.
In these cases, you can choose a faster encryption cipher to speed up that part of your data transfer via options to ssh/sftp. In our tests, these ciphers have had the listed transfer speedups over the default. If speedups are noticeable for you depends on processor type, network connection and the used hard disk.
| Cipher | performance |
|---|---|
| chacha20-poly1305@openssh.com (default) | 100% |
| aes128-gcm@openssh.com | ~200% |
| aes128-ctr | ~188% |
With ssh/sshfs you can use different ciphers with the -c option:
ssh -c aes128-gcm@openssh.com
A list of available ciphers should be available with the command
ssh -Q cipher