Data Transfer: Difference between revisions
K Siegmund (talk | contribs) m (K Siegmund moved page BwFileStorage SFTP to Data Transfer) |
K Siegmund (talk | contribs) No edit summary |
||
| Line 84: | Line 84: | ||
== Linux/Unix/Mac commandline sftp/scp Usage Examples == |
|||
=== sftp=== |
|||
<H1>Using SFTP from Unix client</H1> |
|||
<code> |
|||
'''Example:''' |
|||
<pre> |
|||
> sftp ka_xy1234@bwfilestorage.lsdf.kit.edu |
> sftp ka_xy1234@bwfilestorage.lsdf.kit.edu |
||
Connecting to bwfilestorage.lsdf.kit.edu<br> |
Connecting to bwfilestorage.lsdf.kit.edu<br> |
||
| Line 100: | Line 98: | ||
sftp> put myfile |
sftp> put myfile |
||
sftp> get myfile |
sftp> get myfile |
||
</ |
</code> |
||
=== scp === |
|||
<code> |
|||
> scp mylocalfile ul_xy1234@justus.uni-ulm.de: # copies to home directory |
|||
</code> |
|||
== Using SFTP from Windows and Mac graphical clients == |
|||
Windows clients do not have a SCP/SFTP client installed by default, so it needs to be installed before this protocol can be used. |
Windows clients do not have a SCP/SFTP client installed by default, so it needs to be installed before this protocol can be used. |
||
| Line 119: | Line 121: | ||
*[https://www.netdrive.net/ NetDrive] (for Windows) |
*[https://www.netdrive.net/ NetDrive] (for Windows) |
||
*[https://www.expandrive.com/expandrive ExpanDrive] (for Windows and Mac) |
*[https://www.expandrive.com/expandrive ExpanDrive] (for Windows and Mac) |
||
<hr> |
|||
== Best practices == |
|||
<br> |
|||
<br> |
|||
=== Ciphers === |
|||
<br> |
|||
<br> |
|||
Encrypting all the transferred data via scp/sftp takes time, which can become significant for really large data transfers. |
|||
[[Category:bwFileStorage|SFTP]] |
|||
In these cases, you can choose a faster encryption cipher to speed up that part of your data transfer via options to ssh/sftp. |
|||
In our tests, these ciphers have had the listed transfer speedups over the default. If speedups are noticeable for you depends on processor type, network connection and the used hard disk. |
|||
{| class="wikitable" |
|||
!Cipher |
|||
!style="text-align:left;"| performance |
|||
|- |
|||
|chacha20-poly1305@openssh.com (default) |
|||
| 100% |
|||
|- |
|||
|aes128-gcm@openssh.com |
|||
|~200% |
|||
|- |
|||
|aes128-ctr |
|||
|~188% |
|||
|- |
|||
|} |
|||
With ssh/sshfs you can use different ciphers with the -c option: |
|||
<pre>ssh -c aes128-gcm@openssh.com</pre> |
|||
A list of available ciphers should be available with the command |
|||
<pre>ssh -Q cipher</pre> |
|||
Revision as of 14:47, 14 July 2022
Transfer Tools
| Type | Software | Remarks | Executable on | Transfer from/to | |||||
|---|---|---|---|---|---|---|---|---|---|
| Local° | bwUniCluster | bwForCluster | www | bwHPC cluster | SDS@hd | ||||
| Command-line | scp | Throughput < 150 MB/s (depending on cipher) | + | + | + | + | |||
| sftp | + | + | + | + | + | ||||
| rsync | + | + | + | + | |||||
| rdata | Throughput of 350-400 MB/s | + | + | ||||||
| wget | Download from http/ftp address only | + | + | + | + | ||||
| Graphical | WinSCP | based on SCP/SFTP, Windows only | + | + | + | ||||
| FileZilla | based on SFTP | + | + | + | |||||
° Depending on the installed operating system (OS).
Linux/Unix/Mac commandline sftp/scp Usage Examples
sftp
> sftp ka_xy1234@bwfilestorage.lsdf.kit.edu
Connecting to bwfilestorage.lsdf.kit.edu
ka_xy1234@bwfilestorage.lsdf.kit.edu's password:
sftp> ls
snapshots
temp test
sftp> help
...
sftp> put myfile
sftp> get myfile
scp
> scp mylocalfile ul_xy1234@justus.uni-ulm.de: # copies to home directory
Using SFTP from Windows and Mac graphical clients
Windows clients do not have a SCP/SFTP client installed by default, so it needs to be installed before this protocol can be used.
Tools for example:
- OpenSSH
- Putty suite (for Windows and Unix)
- WinSCP (for Windows)
- FileZilla (for Windows, Mac and Linux)
- Cygwin (for Windows)
network drive over SFTP:
- WebDrive (for Windows and Mac)
- SFTP Net Drive (ELDOS) (for Windows)
- NetDrive (for Windows)
- ExpanDrive (for Windows and Mac)
Best practices
Ciphers
Encrypting all the transferred data via scp/sftp takes time, which can become significant for really large data transfers.
In these cases, you can choose a faster encryption cipher to speed up that part of your data transfer via options to ssh/sftp. In our tests, these ciphers have had the listed transfer speedups over the default. If speedups are noticeable for you depends on processor type, network connection and the used hard disk.
| Cipher | performance |
|---|---|
| chacha20-poly1305@openssh.com (default) | 100% |
| aes128-gcm@openssh.com | ~200% |
| aes128-ctr | ~188% |
With ssh/sshfs you can use different ciphers with the -c option:
ssh -c aes128-gcm@openssh.com
A list of available ciphers should be available with the command
ssh -Q cipher