Difference between revisions of "Data Transfer"

From bwHPC Wiki
Jump to: navigation, search
(Transfer Tools)
(13 intermediate revisions by 4 users not shown)
Line 1: Line 1:
  +
== Transfer Tools ==
<H1>Using SFTP from Unix client</H1>
 
   
  +
{|class="wikitable"
'''Example:'''
 
  +
|-
  +
! rowspan="2" | Type
  +
! rowspan="2" | Software
  +
! rowspan="2" | Remarks
  +
! colspan="4" style="text-align:center" | Executable on
  +
! colspan="3" style="text-align:center" | Transfer from/to
  +
|-
  +
!Local°
  +
!bwUniCluster
  +
!bwForCluster
  +
!www
  +
!bwHPC cluster
  +
![[SDS@hd]]
  +
|-
  +
| rowspan="5" | Command-line
  +
! scp
  +
| rowspan="3" | Throughput < 150 MB/s (depending on cipher)
  +
| style="text-align:center" | +
  +
| style="text-align:center" | +
  +
| style="text-align:center" | +
  +
|
  +
| style="text-align:center" | +
  +
| style="text-align:center" |
  +
|-
  +
! sftp
  +
| style="text-align:center" | +
  +
| style="text-align:center" | +
  +
| style="text-align:center" | +
  +
|
  +
| style="text-align:center" | +
  +
| style="text-align:center" | +
  +
|-
  +
! rsync
  +
| style="text-align:center" | +
  +
| style="text-align:center" | +
  +
| style="text-align:center" | +
  +
|
  +
| style="text-align:center" | +
  +
| style="text-align:center" |
  +
|-
  +
! rdata
  +
| Throughput of 350-400 MB/s
  +
|
  +
| style="text-align:center" | +
  +
|
  +
|
  +
|
  +
| style="text-align:center" | +
  +
|-
  +
! wget
  +
| Download from http/ftp address only
  +
| style="text-align:center" | +
  +
| style="text-align:center" | +
  +
| style="text-align:center" | +
  +
| style="text-align:center" | +
  +
|
  +
| style="text-align:center" |
  +
|-
  +
| rowspan="2" | Graphical
  +
! [https://winscp.net/eng/download.php WinSCP]
  +
| based on SCP/SFTP, Windows only
  +
| style="text-align:center" | +
  +
|
  +
|
  +
|
  +
| style="text-align:center" | +
  +
| style="text-align:center" | +
  +
|-
  +
! [https://filezilla-project.org/download.php?show_all=1 FileZilla]
  +
| based on SFTP
  +
| style="text-align:center" | +
  +
|
  +
|
  +
|
  +
| style="text-align:center" | +
  +
| style="text-align:center" | +
  +
|-
  +
|}
   
  +
° Depending on the installed operating system (OS).
  +
  +
== Linux/Unix/Mac commandline sftp/scp Usage Examples ==
  +
  +
=== sftp===
 
<pre>
 
<pre>
 
> sftp ka_xy1234@bwfilestorage.lsdf.kit.edu
 
> sftp ka_xy1234@bwfilestorage.lsdf.kit.edu
Line 16: Line 99:
 
</pre>
 
</pre>
   
  +
=== scp ===
  +
<code>
  +
> scp mylocalfile ul_xy1234@justus.uni-ulm.de: # copies to home directory
  +
</code>
   
<H1>Using SFTP from Windows and Mac client</H1>
+
== Using SFTP from Windows and Mac graphical clients ==
   
 
Windows clients do not have a SCP/SFTP client installed by default, so it needs to be installed before this protocol can be used.
 
Windows clients do not have a SCP/SFTP client installed by default, so it needs to be installed before this protocol can be used.
Line 26: Line 113:
 
*[https://winscp.net/eng/download.php WinSCP] (for Windows)
 
*[https://winscp.net/eng/download.php WinSCP] (for Windows)
 
*[https://filezilla-project.org/download.php?show_all=1 FileZilla] (for Windows, Mac and Linux)
 
*[https://filezilla-project.org/download.php?show_all=1 FileZilla] (for Windows, Mac and Linux)
*[http://cygwin.com/install.html Cygwin] (for Windows)
+
*[https://cygwin.com/install.html Cygwin] (for Windows)
 
<br>
 
<br>
 
'''network drive over SFTP:'''
 
'''network drive over SFTP:'''
Line 33: Line 120:
 
*[https://www.netdrive.net/ NetDrive] (for Windows)
 
*[https://www.netdrive.net/ NetDrive] (for Windows)
 
*[https://www.expandrive.com/expandrive ExpanDrive] (for Windows and Mac)
 
*[https://www.expandrive.com/expandrive ExpanDrive] (for Windows and Mac)
  +
<hr>
 
  +
== Best practices ==
<br>
 
  +
<br>
 
  +
=== Ciphers ===
<br>
 
  +
<br>
 
  +
Encrypting all the transferred data via scp/sftp takes time, which can become significant for really large data transfers.
[[Category:bwFileStorage|SFTP]]
 
  +
  +
In these cases, you can choose a faster encryption cipher to speed up that part of your data transfer via options to ssh/sftp.
  +
In our tests, these ciphers have had the listed transfer speedups over the default. If speedups are noticeable for you depends on processor type, network connection and the used hard disk.
  +
  +
{| class="wikitable"
  +
!Cipher
  +
!style="text-align:left;"| performance
  +
|-
  +
|chacha20-poly1305@openssh.com (default)
  +
| 100%
  +
|-
  +
|aes128-gcm@openssh.com
  +
|~200%
  +
|-
  +
|aes128-ctr
  +
|~188%
  +
|-
  +
|}
  +
  +
With ssh/sshfs you can use different ciphers with the -c option:
  +
  +
<pre>ssh -c aes128-gcm@openssh.com</pre>
  +
  +
A list of available ciphers should be available with the command
  +
  +
<pre>ssh -Q cipher</pre>

Revision as of 17:50, 19 August 2022

1 Transfer Tools

Type Software Remarks Executable on Transfer from/to
Local° bwUniCluster bwForCluster www bwHPC cluster SDS@hd
Command-line scp Throughput < 150 MB/s (depending on cipher) + + + +
sftp + + + + +
rsync + + + +
rdata Throughput of 350-400 MB/s + +
wget Download from http/ftp address only + + + +
Graphical WinSCP based on SCP/SFTP, Windows only + + +
FileZilla based on SFTP + + +

° Depending on the installed operating system (OS).

2 Linux/Unix/Mac commandline sftp/scp Usage Examples

2.1 sftp

> sftp  ka_xy1234@bwfilestorage.lsdf.kit.edu
Connecting to bwfilestorage.lsdf.kit.edu<br>
ka_xy1234@bwfilestorage.lsdf.kit.edu's password: 
sftp> ls
snapshots
temp test
sftp> help
...
sftp> put myfile
sftp> get myfile

2.2 scp

> scp mylocalfile ul_xy1234@justus.uni-ulm.de: # copies to home directory

3 Using SFTP from Windows and Mac graphical clients

Windows clients do not have a SCP/SFTP client installed by default, so it needs to be installed before this protocol can be used.

Tools for example:


network drive over SFTP:

4 Best practices

4.1 Ciphers

Encrypting all the transferred data via scp/sftp takes time, which can become significant for really large data transfers.

In these cases, you can choose a faster encryption cipher to speed up that part of your data transfer via options to ssh/sftp. In our tests, these ciphers have had the listed transfer speedups over the default. If speedups are noticeable for you depends on processor type, network connection and the used hard disk.

Cipher performance
chacha20-poly1305@openssh.com (default) 100%
aes128-gcm@openssh.com ~200%
aes128-ctr ~188%

With ssh/sshfs you can use different ciphers with the -c option:

ssh -c aes128-gcm@openssh.com

A list of available ciphers should be available with the command

ssh -Q cipher