Data Transfer: Difference between revisions
Jump to navigation
Jump to search
m (moved SCP to BwFileStorage SFTP) |
|||
| (36 intermediate revisions by 5 users not shown) | |||
| Line 1: | Line 1: | ||
== Transfer Tools == |
|||
#REDIRECT [[:Category:bwFileStorage]] |
|||
{|class="wikitable" |
|||
|- |
|||
! rowspan="2" | Type |
|||
! rowspan="2" | Software |
|||
! rowspan="2" | Remarks |
|||
! colspan="4" style="text-align:center" | Executable on |
|||
! colspan="3" style="text-align:center" | Transfer from/to |
|||
|- |
|||
!Local° |
|||
!bwUniCluster |
|||
!bwForCluster |
|||
!www |
|||
!bwHPC cluster |
|||
![[SDS@hd]] |
|||
|- |
|||
| rowspan="5" | Command-line |
|||
! scp |
|||
| rowspan="3" | Throughput < 150 MB/s (depending on cipher) |
|||
| style="text-align:center" | + |
|||
| style="text-align:center" | + |
|||
| style="text-align:center" | + |
|||
| |
|||
| style="text-align:center" | + |
|||
| style="text-align:center" | |
|||
|- |
|||
! sftp |
|||
| style="text-align:center" | + |
|||
| style="text-align:center" | + |
|||
| style="text-align:center" | + |
|||
| |
|||
| style="text-align:center" | + |
|||
| style="text-align:center" | + |
|||
|- |
|||
! rsync |
|||
| style="text-align:center" | + |
|||
| style="text-align:center" | + |
|||
| style="text-align:center" | + |
|||
| |
|||
| style="text-align:center" | + |
|||
| style="text-align:center" | |
|||
|- |
|||
! rdata |
|||
| Throughput of 350-400 MB/s |
|||
| |
|||
| style="text-align:center" | + |
|||
| |
|||
| |
|||
| |
|||
| style="text-align:center" | + |
|||
|- |
|||
! wget |
|||
| Download from http/ftp address only |
|||
| style="text-align:center" | + |
|||
| style="text-align:center" | + |
|||
| style="text-align:center" | + |
|||
| style="text-align:center" | + |
|||
| |
|||
| style="text-align:center" | |
|||
|- |
|||
| rowspan="2" | Graphical |
|||
! [https://winscp.net/eng/download.php WinSCP] |
|||
| based on SCP/SFTP, Windows only |
|||
| style="text-align:center" | + |
|||
| |
|||
| |
|||
| |
|||
| style="text-align:center" | + |
|||
| style="text-align:center" | + |
|||
|- |
|||
! [https://filezilla-project.org/download.php?show_all=1 FileZilla] |
|||
| based on SFTP |
|||
| style="text-align:center" | + |
|||
| |
|||
| |
|||
| |
|||
| style="text-align:center" | + |
|||
| style="text-align:center" | + |
|||
|- |
|||
|} |
|||
° Depending on the installed operating system (OS). |
|||
== Linux/Unix/Mac commandline sftp/scp Usage Examples == |
|||
=== sftp=== |
|||
<pre> |
|||
> sftp ka_xy1234@bwfilestorage.lsdf.kit.edu |
|||
Connecting to bwfilestorage.lsdf.kit.edu<br> |
|||
ka_xy1234@bwfilestorage.lsdf.kit.edu's password: |
|||
sftp> ls |
|||
snapshots |
|||
temp test |
|||
sftp> help |
|||
... |
|||
sftp> put myfile |
|||
sftp> get myfile |
|||
</pre> |
|||
=== scp === |
|||
<code> |
|||
> scp mylocalfile ul_xy1234@justus2.uni-ulm.de: # copies to home directory |
|||
</code> |
|||
== Using SFTP from Windows and Mac graphical clients == |
|||
Windows clients do not have a SCP/SFTP client installed by default, so it needs to be installed before this protocol can be used. |
|||
'''Tools:''' |
|||
*[https://winscp.net/eng/download.php WinSCP] (for Windows) |
|||
*[https://filezilla-project.org/download.php?show_all=1 FileZilla] (for Windows, Mac and Linux) |
|||
<br> |
|||
'''network drive over SFTP:''' |
|||
*[https://www.southrivertechnologies.com/download/downloadwd.html WebDrive] (for Windows and Mac) |
|||
*[https://www.eldos.com/sftp-net-drive/comparison.php SFTP Net Drive (ELDOS)] (for Windows) |
|||
*[https://www.netdrive.net/ NetDrive] (for Windows) |
|||
*[https://www.expandrive.com/expandrive ExpanDrive] (for Windows and Mac) |
|||
=== Filezilla === |
|||
Start FileZilla, Select "File -> Site Manager..." from the main menu and set up a new connection with the following settings: |
|||
<pre> |
|||
Protocol: SFTP - SSH File Transfer Protocol |
|||
Host: <hostname> |
|||
Logon Typ: Interactive |
|||
User: <username> |
|||
</pre> |
|||
'''Note:''' By default Filezilla will close the connection after 20 seconds of inactivity. In order to increase or disable this timeout, select "Edit -> Settings ... -> Connections" and increase "Timeout in seconds" to a reasonable value or set to 0 to disable connection timeout. |
|||
== Best practices == |
|||
=== Ciphers === |
|||
Encrypting all the transferred data via scp/sftp takes time, which can become significant for really large data transfers. |
|||
In these cases, you can choose a faster encryption cipher to speed up that part of your data transfer via options to ssh/sftp. |
|||
In our tests, these ciphers have had the listed transfer speedups over the default. If speedups are noticeable for you depends on processor type, network connection and the used hard disk. |
|||
{| class="wikitable" |
|||
!Cipher |
|||
!style="text-align:left;"| performance |
|||
|- |
|||
|chacha20-poly1305@openssh.com (default) |
|||
| 100% |
|||
|- |
|||
|aes128-gcm@openssh.com |
|||
|~200% |
|||
|- |
|||
|aes128-ctr |
|||
|~188% |
|||
|- |
|||
|} |
|||
With ssh/sshfs you can use different ciphers with the -c option: |
|||
<pre>ssh -c aes128-gcm@openssh.com</pre> |
|||
A list of available ciphers should be available with the command |
|||
<pre>ssh -Q cipher</pre> |
|||
Revision as of 19:53, 16 March 2023
Transfer Tools
| Type | Software | Remarks | Executable on | Transfer from/to | |||||
|---|---|---|---|---|---|---|---|---|---|
| Local° | bwUniCluster | bwForCluster | www | bwHPC cluster | SDS@hd | ||||
| Command-line | scp | Throughput < 150 MB/s (depending on cipher) | + | + | + | + | |||
| sftp | + | + | + | + | + | ||||
| rsync | + | + | + | + | |||||
| rdata | Throughput of 350-400 MB/s | + | + | ||||||
| wget | Download from http/ftp address only | + | + | + | + | ||||
| Graphical | WinSCP | based on SCP/SFTP, Windows only | + | + | + | ||||
| FileZilla | based on SFTP | + | + | + | |||||
° Depending on the installed operating system (OS).
Linux/Unix/Mac commandline sftp/scp Usage Examples
sftp
> sftp ka_xy1234@bwfilestorage.lsdf.kit.edu Connecting to bwfilestorage.lsdf.kit.edu<br> ka_xy1234@bwfilestorage.lsdf.kit.edu's password: sftp> ls snapshots temp test sftp> help ... sftp> put myfile sftp> get myfile
scp
> scp mylocalfile ul_xy1234@justus2.uni-ulm.de: # copies to home directory
Using SFTP from Windows and Mac graphical clients
Windows clients do not have a SCP/SFTP client installed by default, so it needs to be installed before this protocol can be used.
Tools:
network drive over SFTP:
- WebDrive (for Windows and Mac)
- SFTP Net Drive (ELDOS) (for Windows)
- NetDrive (for Windows)
- ExpanDrive (for Windows and Mac)
Filezilla
Start FileZilla, Select "File -> Site Manager..." from the main menu and set up a new connection with the following settings:
Protocol: SFTP - SSH File Transfer Protocol Host: <hostname> Logon Typ: Interactive User: <username>
Note: By default Filezilla will close the connection after 20 seconds of inactivity. In order to increase or disable this timeout, select "Edit -> Settings ... -> Connections" and increase "Timeout in seconds" to a reasonable value or set to 0 to disable connection timeout.
Best practices
Ciphers
Encrypting all the transferred data via scp/sftp takes time, which can become significant for really large data transfers.
In these cases, you can choose a faster encryption cipher to speed up that part of your data transfer via options to ssh/sftp. In our tests, these ciphers have had the listed transfer speedups over the default. If speedups are noticeable for you depends on processor type, network connection and the used hard disk.
| Cipher | performance |
|---|---|
| chacha20-poly1305@openssh.com (default) | 100% |
| aes128-gcm@openssh.com | ~200% |
| aes128-ctr | ~188% |
With ssh/sshfs you can use different ciphers with the -c option:
ssh -c aes128-gcm@openssh.com
A list of available ciphers should be available with the command
ssh -Q cipher