Data Transfer: Difference between revisions

Latest revision as of 19:53, 16 March 2023

Transfer Tools

Type	Software	Remarks	Executable on				Transfer from/to
Type	Software	Remarks	Local°	bwUniCluster	bwForCluster	www	bwHPC cluster	SDS@hd
Command-line	scp	Throughput < 150 MB/s (depending on cipher)	+	+	+		+
	sftp		+	+	+		+	+
	rsync		+	+	+		+
	rdata	Throughput of 350-400 MB/s		+				+
	wget	Download from http/ftp address only	+	+	+	+
Graphical	WinSCP	based on SCP/SFTP, Windows only	+				+	+
Graphical	FileZilla	based on SFTP	+				+	+

° Depending on the installed operating system (OS).

Linux/Unix/Mac commandline sftp/scp Usage Examples

sftp

> sftp  ka_xy1234@bwfilestorage.lsdf.kit.edu
Connecting to bwfilestorage.lsdf.kit.edu<br>
ka_xy1234@bwfilestorage.lsdf.kit.edu's password: 
sftp> ls
snapshots
temp test
sftp> help
...
sftp> put myfile
sftp> get myfile

scp

> scp mylocalfile ul_xy1234@justus2.uni-ulm.de: # copies to home directory

Using SFTP from Windows and Mac graphical clients

Windows clients do not have a SCP/SFTP client installed by default, so it needs to be installed before this protocol can be used.

Tools:

WinSCP (for Windows)
FileZilla (for Windows, Mac and Linux)

network drive over SFTP:

WebDrive (for Windows and Mac)
SFTP Net Drive (ELDOS) (for Windows)
NetDrive (for Windows)
ExpanDrive (for Windows and Mac)

Filezilla

Start FileZilla, Select "File -> Site Manager..." from the main menu and set up a new connection with the following settings:

Protocol: SFTP - SSH File Transfer Protocol
Host: <hostname>
Logon Typ: Interactive
User: <username>

Note: By default Filezilla will close the connection after 20 seconds of inactivity. In order to increase or disable this timeout, select "Edit -> Settings ... -> Connections" and increase "Timeout in seconds" to a reasonable value or set to 0 to disable connection timeout.

Best practices

Ciphers

Encrypting all the transferred data via scp/sftp takes time, which can become significant for really large data transfers.

In these cases, you can choose a faster encryption cipher to speed up that part of your data transfer via options to ssh/sftp. In our tests, these ciphers have had the listed transfer speedups over the default. If speedups are noticeable for you depends on processor type, network connection and the used hard disk.

Cipher	performance
chacha20-poly1305@openssh.com (default)	100%
aes128-gcm@openssh.com	~200%
aes128-ctr	~188%

With ssh/sshfs you can use different ciphers with the -c option:

ssh -c aes128-gcm@openssh.com

A list of available ciphers should be available with the command

ssh -Q cipher

@@ Line 1: / Line 1: @@
 == Transfer Tools ==
+{|class="wikitable"
-{|{{Table|width=99%}}
 |-
 ! rowspan="2" | Type
@@ Line 14: / Line 14: @@
 !www
 !bwHPC cluster
-![[Sds-hd|SDS@hd]]
+![[SDS@hd]]
 |-
 | rowspan="5" | Command-line
-| scp
+! scp
 | rowspan="3" | Throughput < 150 MB/s (depending on cipher)
 | style="text-align:center" | +
@@ Line 26: / Line 26: @@
 | style="text-align:center" |
 |-
-| sftp
+! sftp
 | style="text-align:center" | +
 | style="text-align:center" | +
@@ Line 34: / Line 34: @@
 | style="text-align:center" | +
 |-
-| rsync
+! rsync
 | style="text-align:center" | +
 | style="text-align:center" | +
@@ Line 42: / Line 42: @@
 | style="text-align:center" |
 |-
-| rdata
+! rdata
 | Throughput of 350-400 MB/s
 |
@@ Line 51: / Line 51: @@
 | style="text-align:center" | +
 |-
-| wget
+! wget
 | Download from http/ftp address only
 | style="text-align:center" | +
@@ Line 61: / Line 61: @@
 |-
 | rowspan="2" | Graphical
-| [https://winscp.net/eng/download.php WinSCP]
+! [https://winscp.net/eng/download.php WinSCP]
 | based on SCP/SFTP, Windows only
 | style="text-align:center" | +
@@ Line 70: / Line 70: @@
 | style="text-align:center" | +
 |-
-| [https://filezilla-project.org/download.php?show_all=1 FileZilla]
+! [https://filezilla-project.org/download.php?show_all=1 FileZilla]
 | based on SFTP
 | style="text-align:center" | +
@@ Line 82: / Line 82: @@
 ° Depending on the installed operating system (OS).
 == Linux/Unix/Mac commandline sftp/scp Usage Examples ==
 === sftp===
-<code>
+<pre>
 > sftp  ka_xy1234@bwfilestorage.lsdf.kit.edu
 Connecting to bwfilestorage.lsdf.kit.edu<br>
@@ Line 98: / Line 97: @@
 sftp> put myfile
 sftp> get myfile
-</code>
+</pre>
 === scp ===
 <code>
-> scp mylocalfile ul_xy1234@justus.uni-ulm.de: # copies to home directory
+> scp mylocalfile ul_xy1234@justus2.uni-ulm.de: # copies to home directory
 </code>
@@ Line 109: / Line 108: @@
 Windows clients do not have a SCP/SFTP client installed by default, so it needs to be installed before this protocol can be used.
-'''Tools for example:'''
+'''Tools:'''
-* [https://www.openssh.com/ OpenSSH]
-*[https://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Putty suite] (for Windows and Unix)
 *[https://winscp.net/eng/download.php WinSCP] (for Windows)
 *[https://filezilla-project.org/download.php?show_all=1 FileZilla] (for Windows, Mac and Linux)
-*[https://cygwin.com/install.html Cygwin] (for Windows)
 <br>
 '''network drive over SFTP:'''
@@ Line 121: / Line 117: @@
 *[https://www.netdrive.net/ NetDrive] (for Windows)
 *[https://www.expandrive.com/expandrive ExpanDrive] (for Windows and Mac)
+=== Filezilla ===
+Start FileZilla, Select "File -> Site Manager..." from the main menu and set up a new connection with the following settings:
+<pre>
+Protocol: SFTP - SSH File Transfer Protocol
+Host: <hostname>
+Logon Typ: Interactive
+User: <username>
+</pre>
+'''Note:''' By default Filezilla will close the connection after 20 seconds of inactivity. In order to increase or disable this timeout, select "Edit -> Settings ... -> Connections" and increase "Timeout in seconds" to a reasonable value or set to 0 to disable connection timeout.
 == Best practices ==

Data Transfer: Difference between revisions

Latest revision as of 19:53, 16 March 2023

Contents

Transfer Tools

Linux/Unix/Mac commandline sftp/scp Usage Examples

sftp

scp

Using SFTP from Windows and Mac graphical clients

Filezilla

Best practices

Ciphers

Navigation menu

Data Transfer: Difference between revisions

Latest revision as of 19:53, 16 March 2023

Transfer Tools

Linux/Unix/Mac commandline sftp/scp Usage Examples

sftp

scp

Using SFTP from Windows and Mac graphical clients

Filezilla

Best practices

Ciphers

Navigation menu

Search