Data Transfer/Advanced Data Transfer

Advanced Data Transfer

This page presents advanced topics that can improve data transfer in special cases.

Adjust ciphers for faster transfer

A cipher is an encryption method. A specific cipher can be specified when using ssh, scp, rsync, sshfs or sftp. Changing the cipher can have a significant impact on the transmission speed, especially with big data files. The effects are difficult to predict because, among other things, they are depending on the client's processor. It appears that usually "aes128-ctr" is one of the fastest ciphers.

Usage

You can specify the cipher by adding the option -o Cipher=<cipher>. For example:

ssh -o Cipher=<cipher> <user>@<remotehost>
sshfs -o Cipher=<cipher> -o reconnect <user>@<remotehost>: <local_mountpoint>
rsync -av --rsh="ssh -o Cipher=<cipher>" ./path/to/local/file <user>@<remotehost>:  

To get a list of available ciphers on the cluster:

ssh -Q cipher

Attention: It has to be noted, that not all encryption methods meet the same security requirements. You have to consider the different performance and security requirements for your indiviual usecase.

Find fastest Cipher

You can test it with your own hardware by using the following code:

source=<provide the path to your file>
destination=<userID>@hostname:<path to destination>

for cipher in aes128-cbc aes256-cbc aes128-ctr aes256-ctr aes128-gcm@openssh.com aes256-gcm@openssh.com chacha20-poly1305@openssh.com; do
        echo "$cipher"
        for try in 1 2 ; do
                scp -c "$cipher" -p  $source $destination
                rsync -P --rsh="ssh -c $cipher" $source $destination
        done
done