Data Transfer/Rsync

Rsync is a command line tool used for singlethreaded, one-directional synchronization. It allows to only transfer files that have changed or that were newly created on the source side. Rsync even allows to only synchronize parts of a file that have changed instead of the whole file because it operates on block level instead of file level.

Usage

• rsync -av dir1/ dir2/
  Local synchronization of all files from dir1 to dir2

• rsync -av --delete dir1/ dir2/
  Local synchronization of all files from dir1 to dir2. Files that do not appear in dir1 are deleted in dir2.

• rsync -av dir1/ <user>@<remotehost>:/home/kit/ifkm/ej4555/dir2/
  Local to remote synchronization of all files from the local dir1 folder to the dir2 folder, which is accessed via the network using SSH.

• rsync -av <user>@<remotehost>:/home/kit/ifkm/ej4555/dir1/ dir2/
  Remote to local synchronization of files from the dir1 folder, which is accessed via the network using SSH, to the local dir2 folder.

In above examples, the following flags are used:

• -v: more verbose output
• -a: archive mode, meaning all file permissions and meta date is preserved, folders are also synced

Please note the trailing slashes / of the directories to be synchronized.

In order to allows rsync to work with partially downloaded files, the flag --partial can be used. Interrupted rsync sessions can be restarted where it left off by repeating the used command again.

# Execute in your local folder, where the file bigdata.tgz is placed.
$ rsync -P -e ssh <username>@<remotehost>:bigdata.tgz ./bigdata.tgz

• The parameter setting -e ssh tells rsync to use SSH as a remote shell to have a secure connection.
• The parameter -P (shortcut for --partial --progress) tells rsync to do partial sync on files and to be verbose.

Best Practices

If Rsync is not found on the remote host:
You can add the Rsync path as additional option:
--rsync-path=/usr/bin/rsync
You can find the path by using which rsync on the remotehost.

When the connection is slow:
Compress the data with the -z option to make the transfer faster.