Data Transfer/Rsync - Revision history

S Braun: /* Usage */

2025-03-31T08:11:37Z

Usage

← Older revision		Revision as of 10:11, 31 March 2025
Line 30:		Line 30:

	* The parameter setting <code>-e ssh</code> tells rsync to use SSH as a remote shell to have a secure connection.		* The parameter setting <code>-e ssh</code> tells rsync to use SSH as a remote shell to have a secure connection.
	* The parameter <code>-P</code> (shortcut for <code>--partial --progress</code>) tells rsync to do partial sync on files.		* The parameter <code>-P</code> (shortcut for <code>--partial --progress</code>) tells rsync to do partial sync on files and to be verbose.

	== Best Practices ==		== Best Practices ==

S Braun at 08:10, 31 March 2025

2025-03-31T08:10:52Z

← Older revision		Revision as of 10:10, 31 March 2025
Line 29:		Line 29:
	$ rsync -P -e ssh <username>@<remotehost>:bigdata.tgz ./bigdata.tgz</pre>		$ rsync -P -e ssh <username>@<remotehost>:bigdata.tgz ./bigdata.tgz</pre>

	* The parameter setting <code>-~~-rsh=~~ssh</code> tells rsync to use SSH as a remote shell to have a secure connection.		* The parameter setting <code>-e ssh</code> tells rsync to use SSH as a remote shell to have a secure connection.
	* The parameter <code>-P</code> (shortcut for <code>--partial --progress</code>) tells rsync to do partial sync on files.		* The parameter <code>-P</code> (shortcut for <code>--partial --progress</code>) tells rsync to do partial sync on files.

S Braun at 08:10, 31 March 2025

2025-03-31T08:10:15Z

← Older revision		Revision as of 10:10, 31 March 2025
Line 8:		Line 8:

	-->		-->
	== Usage ~~Examples~~ ==		== Usage ==
	* <code>rsync -av dir1/ dir2/</code><br>Local synchronization of all files from dir1 to dir2		* <code>rsync -av dir1/ dir2/</code><br>Local synchronization of all files from dir1 to dir2

Line 17:		Line 17:
	* <code>rsync -av <user>@<remotehost>:/home/kit/ifkm/ej4555/dir1/ dir2/</code><br>Remote to local synchronization of files from the dir1 folder, which is accessed via the network using SSH, to the local dir2 folder.		* <code>rsync -av <user>@<remotehost>:/home/kit/ifkm/ej4555/dir1/ dir2/</code><br>Remote to local synchronization of files from the dir1 folder, which is accessed via the network using SSH, to the local dir2 folder.

			In above examples, the following flags are used:<br>
			* <code>-v</code>: more verbose output
			* <code>-a</code>: archive mode, meaning all file permissions and meta date is preserved, folders are also synced

			Please note the trailing slashes <code>/</code> of the directories to be synchronized.
⚫	~~The~~ ~~parameter~~ ~~<code>--partial</code>~~ allows ~~Rsync~~ to work with partially downloaded files. Interrupted ~~Rsync~~ sessions can be restarted where it left off by repeating the used command again. <br />

⚫	The parameter setting <code>--rsh=ssh</code> tells ~~Rsync~~ to use ~~ssh~~ as a remote shell to have a secure connection.
		⚫	In order to allows rsync to work with partially downloaded files, the flag <code>--partial</code> can be used. Interrupted rsync sessions can be restarted where it left off by repeating the used command again.<br />

	<pre>		<pre>
	# Execute in your local folder, where the file bigdata.tgz is placed.		# Execute in your local folder, where the file bigdata.tgz is placed.
	$ rsync -P -~~-rsh=~~ssh <username>@<remotehost>:bigdata.tgz ./bigdata.tgz</pre>		$ rsync -P -e ssh <username>@<remotehost>:bigdata.tgz ./bigdata.tgz</pre>

		⚫	* The parameter setting <code>--rsh=ssh</code> tells rsync to use SSH as a remote shell to have a secure connection.
			* The parameter <code>-P</code> (shortcut for <code>--partial --progress</code>) tells rsync to do partial sync on files.

	== Best Practices ==		== Best Practices ==

S Braun: /* Usage */

2025-03-31T07:53:49Z

Usage

← Older revision		Revision as of 09:53, 31 March 2025
Line 8:		Line 8:

	-->		-->
	== Usage ==		== Usage Examples ==
			* <code>rsync -av dir1/ dir2/</code><br>Local synchronization of all files from dir1 to dir2
⚫	The parameter <code>~~-P (~~--partial~~/--progress)~~</code> allows Rsync to work with partially downloaded files. Interrupted Rsync sessions can be restarted where it left off by repeating the used command again. <br />

			* <code>rsync -av --delete dir1/ dir2/</code><br>Local synchronization of all files from dir1 to dir2. Files that do not appear in dir1 are deleted in dir2.

			* <code>rsync -av dir1/ <user>@<remotehost>:/home/kit/ifkm/ej4555/dir2/</code><br>Local to remote synchronization of all files from the local dir1 folder to the dir2 folder, which is accessed via the network using SSH.

			* <code>rsync -av <user>@<remotehost>:/home/kit/ifkm/ej4555/dir1/ dir2/</code><br>Remote to local synchronization of files from the dir1 folder, which is accessed via the network using SSH, to the local dir2 folder.


		⚫	The parameter <code>--partial</code> allows Rsync to work with partially downloaded files. Interrupted Rsync sessions can be restarted where it left off by repeating the used command again. <br />
	The parameter setting <code>--rsh=ssh</code> tells Rsync to use ssh as a remote shell to have a secure connection.		The parameter setting <code>--rsh=ssh</code> tells Rsync to use ssh as a remote shell to have a secure connection.

S Braun at 07:46, 31 March 2025

2025-03-31T07:46:58Z

← Older revision		Revision as of 09:46, 31 March 2025
Line 1:		Line 1:
	__FORCETOC__		__FORCETOC__
	Rsync is a command line tool used for singlethreaded, one-directional synchronization. It allows to only transfer the parts of a file~~(s)~~ that have changed instead of the whole file~~(s)~~ because it operates on block level instead of file level.		Rsync is a command line tool used for singlethreaded, one-directional synchronization. It allows to only transfer files that have changed or that were newly created on the source side. Rsync even allows to only synchronize parts of a file that have changed instead of the whole file because it operates on block level instead of file level.

			<!--
	If you want to mount a folder or if you want to access data on a cloud storage system like [[SDS@hd\|SDS@hd]] or Nextcloud, you should use [[Data_Transfer/Rclone\|Rclone]]. As Rclone is more broadly usable than Rsync, it is the preferred transfer method. If you only want to synchronize your data with a machine that is accessible via ssh (bwHPC clusters) then you can use Rsync for this.		If you want to mount a folder or if you want to access data on a cloud storage system like [[SDS@hd\|SDS@hd]] or Nextcloud, you should use [[Data_Transfer/Rclone\|Rclone]]. As Rclone is more broadly usable than Rsync, it is the preferred transfer method. If you only want to synchronize your data with a machine that is accessible via ssh (bwHPC clusters) then you can use Rsync for this.

	'''Caution:''' If you work on the data collaboratively, synchronization can lead to merge errors when data was changed by multiple parties.		'''Caution:''' If you work on the data collaboratively, synchronization can lead to merge errors when data was changed by multiple parties.

			-->
	== Usage ==		== Usage ==
	The parameter <code>-P (--partial/--progress)</code> allows Rsync to work with partially downloaded files. Interrupted Rsync sessions can be restarted where it left off by repeating the used command again. <br />		The parameter <code>-P (--partial/--progress)</code> allows Rsync to work with partially downloaded files. Interrupted Rsync sessions can be restarted where it left off by repeating the used command again. <br />

S Braun: /* Usage */

2025-02-26T14:52:36Z

Usage

← Older revision		Revision as of 16:52, 26 February 2025
Line 12:		Line 12:
	<pre>		<pre>
	# Execute in your local folder, where the file bigdata.tgz is placed.		# Execute in your local folder, where the file bigdata.tgz is placed.
	rsync -P --rsh=ssh <username>@<remotehost>:bigdata.tgz ./bigdata.tgz</pre>		$ rsync -P --rsh=ssh <username>@<remotehost>:bigdata.tgz ./bigdata.tgz</pre>

	== Best Practices ==		== Best Practices ==

H Schumacher: created page

2024-11-28T23:25:32Z

created page

New page

__FORCETOC__
Rsync is a command line tool used for singlethreaded, one-directional synchronization. It allows to only transfer the parts of a file(s) that have changed instead of the whole file(s) because it operates on block level instead of file level.

If you want to mount a folder or if you want to access data on a cloud storage system like [[SDS@hd|SDS@hd]] or Nextcloud, you should use [[Data_Transfer/Rclone|Rclone]]. As Rclone is more broadly usable than Rsync, it is the preferred transfer method. If you only want to synchronize your data with a machine that is accessible via ssh (bwHPC clusters) then you can use Rsync for this.

'''Caution:''' If you work on the data collaboratively, synchronization can lead to merge errors when data was changed by multiple parties.

== Usage ==
The parameter <code>-P (--partial/--progress)</code> allows Rsync to work with partially downloaded files. Interrupted Rsync sessions can be restarted where it left off by repeating the used command again. 
The parameter setting <code>--rsh=ssh</code> tells Rsync to use ssh as a remote shell to have a secure connection.

<pre>
# Execute in your local folder, where the file bigdata.tgz is placed.
rsync -P --rsh=ssh <username>@<remotehost>:bigdata.tgz ./bigdata.tgz</pre>
== Best Practices ==

'''If Rsync is not found on the remote host:''' 
You can add the Rsync path as additional option: 
<code>--rsync-path=/usr/bin/rsync</code> 
You can find the path by using <code>which rsync</code> on the remotehost.

'''When the connection is slow:''' 
Compress the data with the <code>-z</code> option to make the transfer faster.