Data Transfer - Revision history

H Schumacher: Added ssh key info.

2026-05-13T14:42:13Z

Added ssh key info.

← Older revision		Revision as of 16:42, 13 May 2026
Line 60:		Line 60:
	* <u>Windows</u>: '''[[Data_Transfer/Graphical_Clients#MobaXterm\|MobaXterm]]''' is a graphical user interface that allows logging in to the cluster with ssh as well as transferring data via a file browser or using command line tools for the transfer.		* <u>Windows</u>: '''[[Data_Transfer/Graphical_Clients#MobaXterm\|MobaXterm]]''' is a graphical user interface that allows logging in to the cluster with ssh as well as transferring data via a file browser or using command line tools for the transfer.
	* <u>MacOS and Linux</u>:		* <u>MacOS and Linux</u>:
	** '''[[Data_Transfer/SSHFS\|sshfs]]''' is ~~the~~ ~~quickest~~ ~~way~~ for mounting under stable connections.		** '''[[Data_Transfer/SSHFS\|sshfs]]''' is quickly set up for mounting under stable connections.
	** '''[[Data_Transfer/Rclone\|Rclone]]''' for mount, copy and sync ~~(not~~ ~~usable~~ with 2FA).		** '''[[Data_Transfer/Rclone\|Rclone]]''' for mount, copy and sync. An [[Registration/SSH \| ssh key]] is needed when used with 2FA.
	* <u>SDS@hd</u>: See the [[SDS@hd/Access\|SDS@hd Access]] page.		* <u>SDS@hd</u>: See the [[SDS@hd/Access\|SDS@hd Access]] page.

H Schumacher: new recommendations

2025-03-25T09:25:43Z

new recommendations

← Older revision		Revision as of 11:25, 25 March 2025
Line 55:		Line 55:

	== Recommended Setup ==		== Recommended Setup ==
			[[File:Bwhpc diagram simplenobox.jpg\|thumb\|Main routes for mounting\|x150px]]
			When you are working with a development environment that allows remote connections, this is the first choice. Otherwise, the main tools/protocols for transferring data are as follows:

		⚫	* <u>Windows</u>: '''[[Data_Transfer/Graphical_Clients#MobaXterm\|MobaXterm]]''' is a graphical user interface that allows logging in to the cluster with ssh as well as transferring data via a file browser or using command line tools for the transfer.
	The main tool/protocol for transferring data to a bwHPC cluster from your local machine is as follows:
			* <u>MacOS and Linux</u>:

			** '''[[Data_Transfer/SSHFS\|sshfs]]''' is the quickest way for mounting under stable connections.
⚫	* '''[[Data_Transfer/Graphical_Clients#MobaXterm\|MobaXterm]]''' ~~for Windows. It~~ is a graphical user interface that allows logging in to the cluster with ssh as well as transferring data via a file browser.
			** '''[[Data_Transfer/Rclone\|Rclone]]''' for mount, copy and sync (not usable with 2FA).
	* '''[[Data_Transfer/SSHFS\|sshfs]]''' for MacOS and Linux. It is the quickest solution for mounting a folder. If you want to use copy and sync as well, it is more convenient to use '''[[Data_Transfer/Rclone\|Rclone]]''' instead. Rclone provides copy, sync and mount functionality for various types of infrastructure.
		⚫	* <u>SDS@hd</u>: See the [[SDS@hd/Access\|SDS@hd Access]] page.


	<p style="text-align: center; margin-top: 10px">[[File:Bwhpc diagram simplenobox.jpg\|x150px]]</p>
	<p style="text-align: center; font-size: small; margin-top: 10px">Figure 2: bwHPC main transfer routes</p>
⚫	~~For~~ SDS@hd ~~you can find the main access options at~~ the [[SDS@hd/Access\|SDS@hd Access]] page.

	== Best Practices ==		== Best Practices ==

H Schumacher: Added links to protocols in table

2025-03-12T15:37:58Z

Added links to protocols in table

← Older revision		Revision as of 17:37, 12 March 2025
Line 39:		Line 39:
	\|-		\|-
	\| ssh		\| ssh
			\| [[Data_Transfer/SCP \| scp]], [[Data_Transfer/SFTP \| sftp]], [[Data_Transfer/Rsync \| rsync]]
	\| scp, sftp, rsync
	\|-		\|-
	\| http(s)		\| http(s)
	\| WebDAV		\| [[Data_Transfer/WebDAV \| WebDAV]]
	\|-		\|-
			\| [[SDS@hd/Access/SMB \| smb]]
	\| smb
	\| -		\| -
	\|-		\|-
			\| [[SDS@hd/Access/NFS \| NFS]]
	\| NFS
	\| -		\| -
	\|}		\|}

K Siegmund: /* Ways of Transfer: Copy, Sync, Mount */

2025-01-28T09:10:51Z

Ways of Transfer: Copy, Sync, Mount

← Older revision		Revision as of 11:10, 28 January 2025
Line 25:		Line 25:
	* '''Copy:''' A simple copy command is the most basic way to transfer data. This is most efficient for very big data files that shall be retrieved from or moved to a remote location. And it can be most convenient, if you prefer moving your files via commandline instead of using a file browser. <br>Examples: [[Data_Transfer/SCP\|scp]], [[Data_Transfer/SFTP\|sftp]]		* '''Copy:''' A simple copy command is the most basic way to transfer data. This is most efficient for very big data files that shall be retrieved from or moved to a remote location. And it can be most convenient, if you prefer moving your files via commandline instead of using a file browser. <br>Examples: [[Data_Transfer/SCP\|scp]], [[Data_Transfer/SFTP\|sftp]]
	* '''Sync:''' If the data is intended to be kept on both systems and undergoes change on only one of the systems, it makes sense to use a synchronization command instead. This way, only the changed files in one location are updated in the other location. Good use cases are backups or data transfers that go mostly in one direction like moving data from a sequencer to a storage space. A disadvantage is that the data needs storage space on both systems. <br>Example: [[Data_Transfer/Rsync\|rsync]]		* '''Sync:''' If the data is intended to be kept on both systems and undergoes change on only one of the systems, it makes sense to use a synchronization command instead. This way, only the changed files in one location are updated in the other location. Good use cases are backups or data transfers that go mostly in one direction like moving data from a sequencer to a storage space. A disadvantage is that the data needs storage space on both systems. <br>Example: [[Data_Transfer/Rsync\|rsync]]
	* '''Mount:''' If the data undergoes change on both systems or is too big to store locally, then mounting is the most convenient solution. This allows you to see and work with the data as if it were stored locally on your computer while it is still placed on the remote system. All changes that you implement happen directly on the original data so that you don't need to copy or synchronize anything. Additionally, you'll see all changes that another party does to the data with just a very short delay. Disadvantages are that you need defined edit sessions starting with a mount and ending with a clean unmount of your files and with a stable network connection during the session. Also, file operations on the remote system become much ~~smaller~~ via a remotely sshfs-mounted system. <br>Example: [[Data_Transfer/SSHFS\|sshfs]]		* '''Mount:''' If the data undergoes change on both systems or is too big to store locally, then mounting is the most convenient solution. This allows you to see and work with the data as if it were stored locally on your computer while it is still placed on the remote system. All changes that you implement happen directly on the original data so that you don't need to copy or synchronize anything. Additionally, you'll see all changes that another party does to the data with just a very short delay. Disadvantages are that you need defined edit sessions starting with a mount and ending with a clean unmount of your files and with a stable network connection during the session. Also, file operations on the remote system become much slower via a remotely sshfs-mounted system. <br>Example: [[Data_Transfer/SSHFS\|sshfs]]
	<p style="text-align: left;">[[File:CopySyncMount.png\|x250px]]</p>		<p style="text-align: left;">[[File:CopySyncMount.png\|x250px]]</p>
	<p style="text-align: left; font-size: small; margin-top: 10px; margin-left: 255px;">Figure 1: Top level transfer routes</p>		<p style="text-align: left; font-size: small; margin-top: 10px; margin-left: 255px;">Figure 1: Top level transfer routes</p>

K Siegmund: /* Ways of Transfer: Copy, Sync, Mount */

2025-01-27T16:51:48Z

Ways of Transfer: Copy, Sync, Mount

← Older revision		Revision as of 18:51, 27 January 2025
Line 23:		Line 23:
	The top level ways of transfer are:		The top level ways of transfer are:

	* '''Copy:''' A simple copy command is the most basic way to transfer data. This is most efficient for very big data files that shall be retrieved from or moved to a remote location. And it can be most convenient, if you prefer moving your files via commandline instead of using a file browser. Examples: [[Data_Transfer/SCP\|scp]], [[Data_Transfer/SFTP\|sftp]]		* '''Copy:''' A simple copy command is the most basic way to transfer data. This is most efficient for very big data files that shall be retrieved from or moved to a remote location. And it can be most convenient, if you prefer moving your files via commandline instead of using a file browser. <br>Examples: [[Data_Transfer/SCP\|scp]], [[Data_Transfer/SFTP\|sftp]]
	* '''Sync:''' If the data is intended to be kept on both systems and undergoes change on only one of the systems, it makes sense to use a synchronization command instead. This way, only the changed files in one location are updated in the other location. Good use cases are backups or data transfers that go mostly in one direction like moving data from a sequencer to a storage space. A disadvantage is that the data needs storage space on both systems. Example: [[Data_Transfer/Rsync\|rsync]]		* '''Sync:''' If the data is intended to be kept on both systems and undergoes change on only one of the systems, it makes sense to use a synchronization command instead. This way, only the changed files in one location are updated in the other location. Good use cases are backups or data transfers that go mostly in one direction like moving data from a sequencer to a storage space. A disadvantage is that the data needs storage space on both systems. <br>Example: [[Data_Transfer/Rsync\|rsync]]
	* '''Mount:''' If the data undergoes change on both systems or is too big to store locally, then mounting is the most convenient solution. This allows you to see and work with the data as if it were stored locally on your computer while it is still placed on the remote system. All changes that you implement happen directly on the original data so that you don't need to copy or synchronize anything. Additionally, you'll see all changes that another party does to the data with just a very short delay. Disadvantages are that you need defined edit sessions starting with a mount and ending with a clean unmount of your files and with a stable network connection during the session. Also, file operations on the remote system become much smaller via a remotely sshfs-mounted system. Example: [[Data_Transfer/SSHFS\|sshfs]]		* '''Mount:''' If the data undergoes change on both systems or is too big to store locally, then mounting is the most convenient solution. This allows you to see and work with the data as if it were stored locally on your computer while it is still placed on the remote system. All changes that you implement happen directly on the original data so that you don't need to copy or synchronize anything. Additionally, you'll see all changes that another party does to the data with just a very short delay. Disadvantages are that you need defined edit sessions starting with a mount and ending with a clean unmount of your files and with a stable network connection during the session. Also, file operations on the remote system become much smaller via a remotely sshfs-mounted system. <br>Example: [[Data_Transfer/SSHFS\|sshfs]]
	<p style="text-align: left;">[[File:CopySyncMount.png\|x250px]]</p>		<p style="text-align: left;">[[File:CopySyncMount.png\|x250px]]</p>
	<p style="text-align: left; font-size: small; margin-top: 10px; margin-left: 255px;">Figure 1: Top level transfer routes</p>		<p style="text-align: left; font-size: small; margin-top: 10px; margin-left: 255px;">Figure 1: Top level transfer routes</p>

K Siegmund: /* Ways of Transfer: Copy, Sync, Mount */

2025-01-27T16:51:19Z

Ways of Transfer: Copy, Sync, Mount

← Older revision		Revision as of 18:51, 27 January 2025
Line 23:		Line 23:
	The top level ways of transfer are:		The top level ways of transfer are:

	* '''Copy:''' A simple copy command is the most basic way to transfer data. This is most efficient for very big data files that shall be retrieved from or moved to a remote location. And it can be most convenient, if you prefer moving your files via commandline instead of using a file browser. Examples: [[Data_Transfer/SCP\|scp]] [[Data_Transfer/SFTP\|sftp]]		* '''Copy:''' A simple copy command is the most basic way to transfer data. This is most efficient for very big data files that shall be retrieved from or moved to a remote location. And it can be most convenient, if you prefer moving your files via commandline instead of using a file browser. Examples: [[Data_Transfer/SCP\|scp]], [[Data_Transfer/SFTP\|sftp]]
	* '''Sync:''' If the data is intended to be kept on both systems and undergoes change on only one of the systems, it makes sense to use a synchronization command instead. This way, only the changed files in one location are updated in the other location. Good use cases are backups or data transfers that go mostly in one direction like moving data from a sequencer to a storage space. A disadvantage is that the data needs storage space on both systems. Example: [[Data_Transfer/Rsync\|rsync]]		* '''Sync:''' If the data is intended to be kept on both systems and undergoes change on only one of the systems, it makes sense to use a synchronization command instead. This way, only the changed files in one location are updated in the other location. Good use cases are backups or data transfers that go mostly in one direction like moving data from a sequencer to a storage space. A disadvantage is that the data needs storage space on both systems. Example: [[Data_Transfer/Rsync\|rsync]]
	* '''Mount:''' If the data undergoes change on both systems or is too big to store locally, then mounting is the most convenient solution. This allows you to see and work with the data as if it were stored locally on your computer while it is still placed on the remote system. All changes that you implement happen directly on the original data so that you don't need to copy or synchronize anything. Additionally, you'll see all changes that another party does to the data with just a very short delay. Disadvantages are that you need defined edit sessions starting with a mount and ending with a clean unmount of your files and with a stable network connection during the session. Also, file operations on the remote system become much smaller via a remotely sshfs-mounted system. Example: [[Data_Transfer/SSHFS\|sshfs]]		* '''Mount:''' If the data undergoes change on both systems or is too big to store locally, then mounting is the most convenient solution. This allows you to see and work with the data as if it were stored locally on your computer while it is still placed on the remote system. All changes that you implement happen directly on the original data so that you don't need to copy or synchronize anything. Additionally, you'll see all changes that another party does to the data with just a very short delay. Disadvantages are that you need defined edit sessions starting with a mount and ending with a clean unmount of your files and with a stable network connection during the session. Also, file operations on the remote system become much smaller via a remotely sshfs-mounted system. Example: [[Data_Transfer/SSHFS\|sshfs]]

K Siegmund at 16:51, 27 January 2025

2025-01-27T16:51:00Z

← Older revision		Revision as of 18:51, 27 January 2025
Line 23:		Line 23:
	The top level ways of transfer are:		The top level ways of transfer are:

	* '''Copy:''' A simple copy command is the most basic way to transfer data. This is most efficient for very big data files that shall be retrieved from or moved to a remote location. And it can be most convenient, if you prefer moving your files via commandline instead of using a file browser.		* '''Copy:''' A simple copy command is the most basic way to transfer data. This is most efficient for very big data files that shall be retrieved from or moved to a remote location. And it can be most convenient, if you prefer moving your files via commandline instead of using a file browser. Examples: [[Data_Transfer/SCP\|scp]] [[Data_Transfer/SFTP\|sftp]]
	* '''Sync:''' If the data is intended to be kept on both systems and undergoes change on only one of the systems, it makes sense to use a synchronization command instead. This way, only the changed files in one location are updated in the other location. Good use cases are backups or data transfers that go mostly in one direction like moving data from a sequencer to a storage space. A disadvantage is that the data needs storage space on both systems.		* '''Sync:''' If the data is intended to be kept on both systems and undergoes change on only one of the systems, it makes sense to use a synchronization command instead. This way, only the changed files in one location are updated in the other location. Good use cases are backups or data transfers that go mostly in one direction like moving data from a sequencer to a storage space. A disadvantage is that the data needs storage space on both systems. Example: [[Data_Transfer/Rsync\|rsync]]
	* '''Mount:''' If the data undergoes change on both systems or is too big to store locally, then mounting is the most convenient solution. This allows you to see and work with the data as if it were stored locally on your computer while it is still placed on the remote system. All changes that you implement happen directly on the original data so that you don't need to copy or synchronize anything. Additionally, you'll see all changes that another party does to the data with just a very short delay.		* '''Mount:''' If the data undergoes change on both systems or is too big to store locally, then mounting is the most convenient solution. This allows you to see and work with the data as if it were stored locally on your computer while it is still placed on the remote system. All changes that you implement happen directly on the original data so that you don't need to copy or synchronize anything. Additionally, you'll see all changes that another party does to the data with just a very short delay. Disadvantages are that you need defined edit sessions starting with a mount and ending with a clean unmount of your files and with a stable network connection during the session. Also, file operations on the remote system become much smaller via a remotely sshfs-mounted system. Example: [[Data_Transfer/SSHFS\|sshfs]]
	<p style="text-align: left;">[[File:CopySyncMount.png\|x250px]]</p>		<p style="text-align: left;">[[File:CopySyncMount.png\|x250px]]</p>
	<p style="text-align: left; font-size: small; margin-top: 10px; margin-left: 255px;">Figure 1: Top level transfer routes</p>		<p style="text-align: left; font-size: small; margin-top: 10px; margin-left: 255px;">Figure 1: Top level transfer routes</p>

H Schumacher: removed in progress message

2024-12-05T09:50:25Z

removed in progress message

← Older revision		Revision as of 11:50, 5 December 2024
Line 1:		Line 1:
	{\| style=" background:#FEF4AB; width:100%;"
	\| style="padding:6px; font-size:100%;text-align:left" \| This page is work in progress. To discover more relevant subpages, please come back at a later timepoint.
	\|-
	\|}
	== Overview ==		== Overview ==

H Schumacher: fixed link

2024-11-29T13:57:05Z

fixed link

← Older revision		Revision as of 15:57, 29 November 2024
Line 56:		Line 56:
	For every data transfer a network protocol to use for the communication between the systems must be chosen. The basic network protocols and the network protocols that build directly upon those are shown in the table on the right. These protocols can either be used rather directly or through tools that provide the protocol together with additional features. A tool can either mean a command line tool or a tool with a graphical user interface.		For every data transfer a network protocol to use for the communication between the systems must be chosen. The basic network protocols and the network protocols that build directly upon those are shown in the table on the right. These protocols can either be used rather directly or through tools that provide the protocol together with additional features. A tool can either mean a command line tool or a tool with a graphical user interface.

	A comprehensive overview of all transfer options (network protocols and tools) can be found on the page [[Data_Transfer/~~All_Transfer_Routes~~\|all transfer routes]].		A comprehensive overview of all transfer options (network protocols and tools) can be found on the page [[Data_Transfer/All_Data_Transfer_Routes\|all data transfer routes]].

	== Recommended Setup ==		== Recommended Setup ==

H Schumacher: fixed link

2024-11-29T13:55:24Z

fixed link

← Older revision		Revision as of 15:55, 29 November 2024
Line 62:		Line 62:
	The main tool/protocol for transferring data to a bwHPC cluster from your local machine is as follows:		The main tool/protocol for transferring data to a bwHPC cluster from your local machine is as follows:

	* '''[[Data_Transfer/MobaXterm\|MobaXterm]]''' for Windows. It is a graphical user interface that allows logging in to the cluster with ssh as well as transferring data via a file browser.		* '''[[Data_Transfer/Graphical_Clients#MobaXterm\|MobaXterm]]''' for Windows. It is a graphical user interface that allows logging in to the cluster with ssh as well as transferring data via a file browser.
	* '''[[Data_Transfer/SSHFS\|sshfs]]''' for MacOS and Linux. It is the quickest solution for mounting a folder. If you want to use copy and sync as well, it is more convenient to use '''[[Data_Transfer/Rclone\|Rclone]]''' instead. Rclone provides copy, sync and mount functionality for various types of infrastructure.		* '''[[Data_Transfer/SSHFS\|sshfs]]''' for MacOS and Linux. It is the quickest solution for mounting a folder. If you want to use copy and sync as well, it is more convenient to use '''[[Data_Transfer/Rclone\|Rclone]]''' instead. Rclone provides copy, sync and mount functionality for various types of infrastructure.

← Older revision		Revision as of 16:42, 13 May 2026
Line 60:		Line 60:
	* <u>Windows</u>: '''[[Data_Transfer/Graphical_Clients#MobaXterm\|MobaXterm]]''' is a graphical user interface that allows logging in to the cluster with ssh as well as transferring data via a file browser or using command line tools for the transfer.		* <u>Windows</u>: '''[[Data_Transfer/Graphical_Clients#MobaXterm\|MobaXterm]]''' is a graphical user interface that allows logging in to the cluster with ssh as well as transferring data via a file browser or using command line tools for the transfer.
	* <u>MacOS and Linux</u>:		* <u>MacOS and Linux</u>:
	** '''[[Data_Transfer/SSHFS\|sshfs]]''' is ~~the~~ ~~quickest~~ ~~way~~ for mounting under stable connections.		** '''[[Data_Transfer/SSHFS\|sshfs]]''' is quickly set up for mounting under stable connections.
	** '''[[Data_Transfer/Rclone\|Rclone]]''' for mount, copy and sync ~~(not~~ ~~usable~~ with 2FA).		** '''[[Data_Transfer/Rclone\|Rclone]]''' for mount, copy and sync. An [[Registration/SSH \| ssh key]] is needed when used with 2FA.
	* <u>SDS@hd</u>: See the [[SDS@hd/Access\|SDS@hd Access]] page.		* <u>SDS@hd</u>: See the [[SDS@hd/Access\|SDS@hd Access]] page.

← Older revision		Revision as of 11:10, 28 January 2025
Line 25:		Line 25:
	* '''Copy:''' A simple copy command is the most basic way to transfer data. This is most efficient for very big data files that shall be retrieved from or moved to a remote location. And it can be most convenient, if you prefer moving your files via commandline instead of using a file browser. <br>Examples: [[Data_Transfer/SCP\|scp]], [[Data_Transfer/SFTP\|sftp]]		* '''Copy:''' A simple copy command is the most basic way to transfer data. This is most efficient for very big data files that shall be retrieved from or moved to a remote location. And it can be most convenient, if you prefer moving your files via commandline instead of using a file browser. <br>Examples: [[Data_Transfer/SCP\|scp]], [[Data_Transfer/SFTP\|sftp]]
	* '''Sync:''' If the data is intended to be kept on both systems and undergoes change on only one of the systems, it makes sense to use a synchronization command instead. This way, only the changed files in one location are updated in the other location. Good use cases are backups or data transfers that go mostly in one direction like moving data from a sequencer to a storage space. A disadvantage is that the data needs storage space on both systems. <br>Example: [[Data_Transfer/Rsync\|rsync]]		* '''Sync:''' If the data is intended to be kept on both systems and undergoes change on only one of the systems, it makes sense to use a synchronization command instead. This way, only the changed files in one location are updated in the other location. Good use cases are backups or data transfers that go mostly in one direction like moving data from a sequencer to a storage space. A disadvantage is that the data needs storage space on both systems. <br>Example: [[Data_Transfer/Rsync\|rsync]]
	* '''Mount:''' If the data undergoes change on both systems or is too big to store locally, then mounting is the most convenient solution. This allows you to see and work with the data as if it were stored locally on your computer while it is still placed on the remote system. All changes that you implement happen directly on the original data so that you don't need to copy or synchronize anything. Additionally, you'll see all changes that another party does to the data with just a very short delay. Disadvantages are that you need defined edit sessions starting with a mount and ending with a clean unmount of your files and with a stable network connection during the session. Also, file operations on the remote system become much ~~smaller~~ via a remotely sshfs-mounted system. <br>Example: [[Data_Transfer/SSHFS\|sshfs]]		* '''Mount:''' If the data undergoes change on both systems or is too big to store locally, then mounting is the most convenient solution. This allows you to see and work with the data as if it were stored locally on your computer while it is still placed on the remote system. All changes that you implement happen directly on the original data so that you don't need to copy or synchronize anything. Additionally, you'll see all changes that another party does to the data with just a very short delay. Disadvantages are that you need defined edit sessions starting with a mount and ending with a clean unmount of your files and with a stable network connection during the session. Also, file operations on the remote system become much slower via a remotely sshfs-mounted system. <br>Example: [[Data_Transfer/SSHFS\|sshfs]]
	<p style="text-align: left;">[[File:CopySyncMount.png\|x250px]]</p>		<p style="text-align: left;">[[File:CopySyncMount.png\|x250px]]</p>
	<p style="text-align: left; font-size: small; margin-top: 10px; margin-left: 255px;">Figure 1: Top level transfer routes</p>		<p style="text-align: left; font-size: small; margin-top: 10px; margin-left: 255px;">Figure 1: Top level transfer routes</p>