Data Transfer/Rclone: Difference between revisions

From bwHPC Wiki
Jump to navigation Jump to search
m (Rename chapter)
(Added more instructions regarding the setup of a remote)
 
(One intermediate revision by the same user not shown)
Line 1: Line 1:


Rclone is a command line tool to manage files on remote systems (e.g. cloud storage systems). Rclone either synchronizes in one direction only or its mounting functionality is used with <code>rclone mount</code>. Data can be piped between two completely remote locations, sometimes without local download. One advantage is that the transfer is multithreaded and it operates on a file level basis.
[https://rclone.org/docs/ Rclone] is a command line tool to manage files on remote systems (e.g. cloud storage systems). Rclone either synchronizes in one direction only or its mounting functionality is used with <code>rclone mount</code>. Data can be piped between two completely remote locations, sometimes without local download. One advantage is that the transfer is multithreaded and it operates on a file level basis.
'''Caution:''' You can't use Rclone with 2FA.


== Installation ==
== Installation ==
Line 18: Line 19:
== Usage Rclone ==
== Usage Rclone ==


To use Rclone you have to define a config file. Afterwards you can connect by using the name of your configured connections.
A detailed explanation on how to use Rclone can be found here <link>.<br />
Before you can get started, you need to set up a client. Therefore, run <code>rclone config</code> to set up a new connection.


=== Configure Remote ===
An example is provided for [[Data_Transfer/WebDAV|setting up a WebDAV connection with a remote system]].

Before you can start using Rclone, you need to set up a remote. This means to configure a specific connection by providing authentication information, the network protocol that you want to use and a name for this configuration so that you can use it later on.

To configure a remote for a specific service, you need the following information:

* <code>&lt;remotehost&gt;</code>
* <code>&lt;username&gt;</code>
* <code>&lt;servicePassword&gt;</code>

Furthermore, you have to decide on:

* <code>network protocol</code> (for example webDAV, smb, sftp)
* <code>remote-name</code> (for example you can use the name of the service you want to connect to)

You have three different options to set up a new remote which are explained by the following sections.

<span style="font-size: 1.1em; text-decoration: underline;">'''Interactive Setup''' </span>

Execute:
<pre>rclone config</pre>
This will guide you through an interactive setup process. You can find detailed instructions at the website:

* [https://rclone.org/webdav/ Connect via sftp]
* [https://rclone.org/smb/ Connect via smb]
* [https://rclone.org/sftp/ Connect via webdav]

<span style="font-size: 1.1em; text-decoration: underline;">'''Oneliner''' </span>

Define all parameters in one command. For example:
<syntaxhighlight lang="bash">rclone config create test sftp host=<remotehost> user=<username> pass=<password> --obscure</syntaxhighlight>

<span style="font-size: 1.1em; text-decoration: underline;">'''Adjust Config File''' </span>

To see, where the file is run: <code>rclone config file</code>.

You can use the following snippet as template for your connections.
<syntaxhighlight>[<remote-name>]
type = webdav
url = <hostURL>
vendor = other
user = <userID>

[<remote-name>]
type = sftp
host = <hostname>
user = <userID>
key_use_agent = false
</syntaxhighlight>
To add the password, please use
<syntaxhighlight>rclone config update <remote-name> pass=<password> --obscure</syntaxhighlight>

=== Use Remote ===


The syntax to use Rclone is like this:
The syntax to use Rclone is like this:
Line 29: Line 81:
List all directories/containers/buckets in the folder XX.
List all directories/containers/buckets in the folder XX.


<pre>rclone lsd <client-name>:XX
<pre>rclone lsd <remote-name>:XX
</pre>
</pre>
Copies /local/path to the remote path
Copies /local/path to the remote path


<pre>rclone copy &lt;/local/path&gt; <client-name>:&lt;remote/path&gt;
<pre>rclone copy &lt;/local/path&gt; <remote-name>:&lt;remote/path&gt;
</pre>
</pre>
Copies fom remote path to /local/path
Copies fom remote path to /local/path


<pre>rclone copy <client-name>:&lt;remote/path&gt; &lt;/local/path&gt;
<pre>rclone copy <remote-name>:&lt;remote/path&gt; &lt;/local/path&gt;
</pre>
</pre>
Moves the contents of the source directory to the destination directory.
Moves the contents of the source directory to the destination directory.


<pre>rclone move <client-name>:&lt;source/path&gt; <client-name>:&lt;destination/path&gt;
<pre>rclone move <remote-name>:&lt;source/path&gt; <remote-name>:&lt;destination/path&gt;
</pre>
</pre>
More subcommands can be found [https://rclone.org/docs/#subcommands here].
More subcommands can be found [https://rclone.org/docs/#subcommands here].
Line 47: Line 99:
== Usage Rclone Mount ==
== Usage Rclone Mount ==


Before you can follow the instructions in this chapter, you need to have set up a [[Data_Transfer/Rclone#Usage_Rclone | remote]].
Detailed information on how to use rclone mount can be found [https://rclone.org/commands/rclone_mount/ here].
Detailed information on how to use rclone mount can be found [https://rclone.org/commands/rclone_mount/ here].


Line 53: Line 106:
To run rclone mount on Windows, you will need to [https://winfsp.dev/rel/ download and install WinFsp]. To mount on drive letter X or a nonexistent subdirectory, use:
To run rclone mount on Windows, you will need to [https://winfsp.dev/rel/ download and install WinFsp]. To mount on drive letter X or a nonexistent subdirectory, use:


<pre>rclone mount &lt;client-name&gt;:path/to/files X:
<pre>rclone mount &lt;remote-name&gt;:path/to/files X:
rclone mount &lt;client-name&gt;:path/to/files C:\path\parent\mount</pre>
rclone mount &lt;remote-name&gt;:path/to/files C:\path\parent\mount</pre>
In contrast to Linux/Mac, there is no background mode.
In contrast to Linux/Mac, there is no background mode.


Line 64: Line 117:


<pre># to mount the root folder:
<pre># to mount the root folder:
rclone mount --vfs-cache-mode full &lt;client-name&gt;: /path/to/empty/folder
rclone mount --vfs-cache-mode full &lt;remote-name&gt;: /path/to/empty/folder
# to mount a subfolder:
# to mount a subfolder:
rclone mount --vfs-cache-mode full &lt;client-name&gt;:folderX/folderY /path/to/empty/folder </pre>
rclone mount --vfs-cache-mode full &lt;remote-name&gt;:folderX/folderY /path/to/empty/folder </pre>


== Best practices ==
== Best Practices ==


Rclone has a lot of useful options.
Rclone has a lot of useful options.

Latest revision as of 18:06, 19 February 2025

Rclone is a command line tool to manage files on remote systems (e.g. cloud storage systems). Rclone either synchronizes in one direction only or its mounting functionality is used with rclone mount. Data can be piped between two completely remote locations, sometimes without local download. One advantage is that the transfer is multithreaded and it operates on a file level basis. Caution: You can't use Rclone with 2FA.

Installation

Rclone is a Go program and comes as a single binary file.

  1. Download the relevant binary.
  2. Extract the rclone executable, rclone.exe on Windows, from the archive.
  3. You can use the executables without further installation. For easy use, it is recommended to add the binary to your PATH environment variable. Information on how to do this can be found below.

Detailed information regarding different operating systems can be found here:

Usage Rclone

To use Rclone you have to define a config file. Afterwards you can connect by using the name of your configured connections.

Configure Remote

Before you can start using Rclone, you need to set up a remote. This means to configure a specific connection by providing authentication information, the network protocol that you want to use and a name for this configuration so that you can use it later on.

To configure a remote for a specific service, you need the following information:

  • <remotehost>
  • <username>
  • <servicePassword>

Furthermore, you have to decide on:

  • network protocol (for example webDAV, smb, sftp)
  • remote-name (for example you can use the name of the service you want to connect to)

You have three different options to set up a new remote which are explained by the following sections.

Interactive Setup

Execute: 

rclone config

This will guide you through an interactive setup process. You can find detailed instructions at the website:

Oneliner

Define all parameters in one command. For example: 

rclone config create test sftp host=<remotehost> user=<username> pass=<password> --obscure

Adjust Config File

To see, where the file is run: rclone config file.

You can use the following snippet as template for your connections. 

[<remote-name>]
type = webdav
url = <hostURL>
vendor = other
user = <userID>

[<remote-name>]
type = sftp
host = <hostname>
user = <userID>
key_use_agent = false

To add the password, please use 

rclone config update <remote-name> pass=<password> --obscure

Use Remote

The syntax to use Rclone is like this: 

rclone [options] subcommand <parameters> <parameters...>

List all directories/containers/buckets in the folder XX. 

rclone lsd <remote-name>:XX

Copies /local/path to the remote path 

rclone copy </local/path> <remote-name>:<remote/path>

Copies fom remote path to /local/path 

rclone copy <remote-name>:<remote/path> </local/path>

Moves the contents of the source directory to the destination directory. 

rclone move <remote-name>:<source/path> <remote-name>:<destination/path>

More subcommands can be found here.

Usage Rclone Mount

Before you can follow the instructions in this chapter, you need to have set up a remote. Detailed information on how to use rclone mount can be found here.

Windows

To run rclone mount on Windows, you will need to download and install WinFsp. To mount on drive letter X or a nonexistent subdirectory, use: 

rclone mount <remote-name>:path/to/files X:
rclone mount <remote-name>:path/to/files C:\path\parent\mount

In contrast to Linux/Mac, there is no background mode.

MacOS & Linux

You can run mount in either foreground or background (aka daemon) mode. Mount runs in foreground mode by default. Use the --daemon flag to force background mode.

Create an empty directory on your local machine and then execute 

# to mount the root folder:
rclone mount --vfs-cache-mode full <remote-name>: /path/to/empty/folder 
# to mount a subfolder:
rclone mount --vfs-cache-mode full <remote-name>:folderX/folderY /path/to/empty/folder

Best Practices

Rclone has a lot of useful options.

Performance

To be able to utilize a larger bandwidth, it is helpful to add the following options for increased performance: 

--transfers <int>

Number of file transfers to run in parallel (default: 4). Depending on the local Network, read and write speeds on the file system, and current load, different values might be best. For large transfers, it is advised to test local performance with different values beforehand.

  • In our tests, we observed the best results between 8 and 32.
  • For regular use cases, we recommend 16 as the default.
  • Values above 64 are not recommended and degrade performance.
--multi-thread-streams <int>

Number of streams to use for multithreaded downloads (default: 4). Only important on very large files. This will cause multithreaded up/download on chunk-sized bits of the file.

The optimal value is highly specific to the local network and used Hardware. For regular use cases, we recommend 4 as the default.

Debugging and Statistics

To get updates on current progress, use: 

--stats

Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable) (default 1m0s).

To get debug information, use: 

--log-level=DEBUG 
--stats-log-level=DEBUG
Retrieved from "https://wiki.bwhpc.de/wiki/index.php?title=Data_Transfer/Rclone&oldid=14019"