Data Transfer/Rclone: Difference between revisions
H Schumacher (talk | contribs) (Created page) |
H Schumacher (talk | contribs) m (fixed a chapter title) |
||
(5 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
<pre class="{contents}"></pre> |
|||
Rclone is a command line tool to manage files on remote systems (e.g. cloud storage systems). Rclone either synchronizes in one direction only or its mounting functionality is used with <code>rclone mount</code>. Data can be piped between two completely remote locations, sometimes without local download. One advantage is that the transfer is multithreaded and it operates on a file level basis. |
Rclone is a command line tool to manage files on remote systems (e.g. cloud storage systems). Rclone either synchronizes in one direction only or its mounting functionality is used with <code>rclone mount</code>. Data can be piped between two completely remote locations, sometimes without local download. One advantage is that the transfer is multithreaded and it operates on a file level basis. |
||
== Installation == |
== Installation == |
||
Line 14: | Line 12: | ||
Detailed information regarding different operating systems can be found here: |
Detailed information regarding different operating systems can be found here: |
||
⚫ | |||
⚫ | |||
* Installation on [https://rclone.org/install/#script-installation Linux] |
* Installation on [https://rclone.org/install/#script-installation Linux] |
||
⚫ | |||
⚫ | |||
== Usage Rclone == |
== Usage Rclone == |
||
Line 48: | Line 44: | ||
</pre> |
</pre> |
||
More subcommands can be found [https://rclone.org/docs/#subcommands here]. |
More subcommands can be found [https://rclone.org/docs/#subcommands here]. |
||
== Usage Rclone Mount == |
== Usage Rclone Mount == |
||
Line 55: | Line 49: | ||
Detailed information on how to use rclone mount can be found [https://rclone.org/commands/rclone_mount/ here]. |
Detailed information on how to use rclone mount can be found [https://rclone.org/commands/rclone_mount/ here]. |
||
=== |
=== Windows === |
||
⚫ | |||
⚫ | |||
⚫ | |||
⚫ | |||
=== MacOS & Linux === |
|||
You can run mount in either foreground or background (aka daemon) mode. Mount runs in foreground mode by default. Use the <code>--daemon</code> flag to force background mode. |
You can run mount in either foreground or background (aka daemon) mode. Mount runs in foreground mode by default. Use the <code>--daemon</code> flag to force background mode. |
||
Line 65: | Line 67: | ||
# to mount a subfolder: |
# to mount a subfolder: |
||
rclone mount --vfs-cache-mode full <client-name>:folderX/folderY /path/to/empty/folder </pre> |
rclone mount --vfs-cache-mode full <client-name>:folderX/folderY /path/to/empty/folder </pre> |
||
=== Windows === |
|||
== Best Practices == |
|||
⚫ | |||
Rclone has a lot of useful options. |
|||
⚫ | |||
⚫ | |||
⚫ | |||
== |
=== Performance === |
||
To be able to utilize a larger bandwidth, it is helpful to add the following options for increased performance: |
|||
see wiki |
|||
<pre> |
|||
--transfers <int> |
|||
</pre> |
|||
Number of file transfers to run in parallel (default: 4). Depending on the local Network, read and write speeds on the file system, and current load, different values might be best. For large transfers, it is advised to test local performance with different values beforehand. |
|||
* In our tests, we observed the best results between 8 and 32. |
|||
* For regular use cases, we recommend 16 as the default. |
|||
* Values above 64 are not recommended and degrade performance. |
|||
<pre> |
|||
--multi-thread-streams <int> |
|||
</pre> |
|||
Number of streams to use for multithreaded downloads (default: 4). Only important on very large files. This will cause multithreaded up/download on chunk-sized bits of the file. |
|||
The optimal value is highly specific to the local network and used Hardware. For regular use cases, we recommend 4 as the default. |
|||
=== Debugging and Statistics === |
|||
To get updates on current progress, use: |
|||
<pre> |
|||
--stats |
|||
</pre> |
|||
Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable) (default 1m0s). |
|||
To get debug information, use: |
|||
<pre> |
|||
--log-level=DEBUG |
|||
--stats-log-level=DEBUG |
|||
</pre> |
Latest revision as of 00:08, 22 November 2024
Rclone is a command line tool to manage files on remote systems (e.g. cloud storage systems). Rclone either synchronizes in one direction only or its mounting functionality is used with rclone mount
. Data can be piped between two completely remote locations, sometimes without local download. One advantage is that the transfer is multithreaded and it operates on a file level basis.
Installation
Rclone is a Go program and comes as a single binary file.
- Download the relevant binary.
- Extract the
rclone
executable,rclone.exe
on Windows, from the archive. - You can use the executables without further installation. For easy use, it is recommended to add the binary to your PATH environment variable. Information on how to do this can be found below.
Detailed information regarding different operating systems can be found here:
Usage Rclone
A detailed explanation on how to use Rclone can be found here <link>.
Before you can get started, you need to set up a client. Therefore, run rclone config
to set up a new connection.
An example is provided for setting up a WebDAV connection with a remote system.
The syntax to use Rclone is like this:
rclone [options] subcommand <parameters> <parameters...>
List all directories/containers/buckets in the folder XX.
rclone lsd <client-name>:XX
Copies /local/path to the remote path
rclone copy </local/path> <client-name>:<remote/path>
Copies fom remote path to /local/path
rclone copy <client-name>:<remote/path> </local/path>
Moves the contents of the source directory to the destination directory.
rclone move <client-name>:<source/path> <client-name>:<destination/path>
More subcommands can be found here.
Usage Rclone Mount
Detailed information on how to use rclone mount can be found here.
Windows
To run rclone mount on Windows, you will need to download and install WinFsp. To mount on drive letter X or a nonexistent subdirectory, use:
rclone mount <client-name>:path/to/files X: rclone mount <client-name>:path/to/files C:\path\parent\mount
In contrast to Linux/Mac, there is no background mode.
MacOS & Linux
You can run mount in either foreground or background (aka daemon) mode. Mount runs in foreground mode by default. Use the --daemon
flag to force background mode.
Create an empty directory on your local machine and then execute
# to mount the root folder: rclone mount --vfs-cache-mode full <client-name>: /path/to/empty/folder # to mount a subfolder: rclone mount --vfs-cache-mode full <client-name>:folderX/folderY /path/to/empty/folder
Best Practices
Rclone has a lot of useful options.
Performance
To be able to utilize a larger bandwidth, it is helpful to add the following options for increased performance:
--transfers <int>
Number of file transfers to run in parallel (default: 4). Depending on the local Network, read and write speeds on the file system, and current load, different values might be best. For large transfers, it is advised to test local performance with different values beforehand.
- In our tests, we observed the best results between 8 and 32.
- For regular use cases, we recommend 16 as the default.
- Values above 64 are not recommended and degrade performance.
--multi-thread-streams <int>
Number of streams to use for multithreaded downloads (default: 4). Only important on very large files. This will cause multithreaded up/download on chunk-sized bits of the file.
The optimal value is highly specific to the local network and used Hardware. For regular use cases, we recommend 4 as the default.
Debugging and Statistics
To get updates on current progress, use:
--stats
Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable) (default 1m0s).
To get debug information, use:
--log-level=DEBUG --stats-log-level=DEBUG