Data Transfer/Rclone

From bwHPC Wiki
< Data Transfer
Revision as of 18:36, 21 November 2024 by H Schumacher (talk | contribs) (Created page)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Rclone is a command line tool to manage files on remote systems (e.g. cloud storage systems). Rclone either synchronizes in one direction only or its mounting functionality is used with rclone mount. Data can be piped between two completely remote locations, sometimes without local download. One advantage is that the transfer is multithreaded and it operates on a file level basis.


Installation

Rclone is a Go program and comes as a single binary file.

  1. Download the relevant binary.
  2. Extract the rclone executable, rclone.exe on Windows, from the archive.
  3. You can use the executables without further installation. For easy use, it is recommended to add the binary to your PATH environment variable. Information on how to do this can be found below.

Detailed information regarding different operating systems can be found here:


Usage Rclone

A detailed explanation on how to use Rclone can be found here <link>.
Before you can get started, you need to set up a client. Therefore, run rclone config to set up a new connection.

An example is provided for setting up a WebDAV connection with a remote system.

The syntax to use Rclone is like this:

rclone [options] subcommand <parameters> <parameters...>

List all directories/containers/buckets in the folder XX.

rclone lsd <client-name>:XX

Copies /local/path to the remote path

rclone copy </local/path> <client-name>:<remote/path>

Copies fom remote path to /local/path

rclone copy <client-name>:<remote/path> </local/path>

Moves the contents of the source directory to the destination directory.

rclone move <client-name>:<source/path> <client-name>:<destination/path>

More subcommands can be found here.


Usage Rclone Mount

Detailed information on how to use rclone mount can be found here.

Linux/Mac

You can run mount in either foreground or background (aka daemon) mode. Mount runs in foreground mode by default. Use the --daemon flag to force background mode.

Create an empty directory on your local machine and then execute

# to mount the root folder:
rclone mount --vfs-cache-mode full <client-name>: /path/to/empty/folder 
# to mount a subfolder:
rclone mount --vfs-cache-mode full <client-name>:folderX/folderY /path/to/empty/folder 

Windows

To run rclone mount on Windows, you will need to download and install WinFsp. To mount on drive letter X or a nonexistent subdirectory, use:

rclone mount <client-name>:path/to/files X:
rclone mount <client-name>:path/to/files C:\path\parent\mount

In contrast to Linux/Mac, there is no background mode.

Best Practices

see wiki