Data Transfer/Rclone

Rclone is a command line tool to manage files on remote systems (e.g. high performance clusters, data and cloud storage systems, archiving services).
Rclone either synchronizes in one direction only or its mounting functionality is used with rclone mount.
Data can be piped between two completely remote locations. When moving data within the same system and the data transfer protocol allows for server side copy, no local download is necessary.
One advantage is that the transfer is multithreaded and it operates on a file level basis.
Caution: When using Rclone with 2FA it is necessary to connect via ssh key instead of a password.

Installation

Rclone is a Go program and comes as a single binary file.

1. Download the relevant binary.
2. Extract the rclone executable (rclone.exe).
3. You can use the executables without further installation. For easy use, it is recommended to add the binary to your PATH environment variable. Information on how to do this can be found below.

Detailed information regarding different operating systems can be found here:

• Installation on Windows

  To use rclone after the installation, you have to open a terminal (cmd or PowerShell) in the folder where the rclone.exe file is. This can be done with a right click in the empty space wihin the rclone folder and then choosing "open in terminal". Then run all commands with a leading './' (for example ./rclone config. To be able to simply use "rclone" anywhere, you can add it to your path. For instructions see the Troubleshooting section.

• Installation on macOS
  • If Rclone shall be used together with sftp, the sshfs installation instructions must be followed after installing Rclone.
• Installation on Linux

Usage Rclone

To use Rclone you have to define a config file. Afterwards you can connect by using the name of your configured connections.

Configure Remote

Before you can start using Rclone, you need to set up a remote. This means to configure a specific connection by providing authentication information, the network protocol that you want to use and a name for this configuration so that you can use it later on.

To configure a remote for a specific service, you need the following information:

• <remotehost>
• <username>
• <servicePassword>

Furthermore, you have to decide on:

• network protocol (for example webDAV, smb, sftp)
• remote-name (for example you can use the name of the service you want to connect to)

You have three different options to set up a new remote as described in the following sections. It's enough to choose one of those.

Oneliner

Open a terminal and define all parameters in one command. If a ssh key shall be used instead of a password, it might be easier to adjust the config file directly as described in the next section.

# smb
rclone config create <remote-name> smb host=<remotehost> user=<username> domain=<domain> pass=<password> --obscure
# sftp
rclone config create <remote-name> sftp host=<remotehost> user=<username> pass=<password> --obscure
# webdav
rclone config create <remote-name> webdav url=<remotehost> user=<username> pass=<password> --obscure

Adjust Config File

# Make sure that a config file exists. Access rights for the config file are set to 600.  
rclone config touch
# Get the location of the file: 
rclone config file

Open the file and use one of the following snippets as template for your connection:

[<remote-name>]
type = smb
host = <hostname>
user = <userID>
domain = <e.g. BWSERVICESAD>

[<remote-name>]
type = webdav
url = <hostURL>
vendor = other
user = <userID>

[<remote-name>]
type = sftp
host = <hostname>
user = <userID>
key_use_agent = false

[<remote-name-ssh-key-version>]
type = sftp
host = <hostname>
user = <userID>
key_file = ~/.ssh/id_ed25519
pubkey_file = ~/.ssh/id_ed25519.pub
key_use_agent = true
shell_type = none

To add the password, please use the following command. Make sure to keep the prepended space so that the command is not saved in your command history.

 rclone config update <remote-name> pass=<password> --obscure

Interactive Setup

Execute:

rclone config

This will guide you through an interactive setup process. It's quite helpful for checking out which options there are and what the config file looks like after setting values that deviate from the default values. You can find detailed instructions at the website:

• Connect via webdav
• Connect via smb
• Connect via sftp

Use Remote

To see all configured remotes:

rclone config
# Exit with 'q'.

The syntax to use Rclone is like this:

rclone [options] subcommand <parameters> <parameters...>

List all elements in the folder XX:

rclone lsf <remote-name>:XX
# List elements from subfolders as well:
rclone lsf --max-depth 2 <remote-name>:XX

Copies /local/path to the remote path

rclone copy </local/path> <remote-name>:<remote/path>

Copies fom remote path to /local/path

rclone copy <remote-name>:<remote/path> </local/path>

Moves the contents of the source directory to the destination directory.

rclone move <remote-name>:<source/path> <remote-name>:<destination/path>

More subcommands can be found here.

Usage Rclone Mount

Before you can follow the instructions in this chapter, you need to have set up a remote.
More detailed information on how to use rclone mount can be found here.

Windows

To run rclone mount on Windows, you will need to download and install WinFsp first.

# mount under drive letter X (or any other free letter)
rclone mount --vfs-cache-mode full <remote-name>:<path/to/remote/files> X:
# or provide the path to a nonexisting folder:
rclone mount --vfs-cache-mode full <remote-name>:<path/to/remote/files> <C:\path\parentDir\nonexistingFoldername>

To unmount, the command can be stopped in the terminal with the key combination CTRL+C. Alternatively, a mounted drive can be unmounted by making a right click on the drive name in the file explorer and choose "Eject".
In contrast to Linux/Mac, there is no background mode.

MacOS & Linux

You can run mount in either foreground or background (aka daemon) mode. Mount runs in foreground mode by default. Use the --daemon flag to force background mode. If this doesn't work, you can put an & at the end of the command instead.

Create an empty directory on your local machine and then execute

# to mount the root folder:
rclone mount --vfs-cache-mode full <remote-name>: /path/to/empty/folder 
# to mount a subfolder:
rclone mount --vfs-cache-mode full <remote-name>:folderX/folderY /path/to/empty/folder 
# to unmount:
fusermount -uz /path/to/mounted/folder

Best Practices

Rclone has a lot of useful options.

Performance

To be able to utilize a larger bandwidth, it is helpful to add the following options for increased performance:

--transfers <int>

Number of file transfers to run in parallel (default: 4). Depending on the local Network, read and write speeds on the file system, and current load, different values might be best. For large transfers, it is advised to test local performance with different values beforehand.

• In our tests, we observed the best results between 8 and 32.
• For regular use cases, we recommend 16 as the default.
• Values above 64 are not recommended and degrade performance.

--multi-thread-streams <int> 

Number of streams to use for multithreaded downloads (default: 4). Only important on very large files. This will cause multithreaded up/download on chunk-sized bits of the file.

The optimal value is highly specific to the local network and used Hardware. For regular use cases, we recommend 4 as the default.

Debugging and Statistics

To get updates on current progress, use:

--stats

Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable) (default 1m0s).

To get debug information, use:

--log-level=DEBUG 
--stats-log-level=DEBUG

Troubleshooting

1. Windows: Adding rclone to the path

   Needed information: path to the rclone.exe file. If you chose to install via 'winget', the path is "C:\Users\<user>\AppData\Local\Microsoft\WinGet\Packages\Rclone.Rclone_Microsoft.Winget.Source.... . Please look up the exact path.

   • Option 1: Add the rclone.exe file to a folder that is already in the path. You can move it to "C:\Windows\System32\". Then open a new terminal window and test if the command "rclone config" works.
   • Option 2: Add the path of the rclone.exe file to your Windows Path variable:

     Open the Start Search and type env. Choose "Edit environment variables"

     Click the "Environment Variables" button.

     Edit the user variables and add the path to your rclone.exe file.