NEMO2/SSH-Login-Disabled-Data-Transfer: Difference between revisions

From bwHPC Wiki
Jump to navigation Jump to search
mNo edit summary
 
(2 intermediate revisions by the same user not shown)
Line 30: Line 30:


'''Note:''' Replace <code>X.X.0.0/16</code> with the actual IP range of your own
'''Note:''' Replace <code>X.X.0.0/16</code> with the actual IP range of your own
university/institution.
university/institution. If you need to allow more than one range (e.g. an
internal and an external one), you can specify multiple IP addresses/ranges at
once, separated by a comma, e.g. <code>10.x.x.x/y,132.230.0.0/16</code>.


'''Tip:''' You can look up your current IP address at
'''Tip:''' You can look up your current IP address at
Line 38: Line 40:
<div style="border: 3px solid #ffc107; background-color: #fff3cd; margin: 10px 0px 10px 0px;">
<div style="border: 3px solid #ffc107; background-color: #fff3cd; margin: 10px 0px 10px 0px;">
<p style="padding-left: 5px; margin: 0px;">'''University of Freiburg users:'''</p>
<p style="padding-left: 5px; margin: 0px;">'''University of Freiburg users:'''</p>
When you visit [https://wieistmeineip.scc.kit.edu/ wieistmeineip.scc.kit.edu], it
[https://wieistmeineip.scc.kit.edu/ wieistmeineip.scc.kit.edu] shows your
shows your '''external (public) IP address'''. However, if you reach NEMO2 over an
'''external''' IP. Inside the university network you usually have an '''internal'''
'''internal IP address''' (e.g. a <code>10.x.x.x</code> network), you must register
<code>10.x.x.x</code> address instead register that one, or use a jump host
that internal/local IP address instead. If you want to use your external IP
(which has its own external-facing IP) or VPN. If you might use both, register
both ranges separated by a comma.
address, you need to use the external university network — either via a jump host or
by activating the '''VPN'''.
</div>
</div>


Line 73: Line 74:
<code>/home/fr/fr_ab1234</code> with your own):
<code>/home/fr/fr_ab1234</code> with your own):
<pre>
<pre>
rsync -e "ssh -i ~/.ssh/id_ed25519_rrsync" <USER>@login2.nemo.uni-freiburg.de:/home/fr/fr_ab1234/ myfiles/
rsync -ahzP -e "ssh -i ~/.ssh/id_ed25519_rrsync" <USER>@login2.nemo.uni-freiburg.de:/home/fr/fr_ab1234/ myfiles/
</pre>
</pre>


Replace <code><USER></code> with your username and <code>id_ed25519_rrsync</code>
Replace <code><USER></code> with your username and <code>id_ed25519_rrsync</code>
with the private key you registered as the command key.
with the private key you registered as the command key. The <code>-a</code> option
is required to copy folders recursively — without it, <code>rsync</code> only
copies top-level files and skips directories. See
[[#Useful_rsync_Options|Useful rsync Options]] below for what the other options do.


'''Workspace example:''' To view or copy the contents of a workspace, use its full
'''Workspace example:''' To view or copy the contents of a workspace, use its full
Line 89: Line 93:
To copy the workspace contents:
To copy the workspace contents:
<pre>
<pre>
rsync -e "ssh -i ~/.ssh/id_ed25519_rrsync" <USER>@login2.nemo.uni-freiburg.de:/work/classic/<USER>-<WS_NAME>/ myfiles/
rsync -ahzP -e "ssh -i ~/.ssh/id_ed25519_rrsync" <USER>@login2.nemo.uni-freiburg.de:/work/classic/<USER>-<WS_NAME>/ myfiles/
</pre>
</pre>


== Useful rsync Options ==
== Useful rsync Options ==


For larger transfers, the following options are recommended:
The copy commands above already use these recommended options:
<pre>
<pre>
rsync -ahzP -e "ssh -i ~/.ssh/id_ed25519_rrsync" <USER>@login2.nemo.uni-freiburg.de:/home/fr/fr_ab1234/ myfiles/
rsync -ahzP -e "ssh -i ~/.ssh/id_ed25519_rrsync" ...
</pre>
</pre>


Line 103: Line 107:
* <code>-z</code> — compress data during transfer (helps on slower connections)
* <code>-z</code> — compress data during transfer (helps on slower connections)
* <code>-P</code> — show progress and allow resuming partially transferred files (equivalent to <code>--partial --progress</code>, so you don't need to add <code>--progress</code> separately)
* <code>-P</code> — show progress and allow resuming partially transferred files (equivalent to <code>--partial --progress</code>, so you don't need to add <code>--progress</code> separately)

'''Not needed on NEMO2 (WEKA storage):''' <code>-x</code>/<code>--one-file-system</code>
prevents rsync from crossing into other mounted filesystems while reading the
source tree. Your NEMO2 home and workspace directories live entirely on the
WEKA filesystem, so there are no other mounts nested underneath them — this
option has no effect here and can be omitted. Also make sure not to pass
<code>-a</code> (or any other option) twice.


<div style="border: 3px solid #ffc107; background-color: #fff3cd; margin: 10px 0px 10px 0px;">
<div style="border: 3px solid #ffc107; background-color: #fff3cd; margin: 10px 0px 10px 0px;">

Latest revision as of 14:02, 11 July 2026

Copying Data from NEMO2 While SSH Login is Disabled

While interactive SSH login is temporarily disabled, you can still transfer data to and from NEMO2 by registering a command SSH key restricted to rsync. This page walks you through the setup.

Important:

  • This only works on login2.
  • Your running jobs are not affected and continue to run normally.

Step 1: Create a Command SSH Key

Follow the instructions for registering a Command Key, using rrsync to restrict the key to read-only access. With -ro, the key only allows downloading (copying) data from NEMO2 to your local machine; uploading data to NEMO2 with this key is not possible.

Use the following values when registering the key:

Field Value
Command
/usr/local/bin/rrsync -ro /
From (IP range) University IP range X.X.0.0/16

Note: Replace X.X.0.0/16 with the actual IP range of your own university/institution. If you need to allow more than one range (e.g. an internal and an external one), you can specify multiple IP addresses/ranges at once, separated by a comma, e.g. 10.x.x.x/y,132.230.0.0/16.

Tip: You can look up your current IP address at wieistmeineip.scc.kit.edu to determine the correct range for your institution.

University of Freiburg users:

wieistmeineip.scc.kit.edu shows your external IP. Inside the university network you usually have an internal 10.x.x.x address instead — register that one, or use a jump host (which has its own external-facing IP) or VPN. If you might use both, register both ranges separated by a comma.

Step 2: Use rsync on login2

Once your command key has been approved, you can use it with rsync on login2 as follows.

To view the available files:

rsync -e "ssh -i ~/.ssh/id_ed25519_rrsync" <USER>@login2.nemo.uni-freiburg.de:

This lists the restricted base directory (/, as set on the command key) itself, so it always works without specifying a path.

Why do I need a full path to copy, but not to view?

The command key restricts rrsync to the base directory / (as configured in Step 1). Any relative path you pass (e.g. myfiles/) is resolved relative to that base directory /, not your home directory — so myfiles/ would actually be looked up as /myfiles, which does not exist. You must therefore always use the full absolute path, starting with /, e.g. your home directory (/home/fr/fr_ab1234).

To copy files, use the full absolute path of your home directory (replace /home/fr/fr_ab1234 with your own):

rsync -ahzP -e "ssh -i ~/.ssh/id_ed25519_rrsync" <USER>@login2.nemo.uni-freiburg.de:/home/fr/fr_ab1234/ myfiles/

Replace <USER> with your username and id_ed25519_rrsync with the private key you registered as the command key. The -a option is required to copy folders recursively — without it, rsync only copies top-level files and skips directories. See Useful rsync Options below for what the other options do.

Workspace example: To view or copy the contents of a workspace, use its full path (/work/classic/<USER>-<WS_NAME>) instead of myfiles/.

To view the workspace contents:

rsync -e "ssh -i ~/.ssh/id_ed25519_rrsync" <USER>@login2.nemo.uni-freiburg.de:/work/classic/<USER>-<WS_NAME>/

To copy the workspace contents:

rsync -ahzP -e "ssh -i ~/.ssh/id_ed25519_rrsync" <USER>@login2.nemo.uni-freiburg.de:/work/classic/<USER>-<WS_NAME>/ myfiles/

Useful rsync Options

The copy commands above already use these recommended options:

rsync -ahzP -e "ssh -i ~/.ssh/id_ed25519_rrsync" ...
  • -a — archive mode (recursive, preserves permissions/times/symlinks/...)
  • -h — human-readable sizes in the output
  • -z — compress data during transfer (helps on slower connections)
  • -P — show progress and allow resuming partially transferred files (equivalent to --partial --progress, so you don't need to add --progress separately)

Passphrase-protected key?

If your private key is protected with a passphrase, add it to your local ssh-agent once before running the commands above, otherwise you will be prompted for the passphrase on every rsync call:

ssh-add ~/.ssh/id_ed25519_rrsync

Note: You can also configure a matching entry in your ~/.ssh/config instead of passing -e "ssh -i ..." on every command. Remember that this only works with login2.