NEMO2/SSH-Login-Disabled-Data-Transfer: Difference between revisions

From bwHPC Wiki
Jump to navigation Jump to search
 
(3 intermediate revisions by the same user not shown)
Line 30: Line 30:


'''Note:''' Replace <code>X.X.0.0/16</code> with the actual IP range of your own
'''Note:''' Replace <code>X.X.0.0/16</code> with the actual IP range of your own
university/institution.
university/institution. If you need to allow more than one range (e.g. an
internal and an external one), you can specify multiple IP addresses/ranges at
once, separated by a comma, e.g. <code>10.x.x.x/y,132.230.0.0/16</code>.


'''Tip:''' You can look up your current IP address at
'''Tip:''' You can look up your current IP address at
Line 38: Line 40:
<div style="border: 3px solid #ffc107; background-color: #fff3cd; margin: 10px 0px 10px 0px;">
<div style="border: 3px solid #ffc107; background-color: #fff3cd; margin: 10px 0px 10px 0px;">
<p style="padding-left: 5px; margin: 0px;">'''University of Freiburg users:'''</p>
<p style="padding-left: 5px; margin: 0px;">'''University of Freiburg users:'''</p>
When you visit [https://wieistmeineip.scc.kit.edu/ wieistmeineip.scc.kit.edu], it
[https://wieistmeineip.scc.kit.edu/ wieistmeineip.scc.kit.edu] shows your
shows your '''external (public) IP address'''. However, if you reach NEMO2 over an
'''external''' IP. Inside the university network you usually have an '''internal'''
'''internal IP address''' (e.g. a <code>10.x.x.x</code> network), you must register
<code>10.x.x.x</code> address instead register that one, or use a jump host
that internal/local IP address instead. If you want to use your external IP
(which has its own external-facing IP) or VPN. If you might use both, register
both ranges separated by a comma.
address, you need to use the external university network — either via a jump host or
by activating the '''VPN'''.
</div>
</div>


Line 56: Line 57:
</pre>
</pre>


This lists the restricted base directory (<code>/</code>, as set on the command key)
To copy files:
itself, so it always works without specifying a path.

<div style="border: 3px solid #ffc107; background-color: #fff3cd; margin: 10px 0px 10px 0px;">
<p style="padding-left: 5px; margin: 0px;">'''Why do I need a full path to copy, but not to view?'''</p>
The command key restricts <code>rrsync</code> to the base directory <code>/</code>
(as configured in [[#Step_1:_Create_a_Command_SSH_Key|Step 1]]). Any '''relative'''
path you pass (e.g. <code>myfiles/</code>) is resolved relative to that base
directory <code>/</code>, '''not''' your home directory — so
<code>myfiles/</code> would actually be looked up as <code>/myfiles</code>, which
does not exist. You must therefore always use the '''full absolute path''', starting
with <code>/</code>, e.g. your home directory (<code>/home/fr/fr_ab1234</code>).
</div>

To copy files, use the full absolute path of your home directory (replace
<code>/home/fr/fr_ab1234</code> with your own):
<pre>
<pre>
rsync -e "ssh -i ~/.ssh/id_ed25519_rrsync" <USER>@login2.nemo.uni-freiburg.de:myfiles/ myfiles/
rsync -ahzP -e "ssh -i ~/.ssh/id_ed25519_rrsync" <USER>@login2.nemo.uni-freiburg.de:/home/fr/fr_ab1234/ myfiles/
</pre>
</pre>


Replace <code><USER></code> with your username and <code>id_ed25519_rrsync</code>
Replace <code><USER></code> with your username and <code>id_ed25519_rrsync</code>
with the private key you registered as the command key.
with the private key you registered as the command key. The <code>-a</code> option
is required to copy folders recursively — without it, <code>rsync</code> only
copies top-level files and skips directories. See
[[#Useful_rsync_Options|Useful rsync Options]] below for what the other options do.


'''Workspace example:''' To view or copy the contents of a workspace, use its full
'''Workspace example:''' To view or copy the contents of a workspace, use its full
Line 74: Line 93:
To copy the workspace contents:
To copy the workspace contents:
<pre>
<pre>
rsync -e "ssh -i ~/.ssh/id_ed25519_rrsync" <USER>@login2.nemo.uni-freiburg.de:/work/classic/<USER>-<WS_NAME>/ myfiles/
rsync -ahzP -e "ssh -i ~/.ssh/id_ed25519_rrsync" <USER>@login2.nemo.uni-freiburg.de:/work/classic/<USER>-<WS_NAME>/ myfiles/
</pre>
</pre>

== Useful rsync Options ==

The copy commands above already use these recommended options:
<pre>
rsync -ahzP -e "ssh -i ~/.ssh/id_ed25519_rrsync" ...
</pre>

* <code>-a</code> — archive mode (recursive, preserves permissions/times/symlinks/...)
* <code>-h</code> — human-readable sizes in the output
* <code>-z</code> — compress data during transfer (helps on slower connections)
* <code>-P</code> — show progress and allow resuming partially transferred files (equivalent to <code>--partial --progress</code>, so you don't need to add <code>--progress</code> separately)


<div style="border: 3px solid #ffc107; background-color: #fff3cd; margin: 10px 0px 10px 0px;">
<div style="border: 3px solid #ffc107; background-color: #fff3cd; margin: 10px 0px 10px 0px;">

Latest revision as of 14:02, 11 July 2026

Copying Data from NEMO2 While SSH Login is Disabled

While interactive SSH login is temporarily disabled, you can still transfer data to and from NEMO2 by registering a command SSH key restricted to rsync. This page walks you through the setup.

Important:

  • This only works on login2.
  • Your running jobs are not affected and continue to run normally.

Step 1: Create a Command SSH Key

Follow the instructions for registering a Command Key, using rrsync to restrict the key to read-only access. With -ro, the key only allows downloading (copying) data from NEMO2 to your local machine; uploading data to NEMO2 with this key is not possible.

Use the following values when registering the key:

Field Value
Command
/usr/local/bin/rrsync -ro /
From (IP range) University IP range X.X.0.0/16

Note: Replace X.X.0.0/16 with the actual IP range of your own university/institution. If you need to allow more than one range (e.g. an internal and an external one), you can specify multiple IP addresses/ranges at once, separated by a comma, e.g. 10.x.x.x/y,132.230.0.0/16.

Tip: You can look up your current IP address at wieistmeineip.scc.kit.edu to determine the correct range for your institution.

University of Freiburg users:

wieistmeineip.scc.kit.edu shows your external IP. Inside the university network you usually have an internal 10.x.x.x address instead — register that one, or use a jump host (which has its own external-facing IP) or VPN. If you might use both, register both ranges separated by a comma.

Step 2: Use rsync on login2

Once your command key has been approved, you can use it with rsync on login2 as follows.

To view the available files:

rsync -e "ssh -i ~/.ssh/id_ed25519_rrsync" <USER>@login2.nemo.uni-freiburg.de:

This lists the restricted base directory (/, as set on the command key) itself, so it always works without specifying a path.

Why do I need a full path to copy, but not to view?

The command key restricts rrsync to the base directory / (as configured in Step 1). Any relative path you pass (e.g. myfiles/) is resolved relative to that base directory /, not your home directory — so myfiles/ would actually be looked up as /myfiles, which does not exist. You must therefore always use the full absolute path, starting with /, e.g. your home directory (/home/fr/fr_ab1234).

To copy files, use the full absolute path of your home directory (replace /home/fr/fr_ab1234 with your own):

rsync -ahzP -e "ssh -i ~/.ssh/id_ed25519_rrsync" <USER>@login2.nemo.uni-freiburg.de:/home/fr/fr_ab1234/ myfiles/

Replace <USER> with your username and id_ed25519_rrsync with the private key you registered as the command key. The -a option is required to copy folders recursively — without it, rsync only copies top-level files and skips directories. See Useful rsync Options below for what the other options do.

Workspace example: To view or copy the contents of a workspace, use its full path (/work/classic/<USER>-<WS_NAME>) instead of myfiles/.

To view the workspace contents:

rsync -e "ssh -i ~/.ssh/id_ed25519_rrsync" <USER>@login2.nemo.uni-freiburg.de:/work/classic/<USER>-<WS_NAME>/

To copy the workspace contents:

rsync -ahzP -e "ssh -i ~/.ssh/id_ed25519_rrsync" <USER>@login2.nemo.uni-freiburg.de:/work/classic/<USER>-<WS_NAME>/ myfiles/

Useful rsync Options

The copy commands above already use these recommended options:

rsync -ahzP -e "ssh -i ~/.ssh/id_ed25519_rrsync" ...
  • -a — archive mode (recursive, preserves permissions/times/symlinks/...)
  • -h — human-readable sizes in the output
  • -z — compress data during transfer (helps on slower connections)
  • -P — show progress and allow resuming partially transferred files (equivalent to --partial --progress, so you don't need to add --progress separately)

Passphrase-protected key?

If your private key is protected with a passphrase, add it to your local ssh-agent once before running the commands above, otherwise you will be prompted for the passphrase on every rsync call:

ssh-add ~/.ssh/id_ed25519_rrsync

Note: You can also configure a matching entry in your ~/.ssh/config instead of passing -e "ssh -i ..." on every command. Remember that this only works with login2.