User:M Janczyk/Migrate NEMO1 workspaces to NEMO2 workspaces - Revision history

M Janczyk: M Janczyk moved page NEMO2/Migrate NEMO1 workspaces to NEMO2 workspaces to User:M Janczyk/Migrate NEMO1 workspaces to NEMO2 workspaces without leaving a redirect

2026-04-21T08:13:19Z

M Janczyk moved page NEMO2/Migrate NEMO1 workspaces to NEMO2 workspaces to User:M Janczyk/Migrate NEMO1 workspaces to NEMO2 workspaces without leaving a redirect

← Older revision	Revision as of 10:13, 21 April 2026
(No difference)

M Janczyk at 09:55, 25 March 2025

2025-03-25T09:55:34Z

← Older revision		Revision as of 11:55, 25 March 2025
Line 1:		Line 1:
			{\|style="background:#ffa833; width:100%;"
			\|style="padding:5px; background:##ffa833; text-align:left"\|
			[[Image:Attention.svg\|center\|25px]]
			\|style="padding:5px; background:#ffa833; text-align:left"\|
			NEMO2 workspaces are currently limited to 5 TB per workspace.
			Larger workspaces should be split up first.
			\|}

	You can migrate your NEMO1 workspace to NEMO2 using <code>rsync</code> or the <code>migratenemo1ws</code> script.		You can migrate your NEMO1 workspace to NEMO2 using <code>rsync</code> or the <code>migratenemo1ws</code> script.

M Janczyk at 15:11, 4 March 2025

2025-03-04T15:11:42Z

← Older revision		Revision as of 17:11, 4 March 2025
Line 5:		Line 5:
	ssh -l <username> nemo2-login.nemo.uni-freiburg.de		ssh -l <username> nemo2-login.nemo.uni-freiburg.de
	</pre>		</pre>

			__TOC__

	== Using migratenemo1ws to migrate workspaces ==		== Using migratenemo1ws to migrate workspaces ==
Line 22:		Line 24:
	This leads to the following command:		This leads to the following command:
	<pre>		<pre>
	rsync -ahxzP --progress ~~NEMO1~~:/work/ws/nemo/<username>-<nemo1wsname>-0/ /work/classic/<username>-<nemo2wsname>/		rsync -ahxzP --progress workflow.nemo.uni-freiburg.de:/work/ws/nemo/<username>-<nemo1wsname>-0/ /work/classic/<username>-<nemo2wsname>/
	</pre>		</pre>

Line 85:		Line 87:
	Ctrl+D		Ctrl+D
	</pre>		</pre>


	== Using rsync manually ==		== Using rsync manually ==
Line 93:		Line 94:
	You can use the above command to migrate your NEMO1 workspace to NEMO2:		You can use the above command to migrate your NEMO1 workspace to NEMO2:
	<pre>		<pre>
	rsync -ahxzP --progress ~~NEMO1~~:/work/ws/nemo/<username>-<nemo1wsname>-0/ /work/classic/<username>-<nemo2wsname>/		rsync -ahxzP --progress login.nemo.uni-freiburg.de:/work/ws/nemo/<username>-<nemo1wsname>-0/ /work/classic/<username>-<nemo2wsname>/
	</pre>		</pre>

	To test which files will be transferred without actually transferring them, use the <code>--dry-run</code> option:		To test which files will be transferred without actually transferring them, use the <code>--dry-run</code> option:
	<pre>		<pre>
	rsync -ahxzP --progress --dry-run ~~NEMO1~~:/work/ws/nemo/<username>-<nemo1wsname>-0/ /work/classic/<username>-<nemo2wsname>/		rsync -ahxzP --progress --dry-run login.nemo.uni-freiburg.de:/work/ws/nemo/<username>-<nemo1wsname>-0/ /work/classic/<username>-<nemo2wsname>/
	</pre>		</pre>
	To delete files in the target that are not in the source, use the <code>--delete</code> option:		To delete files in the target that are not in the source, use the <code>--delete</code> option:
	<pre>		<pre>
	rsync -ahxzP --progress --delete ~~NEMO1~~:/work/ws/nemo/<username>-<nemo1wsname>-0/ /work/classic/<username>-<nemo2wsname>/		rsync -ahxzP --progress --delete login.nemo.uni-freiburg.de:/work/ws/nemo/<username>-<nemo1wsname>-0/ /work/classic/<username>-<nemo2wsname>/
	</pre>		</pre>

Line 111:		Line 112:

	You can migrate directories in your old NEMO1 home directory to NEMO2 using <code>rsync</code> with the following command:		You can migrate directories in your old NEMO1 home directory to NEMO2 using <code>rsync</code> with the following command:
	<code>rsync -ahxzP --progress ~~NEMO1~~:<dirtomigrate>/ $HOME/nemo1homebak/<dirtomigrate>/</code>		<code>rsync -ahxzP --progress login.nemo.uni-freiburg.de:<dirtomigrate>/ $HOME/nemo1homebak/<dirtomigrate>/</code>

	Example:		Example:
Line 117:		Line 118:
	# migrate some job directory from old NEMO1 home to NEMO2 (replace <some_job> with the actual directory name)		# migrate some job directory from old NEMO1 home to NEMO2 (replace <some_job> with the actual directory name)
	mkdir -p $HOME/nemo1homebak/<some_job>/		mkdir -p $HOME/nemo1homebak/<some_job>/
	rsync -ahxzP --progress ~~NEMO1~~:<some_job>/ $HOME/nemo1homebak/<some_job>/		rsync -ahxzP --progress login.nemo.uni-freiburg.de:<some_job>/ $HOME/nemo1homebak/<some_job>/
	</pre>		</pre>

	To test which files will be transferred without actually transferring them, use the <code>--dry-run</code> option:		To test which files will be transferred without actually transferring them, use the <code>--dry-run</code> option:
	<pre>		<pre>
	rsync -ahxzP --progress --dry-run ~~NEMO1~~:<dirtomigrate>/ $HOME/nemo1homebak/<dirtomigrate>/		rsync -ahxzP --progress --dry-run login.nemo.uni-freiburg.de:<dirtomigrate>/ $HOME/nemo1homebak/<dirtomigrate>/
	</pre>		</pre>
	To delete files in the target that are not in the source, use the <code>--delete</code> option:		To delete files in the target that are not in the source, use the <code>--delete</code> option:
	<pre>		<pre>
	rsync -ahxzP --progress --delete ~~NEMO1~~:<dirtomigrate>/ $HOME/nemo1homebak/<dirtomigrate>/		rsync -ahxzP --progress --delete login.nemo.uni-freiburg.de:<dirtomigrate>/ $HOME/nemo1homebak/<dirtomigrate>/
	</pre>		</pre>

M Janczyk at 14:36, 4 March 2025

2025-03-04T14:36:53Z

← Older revision		Revision as of 16:36, 4 March 2025
Line 22:		Line 22:
	This leads to the following command:		This leads to the following command:
	<pre>		<pre>
	rsync -ahxzP --progress NEMO1:/work/ws/nemo/<username>-<~~nemo2wsname~~>-0/ /work/classic/<username>-<nemo2wsname>/		rsync -ahxzP --progress NEMO1:/work/ws/nemo/<username>-<nemo1wsname>-0/ /work/classic/<username>-<nemo2wsname>/
	</pre>		</pre>

Line 93:		Line 93:
	You can use the above command to migrate your NEMO1 workspace to NEMO2:		You can use the above command to migrate your NEMO1 workspace to NEMO2:
	<pre>		<pre>
	rsync -ahxzP --progress NEMO1:/work/ws/nemo/<username>-<~~nemo2wsname~~>-0/ /work/classic/<username>-<nemo2wsname>/		rsync -ahxzP --progress NEMO1:/work/ws/nemo/<username>-<nemo1wsname>-0/ /work/classic/<username>-<nemo2wsname>/
	</pre>		</pre>

	To test which files will be transferred without actually transferring them, use the <code>--dry-run</code> option:		To test which files will be transferred without actually transferring them, use the <code>--dry-run</code> option:
	<pre>		<pre>
	rsync -ahxzP --progress --dry-run NEMO1:/work/ws/nemo/<username>-<~~nemo2wsname~~>-0/ /work/classic/<username>-<nemo2wsname>/		rsync -ahxzP --progress --dry-run NEMO1:/work/ws/nemo/<username>-<nemo1wsname>-0/ /work/classic/<username>-<nemo2wsname>/
	</pre>		</pre>
	To delete files in the target that are not in the source, use the <code>--delete</code> option:		To delete files in the target that are not in the source, use the <code>--delete</code> option:
	<pre>		<pre>
	rsync -ahxzP --progress --delete NEMO1:/work/ws/nemo/<username>-<~~nemo2wsname~~>-0/ /work/classic/<username>-<nemo2wsname>/		rsync -ahxzP --progress --delete NEMO1:/work/ws/nemo/<username>-<nemo1wsname>-0/ /work/classic/<username>-<nemo2wsname>/
	</pre>		</pre>

M Janczyk: M Janczyk moved page Migrate NEMO1 workspaces to NEMO2 workspaces to NEMO2/Migrate NEMO1 workspaces to NEMO2 workspaces without leaving a redirect

2025-03-03T18:40:39Z

M Janczyk moved page Migrate NEMO1 workspaces to NEMO2 workspaces to NEMO2/Migrate NEMO1 workspaces to NEMO2 workspaces without leaving a redirect

← Older revision	Revision as of 20:40, 3 March 2025
(No difference)

M Janczyk: Created page with "You can migrate your NEMO1 workspace to NEMO2 using rsync or the migratenemo1ws script. Login to NEMO2 (after registration):

 ssh -l  nemo2-login.nemo.uni-freiburg.de

== Using migratenemo1ws to migrate workspaces == You can migrate your NEMO1 workspace to NEMO2 using the migratenemo1ws script. It will migrate your NEMO1 workspace to NEMO2 using rsync with predefined options-a -h -x -z -P -..."


2025-03-03T18:40:06Z
Created page with "You can migrate your NEMO1 workspace to NEMO2 using <code>rsync</code> or the <code>migratenemo1ws</code> script.  Login to NEMO2 (after registration): <pre> ssh -l <username> nemo2-login.nemo.uni-freiburg.de </pre>  == Using migratenemo1ws to migrate workspaces ==  You can migrate your NEMO1 workspace to NEMO2 using the <code>migratenemo1ws</code> script. It will migrate your NEMO1 workspace to NEMO2 using <code>rsync</code> with predefined options<code>-a -h -x -z -P -..."
New page
You can migrate your NEMO1 workspace to NEMO2 using <code>rsync</code> or the <code>migratenemo1ws</code> script.



Login to NEMO2 (after registration):

<pre>

ssh -l <username> nemo2-login.nemo.uni-freiburg.de

</pre>



== Using migratenemo1ws to migrate workspaces ==



You can migrate your NEMO1 workspace to NEMO2 using the <code>migratenemo1ws</code> script.

It will migrate your NEMO1 workspace to NEMO2 using <code>rsync</code> with predefined options<code>-a -h -x -z -P --progress</code>:



<pre>

-a: archive mode; equals -rlptgoD (no -H, -A, -X)

-h: output numbers in a human-readable format

-x: don't cross filesystem boundaries

-z: compress file data during the transfer

-P: keep partially transferred files

--progress: show progress during transfer

</pre>



This leads to the following command:

<pre>

rsync -ahxzP --progress NEMO1:/work/ws/nemo/<username>-<nemo2wsname>-0/ /work/classic/<username>-<nemo2wsname>/

</pre>



The script will also delete files in the target that are not in the source, if you use the <code>-d</code> option.

To test the migration without transferring any files, use the <code>-n</code> option for a dry run.

If you want to clean up the installation files and exit, use the <code>-c</code> option.



The help message is as follows:

<pre>

user@login3 ~> migratenemo1ws -h

Usage:

  /usr/local/bin/migratenemo1ws [-h] [-c] [-d] [-n] [-s <source>] [-t <target>]



  -h            this help

  -c            cleanup installation files and exit

  -d            delete files in target that are not in source

  -n            perform a dry run without transferring any files

  -s <source>   use this NEMO1 workspace name as source

  -t <target>   use this NEMO2 workspace name as target



Example:

  /usr/local/bin/migratenemo1ws -s nemo1ws -t nemo2ws



Migrating workspaces may take a while, depending on the amount of data to be transferred.

We suggest to run this script in a screen/tmux session.

You can detach from the session by pressing Ctrl+A, D (Ctrl+B, D, for tmux).

You can reattach to the session by running <code>screen -r</code> (<code>tmux attach</code>).

You can cancel the migration by pressing Ctrl+C.

</pre>



It is recommended to run this script in a <code>screen</code> or <code>tmux</code> session, as the migration may take a while.

You can detach from the session by pressing Ctrl+A, D (Ctrl+B, D, for tmux).

You can reattach to the session by running <code>screen -r</code> (<code>tmux attach</code>).

You can cancel the migration by pressing Ctrl+C.

<pre>

# For screen users:

# start a screen session

screen -S migratenemo1ws

# start data migration

migratenemo1ws -s nemo1ws -t nemo2ws

# detach from the screen session

Ctrl+A, D

# reattach to the screen session

screen -r migratenemo1ws

# cancel migration, if needed

Ctrl+C

# exit screen session, when done

Ctrl+D



# For tmux users:

# start a tmux session

tmux new -s migratenemo1ws

# start data migration

migratenemo1ws -s nemo1ws -t nemo2ws

# detach from the tmux session

Ctrl+B, D

# reattach to the tmux session

tmux attach -t migratenemo1ws

# cancel migration, if needed

Ctrl+C

# exit tmux session, when done

Ctrl+D

</pre>





== Using rsync manually ==



<code>rsync</code> is a fast and versatile file copying tool. It can be used to migrate your NEMO1 workspace to NEMO2.



You can use the above command to migrate your NEMO1 workspace to NEMO2:

<pre>

rsync -ahxzP --progress NEMO1:/work/ws/nemo/<username>-<nemo2wsname>-0/ /work/classic/<username>-<nemo2wsname>/

</pre>



To test which files will be transferred without actually transferring them, use the <code>--dry-run</code> option:

<pre>

rsync -ahxzP --progress --dry-run NEMO1:/work/ws/nemo/<username>-<nemo2wsname>-0/ /work/classic/<username>-<nemo2wsname>/

</pre>

To delete files in the target that are not in the source, use the <code>--delete</code> option:

<pre>

rsync -ahxzP --progress --delete NEMO1:/work/ws/nemo/<username>-<nemo2wsname>-0/ /work/classic/<username>-<nemo2wsname>/

</pre>



== Migrating directories from old NEMO1 HOME ==



We suggest migrating only files that are needed for your work.

Try not to copy everything from your old NEMO1 home directory, since it may contain a lot of unnecessary files.



You can migrate directories in your old NEMO1 home directory to NEMO2 using <code>rsync</code> with the following command:

<code>rsync -ahxzP --progress NEMO1:<dirtomigrate>/ $HOME/nemo1homebak/<dirtomigrate>/</code>



Example:

<pre>

# migrate some job directory from old NEMO1 home to NEMO2 (replace <some_job> with the actual directory name)

mkdir -p $HOME/nemo1homebak/<some_job>/

rsync -ahxzP --progress NEMO1:<some_job>/ $HOME/nemo1homebak/<some_job>/

</pre>



To test which files will be transferred without actually transferring them, use the <code>--dry-run</code> option:

<pre>

rsync -ahxzP --progress --dry-run NEMO1:<dirtomigrate>/ $HOME/nemo1homebak/<dirtomigrate>/

</pre>

To delete files in the target that are not in the source, use the <code>--delete</code> option:

<pre>

rsync -ahxzP --progress --delete NEMO1:<dirtomigrate>/ $HOME/nemo1homebak/<dirtomigrate>/

</pre>