BinAC2/Migrate BinAC 1 workspaces to BinAC 2 workspaces

Preamble

Remove unnecessary data

Now is a good possibility getting rid of unnecessary data. Don't migrate data to BinAC 2 that you know you won't ever touch again. Instead backup it if needed and delete it from BinAC 1.

BinAC 2 storage architecture

Please read about the BinAC 2 file systems and understand the difference between project and work file system before migrating any data.

About rsync

You will use rsync for migrating your data from BinAC 1 to BinAC 2. You may want to use the following options:

-a: archive mode; equals -rlptgoD (no -H, -A, -X)
-h: output numbers in a human-readable format
-x: don't cross filesystem boundaries
-z: compress file data during the transfer
-P: keep partially transferred files
--progress: show progress during transfer
--dry-run: perform a trial run that doesn’t make any changes (and produces mostly the same output as a real run)

Mind trailing / in the commands. They make a big difference when working with rsync !

If you are unsure, run rsync add the --dry-run to your rsync command. This makes rsync perform a trial run that doesn’t make any changes (and produces mostly the same output as a real run).

About screen and tmux

It is recommended to migrate data in a screen or tmux session, as the migration may take a while. You can detach from the session by pressing Ctrl+A, D (Ctrl+B, D, for tmux). You can reattach to the session by running screen -r (tmux attach). You can cancel the migration by pressing Ctrl+C.

# For screen users:
# start a screen session
screen -S migratebinacws

# start data migration, commands see below

# detach from the screen session
Ctrl+A, D
# reattach to the screen session
screen -r migratenemo1ws
# cancel migration, if needed
Ctrl+C
# exit screen session, when done
Ctrl+D

# For tmux users:
# start a tmux session
tmux new -s migratebinacws

# start data migration, commands see below

# detach from the tmux session
Ctrl+B, D
# reattach to the tmux session
tmux attach -t migratenemo1ws
# cancel migration, if needed
Ctrl+C
# exit tmux session, when done
Ctrl+D

Migration

All commands assume that you are logged in on BinAC 1 and you also have a working account on BinAC 2.

Migrate BinAC 1 shared folder

If you used a directory at /beegfs/work for sharing data with people from the same compute project you can sync this directory to the new project file system at /pfs/10/project/.

Depending on how you want to organize your data in the new project file system we will give some example commands:

This command will copy the content of the shared directory on BinAC 1 into the project directory on BinAC 2:

rsync -ahxzP --progress /beegfs/work/<your shared directory>/* ${USER}@login.binac2.uni-tuebingen.de:/pfs/10/project/<RV acronym>/

This command will copy the shared directory on BinAC 1 into the projet directory on BinAC 2. E.g. it will create a subdirectory /pfs/10/project/<RV acronym>/

rsync -ahxzP --progress /beegfs/work/<your shared directory>/ ${USER}@login.binac2.uni-tuebingen.de:/pfs/10/project/<RV acronym>/<subdir name>

Migrate personal BinAC 1 workspaces

This can be a directory at /beegfs/work or a workspace created by the workspace tools at /beegfs/work/workspace/ws.

If the directory contains data you don't need for computations in the next time, please migrate it to a personal directory in the projects file system: Although project is accessible for every compute project member, you can create directories only you can access. This will save precious space on the fast (but not tinier) BinAC 2 work.

# On BinAC 1
rsync -ahxzP --progress /beegfs/work/workspace/ws/<directory name>/ ${USER}@login.binac2.uni-tuebingen.de:/pfs/10/project/<RV acronym>/<directory name>

# On BinAC 2: Change permissions. Only you can access the directory, nobody else.
chmod 700 /pfs/10/project/<RV acronym>/<directory name>

If the directory contains data you will use for computations it's totally fine to migrate it to BinAC 2 work file system. Please not that (in contrast to BinAC 1) you cannot create directories yourself in the work file system with mkdir. Everything at /pfs/10/work is a workspace managed by the workspace tools.

For each of your workspaces on BinAC 1 you want to migrate you will have to create a workspace on BinAC 2:

ws_allocate <workspace name> 30

You can now sync the files from BinAC 1 to BinAC 2:

This command will copy the content of the workspace on BinAC 1 into the workspace on BinAC 2:

rsync -ahxzP --progress /beegfs/work/workspace/ws/<directory name>/* ${USER}@login.binac2.uni-tuebingen.de:/pfs/10/work/<workspace name>/