BwUniCluster2.0/File System Migration Guide - Revision history

R Barthel: R Barthel moved page BwUniCluster 2.0 File System Migration Guide to BwUniCluster2.0/File System Migration Guide

2022-10-07T21:19:01Z

R Barthel moved page BwUniCluster 2.0 File System Migration Guide to BwUniCluster2.0/File System Migration Guide

← Older revision	Revision as of 23:19, 7 October 2022
(No difference)

R Laifer: /* Hints for migration of $WORK data */

2020-03-23T13:32:46Z

Hints for migration of $WORK data

← Older revision		Revision as of 15:32, 23 March 2020
Line 51:		Line 51:
	</pre>		</pre>
	Of course, if only a part of your old $WORK is needed you should only copy that part. As shown above you can redefine the WORK environment variable. For example, such a definition could be added to old scripts and then no further change on the script should be required.		Of course, if only a part of your old $WORK is needed you should only copy that part. As shown above you can redefine the WORK environment variable. For example, such a definition could be added to old scripts and then no further change on the script should be required.

			The environment variable $WORK will exist and is set after login while the old $WORK file system is mounted during the next 4 weeks. Afterwards it will no longer exist or point to an empty string.

	= LSDF Online Storage =		= LSDF Online Storage =

R Laifer: /* Lustre file systems */

2020-03-18T08:34:12Z

Lustre file systems

← Older revision		Revision as of 10:34, 18 March 2020
Line 3:		Line 3:
	= Lustre file systems =		= Lustre file systems =

	With bwUniCluster 2.0 new file systems for $HOME and workspaces with a total capacity of 5 PB and aggregate throughput of 72 GB/s have been purchased. The user data of the corresponding old file systems has been copied by KIT/SCC to these new file systems. The content of $WORK has to be migrated manually by the users to workspaces during the next 4 weeks, for details see below.		With bwUniCluster 2.0 new file systems for $HOME and workspaces with a total capacity of 5 PB and aggregate throughput of 72 GB/s have been purchased. The user data of the corresponding old file systems has been copied by KIT/SCC to these new file systems. This means that you should see the same data on $HOME and on workspaces as on bwUniCluster 1. The content of $WORK has to be migrated manually by the users to workspaces during the next 4 weeks, for details see below.

	== Changes on $HOME and workspace directories ==		== Changes on $HOME and workspace directories ==

R Laifer: /* New user quota limits for $HOME */

2020-03-16T11:07:24Z

New user quota limits for $HOME

← Older revision		Revision as of 13:07, 16 March 2020
Line 11:		Line 11:
	=== New user quota limits for $HOME ===		=== New user quota limits for $HOME ===

	There are new user quota limits of 2 TiB and 10 million files on $HOME. Users which have currently more data have to reduce their data during the next 4 weeks. You can show your current usage and limits with the following command:		There are new user quota limits of 1 TiB and 10 million files on $HOME. Users which have currently more data have to reduce their data during the next 4 weeks. You can show your current usage and limits with the following command:
	<pre>		<pre>
	$ lfs quota -uh $(whoami) $HOME		$ lfs quota -uh $(whoami) $HOME

R Laifer: /* New user quota limits for $HOME */

2020-03-16T09:39:13Z

New user quota limits for $HOME

← Older revision		Revision as of 11:39, 16 March 2020
Line 15:		Line 15:
	$ lfs quota -uh $(whoami) $HOME		$ lfs quota -uh $(whoami) $HOME
	</pre>		</pre>
	Note that in addition to the user limit there is a limit of your organization which depends on the ~~finiancial~~ share. This limit existed before, too, but it is now enforced with so-called Lustre project quotas and before Lustre group quotas were used. You can show the current usage and limits of your organization with the following command:		Note that in addition to the user limit there is a limit of your organization which depends on the financial share. This limit existed before, too, but it is now enforced with so-called Lustre project quotas and before Lustre group quotas were used. You can show the current usage and limits of your organization with the following command:
	<pre>		<pre>
	lfs quota -ph $(grep $(echo $HOME \| sed -e "s\|/[^/]/[^/]$\|\|") /pfs/data5/project_ids.txt \| cut -f 1 -d\ ) $HOME		lfs quota -ph $(grep $(echo $HOME \| sed -e "s\|/[^/]/[^/]$\|\|") /pfs/data5/project_ids.txt \| cut -f 1 -d\ ) $HOME

R Laifer: /* Changes on $HOME and workspace directories */

2020-03-16T09:37:49Z

Changes on $HOME and workspace directories

← Older revision		Revision as of 11:37, 16 March 2020
Line 7:		Line 7:
	== Changes on $HOME and workspace directories ==		== Changes on $HOME and workspace directories ==

	You should see the same data as before but if this is not true - which is possible in rare cases since the data of expired users has been cleaned up - please contact the hotline or open a ticket. The virtual path to $HOME (/home/ORG/GROUP/USER) is still the same (except for University of Stuttgart users, see below) but the physical path (which can be seen by ''/bin/pwd'') has changed. If one of ~~yours~~ scripts includes a physical path (which is not recommended) you should modify these scripts. If applications complain about the old physical path (which started with ''/pfs/data1/'' or ''/pfs/data2/'') you should modify or recompile the application.		You should see the same data as before but if this is not true - which is possible in rare cases since the data of expired users has been cleaned up - please contact the hotline or open a ticket. The virtual path to $HOME (/home/ORG/GROUP/USER) is still the same (except for University of Stuttgart users, see below) but the physical path (which can be seen by ''/bin/pwd'') has changed. If one of your scripts includes a physical path (which is not recommended) you should modify these scripts. If applications complain about the old physical path (which started with ''/pfs/data1/'' or ''/pfs/data2/'') you should modify or recompile the application.

	=== New user quota limits for $HOME ===		=== New user quota limits for $HOME ===

R Laifer: /* Changes on $HOME and workspace directories */

2020-03-16T09:15:46Z

Changes on $HOME and workspace directories

← Older revision		Revision as of 11:15, 16 March 2020
Line 7:		Line 7:
	== Changes on $HOME and workspace directories ==		== Changes on $HOME and workspace directories ==

	You should see the same data as before but if this is not true - which is possible in rare cases since the data of expired users has been cleaned up - please contact the hotline or open a ticket. The virtual path to $HOME (/home/ORG/GROUP/USER) is still the same (except for University of Stuttgart users, see below) but the physical path (which can be seen by ''/bin/pwd'') has changed. If one of yours scripts includes a physical path (~~wich~~ is not recommended) you should modify these scripts. If applications complain about the old physical path (which started with ''/pfs/data1/'' or ''/pfs/data2/'') you should modify or recompile the application.		You should see the same data as before but if this is not true - which is possible in rare cases since the data of expired users has been cleaned up - please contact the hotline or open a ticket. The virtual path to $HOME (/home/ORG/GROUP/USER) is still the same (except for University of Stuttgart users, see below) but the physical path (which can be seen by ''/bin/pwd'') has changed. If one of yours scripts includes a physical path (which is not recommended) you should modify these scripts. If applications complain about the old physical path (which started with ''/pfs/data1/'' or ''/pfs/data2/'') you should modify or recompile the application.

	=== New user quota limits for $HOME ===		=== New user quota limits for $HOME ===
Line 35:		Line 35:
	=== New group membership for users of University of Stuttgart ===		=== New group membership for users of University of Stuttgart ===

	For users of University of Stuttgart the group membership has usually changed and the owning group for all data below $HOME and the old $WORK has been changed accordingly. This means that the virtual path to $HOME (/home/ORG/GROUP/USER) has changed, too. However, if you use environment variables (for example $HOME) in your scripts (which is recommended) nothing needs to be done. Otherwise you have to change your scripts accordingly.		For users of University of Stuttgart the group membership has usually changed and the owning group for all data below $HOME and the old $WORK has been changed accordingly. This means that the virtual path to $HOME (/home/ORG/GROUP/USER) has changed, too. However, if you use environment variables (for example $HOME) in your scripts (which is recommended) nothing needs to be done. Otherwise you have to change your scripts accordingly.

	== Hints for migration of $WORK data ==		== Hints for migration of $WORK data ==

R Laifer: /* New user quota limits for $HOME */

2020-03-16T08:21:42Z

New user quota limits for $HOME

← Older revision		Revision as of 10:21, 16 March 2020
Line 19:		Line 19:
	lfs quota -ph $(grep $(echo $HOME \| sed -e "s\|/[^/]/[^/]$\|\|") /pfs/data5/project_ids.txt \| cut -f 1 -d\ ) $HOME		lfs quota -ph $(grep $(echo $HOME \| sed -e "s\|/[^/]/[^/]$\|\|") /pfs/data5/project_ids.txt \| cut -f 1 -d\ ) $HOME
	</pre>		</pre>
			With the new quota limits there is no longer a need to check the quotas of other users of the main group. Hence the diskusage files will no longer be created.

	=== New user quota limits for the workspace file system ===		=== New user quota limits for the workspace file system ===

R Laifer: /* New stripe count settings for the workspace file system */

2020-03-16T08:12:25Z

New stripe count settings for the workspace file system

← Older revision		Revision as of 10:12, 16 March 2020
Line 30:		Line 30:
	=== New stripe count settings for the workspace file system ===		=== New stripe count settings for the workspace file system ===

	For the workspace file system directories ~~with help of~~ the new Lustre feature ''Progressive File Layouts'' has been used to define file striping parameters. This means that the stripe count is adapted if the file size is growing. In normal cases users no longer need to adapt file striping parameters in case they have very huge files or in order to reach better performance.		For the workspace file system directories the new Lustre feature ''Progressive File Layouts'' has been used to define file striping parameters. This means that the stripe count is adapted if the file size is growing. In normal cases users no longer need to adapt file striping parameters in case they have very huge files or in order to reach better performance.

	=== New group membership for users of University of Stuttgart ===		=== New group membership for users of University of Stuttgart ===

R Laifer: /* Lustre file systems */

2020-03-13T16:02:09Z

Lustre file systems

← Older revision		Revision as of 18:02, 13 March 2020
Line 3:		Line 3:
	= Lustre file systems =		= Lustre file systems =

	With bwUniCluster 2.0 new file systems for $HOME and workspaces have been purchased. The user data of the corresponding old file systems has been copied by KIT/SCC to these new file systems. The content of $WORK has to be migrated manually by the users to workspaces during the next 4 weeks, for details see below.		With bwUniCluster 2.0 new file systems for $HOME and workspaces with a total capacity of 5 PB and aggregate throughput of 72 GB/s have been purchased. The user data of the corresponding old file systems has been copied by KIT/SCC to these new file systems. The content of $WORK has to be migrated manually by the users to workspaces during the next 4 weeks, for details see below.

	== Changes on $HOME and workspace directories ==		== Changes on $HOME and workspace directories ==