Helix/Filesystems

Overview

The cluster storage system provides a large parallel file system based on IBM Spectrum Scale for $HOME, for workspaces, and for temporary storage via the $TMPDIR environment variable.

	$HOME	Workspaces	$TMPDIR
Visibility	global	global	local
Lifetime	permanent	workspace lifetime	batch job walltime
Quotas	200 GB	10 TB	none
Backup	no	no	no

• global: all nodes access the same file system.
• local: each node has its own temporary file space.
• permanent: files are stored permanently.
• workspace lifetime: files are removed at end of workspace lifetime.
• batch job walltime: files are removed at end of the batch job.

$HOME

Home directories are meant for permanent storage of files that are kept being used like source codes, configuration files, executable programs. There is currently no backup for the home directory. The disk space per user is limited to 200 GB. The used disk space is displayed with the command:

homequotainfo

Workspaces

Workspace tools can be used to get temporary space for larger amounts of data necessary for or produced by running jobs. To create a workspace you need to supply a name for the workspace and a lifetime in days. The maximum lifetime is 30 days. It is possible to extend the lifetime 10 times.

Command	Action
ws_allocate -r 7 -m <email> foo 10	Allocate a workspace named foo for 10 days and set a email reminder 7 days before expiring. Important: You should add your email address and a relative date for notification!
ws_list -a	List all your workspaces.
ws_find foo	Get absolute path of workspace foo.
ws_extend foo 5	Extend lifetime of workspace foo by 5 days from now.
ws_release foo	Manually erase your workspace foo.
ws_send_ical -m <email> foo	sending a .ical calendar entry for reminding workspace expiring
ws_share share/unshare/unshare-all <workspacename> <username> [username2]	ws_share allows to share an existing workspace with other users.

If you plan to produce or copy large amounts of data in workspaces, please check the availability. The used and free disk space on the workspace filesystem is displayed with the command:

workquotainfo

Restoring expired Workspaces

At expiration time your workspace will be moved to a special, hidden directory. For a short period of time you can still restore your data into a valid workspace. For that, use

ws_restore -l

to get a list of your expired workspaces, and then restore them into an existing, active workspace restored_ws:

ws_restore <deleted_workspacename> restored_ws

NOTE: the expired workspace has to be specified using the full name as listed by ws_restore -l, including username prefix and timestamp suffix (otherwise, it cannot be uniquely identified). The target workspace, on the other hand, must be given with just its short name as listed by ws_list, without the username prefix.

NOTE: ws_restore can only work on the same filesystem! So you have to ensure that the new workspace allocated with ws_allocate is placed on the same filesystem as the expired workspace. Therefor you can use -F <filesystem> flag if needed.

Linking workspaces in Home

It might be valuable to have links to personal workspaces within a certain directory, e.g., the user home directory. The command

ws_register <DIR>

will create and manage links to all personal workspaces within in the directory <DIR>. Calling this command will do the following:

• The directory <DIR> will be created if necessary
• Links to all personal workspaces will be managed:
  • Creates links to all available workspaces if not already present
  • Removes links to released workspaces

Sharing Workspaces

To simplify sharing of workspaces you can use the command

ws_share

This tool setup the needed ACLs on the workspace to allow read-only access to the specified users.

If you need more specific permissions, you can also setup the ACLs on your own, like described here: https://wiki.bwhpc.de/e/Workspace#Sharing_Workspace_Data_within_your_Workgroup

Troubleshooting

If you are getting the error:

Error: could not create workspace directory!

you should check the localesetting of your ssh client. Some clients (e.g. the one from MacOSX) set values that are not valid. You should overwrite LC_CTYPE and set it to a valid locale value like:

export LC_CTYPE=de_DE.UTF-8

To make this change persistent you can add this line also to your .bashrc file.

A list of valid locales can be retrieved via

locale -a

If you are getting the error:

Error: only root can do this <workspacename>

ensure that you are using the FULL workspacename, including your account_prefix, e.g. "hd_ab123-workspacename"

$TMPDIR

The variable $TMPDIR provides file space for temporary data of running jobs. Each node has its own $TMPDIR. The data in $TMPDIR become unavailable as soon as the job has finished.

Access to SDS@hd

It is possible to access your storage space on SDS@hd directly on the bwForCluster Helix in /mnt/sds-hd/ on all login and compute nodes.