The cluster storage system provides a large parallel file system based on IBM Spectrum Scale for $HOME, for workspaces, and for temporary storage via the $TMPDIR environment variable.
|Lifetime||permanent||workspace lifetime||batch job walltime|
|Quotas||200 GB||10 TB||none|
- global: all nodes access the same file system.
- local: each node has its own temporary file space.
- permanent: files are stored permanently.
- workspace lifetime: files are removed at end of workspace lifetime.
- batch job walltime: files are removed at end of the batch job.
Home directories are meant for permanent storage of files that are kept being used like source codes, configuration files, executable programs. There is currently no backup for the home directory. The disk space per user is limited to 200 GB. The used disk space is displayed with the command:
Workspace tools can be used to get temporary space for larger amounts of data necessary for or produced by running jobs. To create a workspace you need to supply a name for the workspace and a lifetime in days. The maximum lifetime is 30 days. It is possible to extend the lifetime 10 times.
|ws_allocate -r 7 -m <email> foo 10||Allocate a workspace named foo for 10 days and set a email reminder 7 days before expiring.
Important: You should add your email address and a relative date for notification!
|ws_list -a||List all your workspaces.|
|ws_find foo||Get absolute path of workspace foo.|
|ws_extend foo 5||Extend lifetime of workspace foo by 5 days from now.|
|ws_release foo||Manually erase your workspace foo.|
|ws_send_ical -m <email> foo||sending a .ical calendar entry for reminding workspace expiring|
|ws_share share/unshare/unshare-all <workspacename> <username> [username2]||ws_share allows to share an existing workspace with other users.|
If you plan to produce or copy large amounts of data in workspaces, please check the availability. The used and free disk space on the workspace filesystem is displayed with the command:
3.1 Restoring expired Workspaces
At expiration time your workspace will be moved to a special, hidden directory. For a short period of time you can still restore your data into a valid workspace. For that, use
to get a list of your expired workspaces, and then restore them into an existing, active workspace
ws_restore <deleted_workspacename> restored_ws
NOTE: the expired workspace has to be specified using the full name as listed by
ws_restore -l, including username prefix and timestamp suffix (otherwise, it cannot be uniquely identified).
The target workspace, on the other hand, must be given with just its short name as listed by
ws_list, without the username prefix.
ws_restore can only work on the same filesystem! So you have to ensure that the new workspace allocated with
ws_allocate is placed on the same filesystem as the expired workspace. Therefor you can use
-F <filesystem> flag if needed.
3.2 Linking workspaces in Home
It might be valuable to have links to personal workspaces within a certain directory, e.g., the user home directory. The command
will create and manage links to all personal workspaces within in the directory <DIR>. Calling this command will do the following:
- The directory <DIR> will be created if necessary
- Links to all personal workspaces will be managed:
- Creates links to all available workspaces if not already present
- Removes links to released workspaces
3.3 Sharing Workspaces
To simplify sharing of workspaces you can use the command
This tool setup the needed ACLs on the workspace to allow read-only access to the specified users.
If you need more specific permissions, you can also setup the ACLs on your own, like described here: https://wiki.bwhpc.de/e/Workspace#Sharing_Workspace_Data_within_your_Workgroup
If you are getting the error:
Error: could not create workspace directory!
you should check the localesetting of your ssh client. Some clients (e.g. the one from MacOSX) set values that are not valid. You should overwrite LC_CTYPE and set it to a valid locale value like:
To make this change persistent you can add this line also to your
A list of valid locales can be retrieved via
If you are getting the error:
Error: only root can do this <workspacename>
ensure that you are using the FULL workspacename, including your account_prefix, e.g. "hd_ab123-workspacename"
The variable $TMPDIR provides file space for temporary data of running jobs. Each node has its own $TMPDIR. The data in $TMPDIR become unavailable as soon as the job has finished.
5 Access to SDS@hd
It is possible to access your storage space on SDS@hd directly on the bwForCluster Helix in /mnt/sds-hd/ on all login and compute nodes.