NEMO2/Workspaces: Difference between revisions

From bwHPC Wiki
Jump to navigation Jump to search
mNo edit summary
mNo edit summary
 
(19 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{| style=" background:#FFD28A; width:100%;"
<div style="border: 3px solid #ffc107; padding: 15px; background-color: #fff3cd; margin: 10px 0;">
'''Note:''' This is the updated Workspaces guide for NEMO2. For other clusters please use: [[Workspace]].
| style="padding:8px; background:#FFC05C; font-size:120%; font-weight:bold; text-align:left" | New Workspace Page
</div>
|-
|
'''WARNING:''' This is a new Workspaces page, the old safe-to-use page can be found here: [[Workspace]].
|}


'''Workspace tools''' provide temporary storage on NEMO's fast parallel filesystem (Weka).
'''Workspace tools''' provide temporary scratch spaces called '''workspaces''' for your calculations on a central file storage. They are meant to keep data for a limited time – but usually longer than the time of a single job run.
They are meant for data that needs to persist longer than a single job, but not permanently.


For detailed information about advanced workspace features, configuration options, and less frequently used commands, see the separate '''[[Workspaces/Advanced Features]] guide'''.
For advanced features — user config (<tt>~/.ws_user.conf</tt>), reminders, quotas, workspace handover, and more — see [[NEMO2/Workspaces/Advanced_Features|Advanced Features]].


== Important ==
== What are Workspaces? ==


'''Use workspaces for:'''
* '''No Backup:''' Data in workspaces is '''not backed up''' and will be '''automatically deleted''' after expiration
* Jobs generating intermediate data
* '''Time-limited:''' Every workspace has a limited lifetime (typically 30-100 days depending on cluster, see the [[Workspaces/Advanced_Features#Cluster-Specific_Workspace_Limits|Cluster-Specific Workspace Limits]])
* Data shared between multiple compute nodes
* '''Automatic Email Reminders:''' You will receive email notifications before expiration
* Multi-step workflows
* '''Backup Important Data:''' Copy important results to appropriate permanent storage before expiration (location depends on your cluster/site policies)


'''Don't use workspaces for:'''
== Quick Start - Most Common Commands ==
* Permanent storage (use HOME or project directories)
* Single-node temporary files (use <tt>$TMPDIR</tt> instead)


== Important - Read First ==
The workspace tools consist of several commands:


* No Backup: Data is '''not backed up''' and will be '''automatically deleted''' after expiration
* <tt>ws_allocate</tt> - Create or extend a workspace
* Time-limited: Maximum lifetime is 100 days, up to 100 extensions
* <tt>ws_list</tt> - List all your workspaces
* Email Reminders: You receive email notifications before expiration
* <tt>ws_find</tt> - Find the path to a workspace
* Backup Important Data: Copy results to permanent storage before expiration
* <tt>ws_extend</tt> - Extend the lifetime of a workspace
* <tt>ws_release</tt> - Release (delete) a workspace
* <tt>ws_restore</tt> - Restore an expired or released workspace
* <tt>ws_register</tt> - Create symbolic links to workspaces


== Command Overview ==
All commands support <tt>-h</tt> or <tt>--help</tt> to show detailed usage information.

* <tt>ws_allocate</tt> - Create or extend workspace
* <tt>ws_list</tt> - List your workspaces
* <tt>ws_find</tt> - Find workspace path (for scripts)
* <tt>ws_extend</tt> - Extend workspace lifetime
* <tt>ws_release</tt> - Release (delete) workspace
* <tt>ws_restore</tt> - Restore expired/released workspace
* <tt>ws_register</tt> - Create symbolic links

All commands support <tt>-h</tt> for help.

== Quick Start ==


{| class="wikitable"
{| class="wikitable"
Line 36: Line 45:
!style="width:60%" | Command
!style="width:60%" | Command
|-
|-
|Create workspace for 30 days
|Create workspace (100 days)
|<tt>ws_allocate myWs 30</tt>
|<tt>ws_allocate myWs 100</tt>
|-
|-
|Create group-writable workspace
|Create group workspace
|<tt>ws_allocate -G groupname myWs 30</tt>
|<tt>ws_allocate -G groupname myWs 100</tt>
|-
|-
|List all your workspaces
|List all workspaces
|<tt>ws_list</tt>
|<tt>ws_list</tt>
|-
|-
|See what expires soon
|Find workspace path (for scripts)
|<tt>ws_list -Rr</tt>
|-
|Find path (for scripts)
|<tt>ws_find myWs</tt>
|<tt>ws_find myWs</tt>
|-
|-
|Extend by 100 days
|Check which expire soon
|<tt>ws_list -R</tt>
|<tt>ws_extend myWs 100</tt>
|-
|-
|Extend workspace by 30 days
|Delete workspace (permanent, next nightly run)
|<tt>ws_extend myWs 30</tt>
|-
|Delete/release workspace
|<tt>ws_release myWs</tt>
|<tt>ws_release myWs</tt>
|-
|-
|Restore released workspace
|Restore expired workspace (30d grace)
|<tt>ws_restore -l</tt> then <tt>ws_restore oldname newname</tt>
|<tt>ws_restore -l</tt> then <tt>ws_restore oldname newname</tt>
|}
|}


== Create Workspace ==
== Creating Workspaces ==


To create a workspace you need to specify a '''name''' and '''lifetime''' in days:
Create a workspace with a '''name''' and '''lifetime''' in days:


$ ws_allocate myWs 100
'''Basic usage:'''


Returns:
$ ws_allocate myWs 30 # Create workspace for 30 days


/work/classic/$USER-myWs
'''Common variations:'''


'''Capture path in variable:'''
$ ws_allocate -G groupname myWs 30 # Create group-writable workspace (recommended for teams)
$ ws_allocate -g myWs 30 # Create group-readable workspace
$ ws_allocate -F ffuc myWs 30 # bwUniCluster 3.0: Use flash filesystem for better performance


$ WORKSPACE=$(ws_allocate myWs 100)
This returns:
$ cd "$WORKSPACE"
Workspace created. Duration is 720 hours.
Further extensions available: 3
/work/workspace/scratch/username-myWs-0


'''Important:''' Creating a workspace a second time with the same command is safe - it always returns the same path.
'''Important:''' Running the same command again is safe - returns the existing workspace path.


== Listing Workspaces ==
'''Capture the path in a variable:'''


$ ws_list # List all workspaces
$ WORKSPACE=$(ws_allocate myWs 30)
$ ws_list -Rr # Sort by remaining time, soonest first
$ cd $WORKSPACE
$ ws_list -g # Show group workspaces


== Extending Workspaces ==
'''For all options and advanced usage,''' see the [[Workspaces/Advanced_Features#ws_allocate_-_Create_and_Extend_Workspaces|Advanced Features guide]].


$ ws_extend myWs 100 # Extend by 100 days from now
== List Your Workspaces ==


'''Alternative:''' <tt>ws_allocate -x myWs 100</tt>
'''Basic usage:'''


Each extension consumes one of your available extensions (100 total).
$ ws_list # List all your workspaces


== Releasing Workspaces ==
Shows:
* Workspace ID
* Workspace location
* Available extensions
* Creation date and remaining time


$ ws_release myWs
'''Common variations:'''


The workspace becomes inaccessible immediately and is permanently deleted at the next nightly expirer run. '''Do not rely on recovering a released workspace.'''
$ ws_list -R # Sort by remaining time (see what expires soon)
$ ws_list -s # Short format (only names, good for scripts)
$ ws_list -g # List group workspaces (if you're in the same group)


== Restoring Workspaces ==
'''Note:''' To list expired workspaces that can be restored, see [[#Restore_Workspace|Restore Workspace]].


Recover workspaces that '''expired naturally''' (reached end of lifetime) within the 30-day grace period:
'''For all options,''' see the [[Workspaces/Advanced_Features#ws_list_-_List_Workspaces|Advanced Features guide]].

== Extend Workspace Lifetime ==

Extend a workspace before it expires:

'''Basic usage:'''

$ ws_extend myWs 30 # Extend by 30 days from now

'''Alternative commands:'''

$ ws_allocate -x myWs 30 # Same as ws_extend

'''Update reminder only (without extending):'''

$ ws_allocate -r 7 -x myWs 0 # Update reminder time to 7 days

'''Note:''' Each extension consumes one of your available extensions (see the [[Workspaces/Advanced_Features#Cluster-Specific_Workspace_Limits|Cluster-Specific Workspace Limits]]).

'''For all options,''' see the [[Workspaces/Advanced_Features#ws_extend_-_Extend_Workspace_Lifetime|Advanced Features guide]].

== Release (Delete) Workspace ==

When you no longer need a workspace:

'''Basic usage:'''

$ ws_release myWs # Release workspace (recoverable during grace period)

'''What happens:'''
* Workspace becomes inaccessible immediately
* Data is kept for a short grace period (usually until next cleanup run at nighttime) and can be restored with <tt>ws_restore</tt>
* Released data may still count toward quota until final deletion

{| class="wikitable"
|-
!style="width:40%" | Works on cluster
!style="width:10%" | bwUC 3.0
!style="width:10%" | BinAC2
!style="width:10%" | Helix
!style="width:10%" | JUSTUS 2
!style="width:10%" | NEMO2
|-
|<tt>ws_release --delete-data</tt> (immediate deletion)
|style="background-color:#90EE90; text-align:center;" | ✓
| style="text-align:center;" |
| style="text-align:center;" |
| style="text-align:center;" |
|style="background-color:#90EE90; text-align:center;" | ✓
|}

'''If you need to free quota immediately:'''

$ ws_release --delete-data myWs # WARNING: Permanently deletes data, cannot be recovered!

'''IMPORTANT:''' Only use immediate deletion if you're absolutely certain you don't need the data and have verified any backups. Regular <tt>ws_release</tt> is safer and recommended for most cases.

'''For all options,''' see the [[Workspaces/Advanced_Features#ws_release_-_Release_.28Delete.29_Workspace|Advanced Features guide]].

== Restore Workspace ==

{| class="wikitable"
|-
!style="width:40%" | Works on cluster
!style="width:10%" | bwUC 3.0
!style="width:10%" | BinAC2
!style="width:10%" | Helix
!style="width:10%" | JUSTUS 2
!style="width:10%" | NEMO2
|-
|<tt>ws_restore</tt>
|style="background-color:#90EE90; text-align:center;" | ✓
|style="background-color:#90EE90; text-align:center;" | ✓
|style="background-color:#90EE90; text-align:center;" | ✓
|style="background-color:#90EE90; text-align:center;" | ✓
|style="background-color:#90EE90; text-align:center;" | ✓
|}

If you released a workspace by accident or need to recover an expired one, you can restore it within a grace period:

'''Basic workflow:'''


$ ws_restore -l # (1) List restorable workspaces
$ ws_restore -l # (1) List restorable workspaces
$ ws_allocate restored 60 # (2) Create a new target workspace
$ ws_allocate restored 100 # (2) Create target workspace
$ ws_restore username-myWs-0 restored # (3) Restore the expired workspace
$ ws_restore username-myWs-0 restored # (3) Restore


'''Note:''' Use the '''full name''' from <tt>ws_restore -l</tt> (including username and timestamp), not the short name from <tt>ws_list</tt>.
'''Important:''' Use the '''full name''' from <tt>ws_restore -l</tt> (with username and timestamp), not the short name.
Released workspaces (via <tt>ws_release</tt>) can also be restored, but only until the next nightly expirer run — after that they are permanently deleted.


== Sharing Workspaces ==
'''For detailed restore options,''' see the [[Workspaces/Advanced_Features#ws_restore_-_Restore_Expired_Workspace|Advanced Features guide]].


== Work with Groups (Share Workspaces) ==
=== Group workspace (recommended) ===


$ ws_allocate -g myWs 100 # Group-readable (read-only for group)
{| class="wikitable"
$ ws_allocate -G projectgroup myWs 100 # Group-writable (recommended for teams)
|-
!style="width:40%" | Works on cluster
!style="width:10%" | bwUC 3.0
!style="width:10%" | BinAC2
!style="width:10%" | Helix
!style="width:10%" | JUSTUS 2
!style="width:10%" | NEMO2
|-
|<tt>-g</tt> option (group-readable)
| style="text-align:center;" |
| style="text-align:center;" |
| style="text-align:center;" |
| style="text-align:center;" |
|style="background-color:#90EE90; text-align:center;" | ✓
|-
|<tt>-G</tt> option (group-writable)
| style="text-align:center;" |
| style="text-align:center;" |
| style="text-align:center;" |
| style="text-align:center;" |
|style="background-color:#90EE90; text-align:center;" | ✓
|}


Anyone in the group can use <tt>ws_list -g</tt> to see the workspace and extend it with <tt>ws_allocate -x -u owner myWs 100</tt>.
Working with team members is simple using group workspaces:
Using <tt>-G</tt> also enables smooth handover when team members leave — see [[NEMO2/Workspaces/Advanced_Features#Workspace_Handover|Workspace Handover]].


'''Set default group in <tt>~/.ws_user.conf</tt>:'''
=== Create Group Workspace ===

'''Group-readable workspace''' (team can read):

$ ws_allocate -g myWs 30

'''Group-writable workspace''' (team can read and write, recommended):

$ ws_allocate -G projectgroup myWs 30

Replace <tt>projectgroup</tt> with your actual group name (e.g., <tt>bw11a000</tt>).

'''Tip:''' Set your default group in <tt>~/.ws_user.conf</tt> to avoid typing it every time:


<pre>
<pre>
Line 247: Line 134:
</pre>
</pre>


=== Share after creation ===
Then you only need: <tt>ws_allocate myWs 30</tt>

=== List Group Workspaces ===

See all workspaces from your group:

$ ws_list -g

This shows workspaces that were created with <tt>-g</tt> or <tt>-G</tt> by anyone in your group.

=== Extend Group Workspace ===

Anyone in the group can extend a group-writable workspace (<tt>-G</tt>):

$ ws_extend myWs 30 # If you created it
$ ws_allocate -x -u username myWs 30 # If colleague created it

Replace <tt>username</tt> with the workspace owner's username. This is useful when they're unavailable.

=== Manage Reminders for Group Workspace ===

You can update reminder settings and take over responsibility for reminders on a colleague's workspace:

$ ws_allocate -r 7 -u username -x myWs 0 # Update reminder time and take over
# another user's workspace reminders

This changes the reminder timing to 7 days before expiration and redirects reminder emails to you instead of the original creator. Useful when you're taking over responsibility for a shared workspace.


If you didn't use <tt>-g</tt>/<tt>-G</tt> at creation, share read-only with <tt>ws_share</tt>:
=== Why Use Group Workspaces? ===


$ ws_share share myWs alice bob # Grant read access
* '''Simple collaboration:''' Everyone can access the same data
$ ws_share list myWs # Show who has access
* '''No permission problems:''' Files automatically get group permissions
$ ws_share unshare myWs alice # Remove access
* '''Independent extensions:''' Team members can extend without original creator
* '''Easy to find:''' Use <tt>ws_list -g</tt> to see all team workspaces


'''For advanced sharing options''' (sharing with specific users outside your group, ACL-based methods), see the [[Workspaces/Advanced_Features#Cooperative_Usage_.28Group_Workspaces_and_Sharing.29|Advanced Features guide]].
'''Advanced sharing:''' [[NEMO2/Workspaces/Advanced_Features#Sharing|Sharing guide]] for ACL-based per-user permissions.

Latest revision as of 17:37, 12 May 2026

Note: This is the updated Workspaces guide for NEMO2. For other clusters please use: Workspace.

Workspace tools provide temporary storage on NEMO's fast parallel filesystem (Weka). They are meant for data that needs to persist longer than a single job, but not permanently.

For advanced features — user config (~/.ws_user.conf), reminders, quotas, workspace handover, and more — see Advanced Features.

What are Workspaces?

Use workspaces for:

  • Jobs generating intermediate data
  • Data shared between multiple compute nodes
  • Multi-step workflows

Don't use workspaces for:

  • Permanent storage (use HOME or project directories)
  • Single-node temporary files (use $TMPDIR instead)

Important - Read First

  • No Backup: Data is not backed up and will be automatically deleted after expiration
  • Time-limited: Maximum lifetime is 100 days, up to 100 extensions
  • Email Reminders: You receive email notifications before expiration
  • Backup Important Data: Copy results to permanent storage before expiration

Command Overview

  • ws_allocate - Create or extend workspace
  • ws_list - List your workspaces
  • ws_find - Find workspace path (for scripts)
  • ws_extend - Extend workspace lifetime
  • ws_release - Release (delete) workspace
  • ws_restore - Restore expired/released workspace
  • ws_register - Create symbolic links

All commands support -h for help.

Quick Start

Task Command
Create workspace (100 days) ws_allocate myWs 100
Create group workspace ws_allocate -G groupname myWs 100
List all workspaces ws_list
See what expires soon ws_list -Rr
Find path (for scripts) ws_find myWs
Extend by 100 days ws_extend myWs 100
Delete workspace (permanent, next nightly run) ws_release myWs
Restore expired workspace (30d grace) ws_restore -l then ws_restore oldname newname

Creating Workspaces

Create a workspace with a name and lifetime in days: 

  $ ws_allocate myWs 100

Returns: 

  /work/classic/$USER-myWs

Capture path in variable: 

  $ WORKSPACE=$(ws_allocate myWs 100)
  $ cd "$WORKSPACE"

Important: Running the same command again is safe - returns the existing workspace path.

Listing Workspaces

  $ ws_list                                # List all workspaces
  $ ws_list -Rr                            # Sort by remaining time, soonest first
  $ ws_list -g                             # Show group workspaces

Extending Workspaces

  $ ws_extend myWs 100                      # Extend by 100 days from now

Alternative: ws_allocate -x myWs 100

Each extension consumes one of your available extensions (100 total).

Releasing Workspaces

  $ ws_release myWs

The workspace becomes inaccessible immediately and is permanently deleted at the next nightly expirer run. Do not rely on recovering a released workspace.

Restoring Workspaces

Recover workspaces that expired naturally (reached end of lifetime) within the 30-day grace period: 

  $ ws_restore -l                          # (1) List restorable workspaces
  $ ws_allocate restored 100               # (2) Create target workspace
  $ ws_restore username-myWs-0 restored    # (3) Restore

Important: Use the full name from ws_restore -l (with username and timestamp), not the short name. Released workspaces (via ws_release) can also be restored, but only until the next nightly expirer run — after that they are permanently deleted.

Sharing Workspaces

Group workspace (recommended)

  $ ws_allocate -g myWs 100                # Group-readable (read-only for group)
  $ ws_allocate -G projectgroup myWs 100   # Group-writable (recommended for teams)

Anyone in the group can use ws_list -g to see the workspace and extend it with ws_allocate -x -u owner myWs 100. Using -G also enables smooth handover when team members leave — see Workspace Handover.

Set default group in ~/.ws_user.conf: 

groupname: projectgroup

Share after creation

If you didn't use -g/-G at creation, share read-only with ws_share: 

  $ ws_share share myWs alice bob          # Grant read access
  $ ws_share list myWs                     # Show who has access
  $ ws_share unshare myWs alice            # Remove access

Advanced sharing: Sharing guide for ACL-based per-user permissions.

Retrieved from "https://wiki.bwhpc.de/wiki/index.php?title=NEMO2/Workspaces&oldid=16069"