NEMO2/Workspaces: Difference between revisions

From bwHPC Wiki
Jump to navigation Jump to search
mNo edit summary
mNo edit summary
 
(9 intermediate revisions by 3 users not shown)
Line 1: Line 1:
<div style="border: 3px solid #ffc107; padding: 15px; background-color: #fff3cd; margin: 10px 0;">
<div style="border: 3px solid #ffc107; padding: 15px; background-color: #fff3cd; margin: 10px 0;">
'''Note:''' This is a new Workspaces page. The old wiki can be found here: [[Workspace]].
'''Note:''' This is the updated Workspaces guide for NEMO2. For other clusters please use: [[Workspace]].
</div>
</div>


'''Workspace tools''' provide temporary storage on NEMO's fast parallel filesystem (Weka).
'''Workspace tools''' provide temporary scratch spaces called '''workspaces''' for your calculations on central file storage. They are meant to keep data for a limited time – but usually longer than the time of a single job run.
They are meant for data that needs to persist longer than a single job, but not permanently.


For advanced features — user config (<tt>~/.ws_user.conf</tt>), reminders, quotas, workspace handover, and more — see [[NEMO2/Workspaces/Advanced_Features|Advanced Features]].
== Important - Read Before Using Workspaces ==


== What are Workspaces? ==
* '''No Backup:''' Data in workspaces is '''not backed up''' and will be '''automatically deleted''' after expiration
* '''Time-limited:''' Every workspace has a limited lifetime (typically 30-100 days depending on cluster), see the [[Workspaces/Advanced_Features/Quotas#Cluster-Specific_Workspace_Limits|Cluster-Specific Workspace Limits]])
* '''Automatic Reminders:''' You will receive email notifications before expiration
* '''Backup Important Data:''' Copy important results to permanent storage before expiration (check your cluster/site policies for backup locations)


'''Use workspaces for:'''
'''For detailed information''' about advanced features, configuration options, and less frequently used commands, see the '''[[Workspaces/Advanced Features]]''' guide.
* Jobs generating intermediate data
* Data shared between multiple compute nodes
* Multi-step workflows

'''Don't use workspaces for:'''
* Permanent storage (use HOME or project directories)
* Single-node temporary files (use <tt>$TMPDIR</tt> instead)

== Important - Read First ==

* No Backup: Data is '''not backed up''' and will be '''automatically deleted''' after expiration
* Time-limited: Maximum lifetime is 100 days, up to 100 extensions
* Email Reminders: You receive email notifications before expiration
* Backup Important Data: Copy results to permanent storage before expiration


== Command Overview ==
== Command Overview ==


* <tt>ws_allocate</tt> - Create or extend workspace
The workspace tools consist of several commands:
* <tt>ws_list</tt> - List your workspaces
* <tt>ws_find</tt> - Find workspace path (for scripts)
* <tt>ws_extend</tt> - Extend workspace lifetime
* <tt>ws_release</tt> - Release (delete) workspace
* <tt>ws_restore</tt> - Restore expired/released workspace
* <tt>ws_register</tt> - Create symbolic links


All commands support <tt>-h</tt> for help.
* <tt>ws_allocate</tt> - Create or extend a workspace
* <tt>ws_list</tt> - List all your workspaces
* <tt>ws_find</tt> - Find the path to a workspace
* <tt>ws_extend</tt> - Extend the lifetime of a workspace
* <tt>ws_release</tt> - Release (delete) a workspace
* <tt>ws_restore</tt> - Restore an expired or released workspace
* <tt>ws_register</tt> - Create symbolic links to workspaces


== Quick Start ==
All commands support <tt>-h</tt> or <tt>--help</tt> to show detailed usage information.

== Quick Start - Most Common Commands ==


{| class="wikitable"
{| class="wikitable"
Line 35: Line 45:
!style="width:60%" | Command
!style="width:60%" | Command
|-
|-
|Create workspace for 30 days
|Create workspace (100 days)
|<tt>ws_allocate myWs 30</tt>
|<tt>ws_allocate myWs 100</tt>
|-
|-
|Create group-writable workspace
|Create group workspace
|<tt>ws_allocate -G groupname myWs 30</tt>
|<tt>ws_allocate -G groupname myWs 100</tt>
|-
|-
|List all your workspaces
|List all workspaces
|<tt>ws_list</tt>
|<tt>ws_list</tt>
|-
|-
|Check which expire soon
|See what expires soon
|<tt>ws_list -Rr</tt>
|<tt>ws_list -Rr</tt>
|-
|-
|Find workspace path (for scripts)
|Find path (for scripts)
|<tt>ws_find myWs</tt>
|<tt>ws_find myWs</tt>
|-
|-
|Extend workspace by 30 days
|Extend by 100 days
|<tt>ws_extend myWs 30</tt>
|<tt>ws_extend myWs 100</tt>
|-
|-
|Delete/release workspace
|Delete workspace (permanent, next nightly run)
|<tt>ws_release myWs</tt>
|<tt>ws_release myWs</tt>
|-
|-
|Restore released workspace
|Restore expired workspace (30d grace)
|<tt>ws_restore -l</tt> then <tt>ws_restore oldname newname</tt>
|<tt>ws_restore -l</tt> then <tt>ws_restore oldname newname</tt>
|}
|}


== Create Workspace ==
== Creating Workspaces ==


To create a workspace you need to specify a '''name''' and '''lifetime''' in days:
Create a workspace with a '''name''' and '''lifetime''' in days:


$ ws_allocate myWs 100
'''Basic usage:'''


Returns:
$ ws_allocate myWs 30 # Create workspace for 30 days


/work/classic/$USER-myWs
'''Common variations:'''


'''Capture path in variable:'''
$ ws_allocate -G groupname myWs 30 # Create group-writable workspace (recommended for teams)
$ ws_allocate -g myWs 30 # Create group-readable workspace
$ ws_allocate -F ffuc myWs 30 # bwUniCluster 3.0: Use flash filesystem


$ WORKSPACE=$(ws_allocate myWs 100)
This returns:
$ cd "$WORKSPACE"
Workspace created. Duration is 720 hours.
Further extensions available: 3
/work/workspace/scratch/username-myWs-0


'''Important:''' Creating a workspace a second time with the same command is safe - it always returns the same path.
'''Important:''' Running the same command again is safe - returns the existing workspace path.


== Listing Workspaces ==
'''Capture the path in a variable:'''


$ ws_list # List all workspaces
$ WORKSPACE=$(ws_allocate myWs 30)
$ ws_list -Rr # Sort by remaining time, soonest first
$ cd $WORKSPACE
$ ws_list -g # Show group workspaces


== Extending Workspaces ==
'''For all options and advanced usage,''' see the [[Workspaces/Advanced_Features/ws_allocate|Advanced Features guide]].


$ ws_extend myWs 100 # Extend by 100 days from now
== List Your Workspaces ==


'''Alternative:''' <tt>ws_allocate -x myWs 100</tt>
'''Basic usage:'''


Each extension consumes one of your available extensions (100 total).
$ ws_list # List all your workspaces


== Releasing Workspaces ==
Shows:
* Workspace ID
* Workspace location
* Available extensions
* Creation date and remaining time


$ ws_release myWs
'''Common variations:'''


The workspace becomes inaccessible immediately and is permanently deleted at the next nightly expirer run. '''Do not rely on recovering a released workspace.'''
$ ws_list -Rr # Sort by remaining time (see what expires soon)
$ ws_list -s # Short format (only names, good for scripts)
$ ws_list -g # List group workspaces


== Restoring Workspaces ==
'''Note:''' To list expired workspaces that can be restored, see [[#Restore_Workspace|Restore Workspace]].


Recover workspaces that '''expired naturally''' (reached end of lifetime) within the 30-day grace period:
'''For all options,''' see the [[Workspaces/Advanced_Features/ws_list|Advanced Features guide]].

== Extend Workspace Lifetime ==

Extend a workspace before it expires:

'''Basic usage:'''

$ ws_extend myWs 30 # Extend by 30 days from now

'''Alternative commands:'''

$ ws_allocate -x myWs 30 # Same as ws_extend

'''Note:''' Each extension consumes one of your available extensions (see the [[Workspaces/Advanced_Features/Quotas#Cluster-Specific_Workspace_Limits|Cluster-Specific Workspace Limits]]).

'''For all options,''' see the [[Workspaces/Advanced_Features/ws_extend|Advanced Features guide]].

== Release (Delete) Workspace ==

When you no longer need a workspace:

'''Basic usage:'''

$ ws_release myWs # Release workspace (recoverable during grace period)

'''What happens:'''
* Workspace becomes inaccessible immediately
* Data is kept for a short grace period (usually until next cleanup run) and can be restored with <tt>ws_restore</tt>
* Released data may still count toward quota until final deletion

'''For all options,''' see the [[Workspaces/Advanced_Features/ws_release|Advanced Features guide]].

== Restore Workspace ==

{| class="wikitable"
|-
!style="width:40%" | Works on cluster
!style="width:10%" | bwUC 3.0
!style="width:10%" | BinAC2
!style="width:10%" | Helix
!style="width:10%" | JUSTUS 2
!style="width:10%" | NEMO2
|-
|<tt>ws_restore</tt>
|style="background-color:#90EE90; text-align:center;" | ✓
|style="background-color:#90EE90; text-align:center;" | ✓
|style="background-color:#90EE90; text-align:center;" | ✓
|style="background-color:#90EE90; text-align:center;" | ✓
|style="background-color:#90EE90; text-align:center;" | ✓
|}

If you released a workspace by accident or need to recover an expired one, you can restore it within a grace period:

'''Basic workflow:'''


$ ws_restore -l # (1) List restorable workspaces
$ ws_restore -l # (1) List restorable workspaces
$ ws_allocate restored 60 # (2) Create a new target workspace
$ ws_allocate restored 100 # (2) Create target workspace
$ ws_restore username-myWs-0 restored # (3) Restore the expired workspace
$ ws_restore username-myWs-0 restored # (3) Restore


'''Note:''' Use the '''full name''' from <tt>ws_restore -l</tt> (including username and timestamp), not the short name from <tt>ws_list</tt>.
'''Important:''' Use the '''full name''' from <tt>ws_restore -l</tt> (with username and timestamp), not the short name.
Released workspaces (via <tt>ws_release</tt>) can also be restored, but only until the next nightly expirer run — after that they are permanently deleted.


== Sharing Workspaces ==
'''For detailed restore options,''' see the [[Workspaces/Advanced_Features/ws_restore|Advanced Features guide]].


== Work with Groups (Share Workspaces) ==
=== Group workspace (recommended) ===


$ ws_allocate -g myWs 100 # Group-readable (read-only for group)
{| class="wikitable"
$ ws_allocate -G projectgroup myWs 100 # Group-writable (recommended for teams)
|-
!style="width:40%" | Works on cluster
!style="width:10%" | bwUC 3.0
!style="width:10%" | BinAC2
!style="width:10%" | Helix
!style="width:10%" | JUSTUS 2
!style="width:10%" | NEMO2
|-
|<tt>-g</tt> option (group-readable)
| style="text-align:center;" |
| style="text-align:center;" |
| style="text-align:center;" |
| style="text-align:center;" |
|style="background-color:#90EE90; text-align:center;" | ✓
|-
|<tt>-G</tt> option (group-writable)
| style="text-align:center;" |
| style="text-align:center;" |
| style="text-align:center;" |
| style="text-align:center;" |
|style="background-color:#90EE90; text-align:center;" | ✓
|}


Anyone in the group can use <tt>ws_list -g</tt> to see the workspace and extend it with <tt>ws_allocate -x -u owner myWs 100</tt>.
Working with team members is simple using group workspaces:
Using <tt>-G</tt> also enables smooth handover when team members leave — see [[NEMO2/Workspaces/Advanced_Features#Workspace_Handover|Workspace Handover]].


'''Set default group in <tt>~/.ws_user.conf</tt>:'''
=== Create Group Workspace ===

'''Group-readable workspace''' (team can read):

$ ws_allocate -g myWs 30

'''Group-writable workspace''' (team can read and write, recommended):

$ ws_allocate -G projectgroup myWs 30

Replace <tt>projectgroup</tt> with your actual group name (e.g., <tt>bw11a000</tt>).

'''Tip:''' Set your default group in <tt>~/.ws_user.conf</tt> to avoid typing it every time:


<pre>
<pre>
Line 219: Line 134:
</pre>
</pre>


=== Share after creation ===
Then you only need: <tt>ws_allocate myWs 30</tt>

=== List Group Workspaces ===

See all workspaces from your group:

$ ws_list -g

This shows workspaces that were created with <tt>-g</tt> or <tt>-G</tt> by anyone in your group.

=== Extend Group Workspace ===

Anyone in the group can extend a group-writable workspace (<tt>-G</tt>):

$ ws_extend myWs 30 # If you created it
$ ws_allocate -x -u username myWs 30 # If colleague created it

Replace <tt>username</tt> with the workspace owner's username. This is useful when they're unavailable.

=== Manage Reminders for Group Workspace ===

You can update reminder settings and take over responsibility for reminders on a colleague's workspace:

$ ws_allocate -r 7 -u username -x myWs 0 # Update reminder time and take over
# another user's workspace reminders

This changes the reminder timing to 7 days before expiration and redirects reminder emails to you instead of the original creator. Useful when you're taking over responsibility for a shared workspace.


If you didn't use <tt>-g</tt>/<tt>-G</tt> at creation, share read-only with <tt>ws_share</tt>:
=== Why Use Group Workspaces? ===


$ ws_share share myWs alice bob # Grant read access
* Simple collaboration: Everyone can access the same data
$ ws_share list myWs # Show who has access
* No permission problems: Files automatically get group permissions
$ ws_share unshare myWs alice # Remove access
* Independent extensions: Team members can extend without original creator
* Easy to find: Use <tt>ws_list -g</tt> to see all team workspaces


'''For advanced sharing options''' (sharing with specific users outside your group, ACL-based methods), see the [[Workspaces/Advanced_Features/Sharing|Advanced Features guide]].
'''Advanced sharing:''' [[NEMO2/Workspaces/Advanced_Features#Sharing|Sharing guide]] for ACL-based per-user permissions.

Latest revision as of 17:37, 12 May 2026

Note: This is the updated Workspaces guide for NEMO2. For other clusters please use: Workspace.

Workspace tools provide temporary storage on NEMO's fast parallel filesystem (Weka). They are meant for data that needs to persist longer than a single job, but not permanently.

For advanced features — user config (~/.ws_user.conf), reminders, quotas, workspace handover, and more — see Advanced Features.

What are Workspaces?

Use workspaces for:

  • Jobs generating intermediate data
  • Data shared between multiple compute nodes
  • Multi-step workflows

Don't use workspaces for:

  • Permanent storage (use HOME or project directories)
  • Single-node temporary files (use $TMPDIR instead)

Important - Read First

  • No Backup: Data is not backed up and will be automatically deleted after expiration
  • Time-limited: Maximum lifetime is 100 days, up to 100 extensions
  • Email Reminders: You receive email notifications before expiration
  • Backup Important Data: Copy results to permanent storage before expiration

Command Overview

  • ws_allocate - Create or extend workspace
  • ws_list - List your workspaces
  • ws_find - Find workspace path (for scripts)
  • ws_extend - Extend workspace lifetime
  • ws_release - Release (delete) workspace
  • ws_restore - Restore expired/released workspace
  • ws_register - Create symbolic links

All commands support -h for help.

Quick Start

Task Command
Create workspace (100 days) ws_allocate myWs 100
Create group workspace ws_allocate -G groupname myWs 100
List all workspaces ws_list
See what expires soon ws_list -Rr
Find path (for scripts) ws_find myWs
Extend by 100 days ws_extend myWs 100
Delete workspace (permanent, next nightly run) ws_release myWs
Restore expired workspace (30d grace) ws_restore -l then ws_restore oldname newname

Creating Workspaces

Create a workspace with a name and lifetime in days: 

  $ ws_allocate myWs 100

Returns: 

  /work/classic/$USER-myWs

Capture path in variable: 

  $ WORKSPACE=$(ws_allocate myWs 100)
  $ cd "$WORKSPACE"

Important: Running the same command again is safe - returns the existing workspace path.

Listing Workspaces

  $ ws_list                                # List all workspaces
  $ ws_list -Rr                            # Sort by remaining time, soonest first
  $ ws_list -g                             # Show group workspaces

Extending Workspaces

  $ ws_extend myWs 100                      # Extend by 100 days from now

Alternative: ws_allocate -x myWs 100

Each extension consumes one of your available extensions (100 total).

Releasing Workspaces

  $ ws_release myWs

The workspace becomes inaccessible immediately and is permanently deleted at the next nightly expirer run. Do not rely on recovering a released workspace.

Restoring Workspaces

Recover workspaces that expired naturally (reached end of lifetime) within the 30-day grace period: 

  $ ws_restore -l                          # (1) List restorable workspaces
  $ ws_allocate restored 100               # (2) Create target workspace
  $ ws_restore username-myWs-0 restored    # (3) Restore

Important: Use the full name from ws_restore -l (with username and timestamp), not the short name. Released workspaces (via ws_release) can also be restored, but only until the next nightly expirer run — after that they are permanently deleted.

Sharing Workspaces

Group workspace (recommended)

  $ ws_allocate -g myWs 100                # Group-readable (read-only for group)
  $ ws_allocate -G projectgroup myWs 100   # Group-writable (recommended for teams)

Anyone in the group can use ws_list -g to see the workspace and extend it with ws_allocate -x -u owner myWs 100. Using -G also enables smooth handover when team members leave — see Workspace Handover.

Set default group in ~/.ws_user.conf: 

groupname: projectgroup

Share after creation

If you didn't use -g/-G at creation, share read-only with ws_share: 

  $ ws_share share myWs alice bob          # Grant read access
  $ ws_share list myWs                     # Show who has access
  $ ws_share unshare myWs alice            # Remove access

Advanced sharing: Sharing guide for ACL-based per-user permissions.

Retrieved from "https://wiki.bwhpc.de/wiki/index.php?title=NEMO2/Workspaces&oldid=16069"