NEMO2/Workspaces: Difference between revisions

From bwHPC Wiki
Jump to navigation Jump to search
mNo edit summary
 
(7 intermediate revisions by 3 users not shown)
Line 1: Line 1:
<div style="border: 3px solid #ffc107; padding: 15px; background-color: #fff3cd; margin: 10px 0;">
<div style="border: 3px solid #ffc107; padding: 15px; background-color: #fff3cd; margin: 10px 0;">
'''Note:''' This is the updated Workspaces guide. The old wiki: [[Workspace]].
'''Note:''' This is the updated Workspaces guide for NEMO2. For other clusters please use: [[Workspace]].
</div>
</div>


'''Workspace tools''' provide temporary storage spaces called '''workspaces''' for your calculations. They are meant for data that needs to persist longer than a single job, but not permanently.
'''Workspace tools''' provide temporary storage on NEMO's fast parallel filesystem (Weka).
They are meant for data that needs to persist longer than a single job, but not permanently.


For advanced features — user config (<tt>~/.ws_user.conf</tt>), reminders, quotas, workspace handover, and more — see [[NEMO2/Workspaces/Advanced_Features|Advanced Features]].
== What are Workspaces? ==


== What are Workspaces? ==
Workspaces give you access to the cluster's fast parallel filesystems (like Lustre or Weka). '''You cannot write directly to these parallel filesystems''' - workspaces provide your designated area.


'''Use workspaces for:'''
'''Use workspaces for:'''
Line 13: Line 14:
* Data shared between multiple compute nodes
* Data shared between multiple compute nodes
* Multi-step workflows
* Multi-step workflows
* Temporary scratch space during calculations


'''Don't use workspaces for:'''
'''Don't use workspaces for:'''
Line 21: Line 21:
== Important - Read First ==
== Important - Read First ==


* '''No Backup:''' Data is '''not backed up''' and will be '''automatically deleted''' after expiration
* No Backup: Data is '''not backed up''' and will be '''automatically deleted''' after expiration
* Time-limited: Maximum lifetime is 100 days, up to 100 extensions
* '''Time-limited:''' Lifetime typically 30-100 days (cluster-specific). See [[Workspaces/Advanced_Features/Quotas#Cluster-Specific_Workspace_Limits|Cluster Limits]]
* '''Automatic Reminders:''' Email notifications before expiration
* Email Reminders: You receive email notifications before expiration
* '''Backup Important Data:''' Copy results to permanent storage before expiration
* Backup Important Data: Copy results to permanent storage before expiration

'''For advanced features and detailed options:''' [[Workspaces/Advanced Features]]


== Command Overview ==
== Command Overview ==

Main commands:


* <tt>ws_allocate</tt> - Create or extend workspace
* <tt>ws_allocate</tt> - Create or extend workspace
Line 49: Line 45:
!style="width:60%" | Command
!style="width:60%" | Command
|-
|-
|Create workspace (30 days)
|Create workspace (100 days)
|<tt>ws_allocate myWs 30</tt>
|<tt>ws_allocate myWs 100</tt>
|-
|-
|Create group workspace
|Create group workspace
|<tt>ws_allocate -G groupname myWs 30</tt>
|<tt>ws_allocate -G groupname myWs 100</tt>
|-
|-
|List all workspaces
|List all workspaces
Line 59: Line 55:
|-
|-
|See what expires soon
|See what expires soon
|<tt>ws_list -Rr</tt> (<tt>-R</tt>=by time, <tt>-r</tt>=reverse)
|<tt>ws_list -Rr</tt>
|-
|-
|Find path (for scripts)
|Find path (for scripts)
|<tt>ws_find myWs</tt>
|<tt>ws_find myWs</tt>
|-
|-
|Extend by 30 days
|Extend by 100 days
|<tt>ws_extend myWs 30</tt>
|<tt>ws_extend myWs 100</tt>
|-
|-
|Delete workspace
|Delete workspace (permanent, next nightly run)
|<tt>ws_release myWs</tt>
|<tt>ws_release myWs</tt>
|-
|-
|Restore workspace
|Restore expired workspace (30d grace)
|<tt>ws_restore -l</tt> then <tt>ws_restore oldname newname</tt>
|<tt>ws_restore -l</tt> then <tt>ws_restore oldname newname</tt>
|}
|}


== Create Workspace ==
== Creating Workspaces ==


Create a workspace with a '''name''' and '''lifetime''' in days:
Create a workspace with a '''name''' and '''lifetime''' in days:


$ ws_allocate myWs 30 # Create for 30 days
$ ws_allocate myWs 100


Returns:
Returns:
Workspace created. Duration is 720 hours.
Further extensions available: 3
/work/workspace/scratch/username-myWs-0


/work/classic/$USER-myWs
'''Common options:'''

$ ws_allocate -G groupname myWs 30 # Group-writable (for teams)
$ ws_allocate -g myWs 30 # Group-readable
$ ws_allocate -F ffuc myWs 30 # bwUniCluster 3.0: Flash filesystem


'''Capture path in variable:'''
'''Capture path in variable:'''


$ WORKSPACE=$(ws_allocate myWs 30)
$ WORKSPACE=$(ws_allocate myWs 100)
$ cd "$WORKSPACE"
$ cd "$WORKSPACE"


'''Important:''' Running the same command again is safe - returns the existing workspace path.
'''Important:''' Running the same command again is safe - returns the existing workspace path.


== Listing Workspaces ==
'''Details:''' [[Workspaces/Advanced_Features/ws_allocate|Advanced Features guide]]

== List Your Workspaces ==


$ ws_list # List all workspaces
$ ws_list # List all workspaces
$ ws_list -Rr # Sort by remaining time, soonest first

Shows workspace ID, location, extensions, creation date, and remaining time.

'''Common options:'''

$ ws_list -Rr # Sort by remaining time, reverse (last to expire first)
$ ws_list -s # Short format (names only, for scripts)
$ ws_list -g # Show group workspaces
$ ws_list -g # Show group workspaces


== Extending Workspaces ==
'''Note:''' To list expired workspaces for restore, use <tt>ws_restore -l</tt>. See [[#Restore_Workspace|Restore]].


$ ws_extend myWs 100 # Extend by 100 days from now
'''Details:''' [[Workspaces/Advanced_Features/ws_list|Advanced Features guide]]


'''Alternative:''' <tt>ws_allocate -x myWs 100</tt>
== Extend Workspace Lifetime ==


Each extension consumes one of your available extensions (100 total).
Extend before workspace expires:


== Releasing Workspaces ==
$ ws_extend myWs 30 # Extend by 30 days from now


$ ws_release myWs
'''Alternative:'''


The workspace becomes inaccessible immediately and is permanently deleted at the next nightly expirer run. '''Do not rely on recovering a released workspace.'''
$ ws_allocate -x myWs 30 # Same result
$ ws_allocate -x -u alice projectWs 30 # Extend Alice's workspace by 30 days
# For group workspaces created with <tt>-g</tt>/<tt>-G</tt>


== Restoring Workspaces ==
'''Note:''' Each extension consumes one available extension. See [[Workspaces/Advanced_Features/Quotas#Cluster-Specific_Workspace_Limits|Cluster Limits]].


Recover workspaces that '''expired naturally''' (reached end of lifetime) within the 30-day grace period:
'''Details:''' [[Workspaces/Advanced_Features/ws_extend|Advanced Features guide]]

== Release (Delete) Workspace ==

When no longer needed:

$ ws_release myWs # Release workspace

'''What happens:'''
* Workspace becomes inaccessible immediately
* Data kept for short grace period (typically until next cleanup)
* Can be restored with <tt>ws_restore</tt> during grace period
* May still count toward quota until final deletion

'''Details:''' [[Workspaces/Advanced_Features/ws_release|Advanced Features guide]]

== Restore Workspace ==

{| class="wikitable"
|-
!style="width:40%" | Works on cluster
!style="width:10%" | bwUC 3.0
!style="width:10%" | BinAC2
!style="width:10%" | Helix
!style="width:10%" | JUSTUS 2
!style="width:10%" | NEMO2
|-
|<tt>ws_restore</tt>
|style="background-color:#90EE90; text-align:center;" | ✓
|style="background-color:#90EE90; text-align:center;" | ✓
|style="background-color:#90EE90; text-align:center;" | ✓
|style="background-color:#90EE90; text-align:center;" | ✓
|style="background-color:#90EE90; text-align:center;" | ✓
|}

Recover expired or released workspaces within grace period:

'''Restore procedure:'''


$ ws_restore -l # (1) List restorable workspaces
$ ws_restore -l # (1) List restorable workspaces
$ ws_allocate restored 60 # (2) Create target workspace
$ ws_allocate restored 100 # (2) Create target workspace
$ ws_restore username-myWs-0 restored # (3) Restore
$ ws_restore username-myWs-0 restored # (3) Restore


'''Important:''' Use the '''full name''' from <tt>ws_restore -l</tt> (with username and timestamp), not the short name.
'''Important:''' Use the '''full name''' from <tt>ws_restore -l</tt> (with username and timestamp), not the short name.
Released workspaces (via <tt>ws_release</tt>) can also be restored, but only until the next nightly expirer run — after that they are permanently deleted.


== Sharing Workspaces ==
'''Details:''' [[Workspaces/Advanced_Features/ws_restore|Advanced Features guide]]


== Work with Groups (Share Workspaces) ==
=== Group workspace (recommended) ===


$ ws_allocate -g myWs 100 # Group-readable (read-only for group)
{| class="wikitable"
$ ws_allocate -G projectgroup myWs 100 # Group-writable (recommended for teams)
|-
!style="width:40%" | Works on cluster
!style="width:10%" | bwUC 3.0
!style="width:10%" | BinAC2
!style="width:10%" | Helix
!style="width:10%" | JUSTUS 2
!style="width:10%" | NEMO2
|-
|<tt>-g</tt> (group-readable)
| style="text-align:center;" |
| style="text-align:center;" |
| style="text-align:center;" |
| style="text-align:center;" |
|style="background-color:#90EE90; text-align:center;" | ✓
|-
|<tt>-G</tt> (group-writable)
| style="text-align:center;" |
| style="text-align:center;" |
| style="text-align:center;" |
| style="text-align:center;" |
|style="background-color:#90EE90; text-align:center;" | ✓
|}


Anyone in the group can use <tt>ws_list -g</tt> to see the workspace and extend it with <tt>ws_allocate -x -u owner myWs 100</tt>.
Simple team collaboration with group workspaces:
Using <tt>-G</tt> also enables smooth handover when team members leave — see [[NEMO2/Workspaces/Advanced_Features#Workspace_Handover|Workspace Handover]].


'''Set default group in <tt>~/.ws_user.conf</tt>:'''
=== Create Group Workspace ===

'''Group-readable (read-only):'''

$ ws_allocate -g myWs 30

'''Group-writable (recommended):'''

$ ws_allocate -G projectgroup myWs 30 # Replace 'projectgroup' with your group

'''Tip:''' Set default in <tt>~/.ws_user.conf</tt>:


<pre>
<pre>
Line 222: Line 134:
</pre>
</pre>


=== Share after creation ===
Then simply: <tt>ws_allocate myWs 30</tt>

=== List Group Workspaces ===

$ ws_list -g # Show all group workspaces

=== Extend Group Workspace ===

Anyone in the group can extend group-writable workspaces (<tt>-G</tt>):

$ ws_extend myWs 30 # If you created it
$ ws_allocate -x -u alice myWs 30 # If colleague created it

=== Manage Reminders ===

Take over reminder responsibility:

$ ws_allocate -r 7 -u alice -x myWs 0 # Update timing and take over reminders

Changes reminder to 7 days before expiration and redirects emails to you.


If you didn't use <tt>-g</tt>/<tt>-G</tt> at creation, share read-only with <tt>ws_share</tt>:
=== Why Use Group Workspaces? ===


$ ws_share share myWs alice bob # Grant read access
* Simple collaboration - everyone accesses same data
$ ws_share list myWs # Show who has access
* No permission problems - automatic group permissions
$ ws_share unshare myWs alice # Remove access
* Independent extensions - team can extend without creator
* Easy discovery - <tt>ws_list -g</tt> shows all team workspaces


'''Advanced sharing:''' [[Workspaces/Advanced_Features/Sharing|Sharing guide]] for ACLs and ws_share
'''Advanced sharing:''' [[NEMO2/Workspaces/Advanced_Features#Sharing|Sharing guide]] for ACL-based per-user permissions.

Latest revision as of 17:37, 12 May 2026

Note: This is the updated Workspaces guide for NEMO2. For other clusters please use: Workspace.

Workspace tools provide temporary storage on NEMO's fast parallel filesystem (Weka). They are meant for data that needs to persist longer than a single job, but not permanently.

For advanced features — user config (~/.ws_user.conf), reminders, quotas, workspace handover, and more — see Advanced Features.

What are Workspaces?

Use workspaces for:

  • Jobs generating intermediate data
  • Data shared between multiple compute nodes
  • Multi-step workflows

Don't use workspaces for:

  • Permanent storage (use HOME or project directories)
  • Single-node temporary files (use $TMPDIR instead)

Important - Read First

  • No Backup: Data is not backed up and will be automatically deleted after expiration
  • Time-limited: Maximum lifetime is 100 days, up to 100 extensions
  • Email Reminders: You receive email notifications before expiration
  • Backup Important Data: Copy results to permanent storage before expiration

Command Overview

  • ws_allocate - Create or extend workspace
  • ws_list - List your workspaces
  • ws_find - Find workspace path (for scripts)
  • ws_extend - Extend workspace lifetime
  • ws_release - Release (delete) workspace
  • ws_restore - Restore expired/released workspace
  • ws_register - Create symbolic links

All commands support -h for help.

Quick Start

Task Command
Create workspace (100 days) ws_allocate myWs 100
Create group workspace ws_allocate -G groupname myWs 100
List all workspaces ws_list
See what expires soon ws_list -Rr
Find path (for scripts) ws_find myWs
Extend by 100 days ws_extend myWs 100
Delete workspace (permanent, next nightly run) ws_release myWs
Restore expired workspace (30d grace) ws_restore -l then ws_restore oldname newname

Creating Workspaces

Create a workspace with a name and lifetime in days: 

  $ ws_allocate myWs 100

Returns: 

  /work/classic/$USER-myWs

Capture path in variable: 

  $ WORKSPACE=$(ws_allocate myWs 100)
  $ cd "$WORKSPACE"

Important: Running the same command again is safe - returns the existing workspace path.

Listing Workspaces

  $ ws_list                                # List all workspaces
  $ ws_list -Rr                            # Sort by remaining time, soonest first
  $ ws_list -g                             # Show group workspaces

Extending Workspaces

  $ ws_extend myWs 100                      # Extend by 100 days from now

Alternative: ws_allocate -x myWs 100

Each extension consumes one of your available extensions (100 total).

Releasing Workspaces

  $ ws_release myWs

The workspace becomes inaccessible immediately and is permanently deleted at the next nightly expirer run. Do not rely on recovering a released workspace.

Restoring Workspaces

Recover workspaces that expired naturally (reached end of lifetime) within the 30-day grace period: 

  $ ws_restore -l                          # (1) List restorable workspaces
  $ ws_allocate restored 100               # (2) Create target workspace
  $ ws_restore username-myWs-0 restored    # (3) Restore

Important: Use the full name from ws_restore -l (with username and timestamp), not the short name. Released workspaces (via ws_release) can also be restored, but only until the next nightly expirer run — after that they are permanently deleted.

Sharing Workspaces

Group workspace (recommended)

  $ ws_allocate -g myWs 100                # Group-readable (read-only for group)
  $ ws_allocate -G projectgroup myWs 100   # Group-writable (recommended for teams)

Anyone in the group can use ws_list -g to see the workspace and extend it with ws_allocate -x -u owner myWs 100. Using -G also enables smooth handover when team members leave — see Workspace Handover.

Set default group in ~/.ws_user.conf: 

groupname: projectgroup

Share after creation

If you didn't use -g/-G at creation, share read-only with ws_share: 

  $ ws_share share myWs alice bob          # Grant read access
  $ ws_share list myWs                     # Show who has access
  $ ws_share unshare myWs alice            # Remove access

Advanced sharing: Sharing guide for ACL-based per-user permissions.

Retrieved from "https://wiki.bwhpc.de/wiki/index.php?title=NEMO2/Workspaces&oldid=16069"