Workspaces/Advanced Features: Difference between revisions

From bwHPC Wiki
Jump to navigation Jump to search
mNo edit summary
Tag: Reverted
mNo edit summary
 
(5 intermediate revisions by the same user not shown)
Line 1: Line 1:
This document covers advanced features and detailed configuration options for the HPC workspace tools. For basic daily usage, see the main [[Workspaces]] guide.
<div style="border: 3px solid #ffc107; padding: 15px; background-color: #fff3cd; margin: 10px 0;">
'''Note:''' This is a new Workspaces page. The old wiki can be found here: [[Workspace]].
</div>


== Almost Complete Command Reference ==
'''Workspace tools''' provide temporary scratch spaces called '''workspaces''' for your calculations on central file storage. They are meant to keep data for a limited time – but usually longer than the time of a single job run.

<div style="border: 3px solid #dc3545; padding: 15px; background-color: #f8d7da; margin: 10px 0;">
'''Important - Read Before Using Workspaces:'''

* '''No Backup:''' Data in workspaces is '''not backed up''' and will be '''automatically deleted''' after expiration
* '''Time-limited:''' Every workspace has a limited lifetime (typically 30-100 days depending on cluster), see the [[Workspaces/Advanced_Features#Cluster-Specific_Workspace_Limits|Cluster-Specific Workspace Limits]])
* '''Automatic Reminders:''' You will receive email notifications before expiration
* '''Backup Important Data:''' Copy important results to permanent storage before expiration (check your cluster/site policies for backup locations)
</div>

'''For detailed information''' about advanced features, configuration options, and less frequently used commands, see the '''[[Workspaces/Advanced Features]]''' guide.

== Command Overview ==

The workspace tools consist of several commands:

* <tt>ws_allocate</tt> - Create or extend a workspace
* <tt>ws_list</tt> - List all your workspaces
* <tt>ws_find</tt> - Find the path to a workspace
* <tt>ws_extend</tt> - Extend the lifetime of a workspace
* <tt>ws_release</tt> - Release (delete) a workspace
* <tt>ws_restore</tt> - Restore an expired or released workspace
* <tt>ws_register</tt> - Create symbolic links to workspaces

All commands support <tt>-h</tt> or <tt>--help</tt> to show detailed usage information.

== Quick Start - Most Common Commands ==


{| class="wikitable"
{| class="wikitable"
Line 39: Line 10:
|Create workspace for 30 days
|Create workspace for 30 days
|<tt>ws_allocate myWs 30</tt>
|<tt>ws_allocate myWs 30</tt>
|-
|Create with custom email
|<tt>ws_allocate -m custom@example.com -r 3 myWs 30</tt>
|-
|-
|Create group-writable workspace
|Create group-writable workspace
|<tt>ws_allocate -G groupname myWs 30</tt>
|<tt>ws_allocate -G groupname myWs 30</tt>
|-
|Create on specific filesystem
|<tt>ws_allocate -F filesystem myWs 30</tt>
|-
|-
|List all your workspaces
|List all your workspaces
|<tt>ws_list</tt>
|<tt>ws_list</tt>
|-
|-
|List by remaining time
|Check which expire soon
|<tt>ws_list -Rr</tt>
|<tt>ws_list -R</tt>
|-
|-
|List available filesystems
|Find workspace path (for scripts)
|<tt>ws_list -l</tt> or <tt>ws_find -l</tt>
|-
|Find workspace path
|<tt>ws_find myWs</tt>
|<tt>ws_find myWs</tt>
|-
|-
|Extend workspace by 30 days
|Extend workspace by 40 days
|<tt>ws_extend myWs 30</tt>
|<tt>ws_extend myWs 40</tt> or <tt>ws_allocate -x myWs 40</tt>
|-
|-
|Share with another user
|Delete/release workspace
|<tt>ws_release myWs</tt>
|<tt>ws_share share myWs username</tt>
|-
|-
|List shared users
|Restore released workspace
|<tt>ws_share list myWs</tt>
|<tt>ws_restore -l</tt> then <tt>ws_restore oldname newname</tt>
|}

== Create Workspace ==

To create a workspace you need to specify a '''name''' and '''lifetime''' in days:

'''Basic usage:'''

$ ws_allocate myWs 30 # Create workspace for 30 days

'''Common variations:'''

$ ws_allocate -G groupname myWs 30 # Create group-writable workspace (recommended for teams)
$ ws_allocate -g myWs 30 # Create group-readable workspace
$ ws_allocate -F ffuc myWs 30 # bwUniCluster 3.0: Use flash filesystem

This returns:
Workspace created. Duration is 720 hours.
Further extensions available: 3
/work/workspace/scratch/username-myWs-0

'''Important:''' Creating a workspace a second time with the same command is safe - it always returns the same path.

'''Capture the path in a variable:'''

$ WORKSPACE=$(ws_allocate myWs 30)
$ cd $WORKSPACE

'''For all options and advanced usage,''' see the [[Workspaces/Advanced_Features#ws_allocate_-_Create_and_Extend_Workspaces|Advanced Features guide]].

== List Your Workspaces ==

'''Basic usage:'''

$ ws_list # List all your workspaces

Shows:
* Workspace ID
* Workspace location
* Available extensions
* Creation date and remaining time

'''Common variations:'''

$ ws_list -Rr # Sort by remaining time (see what expires soon)
$ ws_list -s # Short format (only names, good for scripts)
$ ws_list -g # List group workspaces

'''Note:''' To list expired workspaces that can be restored, see [[#Restore_Workspace|Restore Workspace]].

'''For all options,''' see the [[Workspaces/Advanced_Features#ws_list_-_List_Workspaces|Advanced Features guide]].

== Extend Workspace Lifetime ==

Extend a workspace before it expires:

'''Basic usage:'''

$ ws_extend myWs 30 # Extend by 30 days from now

'''Alternative commands:'''

$ ws_allocate -x myWs 30 # Same as ws_extend

'''Note:''' Each extension consumes one of your available extensions (see the [[Workspaces/Advanced_Features#Cluster-Specific_Workspace_Limits|Cluster-Specific Workspace Limits]]).

'''For all options,''' see the [[Workspaces/Advanced_Features#ws_extend_-_Extend_Workspace_Lifetime|Advanced Features guide]].

== Release (Delete) Workspace ==

When you no longer need a workspace:

'''Basic usage:'''

$ ws_release myWs # Release workspace (recoverable during grace period)

'''What happens:'''
* Workspace becomes inaccessible immediately
* Data is kept for a short grace period (usually until next cleanup run) and can be restored with <tt>ws_restore</tt>
* Released data may still count toward quota until final deletion

'''For all options,''' see the [[Workspaces/Advanced_Features#ws_release_-_Release_.28Delete.29_Workspace|Advanced Features guide]].

== Restore Workspace ==

{| class="wikitable"
|-
!style="width:40%" | Works on cluster
!style="width:10%" | bwUC 3.0
!style="width:10%" | BinAC2
!style="width:10%" | Helix
!style="width:10%" | JUSTUS 2
!style="width:10%" | NEMO2
|-
|-
|Send calendar reminder
|<tt>ws_restore</tt>
|<tt>ws_send_ical myWs user@example.com</tt>
|style="background-color:#90EE90; text-align:center;" | ✓
|style="background-color:#90EE90; text-align:center;" | ✓
|style="background-color:#90EE90; text-align:center;" | ✓
|style="background-color:#90EE90; text-align:center;" | ✓
|style="background-color:#90EE90; text-align:center;" | ✓
|}

If you released a workspace by accident or need to recover an expired one, you can restore it within a grace period:

'''Basic workflow:'''

$ ws_restore -l # (1) List restorable workspaces
$ ws_allocate restored 60 # (2) Create a new target workspace
$ ws_restore username-myWs-0 restored # (3) Restore the expired workspace

'''Note:''' Use the '''full name''' from <tt>ws_restore -l</tt> (including username and timestamp), not the short name from <tt>ws_list</tt>.

'''For detailed restore options,''' see the [[Workspaces/Advanced_Features#ws_restore_-_Restore_Expired_Workspace|Advanced Features guide]].

== Work with Groups (Share Workspaces) ==

{| class="wikitable"
|-
!style="width:40%" | Works on cluster
!style="width:10%" | bwUC 3.0
!style="width:10%" | BinAC2
!style="width:10%" | Helix
!style="width:10%" | JUSTUS 2
!style="width:10%" | NEMO2
|-
|-
|Release workspace
|<tt>-g</tt> option (group-readable)
|<tt>ws_release myWs</tt>
| style="text-align:center;" |
| style="text-align:center;" |
| style="text-align:center;" |
| style="text-align:center;" |
|style="background-color:#90EE90; text-align:center;" | ✓
|-
|-
|List restorable workspaces
|<tt>-G</tt> option (group-writable)
|<tt>ws_restore -l</tt>
| style="text-align:center;" |
|-
| style="text-align:center;" |
|Register workspace links
| style="text-align:center;" |
|<tt>ws_register ~/workspaces</tt>
| style="text-align:center;" |
|style="background-color:#90EE90; text-align:center;" | ✓
|}
|}


== Detailed Topics ==
Working with team members is simple using group workspaces:

=== Create Group Workspace ===

'''Group-readable workspace''' (team can read):

$ ws_allocate -g myWs 30

'''Group-writable workspace''' (team can read and write, recommended):

$ ws_allocate -G projectgroup myWs 30

Replace <tt>projectgroup</tt> with your actual group name (e.g., <tt>bw11a000</tt>).

'''Tip:''' Set your default group in <tt>~/.ws_user.conf</tt> to avoid typing it every time:

<pre>
groupname: projectgroup
</pre>


=== Commands ===
Then you only need: <tt>ws_allocate myWs 30</tt>


* '''[[Workspaces/Advanced_Features/ws_allocate|ws_allocate]]''' - Create and extend workspaces (detailed options)
=== List Group Workspaces ===
* '''[[Workspaces/Advanced_Features/ws_list|ws_list]]''' - List workspaces (sorting, filtering)
* '''[[Workspaces/Advanced_Features/ws_find|ws_find]]''' - Find workspace paths for scripts
* '''[[Workspaces/Advanced_Features/ws_extend|ws_extend]]''' - Extend workspace lifetime
* '''[[Workspaces/Advanced_Features/ws_release|ws_release]]''' - Release (delete) workspaces with immediate deletion options
* '''[[Workspaces/Advanced_Features/ws_restore|ws_restore]]''' - Restore expired or released workspaces
* '''[[Workspaces/Advanced_Features/ws_register|ws_register]]''' - Create symbolic links to workspaces


=== Configuration & Features ===
See all workspaces from your group:


* '''[[Workspaces/Advanced_Features/Filesystems|Multiple Filesystem Locations]]''' - Choosing the right filesystem for your workload
$ ws_list -g
* '''[[Workspaces/Advanced_Features/Reminders|Reminders & Configuration]]''' - Email reminders and ~/.ws_user.conf setup
* '''[[Workspaces/Advanced_Features/Sharing|Sharing Workspaces]]''' - Group workspaces, ws_share, ACLs, Unix permissions
* '''[[Workspaces/Advanced_Features/Quotas|Quotas & Limits]]''' - Cluster-specific limits and checking quotas


=== Best Practices ===
This shows workspaces that were created with <tt>-g</tt> or <tt>-G</tt> by anyone in your group.


* '''[[Workspaces/Advanced_Features/Best_Practices|Best Practices]]''' - Recommendations for different use cases
=== Extend Group Workspace ===


== Quick Navigation ==
Anyone in the group can extend a group-writable workspace (<tt>-G</tt>):


=== By Task ===
$ ws_extend myWs 30 # If you created it
$ ws_allocate -x -u username myWs 30 # If colleague created it


'''Creating and Managing:'''
Replace <tt>username</tt> with the workspace owner's username. This is useful when they're unavailable.
* Create workspace → [[Workspaces#Create_Workspace|Basic]] | [[Workspaces/Advanced_Features/ws_allocate|Advanced]]
* List workspaces → [[Workspaces#List_Your_Workspaces|Basic]] | [[Workspaces/Advanced_Features/ws_list|Advanced]]
* Extend lifetime → [[Workspaces#Extend_Workspace_Lifetime|Basic]] | [[Workspaces/Advanced_Features/ws_extend|Advanced]]
* Release workspace → [[Workspaces#Release_.28Delete.29_Workspace|Basic]] | [[Workspaces/Advanced_Features/ws_release|Advanced]]
* Restore workspace → [[Workspaces#Restore_Workspace|Basic]] | [[Workspaces/Advanced_Features/ws_restore|Advanced]]


'''Collaboration:'''
=== Manage Reminders for Group Workspace ===
* Work with groups → [[Workspaces#Work_with_Groups_.28Share_Workspaces.29|Basic]] | [[Workspaces/Advanced_Features/Sharing#Group_Workspaces|Advanced]]
* Share with users → [[Workspaces/Advanced_Features/Sharing#Sharing_with_ws_share|ws_share]] | [[Workspaces/Advanced_Features/Sharing#ACLs:_Access_Control_Lists|ACLs]] | [[Workspaces/Advanced_Features/Sharing#Regular_Unix_Permissions|Unix Permissions]]


'''Configuration:'''
You can update reminder settings and take over responsibility for reminders on a colleague's workspace:
* Email reminders → [[Workspaces/Advanced_Features/Reminders|Reminders]]
* Default settings → [[Workspaces/Advanced_Features/Reminders#Configuration_File:_.7E.2F.ws_user.conf|~/.ws_user.conf]]
* Workspace links → [[Workspaces/Advanced_Features/ws_register|ws_register]]


'''Resources:'''
$ ws_allocate -r 7 -u username -x myWs 0 # Update reminder time and take over
* Filesystem selection → [[Workspaces/Advanced_Features/Filesystems|Filesystems]]
# another user's workspace reminders
* Check quotas → [[Workspaces/Advanced_Features/Quotas#Checking_Workspace_Quotas|Quota Commands]]
* Cluster limits → [[Workspaces/Advanced_Features/Quotas|Quotas & Limits]]


=== By Experience Level ===
This changes the reminder timing to 7 days before expiration and redirects reminder emails to you instead of the original creator. Useful when you're taking over responsibility for a shared workspace.


'''Beginners:'''
=== Why Use Group Workspaces? ===
* Start with the main [[Workspaces]] guide
* Review [[Workspaces/Advanced_Features/Best_Practices|Best Practices]]


'''Intermediate Users:'''
* Simple collaboration: Everyone can access the same data
* [[Workspaces/Advanced_Features/Filesystems|Choose optimal filesystem]]
* No permission problems: Files automatically get group permissions
* [[Workspaces#Work_with_Groups_.28Share_Workspaces.29|Set up group workspaces]]
* Independent extensions: Team members can extend without original creator
* [[Workspaces/Advanced_Features/ws_allocate#Using_Workspaces_in_Batch_Jobs|Use in batch jobs]]
* Easy to find: Use <tt>ws_list -g</tt> to see all team workspaces
* [[Workspaces/Advanced_Features/ws_restore#Helix-specific_-_Workspace_Snapshots|Snapshot recovery (Helix)]]


'''Advanced Users:'''
'''For advanced sharing options''' (sharing with specific users outside your group, ACL-based methods), see the [[Workspaces/Advanced_Features#Cooperative_Usage_.28Group_Workspaces_and_Sharing.29|Advanced Features guide]].
* [[Workspaces/Advanced_Features/Sharing|Advanced sharing methods]] (ACLs)
* [[Workspaces/Advanced_Features/ws_release#Immediate_Deletion_.28Free_Quota_Instantly.29|Immediate deletion for quota management]]
* Set up [[Workspaces/Advanced_Features/Reminders#Configuration_File:_.7E.2F.ws_user.conf|~/.ws_user.conf]]

Latest revision as of 17:43, 2 December 2025

This document covers advanced features and detailed configuration options for the HPC workspace tools. For basic daily usage, see the main Workspaces guide.

Almost Complete Command Reference

Task Command
Create workspace for 30 days ws_allocate myWs 30
Create with custom email ws_allocate -m custom@example.com -r 3 myWs 30
Create group-writable workspace ws_allocate -G groupname myWs 30
Create on specific filesystem ws_allocate -F filesystem myWs 30
List all your workspaces ws_list
List by remaining time ws_list -R
List available filesystems ws_list -l or ws_find -l
Find workspace path ws_find myWs
Extend workspace by 40 days ws_extend myWs 40 or ws_allocate -x myWs 40
Share with another user ws_share share myWs username
List shared users ws_share list myWs
Send calendar reminder ws_send_ical myWs user@example.com
Release workspace ws_release myWs
List restorable workspaces ws_restore -l
Register workspace links ws_register ~/workspaces

Detailed Topics

Commands

  • ws_allocate - Create and extend workspaces (detailed options)
  • ws_list - List workspaces (sorting, filtering)
  • ws_find - Find workspace paths for scripts
  • ws_extend - Extend workspace lifetime
  • ws_release - Release (delete) workspaces with immediate deletion options
  • ws_restore - Restore expired or released workspaces
  • ws_register - Create symbolic links to workspaces

Configuration & Features

Best Practices

Quick Navigation

By Task

Creating and Managing:

Collaboration:

Configuration:

Resources:

By Experience Level

Beginners:

Intermediate Users:

Advanced Users:

Retrieved from "https://wiki.bwhpc.de/wiki/index.php?title=Workspaces/Advanced_Features&oldid=15614"