M Janczyk: M Janczyk moved page Workspaces/Advanced Features/Filesystems to NEMO2/Workspaces/Advanced Features/Filesystems

2026-03-02T19:39:29Z

M Janczyk moved page Workspaces/Advanced Features/Filesystems to NEMO2/Workspaces/Advanced Features/Filesystems

← Older revision	Revision as of 21:39, 2 March 2026
(No difference)

M Janczyk at 15:32, 2 December 2025

2025-12-02T15:32:53Z

Show changes

M Janczyk: Created page with "= Multiple Filesystem Locations = {| class="wikitable" |- !style="width:40%" | Works on cluster !style="width:10%" | bwUC 3.0 !style="width:10%" | BinAC2 !style="width:10%" | Helix !style="width:10%" | JUSTUS 2 !style="width:10%" | NEMO2 |- |`-F` option (multiple filesystems) |style="background-color:#90EE90; text-align:center;" | ✓ |style="background-color:#FFB6C1; text-align:center;" | ✗ |style="background-color:#FFB6C1; text-align:center;" | ✗ |style="b..."

2025-12-02T13:11:55Z

Created page with "= Multiple Filesystem Locations = {| class="wikitable" |- !style="width:40%" | Works on cluster !style="width:10%" | bwUC 3.0 !style="width:10%" | BinAC2 !style="width:10%" | Helix !style="width:10%" | JUSTUS 2 !style="width:10%" | NEMO2 |- |<tt>-F</tt> option (multiple filesystems) |style="background-color:#90EE90; text-align:center;" | ✓ |style="background-color:#FFB6C1; text-align:center;" | ✗ |style="background-color:#FFB6C1; text-align:center;" | ✗ |style="b..."

New page

= Multiple Filesystem Locations =

{| class="wikitable"
|-
!style="width:40%" | Works on cluster
!style="width:10%" | bwUC 3.0
!style="width:10%" | BinAC2
!style="width:10%" | Helix
!style="width:10%" | JUSTUS 2
!style="width:10%" | NEMO2
|-
|<tt>-F</tt> option (multiple filesystems)
|style="background-color:#90EE90; text-align:center;" | ✓
|style="background-color:#FFB6C1; text-align:center;" | ✗
|style="background-color:#FFB6C1; text-align:center;" | ✗
|style="background-color:#FFB6C1; text-align:center;" | ✗
|style="background-color:#FFB6C1; text-align:center;" | ✗
|}

Some clusters offer multiple filesystem locations for workspaces with different characteristics:

'''bwUniCluster 3.0:'''
* Default workspace filesystem (Lustre)
* Flash-based workspace filesystem (<tt>ffuc</tt>) - for KIT/HoreKa users only
** Lower latency and better performance for small files
** SSDs instead of hard disks
** Shared between bwUniCluster 3.0 and HoreKa

'''Example creating workspace on flash filesystem:'''

$ ws_allocate -F ffuc myworkspace 60

Use <tt>ws_list -l</tt> or <tt>ws_find -l</tt> to see available filesystem locations on your cluster.

== Choosing the Right Filesystem ==

'''Note:''' Most clusters have only one default filesystem - the <tt>-F</tt> option is rarely needed. Use <tt>ws_list -l</tt> first to check if multiple filesystems are available on your cluster.

=== bwUniCluster 3.0 Filesystems ===

'''Default Lustre filesystem:'''
* Standard workspace location
* Best for large files and sequential I/O
* General-purpose storage

'''Flash filesystem (ffuc):'''
* SSD-based storage for KIT/HoreKa users
* Shared between bwUniCluster 3.0 and HoreKa
* Use for workloads with:
** Many small files
** Random I/O patterns
** AI/ML training
** Compilation and builds
* Balance load: use <tt>-F ffuc</tt> when appropriate to reduce load on default filesystem

=== General Guidelines ===

'''Flash-based filesystems (SSD/NVMe):'''
* Use for many small files
* Best for low-latency requirements
* Ideal for random I/O patterns
* Examples: Weka (NEMO2), ffuc (bwUniCluster 3.0)

'''Standard Lustre/parallel filesystems:'''
* Best for large files
* Optimized for sequential I/O patterns
* General-purpose workload support

=== Data Type Recommendations ===

'''Large sequential I/O:'''
* Use standard workspace filesystem
* Lustre: best for very large files
* Weka: excellent for both large and small files

'''Many small files or random access:'''
* Use flash-based workspace filesystem (Weka, ffuc)
* Or stage to <tt>$TMPDIR</tt> on compute nodes

'''Data read multiple times on single node:'''
* Copy to <tt>$TMPDIR</tt> at job start for best performance

'''Temporary data for single node:'''
* Always use <tt>$TMPDIR</tt>, not workspaces

'''Multi-node temporary data:'''
* Use workspaces (not suitable for <tt>$TMPDIR</tt>)

'''AI/ML training data:'''
* Use Weka (NEMO2) or flash filesystems for best performance
* Or stage to <tt>$TMPDIR</tt> for repeated access

'''Compilation/build directories:'''
* Use flash-based filesystems (Weka, ffuc)
* Or <tt>$TMPDIR</tt> for better performance

For more information about specific filesystems, see the [[Workspaces/Advanced_Features/Quotas|Quotas & Limits]] page.

NEMO2/Workspaces/Advanced Features/Filesystems - Revision history

M Janczyk: M Janczyk moved page Workspaces/Advanced Features/Filesystems to NEMO2/Workspaces/Advanced Features/Filesystems

M Janczyk at 15:32, 2 December 2025