Workspaces/Advanced Features/Filesystems

From bwHPC Wiki
Jump to navigation Jump to search

Multiple Filesystem Locations

Most users don't need special filesystem options. On all clusters, workspaces are created on the default high-performance filesystem without any options - this works for standard I/O workloads.

Do I Need the -F Option?

For standard I/O (large files, sequential access):

  • All clusters: Just use ws_allocate myWs 30 (no -F needed)
  • The default filesystem handles standard workloads well

For special workloads (AI/ML, many small files, random I/O):

  • NEMO2: Default Weka filesystem works great - no -F needed
  • bwUniCluster 3.0: Use -F ffuc for flash filesystem
  • Other clusters: Use $TMPDIR or default workspace

Check Available Filesystems

  $ ws_list -l                             # List available filesystems

If only one filesystem is listed, you're all set - just use ws_allocate without -F.

When -F Option is Available

Works on cluster bwUC 3.0 BinAC2 Helix JUSTUS 2 NEMO2
-F option

Only bwUniCluster 3.0 offers multiple filesystems via -F option.

Cluster-Specific Information

NEMO2

Default Weka filesystem (no -F needed):

  • Excellent for all workloads - standard I/O, small files, random access
  • Handles AI/ML training, compilation, and general workloads efficiently
  • Just use: ws_allocate myWs 30

bwUniCluster 3.0

Default Lustre filesystem (no -F needed):

  • Best for standard I/O: large files, sequential access
  • General-purpose workload
  • Use: ws_allocate myWs 30

Flash filesystem with -F ffuc:

  • SSD-based storage for special workloads
  • Shared between bwUniCluster 3.0 and HoreKa (KIT/HoreKa users only)
  • Use for: AI/ML datasets, many small files, random I/O, compilation
  • Use: ws_allocate -F ffuc myWs 30

Other Clusters (BinAC2, Helix, JUSTUS 2)

  • Single default filesystem (no -F option available)
  • Good for all standard workloads
  • For special workloads with many small files, consider using $TMPDIR

Simple Decision Guide

Your Workload NEMO2 bwUniCluster 3.0
Standard I/O (large files) ws_allocate myWs 30 ws_allocate myWs 30
AI/ML training ws_allocate myWs 30 ws_allocate -F ffuc myWs 30
Many small files ws_allocate myWs 30 ws_allocate -F ffuc myWs 30
Random I/O ws_allocate myWs 30 ws_allocate -F ffuc myWs 30
Compilation/builds ws_allocate myWs 30 ws_allocate -F ffuc myWs 30
Single-node temporary Use $TMPDIR, not workspaces

Quick Reference by Data Type

Data Type Where to Store
Large files, standard I/O Default workspace (no -F) on all clusters
AI/ML datasets NEMO2: default workspace; bwUniCluster 3.0: -F ffuc
Many small files NEMO2: default workspace; bwUniCluster 3.0: -F ffuc
Random I/O patterns NEMO2: default workspace; bwUniCluster 3.0: -F ffuc
Single-node temporary Always $TMPDIR, not workspaces
Multi-node shared data Default workspace on all clusters
Compilation/builds NEMO2: default workspace; bwUniCluster 3.0: -F ffuc or $TMPDIR

For quota information, see Quotas & Limits.