Workspaces/Advanced Features/Filesystems
Jump to navigation
Jump to search
Multiple Filesystem Locations
Most users don't need special filesystem options. On all clusters, workspaces are created on the default high-performance filesystem without any options - this works for standard I/O workloads.
Do I Need the -F Option?
For standard I/O (large files, sequential access):
- All clusters: Just use ws_allocate myWs 30 (no -F needed)
- The default filesystem handles standard workloads well
For special workloads (AI/ML, many small files, random I/O):
- NEMO2: Default Weka filesystem works great - no -F needed
- bwUniCluster 3.0: Use -F ffuc for flash filesystem
- Other clusters: Use $TMPDIR or default workspace
Check Available Filesystems
$ ws_list -l # List available filesystems
If only one filesystem is listed, you're all set - just use ws_allocate without -F.
When -F Option is Available
| Works on cluster | bwUC 3.0 | BinAC2 | Helix | JUSTUS 2 | NEMO2 |
|---|---|---|---|---|---|
| -F option | ✓ | ✗ | ✗ | ✗ | ✗ |
Only bwUniCluster 3.0 offers multiple filesystems via -F option.
Cluster-Specific Information
NEMO2
Default Weka filesystem (no -F needed):
- Excellent for all workloads - standard I/O, small files, random access
- Handles AI/ML training, compilation, and general workloads efficiently
- Just use: ws_allocate myWs 30
bwUniCluster 3.0
Default Lustre filesystem (no -F needed):
- Best for standard I/O: large files, sequential access
- General-purpose workload
- Use: ws_allocate myWs 30
Flash filesystem with -F ffuc:
- SSD-based storage for special workloads
- Shared between bwUniCluster 3.0 and HoreKa (KIT/HoreKa users only)
- Use for: AI/ML datasets, many small files, random I/O, compilation
- Use: ws_allocate -F ffuc myWs 30
Other Clusters (BinAC2, Helix, JUSTUS 2)
- Single default filesystem (no -F option available)
- Good for all standard workloads
- For special workloads with many small files, consider using $TMPDIR
Simple Decision Guide
| Your Workload | NEMO2 | bwUniCluster 3.0 |
|---|---|---|
| Standard I/O (large files) | ws_allocate myWs 30 | ws_allocate myWs 30 |
| AI/ML training | ws_allocate myWs 30 | ws_allocate -F ffuc myWs 30 |
| Many small files | ws_allocate myWs 30 | ws_allocate -F ffuc myWs 30 |
| Random I/O | ws_allocate myWs 30 | ws_allocate -F ffuc myWs 30 |
| Compilation/builds | ws_allocate myWs 30 | ws_allocate -F ffuc myWs 30 |
| Single-node temporary | Use $TMPDIR, not workspaces | |
Quick Reference by Data Type
| Data Type | Where to Store |
|---|---|
| Large files, standard I/O | Default workspace (no -F) on all clusters |
| AI/ML datasets | NEMO2: default workspace; bwUniCluster 3.0: -F ffuc |
| Many small files | NEMO2: default workspace; bwUniCluster 3.0: -F ffuc |
| Random I/O patterns | NEMO2: default workspace; bwUniCluster 3.0: -F ffuc |
| Single-node temporary | Always $TMPDIR, not workspaces |
| Multi-node shared data | Default workspace on all clusters |
| Compilation/builds | NEMO2: default workspace; bwUniCluster 3.0: -F ffuc or $TMPDIR |
For quota information, see Quotas & Limits.