BinAC2/Hardware and Architecture and NEMO2/Hardware: Difference between pages

From bwHPC Wiki
< BinAC2(Difference between pages)
Jump to navigation Jump to search
No edit summary
 
mNo edit summary
 
Line 1: Line 1:
= Hardware and Architecture =
== Operating System and Software ==


{| class="wikitable"
The bwForCluster BinAC 2 supports researchers from the broader fields of Bioinformatics, Medical Informatics, Astrophysics, Geosciences and Pharmacy.
|-
!scope="column" | Operating System
| [https://rockylinux.org Rocky Linux 9] (similar to RHEL 9)
|-
!scope="column" | Queuing System
| [https://slurm.schedmd.com SLURM] (see [[NEMO2/Slurm]] for help)
|-
!scope="column" | (Scientific) Libraries and Software
| [[Environment Modules]]
|-
!scope="column" | Own Software Modules using EasyBuild and Spack
| [https://docs.easybuild.io/ EasyBuild]
|-
!scope="column" | Own (Python) Environments with Conda
| [[Conda]]
|-
!scope="column" | Containers with Apptianer/Singularity (and enroot in the future)
| [[Development/Containers]]
|-
|}


== Operating System and Software ==


__TOC__
* Operating System: Rocky Linux 9.5
* Queuing System: [https://slurm.schedmd.com/documentation.html Slurm] (see [[BinAC2/Slurm]] for help)
* (Scientific) Libraries and Software: [[Environment Modules]]


== Compute Nodes ==


== Compute and Special Purpose Nodes ==
BinAC 2 offers compute nodes, high-mem nodes, and three types of GPU nodes.
* 180 compute nodes
* 16 SMP node
* 32 GPU nodes (2xA30)
* 8 GPU nodes (4xA100)
* 4 GPU nodes (4xH200)
* plus several special purpose nodes for login, interactive jobs, etc.


For researchers from the scientific fields '''N'''euroscience, '''E'''lementary Particle Physics, '''M'''icrosystems Engineering and '''M'''aterials Science the bwForCluster '''NEMO''' offers about 240 compute nodes plus several special purpose nodes for login, interactive jobs, visualization, machine learning and ai.
Compute node specification:

{| class="wikitable"
Node specification (see [[NEMO2/Slurm]] for slurm partitons):
{| class="wikitable" style="text-align:center;"
|-
|-
! style="width:10%"|
! style="width:15%"|
! style="width:10%"| Standard
! style="width:15%"| Genoa Partition
! style="width:10%"| High-Mem
! style="width:15%"| Milan Partition
! style="width:10%"| GPU (A30)
! style="width:15%"| L40S Partition
! style="width:10%"| GPU (A100)
! style="width:15%"| MI300A Partition
! style="width:10%"| GPU (H200)
! style="width:15%"| H200 Partition (August 2025)
! style="width:15%"| Login Partition
|-
|-
!scope="column"| Quantity
!scope="column"| Quantity
| 106
| 168 / 12
| 14 / 2
| 137
| 32
| 9
| 8
| 4
| 4
| 2
| 2
|-
|-
!scope="column" | Processors
!scope="column" | Processors / APU/GPU
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7543.html AMD EPYC Milan 7543] / 2 x [https://www.amd.com/en/products/processors/server/epyc/7003-series/amd-epyc-75f3.html AMD EPYC Milan 75F3]
| 2x [https://www.amd.com/en/products/processors/server/epyc/4th-generation-9004-and-8004-series/amd-epyc-9654.html AMD EPYC 9654 (Genoa)]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7443.html AMD EPYC Milan 7443] / 2 x [https://www.amd.com/en/products/processors/server/epyc/7003-series/amd-epyc-75f3.html AMD EPYC Milan 75F3]
| 2x [https://www.amd.com/en/products/processors/server/epyc/7003-series/amd-epyc-7763.html AMD EPYC 7763 (Milan)]
| 2x [https://www.intel.com/content/www/us/en/products/sku/237558/intel-xeon-platinum-8562y-processor-60m-cache-2-80-ghz/specifications.html Intel Xeon Platinum 8562Y+ (5th Gen)]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7543.html AMD EPYC Milan 7543]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7543.html AMD EPYC Milan 7543]
4x [https://www.nvidia.com/en-us/data-center/l40s/ NVIDIA L40S]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/9005-series/amd-epyc-9555.html AMD EPYC Milan 9555]
| 4x [https://www.amd.com/en/products/accelerators/instinct/mi300/mi300a.html AMD Instinct MI300A]
| 2x [https://www.amd.com/en/products/processors/server/epyc/4th-generation-9004-and-8004-series/amd-epyc-9654.html AMD EPYC 9654 (Genoa)]
8x [https://www.nvidia.com/en-us/data-center/h200/ NVIDIA H200]
| 1x [https://www.amd.com/en/products/processors/server/epyc/4th-generation-9004-and-8004-series/amd-epyc-9354.html AMD EPYC 9354 (Genoa)]
|-
|-
!scope="column" | Processor Base Frequency (GHz)
!scope="column" | Base Frequency/Boost Frequency (GHz) / APU/GPU Performance (TFLOPs/TOPs)
| 2.80 / 2.95
| 2.4/3.55
| 2.85 / 2.95
| 2.45/3.5
| 2.80
| 2.8/3.8
91.6 (FP32) / 733 (INT8)
| 2.80
| 3.20
| -/3.7
61.3 (FP64) / 122.6 (FP32) / 1960 (INT8)
| 2.4/3.55
34 (FP64) / 67 (FP32) / 3958 (INT8)
| 3.25/3.75
|-
|-
!scope="column" | Number of Physical Cores / Hypertreads
!scope="column" | CPU Cores per Node
| 192
| 64 / 128
'''Usable in Slurm: 190'''
| 48 / 96 // 64 / 128
| 64 / 128
| 128
'''Usable in Slurm: 126'''
| 64 / 128
| 64
| 128 / 256
'''Usable in Slurm: 62'''
| 4x 24
'''Usable in Slurm: 92'''
| 192
'''Usable in Slurm: 190'''
| 32 (SMT on)
'''Usable for Users: 4'''
|-
|-
!scope="column" | Working Memory (GB)
!scope="column" | CPU Cores per APU/GPU
| 512
| ---
| 2048
| ---
| 512
| 15
| 512
| 23
| 1536
| 23
| ---
|-
|-
!scope="column" | Local Disk (GiB)
!scope="column" | Memory
| 768 GiB, 4.8 GHz (DDR5)
| 450 (NVMe-SSD)
'''Usable in Slurm: 727 GiB, 745000 MiB, per Core: 3900 MiB'''
| 14000 (NVMe-SSD)
| 512 GiB, 3.2 GHZ (DDR4)
| 450 (NVMe-SSD)
'''Usable in Slurm: 495 GiB, 507000 MiB, per Core: 4000 MiB'''
| 14000 (NVMe-SSD)
| 512 GiB, 5.6 GHz (DDR5)
| 28000 (NVMe-SSD)
4x 48 GB, 864 GB/s (GDDR6)
'''Usable in Slurm: 495 GiB, 507000 MiB, per Core: 8100 MiB'''
| 4x 128 GB, 5300 GB/s (HBM3)
'''Usable in Slurm: 495 GiB, 507000 MiB, per Core: 5300 MiB'''
| 1536 GiB, 4.8 GHz (DDR5)
8x 141 GB, 4813 GB/s (HBM3e)
'''Usable in Slurm: 1459 GiB, 1495000 MiB, per Core:7800 MiB'''
| 384 GiB, 4.8 GHz (DDR5)
'''Usable for Users: 50 GiB, 51200 MiB, per Core: 12800 MiB'''
|-
!scope="column" | Local NVMe (GB)
| 3840
| 1920
| 3840
| 3840
| 3840
| 480
|-
|-
!scope="column" | Interconnect
!scope="column" | Interconnect
| HDR 100 IB (84 nodes) / 100GbE (96 nodes)
| 100 GbE (RoCEv2)
| Omni-Path 100
| 100GbE
100 GbE (RoCEv2)
| 100GbE
| 100 GbE (RoCEv2)
| 100GbE
| 100 GbE (RoCEv2)
| HDR 200 IB + 100GbE
| 100 GbE (RoCEv2)
| 100 GbE (RoCEv2)
|-
!scope="column" | Job Example Genoa Partition
|colspan="6" | Maximum resources for a single node job (*):
<code>--partition=genoa --ntasks=190 --mem=727GB # or: --mem=745000MB, or: --mem-per-cpu=3900MB</code>
|-
!scope="column" | Job Example Milan Partition
|colspan="6" | Maximum resources for a single node job (*):
<code>--partition=milan --ntasks=126 --mem=495GB # or: --mem=507000MB, or: --mem-per-cpu=4000MB</code>
|-
!scope="column" | Job Example L40S Partition
|colspan="6" | Maximum resources for a single node job (*):
<code>--partition=l40s --ntasks=62 --gres=gpu:4 --mem=495GB # or: --mem=507000MB, or: --mem-per-cpu=8100MB</code>
|-
!scope="column" | Job Example MI300A Partition
|colspan="6" | Maximum resources for a single node job (*):
<code>--partition=mi300a --ntasks=94 --gres=gpu:4 --mem=495GB # or: --mem=507000MB, or: --mem-per-cpu=5300MB</code>
|-
!scope="column" | Job Example H200 Partition
|colspan="6" | Maximum resources for a single node job (*):
<code>--partition=h200 --ntasks=190 --gres=gpu:8 --mem=1459GB # or: --mem=1495000MB, or: --mem-per-cpu=7800MB</code>
|-
!scope="column" | Job Example Login Partition
|colspan="6" | Maximum resources for a single node job (*):
<code>--partition=login --ntasks=4 --time=15:00 --mem=50GB + or: --mem=51200MB, or: --mem-per-cpu=12800MB</code>
|-
|-
!scope="column" | Coprocessors
| -
| -
| 2 x [https://www.nvidia.com/de-de/data-center/products/a30-gpu/ NVIDIA A30 (24 GB ECC HBM2, NVLink)]
| 4 x [https://www.nvidia.com/de-de/data-center/a100/ NVIDIA A100 (80 GB ECC HBM2e)]
| 4 x [https://www.nvidia.com/de-de/data-center/h200/ NVIDIA H200 NVL (141 GB ECC HBM3e, NVLink)]
|}
|}
(*) Slurm internally uses Mebi- (MiB) and Gibibyte (GiB), please multiply/divide with 1024 for (M/G).


== File Systems ==
= Network =


NEMO2 offers a fast [https://docs.weka.io/ Weka] parallel filesystem, which is only limited by the uplink to this storage (>90GB/s).
The compute nodes and the parallel file system are connected via 100GbE ethernet</br>
The storage is used for <tt>$HOME</tt> and '''[[workspace]]s'''.
In contrast to BinAC 1 not all compute nodes are connected via Infiniband, but there are 84 standard compute nodes connected via HDR Infiniband (100 GbE). In order to get your jobs onto the Infiniband nodes, submit your job with <code>--constraint=ib</code>.
There sill be no backups, but we plan to implement Snapshots for the last 7 days in the next months.

Additionally, each compute node provides temporary storage on the node-local NVMe disk.
= File Systems =

The bwForCluster BinAC 2 consists of two separate storage systems, one for the user's home directory $HOME and one serving as a project/work space.
The home directory is limited in space and parallel access but offers snapshots of your files and backup.

The project/work is a parallel file system (PFS) which offers fast and parallel file access and a bigger capacity than the home directory. It is mounted at <code>/pfs/10</code> on the login and compute nodes. This storage is based on Lustre and can be accessed parallel from many nodes. The PFS contains the project and the work directory. Each compute project has its own directory at <code>/pfs/10/project</code> that is accessible for all members of the compute project.
Each user can create workspaces under <code>/pfs/10/work</code> using the workspace tools. These directories are only accessible for the user who created the workspace.

Additionally, each compute node provides high-speed temporary storage (SSD) on the node-local solid state disk via the $TMPDIR environment variable.


{| class="wikitable"
{| class="wikitable"
Line 105: Line 164:
! style="width:10%"|
! style="width:10%"|
! style="width:10%"| <tt>$HOME</tt>
! style="width:10%"| <tt>$HOME</tt>
! style="width:10%"| project
! style="width:10%"| '''[[Workspace]]s'''
! style="width:10%"| work
! style="width:10%"| NVMe
! style="width:10%"| <tt>$TMPDIR</tt>
|-
|-
!scope="column" | Visibility
!scope="column" | Visibility
|colspan="2" style="text-align:center;"| global (100 GbE)
| global
| global
| global
| node local
| node local
|-
|-
!scope="column" | Lifetime
!scope="column" | Lifetime
| permanent
| permanent
| workspace lifetime
| permanent
| work space lifetime (max. 30 days, max. 5 extensions)
(max. 100 days, extensions possible)
| batch job walltime
| batch job walltime
|-
|-
!scope="column" | Capacity
!scope="column" | Capacity
|colspan="2" style="text-align:center;"| 1 PB
| -
| 1.92/3.84 TB (logins 480 GB)
| 8.1 PB
| 1000 TB
| 480 GB (compute nodes); 7.7 TB (GPU-A30 nodes); 16 TB (GPU-A100 and SMP nodes); 31 TB (GPU-H200 nodes)
|-
|-
!scope="column" | Speed (read)
!scope="column" | Quotas per <tt>$HOME</tt>/Workspace
| 100 GB
| ≈ 1 GB/s, shared by all nodes
| 5 TB (per workspace)
| max. 12 GB/s
| ---
| ≈ 145 GB/s peak, aggregated over 56 nodes, ideal striping
| ≈ 3 GB/s (compute)/ ≈5 GB/S (GPUA-30)/ ≈ 26 GB/s (GPU-A100 + SMP)/ ≈ 42 GB/s (GPU-H200) per node
|-
|-
!scope="column" | [https://en.wikipedia.org/wiki/Disk_quota#Quotas Quotas]
!scope="column" | Snapshots
| daily (7 snapshots) (not yet implemented)
| 40 GB per user
| ---
| not yet, maybe in the future
| none
| ---
|-
| none
!scope="column" | Backups
|colspan="3" style="text-align:center;"| '''There is NO storage backup!'''
|-
|-
!scope="column" | Backup
| yes (nightly)
| '''no'''
| '''no'''
| '''no'''
|}
|}


<pre>
global : all nodes access the same file system
global : all nodes access the same file system
local : each node has its own file system
local : each node has its own file system
permanent : files are stored permanently
permanent : files are stored permanently
however, if an account has lost access,
the remaining data will be deleted after 6 months
batch job walltime : files are removed at end of the batch job
batch job walltime : files are removed at end of the batch job
</pre>


=== Listing File System Quotas ===
{| class="wikitable" style="color:red; background-color:#ffffcc;" cellpadding="10"
|
Please note that due to the large capacity of '''work''' and '''project''' and due to frequent file changes on these file systems, no backup can be provided.</br>
Backing up these file systems would require a redundant storage facility with multiple times the capacity of '''project'''. Furthermore, regular backups would significantly degrade the performance.</br>
Data is stored redundantly, i.e. immune against disk failures but not immune against catastrophic incidents like cyber attacks or a fire in the server room.</br>
Please consider to use on of the remote storage facilities like [https://wiki.bwhpc.de/e/SDS@hd SDS@hd], [https://uni-tuebingen.de/einrichtungen/zentrum-fuer-datenverarbeitung/projekte/laufende-projekte/bwsfs bwSFS], [https://www.scc.kit.edu/en/services/lsdf.php LSFD Online Storage] or the [https://www.rda.kit.edu/english/ bwDataArchive] to back up your valuable data.
|}


You can list the quotas for your users using <code>nemoquota</code> or <code>df <path_to_directory></code>.


Examples:
=== Home ===
<pre>
$ nemoquota
HOME / WORKSPACE Used% Used Free Quota
HOME 8% 7.5G 93G 101G
conda 2% 59G 5.0T 5.0T
container 6% 252G 4.8T 5.0T


$ df --si $HOME
Home directories are meant for permanent file storage of files that are keep being used like source codes, configuration files, executable programs etc.; the content of home directories will be backed up on a regular basis.
Filesystem Size Used Avail Use% Mounted on
Because the backup space is limited we enforce a quota of 40GB on the home directories.
10.14.101.1/home 101G 7.5G 93G 8% /home


$ df --si $(ws_find conda)
'''NOTE:'''
Filesystem Size Used Avail Use% Mounted on
Compute jobs on nodes must not write temporary data to $HOME.
10.14.101.1/work 5.0T 59G 5.0T 2% /work
Instead they should use the local $TMPDIR directory for I/O-heavy use cases
and work spaces for less I/O intense multinode-jobs.

<!--
Current disk usage on home directory and quota status can be checked with the '''diskusage''' command:
$ diskusage
User Used (GB) Quota (GB) Used (%)
------------------------------------------------------------------------
<username> 4.38 100.00 4.38
-->
=== Project ===

Each compute project has its own project directory at <code>/pfs/10/project</code>.

<pre>
$ ls -lh /pfs/10/project/
drwxrwx---. 2 root bw16f003 33K Dec 12 16:46 bw16f003
[...]
</pre>
</pre>

As you can see the directory is owned by a group representing your compute project (here bw16f003) and the directory is accessible by all group members. It is upon your group to decide how to use the space inside this directory: shared data folders, personal directories for each project member, software containers, etc.

The data is stored on HDDs. The primary focus of <code>/pfs/10/project</code> is pure capacity, not speed.

=== Workspaces ===

Data on the fast storage pool at <code>/pfs/10/work</code> is stored on SSDs.
The primary focus is speed, not capacity.

In contrast to BinAC 1 we will enforce work space lifetime, as the capacity is limited.
We ask you to only store data you actively use for computations on <code>/pfs/10/work</code>.
Please move data to <code>/pfs/10/project</code> when you don't need it on the fast storage any more.

Each user should create workspaces at <code>/pfs/10/work</code> through the workspace tools

You can find more info on workspace tools on our general page:

:: &rarr; '''[[Workspace]]s'''

To create a work space you'll need to supply a name for your work space area and a lifetime in days.
For more information read the corresponding help, e.g: <code>ws_allocate -h.</code>
{| class="wikitable"
|-
!style="width:30%" | Command
!style="width:70%" | Action
|-
|<code>ws_allocate mywork 30</code>
|Allocate a work space named "mywork" for 30 days.
|-
|<code>ws_allocate myotherwork</code>
|Allocate a work space named "myotherwork" with maximum lifetime.
|-
|<code>ws_list -a</code>
|List all your work spaces.
|-
|<code>ws_find mywork</code>
|Get absolute path of work space "mywork".
|-
|<code>ws_extend mywork 30</code>
|Extend life me of work space mywork by 30 days from now.
|-
|<code>ws_release mywork</code>
|Manually erase your work space "mywork". Please remove directory content first.
|-
|}

=== Scratch ===

Please use the fast local scratch space for storing temporary data during your jobs.

For each job a scratch directory will be created on the compute nodes. It is available via the environment variable <code>$TMPDIR</code>, which points to <code>/scratch/<jobID></code>.

Especially the SMP nodes and the GPU nodes are equipped with large and fast local disks that should be used for temporary data, scratch data or data staging for ML model training.
The Lustre file system (<code>WORK</code> and <code>PROJECT</code>) is unsuited for repetitive random I/O, I/O sizes smaller than the Lustre and ZFS block size (1M) or I/O patterns where files are opened and closed in rapid succession. The XFS file system of the local scratch drives is better suited for typical scratch workloads and access patterns. Moreover, the local scratch drives offer a lower latency and a higher bandwidth than <code>WORK</code>.


=== SDS@hd ===

SDS@hd is mounted via NFS on login and compute nodes at <syntaxhighlight inline>/mnt/sds-hd</syntaxhighlight>.

To access your Speichervorhaben, the export to BinAC 2 must first be enabled by the SDS@hd-Team. Please contact [mailto:sds-hd-support@urz.uni-heidelberg.de SDS@hd support] and provide the acronym of your Speichervorhaben, along with a request to enable the export to BinAC 2.

Once this has been done, you can access your Speichervorhaben as described in the [https://wiki.bwhpc.de/e/SDS@hd/Access/NFS#Access_your_data SDS documentation].

<syntaxhighlight>
$ kinit $USER
Password for <user>@BWSERVICES.UNI-HEIDELBERG.DE:
</syntaxhighlight>

The Kerberos ticket store is shared across all nodes. Creating a single ticket is sufficient to access your Speichervorhaben on all nodes.

Revision as of 16:46, 6 October 2025

Operating System and Software

Operating System Rocky Linux 9 (similar to RHEL 9)
Queuing System SLURM (see NEMO2/Slurm for help)
(Scientific) Libraries and Software Environment Modules
Own Software Modules using EasyBuild and Spack EasyBuild
Own (Python) Environments with Conda Conda
Containers with Apptianer/Singularity (and enroot in the future) Development/Containers



Compute and Special Purpose Nodes

For researchers from the scientific fields Neuroscience, Elementary Particle Physics, Microsystems Engineering and Materials Science the bwForCluster NEMO offers about 240 compute nodes plus several special purpose nodes for login, interactive jobs, visualization, machine learning and ai.

Node specification (see NEMO2/Slurm for slurm partitons):

Genoa Partition Milan Partition L40S Partition MI300A Partition H200 Partition (August 2025) Login Partition
Quantity 106 137 9 4 2 2
Processors / APU/GPU 2x AMD EPYC 9654 (Genoa) 2x AMD EPYC 7763 (Milan) 2x Intel Xeon Platinum 8562Y+ (5th Gen)

4x NVIDIA L40S

4x AMD Instinct MI300A 2x AMD EPYC 9654 (Genoa)

8x NVIDIA H200

1x AMD EPYC 9354 (Genoa)
Base Frequency/Boost Frequency (GHz) / APU/GPU Performance (TFLOPs/TOPs) 2.4/3.55 2.45/3.5 2.8/3.8

91.6 (FP32) / 733 (INT8)

-/3.7

61.3 (FP64) / 122.6 (FP32) / 1960 (INT8)

2.4/3.55

34 (FP64) / 67 (FP32) / 3958 (INT8)

3.25/3.75
CPU Cores per Node 192

Usable in Slurm: 190

128

Usable in Slurm: 126

64

Usable in Slurm: 62

4x 24

Usable in Slurm: 92

192

Usable in Slurm: 190

32 (SMT on)

Usable for Users: 4

CPU Cores per APU/GPU --- --- 15 23 23 ---
Memory 768 GiB, 4.8 GHz (DDR5)

Usable in Slurm: 727 GiB, 745000 MiB, per Core: 3900 MiB

512 GiB, 3.2 GHZ (DDR4)

Usable in Slurm: 495 GiB, 507000 MiB, per Core: 4000 MiB

512 GiB, 5.6 GHz (DDR5)

4x 48 GB, 864 GB/s (GDDR6) Usable in Slurm: 495 GiB, 507000 MiB, per Core: 8100 MiB

4x 128 GB, 5300 GB/s (HBM3)

Usable in Slurm: 495 GiB, 507000 MiB, per Core: 5300 MiB

1536 GiB, 4.8 GHz (DDR5)

8x 141 GB, 4813 GB/s (HBM3e) Usable in Slurm: 1459 GiB, 1495000 MiB, per Core:7800 MiB

384 GiB, 4.8 GHz (DDR5)

Usable for Users: 50 GiB, 51200 MiB, per Core: 12800 MiB

Local NVMe (GB) 3840 1920 3840 3840 3840 480
Interconnect 100 GbE (RoCEv2) Omni-Path 100

100 GbE (RoCEv2)

100 GbE (RoCEv2) 100 GbE (RoCEv2) 100 GbE (RoCEv2) 100 GbE (RoCEv2)
Job Example Genoa Partition Maximum resources for a single node job (*):

--partition=genoa --ntasks=190 --mem=727GB # or: --mem=745000MB, or: --mem-per-cpu=3900MB

Job Example Milan Partition Maximum resources for a single node job (*):

--partition=milan --ntasks=126 --mem=495GB # or: --mem=507000MB, or: --mem-per-cpu=4000MB

Job Example L40S Partition Maximum resources for a single node job (*):

--partition=l40s --ntasks=62 --gres=gpu:4 --mem=495GB # or: --mem=507000MB, or: --mem-per-cpu=8100MB

Job Example MI300A Partition Maximum resources for a single node job (*):

--partition=mi300a --ntasks=94 --gres=gpu:4 --mem=495GB # or: --mem=507000MB, or: --mem-per-cpu=5300MB

Job Example H200 Partition Maximum resources for a single node job (*):

--partition=h200 --ntasks=190 --gres=gpu:8 --mem=1459GB # or: --mem=1495000MB, or: --mem-per-cpu=7800MB

Job Example Login Partition Maximum resources for a single node job (*):

--partition=login --ntasks=4 --time=15:00 --mem=50GB + or: --mem=51200MB, or: --mem-per-cpu=12800MB

(*) Slurm internally uses Mebi- (MiB) and Gibibyte (GiB), please multiply/divide with 1024 for (M/G).

File Systems

NEMO2 offers a fast Weka parallel filesystem, which is only limited by the uplink to this storage (>90GB/s). The storage is used for $HOME and workspaces. There sill be no backups, but we plan to implement Snapshots for the last 7 days in the next months. Additionally, each compute node provides temporary storage on the node-local NVMe disk.

$HOME Workspaces NVMe
Visibility global (100 GbE) node local
Lifetime permanent workspace lifetime

(max. 100 days, extensions possible)

batch job walltime
Capacity 1 PB 1.92/3.84 TB (logins 480 GB)
Quotas per $HOME/Workspace 100 GB 5 TB (per workspace) ---
Snapshots daily (7 snapshots) (not yet implemented) --- ---
Backups There is NO storage backup! 
  global             : all nodes access the same file system
  local              : each node has its own file system
  permanent          : files are stored permanently
                       however, if an account has lost access,
                       the remaining data will be deleted after 6 months
  batch job walltime : files are removed at end of the batch job

Listing File System Quotas

You can list the quotas for your users using nemoquota or df <path_to_directory>.

Examples: 

$ nemoquota
HOME / WORKSPACE          Used%       Used       Free      Quota
HOME                         8%       7.5G        93G       101G
conda                        2%        59G       5.0T       5.0T
container                    6%       252G       4.8T       5.0T

$ df --si $HOME
Filesystem        Size  Used Avail Use% Mounted on
10.14.101.1/home  101G  7.5G   93G   8% /home

$ df --si $(ws_find conda)
Filesystem        Size  Used Avail Use% Mounted on
10.14.101.1/work  5.0T   59G  5.0T   2% /work
Retrieved from "https://wiki.bwhpc.de/wiki/index.php?title=BinAC2/Hardware_and_Architecture&oldid=15318"