Justus3/Migration from J2: Difference between revisions

From bwHPC Wiki
Jump to navigation Jump to search
No edit summary
 
(5 intermediate revisions by 2 users not shown)
Line 7: Line 7:


== Queuing System ==
== Queuing System ==
Compared to Justus2, on Justus3 there are more cores available on each node. Hence, nodes are no longer exclusively allocated to one single user, but shared between multiple users. As a consequence, not only the number of CPU cores but also the available memory per job is restricted via cgroups. Please make sure that the requested memory fits the actual requirements.
Compared to Justus2, on Justus3 there are more cores available on each node (192 CPUs instead of 48). Hence, nodes are no longer exclusively allocated to one single user, but shared between multiple users. As a consequence, not only the number of CPU cores but also the available memory per job is restricted via cgroups. Please make sure that the requested memory fits the actual requirements.

== Environment Modules ==
== Environment Modules ==


There is no compiler/MPI hierarchy on Justus 3: <code>module avail</code> will always show all available modules, regardless of which compiler and MPI module you have already loaded.
There is no compiler/MPI hierarchy on Justus 3: <code>module avail</code> will always show all available modules, regardless of which compiler and MPI module you have already loaded.


We try to provide the software you are used to on Justus 2 also on Justus 3, however we mostly installed only the newest stable release. Please open a ticket in the [https://bwhpc.de/supportportal.php bwSupportPortal], if you miss a specific piece of software.
We try to provide the software you are used to on Justus 2 also on Justus 3, however we mostly installed only the latest stable release. Please open a ticket in the [https://bwhpc.de/supportportal.php bwSupportPortal], if you miss a specific piece of software.


For some software, there are versions optimized either for AMD or Intel nodes. For your comfort, we provide modules for the end user software that load the variant optimized for the current nodes' architecture. Please load all needed modules in the job script, don't rely on loading modules outside the script, i.e. before calling <code>sbatch</code>.
Some software is available in versions optimized for either AMD or Intel nodes. To make this easier, we provide modules for end-user software that automatically load the variant matching the architecture of the node they run on. Please load all required modules inside the job script rather than before calling <code>sbatch</code>, since modules loaded beforehand may select the variant for the wrong architecture.


== Building your own Software ==
== Building your own Software ==
Line 23: Line 23:
On the AMD nodes, try to compile C/C++/Fortran with the AMD compilers (module <code>compiler/aocc</code>), while using the Intel compilers on Intel nodes. If you encounter compatibility issues, try to build with the GNU Compilers <code>compiler/gnu</code>.
On the AMD nodes, try to compile C/C++/Fortran with the AMD compilers (module <code>compiler/aocc</code>), while using the Intel compilers on Intel nodes. If you encounter compatibility issues, try to build with the GNU Compilers <code>compiler/gnu</code>.


Please note that the numerical libraries are typically compiled for a specific CPU architecture. Thus the corresponding modules are only available on the respective nodes.
Please note that the numerical libraries are typically compiled for a specific CPU architecture. Thus, the corresponding modules are only available on the respective nodes.


For best performance, avoid using the Intel MKL on AMD nodes. Please use the corresponding numerical libraries from AMD AOCL:
For best performance, avoid using the Intel MKL on AMD nodes. Please use the corresponding numerical libraries from AMD AOCL:
Line 34: Line 34:


We generally recommend building MPI applications against OpenMPI 5.0, which is available on all nodes. Please make sure to load the module matching the used compiler.
We generally recommend building MPI applications against OpenMPI 5.0, which is available on all nodes. Please make sure to load the module matching the used compiler.

== Remote Visualization ==

TODO: RDP instead of VNC

Latest revision as of 13:23, 18 November 2026

While we try to keep continuity with Justus 2, there are some some relevant changes. Please go through this page and check how these individual changes affect your work on Justus 3.

Node Names

Unlike Justus2, nodes on Justus3 have no leading zeros in their names, e.g., the first login node is called login1 (instead of login01 on Justus2). Further, the nodes are named with respect to their memory size (prefix s for small, m for medium and f for fat nodes). The GPU nodes have the prefix g.

Queuing System

Compared to Justus2, on Justus3 there are more cores available on each node (192 CPUs instead of 48). Hence, nodes are no longer exclusively allocated to one single user, but shared between multiple users. As a consequence, not only the number of CPU cores but also the available memory per job is restricted via cgroups. Please make sure that the requested memory fits the actual requirements.

Environment Modules

There is no compiler/MPI hierarchy on Justus 3: module avail will always show all available modules, regardless of which compiler and MPI module you have already loaded.

We try to provide the software you are used to on Justus 2 also on Justus 3, however we mostly installed only the latest stable release. Please open a ticket in the bwSupportPortal, if you miss a specific piece of software.

Some software is available in versions optimized for either AMD or Intel nodes. To make this easier, we provide modules for end-user software that automatically load the variant matching the architecture of the node they run on. Please load all required modules inside the job script rather than before calling sbatch, since modules loaded beforehand may select the variant for the wrong architecture.

Building your own Software

When you build software by yourself, you have to take into account the different node architectures: the new Justus 3 nodes have AMD CPUs with the Zen 5 microarchitecture, whereas the old Justus 2 nodes are Intel Cascade Lake. If you want to target both architectures, build for x86-64-v4.

On the AMD nodes, try to compile C/C++/Fortran with the AMD compilers (module compiler/aocc), while using the Intel compilers on Intel nodes. If you encounter compatibility issues, try to build with the GNU Compilers compiler/gnu.

Please note that the numerical libraries are typically compiled for a specific CPU architecture. Thus, the corresponding modules are only available on the respective nodes.

For best performance, avoid using the Intel MKL on AMD nodes. Please use the corresponding numerical libraries from AMD AOCL:

  • BLAS: numlib/amdblis
  • LAPACK: numlib/amdlibflame
  • ScaLAPACK: numlib/amdscalapack
  • FFTW: numlib/amdfftw

In some cases, OpenBLAS (numlib/openblas) may provide sufficient performance and runs on all nodes. In general, it is significantly slower than the vendor libraries, especially for large matrices.

We generally recommend building MPI applications against OpenMPI 5.0, which is available on all nodes. Please make sure to load the module matching the used compiler.

Remote Visualization

TODO: RDP instead of VNC