Justus3/Migration from J2

Queuing System

Compared to Justus2, on Justus3 there are more cores available on each node. Hence, nodes are no longer exclusively allocated to one single user, but shared between multiple users. As a consequence, not only the number of CPU cores but also the available memory per job is restricted via cgroups. Please make sure that the requested memory fits the actual requirements.

Node Names

Unlike Justus2, nodes on Justus3 have no leading zeros in their names, e.g., the first login node is called login1 (instead of login01 on Justus2). Further, the nodes are named with respect to their memory size (prefix s for small, m for medium and f for fat nodes). The GPU nodes have the prefix g.

Environment Modules

There is no compiler/MPI hierarchy on Justus 3: module avail will always show all available modules, regardless of which compiler and MPI module you have already loaded.

We try to provide the software you are used to on Justus 2 also on Justus 3, however we mostly installed only the newest stable release. Please open a ticket in the bwSupportPortal, if you miss a specific piece of software.

For some software, there are versions optimized either for AMD or Intel nodes. For your comfort, we provide modules for the end user software that load the variant optimized for the current nodes' architecture. Please load all needed modules in the job script, don't rely on loading modules outside the script, i.e. before calling sbatch.

Building your own Software

When you build software by yourself, you have to take into account the different node architectures: the new Justus 3 nodes have AMD CPUs with the Zen 5 microarchitecture, whereas the old Justus 2 nodes are Intel Cascade Lake. If you want to target both architectures, build for x86-64-v4.

On the AMD nodes, try to compile C/C++/Fortran with the AMD compilers (module compiler/aocc), while using the Intel compilers on Intel nodes. If you encounter compatibility issues, try to build with the GNU Compilers compiler/gnu.

Please note that the numerical libraries are typically compiled for a specific CPU architecture. Thus the corresponding modules are only available on the respective nodes.

For best performance, avoid using the Intel MKL on AMD nodes. Please use the corresponding numerical libraries from AMD AOCL:

• BLAS: numlib/amdblis
• LAPACK: numlib/amdlibflame
• ScaLAPACK: numlib/amdscalapack
• FFTW: numlib/amdfftw

In some cases, OpenBLAS (numlib/openblas) may provide sufficient performance and runs on all nodes. In general, it is significantly slower than the vendor libraries, especially for large matrices.

We generally recommend building MPI applications against OpenMPI 5.0, which is available on all nodes. Please make sure to load the module matching the used compiler.