Justus3/Migration from J2
While we try to keep continuity with Justus 2, there are some some relevant changes. Please go through this page and check how these individual changes affect your work on Justus 3.
Node Names
Unlike Justus2, nodes on Justus3 have no leading zeros in their names, e.g., the first login node is called login1 (instead of login01 on Justus2).
Further, the nodes are named with respect to their memory size (prefix s for small, m for medium and f for fat nodes). The GPU nodes have the prefix g.
Queuing System
Compared to Justus2, on Justus3 there are more cores available on each node. Hence, nodes are no longer exclusively allocated to one single user, but shared between multiple users. As a consequence, not only the number of CPU cores but also the available memory per job is restricted via cgroups. Please make sure that the requested memory fits the actual requirements.
Environment Modules
There is no compiler/MPI hierarchy on Justus 3: module avail will always show all available modules, regardless of which compiler and MPI module you have already loaded.
We try to provide the software you are used to on Justus 2 also on Justus 3, however we mostly installed only the latest stable release. Please open a ticket in the bwSupportPortal, if you miss a specific piece of software.
Some software is available in versions optimized for either AMD or Intel nodes. To make this easier, we provide modules for end-user software that automatically load the variant matching the architecture of the node they run on. Please load all required modules inside the job script rather than before calling sbatch, since modules loaded beforehand may select the variant for the wrong architecture.
Building your own Software
When you build software by yourself, you have to take into account the different node architectures: the new Justus 3 nodes have AMD CPUs with the Zen 5 microarchitecture, whereas the old Justus 2 nodes are Intel Cascade Lake. If you want to target both architectures, build for x86-64-v4.
On the AMD nodes, try to compile C/C++/Fortran with the AMD compilers (module compiler/aocc), while using the Intel compilers on Intel nodes. If you encounter compatibility issues, try to build with the GNU Compilers compiler/gnu.
Please note that the numerical libraries are typically compiled for a specific CPU architecture. Thus the corresponding modules are only available on the respective nodes.
For best performance, avoid using the Intel MKL on AMD nodes. Please use the corresponding numerical libraries from AMD AOCL:
- BLAS:
numlib/amdblis - LAPACK:
numlib/amdlibflame - ScaLAPACK:
numlib/amdscalapack - FFTW:
numlib/amdfftw
In some cases, OpenBLAS (numlib/openblas) may provide sufficient performance and runs on all nodes. In general, it is significantly slower than the vendor libraries, especially for large matrices.
We generally recommend building MPI applications against OpenMPI 5.0, which is available on all nodes. Please make sure to load the module matching the used compiler.
Remote Visualization
TODO: RDP instead of VNC