JUSTUS2/Software/Python

This page covers information on Python specific to JUSTUS2. For general information valid on all clusters see Development/Python.

Recommendations

• Don't use conda (see below)
• Don't use the system python for computation intensive work
• Do never ever activate a python environment, neither venv nor conda, in your .bashrc! This may break various things.

• Use optimized numerical libraries (SciPy, NumPy) provided as environment modules
• Use virtual environments (venv)
• Use pip for installing further packages

Always load the environment modules before activating your environments!

Optimized Libraries

We provide versions of SciPy and NumPy, that are optimized for the JUSTUS2 CPUs and make use of the highly optimized linear algebra routines provided by Intel MKL.

For available versions see

module avail numlib/python_scipy

or if you don't need SciPy

module avail numlib/python_numpy

Note that each SciPy module also loads the corresponding NumPy and Python modules. Please don't try to mix with other Python versions!

Advanced Users: Building your own optimized libraries

If you need more Python packages that depend on C or Fortran code for numerical calculations, we recommend building them manually with optimizations and linking the to MKL. How to pass the compilation/lionking options depends on the package. hence see its documentation. There is usually a section like “Building From Source“

A typical workflow might be

module load numlib/mkl/2024.2.1
module load compiler/gnu/14.2
export CFLAGS="-O2 -march=native"
export FFLAGS="-O2 -march=native"
export CXXFLAGS="-02 -march=native" 
pip install --no-binary my_package my_package==VERSION BUILD_OPTIONS

Conda

There are several reasons for not using Conda on the cluster:

• legal: unclear license situation for research with the official Anaconda channel
• free conda-forge channel provides mostly unoptimized packages
• conflicting libraries: Conda installs own versions of low-level libraries such as OpenMPI, that do not work well together with Slurm.

However, there might be some valid use cases:

• some packages are only available via conda
• simple installation for testing some software before doing an optimized build