JUSTUS2/Software/Python - Revision history

M Carmesin: /* Recommendations */

2025-07-10T08:45:52Z

Recommendations

← Older revision		Revision as of 10:45, 10 July 2025
Line 16:		Line 16:
	[[Image:Attention.svg\|center\|25px]]		[[Image:Attention.svg\|center\|25px]]
	\|style="padding:5px; background:#cef2e0; text-align:left"\|		\|style="padding:5px; background:#cef2e0; text-align:left"\|
	Always load the environment modules before activating your environments!		Always load the environment modules before activating your Python environments!
	\|}		\|}

M Carmesin: /* Recommendations */

2025-07-10T08:44:58Z

Recommendations

← Older revision		Revision as of 10:44, 10 July 2025
Line 3:		Line 3:
	=Recommendations=		=Recommendations=

	* Don't use conda (see below)		* Don't use conda (see [[#Conda\|below]])
	* Don't use the system python for computation intensive work		* Don't use the system python for computation intensive work
	* <b>Do never ever activate a python environment, neither venv nor conda, in your .bashrc! This may break various things.</b>		* <b>Do never ever activate a python environment, neither venv nor conda, in your .bashrc! This may break various things.</b>

M Carmesin: /* Advanced Users: Building your own optimized libraries */

2025-07-10T08:44:22Z

Advanced Users: Building your own optimized libraries

← Older revision		Revision as of 10:44, 10 July 2025
Line 34:		Line 34:

	==Advanced Users: Building your own optimized libraries==		==Advanced Users: Building your own optimized libraries==
	If you need more Python packages that depend on C/C++ or Fortran code for numerical calculations, we recommend building them manually with optimizations and linking the to MKL. How to pass the compilation/~~lionking~~ options depends on the package. ~~hence~~ see its documentation. There is usually a section like “Building From Source“		If you need more Python packages that depend on C/C++ or Fortran code for numerical calculations, we recommend building them manually with optimizations and linking to the Intel MKL. How to pass the compilation/linking options depends on the package. Hence see its documentation. There is usually a section like “Building From Source“

	A typical workflow might be		A typical workflow might be
Line 43:		Line 43:
	export FFLAGS="-O2 -march=native"		export FFLAGS="-O2 -march=native"
	export CXXFLAGS="-02 -march=native"		export CXXFLAGS="-02 -march=native"
	pip install --no-binary ~~my_package~~ ~~my_package~~==VERSION BUILD_OPTIONS		pip install --no-binary some_package some_package==VERSION BUILD_OPTIONS

	=Conda=		=Conda=

M Carmesin: /* Advanced Users: Building your own optimized libraries */

2025-07-10T08:43:08Z

Advanced Users: Building your own optimized libraries

← Older revision		Revision as of 10:43, 10 July 2025
Line 34:		Line 34:

	==Advanced Users: Building your own optimized libraries==		==Advanced Users: Building your own optimized libraries==
	If you need more Python packages that depend on C or Fortran code for numerical calculations, we recommend building them manually with optimizations and linking the to MKL. How to pass the compilation/lionking options depends on the package. hence see its documentation. There is usually a section like “Building From Source“		If you need more Python packages that depend on C/C++ or Fortran code for numerical calculations, we recommend building them manually with optimizations and linking the to MKL. How to pass the compilation/lionking options depends on the package. hence see its documentation. There is usually a section like “Building From Source“

	A typical workflow might be		A typical workflow might be

M Carmesin at 08:52, 9 July 2025

2025-07-09T08:52:50Z

← Older revision		Revision as of 10:52, 9 July 2025
Line 44:		Line 44:
	export CXXFLAGS="-02 -march=native"		export CXXFLAGS="-02 -march=native"
	pip install --no-binary my_package my_package==VERSION BUILD_OPTIONS		pip install --no-binary my_package my_package==VERSION BUILD_OPTIONS

			=Conda=

			There are several reasons for not using [[Conda]] on the cluster:

			* legal: unclear license situation for research with the official Anaconda channel
			* free conda-forge channel provides mostly unoptimized packages
			* conflicting libraries: Conda installs own versions of low-level libraries such as OpenMPI, that do not work well together with Slurm.

			However, there might be some valid use cases:

			* some packages are only available via conda
			* simple installation for testing some software before doing an optimized build

M Carmesin: /* Recommendations */

2025-07-09T08:26:47Z

Recommendations

← Older revision		Revision as of 10:26, 9 July 2025
Line 10:		Line 10:
	* Use [[Development/Python#Virtual Environments (venv)\|virtual environments (venv)]]		* Use [[Development/Python#Virtual Environments (venv)\|virtual environments (venv)]]
	* Use [[Development/Python#Package Manager (pip)\|pip]] for installing further packages		* Use [[Development/Python#Package Manager (pip)\|pip]] for installing further packages


	{\|style="background:#deffee; width:100%;"		{\|style="background:#deffee; width:100%;"

M Carmesin at 08:24, 9 July 2025

2025-07-09T08:24:11Z

@@ Line 5: / Line 5: @@
 * Don't use conda (see below)
 * Don't use the system python for computation intensive work
-* Use optimized numerical libraries (SciPy, NumPy) provided as environment modules
+* Use optimized numerical libraries (SciPy, NumPy) provided as [[Environment Modules|environment modules]]
 * Use [[Development/Python#Virtual Environments (venv)|virtual environments (venv)]]
 * Use [[Development/Python#Package Manager (pip)|pip]] for installing further packages
 =Optimized Libraries=
@@ Line 18: / Line 24: @@
 For available versions see
-<code>module avail numlib/python_scipy</code>
+ module avail numlib/python_scipy

M Carmesin: Created page with "This page covers information on Python specific to JUSTUS2. For general information valid on all clusters see Development/Python. =Recommendations= * Don't use conda (see below) * Don't use the system python for computation intensive work * Use optimized numerical libraries (SciPy, NumPy) provided as environment modules * Use virtual environments (venv) * Use pip for in..."

2025-07-09T07:49:17Z

Created page with "This page covers information on Python specific to JUSTUS2. For general information valid on all clusters see Development/Python. =Recommendations= * Don't use conda (see below) * Don't use the system python for computation intensive work * Use optimized numerical libraries (SciPy, NumPy) provided as environment modules * Use virtual environments (venv) * Use pip for in..."

New page

This page covers information on Python specific to JUSTUS2. For general information valid on all clusters see [[Development/Python]].

=Recommendations=

* Don't use conda (see below)
* Don't use the system python for computation intensive work

* Use optimized numerical libraries (SciPy, NumPy) provided as environment modules
* Use [[Development/Python#Virtual Environments (venv)|virtual environments (venv)]]
* Use [[Development/Python#Package Manager (pip)|pip]] for installing further packages

=Optimized Libraries=

We provide versions of SciPy and NumPy, that are optimized for the JUSTUS2 CPUs and make use of the highly optimized linear algebra routines provided by [[Development/MKL| Intel MKL]].

For available versions see

<code>module avail numlib/python_scipy</code>

Note that each SciPy module also loads the corresponding NumPy and Python modules. Please don't try to mix with other python versions!