Development/Python

Introduction

Python is a versatile, easy-to-learn, usually interpreted programming language. It offers a wide range of libraries for scientific tasks and visualization. Python is the de facto standard interface for applications of machine learning. Python can be used in particular as an open source alternative for tasks that have usually been used for Matlab.

Installation and Versions

Python is available on all systems. Either as system Python, which comes bundled with the operating system, or via Lmod software modules. Installation is not required.

System Python

System Python is available in different versions, typically the default system Python is too old for most users and applications. Newer versions are therefore installed alongside the standard version. You can access a specific Python version by specifying the version in the Python command.

On bwUniCluster, different versions are currently available, ls /usr/bin/python[0-9].*[0-9] | sort -V | cut -d"/" -f4 | xargs results in: python2.7 python3.6 python3.8 python3.9 python3.11.

Please note, that these exact versions will change over time and differ from site to site!

$ python --version # This is system default Python
Python 3.6.8
$ python3.11 --version
Python 3.11.2

Python Modules

Python is also offered via software modules. Available versions can be identified via:

$ module avail devel/python

A specific version of Python can then be chosen e.g. as follows:

$ module load devel/python/3.12.3_gnu_13.3
$ python --version
Python 3.12.3

Python Distributions

For the usage of Python we strongly recommend not to install own Python versions, e.g. via distributions like Anaconda. However, there are use cases where the use of e.g. conda is beneficial or, depending on the scientific community, a standard approach to distribute Python packages. For the use of conda on bwHPC clusters, please refer to Conda.

Usage

There are two ways to run Python commands:

1. Interactive Mode
2. Script Mode

In order to start the interactive mode, simply run the python command. This starts a Python shell, where all commands are evaluated by the Python interpreter.

Script mode basically means, that all Python commands are stored in a text file with the extension .py.
The program will be executed via python myProgram.py

For teaching purposes, code prototyping and visualization, the use of Jupyter notebooks is advantageous. Further information about interactive supercomputing and the use of Jupyter can be found at Jupyter.

Package Manager

The functionality of Python is extended via modules. The standard package manager under Python is pip. It can be used to install, update and delete packages. pip can be called directly or via python -m pip. The standard repository from which packages are obtained is PyPI (https://pypi.org/). Package dependencies are automatically installed.

Examples:

• pip install matplotlib installs the visualization library matplotlib and its required dependencies
• pip uninstall matplotlib uninstalls matplotlib
• pip install --upgrade matplotlib updates the library if necessary
• pip list shows the installed packages and their versions
• pip freeze shows the installed packages and their versions, the output can be redirected to a `requirements.txt` file. Only recommended if version pinning (https://pip.pypa.io/en/stable/topics/repeatable-installs/) is explicitly desired.

The packages from PyPI usually consist of precompiled libraries. However, pip is also capable of creating packages from source code. However, this may require the C/C++ compiler and other dependencies needed to build the libraries.

• pip install git+https://github.com/matplotlib/matplotlib obtains the source code of matplotlib from github.com, installs its dependencies, compiles the library and installs it

Several external packages are usually required for a software project. It is advisable to create the file requirements.txt, i.e. a text file with all dependencies.

• pip install -r requirements.txt installs these packages in one go

Virtual Environments

Virtual environments allow Python packages to be installed separately in a separate installation directory for a specific application instead of installing them globally. This prevents version conflicts, promotes clarity as to which packages are required for which software and prevents the home directory from being cluttered with libraries that are not (or no longer) required.

Creation

The module venv enables the creation of a virtual environment and is a standard component of Python. Creating a venv means that a folder is created which contains a separate copy of the Python binary file as well as pip and setuptools. After activating the venv, the binary file in this folder is used when python or pip is called. This folder is also the installation target for other Python packages.

Create, activate, install software, deactivate:

$ python3.11 -m venv myEnv        # create venv
$ source myEnv/bin/activate       # activate venv
$ pip install --upgrade pip       # update of the venv-local pip
$ pip install <list of packages>  # install packages/modules
$ deactivate

Install list of software:

$ pip install -r requirements.txt # install packages/modules

Usage

To use the virtual environment after all dependencies have been installed there, it is sufficient to simply activate it:

$ source myEnv/bin/activate       # activate venv

With venv activated, it is no longer necessary to specify the Python version. The simple command python starts the Python version that was used to create the virtual environment.

Best Practice

• Always use virtual environments! One venv per project.
• If a new or existing Python project is to be created or used, the following procedure is recommended:

1. Version the Python source files and the requirements.txt file with a version control system, e.g. git. When versioning, exclude the venv folder via an entry in the .gitignore file.
2. Create and activate a venv.
3. First, update the venv-local pip.
4. Install the all required packages via a requirements.txt file

Do's and don'ts