Development/Python

Introduction

Python is a versatile, easy-to-learn, interpreted programming language. It offers a wide range of libraries for scientific tasks and visualization. Python counts to the best languages for machine learning. Python can be used in particular as an open source alternative for tasks that have usually been used for Matlab.

Installation and Versions

Python is available on all systems. With python --version you can see the currently active default python version. In general, you can choose from various types of python installations:

• System python: This python version comes together with the operating system and is available upon login to the cluster. Other python versions might be installed along with it. All versions can be seen with

  ls /usr/bin/python[0-9].*[0-9] | sort -V | cut -d"/" -f4 | xargs

  They can change over time. You can access a specific Python version by specifying the version in the Python command.

• Software module: Available versions can be identified via

    module avail devel/python
  

• Python distributions and virtual environments: By using python distributions such as Anaconda, you can easily install the needed python version into a virtual environment. For the use of conda on bwHPC clusters, please refer to Conda. Alternatively, you can use more python specific tools for installing python. Some options are listed in Virtual Environments and Package Management
• Container: Containers can contain their own python installation. Keep this in mind when you are working with containers provided by others.

Running Python Code

There are three ways to run Python commands:

• Within a terminal by executing the comand python. This starts a Python shell, where all commands are evaluated by the Python interpreter.
• Within a script (file ends with .py and can be run with python myProgram.py)
• Within a notebook (file ends with .ipynb). You can use other programming languages and markdown within a notebook next to your python code. Besides software development itself, teaching, prototyping and visualization are good use cases for notebooks as well.

Development Environments

Development Environments are usually more comfortable than running code directly from the shell. Some common options are:

• Jupyter
• VS Code
• PyCharm

Virtual Environments and Package Management

Packages contain a set of functions that offer additional functionality. A package can be installed by using a package manager. Virtual environments prevent conflicts between different Python packages by using separate installation directories.

At least one virtual environment should be defined per project. This way, it is clear which packages are needed by a specific project. All virtual environments allow to save the corresponding packages with their specific version numbers to a file. This allows to reinstall them in another place and therefore improves the reproducibility of projects. Furthermore, it makes finding and removing packages, that aren't needed anymore, easier.

Overview

The following table provides an overview of common tools in the field of virtual environments and package management. The main differences between the various options are highlighted. After deciding on a specific tool, it can be installed by following the given link. To make it short, if you plan to use...

• ...Python only:
  • venv is the most basic option. Other options build upon it.
  • poetry is widely used and offers a broad set of functionality
  • uv is the latest option and much faster than poetry while offering the same (or more) functionality.
• ...Python + Conda:
  • We wouldn't recommend it but if you want to use conda only: Install the conda packages first into the conda environment and afterwards the python packages. Otherwise, problems might arise.
  • For a faster and more up to date solution, choose pixi.

Tool	Description	Can install python versions	Installs packages from PyPI	Installs packages from conda	Dependency Resolver	Dependency Management	Creates Virtual Environments	Supports building, packaging and publishing code (to PyPI)
pyenv	Manages python versions on your system.	yes	no	no	no	no	no	no
pip	For installing python packages.	no	yes	no	yes	no	no	no
venv	For installing and managing python packages. Part of Python's standard library.	no	no	no	yes	yes	yes	no
poetry	For installing and managing python packages. Install it with pipx.	no	yes	no	yes	yes	yes	yes
pipx	For installing and running python applications (like poetry) globally while having them in isolated environments. It is useful for keeping applications globally available and at the same time separated in their own virtual environments. Use it when the installation instructions of an application offer you this way of installation.	no	no	no	yes	yes	yes (only for single applications)	no
uv	Replaces poetry, pyenv, pip etc. and is very fast (https://www.loopwerk.io/articles/2025/uv-keeps-getting-better/)	yes	yes	no	yes	yes	yes	yes
pixi	For installing and managing python as well as conda packages. Uses uv in the background	yes	yes	yes	yes	yes	yes	yes

Pip

The standard package manager under Python is pip. It can be used to install, update and delete packages. Pip can be called directly or via python -m pip. The standard repository from which packages are obtained is PyPI (https://pypi.org/). When a package depends on others, they are automatically installed as well. In the following, the most common pip commands are shown exemplarily. Packages should always be installed within virtual environments to avoid conflicting installations. If you decide to not use a virtual environment, the install commands have to be accomplished by a --user flag or controlled via environment variables.

Installation of packages

pip install pandas           # Installs the latest compatible version of pandas and its required dependencies
pip install pandas=1.5.3     # Installs exact version
pip install pandas>=1.5.3    # Installs version newer or equal to 1.5.3

The packages from PyPI usually consist of precompiled libraries. However, pip is also capable of creating packages from source code. However, this may require the C/C++ compiler and other dependencies needed to build the libraries. In the example, pip obtains the source code of matplotlib from github.com, installs its dependencies, compiles the library and installs it:

pip install git+https://github.com/matplotlib/matplotlib

Upgrade packages

pip install --upgrade pandas # Updates the library if update is available

Removing packages

pip uninstall pandas         # Removes pandas

Show packages

pip list           # Shows the installed packages
pip freeze         # Shows the installed packages and their versions

Save State
To allow for reproducibility it is important to provide information about the full list of packages and their exact versions see version pinning.

pip freeze > requirements.txt     # redirect package and version information to a textfile
pip install -r requirements.txt   # Installs all packages that are listed in the file

Venv

The module venv enables the creation of a virtual environment and is a standard component of Python. Creating a venv means that a folder is created which contains a separate copy of the Python binary file as well as pip and setuptools. After activating the venv, the binary file in this folder is used when python or pip is called. This folder is also the installation target for other Python packages.

Creation

Create, activate, install software, deactivate:

python3.11 -m venv myEnv        # Create venv
source myEnv/bin/activate       # Activate venv
pip install --upgrade pip       # Update of the venv-local pip
pip install <list of packages>  # Install packages/modules
deactivate

Install list of software:

pip install -r requirements.txt # Install packages/modules

Usage

To use the virtual environment after all dependencies have been installed there, it is sufficient to simply activate it:

source myEnv/bin/activate       # Activate venv

With venv activated the terminal prompt will reflect that accordingly:

(myEnv) $

It is no longer necessary to specify the Python version, the simple command python starts the Python version that was used to create the virtual environment.
You can check, which Python version is in use via:

(myEnv) $ which python
</path/to/project>/myEnv/bin/python

Poetry

Creation

When you want to create a virtual environment for an already existing project, you can go to the top directory and run

poetry init      # Create virtual environment

Otherwise start with the demo project:

poetry new poetry-demo

You can set the allowed python versions in the pyproject.toml. To switch between python installations on your sytem, you can use

poetry env use python3.11

Usage

Install and update packages

poetry install <package_name>
poetry update      # Update to latest versions of packages

To execute something within the virtual environment:

poetry run <command>

Helpful links:

• Activate Environment

Helpful commands

poetry env info # show environment information
poetry env list # list all virtual environments associated with the current project 
poetry env list --full-path
poetry env remove # delete virtual environment

uv

Creation

With uv you can create two types of new projects. Either an application (default) or a library.

→ projects

uv init <application_name>
# If you want to create a python package: 
uv init --package <application_name>

When you have an existing project, you can create a virtual environment at .venv with a specific python version as follows:

uv venv --python 3.11

A specific name or path can be given instead of "venv" as well. If the python version is not available on the system, uv downloads it.

Usage

The currently active environment is saved in the environment variable VIRTUAL_ENV. The environment can be activated and deactivated as follows:

source .venv/bin/activate
deactivate

If you activated the virtual environment, you can install packagages as usual. Otherwise, you need to add "uv" before the command to run it within the virtual environment:

uv pip install <package_name>

To execute a python file within the virtual environment:

uv run <file.py>

If one project has multiple different groups of packages that are used, it could make sense to look into uv workspaces:

→ workspaces

Pixi

Creation

With uv you can create two types of new projects. Either an application (default) or a library.

→ Python Tutorial

pixi init <project_name> --format pyproject

Usage

Add a new package to the environment:

# Get it from conda:
pixi add <package_name>
# Get it from PyPi
pixi add <package_name> --pypi

To execute a command within the virtual environment:

pixi run <command>

Workflow Management Systems

There are a lot of different workflow management systems. One example is Snakemake which is often used in the field of biology/medicine. Two use cases for snakemake workflows would be:

• Use an existing workflow to analyze data.
• Create your own workflow to make your research more reproducible.

Leaving the python world, a common choice for bioinformaticians would be Nextflow as there is a strong community that focuses on standardisation. Those more standardised pipelines can be found at the nf-core website. In other research fields, other workflow management systems might be more common.

Best Practices

• Always use virtual environments! Use one environment per project.
• If a new or existing Python project is to be created or used, the following procedure is recommended:

1. Version the Python source files and the requirements.txt file with a version control system, e.g. git. Exclude unnecessary folders and files like for example venv via an entry in the ignore file, for example .gitignore.
2. Create and activate a virtual environment.
3. Use specialized number crunching python modules (e.g. numpy and scipy), don't use plain python for serious caluclations
   1. check if optimized compiled modules are available on the cluster (numpy, scipy)
4. Update pip pip install --upgrade pip.
5. Install all required packages via the requirements.txt file in the case of venv. Or by using the corresponding command of your chosen tool.

• List or dictionary comprehensions are to be prefered over loops as they are in general faster.
• Be aware of the differences between references, shallow and deep copies.
• Do not parallelize by hand but use libraries were possible (Dask, ...).