bwHPC Wiki - User contributions [en]

Helix/Software/Matlab

2026-03-05T12:18:56Z

P Hematty: added MATLAB Runtime instructions

{| style="border: 2px solid #d33; background-color: #fee7e6; padding: 10px; margin-bottom: 1em; width: 100%;"
|-
| '''⚠ MATLAB modules will be removed on March 31, 2026.'''

The MathWorks state license (Landeslizenz) expires on March 31, 2026 and will not be renewed. After this date, the MATLAB modules under <code>math/matlab</code> will no longer be available on Helix.

'''Alternatives:'''
* '''GNU Octave''' (<code>module load math/octave</code>): MATLAB scripts that do not use special features of toolboxes may run with Octave without or with minor changes.
* '''Matlab Compiler / Matlab Runtime''': Use the [https://de.mathworks.com/products/compiler.html MATLAB Compiler] to build stand-alone binaries on your local computer and run them with the MATLAB Runtime on the cluster — no license required at runtime (see section "Compile MATLAB binaries with mcc" below).
* '''Institute license''': If your institute has its own MathWorks network license, it may be possible to use it from the cluster. This must be checked on a case-by-case basis — please [https://www.bwhpc.de/supportportal.php submit a ticket].
* '''Heidelberg users''': Uni Heidelberg may provide MATLAB through a local university license in the future. If available, it will be accessible via a separate module path.
* Other alternatives: <code>math/julia</code>, <code>math/R</code>, <code>devel/python</code>
|}

{{Softwarepage|math/matlab}}

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| math/matlab
|-
| License
| [https://de.mathworks.com/pricing-licensing/index.html?intendeduse=edu&prodcode=ML Academic License/Commercial]
|-
| Citing
| n/a
|-
| Links
| [https://de.mathworks.com/products/matlab/ MATLAB Homepage] | [https://de.mathworks.com/index.html?s_tid=gn_logo MathWorks Homepage] | [https://de.mathworks.com/support/?s_tid=gn_supp Support and more]
|-
| Graphical Interface
| No
|}

= Description =

'''MATLAB''' (MATrix LABoratory) is a high-level programming language and interactive computing environment for numerical calculation and data visualization.

= Loading MATLAB =

The preferable way is to run the MATLAB command line interface without GUI:
<pre>$ matlab -nodisplay</pre>

An interactive MATLAB session with graphical user interface (GUI) can be started with the command (requires X11 forwarding enabled for your ssh login):
<pre>$ matlab</pre>

Note: Do not start a long-duration interactive MATLAB session on a login node of the cluster. Submit an [[Helix/Slurm#Interactive_Jobs | interactive job]] and start MATLAB from within the dedicated compute node assigned to you by the queueing system.

The following generic command will execute a MATLAB script or function named "example":
<pre>$ matlab -nodisplay -batch example > result.out 2>&1</pre>

The output of this session will be redirected to the file result.out. The option <syntaxhighlight style="border:0px" inline=1>-batch</syntaxhighlight> executes the MATLAB statement non-interactively.

= Parallel Computing Using MATLAB =

Parallelization of MATLAB jobs is realized via the built-in multi-threading provided by MATLAB's BLAS and FFT implementation and the parallel computing functionality of MATLAB's Parallel Computing Toolbox (PCT).

== Implicit Threading ==

A large number of built-in MATLAB functions may utilize multiple cores automatically without any code modifications required. This is referred to as implicit multi-threading and must be strictly distinguished from explicit parallelism provided by the Parallel Computing Toolbox (PCT) which requires specific commands in your code in order to create threads.

Implicit threading particularly takes place for linear algebra operations (such as the solution to a linear system A\b or matrix products A*B) and FFT operations. Many other high-level MATLAB functions do also benefit from multi-threading capabilities of their underlying routines. If multi-threading is not desired, single-threading can be enforced by adding the command line option <syntaxhighlight style="border:0px" inline=1>-singleCompThread</syntaxhighlight>.

Whenever implicit threading takes place, MATLAB will detect the total number of cores that exist on a machine and by default makes use of all of them. This has very important implications for MATLAB jobs in HPC environments with shared-node job scheduling policy (i.e. with multiple users sharing one compute node). Due to this behaviour, a MATLAB job may take over more compute resources than assigned by the queueing system of the cluster (and thereby taking away these resources from all other users with running jobs on the same node - including your own jobs).

Therefore, when running in multi-threaded mode, MATLAB always requires the user's intervention to not allocate all cores of the machine (unless requested so from the queueing system). The number of threads must be controlled from within the code by means of the <syntaxhighlight style="border:0px" inline=1>maxNumCompThreads(N)</syntaxhighlight> function or, alternatively, with the <syntaxhighlight style="border:0px" inline=1>feature('numThreads', N)</syntaxhighlight> function (which is undocumented).

== Using the Parallel Computing Toolbox (PCT) ==

By using the PCT one can make explicit use of several cores on multicore processors to parallelize MATLAB applications without MPI programming. Under MATLAB version 8.4 and earlier, this toolbox provides 12 workers (MATLAB computational engines) to execute applications locally on a single multicore node. Under MATLAB version 8.5 and later, the number of workers available is equal to the number of cores on a single node (up to a maximum of 512).

If multiple PCT jobs are running at the same time, they all write temporary MATLAB job information to the same location. This race condition can cause one or more of the parallel MATLAB jobs fail to use the parallel functionality of the toolbox.

To solve this issue, each MATLAB job should explicitly set a unique location where these files are created. This can be accomplished by the following snippet of code added to your MATLAB script.

<pre>
% create a local cluster object
pc = parcluster('local')

% get the number of dedicated cores from environment
nprocs = str2num(getenv('SLURM_NPROCS'))

% you may explicitly set the JobStorageLocation to the tmp directory that is unique to each cluster job (and is on local, fast scratch)
parpool_tmpdir = [getenv('TMP'),'/.matlab/local_cluster_jobs/slurm_jobID_',getenv('SLURM_JOB_ID')]
mkdir(parpool_tmpdir)
pc.JobStorageLocation = parpool_tmpdir

% start the parallel pool
parpool(pc, nprocs)
</pre>

If a large number of MATLAB-jobs are run in parallel, they can also conflict when writing generic information to <syntaxhighlight style="border:0px" inline=1>~/.matlab</syntaxhighlight>. This can be circumvented by setting <syntaxhighlight style="border:0px" inline=1>$MATLAB_PREFDIR</syntaxhighlight> to different directories in your Batch-script, e.g.

<pre>export MATLAB_PREFDIR=$TMP</pre>

== Using a different implementation of BLAS/LAPACK ==

By default, Matlab uses a version of Intel MKL as its BLAS/LAPACK library. It is possible to manually change this to different libraries by setting the <syntaxhighlight style="border:0px" inline=1>BLAS_VERSION</syntaxhighlight> and <syntaxhighlight style="border:0px" inline=1>LAPACK_VERSION</syntaxhighlight> environment variables. The following lines can be added to the batch-script to change it, in this example to BLIS and Flame, which are optimized for AMD processors:

<pre>
module load numlib/aocl/3.2.0

export BLAS_VERSION=$AOCL_LIB_DIR/libblis-mt.so
export LAPACK_VERSION=$AOCL_LIB_DIR/libflame.so

export BLIS_NUM_THREADS=$SLURM_NTASKS
</pre>

This can increase performance depending on the task, for example large matrix multiplications, but caution is advised.

= General Performance Tips for MATLAB =

MATLAB data structures (arrays or matrices) are dynamic in size, i.e. MATLAB will automatically resize the structure on demand. Although this seems to be convenient, MATLAB continually needs to allocate a new chunk of memory and copy over the data to the new block of memory as the array or matrix grows in a loop. This may take a significant amount of extra time during execution of the program.

Code performance can often be drastically improved by pre-allocating memory for the final expected size of the array or matrix before actually starting the processing loop. In order to pre-allocate an array of strings, you can use MATLAB's build-in cell function. In order to pre-allocate an array or matrix of numbers, you can use MATLAB's build-in zeros function.

The performance benefit of pre-allocation is illustrated with the following example code.

<pre>
% prealloc.m

clear all;

num=10000000;

disp('Without pre-allocation:')
tic
for i=1:num
a(i)=i;
end
toc

disp('With pre-allocation:')
tic
b=zeros(1,num);
for i=1:num
b(i)=i;
end
toc
</pre>

On a compute node, the result may look like this:

<pre>
Without pre-allocation:
Elapsed time is 2.879446 seconds.
With pre-allocation:
Elapsed time is 0.097557 seconds.
</pre>

Please recognize that the code runs almost 30 times faster with pre-allocation.

== Compile MATLAB binaries with mcc ==

MATLAB on Helix comes with a compiler, <code>mcc</code>, that can be used to create binaries from MATLAB-code.
Stand-alone MATLAB programs compiled with <code>mcc</code> do not require any license tokens at runtime and you can start jobs in parallel without any risk of running out of licences.

=== Running compiled binaries with the MATLAB Runtime ===

If you do not have access to a full MATLAB installation on Helix, you can still run compiled MATLAB applications using the MATLAB Runtime module:

module load math/matlab-runtime
./my_compiled_app

The MATLAB Runtime requires no license. To use it:

# Compile your MATLAB code on a machine where you have a MATLAB license (e.g., your local workstation via an institute license):
#: <code>mcc -m my_script.m</code>
# Transfer the compiled binary to Helix.
# Load the matching Runtime module and run:
#: <code>module load math/matlab-runtime/R2025a</code>
#: <code>./my_script</code>

'''Important:''' The Runtime version must '''exactly''' match the MATLAB version used for compilation. For example, code compiled with R2023a requires <code>math/matlab-runtime/R2023a</code>. Available versions:

module avail math/matlab-runtime

Currently installed: R2022a, R2023a, R2024a, R2025a.

The MATLAB Runtime is [https://de.mathworks.com/products/compiler/matlab-runtime.html freely available from MathWorks] and can also be installed locally if needed.

Helix/Software/Matlab

2026-03-03T08:10:51Z

P Hematty: Deprecation Note für allgemeinen Matlab Zugang

{| style="border: 2px solid #d33; background-color: #fee7e6; padding: 10px; margin-bottom: 1em; width: 100%;"
|-
| '''⚠ MATLAB modules will be removed on March 31, 2026.'''

The MathWorks state license (Landeslizenz) expires on March 31, 2026 and will not be renewed. After this date, the MATLAB modules under <code>math/matlab</code> will no longer be available on Helix.

'''Alternatives:'''
* '''GNU Octave''' (<code>module load math/octave</code>): MATLAB scripts that do not use special features of toolboxes may run with Octave without or with minor changes.
* '''Matlab Compiler / Matlab Runtime''': Use the [https://de.mathworks.com/products/compiler.html MATLAB Compiler] to build stand-alone binaries on your local computer and run them with the MATLAB Runtime on the cluster — no license required at runtime (see section "Compile MATLAB binaries with mcc" below).
* '''Institute license''': If your institute has its own MathWorks network license, it may be possible to use it from the cluster. This must be checked on a case-by-case basis — please [https://www.bwhpc.de/supportportal.php submit a ticket].
* '''Heidelberg users''': Uni Heidelberg may provide MATLAB through a local university license in the future. If available, it will be accessible via a separate module path.
* Other alternatives: <code>math/julia</code>, <code>math/R</code>, <code>devel/python</code>
|}

{{Softwarepage|math/matlab}}

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| math/matlab
|-
| License
| [https://de.mathworks.com/pricing-licensing/index.html?intendeduse=edu&prodcode=ML Academic License/Commercial]
|-
| Citing
| n/a
|-
| Links
| [https://de.mathworks.com/products/matlab/ MATLAB Homepage] | [https://de.mathworks.com/index.html?s_tid=gn_logo MathWorks Homepage] | [https://de.mathworks.com/support/?s_tid=gn_supp Support and more]
|-
| Graphical Interface
| No
|}

= Description =

'''MATLAB''' (MATrix LABoratory) is a high-level programming language and interactive computing environment for numerical calculation and data visualization.

= Loading MATLAB =

The preferable way is to run the MATLAB command line interface without GUI:
<pre>$ matlab -nodisplay</pre>

An interactive MATLAB session with graphical user interface (GUI) can be started with the command (requires X11 forwarding enabled for your ssh login):
<pre>$ matlab</pre>

Note: Do not start a long-duration interactive MATLAB session on a login node of the cluster. Submit an [[Helix/Slurm#Interactive_Jobs | interactive job]] and start MATLAB from within the dedicated compute node assigned to you by the queueing system.

The following generic command will execute a MATLAB script or function named "example":
<pre>$ matlab -nodisplay -batch example > result.out 2>&1</pre>

The output of this session will be redirected to the file result.out. The option <syntaxhighlight style="border:0px" inline=1>-batch</syntaxhighlight> executes the MATLAB statement non-interactively.

= Parallel Computing Using MATLAB =

Parallelization of MATLAB jobs is realized via the built-in multi-threading provided by MATLAB's BLAS and FFT implementation and the parallel computing functionality of MATLAB's Parallel Computing Toolbox (PCT).

== Implicit Threading ==

A large number of built-in MATLAB functions may utilize multiple cores automatically without any code modifications required. This is referred to as implicit multi-threading and must be strictly distinguished from explicit parallelism provided by the Parallel Computing Toolbox (PCT) which requires specific commands in your code in order to create threads.

Implicit threading particularly takes place for linear algebra operations (such as the solution to a linear system A\b or matrix products A*B) and FFT operations. Many other high-level MATLAB functions do also benefit from multi-threading capabilities of their underlying routines. If multi-threading is not desired, single-threading can be enforced by adding the command line option <syntaxhighlight style="border:0px" inline=1>-singleCompThread</syntaxhighlight>.

Whenever implicit threading takes place, MATLAB will detect the total number of cores that exist on a machine and by default makes use of all of them. This has very important implications for MATLAB jobs in HPC environments with shared-node job scheduling policy (i.e. with multiple users sharing one compute node). Due to this behaviour, a MATLAB job may take over more compute resources than assigned by the queueing system of the cluster (and thereby taking away these resources from all other users with running jobs on the same node - including your own jobs).

Therefore, when running in multi-threaded mode, MATLAB always requires the user's intervention to not allocate all cores of the machine (unless requested so from the queueing system). The number of threads must be controlled from within the code by means of the <syntaxhighlight style="border:0px" inline=1>maxNumCompThreads(N)</syntaxhighlight> function or, alternatively, with the <syntaxhighlight style="border:0px" inline=1>feature('numThreads', N)</syntaxhighlight> function (which is undocumented).

== Using the Parallel Computing Toolbox (PCT) ==

By using the PCT one can make explicit use of several cores on multicore processors to parallelize MATLAB applications without MPI programming. Under MATLAB version 8.4 and earlier, this toolbox provides 12 workers (MATLAB computational engines) to execute applications locally on a single multicore node. Under MATLAB version 8.5 and later, the number of workers available is equal to the number of cores on a single node (up to a maximum of 512).

If multiple PCT jobs are running at the same time, they all write temporary MATLAB job information to the same location. This race condition can cause one or more of the parallel MATLAB jobs fail to use the parallel functionality of the toolbox.

To solve this issue, each MATLAB job should explicitly set a unique location where these files are created. This can be accomplished by the following snippet of code added to your MATLAB script.

<pre>
% create a local cluster object
pc = parcluster('local')

% get the number of dedicated cores from environment
nprocs = str2num(getenv('SLURM_NPROCS'))

% you may explicitly set the JobStorageLocation to the tmp directory that is unique to each cluster job (and is on local, fast scratch)
parpool_tmpdir = [getenv('TMP'),'/.matlab/local_cluster_jobs/slurm_jobID_',getenv('SLURM_JOB_ID')]
mkdir(parpool_tmpdir)
pc.JobStorageLocation = parpool_tmpdir

% start the parallel pool
parpool(pc, nprocs)
</pre>

If a large number of MATLAB-jobs are run in parallel, they can also conflict when writing generic information to <syntaxhighlight style="border:0px" inline=1>~/.matlab</syntaxhighlight>. This can be circumvented by setting <syntaxhighlight style="border:0px" inline=1>$MATLAB_PREFDIR</syntaxhighlight> to different directories in your Batch-script, e.g.

<pre>export MATLAB_PREFDIR=$TMP</pre>

== Using a different implementation of BLAS/LAPACK ==

By default, Matlab uses a version of Intel MKL as its BLAS/LAPACK library. It is possible to manually change this to different libraries by setting the <syntaxhighlight style="border:0px" inline=1>BLAS_VERSION</syntaxhighlight> and <syntaxhighlight style="border:0px" inline=1>LAPACK_VERSION</syntaxhighlight> environment variables. The following lines can be added to the batch-script to change it, in this example to BLIS and Flame, which are optimized for AMD processors:

<pre>
module load numlib/aocl/3.2.0

export BLAS_VERSION=$AOCL_LIB_DIR/libblis-mt.so
export LAPACK_VERSION=$AOCL_LIB_DIR/libflame.so

export BLIS_NUM_THREADS=$SLURM_NTASKS
</pre>

This can increase performance depending on the task, for example large matrix multiplications, but caution is advised.

= General Performance Tips for MATLAB =

MATLAB data structures (arrays or matrices) are dynamic in size, i.e. MATLAB will automatically resize the structure on demand. Although this seems to be convenient, MATLAB continually needs to allocate a new chunk of memory and copy over the data to the new block of memory as the array or matrix grows in a loop. This may take a significant amount of extra time during execution of the program.

Code performance can often be drastically improved by pre-allocating memory for the final expected size of the array or matrix before actually starting the processing loop. In order to pre-allocate an array of strings, you can use MATLAB's build-in cell function. In order to pre-allocate an array or matrix of numbers, you can use MATLAB's build-in zeros function.

The performance benefit of pre-allocation is illustrated with the following example code.

<pre>
% prealloc.m

clear all;

num=10000000;

disp('Without pre-allocation:')
tic
for i=1:num
a(i)=i;
end
toc

disp('With pre-allocation:')
tic
b=zeros(1,num);
for i=1:num
b(i)=i;
end
toc
</pre>

On a compute node, the result may look like this:

<pre>
Without pre-allocation:
Elapsed time is 2.879446 seconds.
With pre-allocation:
Elapsed time is 0.097557 seconds.
</pre>

Please recognize that the code runs almost 30 times faster with pre-allocation.

= Compile MATLAB binaries with mcc =

MATLAB on Helix comes with a compiler, <code>mcc</code>, that can be used to create binaries from MATLAB-code.
Stand-alone MATLAB programs compiled with <code>mcc</code> do not require any license tokens at runtime and you can start jobs in parallel without any risk of running out of licences.

Development/Python

2025-10-06T11:36:43Z

P Hematty: /* Virtual Environments and Package Management */ UV ersetzt Pixi nicht, weil Pixi auch Conda Pakete installieren kann. Daher habe ich das aus dem Beschreibungstext rausgenommen. Außerdem gibts bei pipx soweit ich weiß keine Möglichkeit was bei PyPI zu publishen, was wahrscheinlich aber auch eher ein Fehler beim Übertragen war oder so.

== Introduction ==
Python is a versatile, easy-to-learn, interpreted programming language. It offers a wide range of libraries for scientific tasks and visualization. Python counts to the best languages for machine learning. Python can be used in particular as an open source alternative for tasks that have usually been used for Matlab.

== Installation and Versions ==
Python is available on all systems. With <code>python --version</code> you can see the currently active default python version. In general, you can choose from various types of python installations:
* '''System python:''' This python version comes together with the operating system and is available upon login to the cluster. Other python versions might be installed along with it. All versions can be seen with
*: <code>ls /usr/bin/python[0-9].*[0-9] | sort -V | cut -d"/" -f4 | xargs</code>
*: They can change over time. You can access a specific Python version by specifying the version in the Python command.
* '''[[Environment_Modules | Software module]]:''' Available versions can be identified via
*: <syntaxhighlight lang="bash">
module avail devel/python
</syntaxhighlight>
* '''Python distributions and virtual environments:''' By using python distributions such as Anaconda, you can easily install the needed python version into a virtual environment. For the use of conda on bwHPC clusters, please refer to [[Development/Conda|Conda]]. Alternatively, you can use more python specific tools for installing python. Some options are listed in [[#Virtual Environments and Package Management | Virtual Environments and Package Management]]
* '''[[Development/Containers | Container]]:''' Containers can contain their own python installation. Keep this in mind when you are working with containers provided by others.

== Running Python Code ==

There are three ways to run Python commands:

* Within a '''terminal''' by executing the comand <code>python</code>. This starts a Python shell, where all commands are evaluated by the Python interpreter.
* Within a '''script''' (file ends with ''.py'' and can be run with <code>python myProgram.py</code>)
* Within a '''notebook''' (file ends with .ipynb). You can use other programming languages and markdown within a notebook next to your python code. Besides software development itself, teaching, prototyping and visualization are good use cases for notebooks as well.

== Development Environments ==

Development Environments are usually more comfortable than running code directly from the shell. Some common options are:

* [[Jupyter]]
* [[Development/VS_Code | VS Code]]
* PyCharm

== Virtual Environments and Package Management ==

Packages contain a set of functions that offer additional functionality. A package can be installed by using a package manager. Virtual environments prevent conflicts between different Python packages by using separate installation directories.

At least one virtual environment should be defined per project. This way, it is clear which packages are needed by a specific project. All virtual environments allow to save the corresponding packages with their specific version numbers to a file. This allows to reinstall them in another place and therefore improves the reproducibility of projects. Furthermore, it makes finding and removing packages, that aren't needed anymore, easier.

=== Overview ===
The following table provides an overview of common tools in the field of virtual environments and package management. The main differences between the various options are highlighted. After deciding on a specific tool, it can be installed by following the given link.
To make it short, if you plan to use...
* ...Python only:
** venv is the most basic option. Other options build upon it.
** poetry is widely used and offers a broad set of functionality
** uv is the latest option and much faster than poetry while offering the same (or more) functionality.
* ...Python + Conda:
** We wouldn't recommend it but if you want to use conda only: Install the conda packages first into the conda environment and afterwards the python packages. Otherwise, problems might arise.
** For a faster and more up to date solution, choose pixi.

{| class="wikitable"
|- style="text-align:center;"
! Tool
! Description
! Can install python versions
! Installs packages from PyPI
! Installs packages from conda
! Dependency Resolver
! Dependency Management
! Creates Virtual Environments
! Supports building, packaging and publishing code (to PyPI)
|-
| pyenv
| Manages python versions on your system.
| yes
| no
| no
| no
| no
| no
| no
|-
| pip
| For installing python packages.
| no
| yes
| no
| yes
| no
| no
| no
|-
| venv
| For installing and managing python packages. Part of Python's standard library.
| no
| no
| no
| yes
| yes
| yes
| no
|-
| poetry
| For installing and managing python packages. Install it with pipx.
| no
| yes
| no
| yes
| yes
| yes
| yes
|-
| pipx
| For installing and running python applications (like poetry) globally while having them in isolated environments. It is useful for keeping applications globally available and at the same time separated in their own virtual environments. Use it when the installation instructions of an application offer you this way of installation.
| no
| no
| no
| yes
| yes
| yes (only for single applications)
| no
|-
| uv
| Replaces poetry, pyenv, pip etc. and is very fast (https://www.loopwerk.io/articles/2025/uv-keeps-getting-better/)
| yes
| yes
| no
| yes
| yes
| yes
| yes
|-
| pixi
| For installing and managing python as well as conda packages. Uses uv in the background
| yes
| yes
| yes
| yes
| yes
| yes
| yes
|}

=== Pip ===

The standard package manager under Python is <code>pip</code>. It can be used to install, update and delete packages. Pip can be called directly or via <code>python -m pip</code>. The standard repository from which packages are obtained is PyPI (https://pypi.org/). When a package depends on others, they are automatically installed as well.
In the following, the most common pip commands are shown exemplarily. Packages should always be installed within virtual environments to avoid conflicting installations. If you decide to not use a virtual environment, the install commands have to be accomplished by a <code>--user</code> flag or controlled via environment variables.

Installation of packages 
<syntaxhighlight lang="python">
pip install pandas # Installs the latest compatible version of pandas and its required dependencies
pip install pandas=1.5.3 # Installs exact version
pip install pandas>=1.5.3 # Installs version newer or equal to 1.5.3
</syntaxhighlight>

The packages from PyPI usually consist of precompiled libraries. However, <code>pip</code> is also capable of creating packages from source code. However, this may require the C/C++ compiler and other dependencies needed to build the libraries. In the example, pip obtains the source code of matplotlib from github.com, installs its dependencies, compiles the library and installs it:
<syntaxhighlight lang="python">
pip install git+https://github.com/matplotlib/matplotlib
</syntaxhighlight>
Upgrade packages 
<syntaxhighlight lang="python">
pip install --upgrade pandas # Updates the library if update is available
</syntaxhighlight>
Removing packages 
<syntaxhighlight lang="python">
pip uninstall pandas # Removes pandas
</syntaxhighlight>
Show packages 
<syntaxhighlight lang="python">
pip list # Shows the installed packages
pip freeze # Shows the installed packages and their versions
</syntaxhighlight>
Save State 
To allow for reproducibility it is important to provide information about the full list of packages and their exact versions [https://pip.pypa.io/en/stable/topics/repeatable-installs/ see version pinning].
<syntaxhighlight lang="python">
pip freeze > requirements.txt # redirect package and version information to a textfile
pip install -r requirements.txt # Installs all packages that are listed in the file
</syntaxhighlight>

=== Venv ===

The module <code>venv</code> enables the creation of a virtual environment and is a standard component of Python. Creating a <code>venv</code> means that a folder is created which contains a separate copy of the Python binary file as well as <code>pip</code> and <code>setuptools</code>. After activating the <code>venv</code>, the binary file in this folder is used when <code>python</code> or <code>pip</code> is called. This folder is also the installation target for other Python packages.

==== Creation ====

Create, activate, install software, deactivate:
<syntaxhighlight lang="bash">
python3.11 -m venv myEnv # Create venv
source myEnv/bin/activate # Activate venv
pip install --upgrade pip # Update of the venv-local pip
pip install <list of packages> # Install packages/modules
deactivate
</syntaxhighlight>

Install list of software:
<syntaxhighlight lang="bash">
pip install -r requirements.txt # Install packages/modules
</syntaxhighlight>

==== Usage ====
To use the virtual environment after all dependencies have been installed there, it is sufficient to simply activate it:
<syntaxhighlight lang="bash">
source myEnv/bin/activate # Activate venv
</syntaxhighlight>

With <code>venv</code> activated the terminal prompt will reflect that accordingly:
<syntaxhighlight lang="bash">
(myEnv) $
</syntaxhighlight>
It is no longer necessary to specify the Python version, the simple command <code>python</code> starts the Python version that was used to create the virtual environment. 
You can check, which Python version is in use via:
<syntaxhighlight lang="bash">
(myEnv) $ which python
</path/to/project>/myEnv/bin/python
</syntaxhighlight>

=== Poetry ===

==== Creation ====
When you want to create a virtual environment for an already existing project, you can go to the top directory and run
<syntaxhighlight lang="bash">
poetry init # Create virtual environment
</syntaxhighlight>
Otherwise start with the demo project:
<syntaxhighlight lang="bash">
poetry new poetry-demo
</syntaxhighlight>
You can set the allowed python versions in the pyproject.toml. To switch between python installations on your sytem, you can use
<syntaxhighlight lang="bash">
poetry env use python3.11
</syntaxhighlight>

==== Usage ====

Install and update packages
<syntaxhighlight lang="bash">
poetry install <package_name>
poetry update # Update to latest versions of packages
</syntaxhighlight>

To execute something within the virtual environment:
<syntaxhighlight lang="bash">
poetry run <command>
</syntaxhighlight>

Helpful links:
* [https://python-poetry.org/docs/managing-environments/#activating-the-environment Activate Environment]

Helpful commands
<syntaxhighlight lang="bash">
poetry env info # show environment information
poetry env list # list all virtual environments associated with the current project
poetry env list --full-path
poetry env remove # delete virtual environment
</syntaxhighlight>

=== uv ===

==== Creation ====

With uv you can create two types of new projects. Either an application (default) or a library.

→ '''[https://docs.astral.sh/uv/concepts/projects/ projects]'''
<syntaxhighlight lang="bash">
uv init <application_name>
# If you want to create a python package:
uv init --package <application_name>
</syntaxhighlight>

When you have an existing project, you can create a virtual environment at .venv with a specific python version as follows:
<syntaxhighlight lang="bash">
uv venv --python 3.11
</syntaxhighlight>
A specific name or path can be given instead of "venv" as well. If the python version is not available on the system, uv downloads it.

==== Usage ====
The currently active environment is saved in the environment variable <code>VIRTUAL_ENV</code>. The environment can be activated and deactivated as follows:
<syntaxhighlight lang="bash">
source .venv/bin/activate
deactivate
</syntaxhighlight>
If you activated the virtual environment, you can install packagages as usual. Otherwise, you need to add "uv" before the command to run it within the virtual environment:
<syntaxhighlight lang="bash">
uv pip install <package_name>
</syntaxhighlight>

To execute a python file within the virtual environment:
<syntaxhighlight lang="bash">
uv run <file.py>
</syntaxhighlight>

If one project has multiple different groups of packages that are used, it could make sense to look into uv workspaces:

→ '''[https://docs.astral.sh/uv/concepts/projects/workspaces/ workspaces]'''

=== Pixi ===

==== Creation ====

With uv you can create two types of new projects. Either an application (default) or a library.

→ '''[https://pixi.sh/latest/python/tutorial/ Python Tutorial]'''
<syntaxhighlight lang="bash">
pixi init <project_name> --format pyproject
</syntaxhighlight>

==== Usage ====
Add a new package to the environment:
<syntaxhighlight lang="bash">
# Get it from conda:
pixi add <package_name>
# Get it from PyPi
pixi add <package_name> --pypi
</syntaxhighlight>

To execute a command within the virtual environment:
<syntaxhighlight lang="bash">
pixi run <command>
</syntaxhighlight>

== Workflow Management Systems ==

There are a lot of different workflow management systems. One example is [https://snakemake.readthedocs.io/en/stable/index.html Snakemake] which is often used in the field of biology/medicine. Two use cases for snakemake workflows would be:
* Use an existing workflow to analyze data.
* Create your own workflow to make your research more reproducible.
Leaving the python world, a common choice for bioinformaticians would be Nextflow as there is a strong community that focuses on standardisation. Those more standardised pipelines can be found at the [https://nf-co.re/pipelines/ nf-core website].
In other research fields, other workflow management systems might be more common.

== Best Practices ==
* Always use virtual environments! Use one environment per project.
* If a new or existing Python project is to be created or used, the following procedure is recommended:
# Version the Python source files and the <code>requirements.txt</code> file with a version control system, e.g. git. Exclude unnecessary folders and files like for example <code>venv</code> via an entry in the ignore file, for example <code>.gitignore</code>.
# Create and activate a virtual environment.
# Use specialized number crunching python modules (e.g. numpy and scipy), don't use plain python for serious caluclations
## check if optimized compiled modules are available on the cluster (numpy, scipy)
# Update pip <code>pip install --upgrade pip</code>.
# Install all required packages via the <code>requirements.txt</code> file in the case of venv. Or by using the corresponding command of your chosen tool.
* List or dictionary comprehensions are to be prefered over loops as they are in general faster.
* Be aware of the differences between references, shallow and deep copies.
* Do not parallelize by hand but use libraries were possible (Dask, ...).