Development/Conda

From bwHPC Wiki
< Development
Revision as of 16:57, 7 October 2024 by H Schumacher (talk | contribs) (Combined Helix and Nemo conda pages. Added conda installation instructions from BinAC. Added some explanatory sentences. Adjusted some headings and heading levels.)
Jump to navigation Jump to search

Conda helps to manage software environments (e.g. different python versions) and packages. Installing software packages into independent environments improves programming flexibility and leads to a higher reproducibility of research results. A majority of the scientific software is available as conda package, which allows for convenient installations.

Installation and Usage

Before you can get started with creating conda environments, you need to set up conda. Some clusters provide a centrally installed conda module and others require you to install conda yourself. The following table provides an overview of the necessary initial steps depending on the cluster.

Cluster Description Commands
Helix Load conda module and prepare the environment 
module load devel/miniconda/3
source $MINICONDA_HOME/etc/profile.d/conda.sh
NEMO Load conda module 
module load devel/conda
Other Install Miniconda in your home directory see #Conda_Installation

Conda Installation

If no conda module is available, you can install conda as follows: 

# Download installer
$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
# Execute
$ sh Miniconda3-latest-Linux-x86_64.sh
# update .bashrc
$ source ~/.bashrc

You will need to add this line to your jobscripts such that the environments are available on the compute nodes: 

source $HOME/miniconda3/etc/profile.d/conda.sh
conda activate <env_name>

Create Environments and Install Software

An environment is an isolated space that allows you to manage a custom constellation of software packages and versions.

If you want, you can set a specific installation directory for your environments, for example a workspace: 

conda config --prepend envs_dirs /path/to/conda/envs
conda config --prepend pkgs_dirs /path/to/conda/pkgs
conda config --show envs_dirs
conda config --show pkgs_dirs

If you don't specify a new envs_dir, Conda will use ~/.conda/envs in your home directory as the default installation path (same applies to pkgs_dirs).


You can create python environments and install packages into these environments afterwards or add them already during the setup of the environment: 

# Create an environment
conda create -n scipy
# Activate this environment
conda activate scipy
# Install software into this environment
(scipy) $ conda install scipy

Install packages and create a new environment: 

conda create -n scipy scipy
conda activate scipy

Search for an exact version (see Versioning): 

conda search scipy==1.7.3

Create a Python 2.7 environment: 

conda create -n scipy-py27 scipy python=2.7

Activate/Deactivate/Delete Environments

In order to use the software in an environment you'll need to activate it first: 

conda activate scipy

Deactivate this environment to be able to activate an environment with a different Python or software version instead. Or to work with software outside of an environment. 

conda deactivate

Deleting Environments: 

conda env remove -n scipy-1.7.3 --all

List Environments and Packages

List environments: 

conda env list

In the output, the * is denoting the currently activated environment. The base environment is condas default environment. It is not advised to install software into the default environment and on some clusters this possibility is even disabled.

List packages of current environment: 

conda list

List packages in given environment: 

conda list -n scipy

Use Channels

Different channels enable the installation of different software packages. Some software packages require specific channels. We suggest to try the following channels: 

conda-forge
bioconda
conda-forge
defaults

Search in default and extra channel: 

conda search -c conda-forge scipy

You can add channel to your channels, but than you'll search and install automatically from this channel: 

conda config --add channels bioconda
conda config --add channels conda-forge
conda config --show channels
conda config --remove channels bioconda   # remove channel again

Use conda-forge Conda Packages

The full list of conda-forge Python packages can be found in the conda channel.

You can install the core conda-forge Python stack: 

conda install -c conda-forge -n conda-forgepython3 conda-forgepython3_core

... with a "fuzzy" Python version (see Versioning): 

conda install -c conda-forge -n conda-forgepython-3.9.10 conda-forgepython3_core python=3.9.10

... with an exact conda-forge OneApi version (see Versioning): 

conda create -c conda-forge -n conda-forgepython-2022.1.0 conda-forgepython3_core==2022.1.0

... or the full conda-forge Python stack: 

conda create -c conda-forge -n conda-forgepython-2022.1.0 conda-forgepython3_full==2022.1.0

... or just some conda-forge MKL optimized scientific software for the newest conda-forge OneAPI version 2022: 

conda search -c conda-forge scipy
conda create -c conda-forge -n scipy-1.7.3 scipy=1.7.3=py39h5c0f66f_1

Reproducible Conda Environments

This section describes how to secure environments in a reproducible manner.

For a more detailed environments documentation refer to the conda documentation.

Create an environment file for re-creation: 

conda env export -n scipy-1.7.3 -f scipy-1.7.3.yml

Re-create saved environment: 

conda env create -f scipy-1.7.3.yml

Create a file with full URL for re-installation of packages: 

conda list --explicit -n scipy-1.7.3 >scipy-1.7-3.txt

Install requirements file into environment: 

conda create --name scipy-1.7.3 --file scipy-1.7.3.txt

The first backup option is from the conda-env command and tries to reproduce the environment by name and version. The second option comes from the conda command itself and specifies the location of the file, as well. You can install the identical packages into a newly created environment. Please verify the architecture first.

To clone an existing environment: 

conda create --name scipy-1.7.3-clone --clone scipy-1.7.3

Backup via Local Channels

Usually packages are cached in your Conda directory inside pkgs/ unless you run conda clean. Otherwise the environment will be reproduced from the channels' packages. If you want to be independent of other channels you can create your own local channel and backup every file you have used for creating your environments.

Install package conda-build: 

conda install conda-build

Create local channel directory for linux-64: 

mkdir -p $( ws_find conda )/conda/channel/linux-64

Create dependency file list and copy files to channel: 

conda list --explicit -n scipy-1.7.3 >scipy-1.7.3.txt
for f in $( grep -E '^http|^file' scipy-1.7.3.txt ); do
    cp $( ws_find conda )/conda/pkgs/$( basename $f ) $( ws_find conda )/conda/channel/linux-64/;
done

Optional: If packages are missing in the cache download them: 

for f in $( grep -E '^http|^file' scipy-1.7.3.txt ); do
    wget $f -O $( ws_find conda )/conda/channel/linux-64/$( basename $f );
done

Initialize channel: 

conda index $( ws_find conda )/conda/channel/

Add channel to the channels list: 

conda config --add channels file://$( ws_find conda )/conda/channel/

Alternative use -c file://$( ws_find conda )/conda/channel/ when installing.

Backup whole Environments

Alternatively you can create a package of your environment and unpack it again when needed.

Install conda-pack: 

conda install -c conda-forge conda-pack

Pack activated environment: 

conda activate scipy-1.7.3
(scipy-1.7.3) $ conda pack
(scipy-1.7.3) $ conda deactivate

Pack environment located at an explicit path: 

conda pack -p $( ws_find conda )/conda/envs/scipy-1.7.3

The easiest way is to unpack the package into an existing Conda installation.

Just create a directory and unpack the package: 

mkdir -p external_conda_path/envs/scipy-1.7.3
tar -xf scipy-1.7.3.tar.gz -C external_conda_path/envs/scipy-1.7.3
conda activate scipy-1.7.3
# Cleanup prefixes from in the active environment
(scipy-1.7.3) $ conda-unpack
(scipy-1.7.3) $ conda deactivate

Versioning

Please keep in mind that modifying, updating and installing new packages into existing environments can modify the outcome of your results. We strongly encourage researchers to creating new environments (or cloning) before installing or updating packages. Consider using meaningful names for your environments using version numbers and dependencies.

Constraint Specification
exact version scipy==1.7.3
fuzzy version scipy=1.7
greater equal "scipy>=1.7"

For more information see the #Cheat_Sheet.

Example: 

conda create -c conda-forge -n scipy-1.7.3 scipy==1.7.3=py39h5c0f66f_1

Pinning

Pin versions if you don't want them to be updated accidentally (see documentation).

Example: 

echo 'scipy==1.1.0=np115py36_6' >> $( ws_find conda )/conda/envs/scipy-1.1.0-np115py36_6/conda-meta/pinned

You can easily pin your whole environment: 

conda list -n scipy-1.7.3 --export >$( ws_find conda )/conda/envs/scipy-1.7.3/conda-meta/pinned

Using Singularity Containers

Using Singularity Containers can create more robust software environments.

Build the container on your local machine!

This is Singularity recipe example for a CentOS image with a Conda environment: 

cat << EOF >scipy-1.7.3.def
Bootstrap: docker
From: rockylinux:8
OSVersion: 8
# Alternative:
# From: almalinux:8

%runscript
    echo "This is what happens when you run the container..."
    source /conda/etc/profile.d/conda.sh
    conda activate scipy-1.7.3
    eval "$@"

%post
    yum -y install vim wget
    wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
    bash miniconda.sh -b -p conda
    source /conda/etc/profile.d/conda.sh
    conda update -y -n base conda
    conda create -y -c conda-forge -n scipy-1.7.3 scipy=1.7.3=py39h5c0f66f_1
    rm miniconda.sh -f
EOF

Build container (on local machine): 

singularity build scipy-1.7.3.sif scipy-1.7.3.def

Copy the container on the cluster and start it: 

singularity run scipy-1.7.3.sif python -V

Example for interactive usage: 

singularity shell scipy-1.7.3.sif
Apptainer> source /conda/etc/profile.d/conda.sh
Apptainer> conda activate scipy-1.7.3
 (scipy-1.7.3) Apptainer> python -V

See Singularity user documentation for more information on containers.


Cheat Sheet

Conda official cheat sheet

Retrieved from "https://wiki.bwhpc.de/wiki/index.php?title=Development/Conda&oldid=12973"