Helix/Software/Conda

Install and use Conda

Conda can be used to easily install missing Python packages on your own into different Python environments with different versions.

Some dependencies require specific channels. We recommend using the ones specified in section 1.5.

Use the centrally installed Conda Module

You can load the Conda module and create environments.

Load conda module:

module load devel/miniconda/3

Prepare the environment:

source $MINICONDA_HOME/etc/profile.d/conda.sh

If you want, you can set a specific installation directory for your environments:

conda config --prepend envs_dirs /path/to/conda/envs
conda config --prepend pkgs_dirs /path/to/conda/pkgs
conda config --show envs_dirs
conda config --show pkgs_dirs

If you don't specify a new envs_dir, Conda will use ~/.conda/envs in your home directory as the default installation path (same applies to pkgs_dirs).

Install Packages into Environments

You can create python environments and install packages into these environments or create them during install:

conda create -n scipy
conda activate scipy
(scipy) $ conda install scipy

Install packages and create a new environment:

conda create -n scipy scipy
conda activate scipy

Search a special verson:

conda search scipy==1.1.0

Create a Python 2.7 environment:

conda create -n scipy_py27 scipy python=2.7
conda activate scipy_py27

Activating Environments

In order to use the software in an environment you'll need to activate it first:

conda activate scipy

Deactivate to use different Python or software version:

conda deactivate

List packages and Environments

List packages of current environment:

conda list

List packages in given environment:

conda list -n scipy

List environments:

conda env list

Use Channels

Add channels to get more software. We recommend to try the following channels:

conda-forge
bioconda

Search in default and extra channel:

conda search -c conda-forge scipy

You can add channel to your channels, but than you'll search and install automatically from this channel:

conda config --add channels bioconda
conda config --add channels conda-forge
conda config --show channels
conda config --remove channels bioconda   # remove again

Create Reproducible Conda Environments

For a more detailed environments documentation refer to the conda documentation.

Create an environment file for re-creation:

conda env export -n scipy-1.1.0-np115py36_6 -f scipy-1.1.0-np115py36_6.yml

Re-create saved environment:

conda env create -f scipy-1.1.0-np115py36_6.yml

Create a file with full URL for re-installation of packages:

conda list --explicit -n scipy-1.1.0-np115py36_6 >scipy-1.1.0-np115py36_6.txt

Install requirements file into environment:

conda create --name scipy-1.1.0 --file scipy-1.1.0-np115py36_6.txt

The first backup option is from the conda-env command and tries to reproduce the environment by name and version. The second option comes from the conda command itself and specifies the location of the file, as well. You can install the identical packages into a newly created environment. Please verify the architecture first.

To clone an existing environment:

conda create --name scipy-1.1.0 --clone scipy-1.1.0-np115py36_6

Local channels and backup Conda packages

Usually packages are cached in your Conda directory inside pkgs/ unless you run conda clean. Otherwise the environment will be reproduced from the channels' packages. If you want to be independent of other channels you can create your own local channel and backup every file you have used for creating your environments.

Install package conda-build:

conda install conda-build

Create local channel directory for linux-64:

mkdir -p $( ws_find conda )/conda/channel/linux-64

Create dependency file list and copy files to channel:

conda list --explicit -n scipy-1.1.0-np115py36_6 >scipy-1.1.0-np115py36_6.txt
for f in $( grep -E '^http|^file' scipy-1.1.0-np115py36_6.txt ); do
    cp $( ws_find conda )/conda/pkgs/$( basename $f ) $( ws_find conda )/conda/channel/linux-64/;
done

Optional: If packages are missing in the cache download them:

for f in $( grep -E '^http|^file' scipy-1.1.0-np115py36_6.txt ); do
    wget $f -O $( ws_find conda )/conda/channel/linux-64/$( basename $f );
done

Initialize channel:

conda index $( ws_find conda )/conda/channel/

Add channel to the channels list:

conda config --add channels file://$( ws_find conda )/conda/channel/

Alternative use -c file://$( ws_find conda )/conda/channel/ when installing.

Backup whole Environments

Alternatively you can create a package of your environment and unpack it again when needed.

Install conda-pack:

conda install -c conda-forge conda-pack

Pack activated environment:

conda activate scipy-1.1.0-np115py36_6
(scipy-1.1.0-np115py36_6) $ conda pack
(scipy-1.1.0-np115py36_6) $ conda deactivate

Pack environment located at an explicit path:

conda pack -p $( ws_find conda )/conda/envs/scipy-1.1.0-np115py36_6

The easiest way is to unpack the package into an existing Conda installation.

Just create a directory and unpack the package:

mkdir -p external_conda_path/envs/scipy-1.1.0-np115py36_6
tar -xf scipy-1.1.0-np115py36_6.tar.gz -C external_conda_path/envs/scipy-1.1.0-np115py36_6
conda activate scipy-1.1.0-np115py36_6
# Cleanup prefixes from in the active environment
(scipy-1.1.0-np115py36_6) $ conda-unpack
(scipy-1.1.0-np115py36_6) $ conda deactivate

Versioning

Please keep in mind that modifying, updating and installing new packages into existing environments can modify the outcome of your results. We strongly encourage researchers to creating new environments (or cloning) before installing or updating packages. Consider using meaningful names for your environments using version numbers and dependencies.

Constraint	Specification
exact version	scipy==1.1.0
fuzzy version	scipy=1.1
greater equal	"scipy>=1.1"

For more information see cheat sheet below.

Example:

conda create -n scipy-1.1.0 scipy==1.1.0=np115py36_6

Pinning

Pin versions if you don't want them to be updated accidentally (see documentation).

Example:

echo 'scipy==1.1.0=np115py36_6' >> $( ws_find conda )/conda/envs/scipy-1.1.0-np115py36_6/conda-meta/pinned

You can easily pin your whole environment:

conda list -n scipy-1.1.0-np115py36_6 --export >$( ws_find conda )/conda/envs/scipy-1.1.0-np115py36_6/conda-meta/pinned

Deleting environments

Example:

conda env remove -n scipy-1.1.0-np115py36_6 --all

Cheat Sheet

Conda official cheat sheet