Helix/Software/Conda: Difference between revisions

From bwHPC Wiki
Jump to navigation Jump to search
mNo edit summary
(Redirect)
Tag: New redirect
 
(11 intermediate revisions by the same user not shown)
Line 1: Line 1:
#REDIRECT [[Development/Conda]]
= Install and use Conda =

[https://conda.io/docs/index.html Conda] can be used to easily install missing Python packages on your own into different Python environments with different versions in your own work spaces.

There are two possibilities of using Conda. One to install Conda itself, the other is to load a central installation if possible.

== (A) Installing Conda into a Workspace ==

We suggest using workspaces for Conda installation. Otherwise <syntaxhighlight style="border:0px" inline=1>$HOME/.conda</syntaxhighlight> will be used as default Conda path.

Create workspace for your conda environments:
<pre>
ws_allocate conda &lt;days&gt; # e.g. 30, for 30 days
cd $( ws_find conda )
</pre>

Download script and install conda:
<pre>
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
bash miniconda.sh -b -p $( ws_find conda )/conda
</pre>

Source variables for conda:
<pre>
source $( ws_find conda )/conda/etc/profile.d/conda.sh
</pre>

=== Optional Steps ===

Optional: Add source path in your local <syntaxhighlight style="border:0px" inline=1>~/.bashrc</syntaxhighlight> (edit file):

<pre>
source $( ws_find conda )/conda/etc/profile.d/conda.sh
</pre>

Optional: Update conda (usually base enviroment):
<pre>
conda update -n base conda
</pre>

Optional: Removing installation script:
<pre>
rm miniconda.sh
</pre>

== (B) Use a centrally installed Conda Module ==

Alternatively, you can load the Conda module and create environments in a workspace.

Load conda module:
<pre>
module load devel/miniconda/3
</pre>

Create a workspace:
<pre>
ws_allocate conda 30
</pre>

Prepare the environment:
<pre>
source $MINICONDA_HOME/etc/profile.d/conda.sh
</pre>

The last step defines the newly created workspace as the download and installation path for your environments:
<pre>
conda config --prepend envs_dirs $( ws_find conda )/conda/envs
conda config --prepend pkgs_dirs $( ws_find conda )/conda/pkgs
conda config --show envs_dirs
conda config --show pkgs_dirs
</pre>

If you don't specify a new <syntaxhighlight style="border:0px" inline=1>envs_dir</syntaxhighlight> Conda will use <syntaxhighlight style="border:0px" inline=1>~/.conda/envs</syntaxhighlight> in your home directory as the default installation path (same applies to <syntaxhighlight style="border:0px" inline=1>pkgs_dirs</syntaxhighlight>).

== Install Packages into Environments ==

You can create python environments and install packages into these environments or create them during install:
<pre>
conda create -n scipy
conda activate scipy
(scipy) $ conda install scipy
</pre>

Install packages and create a new environment:
<pre>
conda create -n scipy scipy
conda activate scipy
</pre>

Search a special verson:
<pre>
conda search scipy==1.1.0
</pre>

Create a Python 2.7 environment:
<pre>
conda create -n scipy_py27 scipy python=2.7
conda activate scipy_py27
</pre>

== Activating Environments ==

In order to use the software in an environment you'll need to activate it first:
<pre>
conda activate scipy
</pre>

Deactivate to use different Python or software version:
<pre>
conda deactivate
</pre>

== List packages and Environments ==

List packages of current environment:

<pre>
conda list
</pre>

List packages in given environment:
<pre>
conda list -n scipy
</pre>

List environments:
<pre>
conda env list
</pre>

== Use Channels ==

Add channels to get more software. We suggest to try the following channels:

<pre>
conda-forge
bioconda
</pre>

Search in default and extra channel:
<pre>
conda search -c conda-forge scipy
</pre>

You can add channel to your channels, but than you'll search and install automatically from this channel:
<pre>
conda config --add channels bioconda
conda config --show channels
conda config --remove channels bioconda # remove again
</pre>


== Create Reproducible Conda Environments ==

For a more detailed environments documentation refer to the [https://conda.io/docs/user-guide/tasks/manage-environments.html conda documentation].

Create an environment file for re-creation:
<pre>
conda env export -n scipy-1.1.0-np115py36_6 -f scipy-1.1.0-np115py36_6.yml
</pre>

Re-create saved environment:
<pre>
conda env create -f scipy-1.1.0-np115py36_6.yml
</pre>

Create a file with full URL for re-installation of packages:
<pre>
conda list --explicit -n scipy-1.1.0-np115py36_6 &gt;scipy-1.1.0-np115py36_6.txt
</pre>

Install requirements file into environment:
<pre>
conda create --name scipy-1.1.0 --file scipy-1.1.0-np115py36_6.txt
</pre>

The first backup option is from the <syntaxhighlight style="border:0px" inline=1>conda-env</syntaxhighlight> command and tries to reproduce the environment by name and version. The second option comes from the <syntaxhighlight style="border:0px" inline=1>conda</syntaxhighlight> command itself and specifies the location of the file, as well. You can install the identical packages into a newly created environment. Please verify the architecture first.

To clone an existing environment:
<pre>
conda create --name scipy-1.1.0 --clone scipy-1.1.0-np115py36_6
</pre>

=== Local channels and backup Conda packages ===

Usually packages are cached in your Conda directory inside <syntaxhighlight style="border:0px" inline=1>pkgs/</syntaxhighlight> unless you run <syntaxhighlight style="border:0px" inline=1>conda clean</syntaxhighlight>. Otherwise the environment will be reproduced from the channels' packages. If you want to be independent of other channels you can create your own local channel and backup every file you have used for creating your environments.

Install package <syntaxhighlight style="border:0px" inline=1>conda-build</syntaxhighlight>:
<pre>
conda install conda-build
</pre>

Create local channel directory for <syntaxhighlight style="border:0px" inline=1>linux-64</syntaxhighlight>:
<pre>
mkdir -p $( ws_find conda )/conda/channel/linux-64
</pre>

Create dependency file list and copy files to channel:
<pre>
conda list --explicit -n scipy-1.1.0-np115py36_6 >scipy-1.1.0-np115py36_6.txt
for f in $( grep -E '^http|^file' scipy-1.1.0-np115py36_6.txt ); do
cp $( ws_find conda )/conda/pkgs/$( basename $f ) $( ws_find conda )/conda/channel/linux-64/;
done
</pre>

Optional: If packages are missing in the cache download them:
<pre>
for f in $( grep -E '^http|^file' scipy-1.1.0-np115py36_6.txt ); do
wget $f -O $( ws_find conda )/conda/channel/linux-64/$( basename $f );
done
</pre>

Initialize channel:
<pre>
conda index $( ws_find conda )/conda/channel/
</pre>

Add channel to the channels list:
<pre>
conda config --add channels file://$( ws_find conda )/conda/channel/
</pre>

Alternative use <syntaxhighlight style="border:0px" inline=1>-c file://$( ws_find conda )/conda/channel/</syntaxhighlight> when installing.

=== Backup whole Environments ===

Alternatively you can create a package of your environment and unpack it again when needed.

Install <syntaxhighlight style="border:0px" inline=1>conda-pack</syntaxhighlight>:
<pre>
conda install -c conda-forge conda-pack
</pre>

Pack activated environment:
<pre>
conda activate scipy-1.1.0-np115py36_6
(scipy-1.1.0-np115py36_6) $ conda pack
(scipy-1.1.0-np115py36_6) $ conda deactivate
</pre>

Pack environment located at an explicit path:
<pre>
conda pack -p $( ws_find conda )/conda/envs/scipy-1.1.0-np115py36_6
</pre>

The easiest way is to unpack the package into an existing Conda installation.

Just create a directory and unpack the package:
<pre>
mkdir -p external_conda_path/envs/scipy-1.1.0-np115py36_6
tar -xf scipy-1.1.0-np115py36_6.tar.gz -C external_conda_path/envs/scipy-1.1.0-np115py36_6
conda activate scipy-1.1.0-np115py36_6
# Cleanup prefixes from in the active environment
(scipy-1.1.0-np115py36_6) $ conda-unpack
(scipy-1.1.0-np115py36_6) $ conda deactivate
</pre>

== Versioning ==

Please keep in mind that modifying, updating and installing new packages into existing environments can modify the outcome of your results. We strongly encourage researchers to creating new environments (or cloning) before installing or updating packages. Consider using meaningful names for your environments using version numbers and dependencies.

{| class="wikitable"
!Constraint
!Specification
|-
|exact version
|scipy==1.1.0
|-
|fuzzy version
|scipy=1.1
|-
|greater equal
|&quot;scipy&gt;=1.1&quot;
|}

For more information see cheat sheet below.

Example:
<pre>
conda create -n scipy-1.1.0 scipy==1.1.0=np115py36_6
</pre>

=== Pinning ===

Pin versions if you don't want them to be updated accidentally ([https://conda.io/docs/user-guide/tasks/manage-pkgs.html#preventing-packages-from-updating-pinning see documentation]).

Example:
<pre>
echo 'scipy==1.1.0=np115py36_6' &gt;&gt; $( ws_find conda )/conda/envs/scipy-1.1.0-np115py36_6/conda-meta/pinned
</pre>

You can easily pin your whole environment:
<pre>
conda list -n scipy-1.1.0-np115py36_6 --export &gt;$( ws_find conda )/conda/envs/scipy-1.1.0-np115py36_6/conda-meta/pinned
</pre>

=== Deleting environments ===

Example:
<pre>
conda env remove -n scipy-1.1.0-np115py36_6 --all
</pre>

== Cheat Sheet ==

[https://docs.conda.io/projects/conda/en/latest/user-guide/cheatsheet.html Conda official cheat sheet]

Latest revision as of 16:39, 4 November 2024

Redirect to:

Retrieved from "https://wiki.bwhpc.de/wiki/index.php?title=Helix/Software/Conda&oldid=13125"