User:M Janczyk/Software/Conda - Revision history

M Janczyk: M Janczyk moved page NEMO1/Software/Conda to User:M Janczyk/Software/Conda

2025-07-15T08:20:28Z

M Janczyk moved page NEMO1/Software/Conda to User:M Janczyk/Software/Conda

← Older revision	Revision as of 10:20, 15 July 2025
(No difference)

M Janczyk: M Janczyk moved page NEMO1 Legacy/Software/Conda to NEMO1/Software/Conda

2025-03-03T12:50:34Z

M Janczyk moved page NEMO1 Legacy/Software/Conda to NEMO1/Software/Conda

← Older revision	Revision as of 14:50, 3 March 2025
(No difference)

M Janczyk: M Janczyk moved page NEMO/Software/Conda to NEMO1 Legacy/Software/Conda without leaving a redirect

2025-03-03T12:45:22Z

M Janczyk moved page NEMO/Software/Conda to NEMO1 Legacy/Software/Conda without leaving a redirect

← Older revision	Revision as of 14:45, 3 March 2025
(No difference)

M Janczyk: M Janczyk moved page NEMO/SW/Conda to NEMO/Software/Conda

2022-12-06T09:36:26Z

M Janczyk moved page NEMO/SW/Conda to NEMO/Software/Conda

← Older revision	Revision as of 11:36, 6 December 2022
(No difference)

M Janczyk at 11:45, 7 July 2022

2022-07-07T11:45:29Z

← Older revision		Revision as of 13:45, 7 July 2022
Line 358:		Line 358:

	[https://docs.conda.io/projects/conda/en/latest/user-guide/cheatsheet.html Conda official cheat sheet]		[https://docs.conda.io/projects/conda/en/latest/user-guide/cheatsheet.html Conda official cheat sheet]


	= PDF Document =

	[https://www.overleaf.com/read/sswxjzgpwgfn https://www.overleaf.com/read/sswxjzgpwgfn]

M Janczyk at 15:51, 6 July 2022

2022-07-06T15:51:24Z

← Older revision		Revision as of 17:51, 6 July 2022
Line 43:		Line 43:
	</pre>		</pre>

	Search a ~~special~~ version:		Search for an exact version (see [[NEMO/SW/Conda#Versioning\|Versioning]]):
	<pre>		<pre>
	conda search scipy==1.1.0		conda search scipy==1.7.3
	</pre>		</pre>

	Create a Python 2.7 environment:		Create a Python 2.7 environment:
	<pre>		<pre>
	conda create -n ~~scipy_py27~~ scipy python=2.7		conda create -n scipy-py27 scipy python=2.7
	conda activate ~~scipy_py27~~		conda activate scipy-py27
	</pre>		</pre>


	== Activating Environments ==		== Activating Environments ==
Line 65:		Line 66:
	conda deactivate		conda deactivate
	</pre>		</pre>


	== List packages and Environments ==		== List packages and Environments ==
Line 83:		Line 85:
	conda env list		conda env list
	</pre>		</pre>


	== Use Channels ==		== Use Channels ==
Line 105:		Line 108:
	conda config --remove channels intel # remove channel again		conda config --remove channels intel # remove channel again
	</pre>		</pre>


	== Use Intel Conda Packages ==		== Use Intel Conda Packages ==
Line 115:		Line 119:
	</pre>		</pre>

	... with a ~~special~~ Python version:		... with a "fuzzy" Python version (see [[NEMO/SW/Conda#Versioning\|Versioning]]):
	<pre>		<pre>
	conda install -c intel -n intelpython-3.9.10 intelpython3_core python=3.9.10		conda install -c intel -n intelpython-3.9.10 intelpython3_core python=3.9.10
	</pre>		</pre>

	... with a ~~special~~ Intel OneApi version:		... with an exact Intel OneApi version (see [[NEMO/SW/Conda#Versioning\|Versioning]]):

	<pre>		<pre>
Line 317:		Line 321:
	source /conda/etc/profile.d/conda.sh		source /conda/etc/profile.d/conda.sh
	conda activate scipy-1.7.3		conda activate scipy-1.7.3
	~~python~~ ~~--version~~		eval "$@"

	%startscript
	echo "This is what happens when you start the container..."
⚫	source /conda/etc/profile.d/conda.sh
⚫	conda activate scipy-1.7.3
	python --version

	%post		%post
	echo "Hello from inside the container"
	yum -y install vim wget		yum -y install vim wget
	wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh		wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
Line 334:		Line 331:
	conda create -y -c intel -n scipy-1.7.3 scipy=1.7.3=py39h5c0f66f_1		conda create -y -c intel -n scipy-1.7.3 scipy=1.7.3=py39h5c0f66f_1
	rm miniconda.sh -f		rm miniconda.sh -f

	%test
	source /conda/etc/profile.d/conda.sh
	conda activate scipy-scipy-1.7.3
	python --version
	EOF		EOF
	</pre>		</pre>
Line 349:		Line 341:
	Copy the container on the cluster and start it:		Copy the container on the cluster and start it:
	<pre>		<pre>
	singularity run scipy-1.7.3.sif		singularity run scipy-1.7.3.sif python -V
			</pre>

			Example for interactive usage:
			<pre>
			singularity shell scipy-1.7.3.sif
		⚫	Apptainer> source /conda/etc/profile.d/conda.sh
		⚫	Apptainer> conda activate scipy-1.7.3
			(scipy-1.7.3) Apptainer> python -V
	</pre>		</pre>

	See [https://sylabs.io Singularity] documentation for more information on containers.		See [https://docs.sylabs.io/guides/latest/user-guide/ Singularity user documentation] for more information on containers.

M Janczyk: Created page with "= Use Conda Environments on NEMO = [https://conda.io/docs/index.html Conda] can be used to easily install missing Python packages by yourself in different Python environments..."

2022-07-06T14:44:31Z

Created page with "= Use Conda Environments on NEMO = [https://conda.io/docs/index.html Conda] can be used to easily install missing Python packages by yourself in different Python environments..."

New page

= Use Conda Environments on NEMO =

[https://conda.io/docs/index.html Conda] can be used to easily install missing Python packages by yourself in different Python environments with different versions in their own workspaces.

On NEMO, you can simply load the Conda module and start creating environments.

== Use the centrally installed Conda Module ==

The conda module is provided on NEMO, so you can easily load it and create a workspace.

Load conda module:
<pre>
module load devel/conda
</pre>

Create workspace:
<pre>
ws_allocate conda <days> # e.g. 100
</pre>

The last step defines the newly created workspace as the download and installation path for your environments:
<pre>
conda config --prepend envs_dirs $( ws_find conda )/conda/envs
conda config --prepend pkgs_dirs $( ws_find conda )/conda/pkgs
conda config --show envs_dirs
conda config --show pkgs_dirs
</pre>

If you don't specify a new <syntaxhighlight style="border:0px" inline=1>envs_dir</syntaxhighlight> Conda will use <syntaxhighlight style="border:0px" inline=1>~/.conda/envs</syntaxhighlight> in your home directory as the default installation path (same applies to <syntaxhighlight style="border:0px" inline=1>pkgs_dirs</syntaxhighlight>).

== Install Packages into Environments ==

You can create python environments and install packages into these environments or create them during install:
<pre>
conda create -n scipy
conda activate scipy
</pre>

Create a new environment and install packages into it with a single command:
<pre>
conda create -n scipy scipy
conda activate scipy
</pre>

Search a special version:
<pre>
conda search scipy==1.1.0
</pre>

Create a Python 2.7 environment:
<pre>
conda create -n scipy_py27 scipy python=2.7
conda activate scipy_py27
</pre>

== Activating Environments ==

To use the software in an environment, you must first activate it:
<pre>
conda activate scipy
</pre>

Disable the environment to load a different Python or software version:
<pre>
conda deactivate
</pre>

== List packages and Environments ==

List packages of current environment:

<pre>
conda list
</pre>

List packages in given environment:
<pre>
conda list -n scipy
</pre>

List environments:
<pre>
conda env list
</pre>

== Use Channels ==

Add channels to get more software. We suggest to try the following channels:

<pre>
conda-forge
intel
bioconda
</pre>

Search in default and extra channel:
<pre>
conda search -c intel scipy
</pre>

You can add channel to your channels, but than you'll search and install automatically from this channel:
<pre>
conda config --add channels intel
conda config --show channels
conda config --remove channels intel # remove channel again
</pre>

== Use Intel Conda Packages ==

The full list of Intel Python packages can be found in the [https://anaconda.org/intel/repo?sort=_name&sort_order=asc conda channel].

You can install the core Intel Python stack:
<pre>
conda install -c intel -n intelpython3 intelpython3_core
</pre>

... with a special Python version:
<pre>
conda install -c intel -n intelpython-3.9.10 intelpython3_core python=3.9.10
</pre>

... with a special Intel OneApi version:

<pre>
conda create -c intel -n intelpython-2022.1.0 intelpython3_core==2022.1.0
</pre>

... or the full Intel Python stack:
<pre>
conda create -c intel -n intelpython-2022.1.0 intelpython3_full==2022.1.0
</pre>

... or just some Intel MKL optimized scientific software for the newest Intel OneAPI version 2022:
<pre>
conda search -c intel scipy
conda create -c intel -n scipy-1.7.3 scipy=1.7.3=py39h5c0f66f_1
</pre>

== Deleting environments ==

Example:
<pre>
conda env remove -n scipy-1.7.3 --all
</pre>

----

= Create Reproducible Conda Environments =

This section is intended for advanced users who want to secure environments and create reproducible environments.

For a more detailed environments documentation refer to the [https://conda.io/docs/user-guide/tasks/manage-environments.html conda documentation].

Create an environment file for re-creation:
<pre>
conda env export -n scipy-1.7.3 -f scipy-1.7.3.yml
</pre>

Re-create saved environment:
<pre>
conda env create -f scipy-1.7.3.yml
</pre>

Create a file with full URL for re-installation of packages:
<pre>
conda list --explicit -n scipy-1.7.3 >scipy-1.7.3.txt
</pre>

Install requirements file into environment:
<pre>
conda create --name scipy-1.7.3 --file scipy-1.7.3.txt
</pre>

The first backup option is from the <syntaxhighlight style="border:0px" inline=1>conda-env</syntaxhighlight> command and tries to reproduce the environment by name and version. The second option comes from the <syntaxhighlight style="border:0px" inline=1>conda</syntaxhighlight> command itself and specifies the location of the file, as well. You can install the identical packages into a newly created environment. Please verify the architecture first.

To clone an existing environment:
<pre>
conda create --name scipy-1.7.3-clone --clone scipy-1.7.3
</pre>

== Local channels and backup Conda packages ==

Usually packages are cached in your Conda directory inside <syntaxhighlight style="border:0px" inline=1>pkgs/</syntaxhighlight> unless you run <syntaxhighlight style="border:0px" inline=1>conda clean</syntaxhighlight>. Otherwise the environment will be reproduced from the channels' packages. If you want to be independent of other channels you can create your own local channel and backup every file you have used for creating your environments.

Install package <syntaxhighlight style="border:0px" inline=1>conda-build</syntaxhighlight>:
<pre>
conda install conda-build
</pre>

Create local channel directory for <syntaxhighlight style="border:0px" inline=1>linux-64</syntaxhighlight>:
<pre>
mkdir -p $( ws_find conda )/conda/channel/linux-64
</pre>

Create dependency file list and copy files to channel:
<pre>
conda list --explicit -n scipy-1.7.3 >scipy-1.7.3.txt
for f in $( grep -E '^http|^file' scipy-1.7.3.txt ); do
cp $( ws_find conda )/conda/pkgs/$( basename $f ) $( ws_find conda )/conda/channel/linux-64/;
done
</pre>

Optional: If packages are missing in the cache download them:
<pre>
for f in $( grep -E '^http|^file' scipy-1.7.3.txt ); do
wget $f -O $( ws_find conda )/conda/channel/linux-64/$( basename $f );
done
</pre>

Initialize channel:
<pre>
conda index $( ws_find conda )/conda/channel/
</pre>

Add channel to the channels list:
<pre>
conda config --add channels file://$( ws_find conda )/conda/channel/
</pre>

Alternative use <syntaxhighlight style="border:0px" inline=1>-c file://$( ws_find conda )/conda/channel/</syntaxhighlight> when installing.

== Backup whole Environments ==

Alternatively you can create a package of your environment and unpack it again when needed.

Install <syntaxhighlight style="border:0px" inline=1>conda-pack</syntaxhighlight>:
<pre>
conda install -c conda-forge conda-pack
</pre>

Pack activated environment:
<pre>
conda activate scipy-1.7.3
(scipy-1.7.3) $ conda pack
(scipy-1.7.3) $ conda deactivate
</pre>

Pack environment located at an explicit path:
<pre>
conda pack -p $( ws_find conda )/conda/envs/scipy-1.7.3
</pre>

The easiest way is to unpack the package into an existing Conda installation.

Just create a directory and unpack the package:
<pre>
mkdir -p external_conda_path/envs/scipy-1.7.3
tar -xf scipy-1.7.3.tar.gz -C external_conda_path/envs/scipy-1.7.3
conda activate scipy-1.7.3
# Cleanup prefixes from in the active environment
(scipy-1.7.3) $ conda-unpack
(scipy-1.7.3) $ conda deactivate
</pre>

= Versioning =

Please keep in mind that modifying, updating and installing new packages into existing environments can modify the outcome of your results. We strongly encourage researchers to creating new environments (or cloning) before installing or updating packages. Consider using meaningful names for your environments using version numbers and dependencies.

{| class="wikitable"
!Constraint
!Specification
|-
|exact version
|scipy==1.7.3
|-
|fuzzy version
|scipy=1.7
|-
|greater equal
|"scipy>=1.7"
|}

For more information see cheat sheet below.

Example:
<pre>
conda create -c intel -n scipy-1.7.3 scipy==1.7.3=py39h5c0f66f_1
</pre>

= Pinning =

Pin versions if you don't want them to be updated accidentally ([https://conda.io/docs/user-guide/tasks/manage-pkgs.html#preventing-packages-from-updating-pinning see documentation]).

Example:
<pre>
echo 'scipy==1.7.3=py39h5c0f66f_1' >> $( ws_find conda )/conda/envs/scipy-1.7.3/conda-meta/pinned
</pre>

You can easily pin your whole environment:
<pre>
conda list -n scipy-1.7.3 --export >$( ws_find conda )/conda/envs/scipy-1.7.3/conda-meta/pinned
</pre>

= Using Singularity Containers =

Using [[Singularity_Containers|Singularity Containers]] can create more robust software environments.

Build the container on your local machine!

This is Singularity recipe example for a CentOS image with a Conda environment:
<pre>
cat << EOF >scipy-1.7.3.def
Bootstrap: docker
From: rockylinux:8
OSVersion: 8
# Alternative:
# From: almalinux:8

%runscript
echo "This is what happens when you run the container..."
source /conda/etc/profile.d/conda.sh
conda activate scipy-1.7.3
python --version

%startscript
echo "This is what happens when you start the container..."
source /conda/etc/profile.d/conda.sh
conda activate scipy-1.7.3
python --version

%post
echo "Hello from inside the container"
yum -y install vim wget
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
bash miniconda.sh -b -p conda
source /conda/etc/profile.d/conda.sh
conda update -y -n base conda
conda create -y -c intel -n scipy-1.7.3 scipy=1.7.3=py39h5c0f66f_1
rm miniconda.sh -f

%test
source /conda/etc/profile.d/conda.sh
conda activate scipy-scipy-1.7.3
python --version
EOF
</pre>

Build container (on local machine):
<pre>
singularity build scipy-1.7.3.sif scipy-1.7.3.def
</pre>

Copy the container on the cluster and start it:
<pre>
singularity run scipy-1.7.3.sif
</pre>

See [https://sylabs.io Singularity] documentation for more information on containers.

= Cheat Sheet =

[https://docs.conda.io/projects/conda/en/latest/user-guide/cheatsheet.html Conda official cheat sheet]

= PDF Document =

[https://www.overleaf.com/read/sswxjzgpwgfn https://www.overleaf.com/read/sswxjzgpwgfn]