Difference between revisions of "Development/Conda"
(Created page with "= Install and use Conda = [https://conda.io/docs/index.html Conda] can be used to easily install missing Python packages on your own into different Python environments with d...") |
S Richling (talk | contribs) m (S Richling moved page Conda to Development/Conda: Umzug in Development-Bereich) |
||
| (22 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
| + | With [https://conda.io/docs/index.html Conda], you can easily install missing Python packages yourself in different Python environments with different versions in your own workspaces. |
||
| − | = Install and use Conda = |
||
| + | Each cluster provides a slightly different Conda installation and documentation. |
||
| − | [https://conda.io/docs/index.html Conda] can be used to easily install missing Python packages on your own into different Python environments with different versions in your own work spaces. |
||
| + | To learn more about Conda installations, visit the corresponding cluster page. |
||
| − | There are two possibilities of using Conda. One to install Conda itself, the other is to load a central installation if possible. |
||
| + | * [[Helix/Software/Conda]] |
||
| − | == (A) Installing Conda into a Workspace == |
||
| + | * [[NEMO/Software/Conda]] |
||
| − | |||
| − | We suggest using workspaces for Conda installation. Otherwise <syntaxhighlight style="border:0px" inline=1>$HOME/.conda</syntaxhighlight> will be used as default Conda path. |
||
| − | |||
| − | Create workspace: |
||
| − | <pre> |
||
| − | ws_allocate conda <days> # e.g. 100 |
||
| − | cd $( ws_find conda ) |
||
| − | </pre> |
||
| − | |||
| − | Download script and install conda: |
||
| − | <pre> |
||
| − | wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh |
||
| − | bash miniconda.sh -b -p $( ws_find conda )/conda |
||
| − | </pre> |
||
| − | |||
| − | Source variables for conda: |
||
| − | <pre> |
||
| − | source $( ws_find conda )/conda/etc/profile.d/conda.sh |
||
| − | </pre> |
||
| − | |||
| − | === Optional Steps === |
||
| − | |||
| − | Optional: Add source path in your local <syntaxhighlight style="border:0px" inline=1>~/.bashrc</syntaxhighlight> (edit file): |
||
| − | |||
| − | <pre> |
||
| − | source $( ws_find conda )/conda/etc/profile.d/conda.sh |
||
| − | </pre> |
||
| − | |||
| − | Optional: Update conda (usually base enviroment): |
||
| − | <pre> |
||
| − | conda update -n base conda |
||
| − | </pre> |
||
| − | |||
| − | Optional: Removing installation script: |
||
| − | <pre> |
||
| − | m miniconda.sh |
||
| − | </pre> |
||
| − | |||
| − | == (B) Use a centrally installed Conda Module == |
||
| − | |||
| − | If a Conda Module is provided you can just load it and create environments a workspace. |
||
| − | |||
| − | Load conda module: |
||
| − | <pre> |
||
| − | module avail tools/conda # list Conda modules |
||
| − | module load tools/conda # load module |
||
| − | </pre> |
||
| − | |||
| − | Create workspace: |
||
| − | <pre> |
||
| − | ws_allocate conda <days> # e.g. 100 |
||
| − | </pre> |
||
| − | |||
| − | The last step defines the newly created workspace as the download and installation path for your environments: |
||
| − | <pre> |
||
| − | conda config --append envs_dirs $( ws_find conda )/conda/envs |
||
| − | conda config --append pkgs_dirs $( ws_find conda )/conda/pkgs |
||
| − | conda config --show envs_dirs |
||
| − | conda config --show pkgs_dirs |
||
| − | </pre> |
||
| − | |||
| − | If you don't specify a new <syntaxhighlight style="border:0px" inline=1>envs_dir</syntaxhighlight> Conda will use <syntaxhighlight style="border:0px" inline=1>~/.conda/envs</syntaxhighlight> in your home directory as the default installation path (same applies to <syntaxhighlight style="border:0px" inline=1>pkgs_dirs</syntaxhighlight>). |
||
| − | |||
| − | == Install Packages into Environments == |
||
| − | |||
| − | You can create python environments and install packages into these environments or create them during install: |
||
| − | <pre> |
||
| − | conda create -n scipy |
||
| − | source activate scipy |
||
| − | conda install scipy |
||
| − | </pre> |
||
| − | |||
| − | Install packages and create a new environment: |
||
| − | <pre> |
||
| − | conda create -n scipy scipy |
||
| − | source activate scipy |
||
| − | </pre> |
||
| − | |||
| − | Search a special verson: |
||
| − | <pre> |
||
| − | conda search scipy==1.1.0 |
||
| − | </pre> |
||
| − | |||
| − | Create a Python 2.7 environment: |
||
| − | <pre> |
||
| − | conda create -n scipy_py27 scipy python=2.7 |
||
| − | source activate scipy_py27 |
||
| − | </pre> |
||
| − | |||
| − | == Activating Environments == |
||
| − | |||
| − | In order to use the software in an environment you'll need to activate it first: |
||
| − | <pre> |
||
| − | source activate scipy |
||
| − | </pre> |
||
| − | |||
| − | Deactivate to use different Python or software version: |
||
| − | <pre> |
||
| − | source deactivate |
||
| − | </pre> |
||
| − | |||
| − | == List packages and Environments == |
||
| − | |||
| − | List packages of current environment: |
||
| − | |||
| − | <pre> |
||
| − | conda list |
||
| − | </pre> |
||
| − | |||
| − | List packages in given environment: |
||
| − | <pre> |
||
| − | conda list -n scipy |
||
| − | </pre> |
||
| − | |||
| − | List environments: |
||
| − | <pre> |
||
| − | conda env list |
||
| − | </pre> |
||
| − | |||
| − | == Use Channels == |
||
| − | |||
| − | Add channels to get more software. We suggest to try the following channels: |
||
| − | |||
| − | <pre> |
||
| − | conda-forge |
||
| − | intel |
||
| − | bioconda |
||
| − | </pre> |
||
| − | |||
| − | Search in default and extra channel: |
||
| − | <pre> |
||
| − | conda search -c intel scipy |
||
| − | </pre> |
||
| − | |||
| − | You can add channel to your channels, but than you'll search and install automatically from this channel: |
||
| − | <pre> |
||
| − | conda config --add channels intel |
||
| − | conda config --show channels |
||
| − | conda config --remove channels intel # remove again |
||
| − | </pre> |
||
| − | |||
| − | == Use Intel Conda Packages == |
||
| − | |||
| − | You can find the full list of Intel Python packages on the [https://software.intel.com/en-us/articles/complete-list-of-packages-for-the-intel-distribution-for-python Intel web site]. |
||
| − | |||
| − | You can install the core Intel Python stack: |
||
| − | <pre> |
||
| − | conda install -c intel -n intelpython3 intelpython3_core |
||
| − | </pre> |
||
| − | |||
| − | ... with a special Python version: |
||
| − | <pre> |
||
| − | conda install -c intel -n intelpython-3.6.5 intelpython3_core python=3.6.5 |
||
| − | </pre> |
||
| − | |||
| − | ... with a Intel update version: |
||
| − | |||
| − | <pre> |
||
| − | conda create -c intel -n intelpython-2018.0.3 intelpython3_core==2018.0.3 |
||
| − | </pre> |
||
| − | |||
| − | ... or the full Intel Python stack: |
||
| − | <pre> |
||
| − | conda create -c intel -n intelpython-2018.0.3 intelpython3_full==2018.0.3 |
||
| − | </pre> |
||
| − | |||
| − | ... or just some Intel MKL optimized scientific software for the newest Intel 2019 version: |
||
| − | <pre> |
||
| − | # installs scipy-1.1.0-np115py36_6 |
||
| − | conda create -c intel -n scipy-1.1.0-np115py36_6 intelpython3_core=2019 |
||
| − | </pre> |
||
| − | |||
| − | == Create Reproducible Conda Environments == |
||
| − | |||
| − | For a more detailed environments documentation refer to the [https://conda.io/docs/user-guide/tasks/manage-environments.html conda documentation]. |
||
| − | |||
| − | Create an environment file for re-creation: |
||
| − | <pre> |
||
| − | conda env export -n scipy-1.1.0-np115py36_6 -f scipy-1.1.0-np115py36_6.yml |
||
| − | </pre> |
||
| − | |||
| − | Re-create saved environment: |
||
| − | <pre> |
||
| − | conda env create -f scipy-1.1.0-np115py36_6.yml |
||
| − | </pre> |
||
| − | |||
| − | Create a file with full URL for re-installation of packages: |
||
| − | <pre> |
||
| − | conda list --explicit -n scipy-1.1.0-np115py36_6 >scipy-1.1.0-np115py36_6.txt |
||
| − | </pre> |
||
| − | |||
| − | Install requirements file into environment: |
||
| − | <pre> |
||
| − | conda create --name scipy-1.1.0 --file scipy-1.1.0-np115py36_6.txt |
||
| − | </pre> |
||
| − | |||
| − | The first backup option is from the <syntaxhighlight style="border:0px" inline=1>conda-env</syntaxhighlight> command and tries to reproduce the environment by name and version. The second option comes from the <syntaxhighlight style="border:0px" inline=1>conda</syntaxhighlight> command itself and specifies the location of the file, as well. You can install the identical packages into a newly created environment. Please verify the architecture first. |
||
| − | |||
| − | To delete an environment: |
||
| − | <pre> |
||
| − | conda env remove -n scipy-1.1.0 |
||
| − | </pre> |
||
| − | |||
| − | To clone an existing environment: |
||
| − | <pre> |
||
| − | conda create --name scipy-1.1.0 --clone scipy-1.1.0-np115py36_6 |
||
| − | </pre> |
||
| − | |||
| − | == Versioning == |
||
| − | |||
| − | Please keep in mind that modifying, updating and installing new packages into existing environments can modify the outcome of your results. We strongly encourage researchers to creating new environments (or cloning) before installing or updating packages. Consider using meaningful names for your environments using version numbers and dependencies. |
||
| − | |||
| − | {| |
||
| − | !Constraint |
||
| − | !Specification |
||
| − | |- |
||
| − | |exact version |
||
| − | |scipy==1.1.0 |
||
| − | |- |
||
| − | |fuzzy version |
||
| − | |scipy=1.1 |
||
| − | |- |
||
| − | |greater equal |
||
| − | |"scipy>=1.1" |
||
| − | |} |
||
| − | |||
| − | For more information see cheat sheet below. |
||
| − | |||
| − | Example: |
||
| − | <pre> |
||
| − | conda create -c intel -n scipy-1.1.0 scipy==1.1.0=np115py36_6 |
||
| − | </pre> |
||
| − | |||
| − | === Pinning === |
||
| − | |||
| − | Pin versions if you don't want them to be updated accidentally ([https://conda.io/docs/user-guide/tasks/manage-pkgs.html#preventing-packages-from-updating-pinning see documentation]). |
||
| − | |||
| − | Example: |
||
| − | <pre> |
||
| − | echo 'scipy==1.1.0=np115py36_6' >> $( ws_find conda )/conda/envs/scipy-1.1.0-np115py36_6/conda-meta/pinned |
||
| − | </pre> |
||
| − | |||
| − | You can easily pin your whole environment: |
||
| − | <pre> |
||
| − | conda list -n scipy-1.1.0-np115py36_6 --export >$( ws_find conda )/conda/envs/scipy-1.1.0-np115py36_6/conda-meta/pinned |
||
| − | </pre> |
||
| − | |||
| − | == Cheat Sheet == |
||
| − | |||
| − | [https://conda.io/docs/_downloads/conda-cheatsheet.pdf Conda official cheat sheet] |
||
Latest revision as of 00:14, 15 March 2023
With Conda, you can easily install missing Python packages yourself in different Python environments with different versions in your own workspaces.
Each cluster provides a slightly different Conda installation and documentation.
To learn more about Conda installations, visit the corresponding cluster page.
