<?xml version="1.0"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en">
	<id>https://wiki.bwhpc.de/wiki/api.php?action=feedcontributions&amp;feedformat=atom&amp;user=I+Tachakor</id>
	<title>bwHPC Wiki - User contributions [en]</title>
	<link rel="self" type="application/atom+xml" href="https://wiki.bwhpc.de/wiki/api.php?action=feedcontributions&amp;feedformat=atom&amp;user=I+Tachakor"/>
	<link rel="alternate" type="text/html" href="https://wiki.bwhpc.de/e/Special:Contributions/I_Tachakor"/>
	<updated>2026-11-16T06:01:41Z</updated>
	<subtitle>User contributions</subtitle>
	<generator>MediaWiki 1.39.17</generator>
	<entry>
		<id>https://wiki.bwhpc.de/wiki/index.php?title=BinAC2/Software/Jupyterlab&amp;diff=16472</id>
		<title>BinAC2/Software/Jupyterlab</title>
		<link rel="alternate" type="text/html" href="https://wiki.bwhpc.de/wiki/index.php?title=BinAC2/Software/Jupyterlab&amp;diff=16472"/>
		<updated>2026-10-05T20:44:25Z</updated>

		<summary type="html">&lt;p&gt;I Tachakor: /* Add a new Kernel */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;{{Softwarepage|devel/jupyterlab}}&lt;br /&gt;
&lt;br /&gt;
{| width=700px class=&amp;quot;wikitable&amp;quot;&lt;br /&gt;
|-&lt;br /&gt;
! Description   !! Content&lt;br /&gt;
|-&lt;br /&gt;
| module load&lt;br /&gt;
| devel/jupyterlab&lt;br /&gt;
|-&lt;br /&gt;
| License&lt;br /&gt;
| [https://github.com/jupyterlab/jupyterlab/blob/main/LICENSE JupyterLab License]&lt;br /&gt;
|-&lt;br /&gt;
| Links&lt;br /&gt;
| [https://jupyter.org/ Homepage]&lt;br /&gt;
|-&lt;br /&gt;
| Graphical Interface&lt;br /&gt;
|  Yes&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
= Description = &lt;br /&gt;
&lt;br /&gt;
JupyterLab is a web-based interactive development environment for notebooks, code, and data.&lt;br /&gt;
&lt;br /&gt;
Currently, BinAC 2 provides the following [https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-minimal-notebook JupyterLab Docker images] via Apptainer:&lt;br /&gt;
&lt;br /&gt;
* &amp;lt;syntaxhighlight inline&amp;gt;minimal-notebook&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
* &amp;lt;syntaxhighlight inline&amp;gt;r-notebook&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
* &amp;lt;syntaxhighlight inline&amp;gt;julia-notebook&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
* &amp;lt;syntaxhighlight inline&amp;gt;scipy-notebook&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
&lt;br /&gt;
= Usage =&lt;br /&gt;
&lt;br /&gt;
This guide is valid for &amp;lt;syntaxhighlight inline&amp;gt;minimal-notebook&amp;lt;/syntaxhighlight&amp;gt;. You can also follow it for the other notebook flavors &amp;amp;mdash; just replace &amp;lt;syntaxhighlight inline&amp;gt;minimal-notebook&amp;lt;/syntaxhighlight&amp;gt; with one of the other notebooks listed above.&lt;br /&gt;
&lt;br /&gt;
You will start a job on the cluster as usual, create an SSH tunnel, and connect to the running JupyterLab instance via your browser.&lt;br /&gt;
&lt;br /&gt;
== Start JupyterLab ==&lt;br /&gt;
&lt;br /&gt;
The module &amp;lt;syntaxhighlight inline&amp;gt;devel/jupyterlab&amp;lt;/syntaxhighlight&amp;gt; provides a job script for starting a JupyterLab instance on BinAC 2. Load the module and copy the template job script into your workspace:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;syntaxhighlight&amp;gt;&lt;br /&gt;
$ module load devel/jupyterlab&lt;br /&gt;
&lt;br /&gt;
&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Upon loading the module, you can list the available template job scripts and copy the one you need into your workspace:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;syntaxhighlight&amp;gt;&lt;br /&gt;
$ ls $JUPYTERLAB_EXA_DIR &lt;br /&gt;
binac2-julia-notebook.slurm  binac2-minimal-notebook.slurm  binac2-r-notebook.slurm  binac2-scipy-notebook.slurm&lt;br /&gt;
&lt;br /&gt;
$ cp $JUPYTERLAB_EXA_DIR/binac2-minimal-notebook.slurm &amp;lt;your workspace&amp;gt;&lt;br /&gt;
&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The job script is very simple; you only need to adjust the hardware resources according to your needs:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot; line&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
&lt;br /&gt;
# Adjust these values as needed&lt;br /&gt;
#SBATCH --cpus-per-task=1&lt;br /&gt;
#SBATCH --mem=2gb&lt;br /&gt;
#SBATCH --time=6:00:00&lt;br /&gt;
&lt;br /&gt;
# Don&#039;t change these settings&lt;br /&gt;
#SBATCH --nodes=1&lt;br /&gt;
#SBATCH --partition=interactive&lt;br /&gt;
#SBATCH --job-name=minimal-notebook-7.4.1&lt;br /&gt;
#PBS -j oe&lt;br /&gt;
&lt;br /&gt;
# Load Jupyterlab module&lt;br /&gt;
module load devel/jupyterlab/7.4.1&lt;br /&gt;
&lt;br /&gt;
# Start jupyterlab&lt;br /&gt;
$ ${JUPYTERLAB_BIN_DIR}/minimal-notebook.sh&lt;br /&gt;
&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Now submit the job to SLURM. This command will store the job ID in the variable &amp;lt;syntaxhighlight inline&amp;gt;jobid&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;syntaxhighlight&amp;gt;&lt;br /&gt;
$ jobid=$(sbatch --parsable binac2-minimal-notebook.slurm)&lt;br /&gt;
&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
&lt;br /&gt;
{|style=&amp;quot;background:#deffee; width:100%;&amp;quot;&lt;br /&gt;
|style=&amp;quot;padding:5px; background:#cef2e0; text-align:left&amp;quot;|&lt;br /&gt;
[[Image:Attention.svg|center|25px]]&lt;br /&gt;
|style=&amp;quot;padding:5px; background:#cef2e0; text-align:left&amp;quot;|&lt;br /&gt;
Depending on the current load on BinAC 2 and the resources you requested in your job script, it may take some time for the job to start.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
== Create SSH tunnel ==&lt;br /&gt;
&lt;br /&gt;
The compute node on which JupyterLab is running is not directly reachable from your local workstation. Therefore, you need to create an SSH tunnel from your workstation to the compute node via the BinAC 2 login node.&lt;br /&gt;
&lt;br /&gt;
The job&#039;s standard output file (&amp;lt;syntaxhighlight inline&amp;gt;slurm-${jobid}.out&amp;lt;/syntaxhighlight&amp;gt;) contains all the information you need. Please note that details such as the IP address, port number, and access URL will differ in your case.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;&lt;br /&gt;
$ cat slurm-${jobid}.out&lt;br /&gt;
&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
&lt;br /&gt;
[[File:Binac2_jupyterlab_connection_details.png | 800px | center | JupyterLab connection info]]&lt;br /&gt;
&lt;br /&gt;
=== Linux Users ===&lt;br /&gt;
&lt;br /&gt;
Copy the &amp;lt;syntaxhighlight inline&amp;gt;ssh -N -L ... &amp;lt;/syntaxhighlight&amp;gt; command and execute it in a shell on your workstation. After successful authentication, the SSH tunnel will be ready to use. The ssh command does not return any output. If there are no error messages, everything should be fine:&lt;br /&gt;
&lt;br /&gt;
[[File:Binac2_jupyterlab_ssh_tunnel_linux.png | 800px | center | SSH tunnel creation with Linux]]&lt;br /&gt;
&lt;br /&gt;
=== Windows Users ===&lt;br /&gt;
&lt;br /&gt;
If you are using Windows, you will need to create the SSH tunnel using an SSH client of your choice (e.g. MobaXTerm, PuTTY, etc.). Her, we will show how to create an SSH tunnel with MobaXTerm.&lt;br /&gt;
&lt;br /&gt;
Select &amp;lt;syntaxhighlight inline&amp;gt;Tunneling&amp;lt;/syntaxhighlight&amp;gt; in the top ribbon. Then click &amp;lt;syntaxhighlight inline&amp;gt;New SSH tunnel&amp;lt;/syntaxhighlight&amp;gt;.&lt;br /&gt;
Configure the SSH tunnel with the correct values taken from the SSH tunnel information provided above.&lt;br /&gt;
For the example in this tutorial it looks as follows:&lt;br /&gt;
&lt;br /&gt;
[[File:Binac2_jupyterlab_mobaxterm.png | 800px | center | SSH tunnel creation with MobaXTerm]]&lt;br /&gt;
&lt;br /&gt;
== Access JupyterLab ==&lt;br /&gt;
&lt;br /&gt;
JupyterLab is now running on a BinAC 2 compute node, and you have created an SSH tunnel from your workstation to that compute node. Open a browser and paste the URL with the access token into the address bar:&lt;br /&gt;
&lt;br /&gt;
[[File:Binac2_jupyterlab_browser_url.png | 800px | center | JupyterLab URL in browser ]]&lt;br /&gt;
&lt;br /&gt;
Your browser should now display the JupyterLab interface:&lt;br /&gt;
&lt;br /&gt;
[[File:Binac2_jupyterlab_browser_lab.png | 800px | center ]]&lt;br /&gt;
&lt;br /&gt;
== Shut Down JupyterLab ==&lt;br /&gt;
&lt;br /&gt;
You can shut down JupyterLab via &amp;lt;syntaxhighlight inline&amp;gt;File -&amp;gt; Shut Down&amp;lt;/syntaxhighlight&amp;gt;.&lt;br /&gt;
Please note that this will also terminate your compute job on BinAC 2!&lt;br /&gt;
&lt;br /&gt;
[[ File:Binac2_jupyterlab_browser_shutdown.png | 800px | center | JupyterLab user interface]]&lt;br /&gt;
&lt;br /&gt;
== Tips &amp;amp; Tricks ==&lt;br /&gt;
&lt;br /&gt;
=== Managing Notebooks ===&lt;br /&gt;
&lt;br /&gt;
JupyterLab&#039;s root directory will be your home directory. Since your home directory is backed up daily, you may want to store your notebooks there.&lt;br /&gt;
It may also be a good idea to place your notebooks under proper version control using Git.&lt;br /&gt;
&lt;br /&gt;
=== Access &amp;lt;syntaxhighlight inline&amp;gt;work&amp;lt;/syntaxhighlight&amp;gt; and &amp;lt;syntaxhighlight inline&amp;gt;project&amp;lt;/syntaxhighlight&amp;gt; ===&lt;br /&gt;
&lt;br /&gt;
The notebooks will always be able to access your data stored in &amp;lt;syntaxhighlight inline&amp;gt;work&amp;lt;/syntaxhighlight&amp;gt; and &amp;lt;syntaxhighlight inline&amp;gt;project&amp;lt;/syntaxhighlight&amp;gt;. However, the file browser shows the content in your home directory and you won’t be able to access &amp;lt;syntaxhighlight inline&amp;gt;work&amp;lt;/syntaxhighlight&amp;gt; and &amp;lt;syntaxhighlight inline&amp;gt;project&amp;lt;/syntaxhighlight&amp;gt; initially.&lt;br /&gt;
You can create symbolic links in your home directory to work and project, making these two partitions available in JupyterLab&#039;s file browser.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;&lt;br /&gt;
$ cd ~&lt;br /&gt;
$ ln -s /pfs/10/work/ $HOME/work&lt;br /&gt;
$ ln -s /pfs/10/project $HOME/project&lt;br /&gt;
&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Through that link in your home directory you can navigate to your research data in Jupyterlab&#039;s file browser.&lt;br /&gt;
Here is an example how I linked to my directory:&lt;br /&gt;
&lt;br /&gt;
[[ File:Binac2_jupyterlab_link.png | 800px | center | Access work and project storage in JupyterLab file browser ]]&lt;br /&gt;
&lt;br /&gt;
=== Managing Kernels ===&lt;br /&gt;
&lt;br /&gt;
Depending on the notebook you started, the available kernels will differ.&lt;br /&gt;
The Python notebook, for example, only has one Python kernel, whereas the R notebook has only one R kernel, and so on.&lt;br /&gt;
Given the nearly endless combinations of programming languages and packages they provide, we suggest creating your own kernels if needed.&lt;br /&gt;
The kernels are stored in your hone directory on BinAC 2: &amp;lt;syntaxhighlight inline&amp;gt;$HOME/.local/share/jupyter/kernels/&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
You cannot install new kernels directly from within JupyterLab; this must be done from the command line in your usual BinAC 2 shell session.&lt;br /&gt;
&lt;br /&gt;
=== Add a new Kernel ===&lt;br /&gt;
&lt;br /&gt;
We will show you how to create new kernels for Python and R. For Python, you can use either Conda or &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt;. For R, &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; is not supported, so R kernels must be created using Conda.&lt;br /&gt;
&lt;br /&gt;
==== Python ====&lt;br /&gt;
&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;&lt;br /&gt;
module load devel/miniforge&lt;br /&gt;
conda create --name kernel_env python=3.13 pandas numpy matplotlib ipykernel            # 1&lt;br /&gt;
conda activate kernel_env                                                               # 2&lt;br /&gt;
python -m ipykernel install --user --name pandas --display-name=&amp;quot;Python 3.13 (pandas)&amp;quot;  # 3&lt;br /&gt;
# Installed kernelspec pandas in $HOME/.local/share/jupyter/kernels/pandas&lt;br /&gt;
&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The first command creates a new Conda environment called &amp;lt;syntaxhighlight inline&amp;gt;kernel_env&amp;lt;/syntaxhighlight&amp;gt; and installs a specific Python version, along with a few additional Python packages.&lt;br /&gt;
It&#039;s important that you also install &amp;lt;syntaxhighlight inline&amp;gt;ipykernel&amp;lt;/syntaxhighlight&amp;gt;, as we will need it later to create the JupyterLab kernel.&lt;br /&gt;
The second command activates the &amp;lt;syntaxhighlight inline&amp;gt;kernel_env&amp;lt;/syntaxhighlight&amp;gt; Conda environment.&lt;br /&gt;
The third command creates the new JupyterLab kernel.&lt;br /&gt;
&lt;br /&gt;
Alternatively, the same Python environment can be created and registered as a JupyterLab kernel using &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt;:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;&lt;br /&gt;
uv venv kernel_env --python 3.13&lt;br /&gt;
uv pip install --python kernel_env/bin/python pandas numpy matplotlib ipykernel&lt;br /&gt;
uv run --directory kernel_env python -m ipykernel install --user --name pandas --display-name=&amp;quot;Python 3.13 (pandas)&amp;quot;&lt;br /&gt;
# Installed kernelspec pandas in $HOME/.local/share/jupyter/kernels/pandas&lt;br /&gt;
&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The first command creates a new &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; virtual environment called &amp;lt;syntaxhighlight inline&amp;gt;kernel_env&amp;lt;/syntaxhighlight&amp;gt; using Python 3.13.&lt;br /&gt;
The second command installs the required Python packages, including &amp;lt;syntaxhighlight inline&amp;gt;ipykernel&amp;lt;/syntaxhighlight&amp;gt;.&lt;br /&gt;
The third command creates the new JupyterLab kernel.&lt;br /&gt;
&lt;br /&gt;
Unlike with Conda, the &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; environment does not have to be activated. The &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; commands directly use the Python interpreter from the environment.&lt;br /&gt;
&lt;br /&gt;
[[ File:Binac2_jupyterlab_new_kernel.png | 800px | center | New Python kernel in JupyterLab]]&lt;br /&gt;
&lt;br /&gt;
==== R ====&lt;br /&gt;
&lt;br /&gt;
The instructions for creating new R kernels are slightly different.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;&lt;br /&gt;
module load devel/miniforge&lt;br /&gt;
conda create --name r_kernel_env r-base=4.4.3 jupyter r-irkernel&lt;br /&gt;
conda activate r_kernel_env &lt;br /&gt;
R&lt;br /&gt;
# In the R-Session&lt;br /&gt;
install.packages(...)&lt;br /&gt;
IRkernel::installspec(name = &#039;ir44&#039;, displayname = &#039;R 4.4.3&#039;)&lt;br /&gt;
&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The first command creates a new Conda environment called &amp;lt;syntaxhighlight inline&amp;gt;r_kernel_env&amp;lt;/syntaxhighlight&amp;gt; and installs a specific R version.&lt;br /&gt;
It&#039;s important to also install &amp;lt;syntaxhighlight inline&amp;gt;r-irkernel&amp;lt;/syntaxhighlight&amp;gt;, as we will need it later to create the JupyterLab kernel.&lt;br /&gt;
The second command activates the &amp;lt;syntaxhighlight inline&amp;gt;r_kernel_env&amp;lt;/syntaxhighlight&amp;gt; Conda environment and open an R session.&lt;br /&gt;
In this session, you can install any R packages you need for your kernel.&lt;br /&gt;
Finally, create the new kernel with the &amp;lt;syntaxhighlight inline&amp;gt;installspec&amp;lt;/syntaxhighlight&amp;gt; command.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; does not provide an equivalent for creating and managing R environments. Therefore, the Conda-based procedure above should be used for R kernels.&lt;br /&gt;
&lt;br /&gt;
=== Remove a Kernel ===&lt;br /&gt;
&lt;br /&gt;
To remove a kernel from JupyterLab, simply delete the corresponding directory at &amp;lt;syntaxhighlight inline&amp;gt;$HOME/.local/share/jupyter/kernels/&amp;lt;/syntaxhighlight&amp;gt;:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;&lt;br /&gt;
# Remove the JupyterLab kernel installed in the previous example&lt;br /&gt;
rm -rf $HOME/.local/share/jupyter/kernels/pandas&lt;br /&gt;
&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Also, remove the corresponding Conda environment if you do not need it any more:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;&lt;br /&gt;
conda env remove --name kernel_env&lt;br /&gt;
&amp;lt;/syntaxhighlight&amp;gt;&lt;/div&gt;</summary>
		<author><name>I Tachakor</name></author>
	</entry>
	<entry>
		<id>https://wiki.bwhpc.de/wiki/index.php?title=Helix/bwVisu/JupyterLab&amp;diff=16471</id>
		<title>Helix/bwVisu/JupyterLab</title>
		<link rel="alternate" type="text/html" href="https://wiki.bwhpc.de/wiki/index.php?title=Helix/bwVisu/JupyterLab&amp;diff=16471"/>
		<updated>2026-10-05T20:21:16Z</updated>

		<summary type="html">&lt;p&gt;I Tachakor: /* Create a kernel from a virtual environment */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;[https://jupyter.org/ JupyterLab] is an integrated development environment (IDE) that provides a flexible and scalable interface for the Jupyter Notebook system. It supports interactive data science and scientific computing across over 40 programming languages (including Python, Julia, and R).&lt;br /&gt;
&lt;br /&gt;
== Change Python Version ==&lt;br /&gt;
&lt;br /&gt;
The default python version can be seen by running &amp;lt;code&amp;gt;python --version&amp;lt;/code&amp;gt; in the terminal.&lt;br /&gt;
&lt;br /&gt;
A different python version can be installed into a new virtual environment and then registered as IPython kernel for the usage in JupyterLab. This is explained in the chapter [[#Add_packages_via_conda_environments | add packages via conda environments]].&lt;br /&gt;
&lt;br /&gt;
== Install Python Packages ==&lt;br /&gt;
&lt;br /&gt;
Python packages can be added by installing them into a virtual environment and then creating an IPython kernel from the virtual environment. &amp;lt;/br&amp;gt;&lt;br /&gt;
&amp;lt;u&amp;gt;Kernels can be shared&amp;lt;/u&amp;gt;. See the notes below. &amp;lt;/br&amp;gt;&lt;br /&gt;
If you want to move a virtual environment, it is adivsed to recreate it in the new place. Otherwise, dependencies based on relative paths will break.&lt;br /&gt;
&lt;br /&gt;
# Create a virtual environment with...&lt;br /&gt;
#* [[#Add_packages_via_venv_virtual_environments | ...venv]] or&lt;br /&gt;
#* [[#Add_packages_via_Conda_virtual_environments | ...conda]] (choose this option if you want to install a different python version) or&lt;br /&gt;
#* [[#Add_packages_via_uv_virtual_environments | ...uv]] if you want to install a different python version but don&#039;t want to use conda.&lt;br /&gt;
# [[#Create_an_IPython_Kernel | Create an IPython kernel]] from the virtual environment&lt;br /&gt;
# Use the kernel within JupyterLab&lt;br /&gt;
#* By default new kernels are saved under &amp;lt;code&amp;gt;~/.local/share/jupyter&amp;lt;/code&amp;gt; and this location is automatically detected. Therefore, new kernels are directly available.&lt;br /&gt;
#*:[[File:BwVisu JuypterLab KernelPath.png|In the JupyterLab job configuration form, a custom kernel path can be provided.|right|thumb|x150px]]&lt;br /&gt;
#* If the kernel is saved somewhere else, the path can be provided in the &amp;quot;Kernel path&amp;quot; field when configuring the JupyterLab job (see image). For a kernel placed under &amp;lt;code&amp;gt;path_to_parent_dir/share/jupyter/kernels/my_kernel&amp;lt;/code&amp;gt; the needed &amp;quot;Kernel path&amp;quot; would be &amp;lt;code&amp;gt;path_to_parent_dir/share/jupyter&amp;lt;/code&amp;gt;.&lt;br /&gt;
# When the kernel is used the first time, the file &amp;lt;code&amp;gt;notebook_secrets&amp;lt;/code&amp;gt; is created automatically. It can be found under &amp;quot;Kernel path&amp;quot;. For others to use the kernel, they must have read access to this file. The command &amp;lt;code&amp;gt;chmod 750 notebook_secrets&amp;lt;/code&amp;gt; would for example allow the whole SDS@hd SV read access. For the access management in workspaces, please see [[Workspace#Setting_Permissions_for_Sharing_Files| Workspace Permissions]].&lt;br /&gt;
&lt;br /&gt;
&amp;lt;u&amp;gt;Notes regarding the sharing of IPython kernels&amp;lt;/u&amp;gt;&lt;br /&gt;
* The virtual environment and the kernel need to be placed in a shared directory. For example at SDS@hd.&lt;br /&gt;
* There could be a subdirectory for the virtual environments and one for the kernels.&lt;br /&gt;
* The path to the kernels is saved in the environment variable $JUPYTER_DATA_DIR. Jupyter relevant paths can be seen with &amp;lt;code&amp;gt;jupyter --paths&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Add packages via &#039;&#039;venv&#039;&#039; virtual environments ===&lt;br /&gt;
More information about venv or other python virtual environments can be found at the [[Development/Python | Python]] page.&lt;br /&gt;
&lt;br /&gt;
Steps for creating a &#039;&#039;&#039;venv&#039;&#039;&#039; virtual environment:&lt;br /&gt;
&amp;lt;ol style=&amp;quot;list-style-type: decimal;&amp;quot;&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Open a terminal.&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Create a new virtual evironment:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;python3 -m venv &amp;lt;env_parent_dir&amp;gt;/&amp;lt;env_name&amp;gt;&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
* &amp;lt;code&amp;gt;env_parent_dir&amp;lt;/code&amp;gt; is the path to the folder where the virtual environment shall be created. Relative paths can be used.&lt;br /&gt;
* Caution: If you you want to share the environment with others, make sure to already create it in the shared place.&lt;br /&gt;
&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Activate the environment:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;source &amp;lt;env_parent_dir&amp;gt;/&amp;lt;env_name&amp;gt;/bin/activate&amp;lt;/syntaxhighlight&amp;gt;&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Update pip and install packages:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;pip install -U pip --no-user&lt;br /&gt;
# when the environment is installed in home&lt;br /&gt;
pip install &amp;lt;packagename&amp;gt;&lt;br /&gt;
# when the environment is installed somewhere else and shall not have dependencies in home so that others can access it as well&lt;br /&gt;
pip install &amp;lt;packagename&amp;gt; --no-user&amp;lt;/syntaxhighlight&amp;gt;&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt; [[#Create_an_IPython_Kernel | Create an IPython kernel]]&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;/ol&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Add packages via Conda virtual environments ===&lt;br /&gt;
More information about using conda can be found at the [[Development/Conda | Conda]] page.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;ol style=&amp;quot;list-style-type: decimal;&amp;quot;&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Load the miniforge module by clicking first on the double hexagon icon on the left-hand side of Jupyter&#039;s start page and then on the &amp;amp;quot;load&amp;amp;quot; button right of the entry for miniforge in the software module menu.&lt;br /&gt;
&amp;lt;li&amp;gt;Open a terminal.&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Create a new virtual environment:&lt;br /&gt;
* If you are the only person using the environment, you can install it in your home directory:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;conda create --name &amp;lt;env_name&amp;gt; python=&amp;lt;python version&amp;gt;&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
* If you want to install it into a different directory, for example a shared place:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;conda create --prefix &amp;lt;path_to_shared_directory&amp;gt;/&amp;lt;env_name&amp;gt; python=&amp;lt;python version&amp;gt;&lt;br /&gt;
&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Activate your environment:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;conda activate &amp;lt;myenv&amp;gt;&amp;lt;/syntaxhighlight&amp;gt;&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Install your packages:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;conda install &amp;lt;mypackage&amp;gt;&amp;lt;/syntaxhighlight&amp;gt;&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt; [[#Create_an_IPython_Kernel | Create an IPython kernel]]&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;/ol&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Add packages via uv virtual environments ===&lt;br /&gt;
More information about using &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; can be found at the [[Development/Python | Python]] page.&lt;br /&gt;
&lt;br /&gt;
Steps for creating a &#039;&#039;&#039;uv&#039;&#039;&#039; virtual environment:&lt;br /&gt;
&amp;lt;ol style=&amp;quot;list-style-type: decimal;&amp;quot;&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Open a terminal.&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Create a new virtual environment:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;uv venv &amp;lt;env_parent_dir&amp;gt;/&amp;lt;env_name&amp;gt; --python &amp;lt;python version&amp;gt;&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
* &amp;lt;code&amp;gt;env_parent_dir&amp;lt;/code&amp;gt; is the path to the folder where the virtual environment shall be created. Relative paths can be used.&lt;br /&gt;
* The &amp;lt;code&amp;gt;--python&amp;lt;/code&amp;gt; option can be omitted if the default Python version should be used.&lt;br /&gt;
* Caution: If you want to share the environment with others, make sure to already create it in the shared place.&lt;br /&gt;
&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Install packages into the environment:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;uv pip install --python &amp;lt;env_parent_dir&amp;gt;/&amp;lt;env_name&amp;gt;/bin/python &amp;lt;packagename&amp;gt;&amp;lt;/syntaxhighlight&amp;gt;&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt; [[#Create_an_IPython_Kernel | Create an IPython kernel]]&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;/ol&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Unlike a traditional &amp;lt;code&amp;gt;venv&amp;lt;/code&amp;gt; environment, a &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; environment does not have to be activated with &amp;lt;code&amp;gt;source&amp;lt;/code&amp;gt;. The &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; commands can directly use the Python interpreter from the environment.&lt;br /&gt;
&lt;br /&gt;
== Create an IPython Kernel ==&lt;br /&gt;
&lt;br /&gt;
Python kernels are implementations of the Jupyter notebook environment for different languages or virtual environments. You can switch between kernels easily, allowing you to use the best tool for a specific task.&lt;br /&gt;
conda_kernels&lt;br /&gt;
&lt;br /&gt;
=== Create a kernel from a venv or Conda virtual environment ===&lt;br /&gt;
&lt;br /&gt;
For a &#039;&#039;&#039;venv&#039;&#039;&#039; or &#039;&#039;&#039;Conda&#039;&#039;&#039; environment, the environment has to be activated before creating the kernel.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;ol&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Activate the virtual environment.&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Install the ipykernel package:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;pip install ipykernel&amp;lt;/syntaxhighlight&amp;gt;&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Register the virtual environment as custom kernel to Jupyter.&lt;br /&gt;
* If you are the only person using the environment:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;python3 -m ipykernel install --user --name=&amp;lt;kernel_name&amp;gt;&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
The kernel can be found under &amp;lt;code&amp;gt;~/.local/share/jupyter/kernels/&amp;lt;/code&amp;gt;.&lt;br /&gt;
* If you installed the environment in a shared place and want to have the kernel there as well:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;python3 -m ipykernel install --prefix &amp;lt;path_to_kernel_folder&amp;gt; --name=&amp;lt;kernel_name&amp;gt;&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
The kernel can then be found under &amp;lt;code&amp;gt;path_to_kernel_folder/share/jupyter/kernels/&amp;lt;kernel_name&amp;gt;&amp;lt;/code&amp;gt;. As long as the same &amp;lt;code&amp;gt;path_to_kernel_folder&amp;lt;/code&amp;gt; is used, all kernels will be saved next to each other in &amp;quot;kernels&amp;quot;.&lt;br /&gt;
&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;/ol&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Create a kernel from a uv virtual environment ===&lt;br /&gt;
&lt;br /&gt;
A kernel can also be created from a &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; virtual environment. The environment does not have to be activated.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;ol&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Install the &amp;lt;code&amp;gt;ipykernel&amp;lt;/code&amp;gt; package into the &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; environment:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;uv pip install --python &amp;lt;env_parent_dir&amp;gt;/&amp;lt;env_name&amp;gt;/bin/python ipykernel&amp;lt;/syntaxhighlight&amp;gt;&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Register the environment as a custom kernel using &amp;lt;code&amp;gt;uv run&amp;lt;/code&amp;gt;:&lt;br /&gt;
* If you are the only person using the environment:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;uv run --directory &amp;lt;env_parent_dir&amp;gt;/&amp;lt;env_name&amp;gt; python -m ipykernel install --user --name=&amp;lt;kernel_name&amp;gt;&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
The kernel can be found under &amp;lt;code&amp;gt;~/.local/share/jupyter/kernels/&amp;lt;/code&amp;gt;.&lt;br /&gt;
* If you installed the environment in a shared place and want to have the kernel there as well:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;uv run --directory &amp;lt;env_parent_dir&amp;gt;/&amp;lt;env_name&amp;gt; python -m ipykernel install --prefix &amp;lt;path_to_kernel_folder&amp;gt; --name=&amp;lt;kernel_name&amp;gt;&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
The kernel can then be found under &amp;lt;code&amp;gt;path_to_kernel_folder/share/jupyter/kernels/&amp;lt;kernel_name&amp;gt;&amp;lt;/code&amp;gt;. As long as the same &amp;lt;code&amp;gt;path_to_kernel_folder&amp;lt;/code&amp;gt; is used, all kernels will be saved next to each other in &amp;quot;kernels&amp;quot;.&lt;br /&gt;
&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;/ol&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Multi-Language Support ===&lt;br /&gt;
&lt;br /&gt;
JupyterLab supports over 40 programming languages including Python, R, Julia, and Scala. This is achieved through the use of different kernels.&lt;br /&gt;
&lt;br /&gt;
==== R Kernel ====&lt;br /&gt;
&lt;br /&gt;
&amp;lt;ol style=&amp;quot;list-style-type: decimal;&amp;quot;&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;On the cluster:&lt;br /&gt;
&amp;lt;pre&amp;gt;$ module load math/R&lt;br /&gt;
$ R&lt;br /&gt;
&amp;amp;gt; install.packages(&#039;IRkernel&#039;)&amp;lt;/pre&amp;gt;&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;On bwVisu:&lt;br /&gt;
&amp;lt;ol style=&amp;quot;list-style-type: decimal;&amp;quot;&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Start Jupyter App&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;In left menu: load math/R&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Open Console:&amp;lt;/li&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;$ R&lt;br /&gt;
&amp;amp;gt; IRkernel::installspec(displayname = &#039;R 4.2&#039;)&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;li&amp;gt;Start kernel &#039;R 4.2&#039; as console or notebook&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;/ol&amp;gt;&lt;br /&gt;
&amp;lt;/ol&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==== Julia Kernel ====&lt;br /&gt;
&lt;br /&gt;
Load the math/julia module. Open the Terminal.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
julia&lt;br /&gt;
]&lt;br /&gt;
add IJulia&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
After that, Julia is available as a kernel.&lt;br /&gt;
&lt;br /&gt;
== Interactive Widgets ==&lt;br /&gt;
&lt;br /&gt;
JupyterLab supports interactive widgets that can create UI controls for interactive data visualization and manipulation within the notebooks. Example of using an interactive widget:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre class=&amp;quot;{.python&amp;quot;&amp;gt;from ipywidgets import IntSlider&lt;br /&gt;
slider = IntSlider()&lt;br /&gt;
display(slider)&amp;lt;/pre&amp;gt;&lt;br /&gt;
These widgets can be sliders, dropdowns, buttons, etc., which can be connected to Python code running in the backend.&lt;br /&gt;
&lt;br /&gt;
== FAQ ==&lt;br /&gt;
&lt;br /&gt;
&amp;lt;ol&amp;gt;&amp;lt;li&amp;gt;&#039;&#039;&#039;How can I use bwForCluster Helix software modules?&#039;&#039;&#039;&amp;lt;/br&amp;gt;&lt;br /&gt;
First click on the double hexagon icon on the left-hand side of Jupyter&#039;s start page. Then load a module by clicking on the &amp;quot;load&amp;quot; button next to the corresponding module entry.&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;&#039;&#039;&#039;My virtual environment works on Helix but not in the bwVisu JupyterLab job.&#039;&#039;&#039;&amp;lt;/br&amp;gt;&lt;br /&gt;
If the environment uses Helix modules, you have to load the modules in the bwVisu job first. For example, when you used the python 3.13 module for creating the environment, the path to this python version is saved in the environment. When the module is not loadedin bwVisu, the path is not available and you might get an error like &amp;quot;&#039;&#039;[...]/bin/python: error while loading shared libraries: libpython3.13.so.1.0: cannot open shared object file: No such file or directory&#039;&#039;&amp;quot;&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;&#039;&#039;&#039;How can I navigate to my SDS@hd folder in the file browser?&#039;&#039;&#039;&amp;lt;/br&amp;gt;&lt;br /&gt;
Please see the instructions at the [[Helix/bwVisu/Usage#Files | Usage]] page.&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;&#039;&#039;&#039;I prefer VSCode over JupyterLab. Can I start a JupyterLab job and then connect with it via VSCode?&#039;&#039;&#039;&amp;lt;/br&amp;gt;&lt;br /&gt;
This is not possible. Please start the job directly on Helix instead. You can find the instructions at the [[Development/VS_Code#Connect_to_Remote_Jupyter_Kernel |VSCode page]].&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;&#039;&#039;&#039;My conda commands are interrupted with message &#039;Killed&#039;.&#039;&#039;&#039;&amp;lt;/br&amp;gt;&lt;br /&gt;
Request more memory when starting Jupyter.&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;&#039;&#039;&#039;Jupyterlab doesn&#039;t let me in but asks for a password.&#039;&#039;&#039;&amp;lt;/br&amp;gt;&lt;br /&gt;
Try using more memory for the job. If this doesn&#039;t help, try using the inkognito mode of your browser as the browser cache might be the problem.&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;/ol&amp;gt;&lt;/div&gt;</summary>
		<author><name>I Tachakor</name></author>
	</entry>
	<entry>
		<id>https://wiki.bwhpc.de/wiki/index.php?title=Helix/bwVisu/JupyterLab&amp;diff=16470</id>
		<title>Helix/bwVisu/JupyterLab</title>
		<link rel="alternate" type="text/html" href="https://wiki.bwhpc.de/wiki/index.php?title=Helix/bwVisu/JupyterLab&amp;diff=16470"/>
		<updated>2026-10-05T20:11:51Z</updated>

		<summary type="html">&lt;p&gt;I Tachakor: Helix/bwVisu/JupyterLab&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;[https://jupyter.org/ JupyterLab] is an integrated development environment (IDE) that provides a flexible and scalable interface for the Jupyter Notebook system. It supports interactive data science and scientific computing across over 40 programming languages (including Python, Julia, and R).&lt;br /&gt;
&lt;br /&gt;
== Change Python Version ==&lt;br /&gt;
&lt;br /&gt;
The default python version can be seen by running &amp;lt;code&amp;gt;python --version&amp;lt;/code&amp;gt; in the terminal.&lt;br /&gt;
&lt;br /&gt;
A different python version can be installed into a new virtual environment and then registered as IPython kernel for the usage in JupyterLab. This is explained in the chapter [[#Add_packages_via_conda_environments | add packages via conda environments]].&lt;br /&gt;
&lt;br /&gt;
== Install Python Packages ==&lt;br /&gt;
&lt;br /&gt;
Python packages can be added by installing them into a virtual environment and then creating an IPython kernel from the virtual environment. &amp;lt;/br&amp;gt;&lt;br /&gt;
&amp;lt;u&amp;gt;Kernels can be shared&amp;lt;/u&amp;gt;. See the notes below. &amp;lt;/br&amp;gt;&lt;br /&gt;
If you want to move a virtual environment, it is adivsed to recreate it in the new place. Otherwise, dependencies based on relative paths will break.&lt;br /&gt;
&lt;br /&gt;
# Create a virtual environment with...&lt;br /&gt;
#* [[#Add_packages_via_venv_virtual_environments | ...venv]] or&lt;br /&gt;
#* [[#Add_packages_via_Conda_virtual_environments | ...conda]] (choose this option if you want to install a different python version) or&lt;br /&gt;
#* [[#Add_packages_via_uv_virtual_environments | ...uv]] if you want to install a different python version but don&#039;t want to use conda.&lt;br /&gt;
# [[#Create_an_IPython_Kernel | Create an IPython kernel]] from the virtual environment&lt;br /&gt;
# Use the kernel within JupyterLab&lt;br /&gt;
#* By default new kernels are saved under &amp;lt;code&amp;gt;~/.local/share/jupyter&amp;lt;/code&amp;gt; and this location is automatically detected. Therefore, new kernels are directly available.&lt;br /&gt;
#*:[[File:BwVisu JuypterLab KernelPath.png|In the JupyterLab job configuration form, a custom kernel path can be provided.|right|thumb|x150px]]&lt;br /&gt;
#* If the kernel is saved somewhere else, the path can be provided in the &amp;quot;Kernel path&amp;quot; field when configuring the JupyterLab job (see image). For a kernel placed under &amp;lt;code&amp;gt;path_to_parent_dir/share/jupyter/kernels/my_kernel&amp;lt;/code&amp;gt; the needed &amp;quot;Kernel path&amp;quot; would be &amp;lt;code&amp;gt;path_to_parent_dir/share/jupyter&amp;lt;/code&amp;gt;.&lt;br /&gt;
# When the kernel is used the first time, the file &amp;lt;code&amp;gt;notebook_secrets&amp;lt;/code&amp;gt; is created automatically. It can be found under &amp;quot;Kernel path&amp;quot;. For others to use the kernel, they must have read access to this file. The command &amp;lt;code&amp;gt;chmod 750 notebook_secrets&amp;lt;/code&amp;gt; would for example allow the whole SDS@hd SV read access. For the access management in workspaces, please see [[Workspace#Setting_Permissions_for_Sharing_Files| Workspace Permissions]].&lt;br /&gt;
&lt;br /&gt;
&amp;lt;u&amp;gt;Notes regarding the sharing of IPython kernels&amp;lt;/u&amp;gt;&lt;br /&gt;
* The virtual environment and the kernel need to be placed in a shared directory. For example at SDS@hd.&lt;br /&gt;
* There could be a subdirectory for the virtual environments and one for the kernels.&lt;br /&gt;
* The path to the kernels is saved in the environment variable $JUPYTER_DATA_DIR. Jupyter relevant paths can be seen with &amp;lt;code&amp;gt;jupyter --paths&amp;lt;/code&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Add packages via &#039;&#039;venv&#039;&#039; virtual environments ===&lt;br /&gt;
More information about venv or other python virtual environments can be found at the [[Development/Python | Python]] page.&lt;br /&gt;
&lt;br /&gt;
Steps for creating a &#039;&#039;&#039;venv&#039;&#039;&#039; virtual environment:&lt;br /&gt;
&amp;lt;ol style=&amp;quot;list-style-type: decimal;&amp;quot;&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Open a terminal.&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Create a new virtual evironment:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;python3 -m venv &amp;lt;env_parent_dir&amp;gt;/&amp;lt;env_name&amp;gt;&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
* &amp;lt;code&amp;gt;env_parent_dir&amp;lt;/code&amp;gt; is the path to the folder where the virtual environment shall be created. Relative paths can be used.&lt;br /&gt;
* Caution: If you you want to share the environment with others, make sure to already create it in the shared place.&lt;br /&gt;
&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Activate the environment:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;source &amp;lt;env_parent_dir&amp;gt;/&amp;lt;env_name&amp;gt;/bin/activate&amp;lt;/syntaxhighlight&amp;gt;&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Update pip and install packages:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;pip install -U pip --no-user&lt;br /&gt;
# when the environment is installed in home&lt;br /&gt;
pip install &amp;lt;packagename&amp;gt;&lt;br /&gt;
# when the environment is installed somewhere else and shall not have dependencies in home so that others can access it as well&lt;br /&gt;
pip install &amp;lt;packagename&amp;gt; --no-user&amp;lt;/syntaxhighlight&amp;gt;&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt; [[#Create_an_IPython_Kernel | Create an IPython kernel]]&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;/ol&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Add packages via Conda virtual environments ===&lt;br /&gt;
More information about using conda can be found at the [[Development/Conda | Conda]] page.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;ol style=&amp;quot;list-style-type: decimal;&amp;quot;&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Load the miniforge module by clicking first on the double hexagon icon on the left-hand side of Jupyter&#039;s start page and then on the &amp;amp;quot;load&amp;amp;quot; button right of the entry for miniforge in the software module menu.&lt;br /&gt;
&amp;lt;li&amp;gt;Open a terminal.&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Create a new virtual environment:&lt;br /&gt;
* If you are the only person using the environment, you can install it in your home directory:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;conda create --name &amp;lt;env_name&amp;gt; python=&amp;lt;python version&amp;gt;&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
* If you want to install it into a different directory, for example a shared place:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;conda create --prefix &amp;lt;path_to_shared_directory&amp;gt;/&amp;lt;env_name&amp;gt; python=&amp;lt;python version&amp;gt;&lt;br /&gt;
&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Activate your environment:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;conda activate &amp;lt;myenv&amp;gt;&amp;lt;/syntaxhighlight&amp;gt;&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Install your packages:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;conda install &amp;lt;mypackage&amp;gt;&amp;lt;/syntaxhighlight&amp;gt;&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt; [[#Create_an_IPython_Kernel | Create an IPython kernel]]&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;/ol&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Add packages via uv virtual environments ===&lt;br /&gt;
More information about using &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; can be found at the [[Development/Python | Python]] page.&lt;br /&gt;
&lt;br /&gt;
Steps for creating a &#039;&#039;&#039;uv&#039;&#039;&#039; virtual environment:&lt;br /&gt;
&amp;lt;ol style=&amp;quot;list-style-type: decimal;&amp;quot;&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Open a terminal.&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Create a new virtual environment:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;uv venv &amp;lt;env_parent_dir&amp;gt;/&amp;lt;env_name&amp;gt; --python &amp;lt;python version&amp;gt;&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
* &amp;lt;code&amp;gt;env_parent_dir&amp;lt;/code&amp;gt; is the path to the folder where the virtual environment shall be created. Relative paths can be used.&lt;br /&gt;
* The &amp;lt;code&amp;gt;--python&amp;lt;/code&amp;gt; option can be omitted if the default Python version should be used.&lt;br /&gt;
* Caution: If you want to share the environment with others, make sure to already create it in the shared place.&lt;br /&gt;
&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Install packages into the environment:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;uv pip install --python &amp;lt;env_parent_dir&amp;gt;/&amp;lt;env_name&amp;gt;/bin/python &amp;lt;packagename&amp;gt;&amp;lt;/syntaxhighlight&amp;gt;&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt; [[#Create_an_IPython_Kernel | Create an IPython kernel]]&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;/ol&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Unlike a traditional &amp;lt;code&amp;gt;venv&amp;lt;/code&amp;gt; environment, a &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; environment does not have to be activated with &amp;lt;code&amp;gt;source&amp;lt;/code&amp;gt;. The &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; commands can directly use the Python interpreter from the environment.&lt;br /&gt;
&lt;br /&gt;
== Create an IPython Kernel ==&lt;br /&gt;
&lt;br /&gt;
Python kernels are implementations of the Jupyter notebook environment for different languages or virtual environments. You can switch between kernels easily, allowing you to use the best tool for a specific task.&lt;br /&gt;
conda_kernels&lt;br /&gt;
&lt;br /&gt;
=== Create a kernel from a virtual environment ===&lt;br /&gt;
&lt;br /&gt;
For a &#039;&#039;&#039;venv&#039;&#039;&#039; or &#039;&#039;&#039;Conda&#039;&#039;&#039; environment, the environment has to be activated before creating the kernel.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;ol&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Activate the virtual environment.&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Install the ipykernel package:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;pip install ipykernel&amp;lt;/syntaxhighlight&amp;gt;&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Register the virtual environment as custom kernel to Jupyter.&lt;br /&gt;
* If you are the only person using the environment:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;python3 -m ipykernel install --user --name=&amp;lt;kernel_name&amp;gt;&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
The kernel can be found under &amp;lt;code&amp;gt;~/.local/share/jupyter/kernels/&amp;lt;/code&amp;gt;.&lt;br /&gt;
* If you installed the environment in a shared place and want to have the kernel there as well:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;python3 -m ipykernel install --prefix &amp;lt;path_to_kernel_folder&amp;gt; --name=&amp;lt;kernel_name&amp;gt;&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
The kernel can then be found under &amp;lt;code&amp;gt;path_to_kernel_folder/share/jupyter/kernels/&amp;lt;kernel_name&amp;gt;&amp;lt;/code&amp;gt;. As long as the same &amp;lt;code&amp;gt;path_to_kernel_folder&amp;lt;/code&amp;gt; is used, all kernels will be saved next to each other in &amp;quot;kernels&amp;quot;.&lt;br /&gt;
&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;/ol&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Create a kernel from a uv virtual environment ===&lt;br /&gt;
&lt;br /&gt;
A kernel can also be created from a &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; virtual environment. The environment does not have to be activated.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;ol&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Install the &amp;lt;code&amp;gt;ipykernel&amp;lt;/code&amp;gt; package into the &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; environment:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;uv pip install --python &amp;lt;env_parent_dir&amp;gt;/&amp;lt;env_name&amp;gt;/bin/python ipykernel&amp;lt;/syntaxhighlight&amp;gt;&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Register the environment as a custom kernel using &amp;lt;code&amp;gt;uv run&amp;lt;/code&amp;gt;:&lt;br /&gt;
* If you are the only person using the environment:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;uv run --directory &amp;lt;env_parent_dir&amp;gt;/&amp;lt;env_name&amp;gt; python -m ipykernel install --user --name=&amp;lt;kernel_name&amp;gt;&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
The kernel can be found under &amp;lt;code&amp;gt;~/.local/share/jupyter/kernels/&amp;lt;/code&amp;gt;.&lt;br /&gt;
* If you installed the environment in a shared place and want to have the kernel there as well:&lt;br /&gt;
&amp;lt;syntaxhighlight lang=&amp;quot;bash&amp;quot;&amp;gt;uv run --directory &amp;lt;env_parent_dir&amp;gt;/&amp;lt;env_name&amp;gt; python -m ipykernel install --prefix &amp;lt;path_to_kernel_folder&amp;gt; --name=&amp;lt;kernel_name&amp;gt;&amp;lt;/syntaxhighlight&amp;gt;&lt;br /&gt;
The kernel can then be found under &amp;lt;code&amp;gt;path_to_kernel_folder/share/jupyter/kernels/&amp;lt;kernel_name&amp;gt;&amp;lt;/code&amp;gt;. As long as the same &amp;lt;code&amp;gt;path_to_kernel_folder&amp;lt;/code&amp;gt; is used, all kernels will be saved next to each other in &amp;quot;kernels&amp;quot;.&lt;br /&gt;
&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;/ol&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Multi-Language Support ===&lt;br /&gt;
&lt;br /&gt;
JupyterLab supports over 40 programming languages including Python, R, Julia, and Scala. This is achieved through the use of different kernels.&lt;br /&gt;
&lt;br /&gt;
==== R Kernel ====&lt;br /&gt;
&lt;br /&gt;
&amp;lt;ol style=&amp;quot;list-style-type: decimal;&amp;quot;&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;On the cluster:&lt;br /&gt;
&amp;lt;pre&amp;gt;$ module load math/R&lt;br /&gt;
$ R&lt;br /&gt;
&amp;amp;gt; install.packages(&#039;IRkernel&#039;)&amp;lt;/pre&amp;gt;&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;On bwVisu:&lt;br /&gt;
&amp;lt;ol style=&amp;quot;list-style-type: decimal;&amp;quot;&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Start Jupyter App&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;In left menu: load math/R&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Open Console:&amp;lt;/li&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;$ R&lt;br /&gt;
&amp;amp;gt; IRkernel::installspec(displayname = &#039;R 4.2&#039;)&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;li&amp;gt;Start kernel &#039;R 4.2&#039; as console or notebook&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;/ol&amp;gt;&lt;br /&gt;
&amp;lt;/ol&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==== Julia Kernel ====&lt;br /&gt;
&lt;br /&gt;
Load the math/julia module. Open the Terminal.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
julia&lt;br /&gt;
]&lt;br /&gt;
add IJulia&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
After that, Julia is available as a kernel.&lt;br /&gt;
&lt;br /&gt;
== Interactive Widgets ==&lt;br /&gt;
&lt;br /&gt;
JupyterLab supports interactive widgets that can create UI controls for interactive data visualization and manipulation within the notebooks. Example of using an interactive widget:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre class=&amp;quot;{.python&amp;quot;&amp;gt;from ipywidgets import IntSlider&lt;br /&gt;
slider = IntSlider()&lt;br /&gt;
display(slider)&amp;lt;/pre&amp;gt;&lt;br /&gt;
These widgets can be sliders, dropdowns, buttons, etc., which can be connected to Python code running in the backend.&lt;br /&gt;
&lt;br /&gt;
== FAQ ==&lt;br /&gt;
&lt;br /&gt;
&amp;lt;ol&amp;gt;&amp;lt;li&amp;gt;&#039;&#039;&#039;How can I use bwForCluster Helix software modules?&#039;&#039;&#039;&amp;lt;/br&amp;gt;&lt;br /&gt;
First click on the double hexagon icon on the left-hand side of Jupyter&#039;s start page. Then load a module by clicking on the &amp;quot;load&amp;quot; button next to the corresponding module entry.&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;&#039;&#039;&#039;My virtual environment works on Helix but not in the bwVisu JupyterLab job.&#039;&#039;&#039;&amp;lt;/br&amp;gt;&lt;br /&gt;
If the environment uses Helix modules, you have to load the modules in the bwVisu job first. For example, when you used the python 3.13 module for creating the environment, the path to this python version is saved in the environment. When the module is not loadedin bwVisu, the path is not available and you might get an error like &amp;quot;&#039;&#039;[...]/bin/python: error while loading shared libraries: libpython3.13.so.1.0: cannot open shared object file: No such file or directory&#039;&#039;&amp;quot;&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;&#039;&#039;&#039;How can I navigate to my SDS@hd folder in the file browser?&#039;&#039;&#039;&amp;lt;/br&amp;gt;&lt;br /&gt;
Please see the instructions at the [[Helix/bwVisu/Usage#Files | Usage]] page.&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;&#039;&#039;&#039;I prefer VSCode over JupyterLab. Can I start a JupyterLab job and then connect with it via VSCode?&#039;&#039;&#039;&amp;lt;/br&amp;gt;&lt;br /&gt;
This is not possible. Please start the job directly on Helix instead. You can find the instructions at the [[Development/VS_Code#Connect_to_Remote_Jupyter_Kernel |VSCode page]].&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;&#039;&#039;&#039;My conda commands are interrupted with message &#039;Killed&#039;.&#039;&#039;&#039;&amp;lt;/br&amp;gt;&lt;br /&gt;
Request more memory when starting Jupyter.&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;&#039;&#039;&#039;Jupyterlab doesn&#039;t let me in but asks for a password.&#039;&#039;&#039;&amp;lt;/br&amp;gt;&lt;br /&gt;
Try using more memory for the job. If this doesn&#039;t help, try using the inkognito mode of your browser as the browser cache might be the problem.&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;/ol&amp;gt;&lt;/div&gt;</summary>
		<author><name>I Tachakor</name></author>
	</entry>
	<entry>
		<id>https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster3.0/Jupyter&amp;diff=16469</id>
		<title>BwUniCluster3.0/Jupyter</title>
		<link rel="alternate" type="text/html" href="https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster3.0/Jupyter&amp;diff=16469"/>
		<updated>2026-10-05T19:46:17Z</updated>

		<summary type="html">&lt;p&gt;I Tachakor: /* Virtual environments */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;Jupyter can be used as an alternative to accessing HPC resources via SSH. For this purpose only a web browser is required. Within the website source code of different programming languages can be edited and executed. Furthermore different user interfaces and terminals are available.&lt;br /&gt;
&lt;br /&gt;
= Short description of Jupyter =&lt;br /&gt;
&lt;br /&gt;
Jupyter is a web application, central component of Jupyter is the &#039;&#039;&#039;Jupyter Notebook&#039;&#039;&#039;. It is a document, which can contain formatted text, executable code sections and (interactive) visualizations (image, sound, video, 3D views).&lt;br /&gt;
&lt;br /&gt;
The Jupyter notebooks are executed in an interactive session on the compute nodes of the respective cluster. Access is via any modern web browser. Data is prepared and visualized on the server and therefore does not have to be transmitted over the network. Only the resulting text, image, sound and video data is transmitted. Starting point of a Jupyter session is the HOME directory of the user on the respective cluster. &lt;br /&gt;
&lt;br /&gt;
&#039;&#039;&#039;JupyterLab&#039;&#039;&#039; is a modern user interface, within which one or more Jupyter notebooks can be opened, edited and executed. The individual notebooks can be arranged as tabs or tiled. JupyterLab is the standard user interface. Besides JupyterLab the classic notebook user interface is available, in which only one Jupyter notebook per browser tab can be opened at a time.&lt;br /&gt;
&lt;br /&gt;
A &#039;&#039;&#039;Jupyter Kernel&#039;&#039;&#039; describes a separate process, in which one Jupyter Notebook is executed at a time. Different kernels are available for different programming languages or language versions.&lt;br /&gt;
&lt;br /&gt;
Before a Jupyter session is started, the access authorization must be checked first. This is done via &#039;&#039;&#039;JupyterHub&#039;&#039;&#039;, where the resources are selected, for example the number of CPU cores, GPUs or the required main memory.&lt;br /&gt;
&lt;br /&gt;
A detailed documentation of the Jupyter project can be found at [https://jupyter.readthedocs.io https://jupyter.readthedocs.io].&lt;br /&gt;
&lt;br /&gt;
= Access requirements =&lt;br /&gt;
&lt;br /&gt;
{|style=&amp;quot;background:#deffee; width:100%;&amp;quot;&lt;br /&gt;
|style=&amp;quot;padding:5px; background:#cef2e0; text-align:left&amp;quot;|&lt;br /&gt;
[[Image:Attention.svg|center|25px]]&lt;br /&gt;
|style=&amp;quot;padding:5px; background:#cef2e0; text-align:left&amp;quot;|&lt;br /&gt;
Access to Jupyter is &#039;&#039;&#039;limited to IP addresses from the BelWü network&#039;&#039;&#039;.&lt;br /&gt;
All home institutions of our current users are connected to BelWü, so if you are on your campus network (e.g. in your office or on the Campus WiFi) you should be able to connect to bwUniCluster 3.0 without restrictions.&lt;br /&gt;
If you are outside one of the BelWü networks (e.g. at home), a VPN connection to the home institution or a connection to an SSH jump host at the home institution must be established first.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
To use Jupyter on the HPC resources of SCC, the access requirements for [https://wiki.bwhpc.de/e/Registration/bwUniCluster bwUniCluster 3.0] apply. A [https://wiki.bwhpc.de/e/Registration/bwUniCluster registration] is required. Please note, You should&#039;ve completed registration and tested your login once using [https://wiki.bwhpc.de/e/Registration/SSH Secure Shell (ssh)].&lt;br /&gt;
&lt;br /&gt;
= Login process =&lt;br /&gt;
&lt;br /&gt;
Login takes place at &lt;br /&gt;
* bwUniCluster 3.0: [https://uc3-jupyter.scc.kit.edu uc3-jupyter.scc.kit.edu]&lt;br /&gt;
* SDIL: [https://sdil-jupyter.scc.kit.edu sdil-jupyter.scc.kit.edu]&lt;br /&gt;
* HoreKa: [https://hk-jupyter.scc.kit.edu hk-jupyter.scc.kit.edu]&lt;br /&gt;
* HAICORE: [https://haicore-jupyter.scc.kit.edu haicore-jupyter.scc.kit.edu]&lt;br /&gt;
&lt;br /&gt;
For login, your username, your password and a 2-factor authentication are required.&lt;br /&gt;
&lt;br /&gt;
You will first find yourself on a landing page that also gives more information about the currently installed software versions.&lt;br /&gt;
By pressing the login button you will be redirected to the JupyterHub page. Click on Enter JupyterHub to start the login process. Select the organization (e.g. KIT) that has granted you access to the HPC system and press Continue. In the Login section that appears, enter your username and password (not the service password). &lt;br /&gt;
After pressing the Login button you will be redirected to the second factor query page. Enter the one-time password (e.g. from KIT Token or Google Authenticator App) and press Validate. Now you are done with the login process and can start selecting your computing resources.&lt;br /&gt;
&lt;br /&gt;
[[File:Jupyter_Anmeldung.gif|700px]]&lt;br /&gt;
&lt;br /&gt;
= Selection of the compute resources =&lt;br /&gt;
&lt;br /&gt;
The Jupyter notebooks are executed in an interactive session on the compute nodes of the HPC clusters. Just like accessing an interactive session with SSH, resource allocation is done by the Workload Manager Slurm. The selection of resources for Jupyter is realized via drop-down menus. Only jobs with a maximum of one node are possible.&lt;br /&gt;
&lt;br /&gt;
Available resources for selection are&lt;br /&gt;
&lt;br /&gt;
* Number of CPU cores&lt;br /&gt;
* Number of GPUs&lt;br /&gt;
* Runtime&lt;br /&gt;
* Partition/Queue&lt;br /&gt;
* Amount of main memory&lt;br /&gt;
&lt;br /&gt;
If Auto-Reservation is selected the automatic Jupyter reservation of the cluster is enabled.&lt;br /&gt;
&lt;br /&gt;
In normal mode, the grayed-out fields contain reasonable presets, depending on the number of required CPU cores or GPUs respectively. The presets can be bypassed in advanced mode, where further options are available. &lt;br /&gt;
&lt;br /&gt;
Advanced Mode can be activated by clicking on the checkbox of the same name. The following additional options then become available:&lt;br /&gt;
&lt;br /&gt;
* Specification of a reservation&lt;br /&gt;
* LSDF mount option&lt;br /&gt;
* BEEOND mount option&lt;br /&gt;
&lt;br /&gt;
After the selection is made, the interactive job is started with the spawn button. As when requesting interactive compute resources with the `salloc` command, waiting times may occur. These are usually the longer the larger the requested resources are.&lt;br /&gt;
Even if the chosen resources are available immediately, the spawning process may take up to one minute.&lt;br /&gt;
&lt;br /&gt;
[[File:Ressources_neu.gif|500px]]&lt;br /&gt;
&lt;br /&gt;
Please note that in advanced mode, resource combinations can be selected that are impossible to be met. In this case, an error message will appear when the job is spawned.&lt;br /&gt;
&lt;br /&gt;
[[File:Jupyter_Falsche_ressourcen.gif|500px]]&lt;br /&gt;
&lt;br /&gt;
The spawning timeout is currently set to 10 minutes. With a normal workload of the HPC facility, this time is usually sufficient to get interactive resources.&lt;br /&gt;
&lt;br /&gt;
== Prioritized access to computing resources on bwUniCluster 3.0 ==&lt;br /&gt;
The use of Jupyter requires the immediate availability of computing resources since the JupyterLab server is started within an interactive Slurm session. To improve the availability of CPUs/GPUs for interactive supercomputing with Jupyter, &#039;&#039;&#039;automatic reservation&#039;&#039;&#039; for CPU (cpu_il) and GPU (gpu_a100_il) resources has been set up on &#039;&#039;&#039;bwUniCluster 3.0&#039;&#039;&#039;. It is active &#039;&#039;&#039;between 8am and 8pm&#039;&#039;&#039; every weekday. The reservation is automatically active if&lt;br /&gt;
&lt;br /&gt;
* no other reservation is set manually&lt;br /&gt;
* Auto-Reservation is enabled&lt;br /&gt;
&lt;br /&gt;
To give you a better overview of the currently available resources, a status indicator has been implemented. It appears when selecting the number of required CPUs/GPUs and shows whether a Jupyter job of the selected size can currently be started or not. Green means the selected CPU/GPU resources are available instantly. Yellow means only a single additional job of the selected size can be started. Red means there are no GPU resources left that could satisfy the selected amount of resources.&lt;br /&gt;
&lt;br /&gt;
If there are no more resources available within the reservation, you can try selecting a different amount of CPUs/GPUs or activate Advanced Mode and select a different partition. Availability can be estimated using sinfo_t_idle, which is available when logging in via SSH.&lt;br /&gt;
&lt;br /&gt;
= JupyterLab =&lt;br /&gt;
&lt;br /&gt;
JupyterLab is the standard user interface. In the following only its essential functions are briefly introduced. A detailed documentation is available at &lt;br /&gt;
[https://jupyterlab.readthedocs.io https://jupyterlab.readthedocs.io].&lt;br /&gt;
&lt;br /&gt;
== Menu bar ==&lt;br /&gt;
&lt;br /&gt;
The menu bar at the upper edge of JupyterLab has higher-level menus that display the actions available in JupyterLab along with their shortcut keys. The default menus are:&lt;br /&gt;
&lt;br /&gt;
* File: Actions related to files and directories&lt;br /&gt;
* Edit: Actions related to editing documents and other activities&lt;br /&gt;
* View: actions that change the appearance of JupyterLab&lt;br /&gt;
* Run: Actions to execute code in various activities like notebooks and code consoles&lt;br /&gt;
* Kernel: Actions to manage kernels that are separate processes for executing code&lt;br /&gt;
* Tabs: a list of open documents and activities in the Dock Panel&lt;br /&gt;
* Settings: general settings and an editor for advanced settings&lt;br /&gt;
* Help: a list of help links to JupyterLab and the kernel&lt;br /&gt;
&lt;br /&gt;
== Left sidebar ==&lt;br /&gt;
&lt;br /&gt;
In the left sidebar there are foldable tabs. The most relevant ones are:&lt;br /&gt;
&lt;br /&gt;
* File browser: Switch to directories and open files with left mouse button, context menu with right mouse button&lt;br /&gt;
* Running kernels: Overview of running kernels&lt;br /&gt;
* Command overview&lt;br /&gt;
* Tab Overview&lt;br /&gt;
* Lmod software selection: Search and load/unload Lmod software modules&lt;br /&gt;
&lt;br /&gt;
== Main working area ==&lt;br /&gt;
The main working area in JupyterLab allows to arrange, resize and divide documents (notebooks, text files, etc.) and other activities (terminals, code consoles, etc.) in tabs. By holding down the left mouse button, the tabs can be grabbed and repositioned.&lt;br /&gt;
&lt;br /&gt;
In a new JupyterLab session the Launcher tab is opened first. It contains buttons for starting new notebooks, code consoles and other functions. When a notebook is open, a new Launcher tab can be started by pressing the plus symbol in the file browser tab of the left sidebar, by calling &#039;&#039;File &amp;gt; New Launcher&#039;&#039; in the upper menu bar or by the key combination &#039;&#039;Ctrl+Shift+L&#039;&#039;.&lt;br /&gt;
&lt;br /&gt;
== Classic Notebook ==&lt;br /&gt;
&lt;br /&gt;
The classic Jupyter Notebook user interface offers only one open Jupyter Notebook or terminal per browser tab. From the JupyterLab user interface the classic display can be reached in the menu bar under &#039;&#039;Help &amp;gt; Launch Classic Notebook&#039;&#039;. Clicking on the JupyterHub logo in the upper left corner will take you back to the JupyterLab interface.&lt;br /&gt;
&lt;br /&gt;
= Log out =&lt;br /&gt;
&lt;br /&gt;
You can log out from a running Jupyter session by calling &#039;&#039;File &amp;gt; Log Out&#039;&#039; in the upper menu bar. &lt;br /&gt;
&lt;br /&gt;
{| style=&amp;quot;width: 100%; margin:4px 0 0 0; background:none; border-spacing: 0px;&amp;quot;&lt;br /&gt;
| style=&amp;quot;width:100%; border:1px solid #BBBBBB; background:#fff5fa; vertical-align:top; color:#000;&amp;quot; |&lt;br /&gt;
{| style=&amp;quot;width:100%; vertical-align:top; border:0px solid #BBBBBB; padding:4px;&amp;quot; |&lt;br /&gt;
|-&lt;br /&gt;
|{{Red}}| Attention&lt;br /&gt;
|-&lt;br /&gt;
|&lt;br /&gt;
Please note that your interactive session will continue in the background!  &lt;br /&gt;
&amp;lt;!--For example, this affects your computing time quota on the ForHLR.--&amp;gt;&lt;br /&gt;
|}&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
As long as the interactive session is running, you can re-enter it at any time. Depending on the duration of your absence, it may be necessary to re-enter your one-time password and possibly KIT password.&lt;br /&gt;
&lt;br /&gt;
If you want to end the interactive session before it has reached its runtime, you can do so via the Hub Control Panel. Under &#039;&#039;File &amp;gt; Hub Control Panel&#039;&#039; in the upper menu bar, it is opened in a new browser tab. By pressing the &#039;&#039;Stop My Server&#039;&#039; button the session will be terminated. You can now log out using the &#039;&#039;Logout&#039;&#039; button in the upper right corner or start a new session directly using the &#039;&#039;Start My Server&#039;&#039; button, for example with a changed resource selection.&lt;br /&gt;
&lt;br /&gt;
[[File:logout_small.gif|750px]]&lt;br /&gt;
&lt;br /&gt;
= Selection of software =&lt;br /&gt;
&lt;br /&gt;
For the selection of the required Lmod software modules the corresponding tab &#039;&#039;Softwares&#039;&#039; is available in the left sidebar. The list of available modules can be narrowed down by entering the search field. The desired module is loaded by pressing the &#039;&#039;Load&#039;&#039; button. In the list with the loaded modules you can remove them with the &#039;&#039;Unload&#039;&#039; button.&lt;br /&gt;
&lt;br /&gt;
{| style=&amp;quot;width: 100%; margin:4px 0 0 0; background:none; border-spacing: 0px;&amp;quot;&lt;br /&gt;
| style=&amp;quot;width:100%; border:1px solid #BBBBBB; background:#f5fffa; vertical-align:top; color:#000;&amp;quot; |&lt;br /&gt;
{| style=&amp;quot;width:100%; vertical-align:top; border:0px solid #BBBBBB; padding:4px;&amp;quot; |&lt;br /&gt;
|-&lt;br /&gt;
|{{Green}}| Note&lt;br /&gt;
|-&lt;br /&gt;
|&lt;br /&gt;
On already opened Jupyter Notebooks, newly loaded software modules become active only after restarting the kernel (&#039;&#039;Kernel &amp;gt; Restart Kernel&#039;&#039; in the upper menu bar). Terminals must be closed and reopened.&lt;br /&gt;
|}&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
[[File:software_small.gif|750px]]&lt;br /&gt;
&lt;br /&gt;
== Software Stacks for Jupyter ==&lt;br /&gt;
Currently 3 special Jupyter software stacks are available via Lmod:&lt;br /&gt;
&lt;br /&gt;
* &amp;lt;code&amp;gt;jupyter/minimal&amp;lt;/code&amp;gt;&lt;br /&gt;
*: Minimal installation of JupyterLab&lt;br /&gt;
&lt;br /&gt;
* &amp;lt;code&amp;gt;jupyter/base&amp;lt;/code&amp;gt; &lt;br /&gt;
*: Basic installation of JupyterLab.&lt;br /&gt;
*: For a complete list of pre-installed packages, please refer to [https://uc3-jupyter.scc.kit.edu/software-modules/#pre-installed-software-packages this site].&lt;br /&gt;
&lt;br /&gt;
* &amp;lt;code&amp;gt;jupyter/tensorflow&amp;lt;/code&amp;gt; (default at login, will be deprecated with the advent of bwUniCluster 3.0)&lt;br /&gt;
*: Preinstalled software packages for machine learning applications. Includes among others TensorFlow, Keras, Torch, Pandas, Matplotlib, SKLearn.&lt;br /&gt;
*: For a complete list of pre-installed packages and their respective version, please refer to [https://uc3-jupyter.scc.kit.edu/software-modules/#pre-installed-software-packages this site].&lt;br /&gt;
&lt;br /&gt;
* &amp;lt;code&amp;gt;jupyter/ai&amp;lt;/code&amp;gt; (will be the new default at login für bwUniCluster 3.0, contains all latest and greatest software for AI workflows)&lt;br /&gt;
*: Preinstalled software packages for machine learning applications. Includes among others TensorFlow, Keras, Torch, Torchvision, Lighning, Pandas, Matplotlib, SKLearn.&lt;br /&gt;
*: For a complete list of pre-installed packages and their respective version, please refer to [https://uc3-jupyter.scc.kit.edu/software-modules/#pre-installed-software-packages this site].&lt;br /&gt;
&lt;br /&gt;
* &amp;lt;code&amp;gt;jupyter/extensions&amp;lt;/code&amp;gt;&lt;br /&gt;
*: Same packages as tensorflow + extensions&lt;br /&gt;
&lt;br /&gt;
These software stacks can be used both when accessing the cluster via JupyterHub, as well as for conventional access via SSH via module load.&lt;br /&gt;
&lt;br /&gt;
A continuously updated list with the installed packages can be found on the corresponding subpage of the respective cluster:&lt;br /&gt;
&lt;br /&gt;
* bwUniCluster 3.0: [https://uc3-jupyter.scc.kit.edu/software-modules uc3-jupyter.scc.kit.edu/software-modules]&lt;br /&gt;
* HoreKa: [https://hk-jupyter.scc.kit.edu/software-modules hk-jupyter.scc.kit.edu/software-modules]&lt;br /&gt;
&lt;br /&gt;
= Installation of further software =&lt;br /&gt;
The software provided by the Lmod modules jupyter/minimal, jupyter/base and jupyter/tensorflow can be easily supplemented by additional Python packages. There are 2 procedures for this.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;ul&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;User-Installation (not recommended)&amp;lt;br&amp;gt;&lt;br /&gt;
&amp;lt;code&amp;gt;pip install --user &amp;lt;packageName&amp;gt; &amp;lt;/code&amp;gt;&amp;lt;br&amp;gt;&lt;br /&gt;
The additional packages are installed under $HOME/.local/lib/python3.11/site-packages/ which is part of PYTHONPATH.&lt;br /&gt;
&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Virtual environments (recommended)&amp;lt;br&amp;gt;&lt;br /&gt;
The user can create and use virtual environments (cf. Virtual environments). Packages provided by the jupyter Lmod modules remain visible and usable.&lt;br /&gt;
&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;/ul&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Virtual environments ==&lt;br /&gt;
&lt;br /&gt;
Python virtual environments allow to use different versions of a package and to keep your local site-packages (accessible under &amp;lt;code&amp;gt;$PYTHONPATH&amp;lt;/code&amp;gt;) free from conflicts.&lt;br /&gt;
&lt;br /&gt;
=== Creation of virtual environment ===&lt;br /&gt;
&lt;br /&gt;
A virtual environment can be created using the standard Python &amp;lt;code&amp;gt;venv&amp;lt;/code&amp;gt; module:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
python -m venv &amp;lt;myEnv&amp;gt;&lt;br /&gt;
source &amp;lt;myEnv&amp;gt;/bin/activate&lt;br /&gt;
pip install &amp;lt;packageName&amp;gt;&lt;br /&gt;
deactivate&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The additional packages are installed under &amp;lt;code&amp;gt;&amp;lt;myEnv&amp;gt;/lib/python3.11/site-packages/&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
Alternatively, a virtual environment can be created and managed using &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt;:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
uv venv &amp;lt;myEnv&amp;gt;&lt;br /&gt;
uv add --directory &amp;lt;myEnv&amp;gt; &amp;lt;packageName&amp;gt;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
=== Usage of virtual environment ===&lt;br /&gt;
&lt;br /&gt;
In order to use a virtual environment created with &amp;lt;code&amp;gt;venv&amp;lt;/code&amp;gt;, it has to be activated via &amp;lt;code&amp;gt;source &amp;lt;myEnv&amp;gt;/bin/activate&amp;lt;/code&amp;gt;. &amp;lt;code&amp;gt;PYTHONPATH&amp;lt;/code&amp;gt; is set accordingly. Deactivation of the venv is done via &amp;lt;code&amp;gt;deactivate&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
For environments managed by &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt;, activation is not required. Python and other commands can be executed using &amp;lt;code&amp;gt;uv run&amp;lt;/code&amp;gt;, which automatically uses the environment managed by &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
=== Usage of virtual environment in JupyterLab ===&lt;br /&gt;
&lt;br /&gt;
To be able to use the virtual environments within JupyterLab, a corresponding kernel has to be installed:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
source &amp;lt;myEnv&amp;gt;/bin/activate&lt;br /&gt;
python -m ipykernel install \&lt;br /&gt;
   --user \&lt;br /&gt;
   --name myEnv \&lt;br /&gt;
   --display-name &amp;quot;Python (myEnv)&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
After installing the kernel (and possibly refreshing the browser window), a button named &amp;quot;myEnv&amp;quot; is available in JupyterLab. The kernel can also be selected from the drop-down menu.&lt;br /&gt;
&lt;br /&gt;
&#039;&#039;&#039;Attention&#039;&#039;&#039;&lt;br /&gt;
The (Lmod) base module you used in the Creation of virtual environment step must be loaded to use the venv. However, to be on the safe side, you can also use the system Python (&amp;lt;code&amp;gt;/usr/bin/python3.11&amp;lt;/code&amp;gt;) at creation time, which is available even without any &amp;lt;code&amp;gt;jupyter/{base,tensorflow}&amp;lt;/code&amp;gt; module loaded.&lt;br /&gt;
&lt;br /&gt;
As an alternative, an environment managed by &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; can be used. The corresponding Jupyter kernel can be installed with:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
uv run python -m ipykernel install \&lt;br /&gt;
   --user \&lt;br /&gt;
   --name myEnv \&lt;br /&gt;
   --display-name &amp;quot;Python (myEnv)&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&#039;&#039;&#039;Attention&#039;&#039;&#039;&lt;br /&gt;
The &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; environment must contain &amp;lt;code&amp;gt;ipykernel&amp;lt;/code&amp;gt;. If it is not already installed, it can be added to the environment with:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
uv add ipykernel&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Examples on Data processing, Machine Learning &amp;amp; Visualization ==&lt;br /&gt;
&lt;br /&gt;
In the [https://github.com/hpcraink/workshop-parallel-jupyter/ workshop repository] the usage and best practices on Python in general, and the packages NumPy, Pandas, SciKit and Dask are provided, containing running examples based on open data. It also explains, how Jupyter interacts with pre-installed and your own provided environments.&lt;br /&gt;
&lt;br /&gt;
== R language ==&lt;br /&gt;
&lt;br /&gt;
In order to use R language in JupyterLab, the Lmod module &amp;lt;code&amp;gt;math/R&amp;lt;/code&amp;gt; has to be loaded (blue button in JupyterLab or &amp;lt;code&amp;gt;module add math/R&amp;lt;/code&amp;gt; in terminal) and a corresponding kernel has to be installed.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
R&lt;br /&gt;
install.packages(&#039;IRkernel&#039;)&lt;br /&gt;
IRkernel::installspec()&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
After installing the kernel , a button named &amp;quot;R&amp;quot; is available in JupyterLab. The kernel can also be selected from the drop-down menu.&lt;br /&gt;
&lt;br /&gt;
&#039;&#039;&#039;Attention:&#039;&#039;&#039;&lt;br /&gt;
Don&#039;t forget to load the &amp;lt;code&amp;gt;math/R&amp;lt;/code&amp;gt; module (blue button) before using the kernel.&lt;br /&gt;
&lt;br /&gt;
== Julia language ==&lt;br /&gt;
&lt;br /&gt;
In order to use Julia language in JupyterLab, the Lmod module &amp;lt;code&amp;gt;math/julia/1.10.8&amp;lt;/code&amp;gt; has to be loaded (blue button in JupyterLab or &amp;lt;code&amp;gt;module math/julia/1.10.8&amp;lt;/code&amp;gt; in terminal). When the module is loaded from the JupytherLab UI, the corresponding kernel will be installed. If you use the terminal, this has to be done manually&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
julia&lt;br /&gt;
]&lt;br /&gt;
add IJulia&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
After installing the kernel, a button named &amp;quot;Julia 1.10.8&amp;quot; is available in JupyterLab. The kernel can also be selected from the drop-down menu.&lt;br /&gt;
&lt;br /&gt;
&#039;&#039;&#039;Attention:&#039;&#039;&#039;&lt;br /&gt;
Don&#039;t forget to load the &amp;lt;code&amp;gt;math/julia/1.10.8&amp;lt;/code&amp;gt; module (blue button) before using the kernel.&lt;br /&gt;
&lt;br /&gt;
= Jupyter Container Mode =&lt;br /&gt;
&lt;br /&gt;
The container integration on the jupyterhub is done via pyxis. In order to use it the checkmark for container mode has to be clicked. Available options are:&lt;br /&gt;
&lt;br /&gt;
* &amp;lt;code&amp;gt; --container-image:&amp;lt;/code&amp;gt; The container image to use. Corresponds to the pyxis option --container-image&lt;br /&gt;
* &amp;lt;code&amp;gt; --container-name:&amp;lt;/code&amp;gt; The name of the image to use. Corresponds to the pyxis option --container-name. Already downloaded containers in ~/.local/share/enroot can be startet by simply specifing their name.&lt;br /&gt;
* &amp;lt;code&amp;gt; --container-mount-home:&amp;lt;/code&amp;gt; Corresponds to the pyxis option --container-mount-home. Mounts the home-directory&lt;br /&gt;
&lt;br /&gt;
&#039;&#039;&#039;Attention:&#039;&#039;&#039;&lt;br /&gt;
Make sure Python3.11 and pip are installed in the Container or the notebook will not spawn. This can be checked via command python3.11 -m pip list inside your container&lt;br /&gt;
&lt;br /&gt;
It is advised to create the container prior via e.G. enroot and install all necessary software. For more information see [https://wiki.bwhpc.de/e/BwUniCluster2.0/Containers#SLURM_Integration here]&lt;br /&gt;
&lt;br /&gt;
----&lt;/div&gt;</summary>
		<author><name>I Tachakor</name></author>
	</entry>
	<entry>
		<id>https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster3.0/Jupyter&amp;diff=16468</id>
		<title>BwUniCluster3.0/Jupyter</title>
		<link rel="alternate" type="text/html" href="https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster3.0/Jupyter&amp;diff=16468"/>
		<updated>2026-10-05T19:35:48Z</updated>

		<summary type="html">&lt;p&gt;I Tachakor: /* Virtual environments */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;Jupyter can be used as an alternative to accessing HPC resources via SSH. For this purpose only a web browser is required. Within the website source code of different programming languages can be edited and executed. Furthermore different user interfaces and terminals are available.&lt;br /&gt;
&lt;br /&gt;
= Short description of Jupyter =&lt;br /&gt;
&lt;br /&gt;
Jupyter is a web application, central component of Jupyter is the &#039;&#039;&#039;Jupyter Notebook&#039;&#039;&#039;. It is a document, which can contain formatted text, executable code sections and (interactive) visualizations (image, sound, video, 3D views).&lt;br /&gt;
&lt;br /&gt;
The Jupyter notebooks are executed in an interactive session on the compute nodes of the respective cluster. Access is via any modern web browser. Data is prepared and visualized on the server and therefore does not have to be transmitted over the network. Only the resulting text, image, sound and video data is transmitted. Starting point of a Jupyter session is the HOME directory of the user on the respective cluster. &lt;br /&gt;
&lt;br /&gt;
&#039;&#039;&#039;JupyterLab&#039;&#039;&#039; is a modern user interface, within which one or more Jupyter notebooks can be opened, edited and executed. The individual notebooks can be arranged as tabs or tiled. JupyterLab is the standard user interface. Besides JupyterLab the classic notebook user interface is available, in which only one Jupyter notebook per browser tab can be opened at a time.&lt;br /&gt;
&lt;br /&gt;
A &#039;&#039;&#039;Jupyter Kernel&#039;&#039;&#039; describes a separate process, in which one Jupyter Notebook is executed at a time. Different kernels are available for different programming languages or language versions.&lt;br /&gt;
&lt;br /&gt;
Before a Jupyter session is started, the access authorization must be checked first. This is done via &#039;&#039;&#039;JupyterHub&#039;&#039;&#039;, where the resources are selected, for example the number of CPU cores, GPUs or the required main memory.&lt;br /&gt;
&lt;br /&gt;
A detailed documentation of the Jupyter project can be found at [https://jupyter.readthedocs.io https://jupyter.readthedocs.io].&lt;br /&gt;
&lt;br /&gt;
= Access requirements =&lt;br /&gt;
&lt;br /&gt;
{|style=&amp;quot;background:#deffee; width:100%;&amp;quot;&lt;br /&gt;
|style=&amp;quot;padding:5px; background:#cef2e0; text-align:left&amp;quot;|&lt;br /&gt;
[[Image:Attention.svg|center|25px]]&lt;br /&gt;
|style=&amp;quot;padding:5px; background:#cef2e0; text-align:left&amp;quot;|&lt;br /&gt;
Access to Jupyter is &#039;&#039;&#039;limited to IP addresses from the BelWü network&#039;&#039;&#039;.&lt;br /&gt;
All home institutions of our current users are connected to BelWü, so if you are on your campus network (e.g. in your office or on the Campus WiFi) you should be able to connect to bwUniCluster 3.0 without restrictions.&lt;br /&gt;
If you are outside one of the BelWü networks (e.g. at home), a VPN connection to the home institution or a connection to an SSH jump host at the home institution must be established first.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
To use Jupyter on the HPC resources of SCC, the access requirements for [https://wiki.bwhpc.de/e/Registration/bwUniCluster bwUniCluster 3.0] apply. A [https://wiki.bwhpc.de/e/Registration/bwUniCluster registration] is required. Please note, You should&#039;ve completed registration and tested your login once using [https://wiki.bwhpc.de/e/Registration/SSH Secure Shell (ssh)].&lt;br /&gt;
&lt;br /&gt;
= Login process =&lt;br /&gt;
&lt;br /&gt;
Login takes place at &lt;br /&gt;
* bwUniCluster 3.0: [https://uc3-jupyter.scc.kit.edu uc3-jupyter.scc.kit.edu]&lt;br /&gt;
* SDIL: [https://sdil-jupyter.scc.kit.edu sdil-jupyter.scc.kit.edu]&lt;br /&gt;
* HoreKa: [https://hk-jupyter.scc.kit.edu hk-jupyter.scc.kit.edu]&lt;br /&gt;
* HAICORE: [https://haicore-jupyter.scc.kit.edu haicore-jupyter.scc.kit.edu]&lt;br /&gt;
&lt;br /&gt;
For login, your username, your password and a 2-factor authentication are required.&lt;br /&gt;
&lt;br /&gt;
You will first find yourself on a landing page that also gives more information about the currently installed software versions.&lt;br /&gt;
By pressing the login button you will be redirected to the JupyterHub page. Click on Enter JupyterHub to start the login process. Select the organization (e.g. KIT) that has granted you access to the HPC system and press Continue. In the Login section that appears, enter your username and password (not the service password). &lt;br /&gt;
After pressing the Login button you will be redirected to the second factor query page. Enter the one-time password (e.g. from KIT Token or Google Authenticator App) and press Validate. Now you are done with the login process and can start selecting your computing resources.&lt;br /&gt;
&lt;br /&gt;
[[File:Jupyter_Anmeldung.gif|700px]]&lt;br /&gt;
&lt;br /&gt;
= Selection of the compute resources =&lt;br /&gt;
&lt;br /&gt;
The Jupyter notebooks are executed in an interactive session on the compute nodes of the HPC clusters. Just like accessing an interactive session with SSH, resource allocation is done by the Workload Manager Slurm. The selection of resources for Jupyter is realized via drop-down menus. Only jobs with a maximum of one node are possible.&lt;br /&gt;
&lt;br /&gt;
Available resources for selection are&lt;br /&gt;
&lt;br /&gt;
* Number of CPU cores&lt;br /&gt;
* Number of GPUs&lt;br /&gt;
* Runtime&lt;br /&gt;
* Partition/Queue&lt;br /&gt;
* Amount of main memory&lt;br /&gt;
&lt;br /&gt;
If Auto-Reservation is selected the automatic Jupyter reservation of the cluster is enabled.&lt;br /&gt;
&lt;br /&gt;
In normal mode, the grayed-out fields contain reasonable presets, depending on the number of required CPU cores or GPUs respectively. The presets can be bypassed in advanced mode, where further options are available. &lt;br /&gt;
&lt;br /&gt;
Advanced Mode can be activated by clicking on the checkbox of the same name. The following additional options then become available:&lt;br /&gt;
&lt;br /&gt;
* Specification of a reservation&lt;br /&gt;
* LSDF mount option&lt;br /&gt;
* BEEOND mount option&lt;br /&gt;
&lt;br /&gt;
After the selection is made, the interactive job is started with the spawn button. As when requesting interactive compute resources with the `salloc` command, waiting times may occur. These are usually the longer the larger the requested resources are.&lt;br /&gt;
Even if the chosen resources are available immediately, the spawning process may take up to one minute.&lt;br /&gt;
&lt;br /&gt;
[[File:Ressources_neu.gif|500px]]&lt;br /&gt;
&lt;br /&gt;
Please note that in advanced mode, resource combinations can be selected that are impossible to be met. In this case, an error message will appear when the job is spawned.&lt;br /&gt;
&lt;br /&gt;
[[File:Jupyter_Falsche_ressourcen.gif|500px]]&lt;br /&gt;
&lt;br /&gt;
The spawning timeout is currently set to 10 minutes. With a normal workload of the HPC facility, this time is usually sufficient to get interactive resources.&lt;br /&gt;
&lt;br /&gt;
== Prioritized access to computing resources on bwUniCluster 3.0 ==&lt;br /&gt;
The use of Jupyter requires the immediate availability of computing resources since the JupyterLab server is started within an interactive Slurm session. To improve the availability of CPUs/GPUs for interactive supercomputing with Jupyter, &#039;&#039;&#039;automatic reservation&#039;&#039;&#039; for CPU (cpu_il) and GPU (gpu_a100_il) resources has been set up on &#039;&#039;&#039;bwUniCluster 3.0&#039;&#039;&#039;. It is active &#039;&#039;&#039;between 8am and 8pm&#039;&#039;&#039; every weekday. The reservation is automatically active if&lt;br /&gt;
&lt;br /&gt;
* no other reservation is set manually&lt;br /&gt;
* Auto-Reservation is enabled&lt;br /&gt;
&lt;br /&gt;
To give you a better overview of the currently available resources, a status indicator has been implemented. It appears when selecting the number of required CPUs/GPUs and shows whether a Jupyter job of the selected size can currently be started or not. Green means the selected CPU/GPU resources are available instantly. Yellow means only a single additional job of the selected size can be started. Red means there are no GPU resources left that could satisfy the selected amount of resources.&lt;br /&gt;
&lt;br /&gt;
If there are no more resources available within the reservation, you can try selecting a different amount of CPUs/GPUs or activate Advanced Mode and select a different partition. Availability can be estimated using sinfo_t_idle, which is available when logging in via SSH.&lt;br /&gt;
&lt;br /&gt;
= JupyterLab =&lt;br /&gt;
&lt;br /&gt;
JupyterLab is the standard user interface. In the following only its essential functions are briefly introduced. A detailed documentation is available at &lt;br /&gt;
[https://jupyterlab.readthedocs.io https://jupyterlab.readthedocs.io].&lt;br /&gt;
&lt;br /&gt;
== Menu bar ==&lt;br /&gt;
&lt;br /&gt;
The menu bar at the upper edge of JupyterLab has higher-level menus that display the actions available in JupyterLab along with their shortcut keys. The default menus are:&lt;br /&gt;
&lt;br /&gt;
* File: Actions related to files and directories&lt;br /&gt;
* Edit: Actions related to editing documents and other activities&lt;br /&gt;
* View: actions that change the appearance of JupyterLab&lt;br /&gt;
* Run: Actions to execute code in various activities like notebooks and code consoles&lt;br /&gt;
* Kernel: Actions to manage kernels that are separate processes for executing code&lt;br /&gt;
* Tabs: a list of open documents and activities in the Dock Panel&lt;br /&gt;
* Settings: general settings and an editor for advanced settings&lt;br /&gt;
* Help: a list of help links to JupyterLab and the kernel&lt;br /&gt;
&lt;br /&gt;
== Left sidebar ==&lt;br /&gt;
&lt;br /&gt;
In the left sidebar there are foldable tabs. The most relevant ones are:&lt;br /&gt;
&lt;br /&gt;
* File browser: Switch to directories and open files with left mouse button, context menu with right mouse button&lt;br /&gt;
* Running kernels: Overview of running kernels&lt;br /&gt;
* Command overview&lt;br /&gt;
* Tab Overview&lt;br /&gt;
* Lmod software selection: Search and load/unload Lmod software modules&lt;br /&gt;
&lt;br /&gt;
== Main working area ==&lt;br /&gt;
The main working area in JupyterLab allows to arrange, resize and divide documents (notebooks, text files, etc.) and other activities (terminals, code consoles, etc.) in tabs. By holding down the left mouse button, the tabs can be grabbed and repositioned.&lt;br /&gt;
&lt;br /&gt;
In a new JupyterLab session the Launcher tab is opened first. It contains buttons for starting new notebooks, code consoles and other functions. When a notebook is open, a new Launcher tab can be started by pressing the plus symbol in the file browser tab of the left sidebar, by calling &#039;&#039;File &amp;gt; New Launcher&#039;&#039; in the upper menu bar or by the key combination &#039;&#039;Ctrl+Shift+L&#039;&#039;.&lt;br /&gt;
&lt;br /&gt;
== Classic Notebook ==&lt;br /&gt;
&lt;br /&gt;
The classic Jupyter Notebook user interface offers only one open Jupyter Notebook or terminal per browser tab. From the JupyterLab user interface the classic display can be reached in the menu bar under &#039;&#039;Help &amp;gt; Launch Classic Notebook&#039;&#039;. Clicking on the JupyterHub logo in the upper left corner will take you back to the JupyterLab interface.&lt;br /&gt;
&lt;br /&gt;
= Log out =&lt;br /&gt;
&lt;br /&gt;
You can log out from a running Jupyter session by calling &#039;&#039;File &amp;gt; Log Out&#039;&#039; in the upper menu bar. &lt;br /&gt;
&lt;br /&gt;
{| style=&amp;quot;width: 100%; margin:4px 0 0 0; background:none; border-spacing: 0px;&amp;quot;&lt;br /&gt;
| style=&amp;quot;width:100%; border:1px solid #BBBBBB; background:#fff5fa; vertical-align:top; color:#000;&amp;quot; |&lt;br /&gt;
{| style=&amp;quot;width:100%; vertical-align:top; border:0px solid #BBBBBB; padding:4px;&amp;quot; |&lt;br /&gt;
|-&lt;br /&gt;
|{{Red}}| Attention&lt;br /&gt;
|-&lt;br /&gt;
|&lt;br /&gt;
Please note that your interactive session will continue in the background!  &lt;br /&gt;
&amp;lt;!--For example, this affects your computing time quota on the ForHLR.--&amp;gt;&lt;br /&gt;
|}&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
As long as the interactive session is running, you can re-enter it at any time. Depending on the duration of your absence, it may be necessary to re-enter your one-time password and possibly KIT password.&lt;br /&gt;
&lt;br /&gt;
If you want to end the interactive session before it has reached its runtime, you can do so via the Hub Control Panel. Under &#039;&#039;File &amp;gt; Hub Control Panel&#039;&#039; in the upper menu bar, it is opened in a new browser tab. By pressing the &#039;&#039;Stop My Server&#039;&#039; button the session will be terminated. You can now log out using the &#039;&#039;Logout&#039;&#039; button in the upper right corner or start a new session directly using the &#039;&#039;Start My Server&#039;&#039; button, for example with a changed resource selection.&lt;br /&gt;
&lt;br /&gt;
[[File:logout_small.gif|750px]]&lt;br /&gt;
&lt;br /&gt;
= Selection of software =&lt;br /&gt;
&lt;br /&gt;
For the selection of the required Lmod software modules the corresponding tab &#039;&#039;Softwares&#039;&#039; is available in the left sidebar. The list of available modules can be narrowed down by entering the search field. The desired module is loaded by pressing the &#039;&#039;Load&#039;&#039; button. In the list with the loaded modules you can remove them with the &#039;&#039;Unload&#039;&#039; button.&lt;br /&gt;
&lt;br /&gt;
{| style=&amp;quot;width: 100%; margin:4px 0 0 0; background:none; border-spacing: 0px;&amp;quot;&lt;br /&gt;
| style=&amp;quot;width:100%; border:1px solid #BBBBBB; background:#f5fffa; vertical-align:top; color:#000;&amp;quot; |&lt;br /&gt;
{| style=&amp;quot;width:100%; vertical-align:top; border:0px solid #BBBBBB; padding:4px;&amp;quot; |&lt;br /&gt;
|-&lt;br /&gt;
|{{Green}}| Note&lt;br /&gt;
|-&lt;br /&gt;
|&lt;br /&gt;
On already opened Jupyter Notebooks, newly loaded software modules become active only after restarting the kernel (&#039;&#039;Kernel &amp;gt; Restart Kernel&#039;&#039; in the upper menu bar). Terminals must be closed and reopened.&lt;br /&gt;
|}&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
[[File:software_small.gif|750px]]&lt;br /&gt;
&lt;br /&gt;
== Software Stacks for Jupyter ==&lt;br /&gt;
Currently 3 special Jupyter software stacks are available via Lmod:&lt;br /&gt;
&lt;br /&gt;
* &amp;lt;code&amp;gt;jupyter/minimal&amp;lt;/code&amp;gt;&lt;br /&gt;
*: Minimal installation of JupyterLab&lt;br /&gt;
&lt;br /&gt;
* &amp;lt;code&amp;gt;jupyter/base&amp;lt;/code&amp;gt; &lt;br /&gt;
*: Basic installation of JupyterLab.&lt;br /&gt;
*: For a complete list of pre-installed packages, please refer to [https://uc3-jupyter.scc.kit.edu/software-modules/#pre-installed-software-packages this site].&lt;br /&gt;
&lt;br /&gt;
* &amp;lt;code&amp;gt;jupyter/tensorflow&amp;lt;/code&amp;gt; (default at login, will be deprecated with the advent of bwUniCluster 3.0)&lt;br /&gt;
*: Preinstalled software packages for machine learning applications. Includes among others TensorFlow, Keras, Torch, Pandas, Matplotlib, SKLearn.&lt;br /&gt;
*: For a complete list of pre-installed packages and their respective version, please refer to [https://uc3-jupyter.scc.kit.edu/software-modules/#pre-installed-software-packages this site].&lt;br /&gt;
&lt;br /&gt;
* &amp;lt;code&amp;gt;jupyter/ai&amp;lt;/code&amp;gt; (will be the new default at login für bwUniCluster 3.0, contains all latest and greatest software for AI workflows)&lt;br /&gt;
*: Preinstalled software packages for machine learning applications. Includes among others TensorFlow, Keras, Torch, Torchvision, Lighning, Pandas, Matplotlib, SKLearn.&lt;br /&gt;
*: For a complete list of pre-installed packages and their respective version, please refer to [https://uc3-jupyter.scc.kit.edu/software-modules/#pre-installed-software-packages this site].&lt;br /&gt;
&lt;br /&gt;
* &amp;lt;code&amp;gt;jupyter/extensions&amp;lt;/code&amp;gt;&lt;br /&gt;
*: Same packages as tensorflow + extensions&lt;br /&gt;
&lt;br /&gt;
These software stacks can be used both when accessing the cluster via JupyterHub, as well as for conventional access via SSH via module load.&lt;br /&gt;
&lt;br /&gt;
A continuously updated list with the installed packages can be found on the corresponding subpage of the respective cluster:&lt;br /&gt;
&lt;br /&gt;
* bwUniCluster 3.0: [https://uc3-jupyter.scc.kit.edu/software-modules uc3-jupyter.scc.kit.edu/software-modules]&lt;br /&gt;
* HoreKa: [https://hk-jupyter.scc.kit.edu/software-modules hk-jupyter.scc.kit.edu/software-modules]&lt;br /&gt;
&lt;br /&gt;
= Installation of further software =&lt;br /&gt;
The software provided by the Lmod modules jupyter/minimal, jupyter/base and jupyter/tensorflow can be easily supplemented by additional Python packages. There are 2 procedures for this.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;ul&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;User-Installation (not recommended)&amp;lt;br&amp;gt;&lt;br /&gt;
&amp;lt;code&amp;gt;pip install --user &amp;lt;packageName&amp;gt; &amp;lt;/code&amp;gt;&amp;lt;br&amp;gt;&lt;br /&gt;
The additional packages are installed under $HOME/.local/lib/python3.11/site-packages/ which is part of PYTHONPATH.&lt;br /&gt;
&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Virtual environments (recommended)&amp;lt;br&amp;gt;&lt;br /&gt;
The user can create and use virtual environments (cf. Virtual environments). Packages provided by the jupyter Lmod modules remain visible and usable.&lt;br /&gt;
&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;/ul&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Virtual environments ==&lt;br /&gt;
&lt;br /&gt;
Python virtual environments allow to use different versions of a package and to keep your local site-packages (accessible under &amp;lt;code&amp;gt;$PYTHONPATH&amp;lt;/code&amp;gt;) free from conflicts.&lt;br /&gt;
&lt;br /&gt;
=== Creation of virtual environment ===&lt;br /&gt;
&lt;br /&gt;
A virtual environment can be created using the standard Python &amp;lt;code&amp;gt;venv&amp;lt;/code&amp;gt; module:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
python -m venv &amp;lt;myEnv&amp;gt;&lt;br /&gt;
source &amp;lt;myEnv&amp;gt;/bin/activate&lt;br /&gt;
pip install &amp;lt;packageName&amp;gt;&lt;br /&gt;
deactivate&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The additional packages are installed under &amp;lt;code&amp;gt;&amp;lt;myEnv&amp;gt;/lib/python3.11/site-packages/&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
Alternatively, a virtual environment can be created and managed using &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt;:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
uv venv &amp;lt;myEnv&amp;gt;&lt;br /&gt;
uv add --directory &amp;lt;myEnv&amp;gt; &amp;lt;packageName&amp;gt;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
When using &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt;, the environment does not have to be activated with &amp;lt;code&amp;gt;source&amp;lt;/code&amp;gt;. Commands can be executed directly through &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt;, which automatically uses the corresponding environment.&lt;br /&gt;
&lt;br /&gt;
=== Usage of virtual environment ===&lt;br /&gt;
&lt;br /&gt;
In order to use a virtual environment created with &amp;lt;code&amp;gt;venv&amp;lt;/code&amp;gt;, it has to be activated via &amp;lt;code&amp;gt;source &amp;lt;myEnv&amp;gt;/bin/activate&amp;lt;/code&amp;gt;. &amp;lt;code&amp;gt;PYTHONPATH&amp;lt;/code&amp;gt; is set accordingly. Deactivation of the venv is done via &amp;lt;code&amp;gt;deactivate&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
For environments managed by &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt;, activation is not required. Python and other commands can be executed using &amp;lt;code&amp;gt;uv run&amp;lt;/code&amp;gt;, which automatically uses the environment managed by &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
=== Usage of virtual environment in JupyterLab ===&lt;br /&gt;
&lt;br /&gt;
To be able to use the virtual environments within JupyterLab, a corresponding kernel has to be installed:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
source &amp;lt;myEnv&amp;gt;/bin/activate&lt;br /&gt;
python -m ipykernel install \&lt;br /&gt;
   --user \&lt;br /&gt;
   --name myEnv \&lt;br /&gt;
   --display-name &amp;quot;Python (myEnv)&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
After installing the kernel (and possibly refreshing the browser window), a button named &amp;quot;myEnv&amp;quot; is available in JupyterLab. The kernel can also be selected from the drop-down menu.&lt;br /&gt;
&lt;br /&gt;
&#039;&#039;&#039;Attention&#039;&#039;&#039;&lt;br /&gt;
The (Lmod) base module you used in the Creation of virtual environment step must be loaded to use the venv. However, to be on the safe side, you can also use the system Python (&amp;lt;code&amp;gt;/usr/bin/python3.11&amp;lt;/code&amp;gt;) at creation time, which is available even without any &amp;lt;code&amp;gt;jupyter/{base,tensorflow}&amp;lt;/code&amp;gt; module loaded.&lt;br /&gt;
&lt;br /&gt;
As an alternative, an environment managed by &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; can be used. The corresponding Jupyter kernel can be installed with:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
uv run python -m ipykernel install \&lt;br /&gt;
   --user \&lt;br /&gt;
   --name myEnv \&lt;br /&gt;
   --display-name &amp;quot;Python (myEnv)&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
After installing the kernel (and possibly refreshing the browser window), a button named &amp;quot;Python (myEnv)&amp;quot; is available in JupyterLab. The kernel can also be selected from the drop-down menu.&lt;br /&gt;
&lt;br /&gt;
&#039;&#039;&#039;Attention&#039;&#039;&#039;&lt;br /&gt;
The &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; environment must contain &amp;lt;code&amp;gt;ipykernel&amp;lt;/code&amp;gt;. If it is not already installed, it can be added to the environment with:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
uv add ipykernel&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Examples on Data processing, Machine Learning &amp;amp; Visualization ==&lt;br /&gt;
&lt;br /&gt;
In the [https://github.com/hpcraink/workshop-parallel-jupyter/ workshop repository] the usage and best practices on Python in general, and the packages NumPy, Pandas, SciKit and Dask are provided, containing running examples based on open data. It also explains, how Jupyter interacts with pre-installed and your own provided environments.&lt;br /&gt;
&lt;br /&gt;
== R language ==&lt;br /&gt;
&lt;br /&gt;
In order to use R language in JupyterLab, the Lmod module &amp;lt;code&amp;gt;math/R&amp;lt;/code&amp;gt; has to be loaded (blue button in JupyterLab or &amp;lt;code&amp;gt;module add math/R&amp;lt;/code&amp;gt; in terminal) and a corresponding kernel has to be installed.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
R&lt;br /&gt;
install.packages(&#039;IRkernel&#039;)&lt;br /&gt;
IRkernel::installspec()&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
After installing the kernel , a button named &amp;quot;R&amp;quot; is available in JupyterLab. The kernel can also be selected from the drop-down menu.&lt;br /&gt;
&lt;br /&gt;
&#039;&#039;&#039;Attention:&#039;&#039;&#039;&lt;br /&gt;
Don&#039;t forget to load the &amp;lt;code&amp;gt;math/R&amp;lt;/code&amp;gt; module (blue button) before using the kernel.&lt;br /&gt;
&lt;br /&gt;
== Julia language ==&lt;br /&gt;
&lt;br /&gt;
In order to use Julia language in JupyterLab, the Lmod module &amp;lt;code&amp;gt;math/julia/1.10.8&amp;lt;/code&amp;gt; has to be loaded (blue button in JupyterLab or &amp;lt;code&amp;gt;module math/julia/1.10.8&amp;lt;/code&amp;gt; in terminal). When the module is loaded from the JupytherLab UI, the corresponding kernel will be installed. If you use the terminal, this has to be done manually&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
julia&lt;br /&gt;
]&lt;br /&gt;
add IJulia&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
After installing the kernel, a button named &amp;quot;Julia 1.10.8&amp;quot; is available in JupyterLab. The kernel can also be selected from the drop-down menu.&lt;br /&gt;
&lt;br /&gt;
&#039;&#039;&#039;Attention:&#039;&#039;&#039;&lt;br /&gt;
Don&#039;t forget to load the &amp;lt;code&amp;gt;math/julia/1.10.8&amp;lt;/code&amp;gt; module (blue button) before using the kernel.&lt;br /&gt;
&lt;br /&gt;
= Jupyter Container Mode =&lt;br /&gt;
&lt;br /&gt;
The container integration on the jupyterhub is done via pyxis. In order to use it the checkmark for container mode has to be clicked. Available options are:&lt;br /&gt;
&lt;br /&gt;
* &amp;lt;code&amp;gt; --container-image:&amp;lt;/code&amp;gt; The container image to use. Corresponds to the pyxis option --container-image&lt;br /&gt;
* &amp;lt;code&amp;gt; --container-name:&amp;lt;/code&amp;gt; The name of the image to use. Corresponds to the pyxis option --container-name. Already downloaded containers in ~/.local/share/enroot can be startet by simply specifing their name.&lt;br /&gt;
* &amp;lt;code&amp;gt; --container-mount-home:&amp;lt;/code&amp;gt; Corresponds to the pyxis option --container-mount-home. Mounts the home-directory&lt;br /&gt;
&lt;br /&gt;
&#039;&#039;&#039;Attention:&#039;&#039;&#039;&lt;br /&gt;
Make sure Python3.11 and pip are installed in the Container or the notebook will not spawn. This can be checked via command python3.11 -m pip list inside your container&lt;br /&gt;
&lt;br /&gt;
It is advised to create the container prior via e.G. enroot and install all necessary software. For more information see [https://wiki.bwhpc.de/e/BwUniCluster2.0/Containers#SLURM_Integration here]&lt;br /&gt;
&lt;br /&gt;
----&lt;/div&gt;</summary>
		<author><name>I Tachakor</name></author>
	</entry>
	<entry>
		<id>https://wiki.bwhpc.de/wiki/index.php?title=JUSTUS2/Software/Python&amp;diff=16467</id>
		<title>JUSTUS2/Software/Python</title>
		<link rel="alternate" type="text/html" href="https://wiki.bwhpc.de/wiki/index.php?title=JUSTUS2/Software/Python&amp;diff=16467"/>
		<updated>2026-10-05T19:17:58Z</updated>

		<summary type="html">&lt;p&gt;I Tachakor: /* uv */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page covers information on Python specific to JUSTUS2. For general information valid on all clusters see [[Development/Python]].&lt;br /&gt;
&lt;br /&gt;
=Recommendations=&lt;br /&gt;
&lt;br /&gt;
* Don&#039;t use conda (see [[#Conda|below]])&lt;br /&gt;
* Don&#039;t use the system python for computation intensive work&lt;br /&gt;
* &#039;&#039;&#039;Do never ever activate a python environment, neither venv nor conda, in your .bashrc! This may break various things.&#039;&#039;&#039;&lt;br /&gt;
&lt;br /&gt;
* Use optimized numerical libraries (SciPy, NumPy) provided as [[Environment Modules|environment modules]]&lt;br /&gt;
* Use [[Development/Python#Virtual Environments (venv)|virtual environments (venv)]]&lt;br /&gt;
* Use [[Development/Python#Package Manager (pip)|pip]] for installing further packages&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
{|style=&amp;quot;background:#deffee; width:100%;&amp;quot;&lt;br /&gt;
|style=&amp;quot;padding:5px; background:#cef2e0; text-align:left&amp;quot;|&lt;br /&gt;
[[Image:Attention.svg|center|25px]]&lt;br /&gt;
|style=&amp;quot;padding:5px; background:#cef2e0; text-align:left&amp;quot;|&lt;br /&gt;
Always load the environment modules before activating your Python environments!&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
=Optimized Libraries=&lt;br /&gt;
&lt;br /&gt;
We provide versions of SciPy and NumPy, that are optimized for the JUSTUS2 CPUs and make use of the highly optimized linear algebra routines provided by [[Development/MKL| Intel MKL]].&lt;br /&gt;
&lt;br /&gt;
For available versions see&lt;br /&gt;
&lt;br /&gt;
 module avail numlib/python_scipy&lt;br /&gt;
&lt;br /&gt;
or if you don&#039;t need SciPy&lt;br /&gt;
&lt;br /&gt;
 module avail numlib/python_numpy&lt;br /&gt;
&lt;br /&gt;
Note that each SciPy module also loads the corresponding NumPy and Python modules. Please don&#039;t try to mix with other Python versions!&lt;br /&gt;
&lt;br /&gt;
==Advanced Users: Building your own optimized libraries==&lt;br /&gt;
If you need more Python packages that depend on C/C++ or Fortran code for numerical calculations, we recommend building them manually with optimizations and linking to the Intel MKL. How to pass the compilation/linking options depends on the package. Hence see its documentation. There is usually a section like “Building From Source“&lt;br /&gt;
&lt;br /&gt;
A typical workflow might be&lt;br /&gt;
&lt;br /&gt;
 module load numlib/mkl/2024.2.1&lt;br /&gt;
 module load compiler/gnu/14.2&lt;br /&gt;
 export CFLAGS=&amp;quot;-O2 -march=native&amp;quot;&lt;br /&gt;
 export FFLAGS=&amp;quot;-O2 -march=native&amp;quot;&lt;br /&gt;
 export CXXFLAGS=&amp;quot;-02 -march=native&amp;quot;&lt;br /&gt;
 pip install --no-binary some_package some_package==VERSION BUILD_OPTIONS&lt;br /&gt;
&lt;br /&gt;
=Conda=&lt;br /&gt;
&lt;br /&gt;
There are several reasons for not using [[Conda]] on the cluster:&lt;br /&gt;
&lt;br /&gt;
* legal: unclear license situation for research with the official Anaconda channel&lt;br /&gt;
* free conda-forge channel provides mostly unoptimized packages&lt;br /&gt;
* conflicting libraries: Conda installs own versions of low-level libraries such as OpenMPI, that do not work well together with Slurm.&lt;br /&gt;
&lt;br /&gt;
However, there might be some valid use cases:&lt;br /&gt;
&lt;br /&gt;
* some packages are only available via conda&lt;br /&gt;
* simple installation for testing some software before doing an optimized build&lt;br /&gt;
&lt;br /&gt;
=uv=&lt;br /&gt;
&lt;br /&gt;
In contrast to Conda, we recommend using [https://docs.astral.sh/uv/ uv] for managing Python environments and packages on JUSTUS2.&lt;br /&gt;
&lt;br /&gt;
There are several advantages to using &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; on the cluster:&lt;br /&gt;
&lt;br /&gt;
* fast environment and package management&lt;br /&gt;
* uses the standard Python packaging ecosystem and PyPI&lt;br /&gt;
* creates isolated virtual environments without requiring Conda&lt;br /&gt;
* supports reproducible environments through lock files&lt;br /&gt;
* does not install its own low-level system libraries such as OpenMPI, avoiding potential conflicts with the cluster&#039;s environment&lt;br /&gt;
* works well with the existing Python and [[Environment Modules|environment modules]] infrastructure&lt;br /&gt;
&lt;br /&gt;
For instructions on how to use &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; on bwHPC systems, see [https://wiki.bwhpc.de/e/Development/Python#uv Development/Python: uv].&lt;/div&gt;</summary>
		<author><name>I Tachakor</name></author>
	</entry>
	<entry>
		<id>https://wiki.bwhpc.de/wiki/index.php?title=JUSTUS2/Software/Python&amp;diff=16466</id>
		<title>JUSTUS2/Software/Python</title>
		<link rel="alternate" type="text/html" href="https://wiki.bwhpc.de/wiki/index.php?title=JUSTUS2/Software/Python&amp;diff=16466"/>
		<updated>2026-10-05T19:12:22Z</updated>

		<summary type="html">&lt;p&gt;I Tachakor: Update uv-section&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page covers information on Python specific to JUSTUS2. For general information valid on all clusters see [[Development/Python]].&lt;br /&gt;
&lt;br /&gt;
=Recommendations=&lt;br /&gt;
&lt;br /&gt;
* Don&#039;t use conda (see [[#Conda|below]])&lt;br /&gt;
* Don&#039;t use the system python for computation intensive work&lt;br /&gt;
* &#039;&#039;&#039;Do never ever activate a python environment, neither venv nor conda, in your .bashrc! This may break various things.&#039;&#039;&#039;&lt;br /&gt;
&lt;br /&gt;
* Use optimized numerical libraries (SciPy, NumPy) provided as [[Environment Modules|environment modules]]&lt;br /&gt;
* Use [[Development/Python#Virtual Environments (venv)|virtual environments (venv)]]&lt;br /&gt;
* Use [[Development/Python#Package Manager (pip)|pip]] for installing further packages&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
{|style=&amp;quot;background:#deffee; width:100%;&amp;quot;&lt;br /&gt;
|style=&amp;quot;padding:5px; background:#cef2e0; text-align:left&amp;quot;|&lt;br /&gt;
[[Image:Attention.svg|center|25px]]&lt;br /&gt;
|style=&amp;quot;padding:5px; background:#cef2e0; text-align:left&amp;quot;|&lt;br /&gt;
Always load the environment modules before activating your Python environments!&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
=Optimized Libraries=&lt;br /&gt;
&lt;br /&gt;
We provide versions of SciPy and NumPy, that are optimized for the JUSTUS2 CPUs and make use of the highly optimized linear algebra routines provided by [[Development/MKL| Intel MKL]].&lt;br /&gt;
&lt;br /&gt;
For available versions see&lt;br /&gt;
&lt;br /&gt;
 module avail numlib/python_scipy&lt;br /&gt;
&lt;br /&gt;
or if you don&#039;t need SciPy&lt;br /&gt;
&lt;br /&gt;
 module avail numlib/python_numpy&lt;br /&gt;
&lt;br /&gt;
Note that each SciPy module also loads the corresponding NumPy and Python modules. Please don&#039;t try to mix with other Python versions!&lt;br /&gt;
&lt;br /&gt;
==Advanced Users: Building your own optimized libraries==&lt;br /&gt;
If you need more Python packages that depend on C/C++ or Fortran code for numerical calculations, we recommend building them manually with optimizations and linking to the Intel MKL. How to pass the compilation/linking options depends on the package. Hence see its documentation. There is usually a section like “Building From Source“&lt;br /&gt;
&lt;br /&gt;
A typical workflow might be&lt;br /&gt;
&lt;br /&gt;
 module load numlib/mkl/2024.2.1&lt;br /&gt;
 module load compiler/gnu/14.2&lt;br /&gt;
 export CFLAGS=&amp;quot;-O2 -march=native&amp;quot;&lt;br /&gt;
 export FFLAGS=&amp;quot;-O2 -march=native&amp;quot;&lt;br /&gt;
 export CXXFLAGS=&amp;quot;-02 -march=native&amp;quot;&lt;br /&gt;
 pip install --no-binary some_package some_package==VERSION BUILD_OPTIONS&lt;br /&gt;
&lt;br /&gt;
=Conda=&lt;br /&gt;
&lt;br /&gt;
There are several reasons for not using [[Conda]] on the cluster:&lt;br /&gt;
&lt;br /&gt;
* legal: unclear license situation for research with the official Anaconda channel&lt;br /&gt;
* free conda-forge channel provides mostly unoptimized packages&lt;br /&gt;
* conflicting libraries: Conda installs own versions of low-level libraries such as OpenMPI, that do not work well together with Slurm.&lt;br /&gt;
&lt;br /&gt;
However, there might be some valid use cases:&lt;br /&gt;
&lt;br /&gt;
* some packages are only available via conda&lt;br /&gt;
* simple installation for testing some software before doing an optimized build&lt;br /&gt;
&lt;br /&gt;
=uv=&lt;br /&gt;
&lt;br /&gt;
In contrast to Conda, we recommend using [[https://docs.astral.sh/uv/|uv]] for managing Python environments and packages on JUSTUS2.&lt;br /&gt;
&lt;br /&gt;
There are several advantages to using &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; on the cluster:&lt;br /&gt;
&lt;br /&gt;
* fast environment and package management&lt;br /&gt;
* uses the standard Python packaging ecosystem and PyPI&lt;br /&gt;
* creates isolated virtual environments without requiring Conda&lt;br /&gt;
* supports reproducible environments through lock files&lt;br /&gt;
* does not install its own low-level system libraries such as OpenMPI, avoiding potential conflicts with the cluster&#039;s environment&lt;br /&gt;
* works well with the existing Python and [[Environment Modules|environment modules]] infrastructure&lt;br /&gt;
&lt;br /&gt;
For instructions on how to use &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; on bwHPC systems, see [[https://wiki.bwhpc.de/e/Development/Python#uv|Development/Python: uv]].&lt;/div&gt;</summary>
		<author><name>I Tachakor</name></author>
	</entry>
	<entry>
		<id>https://wiki.bwhpc.de/wiki/index.php?title=JUSTUS2/Software/Python&amp;diff=16465</id>
		<title>JUSTUS2/Software/Python</title>
		<link rel="alternate" type="text/html" href="https://wiki.bwhpc.de/wiki/index.php?title=JUSTUS2/Software/Python&amp;diff=16465"/>
		<updated>2026-10-05T19:10:39Z</updated>

		<summary type="html">&lt;p&gt;I Tachakor: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page covers information on Python specific to JUSTUS2. For general information valid on all clusters see [[Development/Python]].&lt;br /&gt;
&lt;br /&gt;
=Recommendations=&lt;br /&gt;
&lt;br /&gt;
* Don&#039;t use conda (see [[#Conda|below]])&lt;br /&gt;
* Don&#039;t use the system python for computation intensive work&lt;br /&gt;
* &amp;lt;b&amp;gt;Do never ever activate a python environment, neither venv nor conda, in your .bashrc! This may break various things.&amp;lt;/b&amp;gt;&lt;br /&gt;
&lt;br /&gt;
* Use optimized numerical libraries (SciPy, NumPy) provided as [[Environment Modules|environment modules]]&lt;br /&gt;
* Use [[Development/Python#Virtual Environments (venv)|virtual environments (venv)]]&lt;br /&gt;
* Use [[Development/Python#Package Manager (pip)|pip]] for installing further packages&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
{|style=&amp;quot;background:#deffee; width:100%;&amp;quot;&lt;br /&gt;
|style=&amp;quot;padding:5px; background:#cef2e0; text-align:left&amp;quot;|&lt;br /&gt;
[[Image:Attention.svg|center|25px]]&lt;br /&gt;
|style=&amp;quot;padding:5px; background:#cef2e0; text-align:left&amp;quot;|&lt;br /&gt;
Always load the environment modules before activating your Python environments! &lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
=Optimized Libraries=&lt;br /&gt;
&lt;br /&gt;
We provide versions of SciPy and NumPy, that are optimized for the JUSTUS2 CPUs and make use of the highly optimized linear algebra routines provided by [[Development/MKL| Intel MKL]].&lt;br /&gt;
&lt;br /&gt;
For available versions see&lt;br /&gt;
&lt;br /&gt;
 module avail numlib/python_scipy&lt;br /&gt;
&lt;br /&gt;
or if you don&#039;t need SciPy&lt;br /&gt;
&lt;br /&gt;
 module avail numlib/python_numpy&lt;br /&gt;
&lt;br /&gt;
Note that each SciPy module also loads the corresponding NumPy and Python modules. Please don&#039;t try to mix with other Python versions!&lt;br /&gt;
&lt;br /&gt;
==Advanced Users: Building your own optimized libraries==&lt;br /&gt;
If you need more Python packages that depend on C/C++ or Fortran code for numerical calculations, we recommend building them manually with optimizations and linking to the Intel MKL. How to pass the compilation/linking options depends on the package. Hence see its documentation. There is usually a section like “Building From Source“&lt;br /&gt;
&lt;br /&gt;
A typical workflow might be&lt;br /&gt;
&lt;br /&gt;
 module load numlib/mkl/2024.2.1&lt;br /&gt;
 module load compiler/gnu/14.2&lt;br /&gt;
 export CFLAGS=&amp;quot;-O2 -march=native&amp;quot;&lt;br /&gt;
 export FFLAGS=&amp;quot;-O2 -march=native&amp;quot;&lt;br /&gt;
 export CXXFLAGS=&amp;quot;-02 -march=native&amp;quot; &lt;br /&gt;
 pip install --no-binary some_package some_package==VERSION BUILD_OPTIONS&lt;br /&gt;
&lt;br /&gt;
=Conda=&lt;br /&gt;
&lt;br /&gt;
There are several reasons for not using [[Conda]] on the cluster:&lt;br /&gt;
&lt;br /&gt;
* legal: unclear license situation for research with the official Anaconda channel&lt;br /&gt;
* free conda-forge channel provides mostly unoptimized packages &lt;br /&gt;
* conflicting libraries: Conda installs own versions of low-level libraries such as OpenMPI, that do not work well together with Slurm.&lt;br /&gt;
&lt;br /&gt;
However, there might be some valid use cases:&lt;br /&gt;
&lt;br /&gt;
* some packages are only available via conda&lt;br /&gt;
* simple installation for testing some software before doing an optimized build&lt;br /&gt;
&lt;br /&gt;
\=uv=&lt;br /&gt;
&lt;br /&gt;
In contrast to Conda, we recommend using [[https://docs.astral.sh/uv/|uv]] for managing Python environments and packages on JUSTUS2.&lt;br /&gt;
&lt;br /&gt;
There are several advantages to using &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; on the cluster:&lt;br /&gt;
&lt;br /&gt;
\* fast environment and package management&lt;br /&gt;
\* uses the standard Python packaging ecosystem and PyPI&lt;br /&gt;
\* creates isolated virtual environments without requiring Conda&lt;br /&gt;
\* supports reproducible environments through lock files&lt;br /&gt;
\* does not install its own low-level system libraries such as OpenMPI, avoiding potential conflicts with the cluster&#039;s environment&lt;br /&gt;
\* works well with the existing Python and [[Environment Modules|environment modules]] infrastructure&lt;br /&gt;
&lt;br /&gt;
For instructions on how to use &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; on bwHPC systems, see [[https://wiki.bwhpc.de/e/Development/Python#uv|Development/Python: uv]].&lt;/div&gt;</summary>
		<author><name>I Tachakor</name></author>
	</entry>
	<entry>
		<id>https://wiki.bwhpc.de/wiki/index.php?title=JUSTUS2/Software/Python&amp;diff=16464</id>
		<title>JUSTUS2/Software/Python</title>
		<link rel="alternate" type="text/html" href="https://wiki.bwhpc.de/wiki/index.php?title=JUSTUS2/Software/Python&amp;diff=16464"/>
		<updated>2026-10-05T19:09:27Z</updated>

		<summary type="html">&lt;p&gt;I Tachakor: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;This page covers information on Python specific to JUSTUS2. For general information valid on all clusters see [[Development/Python]].&lt;br /&gt;
&lt;br /&gt;
=Recommendations=&lt;br /&gt;
&lt;br /&gt;
* Don&#039;t use conda (see [[#Conda|below]])&lt;br /&gt;
* Don&#039;t use the system python for computation intensive work&lt;br /&gt;
* &amp;lt;b&amp;gt;Do never ever activate a python environment, neither venv nor conda, in your .bashrc! This may break various things.&amp;lt;/b&amp;gt;&lt;br /&gt;
&lt;br /&gt;
* Use optimized numerical libraries (SciPy, NumPy) provided as [[Environment Modules|environment modules]]&lt;br /&gt;
* Use [[Development/Python#Virtual Environments (venv)|virtual environments (venv)]]&lt;br /&gt;
* Use [[Development/Python#Package Manager (pip)|pip]] for installing further packages&lt;br /&gt;
&lt;br /&gt;
&lt;br /&gt;
{|style=&amp;quot;background:#deffee; width:100%;&amp;quot;&lt;br /&gt;
|style=&amp;quot;padding:5px; background:#cef2e0; text-align:left&amp;quot;|&lt;br /&gt;
[[Image:Attention.svg|center|25px]]&lt;br /&gt;
|style=&amp;quot;padding:5px; background:#cef2e0; text-align:left&amp;quot;|&lt;br /&gt;
Always load the environment modules before activating your Python environments! &lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
=Optimized Libraries=&lt;br /&gt;
&lt;br /&gt;
We provide versions of SciPy and NumPy, that are optimized for the JUSTUS2 CPUs and make use of the highly optimized linear algebra routines provided by [[Development/MKL| Intel MKL]].&lt;br /&gt;
&lt;br /&gt;
For available versions see&lt;br /&gt;
&lt;br /&gt;
 module avail numlib/python_scipy&lt;br /&gt;
&lt;br /&gt;
or if you don&#039;t need SciPy&lt;br /&gt;
&lt;br /&gt;
 module avail numlib/python_numpy&lt;br /&gt;
&lt;br /&gt;
Note that each SciPy module also loads the corresponding NumPy and Python modules. Please don&#039;t try to mix with other Python versions!&lt;br /&gt;
&lt;br /&gt;
==Advanced Users: Building your own optimized libraries==&lt;br /&gt;
If you need more Python packages that depend on C/C++ or Fortran code for numerical calculations, we recommend building them manually with optimizations and linking to the Intel MKL. How to pass the compilation/linking options depends on the package. Hence see its documentation. There is usually a section like “Building From Source“&lt;br /&gt;
&lt;br /&gt;
A typical workflow might be&lt;br /&gt;
&lt;br /&gt;
 module load numlib/mkl/2024.2.1&lt;br /&gt;
 module load compiler/gnu/14.2&lt;br /&gt;
 export CFLAGS=&amp;quot;-O2 -march=native&amp;quot;&lt;br /&gt;
 export FFLAGS=&amp;quot;-O2 -march=native&amp;quot;&lt;br /&gt;
 export CXXFLAGS=&amp;quot;-02 -march=native&amp;quot; &lt;br /&gt;
 pip install --no-binary some_package some_package==VERSION BUILD_OPTIONS&lt;br /&gt;
&lt;br /&gt;
=Conda=&lt;br /&gt;
&lt;br /&gt;
There are several reasons for not using [[Conda]] on the cluster:&lt;br /&gt;
&lt;br /&gt;
* legal: unclear license situation for research with the official Anaconda channel&lt;br /&gt;
* free conda-forge channel provides mostly unoptimized packages &lt;br /&gt;
* conflicting libraries: Conda installs own versions of low-level libraries such as OpenMPI, that do not work well together with Slurm.&lt;br /&gt;
&lt;br /&gt;
However, there might be some valid use cases:&lt;br /&gt;
&lt;br /&gt;
* some packages are only available via conda&lt;br /&gt;
* simple installation for testing some software before doing an optimized build&lt;br /&gt;
&lt;br /&gt;
=uv=&lt;br /&gt;
&lt;br /&gt;
In contrast to Conda, we recommend using [[https://docs.astral.sh/uv/|uv]] for managing Python environments and packages on JUSTUS2.&lt;br /&gt;
&lt;br /&gt;
There are several advantages to using &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; on the cluster:&lt;br /&gt;
&lt;br /&gt;
\* fast environment and package management&lt;br /&gt;
\* uses the standard Python packaging ecosystem and PyPI&lt;br /&gt;
\* creates isolated virtual environments without requiring Conda&lt;br /&gt;
\* supports reproducible environments through lock files&lt;br /&gt;
\* does not install its own low-level system libraries such as OpenMPI, avoiding potential conflicts with the cluster&#039;s environment&lt;br /&gt;
\* works well with the existing Python and [[Environment Modules|environment modules]] infrastructure&lt;br /&gt;
&lt;br /&gt;
For instructions on how to use &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; on bwHPC systems, see [[https://wiki.bwhpc.de/e/Development/Python#uv|Development/Python: uv]].&lt;/div&gt;</summary>
		<author><name>I Tachakor</name></author>
	</entry>
	<entry>
		<id>https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster3.0/Jupyter&amp;diff=16463</id>
		<title>BwUniCluster3.0/Jupyter</title>
		<link rel="alternate" type="text/html" href="https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster3.0/Jupyter&amp;diff=16463"/>
		<updated>2026-10-05T18:54:42Z</updated>

		<summary type="html">&lt;p&gt;I Tachakor: /* Usage of virtual environment in JupyterLab */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;Jupyter can be used as an alternative to accessing HPC resources via SSH. For this purpose only a web browser is required. Within the website source code of different programming languages can be edited and executed. Furthermore different user interfaces and terminals are available.&lt;br /&gt;
&lt;br /&gt;
= Short description of Jupyter =&lt;br /&gt;
&lt;br /&gt;
Jupyter is a web application, central component of Jupyter is the &#039;&#039;&#039;Jupyter Notebook&#039;&#039;&#039;. It is a document, which can contain formatted text, executable code sections and (interactive) visualizations (image, sound, video, 3D views).&lt;br /&gt;
&lt;br /&gt;
The Jupyter notebooks are executed in an interactive session on the compute nodes of the respective cluster. Access is via any modern web browser. Data is prepared and visualized on the server and therefore does not have to be transmitted over the network. Only the resulting text, image, sound and video data is transmitted. Starting point of a Jupyter session is the HOME directory of the user on the respective cluster. &lt;br /&gt;
&lt;br /&gt;
&#039;&#039;&#039;JupyterLab&#039;&#039;&#039; is a modern user interface, within which one or more Jupyter notebooks can be opened, edited and executed. The individual notebooks can be arranged as tabs or tiled. JupyterLab is the standard user interface. Besides JupyterLab the classic notebook user interface is available, in which only one Jupyter notebook per browser tab can be opened at a time.&lt;br /&gt;
&lt;br /&gt;
A &#039;&#039;&#039;Jupyter Kernel&#039;&#039;&#039; describes a separate process, in which one Jupyter Notebook is executed at a time. Different kernels are available for different programming languages or language versions.&lt;br /&gt;
&lt;br /&gt;
Before a Jupyter session is started, the access authorization must be checked first. This is done via &#039;&#039;&#039;JupyterHub&#039;&#039;&#039;, where the resources are selected, for example the number of CPU cores, GPUs or the required main memory.&lt;br /&gt;
&lt;br /&gt;
A detailed documentation of the Jupyter project can be found at [https://jupyter.readthedocs.io https://jupyter.readthedocs.io].&lt;br /&gt;
&lt;br /&gt;
= Access requirements =&lt;br /&gt;
&lt;br /&gt;
{|style=&amp;quot;background:#deffee; width:100%;&amp;quot;&lt;br /&gt;
|style=&amp;quot;padding:5px; background:#cef2e0; text-align:left&amp;quot;|&lt;br /&gt;
[[Image:Attention.svg|center|25px]]&lt;br /&gt;
|style=&amp;quot;padding:5px; background:#cef2e0; text-align:left&amp;quot;|&lt;br /&gt;
Access to Jupyter is &#039;&#039;&#039;limited to IP addresses from the BelWü network&#039;&#039;&#039;.&lt;br /&gt;
All home institutions of our current users are connected to BelWü, so if you are on your campus network (e.g. in your office or on the Campus WiFi) you should be able to connect to bwUniCluster 3.0 without restrictions.&lt;br /&gt;
If you are outside one of the BelWü networks (e.g. at home), a VPN connection to the home institution or a connection to an SSH jump host at the home institution must be established first.&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
To use Jupyter on the HPC resources of SCC, the access requirements for [https://wiki.bwhpc.de/e/Registration/bwUniCluster bwUniCluster 3.0] apply. A [https://wiki.bwhpc.de/e/Registration/bwUniCluster registration] is required. Please note, You should&#039;ve completed registration and tested your login once using [https://wiki.bwhpc.de/e/Registration/SSH Secure Shell (ssh)].&lt;br /&gt;
&lt;br /&gt;
= Login process =&lt;br /&gt;
&lt;br /&gt;
Login takes place at &lt;br /&gt;
* bwUniCluster 3.0: [https://uc3-jupyter.scc.kit.edu uc3-jupyter.scc.kit.edu]&lt;br /&gt;
* SDIL: [https://sdil-jupyter.scc.kit.edu sdil-jupyter.scc.kit.edu]&lt;br /&gt;
* HoreKa: [https://hk-jupyter.scc.kit.edu hk-jupyter.scc.kit.edu]&lt;br /&gt;
* HAICORE: [https://haicore-jupyter.scc.kit.edu haicore-jupyter.scc.kit.edu]&lt;br /&gt;
&lt;br /&gt;
For login, your username, your password and a 2-factor authentication are required.&lt;br /&gt;
&lt;br /&gt;
You will first find yourself on a landing page that also gives more information about the currently installed software versions.&lt;br /&gt;
By pressing the login button you will be redirected to the JupyterHub page. Click on Enter JupyterHub to start the login process. Select the organization (e.g. KIT) that has granted you access to the HPC system and press Continue. In the Login section that appears, enter your username and password (not the service password). &lt;br /&gt;
After pressing the Login button you will be redirected to the second factor query page. Enter the one-time password (e.g. from KIT Token or Google Authenticator App) and press Validate. Now you are done with the login process and can start selecting your computing resources.&lt;br /&gt;
&lt;br /&gt;
[[File:Jupyter_Anmeldung.gif|700px]]&lt;br /&gt;
&lt;br /&gt;
= Selection of the compute resources =&lt;br /&gt;
&lt;br /&gt;
The Jupyter notebooks are executed in an interactive session on the compute nodes of the HPC clusters. Just like accessing an interactive session with SSH, resource allocation is done by the Workload Manager Slurm. The selection of resources for Jupyter is realized via drop-down menus. Only jobs with a maximum of one node are possible.&lt;br /&gt;
&lt;br /&gt;
Available resources for selection are&lt;br /&gt;
&lt;br /&gt;
* Number of CPU cores&lt;br /&gt;
* Number of GPUs&lt;br /&gt;
* Runtime&lt;br /&gt;
* Partition/Queue&lt;br /&gt;
* Amount of main memory&lt;br /&gt;
&lt;br /&gt;
If Auto-Reservation is selected the automatic Jupyter reservation of the cluster is enabled.&lt;br /&gt;
&lt;br /&gt;
In normal mode, the grayed-out fields contain reasonable presets, depending on the number of required CPU cores or GPUs respectively. The presets can be bypassed in advanced mode, where further options are available. &lt;br /&gt;
&lt;br /&gt;
Advanced Mode can be activated by clicking on the checkbox of the same name. The following additional options then become available:&lt;br /&gt;
&lt;br /&gt;
* Specification of a reservation&lt;br /&gt;
* LSDF mount option&lt;br /&gt;
* BEEOND mount option&lt;br /&gt;
&lt;br /&gt;
After the selection is made, the interactive job is started with the spawn button. As when requesting interactive compute resources with the `salloc` command, waiting times may occur. These are usually the longer the larger the requested resources are.&lt;br /&gt;
Even if the chosen resources are available immediately, the spawning process may take up to one minute.&lt;br /&gt;
&lt;br /&gt;
[[File:Ressources_neu.gif|500px]]&lt;br /&gt;
&lt;br /&gt;
Please note that in advanced mode, resource combinations can be selected that are impossible to be met. In this case, an error message will appear when the job is spawned.&lt;br /&gt;
&lt;br /&gt;
[[File:Jupyter_Falsche_ressourcen.gif|500px]]&lt;br /&gt;
&lt;br /&gt;
The spawning timeout is currently set to 10 minutes. With a normal workload of the HPC facility, this time is usually sufficient to get interactive resources.&lt;br /&gt;
&lt;br /&gt;
== Prioritized access to computing resources on bwUniCluster 3.0 ==&lt;br /&gt;
The use of Jupyter requires the immediate availability of computing resources since the JupyterLab server is started within an interactive Slurm session. To improve the availability of CPUs/GPUs for interactive supercomputing with Jupyter, &#039;&#039;&#039;automatic reservation&#039;&#039;&#039; for CPU (cpu_il) and GPU (gpu_a100_il) resources has been set up on &#039;&#039;&#039;bwUniCluster 3.0&#039;&#039;&#039;. It is active &#039;&#039;&#039;between 8am and 8pm&#039;&#039;&#039; every weekday. The reservation is automatically active if&lt;br /&gt;
&lt;br /&gt;
* no other reservation is set manually&lt;br /&gt;
* Auto-Reservation is enabled&lt;br /&gt;
&lt;br /&gt;
To give you a better overview of the currently available resources, a status indicator has been implemented. It appears when selecting the number of required CPUs/GPUs and shows whether a Jupyter job of the selected size can currently be started or not. Green means the selected CPU/GPU resources are available instantly. Yellow means only a single additional job of the selected size can be started. Red means there are no GPU resources left that could satisfy the selected amount of resources.&lt;br /&gt;
&lt;br /&gt;
If there are no more resources available within the reservation, you can try selecting a different amount of CPUs/GPUs or activate Advanced Mode and select a different partition. Availability can be estimated using sinfo_t_idle, which is available when logging in via SSH.&lt;br /&gt;
&lt;br /&gt;
= JupyterLab =&lt;br /&gt;
&lt;br /&gt;
JupyterLab is the standard user interface. In the following only its essential functions are briefly introduced. A detailed documentation is available at &lt;br /&gt;
[https://jupyterlab.readthedocs.io https://jupyterlab.readthedocs.io].&lt;br /&gt;
&lt;br /&gt;
== Menu bar ==&lt;br /&gt;
&lt;br /&gt;
The menu bar at the upper edge of JupyterLab has higher-level menus that display the actions available in JupyterLab along with their shortcut keys. The default menus are:&lt;br /&gt;
&lt;br /&gt;
* File: Actions related to files and directories&lt;br /&gt;
* Edit: Actions related to editing documents and other activities&lt;br /&gt;
* View: actions that change the appearance of JupyterLab&lt;br /&gt;
* Run: Actions to execute code in various activities like notebooks and code consoles&lt;br /&gt;
* Kernel: Actions to manage kernels that are separate processes for executing code&lt;br /&gt;
* Tabs: a list of open documents and activities in the Dock Panel&lt;br /&gt;
* Settings: general settings and an editor for advanced settings&lt;br /&gt;
* Help: a list of help links to JupyterLab and the kernel&lt;br /&gt;
&lt;br /&gt;
== Left sidebar ==&lt;br /&gt;
&lt;br /&gt;
In the left sidebar there are foldable tabs. The most relevant ones are:&lt;br /&gt;
&lt;br /&gt;
* File browser: Switch to directories and open files with left mouse button, context menu with right mouse button&lt;br /&gt;
* Running kernels: Overview of running kernels&lt;br /&gt;
* Command overview&lt;br /&gt;
* Tab Overview&lt;br /&gt;
* Lmod software selection: Search and load/unload Lmod software modules&lt;br /&gt;
&lt;br /&gt;
== Main working area ==&lt;br /&gt;
The main working area in JupyterLab allows to arrange, resize and divide documents (notebooks, text files, etc.) and other activities (terminals, code consoles, etc.) in tabs. By holding down the left mouse button, the tabs can be grabbed and repositioned.&lt;br /&gt;
&lt;br /&gt;
In a new JupyterLab session the Launcher tab is opened first. It contains buttons for starting new notebooks, code consoles and other functions. When a notebook is open, a new Launcher tab can be started by pressing the plus symbol in the file browser tab of the left sidebar, by calling &#039;&#039;File &amp;gt; New Launcher&#039;&#039; in the upper menu bar or by the key combination &#039;&#039;Ctrl+Shift+L&#039;&#039;.&lt;br /&gt;
&lt;br /&gt;
== Classic Notebook ==&lt;br /&gt;
&lt;br /&gt;
The classic Jupyter Notebook user interface offers only one open Jupyter Notebook or terminal per browser tab. From the JupyterLab user interface the classic display can be reached in the menu bar under &#039;&#039;Help &amp;gt; Launch Classic Notebook&#039;&#039;. Clicking on the JupyterHub logo in the upper left corner will take you back to the JupyterLab interface.&lt;br /&gt;
&lt;br /&gt;
= Log out =&lt;br /&gt;
&lt;br /&gt;
You can log out from a running Jupyter session by calling &#039;&#039;File &amp;gt; Log Out&#039;&#039; in the upper menu bar. &lt;br /&gt;
&lt;br /&gt;
{| style=&amp;quot;width: 100%; margin:4px 0 0 0; background:none; border-spacing: 0px;&amp;quot;&lt;br /&gt;
| style=&amp;quot;width:100%; border:1px solid #BBBBBB; background:#fff5fa; vertical-align:top; color:#000;&amp;quot; |&lt;br /&gt;
{| style=&amp;quot;width:100%; vertical-align:top; border:0px solid #BBBBBB; padding:4px;&amp;quot; |&lt;br /&gt;
|-&lt;br /&gt;
|{{Red}}| Attention&lt;br /&gt;
|-&lt;br /&gt;
|&lt;br /&gt;
Please note that your interactive session will continue in the background!  &lt;br /&gt;
&amp;lt;!--For example, this affects your computing time quota on the ForHLR.--&amp;gt;&lt;br /&gt;
|}&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
As long as the interactive session is running, you can re-enter it at any time. Depending on the duration of your absence, it may be necessary to re-enter your one-time password and possibly KIT password.&lt;br /&gt;
&lt;br /&gt;
If you want to end the interactive session before it has reached its runtime, you can do so via the Hub Control Panel. Under &#039;&#039;File &amp;gt; Hub Control Panel&#039;&#039; in the upper menu bar, it is opened in a new browser tab. By pressing the &#039;&#039;Stop My Server&#039;&#039; button the session will be terminated. You can now log out using the &#039;&#039;Logout&#039;&#039; button in the upper right corner or start a new session directly using the &#039;&#039;Start My Server&#039;&#039; button, for example with a changed resource selection.&lt;br /&gt;
&lt;br /&gt;
[[File:logout_small.gif|750px]]&lt;br /&gt;
&lt;br /&gt;
= Selection of software =&lt;br /&gt;
&lt;br /&gt;
For the selection of the required Lmod software modules the corresponding tab &#039;&#039;Softwares&#039;&#039; is available in the left sidebar. The list of available modules can be narrowed down by entering the search field. The desired module is loaded by pressing the &#039;&#039;Load&#039;&#039; button. In the list with the loaded modules you can remove them with the &#039;&#039;Unload&#039;&#039; button.&lt;br /&gt;
&lt;br /&gt;
{| style=&amp;quot;width: 100%; margin:4px 0 0 0; background:none; border-spacing: 0px;&amp;quot;&lt;br /&gt;
| style=&amp;quot;width:100%; border:1px solid #BBBBBB; background:#f5fffa; vertical-align:top; color:#000;&amp;quot; |&lt;br /&gt;
{| style=&amp;quot;width:100%; vertical-align:top; border:0px solid #BBBBBB; padding:4px;&amp;quot; |&lt;br /&gt;
|-&lt;br /&gt;
|{{Green}}| Note&lt;br /&gt;
|-&lt;br /&gt;
|&lt;br /&gt;
On already opened Jupyter Notebooks, newly loaded software modules become active only after restarting the kernel (&#039;&#039;Kernel &amp;gt; Restart Kernel&#039;&#039; in the upper menu bar). Terminals must be closed and reopened.&lt;br /&gt;
|}&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
[[File:software_small.gif|750px]]&lt;br /&gt;
&lt;br /&gt;
== Software Stacks for Jupyter ==&lt;br /&gt;
Currently 3 special Jupyter software stacks are available via Lmod:&lt;br /&gt;
&lt;br /&gt;
* &amp;lt;code&amp;gt;jupyter/minimal&amp;lt;/code&amp;gt;&lt;br /&gt;
*: Minimal installation of JupyterLab&lt;br /&gt;
&lt;br /&gt;
* &amp;lt;code&amp;gt;jupyter/base&amp;lt;/code&amp;gt; &lt;br /&gt;
*: Basic installation of JupyterLab.&lt;br /&gt;
*: For a complete list of pre-installed packages, please refer to [https://uc3-jupyter.scc.kit.edu/software-modules/#pre-installed-software-packages this site].&lt;br /&gt;
&lt;br /&gt;
* &amp;lt;code&amp;gt;jupyter/tensorflow&amp;lt;/code&amp;gt; (default at login, will be deprecated with the advent of bwUniCluster 3.0)&lt;br /&gt;
*: Preinstalled software packages for machine learning applications. Includes among others TensorFlow, Keras, Torch, Pandas, Matplotlib, SKLearn.&lt;br /&gt;
*: For a complete list of pre-installed packages and their respective version, please refer to [https://uc3-jupyter.scc.kit.edu/software-modules/#pre-installed-software-packages this site].&lt;br /&gt;
&lt;br /&gt;
* &amp;lt;code&amp;gt;jupyter/ai&amp;lt;/code&amp;gt; (will be the new default at login für bwUniCluster 3.0, contains all latest and greatest software for AI workflows)&lt;br /&gt;
*: Preinstalled software packages for machine learning applications. Includes among others TensorFlow, Keras, Torch, Torchvision, Lighning, Pandas, Matplotlib, SKLearn.&lt;br /&gt;
*: For a complete list of pre-installed packages and their respective version, please refer to [https://uc3-jupyter.scc.kit.edu/software-modules/#pre-installed-software-packages this site].&lt;br /&gt;
&lt;br /&gt;
* &amp;lt;code&amp;gt;jupyter/extensions&amp;lt;/code&amp;gt;&lt;br /&gt;
*: Same packages as tensorflow + extensions&lt;br /&gt;
&lt;br /&gt;
These software stacks can be used both when accessing the cluster via JupyterHub, as well as for conventional access via SSH via module load.&lt;br /&gt;
&lt;br /&gt;
A continuously updated list with the installed packages can be found on the corresponding subpage of the respective cluster:&lt;br /&gt;
&lt;br /&gt;
* bwUniCluster 3.0: [https://uc3-jupyter.scc.kit.edu/software-modules uc3-jupyter.scc.kit.edu/software-modules]&lt;br /&gt;
* HoreKa: [https://hk-jupyter.scc.kit.edu/software-modules hk-jupyter.scc.kit.edu/software-modules]&lt;br /&gt;
&lt;br /&gt;
= Installation of further software =&lt;br /&gt;
The software provided by the Lmod modules jupyter/minimal, jupyter/base and jupyter/tensorflow can be easily supplemented by additional Python packages. There are 2 procedures for this.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;ul&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;User-Installation (not recommended)&amp;lt;br&amp;gt;&lt;br /&gt;
&amp;lt;code&amp;gt;pip install --user &amp;lt;packageName&amp;gt; &amp;lt;/code&amp;gt;&amp;lt;br&amp;gt;&lt;br /&gt;
The additional packages are installed under $HOME/.local/lib/python3.11/site-packages/ which is part of PYTHONPATH.&lt;br /&gt;
&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;li&amp;gt;Virtual environments (recommended)&amp;lt;br&amp;gt;&lt;br /&gt;
The user can create and use virtual environments (cf. Virtual environments). Packages provided by the jupyter Lmod modules remain visible and usable.&lt;br /&gt;
&amp;lt;/li&amp;gt;&lt;br /&gt;
&amp;lt;/ul&amp;gt;&lt;br /&gt;
&lt;br /&gt;
== Virtual environments ==&lt;br /&gt;
&lt;br /&gt;
Python virtual environments allow to use different versions of a package and to keep your local site-packages (accessible under &amp;lt;code&amp;gt;$PYTHONPATH&amp;lt;/code&amp;gt;) free from conflicts.&lt;br /&gt;
&lt;br /&gt;
=== Creation of virtual environment ===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
python -m venv &amp;lt;myEnv&amp;gt;&lt;br /&gt;
source &amp;lt;myEnv&amp;gt;/bin/activate  &lt;br /&gt;
pip install &amp;lt;packageName&amp;gt;  &lt;br /&gt;
deactivate&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The additional packages are installed under &amp;lt;code&amp;gt;&amp;lt;myEnv&amp;gt;/lib/python3.11/site-packages/&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
=== Usage of virtual environment ===&lt;br /&gt;
&lt;br /&gt;
In order to use the virtual environment, it has to be activated via &amp;lt;code&amp;gt;source &amp;lt;myEnv&amp;gt;/bin/activate&amp;lt;/code&amp;gt;. &amp;lt;code&amp;gt;PYTHONPATH&amp;lt;/code&amp;gt; is set accordingly. Deactivation of the venv is done via &amp;lt;code&amp;gt;deactivate&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
=== Usage of virtual environment in JupyterLab ===&lt;br /&gt;
&lt;br /&gt;
To be able to use the virtual environments within JupyterLab, a corresponding kernel has to be installed:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
source &amp;lt;myEnv&amp;gt;/bin/activate&lt;br /&gt;
python -m ipykernel install \&lt;br /&gt;
    --user \&lt;br /&gt;
    --name myEnv \&lt;br /&gt;
    --display-name &amp;quot;Python (myEnv)&amp;quot; &lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
After installing the kernel (and possibly refreshing the browser window), a button named &amp;quot;myEnv&amp;quot; is available in JupyterLab. The kernel can also be selected from the drop-down menu.&lt;br /&gt;
&lt;br /&gt;
&#039;&#039;&#039;Attention&#039;&#039;&#039;&lt;br /&gt;
The (Lmod) base module you used in the Creation of virtual environment step must be loaded to use the venv. However, to be on the safe side, you can also use the system Python (&amp;lt;code&amp;gt;/usr/bin/python3.11&amp;lt;/code&amp;gt;) at creation time, which is available even without any &amp;lt;code&amp;gt;jupyter/{base,tensorflow}&amp;lt;/code&amp;gt; module loaded.&lt;br /&gt;
&lt;br /&gt;
As an alternative, an environment managed by &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; can be used. The corresponding Jupyter kernel can be installed with:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
uv run python -m ipykernel install \&lt;br /&gt;
    --user \&lt;br /&gt;
    --name myEnv \&lt;br /&gt;
    --display-name &amp;quot;Python (myEnv)&amp;quot;&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
After installing the kernel (and possibly refreshing the browser window), a button named &amp;quot;Python (myEnv)&amp;quot; is available in JupyterLab. The kernel can also be selected from the drop-down menu.&lt;br /&gt;
&lt;br /&gt;
&#039;&#039;&#039;Attention&#039;&#039;&#039;&lt;br /&gt;
The &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt; environment must contain &amp;lt;code&amp;gt;ipykernel&amp;lt;/code&amp;gt;. If it is not already installed, it can be added to the environment with:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
uv add ipykernel&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
When using &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt;, the environment does not have to be activated with &amp;lt;code&amp;gt;source&amp;lt;/code&amp;gt; before installing the kernel. The &amp;lt;code&amp;gt;uv run&amp;lt;/code&amp;gt; command automatically runs Python from the environment managed by &amp;lt;code&amp;gt;uv&amp;lt;/code&amp;gt;.&lt;br /&gt;
&lt;br /&gt;
== Examples on Data processing, Machine Learning &amp;amp; Visualization ==&lt;br /&gt;
&lt;br /&gt;
In the [https://github.com/hpcraink/workshop-parallel-jupyter/ workshop repository] the usage and best practices on Python in general, and the packages NumPy, Pandas, SciKit and Dask are provided, containing running examples based on open data. It also explains, how Jupyter interacts with pre-installed and your own provided environments.&lt;br /&gt;
&lt;br /&gt;
== R language ==&lt;br /&gt;
&lt;br /&gt;
In order to use R language in JupyterLab, the Lmod module &amp;lt;code&amp;gt;math/R&amp;lt;/code&amp;gt; has to be loaded (blue button in JupyterLab or &amp;lt;code&amp;gt;module add math/R&amp;lt;/code&amp;gt; in terminal) and a corresponding kernel has to be installed.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
R&lt;br /&gt;
install.packages(&#039;IRkernel&#039;)&lt;br /&gt;
IRkernel::installspec()&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
After installing the kernel , a button named &amp;quot;R&amp;quot; is available in JupyterLab. The kernel can also be selected from the drop-down menu.&lt;br /&gt;
&lt;br /&gt;
&#039;&#039;&#039;Attention:&#039;&#039;&#039;&lt;br /&gt;
Don&#039;t forget to load the &amp;lt;code&amp;gt;math/R&amp;lt;/code&amp;gt; module (blue button) before using the kernel.&lt;br /&gt;
&lt;br /&gt;
== Julia language ==&lt;br /&gt;
&lt;br /&gt;
In order to use Julia language in JupyterLab, the Lmod module &amp;lt;code&amp;gt;math/julia/1.10.8&amp;lt;/code&amp;gt; has to be loaded (blue button in JupyterLab or &amp;lt;code&amp;gt;module math/julia/1.10.8&amp;lt;/code&amp;gt; in terminal). When the module is loaded from the JupytherLab UI, the corresponding kernel will be installed. If you use the terminal, this has to be done manually&lt;br /&gt;
&lt;br /&gt;
&amp;lt;pre&amp;gt;&lt;br /&gt;
julia&lt;br /&gt;
]&lt;br /&gt;
add IJulia&lt;br /&gt;
&amp;lt;/pre&amp;gt;&lt;br /&gt;
&lt;br /&gt;
After installing the kernel, a button named &amp;quot;Julia 1.10.8&amp;quot; is available in JupyterLab. The kernel can also be selected from the drop-down menu.&lt;br /&gt;
&lt;br /&gt;
&#039;&#039;&#039;Attention:&#039;&#039;&#039;&lt;br /&gt;
Don&#039;t forget to load the &amp;lt;code&amp;gt;math/julia/1.10.8&amp;lt;/code&amp;gt; module (blue button) before using the kernel.&lt;br /&gt;
&lt;br /&gt;
= Jupyter Container Mode =&lt;br /&gt;
&lt;br /&gt;
The container integration on the jupyterhub is done via pyxis. In order to use it the checkmark for container mode has to be clicked. Available options are:&lt;br /&gt;
&lt;br /&gt;
* &amp;lt;code&amp;gt; --container-image:&amp;lt;/code&amp;gt; The container image to use. Corresponds to the pyxis option --container-image&lt;br /&gt;
* &amp;lt;code&amp;gt; --container-name:&amp;lt;/code&amp;gt; The name of the image to use. Corresponds to the pyxis option --container-name. Already downloaded containers in ~/.local/share/enroot can be startet by simply specifing their name.&lt;br /&gt;
* &amp;lt;code&amp;gt; --container-mount-home:&amp;lt;/code&amp;gt; Corresponds to the pyxis option --container-mount-home. Mounts the home-directory&lt;br /&gt;
&lt;br /&gt;
&#039;&#039;&#039;Attention:&#039;&#039;&#039;&lt;br /&gt;
Make sure Python3.11 and pip are installed in the Container or the notebook will not spawn. This can be checked via command python3.11 -m pip list inside your container&lt;br /&gt;
&lt;br /&gt;
It is advised to create the container prior via e.G. enroot and install all necessary software. For more information see [https://wiki.bwhpc.de/e/BwUniCluster2.0/Containers#SLURM_Integration here]&lt;br /&gt;
&lt;br /&gt;
----&lt;/div&gt;</summary>
		<author><name>I Tachakor</name></author>
	</entry>
</feed>