Helix/bwVisu/JupyterLab

JupyterLab is an integrated development environment (IDE) that provides a flexible and scalable interface for the Jupyter Notebook system. It supports interactive data science and scientific computing across over 40 programming languages (including Python, Julia, and R).

Change Python Version

The default python version can be seen by running python --version in the terminal.

A different python version can be installed into a new virtual environment and then registered as IPython kernel for the usage in JupyterLab. This is explained in the chapter add packages via conda environments.

Install Python Packages

Python packages can be added by installing them into a virtual environment and then creating an IPython kernel from the virtual environment.
Kernels can be shared. See the notes below.
If you want to move a virtual environment, it is adivsed to recreate it in the new place. Otherwise, dependencies based on relative paths will break.

1. Create a virtual environment with...
   • ...venv or
   • ...conda (choose this option if you want to install a different python version) or
   • ...uv if you want to install a different python version but don't want to use conda.
2. Create an IPython kernel from the virtual environment
3. Use the kernel within JupyterLab
   • By default new kernels are saved under ~/.local/share/jupyter and this location is automatically detected. Therefore, new kernels are directly available.

     

     In the JupyterLab job configuration form, a custom kernel path can be provided.

   • If the kernel is saved somewhere else, the path can be provided in the "Kernel path" field when configuring the JupyterLab job (see image). For a kernel placed under path_to_parent_dir/share/jupyter/kernels/my_kernel the needed "Kernel path" would be path_to_parent_dir/share/jupyter.
4. When the kernel is used the first time, the file notebook_secrets is created automatically. It can be found under "Kernel path". For others to use the kernel, they must have read access to this file. The command chmod 750 notebook_secrets would for example allow the whole SDS@hd SV read access. For the access management in workspaces, please see Workspace Permissions.

Notes regarding the sharing of IPython kernels

• The virtual environment and the kernel need to be placed in a shared directory. For example at SDS@hd.
• There could be a subdirectory for the virtual environments and one for the kernels.
• The path to the kernels is saved in the environment variable $JUPYTER_DATA_DIR. Jupyter relevant paths can be seen with jupyter --paths

Add packages via venv virtual environments

More information about venv or other python virtual environments can be found at the Python page.

Steps for creating a venv virtual environment:

1. Open a terminal.
2. Create a new virtual evironment:

   python3 -m venv <env_parent_dir>/<env_name>
   

   • env_parent_dir is the path to the folder where the virtual environment shall be created. Relative paths can be used.
   • Caution: If you you want to share the environment with others, make sure to already create it in the shared place.
3. Activate the environment:

   source <env_parent_dir>/<env_name>/bin/activate
   

4. Update pip and install packages:

   pip install -U pip --no-user
   # when the environment is installed in home
   pip install <packagename>
   # when the environment is installed somewhere else and shall not have dependencies in home so that others can access it as well
   pip install <packagename> --no-user
   

5. Create an IPython kernel

Add packages via Conda virtual environments

More information about using conda can be found at the Conda page.

1. Load the miniforge module by clicking first on the double hexagon icon on the left-hand side of Jupyter's start page and then on the "load" button right of the entry for miniforge in the software module menu.
2. Open a terminal.
3. Create a new virtual environment:
   • If you are the only person using the environment, you can install it in your home directory:

   conda create --name <env_name> python=<python version>
   

   • If you want to install it into a different directory, for example a shared place:

   conda create --prefix <path_to_shared_directory>/<env_name> python=<python version>
   

4. Activate your environment:

   conda activate <env_name>
   

5. Install your packages:

   conda install <packagename>
   

6. Create an IPython kernel

Add packages via uv virtual environments

More information about using uv can be found at the Python page.

Steps for creating a uv virtual environment:

1. Open a terminal
2. Create a new project and virtual environment:

   uv init <env_parent_dir>/<env_name>
   uv venv <env_parent_dir>/<env_name> --python <python version>
   

   • env_parent_dir is the path to the folder where the virtual environment shall be created. Relative paths can be used.
   • The --python <python_version> option can be omitted if the default Python version should be used.
   • Caution: If you want to share the environment with others, make sure to
     • Set your $HOME to the shared place before creating the environment: HOME=<env_name>
     • Create the environment directly in the shared place.
3. Install packages into the environment:

   uv add <packagename>
   

4. Create an IPython kernel

Unlike a traditional venv environment, a uv environment does not have to be activated with source. The uv commands can directly use the Python interpreter from the environment.

Create an IPython Kernel

Python kernels are implementations of the Jupyter notebook environment for different languages or virtual environments. You can switch between kernels easily, allowing you to use the best tool for a specific task. conda_kernels

Create a kernel from a virtual environment

Please follow these instructions to create and find the kernel.

Step 1: Create the kernel

Using venv or Conda

1. Activate the virtual environment.
2. Install the ipykernel package:

   pip install ipykernel
   

3. Register the virtual environment as custom kernel to Jupyter.
   • If you are the only person using the environment:

   python3 -m ipykernel install --user --name=<kernel_name>
   

   • If you installed the environment in a shared place and want to have the kernel there as well:

   python3 -m ipykernel install --prefix <path_to_kernel_folder> --name=<kernel_name>
   

Using uv

1. Install the ipykernel package:

   uv add ipykernel
   

2. Register the virtual environment as custom kernel to Jupyter.
   • If you are the only person using the environment:

   uv run ipython kernel install --user --env VIRTUAL_ENV $(pwd)/.venv --name=<kernel_name>
   

   • If you installed the environment in a shared place and want to have the kernel there as well:

   uv run ipython kernel install --prefix <path_to_kernel_folder> --env VIRTUAL_ENV $(pwd)/.venv --name=<kernel_name>
   

Step 2: Find the kernel

• Private kernel:

  The kernel can be found under ~/.local/share/jupyter/kernels/.

• Shared kernel:

  The kernel can be found under path_to_kernel_folder/share/jupyter/kernels/<kernel_name>. As long as the same path_to_kernel_folder is used, all kernels will be saved next to each other in "kernels".

Multi-Language Support

JupyterLab supports over 40 programming languages including Python, R, Julia, and Scala. This is achieved through the use of different kernels.

R Kernel

1. On the cluster:

   $ module load math/R
   $ R
   > install.packages('IRkernel')

2. On bwVisu:
   1. Start Jupyter App
   2. In left menu: load math/R
   3. Open Console:

   $ R
   > IRkernel::installspec(displayname = 'R 4.2')

   4. Start kernel 'R 4.2' as console or notebook

Julia Kernel

Load the math/julia module. Open the Terminal.

julia
]
add IJulia

After that, Julia is available as a kernel.

Interactive Widgets

JupyterLab supports interactive widgets that can create UI controls for interactive data visualization and manipulation within the notebooks. Example of using an interactive widget:

from ipywidgets import IntSlider
slider = IntSlider()
display(slider)

These widgets can be sliders, dropdowns, buttons, etc., which can be connected to Python code running in the backend.

FAQ

1. How can I use bwForCluster Helix software modules?
   First click on the double hexagon icon on the left-hand side of Jupyter's start page. Then load a module by clicking on the "load" button next to the corresponding module entry.
2. My virtual environment works on Helix but not in the bwVisu JupyterLab job.
   If the environment uses Helix modules, you have to load the modules in the bwVisu job first. For example, when you used the python 3.13 module for creating the environment, the path to this python version is saved in the environment. When the module is not loadedin bwVisu, the path is not available and you might get an error like "[...]/bin/python: error while loading shared libraries: libpython3.13.so.1.0: cannot open shared object file: No such file or directory"
3. How can I navigate to my SDS@hd folder in the file browser?
   Please see the instructions at the Usage page.
4. I prefer VSCode over JupyterLab. Can I start a JupyterLab job and then connect with it via VSCode?
   This is not possible. Please start the job directly on Helix instead. You can find the instructions at the VSCode page.
5. My conda commands are interrupted with message 'Killed'.
   Request more memory when starting Jupyter.
6. Jupyterlab doesn't let me in but asks for a password.
   Try using more memory for the job. If this doesn't help, try using the inkognito mode of your browser as the browser cache might be the problem.