Development/ollama

Using LLMs even for inferencing requires large computational resources - currently at best a powerful GPU -- as provided by the bwHPC clusters. This page explains to how to make usage of bwHPC resources, using Ollama as an example to show best practices at work.

Introduction

Ollama is an inferencing framework that provides access to a multitude of powerful, large models and allows performant access to a variety of accelerators, e.g. from CPUs using AVX-512 to APUs like the AMD MI-300A, as well as GPUs like multiple NVIDIA H100.

Installing the inference server Ollama by default assumes you have root permission to install the server globally for all users into the directory /usr/local/bin. Of course, this is not sensible. Therefore the clusters provide the Environment Modules including binaries and libraries for CPU (if available AVX-512), AMD ROCm (if available) and NVIDIA CUDA using:

module load devel/ollama

More information is available in Ollamas Github documentation page.

The inference server Ollama opens the well-known port 11434. The compute node's IP is on the internal network, e.g. 10.1.0.101, which is not visible to any outside computer like Your laptop. Therefore we need a way to forward this port on an IP visible to the outside, aka the login nodes.

Preparation

Prior to starting and pulling models, it is a good idea to allocate a proper Workspace for the (multi-gigabyte) models and create a soft-link into this directory for Ollama:

ws_allocate ollama_models 60
ln -s `ws_find ollama_models`/ ~/.ollama

Now we may allocate a compute node using Slurm. At first You may start with interactively checking out the method in one terminal:

srun --time=00:30:00 --gres=gpu:1 --pty /bin/bash

Please note that on bwUniCluster, You need to provide a partition, here containing a GPU, e.g. for this 30 minute run, we may select --partition=dev_gpu_4, on DACHS --partition=gpu1.

Your Shell's prompt will list the node's name, e.g. on bwUniCluster node uc2n520:

[USERNAME@uc2n520 ~]$

Now You may load the Ollama module and start the server on the compute node and make sure using OLLAMA_HOST that it serves to the external IP address:

module load devel/ollama
export OLLAMA_HOST=0.0.0.0:11434
ollama serve

You should be able to see the usage of the accelerator:

Accessing from login nodes

From another terminal You may log into the Cluster's login node a second time and pull a LLM (please check [1] for available models):

module load devel/ollama
export OLLAMA_HOST=uc2n520
ollama pull deepseek-r1

On the previous terminal on the compute node, You should see the model being downloaded and installed into the workspace. Of course developing on the login nodes is not viable, therefore You may want to forward the ports.

Port forwarding

The login nodes of course have externally visible IP addresses, e.g. bwunicluster.scc.kit.edu which get to resolved to one of the multiple login nodes. Using the Secure shell ssh one may forward a port from the login node to the compute node.

Of course, You may want to locally on Your laptop. Open another terminal and start the Secure shell using the port forwarding:

ssh -L 11434:uc2n520:11434 USERNAME@bwunicluster.scc.kit.edu
Your OTP: 123456
Password:

You may check using whether this worked using Your local browser on Your Laptop:

Local programming

Now that You made sure You have access to the compute nodes GPU, you may develop on your local system:

python -m venv ollama_test
. ollama_test/bin/activate
python -m pip install ollama
export OLLAMA_HOST=localhost

and call python to run the following code:

import ollama
response = ollama.chat(model='deepseek-r1', messages=[ { 'role': 'user', 'content': 'why is the sky blue?'},])
print(response)

You should now see DeepSeek's response regarding Rayleigh Scattering.

On the compute node, You will see the computation:

Enjoy!