Development/ollama - Revision history

H Winkhardt: UC2 -> UC3

2025-09-01T13:24:51Z

UC2 -> UC3

← Older revision		Revision as of 15:24, 1 September 2025
Line 34:		Line 34:
	ln -s `ws_find ollama_models`/ ~/.ollama		ln -s `ws_find ollama_models`/ ~/.ollama

	Now we may allocate a compute node using [[~~BwUniCluster2~~.0/Slurm\|Slurm]].		Now we may allocate a compute node using [[BwUniCluster3.0/Slurm\|Slurm]].
	At first You may start with interactively checking out the method in one terminal:		At first You may start with interactively checking out the method in one terminal:
	srun --time=00:30:00 --gres=gpu:1 --pty /bin/bash		srun --time=00:30:00 --gres=gpu:1 --pty /bin/bash

R Keller: /* Best Practice */

2025-02-21T15:37:21Z

Best Practice

← Older revision		Revision as of 17:37, 21 February 2025
Line 110:		Line 110:
	#SBATCH --mem=500G # considering bwUniCluster3 768GB, enough on DACHS		#SBATCH --mem=500G # considering bwUniCluster3 768GB, enough on DACHS
	#SBATCH --time=2:00:00 # Please be courteous to other users		#SBATCH --time=2:00:00 # Please be courteous to other users
			#SBATCH --mail-type=BEGIN # Email when the job starts
			#SBATCH --mail-user=my@mail.de # Your email address

	module load cs/ollama # Load the the		module load cs/ollama # Load the the
	export OLLAMA_HOST=0.0.0.0 # Serve on global interface		export OLLAMA_HOST=0.0.0.0 # Serve on global interface

R Keller: /* Best Practice */

2025-02-21T15:35:54Z

Best Practice

← Older revision		Revision as of 17:35, 21 February 2025
Line 103:		Line 103:

	== Best Practice ==		== Best Practice ==
	Running interactively is generally not a good idea, especially not with very large models. Better submit Your job with mail notification, here file <code>ollama.slurm</code>:		Running interactively is generally '''not''' a good idea, especially not with very large models. Better submit Your job with mail notification, here file <code>ollama.slurm</code>:
	#!/bin/bash		#!/bin/bash
	#SBATCH --partition=gpu_h100 # bwUniCluster3, for DACHS: gpu8		#SBATCH --partition=gpu_h100 # bwUniCluster3, for DACHS: gpu8

R Keller: /* Local programming */

2025-02-21T15:35:16Z

Local programming

← Older revision		Revision as of 17:35, 21 February 2025
Line 101:		Line 101:

	'''Enjoy!'''		'''Enjoy!'''

			== Best Practice ==
			Running interactively is generally not a good idea, especially not with very large models. Better submit Your job with mail notification, here file <code>ollama.slurm</code>:
			#!/bin/bash
			#SBATCH --partition=gpu_h100 # bwUniCluster3, for DACHS: gpu8
			#SBATCH --gres=gpu:h100:4 # bwUniCluster3, for DACHS: gpu:h100:8
			#SBATCH --ntasks-per-node=96 # considering bwUniCluster3 AMD EPYC9454, same on DACHS
			#SBATCH --mem=500G # considering bwUniCluster3 768GB, enough on DACHS
			#SBATCH --time=2:00:00 # Please be courteous to other users
			module load cs/ollama # Load the the
			export OLLAMA_HOST=0.0.0.0 # Serve on global interface
			export OLLAMA_KEEP_ALIVE=-1 # Do not unload model (default is 5 minutes)
			ollama serve

			After starting the SSH for portforwarding or on the login-node after setting <code>export OLLAMA_HOST=</code> to the allocated node (see output of <code>squeue</code>):
			ollama run deepseek-r1:671b
			>>> /? # Shows the help
			>>> /? shortcuts # Shows the keyboard shortcuts
			>>> /show # Show information regarding model, prompt
			>>> What is log(e)? # Returns explanation of logarithm under the assumption of base 10 and the natural logarithm including LaTeX Math notation.

R Keller: /* Local programming */

2025-02-20T13:07:21Z

Local programming

← Older revision		Revision as of 15:07, 20 February 2025
Line 85:		Line 85:
	Now that You made sure You have access to the compute nodes GPU, you may develop on your local system:		Now that You made sure You have access to the compute nodes GPU, you may develop on your local system:
	python -m venv ollama_test		python -m venv ollama_test
	. ollama_test/bin/activate		source ollama_test/bin/activate
	python -m pip install ollama		python -m pip install ollama
	export OLLAMA_HOST=localhost		export OLLAMA_HOST=localhost

R Keller at 13:06, 20 February 2025

2025-02-20T13:06:13Z

← Older revision		Revision as of 15:06, 20 February 2025
Line 11:		Line 11:
	into the directory <code>/usr/local/bin</code>. Of course, this is '''not''' sensible.		into the directory <code>/usr/local/bin</code>. Of course, this is '''not''' sensible.
	Therefore the clusters provide the [[Environment_Modules\|Environment Modules]] including binaries and libraries for CPU (if available AVX-512), AMD ROCm (if available) and NVIDIA CUDA using:		Therefore the clusters provide the [[Environment_Modules\|Environment Modules]] including binaries and libraries for CPU (if available AVX-512), AMD ROCm (if available) and NVIDIA CUDA using:
	module load ~~devel~~/ollama		module load cs/ollama

	More information is available in [https://github.com/ollama/ollama/tree/main/docs Ollamas Github documentation] page.		More information is available in [https://github.com/ollama/ollama/tree/main/docs Ollamas Github documentation] page.
Line 18:		Line 18:
	which is not visible to any outside computer like Your laptop.		which is not visible to any outside computer like Your laptop.
	Therefore we need a way to forward this port on an IP visible to the outside, aka the login nodes.		Therefore we need a way to forward this port on an IP visible to the outside, aka the login nodes.

			{\|style="background:#FEF4AB; width:100%;"
			\|style="padding:5px; background:#FEF4AB; text-align:left"\|
			[[Image:Attention.svg\|center\|25px]]
			\|style="padding:5px; background:#FEF4AB; text-align:left"\|
			Please note: this module started off in the Category <code>devel</code>, but has been moved to the correct category computer science, or short <code>cs</code>.
			\|}


	== Preparation ==		== Preparation ==
Line 35:		Line 43:

	Now You may load the Ollama module and start the server on the compute node and make sure using <code>OLLAMA_HOST</code> that it serves to the external IP address:		Now You may load the Ollama module and start the server on the compute node and make sure using <code>OLLAMA_HOST</code> that it serves to the external IP address:
	module load ~~devel~~/ollama		module load cs/ollama
	export OLLAMA_HOST=0.0.0.0:11434		export OLLAMA_HOST=0.0.0.0:11434
	ollama serve		ollama serve

R Keller: /* Accessing from login nodes */

2025-02-20T13:02:09Z

Accessing from login nodes

← Older revision		Revision as of 15:02, 20 February 2025
Line 45:		Line 45:
	== Accessing from login nodes ==		== Accessing from login nodes ==
	From another terminal You may log into the Cluster's login node a second time and pull a LLM (please check [https://ollama.com/search] for available models):		From another terminal You may log into the Cluster's login node a second time and pull a LLM (please check [https://ollama.com/search] for available models):
	module load ~~devel~~/ollama		module load cs/ollama
	export OLLAMA_HOST=uc2n520		export OLLAMA_HOST=uc2n520
	ollama pull deepseek-r1		ollama pull deepseek-r1
Line 51:		Line 51:
	On the previous terminal on the compute node, You should see the model being downloaded and installed into the workspace.		On the previous terminal on the compute node, You should see the model being downloaded and installed into the workspace.
	Of course developing on the login nodes is not viable, therefore You may want to forward the ports.		Of course developing on the login nodes is not viable, therefore You may want to forward the ports.


			{\|style="background:#dedefe; width:100%;"
			\|style="padding:5px; background:#dedefe; text-align:left"\|
			[[Image:Info.svg\|center]]
			\|style="padding:5px; background:#dedefe; text-align:left"\|
			On GPUs with 48GB VRAM like NVIDIA L40S, you may want to use the 70b model of Deepseek, i.e. <code>ollama pull deepseek-r1:70b</code> and amend the below commands accordingly.
			\|}

	== Port forwarding ==		== Port forwarding ==

H Winkhardt: Typo

2025-02-13T14:06:39Z

Typo

← Older revision		Revision as of 16:06, 13 February 2025
Line 1:		Line 1:
	Using LLMs even for inferencing requires large computational resources - currently at best a powerful GPU --		Using LLMs even for inferencing requires large computational resources - currently at best a powerful GPU -
	as provided by the bwHPC clusters.		as provided by the bwHPC clusters.
	This page explains to how to make usage of bwHPC resources, using Ollama as an example to show best practices at work.		This page explains to how to make usage of bwHPC resources, using Ollama as an example to show best practices at work.

R Keller: /* Preparation */

2025-02-13T08:48:42Z

Preparation

← Older revision		Revision as of 10:48, 13 February 2025
Line 24:		Line 24:
	and create a soft-link into this directory for Ollama:		and create a soft-link into this directory for Ollama:
	ws_allocate ollama_models 60		ws_allocate ollama_models 60
	ln -s `ws_find ollama_models`/ .ollama		ln -s `ws_find ollama_models`/ ~/.ollama

	Now we may allocate a compute node using [[BwUniCluster2.0/Slurm\|Slurm]].		Now we may allocate a compute node using [[BwUniCluster2.0/Slurm\|Slurm]].

R Keller: /* Preparation */

2025-02-11T20:06:53Z

Preparation

← Older revision		Revision as of 22:06, 11 February 2025
Line 31:		Line 31:
	Please note that on bwUniCluster, You need to provide a partition, here containing a GPU, e.g. for this 30 minute run, we may select <code>--partition=dev_gpu_4</code>, on DACHS <code>--partition=gpu1</code>.		Please note that on bwUniCluster, You need to provide a partition, here containing a GPU, e.g. for this 30 minute run, we may select <code>--partition=dev_gpu_4</code>, on DACHS <code>--partition=gpu1</code>.

	Your Shell's prompt will list the ~~nodes~~ name, e.g. on bwUniCluster node <code>uc2n520</code>:		Your Shell's prompt will list the node's name, e.g. on bwUniCluster node <code>uc2n520</code>:
	[USERNAME@uc2n520 ~]$		[USERNAME@uc2n520 ~]$