DACHS/Queues - Revision history

M Kunzelmann: /* Multiple nodes */

2025-11-25T07:37:13Z

Multiple nodes

← Older revision		Revision as of 09:37, 25 November 2025
Line 88:		Line 88:
	== Multiple nodes ==		== Multiple nodes ==
	Of course You may allocate multiple GPUs across nodes running:		Of course You may allocate multiple GPUs across nodes running:

	sbatch --nodes 4 ./python_run.slurm		sbatch --nodes 4 ./python_run.slurm

	Please be aware, that TMPDIR is still local. For the time being run from Your $HOME or better yet from an allocated [[Workspace]].		Please be aware, that TMPDIR is still local. For the time being run from Your $HOME or better yet from an allocated [[Workspace]].

M Kunzelmann: /* Interactive usage */

2025-11-25T07:37:00Z

Interactive usage

← Older revision		Revision as of 09:37, 25 November 2025
Line 80:		Line 80:
	== Interactive usage ==		== Interactive usage ==
	To '''get a good estimation''' of runtime, You may first want to try the resource ''interactively'':		To '''get a good estimation''' of runtime, You may first want to try the resource ''interactively'':

	srun --partition=gpu1 --ntasks-per-gpu=48 --gres=gpu:1 --pty /bin/bash		srun --partition=gpu1 --ntasks-per-gpu=48 --gres=gpu:1 --pty /bin/bash

	Then You may execute the steps in <code>python_run.slurm</code> script interactively, noting differences and amend your Slurm batch script.		Then You may execute the steps in <code>python_run.slurm</code> script interactively, noting differences and amend your Slurm batch script.

M Kunzelmann: /* sinfo_t_idle */

2025-11-25T07:36:25Z

sinfo_t_idle

← Older revision		Revision as of 09:36, 25 November 2025
Line 10:		Line 10:
	== sinfo_t_idle ==		== sinfo_t_idle ==
	To see the available nodes, DACHS offers the tool ''sinfo_t_info'', which any user may call.		To see the available nodes, DACHS offers the tool ''sinfo_t_info'', which any user may call.

			<syntaxhighlight lang="sh">
			$ sinfo_t_idle
			Partition gpu1* : 7 nodes idle
			Partition gpu4 : 1 nodes idle
			Partition gpu8 : 0 nodes idle
			</syntaxhighlight>

	== sbatch -p ''partition'' ==		== sbatch -p ''partition'' ==

M Kunzelmann: /* Partitions */

2025-11-25T07:34:25Z

Partitions

← Older revision		Revision as of 09:34, 25 November 2025
Line 2:		Line 2:

	== Partitions ==		== Partitions ==
	DACHS offers three partitions in Slurm, which map directly to the node types: ~~nodes with one NVIDIA L40S GPU, a node with 4 AMD MI300A APUs and the node with 8 NVIDIA H100 GPUs.~~		DACHS offers three partitions in Slurm, which map directly to the node types:

			* 45 nodes with one NVIDIA L40S GPU
			* one node with 4 AMD MI300A APUs
			* one node with 8 NVIDIA H100 GPUs

	== sinfo_t_idle ==		== sinfo_t_idle ==

R Keller at 09:53, 12 August 2025

2025-08-12T09:53:41Z

← Older revision		Revision as of 11:53, 12 August 2025
Line 41:		Line 41:

	A typical Slurm batch script (called for brevity <code>python_run.slurm</code>) for 1-node requiring one NVIDIA L40S GPU:		A typical Slurm batch script (called for brevity <code>python_run.slurm</code>) for 1-node requiring one NVIDIA L40S GPU:
	#!/bin/bash		#!/bin/bash
	#SBATCH --partition=gpu1		#SBATCH --partition=gpu1
	#SBATCH --ntasks-per-gpu=48		#SBATCH --ntasks-per-gpu=48
	#SBATCH --gres=gpu:1		#SBATCH --gres=gpu:1
	#SBATCH --time=1:00:00		#SBATCH --time=1:00:00
	#SBATCH --mail-type=all		#SBATCH --mail-type=all
	#SBATCH --mail-user=my_email@hs-esslingen.de		#SBATCH --mail-user=my_email@hs-esslingen.de
	module load devel/cuda/12.4		module load devel/cuda/12.4
	cd $TMPDIR		cd $TMPDIR
	python3 -m venv my_environment		python3 -m venv my_environment
	. my_environment/bin/activate		. my_environment/bin/activate
	python3 -m pip install -r $HOME/my_requirements.txt		python3 -m pip install -r $HOME/my_requirements.txt
	rsync -avz $HOME/my_data_dir/ .		rsync -avz $HOME/my_data_dir/ .
	time python3 $HOME/python_script.py		time python3 $HOME/python_script.py

	Submitting <code>sbatch python_run.slurm</code> will allocate one compute node and allocate the one available GPU for 1 hour. Furthermore, this will load the CUDA module version 12.4. It will then change to the '''fast''' scratch directory specified in the environment variable <code>TMPDIR</code>.		Submitting <code>sbatch python_run.slurm</code> will allocate one compute node and allocate the one available GPU for 1 hour. Furthermore, this will load the CUDA module version 12.4. It will then change to the '''fast''' scratch directory specified in the environment variable <code>TMPDIR</code>.

R Keller: /* sbatch -p partition */

2025-05-14T08:46:29Z

sbatch -p partition

← Older revision		Revision as of 10:46, 14 May 2025
Line 12:		Line 12:
	These are restricted and must fit the available '''partitions'''.		These are restricted and must fit the available '''partitions'''.
	Since requested compute resources are NOT always automatically mapped to the correct queue class, '''you must add the correct queue class to your sbatch command '''.		Since requested compute resources are NOT always automatically mapped to the correct queue class, '''you must add the correct queue class to your sbatch command '''.
	<font color=red>As with ~~BwUniCluster 2.0~~, the specification of a partition is required.</font>		<font color=red>As with bwUniCluster, the specification of a partition is required.</font>
	<br>		<br>
	Details are:		Details are:

R Keller: /* Multiple nodes */

2025-05-14T08:45:55Z

Multiple nodes

← Older revision		Revision as of 10:45, 14 May 2025
Line 77:		Line 77:
	Of course You may allocate multiple GPUs across nodes running:		Of course You may allocate multiple GPUs across nodes running:
	sbatch --nodes 4 ./python_run.slurm		sbatch --nodes 4 ./python_run.slurm
	Please be aware, that TMPDIR is still local. For the time being run from Your $HOME.		Please be aware, that TMPDIR is still local. For the time being run from Your $HOME or better yet from an allocated [[Workspace]].


	== Nodes with multiple GPUs ==		== Nodes with multiple GPUs ==

R Keller: /* Nodes with multiple GPUs */

2025-05-14T08:44:51Z

Nodes with multiple GPUs

← Older revision		Revision as of 10:44, 14 May 2025
Line 82:		Line 82:
	== Nodes with multiple GPUs ==		== Nodes with multiple GPUs ==
	The partitions <code>gpu4</code> and <code>gpu8</code> feature multiple GPUs.		The partitions <code>gpu4</code> and <code>gpu8</code> feature multiple GPUs.
	The <code>gpu4</code> partition contains the node <code>gpu401</code> featuring 4 AMD MI300A APUs with 128GB of memory ~~each~~ ~~using~~ ~~ROCm~~.		The <code>gpu4</code> partition contains the node <code>gpu401</code> featuring 4x AMD MI300A APUs each with 128GB of fast HMB3e memory shared between the 24 cores and the GPU.
	Please refer to the documentation on this node.		You may use AMD's ROCm employing HIP, OpenACC or OpenCL to parallelize for the GPU. Please refer to the documentation on this node.

	The <code>gpu8</code> partition contains the node <code>~~gpu401~~</code> featuring 4 ~~AMD~~ ~~MI300A~~ ~~APUs~~ ~~with 128GB~~ of ~~memory~~ each using ~~ROCm~~.		The <code>gpu8</code> partition contains the node <code>gpu801</code> featuring 8x NVIDIA H100 offering 80GB of VRAM each, interconnected using SXM5.
	Please refer to the documentation on this node.		Please refer to the documentation on this node.

R Keller: /* Interactive usage */

2025-03-18T13:25:20Z

Interactive usage

← Older revision		Revision as of 15:25, 18 March 2025
Line 69:		Line 69:
	== Interactive usage ==		== Interactive usage ==
	To '''get a good estimation''' of runtime, You may first want to try the resource ''interactively'':		To '''get a good estimation''' of runtime, You may first want to try the resource ''interactively'':
	srun --partition=gpu1 --ntasks-per-gpu=48 --gres=~~gpu1~~ --pty /bin/bash		srun --partition=gpu1 --ntasks-per-gpu=48 --gres=gpu:1 --pty /bin/bash

	Then You may execute the steps in <code>python_run.slurm</code> script interactively, noting differences and amend your Slurm batch script.		Then You may execute the steps in <code>python_run.slurm</code> script interactively, noting differences and amend your Slurm batch script.

R Keller: /* sbatch -p partition */

2024-12-18T11:46:57Z

sbatch -p partition

← Older revision		Revision as of 13:46, 18 December 2024
Line 19:		Line 19:
	! colspan="5" \| DACHS <br> sbatch -p ''partition''		! colspan="5" \| DACHS <br> sbatch -p ''partition''
	\|- style="text-align:left;"		\|- style="text-align:left;"
	! ~~queue~~ !! node !! default resources !! maximum resources		! partition !! node !! default resources !! maximum resources
	\|- style="text-align:left"		\|- style="text-align:left"
	\| gpu1		\| gpu1