NEMO2/Slurm - Revision history

M Janczyk: /* Accounting */

2026-06-16T15:04:49Z

Accounting

← Older revision		Revision as of 17:04, 16 June 2026
Line 451:		Line 451:
	= Accounting =		= Accounting =

			Slurm tracks resource usage per job, user, and research project (RV/account). No CPU-hour budget is consumed — the accounting data is purely informational and useful for analysing your usage.
	Jobs are billed for allocated CPU cores, memory and GPUs.

			== Job Details ==
⚫	~~To see the~~ accounting data of a specific job:

		⚫	Show accounting data for a specific job:
⚫	<syntaxhighlight lang="shell">$ sacct -j <jobid> --format=user,jobid,account,nnodes,ncpus~~,time~~,elapsed,AllocTRES%50</syntaxhighlight>

		⚫	<syntaxhighlight lang="shell">$ sacct -j <jobid> --format=user,jobid,account,nnodes,ncpus,elapsed,AllocTRES%60</syntaxhighlight>
⚫	~~To retrive~~ the job history for a ~~specific~~ user ~~for~~ a ~~certain~~ time ~~frame~~:

			== Job History for a User ==
⚫	<syntaxhighlight lang="shell">$ sacct -u <user> -S ~~2025~~-03-01 -E ~~2025~~-03-02 --format=user,jobid,account,nnodes,ncpus~~,time~~,elapsed,~~AllocTRES~~%50</syntaxhighlight>

		⚫	Retrieve the job history for your own user over a time period:

		⚫	<syntaxhighlight lang="shell">$ sacct -u <user> -S 2026-06-01 -E 2026-06-16 --format=user,jobid,account,partition,nnodes,ncpus,elapsed,state%12</syntaxhighlight>

			== Core-Hour Usage (own user) ==

			Use <code>sreport</code> for fast aggregated queries — it reads pre-aggregated data from the database and is much faster than <code>sacct -a</code> (which iterates every job step). Without <code>-a</code>, only your own data is shown:

			<syntaxhighlight lang="shell">
			# Your CPU-hours this month, broken down by account/RV
			$ sreport user top -t hour Start=$(date +%Y-%m-01) Users=$USER

			# For a specific time range
			$ sreport user top -t hour Start=2026-05-01 End=2026-05-31 Users=$USER
			</syntaxhighlight>

			== Core-Hour Usage per Research Project (RV / Account) ==

			<syntaxhighlight lang="shell">
			# Your usage broken down by account/RV
			$ sreport cluster AccountUtilizationByUser -t hour Start=$(date +%Y-%m-01) Users=$USER

			# All users of one specific account/RV (e.g. bw12a001)
			$ sreport cluster AccountUtilizationByUser -t hour Start=$(date +%Y-%m-01) Accounts=bw12a001
			</syntaxhighlight>

			== GPU-Hour Usage ==

			For GPU jobs, filter by partition. Without <code>-a</code> only your own jobs are shown:

			<syntaxhighlight lang="shell">
			# Your GPU-hours on the l40s partition this month
			$ sacct -S $(date +%Y-%m-01) -E now -r l40s \
			--format=jobid,elapsed,AllocTRES%60 \
			--state=CD,F,TO,OOM \
			\| awk '/gpu=/ {
			match($3, /gpu=([0-9]+)/, g);
			split($2, t, ":");
			h = t[1]*1 + t[2]/60 + t[3]/3600;
			total += g[1] * h
			}
			END { printf "%.1f GPU-hours\n", total }'
			</syntaxhighlight>

M Janczyk: /* Show free nodes */

2026-06-16T14:37:33Z

Show free nodes

← Older revision		Revision as of 16:37, 16 June 2026
Line 186:		Line 186:
	== Show free nodes ==		== Show free nodes ==

	The Slurm command <code>sinfo</code> is used to view partition and node information for a system running Slurm~~, unfortunately it can only be used by the administrators. For users there is a special binary <code>sinfo_t_idle</code> to find out how many nodes are available on the system for immediate use~~.		The Slurm command <code>sinfo</code> is used to view partition and node information for a system running Slurm.

	<pre>		<pre>
			$ sinfo -t idle
	$ sinfo_t_idle
			PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
	Partition cpu* : 127 nodes idle
	~~Partition~~ ~~genoa~~ ~~: 30 nodes idle~~		cpu* up 4-00:00:00 2 idle n[3303-3304]
	~~Partition~~ ~~milan~~ ~~: 97 nodes idle~~		genoa up 4-00:00:00 0 n/a
	~~Partition~~ ~~l40s~~ ~~: 6 nodes idle~~		milan up 4-00:00:00 2 idle n[3303-3304]
	~~Partition~~ ~~mi300a~~ ~~: 3~~ ~~nodes~~ ~~idle~~		l40s up 2-00:00:00 0 n/a
	~~Partition~~ ~~login~~ ~~: 2 nodes~~ ~~idle~~		h200 up 2-00:00:00 0 n/a
			rtx up 2-00:00:00 0 n/a
			mi300a up 2-00:00:00 3 idle a[3601-3603]
			login up 15:00 2 idle login[1-2]
	</pre>		</pre>

M Janczyk: /* Partitions */

2026-06-16T09:53:00Z

Partitions

← Older revision		Revision as of 11:53, 16 June 2026
Line 180:		Line 180:
	\| ---		\| ---
	\| ntasks=1, time=15:00, mem-per-cpu=1gb		\| ntasks=1, time=15:00, mem-per-cpu=1gb
	\| time=15:00, nodes=1, ntasks=4, mem=50gb		\| time=15:00, nodes=2, ntasks=4, mem=50gb
	\|}		\|}
	<code>*</code> default partition		<code>*</code> default partition

M Janczyk: /* Partitions */

2026-06-16T09:52:41Z

Partitions

← Older revision		Revision as of 11:52, 16 June 2026
Line 175:		Line 175:
	\|-		\|-
	\| login		\| login
	\| ~~single~~ node		\| multi node
	\| 1x AMD EPYC 9354 (Genoa) (SMT on)		\| 1x AMD EPYC 9354 (Genoa) (SMT on)
	\| ---		\| ---

M Janczyk at 09:37, 16 June 2026

2026-06-16T09:37:10Z

← Older revision		Revision as of 11:37, 16 June 2026
Line 84:		Line 84:
	'''GPUs'''		'''GPUs'''

	GPUs are requested as "generic resources" with <code>--gres:gpu:<count></code> or <code>--gpus:<count></code>.		GPUs are requested as "generic resources" with <code>--gres=gpu:<count></code> or <code>--gpus=<count></code>.

	== Default Values ==		== Default Values ==
Line 104:		Line 104:
	'''All''' partitions are operated in shared mode, i.e. jobs from different users can run on the same node. Jobs can get exclusive access to compute nodes in these partitions with the <code>--exclusive</code> option.		'''All''' partitions are operated in shared mode, i.e. jobs from different users can run on the same node. Jobs can get exclusive access to compute nodes in these partitions with the <code>--exclusive</code> option.

	The login partition is intended for quick tests on the login ~~lodes~~ with the Slurm environment.		The login partition is intended for quick tests on the login nodes with the Slurm environment.

	See [[NEMO2/Hardware]] for more information on the exact hardware specifications.		See [[NEMO2/Hardware]] for more information on the exact hardware specifications.
Line 264:		Line 264:

	'''Notes:'''		'''Notes:'''
	* Jobs with <code>--mem</code> up to ~~500gb~~ can run on all node types associated with the cpu partition.		* Jobs with <code>--mem</code> up to 495GB can run on all node types associated with the cpu partition (routing to either milan or genoa nodes).

	=== Multi-threaded Programs ===		=== Multi-threaded Programs ===
Line 318:		Line 318:
	== Interactive Tests on the Login Nodes ==		== Interactive Tests on the Login Nodes ==

	The login partition is intended for quick tests on the login ~~lodes~~ with the Slurm environment. The limits are 4 cores and 15 minutes of walltime.		The login partition is intended for quick tests on the login nodes with the Slurm environment. The limits are 4 cores and 15 minutes of walltime.

	<syntaxhighlight lang="shell">		<syntaxhighlight lang="shell">
Line 402:		Line 402:
	= Queue / Partition Limits =		= Queue / Partition Limits =

			'''Per-user CPU and core-minute limits depend on the partition/QOS:'''
	'''Currently, each user can use 8000 cores (QOSMaxCpuPerUserLimit) and 7,2 million core minutes (MaxCpuRunMinsPerUser) at any time.'''

			{\| class="wikitable"
⚫	The core minutes ~~can~~ be calculated by (running CPU cores) * (remaining runtime of all jobs in minutes) = ~~CpuRunMinsPerUser,~~ ~~example~~:
			\|-
	<code>5000cores * 24h * 60min = 7.200.000 CpuRunMinsPerUser</code>
			! Partition/QOS !! Max Cores (QOSMaxCpuPerUserLimit) !! Max Core-Minutes Running (MaxCpuRunMinsPerUser)
			\|-
			\| cpu (default) \|\| 12,000 \|\| 14,500,800
			\|-
			\| genoa \|\| 8,000 \|\| 14,500,800
			\|-
			\| milan \|\| 8,000 \|\| 12,428,640
			\|}

		⚫	The core-minutes limit is calculated as (running CPU cores) × (remaining runtime of all jobs in minutes). Example for genoa:
			<code>8000 cores × 30h × 60min = 14,400,000 core-minutes</code>

	This is constantly evaluated, you do not have to do anything.		This is constantly evaluated, you do not have to do anything.
	This value decreases after a few minutes (running CPU cores * remaining runtime of all jobs in minutes) or when a job ends.		This value decreases after a few minutes (running CPU cores * remaining runtime of all jobs in minutes) or when a job ends.

	'''The partitions 'cpu', 'l40s', 'mi300a' and '~~h200~~' only allow the use of single nodes.'''		'''The partitions 'cpu', 'l40s', 'mi300a', 'h200' and 'rtx' only allow the use of single nodes.'''
	You cannot start jobs that use more than one node, either by specifying <code>--nodes</code> (>1) or <code>--ntasks</code> (more than available on one node).		You cannot start jobs that use more than one node, either by specifying <code>--nodes</code> (>1) or <code>--ntasks</code> (more than available on one node).
	If you do so, your jobs will be blocked with the reason (PartitionNodeLimit).		If you do so, your jobs will be blocked with the reason (PartitionNodeLimit).
Line 428:		Line 439:
	\| QOSMaxCpuPerUserLimit		\| QOSMaxCpuPerUserLimit
	\| Your job cannot be started as your user would otherwise exceed the maximum permitted cores for one user.		\| Your job cannot be started as your user would otherwise exceed the maximum permitted cores for one user.
			\|-
			\| QOSMaxGRESPerUser
			\| Your job exceeds the per-user GPU limit (l40s: 16 GPUs, h200: 8 GPUs, rtx: 4 GPUs, mi300a: 8 GPUs).
	\|}		\|}

	Check additional [https://slurm.schedmd.com/job_reason_codes.html reason codes] at Schedmd.		Check additional [https://slurm.schedmd.com/job_reason_codes.html reason codes] at Schedmd.

	= Accounting ~~(not implemented)~~ =		= Accounting =

	Jobs are billed for allocated CPU cores, memory and GPUs.		Jobs are billed for allocated CPU cores, memory and GPUs.

M Janczyk: /* Partitions */

2026-04-09T13:06:04Z

Partitions

← Older revision		Revision as of 15:06, 9 April 2026
Line 169:		Line 169:
	\| single node		\| single node
	\| 2x AMD EPYC 9655 (Turin)		\| 2x AMD EPYC 9655 (Turin)
	\| 8x 4x NVIDIA RTX PRO 6000 Blackwell Server Edition		\| 4x NVIDIA RTX PRO 6000 Blackwell Server Edition
	\| 47		\| 47
	\| ntasks=1, time=01:00:00, mem-per-cpu=1gb		\| ntasks=1, time=01:00:00, mem-per-cpu=1gb

M Janczyk: /* Monitoring Jobs with squeue */

2026-02-02T16:46:00Z

Monitoring Jobs with squeue

← Older revision		Revision as of 18:46, 2 February 2026
Line 206:		Line 206:
	<syntaxhighlight lang='shell'>		<syntaxhighlight lang='shell'>
	> squeue		> squeue
	JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)		JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
	426 cpu 20k fr_xy0123 R 0:02 2 n[4101-4102]		426 cpu 20k fr_xy0123 R 0:02 2 n[4101-4102]
	</syntaxhighlight>		</syntaxhighlight>

M Janczyk: /* Monitoring Jobs with squeue */

2026-02-02T16:45:30Z

Monitoring Jobs with squeue

← Older revision		Revision as of 18:45, 2 February 2026
Line 207:		Line 207:
	> squeue		> squeue
	JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)		JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
	426 cpu 20k ~~fr_0123~~ R 0:02 2 n[4101-4102]		426 cpu 20k fr_xy0123 R 0:02 2 n[4101-4102]
	</syntaxhighlight>		</syntaxhighlight>

M Janczyk: /* Partitions */

2026-02-02T16:45:06Z

Partitions

← Older revision		Revision as of 18:45, 2 February 2026
Line 110:		Line 110:
	{\| class="wikitable"		{\| class="wikitable"
	\|-		\|-
	! style="width:7%"\| Partition		! style="width:5%"\| Partition
	! style="width:8%"\| Multi Node		! style="width:10%"\| Multi Node
	! style="width:18%"\| Nodes		! style="width:20%"\| Nodes
	! style="width:12%"\| APU/GPU		! style="width:20%"\| APU/GPU
	! style="width:5%"\| Cores per APU/GPU		! style="width:10%"\| Cores per APU/GPU
	! style="width:20%"\| Defaults		! style="width:20%"\| Defaults
	! style="width:20%"\| Limits		! style="width:15%"\| Limits
	\|-		\|-
	\| cpu*		\| cpu*

M Janczyk at 16:39, 2 February 2026

2026-02-02T16:39:32Z

← Older revision		Revision as of 18:39, 2 February 2026
Line 100:		Line 100:
	On bwForCluster NEMO 2 it is '''optional''' to request a partition with <code>--partition=<partition_name></code> on job submission. Within a partition job allocations are routed automatically to the most suitable compute node(s) for the requested resources (e.g. amount of nodes and cores, memory, number of GPUs). The 'cpu' partition is the default partition, if no partition is requested. It will start jobs either on 'milan' or 'genoa' nodes.		On bwForCluster NEMO 2 it is '''optional''' to request a partition with <code>--partition=<partition_name></code> on job submission. Within a partition job allocations are routed automatically to the most suitable compute node(s) for the requested resources (e.g. amount of nodes and cores, memory, number of GPUs). The 'cpu' partition is the default partition, if no partition is requested. It will start jobs either on 'milan' or 'genoa' nodes.

	The partitions 'cpu', 'l40s', 'mi300a' and '~~h200~~' only allow the use of single nodes. You cannot start jobs that use more than one node, either by specifying <code>--nodes</code> (>1) or <code>--ntasks</code> (more than available on one node). If you do so, your jobs will be blocked with the reason (PartitionNodeLimit).		The partitions 'cpu', 'l40s', 'mi300a', 'h200' and 'rtx' only allow the use of single nodes. You cannot start jobs that use more than one node, either by specifying <code>--nodes</code> (>1) or <code>--ntasks</code> (more than available on one node). If you do so, your jobs will be blocked with the reason (PartitionNodeLimit).

	All partitions ~~cpu, milan, genoa, l40s, mi300a and h200~~ are operated in shared mode, i.e. jobs from different users can run on the same node. Jobs can get exclusive access to compute nodes in these partitions with the <code>--exclusive</code> option.		'''All''' partitions are operated in shared mode, i.e. jobs from different users can run on the same node. Jobs can get exclusive access to compute nodes in these partitions with the <code>--exclusive</code> option.

	The login partition is intended for quick tests on the login lodes with the Slurm environment.		The login partition is intended for quick tests on the login lodes with the Slurm environment.
Line 165:		Line 165:
	\| ntasks=1, time=01:00:00, mem-per-cpu=1gb		\| ntasks=1, time=01:00:00, mem-per-cpu=1gb
	\| time=48:00:00, nodes=1, gpus=8		\| time=48:00:00, nodes=1, gpus=8
			\|-
			\| rtx
			\| single node
			\| 2x AMD EPYC 9655 (Turin)
			\| 8x 4x NVIDIA RTX PRO 6000 Blackwell Server Edition
			\| 47
			\| ntasks=1, time=01:00:00, mem-per-cpu=1gb
			\| time=48:00:00, nodes=1, gpus=4
	\|-		\|-
	\| login		\| login

← Older revision		Revision as of 18:45, 2 February 2026
Line 110:		Line 110:
	{\| class="wikitable"		{\| class="wikitable"
	\|-		\|-
	! style="width:7%"\| Partition		! style="width:5%"\| Partition
	! style="width:8%"\| Multi Node		! style="width:10%"\| Multi Node
	! style="width:18%"\| Nodes		! style="width:20%"\| Nodes
	! style="width:12%"\| APU/GPU		! style="width:20%"\| APU/GPU
	! style="width:5%"\| Cores per APU/GPU		! style="width:10%"\| Cores per APU/GPU
	! style="width:20%"\| Defaults		! style="width:20%"\| Defaults
	! style="width:20%"\| Limits		! style="width:15%"\| Limits
	\|-		\|-
	\| cpu*		\| cpu*