Energy Efficient Cluster Usage - Revision history

A Flachmueller: Dos and Don'ts von Running Calculations hierher geschoben - Abgleich und Integration mit "Things to avoid" folgt noch

2026-10-23T15:55:22Z

Dos and Don'ts von Running Calculations hierher geschoben - Abgleich und Integration mit "Things to avoid" folgt noch

@@ Line 57: / Line 57: @@
 = 10.5 minutes job runtime with just the overhead of creating a single job and 16 processes without switching between processes.
 ** &rarr;  Simple parallelization by hand is advisable. See: A basic introduction to [[Development/Parallel_Programming | Parallel Programming]].
 == Fair Share and Scheduler ==

H Schumacher: Added job monitoring link

2026-09-21T13:15:14Z

Added job monitoring link

← Older revision		Revision as of 15:15, 21 September 2026
Line 27:		Line 27:
	== Recommendations ==		== Recommendations ==

	* Use the '''job monitoring''' options of your respective cluster to see the job efficiency and ways to improve it.		* Use the '''[[Job_Monitoring \| job monitoring]]''' options of your respective cluster to see the job efficiency and ways to improve it.
	* '''Test''' new setups first before submitting a lot of similar jobs or resource demanding jobs.		* '''Test''' new setups first before submitting a lot of similar jobs or resource demanding jobs.
	** Run a single job first before sending many.		** Run a single job first before sending many.

H Schumacher: Restructured content and added more advice

2026-07-14T07:51:47Z

Restructured content and added more advice

Show changes

K Siegmund: /* How many and which kind of hardware resources do I require for it */

2026-06-16T10:07:25Z

How many and which kind of hardware resources do I require for it

← Older revision		Revision as of 12:07, 16 June 2026
Line 34:		Line 34:
	As this is dependent on both the job as well as the specific cluster hardware and architecture available.		As this is dependent on both the job as well as the specific cluster hardware and architecture available.

		⚫	'''A small number of jobs and few resources'''
	'''
⚫	A small number of jobs and few resources'''

	→ Submit to the scheduler. No extended testing and resource scaling analysis are needed.		→ Submit to the scheduler. No extended testing and resource scaling analysis are needed.

		⚫	'''Medium-sized projects'''
	'''
⚫	Medium-sized projects'''

	→ Run only necessary jobs: Please consider testing new setups and their output for validity prior to submitting a huge amount of similar jobs		→ Run only necessary jobs: Please consider testing new setups and their output for validity prior to submitting a huge amount of similar jobs

S Fischer: /* Summary: General Recommendations */

2026-05-29T09:10:12Z

Summary: General Recommendations

← Older revision		Revision as of 11:10, 29 May 2026
Line 81:		Line 81:
	= Summary: General Recommendations =		= Summary: General Recommendations =

	* Choose the most '''efficient algorithms''' for the given problem		* Choose the most '''efficient algorithms''' for the given problem.
	* Run only '''necessary''' jobs: Please consider testing new setups and their output for validity prior to submitting a huge amount of similar jobs		* Run only '''necessary''' jobs: Please consider testing new setups and their output for validity prior to submitting a huge amount of similar jobs.
	* Start '''small''': Run ~~Your~~ problem on a small number of parallel entities (be it processes or threads) first.		* Start '''small''': Run your problem on a small number of parallel entities (be it processes or threads) first.
	* '''Estimate''' the runtime of the parallel job as '''exactly''' as possible to increase the efficiency of the scheduling of the whole system		* '''Estimate''' the runtime of the parallel job as '''exactly''' as possible to increase the efficiency of the scheduling of the whole system.
	* Use the proper tools for development: If ~~You~~ develop your own code, please use the proper tools for debugging and parallel performance analysis. More information is available on the bwHPC Wiki.		* Use the proper tools for development: If you develop your own code, please use the proper tools for debugging and parallel performance analysis. More information is available on the bwHPC Wiki.
	* A look at the '''job feedback''' can help you determine if you are using the cluster efficiently		* A look at the '''job feedback''' can help you determine if you are using the cluster efficiently.

A Flachmueller: /* What do I want to do and why do I need an HPC Cluster for it? */

2026-04-22T10:35:53Z

What do I want to do and why do I need an HPC Cluster for it?

← Older revision		Revision as of 12:35, 22 April 2026
Line 24:		Line 24:
	Please consider testing new setups and their output for validity prior to submitting jobs that require lots of resources. This also includes projects where a lot of (smaller) similar jobs are submitted.		Please consider testing new setups and their output for validity prior to submitting jobs that require lots of resources. This also includes projects where a lot of (smaller) similar jobs are submitted.

	Make sure to double-check your jobs prior to the submission, having to discard the output data of an HPC project due to faulty input files is wasting a lot of computational resources.		Make sure to double-check your jobs prior to the submission, as having to discard the output data of an HPC project due to faulty input files is wasting a lot of computational resources.

⚫	Finally, identifying the specific resource requirements for a given job is important to allocate the optimal your compute job, and to decide if an HPC cluster is needed at all.

		⚫	Finally, identifying the specific resource requirements for a given job is important to allocate the optimal amount for your compute job, and to decide if an HPC cluster is needed at all.

	= How many and which kind of hardware resources do I require for it =		= How many and which kind of hardware resources do I require for it =

A Flachmueller: /* Introduction */

2026-04-22T10:30:03Z

Introduction

← Older revision		Revision as of 12:30, 22 April 2026
Line 6:		Line 6:
	Assuming that a typical bwHPC cluster has a few hundred compute nodes, this amounts to the energy consumption of a village for each cluster.		Assuming that a typical bwHPC cluster has a few hundred compute nodes, this amounts to the energy consumption of a village for each cluster.

	Although a large amount of this energy consumption is an intrinsic requirement of running large HPC clusters (even when ~~ist~~ processors are idle, a cluster uses a lot of energy), efficient use of the available resources is important. Using as many resources as possible does not make a power user. Using them wisely does.		Although a large amount of this energy consumption is an intrinsic requirement of running large HPC clusters (even when it's processors are idle, a cluster uses a lot of energy), efficient use of the available resources is important. Using as many resources as possible does not make a power user. Using them wisely does.
	In the following, a basic introduction to some of the most important aspects of energy-efficient HPC usage from a user perspective is given.		In the following, a basic introduction to some of the most important aspects of energy-efficient HPC usage from a user perspective is given.

H Schumacher: fixed link

2026-04-13T08:20:19Z

fixed link

← Older revision		Revision as of 10:20, 13 April 2026
Line 63:		Line 63:
	→ Handling via the scheduler is inefficient.		→ Handling via the scheduler is inefficient.

	→ Simple parallelization by hand is advisable. See: A basic introduction to [[Parallel Programming]].		→ Simple parallelization by hand is advisable. See: A basic introduction to [[Development/Parallel_Programming \| Parallel Programming]].

	= How do I optimize my code to use these resources most efficiently? =		= How do I optimize my code to use these resources most efficiently? =

J Steuer: /* How many and which kind of hardware resources do I require for it */

2023-08-21T11:58:23Z

How many and which kind of hardware resources do I require for it

← Older revision		Revision as of 13:58, 21 August 2023
Line 35:		Line 35:
	As this is dependent on both the job as well as the specific cluster hardware and architecture available.		As this is dependent on both the job as well as the specific cluster hardware and architecture available.

			'''

	A small number of jobs and few resources		A small number of jobs and few resources'''

	→ Submit to the scheduler. No extended testing and resource scaling analysis are needed.		→ Submit to the scheduler. No extended testing and resource scaling analysis are needed.

			'''

	Medium-sized projects		Medium-sized projects'''

	→ Run only necessary jobs: Please consider testing new setups and their output for validity prior to submitting a huge amount of similar jobs		→ Run only necessary jobs: Please consider testing new setups and their output for validity prior to submitting a huge amount of similar jobs
Line 52:		Line 52:


	Large projects		'''Large projects'''

	→ Same approach as for medium-sized projects.		→ Same approach as for medium-sized projects.
Line 59:		Line 59:


	Many short jobs		'''Many short jobs'''

	→ Handling via the scheduler is inefficient.		→ Handling via the scheduler is inefficient.

J Steuer: /* How many and which kind of hardware resources do I require for it */

2023-08-21T11:57:58Z

How many and which kind of hardware resources do I require for it

← Older revision		Revision as of 13:57, 21 August 2023
Line 34:		Line 34:
	Resource allocation is a crucial part when working on an HPC cluster.		Resource allocation is a crucial part when working on an HPC cluster.
	As this is dependent on both the job as well as the specific cluster hardware and architecture available.		As this is dependent on both the job as well as the specific cluster hardware and architecture available.


	A small number of jobs and few resources		A small number of jobs and few resources

	* Submit to the scheduler. No extended testing and resource scaling analysis are needed.		→ Submit to the scheduler. No extended testing and resource scaling analysis are needed.


	Medium-sized projects		Medium-sized projects

	* Run only necessary jobs: Please consider testing new setups and their output for validity prior to submitting a huge amount of similar jobs		→ Run only necessary jobs: Please consider testing new setups and their output for validity prior to submitting a huge amount of similar jobs
⚫	* Start small: Run your problem on a small set of resources first.

⚫	* Use the proper tools for development: If you develop your own code, please use the proper tools for debugging and parallel performance analysis. See: [[Development#Documentation_in_the_Wiki\|Development]].
		⚫	→ Start small: Run your problem on a small set of resources first.
⚫	* A look at the job feedback can help you determine if you are using the cluster efficiently

		⚫	→ Use the proper tools for development: If you develop your own code, please use the proper tools for debugging and parallel performance analysis. See: [[Development#Documentation_in_the_Wiki\|Development]].

		⚫	→ A look at the job feedback can help you determine if you are using the cluster efficiently


	Large projects		Large projects

	* Same approach as for medium-sized projects.		→ Same approach as for medium-sized projects.
⚫	* Run a scaling analysis for your project with regard to how many resources work best. See: [[Scaling]].

		⚫	→ Run a scaling analysis for your project with regard to how many resources work best. See: [[Scaling]].


	Many short jobs		Many short jobs

	* Handling via the scheduler is inefficient.		→ Handling via the scheduler is inefficient.
⚫	* Simple parallelization by hand is advisable. See: A basic introduction to [[Parallel Programming]].

		⚫	→ Simple parallelization by hand is advisable. See: A basic introduction to [[Parallel Programming]].

	= How do I optimize my code to use these resources most efficiently? =		= How do I optimize my code to use these resources most efficiently? =