Energy Efficient Cluster Usage

Poor job efficiency means that hardware resources are wasted and a similar overall result could have been achieved using fewer hardware resources, leaving those for other jobs and reducing the queue wait time for all users. Efficient cluster usage means choosing optimal values for job resources:

• CPU cores
• GPUs
• Memory
• (temporary) storage
• Time

Motivation

User perspective

Short waiting times including short cycles of trial and error and fast results. The lower the job efficiency the longer the waiting times relative to the shortest possible time.

Energy perspective

Energy consumption of data centers has been increasing continuously throughout the last decade. In 2020, the energy consumption of all data centers in Germany amounted to around 3 percent of the total electricity produced. Accompanying this large energy consumption are large-scale emissions of CO2 to the atmosphere and thus significant contributions to climate change. To illustrate this, an average compute job running on a single node for one day may easily consume 10 kWh or even more. That translates roughly to brewing 700 cups of coffee. Assuming that a typical bwHPC cluster has a few hundred compute nodes, this amounts to the energy consumption of a village for each cluster. Although a large amount of this energy consumption is an intrinsic requirement of running large HPC clusters (even when it's processors are idle, a cluster uses a lot of energy), efficient use of the available resources is important.

Something to keep in mind:

‎

Using as many resources as possible does not make a power user. Using them wisely does.‎

Recommendations

• Use the job monitoring options of your respective cluster to see the job efficiency and ways to improve it.
• Test new setups first before submitting a lot of similar jobs or resource demanding jobs.
  • Run a single job first before sending many.
  • Run a simplified problem on a small number of parallel entities (be it processes or threads) first before requesting many resources.
  • Run a time intensive task with a short runtime first to see if it already fails in the first minutes.
• Follow the best practices of your chosen tool, software or programming language and choose the most efficient algorithms for the given problem. There are several places where you might find related documentation:
  • Software Module help.
  • Software page of your cluster.
  • Pages linked at the Development page. For example information about debugging, performance analysis, specific programming languages etc. . → Use an efficient programming language such as Rust, C, and C++ -- well any compiled language. Do not use any interpreted language like Perl or Python. Since Machine Learning is a hot topic, this deserves a few words: Any ML-Python code using Tensorflow or other libraries will make heavy usage of NumPy and other math packages, which will use C-based implementations. Please make sure, you use the provided Python modules, which are optimized to use Intel MKL and other mathematical libraries.

Further reading: Rui Pereira, et al: "Energy efficiency across programming languages: how do energy, time, and memory relate?", SLE 2017: Proc. of the 10th ACM SIGPLAN Int. Conf. on SW Language Eng., Oct. 2017, pp. 256–267, doi:10.1145/3136014.3136031

• If you have a task that can be scaled up, please consider a scaling analysis.
• Analyse memory access patterns: For small tight loops checking for locks, use the pause instruction.

Things to avoid

• Poor choice of resources compared to the size of the nodes leaves part of the node blocked, but doing nothing:
  • Too much (un-needed) memory or disk space requested
• Many small jobs with a short runtime (seconds in extreme cases). A job should run at least 10 minutes.
• When a node has 256 gb memory but only 236 gb are usable (see Hardware table of your cluster) then requesting 256 gb memory needs two nodes.
• GPU: Do not ask for a specific gpu type when your code would be fitting for any type.
• More cores used for a single mpi/openmp parallel computation than useful.
• Writing temporary files to global filesystems when a ram disk or local disk can be used.
• Do not use more parallel processes than there are actual, reasonable parallel tasks. Examples:
  • 4 datasets with values for the same 1000 genes each. Method xx shall be used per gene. This operation needs 10 seconds per gene.
    • Case 1: Provide 64 single threaded cores within one job and use one process/thread per gene: This creates a large overhead as only 64 processes can run in parallel while all 5000 processes try to get computing time so that a process might run for 5 seconds and then makes room for another process that also isn't able to finish because another process comes in between.
    • Case 2: Send 1 job per gene so that each job uses method xx on this gene in all 4 datasets: This creates a large overhead as each job only needs 50 seconds while there is a thousand times the overhead of creating and finishing a job.
    • Case 3: Provide 64 cores in one job. Each dataset uses 16 cores. Each core needs to process 1000/16 =~ 63 genes. This needs 63*10 seconds

= 10.5 minutes job runtime with just the overhead of creating a single job and 16 processes without switching between processes.

• • → Simple parallelization by hand is advisable. See: A basic introduction to Parallel Programming.

Dos and Don'ts

There are a number of pitfalls users might encounter when starting to use compute clusters. To avoid some of them, we've collected some best practices in the form of a Dos and Dont's list. It is meant to serve as guidance, not as an absolute imperative, because sometimes stricly avoiding every undesired behaviour is not feasible, though discussing those cases with the cluster operators beforehand is a good idea.

Dos: Bundle large amounts of tiny calculations (e.g. 1000 short running analyses taking 2 minutes each) together into fewer / a single job(s) that orchestrates them to run consecutively (and maybe even distributes them among different cores) Estimate the total calculation time of your job. A short test run (either on your local machine or on the cluster) can help, though transferability and extrapolation lead to inaccuracies. → Request a calculation time larger than your estimate. For a new calculation about 20%-30% more time than estimated often is a feasable starting point. Good time estimates are below 10% of extra time requested. → Having the requested time be close to the actual calculation time is beneficial for the cluster because it helps the scheduler to be more precise and thus shortens idle times. Plan early what calculations you want to perform and create a rough timetable for when you want to submit them. Keep cluster waiting times in mind and ideally have some buffer before important deadlines (e.g. paper submission deadlines). → The scheduler does not know of your urgency and you have no way of communicating your urgency to the scheduler. Therefore the bwHPC recommends to avoid generating the need for urgent calculations. We however also know that sometimes urgency can not be avoided. In such cases contact the cluster operators to see if they can help you out. Don't ask for this lightly, because if everything is urgent, nothing is! → Many things have a tendency to avoid obliging to plans. Sudden problems can arise both on the user end and on the cluster end, so having a personal time buffer for such cases is good for anxiety prevention. Be mindful when performing very long calculations. For a multitude of reasons a computation job may suddenly end during the run. The longer the calculation the higher the risk of an unexpected error. → If possible create checkpoint files regularly during your calculations, from which you can pick up the calculation again. Keep communication bottlenecks in mind when requesting resources. → With greater physical distance between compute units (cores) comes a longer communication time between them. This leads to efficiency losses depending on the communication volume and frequency.

Don'ts: Do not run very short jobs (single digit minutes) on regular compute nodes. Some clusters offer separate partitions for short running jobs. → For the sake of efficiency small calculations should use few cores and accept runtimes larger than 1 hour. → If a parallelizable compute job calculates in under 1 hour on 1 core you should check your reasoning why it needs to run on the cluster at all instead of your local machine. Most (if not all) modern consumer machines do have multiple cores available and often have higher clock speeds than cluster CPUs. The true power of modern compute clusters comes from splitting large parallelizable calculations over many compute cores, usually at the cost of somewhat lower core clock speeds. Do not unnecessarily split your calculation into many tiny short running jobs to "trick" the backfilling system by using every 5 minute idle gap that may occur. → Allocation of a job onto the compute nodes takes up some time for every job, as does cleanup afterwards. Thus by splitting a calculation its overall efficiency decreases accordingly. → The scheduler assigns and updates priorities and starting times of jobs frequently. A higher number of jobs in the queue means a higher workload for the scheduler, leading to unnecessary strain of the system. Do not request resources that remain unused by your job. → The Resource Management System calculates your recent usage by multiplying the number of requested compute resources (CPU cores / GPUs) by the time the job was allocated. So if 90% of the cores are idle most of the time or not used at all doesn't matter to the system, it will always use 100% of the requested compute resources for the calculation of the Fair Share Value. For the time it however uses the actual allocated time of the jobs instead of what was requested. → Requesting lots of compute resources for a job usually leads to longer waiting times in the queue compared to less resource intensive jobs. Do not spawn too many threads with your jobs. Although multithreading allows some code to run overall more efficient due to better resource usage, spawning too many threads on a compute unit has the contrary effect because the threads have to be managed. → Be mindful when increasing the thread count and analyse your actual performance gain.

Fair Share and Scheduler

Influence of Efficiency on Fair Share and Scheduler:

• Fair Share value:

  The fair share value represents your priority on the cluster. On a busy cluster a low priority leads to longer waiting times as other users with a higher priority have a higher chance of getting free resource slots. The fair share value declines when a lot of cluster resources were used recently. Used is defined as follows:

  • Requested resources: 4 gpus for 2 hours
  • Actual resource usage: 1 gpu and job finishes successfully after 1 hour.
  • Used resources as considered by the fair share value: 4 gpus for 1 hour because the three idle gpus couldn't be used by any other cluster used as they were blocked by this job.

• Job scheduler:

  The longer the requested runtime and the more resources are needed in that time the harder it is to find a fitting timeframe where these resources are free.