https://wiki.bwhpc.de/wiki/api.php?action=feedcontributions&feedformat=atom&user=P+Weisbrod
bwHPC Wiki - User contributions [en]
2026-05-06T08:02:21Z
User contributions
MediaWiki 1.39.17
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster3.0/Running_Jobs&diff=15166
BwUniCluster3.0/Running Jobs
2025-07-22T12:33:23Z
<p>P Weisbrod: </p>
<hr />
<div><br />
= Purpose and function of a queuing system =<br />
<br />
All compute activities on bwUniCluster 3.0 have to be performed on the compute nodes. Compute nodes are only available by requesting the corresponding resources via the queuing system. As soon as the requested resources are available, automated tasks are executed via a batch script or they can be accessed interactively.<br><br />
General procedure: Hint to [[Running_Calculations | Running Calculations]]<br />
<br />
== Job submission process ==<br />
<br />
bwUniCluster 3.0 has installed the workload managing software Slurm. Therefore any job submission by the user is to be executed by commands of the Slurm software. Slurm queues and runs user jobs based on fair sharing policies.<br />
<br />
== Slurm ==<br />
<br />
HPC Workload Manager on bwUniCluster 3.0 is Slurm.<br />
Slurm is a cluster management and job scheduling system. Slurm has three key functions. <br />
* It allocates access to resources (compute cores on nodes) to users for some duration of time so they can perform work. <br />
* It provides a framework for starting, executing, and monitoring work (normally a parallel job) on the set of allocated nodes. <br />
* It arbitrates contention for resources by managing a queue of pending work.<br />
<br />
Any kind of calculation on the compute nodes of bwUniCluster 3.0 requires the user to define calculations as a sequence of commands together with required run time, number of CPU cores and main memory and submit all, i.e., the '''batch job''', to a resource and workload managing software.<br />
<br />
== Terms and definitions ==<br />
<br />
<br />
''' Partitions '''<br />
<br />
Slurm manages job queues for different '''partitions'''. Partitions are used to group similar node types (e.g. nodes with and without accelerators) and to enforce different access policies and resource limits.<br />
<br />
On bwUniCluster 3.0 there are different partitions:<br />
<br />
* CPU-only nodes<br />
** 2-socket nodes, consisting of 2 Intel Ice Lake processors with 32 cores each or 2 AMD processors with 48 cores each<br />
** 2-socket nodes with very high RAM capacity, consisting of 2 AMD processors with 48 cores each<br />
* GPU-accelerated nodes<br />
** 2-socket nodes with 4x NVIDIA A100 or 4x NVIDIA H100 GPUs<br />
** 4-socket node with 4x AMD Instinct accelerator<br />
<br />
''' Queues '''<br />
<br />
Job '''queues''' are used to manage jobs that request access to shared but limited computing resources of a certain kind (partition).<br />
<br />
On bwUniCluster 3.0 there are different main types of queues:<br />
* Regular queues<br />
** cpu: Jobs that request CPU-only nodes.<br />
** gpu: Jobs that request GPU-accelerated nodes.<br />
* Development queues (dev)<br />
** Short, usually interactive jobs that are used for developing, compiling and testing code and workflows. The intention behind development queues is to provide users with immediate access to computer resources without having to wait. This is the place to realize instantaneous heavy compute without affecting other users, as would be the case on the login nodes.<br />
<br />
Requested compute resources such as (wall-)time, number of nodes and amount of memory are restricted and must fit into the boundaries imposed by the queues. The request for compute resources on the bwUniCluster 3.0 <font color=red>requires at least the specification of the '''queue''' and the '''time'''</font>.<br />
<br />
''' Jobs '''<br />
<br />
Jobs can be run non-interactively as '''batch jobs''' or as '''interactive jobs'''.<br><br />
Submitting a batch job means, that all steps of a compute project are defined in a Bash script. This Bash script is queued and executed as soon as the compute resources are available and allocated. Jobs are enqueued with the <code>sbatch</code> command.<br />
For interactive jobs, the resources are requested with the <code>salloc</code> command. As soon as the computing resources are available and allocated, a command line prompt is returned on a computing node and the user can freely dispose of the resources now available to him.<br />
{|style="background:#deffee; width:100%;"<br />
|style="padding:5px; background:#cef2e0; text-align:left"|<br />
[[Image:Attention.svg|center|25px]]<br />
|style="padding:5px; background:#cef2e0; text-align:left"|<br />
'''Please remember:'''<br />
* '''Heavy computations are not allowed on the login nodes'''.<br>Use a developement or a regular job queue instead! Please refer to [[BwUniCluster3.0/Login#Allowed_Activities_on_Login_Nodes|Allowed Activities on Login Nodes]].<br />
* '''Development queues''' are meant for '''development tasks'''.<br>Do not misuse this queue for regular, short-running jobs or chain jobs! Only one running job at a time is enabled. Maximum queue length is reduced to 3.<br />
|}<br />
<br />
= Queues on bwUniCluster 3.0 = <br />
== Policy ==<br />
<br />
The computing time is provided in accordance with the '''fair share policy'''. The individual investment shares of the respective university and the resources already used by its members are taken into account. Furthermore, the following throttling policy is also active: The '''maximum amount of physical cores''' used at any given time from jobs running is '''1920 per user''' (aggregated over all running jobs). This number corresponds to 30 nodes on the Ice Lake partition or 20 nodes on the standard partition. The aim is to minimize waiting times and maximize the number of users who can access computing time at the same time.<br />
<br />
== Regular Queues ==<br />
{| class="wikitable"<br />
|- <br />
! style="width:5%"| Queue<br />
! style="width:13%"| Node-Type<br />
! style="width:23%"| Default Resources<br />
! style="width:13%"| Minimal Resources<br />
! style="width:13%"| Maximum Resources<br />
|-<br />
| <code>cpu_il</code><br />
| CPU nodes<br/>Ice Lake<br />
| mem-per-cpu=2000mb<br />
| <br />
| time=72:00:00, nodes=30, mem=249600mb, ntasks-per-node=64, (threads-per-core=2) <br />
|-<br />
| <code>cpu</code><br />
| CPU nodes<br/>Standard<br />
| mem-per-cpu=2000mb<br />
| <br />
| time=72:00:00, nodes=20, mem=380000mb, ntasks-per-node=96, (threads-per-core=2)<br />
|-<br />
| <code>highmem</code><br />
| CPU nodes<br/>High Memory<br />
| mem-per-cpu=12090mb<br />
| mem=380001mb<br />
| time=72:00:00, nodes=4, mem=2300000mb, ntasks-per-node=96, (threads-per-core=2)<br />
|-<br />
| <code>gpu_h100</code><br />
| GPU nodes<br/>NVIDIA GPU x4<br />
| mem-per-gpu=193300mb<br/>cpus-per-gpu=24<br />
| gres=gpu:1<br />
| time=72:00:00, nodes=12, mem=760000mb, ntasks-per-node=96, (threads-per-core=2)<br />
|-<br />
| <code>gpu_mi300</code><br />
| GPU node<br/>AMD GPU x4<br />
| mem-per-gpu=128200mb<br/>cpus-per-gpu=24<br />
| gres=gpu:1<br />
| time=72:00:00, nodes=1, mem=510000mb, ntasks-per-node=40, (threads-per-core=2)<br />
|-<br />
| <code>gpu_a100_il</code>/<code>gpu_h100_il</code><br />
| GPU nodes<br/>Ice Lake<br/>NVIDIA GPU x4<br />
| mem-per-gpu=127500mb<br/>cpus-per-gpu=16<br />
| gres=gpu:1<br />
| time=48:00:00, nodes=9, mem=510000mb, ntasks-per-node=64, (threads-per-core=2) <br />
|}<br />
Table 1: Regular Queues<br />
<br />
== Short Queues ==<br />
Queues with a short runtime of 30 minutes.<br />
<br />
{| class="wikitable"<br />
|- <br />
! style="width:5%"| Queue<br />
! style="width:13%"| Node Type<br />
! style="width:23%"| Default Resources<br />
! style="width:13%"| Minimal Resources<br />
! style="width:13%"| Maximum Resources<br />
|-<br />
| <code>gpu_a100_short</code><br />
| GPU nodes<br/>Ice Lake<br/>NVIDIA GPU x4<br />
| mem-per-gpu=94000mb<br/>cpus-per-gpu=12<br />
| gres=gpu:1<br />
| time=30, nodes=12, mem=376000mb, ntasks-per-node=48, (threads-per-core=2)<br />
|}<br />
Table 2: Short Queues<br />
<br />
== Development Queues ==<br />
Only for development, i.e. debugging or performance optimization ...<br />
<br />
{| class="wikitable"<br />
|- <br />
! style="width:5%"| Queue<br />
! style="width:13%"| Node Type<br />
! style="width:23%"| Default Resources<br />
! style="width:13%"| Minimal Resources<br />
! style="width:13%"| Maximum Resources<br />
|-<br />
| <code>dev_cpu_il</code><br />
| CPU nodes<br/>Ice Lake<br />
| mem-per-cpu=2000mb<br />
| <br />
| time=30, nodes=8, mem=249600mb, ntasks-per-node=64, (threads-per-core=2)<br />
|-<br />
| <code>dev_cpu</code><br />
| CPU nodes<br/>Standard<br />
| mem-per-cpu=2000mb<br />
| <br />
| time=30, nodes=1, mem=380000mb, ntasks-per-node=96, (threads-per-core=2)<br />
|-<br />
| <code>dev_gpu_h100</code><br />
| GPU nodes<br/>NVIDIA GPU x4<br />
| mem-per-gpu=193300mb<br/>cpus-per-gpu=24<br />
| gres=gpu:1<br />
| time=30, nodes=1, mem=760000mb, ntasks-per-node=96, (threads-per-core=2)<br />
|-<br />
| <code>dev_gpu_a100_il</code><br />
| GPU nodes<br/>NVIDIA GPU x4<br/><br />
| mem-per-gpu=127500mb<br/>cpus-per-gpu=16 <br />
| gres=gpu:1<br />
| time=30, nodes=1, mem=510000mb, ntasks-per-node=64, (threads-per-core=2) <br />
|}<br />
Table 3: Development Queues<br />
<br />
<br />
Default resources of a queue class define number of tasks and memory if not explicitly given with sbatch command. Resource list acronyms ''--time'', ''--ntasks'', ''--nodes'', ''--mem'' and ''--mem-per-cpu'' are described [[BwUniCluster3.0/Running_Jobs/Slurm|here]].<br />
<br />
== Check available resources: sinfo_t_idle ==<br />
The Slurm command sinfo is used to view partition and node information for a system running Slurm. It incorporates down time, reservations, and node state information in determining the available backfill window. The sinfo command can only be used by the administrator.<br />
<br><br />
SCC has prepared a special script (sinfo_t_idle) to find out how many processors are available for immediate use on the system. It is anticipated that users will use this information to submit jobs that meet these criteria and thus obtain quick job turnaround times. <br />
<br><br />
<br><br />
<br />
* The following command displays what resources are available for immediate use for the whole partition.<br />
<pre>$ sinfo_t_idle <br />
Partition dev_cpu : 1 nodes idle<br />
Partition cpu : 1 nodes idle<br />
Partition highmem : 2 nodes idle<br />
Partition dev_gpu_h100 : 0 nodes idle<br />
Partition gpu_h100 : 0 nodes idle<br />
Partition gpu_mi300 : 0 nodes idle<br />
Partition dev_cpu_il : 7 nodes idle<br />
Partition cpu_il : 2 nodes idle<br />
Partition dev_gpu_a100_il : 1 nodes idle<br />
Partition gpu_a100_il : 0 nodes idle<br />
Partition gpu_h100_il : 1 nodes idle<br />
Partition gpu_a100_short : 0 nodes idle<br />
</pre><br />
<br><br />
<br />
= Running Jobs =<br />
<br />
== Slurm Commands (excerpt) ==<br />
Important Slurm commands for non-administrators working on bwUniCluster 3.0.<br />
{| width=850px class="wikitable"<br />
! Slurm commands !! Brief explanation<br />
|-<br />
| [[#Batch Jobs: sbatch|sbatch]] || Submits a job and puts it into the queue [[https://slurm.schedmd.com/sbatch.html sbatch]] <br />
|-<br />
| [[#Interactive Jobs: salloc|salloc]] || Requests resources for an interactive Job [[https://slurm.schedmd.com/salloc.html salloc]]<br />
|-<br />
| [[#Monitor and manage jobs |scontrol show job]] || Displays detailed job state information [[https://slurm.schedmd.com/scontrol.html scontrol]]<br />
|-<br />
| [[#List of your submitted jobs : squeue|squeue]] || Displays information about active, eligible, blocked, and/or recently completed jobs [[https://slurm.schedmd.com/squeue.html squeue]]<br />
|-<br />
| [[#List of your submitted jobs : squeue|squeue --start]] || Returns start time of submitted job [[https://slurm.schedmd.com/squeue.html squeue]]<br />
|-<br />
| [[#Check available resources: sinfo_t_idle|sinfo_t_idle]] || Shows what resources are available for immediate use [[https://slurm.schedmd.com/sinfo.html sinfo]]<br />
|-<br />
| [[#Canceling own jobs : scancel|scancel]] || Cancels a job [[https://slurm.schedmd.com/scancel.html scancel]]<br />
|}<br />
<br />
<br><br />
* [https://slurm.schedmd.com/tutorials.html Slurm Tutorials]<br />
* [https://slurm.schedmd.com/pdfs/summary.pdf Slurm command/option summary (2 pages)]<br />
* [https://slurm.schedmd.com/man_index.html Slurm Commands]<br />
<br><br />
<br />
== Batch Jobs: sbatch ==<br />
<br />
Batch jobs are submitted by using the command '''sbatch'''. The main purpose of the '''sbatch''' command is to specify the resources that are needed to run the job. '''sbatch''' will then queue the batch job. However, starting of batch job depends on the availability of the requested resources and the fair sharing value.<br />
<br><br />
<br><br />
<br />
* The syntax and use of '''sbatch''' can be displayed via:<br />
<pre><br />
$ man sbatch<br />
</pre><br />
<br />
'''sbatch''' options can be used from the command line or in your job script. Different defaults for some of these options are set based on the queue and can be found [[BwUniCluster3.0/Slurm | here]]<br />
<br />
{| class="wikitable"<br />
! colspan="3" | sbatch Options<br />
|-<br />
! style="width:8%"| Command line<br />
! style="width:9%"| Script<br />
! style="width:13%"| Purpose<br />
|- style="vertical-align:top;"<br />
| -t, --time=''time''<br />
| #SBATCH --time=''time''<br />
| Wall clock time limit.<br><br />
|-<br />
|- style="vertical-align:top;"<br />
| -N, --nodes=''count''<br />
| #SBATCH --nodes=''count''<br />
| Number of nodes to be used.<br />
|- style="vertical-align:top;"<br />
| -n, --ntasks=''count''<br />
| #SBATCH --ntasks=''count''<br />
| Number of tasks to be launched.<br />
|-<br />
|- style="vertical-align:top;"<br />
| --ntasks-per-node=''count''<br />
| #SBATCH --ntasks-per-node=''count''<br />
| Maximum count of tasks per node.<br />
|-<br />
|- style="vertical-align:top;"<br />
| -c, --cpus-per-task=''count''<br />
| #SBATCH --cpus-per-task=''count''<br />
| Number of CPUs required per (MPI-)task.<br />
|-<br />
|- style="vertical-align:top;"<br />
| --mem=''value_in_MB''<br />
| #SBATCH --mem=''value_in_MB'' <br />
| Memory in MegaByte per node. (You should omit the setting of this option.)<br />
|-<br />
|- style="vertical-align:top;"<br />
| --mem-per-cpu=''value_in_MB''<br />
| #SBATCH --mem-per-cpu=''value_in_MB'' <br />
| Minimum Memory required per allocated CPU. (You should omit the setting of this option.)<br />
|-<br />
|- style="vertical-align:top;"<br />
| --mail-type=''type''<br />
| #SBATCH --mail-type=''type''<br />
| Notify user by email when certain event types occur.<br>Valid type values are NONE, BEGIN, END, FAIL, REQUEUE, ALL.<br />
|-<br />
|- style="vertical-align:top;"<br />
| --mail-user=''mail-address''<br />
| #SBATCH --mail-user=''mail-address''<br />
| The specified mail-address receives email notification of state changes as defined by --mail-type.<br />
|-<br />
|- style="vertical-align:top;"<br />
| --output=''name''<br />
| #SBATCH --output=''name''<br />
| File in which job output is stored. <br />
|-<br />
|- style="vertical-align:top;"<br />
| --error=''name''<br />
| #SBATCH --error=''name''<br />
| File in which job error messages are stored. <br />
|-<br />
|- style="vertical-align:top;"<br />
| -J, --job-name=''name''<br />
| #SBATCH --job-name=''name''<br />
| Job name.<br />
|-<br />
|- style="vertical-align:top;"<br />
| --export=[ALL,] ''env-variables''<br />
| #SBATCH --export=[ALL,] ''env-variables''<br />
| Identifies which environment variables from the submission environment are propagated to the launched application. Default is ALL.<br />
|-<br />
|- style="vertical-align:top;"<br />
| -A, --account=''group-name''<br />
| #SBATCH --account=''group-name''<br />
| Change resources used by this job to specified group. You may need this option if your account is assigned to more than one group. By command "scontrol show job" the project group the job is accounted on can be seen behind "Account=". <br />
|-<br />
|- style="vertical-align:top;"<br />
| -p, --partition=''queue-name''<br />
| #SBATCH --partition=''queue-name''<br />
| Request a specific queue for the resource allocation.<br />
|-<br />
|- style="vertical-align:top;"<br />
| --reservation=''reservation-name''<br />
| #SBATCH --reservation=''reservation-name''<br />
| Use a specific reservation for the resource allocation.<br />
|-<br />
|- style="vertical-align:top;"<br />
| -C, --constraint=''LSDF''<br />
| #SBATCH --constraint=LSDF<br />
| Job constraint LSDF filesystems.<br />
|-<br />
|- style="vertical-align:top;"<br />
| -C, --constraint=''BEEOND (BEEOND_4MDS, BEEOND_MAXMDS)''<br />
| #SBATCH --constraint=BEEOND (BEEOND_4MDS, BEEOND_MAXMDS)<br />
| Job constraint BeeOND filesystem.<br />
|-<br />
|}<br />
<br><br />
<br />
== Interactive Jobs: salloc ==<br />
<br />
On bwUniCluster 3.0 you are only allowed to run short jobs (<< 1 hour) with little memory requirements (<< 8 GByte) on the logins nodes. If you want to run longer jobs and/or jobs with a request of more than 8 GByte of memory, you must allocate resources for so-called interactive jobs by usage of the command salloc on a login node. Considering a serial application running on a compute node that requires 5000 MByte of memory and limiting the interactive run to 2 hours the following command has to be executed:<br />
<br />
<pre><br />
$ salloc -p cpu -n 1 -t 120 --mem=5000<br />
</pre><br />
<br />
Then you will get one core on a compute node within the partition "cpu". After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system Slurm has granted you the requested resources on the compute system. You will be logged in automatically on the granted core! To run a serial program on the granted core you only have to type the name of the executable.<br />
<br />
<pre><br />
$ ./<my_serial_program><br />
</pre><br />
<br />
Please be aware that your serial job must run less than 2 hours in this example, else the job will be killed during runtime by the system. <br />
<br />
<br />
You can also start now a graphical X11-terminal connecting you to the dedicated resource that is available for 2 hours. You can start it by the command:<br />
<br />
<pre><br />
$ xterm<br />
</pre><br />
<br />
Note that, once the walltime limit has been reached the resources - i.e. the compute node - will automatically be revoked.<br />
<br />
<br />
An interactive parallel application running on one compute node or on many compute nodes (e.g. here 5 nodes) with 96 cores each requires usually an amount of memory in GByte (e.g. 50 GByte) and a maximum time (e.g. 1 hour). E.g. 5 nodes can be allocated by the following command:<br />
<br />
<pre><br />
$ salloc -p cpu -N 5 --ntasks-per-node=96 -t 01:00:00 --mem=50gb<br />
</pre><br />
<br />
Now you can run parallel jobs on 480 cores requiring 50 GByte of memory per node. Please be aware that you will be logged in on core 0 of the first node.<br />
If you want to have access to another node you have to open a new terminal, connect it also to bwUniCluster 3.0 and type the following commands to<br />
connect to the running interactive job and then to a specific node:<br />
<br />
<pre><br />
$ srun --jobid=XXXXXXXX --pty /bin/bash<br />
$ srun --nodelist=uc3nXXX --pty /bin/bash<br />
</pre><br />
<br />
With the command:<br />
<br />
<pre><br />
$ squeue<br />
</pre><br />
<br />
the jobid and the nodelist can be shown.<br />
<br />
If you want to run MPI-programs, you can do it by simply typing mpirun <program_name>. Then your program will be run on 480 cores. A very simple example for starting a parallel job can be:<br />
<br />
<pre><br />
$ mpirun <my_mpi_program><br />
</pre><br />
<br />
You can also start the debugger ddt by the commands:<br />
<br />
<pre><br />
$ module add devel/ddt<br />
$ ddt <my_mpi_program><br />
</pre><br />
<br />
The above commands will execute the parallel program <my_mpi_program> on all available cores. You can also start parallel programs on a subset of cores; an example for this can be:<br />
<br />
<pre><br />
$ mpirun -n 50 <my_mpi_program><br />
</pre><br />
<br />
If you are using Intel MPI you must start <my_mpi_program> by the command mpiexec.hydra (instead of mpirun).<br />
<br />
== Interactive Computing with Jupyter ==<br />
<br />
== Monitor and manage jobs ==<br />
<br />
=== List of your submitted jobs : squeue ===<br />
Displays information about YOUR active, pending and/or recently completed jobs. The command displays all own active and pending jobs. The command squeue is explained in detail on the webpage https://slurm.schedmd.com/squeue.html or via manpage (man squeue).<br />
<br><br />
<br><br />
<br />
* ''squeue'' example on bwUniCluster 3.0 <small>(Only your own jobs are displayed!)</small>.<br />
<pre><br />
$ squeue <br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
1262 cpu wrap ka_ab123 R 8:15 1 uc3n002<br />
1267 dev_gpu_h wrap ka_ab123 PD 0:00 1 (Resources)<br />
1265 highmem wrap ka_ab123 R 2:41 1 uc3n084<br />
$ squeue -l<br />
JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON)<br />
1262 cpu wrap ka_ab123 RUNNING 8:55 20:00 1 uc3n002<br />
1267 dev_gpu_h wrap ka_ab123 PENDING 0:00 20:00 1 (Resources)<br />
1265 highmem wrap ka_ab123 RUNNING 3:21 20:00 1 uc3n084<br />
<br />
</pre><br />
<br />
=== Detailed job information : scontrol show job ===<br />
scontrol show job displays detailed job state information and diagnostic output for all or a specified job of yours. Detailed information is available for active, pending and recently completed jobs. The command scontrol is explained in detail on the webpage https://slurm.schedmd.com/scontrol.html or via manpage (man scontrol). <br />
<br><br />
Display the state of all your jobs in normal mode: scontrol show job<br />
<br><br />
Display the state of a job with <jobid> in normal mode: scontrol show job <jobid><br />
<br><br />
<br><br />
<br />
* Here is an example from bwUniCluster 3.0.<br />
<pre><br />
$ squeue<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
1262 cpu wrap ka_zs040 R 1:12 1 uc3n002<br />
<br />
$<br />
$ # now, see what's up with my pending job with jobid 1262<br />
$ <br />
$ scontrol show job 1262<br />
<br />
JobId=1262 JobName=wrap<br />
UserId=ka_zs0402(241992) GroupId=ka_scc(12345) MCS_label=N/A<br />
Priority=4246 Nice=0 Account=ka QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0<br />
RunTime=00:00:37 TimeLimit=00:20:00 TimeMin=N/A<br />
SubmitTime=2025-04-04T10:01:30 EligibleTime=2025-04-04T10:01:30<br />
AccrueTime=2025-04-04T10:01:30<br />
StartTime=2025-04-04T10:01:31 EndTime=2025-04-04T10:21:31 Deadline=N/A<br />
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2025-04-04T10:01:31 Scheduler=Main<br />
Partition=cpu AllocNode:Sid=uc3n999:2819841<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=uc3n002<br />
BatchHost=uc3n002<br />
NumNodes=1 NumCPUs=2 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*<br />
ReqTRES=cpu=1,mem=2000M,node=1,billing=1<br />
AllocTRES=cpu=2,mem=4000M,node=1,billing=2<br />
Socks/Node=* NtasksPerN:B:S:C=0:0:*:1 CoreSpec=*<br />
MinCPUsNode=1 MinMemoryCPU=2000M MinTmpDiskNode=0<br />
Features=(null) DelayBoot=00:00:00<br />
OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=(null)<br />
WorkDir=/pfs/data6/home/ka/ka_scc/ka_zs0402<br />
StdErr=/pfs/data6/home/ka/ka_scc/ka_zs0402/slurm-1262.out<br />
StdIn=/dev/null<br />
StdOut=/pfs/data6/home/ka/ka_scc/ka_zs0402/slurm-1262.out<br />
<br />
</pre><br />
<br><br />
=== Canceling own jobs : scancel ===<br />
The scancel command is used to cancel jobs. The command scancel is explained in detail on the webpage https://slurm.schedmd.com/scancel.html or via manpage (man scancel). The command is:<br />
<pre><br />
$ scancel [-i] <job-id><br />
$ scancel -t <job_state_name><br />
</pre><br />
<br />
= Slurm Options =<br />
[[BwUniCluster3.0/Running_Jobs/Slurm | Detailed Slurm usage]]<br />
<br />
= Best Practices =<br />
<br />
== Step-by-Step example==<br />
<br />
== Dos and Don'ts ==</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster3.0/Running_Jobs&diff=15165
BwUniCluster3.0/Running Jobs
2025-07-22T12:25:24Z
<p>P Weisbrod: </p>
<hr />
<div><br />
= Purpose and function of a queuing system =<br />
<br />
All compute activities on bwUniCluster 3.0 have to be performed on the compute nodes. Compute nodes are only available by requesting the corresponding resources via the queuing system. As soon as the requested resources are available, automated tasks are executed via a batch script or they can be accessed interactively.<br><br />
General procedure: Hint to [[Running_Calculations | Running Calculations]]<br />
<br />
== Job submission process ==<br />
<br />
bwUniCluster 3.0 has installed the workload managing software Slurm. Therefore any job submission by the user is to be executed by commands of the Slurm software. Slurm queues and runs user jobs based on fair sharing policies.<br />
<br />
== Slurm ==<br />
<br />
HPC Workload Manager on bwUniCluster 3.0 is Slurm.<br />
Slurm is a cluster management and job scheduling system. Slurm has three key functions. <br />
* It allocates access to resources (compute cores on nodes) to users for some duration of time so they can perform work. <br />
* It provides a framework for starting, executing, and monitoring work (normally a parallel job) on the set of allocated nodes. <br />
* It arbitrates contention for resources by managing a queue of pending work.<br />
<br />
Any kind of calculation on the compute nodes of bwUniCluster 3.0 requires the user to define calculations as a sequence of commands together with required run time, number of CPU cores and main memory and submit all, i.e., the '''batch job''', to a resource and workload managing software.<br />
<br />
== Terms and definitions ==<br />
<br />
<br />
''' Partitions '''<br />
<br />
Slurm manages job queues for different '''partitions'''. Partitions are used to group similar node types (e.g. nodes with and without accelerators) and to enforce different access policies and resource limits.<br />
<br />
On bwUniCluster 3.0 there are different partitions:<br />
<br />
* CPU-only nodes<br />
** 2-socket nodes, consisting of 2 Intel Ice Lake processors with 32 cores each or 2 AMD processors with 48 cores each<br />
** 2-socket nodes with very high RAM capacity, consisting of 2 AMD processors with 48 cores each<br />
* GPU-accelerated nodes<br />
** 2-socket nodes with 4x NVIDIA A100 or 4x NVIDIA H100 GPUs<br />
** 4-socket node with 4x AMD Instinct accelerator<br />
<br />
''' Queues '''<br />
<br />
Job '''queues''' are used to manage jobs that request access to shared but limited computing resources of a certain kind (partition).<br />
<br />
On bwUniCluster 3.0 there are different main types of queues:<br />
* Regular queues<br />
** cpu: Jobs that request CPU-only nodes.<br />
** gpu: Jobs that request GPU-accelerated nodes.<br />
* Development queues (dev)<br />
** Short, usually interactive jobs that are used for developing, compiling and testing code and workflows. The intention behind development queues is to provide users with immediate access to computer resources without having to wait. This is the place to realize instantaneous heavy compute without affecting other users, as would be the case on the login nodes.<br />
<br />
Requested compute resources such as (wall-)time, number of nodes and amount of memory are restricted and must fit into the boundaries imposed by the queues. The request for compute resources on the bwUniCluster 3.0 <font color=red>requires at least the specification of the '''queue''' and the '''time'''</font>.<br />
<br />
''' Jobs '''<br />
<br />
Jobs can be run non-interactively as '''batch jobs''' or as '''interactive jobs'''.<br><br />
Submitting a batch job means, that all steps of a compute project are defined in a Bash script. This Bash script is queued and executed as soon as the compute resources are available and allocated. Jobs are enqueued with the <code>sbatch</code> command.<br />
For interactive jobs, the resources are requested with the <code>salloc</code> command. As soon as the computing resources are available and allocated, a command line prompt is returned on a computing node and the user can freely dispose of the resources now available to him.<br />
{|style="background:#deffee; width:100%;"<br />
|style="padding:5px; background:#cef2e0; text-align:left"|<br />
[[Image:Attention.svg|center|25px]]<br />
|style="padding:5px; background:#cef2e0; text-align:left"|<br />
'''Please remember:'''<br />
* '''Heavy computations are not allowed on the login nodes'''.<br>Use a developement or a regular job queue instead! Please refer to [[BwUniCluster3.0/Login#Allowed_Activities_on_Login_Nodes|Allowed Activities on Login Nodes]].<br />
* '''Development queues''' are meant for '''development tasks'''.<br>Do not misuse this queue for regular, short-running jobs or chain jobs! Only one running job at a time is enabled. Maximum queue length is reduced to 3.<br />
|}<br />
<br />
= Queues on bwUniCluster 3.0 = <br />
== Policy ==<br />
<br />
The computing time is provided in accordance with the '''fair share policy'''. The individual investment shares of the respective university and the resources already used by its members are taken into account. Furthermore, the following throttling policy is also active: The '''maximum amount of physical cores''' used at any given time from jobs running is '''1920 per user''' (aggregated over all running jobs). This number corresponds to 30 nodes on the Ice Lake partition or 20 nodes on the standard partition. The aim is to minimize waiting times and maximize the number of users who can access computing time at the same time.<br />
<br />
== Regular Queues ==<br />
{| class="wikitable"<br />
|- <br />
! style="width:5%"| Queue<br />
! style="width:13%"| Node-Type<br />
! style="width:23%"| Default Resources<br />
! style="width:13%"| Minimal Resources<br />
! style="width:13%"| Maximum Resources<br />
|-<br />
| <code>cpu_il</code><br />
| CPU nodes<br/>Ice Lake<br />
| mem-per-cpu=2000mb<br />
| <br />
| time=72:00:00, nodes=30, mem=249600mb, ntasks-per-node=64, (threads-per-core=2) <br />
|-<br />
| <code>cpu</code><br />
| CPU nodes<br/>Standard<br />
| mem-per-cpu=2000mb<br />
| <br />
| time=72:00:00, nodes=20, mem=380000mb, ntasks-per-node=96, (threads-per-core=2)<br />
|-<br />
| <code>highmem</code><br />
| CPU nodes<br/>High Memory<br />
| mem-per-cpu=12090mb<br />
| mem=380001mb<br />
| time=72:00:00, nodes=4, mem=2300000mb, ntasks-per-node=96, (threads-per-core=2)<br />
|-<br />
| <code>gpu_h100</code><br />
| GPU nodes<br/>NVIDIA GPU x4<br />
| mem-per-gpu=193300mb<br/>cpus-per-gpu=24<br />
| <br />
| time=72:00:00, nodes=12, mem=760000mb, ntasks-per-node=96, (threads-per-core=2)<br />
|-<br />
| <code>gpu_mi300</code><br />
| GPU node<br/>AMD GPU x4<br />
| mem-per-gpu=128200mb<br/>cpus-per-gpu=24<br />
| <br />
| time=72:00:00, nodes=1, mem=510000mb, ntasks-per-node=40, (threads-per-core=2)<br />
|-<br />
| <code>gpu_a100_il</code>/<code>gpu_h100_il</code><br />
| GPU nodes<br/>Ice Lake<br/>NVIDIA GPU x4<br />
| mem-per-gpu=127500mb<br/>cpus-per-gpu=16<br />
| <br />
| time=48:00:00, nodes=9, mem=510000mb, ntasks-per-node=64, (threads-per-core=2) <br />
|}<br />
Table 1: Regular Queues<br />
<br />
== Short Queues ==<br />
Queues with a short runtime of 30 minutes.<br />
<br />
{| class="wikitable"<br />
|- <br />
! style="width:5%"| Queue<br />
! style="width:13%"| Node Type<br />
! style="width:23%"| Default Resources<br />
! style="width:13%"| Minimal Resources<br />
! style="width:13%"| Maximum Resources<br />
|-<br />
| <code>gpu_a100_short</code><br />
| GPU nodes<br/>Ice Lake<br/>NVIDIA GPU x4<br />
| mem-per-gpu=94000mb<br/>cpus-per-gpu=12<br />
| <br />
| time=30, nodes=12, mem=376000mb, ntasks-per-node=48, (threads-per-core=2)<br />
|}<br />
Table 2: Short Queues<br />
<br />
== Development Queues ==<br />
Only for development, i.e. debugging or performance optimization ...<br />
<br />
{| class="wikitable"<br />
|- <br />
! style="width:5%"| Queue<br />
! style="width:13%"| Node Type<br />
! style="width:23%"| Default Resources<br />
! style="width:13%"| Minimal Resources<br />
! style="width:13%"| Maximum Resources<br />
|-<br />
| <code>dev_cpu_il</code><br />
| CPU nodes<br/>Ice Lake<br />
| mem-per-cpu=2000mb<br />
| <br />
| time=30, nodes=8, mem=249600mb, ntasks-per-node=64, (threads-per-core=2)<br />
|-<br />
| <code>dev_cpu</code><br />
| CPU nodes<br/>Standard<br />
| mem-per-cpu=2000mb<br />
| <br />
| time=30, nodes=1, mem=380000mb, ntasks-per-node=96, (threads-per-core=2)<br />
|-<br />
| <code>dev_gpu_h100</code><br />
| GPU nodes<br/>NVIDIA GPU x4<br />
| mem-per-gpu=193300mb<br/>cpus-per-gpu=24<br />
| <br />
| time=30, nodes=1, mem=760000mb, ntasks-per-node=96, (threads-per-core=2)<br />
|-<br />
| <code>dev_gpu_a100_il</code><br />
| GPU nodes<br/>NVIDIA GPU x4<br/><br />
| mem-per-gpu=127500mb<br/>cpus-per-gpu=16 <br />
| <br />
| time=30, nodes=1, mem=510000mb, ntasks-per-node=64, (threads-per-core=2) <br />
|}<br />
Table 3: Development Queues<br />
<br />
<br />
Default resources of a queue class define number of tasks and memory if not explicitly given with sbatch command. Resource list acronyms ''--time'', ''--ntasks'', ''--nodes'', ''--mem'' and ''--mem-per-cpu'' are described [[BwUniCluster3.0/Running_Jobs/Slurm|here]].<br />
<br />
== Check available resources: sinfo_t_idle ==<br />
The Slurm command sinfo is used to view partition and node information for a system running Slurm. It incorporates down time, reservations, and node state information in determining the available backfill window. The sinfo command can only be used by the administrator.<br />
<br><br />
SCC has prepared a special script (sinfo_t_idle) to find out how many processors are available for immediate use on the system. It is anticipated that users will use this information to submit jobs that meet these criteria and thus obtain quick job turnaround times. <br />
<br><br />
<br><br />
<br />
* The following command displays what resources are available for immediate use for the whole partition.<br />
<pre>$ sinfo_t_idle <br />
Partition dev_cpu : 1 nodes idle<br />
Partition cpu : 1 nodes idle<br />
Partition highmem : 2 nodes idle<br />
Partition dev_gpu_h100 : 0 nodes idle<br />
Partition gpu_h100 : 0 nodes idle<br />
Partition gpu_mi300 : 0 nodes idle<br />
Partition dev_cpu_il : 7 nodes idle<br />
Partition cpu_il : 2 nodes idle<br />
Partition dev_gpu_a100_il : 1 nodes idle<br />
Partition gpu_a100_il : 0 nodes idle<br />
Partition gpu_h100_il : 1 nodes idle<br />
Partition gpu_a100_short : 0 nodes idle<br />
</pre><br />
<br><br />
<br />
= Running Jobs =<br />
<br />
== Slurm Commands (excerpt) ==<br />
Important Slurm commands for non-administrators working on bwUniCluster 3.0.<br />
{| width=850px class="wikitable"<br />
! Slurm commands !! Brief explanation<br />
|-<br />
| [[#Batch Jobs: sbatch|sbatch]] || Submits a job and puts it into the queue [[https://slurm.schedmd.com/sbatch.html sbatch]] <br />
|-<br />
| [[#Interactive Jobs: salloc|salloc]] || Requests resources for an interactive Job [[https://slurm.schedmd.com/salloc.html salloc]]<br />
|-<br />
| [[#Monitor and manage jobs |scontrol show job]] || Displays detailed job state information [[https://slurm.schedmd.com/scontrol.html scontrol]]<br />
|-<br />
| [[#List of your submitted jobs : squeue|squeue]] || Displays information about active, eligible, blocked, and/or recently completed jobs [[https://slurm.schedmd.com/squeue.html squeue]]<br />
|-<br />
| [[#List of your submitted jobs : squeue|squeue --start]] || Returns start time of submitted job [[https://slurm.schedmd.com/squeue.html squeue]]<br />
|-<br />
| [[#Check available resources: sinfo_t_idle|sinfo_t_idle]] || Shows what resources are available for immediate use [[https://slurm.schedmd.com/sinfo.html sinfo]]<br />
|-<br />
| [[#Canceling own jobs : scancel|scancel]] || Cancels a job [[https://slurm.schedmd.com/scancel.html scancel]]<br />
|}<br />
<br />
<br><br />
* [https://slurm.schedmd.com/tutorials.html Slurm Tutorials]<br />
* [https://slurm.schedmd.com/pdfs/summary.pdf Slurm command/option summary (2 pages)]<br />
* [https://slurm.schedmd.com/man_index.html Slurm Commands]<br />
<br><br />
<br />
== Batch Jobs: sbatch ==<br />
<br />
Batch jobs are submitted by using the command '''sbatch'''. The main purpose of the '''sbatch''' command is to specify the resources that are needed to run the job. '''sbatch''' will then queue the batch job. However, starting of batch job depends on the availability of the requested resources and the fair sharing value.<br />
<br><br />
<br><br />
<br />
* The syntax and use of '''sbatch''' can be displayed via:<br />
<pre><br />
$ man sbatch<br />
</pre><br />
<br />
'''sbatch''' options can be used from the command line or in your job script. Different defaults for some of these options are set based on the queue and can be found [[BwUniCluster3.0/Slurm | here]]<br />
<br />
{| class="wikitable"<br />
! colspan="3" | sbatch Options<br />
|-<br />
! style="width:8%"| Command line<br />
! style="width:9%"| Script<br />
! style="width:13%"| Purpose<br />
|- style="vertical-align:top;"<br />
| -t, --time=''time''<br />
| #SBATCH --time=''time''<br />
| Wall clock time limit.<br><br />
|-<br />
|- style="vertical-align:top;"<br />
| -N, --nodes=''count''<br />
| #SBATCH --nodes=''count''<br />
| Number of nodes to be used.<br />
|- style="vertical-align:top;"<br />
| -n, --ntasks=''count''<br />
| #SBATCH --ntasks=''count''<br />
| Number of tasks to be launched.<br />
|-<br />
|- style="vertical-align:top;"<br />
| --ntasks-per-node=''count''<br />
| #SBATCH --ntasks-per-node=''count''<br />
| Maximum count of tasks per node.<br />
|-<br />
|- style="vertical-align:top;"<br />
| -c, --cpus-per-task=''count''<br />
| #SBATCH --cpus-per-task=''count''<br />
| Number of CPUs required per (MPI-)task.<br />
|-<br />
|- style="vertical-align:top;"<br />
| --mem=''value_in_MB''<br />
| #SBATCH --mem=''value_in_MB'' <br />
| Memory in MegaByte per node. (You should omit the setting of this option.)<br />
|-<br />
|- style="vertical-align:top;"<br />
| --mem-per-cpu=''value_in_MB''<br />
| #SBATCH --mem-per-cpu=''value_in_MB'' <br />
| Minimum Memory required per allocated CPU. (You should omit the setting of this option.)<br />
|-<br />
|- style="vertical-align:top;"<br />
| --mail-type=''type''<br />
| #SBATCH --mail-type=''type''<br />
| Notify user by email when certain event types occur.<br>Valid type values are NONE, BEGIN, END, FAIL, REQUEUE, ALL.<br />
|-<br />
|- style="vertical-align:top;"<br />
| --mail-user=''mail-address''<br />
| #SBATCH --mail-user=''mail-address''<br />
| The specified mail-address receives email notification of state changes as defined by --mail-type.<br />
|-<br />
|- style="vertical-align:top;"<br />
| --output=''name''<br />
| #SBATCH --output=''name''<br />
| File in which job output is stored. <br />
|-<br />
|- style="vertical-align:top;"<br />
| --error=''name''<br />
| #SBATCH --error=''name''<br />
| File in which job error messages are stored. <br />
|-<br />
|- style="vertical-align:top;"<br />
| -J, --job-name=''name''<br />
| #SBATCH --job-name=''name''<br />
| Job name.<br />
|-<br />
|- style="vertical-align:top;"<br />
| --export=[ALL,] ''env-variables''<br />
| #SBATCH --export=[ALL,] ''env-variables''<br />
| Identifies which environment variables from the submission environment are propagated to the launched application. Default is ALL.<br />
|-<br />
|- style="vertical-align:top;"<br />
| -A, --account=''group-name''<br />
| #SBATCH --account=''group-name''<br />
| Change resources used by this job to specified group. You may need this option if your account is assigned to more than one group. By command "scontrol show job" the project group the job is accounted on can be seen behind "Account=". <br />
|-<br />
|- style="vertical-align:top;"<br />
| -p, --partition=''queue-name''<br />
| #SBATCH --partition=''queue-name''<br />
| Request a specific queue for the resource allocation.<br />
|-<br />
|- style="vertical-align:top;"<br />
| --reservation=''reservation-name''<br />
| #SBATCH --reservation=''reservation-name''<br />
| Use a specific reservation for the resource allocation.<br />
|-<br />
|- style="vertical-align:top;"<br />
| -C, --constraint=''LSDF''<br />
| #SBATCH --constraint=LSDF<br />
| Job constraint LSDF filesystems.<br />
|-<br />
|- style="vertical-align:top;"<br />
| -C, --constraint=''BEEOND (BEEOND_4MDS, BEEOND_MAXMDS)''<br />
| #SBATCH --constraint=BEEOND (BEEOND_4MDS, BEEOND_MAXMDS)<br />
| Job constraint BeeOND filesystem.<br />
|-<br />
|}<br />
<br><br />
<br />
== Interactive Jobs: salloc ==<br />
<br />
On bwUniCluster 3.0 you are only allowed to run short jobs (<< 1 hour) with little memory requirements (<< 8 GByte) on the logins nodes. If you want to run longer jobs and/or jobs with a request of more than 8 GByte of memory, you must allocate resources for so-called interactive jobs by usage of the command salloc on a login node. Considering a serial application running on a compute node that requires 5000 MByte of memory and limiting the interactive run to 2 hours the following command has to be executed:<br />
<br />
<pre><br />
$ salloc -p cpu -n 1 -t 120 --mem=5000<br />
</pre><br />
<br />
Then you will get one core on a compute node within the partition "cpu". After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system Slurm has granted you the requested resources on the compute system. You will be logged in automatically on the granted core! To run a serial program on the granted core you only have to type the name of the executable.<br />
<br />
<pre><br />
$ ./<my_serial_program><br />
</pre><br />
<br />
Please be aware that your serial job must run less than 2 hours in this example, else the job will be killed during runtime by the system. <br />
<br />
<br />
You can also start now a graphical X11-terminal connecting you to the dedicated resource that is available for 2 hours. You can start it by the command:<br />
<br />
<pre><br />
$ xterm<br />
</pre><br />
<br />
Note that, once the walltime limit has been reached the resources - i.e. the compute node - will automatically be revoked.<br />
<br />
<br />
An interactive parallel application running on one compute node or on many compute nodes (e.g. here 5 nodes) with 96 cores each requires usually an amount of memory in GByte (e.g. 50 GByte) and a maximum time (e.g. 1 hour). E.g. 5 nodes can be allocated by the following command:<br />
<br />
<pre><br />
$ salloc -p cpu -N 5 --ntasks-per-node=96 -t 01:00:00 --mem=50gb<br />
</pre><br />
<br />
Now you can run parallel jobs on 480 cores requiring 50 GByte of memory per node. Please be aware that you will be logged in on core 0 of the first node.<br />
If you want to have access to another node you have to open a new terminal, connect it also to bwUniCluster 3.0 and type the following commands to<br />
connect to the running interactive job and then to a specific node:<br />
<br />
<pre><br />
$ srun --jobid=XXXXXXXX --pty /bin/bash<br />
$ srun --nodelist=uc3nXXX --pty /bin/bash<br />
</pre><br />
<br />
With the command:<br />
<br />
<pre><br />
$ squeue<br />
</pre><br />
<br />
the jobid and the nodelist can be shown.<br />
<br />
If you want to run MPI-programs, you can do it by simply typing mpirun <program_name>. Then your program will be run on 480 cores. A very simple example for starting a parallel job can be:<br />
<br />
<pre><br />
$ mpirun <my_mpi_program><br />
</pre><br />
<br />
You can also start the debugger ddt by the commands:<br />
<br />
<pre><br />
$ module add devel/ddt<br />
$ ddt <my_mpi_program><br />
</pre><br />
<br />
The above commands will execute the parallel program <my_mpi_program> on all available cores. You can also start parallel programs on a subset of cores; an example for this can be:<br />
<br />
<pre><br />
$ mpirun -n 50 <my_mpi_program><br />
</pre><br />
<br />
If you are using Intel MPI you must start <my_mpi_program> by the command mpiexec.hydra (instead of mpirun).<br />
<br />
== Interactive Computing with Jupyter ==<br />
<br />
== Monitor and manage jobs ==<br />
<br />
=== List of your submitted jobs : squeue ===<br />
Displays information about YOUR active, pending and/or recently completed jobs. The command displays all own active and pending jobs. The command squeue is explained in detail on the webpage https://slurm.schedmd.com/squeue.html or via manpage (man squeue).<br />
<br><br />
<br><br />
<br />
* ''squeue'' example on bwUniCluster 3.0 <small>(Only your own jobs are displayed!)</small>.<br />
<pre><br />
$ squeue <br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
1262 cpu wrap ka_ab123 R 8:15 1 uc3n002<br />
1267 dev_gpu_h wrap ka_ab123 PD 0:00 1 (Resources)<br />
1265 highmem wrap ka_ab123 R 2:41 1 uc3n084<br />
$ squeue -l<br />
JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON)<br />
1262 cpu wrap ka_ab123 RUNNING 8:55 20:00 1 uc3n002<br />
1267 dev_gpu_h wrap ka_ab123 PENDING 0:00 20:00 1 (Resources)<br />
1265 highmem wrap ka_ab123 RUNNING 3:21 20:00 1 uc3n084<br />
<br />
</pre><br />
<br />
=== Detailed job information : scontrol show job ===<br />
scontrol show job displays detailed job state information and diagnostic output for all or a specified job of yours. Detailed information is available for active, pending and recently completed jobs. The command scontrol is explained in detail on the webpage https://slurm.schedmd.com/scontrol.html or via manpage (man scontrol). <br />
<br><br />
Display the state of all your jobs in normal mode: scontrol show job<br />
<br><br />
Display the state of a job with <jobid> in normal mode: scontrol show job <jobid><br />
<br><br />
<br><br />
<br />
* Here is an example from bwUniCluster 3.0.<br />
<pre><br />
$ squeue<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
1262 cpu wrap ka_zs040 R 1:12 1 uc3n002<br />
<br />
$<br />
$ # now, see what's up with my pending job with jobid 1262<br />
$ <br />
$ scontrol show job 1262<br />
<br />
JobId=1262 JobName=wrap<br />
UserId=ka_zs0402(241992) GroupId=ka_scc(12345) MCS_label=N/A<br />
Priority=4246 Nice=0 Account=ka QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0<br />
RunTime=00:00:37 TimeLimit=00:20:00 TimeMin=N/A<br />
SubmitTime=2025-04-04T10:01:30 EligibleTime=2025-04-04T10:01:30<br />
AccrueTime=2025-04-04T10:01:30<br />
StartTime=2025-04-04T10:01:31 EndTime=2025-04-04T10:21:31 Deadline=N/A<br />
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2025-04-04T10:01:31 Scheduler=Main<br />
Partition=cpu AllocNode:Sid=uc3n999:2819841<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=uc3n002<br />
BatchHost=uc3n002<br />
NumNodes=1 NumCPUs=2 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*<br />
ReqTRES=cpu=1,mem=2000M,node=1,billing=1<br />
AllocTRES=cpu=2,mem=4000M,node=1,billing=2<br />
Socks/Node=* NtasksPerN:B:S:C=0:0:*:1 CoreSpec=*<br />
MinCPUsNode=1 MinMemoryCPU=2000M MinTmpDiskNode=0<br />
Features=(null) DelayBoot=00:00:00<br />
OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)<br />
Command=(null)<br />
WorkDir=/pfs/data6/home/ka/ka_scc/ka_zs0402<br />
StdErr=/pfs/data6/home/ka/ka_scc/ka_zs0402/slurm-1262.out<br />
StdIn=/dev/null<br />
StdOut=/pfs/data6/home/ka/ka_scc/ka_zs0402/slurm-1262.out<br />
<br />
</pre><br />
<br><br />
=== Canceling own jobs : scancel ===<br />
The scancel command is used to cancel jobs. The command scancel is explained in detail on the webpage https://slurm.schedmd.com/scancel.html or via manpage (man scancel). The command is:<br />
<pre><br />
$ scancel [-i] <job-id><br />
$ scancel -t <job_state_name><br />
</pre><br />
<br />
= Slurm Options =<br />
[[BwUniCluster3.0/Running_Jobs/Slurm | Detailed Slurm usage]]<br />
<br />
= Best Practices =<br />
<br />
== Step-by-Step example==<br />
<br />
== Dos and Don'ts ==</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0&diff=12823
BwUniCluster2.0
2024-07-09T12:17:05Z
<p>P Weisbrod: </p>
<hr />
<div><!-- Old text: check for permanent removal --><br />
<!--{| style="width: 100%; border-spacing: 5px;"<br />
| style="text-align:center; color:#000;vertical-align:middle;font-size:75%;" |<br />
[[File:BwUniCluster_2.0_Feb2020.jpg|center|border|550px|Close-up of bwUniCluster by Simon Raffeiner, Copyright: KIT (SCC)]] <br />
|-<br />
| style="text-align:center; color:#000;vertical-align:middle;" |<span style="font-size:80%">Close-up of bwUniCluster © KIT (Simon Raffeiner/SCC)</span><br />
|}<br />
<br />
On 17.03.2020, the Scientific Computing Center (SCC) at Karlsruhe Institute of Technology (KIT) commissioned a new parallel computer system called "bwUniCluster 2.0+GFB-HPC" as a state service within the bwHPC framework. The bwUniCluster 2.0 replaces the predecessor system [https://www.scc.kit.edu/dienste/bwUniCluster.php bwUniCluster] and also includes the additional compute nodes which were procured as an extension to the bwUniCluster in November 2016.<br />
<br />
The modern bwUniCluster 2.0 system consists of more than 840 SMP nodes with 64-bit Intel Xeon processors. It provides the universities of the state of Baden-Württemberg with general compute resources and can be used free of charge by the staff of all universities in Baden-Württemberg. Users who currently have access to bwUniCluster will automatically also have access to bwUniCluster 2.0. There is no need to apply for new entitlements or to re-register.<br />
<br />
<br />
--><br />
<!-- <br />
###########################################<br />
## Picture of bwUniCluster - right side ##<br />
###########################################<br />
--><br />
[[File:BwUniCluster_2.0_Feb2020_1024x423.jpg|right|frameless|thumb|alt=bwUniCluster2.0 |upright=1| bwUniCluster 2.0 ]]<br />
<br />
<br />
<!-- <br />
###########################################<br />
## About bwUniCluster ##<br />
###########################################<br />
--><br />
The '''bwUniCluster 2.0''' is the joint high-performance computer system of Baden-Württemberg's Universities and Universities of Applied Sciences for '''general purpose and teaching''' and located at the Scientific Computing Center (SCC) at Karlsruhe Institute of Technology (KIT). The bwUniCluster 2.0 complements the four bwForClusters and their dedicated scientific areas.<br />
<br />
<br />
{| style="background:#FFCCCC; width:100%;"<br />
| '''The following issue is known:''' Due to the hardware configuration, there is currently an already known problem with OpenMPI on the nodes in the "multiple_il" partition. It manifests itself in the warning "No OpenFabrics connection schemes reported" when starting an MPI application and refers to the device "mlx5_2". This is an Ethernet port, which is not supposed to be used by OpenMPI. The warning is informative, we are working on suppressing this message.<br />
|}<br />
<br />
<!-- <br />
###########################################<br />
## bwUniCluster: Maintenance Section ##<br />
###########################################<br />
## Comment out full section if there no upcoming maintenance<br />
<br />
<br />
{| style=" background:#FEF4AB; width:100%;" <br />
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | Next maintenance<br />
|-<br />
| <br />
Due to regular maintenance work the HPC System bwUnicluster 2 will not be available from <br />
<br />
21.05.2024 at 08:30 AM until 24.05.2024 at 15:00 AM<br />
<br />
Please see the [[BwUniCluster2.0/Maintenance/2024-05|maintenance]] page for more information about planned upgrades and other changes<br />
|}<br />
--><br />
<!-- <br />
###########################################<br />
## bwUniCluster: News section ##<br />
###########################################<br />
## Comment out full section if there no news<br />
--><br />
<!-- <br />
{| style=" background:#FEF4AB; width:100%;" <br />
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News<br />
|-<br />
| <br />
* 2022-11-25: All new nodes from bwUniCluster 2.0 Stage 2 are now available.<br />
|}<br />
--><br />
<!-- <br />
###########################################<br />
## bwUniCluster: Training/Support section##<br />
###########################################<br />
--><br />
{| style=" background:#eeeefe; width:100%;" <br />
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support<br />
|-<br />
| <br />
* [[BwUniCluster2.0/First_Steps|Getting Started]]<br />
* [https://training.bwhpc.de E-Learning Courses]<br />
* [[BwUniCluster2.0/Support|Support]]<br />
* [[BwUniCluster2.0/FAQ|FAQ]]<br />
* Send [[:Category:Feedback|Feedback]] about Wiki pages<br />
|}<br />
<!-- <br />
###########################################<br />
## bwUniCluster: User Documentation ##<br />
###########################################<br />
--><br />
{| style=" background:#deffee; width:100%;"<br />
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | User Documentation<br />
|-<br />
|<br />
* Access: [[Registration/bwUniCluster|Registration]], [[Registration/Deregistration|Deregistration]], [[BwUniCluster2.0/Jupyter|Using Jupyter]], [[BwUniCluster2.0/Jupyter (Deutsch)|Using Jupyter (German)]]<br />
* [[BwUniCluster2.0/Login|Login]]<br />
* <br />
* [[BwUniCluster2.0/Hardware_and_Architecture|Hardware and Architecture]]<br />
** [[BwUniCluster2.0/Hardware_and_Architecture#File_Systems|File Systems and Workspaces]] <br />
* [[BwUniCluster2.0/Software|Cluster Specific Software]]<br />
** [[BwUniCluster2.0/Containers|Using Containers]]<br />
* [[BwUniCluster2.0/Slurm|Batch System]]<br />
** [[BwUniCluster2.0/Batch_Queues|Queues and interactive Jobs]]<br />
* [[BwUniCluster2.0/Maintenance|Operational Changes]]<br />
|}<br />
<!-- <br />
###########################################<br />
## bwUniCluster: Acknowledgement ##<br />
###########################################<br />
--><br />
{| style=" background:#e6e9eb; width:100%;"<br />
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Cluster Funding<br />
|-<br />
|<br />
* Please [[BwUniCluster2.0/Acknowledgement|acknowledge]] bwUniCluster 2.0 in your publications.<br />
|}</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0&diff=12822
BwUniCluster2.0
2024-07-09T12:15:36Z
<p>P Weisbrod: </p>
<hr />
<div><!-- Old text: check for permanent removal --><br />
<!--{| style="width: 100%; border-spacing: 5px;"<br />
| style="text-align:center; color:#000;vertical-align:middle;font-size:75%;" |<br />
[[File:BwUniCluster_2.0_Feb2020.jpg|center|border|550px|Close-up of bwUniCluster by Simon Raffeiner, Copyright: KIT (SCC)]] <br />
|-<br />
| style="text-align:center; color:#000;vertical-align:middle;" |<span style="font-size:80%">Close-up of bwUniCluster © KIT (Simon Raffeiner/SCC)</span><br />
|}<br />
<br />
On 17.03.2020, the Scientific Computing Center (SCC) at Karlsruhe Institute of Technology (KIT) commissioned a new parallel computer system called "bwUniCluster 2.0+GFB-HPC" as a state service within the bwHPC framework. The bwUniCluster 2.0 replaces the predecessor system [https://www.scc.kit.edu/dienste/bwUniCluster.php bwUniCluster] and also includes the additional compute nodes which were procured as an extension to the bwUniCluster in November 2016.<br />
<br />
The modern bwUniCluster 2.0 system consists of more than 840 SMP nodes with 64-bit Intel Xeon processors. It provides the universities of the state of Baden-Württemberg with general compute resources and can be used free of charge by the staff of all universities in Baden-Württemberg. Users who currently have access to bwUniCluster will automatically also have access to bwUniCluster 2.0. There is no need to apply for new entitlements or to re-register.<br />
<br />
<br />
--><br />
<!-- <br />
###########################################<br />
## Picture of bwUniCluster - right side ##<br />
###########################################<br />
--><br />
[[File:BwUniCluster_2.0_Feb2020_1024x423.jpg|right|frameless|thumb|alt=bwUniCluster2.0 |upright=1| bwUniCluster 2.0 ]]<br />
<br />
<br />
<!-- <br />
###########################################<br />
## About bwUniCluster ##<br />
###########################################<br />
--><br />
The '''bwUniCluster 2.0''' is the joint high-performance computer system of Baden-Württemberg's Universities and Universities of Applied Sciences for '''general purpose and teaching''' and located at the Steinbuch Centre for Computing (SCC) at Karlsruhe Institute of Technology (KIT). The bwUniCluster 2.0 complements the four bwForClusters and their dedicated scientific areas.<br />
<br />
<br />
{| style="background:#FFCCCC; width:100%;"<br />
| '''The following issue is known:''' Due to the hardware configuration, there is currently an already known problem with OpenMPI on the nodes in the "multiple_il" partition. It manifests itself in the warning "No OpenFabrics connection schemes reported" when starting an MPI application and refers to the device "mlx5_2". This is an Ethernet port, which is not supposed to be used by OpenMPI. The warning is informative, we are working on suppressing this message.<br />
|}<br />
<br />
<!-- <br />
###########################################<br />
## bwUniCluster: Maintenance Section ##<br />
###########################################<br />
## Comment out full section if there no upcoming maintenance<br />
<br />
<br />
{| style=" background:#FEF4AB; width:100%;" <br />
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | Next maintenance<br />
|-<br />
| <br />
Due to regular maintenance work the HPC System bwUnicluster 2 will not be available from <br />
<br />
21.05.2024 at 08:30 AM until 24.05.2024 at 15:00 AM<br />
<br />
Please see the [[BwUniCluster2.0/Maintenance/2024-05|maintenance]] page for more information about planned upgrades and other changes<br />
|}<br />
--><br />
<!-- <br />
###########################################<br />
## bwUniCluster: News section ##<br />
###########################################<br />
## Comment out full section if there no news<br />
--><br />
<!-- <br />
{| style=" background:#FEF4AB; width:100%;" <br />
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News<br />
|-<br />
| <br />
* 2022-11-25: All new nodes from bwUniCluster 2.0 Stage 2 are now available.<br />
|}<br />
--><br />
<!-- <br />
###########################################<br />
## bwUniCluster: Training/Support section##<br />
###########################################<br />
--><br />
{| style=" background:#eeeefe; width:100%;" <br />
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support<br />
|-<br />
| <br />
* [[BwUniCluster2.0/First_Steps|Getting Started]]<br />
* [https://training.bwhpc.de E-Learning Courses]<br />
* [[BwUniCluster2.0/Support|Support]]<br />
* [[BwUniCluster2.0/FAQ|FAQ]]<br />
* Send [[:Category:Feedback|Feedback]] about Wiki pages<br />
|}<br />
<!-- <br />
###########################################<br />
## bwUniCluster: User Documentation ##<br />
###########################################<br />
--><br />
{| style=" background:#deffee; width:100%;"<br />
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | User Documentation<br />
|-<br />
|<br />
* Access: [[Registration/bwUniCluster|Registration]], [[Registration/Deregistration|Deregistration]], [[BwUniCluster2.0/Jupyter|Using Jupyter]], [[BwUniCluster2.0/Jupyter (Deutsch)|Using Jupyter (German)]]<br />
* [[BwUniCluster2.0/Login|Login]]<br />
* <br />
* [[BwUniCluster2.0/Hardware_and_Architecture|Hardware and Architecture]]<br />
** [[BwUniCluster2.0/Hardware_and_Architecture#File_Systems|File Systems and Workspaces]] <br />
* [[BwUniCluster2.0/Software|Cluster Specific Software]]<br />
** [[BwUniCluster2.0/Containers|Using Containers]]<br />
* [[BwUniCluster2.0/Slurm|Batch System]]<br />
** [[BwUniCluster2.0/Batch_Queues|Queues and interactive Jobs]]<br />
* [[BwUniCluster2.0/Maintenance|Operational Changes]]<br />
|}<br />
<!-- <br />
###########################################<br />
## bwUniCluster: Acknowledgement ##<br />
###########################################<br />
--><br />
{| style=" background:#e6e9eb; width:100%;"<br />
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Cluster Funding<br />
|-<br />
|<br />
* Please [[BwUniCluster2.0/Acknowledgement|acknowledge]] bwUniCluster 2.0 in your publications.<br />
|}</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0&diff=12782
BwUniCluster2.0
2024-05-24T08:47:36Z
<p>P Weisbrod: </p>
<hr />
<div><!-- Old text: check for permanent removal --><br />
<!--{| style="width: 100%; border-spacing: 5px;"<br />
| style="text-align:center; color:#000;vertical-align:middle;font-size:75%;" |<br />
[[File:BwUniCluster_2.0_Feb2020.jpg|center|border|550px|Close-up of bwUniCluster by Simon Raffeiner, Copyright: KIT (SCC)]] <br />
|-<br />
| style="text-align:center; color:#000;vertical-align:middle;" |<span style="font-size:80%">Close-up of bwUniCluster © KIT (Simon Raffeiner/SCC)</span><br />
|}<br />
<br />
On 17.03.2020, the Steinbuch Centre for Computing (SCC) at Karlsruhe Institute of Technology (KIT) commissioned a new parallel computer system called "bwUniCluster 2.0+GFB-HPC" as a state service within the bwHPC framework. The bwUniCluster 2.0 replaces the predecessor system [https://www.scc.kit.edu/dienste/bwUniCluster.php bwUniCluster] and also includes the additional compute nodes which were procured as an extension to the bwUniCluster in November 2016.<br />
<br />
The modern bwUniCluster 2.0 system consists of more than 840 SMP nodes with 64-bit Intel Xeon processors. It provides the universities of the state of Baden-Württemberg with general compute resources and can be used free of charge by the staff of all universities in Baden-Württemberg. Users who currently have access to bwUniCluster will automatically also have access to bwUniCluster 2.0. There is no need to apply for new entitlements or to re-register.<br />
<br />
<br />
--><br />
<!-- <br />
###########################################<br />
## Picture of bwUniCluster - right side ##<br />
###########################################<br />
--><br />
[[File:BwUniCluster_2.0_Feb2020_1024x423.jpg|right|frameless|thumb|alt=bwUniCluster2.0 |upright=1| bwUniCluster 2.0 ]]<br />
<br />
<br />
<!-- <br />
###########################################<br />
## About bwUniCluster ##<br />
###########################################<br />
--><br />
The '''bwUniCluster 2.0''' is the joint high-performance computer system of Baden-Württemberg's Universities and Universities of Applied Sciences for '''general purpose and teaching''' and located at the Steinbuch Centre for Computing (SCC) at Karlsruhe Institute of Technology (KIT). The bwUniCluster 2.0 complements the four bwForClusters and their dedicated scientific areas.<br />
<br />
<br />
{| style="background:#FFCCCC; width:100%;"<br />
| '''The following issue is known:''' Due to the hardware configuration, there is currently an already known problem with OpenMPI on the nodes in the "multiple_il" partition. It manifests itself in the warning "No OpenFabrics connection schemes reported" when starting an MPI application and refers to the device "mlx5_2". This is an Ethernet port, which is not supposed to be used by OpenMPI. The warning is informative, we are working on suppressing this message.<br />
|}<br />
<br />
<!-- <br />
###########################################<br />
## bwUniCluster: Maintenance Section ##<br />
###########################################<br />
## Comment out full section if there no upcoming maintenance<br />
<br />
<br />
{| style=" background:#FEF4AB; width:100%;" <br />
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | Next maintenance<br />
|-<br />
| <br />
Due to regular maintenance work the HPC System bwUnicluster 2 will not be available from <br />
<br />
21.05.2024 at 08:30 AM until 24.05.2024 at 15:00 AM<br />
<br />
Please see the [[BwUniCluster2.0/Maintenance/2024-05|maintenance]] page for more information about planned upgrades and other changes<br />
|}<br />
--><br />
<!-- <br />
###########################################<br />
## bwUniCluster: News section ##<br />
###########################################<br />
## Comment out full section if there no news<br />
--><br />
<!-- <br />
{| style=" background:#FEF4AB; width:100%;" <br />
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News<br />
|-<br />
| <br />
* 2022-11-25: All new nodes from bwUniCluster 2.0 Stage 2 are now available.<br />
|}<br />
--><br />
<!-- <br />
###########################################<br />
## bwUniCluster: Training/Support section##<br />
###########################################<br />
--><br />
{| style=" background:#eeeefe; width:100%;" <br />
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support<br />
|-<br />
| <br />
* [[BwUniCluster2.0/First_Steps|Getting Started]]<br />
* [https://training.bwhpc.de E-Learning Courses]<br />
* [[BwUniCluster2.0/Support|Support]]<br />
* [[BwUniCluster2.0/FAQ|FAQ]]<br />
* Send [[:Category:Feedback|Feedback]] about Wiki pages<br />
|}<br />
<!-- <br />
###########################################<br />
## bwUniCluster: User Documentation ##<br />
###########################################<br />
--><br />
{| style=" background:#deffee; width:100%;"<br />
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | User Documentation<br />
|-<br />
|<br />
* Access: [[Registration/bwUniCluster|Registration]], [[Registration/Deregistration|Deregistration]], [[BwUniCluster2.0/Jupyter|Using Jupyter]], [[BwUniCluster2.0/Jupyter (Deutsch)|Using Jupyter (German)]]<br />
* [[BwUniCluster2.0/Login|Login]]<br />
* <br />
* [[BwUniCluster2.0/Hardware_and_Architecture|Hardware and Architecture]]<br />
** [[BwUniCluster2.0/Hardware_and_Architecture#File_Systems|File Systems and Workspaces]] <br />
* [[BwUniCluster2.0/Software|Cluster Specific Software]]<br />
** [[BwUniCluster2.0/Containers|Using Containers]]<br />
* [[BwUniCluster2.0/Slurm|Batch System]]<br />
** [[BwUniCluster2.0/Batch_Queues|Queues and interactive Jobs]]<br />
* [[BwUniCluster2.0/Maintenance|Operational Changes]]<br />
|}<br />
<!-- <br />
###########################################<br />
## bwUniCluster: Acknowledgement ##<br />
###########################################<br />
--><br />
{| style=" background:#e6e9eb; width:100%;"<br />
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Cluster Funding<br />
|-<br />
|<br />
* Please [[BwUniCluster2.0/Acknowledgement|acknowledge]] bwUniCluster 2.0 in your publications.<br />
|}</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0&diff=12781
BwUniCluster2.0
2024-05-24T08:46:53Z
<p>P Weisbrod: </p>
<hr />
<div><!-- Old text: check for permanent removal --><br />
<!--{| style="width: 100%; border-spacing: 5px;"<br />
| style="text-align:center; color:#000;vertical-align:middle;font-size:75%;" |<br />
[[File:BwUniCluster_2.0_Feb2020.jpg|center|border|550px|Close-up of bwUniCluster by Simon Raffeiner, Copyright: KIT (SCC)]] <br />
|-<br />
| style="text-align:center; color:#000;vertical-align:middle;" |<span style="font-size:80%">Close-up of bwUniCluster © KIT (Simon Raffeiner/SCC)</span><br />
|}<br />
<br />
On 17.03.2020, the Steinbuch Centre for Computing (SCC) at Karlsruhe Institute of Technology (KIT) commissioned a new parallel computer system called "bwUniCluster 2.0+GFB-HPC" as a state service within the bwHPC framework. The bwUniCluster 2.0 replaces the predecessor system [https://www.scc.kit.edu/dienste/bwUniCluster.php bwUniCluster] and also includes the additional compute nodes which were procured as an extension to the bwUniCluster in November 2016.<br />
<br />
The modern bwUniCluster 2.0 system consists of more than 840 SMP nodes with 64-bit Intel Xeon processors. It provides the universities of the state of Baden-Württemberg with general compute resources and can be used free of charge by the staff of all universities in Baden-Württemberg. Users who currently have access to bwUniCluster will automatically also have access to bwUniCluster 2.0. There is no need to apply for new entitlements or to re-register.<br />
<br />
<br />
--><br />
<!-- <br />
###########################################<br />
## Picture of bwUniCluster - right side ##<br />
###########################################<br />
--><br />
[[File:BwUniCluster_2.0_Feb2020_1024x423.jpg|right|frameless|thumb|alt=bwUniCluster2.0 |upright=1| bwUniCluster 2.0 ]]<br />
<br />
<br />
<!-- <br />
###########################################<br />
## About bwUniCluster ##<br />
###########################################<br />
<br />
The '''bwUniCluster 2.0''' is the joint high-performance computer system of Baden-Württemberg's Universities and Universities of Applied Sciences for '''general purpose and teaching''' and located at the Steinbuch Centre for Computing (SCC) at Karlsruhe Institute of Technology (KIT). The bwUniCluster 2.0 complements the four bwForClusters and their dedicated scientific areas.<br />
<br />
<br />
{| style="background:#FFCCCC; width:100%;"<br />
| '''The following issue is known:''' Due to the hardware configuration, there is currently an already known problem with OpenMPI on the nodes in the "multiple_il" partition. It manifests itself in the warning "No OpenFabrics connection schemes reported" when starting an MPI application and refers to the device "mlx5_2". This is an Ethernet port, which is not supposed to be used by OpenMPI. The warning is informative, we are working on suppressing this message.<br />
|}<br />
--><br />
<!-- <br />
###########################################<br />
## bwUniCluster: Maintenance Section ##<br />
###########################################<br />
## Comment out full section if there no upcoming maintenance<br />
--><br />
<br />
{| style=" background:#FEF4AB; width:100%;" <br />
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | Next maintenance<br />
|-<br />
| <br />
Due to regular maintenance work the HPC System bwUnicluster 2 will not be available from <br />
<br />
21.05.2024 at 08:30 AM until 24.05.2024 at 15:00 AM<br />
<br />
Please see the [[BwUniCluster2.0/Maintenance/2024-05|maintenance]] page for more information about planned upgrades and other changes<br />
|}<br />
<br />
<!-- <br />
###########################################<br />
## bwUniCluster: News section ##<br />
###########################################<br />
## Comment out full section if there no news<br />
--><br />
<!-- <br />
{| style=" background:#FEF4AB; width:100%;" <br />
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News<br />
|-<br />
| <br />
* 2022-11-25: All new nodes from bwUniCluster 2.0 Stage 2 are now available.<br />
|}<br />
--><br />
<!-- <br />
###########################################<br />
## bwUniCluster: Training/Support section##<br />
###########################################<br />
--><br />
{| style=" background:#eeeefe; width:100%;" <br />
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support<br />
|-<br />
| <br />
* [[BwUniCluster2.0/First_Steps|Getting Started]]<br />
* [https://training.bwhpc.de E-Learning Courses]<br />
* [[BwUniCluster2.0/Support|Support]]<br />
* [[BwUniCluster2.0/FAQ|FAQ]]<br />
* Send [[:Category:Feedback|Feedback]] about Wiki pages<br />
|}<br />
<!-- <br />
###########################################<br />
## bwUniCluster: User Documentation ##<br />
###########################################<br />
--><br />
{| style=" background:#deffee; width:100%;"<br />
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | User Documentation<br />
|-<br />
|<br />
* Access: [[Registration/bwUniCluster|Registration]], [[Registration/Deregistration|Deregistration]], [[BwUniCluster2.0/Jupyter|Using Jupyter]], [[BwUniCluster2.0/Jupyter (Deutsch)|Using Jupyter (German)]]<br />
* [[BwUniCluster2.0/Login|Login]]<br />
* <br />
* [[BwUniCluster2.0/Hardware_and_Architecture|Hardware and Architecture]]<br />
** [[BwUniCluster2.0/Hardware_and_Architecture#File_Systems|File Systems and Workspaces]] <br />
* [[BwUniCluster2.0/Software|Cluster Specific Software]]<br />
** [[BwUniCluster2.0/Containers|Using Containers]]<br />
* [[BwUniCluster2.0/Slurm|Batch System]]<br />
** [[BwUniCluster2.0/Batch_Queues|Queues and interactive Jobs]]<br />
* [[BwUniCluster2.0/Maintenance|Operational Changes]]<br />
|}<br />
<!-- <br />
###########################################<br />
## bwUniCluster: Acknowledgement ##<br />
###########################################<br />
--><br />
{| style=" background:#e6e9eb; width:100%;"<br />
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Cluster Funding<br />
|-<br />
|<br />
* Please [[BwUniCluster2.0/Acknowledgement|acknowledge]] bwUniCluster 2.0 in your publications.<br />
|}</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0&diff=12780
BwUniCluster2.0
2024-05-24T08:46:27Z
<p>P Weisbrod: </p>
<hr />
<div><!-- Old text: check for permanent removal --><br />
<!--{| style="width: 100%; border-spacing: 5px;"<br />
| style="text-align:center; color:#000;vertical-align:middle;font-size:75%;" |<br />
[[File:BwUniCluster_2.0_Feb2020.jpg|center|border|550px|Close-up of bwUniCluster by Simon Raffeiner, Copyright: KIT (SCC)]] <br />
|-<br />
| style="text-align:center; color:#000;vertical-align:middle;" |<span style="font-size:80%">Close-up of bwUniCluster © KIT (Simon Raffeiner/SCC)</span><br />
|}<br />
<br />
On 17.03.2020, the Steinbuch Centre for Computing (SCC) at Karlsruhe Institute of Technology (KIT) commissioned a new parallel computer system called "bwUniCluster 2.0+GFB-HPC" as a state service within the bwHPC framework. The bwUniCluster 2.0 replaces the predecessor system [https://www.scc.kit.edu/dienste/bwUniCluster.php bwUniCluster] and also includes the additional compute nodes which were procured as an extension to the bwUniCluster in November 2016.<br />
<br />
The modern bwUniCluster 2.0 system consists of more than 840 SMP nodes with 64-bit Intel Xeon processors. It provides the universities of the state of Baden-Württemberg with general compute resources and can be used free of charge by the staff of all universities in Baden-Württemberg. Users who currently have access to bwUniCluster will automatically also have access to bwUniCluster 2.0. There is no need to apply for new entitlements or to re-register.<br />
<br />
<br />
--><br />
<!-- <br />
###########################################<br />
## Picture of bwUniCluster - right side ##<br />
###########################################<br />
--><br />
[[File:BwUniCluster_2.0_Feb2020_1024x423.jpg|right|frameless|thumb|alt=bwUniCluster2.0 |upright=1| bwUniCluster 2.0 ]]<br />
<br />
<br />
<!-- <br />
###########################################<br />
## About bwUniCluster ##<br />
###########################################<br />
--><br />
The '''bwUniCluster 2.0''' is the joint high-performance computer system of Baden-Württemberg's Universities and Universities of Applied Sciences for '''general purpose and teaching''' and located at the Steinbuch Centre for Computing (SCC) at Karlsruhe Institute of Technology (KIT). The bwUniCluster 2.0 complements the four bwForClusters and their dedicated scientific areas.<br />
<br />
<br />
{| style="background:#FFCCCC; width:100%;"<br />
| '''The following issue is known:''' Due to the hardware configuration, there is currently an already known problem with OpenMPI on the nodes in the "multiple_il" partition. It manifests itself in the warning "No OpenFabrics connection schemes reported" when starting an MPI application and refers to the device "mlx5_2". This is an Ethernet port, which is not supposed to be used by OpenMPI. The warning is informative, we are working on suppressing this message.<br />
|}<br />
<!-- <br />
###########################################<br />
## bwUniCluster: Maintenance Section ##<br />
###########################################<br />
## Comment out full section if there no upcoming maintenance<br />
--><br />
<br />
{| style=" background:#FEF4AB; width:100%;" <br />
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | Next maintenance<br />
|-<br />
| <br />
Due to regular maintenance work the HPC System bwUnicluster 2 will not be available from <br />
<br />
21.05.2024 at 08:30 AM until 24.05.2024 at 15:00 AM<br />
<br />
Please see the [[BwUniCluster2.0/Maintenance/2024-05|maintenance]] page for more information about planned upgrades and other changes<br />
|}<br />
<br />
<!-- <br />
###########################################<br />
## bwUniCluster: News section ##<br />
###########################################<br />
## Comment out full section if there no news<br />
--><br />
<!-- <br />
{| style=" background:#FEF4AB; width:100%;" <br />
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News<br />
|-<br />
| <br />
* 2022-11-25: All new nodes from bwUniCluster 2.0 Stage 2 are now available.<br />
|}<br />
--><br />
<!-- <br />
###########################################<br />
## bwUniCluster: Training/Support section##<br />
###########################################<br />
--><br />
{| style=" background:#eeeefe; width:100%;" <br />
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support<br />
|-<br />
| <br />
* [[BwUniCluster2.0/First_Steps|Getting Started]]<br />
* [https://training.bwhpc.de E-Learning Courses]<br />
* [[BwUniCluster2.0/Support|Support]]<br />
* [[BwUniCluster2.0/FAQ|FAQ]]<br />
* Send [[:Category:Feedback|Feedback]] about Wiki pages<br />
|}<br />
<!-- <br />
###########################################<br />
## bwUniCluster: User Documentation ##<br />
###########################################<br />
--><br />
{| style=" background:#deffee; width:100%;"<br />
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | User Documentation<br />
|-<br />
|<br />
* Access: [[Registration/bwUniCluster|Registration]], [[Registration/Deregistration|Deregistration]], [[BwUniCluster2.0/Jupyter|Using Jupyter]], [[BwUniCluster2.0/Jupyter (Deutsch)|Using Jupyter (German)]]<br />
* [[BwUniCluster2.0/Login|Login]]<br />
* <br />
* [[BwUniCluster2.0/Hardware_and_Architecture|Hardware and Architecture]]<br />
** [[BwUniCluster2.0/Hardware_and_Architecture#File_Systems|File Systems and Workspaces]] <br />
* [[BwUniCluster2.0/Software|Cluster Specific Software]]<br />
** [[BwUniCluster2.0/Containers|Using Containers]]<br />
* [[BwUniCluster2.0/Slurm|Batch System]]<br />
** [[BwUniCluster2.0/Batch_Queues|Queues and interactive Jobs]]<br />
* [[BwUniCluster2.0/Maintenance|Operational Changes]]<br />
|}<br />
<!-- <br />
###########################################<br />
## bwUniCluster: Acknowledgement ##<br />
###########################################<br />
--><br />
{| style=" background:#e6e9eb; width:100%;"<br />
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Cluster Funding<br />
|-<br />
|<br />
* Please [[BwUniCluster2.0/Acknowledgement|acknowledge]] bwUniCluster 2.0 in your publications.<br />
|}</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0&diff=12779
BwUniCluster2.0
2024-05-24T08:46:03Z
<p>P Weisbrod: </p>
<hr />
<div><!-- Old text: check for permanent removal --><br />
<!--{| style="width: 100%; border-spacing: 5px;"<br />
| style="text-align:center; color:#000;vertical-align:middle;font-size:75%;" |<br />
[[File:BwUniCluster_2.0_Feb2020.jpg|center|border|550px|Close-up of bwUniCluster by Simon Raffeiner, Copyright: KIT (SCC)]] <br />
|-<br />
| style="text-align:center; color:#000;vertical-align:middle;" |<span style="font-size:80%">Close-up of bwUniCluster © KIT (Simon Raffeiner/SCC)</span><br />
|}<br />
<br />
On 17.03.2020, the Steinbuch Centre for Computing (SCC) at Karlsruhe Institute of Technology (KIT) commissioned a new parallel computer system called "bwUniCluster 2.0+GFB-HPC" as a state service within the bwHPC framework. The bwUniCluster 2.0 replaces the predecessor system [https://www.scc.kit.edu/dienste/bwUniCluster.php bwUniCluster] and also includes the additional compute nodes which were procured as an extension to the bwUniCluster in November 2016.<br />
<br />
The modern bwUniCluster 2.0 system consists of more than 840 SMP nodes with 64-bit Intel Xeon processors. It provides the universities of the state of Baden-Württemberg with general compute resources and can be used free of charge by the staff of all universities in Baden-Württemberg. Users who currently have access to bwUniCluster will automatically also have access to bwUniCluster 2.0. There is no need to apply for new entitlements or to re-register.<br />
<br />
<br />
--><br />
<!-- <br />
###########################################<br />
## Picture of bwUniCluster - right side ##<br />
###########################################<br />
--><br />
[[File:BwUniCluster_2.0_Feb2020_1024x423.jpg|right|frameless|thumb|alt=bwUniCluster2.0 |upright=1| bwUniCluster 2.0 ]]<br />
<br />
<br />
<!-- <br />
###########################################<br />
## About bwUniCluster ##<br />
###########################################<br />
--><br />
The '''bwUniCluster 2.0''' is the joint high-performance computer system of Baden-Württemberg's Universities and Universities of Applied Sciences for '''general purpose and teaching''' and located at the Steinbuch Centre for Computing (SCC) at Karlsruhe Institute of Technology (KIT). The bwUniCluster 2.0 complements the four bwForClusters and their dedicated scientific areas.<br />
<br />
<br />
{| style="background:#FFCCCC; width:100%;"<br />
| '''The following issue is known:''' Due to the hardware configuration, there is currently an already known problem with OpenMPI on the nodes in the "multiple_il" partition. It manifests itself in the warning "No OpenFabrics connection schemes reported" when starting an MPI application and refers to the device "mlx5_2". This is an Ethernet port, which is not supposed to be used by OpenMPI. The warning is informative, we are working on suppressing this message.<br />
|}<br />
<!-- <br />
###########################################<br />
## bwUniCluster: Maintenance Section ##<br />
###########################################<br />
## Comment out full section if there no upcoming maintenance<br />
#--><br />
#<br />
#{| style=" background:#FEF4AB; width:100%;" <br />
#| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | Next maintenance<br />
#|-<br />
#| <br />
#Due to regular maintenance work the HPC System bwUnicluster 2 will not be available from <br />
#<br />
#21.05.2024 at 08:30 AM until 24.05.2024 at 15:00 AM<br />
#<br />
#Please see the [[BwUniCluster2.0/Maintenance/2024-05|maintenance]] page for more information about planned upgrades and other changes<br />
#|}<br />
#<br />
#<!-- <br />
###########################################<br />
## bwUniCluster: News section ##<br />
###########################################<br />
## Comment out full section if there no news<br />
--><br />
<!-- <br />
{| style=" background:#FEF4AB; width:100%;" <br />
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News<br />
|-<br />
| <br />
* 2022-11-25: All new nodes from bwUniCluster 2.0 Stage 2 are now available.<br />
|}<br />
--><br />
<!-- <br />
###########################################<br />
## bwUniCluster: Training/Support section##<br />
###########################################<br />
--><br />
{| style=" background:#eeeefe; width:100%;" <br />
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support<br />
|-<br />
| <br />
* [[BwUniCluster2.0/First_Steps|Getting Started]]<br />
* [https://training.bwhpc.de E-Learning Courses]<br />
* [[BwUniCluster2.0/Support|Support]]<br />
* [[BwUniCluster2.0/FAQ|FAQ]]<br />
* Send [[:Category:Feedback|Feedback]] about Wiki pages<br />
|}<br />
<!-- <br />
###########################################<br />
## bwUniCluster: User Documentation ##<br />
###########################################<br />
--><br />
{| style=" background:#deffee; width:100%;"<br />
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | User Documentation<br />
|-<br />
|<br />
* Access: [[Registration/bwUniCluster|Registration]], [[Registration/Deregistration|Deregistration]], [[BwUniCluster2.0/Jupyter|Using Jupyter]], [[BwUniCluster2.0/Jupyter (Deutsch)|Using Jupyter (German)]]<br />
* [[BwUniCluster2.0/Login|Login]]<br />
* <br />
* [[BwUniCluster2.0/Hardware_and_Architecture|Hardware and Architecture]]<br />
** [[BwUniCluster2.0/Hardware_and_Architecture#File_Systems|File Systems and Workspaces]] <br />
* [[BwUniCluster2.0/Software|Cluster Specific Software]]<br />
** [[BwUniCluster2.0/Containers|Using Containers]]<br />
* [[BwUniCluster2.0/Slurm|Batch System]]<br />
** [[BwUniCluster2.0/Batch_Queues|Queues and interactive Jobs]]<br />
* [[BwUniCluster2.0/Maintenance|Operational Changes]]<br />
|}<br />
<!-- <br />
###########################################<br />
## bwUniCluster: Acknowledgement ##<br />
###########################################<br />
--><br />
{| style=" background:#e6e9eb; width:100%;"<br />
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Cluster Funding<br />
|-<br />
|<br />
* Please [[BwUniCluster2.0/Acknowledgement|acknowledge]] bwUniCluster 2.0 in your publications.<br />
|}</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance/2024-05&diff=12778
BwUniCluster2.0/Maintenance/2024-05
2024-05-24T08:36:33Z
<p>P Weisbrod: </p>
<hr />
<div>The following changes were introduced during the maintenance interval between on 21.05.2024 (Tuesday) 08:30 and 24.05.2024 (Friday) 15:00.<br />
<br />
We are not planning to change the host key of the system. You should not receive any warnings by your SSH client(s), but if there should be a warning or if you want to check that you are connecting to the correct system, you can verify the key hashes using the following list:<br />
<br />
{|class="wikitable"<br />
! Algorithm<br />
! Hash (SHA256)<br />
! Hash (MD5)<br />
|-<br />
|RSA<br />
|p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|59:2a:67:44:4a:d7:89:6c:c0:0d:74:ba:3c:c4:63:6d<br />
|-<br />
|ECDSA<br />
|k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|85:d4:d9:97:e0:f0:43:30:6e:66:8e:d0:b6:9b:85:d1<br />
|-<br />
|ED25519<br />
|yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|42:d2:0d:ab:87:48:fc:1d:5d:b3:7c:bf:22:c3:5f:b7<br />
|}<br />
<br />
= Hardware =<br />
<br />
* All firmware versions on all components were upgraded.<br />
<br />
= Operating system =<br />
<br />
* The operating system was upgraded from RHEL 8.6 EUS to RHEL 8.8 EUS. We recommend to re-compile all applications after the upgrade.<br />
<br />
* The Mellanox OFED InfiniBand Stack was updated.<br />
<br />
= Compilers, Libaries and Runtime Environments =<br />
* Certain older compilers are deprecated (hidden, aka module load compiler/gnu/9.3 will only be available as module load compiler/gnu/.9.3)<br />
* Intel OneAPI 2023 and 2024 are now available<br />
* Intel parallel studio 2020 (compiler 19.1, impi 2020, mkl 2020) are deprecated<br />
<br />
= Userspace tools =<br />
<br />
<br />
= Software Modules =<br />
<br />
<br />
= Batch system =<br />
<br />
* The Slurm version was upgraded to version 23.11.5<br />
<br />
= Storage =<br />
<br />
* Lustre client, BeeGFS client and Spectrum Scale client were updated.<br />
<br />
= Graphics stack =<br />
<br />
* The NVIDIA driver was upgraded to 550.54.15<br />
* CUDA 12.2 is now the default version, 12.4 is available<br />
* NVIDIA toolkit: default is now 23.9<br />
<br />
= Containers =<br />
<br />
= JupyterHub =<br />
<br />
* Jupyterhub was upgraded to 4.1.3</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance/2024-05&diff=12777
BwUniCluster2.0/Maintenance/2024-05
2024-05-24T08:35:33Z
<p>P Weisbrod: </p>
<hr />
<div>The following changes were introduced during the maintenance interval between on 21.05.2024 (Tuesday) 08:30 and 24.05.2024 (Friday) 15:00.<br />
<br />
We are not planning to change the host key of the system. You should not receive any warnings by your SSH client(s), but if there should be a warning or if you want to check that you are connecting to the correct system, you can verify the key hashes using the following list:<br />
<br />
{|class="wikitable"<br />
! Algorithm<br />
! Hash (SHA256)<br />
! Hash (MD5)<br />
|-<br />
|RSA<br />
|p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|59:2a:67:44:4a:d7:89:6c:c0:0d:74:ba:3c:c4:63:6d<br />
|-<br />
|ECDSA<br />
|k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|85:d4:d9:97:e0:f0:43:30:6e:66:8e:d0:b6:9b:85:d1<br />
|-<br />
|ED25519<br />
|yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|42:d2:0d:ab:87:48:fc:1d:5d:b3:7c:bf:22:c3:5f:b7<br />
|}<br />
<br />
= Hardware =<br />
<br />
* All firmware versions on all components were upgraded.<br />
<br />
= Operating system =<br />
<br />
* The operating system was upgraded from RHEL 8.6 EUS to RHEL 8.8 EUS. We recommend to re-compile all applications after the upgrade.<br />
<br />
* The Mellanox OFED InfiniBand Stack was updated.<br />
<br />
= Compilers, Libaries and Runtime Environments =<br />
* Certain older compilers are deprecated (hidden, aka module load compiler/gnu/9.3 will only be available as module load compiler/gnu/.9.3)<br />
* Intel OneAPI 2023 and 2024 are now available<br />
* Intel parallel studio 2020 (compiler 19.1, impi 2020, mkl 2020) are deprecated<br />
<br />
= Userspace tools =<br />
<br />
<br />
= Software Modules =<br />
<br />
<br />
= Batch system =<br />
<br />
* The Slurm version was upgraded to version 23.11.5<br />
<br />
= Storage =<br />
<br />
* Lustre client, BeeGFS client and Spectrum Scale client were updated.<br />
<br />
= Graphics stack =<br />
<br />
* The NVIDIA driver will be upgraded.<br />
* CUDA 12.2 is now the default version, 12.4 is available<br />
* NVIDIA toolkit: default is now 23.9<br />
<br />
= Containers =<br />
<br />
= JupyterHub =<br />
<br />
* Jupyterhub was upgraded to 4.1.3</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance/2024-05&diff=12742
BwUniCluster2.0/Maintenance/2024-05
2024-05-13T14:43:33Z
<p>P Weisbrod: </p>
<hr />
<div>The following changes were introduced during the maintenance interval between on 21.05.2024 (Tuesday) 08:30 and 24.05.2024 (Friday) 15:00.<br />
<br />
We are not planning to change the host key of the system. You should not receive any warnings by your SSH client(s), but if there should be a warning or if you want to check that you are connecting to the correct system, you can verify the key hashes using the following list:<br />
<br />
{|class="wikitable"<br />
! Algorithm<br />
! Hash (SHA256)<br />
! Hash (MD5)<br />
|-<br />
|RSA<br />
|p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|59:2a:67:44:4a:d7:89:6c:c0:0d:74:ba:3c:c4:63:6d<br />
|-<br />
|ECDSA<br />
|k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|85:d4:d9:97:e0:f0:43:30:6e:66:8e:d0:b6:9b:85:d1<br />
|-<br />
|ED25519<br />
|yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|42:d2:0d:ab:87:48:fc:1d:5d:b3:7c:bf:22:c3:5f:b7<br />
|}<br />
<br />
= Hardware =<br />
<br />
* All firmware versions on all components will be upgraded.<br />
<br />
= Operating system =<br />
<br />
* The operating system will be upgraded from RHEL 8.6 EUS to RHEL 8.8 EUS. We recommend to re-compile all applications after the upgrade.<br />
<br />
* The Mellanox OFED InfiniBand Stack will be updated.<br />
<br />
= Compilers, Libaries and Runtime Environments =<br />
* Certain older compilers will be deprecated (hidden, aka module load compiler/gnu/9.3 will only be available as module load compiler/gnu/.9.3)<br />
* Intel OneAPI 2023 and 2024 will be available<br />
* Intel parallel studio 2020 (compiler 19.1, impi 2020, mkl 2020) will be removed<br />
<br />
= Userspace tools =<br />
<br />
<br />
= Software Modules =<br />
<br />
<br />
= Batch system =<br />
<br />
* The Slurm version will be upgraded to version 23.11.5<br />
<br />
= Storage =<br />
<br />
* Lustre client, BeeGFS client and Spectrum Scale client will be updated.<br />
<br />
= Graphics stack =<br />
<br />
* The NVIDIA driver will be upgraded.<br />
* CUDA 12.2 will be the default version, 12.4 will be available<br />
* NVIDIA toolkit: default will switch from 21.2 to 23.9<br />
<br />
= Containers =<br />
<br />
= JupyterHub =<br />
<br />
* Jupyterhub will be upgraded</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance/2024-05&diff=12741
BwUniCluster2.0/Maintenance/2024-05
2024-05-13T14:42:13Z
<p>P Weisbrod: </p>
<hr />
<div>The following changes were introduced during the maintenance interval between on 21.05.2024 (Tuesday) 08:30 and 24.05.2024 (Friday) 15:00.<br />
<br />
We are not planning to change the host key of the system. You should not receive any warnings by your SSH client(s), but if there should be a warning or if you want to check that you are connecting to the correct system, you can verify the key hashes using the following list:<br />
<br />
{|class="wikitable"<br />
! Algorithm<br />
! Hash (SHA256)<br />
! Hash (MD5)<br />
|-<br />
|RSA<br />
|p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|59:2a:67:44:4a:d7:89:6c:c0:0d:74:ba:3c:c4:63:6d<br />
|-<br />
|ECDSA<br />
|k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|85:d4:d9:97:e0:f0:43:30:6e:66:8e:d0:b6:9b:85:d1<br />
|-<br />
|ED25519<br />
|yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|42:d2:0d:ab:87:48:fc:1d:5d:b3:7c:bf:22:c3:5f:b7<br />
|}<br />
<br />
= Hardware =<br />
<br />
* All firmware versions on all components will be upgraded.<br />
<br />
= Operating system =<br />
<br />
* The operating system will be upgraded from RHEL 8.6 EUS to RHEL 8.8 EUS. We recommend to re-compile all applications after the upgrade.<br />
<br />
* The Mellanox OFED InfiniBand Stack will be updated.<br />
<br />
= Compilers, Libaries and Runtime Environments =<br />
Certain older compilers will be deprecated (hidden, aka module load compiler/gnu/9.3 will only be available as module load compiler/gnu/.9.3)<br />
Intel OneAPI 2023 and 2024 will be available<br />
Intel parallel studio 2020 (compiler 19.1, impi 2020, mkl 2020) will be removed<br />
<br />
= Userspace tools =<br />
<br />
<br />
= Software Modules =<br />
<br />
<br />
= Batch system =<br />
<br />
* The Slurm version will be upgraded to version 23.11.5<br />
<br />
= Storage =<br />
<br />
* Lustre client, BeeGFS client and Spectrum Scale client will be updated.<br />
<br />
= Graphics stack =<br />
<br />
* The NVIDIA driver will be upgraded.<br />
* CUDA 12.2 will be the default version, 12.4 will be available<br />
<br />
= Containers =<br />
<br />
= JupyterHub =<br />
<br />
* Jupyterhub will be upgraded</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0&diff=12676
BwUniCluster2.0
2024-04-03T13:37:17Z
<p>P Weisbrod: </p>
<hr />
<div><!-- Old text: check for permanent removal --><br />
<!--{| style="width: 100%; border-spacing: 5px;"<br />
| style="text-align:center; color:#000;vertical-align:middle;font-size:75%;" |<br />
[[File:BwUniCluster_2.0_Feb2020.jpg|center|border|550px|Close-up of bwUniCluster by Simon Raffeiner, Copyright: KIT (SCC)]] <br />
|-<br />
| style="text-align:center; color:#000;vertical-align:middle;" |<span style="font-size:80%">Close-up of bwUniCluster © KIT (Simon Raffeiner/SCC)</span><br />
|}<br />
<br />
On 17.03.2020, the Steinbuch Centre for Computing (SCC) at Karlsruhe Institute of Technology (KIT) commissioned a new parallel computer system called "bwUniCluster 2.0+GFB-HPC" as a state service within the bwHPC framework. The bwUniCluster 2.0 replaces the predecessor system [https://www.scc.kit.edu/dienste/bwUniCluster.php bwUniCluster] and also includes the additional compute nodes which were procured as an extension to the bwUniCluster in November 2016.<br />
<br />
The modern bwUniCluster 2.0 system consists of more than 840 SMP nodes with 64-bit Intel Xeon processors. It provides the universities of the state of Baden-Württemberg with general compute resources and can be used free of charge by the staff of all universities in Baden-Württemberg. Users who currently have access to bwUniCluster will automatically also have access to bwUniCluster 2.0. There is no need to apply for new entitlements or to re-register.<br />
<br />
<br />
--><br />
<!-- <br />
###########################################<br />
## Picture of bwUniCluster - right side ##<br />
###########################################<br />
--><br />
[[File:BwUniCluster_2.0_Feb2020_1024x423.jpg|right|frameless|thumb|alt=bwUniCluster2.0 |upright=1| bwUniCluster 2.0 ]]<br />
<br />
<br />
<!-- <br />
###########################################<br />
## About bwUniCluster ##<br />
###########################################<br />
--><br />
The '''bwUniCluster 2.0''' is the joint high-performance computer system of Baden-Württemberg's Universities and Universities of Applied Sciences for '''general purpose and teaching''' and located at the Steinbuch Centre for Computing (SCC) at Karlsruhe Institute of Technology (KIT). The bwUniCluster 2.0 complements the four bwForClusters and their dedicated scientific areas.<br />
<br />
<br />
{| style="background:#FFCCCC; width:100%;"<br />
| '''The following issue is known:''' Due to the hardware configuration, there is currently an already known problem with OpenMPI on the nodes in the "multiple_il" partition. It manifests itself in the warning "No OpenFabrics connection schemes reported" when starting an MPI application and refers to the device "mlx5_2". This is an Ethernet port, which is not supposed to be used by OpenMPI. The warning is informative, we are working on suppressing this message.<br />
|}<br />
<!-- <br />
###########################################<br />
## bwUniCluster: Maintenance Section ##<br />
###########################################<br />
## Comment out full section if there no upcoming maintenance<br />
--><br />
<br />
{| style=" background:#FEF4AB; width:100%;" <br />
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | Next maintenance<br />
|-<br />
| <br />
Due to regular maintenance work the HPC System bwUnicluster 2 will not be available from <br />
<br />
21.05.2024 at 08:30 AM until 24.05.2024 at 15:00 AM<br />
<br />
Please see the [[BwUniCluster2.0/Maintenance/2024-05|maintenance]] page for more information about planned upgrades and other changes<br />
|}<br />
<br />
<!-- <br />
###########################################<br />
## bwUniCluster: News section ##<br />
###########################################<br />
## Comment out full section if there no news<br />
--><br />
<!-- <br />
{| style=" background:#FEF4AB; width:100%;" <br />
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News<br />
|-<br />
| <br />
* 2022-11-25: All new nodes from bwUniCluster 2.0 Stage 2 are now available.<br />
|}<br />
--><br />
<!-- <br />
###########################################<br />
## bwUniCluster: Training/Support section##<br />
###########################################<br />
--><br />
{| style=" background:#eeeefe; width:100%;" <br />
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support<br />
|-<br />
| <br />
* [[BwUniCluster2.0/First_Steps|Getting Started]]<br />
* [https://training.bwhpc.de E-Learning Courses]<br />
* [[BwUniCluster2.0/Support|Support]]<br />
* [[BwUniCluster2.0/FAQ|FAQ]]<br />
* Send [[:Category:Feedback|Feedback]] about Wiki pages<br />
|}<br />
<!-- <br />
###########################################<br />
## bwUniCluster: User Documentation ##<br />
###########################################<br />
--><br />
{| style=" background:#deffee; width:100%;"<br />
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | User Documentation<br />
|-<br />
|<br />
* Access: [[Registration/bwUniCluster|Registration]], [[Registration/Deregistration|Deregistration]], [[BwUniCluster2.0/Jupyter|Using Jupyter]], [[BwUniCluster2.0/Jupyter (Deutsch)|Using Jupyter (German)]]<br />
* [[BwUniCluster2.0/Login|Login]]<br />
* <br />
* [[BwUniCluster2.0/Hardware_and_Architecture|Hardware and Architecture]]<br />
** [[BwUniCluster2.0/Hardware_and_Architecture#File_Systems|File Systems and Workspaces]] <br />
* [[BwUniCluster2.0/Software|Cluster Specific Software]]<br />
** [[BwUniCluster2.0/Containers|Using Containers]]<br />
* [[BwUniCluster2.0/Slurm|Batch System]]<br />
** [[BwUniCluster2.0/Batch_Queues|Queues and interactive Jobs]]<br />
* [[BwUniCluster2.0/Maintenance|Operational Changes]]<br />
|}<br />
<!-- <br />
###########################################<br />
## bwUniCluster: Acknowledgement ##<br />
###########################################<br />
--><br />
{| style=" background:#e6e9eb; width:100%;"<br />
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Cluster Funding<br />
|-<br />
|<br />
* Please [[BwUniCluster2.0/Acknowledgement|acknowledge]] bwUniCluster 2.0 in your publications.<br />
|}</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance/2024-05&diff=12675
BwUniCluster2.0/Maintenance/2024-05
2024-04-03T13:34:59Z
<p>P Weisbrod: Created page with "The following changes were introduced during the maintenance interval between on 21.05.2024 (Tuesday) 08:30 and 24.05.2024 (Friday) 15:00. We are not planning to change the h..."</p>
<hr />
<div>The following changes were introduced during the maintenance interval between on 21.05.2024 (Tuesday) 08:30 and 24.05.2024 (Friday) 15:00.<br />
<br />
We are not planning to change the host key of the system. You should not receive any warnings by your SSH client(s), but if there should be a warning or if you want to check that you are connecting to the correct system, you can verify the key hashes using the following list:<br />
<br />
{|class="wikitable"<br />
! Algorithm<br />
! Hash (SHA256)<br />
! Hash (MD5)<br />
|-<br />
|RSA<br />
|p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|59:2a:67:44:4a:d7:89:6c:c0:0d:74:ba:3c:c4:63:6d<br />
|-<br />
|ECDSA<br />
|k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|85:d4:d9:97:e0:f0:43:30:6e:66:8e:d0:b6:9b:85:d1<br />
|-<br />
|ED25519<br />
|yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|42:d2:0d:ab:87:48:fc:1d:5d:b3:7c:bf:22:c3:5f:b7<br />
|}<br />
<br />
= Hardware =<br />
<br />
* All firmware versions on all components will be upgraded.<br />
<br />
= Operating system =<br />
<br />
* The operating system will be upgraded from RHEL 8.6 EUS to RHEL 8.8 EUS. We recommend to re-compile all applications after the upgrade.<br />
<br />
* The Mellanox OFED InfiniBand Stack will be updated.<br />
<br />
= Compilers, Libaries and Runtime Environments =<br />
<br />
<br />
= Userspace tools =<br />
<br />
<br />
= Software Modules =<br />
<br />
<br />
= Batch system =<br />
<br />
* The Slurm version will be upgraded to version 23.11.x<br />
<br />
= Storage =<br />
<br />
* Lustre client, BeeGFS client and Spectrum Scale client will be updated.<br />
<br />
= Graphics stack =<br />
<br />
* The NVIDIA driver will be upgraded.<br />
<br />
= Containers =<br />
<br />
= JupyterHub =<br />
<br />
* Jupyterhub will be upgraded</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance&diff=12674
BwUniCluster2.0/Maintenance
2024-04-03T13:26:12Z
<p>P Weisbrod: </p>
<hr />
<div>'''2024'''<br />
<br />
* [[BwUniCluster2.0/Maintenance/2024-05]] from 21.05.2024 to 24.05.2024<br />
<br />
'''2023'''<br />
<br />
* [[BwUniCluster2.0/Maintenance/2023-03]] from 20.03.2023 to 24.03.2023<br />
<br />
'''2022'''<br />
<br />
* [[BwUniCluster2.0/Maintenance/2022-11]] from 07.11.2022 to 10.11.2022<br />
<br />
* [[BwUniCluster2.0/Maintenance/2022-03]] from 28.03.2022 to 31.03.2022<br />
<br />
'''2021'''<br />
<br />
* [[BwUniCluster2.0/Maintenance/2021-10]] from 11.10.2021 to 15.10.2021<br />
<br />
'''2020'''<br />
<br />
* [[BwUniCluster2.0/Maintenance/2020-10]] from 06.10.2020 to 13.10.2020<br />
<br />
<br />
=== Maintenance records of retired bwUniCluster 1.0 ===<br />
<br />
<br />
[[Category:BwUniCluster 2.0]]<br />
<br />
'''2019'''<br />
<br />
* [[BwUniCluster/Maintenance/2019-02]] from 02.02.2019 to 08.02.2019<br />
<br />
'''2017'''<br />
<br />
* [[BwUniCluster/Maintenance/2017-05]] from 02.05.2017 to 02.05.2017<br />
* [[BwUniCluster/Maintenance/2017-03]] from 20.03.2017 to 21.03.2017<br />
<br />
<br />
'''2016'''<br />
<br />
* [[BwUniCluster/Maintenance/2016-10]] from 17.10.2016 to 21.10.2016</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Hardware_and_Architecture&diff=12519
BwUniCluster2.0/Hardware and Architecture
2024-01-04T09:49:22Z
<p>P Weisbrod: </p>
<hr />
<div>= Architecture of bwUniCluster 2.0 =<br />
<br />
The bwUniCluster 2.0 is a parallel computer with distributed memory. Each node of system consists of at least two Intel Xeon processor, local memory, disks, network adapters and optionally accelerators (NVIDIA Tesla V100, A100 or H100). All nodes are connected by a fast InfiniBand interconnect. In addition the file system<br />
Lustre, that is connected by coupling the InfiniBand of the file server with the InfiniBand<br />
switch of the compute cluster, is added to bwUniCluster 2.0 to provide a fast and scalable<br />
parallel file system.<br />
<br />
The operating system on each node is Red Hat Enterprise Linux (RHEL) 8.4. A number of additional software packages like e.g. SLURM have been installed on top. Some of these components are of special interest to end users and are briefly<br />
discussed in this document. Others which are of greater importance to system<br />
administrators will not be covered by this document.<br />
<br />
The individual nodes of the system may act in different roles. According to the services supplied by the nodes, they are separated into disjoint groups. From an end users point of view the different groups of nodes are login nodes, compute nodes, file server nodes and administrative server nodes.<br />
<br />
'''Login Nodes'''<br />
<br />
The login nodes are the only nodes that are directly accessible by end users. These nodes<br />
are used for interactive login, file management, program development and interactive pre-<br />
and postprocessing. Three nodes are dedicated to this service but they are all accessible via<br />
one address and a DNS round-robin alias distributes the login sessions to the<br />
different login nodes.<br />
<br />
'''Compute Node'''<br />
<br />
The majority of nodes are compute nodes which are managed by a batch system. Users <br />
submit their jobs to the SLURM batch system and a job is executed when the required resources become available (depending on its fair-share priority).<br />
<br />
'''File Server Nodes'''<br />
<br />
The hardware of the parallel file system Lustre incorporates some file server nodes; the file<br />
system Lustre is connected by coupling the InfiniBand of the file server with the independent InfiniBand switch of the compute cluster. In addition to shared file space there is also local storage on the disks of each node (for details see chapter "File Systems").<br />
<br />
'''Administrative Server Nodes'''<br />
<br />
Some other nodes are delivering additional services like resource management, external<br />
network connection, administration etc. These nodes can be accessed directly by system administrators only.<br />
<br />
= Components of bwUniCluster =<br />
<br />
{| class="wikitable"<br />
|- <br />
! style="width:9%"|<br />
! style="width:13%"| Compute nodes "Thin"<br />
! style="width:13%"| Compute nodes "HPC"<br />
! style="width:13%"| Compute nodes "IceLake"<br />
! style="width:13%"| Compute nodes "Fat"<br />
! style="width:13%"| GPU x4<br />
! style="width:13%"| GPU x8<br />
! style="width:13%"| IceLake + GPU x4<br />
! style="width:13%"| Login<br />
|-<br />
!scope="column"| Number of nodes<br />
| 200 + 60<br />
| 260<br />
| 272<br />
| 6<br />
| 14<br />
| 10<br />
| 15<br />
| 3<br />
|-<br />
!scope="column"| Processors<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Platinum 8358<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6248<br />
| Intel Xeon Platinum 8358<br />
|-<br />
!scope="column"| Number of sockets<br />
| 2<br />
| 2<br />
| 2<br />
| 4<br />
| 2<br />
| 2<br />
| 2<br />
| 2<br />
|-<br />
!scope="column"| Processor frequency (GHz)<br />
| 2.1 Ghz<br />
| 2.1 Ghz<br />
| 2.6 Ghz<br />
| 2.1 Ghz<br />
| 2.1 Ghz<br />
| 2.6 Ghz<br />
| 2.5 Ghz<br />
| <br />
|-<br />
!scope="column"| Total number of cores<br />
| 40<br />
| 40<br />
| 64<br />
| 80<br />
| 40<br />
| 40<br />
| 64<br />
| 40<br />
|-<br />
!scope="column"| Main memory<br />
| 96 GB / 192 GB<br />
| 96 GB<br />
| 256 GB<br />
| 3 TB<br />
| 384 GB<br />
| 768 GB<br />
| 512 GB<br />
| 384 GB<br />
|-<br />
!scope="column"| Local SSD<br />
| 960 GB SATA<br />
| 960 GB SATA<br />
| 1,8 TB NVMe<br />
| 4,8 TB NVMe<br />
| 3,2 TB NVMe<br />
| 15 TB NVMe<br />
| 6,4 TB NVMe<br />
| <br />
|-<br />
!scope="column"| Accelerators<br />
| -<br />
| -<br />
| -<br />
| -<br />
| 4x NVIDIA Tesla V100<br />
| 8x NVIDIA Tesla V100<br />
| 4x NVIDIA A100 / 4x NVIDIA H100<br />
| -<br />
|-<br />
!scope="column"| Accelerator memory<br />
| -<br />
| -<br />
| -<br />
| -<br />
| 32 GB<br />
| 32 GB<br />
| 80 GB / 94 GB<br />
| -<br />
|-<br />
!scope="column"| Interconnect<br />
| IB HDR100 (blocking)<br />
| IB HDR100<br />
| IB HDR200<br />
| IB HDR<br />
| IB HDR<br />
| IB HDR<br />
| IB HDR200<br />
| IB HDR100 (blocking)<br />
|}<br />
Table 1: Properties of the nodes<br />
<br />
= File Systems =<br />
<br />
On bwUniCluster 2.0 the parallel file system Lustre is used for most globally visible user data. Lustre is open source and Lustre solutions and support are available from different vendors. Nowadays, most of the biggest HPC systems are using Lustre. An initial home directory on a Lustre file system is created automatically after account activation, and the environment variable $HOME holds its name. Users can create so-called workspaces on another Lustre file system for non-permanent data with temporary lifetime. There is another workspace file system based on flash storage for special requirements available.<br />
<br />
Within a batch job further file systems are available: <br />
* The directory $TMPDIR is only available and visible on the local node. It is located on fast SSD storage devices.<br />
* On request a parallel on-demand file system (BeeOND) is created which uses the SSDs of the nodes which were allocated to the batch job.<br />
* On request the external LSDF Online Storage is mounted on the nodes which were allocated to the batch job. This file system is based on the parallel file system Spectrum Scale. <br />
<br />
Some of the characteristics of the file systems are shown in Table 2.<br />
<br />
{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"<br />
|- style="width:20%;height=20px; text-align:left;padding:3px"<br />
! style="background-color:#AAA;padding:3px"| Property<br />
! style="background-color:#AAA;padding:3px"| $TMPDIR<br />
! style="background-color:#AAA;padding:3px"| BeeOND<br />
! style="background-color:#AAA;padding:3px"| $HOME<br />
! style="background-color:#AAA;padding:3px"| Workspace<br />
! style="background-color:#AAA;padding:3px"| Workspace <br> on flash<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Visibility <br />
| style="height=20px; text-align:left;padding:3px"| local node<br />
| style="height=20px; text-align:left;padding:3px"| nodes of batch job<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Lifetime <br />
| style="height=20px; text-align:left;padding:3px"| batch job runtime<br />
| style="height=20px; text-align:left;padding:3px"| batch job runtime<br />
| style="height=20px; text-align:left;padding:3px"| permanent<br />
| style="height=20px; text-align:left;padding:3px"| max. 240 days<br />
| style="height=20px; text-align:left;padding:3px"| max. 240 days<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Disk space <br />
| style="height=20px; text-align:left;padding:3px"| 960 GB - 6.4 TB <br> details see table 1<br />
| style="height=20px; text-align:left;padding:3px"| n*750 GB<br />
| style="height=20px; text-align:left;padding:3px"| 1.2 PiB<br />
| style="height=20px; text-align:left;padding:3px"| 4.1 PiB<br />
| style="height=20px; text-align:left;padding:3px"| 236 TiB<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Capacity Quotas<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 1 TiB per user, for <br> MA users 256 GiB <br> also per organization<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 40 TiB per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 1 TiB per user<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Inode Quotas<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 10 million per user <br> for MA users 2.5 million<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 30 million per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 5 million per user<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Backup<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Read perf./node<br />
| style="height=20px; text-align:left;padding:3px"| 500 MB/s - 6 GB/s <br> depends on type of local SSD / job queue: <br> 520 MB/s @ single / multiple <br> 800 MB/s @ multiple_e <br> 6600 MB/s @ fat <br> 6500 MB/s @ gpu_4 <br> 6500 MB/s @ gpu_8<br />
| style="height=20px; text-align:left;padding:3px"| 400 MB/s - 500 MB/s<br> depends on type of local SSDs / job queue: <br> 500 MB/s @ multiple <br> 400 MB/s @ multiple_e<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Write perf./node<br />
| style="height=20px; text-align:left;padding:3px"| 500 MB/s - 4 GB/s <br> depends on type of local SSD / job queue: <br> 500 MB/s @ single / multiple <br> 650 MB/s @ multiple_e <br> 2900 MB/s @ fat <br> 2090 MB/s @ gpu_4 <br> 4060 MB/s @ gpu_8<br />
| style="height=20px; text-align:left;padding:3px"| 250 MB/s - 350 MB/s <br> depends on type of local SSDs / job queue: <br> 350 MB/s @ multiple <br> 250 MB/s @ multiple_e<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Total read perf.<br />
| style="height=20px; text-align:left;padding:3px"| n*500-6000 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| n*400-500 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| 18 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 54 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 45 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Total write perf.<br />
| style="height=20px; text-align:left;padding:3px"| n*500-4000 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| n*250-350 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| 18 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 54 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 38 GB/s<br />
|}<br />
---------------------------------------------------------------------------------------------------------<br />
global: all nodes of UniCluster access the same file system;<br />
local: each node has its own file system;<br />
permanent: files are stored permanently;<br />
batch job: files are removed at end of the batch job.<br />
---------------------------------------------------------------------------------------------------------<br />
Table 2: Properties of the file systems<br />
<br />
== Selecting the appropriate file system ==<br />
<br />
In general, you should separate your data and store it on the appropriate file system.<br />
Permanently needed data like software or important results should be stored below $HOME<br />
but capacity restrictions (quotas) apply. In case you accidentally deleted data on $HOME<br />
you can usually restore it from backup. Permanent data which is not needed for months<br />
or exceeds the capacity restrictions should be sent to the LSDF Online Storage <br />
or to the archive and deleted from the file systems. Temporary data which is only needed on a single<br />
node and which does not exceed the disk space shown in the table above should be stored<br />
below $TMPDIR. Data which is read many times on a single node, e.g. if you are doing AI training, <br />
should be copied to $TMPDIR and read from there. Temporary data which is used from many nodes <br />
of your batch job and which is only needed during job runtime should be stored on a <br />
parallel on-demand file system. Temporary data which can be recomputed or which is the <br />
result of one job and input for another job should be stored in workspaces. The lifetime <br />
of data in workspaces is limited and depends on the lifetime of the workspace which can be <br />
several months.<br />
<br />
For further details please check the chapters below.<br />
<br />
== $HOME ==<br />
<br />
The home directories of bwUniCluster 2.0 (uc2) users are located in the parallel file system Lustre.<br />
You have access to your home directory from all nodes of uc2. A regular backup of these directories <br />
to tape archive is done automatically. The directory $HOME is used to hold those files that are<br />
permanently used like source codes, configuration files, executable programs etc. <br />
<br />
On uc2 there is a default user quota limit of 1 TiB and 10 million inodes (files and directories) per user. <br />
For users of University of Mannheim the limit is 256 GiB and 2.5 million inodes.<br />
You can check your current usage and limits with the command<br />
<pre><br />
$ lfs quota -uh $(whoami) $HOME<br />
</pre><br />
In addition to the user limit there is a limit of your organization (e.g. university) which depends on the financial share. This limit is enforced with so-called Lustre project quotas. You can show the current usage and limits of your organization with the following command:<br />
<pre><br />
lfs quota -ph $(grep $(echo $HOME | sed -e "s|/[^/]*/[^/]*$||") /pfs/data5/project_ids.txt | cut -f 1 -d\ ) $HOME<br />
</pre><br />
<br />
== Workspaces ==<br />
<br />
On uc2 workspaces can be used to store large non-permanent data sets, e.g. restart files or output<br />
data that has to be post-processed. The file system used for workspaces is also the parallel file system Lustre. This file system is especially designed for parallel access and for a high throughput to large<br />
files. It is able to provide high data transfer rates of up to 54 GB/s write and read performance when data access is parallel. <br />
<br />
Workspaces have a lifetime and the data on a workspace expires as a whole after a fixed period. The maximum lifetime of a workspace is 60 days, but it can be renewed at the end of that period 3 times to a total maximum of 240 days after workspace generation. If a workspace has inadvertently expired we can restore the data during a limited time (few weeks). In this case you should create a new workspace and report the name of the new and of the expired workspace in a ticket.<br />
<br />
Creating, deleting, finding and extending workspaces is explained on the [[workspace]] page.<br />
<br />
On uc2 there is a default user quota limit of 40 TiB and 30 million inodes (files and directories) per user. <br />
You can chek your current usage and limits with the command<br />
<pre><br />
$ lfs quota -uh $(whoami) /pfs/work7<br />
</pre><br />
Note that the quotas include data and inodes for all of your workspaces and all of your expired workspaces (as long as they are not yet completely removed).<br />
<br />
=== Reminder for workspace deletion ===<br />
<br />
Normally you will get an email every day starting 7 days before a workspace expires. You can send yourself a calender entry which reminds you when a workspace will be automatically deleted:<br />
<br />
$ ws_send_ical.sh <workspace> <email><br />
<br />
== Improving Performance on $HOME and workspaces ==<br />
<br />
The following recommendations might help to improve throughput and metadata<br />
performance on Lustre filesystems.<br />
<br />
=== '''Improving Throughput Performance''' ===<br />
<br />
Depending on your application some adaptations might be necessary if you want to reach<br />
the full bandwidth of the filesystems. Parallel filesystems typically stripe files over storage subsystems, i.e. large files are separated into stripes and distributed to different storage subsystems. In Lustre, the size of these stripes (sometimes also mentioned as chunks) is called stripe size and the number of used storage subsystems is called stripe count.<br />
<br />
When you are designing your application you should consider that the performance of<br />
parallel filesystems is generally better if data is transferred in large blocks and stored in<br />
few large files. In more detail, to increase throughput performance of a parallel application<br />
following aspects should be considered:<br />
<br />
* collect large chunks of data and write them sequentially at once,<br />
<br />
* to exploit complete filesystem bandwidth use several clients,<br />
<br />
* avoid competitive file access by different tasks or use blocks with boundaries at stripe size (default is 1MB),<br />
<br />
* if files are small enough for the SSDs and are only used by one process store them on $TMPDIR.<br />
<br />
With previous Lustre versions adapting the Lustre stripe count was the most important optimization. However, for the workspaces of uc2 the new Lustre feature Progressive File Layouts has been used to define file striping parameters. This means that the stripe count is adapted if the file size is growing. In normal cases users no longer need to adapt file striping parameters in case they have very huge files or in order to reach better performance. <br />
<br />
If you know what you are doing you can still change striping parameters, e.g. the stripe count, of a directory and of newly created files. New files and directories inherit the stripe count from the parent directory. E.g. if you want to enhance throughput on a single very large file which is created in the directory $HOME/my_output_dir you can use the command<br />
<pre><br />
$ lfs setstripe -c-1 $HOME/my_output_dir<br />
</pre><br />
to change the stripe count to -1 which means that all storage subsystems of the file system are used to store that file. If you change the stripe count of a directory the stripe count of existing files inside this<br />
directory is not changed. If you want to change the stripe count of existing files, change<br />
the stripe count of the parent directory, copy the files to new files, remove the old files<br />
and move the new files back to the old name. In order to check the stripe setting of<br />
the file my_file use<br />
<pre><br />
$ lfs getstripe my_file<br />
</pre><br />
Also note that changes on the striping parameters (e.g. stripe count) are not saved in the<br />
backup, i.e. if directories have to be recreated this information is lost and the default stripe<br />
count will be used. Therefore, you should annotate for which directories you made changes<br />
to the striping parameters so that you can repeat these changes if required.<br />
<br />
=== '''Improving Metadata Performance''' ===<br />
<br />
Metadata performance on parallel file systems is usually not as good as with local<br />
filesystems. In addition, it is usually not scalable, i.e. a limited resource. Therefore,<br />
you should omit metadata operations whenever possible. For example, it is much better<br />
to have few large files than lots of small files. In more detail, to increase metadata<br />
performance of a parallel application following aspects should be considered:<br />
<br />
* avoid creating many small files,<br />
<br />
* avoid competitive directory access, e.g. by creating files in separate subdirectories for each task,<br />
<br />
* if many small files are only used within a batch job and accessed by one process store them on $TMPDIR,<br />
<br />
* change the default colorization setting of the command ls (see below).<br />
<br />
On modern Linux systems, the GNU ls command often uses colorization by default to<br />
visually highlight the file type; this is especially true if the command is run within a terminal<br />
session. This is because the default shell profile initializations usually contain an alias<br />
directive similar to the following for the ls command:<br />
<pre><br />
$ alias ls="ls --color=tty"<br />
</pre><br />
However, running the ls command in this way for files on a Lustre file system requires<br />
a stat() call to be used to determine the file type. This can result in a performance<br />
overhead, because the stat() call always needs to determine the size of a file, and that<br />
in turn means that the client node must query the object size of all the backing objects<br />
that make up a file. As a result of the default colorization setting, running a simple<br />
ls command on a Lustre file system often takes as much time as running the ls command<br />
with the -l option (the same is true if the -F, -p, or the -classify option, or any other option <br />
that requires information from a stat() call, is used). To avoid this performance overhead<br />
when using ls commands, add an alias directive similar to the following<br />
to your shell startup script:<br />
<pre><br />
$ alias ls="ls --color=never"<br />
</pre><br />
<br />
== Workspaces on flash storage ==<br />
<br />
There is another workspace file system for special requirements available. The file system is called ''full flash pfs'' and is based on the parallel file system Lustre.<br />
<br />
=== Advantages of this file system ===<br />
<br />
# All storage devices are based on flash (no hard disks) with low access times. Hence performance is better compared to other parallel file systems for read and write access with small blocks and with small files, i.e. IOPS rates are improved.<br />
# The file system is mounted on bwUniCluster 2.0 and HoreKa, i.e. it can be used to share data between these clusters.<br />
<br />
=== Access restrictions ===<br />
<br />
Only HoreKa users or KIT users of bwUniCluster 2.0 can use this file system.<br />
<br />
=== Using the file system ===<br />
<br />
As KIT or HoreKa user you can use the file system in the same way as a normal workspace. You just have to specify the name of the flash-based workspace file system using the option ''-F'' to all the commands that manage workspaces. On bwUniCluster 2.0 it is called ''ffuc'', on HoreKa it is ''ffhk''. For example, to create a workspace with name myws and a lifetime of 60 days on bwUniCluster 2.0 execute:<br />
ws_allocate -F ffuc myws 60<br />
<br />
If you want to use the full flash pfs on bwUniCluster 2.0 '''and''' HoreKa at the same time, please note that you only have to manage a particular workspace on one of the clusters since the name of the workspace directory is different. However, the path to each workspace is visible and can be used on both clusters.<br />
<br />
Other features are similar to normal workspaces. For example, we are able to restore expired workspaces for few weeks and you have to open a ticket to request the restore. There are quota limits with a default limit of 1 TiB capacity and 5 millions inodes per user. You can check your current usage with<br />
lfs quota -uh $(whoami) /pfs/work8<br />
<br />
== $TMPDIR ==<br />
<br />
The environment variable $TMPDIR contains the name of a directory which is local to each node. This means <br />
that different tasks of a parallel application use different directories when they do not utilize the same node. <br />
Although $TMPDIR points to the same path name for different nodes of a batch job, the physical location and the <br />
content of this directory path on these nodes is different.<br />
<br />
This directory should be used for temporary files being accessed from the local node during job runtime. It should <br />
also be used if you read the same data many times from a single node, e.g. if you are doing AI training. In this <br />
case you should copy the data at the beginning of your batch job to $TMPDIR and read the data from there.<br />
<br />
The $TMPDIR directory is located on extremely fast SSD local storage devices. This means that performance <br />
on small files is much better than on the parallel file systems. The capacity of the local SSDs for each node type <br />
is different and can be checked in Table 1 above. The capacity of $TMPDIR is at least 800 GB.<br />
<br />
Each time a batch job is started, a subdirectory is created on the SSD of each node and assigned to the job. <br />
$TMPDIR is set to the name of the subdirectory and this name contains the job ID so that it is unique <br />
for each job. At the end of the job the subdirectory is removed.<br />
<br />
On login nodes $TMPDIR also points to a fast directory on a local SSD disk but this directory is not unique. <br />
It is recommended to create your own unique subdirectory on these nodes. This directory should be used for the <br />
installation of software packages. This means that the software package to be installed should be unpacked, <br />
compiled and linked in a subdirectory of $TMPDIR. The real installation of the package (e.g. make install) <br />
should be made into the $HOME folder.<br />
<br />
=== Usage example for $TMPDIR ===<br />
<br />
We will provide an example for using $TMPDIR and describe efficient data transfer to and from $TMPDIR. <br />
<br />
If you have a data set with many files which is frequently used by batch jobs you should create <br />
a compressed archive on a workspace. This archive can be extracted on $TMPDIR inside your batch jobs. <br />
Such an archive can be read efficiently from a parallel file system since it is a single huge file. <br />
On a login node you can create such an archive with the following steps: <br />
<source lang="bash"><br />
# Create a workspace to store the archive<br />
[ab1234@uc2n997 ~]$ ws_allocate data-ssd 60<br />
# Create the archive from a local dataset folder (example)<br />
[ab1234@uc2n997 ~]$ tar -cvzf $(ws_find data-ssd)/dataset.tgz dataset/<br />
</source><br />
<br />
Inside a batch job extract the archive on $TMPDIR, read input data from $TMPDIR, store results on $TMPDIR <br />
and save the results on a workspace: <br />
<source lang="bash"><br />
#!/bin/bash<br />
# very simple example on how to use local $TMPDIR<br />
#SBATCH -N 1<br />
#SBATCH -t 24:00:00<br />
<br />
# Extract compressed input dataset on local SSD<br />
tar -C $TMPDIR/ -xvzf $(ws_find data-ssd)/dataset.tgz<br />
<br />
# The application reads data from dataset on $TMPDIR and writes results to $TMPDIR<br />
myapp -input $TMPDIR/dataset/myinput.csv -outputdir $TMPDIR/results<br />
<br />
# Before job completes save results on a workspace<br />
rsync -av $TMPDIR/results $(ws_find data-ssd)/results-${SLURM_JOB_ID}/<br />
</source><br />
<br />
== LSDF Online Storage==<br />
<br />
In some cases it is useful to have access to the LSDF Online Storage on the HPC-Clusters also. Therefore the LSDF Online Storage is mounted on the Login- and Datamover-Nodes.<br />
Furthermore it can be used on the compute nodes during the job runtime with the constraint flag "LSDF" ([[bwUniCluster_2.0_Slurm_common_Features|Slurm common features]]<br />
). There is also an example about the LSDF batch usage: [[bwUniCluster_2.0_Slurm_common_Features#LSDF_Online_Storage|Slurm LSDF example ]] .<br />
<pre><br />
#!/bin/bash<br />
#SBATCH ...<br />
#SBATCH --constraint=LSDF<br />
</pre><br />
<br><br />
For the usage of the LSDF Online Storage the following environment variables are available: $LSDF, $LSDFPROJECTS, $LSDFHOME.<br />
Please request storage projects in the LSDF Online Storage seperately:<br />
[https://www.lsdf.kit.edu/os/storagerequest/: LSDF Storage Request].<br />
<br />
==BeeOND (BeeGFS On-Demand)==<br />
<br />
Users of the UniCluster have possibility to request a private BeeOND (on-demand BeeGFS) parallel filesystem for each job. The file system is created during job startup and purged after your job.<br />
<br />
; '''IMPORTANT: '''<br />
: All data on the private filesystem will be deleted after your job. Make sure you have copied your data back to the global filesystem (within job), e.g., $HOME or any workspace.<br />
<br />
BeeOND/BeeGFS can be used like any other parallel file system. Tools like cp or rsync can be used to copy data in and out. <br />
<br />
For detailed usage see here: [[BwUniCluster_2.0_Slurm_common_Features#BeeOND_.28BeeGFS_On-Demand.29|Request on-demand file system]]<br />
<br />
==Backup and Archiving==<br />
<br />
There are regular backups of all data of the home directories,whereas ACLs and extended attributes will<br />
not be backuped. <br />
<br />
Please open a ticket if you need backuped data.<br />
<br />
<br />
[[Category:bwUniCluster 2.0|File System]][[Category:Hardware_and_Architecture|bwUniCluster 2.0]]</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Login&diff=12518
BwUniCluster2.0/Login
2024-01-04T09:48:11Z
<p>P Weisbrod: </p>
<hr />
<div>{|style="background:#deffee; width:100%;"<br />
|style="padding:5px; background:#cef2e0; text-align:left"|<br />
[[Image:Attention.svg|center|25px]]<br />
|style="padding:5px; background:#cef2e0; text-align:left"|<br />
Access to bwUniCluster 2.0 is '''limited to IP addresses from the BelWü network'''.<br />
All home institutions of our current users are connected to BelWü, so if you are on your campus network (e.g. in your office or on the Campus WiFi) you should be able to connect to bwUniCluster 2.0 without restrictions.<br />
If you are outside one of the BelWü networks (e.g. at home), a VPN connection to the home institution or a connection to an SSH jump host at the home institution must be established first.<br />
|}<br />
<br />
The login nodes of the bwHPC clusters are the access point to the compute system, your <code>$HOME</code> directory and your workspaces.<br />
All users must log in through these nodes to submit jobs to the cluster.<br />
<br />
'''Prerequisites for successful login:'''<br />
<br />
You need to have<br />
* completed the 3-step [[registration]] procedure.<br />
* [[Registration/Password|set a service password]] for bwUniCluster 2.0.<br />
* [[Registration/2FA|set up a second factor]] for the time-based one-time password (TOTP).<br />
<br />
<br />
= Login to the bwUniCluster =<br />
<br />
Login to the bwUniCluster 2.0 is only possible with a Secure Shell (SSH) client for which you must know your username on the cluster and the hostname of the login nodes.<br />
For more general information on SSH clients, visit the [[Registration/Login/Client|SSH clients Guide]].<br />
<br />
== Username ==<br />
<br />
If you want to use the bwUniCluster 2.0 you need to add a prefix to your local username.<br />
Users from the KIT, don't need a prefix on the bwUniCluster.<br />
For prefixes please refer to the [[Registration/Login/Username|'''Username Wiki''']].<br />
<br />
Examples:<br/><br />
* If your local username for the University is <code>vwxyz1234</code> and you are a user from the University of Freiburg this would combine to: <code>fr_vwxyz1234</code>.<br />
* If your KIT username is <code>pxd27239</code>, you can use the same username for the bwUniCluster.<br />
<br />
== Hostnames ==<br />
<br />
The system has three login nodes.<br />
The selection of the login node is done automatically.<br />
If you are logging in multiple times, different sessions might run on different login nodes.<br />
<br />
Login to bwUniClister 2.0:<br />
<br />
{| class="wikitable"<br />
! Hostname !! Node type<br />
|-<br />
| '''bwunicluster.scc.kit.edu''' || login to one of the three login nodes<br />
|-<br />
| '''uc2.scc.kit.edu''' || login to one of the three login nodes<br />
|-<br />
|}<br />
<br />
In general, you should use automatic selection to allow us to balance the load over the three login nodes.<br />
If you need to connect to specific login nodes, you can use the following hostnames:<br />
<br />
{| class="wikitable"<br />
! Hostname !! Node type<br />
|-<br />
| '''uc2-login2.scc.kit.edu''' || bwUniCluster 2.0 first login node<br />
|-<br />
| '''uc2-login3.scc.kit.edu''' || bwUniCluster 2.0 second login node<br />
|-<br />
| '''uc2-login4.scc.kit.edu''' || bwUniCluster 2.0 third login node<br />
|-<br />
|}<br />
<br />
== Host Keys ==<br />
<br />
When you log in, you may receive the message <code>The authenticity of host '<host address>' can't be established.</code> along with the host key fingerprint. This is intended so you can verify the authenticity of the host you are connecting to. Before you continue you should verify, if this fingerprint matches one of the following:<br />
<br />
{| class="wikitable"<br />
! Algorithm !! Fingerprint (SHA256)<br />
|-<br />
| '''RSA''' || SHA256:p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|-<br />
| '''ECDSA''' || SHA256:k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|-<br />
| '''ED25519''' || SHA256:yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|-<br />
|}<br />
<br />
== Login with SSH command (Linux, Mac, Windows) ==<br />
<br />
Most Unix and Unix-like operating systems like Linux, Mac OS and *BSD come with a built-in SSH client provided by the OpenSSH project.<br />
More recent versions of Windows 10 and Windows 11 using the [https://docs.microsoft.com/en-us/windows/wsl/install Windows Subsystem for Linux] (WSL) also come with a built-in OpenSSH client.<br />
<br />
For login use one of the following ssh commands:<br />
<br />
ssh <username>@bwunicluster.scc.kit.edu<br />
ssh -l <username> uc2.scc.kit.edu<br />
<br />
To run graphical applications, you can use the <code>-X</code> or <code>-Y</code> flag to <code>ssh</code>:<br />
<br />
ssh -Y -l <username> bwunicluster.scc.kit.edu<br />
<br />
For better performance, we recommend using [[VNC]].<br />
<br />
== Login with graphical SSH client (Windows) ==<br />
<br />
For Windows we suggest using MobaXterm for login and file transfer.<br />
<br />
Start ''MobaXterm'', fill in the following fields:<br />
<pre><br />
Remote name : bwunicluster.scc.kit.edu # or uc2.scc.kit.edu<br />
Specify user name : <username><br />
Port : 22<br />
</pre><br />
<br />
After that click on 'ok'. Then a terminal will be opened and there you can enter your credentials.<br />
<br />
== Login Example ==<br />
<br />
To log in to bwUniCluster 2.0, you must provide your [[Registration/Password|service password]].<br />
Proceed as follows:<br />
# Use SSH for a login node.<br />
# The system will ask for a one-time password <code>Your OTP:</code>. Please enter your OTP and confirm it with Enter/Return. If you do not have a second factor yet, please create one (see [[Registration/2FA]]).<br />
# The system will ask you for your service password <code>Password:</code>. Please enter it and confirm it with Enter/Return. If you do not have a service password yet or have forgotten it, please create one (see [[Registration/Password]]).<br />
# You will be greeted by the cluster, followed by a shell.<br />
<br />
<pre><br />
~ $ ssh -l fr_vwxyz1234 bwunicluster.scc.kit.edu<br />
(fr_vwxyz1234@bwunicluster.scc.kit.edu) Your OTP: 123456<br />
(fr_vwxyz1234@bwunicluster.scc.kit.edu) Password: <br />
********************************************************************************<br />
* _ _ _ _ ____ _ _ ___ *<br />
* | |____ _| | | |_ __ (_)/ ___| |_ _ ___| |_ ___ _ __(__ \ *<br />
* | '_ \ \ /\ / / | | | '_ \| | | | | | | / __| __/ _ \ '__| / / *<br />
* | |_) \ V V /| |_| | | | | | |___| | |_| \__ \ || __/ | / /_ *<br />
* |_.__/ \_/\_/ \___/|_| |_|_|\____|_|\__,_|___/\__\___|_| (____| *<br />
* *<br />
* (KITE 2.0, RHEL 8.4, Lustre 2.12.6_ddn72) *<br />
* *<br />
* wiki: https://wiki.bwhpc.de/e/bwUniCluster_2.0 *<br />
* *<br />
* ticket system: https://www.bwhpc.de/supportportal *<br />
* email: bwunicluster@bwhpc.de *<br />
* *<br />
* training: https://training.bwhpc.de *<br />
* email: training@bwhpc.de *<br />
* *<br />
********************************************************************************<br />
Last login: Thu Jul 7 18:09:43 2022 from host1.scc.kit.edu<br />
********************************************************************************<br />
[fr_vwxyz1234@uc2n995 ~]$ <br />
</pre><br />
<br />
== Troubleshooting ==<br />
<br />
See [[BwUniCluster_2.0/FAQ#Login_Issues|bwUniCluster FAQ]].<br />
<br />
<br />
= Allowed Activities on Login Nodes =<br />
<br />
{|style="background:#deffee; width:100%;"<br />
|style="padding:5px; background:#cef2e0; text-align:left"|<br />
[[Image:Attention.svg|center|25px]]<br />
|style="padding:5px; background:#cef2e0; text-align:left"|<br />
To guarantee usability for all the users of clusters you must not run your compute jobs on the login nodes.<br />
Compute jobs must be submitted to the queuing system.<br />
Any compute job running on the login nodes will be terminated without any notice.<br />
Any long-running compilation or any long-running pre- or post-processing of batch jobs must also be submitted to the queuing system.<br />
|}<br />
<br />
The login nodes of the bwHPC clusters are the access point to the compute system, your <code>$HOME</code> directory and your workspaces.<br />
These nodes are shared with all the users therefore, your activities on the login nodes are limited to primarily set up your batch jobs.<br />
Your activities may also be:<br />
* '''short''' compilation of your program code and<br />
* '''short''' pre- and post-processing of your batch jobs.<br />
<br />
We advise users to use [[BwUniCluster_2.0_Batch_Queues#Interactive_Jobs|interactive jobs]] for compute and memory intensive tasks like compiling.<br />
<br />
<br />
= Related Information =<br />
<br />
* If you want to reset your service password, consult the [[Registration/Password|Password Guide]].<br />
* If you want to register a new token for the two factor authentication (2FA), consult the [[Registration/2FA|2FA Guide]].<br />
* If you want to de-register, consult the [[Registration/Deregistration|De-registration Guide]].<br />
* If you need an SSH key for your workflow, read [[Registration/SSH|Registering SSH Keys with your Cluster]].</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Containers&diff=11890
BwUniCluster2.0/Containers
2023-03-24T12:44:53Z
<p>P Weisbrod: </p>
<hr />
<div>= Introduction =<br />
To date, only few container runtime environments integrate well with HPC environments due to security concerns and differing assumptions in some areas.<br />
<br />
For example native Docker environments require elevated privileges, which is not an option on shared HPC resources. Docker's "rootless mode" is also currently not supported on our HPC systems because it does not support necessary features such as cgroups resource controls, security profiles, overlay networks, furthermore GPU passthrough is difficult. Necessary subuid (newuidmap) and subgid (newgidmap) settings may impose security issues.<br />
<br />
On bwUniCluster the container runtimes '''Enroot''' and '''Singularity/Apptainer''' are supported.<br />
<br />
Further rootless container runtime environments (Podman, …) might be supported in the future, depending on how support for e.g. network interconnects, security features and HPC file systems develops.<br />
<br />
= ENROOT =<br />
<br />
Enroot enables you to run '''Docker containers''' on HPC systems. It is developed by NVIDIA. It is the '''recommended tool''' to use containers on bwUniCluster and integrates well with GPU usage and has basically no impact on performance.<br />
Enroot is available to all users by default.<br />
[[File:docker_logo.svg|center|100px]]<br />
<br />
== Usage ==<br />
<br />
Excellent documentation is provided on [https://github.com/NVIDIA/enroot/blob/master/doc NVIDIA's github page]. This documentation here therefore confines itself to simple examples to get to know the essential functionalities.<br />
<br />
Using Docker containers with Enroot requires three steps:<br />
<br />
* Importing an image<br />
* Creating a container<br />
* Starting a container<br />
<br />
Optionally containers can also be exported and transferred.<br />
<br />
=== Importing a container image ===<br />
<br />
* <code>enroot import docker://alpine</code><br />This pulls the latest alpine image from dockerhub (default registry). You will obtain the file alpine.sqsh.<br />
<br />
* <code>enroot import docker://nvcr.io#nvidia/pytorch:21.04-py3</code><br />This pulls the pytorch image version 21.04-py3 from [https://ngc.nvidia.com/catalog NVIDIA's NGC registry]. Please note that the NGC registry does not always contain the "latest" tag and instead requires the specification of a dedicated version. You will obtain the file nvidia+pytorch+21.04-py3.sqsh.<br />
<br />
* <code>enroot import docker://registry.scc.kit.edu#myProject/myImage:latest</code><br />This pulls your latest image from the KIT registry. You obtain the file myImage.sqsh.<br />
<br />
=== Creating a container ===<br />
Create a container named "nvidia+pytorch+21.04-py3" by unpacking the .sqsh-file.<br />
<br />
<code>enroot create --name nvidia+pytorch+21.04-py3 nvidia+pytorch+21.04-py3.sqsh</code><br />
<br />
"Creating" a container means that the squashed container image is unpacked inside <code>$ENROOT_DATA_PATH/</code>. By default this variable points to <code>$HOME/.local/share/enroot/</code>.<br />
<br />
=== Starting a container ===<br />
* Start the container nvidia+pytorch+21.04-py3 in read-write mode (<code>--rw</code>) and run bash inside the container.<br /><code>enroot start --rw nvidia+pytorch+21.04-py3 bash</code><br />
<br />
* Start container in <code>--rw</code>-mode and get root access (<code>--root</code>) inside the container.<br /> <code>enroot start --root --rw nvidia+pytorch+21.04-py3 bash</code><br />You can now install software with root privileges, depending on the containerized Linux distribution e.g. with<br /><code>apt-get install … </code>, <code>apk add …</code>, <code>yum install …</code>, <code>pacman -S …</code><br />
<br />
* Start container and mount (<code>-m</code>) a local directory to <code>/work</code> inside the container.<br /><code>enroot start -m <localDir>:/work --rw nvidia+pytorch+21.04-py3 bash</code><br />
<br />
* Start container, mount a directory and start the application <code>jupyter lab</code>.<br /><code>enroot start -m <localDir>:/work --rw nvidia+pytorch+21.04-py3 jupyter lab</code><br />
<br />
=== Exporting and transfering containers ===<br />
<br />
If you intend to use Docker images which you built e.g. on your local desktop, and transfer them somewhere else, there are several possibilities to do so:<br />
<br />
* <code>enroot import --output myImage.sqsh dockerd://myImage</code><br />Import an image from the locally running Docker daemon. Copy the .sqsh-file to bwUniCluster and import it with <code>enroot import</code>.<br />
<br />
* <code>enroot export --output myImage.sqsh myImage</code><br />Export an existing enroot container. Copy the .sqsh-file to bwUniCluster and import it with enroot import.<br />
<br />
* <code>enroot bundle --output myImage.run myImage.sqsh</code><br />Create a self extracting bundle from a container image. Copy the .run-file to bwUniCluster. You can run the self extracting image via ./myImage.run even if enroot is not installed!<br />
<br />
=== Container management ===<br />
<br />
You can list all containers on the system and additional information (<code>--fancy</code> parameter) with the <code>enroot list</code> command.<br />
<br />
The unpacked images can be removed with the enroot remove command.<br />
<br />
== SLURM Integration==<br />
Enroot allows you to run containerized applications non-interactively, including MPI- and multi-node parallelism. The necessary Slurm integration is realized via the [https://github.com/NVIDIA/pyxis Pyxis plugin].<br />
<br />
NOTE: THIS IS WORK IN PROGRESS!<br />
<br />
== FAQ ==<br />
<br />
* ''How can I run JupyterLab in a container and connect to it?''<br />
** Start an interactive session with or without GPUs. Notice the compute node ID the session is running on, and start a container with a running JupyterLab, e.g.:<br /><code>salloc -p gpu_4 --time=01:00:00 --gres=gpu:1</code><br /><code>enroot start -m <localDir>:/work --rw nvidia+pytorch+21.04-py3 jupyter lab</code><br />
** Open a terminal on your desktop and create a SSH-tunnel to the running JupyterLab instance on the compute node. Insert the node ID, where the interactive session is running on:<br /><code>ssh -L8888:<computeNodeID>:8888 <yourAccount>@uc2.scc.kit.edu</code><br />
** Open a web browser and open the URL [http://localhost:8888 localhost:8888]<br />
** Enter the token, which is visible in the output of the first terminal.<br />Copy the string behind the <code>token=</code> and paste it into the input field in the browser.<br />
<br />
* ''Are GPUs accessible from within a running container?''<br />Yes.<br />Unlike Docker, Enroot does not need further command line options to enable GPU passthrough like <code>--runtime=nvidia</code> or <code>--privileged</code>.<br />
<br />
* ''Is there something like <code>enroot-compose</code>?''<br />AFAIK no.<br /> Enroot is mainly intended for HPC workloads, not for operating multi-container applications. However, starting and running these applications separately is possible.<br />
<br />
* ''Can I use workspaces to store containers?''<br />Yes.<br />You can define the location of configuration files and storage with environment variables. The <code>ENROOT_DATA_PATH</code> variable should be set accordingly. Please refer to [https://github.com/NVIDIA/enroot/blob/master/doc/configuration.md#runtime-configuration NVIDIA's documentation] on runtime configuration.<br />
<br />
== Additional resources ==<br />
<br />
Source code: [https://github.com/NVIDIA/enroot https://github.com/NVIDIA/enroot]<br />
<br />
Documentation: [https://github.com/NVIDIA/enroot/blob/master/doc https://github.com/NVIDIA/enroot/blob/master/doc]<br />
<br />
Additional information: <br />
* [https://archive.fosdem.org/2020/schedule/event/containers_hpc_unprivileged/ FOSDEM 2020 talk] + [https://archive.fosdem.org/2020/schedule/event/containers_hpc_unprivileged/attachments/slides/3711/export/events/attachments/containers_hpc_unprivileged/slides/3711/containers_hpc_unprivileged.pdf slides]<br />
* [https://slurm.schedmd.com/SLUG19/NVIDIA_Containers.pdf Slurm User Group Meeting 2019 talk]<br />
<br />
= Singularity/Apptainer =<br />
[[File:singularity_logo.svg|center|100px]]<br />
<br />
<br />
== Usage ==<br />
<br />
Excellent documentation is provided on the [https://sylabs.io/docs/ Documentation&Examples] page provided by Sylabs. This documentation here therefore confines itself to simple examples to get to know the essential functionalities.<br />
<br />
Using Singularity/Apptainer usually involves two steps:<br />
<br />
* Building a container image using singularity build<br />
<br />
* Running a container image using singularity run or singularity exec<br />
<br />
=== Building an image ===<br />
<br />
* <code>singularity build ubuntu.sif library://ubuntu</code><br />This pulls the latest Ubuntu image from Singularity's [https://cloud.sylabs.io/library Container Library] and locally creates a container image file called ubuntu.sif.<br />
<br />
* <code>singularity build alpine.sif docker://alpine</code><br />This pulls the latest alpine image from Dockerhub and locally creates a container image file called alpine.sif.<br />
<br />
* <code>singularity build pytorch-21.04-p3.sif docker://nvcr.io#nvidia/pytorch:21.04-py3</code><br />This pulls the latest pytorch image from NVIDIA's NGC registry and locally creates a container image file called pytorch-21.04-p3.sif.<br />
<br />
=== Running an image ===<br />
<br />
* <code>singularity shell ubuntu.sif</code><br />Start a shell in the Ubuntu container.<br />
<br />
* <code>singularity run alpine.sif</code><br />Start the container alpine.sif and run the default runscript provided by the image.<br />
<br />
* <code>singularity exec alpine.sif /bin/ls</code><br />Start the container alpine.sif and run the /bin/ls command.<br />
<br />
=== Container management ===<br />
<br />
You can use the <code>singularity search</code> command to search for images on Singularity's [https://cloud.sylabs.io/library Container Library].</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Containers&diff=11889
BwUniCluster2.0/Containers
2023-03-24T12:44:37Z
<p>P Weisbrod: </p>
<hr />
<div>= Introduction =<br />
To date, only few container runtime environments integrate well with HPC environments due to security concerns and differing assumptions in some areas.<br />
<br />
For example native Docker environments require elevated privileges, which is not an option on shared HPC resources. Docker's "rootless mode" is also currently not supported on our HPC systems because it does not support necessary features such as cgroups resource controls, security profiles, overlay networks, furthermore GPU passthrough is difficult. Necessary subuid (newuidmap) and subgid (newgidmap) settings may impose security issues.<br />
<br />
On bwUniCluster the container runtimes '''Enroot''' and '''Singularity''' are supported.<br />
<br />
Further rootless container runtime environments (Podman, …) might be supported in the future, depending on how support for e.g. network interconnects, security features and HPC file systems develops.<br />
<br />
= ENROOT =<br />
<br />
Enroot enables you to run '''Docker containers''' on HPC systems. It is developed by NVIDIA. It is the '''recommended tool''' to use containers on bwUniCluster and integrates well with GPU usage and has basically no impact on performance.<br />
Enroot is available to all users by default.<br />
[[File:docker_logo.svg|center|100px]]<br />
<br />
== Usage ==<br />
<br />
Excellent documentation is provided on [https://github.com/NVIDIA/enroot/blob/master/doc NVIDIA's github page]. This documentation here therefore confines itself to simple examples to get to know the essential functionalities.<br />
<br />
Using Docker containers with Enroot requires three steps:<br />
<br />
* Importing an image<br />
* Creating a container<br />
* Starting a container<br />
<br />
Optionally containers can also be exported and transferred.<br />
<br />
=== Importing a container image ===<br />
<br />
* <code>enroot import docker://alpine</code><br />This pulls the latest alpine image from dockerhub (default registry). You will obtain the file alpine.sqsh.<br />
<br />
* <code>enroot import docker://nvcr.io#nvidia/pytorch:21.04-py3</code><br />This pulls the pytorch image version 21.04-py3 from [https://ngc.nvidia.com/catalog NVIDIA's NGC registry]. Please note that the NGC registry does not always contain the "latest" tag and instead requires the specification of a dedicated version. You will obtain the file nvidia+pytorch+21.04-py3.sqsh.<br />
<br />
* <code>enroot import docker://registry.scc.kit.edu#myProject/myImage:latest</code><br />This pulls your latest image from the KIT registry. You obtain the file myImage.sqsh.<br />
<br />
=== Creating a container ===<br />
Create a container named "nvidia+pytorch+21.04-py3" by unpacking the .sqsh-file.<br />
<br />
<code>enroot create --name nvidia+pytorch+21.04-py3 nvidia+pytorch+21.04-py3.sqsh</code><br />
<br />
"Creating" a container means that the squashed container image is unpacked inside <code>$ENROOT_DATA_PATH/</code>. By default this variable points to <code>$HOME/.local/share/enroot/</code>.<br />
<br />
=== Starting a container ===<br />
* Start the container nvidia+pytorch+21.04-py3 in read-write mode (<code>--rw</code>) and run bash inside the container.<br /><code>enroot start --rw nvidia+pytorch+21.04-py3 bash</code><br />
<br />
* Start container in <code>--rw</code>-mode and get root access (<code>--root</code>) inside the container.<br /> <code>enroot start --root --rw nvidia+pytorch+21.04-py3 bash</code><br />You can now install software with root privileges, depending on the containerized Linux distribution e.g. with<br /><code>apt-get install … </code>, <code>apk add …</code>, <code>yum install …</code>, <code>pacman -S …</code><br />
<br />
* Start container and mount (<code>-m</code>) a local directory to <code>/work</code> inside the container.<br /><code>enroot start -m <localDir>:/work --rw nvidia+pytorch+21.04-py3 bash</code><br />
<br />
* Start container, mount a directory and start the application <code>jupyter lab</code>.<br /><code>enroot start -m <localDir>:/work --rw nvidia+pytorch+21.04-py3 jupyter lab</code><br />
<br />
=== Exporting and transfering containers ===<br />
<br />
If you intend to use Docker images which you built e.g. on your local desktop, and transfer them somewhere else, there are several possibilities to do so:<br />
<br />
* <code>enroot import --output myImage.sqsh dockerd://myImage</code><br />Import an image from the locally running Docker daemon. Copy the .sqsh-file to bwUniCluster and import it with <code>enroot import</code>.<br />
<br />
* <code>enroot export --output myImage.sqsh myImage</code><br />Export an existing enroot container. Copy the .sqsh-file to bwUniCluster and import it with enroot import.<br />
<br />
* <code>enroot bundle --output myImage.run myImage.sqsh</code><br />Create a self extracting bundle from a container image. Copy the .run-file to bwUniCluster. You can run the self extracting image via ./myImage.run even if enroot is not installed!<br />
<br />
=== Container management ===<br />
<br />
You can list all containers on the system and additional information (<code>--fancy</code> parameter) with the <code>enroot list</code> command.<br />
<br />
The unpacked images can be removed with the enroot remove command.<br />
<br />
== SLURM Integration==<br />
Enroot allows you to run containerized applications non-interactively, including MPI- and multi-node parallelism. The necessary Slurm integration is realized via the [https://github.com/NVIDIA/pyxis Pyxis plugin].<br />
<br />
NOTE: THIS IS WORK IN PROGRESS!<br />
<br />
== FAQ ==<br />
<br />
* ''How can I run JupyterLab in a container and connect to it?''<br />
** Start an interactive session with or without GPUs. Notice the compute node ID the session is running on, and start a container with a running JupyterLab, e.g.:<br /><code>salloc -p gpu_4 --time=01:00:00 --gres=gpu:1</code><br /><code>enroot start -m <localDir>:/work --rw nvidia+pytorch+21.04-py3 jupyter lab</code><br />
** Open a terminal on your desktop and create a SSH-tunnel to the running JupyterLab instance on the compute node. Insert the node ID, where the interactive session is running on:<br /><code>ssh -L8888:<computeNodeID>:8888 <yourAccount>@uc2.scc.kit.edu</code><br />
** Open a web browser and open the URL [http://localhost:8888 localhost:8888]<br />
** Enter the token, which is visible in the output of the first terminal.<br />Copy the string behind the <code>token=</code> and paste it into the input field in the browser.<br />
<br />
* ''Are GPUs accessible from within a running container?''<br />Yes.<br />Unlike Docker, Enroot does not need further command line options to enable GPU passthrough like <code>--runtime=nvidia</code> or <code>--privileged</code>.<br />
<br />
* ''Is there something like <code>enroot-compose</code>?''<br />AFAIK no.<br /> Enroot is mainly intended for HPC workloads, not for operating multi-container applications. However, starting and running these applications separately is possible.<br />
<br />
* ''Can I use workspaces to store containers?''<br />Yes.<br />You can define the location of configuration files and storage with environment variables. The <code>ENROOT_DATA_PATH</code> variable should be set accordingly. Please refer to [https://github.com/NVIDIA/enroot/blob/master/doc/configuration.md#runtime-configuration NVIDIA's documentation] on runtime configuration.<br />
<br />
== Additional resources ==<br />
<br />
Source code: [https://github.com/NVIDIA/enroot https://github.com/NVIDIA/enroot]<br />
<br />
Documentation: [https://github.com/NVIDIA/enroot/blob/master/doc https://github.com/NVIDIA/enroot/blob/master/doc]<br />
<br />
Additional information: <br />
* [https://archive.fosdem.org/2020/schedule/event/containers_hpc_unprivileged/ FOSDEM 2020 talk] + [https://archive.fosdem.org/2020/schedule/event/containers_hpc_unprivileged/attachments/slides/3711/export/events/attachments/containers_hpc_unprivileged/slides/3711/containers_hpc_unprivileged.pdf slides]<br />
* [https://slurm.schedmd.com/SLUG19/NVIDIA_Containers.pdf Slurm User Group Meeting 2019 talk]<br />
<br />
= Singularity/Apptainer =<br />
[[File:singularity_logo.svg|center|100px]]<br />
<br />
<br />
== Usage ==<br />
<br />
Excellent documentation is provided on the [https://sylabs.io/docs/ Documentation&Examples] page provided by Sylabs. This documentation here therefore confines itself to simple examples to get to know the essential functionalities.<br />
<br />
Using Singularity/Apptainer usually involves two steps:<br />
<br />
* Building a container image using singularity build<br />
<br />
* Running a container image using singularity run or singularity exec<br />
<br />
=== Building an image ===<br />
<br />
* <code>singularity build ubuntu.sif library://ubuntu</code><br />This pulls the latest Ubuntu image from Singularity's [https://cloud.sylabs.io/library Container Library] and locally creates a container image file called ubuntu.sif.<br />
<br />
* <code>singularity build alpine.sif docker://alpine</code><br />This pulls the latest alpine image from Dockerhub and locally creates a container image file called alpine.sif.<br />
<br />
* <code>singularity build pytorch-21.04-p3.sif docker://nvcr.io#nvidia/pytorch:21.04-py3</code><br />This pulls the latest pytorch image from NVIDIA's NGC registry and locally creates a container image file called pytorch-21.04-p3.sif.<br />
<br />
=== Running an image ===<br />
<br />
* <code>singularity shell ubuntu.sif</code><br />Start a shell in the Ubuntu container.<br />
<br />
* <code>singularity run alpine.sif</code><br />Start the container alpine.sif and run the default runscript provided by the image.<br />
<br />
* <code>singularity exec alpine.sif /bin/ls</code><br />Start the container alpine.sif and run the /bin/ls command.<br />
<br />
=== Container management ===<br />
<br />
You can use the <code>singularity search</code> command to search for images on Singularity's [https://cloud.sylabs.io/library Container Library].</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance/2023-03&diff=11888
BwUniCluster2.0/Maintenance/2023-03
2023-03-24T12:39:44Z
<p>P Weisbrod: </p>
<hr />
<div>The following changes were introduced during the maintenance interval between on 20.03.2023 (Monday) 08:30 and 24.03.2023 (Friday) 15:00.<br />
<br />
The host key of the system has not changed. You should not receive any warnings by your SSH client(s), but if there should be a warning or if you want to check that you are connecting to the correct system, you can verify the key hashes using the following list:<br />
<br />
{|class="wikitable"<br />
! Algorithm<br />
! Hash (SHA256)<br />
! Hash (MD5)<br />
|-<br />
|RSA<br />
|p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|59:2a:67:44:4a:d7:89:6c:c0:0d:74:ba:3c:c4:63:6d<br />
|-<br />
|ECDSA<br />
|k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|85:d4:d9:97:e0:f0:43:30:6e:66:8e:d0:b6:9b:85:d1<br />
|-<br />
|ED25519<br />
|yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|42:d2:0d:ab:87:48:fc:1d:5d:b3:7c:bf:22:c3:5f:b7<br />
|}<br />
<br />
= Hardware =<br />
<br />
* All firmware versions on all components were upgraded.<br />
<br />
= Operating system =<br />
<br />
* The operating system was upgraded from RHEL 8.4 EUS to RHEL 8.6 EUS. We recommend to re-compile all applications after the upgrade.<br />
<br />
* The Mellanox OFED InfiniBand Stack was updated.<br />
<br />
= Compilers, Libaries and Runtime Environments =<br />
<br />
<br />
= Userspace tools =<br />
<br />
* pigz and pbzip are not supported anymore. Please use pzstd instead.<br />
<br />
= Software Modules =<br />
<br />
* The Lmod module system was upgraded.<br />
<br />
* Old openmpi 4.0 modules were removed. Please use openmpi 4.1.<br />
<br />
= Batch system =<br />
<br />
* The Slurm version was upgraded to version 23.02.0.<br />
<br />
* The Slurm partitions have changed:<br />
** multiple and multiple_il maximum number of nodes is now 80<br />
** the amount of available nodes in partition single has been increased<br />
** the amount of available nodes in partition multiple has been decreased<br />
<br />
= Storage =<br />
<br />
* Lustre client, BeeGFS client and Spectrum Scale client were updated.<br />
<br />
= Graphics stack =<br />
<br />
* The NVIDIA driver was upgraded.<br />
<br />
= Containers =<br />
<br />
* Enroot was upgraded.<br />
<br />
* Singularity was replaced with its successor Apptainer. (the command 'singularity' still works)<br />
<br />
= JupyterHub =<br />
<br />
* Jupyterhub was upgraded to version 3.1.1<br />
<br />
* python3.9 is now used as the default<br />
<br />
= Resource Limits on Login Nodes =<br />
<br />
* After the maintenance the following per-user limits apply (via cgroups):<br />
** 48 GB phyisical memory<br />
** 400% CPU cycles (100% equals 1 thread)</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance/2023-03&diff=11887
BwUniCluster2.0/Maintenance/2023-03
2023-03-24T12:36:09Z
<p>P Weisbrod: </p>
<hr />
<div>The following changes were introduced during the maintenance interval between on 20.03.2023 (Monday) 08:30 and 24.03.2023 (Friday) 15:00.<br />
<br />
The host key of the system has not changed. You should not receive any warnings by your SSH client(s), but if there should be a warning or if you want to check that you are connecting to the correct system, you can verify the key hashes using the following list:<br />
<br />
{|class="wikitable"<br />
! Algorithm<br />
! Hash (SHA256)<br />
! Hash (MD5)<br />
|-<br />
|RSA<br />
|p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|59:2a:67:44:4a:d7:89:6c:c0:0d:74:ba:3c:c4:63:6d<br />
|-<br />
|ECDSA<br />
|k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|85:d4:d9:97:e0:f0:43:30:6e:66:8e:d0:b6:9b:85:d1<br />
|-<br />
|ED25519<br />
|yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|42:d2:0d:ab:87:48:fc:1d:5d:b3:7c:bf:22:c3:5f:b7<br />
|}<br />
<br />
= Hardware =<br />
<br />
* All firmware versions on all components were upgraded.<br />
<br />
= Operating system =<br />
<br />
* The operating system was upgraded from RHEL 8.4 EUS to RHEL 8.6 EUS. We recommend to re-compile all applications after the upgrade.<br />
<br />
* The Mellanox OFED InfiniBand Stack was updated.<br />
<br />
= Compilers, Libaries and Runtime Environments =<br />
<br />
<br />
= Userspace tools =<br />
<br />
<br />
= Software Modules =<br />
<br />
* The Lmod module system was upgraded.<br />
<br />
* Old openmpi 4.0 modules were removed. Please use openmpi 4.1.<br />
<br />
= Batch system =<br />
<br />
* The Slurm version was upgraded to version 23.02.0.<br />
<br />
* The Slurm partitions have changed:<br />
** multiple and multiple_il maximum number of nodes is now 80<br />
** the amount of available nodes in partition single has been increased<br />
** the amount of available nodes in partition multiple has been decreased<br />
<br />
= Storage =<br />
<br />
* Lustre client, BeeGFS client and Spectrum Scale client were updated.<br />
<br />
= Graphics stack =<br />
<br />
* The NVIDIA driver was upgraded.<br />
<br />
= Containers =<br />
<br />
* Enroot was upgraded.<br />
<br />
* Singularity was replaced with its successor Apptainer. (the command 'singularity' still works)<br />
<br />
= JupyterHub =<br />
<br />
* Jupyterhub was upgraded to version 3.1.1<br />
<br />
* python3.9 is now used as the default<br />
<br />
= Resource Limits on Login Nodes =<br />
<br />
* After the maintenance the following per-user limits apply (via cgroups):<br />
** 48 GB phyisical memory<br />
** 400% CPU cycles (100% equals 1 thread)</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance/2023-03&diff=11886
BwUniCluster2.0/Maintenance/2023-03
2023-03-24T12:32:20Z
<p>P Weisbrod: </p>
<hr />
<div>The following changes were introduced during the maintenance interval between on 20.03.2023 (Monday) 08:30 and 24.03.2023 (Friday) 15:00.<br />
<br />
The host key of the system has not changed. You should not receive any warnings by your SSH client(s), but if there should be a warning or if you want to check that you are connecting to the correct system, you can verify the key hashes using the following list:<br />
<br />
{|class="wikitable"<br />
! Algorithm<br />
! Hash (SHA256)<br />
! Hash (MD5)<br />
|-<br />
|RSA<br />
|p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|59:2a:67:44:4a:d7:89:6c:c0:0d:74:ba:3c:c4:63:6d<br />
|-<br />
|ECDSA<br />
|k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|85:d4:d9:97:e0:f0:43:30:6e:66:8e:d0:b6:9b:85:d1<br />
|-<br />
|ED25519<br />
|yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|42:d2:0d:ab:87:48:fc:1d:5d:b3:7c:bf:22:c3:5f:b7<br />
|}<br />
<br />
= Hardware =<br />
<br />
* All firmware versions on all components were upgraded.<br />
<br />
= Operating system =<br />
<br />
* The operating system was upgraded from RHEL 8.4 EUS to RHEL 8.6 EUS. We recommend to re-compile all applications after the upgrade.<br />
<br />
* The Mellanox OFED InfiniBand Stack was updated.<br />
<br />
= Compilers, Libaries and Runtime Environments =<br />
<br />
<br />
= Userspace tools =<br />
<br />
<br />
= Software Modules =<br />
<br />
* The Lmod module system was upgraded.<br />
<br />
* Old openmpi 4.0 modules were removed. Please use openmpi 4.1.<br />
<br />
= Batch system =<br />
<br />
* The Slurm version was upgraded to version 23.02.0.<br />
<br />
* The Slurm partitions have changed:<br />
<br />
* multiple and multiple_il maximum number of nodes is now 80<br />
<br />
<br />
= Storage =<br />
<br />
* Lustre client, BeeGFS client and Spectrum Scale client were updated.<br />
<br />
= Graphics stack =<br />
<br />
* The NVIDIA driver was upgraded.<br />
<br />
= Containers =<br />
<br />
* Enroot was upgraded.<br />
<br />
* Singularity was replaced with its successor Apptainer. (the command 'singularity' still works)<br />
<br />
= JupyterHub =<br />
<br />
* Jupyterhub was upgraded to version 3.1.1<br />
<br />
* python3.9 is now used as the default<br />
<br />
= Resource Limits on Login Nodes =<br />
<br />
* After the maintenance the following per-user limits apply (via cgroups):<br />
** 48 GB phyisical memory<br />
** 400% CPU cycles (100% equals 1 thread)</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance/2023-03&diff=11885
BwUniCluster2.0/Maintenance/2023-03
2023-03-24T12:32:06Z
<p>P Weisbrod: </p>
<hr />
<div>The following changes were introduced during the maintenance interval between on 20.03.2023 (Monday) 08:30 and 24.03.2023 (Friday) 15:00.<br />
<br />
The host key of the system has not changed. You should not receive any warnings by your SSH client(s), but if there should be a warning or if you want to check that you are connecting to the correct system, you can verify the key hashes using the following list:<br />
<br />
{|class="wikitable"<br />
! Algorithm<br />
! Hash (SHA256)<br />
! Hash (MD5)<br />
|-<br />
|RSA<br />
|p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|59:2a:67:44:4a:d7:89:6c:c0:0d:74:ba:3c:c4:63:6d<br />
|-<br />
|ECDSA<br />
|k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|85:d4:d9:97:e0:f0:43:30:6e:66:8e:d0:b6:9b:85:d1<br />
|-<br />
|ED25519<br />
|yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|42:d2:0d:ab:87:48:fc:1d:5d:b3:7c:bf:22:c3:5f:b7<br />
|}<br />
<br />
= Hardware =<br />
<br />
* All firmware versions on all components were upgraded.<br />
<br />
= Operating system =<br />
<br />
* The operating system was upgraded from RHEL 8.4 EUS to RHEL 8.6 EUS. We recommend to re-compile all applications after the upgrade.<br />
<br />
* The Mellanox OFED InfiniBand Stack was updated.<br />
<br />
= Compilers, Libaries and Runtime Environments =<br />
<br />
<br />
= Userspace tools =<br />
<br />
<br />
= Software Modules =<br />
<br />
* The Lmod module system was upgraded.<br />
<br />
* Old openmpi 4.0 modules were removed. Please use openmpi 4.1.<br />
<br />
= Batch system =<br />
<br />
* The Slurm version was upgraded to version 23.02.0.<br />
<br />
* The Slurm partitions have changed:<br />
<br />
* multiple and multiple_il maximum number of nodes is now 80<br />
<br />
<br />
= Storage =<br />
<br />
* Lustre client, BeeGFS client and Spectrum Scale client were updated.<br />
<br />
= Graphics stack =<br />
<br />
* The NVIDIA driver was upgraded.<br />
<br />
= Containers =<br />
<br />
* Enroot was upgraded.<br />
<br />
* Singularity was replaced with its successor Apptainer. (the command 'singularity' still works)<br />
<br />
= JupyterHub =<br />
<br />
* Jupyterhub was upgraded to version 3.1.1<br />
<br />
* python3.9 is now used as the default<br />
<br />
= Resource Limits on Login Nodes =<br />
<br />
* After the maintenance the following per-user limits apply (via cgroups):<br />
<br />
** 48 GB phyisical memory<br />
<br />
** 400% CPU cycles (100% equals 1 thread)</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance/2023-03&diff=11882
BwUniCluster2.0/Maintenance/2023-03
2023-03-24T12:21:01Z
<p>P Weisbrod: </p>
<hr />
<div>The following changes were introduced during the maintenance interval between on 20.03.2023 (Monday) 08:30 and 24.03.2023 (Friday) 15:00.<br />
<br />
The host key of the system has not changed. You should not receive any warnings by your SSH client(s), but if there should be a warning or if you want to check that you are connecting to the correct system, you can verify the key hashes using the following list:<br />
<br />
{|class="wikitable"<br />
! Algorithm<br />
! Hash (SHA256)<br />
! Hash (MD5)<br />
|-<br />
|RSA<br />
|p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|59:2a:67:44:4a:d7:89:6c:c0:0d:74:ba:3c:c4:63:6d<br />
|-<br />
|ECDSA<br />
|k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|85:d4:d9:97:e0:f0:43:30:6e:66:8e:d0:b6:9b:85:d1<br />
|-<br />
|ED25519<br />
|yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|42:d2:0d:ab:87:48:fc:1d:5d:b3:7c:bf:22:c3:5f:b7<br />
|}<br />
<br />
= Hardware =<br />
<br />
* All firmware versions on all components were upgraded.<br />
<br />
= Operating system =<br />
<br />
* The operating system was upgraded from RHEL 8.4 EUS to RHEL 8.6 EUS. We recommend to re-compile all applications after the upgrade.<br />
<br />
* The Mellanox OFED InfiniBand Stack was updated.<br />
<br />
= Compilers, Libaries and Runtime Environments =<br />
<br />
<br />
= Userspace tools =<br />
<br />
<br />
= Software Modules =<br />
<br />
* The Lmod module system was upgraded.<br />
<br />
* Old openmpi 4.0 modules were removed. Please use openmpi 4.1.<br />
<br />
= Batch system =<br />
<br />
* The Slurm version was upgraded to version 23.02.0.<br />
<br />
* The Slurm partitions have changed.<br />
<br />
= Storage =<br />
<br />
* Lustre client, BeeGFS client and Spectrum Scale client were updated.<br />
<br />
= Graphics stack =<br />
<br />
* The NVIDIA driver was upgraded.<br />
<br />
= Containers =<br />
<br />
* Enroot was upgraded.<br />
<br />
* Singularity was replaced with its successor Apptainer. (the command 'singularity' still works)<br />
<br />
= JupyterHub =<br />
<br />
* Jupyterhub was upgraded to version 3.1.1<br />
<br />
* python3.9 is now used as the default<br />
<br />
= Resource Limits on Login Nodes =<br />
<br />
* After the maintenance the following per-user limits apply (via cgroups):<br />
<br />
* 48 GB phyisical memory<br />
<br />
* 400% CPU cycles (100% equals 1 thread)</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance/2023-03&diff=11881
BwUniCluster2.0/Maintenance/2023-03
2023-03-24T12:19:10Z
<p>P Weisbrod: </p>
<hr />
<div>The following changes were introduced during the maintenance interval between on 20.03.2023 (Monday) 08:30 and 24.03.2023 (Friday) 15:00.<br />
<br />
The host key of the system has not changed. You should not receive any warnings by your SSH client(s), but if there should be a warning or if you want to check that you are connecting to the correct system, you can verify the key hashes using the following list:<br />
<br />
{|class="wikitable"<br />
! Algorithm<br />
! Hash (SHA256)<br />
! Hash (MD5)<br />
|-<br />
|RSA<br />
|p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|59:2a:67:44:4a:d7:89:6c:c0:0d:74:ba:3c:c4:63:6d<br />
|-<br />
|ECDSA<br />
|k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|85:d4:d9:97:e0:f0:43:30:6e:66:8e:d0:b6:9b:85:d1<br />
|-<br />
|ED25519<br />
|yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|42:d2:0d:ab:87:48:fc:1d:5d:b3:7c:bf:22:c3:5f:b7<br />
|}<br />
<br />
= Hardware =<br />
<br />
* All firmware versions on all components were upgraded.<br />
<br />
= Operating system =<br />
<br />
* The operating system was upgraded from RHEL 8.4 EUS to RHEL 8.6 EUS. We recommend to re-compile all applications after the upgrade.<br />
<br />
* The Mellanox OFED InfiniBand Stack was updated.<br />
<br />
= Compilers, Libaries and Runtime Environments =<br />
<br />
<br />
= Userspace tools =<br />
<br />
<br />
= Software Modules =<br />
<br />
* The Lmod module system was upgraded.<br />
<br />
* Old openmpi 4.0 modules were removed. Please use openmpi 4.1.<br />
<br />
= Batch system =<br />
<br />
* The Slurm version was upgraded to version 23.02.0.<br />
<br />
* The Slurm partitions have changed.<br />
<br />
= Storage =<br />
<br />
* Lustre client, BeeGFS client and Spectrum Scale client were updated.<br />
<br />
= Graphics stack =<br />
<br />
* The NVIDIA driver was upgraded.<br />
<br />
= Containers =<br />
<br />
* Enroot was upgraded.<br />
<br />
* Singularity was replaced with its successor Apptainer. (the command 'singularity' still works)<br />
<br />
= JupyterHub =<br />
<br />
* Jupyterhub was upgraded to version 3.1.1<br />
<br />
* python3.9 is now used as the default<br />
<br />
= Resource Limits on Login Nodes =<br />
<br />
* After the maintenance the following per-user limits apply (via cgroups):<br />
<br />
* 48 GB phyisical memory<br />
<br />
* 400% of available CPU cycles (100% equals 1 core)</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance/2023-03&diff=11880
BwUniCluster2.0/Maintenance/2023-03
2023-03-24T12:14:35Z
<p>P Weisbrod: </p>
<hr />
<div>The following changes were introduced during the maintenance interval between on 20.03.2023 (Monday) 08:30 and 24.03.2023 (Friday) 15:00.<br />
<br />
The host key of the system has not changed. You should not receive any warnings by your SSH client(s), but if there should be a warning or if you want to check that you are connecting to the correct system, you can verify the key hashes using the following list:<br />
<br />
{|class="wikitable"<br />
! Algorithm<br />
! Hash (SHA256)<br />
! Hash (MD5)<br />
|-<br />
|RSA<br />
|p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|59:2a:67:44:4a:d7:89:6c:c0:0d:74:ba:3c:c4:63:6d<br />
|-<br />
|ECDSA<br />
|k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|85:d4:d9:97:e0:f0:43:30:6e:66:8e:d0:b6:9b:85:d1<br />
|-<br />
|ED25519<br />
|yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|42:d2:0d:ab:87:48:fc:1d:5d:b3:7c:bf:22:c3:5f:b7<br />
|}<br />
<br />
= Hardware =<br />
<br />
* All firmware versions on all components were upgraded.<br />
<br />
= Operating system =<br />
<br />
* The operating system was upgraded from RHEL 8.4 EUS to RHEL 8.6 EUS. We recommend to re-compile all applications after the upgrade.<br />
<br />
* The Mellanox OFED InfiniBand Stack was updated.<br />
<br />
= Compilers, Libaries and Runtime Environments =<br />
<br />
<br />
= Userspace tools =<br />
<br />
<br />
= Software Modules =<br />
<br />
* The Lmod module system was upgraded.<br />
<br />
* Old openmpi 4.0 modules were removed. Please use openmpi 4.1.<br />
<br />
= Batch system =<br />
<br />
* The Slurm version was upgraded to version 23.02.0.<br />
<br />
* The Slurm partitions have changed.<br />
<br />
= Storage =<br />
<br />
* Lustre client, BeeGFS client and Spectrum Scale client were updated.<br />
<br />
= Graphics stack =<br />
<br />
* The NVIDIA driver was upgraded.<br />
<br />
= Containers =<br />
<br />
* Enroot was upgraded.<br />
<br />
* Singularity was replaced with its successor Apptainer. (the command 'singularity' still works)<br />
<br />
= JupyterHub =<br />
<br />
* Jupyterhub was upgraded to version 3.1.1<br />
<br />
* python3.9 is now used as the default</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance/2023-03&diff=11879
BwUniCluster2.0/Maintenance/2023-03
2023-03-24T11:52:09Z
<p>P Weisbrod: </p>
<hr />
<div>The following changes were introduced during the maintenance interval between on 20.03.2023 (Monday) 08:30 and 24.03.2023 (Friday) 15:00.<br />
<br />
The host key of the system has not changed. You should not receive any warnings by your SSH client(s), but if there should be a warning or if you want to check that you are connecting to the correct system, you can verify the key hashes using the following list:<br />
<br />
{|class="wikitable"<br />
! Algorithm<br />
! Hash (SHA256)<br />
! Hash (MD5)<br />
|-<br />
|RSA<br />
|p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|59:2a:67:44:4a:d7:89:6c:c0:0d:74:ba:3c:c4:63:6d<br />
|-<br />
|ECDSA<br />
|k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|85:d4:d9:97:e0:f0:43:30:6e:66:8e:d0:b6:9b:85:d1<br />
|-<br />
|ED25519<br />
|yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|42:d2:0d:ab:87:48:fc:1d:5d:b3:7c:bf:22:c3:5f:b7<br />
|}<br />
<br />
= Hardware =<br />
<br />
* All firmware versions on all components were upgraded.<br />
<br />
= Operating system =<br />
<br />
* The operating system was upgraded from RHEL 8.4 EUS to RHEL 8.6 EUS. We recommend to re-compile all applications after the upgrade.<br />
<br />
* The Mellanox OFED InfiniBand Stack was updated.<br />
<br />
= Compilers, Libaries and Runtime Environments =<br />
<br />
<br />
= Userspace tools =<br />
<br />
<br />
= Software Modules =<br />
<br />
* The Lmod module system was upgraded.<br />
<br />
* Old openmpi 4.0 modules were removed. Please use openmpi 4.1.<br />
<br />
= Batch system =<br />
<br />
* The Slurm version was upgraded to version 23.02.0.<br />
<br />
= Storage =<br />
<br />
* Lustre client, BeeGFS client and Spectrum Scale client were updated.<br />
<br />
= Graphics stack =<br />
<br />
* The NVIDIA driver was upgraded.<br />
<br />
= Containers =<br />
<br />
* Enroot was upgraded.<br />
<br />
* Singularity was replaced with its successor Apptainer. (the command 'singularity' still works)<br />
<br />
= JupyterHub =<br />
<br />
* Jupyterhub was upgraded to version 3.1.1<br />
<br />
* python3.9 is now used as the default</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance/2023-03&diff=11874
BwUniCluster2.0/Maintenance/2023-03
2023-03-16T14:20:27Z
<p>P Weisbrod: </p>
<hr />
<div>The following changes will be introduced during the maintenance interval between on 20.03.2023 (Monday) 08:30 and 24.03.2023 (Friday) 15:00.<br />
<br />
The host key of the system will not change. You should not receive any warnings by your SSH client(s), but if there should be a warning or if you want to check that you are connecting to the correct system, you can verify the key hashes using the following list:<br />
<br />
{|class="wikitable"<br />
! Algorithm<br />
! Hash (SHA256)<br />
! Hash (MD5)<br />
|-<br />
|RSA<br />
|p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|59:2a:67:44:4a:d7:89:6c:c0:0d:74:ba:3c:c4:63:6d<br />
|-<br />
|ECDSA<br />
|k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|85:d4:d9:97:e0:f0:43:30:6e:66:8e:d0:b6:9b:85:d1<br />
|-<br />
|ED25519<br />
|yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|42:d2:0d:ab:87:48:fc:1d:5d:b3:7c:bf:22:c3:5f:b7<br />
|}<br />
<br />
= Hardware =<br />
<br />
* All firmware versions on all components will be upgraded.<br />
<br />
= Operating system =<br />
<br />
* The operating system will be upgraded from RHEL 8.4 EUS to RHEL 8.6 EUS. We recommend to re-compile all applications after the upgrade.<br />
<br />
* The Mellanox OFED InfiniBand Stack will be updated.<br />
<br />
= Compilers, Libaries and Runtime Environments =<br />
<br />
<br />
= Userspace tools =<br />
<br />
<br />
= Software Modules =<br />
<br />
* The Lmod module system will be upgraded.<br />
<br />
* Old openmpi 4.0 modules will be removed. Please use openmpi 4.1.<br />
<br />
= Batch system =<br />
<br />
* The Slurm version will be upgraded to version 23.02.0.<br />
<br />
= Storage =<br />
<br />
* Lustre client, BeeGFS client and Spectrum Scale client will be updated.<br />
<br />
= Graphics stack =<br />
<br />
* The NVIDIA driver will be upgraded.<br />
<br />
= Containers =<br />
<br />
* Enroot will be upgraded.<br />
<br />
* Singularity will be replaced with its successor Apptainer. (the command 'singularity' will still work)<br />
<br />
= JupyterHub =<br />
<br />
* Jupyterhub will be upgraded to version 3.1.1<br />
<br />
* python3.9 will used as the default</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance/2023-03&diff=11792
BwUniCluster2.0/Maintenance/2023-03
2023-03-13T14:00:46Z
<p>P Weisbrod: </p>
<hr />
<div>The following changes will be introduced during the maintenance interval between on 20.03.2023 (Monday) 08:30 and 24.03.2023 (Friday) 15:00.<br />
<br />
The host key of the system will not change. You should not receive any warnings by your SSH client(s), but if there should be a warning or if you want to check that you are connecting to the correct system, you can verify the key hashes using the following list:<br />
<br />
{|class="wikitable"<br />
! Algorithm<br />
! Hash (SHA256)<br />
! Hash (MD5)<br />
|-<br />
|RSA<br />
|p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|59:2a:67:44:4a:d7:89:6c:c0:0d:74:ba:3c:c4:63:6d<br />
|-<br />
|ECDSA<br />
|k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|85:d4:d9:97:e0:f0:43:30:6e:66:8e:d0:b6:9b:85:d1<br />
|-<br />
|ED25519<br />
|yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|42:d2:0d:ab:87:48:fc:1d:5d:b3:7c:bf:22:c3:5f:b7<br />
|}<br />
<br />
= Hardware =<br />
<br />
* All firmware versions on all components will be upgraded.<br />
<br />
= Operating system =<br />
<br />
* The operating system will be upgraded from RHEL 8.4 EUS to RHEL 8.6 EUS. We recommend to re-compile all applications after the upgrade.<br />
<br />
* The Mellanox OFED InfiniBand Stack will be updated.<br />
<br />
= Compilers, Libaries and Runtime Environments =<br />
<br />
<br />
= Userspace tools =<br />
<br />
* The hpc-workspace tools will be updated to version 1.4.0.<br />
<br />
= Software Modules =<br />
<br />
* The Lmod module system will be upgraded.<br />
<br />
* Old openmpi 4.0 modules will be removed. Please use openmpi 4.1.<br />
<br />
= Batch system =<br />
<br />
* The Slurm version will be upgraded to version 23.02.0.<br />
<br />
= Storage =<br />
<br />
* Lustre client, BeeGFS client and Spectrum Scale client will be updated.<br />
<br />
= Graphics stack =<br />
<br />
* The NVIDIA driver will be upgraded.<br />
<br />
= Containers =<br />
<br />
* Enroot will be upgraded.<br />
<br />
* Singularity will be replaced with its successor Apptainer. (the command 'singularity' will still work)<br />
<br />
= JupyterHub =</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance/2023-03&diff=11791
BwUniCluster2.0/Maintenance/2023-03
2023-03-13T13:53:10Z
<p>P Weisbrod: </p>
<hr />
<div>The following changes will be introduced during the maintenance interval between on 20.03.2023 (Monday) 08:30 and 24.03.2023 (Friday) 15:00.<br />
<br />
The host key of the system will not change. You should not receive any warnings by your SSH client(s), but if there should be a warning or if you want to check that you are connecting to the correct system, you can verify the key hashes using the following list:<br />
<br />
{|class="wikitable"<br />
! Algorithm<br />
! Hash (SHA256)<br />
! Hash (MD5)<br />
|-<br />
|RSA<br />
|p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|59:2a:67:44:4a:d7:89:6c:c0:0d:74:ba:3c:c4:63:6d<br />
|-<br />
|ECDSA<br />
|k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|85:d4:d9:97:e0:f0:43:30:6e:66:8e:d0:b6:9b:85:d1<br />
|-<br />
|ED25519<br />
|yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|42:d2:0d:ab:87:48:fc:1d:5d:b3:7c:bf:22:c3:5f:b7<br />
|}<br />
<br />
= Hardware =<br />
<br />
* All firmware versions on all components will be upgraded.<br />
<br />
= Operating system =<br />
<br />
* The operating system will be upgraded from RHEL 8.4 EUS to RHEL 8.6 EUS. We recommend to re-compile all applications after the upgrade.<br />
<br />
* The Mellanox OFED InfiniBand Stack will be updated.<br />
<br />
= Compilers, Libaries and Runtime Environments =<br />
<br />
<br />
= Userspace tools =<br />
<br />
* The hpc-workspace tools will be updated to version 1.4.0.<br />
<br />
= Software Modules =<br />
<br />
* The Lmod module system will be upgraded.<br />
<br />
* Old openmpi 4.0 modules will be removed. Please use openmpi 4.1.<br />
<br />
= Batch system =<br />
<br />
* The Slurm version will be upgraded to version 23.02.0.<br />
<br />
= Storage =<br />
<br />
* Lustre client, BeeGFS client and Spectrum Scale client will be updated.<br />
<br />
= Graphics stack =<br />
<br />
* The NVIDIA driver will be upgraded.<br />
<br />
= Containers =<br />
<br />
* Enroot will be upgraded.<br />
<br />
* Singularity will be replaced with its successor Apptainer. (the command 'singularity' will still work)<br />
<br />
= JupyterHub =</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Slurm&diff=11778
BwUniCluster2.0/Slurm
2023-02-28T09:34:20Z
<p>P Weisbrod: </p>
<hr />
<div><div id="top"></div><br />
= Slurm HPC Workload Manager = <br />
== Specification == <br />
Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters. Slurm requires no kernel modifications for its operation and is relatively self-contained. As a cluster workload manager, Slurm has three key functions. First, it allocates exclusive and/or non-exclusive access to resources (compute nodes) to users for some duration of time so they can perform work. Second, it provides a framework for starting, executing, and monitoring work (normally a parallel job) on the set of allocated nodes. Finally, it arbitrates contention for resources by managing a queue of pending work.<br />
<br><br />
<br><br />
Any kind of calculation on the compute nodes of [[bwUniCluster 2.0|bwUniCluster 2.0]] requires the user to define calculations as a sequence of commands or single command together with required run time, number of CPU cores and main memory and submit all, i.e., the '''batch job''', to a resource and workload managing software. bwUniCluster 2.0 has installed the workload managing software Slurm. Therefore any job submission by the user is to be executed by commands of the Slurm software. Slurm queues and runs user jobs based on fair sharing policies. <br />
<br><br />
<br><br />
<br />
== Slurm Commands (excerpt) ==<br />
Some of the most used Slurm commands for non-administrators working on bwUniCluster 2.0.<br />
{| width=750px class="wikitable"<br />
! Slurm commands !! Brief explanation<br />
|-<br />
| [[#Job Submission : sbatch|sbatch]] || Submits a job and queues it in an input queue [[https://slurm.schedmd.com/sbatch.html sbatch]] <br />
|-<br />
| [[#Detailed job information : scontrol show job|scontrol show job]] || Displays detailed job state information [[https://slurm.schedmd.com/scontrol.html scontrol]]<br />
|-<br />
| [[#List of your submitted jo/bs : squeue|squeue]] || Displays information about active, eligible, blocked, and/or recently completed jobs [[https://slurm.schedmd.com/squeue.html squeue]]<br />
|-<br />
| [[#Start time of job or resources : squeue|squeue --start]] || Returns start time of submitted job or requested resources [[https://slurm.schedmd.com/squeue.html squeue]]<br />
|-<br />
| [[#Shows free resources : sinfo_t_idle|sinfo_t_idle]] || Shows what resources are available for immediate use [[https://slurm.schedmd.com/sinfo.html sinfo]]<br />
|-<br />
| [[#Canceling own jobs : scancel|scancel]] || Cancels a job (obsoleted!) [[https://slurm.schedmd.com/scancel.html scancel]]<br />
|}<br />
If your job was submitted to the "multiple" queue you can log into the allocated nodes via SSH as soon as the job is running.<br />
<br />
<br><br />
* [https://slurm.schedmd.com/tutorials.html Slurm Tutorials]<br />
* [https://slurm.schedmd.com/pdfs/summary.pdf Slurm command/option summary (2 pages)]<br />
* [https://slurm.schedmd.com/man_index.html Slurm Commands]<br />
<br><br />
<br />
== Job Submission : sbatch ==<br />
Batch jobs are submitted by using the command '''sbatch'''. The main purpose of the '''sbatch''' command is to specify the resources that are needed to run the job. '''sbatch''' will then queue the batch job. However, starting of batch job depends on the availability of the requested resources and the fair sharing value.<br />
<br><br />
<br><br />
=== sbatch Command Parameters ===<br />
The syntax and use of '''sbatch''' can be displayed via:<br />
<pre><br />
$ man sbatch<br />
</pre><br />
'''sbatch''' options can be used from the command line or in your job script.<br />
{| width=750px class="wikitable"<br />
! colspan="3" | sbatch Options<br />
|-<br />
! Command line<br />
! Script<br />
! Purpose<br />
|- style="vertical-align:top;"<br />
| -t ''time'' or --time=''time''<br />
| #SBATCH --time=''time''<br />
| Wall clock time limit.<br><br />
|-<br />
|- style="vertical-align:top;"<br />
| -N ''count'' or --nodes=''count''<br />
| #SBATCH --nodes=''count''<br />
| Number of nodes to be used.<br />
|- style="vertical-align:top;"<br />
| -n ''count'' or --ntasks=''count''<br />
| #SBATCH --ntasks=''count''<br />
| Number of tasks to be launched.<br />
|-<br />
|- style="vertical-align:top;"<br />
| --ntasks-per-node=''count''<br />
| #SBATCH --ntasks-per-node=''count''<br />
| Maximum count (<= 28 and <= 40 resp.) of tasks per node.<br>(Replaces the option ppn of MOAB.)<br />
|-<br />
|- style="vertical-align:top;"<br />
| -c ''count'' or --cpus-per-task=''count''<br />
| #SBATCH --cpus-per-task=''count''<br />
| Number of CPUs required per (MPI-)task.<br />
|-<br />
|- style="vertical-align:top;"<br />
| --mem=''value_in_MB''<br />
| #SBATCH --mem=''value_in_MB'' <br />
| Memory in MegaByte per node.<br>(Default value is 128000 and 96000 MB resp., i.e. you should omit <br> the setting of this option.)<br />
|-<br />
|- style="vertical-align:top;"<br />
| --mem-per-cpu=''value_in_MB''<br />
| #SBATCH --mem-per-cpu=''value_in_MB'' <br />
| Minimum Memory required per allocated CPU.<br>(Replaces the option pmem of MOAB. You should omit <br> the setting of this option.)<br />
|-<br />
|- style="vertical-align:top;"<br />
| --mail-type=''type''<br />
| #SBATCH --mail-type=''type''<br />
| Notify user by email when certain event types occur.<br>Valid type values are NONE, BEGIN, END, FAIL, REQUEUE, ALL.<br />
|-<br />
|- style="vertical-align:top;"<br />
| --mail-user=''mail-address''<br />
| #SBATCH --mail-user=''mail-address''<br />
| The specified mail-address receives email notification of state<br>changes as defined by --mail-type.<br />
|-<br />
|- style="vertical-align:top;"<br />
| --output=''name''<br />
| #SBATCH --output=''name''<br />
| File in which job output is stored. <br />
|-<br />
|- style="vertical-align:top;"<br />
| --error=''name''<br />
| #SBATCH --error=''name''<br />
| File in which job error messages are stored. <br />
|-<br />
|- style="vertical-align:top;"<br />
| -J ''name'' or --job-name=''name''<br />
| #SBATCH --job-name=''name''<br />
| Job name.<br />
|-<br />
|- style="vertical-align:top;"<br />
| --export=[ALL,] ''env-variables''<br />
| #SBATCH --export=[ALL,] ''env-variables''<br />
| Identifies which environment variables from the submission <br> environment are propagated to the launched application. Default <br> is ALL. If adding an environment variable to the submission<br> environment is intended, the argument ALL must be added.<br />
|-<br />
|- style="vertical-align:top;"<br />
| -A ''group-name'' or --account=''group-name''<br />
| #SBATCH --account=''group-name''<br />
| Change resources used by this job to specified group. You may <br> need this option if your account is assigned to more <br> than one group. By command "scontrol show job" the project <br> group the job is accounted on can be seen behind "Account=". <br />
|-<br />
|- style="vertical-align:top;"<br />
| -p ''queue-name'' or --partition=''queue-name''<br />
| #SBATCH --partition=''queue-name''<br />
| Request a specific queue for the resource allocation.<br />
|-<br />
|- style="vertical-align:top;"<br />
| --reservation=''reservation-name''<br />
| #SBATCH --reservation=''reservation-name''<br />
| Use a specific reservation for the resource allocation.<br />
|-<br />
|- style="vertical-align:top;"<br />
| -C ''LSDF'' or --constraint=''LSDF''<br />
| #SBATCH --constraint=LSDF<br />
| Job constraint LSDF Filesystems.<br />
|-<br />
|- style="vertical-align:top;"<br />
| -C ''BEEOND (BEEOND_4MDS, BEEOND_MAXMDS)'' or --constraint=''BEEOND (BEEOND_4MDS, BEEOND_MAXMDS''<br />
| #SBATCH --constraint=BEEOND (BEEOND_4MDS, BEEOND_MAXMDS)<br />
| Job constraint BeeOND file system.<br />
|-<br />
|}<br />
<br><br />
<br />
==== sbatch --partition ''queues'' ====<br />
Queue classes define maximum resources such as walltime, nodes and processes per node and queue of the compute system. Details can be found here:<br />
* [[BwUniCluster_2.0_Batch_Queues#sbatch_-p_queue|bwUniCluster 2.0 queue settings]]<br />
<br><br />
<br />
=== sbatch Examples ===<br />
==== Serial Programs ====<br />
To submit a serial job that runs the script '''job.sh''' and that requires 5000 MB of main memory and 10 minutes of wall clock time<br />
<br />
a) execute:<br />
<pre><br />
$ sbatch -p dev_single -n 1 -t 10:00 --mem=5000 job.sh<br />
</pre><br />
or<br />
b) add after the initial line of your script '''job.sh''' the lines (here with a high memory request):<br />
<source lang="bash"><br />
#SBATCH --ntasks=1<br />
#SBATCH --time=10<br />
#SBATCH --mem=180gb<br />
#SBATCH --job-name=simple<br />
</source><br />
and execute the modified script with the command line option ''--partition=fat'' (with ''--partition=(dev_)single'' maximum ''--mem=96gb'' is possible):<br />
<pre><br />
$ sbatch --partition=fat job.sh<br />
</pre><br />
Note, that sbatch command line options overrule script options.<br />
<br><br />
<br><br />
<br />
==== Multithreaded Programs ====<br />
Multithreaded programs operate faster than serial programs on CPUs with multiple cores.<br><br />
Moreover, multiple threads of one process share resources such as memory.<br />
<br><br />
For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) a number of threads is defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).<br />
<br><br />
'''Because hyperthreading is switched on on bwUniCluster 2.0, the option --cpus-per-task (-c) must be set to 2*n, if you want to use n threads.'''<br />
<br><br />
To submit a batch job called ''OpenMP_Test'' that runs a 40-fold threaded program ''omp_exe'' which requires 6000 MByte of total physical memory and total wall clock time of 40 minutes:<br />
<br><br />
a) execute:<br />
<pre><br />
$ sbatch -p single --export=ALL,OMP_NUM_THREADS=40 -J OpenMP_Test -N 1 -c 80 -t 40 --mem=6000 ./omp_exe<br />
</pre><br />
or<br />
--><br />
* generate the script '''job_omp.sh''' containing the following lines:<br />
<source lang="bash"><br />
#!/bin/bash<br />
#SBATCH --nodes=1<br />
#SBATCH --cpus-per-task=80<br />
#SBATCH --time=40:00<br />
#SBATCH --mem=6000mb <br />
#SBATCH --export=ALL,EXECUTABLE=./omp_exe<br />
#SBATCH -J OpenMP_Test<br />
<br />
#Usually you should set<br />
export KMP_AFFINITY=compact,1,0<br />
#export KMP_AFFINITY=verbose,compact,1,0 prints messages concerning the supported affinity<br />
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE<br />
<br />
export OMP_NUM_THREADS=$((${SLURM_JOB_CPUS_PER_NODE}/2))<br />
echo "Executable ${EXECUTABLE} running on ${SLURM_JOB_CPUS_PER_NODE} cores with ${OMP_NUM_THREADS} threads"<br />
startexe=${EXECUTABLE}<br />
echo $startexe<br />
exec $startexe<br />
</source><br />
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores and, if necessary, replace <placeholder> with the required modulefile to enable the OpenMP environment and execute the script '''job_omp.sh''' adding the queue class ''single'' as sbatch option:<br />
<pre><br />
$ sbatch -p single job_omp.sh<br />
</pre><br />
Note, that sbatch command line options overrule script options, e.g.,<br />
<pre><br />
$ sbatch --partition=single --mem=200 job_omp.sh<br />
</pre><br />
overwrites the script setting of 6000 MByte with 200 MByte.<br />
<br><br />
<br><br />
<br />
==== MPI Parallel Programs ====<br />
MPI parallel programs run faster than serial programs on multi CPU and multi core systems. N-fold spawned processes of the MPI program, i.e., '''MPI tasks''', run simultaneously and communicate via the Message Passing Interface (MPI) paradigm. MPI tasks do not share memory but can be spawned over different nodes.<br />
<br><br />
Multiple MPI tasks must be launched via '''mpirun''', e.g. 4 MPI tasks of ''my_par_program'':<br />
<pre><br />
$ mpirun -n 4 my_par_program<br />
</pre><br />
This command runs 4 MPI tasks of ''my_par_program'' on the node you are logged in.<br />
To run this command with a loaded Intel MPI the environment variable I_MPI_HYDRA_BOOTSTRAP must be unset ( --> $ unset I_MPI_HYDRA_BOOTSTRAP).<br />
<br />
Running MPI parallel programs in a batch job the interactive environment - particularly the loaded modules - will also be set in the batch job. If you want to set a defined module environment in your batch job you have to purge all modules before setting the wished modules. <br />
<br><br />
<br><br />
===== OpenMPI =====<br />
<br />
If you want to run jobs on batch nodes, generate a wrapper script ''job_ompi.sh'' for '''OpenMPI''' containing the following lines:<br />
<source lang="bash"><br />
#!/bin/bash<br />
# Use when a defined module environment related to OpenMPI is wished<br />
module load mpi/openmpi/<placeholder_for_version><br />
mpirun --bind-to core --map-by core -report-bindings my_par_program<br />
</source><br />
'''Attention:''' Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since Slurm instructs mpirun about number of processes and node hostnames. Use '''ALWAYS''' the MPI options '''''--bind-to core''''' and '''''--map-by core|socket|node'''''. Please type ''mpirun --help'' for an explanation of the meaning of the different options of mpirun option ''--map-by''.<br />
<br><br />
Considering 4 OpenMPI tasks on a single node, each requiring 2000 MByte, and running for 1 hour, execute:<br />
<pre><br />
$ sbatch -p single -N 1 -n 4 --mem-per-cpu=2000 --time=01:00:00 ./job_ompi.sh<br />
</pre><br />
<br><br />
<br />
===== Intel MPI =====<br />
<br />
Generate a wrapper script for '''Intel MPI''', ''job_impi.sh'' containing the following lines:<br />
<source lang="bash"><br />
#!/bin/bash<br />
# Use when a defined module environment related to Intel MPI is wished<br />
module load mpi/impi/<placeholder_for_version> <br />
mpiexec.hydra -bootstrap slurm my_par_program<br />
</source><br />
<font color=red>'''Attention:'''</font><br><br />
Do '''NOT''' add mpirun options ''-n <number_of_processes>'' or any other option defining processes or nodes, since Slurm instructs mpirun about number of processes and node hostnames.<br />
<br><br />
Launching and running 200 Intel MPI tasks on 5 nodes, each requiring 80 GByte, and running for 5 hours, execute:<br />
<pre><br />
$ sbatch --partition=multiple -N 5 --ntasks-per-node=40 --mem=80gb -t 300 ./job_impi.sh<br />
</pre> <br />
<br><br />
If you want to use 128 or more nodes, you must also set the environment variable as follows: <BR><br />
export I_MPI_HYDRA_BRANCH_COUNT=-1<br />
<br><br />
If you want to use the options perhost, ppn or rr, you must additionally set the environment variable I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=off.<br />
<br><br />
<br><br />
<br />
==== Multithreaded + MPI parallel Programs ====<br />
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes. '''Because hyperthreading is switched on on bwUniCluster 2.0, the option --cpus-per-task (-c) must be set to 2*n, if you want to use n threads.'''<br />
<br><br />
<br><br />
===== OpenMPI with Multithreading =====<br />
Multiple MPI tasks using '''OpenMPI''' must be launched by the MPI parallel program '''mpirun'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).<br />
<br><br />
'''For OpenMPI''' a job-script to submit a batch job called ''job_ompi_omp.sh'' that runs a MPI program with 4 tasks and a 28-fold threaded program ''ompi_omp_program'' requiring 3000 MByte of physical memory per thread (using 28 threads per MPI task you will get 28*3000 MByte = 84000 MByte per MPI task) and total wall clock time of 3 hours looks like:<br />
<!--b)--><br />
<source lang="bash"><br />
#!/bin/bash<br />
#SBATCH --ntasks=4<br />
#SBATCH --cpus-per-task=56<br />
#SBATCH --time=03:00:00<br />
#SBATCH --mem=83gb # 84000 MB = 84000/1024 GB = 82.1 GB<br />
#SBATCH --export=ALL,MPI_MODULE=mpi/openmpi/3.1,EXECUTABLE=./ompi_omp_program<br />
#SBATCH --output="parprog_hybrid_%j.out" <br />
<br />
# Use when a defined module environment related to OpenMPI is wished<br />
module load ${MPI_MODULE}<br />
export OMP_NUM_THREADS=$((${SLURM_CPUS_PER_TASK}/2))<br />
export MPIRUN_OPTIONS="--bind-to core --map-by socket:PE=${OMP_NUM_THREADS} -report-bindings"<br />
export NUM_CORES=${SLURM_NTASKS}*${OMP_NUM_THREADS}<br />
echo "${EXECUTABLE} running on ${NUM_CORES} cores with ${SLURM_NTASKS} MPI-tasks and ${OMP_NUM_THREADS} threads"<br />
startexe="mpirun -n ${SLURM_NTASKS} ${MPIRUN_OPTIONS} ${EXECUTABLE}"<br />
echo $startexe<br />
exec $startexe<br />
</source><br />
Execute the script '''job_ompi_omp.sh''' by command sbatch:<br />
<pre><br />
$ sbatch -p multiple_e ./job_ompi_omp.sh<br />
</pre><br />
* With the mpirun option ''--bind-to core'' MPI tasks and OpenMP threads are bound to physical cores.<br />
* With the option ''--map-by node:PE=<value>'' (neighbored) MPI tasks will be attached to different nodes and each MPI task is bound to the first core of a node. <value> must be set to ${OMP_NUM_THREADS}.<br />
* The option ''-report-bindings'' shows the bindings between MPI tasks and physical cores.<br />
* The mpirun-options '''--bind-to core''', '''--map-by socket|...|node:PE=<value>''' should always be used when running a multithreaded MPI program.<br />
<br><br />
<br />
===== Intel MPI with Multithreading =====<br />
Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes. <br />
<br />
Multiple Intel MPI tasks must be launched by the MPI parallel program '''mpiexec.hydra'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).<br />
<br />
'''For Intel MPI''' a job-script to submit a batch job called ''job_impi_omp.sh'' that runs a Intel MPI program with 10 tasks and a 40-fold threaded program ''impi_omp_program'' requiring 96000 MByte of total physical memory per task and total wall clock time of 1 hours looks like: <br />
<br />
<!--b)--> <br />
<source lang="bash"><br />
#!/bin/bash<br />
#SBATCH --ntasks=10<br />
#SBATCH --cpus-per-task=80<br />
#SBATCH --time=60<br />
#SBATCH --mem=96000<br />
#SBATCH --export=ALL,MPI_MODULE=mpi/impi,EXE=./impi_omp_program<br />
#SBATCH --output="parprog_impi_omp_%j.out"<br />
<br />
#If using more than one MPI task per node please set<br />
export KMP_AFFINITY=compact,1,0<br />
#export KMP_AFFINITY=verbose,scatter prints messages concerning the supported affinity <br />
#KMP_AFFINITY Description: https://software.intel.com/en-us/node/524790#KMP_AFFINITY_ENVIRONMENT_VARIABLE<br />
<br />
# Use when a defined module environment related to Intel MPI is wished <br />
module load ${MPI_MODULE}<br />
export OMP_NUM_THREADS=$((${SLURM_CPUS_PER_TASK}/2))<br />
export MPIRUN_OPTIONS="-binding "domain=omp:compact" -print-rank-map -envall"<br />
export NUM_PROCS=eval(${SLURM_NTASKS}*${OMP_NUM_THREADS})<br />
echo "${EXE} running on ${NUM_PROCS} cores with ${SLURM_NTASKS} MPI-tasks and ${OMP_NUM_THREADS} threads"<br />
startexe="mpiexec.hydra -bootstrap slurm ${MPIRUN_OPTIONS} -n ${SLURM_NTASKS} ${EXE}"<br />
echo $startexe<br />
exec $startexe<br />
</source><br />
Using Intel compiler the environment variable KMP_AFFINITY switches on binding of threads to specific cores. If you only run one MPI task per node please set KMP_AFFINITY=compact,1,0.<br />
<BR><br />
If you want to use 128 or more nodes, you must also set the environment variable as follows: <BR><br />
export I_MPI_HYDRA_BRANCH_COUNT=-1<br />
<br><br />
If you want to use the options perhost, ppn or rr, you must additionally set the environment variable I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=off. <br />
<br><br />
<br><br />
Execute the script '''job_impi_omp.sh''' by command sbatch:<br />
<pre><br />
$ sbatch -p multiple ./job_impi_omp.sh<br />
</pre><br />
<br><br />
The mpirun option ''-print-rank-map'' shows the bindings between MPI tasks and nodes (not very beneficial). The option ''-binding'' binds MPI tasks (processes) to a particular processor; ''domain=omp'' means that the domain size is determined by the number of threads. If you would choose 2 MPI tasks per node, you should choose ''-binding "cell=unit;map=bunch"''; this binding maps one MPI process to each socket. <br />
<br><br />
<br><br />
<br />
==== Chain jobs ====<br />
The CPU time requirements of many applications exceed the limits of the job classes. In those situations it is recommended to solve the problem by a job chain. A job chain is a sequence of jobs where each job automatically starts its successor. <br />
<source lang="bash"><br />
#!/bin/bash<br />
####################################<br />
## simple Slurm submitter script to setup ## <br />
## a chain of jobs using Slurm ##<br />
####################################<br />
## ver. : 2018-11-27, KIT, SCC<br />
<br />
## Define maximum number of jobs via positional parameter 1, default is 5<br />
max_nojob=${1:-5}<br />
<br />
## Define your jobscript (e.g. "~/chain_job.sh")<br />
chain_link_job=${PWD}/chain_job.sh<br />
<br />
## Define type of dependency via positional parameter 2, default is 'afterok'<br />
dep_type="${2:-afterok}"<br />
## -> List of all dependencies:<br />
## https://slurm.schedmd.com/sbatch.html<br />
<br />
myloop_counter=1<br />
## Submit loop<br />
while [ ${myloop_counter} -le ${max_nojob} ] ; do<br />
##<br />
## Differ msub_opt depending on chain link number<br />
if [ ${myloop_counter} -eq 1 ] ; then<br />
slurm_opt=""<br />
else<br />
slurm_opt="-d ${dep_type}:${jobID}"<br />
fi<br />
##<br />
## Print current iteration number and sbatch command<br />
echo "Chain job iteration = ${myloop_counter}"<br />
echo " sbatch --export=myloop_counter=${myloop_counter} ${slurm_opt} ${chain_link_job}"<br />
## Store job ID for next iteration by storing output of sbatch command with empty lines<br />
jobID=$(sbatch -p <queue> --export=ALL,myloop_counter=${myloop_counter} ${slurm_opt} ${chain_link_job} 2>&1 | sed 's/[S,a-z]* //g')<br />
## <br />
## Check if ERROR occured<br />
if [[ "${jobID}" =~ "ERROR" ]] ; then<br />
echo " -> submission failed!" ; exit 1<br />
else<br />
echo " -> job number = ${jobID}"<br />
fi<br />
##<br />
## Increase counter<br />
let myloop_counter+=1<br />
done<br />
</source><br />
<br><br />
<br />
==== GPU jobs ====<br />
<br />
The nodes in the gpu_4 and gpu_8 queues have 4 or 8 NVIDIA Tesla V100 GPUs. Just submitting a job to these queues is not enough to also allocate one or more GPUs, you have to do so using the "--gres=gpu" parameter. You have to specifiy how many GPUs your job needs, e.g. "--gres=gpu:2" will request two GPUs.<br />
<br />
The GPU nodes are shared between multiple jobs if the jobs don't request all the GPUs in a node and there are enough ressources to run more than one job. The individual GPUs are always bound to a single job and will not be shared between different jobs.<br />
<br />
a) add after the initial line of your script job.sh the line including the<br />
information about the GPU usage:<br> #SBATCH --gres=gpu:2<br />
<pre><br />
#!/bin/bash<br />
#SBATCH --ntasks=40<br />
#SBATCH --time=02:00:00<br />
#SBATCH --mem=4000<br />
#SBATCH --gres=gpu:2<br />
</pre><br />
<br />
or b) execute:<br />
<pre><br />
$ sbatch -p <queue> -n 40 -t 02:00:00 --mem 4000 --gres=gpu:2 job.sh<br />
</pre><br />
<br/><br />
If you start an interactive session on of the GPU nodes, you can use the "nvidia-smi" command to list the GPUs allocated to your job:<br />
<pre><br />
$ nvidia-smi<br />
Sun Mar 29 15:20:05 2020 <br />
+-----------------------------------------------------------------------------+<br />
| NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2 |<br />
|-------------------------------+----------------------+----------------------+<br />
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |<br />
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |<br />
|===============================+======================+======================|<br />
| 0 Tesla V100-SXM2... Off | 00000000:3A:00.0 Off | 0 |<br />
| N/A 29C P0 39W / 300W | 9MiB / 32510MiB | 0% Default |<br />
+-------------------------------+----------------------+----------------------+<br />
| 1 Tesla V100-SXM2... Off | 00000000:3B:00.0 Off | 0 |<br />
| N/A 30C P0 41W / 300W | 8MiB / 32510MiB | 0% Default |<br />
+-------------------------------+----------------------+----------------------+<br />
<br />
+-----------------------------------------------------------------------------+<br />
| Processes: GPU Memory |<br />
| GPU PID Type Process name Usage |<br />
|=============================================================================|<br />
| 0 14228 G /usr/bin/X 8MiB |<br />
| 1 14228 G /usr/bin/X 8MiB |<br />
+-----------------------------------------------------------------------------+<br />
</pre><br />
<br />
<br/><br />
In case of using Open MPI, the underlying communication infrastructure (UCX and Open MPI's BTL) is CUDA-aware.<br />
However, there may be warnings, e.g. when running<br />
<pre><br />
$ module load compiler/gnu/10.3 mpi/openmpi devel/cuad<br />
$ mpirun mpirun -np 2 ./mpi_cuda_app<br />
--------------------------------------<br />
WARNING: There are more than one active ports on host 'uc2n520', but the<br />
default subnet GID prefix was detected on more than one of these<br />
ports. If these ports are connected to different physical IB<br />
networks, this configuration will fail in Open MPI. This version of<br />
Open MPI requires that every physically separate IB subnet that is<br />
used between connected MPI processes must have different subnet ID<br />
values.<br />
<br />
Please see this FAQ entry for more details:<br />
<br />
http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid<br />
<br />
NOTE: You can turn off this warning by setting the MCA parameter<br />
btl_openib_warn_default_gid_prefix to 0.<br />
--------------------------------------------------------------------------<br />
</pre><br />
<br />
Please run Open MPI's mpirun using the following command:<br />
<pre><br />
$ mpirun --mca pml ucx --mca btl_openib_warn_default_gid_prefix 0 -np 2 ./mpi_cuda_app<br />
</pre><br />
or disabling the (older) communication layer Bit-Transfer-Layer (short BTL) alltogether:<br />
<pre><br />
$ mpirun --mca pml ucx --mca btl ^openib -np 2 ./mpi_cuda_app<br />
</pre><br />
<br />
(Please note, that CUDA per v11.4 is only available with up to GCC-10)<br />
<br><br />
<br><br />
<br />
==== LSDF Online Storage ====<br />
On bwUniCluster 2.0 you can use for special cases the LSDF Online Storage on the HPC cluster nodes. Please request for this service separately ([https://www.lsdf.kit.edu/os/storagerequest/: LSDF Storage Request]).<br />
To mount the LSDF Online Storage on the compute nodes during the job runtime the<br />
the constraint flag "LSDF" has to be set. <br />
<br />
a) add after the initial line of your script job.sh the line including the<br />
information about the LSDF Online Storage usage:<br> #SBATCH --constraint=LSDF<br />
<pre><br />
#!/bin/bash<br />
#SBATCH --ntasks=1<br />
#SBATCH --time=120<br />
#SBATCH --mem=200<br />
#SBATCH --constraint=LSDF<br />
</pre><br />
<br />
or b) execute:<br />
<pre><br />
$ sbatch -p <queue> -n 1 -t 2:00:00 --mem 200 job.sh -C LSDF<br />
</pre><br />
<br><br />
For the usage of the LSDF Online Storage<br />
the following environment variables are available: $LSDF, $LSDFPROJECTS, $LSDFHOME.<br />
<br><br />
<br><br />
<br />
====BeeOND (BeeGFS On-Demand)====<br />
<br />
BeeOND instances are integrated into the prolog and epilog script of the cluster batch system Slurm. It can be used on the compute nodes during the job runtime with the constraint flag "BEEOND", "BEEOND_4MDS" or "BEEOND_MAXMDS" ([[BwUniCluster_2.0_Slurm_common_Features#sbatch_Command_Parameters|Slurm Command Parameters]])<br />
* BEEOND: one metadata server is started on the first node<br />
* BEEOND_4MDS: 4 metadata servers are started within your job. If you have less than 4 nodes less metadata servers are started.<br />
* BEEOND_MAXMDS: on every node of your job a metadata server for the on_demand file system is started<br />
<br />
As starting point we recommend using the "BEEOND" option. If you are unsure if this is sufficient for you feel free to contact the support team.<br />
<source lang="bash"><br />
#!/bin/bash<br />
#SBATCH ...<br />
#SBATCH --constraint=BEEOND # or BEEOND_4MDS or BEEOND_MAXMDS<br />
</source><br />
<br />
After your job has started you can find the private on-demand file system in '''/mnt/odfs/${SLURM_JOB_ID}''' directory. The mountpoint comes with five pre-configured directories:<br />
<source lang="bash"><br />
# For small files (stripe count = 1)<br />
/mnt/odfs/${SLURM_JOB_ID}/stripe_1<br />
# Stripe count = 4<br />
/mnt/odfs/${SLURM_JOB_ID}/stripe_default <br />
# or <br />
/mnt/odfs/${SLURM_JOB_ID}/stripe_4<br />
# Stripe count = 8, 16 or 32, use this directories for medium sized and large files or when using MPI-IO<br />
/mnt/odfs/${SLURM_JOB_ID}/stripe_8<br />
/mnt/odfs/${SLURM_JOB_ID}/stripe_16 <br />
# or <br />
/mnt/odfs/${SLURM_JOB_ID}/stripe_32<br />
</source><br />
<br />
If you request less nodes than stripe count, the stripe count will be the number of nodes. For example, if you only request 8 nodes the directory stripe_16 has only a stripe count 8.<br />
<br />
The capacity of the private file system depends on the number of nodes. For each node you get 750 Gbyte.<br />
<br />
!!! Be careful when creating large files: use always the directory with the max stripe count for large files.<br />
If you create large files use a higher stripe count. For example, if your largest file is 1.1 Tb, then you have to use a stripe count larger than 2, otherwise the used disk space is exceeded. <br />
<br />
If you request 100 nodes for your job, the private file system is 100 * 750 Gbyte ~ 75 Tbyte (approx) capacity.<br />
<br />
'''Possible optimization:'''<br />
<br />
A possible optimization is that the private file system uses its own metadata server. With the constraint BEEOND the metadata server is started on the first node. Depending on your application, the metadata server could consume a decent amount of CPU power. In this case, adding a extra node to your job could improve the performance of the on-demand file system and the total runtime of your application. In order to use this option, start your application with the MPI option:<br />
<pre><br />
mpirun -nolocal myapplication<br />
</pre><br />
With the -nolocal option the node where mpirun is initiated is not used for your application. This node is fully available for the meta data server of your requested on-demand file system.<br />
<br />
Example job script:<br />
<source lang="bash"><br />
#!/bin/bash<br />
# Very simple example on how to use a private on-demand file system.<br />
#SBATCH -N 10<br />
#SBATCH --constraint=BEEOND<br />
<br />
# Create a workspace. <br />
ws_allocate myresults-${SLURM_JOB_ID} 90<br />
RESULTDIR=$(ws_find myresults-${SLURM_JOB_ID})<br />
<br />
# Set ENV variable to on-demand file system.<br />
ODFSDIR=/mnt/odfs/${SLURM_JOB_ID}/stripe_16/<br />
<br />
# Start application and write results to on-demand file system.<br />
mpirun -nolocal myapplication -o $ODFSDIR/results<br />
<br />
# Copy back data after your job application end.<br />
rsync -av $ODFSDIR/results $RESULTDIR<br />
</source><br />
<br><br />
<br><br />
<br />
== Start time of job or resources : squeue --start ==<br />
The command can be used by any user to displays the estimated start time of a job based a number of analysis types based on historical usage, earliest available reservable resources, and priority based backlog. The command squeue is explained in detail on the webpage https://slurm.schedmd.com/squeue.html or via manpage (man squeue). <br />
<br><br />
<br><br />
=== Access ===<br />
By default, this command can be run by '''any user'''. <br />
<br><br />
<br><br />
<br />
== List of your submitted jobs : squeue ==<br />
Displays information about YOUR active, pending and/or recently completed jobs. The command displays all own active and pending jobs. The command squeue is explained in detail on the webpage https://slurm.schedmd.com/squeue.html or via manpage (man squeue).<br />
<br><br />
<br><br />
=== Access ===<br />
By default, this command can be run by any user.<br />
<br><br />
<br><br />
<br />
=== Flags ===<br />
{| width=750px class="wikitable"<br />
|-<br />
! Flag !! Description<br />
|-<br />
| -l, --long<br />
| Report more of the available information for the selected jobs or job steps, subject to any constraints specified.<br />
|}<br />
<br><br />
<br />
=== Examples ===<br />
''squeue'' example on bwUniCluster 2.0 <small>(Only your own jobs are displayed!)</small>.<br />
<pre><br />
$ squeue <br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
18088744 single CPV.sbat ab1234 PD 0:00 1 (Priority)<br />
18098414 multiple CPV.sbat ab1234 PD 0:00 2 (Priority) <br />
18090089 multiple CPV.sbat ab1234 R 2:27 2 uc2n[127-128]<br />
$ squeue -l<br />
JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON) <br />
18088654 single CPV.sbat ab1234 COMPLETI 4:29 2:00:00 1 uc2n374<br />
18088785 single CPV.sbat ab1234 PENDING 0:00 2:00:00 1 (Priority)<br />
18098414 multiple CPV.sbat ab1234 PENDING 0:00 2:00:00 2 (Priority)<br />
18088683 single CPV.sbat ab1234 RUNNING 0:14 2:00:00 1 uc2n413 <br />
</pre><br />
* The output of ''squeue'' shows how many jobs of yours are running or pending and how many nodes are in use by your jobs.<br />
<br><br />
<br />
== Shows free resources : sinfo_t_idle ==<br />
The Slurm command sinfo is used to view partition and node information for a system running Slurm. It incorporates down time, reservations, and node state information in determining the available backfill window. The sinfo command can only be used by the administrator.<br />
<br><br />
SCC has prepared a special script (sinfo_t_idle) to find out how many processors are available for immediate use on the system. It is anticipated that users will use this information to submit jobs that meet these criteria and thus obtain quick job turnaround times. <br />
<br><br />
<br><br />
=== Access ===<br />
By default, this command can be used by any user or administrator. <br />
<br><br />
<br><br />
=== Example ===<br />
* The following command displays what resources are available for immediate use for the whole partition.<br />
<pre>$ sinfo_t_idle<br />
Partition dev_multiple : 8 nodes idle<br />
Partition multiple : 332 nodes idle<br />
Partition dev_single : 4 nodes idle<br />
Partition single : 76 nodes idle<br />
Partition long : 80 nodes idle<br />
Partition fat : 5 nodes idle<br />
Partition dev_special : 342 nodes idle<br />
Partition special : 342 nodes idle<br />
Partition dev_multiple_e: 7 nodes idle<br />
Partition multiple_e : 335 nodes idle<br />
Partition gpu_4 : 12 nodes idle<br />
Partition gpu_8 : 6 nodes idle<br />
</pre><br />
* For the above example jobs in all partitions can be run immediately.<br />
<br><br />
<br />
== Detailed job information : scontrol show job ==<br />
scontrol show job displays detailed job state information and diagnostic output for all or a specified job of yours. Detailed information is available for active, pending and recently completed jobs. The command scontrol is explained in detail on the webpage https://slurm.schedmd.com/scontrol.html or via manpage (man scontrol). <br />
<br><br />
Display the state of all your jobs in normal mode: scontrol show job<br />
<br><br />
Display the state of a job with <jobid> in normal mode: scontrol show job <jobid><br />
<br><br />
<br><br />
=== Access ===<br />
* End users can use scontrol show job to view the status of their '''own jobs''' only. <br />
<br><br />
<br />
=== Arguments ===<br />
{| width=750px class="wikitable"<br />
|-<br />
! Option !! Default !! Description !! Example<br />
|- style="vertical-align:top;"<br />
|- style="width:12%;" <br />
| -d<br />
| (n/a)<br />
| Detailed mode<br />
| Example: Display the state with jobid 18089884 in detailed mode. <br> <pre>scontrol -d show job 18089884</pre><br />
|}<br />
<br><br />
<br><br />
<br />
=== Scontrol show job Example ===<br />
Here is an example from bwUniCluster 2.0.<br />
<pre><br />
squeue # show my own jobs (here the userid is replaced!)<br />
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br />
18089884 multiple CPV.sbat bq0742 R 33:44 2 uc2n[165-166]<br />
<br />
$<br />
$ # now, see what's up with my pending job with jobid 18089884<br />
$ <br />
$ scontrol show job 18089884<br />
<br />
JobId=18089884 JobName=CPV.sbatch<br />
UserId=bq0742(8946) GroupId=scc(12345) MCS_label=N/A<br />
Priority=3 Nice=0 Account=kit QOS=normal<br />
JobState=RUNNING Reason=None Dependency=(null)<br />
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0<br />
RunTime=00:35:06 TimeLimit=02:00:00 TimeMin=N/A<br />
SubmitTime=2020-03-16T14:14:54 EligibleTime=2020-03-16T14:14:54<br />
AccrueTime=2020-03-16T14:14:54<br />
StartTime=2020-03-16T15:12:51 EndTime=2020-03-16T17:12:51 Deadline=N/A<br />
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2020-03-16T15:12:51<br />
Partition=multiple AllocNode:Sid=uc2n995:5064<br />
ReqNodeList=(null) ExcNodeList=(null)<br />
NodeList=uc2n[165-166]<br />
BatchHost=uc2n165<br />
NumNodes=2 NumCPUs=160 NumTasks=80 CPUs/Task=1 ReqB:S:C:T=0:0:*:1<br />
TRES=cpu=160,mem=96320M,node=2,billing=160<br />
Socks/Node=* NtasksPerN:B:S:C=40:0:*:1 CoreSpec=*<br />
MinCPUsNode=40 MinMemoryCPU=1204M MinTmpDiskNode=0<br />
Features=(null) DelayBoot=00:00:00<br />
OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)<br />
Command=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/CPV.sbatch<br />
WorkDir=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin<br />
StdErr=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/slurm-18089884.out<br />
StdIn=/dev/null<br />
StdOut=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/slurm-18089884.out<br />
Power=<br />
MailUser=(null) MailType=NONE<br />
</pre><br />
<br><br />
You can use standard Linux pipe commands to filter the very detailed scontrol show job output.<br />
* In which state the job is?<br />
<pre>$ scontrol show job 18089884 | grep -i State<br />
JobState=COMPLETED Reason=None Dependency=(null)<br />
</pre><br />
<br><br />
<br />
== Cancel Slurm Jobs ==<br />
The scancel command is used to cancel jobs. The command scancel is explained in detail on the webpage https://slurm.schedmd.com/scancel.html or via manpage (man scancel). <br />
<br><br />
<br><br />
=== Canceling own jobs : scancel ===<br />
scancel is used to signal or cancel jobs, job arrays or job steps. The command is:<br />
<pre><br />
$ scancel [-i] <job-id><br />
$ scancel -t <job_state_name><br />
</pre><br />
<br />
<br><br />
{| width=750px class="wikitable"<br />
! Flag !! Default !! Description !! Example<br />
|- style="vertical-align:top;"<br />
| -i, --interactive<br />
| (n/a)<br />
| Interactive mode.<br />
| Cancel the job 987654 interactively. <br> <pre> scancel -i 987654 </pre><br />
|-<br />
| -t, --state<br />
| (n/a)<br />
| Restrict the scancel operation to jobs in a certain state. <br> "job_state_name" may have a value of either "PENDING", "RUNNING" or "SUSPENDED".<br />
| Cancel all jobs in state "PENDING". <br> <pre> scancel -t "PENDING" </pre><br />
|}<br />
<br><br />
<br />
= Resource Managers =<br />
=== Batch Job (Slurm) Variables ===<br />
The following environment variables of Slurm are added to your environment once your job has started<br />
<small>(only an excerpt of the most important ones)</small>.<br />
{| width=750px class="wikitable"<br />
! Environment !! Brief explanation<br />
|- <br />
| SLURM_JOB_CPUS_PER_NODE <br />
| Number of processes per node dedicated to the job<br />
|- <br />
| SLURM_JOB_NODELIST <br />
| List of nodes dedicated to the job<br />
|- <br />
| SLURM_JOB_NUM_NODES <br />
| Number of nodes dedicated to the job<br />
|- <br />
| SLURM_MEM_PER_NODE <br />
| Memory per node dedicated to the job <br />
|- <br />
| SLURM_NPROCS<br />
| Total number of processes dedicated to the job <br />
|-<br />
| SLURM_CLUSTER_NAME<br />
| Name of the cluster executing the job<br />
|-<br />
| SLURM_CPUS_PER_TASK <br />
| Number of CPUs requested per task<br />
|-<br />
| SLURM_JOB_ACCOUNT<br />
| Account name <br />
|-<br />
| SLURM_JOB_ID<br />
| Job ID<br />
|-<br />
| SLURM_JOB_NAME<br />
| Job Name<br />
|-<br />
| SLURM_JOB_PARTITION<br />
| Partition/queue running the job<br />
|-<br />
| SLURM_JOB_UID<br />
| User ID of the job's owner<br />
|-<br />
| SLURM_SUBMIT_DIR<br />
| Job submit folder. The directory from which msub was invoked. <br />
|-<br />
| SLURM_JOB_USER<br />
| User name of the job's owner<br />
|-<br />
| SLURM_RESTART_COUNT<br />
| Number of times job has restarted<br />
|-<br />
| SLURM_PROCID<br />
| Task ID (MPI rank)<br />
|-<br />
| SLURM_NTASKS<br />
| The total number of tasks available for the job<br />
|-<br />
| SLURM_STEP_ID<br />
| Job step ID<br />
|-<br />
| SLURM_STEP_NUM_TASKS<br />
| Task count (number of MPI ranks)<br />
|-<br />
| SLURM_JOB_CONSTRAINT<br />
| Job constraints<br />
|}<br />
See also:<br />
* [https://slurm.schedmd.com/sbatch.html#lbAI Slurm input and output environment variables]<br />
<br><br />
<br />
=== Job Exit Codes ===<br />
A job's exit code (also known as exit status, return code and completion code) is captured by SLURM and saved as part of the job record. <br />
<br><br />
Any non-zero exit code will be assumed to be a job failure and will result in a Job State of FAILED with a reason of "NonZeroExitCode".<br />
<br><br />
The exit code is an 8 bit unsigned number ranging between 0 and 255. While it is possible for a job to return a negative exit code, SLURM will display it as an unsigned value in the 0 - 255 range.<br />
<br><br />
<br><br />
==== Displaying Exit Codes and Signals ====<br />
SLURM displays a job's exit code in the output of the '''scontrol show job''' and the sview utility.<br />
<br> <br />
When a signal was responsible for a job or step's termination, the signal number will be displayed after the exit code, delineated by a colon(:).<br />
<br><br />
<br><br />
==== Submitting Termination Signal ====<br />
Here is an example, how to 'save' a Slurm termination signal in a typical jobscript.<br />
<source lang="bash"><br />
[...]<br />
exit_code=$?<br />
mpirun -np <#cores> <EXE_BIN_DIR>/<executable> ... (options) 2>&1<br />
[ "$exit_code" -eq 0 ] && echo "all clean..." || \<br />
echo "Executable <EXE_BIN_DIR>/<executable> finished with exit code ${$exit_code}"<br />
[...]<br />
</source><br />
* Do not use ''''time'''' mpirun! The exit code will be the one submitted by the first (time) program.<br />
* You do not need an '''exit $exit_code''' in the scripts.<br />
<br><br />
<br><br />
----<br />
[[Category:bwUniCluster 2.0|bwUniCluster 2.0]]<br />
[[#top|Back to top]]</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance/2023-03&diff=11746
BwUniCluster2.0/Maintenance/2023-03
2023-02-07T13:15:01Z
<p>P Weisbrod: </p>
<hr />
<div>The following changes will be introduced during the maintenance interval between on 20.03.2023 (Monday) 08:30 and 24.03.2023 (Friday) 15:00.<br />
<br />
The host key of the system will not change. You should not receive any warnings by your SSH client(s), but if there should be a warning or if you want to check that you are connecting to the correct system, you can verify the key hashes using the following list:<br />
<br />
{|class="wikitable"<br />
! Algorithm<br />
! Hash (SHA256)<br />
! Hash (MD5)<br />
|-<br />
|RSA<br />
|p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|59:2a:67:44:4a:d7:89:6c:c0:0d:74:ba:3c:c4:63:6d<br />
|-<br />
|ECDSA<br />
|k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|85:d4:d9:97:e0:f0:43:30:6e:66:8e:d0:b6:9b:85:d1<br />
|-<br />
|ED25519<br />
|yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|42:d2:0d:ab:87:48:fc:1d:5d:b3:7c:bf:22:c3:5f:b7<br />
|}<br />
<br />
= Hardware =<br />
<br />
* All firmware versions on all components will be upgraded<br />
<br />
= Operating system =<br />
<br />
* The operating system will be upgraded from RHEL 8.4 EUS to RHEL 8.6 EUS<br />
<br />
* The Mellanox OFED InfiniBand Stack will be updated<br />
<br />
= Compilers, Libaries and Runtime Environments =<br />
<br />
<br />
= Userspace tools =<br />
<br />
<br />
= Software Modules =<br />
<br />
<br />
= Batch system =<br />
<br />
<br />
= Storage =<br />
<br />
<br />
= Graphics stack =<br />
<br />
* The NVIDIA driver will be upgraded<br />
<br />
= Containers =<br />
<br />
<br />
<br />
= JupyterHub =</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance/2023-03&diff=11745
BwUniCluster2.0/Maintenance/2023-03
2023-02-07T13:11:44Z
<p>P Weisbrod: Created page with "The following changes have been introduced during the maintenance interval between on 07.11.2022 (Monday) 08:00 and 10.11.2022 (Thursday) 17:00. The host key of the system ha..."</p>
<hr />
<div>The following changes have been introduced during the maintenance interval between on 07.11.2022 (Monday) 08:00 and 10.11.2022 (Thursday) 17:00.<br />
<br />
The host key of the system have not changed. You should not receive any warnings by your SSH client(s), but if there should be a warning or if you want to check that you are connecting to the correct system, you can verify the key hashes using the following list:<br />
<br />
{|class="wikitable"<br />
! Algorithm<br />
! Hash (SHA256)<br />
! Hash (MD5)<br />
|-<br />
|RSA<br />
|p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|59:2a:67:44:4a:d7:89:6c:c0:0d:74:ba:3c:c4:63:6d<br />
|-<br />
|ECDSA<br />
|k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|85:d4:d9:97:e0:f0:43:30:6e:66:8e:d0:b6:9b:85:d1<br />
|-<br />
|ED25519<br />
|yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|42:d2:0d:ab:87:48:fc:1d:5d:b3:7c:bf:22:c3:5f:b7<br />
|}<br />
<br />
= Hardware =<br />
<br />
* All firmware versions on all components were upgraded<br />
<br />
= Operating system =<br />
<br />
* The operating system remains at RHEL 8.4 EUS<br />
<br />
* The Mellanox OFED InfiniBand Stack was updated to version 5.5-2.1.7.0<br />
<br />
= Compilers, Libaries and Runtime Environments =<br />
* The Intel Parallel Studio (Compiler, MKL, MPI) version 2019 modules were removed during the maintenance.<br />
* clang 9 and llvm 10 modules were removed<br />
<br />
= Userspace tools =<br />
<br />
<br />
= Software Modules =<br />
<br />
<br />
= Batch system =<br />
<br />
* The Slurm version was upgraded to version 22.05.5<br />
* Pyxis Plugin version was upgraded to 0.14.0<br />
<br />
= Storage =<br />
* Lustre client, BeeGFS client and Spectrum Scale client were updated<br />
<br />
= Graphics stack =<br />
<br />
* The NVIDIA driver was upgraded to version 515.65.07<br />
* Cuda 11.7 was installed<br />
<br />
= Containers =<br />
<br />
<br />
<br />
= JupyterHub =</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Batch_Queues&diff=11488
BwUniCluster2.0/Batch Queues
2022-11-25T14:57:09Z
<p>P Weisbrod: </p>
<hr />
<div>=== sbatch -p ''queue'' ===<br />
Compute resources such as (wall-)time, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, '''you must add the correct queue class to your sbatch command '''. <font color=red>The specification of a queue is obligatory on BwUniCluster 2.0.</font> <br />
<br><br />
Details are:<br />
<br />
{| width=750px class="wikitable"<br />
! colspan="5" | bwUniCluster 2.0 <br> sbatch -p ''queue''<br />
|- style="text-align:left;"<br />
! queue !! node !! default resources !! minimum resources !! maximum resources<br />
|- style="text-align:left"<br />
| dev_single<br />
| thin<br />
| time=10, mem-per-cpu=1125mb<br />
| <br />
| time=30, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core=2)<br>6 nodes are reserved for this queue. <br> Only for development, i.e. debugging or performance optimization ...<br />
|- style="text-align:left;"<br />
| single<br />
| thin<br />
| time=30, mem-per-cpu=1125mb<br />
| <br />
| time=72:00:00, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core)=2<br />
|- style="text-align:left;"<br />
| dev_multiple<br />
| thin<br />
| time=10, mem-per-cpu=1125mb<br />
| nodes=2<br />
| time=30, nodes=4, mem=90000mb, ntasks-per-node=40, (threads-per-core=2)<br>8 nodes are reserved for this queue.<br> Only for development, i.e. debugging or performance optimization ...<br />
|- style="text-align:left;"<br />
| multiple<br />
| thin<br />
| time=30, mem-per-cpu=1125mb<br />
| nodes=2<br />
| time=72:00:00, mem=90000mb, nodes=128, ntasks-per-node=40, (threads-per-core=2) <br />
|- style="text-align:left;"<br />
| dev_multiple_il<br />
| thin (IceLake)<br />
| time=10, mem-per-cpu=1950mb<br />
| nodes=2<br />
| time=30, nodes=8, mem=249600mb, ntasks-per-node=64, (threads-per-core=2)<br>8 nodes are reserved for this queue <br> Only for development, i.e. debugging or performance optimization ...<br />
|- style="text-align:left;"<br />
| multiple_il<br />
| thin (IceLake)<br />
| time=10, mem-per-cpu=1950mb<br />
| nodes=2<br />
| time=72:00:00, nodes=128, mem=249600mb, ntasks-per-node=64, (threads-per-core=2)<br />
|- style="text-align:left;"<br />
| dev_gpu_4_a100<br />
| gpu4 (IceLake + A100)<br />
| time=10, mem-per-gpu=127500mb, cpu-per-gpu=16<br />
| <br />
| time=30, nodes=1, mem=510000mb, ntasks-per-node=64, (threads-per-core=2)<br />
|- style="text-align:left;"<br />
| gpu_4_a100<br />
| gpu4 (IceLake + A100)<br />
| time=10, mem-per-gpu=127500mb, cpu-per-gpu=16<br />
| <br />
| time=48:00:00, nodes=9, mem=510000mb, ntasks-per-node=64, (threads-per-core=2)<br />
|- style="vertical-align:top; text-align:left"<br />
| fat <br />
| fat<br />
| time=10, mem-per-cpu=18750mb<br />
| mem=180001mb<br />
| time=72:00:00, nodes=1, ntasks-per-node=80, (threads-per-core=2)<br />
|- style="vertical-align:top; text-align:left"<br />
| dev_gpu_4<br />
| gpu4<br />
| time=10, mem-per-gpu=94000mb, cpu-per-gpu=10<br />
| <br />
| time=30, nodes=1, mem=376000, ntasks-per-node=40, (threads-per-core=2)<br>1 node is reserved for this queue <br> Only for development, i.e. debugging or performance optimization ...<br />
|- style="text-align:left;"<br />
| gpu_4<br />
| gpu4<br />
| time=10, mem-per-gpu=94000mb, cpu-per-gpu=10<br />
| <br />
| time=48:00:00, mem=376000, nodes=14, ntasks-per-node=40, (threads-per-core=2)<br />
|- style="vertical-align:top; text-align:left"<br />
| gpu_8<br />
| gpu8<br />
| time=10, mem-per-cpu=94000mb, cpu-per-gpu=10<br />
|<br />
| time=48:00:00, mem=752000, nodes=10, ntasks-per-node=40, (threads-per-core=2)<br />
|- <br />
|}<br />
<br />
Default resources of a queue class defines time, #tasks and memory if not explicitly given with sbatch command. Resource list acronyms ''--time'', ''--ntasks'', ''--nodes'', ''--mem'' and ''--mem-per-cpu'' are described [[BwUniCluster_2.0_Slurm_common_Features|here]].<br />
<br><br />
<br><br />
==== Queue class examples ====<br />
To run your batch job on one of the thin nodes, please use:<br />
<br />
<pre><br />
$ sbatch --partition=dev_multiple<br />
or <br />
$ sbatch -p dev_multiple<br />
</pre><br />
<br><br />
<br />
==== Interactive Jobs ====<br />
On bwUniCluster 2.0 you are only allowed to run short jobs (<< 1 hour) with little memory requirements (<< 8 GByte) on the logins nodes. If you want to run longer jobs and/or jobs with a request of more than 8 GByte of memory, you must allocate resources for so-called interactive jobs by usage of the command salloc on a login node. Considering a serial application running on a compute node that requires 5000 MByte of memory and limiting the interactive run to 2 hours the following command has to be executed:<br />
<pre><br />
$ salloc -p single -n 1 -t 120 --mem=5000<br />
</pre><br />
Then you will get one core on a compute node within the partition "single". After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system Slurm has granted you the requested resources on the compute system. You will be logged in automatically on the granted core! To run a serial program on the granted core you only have to type the name of the executable.<br />
<pre><br />
$ ./<my_serial_program><br />
</pre><br />
Please be aware that your serial job must run less than 2 hours in this example, else the job will be killed during runtime by the system. <br />
<br><br />
You can also start now a graphical X11-terminal connecting you to the dedicated resource that is available for 2 hours. You can start it by the command:<br />
<pre><br />
$ xterm<br />
</pre><br />
Note that, once the walltime limit has been reached the resources - i.e. the compute node - will automatically be revoked.<br />
<br><br />
An interactive parallel application running on one compute node or on many compute nodes (e.g. here 5 nodes) with 40 cores each requires usually an amount of memory in GByte (e.g. 50 GByte) and a maximum time (e.g. 1 hour). E.g. 5 nodes can be allocated by the following command:<br />
<pre><br />
$ salloc -p multiple -N 5 --ntasks-per-node=40 -t 01:00:00 --mem=50gb<br />
</pre><br />
Now you can run parallel jobs on 200 cores requiring 50 GByte of memory per node. Please be aware that you will be logged in on core 0 of the first node.<br />
If you want to have access to another node you have to open a new terminal, connect it also to BwUniCluster 2.0 and type the following commands to<br />
connect to the running interactive job and then to a specific node:<br />
<pre><br />
$ srun --jobid=XXXXXXXX --pty /bin/bash<br />
$ srun --nodelist=uc2nXXX --pty /bin/bash<br />
</pre><br />
With the command:<br />
<pre><br />
$ squeue<br />
</pre><br />
the jobid and the nodelist can be shown.<br />
<br />
If you want to run MPI-programs, you can do it by simply typing mpirun <program_name>. Then your program will be run on 200 cores. A very simple example for starting a parallel job can be:<br />
<pre><br />
$ mpirun <my_mpi_program><br />
</pre><br />
You can also start the debugger ddt by the commands:<br />
<pre><br />
$ module add devel/ddt<br />
$ ddt <my_mpi_program><br />
</pre><br />
The above commands will execute the parallel program <my_mpi_program> on all available cores. You can also start parallel programs on a subset of cores; an example for this can be:<br />
<pre><br />
$ mpirun -n 50 <my_mpi_program><br />
</pre><br />
If you are using Intel MPI you must start <my_mpi_program> by the command mpiexec.hydra (instead of mpirun).<br />
<br><br />
<br><br />
<br />
<br />
----<br />
[[Category:bwUniCluster 2.0|Batch Jobs - bwUniCluster 2.0 Features]]</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Batch_Queues&diff=11486
BwUniCluster2.0/Batch Queues
2022-11-25T14:56:30Z
<p>P Weisbrod: </p>
<hr />
<div>=== sbatch -p ''queue'' ===<br />
Compute resources such as (wall-)time, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, '''you must add the correct queue class to your sbatch command '''. <font color=red>The specification of a queue is obligatory on BwUniCluster 2.0.</font> <br />
<br><br />
Details are:<br />
<br />
{| width=750px class="wikitable"<br />
! colspan="5" | bwUniCluster 2.0 <br> sbatch -p ''queue''<br />
|- style="text-align:left;"<br />
! queue !! node !! default resources !! minimum resources !! maximum resources<br />
|- style="text-align:left"<br />
| dev_single<br />
| thin<br />
| time=10, mem-per-cpu=1125mb<br />
| <br />
| time=30, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core=2)<br>6 nodes are reserved for this queue. <br> Only for development, i.e. debugging or performance optimization ...<br />
|- style="text-align:left;"<br />
| single<br />
| thin<br />
| time=30, mem-per-cpu=1125mb<br />
| <br />
| time=72:00:00, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core)=2<br />
|- style="text-align:left;"<br />
| dev_multiple<br />
| thin<br />
| time=10, mem-per-cpu=1125mb<br />
| nodes=2<br />
| time=30, nodes=4, mem=90000mb, ntasks-per-node=40, (threads-per-core=2)<br>8 nodes are reserved for this queue.<br> Only for development, i.e. debugging or performance optimization ...<br />
|- style="text-align:left;"<br />
| multiple<br />
| thin<br />
| time=30, mem-per-cpu=1125mb<br />
| nodes=2<br />
| time=72:00:00, mem=90000mb, nodes=128, ntasks-per-node=40, (threads-per-core=2) <br />
|- style="text-align:left;"<br />
| dev_multiple_il<br />
| thin (IceLake)<br />
| time=10, mem-per-cpu=1950mb<br />
| nodes=2<br />
| time=30, nodes=8, mem=249600mb, ntasks-per-node=64, (threads-per-core=2)<br>8 nodes are reserved for this queue <br> Only for development, i.e. debugging or performance optimization ...<br />
|- style="text-align:left;"<br />
| multiple_il<br />
| thin (IceLake)<br />
| time=10, mem-per-cpu=1950mb<br />
| nodes=2<br />
| time=72:00:00, nodes=128, mem=249600mb, ntasks-per-node=64, (threads-per-core=2)<br />
|- style="text-align:left;"<br />
| dev_gpu_4_a100<br />
| gpu4 (IceLake + A100)<br />
| time=10, mem-per-gpu=127500mb, cpu-per-gpu=16<br />
| <br />
| time=30, nodes=1, mem=510000mb, ntasks-per-node=64, (threads-per-core=2)<br />
|- style="text-align:left;"<br />
| gpu_4_a100<br />
| gpu4 (IceLake + A100)<br />
| time=10, mem-per-gpu=127500mb, cpu-per-gpu=16<br />
| <br />
| time=48:00:00, nodes=9, mem=510000mb, ntasks-per-node=64, (threads-per-core=2)<br />
|- style="vertical-align:top; text-align:left"<br />
| fat <br />
| fat<br />
| time=10, mem-per-cpu=18750mb<br />
| mem=180001mb<br />
| time=72:00:00, nodes=1, ntasks-per-node=80, (threads-per-core=2)<br />
|- style="vertical-align:top; text-align:left"<br />
| dev_gpu_4<br />
| gpu4<br />
| time=10, mem-per-gpu=94000mb, cpu-per-gpu=10<br />
| <br />
| time=30, nodes=1, mem=376000, ntasks-per-node=40, (threads-per-core=2)<br>1 node is reserved for this queue <br> Only for development, i.e. debugging or performance optimization ...<br />
|- style="text-align:left;"<br />
| gpu_4<br />
| gpu4<br />
| time=10, mem-per-gpu=94000mb, cpu-per-gpu=10<br />
| <br />
| time=48:00:00, mem=376000, nodes=14, ntasks-per-node=40, (threads-per-core=2)<br />
|- style="vertical-align:top; text-align:left"<br />
| gpu_8<br />
| gpu8<br />
| time=10, mem-per-cpu=94000mb, cpu-per-gpu=10<br />
|<br />
| time=48:00:00, mem=752000, nodes=10, ntasks-per-node=40, (threads-per-core=2)<br />
|- <br />
|}<br />
<br />
Default resources of a queue class defines time, #tasks and memory if not explicitly given with sbatch command. Resource list acronyms ''--time'', ''--ntasks'', ''--nodes'', ''--mem'' and ''--mem-per-cpu'' are described [[BwUniCluster_2.0_Slurm_common_Features|here]].<br />
<br />
Access to the "special" and "dev_special" partitions on the bwUniCluster 2.0 is restricted to members of the institutions which participated in the procurement of the extension partition specifically for this purpose. Please contact the support team if your institution participated in the procurement and your account should be able to run jobs in this partition.<br />
<br><br />
<br><br />
==== Queue class examples ====<br />
To run your batch job on one of the thin nodes, please use:<br />
<br />
<pre><br />
$ sbatch --partition=dev_multiple<br />
or <br />
$ sbatch -p dev_multiple<br />
</pre><br />
<br><br />
<br />
==== Interactive Jobs ====<br />
On bwUniCluster 2.0 you are only allowed to run short jobs (<< 1 hour) with little memory requirements (<< 8 GByte) on the logins nodes. If you want to run longer jobs and/or jobs with a request of more than 8 GByte of memory, you must allocate resources for so-called interactive jobs by usage of the command salloc on a login node. Considering a serial application running on a compute node that requires 5000 MByte of memory and limiting the interactive run to 2 hours the following command has to be executed:<br />
<pre><br />
$ salloc -p single -n 1 -t 120 --mem=5000<br />
</pre><br />
Then you will get one core on a compute node within the partition "single". After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system Slurm has granted you the requested resources on the compute system. You will be logged in automatically on the granted core! To run a serial program on the granted core you only have to type the name of the executable.<br />
<pre><br />
$ ./<my_serial_program><br />
</pre><br />
Please be aware that your serial job must run less than 2 hours in this example, else the job will be killed during runtime by the system. <br />
<br><br />
You can also start now a graphical X11-terminal connecting you to the dedicated resource that is available for 2 hours. You can start it by the command:<br />
<pre><br />
$ xterm<br />
</pre><br />
Note that, once the walltime limit has been reached the resources - i.e. the compute node - will automatically be revoked.<br />
<br><br />
An interactive parallel application running on one compute node or on many compute nodes (e.g. here 5 nodes) with 40 cores each requires usually an amount of memory in GByte (e.g. 50 GByte) and a maximum time (e.g. 1 hour). E.g. 5 nodes can be allocated by the following command:<br />
<pre><br />
$ salloc -p multiple -N 5 --ntasks-per-node=40 -t 01:00:00 --mem=50gb<br />
</pre><br />
Now you can run parallel jobs on 200 cores requiring 50 GByte of memory per node. Please be aware that you will be logged in on core 0 of the first node.<br />
If you want to have access to another node you have to open a new terminal, connect it also to BwUniCluster 2.0 and type the following commands to<br />
connect to the running interactive job and then to a specific node:<br />
<pre><br />
$ srun --jobid=XXXXXXXX --pty /bin/bash<br />
$ srun --nodelist=uc2nXXX --pty /bin/bash<br />
</pre><br />
With the command:<br />
<pre><br />
$ squeue<br />
</pre><br />
the jobid and the nodelist can be shown.<br />
<br />
If you want to run MPI-programs, you can do it by simply typing mpirun <program_name>. Then your program will be run on 200 cores. A very simple example for starting a parallel job can be:<br />
<pre><br />
$ mpirun <my_mpi_program><br />
</pre><br />
You can also start the debugger ddt by the commands:<br />
<pre><br />
$ module add devel/ddt<br />
$ ddt <my_mpi_program><br />
</pre><br />
The above commands will execute the parallel program <my_mpi_program> on all available cores. You can also start parallel programs on a subset of cores; an example for this can be:<br />
<pre><br />
$ mpirun -n 50 <my_mpi_program><br />
</pre><br />
If you are using Intel MPI you must start <my_mpi_program> by the command mpiexec.hydra (instead of mpirun).<br />
<br><br />
<br><br />
<br />
<br />
----<br />
[[Category:bwUniCluster 2.0|Batch Jobs - bwUniCluster 2.0 Features]]</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Batch_Queues&diff=11484
BwUniCluster2.0/Batch Queues
2022-11-25T14:53:07Z
<p>P Weisbrod: </p>
<hr />
<div>=== sbatch -p ''queue'' ===<br />
Compute resources such as (wall-)time, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, '''you must add the correct queue class to your sbatch command '''. <font color=red>The specification of a queue is obligatory on BwUniCluster 2.0.</font> <br />
<br><br />
Details are:<br />
<br />
{| width=750px class="wikitable"<br />
! colspan="5" | bwUniCluster 2.0 <br> sbatch -p ''queue''<br />
|- style="text-align:left;"<br />
! queue !! node !! default resources !! minimum resources !! maximum resources<br />
|- style="text-align:left"<br />
| dev_single<br />
| thin<br />
| time=10, mem-per-cpu=1125mb<br />
| <br />
| time=30, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core=2)<br>6 nodes are reserved for this queue. <br> Only for development, i.e. debugging or performance optimization ...<br />
|- style="text-align:left;"<br />
| single<br />
| thin<br />
| time=30, mem-per-cpu=1125mb<br />
| <br />
| time=72:00:00, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core)=2<br />
|- style="text-align:left;"<br />
| dev_multiple<br />
| thin<br />
| time=10, mem-per-cpu=1125mb<br />
| nodes=2<br />
| time=30, nodes=4, mem=90000mb, ntasks-per-node=40, (threads-per-core=2)<br>8 nodes are reserved for this queue.<br> Only for development, i.e. debugging or performance optimization ...<br />
|- style="text-align:left;"<br />
| multiple<br />
| thin<br />
| time=30, mem-per-cpu=1125mb<br />
| nodes=2<br />
| time=72:00:00, mem=90000mb, nodes=128, ntasks-per-node=40, (threads-per-core=2) <br />
|- style="text-align:left;"<br />
| dev_multiple_il<br />
| thin (IceLake)<br />
| time=10, mem-per-cpu=1950mb<br />
| nodes=2<br />
| time=30, nodes=8, mem=249600mb, ntasks-per-node=64, (threads-per-core=2)<br>8 nodes are reserved for this queue <br> Only for development, i.e. debugging or performance optimization ...<br />
|- style="text-align:left;"<br />
| multiple_il<br />
| thin (IceLake)<br />
| time=10, mem-per-cpu=1950mb<br />
| nodes=2<br />
| time=72:00:00, nodes=128, mem=249600mb, ntasks-per-node=64, (threads-per-core=2)<br />
|- style="text-align:left;"<br />
| dev_gpu_4_a100<br />
| gpu4 (IceLake + A100)<br />
| time=10, mem-per-gpu=127500mb, cpu-per-gpu=16<br />
| <br />
| time=30, nodes=1, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)<br />
|- style="text-align:left;"<br />
| gpu_4_a100<br />
| gpu4 (IceLake + A100)<br />
| time=10, mem-per-gpu=127500mb, cpu-per-gpu=16<br />
| <br />
| time=48:00:00, nodes=9, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)<br />
|- style="vertical-align:top; text-align:left"<br />
| fat <br />
| fat<br />
| time=10, mem-per-cpu=18750mb<br />
| mem=180001mb<br />
| time=72:00:00, nodes=1, ntasks-per-node=80, (threads-per-core=2)<br />
|- style="vertical-align:top; text-align:left"<br />
| dev_gpu_4<br />
| gpu4<br />
| time=10, mem-per-gpu=94000mb, cpu-per-gpu=10<br />
| <br />
| time=30, nodes=1, mem=376000, ntasks-per-node=40, (threads-per-core=2)<br>1 node is reserved for this queue <br> Only for development, i.e. debugging or performance optimization ...<br />
|- style="text-align:left;"<br />
| gpu_4<br />
| gpu4<br />
| time=10, mem-per-gpu=94000mb, cpu-per-gpu=10<br />
| <br />
| time=48:00:00, mem=376000, nodes=14, ntasks-per-node=40, (threads-per-core=2)<br />
|- style="vertical-align:top; text-align:left"<br />
| gpu_8<br />
| gpu8<br />
| time=10, mem-per-cpu=94000mb, cpu-per-gpu=10<br />
|<br />
| time=48:00:00, mem=752000, nodes=10, ntasks-per-node=40, (threads-per-core=2)<br />
|- <br />
|}<br />
<br />
Default resources of a queue class defines time, #tasks and memory if not explicitly given with sbatch command. Resource list acronyms ''--time'', ''--ntasks'', ''--nodes'', ''--mem'' and ''--mem-per-cpu'' are described [[BwUniCluster_2.0_Slurm_common_Features|here]].<br />
<br />
Access to the "special" and "dev_special" partitions on the bwUniCluster 2.0 is restricted to members of the institutions which participated in the procurement of the extension partition specifically for this purpose. Please contact the support team if your institution participated in the procurement and your account should be able to run jobs in this partition.<br />
<br><br />
<br><br />
==== Queue class examples ====<br />
To run your batch job on one of the thin nodes, please use:<br />
<br />
<pre><br />
$ sbatch --partition=dev_multiple<br />
or <br />
$ sbatch -p dev_multiple<br />
</pre><br />
<br><br />
<br />
==== Interactive Jobs ====<br />
On bwUniCluster 2.0 you are only allowed to run short jobs (<< 1 hour) with little memory requirements (<< 8 GByte) on the logins nodes. If you want to run longer jobs and/or jobs with a request of more than 8 GByte of memory, you must allocate resources for so-called interactive jobs by usage of the command salloc on a login node. Considering a serial application running on a compute node that requires 5000 MByte of memory and limiting the interactive run to 2 hours the following command has to be executed:<br />
<pre><br />
$ salloc -p single -n 1 -t 120 --mem=5000<br />
</pre><br />
Then you will get one core on a compute node within the partition "single". After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system Slurm has granted you the requested resources on the compute system. You will be logged in automatically on the granted core! To run a serial program on the granted core you only have to type the name of the executable.<br />
<pre><br />
$ ./<my_serial_program><br />
</pre><br />
Please be aware that your serial job must run less than 2 hours in this example, else the job will be killed during runtime by the system. <br />
<br><br />
You can also start now a graphical X11-terminal connecting you to the dedicated resource that is available for 2 hours. You can start it by the command:<br />
<pre><br />
$ xterm<br />
</pre><br />
Note that, once the walltime limit has been reached the resources - i.e. the compute node - will automatically be revoked.<br />
<br><br />
An interactive parallel application running on one compute node or on many compute nodes (e.g. here 5 nodes) with 40 cores each requires usually an amount of memory in GByte (e.g. 50 GByte) and a maximum time (e.g. 1 hour). E.g. 5 nodes can be allocated by the following command:<br />
<pre><br />
$ salloc -p multiple -N 5 --ntasks-per-node=40 -t 01:00:00 --mem=50gb<br />
</pre><br />
Now you can run parallel jobs on 200 cores requiring 50 GByte of memory per node. Please be aware that you will be logged in on core 0 of the first node.<br />
If you want to have access to another node you have to open a new terminal, connect it also to BwUniCluster 2.0 and type the following commands to<br />
connect to the running interactive job and then to a specific node:<br />
<pre><br />
$ srun --jobid=XXXXXXXX --pty /bin/bash<br />
$ srun --nodelist=uc2nXXX --pty /bin/bash<br />
</pre><br />
With the command:<br />
<pre><br />
$ squeue<br />
</pre><br />
the jobid and the nodelist can be shown.<br />
<br />
If you want to run MPI-programs, you can do it by simply typing mpirun <program_name>. Then your program will be run on 200 cores. A very simple example for starting a parallel job can be:<br />
<pre><br />
$ mpirun <my_mpi_program><br />
</pre><br />
You can also start the debugger ddt by the commands:<br />
<pre><br />
$ module add devel/ddt<br />
$ ddt <my_mpi_program><br />
</pre><br />
The above commands will execute the parallel program <my_mpi_program> on all available cores. You can also start parallel programs on a subset of cores; an example for this can be:<br />
<pre><br />
$ mpirun -n 50 <my_mpi_program><br />
</pre><br />
If you are using Intel MPI you must start <my_mpi_program> by the command mpiexec.hydra (instead of mpirun).<br />
<br><br />
<br><br />
<br />
<br />
----<br />
[[Category:bwUniCluster 2.0|Batch Jobs - bwUniCluster 2.0 Features]]</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Batch_Queues&diff=11483
BwUniCluster2.0/Batch Queues
2022-11-25T14:45:27Z
<p>P Weisbrod: </p>
<hr />
<div>=== sbatch -p ''queue'' ===<br />
Compute resources such as (wall-)time, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, '''you must add the correct queue class to your sbatch command '''. <font color=red>The specification of a queue is obligatory on BwUniCluster 2.0.</font> <br />
<br><br />
Details are:<br />
<br />
{| width=750px class="wikitable"<br />
! colspan="5" | bwUniCluster 2.0 <br> sbatch -p ''queue''<br />
|- style="text-align:left;"<br />
! queue !! node !! default resources !! minimum resources !! maximum resources<br />
|- style="text-align:left"<br />
| dev_single<br />
| thin<br />
| time=10, mem-per-cpu=1125mb<br />
| <br />
| time=30, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core=2)<br>6 nodes are reserved for this queue. <br> Only for development, i.e. debugging or performance optimization ...<br />
|- style="text-align:left;"<br />
| single<br />
| thin<br />
| time=30, mem-per-cpu=1125mb<br />
| <br />
| time=72:00:00, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core)=2<br />
|- style="text-align:left;"<br />
| dev_multiple<br />
| thin<br />
| time=10, mem-per-cpu=1125mb<br />
| nodes=2<br />
| time=30, nodes=4, mem=90000mb, ntasks-per-node=40, (threads-per-core=2)<br>8 nodes are reserved for this queue.<br> Only for development, i.e. debugging or performance optimization ...<br />
|- style="text-align:left;"<br />
| multiple<br />
| thin<br />
| time=30, mem-per-cpu=1125mb<br />
| nodes=2<br />
| time=72:00:00, mem=90000mb, nodes=128, ntasks-per-node=40, (threads-per-core=2) <br />
|- style="text-align:left;"<br />
| dev_multiple_il<br />
| thin (IceLake)<br />
| time=10, mem-per-cpu=1950mb<br />
| nodes=2<br />
| time=30, nodes=8, mem=249600mb, ntasks-per-node=64, (threads-per-core=2)<br>8 nodes are reserved for this queue <br> Only for development, i.e. debugging or performance optimization ...<br />
|- style="text-align:left;"<br />
| multiple_il<br />
| thin (IceLake)<br />
| time=10, mem-per-cpu=1950mb<br />
| nodes=2<br />
| time=72:00:00, nodes=128, mem=249600mb, ntasks-per-node=64, (threads-per-core=2)<br />
|- style="text-align:left;"<br />
| dev_gpu_4_a100<br />
| gpu4<br />
| time=10, mem-per-cpu=2178mb<br />
| <br />
| time=30, nodes=1, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)<br />
|- style="text-align:left;"<br />
| gpu_4_a100<br />
| gpu4<br />
| time=10, mem-per-cpu=2178mb<br />
| <br />
| time=72:00:00, nodes=1, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)<br />
|- style="vertical-align:top; text-align:left"<br />
| fat <br />
| fat<br />
| time=10, mem-per-cpu=18750mb<br />
| mem=180001mb<br />
| time=72:00:00, nodes=1, ntasks-per-node=80, (threads-per-core=2)<br />
|- style="vertical-align:top; text-align:left"<br />
| dev_gpu_4<br />
| gpu_4<br />
| time=10, mem-per-gpu=94000mb, cpu-per-gpu=20<br />
| <br />
| time=30, nodes=1, mem=376000, ntasks-per-node=40, (threads-per-core=2)<br>1 node is reserved for this queue <br> Only for development, i.e. debugging or performance optimization ...<br />
|- style="text-align:left;"<br />
| gpu_4<br />
| gpu4<br />
| time=10, mem-per-gpu=94000mb, cpu-per-gpu=20<br />
| <br />
| time=48:00:00, mem=376000, nodes=14, ntasks-per-node=40, (threads-per-core=2)<br />
|- style="vertical-align:top; text-align:left"<br />
| gpu_8<br />
| gpu8<br />
| time=10, mem-per-cpu=94000mb, cpu-per-gpu=10<br />
|<br />
| time=48:00:00, mem=752000, nodes=10, ntasks-per-node=40, (threads-per-core=2)<br />
|- <br />
|}<br />
<br />
Default resources of a queue class defines time, #tasks and memory if not explicitly given with sbatch command. Resource list acronyms ''--time'', ''--ntasks'', ''--nodes'', ''--mem'' and ''--mem-per-cpu'' are described [[BwUniCluster_2.0_Slurm_common_Features|here]].<br />
<br />
Access to the "special" and "dev_special" partitions on the bwUniCluster 2.0 is restricted to members of the institutions which participated in the procurement of the extension partition specifically for this purpose. Please contact the support team if your institution participated in the procurement and your account should be able to run jobs in this partition.<br />
<br><br />
<br><br />
==== Queue class examples ====<br />
To run your batch job on one of the thin nodes, please use:<br />
<br />
<pre><br />
$ sbatch --partition=dev_multiple<br />
or <br />
$ sbatch -p dev_multiple<br />
</pre><br />
<br><br />
<br />
==== Interactive Jobs ====<br />
On bwUniCluster 2.0 you are only allowed to run short jobs (<< 1 hour) with little memory requirements (<< 8 GByte) on the logins nodes. If you want to run longer jobs and/or jobs with a request of more than 8 GByte of memory, you must allocate resources for so-called interactive jobs by usage of the command salloc on a login node. Considering a serial application running on a compute node that requires 5000 MByte of memory and limiting the interactive run to 2 hours the following command has to be executed:<br />
<pre><br />
$ salloc -p single -n 1 -t 120 --mem=5000<br />
</pre><br />
Then you will get one core on a compute node within the partition "single". After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system Slurm has granted you the requested resources on the compute system. You will be logged in automatically on the granted core! To run a serial program on the granted core you only have to type the name of the executable.<br />
<pre><br />
$ ./<my_serial_program><br />
</pre><br />
Please be aware that your serial job must run less than 2 hours in this example, else the job will be killed during runtime by the system. <br />
<br><br />
You can also start now a graphical X11-terminal connecting you to the dedicated resource that is available for 2 hours. You can start it by the command:<br />
<pre><br />
$ xterm<br />
</pre><br />
Note that, once the walltime limit has been reached the resources - i.e. the compute node - will automatically be revoked.<br />
<br><br />
An interactive parallel application running on one compute node or on many compute nodes (e.g. here 5 nodes) with 40 cores each requires usually an amount of memory in GByte (e.g. 50 GByte) and a maximum time (e.g. 1 hour). E.g. 5 nodes can be allocated by the following command:<br />
<pre><br />
$ salloc -p multiple -N 5 --ntasks-per-node=40 -t 01:00:00 --mem=50gb<br />
</pre><br />
Now you can run parallel jobs on 200 cores requiring 50 GByte of memory per node. Please be aware that you will be logged in on core 0 of the first node.<br />
If you want to have access to another node you have to open a new terminal, connect it also to BwUniCluster 2.0 and type the following commands to<br />
connect to the running interactive job and then to a specific node:<br />
<pre><br />
$ srun --jobid=XXXXXXXX --pty /bin/bash<br />
$ srun --nodelist=uc2nXXX --pty /bin/bash<br />
</pre><br />
With the command:<br />
<pre><br />
$ squeue<br />
</pre><br />
the jobid and the nodelist can be shown.<br />
<br />
If you want to run MPI-programs, you can do it by simply typing mpirun <program_name>. Then your program will be run on 200 cores. A very simple example for starting a parallel job can be:<br />
<pre><br />
$ mpirun <my_mpi_program><br />
</pre><br />
You can also start the debugger ddt by the commands:<br />
<pre><br />
$ module add devel/ddt<br />
$ ddt <my_mpi_program><br />
</pre><br />
The above commands will execute the parallel program <my_mpi_program> on all available cores. You can also start parallel programs on a subset of cores; an example for this can be:<br />
<pre><br />
$ mpirun -n 50 <my_mpi_program><br />
</pre><br />
If you are using Intel MPI you must start <my_mpi_program> by the command mpiexec.hydra (instead of mpirun).<br />
<br><br />
<br><br />
<br />
<br />
----<br />
[[Category:bwUniCluster 2.0|Batch Jobs - bwUniCluster 2.0 Features]]</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Batch_Queues&diff=11482
BwUniCluster2.0/Batch Queues
2022-11-25T14:43:11Z
<p>P Weisbrod: </p>
<hr />
<div>=== sbatch -p ''queue'' ===<br />
Compute resources such as (wall-)time, nodes and memory are restricted and must fit into '''queues'''. Since requested compute resources are NOT always automatically mapped to the correct queue class, '''you must add the correct queue class to your sbatch command '''. <font color=red>The specification of a queue is obligatory on BwUniCluster 2.0.</font> <br />
<br><br />
Details are:<br />
<br />
{| width=750px class="wikitable"<br />
! colspan="5" | bwUniCluster 2.0 <br> sbatch -p ''queue''<br />
|- style="text-align:left;"<br />
! queue !! node !! default resources !! minimum resources !! maximum resources<br />
|- style="text-align:left"<br />
| dev_single<br />
| thin<br />
| time=10, mem-per-cpu=1125mb<br />
| <br />
| time=30, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core=2)<br>6 nodes are reserved for this queue. <br> Only for development, i.e. debugging or performance optimization ...<br />
|- style="text-align:left;"<br />
| single<br />
| thin<br />
| time=30, mem-per-cpu=1125mb<br />
| <br />
| time=72:00:00, nodes=1, mem=180000mb, ntasks-per-node=40, (threads-per-core)=2<br />
|- style="text-align:left;"<br />
| dev_multiple<br />
| thin<br />
| time=10, mem-per-cpu=1125mb<br />
| nodes=2<br />
| time=30, nodes=4, mem=90000mb, ntasks-per-node=40, (threads-per-core=2)<br>8 nodes are reserved for this queue.<br> Only for development, i.e. debugging or performance optimization ...<br />
|- style="text-align:left;"<br />
| multiple<br />
| thin<br />
| time=30, mem-per-cpu=1125mb<br />
| nodes=2<br />
| time=72:00:00, mem=90000mb, nodes=128, ntasks-per-node=40, (threads-per-core=2) <br />
|- style="text-align:left;"<br />
| dev_multiple_il<br />
| thin (IceLake)<br />
| time=10, mem-per-cpu=1950mb<br />
| nodes=2<br />
| time=30, nodes=8, mem=249600mb, ntasks-per-node=64, (threads-per-core=2)<br>8 nodes are reserved for this queue <br> Only for development, i.e. debugging or performance optimization ...<br />
|- style="text-align:left;"<br />
| multiple_il<br />
| thin (IceLake)<br />
| time=10, mem-per-cpu=1950mb<br />
| nodes=2<br />
| time=72:00:00, nodes=128, mem=249600mb, ntasks-per-node=64, (threads-per-core=2)<br />
|- style="text-align:left;"<br />
| dev_special<br/>(<i>restricted access!</i>)<br />
| thin (Broadwell)<br />
| time=10, mem-per-cpu=2178mb<br />
| <br />
| time=30, nodes=1, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)<br />
|- style="text-align:left;"<br />
| special<br/>(<i>restricted access!</i>)<br />
| thin (Broadwell)<br />
| time=10, mem-per-cpu=2178mb<br />
| <br />
| time=72:00:00, nodes=1, mem=122000mb, ntasks-per-node=28, (threads-per-core=2)<br />
|- style="vertical-align:top; text-align:left"<br />
| fat <br />
| fat<br />
| time=10, mem-per-cpu=18750mb<br />
| mem=180001mb<br />
| time=72:00:00, nodes=1, ntasks-per-node=80, (threads-per-core=2)<br />
|- style="vertical-align:top; text-align:left"<br />
| dev_gpu_4<br />
| gpu_4<br />
| time=10, mem-per-gpu=94000mb, cpu-per-gpu=20<br />
| <br />
| time=30, nodes=1, mem=376000, ntasks-per-node=40, (threads-per-core=2)<br>1 node is reserved for this queue <br> Only for development, i.e. debugging or performance optimization ...<br />
|- style="text-align:left;"<br />
| gpu_4<br />
| gpu4<br />
| time=10, mem-per-gpu=94000mb, cpu-per-gpu=20<br />
| <br />
| time=48:00:00, mem=376000, nodes=14, ntasks-per-node=40, (threads-per-core=2)<br />
|- style="vertical-align:top; text-align:left"<br />
| gpu_8<br />
| gpu8<br />
| time=10, mem-per-cpu=94000mb, cpu-per-gpu=10<br />
|<br />
| time=48:00:00, mem=752000, nodes=10, ntasks-per-node=40, (threads-per-core=2)<br />
|- <br />
|}<br />
<br />
Default resources of a queue class defines time, #tasks and memory if not explicitly given with sbatch command. Resource list acronyms ''--time'', ''--ntasks'', ''--nodes'', ''--mem'' and ''--mem-per-cpu'' are described [[BwUniCluster_2.0_Slurm_common_Features|here]].<br />
<br />
Access to the "special" and "dev_special" partitions on the bwUniCluster 2.0 is restricted to members of the institutions which participated in the procurement of the extension partition specifically for this purpose. Please contact the support team if your institution participated in the procurement and your account should be able to run jobs in this partition.<br />
<br><br />
<br><br />
==== Queue class examples ====<br />
To run your batch job on one of the thin nodes, please use:<br />
<br />
<pre><br />
$ sbatch --partition=dev_multiple<br />
or <br />
$ sbatch -p dev_multiple<br />
</pre><br />
<br><br />
<br />
==== Interactive Jobs ====<br />
On bwUniCluster 2.0 you are only allowed to run short jobs (<< 1 hour) with little memory requirements (<< 8 GByte) on the logins nodes. If you want to run longer jobs and/or jobs with a request of more than 8 GByte of memory, you must allocate resources for so-called interactive jobs by usage of the command salloc on a login node. Considering a serial application running on a compute node that requires 5000 MByte of memory and limiting the interactive run to 2 hours the following command has to be executed:<br />
<pre><br />
$ salloc -p single -n 1 -t 120 --mem=5000<br />
</pre><br />
Then you will get one core on a compute node within the partition "single". After execution of this command '''DO NOT CLOSE''' your current terminal session but wait until the queueing system Slurm has granted you the requested resources on the compute system. You will be logged in automatically on the granted core! To run a serial program on the granted core you only have to type the name of the executable.<br />
<pre><br />
$ ./<my_serial_program><br />
</pre><br />
Please be aware that your serial job must run less than 2 hours in this example, else the job will be killed during runtime by the system. <br />
<br><br />
You can also start now a graphical X11-terminal connecting you to the dedicated resource that is available for 2 hours. You can start it by the command:<br />
<pre><br />
$ xterm<br />
</pre><br />
Note that, once the walltime limit has been reached the resources - i.e. the compute node - will automatically be revoked.<br />
<br><br />
An interactive parallel application running on one compute node or on many compute nodes (e.g. here 5 nodes) with 40 cores each requires usually an amount of memory in GByte (e.g. 50 GByte) and a maximum time (e.g. 1 hour). E.g. 5 nodes can be allocated by the following command:<br />
<pre><br />
$ salloc -p multiple -N 5 --ntasks-per-node=40 -t 01:00:00 --mem=50gb<br />
</pre><br />
Now you can run parallel jobs on 200 cores requiring 50 GByte of memory per node. Please be aware that you will be logged in on core 0 of the first node.<br />
If you want to have access to another node you have to open a new terminal, connect it also to BwUniCluster 2.0 and type the following commands to<br />
connect to the running interactive job and then to a specific node:<br />
<pre><br />
$ srun --jobid=XXXXXXXX --pty /bin/bash<br />
$ srun --nodelist=uc2nXXX --pty /bin/bash<br />
</pre><br />
With the command:<br />
<pre><br />
$ squeue<br />
</pre><br />
the jobid and the nodelist can be shown.<br />
<br />
If you want to run MPI-programs, you can do it by simply typing mpirun <program_name>. Then your program will be run on 200 cores. A very simple example for starting a parallel job can be:<br />
<pre><br />
$ mpirun <my_mpi_program><br />
</pre><br />
You can also start the debugger ddt by the commands:<br />
<pre><br />
$ module add devel/ddt<br />
$ ddt <my_mpi_program><br />
</pre><br />
The above commands will execute the parallel program <my_mpi_program> on all available cores. You can also start parallel programs on a subset of cores; an example for this can be:<br />
<pre><br />
$ mpirun -n 50 <my_mpi_program><br />
</pre><br />
If you are using Intel MPI you must start <my_mpi_program> by the command mpiexec.hydra (instead of mpirun).<br />
<br><br />
<br><br />
<br />
<br />
----<br />
[[Category:bwUniCluster 2.0|Batch Jobs - bwUniCluster 2.0 Features]]</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Hardware_and_Architecture&diff=11479
BwUniCluster2.0/Hardware and Architecture
2022-11-25T14:04:21Z
<p>P Weisbrod: </p>
<hr />
<div>= Architecture of bwUniCluster 2.0 =<br />
<br />
The bwUniCluster 2.0 is a parallel computer with distributed memory. Each node of system consists of at least two Intel Xeon processor, local memory, disks, network adapters and optionally accelerators (NVIDIA Tesla V100, A100 or H100). All nodes are connected by a fast InfiniBand interconnect. In addition the file system<br />
Lustre, that is connected by coupling the InfiniBand of the file server with the InfiniBand<br />
switch of the compute cluster, is added to bwUniCluster 2.0 to provide a fast and scalable<br />
parallel file system.<br />
<br />
The operating system on each node is Red Hat Enterprise Linux (RHEL) 8.4. A number of additional software packages like e.g. SLURM have been installed on top. Some of these components are of special interest to end users and are briefly<br />
discussed in this document. Others which are of greater importance to system<br />
administrators will not be covered by this document.<br />
<br />
The individual nodes of the system may act in different roles. According to the services supplied by the nodes, they are separated into disjoint groups. From an end users point of view the different groups of nodes are login nodes, compute nodes, file server nodes and administrative server nodes.<br />
<br />
'''Login Nodes'''<br />
<br />
The login nodes are the only nodes that are directly accessible by end users. These nodes<br />
are used for interactive login, file management, program development and interactive pre-<br />
and postprocessing. Two nodes are dedicated to this service but they are all accessible via<br />
one address and a DNS round-robin alias distributes the login sessions to the<br />
different login nodes.<br />
<br />
'''Compute Node'''<br />
<br />
The majority of nodes are compute nodes which are managed by a batch system. Users <br />
submit their jobs to the SLURM batch system and a job is executed when the required resources become available (depending on its fair-share priority).<br />
<br />
'''File Server Nodes'''<br />
<br />
The hardware of the parallel file system Lustre incorporates some file server nodes; the file<br />
system Lustre is connected by coupling the InfiniBand of the file server with the independent InfiniBand switch of the compute cluster. In addition to shared file space there is also local storage on the disks of each node (for details see chapter "File Systems").<br />
<br />
'''Administrative Server Nodes'''<br />
<br />
Some other nodes are delivering additional services like resource management, external<br />
network connection, administration etc. These nodes can be accessed directly by system administrators only.<br />
<br />
= Components of bwUniCluster =<br />
<br />
{| class="wikitable"<br />
|- <br />
! style="width:9%"|<br />
! style="width:13%"| Compute nodes "Thin"<br />
! style="width:13%"| Compute nodes "HPC"<br />
! style="width:13%"| Compute nodes "IceLake"<br />
! style="width:13%"| Compute nodes "Fat"<br />
! style="width:13%"| GPU x4<br />
! style="width:13%"| GPU x8<br />
! style="width:13%"| IceLake + GPU x4<br />
! style="width:13%"| Login<br />
|-<br />
!scope="column"| Number of nodes<br />
| 100 + 60<br />
| 360<br />
| 272<br />
| 6<br />
| 14<br />
| 10<br />
| 15<br />
| 4<br />
|-<br />
!scope="column"| Processors<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Platinum 8358<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Platinum 8358<br />
| Intel Xeon Gold 6248<br />
|-<br />
!scope="column"| Number of sockets<br />
| 2<br />
| 2<br />
| 2<br />
| 4<br />
| 2<br />
| 2<br />
| 2<br />
| 2<br />
|-<br />
!scope="column"| Processor frequency (GHz)<br />
| 2.1 Ghz<br />
| 2.1 Ghz<br />
| 2.6 Ghz<br />
| 2.1 Ghz<br />
| 2.1 Ghz<br />
| 2.6 Ghz<br />
| 2.5 Ghz<br />
| <br />
|-<br />
!scope="column"| Total number of cores<br />
| 40<br />
| 40<br />
| 64<br />
| 80<br />
| 40<br />
| 40<br />
| 64<br />
| 40<br />
|-<br />
!scope="column"| Main memory<br />
| 96 GB / 192 GB<br />
| 96 GB<br />
| 256 GB<br />
| 3 TB<br />
| 384 GB<br />
| 768 GB<br />
| 512 GB<br />
| 384 GB<br />
|-<br />
!scope="column"| Local SSD<br />
| 960 GB SATA<br />
| 960 GB SATA<br />
| 1,8 TB NVMe<br />
| 4,8 TB NVMe<br />
| 3,2 TB NVMe<br />
| 15 TB NVMe<br />
| 6,4 TB NVMe<br />
| <br />
|-<br />
!scope="column"| Accelerators<br />
| -<br />
| -<br />
| -<br />
| -<br />
| 4x NVIDIA Tesla V100<br />
| 8x NVIDIA Tesla V100<br />
| 4x NVIDIA H100 / 4x NVIDIA H100<br />
|-<br />
!scope="column"| Interconnect<br />
| IB HDR100 (blocking)<br />
| IB HDR100<br />
| IB HDR200<br />
| IB HDR<br />
| IB HDR<br />
| IB HDR<br />
| IB HDR200<br />
| IB HDR100 (blocking)<br />
|}<br />
Table 1: Properties of the nodes<br />
<br />
= File Systems =<br />
<br />
On bwUniCluster 2.0 the parallel file system Lustre is used for most globally visible user data. Lustre is open source and Lustre solutions and support are available from different vendors. Nowadays, most of the biggest HPC systems are using Lustre. An initial home directory on a Lustre file system is created automatically after account activation, and the environment variable $HOME holds its name. Users can create so-called workspaces on another Lustre file system for non-permanent data with temporary lifetime. There is another workspace file system based on flash storage for special requirements available.<br />
<br />
Within a batch job further file systems are available: <br />
* The directory $TMP is only available and visible on the local node. It is located on fast SSD storage devices.<br />
* On request a parallel on-demand file system (BeeOND) is created which uses the SSDs of the nodes which were allocated to the batch job.<br />
* On request the external LSDF Online Storage is mounted on the nodes which were allocated to the batch job. This file system is based on the parallel file system Spectrum Scale. <br />
<br />
Some of the characteristics of the file systems are shown in Table 2.<br />
<br />
{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"<br />
|- style="width:20%;height=20px; text-align:left;padding:3px"<br />
! style="background-color:#AAA;padding:3px"| Property<br />
! style="background-color:#AAA;padding:3px"| $TMP<br />
! style="background-color:#AAA;padding:3px"| BeeOND<br />
! style="background-color:#AAA;padding:3px"| $HOME<br />
! style="background-color:#AAA;padding:3px"| Workspace<br />
! style="background-color:#AAA;padding:3px"| Workspace <br> on flash<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Visibility <br />
| style="height=20px; text-align:left;padding:3px"| local node<br />
| style="height=20px; text-align:left;padding:3px"| nodes of batch job<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Lifetime <br />
| style="height=20px; text-align:left;padding:3px"| batch job runtime<br />
| style="height=20px; text-align:left;padding:3px"| batch job runtime<br />
| style="height=20px; text-align:left;padding:3px"| permanent<br />
| style="height=20px; text-align:left;padding:3px"| max. 240 days<br />
| style="height=20px; text-align:left;padding:3px"| max. 240 days<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Disk space <br />
| style="height=20px; text-align:left;padding:3px"| 960 GB - 6.4 TB <br> details see table 1<br />
| style="height=20px; text-align:left;padding:3px"| n*750 GB<br />
| style="height=20px; text-align:left;padding:3px"| 1.2 PiB<br />
| style="height=20px; text-align:left;padding:3px"| 4.1 PiB<br />
| style="height=20px; text-align:left;padding:3px"| 236 TiB<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Capacity Quotas<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 1 TiB per user <br> also per organization<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 40 TiB per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 5 TiB per user<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Inode Quotas<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 10 million per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 30 million per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 25 million per user<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Backup<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Read perf./node<br />
| style="height=20px; text-align:left;padding:3px"| 500 MB/s - 6 GB/s <br> depends on type of local SSD / job queue: <br> 520 MB/s @ single / multiple <br> 800 MB/s @ multiple_e <br> 6600 MB/s @ fat <br> 6500 MB/s @ gpu_4 <br> 6500 MB/s @ gpu_8<br />
| style="height=20px; text-align:left;padding:3px"| 400 MB/s - 500 MB/s<br> depends on type of local SSDs / job queue: <br> 500 MB/s @ multiple <br> 400 MB/s @ multiple_e<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Write perf./node<br />
| style="height=20px; text-align:left;padding:3px"| 500 MB/s - 4 GB/s <br> depends on type of local SSD / job queue: <br> 500 MB/s @ single / multiple <br> 650 MB/s @ multiple_e <br> 2900 MB/s @ fat <br> 2090 MB/s @ gpu_4 <br> 4060 MB/s @ gpu_8<br />
| style="height=20px; text-align:left;padding:3px"| 250 MB/s - 350 MB/s <br> depends on type of local SSDs / job queue: <br> 350 MB/s @ multiple <br> 250 MB/s @ multiple_e<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Total read perf.<br />
| style="height=20px; text-align:left;padding:3px"| n*500-6000 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| n*400-500 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| 18 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 54 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 45 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Total write perf.<br />
| style="height=20px; text-align:left;padding:3px"| n*500-4000 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| n*250-350 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| 18 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 54 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 38 GB/s<br />
|}<br />
---------------------------------------------------------------------------------------------------------<br />
global: all nodes of UniCluster access the same file system;<br />
local: each node has its own file system;<br />
permanent: files are stored permanently;<br />
batch job: files are removed at end of the batch job.<br />
---------------------------------------------------------------------------------------------------------<br />
Table 2: Properties of the file systems<br />
<br />
== Selecting the appropriate file system ==<br />
<br />
In general, you should separate your data and store it on the appropriate file system.<br />
Permanently needed data like software or important results should be stored below $HOME<br />
but capacity restrictions (quotas) apply. In case you accidentally deleted data on $HOME<br />
you can usually restore it from backup. Permanent data which is not needed for months<br />
or exceeds the capacity restrictions should be sent to the LSDF Online Storage <br />
or to the archive and deleted from the file systems. Temporary data which is only needed on a single<br />
node and which does not exceed the disk space shown in the table above should be stored<br />
below $TMP. Data which is read many times on a single node, e.g. if you are doing AI training, <br />
should be copied to $TMP and read from there. Temporary data which is used from many nodes <br />
of your batch job and which is only needed during job runtime should be stored on a <br />
parallel on-demand file system. Temporary data which can be recomputed or which is the <br />
result of one job and input for another job should be stored in workspaces. The lifetime <br />
of data in workspaces is limited and depends on the lifetime of the workspace which can be <br />
several months.<br />
<br />
For further details please check the chapters below.<br />
<br />
== $HOME ==<br />
<br />
The home directories of bwUniCluster 2.0 (uc2) users are located in the parallel file system Lustre.<br />
You have access to your home directory from all nodes of uc2. A regular backup of these directories <br />
to tape archive is done automatically. The directory $HOME is used to hold those files that are<br />
permanently used like source codes, configuration files, executable programs etc. <br />
<br />
On uc2 there is a default user quota limit of 1 TiB and 10 million inodes (files and directories) per user. <br />
You can chek your current usage and limits with the command<br />
<pre><br />
$ lfs quota -uh $(whoami) $HOME<br />
</pre><br />
In addition to the user limit there is a limit of your organization (e.g. university) which depends on the financial share. This limit is enforced with so-called Lustre project quotas. You can show the current usage and limits of your organization with the following command:<br />
<pre><br />
lfs quota -ph $(grep $(echo $HOME | sed -e "s|/[^/]*/[^/]*$||") /pfs/data5/project_ids.txt | cut -f 1 -d\ ) $HOME<br />
</pre><br />
<br />
== Workspaces ==<br />
<br />
On uc2 workspaces can be used to store large non-permanent data sets, e.g. restart files or output<br />
data that has to be post-processed. The file system used for workspaces is also the parallel file system Lustre. This file system is especially designed for parallel access and for a high throughput to large<br />
files. It is able to provide high data transfer rates of up to 54 GB/s write and read performance when data access is parallel. <br />
<br />
Workspaces have a lifetime and the data on a workspace expires as a whole after a fixed period. The maximum lifetime of a workspace is 60 days, but it can be renewed at the end of that period 3 times to a total maximum of 240 days after workspace generation. If a workspace has inadvertently expired we can restore the data during a limited time (few weeks). In this case you should create a new workspace and report the name of the new and of the expired workspace in a ticket.<br />
<br />
Creating, deleting, finding and extending workspaces is explained on the [[workspace]] page.<br />
<br />
On uc2 there is a default user quota limit of 40 TiB and 30 million inodes (files and directories) per user. <br />
You can chek your current usage and limits with the command<br />
<pre><br />
$ lfs quota -uh $(whoami) /pfs/work7<br />
</pre><br />
Note that the quotas include data and inodes for all of your workspaces and all of your expired workspaces (as long as they are not yet completely removed).<br />
<br />
=== Reminder for workspace deletion ===<br />
<br />
Normally you will get an email every day starting 7 days before a workspace expires. You can send yourself a calender entry which reminds you when a workspace will be automatically deleted:<br />
<br />
$ ws_send_ical.sh <workspace> <email><br />
<br />
== Improving Performance on $HOME and workspaces ==<br />
<br />
The following recommendations might help to improve throughput and metadata<br />
performance on Lustre filesystems.<br />
<br />
=== '''Improving Throughput Performance''' ===<br />
<br />
Depending on your application some adaptations might be necessary if you want to reach<br />
the full bandwidth of the filesystems. Parallel filesystems typically stripe files over storage subsystems, i.e. large files are separated into stripes and distributed to different storage subsystems. In Lustre, the size of these stripes (sometimes also mentioned as chunks) is called stripe size and the number of used storage subsystems is called stripe count.<br />
<br />
When you are designing your application you should consider that the performance of<br />
parallel filesystems is generally better if data is transferred in large blocks and stored in<br />
few large files. In more detail, to increase throughput performance of a parallel application<br />
following aspects should be considered:<br />
<br />
* collect large chunks of data and write them sequentially at once,<br />
<br />
* to exploit complete filesystem bandwidth use several clients,<br />
<br />
* avoid competitive file access by different tasks or use blocks with boundaries at stripe size (default is 1MB),<br />
<br />
* if files are small enough for the SSDs and are only used by one process store them on $TMP.<br />
<br />
With previous Lustre versions adapting the Lustre stripe count was the most important optimization. However, for the workspaces of uc2 the new Lustre feature Progressive File Layouts has been used to define file striping parameters. This means that the stripe count is adapted if the file size is growing. In normal cases users no longer need to adapt file striping parameters in case they have very huge files or in order to reach better performance. <br />
<br />
If you know what you are doing you can still change striping parameters, e.g. the stripe count, of a directory and of newly created files. New files and directories inherit the stripe count from the parent directory. E.g. if you want to enhance throughput on a single very large file which is created in the directory $HOME/my_output_dir you can use the command<br />
<pre><br />
$ lfs setstripe -c-1 $HOME/my_output_dir<br />
</pre><br />
to change the stripe count to -1 which means that all storage subsystems of the file system are used to store that file. If you change the stripe count of a directory the stripe count of existing files inside this<br />
directory is not changed. If you want to change the stripe count of existing files, change<br />
the stripe count of the parent directory, copy the files to new files, remove the old files<br />
and move the new files back to the old name. In order to check the stripe setting of<br />
the file my_file use<br />
<pre><br />
$ lfs getstripe my_file<br />
</pre><br />
Also note that changes on the striping parameters (e.g. stripe count) are not saved in the<br />
backup, i.e. if directories have to be recreated this information is lost and the default stripe<br />
count will be used. Therefore, you should annotate for which directories you made changes<br />
to the striping parameters so that you can repeat these changes if required.<br />
<br />
=== '''Improving Metadata Performance''' ===<br />
<br />
Metadata performance on parallel file systems is usually not as good as with local<br />
filesystems. In addition, it is usually not scalable, i.e. a limited resource. Therefore,<br />
you should omit metadata operations whenever possible. For example, it is much better<br />
to have few large files than lots of small files. In more detail, to increase metadata<br />
performance of a parallel application following aspects should be considered:<br />
<br />
* avoid creating many small files,<br />
<br />
* avoid competitive directory access, e.g. by creating files in separate subdirectories for each task,<br />
<br />
* if many small files are only used within a batch job and accessed by one process store them on $TMP,<br />
<br />
* change the default colorization setting of the command ls (see below).<br />
<br />
On modern Linux systems, the GNU ls command often uses colorization by default to<br />
visually highlight the file type; this is especially true if the command is run within a terminal<br />
session. This is because the default shell profile initializations usually contain an alias<br />
directive similar to the following for the ls command:<br />
<pre><br />
$ alias ls="ls --color=tty"<br />
</pre><br />
However, running the ls command in this way for files on a Lustre file system requires<br />
a stat() call to be used to determine the file type. This can result in a performance<br />
overhead, because the stat() call always needs to determine the size of a file, and that<br />
in turn means that the client node must query the object size of all the backing objects<br />
that make up a file. As a result of the default colorization setting, running a simple<br />
ls command on a Lustre file system often takes as much time as running the ls command<br />
with the -l option (the same is true if the -F, -p, or the -classify option, or any other option <br />
that requires information from a stat() call, is used). To avoid this performance overhead<br />
when using ls commands, add an alias directive similar to the following<br />
to your shell startup script:<br />
<pre><br />
$ alias ls="ls --color=never"<br />
</pre><br />
<br />
== Workspaces on flash storage ==<br />
<br />
There is another workspace file system for special reqirements available. The file system is called ''full flash pfs'' and is based on the parallel file system Lustre. Access will be granted on special request.<br />
<br />
=== Advantages of this file system ===<br />
<br />
# All storage devices are based on flash (no hard disks) with low access times. Hence performance is better compared to other parallel file systems for read and write access with small blocks and with small files, i.e. IOPS rates are improved.<br />
# The file system is mounted on bwUniCluster 2.0 and HoreKa, i.e. it can be used to share data between these clusters.<br />
<br />
=== Access restrictions ===<br />
<br />
* Only HoreKa users or KIT users of bwUniCluster 2.0 can use this file system.<br />
* Access is granted on request and for appropriate requirements. In order to request access, please open a ticket with the subject ''full flash pfs access required'' and describe the following topics:<br />
** number of files you want to store on this file system<br />
** needed capacity<br />
** number of nodes used by your typical jobs<br />
** special I/O requirements of your jobs<br />
<br />
=== Using the file system ===<br />
<br />
After access is granted, you can use the file system in the same way as a normal workspace. You just have to specify the name of the flash-based workspace file system using the option ''-F'' to all the commands that manage workspaces. On bwUniCluster 2.0 it is called ''ffuc'', on HoreKa it is ''ffhk''. For example, to create a workspace with name myws and a lifetime of 60 days on bwUniCluster 2.0 execute:<br />
ws_allocate -F ffuc myws 60<br />
<br />
If you want to use the full flash pfs on bwUniCluster 2.0 '''and''' HoreKa at the same time, please note that you only have to manage a particular workspace on one of the clusters since the name of the workspace directory is different. However, the path to each workspace is visible and can be used on both clusters.<br />
<br />
Other features are similar to normal workspaces. For example, we are able to restore expired workspaces for few weeks and you have to open a ticket to request the restore. There are quota limits with a default limit of 5 TB capacity and 25 millions inodes per user. You can check your current usage with<br />
lfs quota -uh $(whoami) /pfs/work8<br />
<br />
== $TMP ==<br />
<br />
While all tasks of a parallel application access the same $HOME and workspace directory, the <br />
$TMP directory is local to each node on bwUniCluster 2.0. All nodes have fast SSDs <br />
local storage devices which are used to store data below $TMP. Different tasks of a parallel<br />
application use different $TMP directories when they do not utilize one node. This directory should<br />
be used for temporary files being accessed by single tasks. It should also be used if you read the <br />
same data many times from a single node, e.g. if you are doing AI training. In this case you should <br />
copy the data at the beginning of your batch job to $TMP and read the data from there.<br />
In addition, this directory should be used for the installation<br />
of software packages. This means that the software package to be installed should be<br />
unpacked, compiled and linked in a subdirectory of $TMP. The real installation of the<br />
package (e.g. make install) should be made in(to) the Lustre filesystem. <br />
<br />
Each time a batch job is started, a subdirectory is created on each node and assigned to the job. <br />
$TMP is newly set and the name of the subdirectory contains the Job-id so that the<br />
subdirectory name is unique for each job. This unique name is then assigned to the<br />
environment variable $TMP within the job. At the end of the job the subdirectory is removed.<br />
Although $TMP points to the same path name for different nodes of a job, the physical location <br />
on these nodes is different.<br />
<br />
== LSDF Online Storage==<br />
<br />
In some cases it is useful to have access to the LSDF Online Storage on the HPC-Clusters also. Therefore the LSDF Online Storage is mounted on the Login- and Datamover-Nodes.<br />
Furthermore it can be used on the compute nodes during the job runtime with the constraint flag "LSDF" ([[bwUniCluster_2.0_Slurm_common_Features|Slurm common features]]<br />
). There is also an example about the LSDF batch usage: [[bwUniCluster_2.0_Slurm_common_Features#LSDF_Online_Storage|Slurm LSDF example ]] .<br />
<pre><br />
#!/bin/bash<br />
#SBATCH ...<br />
#SBATCH --constraint=LSDF<br />
</pre><br />
<br><br />
For the usage of the LSDF Online Storage the following environment variables are available: $LSDF, $LSDFPROJECTS, $LSDFHOME.<br />
Please request storage projects in the LSDF Online Storage seperately:<br />
[https://www.lsdf.kit.edu/os/storagerequest/: LSDF Storage Request].<br />
<br />
==BeeOND (BeeGFS On-Demand)==<br />
<br />
Users of the UniCluster have possibility to request a private BeeOND (on-demand BeeGFS) parallel filesystem for each job. The file system is created during job startup and purged after your job.<br />
<br />
'''IMPORTANT: All data on the private filesystem will be deleted after your job. Make sure you have copied your data back to the global filesystem (within job), e.g., $HOME or any workspace.'''<br />
<br />
BeeOND/BeeGFS can be used like any other parallel file system. Tools like cp or rsync can be used to copy data in and out. <br />
<br />
For detailed usage see here:[[BwUniCluster_2.0_Slurm_common_Features#BeeOND_.28BeeGFS_On-Demand.29|Request on-demand file system]]<br />
<br />
==Backup and Archiving==<br />
<br />
There are regular backups of all data of the home directories,whereas ACLs and extended attributes will<br />
not be backuped. <br />
<br />
Please open a ticket if you need backuped data.<br />
<br />
<br />
[[Category:bwUniCluster 2.0|File System]][[Category:Hardware_and_Architecture|bwUniCluster 2.0]]</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Hardware_and_Architecture&diff=11478
BwUniCluster2.0/Hardware and Architecture
2022-11-25T13:49:30Z
<p>P Weisbrod: /* Components of bwUniCluster */</p>
<hr />
<div>= Architecture of bwUniCluster 2.0 =<br />
<br />
The bwUniCluster 2.0 is a parallel computer with distributed memory. Each node of system consists of at least two Intel Xeon processor, local memory, disks, network adapters and optionally accelerators (NVIDIA Tesla V100). All nodes are connected by a fast InfiniBand interconnect. In addition the file system<br />
Lustre, that is connected by coupling the InfiniBand of the file server with the InfiniBand<br />
switch of the compute cluster, is added to bwUniCluster 2.0 to provide a fast and scalable<br />
parallel file system.<br />
<br />
The operating system on each node is Red Hat Enterprise Linux (RHEL) 8.4. A number of additional software packages like e.g. SLURM have been installed on top. Some of these components are of special interest to end users and are briefly<br />
discussed in this document. Others which are of greater importance to system<br />
administrators will not be covered by this document.<br />
<br />
The individual nodes of the system may act in different roles. According to the services supplied by the nodes, they are separated into disjoint groups. From an end users point of view the different groups of nodes are login nodes, compute nodes, file server nodes and administrative server nodes.<br />
<br />
'''Login Nodes'''<br />
<br />
The login nodes are the only nodes that are directly accessible by end users. These nodes<br />
are used for interactive login, file management, program development and interactive pre-<br />
and postprocessing. Two nodes are dedicated to this service but they are all accessible via<br />
one address and a DNS round-robin alias distributes the login sessions to the<br />
different login nodes.<br />
<br />
'''Compute Node'''<br />
<br />
The majority of nodes are compute nodes which are managed by a batch system. Users <br />
submit their jobs to the SLURM batch system and a job is executed when the required resources become available (depending on its fair-share priority).<br />
<br />
'''File Server Nodes'''<br />
<br />
The hardware of the parallel file system Lustre incorporates some file server nodes; the file<br />
system Lustre is connected by coupling the InfiniBand of the file server with the independent InfiniBand switch of the compute cluster. In addition to shared file space there is also local storage on the disks of each node (for details see chapter "File Systems").<br />
<br />
'''Administrative Server Nodes'''<br />
<br />
Some other nodes are delivering additional services like resource management, external<br />
network connection, administration etc. These nodes can be accessed directly by system administrators only.<br />
<br />
= Components of bwUniCluster =<br />
<br />
{| class="wikitable"<br />
|- <br />
! style="width:9%"|<br />
! style="width:13%"| Compute nodes "Thin"<br />
! style="width:13%"| Compute nodes "HPC"<br />
! style="width:13%"| Compute nodes "IceLake"<br />
! style="width:13%"| Compute nodes "Fat"<br />
! style="width:13%"| GPU x4<br />
! style="width:13%"| GPU x8<br />
! style="width:13%"| IceLake + GPU x4<br />
! style="width:13%"| Login<br />
|-<br />
!scope="column"| Number of nodes<br />
| 100 + 60<br />
| 360<br />
| 272<br />
| 6<br />
| 14<br />
| 10<br />
| 15<br />
| 4<br />
|-<br />
!scope="column"| Processors<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Platinum 8358<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Platinum 8358<br />
| Intel Xeon Gold 6248<br />
|-<br />
!scope="column"| Number of sockets<br />
| 2<br />
| 2<br />
| 2<br />
| 4<br />
| 2<br />
| 2<br />
| 2<br />
| 2<br />
|-<br />
!scope="column"| Processor frequency (GHz)<br />
| 2.1 Ghz<br />
| 2.1 Ghz<br />
| 2.6 Ghz<br />
| 2.1 Ghz<br />
| 2.1 Ghz<br />
| 2.6 Ghz<br />
| 2.5 Ghz<br />
| <br />
|-<br />
!scope="column"| Total number of cores<br />
| 40<br />
| 40<br />
| 64<br />
| 80<br />
| 40<br />
| 40<br />
| 64<br />
| 40<br />
|-<br />
!scope="column"| Main memory<br />
| 96 GB / 192 GB<br />
| 96 GB<br />
| 256 GB<br />
| 3 TB<br />
| 384 GB<br />
| 768 GB<br />
| 512 GB<br />
| 384 GB<br />
|-<br />
!scope="column"| Local SSD<br />
| 960 GB SATA<br />
| 960 GB SATA<br />
| 1,8 TB NVMe<br />
| 4,8 TB NVMe<br />
| 3,2 TB NVMe<br />
| 15 TB NVMe<br />
| 6,4 TB NVMe<br />
| <br />
|-<br />
!scope="column"| Accelerators<br />
| -<br />
| -<br />
| -<br />
| -<br />
| 4x NVIDIA Tesla V100<br />
| 8x NVIDIA Tesla V100<br />
| 4x NVIDIA H100 / 4x NVIDIA H100<br />
|-<br />
!scope="column"| Interconnect<br />
| IB HDR100 (blocking)<br />
| IB HDR100<br />
| IB HDR200<br />
| IB HDR<br />
| IB HDR<br />
| IB HDR<br />
| IB HDR200<br />
| IB HDR100 (blocking)<br />
|}<br />
Table 1: Properties of the nodes<br />
<br />
= File Systems =<br />
<br />
On bwUniCluster 2.0 the parallel file system Lustre is used for most globally visible user data. Lustre is open source and Lustre solutions and support are available from different vendors. Nowadays, most of the biggest HPC systems are using Lustre. An initial home directory on a Lustre file system is created automatically after account activation, and the environment variable $HOME holds its name. Users can create so-called workspaces on another Lustre file system for non-permanent data with temporary lifetime. There is another workspace file system based on flash storage for special requirements available.<br />
<br />
Within a batch job further file systems are available: <br />
* The directory $TMP is only available and visible on the local node. It is located on fast SSD storage devices.<br />
* On request a parallel on-demand file system (BeeOND) is created which uses the SSDs of the nodes which were allocated to the batch job.<br />
* On request the external LSDF Online Storage is mounted on the nodes which were allocated to the batch job. This file system is based on the parallel file system Spectrum Scale. <br />
<br />
Some of the characteristics of the file systems are shown in Table 2.<br />
<br />
{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"<br />
|- style="width:20%;height=20px; text-align:left;padding:3px"<br />
! style="background-color:#AAA;padding:3px"| Property<br />
! style="background-color:#AAA;padding:3px"| $TMP<br />
! style="background-color:#AAA;padding:3px"| BeeOND<br />
! style="background-color:#AAA;padding:3px"| $HOME<br />
! style="background-color:#AAA;padding:3px"| Workspace<br />
! style="background-color:#AAA;padding:3px"| Workspace <br> on flash<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Visibility <br />
| style="height=20px; text-align:left;padding:3px"| local node<br />
| style="height=20px; text-align:left;padding:3px"| nodes of batch job<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Lifetime <br />
| style="height=20px; text-align:left;padding:3px"| batch job runtime<br />
| style="height=20px; text-align:left;padding:3px"| batch job runtime<br />
| style="height=20px; text-align:left;padding:3px"| permanent<br />
| style="height=20px; text-align:left;padding:3px"| max. 240 days<br />
| style="height=20px; text-align:left;padding:3px"| max. 240 days<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Disk space <br />
| style="height=20px; text-align:left;padding:3px"| 960 GB - 6.4 TB <br> details see table 1<br />
| style="height=20px; text-align:left;padding:3px"| n*750 GB<br />
| style="height=20px; text-align:left;padding:3px"| 1.2 PiB<br />
| style="height=20px; text-align:left;padding:3px"| 4.1 PiB<br />
| style="height=20px; text-align:left;padding:3px"| 236 TiB<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Capacity Quotas<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 1 TiB per user <br> also per organization<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 40 TiB per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 5 TiB per user<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Inode Quotas<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 10 million per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 30 million per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 25 million per user<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Backup<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Read perf./node<br />
| style="height=20px; text-align:left;padding:3px"| 500 MB/s - 6 GB/s <br> depends on type of local SSD / job queue: <br> 520 MB/s @ single / multiple <br> 800 MB/s @ multiple_e <br> 6600 MB/s @ fat <br> 6500 MB/s @ gpu_4 <br> 6500 MB/s @ gpu_8<br />
| style="height=20px; text-align:left;padding:3px"| 400 MB/s - 500 MB/s<br> depends on type of local SSDs / job queue: <br> 500 MB/s @ multiple <br> 400 MB/s @ multiple_e<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Write perf./node<br />
| style="height=20px; text-align:left;padding:3px"| 500 MB/s - 4 GB/s <br> depends on type of local SSD / job queue: <br> 500 MB/s @ single / multiple <br> 650 MB/s @ multiple_e <br> 2900 MB/s @ fat <br> 2090 MB/s @ gpu_4 <br> 4060 MB/s @ gpu_8<br />
| style="height=20px; text-align:left;padding:3px"| 250 MB/s - 350 MB/s <br> depends on type of local SSDs / job queue: <br> 350 MB/s @ multiple <br> 250 MB/s @ multiple_e<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Total read perf.<br />
| style="height=20px; text-align:left;padding:3px"| n*500-6000 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| n*400-500 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| 18 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 54 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 45 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Total write perf.<br />
| style="height=20px; text-align:left;padding:3px"| n*500-4000 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| n*250-350 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| 18 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 54 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 38 GB/s<br />
|}<br />
---------------------------------------------------------------------------------------------------------<br />
global: all nodes of UniCluster access the same file system;<br />
local: each node has its own file system;<br />
permanent: files are stored permanently;<br />
batch job: files are removed at end of the batch job.<br />
---------------------------------------------------------------------------------------------------------<br />
Table 2: Properties of the file systems<br />
<br />
== Selecting the appropriate file system ==<br />
<br />
In general, you should separate your data and store it on the appropriate file system.<br />
Permanently needed data like software or important results should be stored below $HOME<br />
but capacity restrictions (quotas) apply. In case you accidentally deleted data on $HOME<br />
you can usually restore it from backup. Permanent data which is not needed for months<br />
or exceeds the capacity restrictions should be sent to the LSDF Online Storage <br />
or to the archive and deleted from the file systems. Temporary data which is only needed on a single<br />
node and which does not exceed the disk space shown in the table above should be stored<br />
below $TMP. Data which is read many times on a single node, e.g. if you are doing AI training, <br />
should be copied to $TMP and read from there. Temporary data which is used from many nodes <br />
of your batch job and which is only needed during job runtime should be stored on a <br />
parallel on-demand file system. Temporary data which can be recomputed or which is the <br />
result of one job and input for another job should be stored in workspaces. The lifetime <br />
of data in workspaces is limited and depends on the lifetime of the workspace which can be <br />
several months.<br />
<br />
For further details please check the chapters below.<br />
<br />
== $HOME ==<br />
<br />
The home directories of bwUniCluster 2.0 (uc2) users are located in the parallel file system Lustre.<br />
You have access to your home directory from all nodes of uc2. A regular backup of these directories <br />
to tape archive is done automatically. The directory $HOME is used to hold those files that are<br />
permanently used like source codes, configuration files, executable programs etc. <br />
<br />
On uc2 there is a default user quota limit of 1 TiB and 10 million inodes (files and directories) per user. <br />
You can chek your current usage and limits with the command<br />
<pre><br />
$ lfs quota -uh $(whoami) $HOME<br />
</pre><br />
In addition to the user limit there is a limit of your organization (e.g. university) which depends on the financial share. This limit is enforced with so-called Lustre project quotas. You can show the current usage and limits of your organization with the following command:<br />
<pre><br />
lfs quota -ph $(grep $(echo $HOME | sed -e "s|/[^/]*/[^/]*$||") /pfs/data5/project_ids.txt | cut -f 1 -d\ ) $HOME<br />
</pre><br />
<br />
== Workspaces ==<br />
<br />
On uc2 workspaces can be used to store large non-permanent data sets, e.g. restart files or output<br />
data that has to be post-processed. The file system used for workspaces is also the parallel file system Lustre. This file system is especially designed for parallel access and for a high throughput to large<br />
files. It is able to provide high data transfer rates of up to 54 GB/s write and read performance when data access is parallel. <br />
<br />
Workspaces have a lifetime and the data on a workspace expires as a whole after a fixed period. The maximum lifetime of a workspace is 60 days, but it can be renewed at the end of that period 3 times to a total maximum of 240 days after workspace generation. If a workspace has inadvertently expired we can restore the data during a limited time (few weeks). In this case you should create a new workspace and report the name of the new and of the expired workspace in a ticket.<br />
<br />
Creating, deleting, finding and extending workspaces is explained on the [[workspace]] page.<br />
<br />
On uc2 there is a default user quota limit of 40 TiB and 30 million inodes (files and directories) per user. <br />
You can chek your current usage and limits with the command<br />
<pre><br />
$ lfs quota -uh $(whoami) /pfs/work7<br />
</pre><br />
Note that the quotas include data and inodes for all of your workspaces and all of your expired workspaces (as long as they are not yet completely removed).<br />
<br />
=== Reminder for workspace deletion ===<br />
<br />
Normally you will get an email every day starting 7 days before a workspace expires. You can send yourself a calender entry which reminds you when a workspace will be automatically deleted:<br />
<br />
$ ws_send_ical.sh <workspace> <email><br />
<br />
== Improving Performance on $HOME and workspaces ==<br />
<br />
The following recommendations might help to improve throughput and metadata<br />
performance on Lustre filesystems.<br />
<br />
=== '''Improving Throughput Performance''' ===<br />
<br />
Depending on your application some adaptations might be necessary if you want to reach<br />
the full bandwidth of the filesystems. Parallel filesystems typically stripe files over storage subsystems, i.e. large files are separated into stripes and distributed to different storage subsystems. In Lustre, the size of these stripes (sometimes also mentioned as chunks) is called stripe size and the number of used storage subsystems is called stripe count.<br />
<br />
When you are designing your application you should consider that the performance of<br />
parallel filesystems is generally better if data is transferred in large blocks and stored in<br />
few large files. In more detail, to increase throughput performance of a parallel application<br />
following aspects should be considered:<br />
<br />
* collect large chunks of data and write them sequentially at once,<br />
<br />
* to exploit complete filesystem bandwidth use several clients,<br />
<br />
* avoid competitive file access by different tasks or use blocks with boundaries at stripe size (default is 1MB),<br />
<br />
* if files are small enough for the SSDs and are only used by one process store them on $TMP.<br />
<br />
With previous Lustre versions adapting the Lustre stripe count was the most important optimization. However, for the workspaces of uc2 the new Lustre feature Progressive File Layouts has been used to define file striping parameters. This means that the stripe count is adapted if the file size is growing. In normal cases users no longer need to adapt file striping parameters in case they have very huge files or in order to reach better performance. <br />
<br />
If you know what you are doing you can still change striping parameters, e.g. the stripe count, of a directory and of newly created files. New files and directories inherit the stripe count from the parent directory. E.g. if you want to enhance throughput on a single very large file which is created in the directory $HOME/my_output_dir you can use the command<br />
<pre><br />
$ lfs setstripe -c-1 $HOME/my_output_dir<br />
</pre><br />
to change the stripe count to -1 which means that all storage subsystems of the file system are used to store that file. If you change the stripe count of a directory the stripe count of existing files inside this<br />
directory is not changed. If you want to change the stripe count of existing files, change<br />
the stripe count of the parent directory, copy the files to new files, remove the old files<br />
and move the new files back to the old name. In order to check the stripe setting of<br />
the file my_file use<br />
<pre><br />
$ lfs getstripe my_file<br />
</pre><br />
Also note that changes on the striping parameters (e.g. stripe count) are not saved in the<br />
backup, i.e. if directories have to be recreated this information is lost and the default stripe<br />
count will be used. Therefore, you should annotate for which directories you made changes<br />
to the striping parameters so that you can repeat these changes if required.<br />
<br />
=== '''Improving Metadata Performance''' ===<br />
<br />
Metadata performance on parallel file systems is usually not as good as with local<br />
filesystems. In addition, it is usually not scalable, i.e. a limited resource. Therefore,<br />
you should omit metadata operations whenever possible. For example, it is much better<br />
to have few large files than lots of small files. In more detail, to increase metadata<br />
performance of a parallel application following aspects should be considered:<br />
<br />
* avoid creating many small files,<br />
<br />
* avoid competitive directory access, e.g. by creating files in separate subdirectories for each task,<br />
<br />
* if many small files are only used within a batch job and accessed by one process store them on $TMP,<br />
<br />
* change the default colorization setting of the command ls (see below).<br />
<br />
On modern Linux systems, the GNU ls command often uses colorization by default to<br />
visually highlight the file type; this is especially true if the command is run within a terminal<br />
session. This is because the default shell profile initializations usually contain an alias<br />
directive similar to the following for the ls command:<br />
<pre><br />
$ alias ls="ls --color=tty"<br />
</pre><br />
However, running the ls command in this way for files on a Lustre file system requires<br />
a stat() call to be used to determine the file type. This can result in a performance<br />
overhead, because the stat() call always needs to determine the size of a file, and that<br />
in turn means that the client node must query the object size of all the backing objects<br />
that make up a file. As a result of the default colorization setting, running a simple<br />
ls command on a Lustre file system often takes as much time as running the ls command<br />
with the -l option (the same is true if the -F, -p, or the -classify option, or any other option <br />
that requires information from a stat() call, is used). To avoid this performance overhead<br />
when using ls commands, add an alias directive similar to the following<br />
to your shell startup script:<br />
<pre><br />
$ alias ls="ls --color=never"<br />
</pre><br />
<br />
== Workspaces on flash storage ==<br />
<br />
There is another workspace file system for special reqirements available. The file system is called ''full flash pfs'' and is based on the parallel file system Lustre. Access will be granted on special request.<br />
<br />
=== Advantages of this file system ===<br />
<br />
# All storage devices are based on flash (no hard disks) with low access times. Hence performance is better compared to other parallel file systems for read and write access with small blocks and with small files, i.e. IOPS rates are improved.<br />
# The file system is mounted on bwUniCluster 2.0 and HoreKa, i.e. it can be used to share data between these clusters.<br />
<br />
=== Access restrictions ===<br />
<br />
* Only HoreKa users or KIT users of bwUniCluster 2.0 can use this file system.<br />
* Access is granted on request and for appropriate requirements. In order to request access, please open a ticket with the subject ''full flash pfs access required'' and describe the following topics:<br />
** number of files you want to store on this file system<br />
** needed capacity<br />
** number of nodes used by your typical jobs<br />
** special I/O requirements of your jobs<br />
<br />
=== Using the file system ===<br />
<br />
After access is granted, you can use the file system in the same way as a normal workspace. You just have to specify the name of the flash-based workspace file system using the option ''-F'' to all the commands that manage workspaces. On bwUniCluster 2.0 it is called ''ffuc'', on HoreKa it is ''ffhk''. For example, to create a workspace with name myws and a lifetime of 60 days on bwUniCluster 2.0 execute:<br />
ws_allocate -F ffuc myws 60<br />
<br />
If you want to use the full flash pfs on bwUniCluster 2.0 '''and''' HoreKa at the same time, please note that you only have to manage a particular workspace on one of the clusters since the name of the workspace directory is different. However, the path to each workspace is visible and can be used on both clusters.<br />
<br />
Other features are similar to normal workspaces. For example, we are able to restore expired workspaces for few weeks and you have to open a ticket to request the restore. There are quota limits with a default limit of 5 TB capacity and 25 millions inodes per user. You can check your current usage with<br />
lfs quota -uh $(whoami) /pfs/work8<br />
<br />
== $TMP ==<br />
<br />
While all tasks of a parallel application access the same $HOME and workspace directory, the <br />
$TMP directory is local to each node on bwUniCluster 2.0. All nodes have fast SSDs <br />
local storage devices which are used to store data below $TMP. Different tasks of a parallel<br />
application use different $TMP directories when they do not utilize one node. This directory should<br />
be used for temporary files being accessed by single tasks. It should also be used if you read the <br />
same data many times from a single node, e.g. if you are doing AI training. In this case you should <br />
copy the data at the beginning of your batch job to $TMP and read the data from there.<br />
In addition, this directory should be used for the installation<br />
of software packages. This means that the software package to be installed should be<br />
unpacked, compiled and linked in a subdirectory of $TMP. The real installation of the<br />
package (e.g. make install) should be made in(to) the Lustre filesystem. <br />
<br />
Each time a batch job is started, a subdirectory is created on each node and assigned to the job. <br />
$TMP is newly set and the name of the subdirectory contains the Job-id so that the<br />
subdirectory name is unique for each job. This unique name is then assigned to the<br />
environment variable $TMP within the job. At the end of the job the subdirectory is removed.<br />
Although $TMP points to the same path name for different nodes of a job, the physical location <br />
on these nodes is different.<br />
<br />
== LSDF Online Storage==<br />
<br />
In some cases it is useful to have access to the LSDF Online Storage on the HPC-Clusters also. Therefore the LSDF Online Storage is mounted on the Login- and Datamover-Nodes.<br />
Furthermore it can be used on the compute nodes during the job runtime with the constraint flag "LSDF" ([[bwUniCluster_2.0_Slurm_common_Features|Slurm common features]]<br />
). There is also an example about the LSDF batch usage: [[bwUniCluster_2.0_Slurm_common_Features#LSDF_Online_Storage|Slurm LSDF example ]] .<br />
<pre><br />
#!/bin/bash<br />
#SBATCH ...<br />
#SBATCH --constraint=LSDF<br />
</pre><br />
<br><br />
For the usage of the LSDF Online Storage the following environment variables are available: $LSDF, $LSDFPROJECTS, $LSDFHOME.<br />
Please request storage projects in the LSDF Online Storage seperately:<br />
[https://www.lsdf.kit.edu/os/storagerequest/: LSDF Storage Request].<br />
<br />
==BeeOND (BeeGFS On-Demand)==<br />
<br />
Users of the UniCluster have possibility to request a private BeeOND (on-demand BeeGFS) parallel filesystem for each job. The file system is created during job startup and purged after your job.<br />
<br />
'''IMPORTANT: All data on the private filesystem will be deleted after your job. Make sure you have copied your data back to the global filesystem (within job), e.g., $HOME or any workspace.'''<br />
<br />
BeeOND/BeeGFS can be used like any other parallel file system. Tools like cp or rsync can be used to copy data in and out. <br />
<br />
For detailed usage see here:[[BwUniCluster_2.0_Slurm_common_Features#BeeOND_.28BeeGFS_On-Demand.29|Request on-demand file system]]<br />
<br />
==Backup and Archiving==<br />
<br />
There are regular backups of all data of the home directories,whereas ACLs and extended attributes will<br />
not be backuped. <br />
<br />
Please open a ticket if you need backuped data.<br />
<br />
<br />
[[Category:bwUniCluster 2.0|File System]][[Category:Hardware_and_Architecture|bwUniCluster 2.0]]</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/FAQ&diff=11451
BwUniCluster2.0/FAQ
2022-11-14T16:09:02Z
<p>P Weisbrod: </p>
<hr />
<div>Frequently asked questions (FAQs) concerning best practice of bwUniCluster 2.0<br />
<br />
= Login =<br />
<br />
* In case of having issues with OTP, service password or denied access &rarr; [[BwUniCluster2.0/Login#Troubleshooting|Troubleshooting]]<br />
<br />
= Hardware and Architecture =<br />
<br />
* Migration from bwUniCluster 1.0 to 2.0<br />
** Concerning guide to migrate your data &rarr; [[BwUniCluster2.0/File_System_Migration_Guide|file system migration guide]]<br />
<br />
= Batch System =<br />
<br />
* Migration from bwUniCluster 1.0 to 2.0<br />
** Concerning switching from MOAB to Slurm and adapting your job scripts &rarr; [[BwUniCluster2.0/Batch_System_Migration_Guide|MOAB to Slurm migration guide]]</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Hardware_and_Architecture&diff=11450
BwUniCluster2.0/Hardware and Architecture
2022-11-14T16:06:26Z
<p>P Weisbrod: /* File Systems */</p>
<hr />
<div>= Architecture of bwUniCluster 2.0 =<br />
<br />
The bwUniCluster 2.0 is a parallel computer with distributed memory. Each node of system consists of at least two Intel Xeon processor, local memory, disks, network adapters and optionally accelerators (NVIDIA Tesla V100). All nodes are connected by a fast InfiniBand interconnect. In addition the file system<br />
Lustre, that is connected by coupling the InfiniBand of the file server with the InfiniBand<br />
switch of the compute cluster, is added to bwUniCluster 2.0 to provide a fast and scalable<br />
parallel file system.<br />
<br />
The operating system on each node is Red Hat Enterprise Linux (RHEL) 8.4. A number of additional software packages like e.g. SLURM have been installed on top. Some of these components are of special interest to end users and are briefly<br />
discussed in this document. Others which are of greater importance to system<br />
administrators will not be covered by this document.<br />
<br />
The individual nodes of the system may act in different roles. According to the services supplied by the nodes, they are separated into disjoint groups. From an end users point of view the different groups of nodes are login nodes, compute nodes, file server nodes and administrative server nodes.<br />
<br />
'''Login Nodes'''<br />
<br />
The login nodes are the only nodes that are directly accessible by end users. These nodes<br />
are used for interactive login, file management, program development and interactive pre-<br />
and postprocessing. Two nodes are dedicated to this service but they are all accessible via<br />
one address and a DNS round-robin alias distributes the login sessions to the<br />
different login nodes.<br />
<br />
'''Compute Node'''<br />
<br />
The majority of nodes are compute nodes which are managed by a batch system. Users <br />
submit their jobs to the SLURM batch system and a job is executed when the required resources become available (depending on its fair-share priority).<br />
<br />
'''File Server Nodes'''<br />
<br />
The hardware of the parallel file system Lustre incorporates some file server nodes; the file<br />
system Lustre is connected by coupling the InfiniBand of the file server with the independent InfiniBand switch of the compute cluster. In addition to shared file space there is also local storage on the disks of each node (for details see chapter "File Systems").<br />
<br />
'''Administrative Server Nodes'''<br />
<br />
Some other nodes are delivering additional services like resource management, external<br />
network connection, administration etc. These nodes can be accessed directly by system administrators only.<br />
<br />
= Components of bwUniCluster =<br />
<br />
{| class="wikitable"<br />
|- <br />
! style="width:9%"|<br />
! style="width:13%"| Compute nodes "Thin"<br />
! style="width:13%"| Compute nodes "HPC"<br />
! style="width:13%"| Compute nodes "Fat"<br />
! style="width:13%"| GPU x4<br />
! style="width:13%"| GPU x8<br />
! style="width:13%"| Login<br />
|-<br />
!scope="column"| Number of nodes<br />
| 100 + 60<br />
| 360<br />
| 6<br />
| 14<br />
| 10<br />
| 4<br />
|-<br />
!scope="column"| Processors<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6248<br />
|-<br />
!scope="column"| Number of sockets<br />
| 2<br />
| 2<br />
| 4<br />
| 2<br />
| 2<br />
| 2<br />
|-<br />
!scope="column"| Processor frequency (GHz)<br />
| 2.1 Ghz<br />
| 2.1 Ghz<br />
| 2.1 Ghz<br />
| 2.1 Ghz<br />
| 2.5 Ghz<br />
| <br />
|-<br />
!scope="column"| Total number of cores<br />
| 40<br />
| 40<br />
| 80<br />
| 40<br />
| 40<br />
| 40<br />
|-<br />
!scope="column"| Main memory<br />
| 96 GB / 192 GB<br />
| 96 GB<br />
| 3 TB<br />
| 384 GB<br />
| 768 GB<br />
| 384 GB<br />
|-<br />
!scope="column"| Local SSD<br />
| 960 GB SATA<br />
| 960 GB SATA<br />
| 4,8 TB NVMe<br />
| 3,2 TB NVMe<br />
| 6,4 TB NVMe<br />
| <br />
|-<br />
!scope="column"| Accelerators<br />
| -<br />
| -<br />
| -<br />
| 4x NVIDIA Tesla V100<br />
| 8x NVIDIA Tesla V100<br />
| <br />
|-<br />
!scope="column"| Interconnect<br />
| IB HDR100 (blocking)<br />
| IB HDR100<br />
| IB HDR<br />
| IB HDR<br />
| IB HDR<br />
| IB HDR100 (blocking)<br />
|}<br />
Table 1: Properties of the nodes<br />
<br />
= File Systems =<br />
<br />
On bwUniCluster 2.0 the parallel file system Lustre is used for most globally visible user data. Lustre is open source and Lustre solutions and support are available from different vendors. Nowadays, most of the biggest HPC systems are using Lustre. An initial home directory on a Lustre file system is created automatically after account activation, and the environment variable $HOME holds its name. Users can create so-called workspaces on another Lustre file system for non-permanent data with temporary lifetime. There is another workspace file system based on flash storage for special requirements available.<br />
<br />
Within a batch job further file systems are available: <br />
* The directory $TMP is only available and visible on the local node. It is located on fast SSD storage devices.<br />
* On request a parallel on-demand (BeeOND) file system is created which uses the SSDs of the nodes which were allocated to the batch job.<br />
* On request the external LSDF Online Storage is mounted on the nodes which were allocated to the batch job. This file system is based on the parallel file system Spectrum Scale. <br />
<br />
Some of the characteristics of the file systems are shown in Table 2.<br />
<br />
{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"<br />
|- style="width:20%;height=20px; text-align:left;padding:3px"<br />
! style="background-color:#AAA;padding:3px"| Property<br />
! style="background-color:#AAA;padding:3px"| $TMP<br />
! style="background-color:#AAA;padding:3px"| BeeOND<br />
! style="background-color:#AAA;padding:3px"| $HOME<br />
! style="background-color:#AAA;padding:3px"| Workspace<br />
! style="background-color:#AAA;padding:3px"| Workspace <br> on flash<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Visibility <br />
| style="height=20px; text-align:left;padding:3px"| local node<br />
| style="height=20px; text-align:left;padding:3px"| nodes of batch job<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Lifetime <br />
| style="height=20px; text-align:left;padding:3px"| batch job runtime<br />
| style="height=20px; text-align:left;padding:3px"| batch job runtime<br />
| style="height=20px; text-align:left;padding:3px"| permanent<br />
| style="height=20px; text-align:left;padding:3px"| max. 240 days<br />
| style="height=20px; text-align:left;padding:3px"| max. 240 days<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Disk space <br />
| style="height=20px; text-align:left;padding:3px"| 960 GB - 6.4 TB <br> details see table 1<br />
| style="height=20px; text-align:left;padding:3px"| n*750 GB<br />
| style="height=20px; text-align:left;padding:3px"| 1.2 PiB<br />
| style="height=20px; text-align:left;padding:3px"| 4.1 PiB<br />
| style="height=20px; text-align:left;padding:3px"| 236 TiB<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Capacity Quotas<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 1 TiB per user <br> also per organization<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 40 TiB per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 5 TiB per user<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Inode Quotas<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 10 million per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 30 million per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 25 million per user<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Backup<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Read perf./node<br />
| style="height=20px; text-align:left;padding:3px"| 500 MB/s - 6 GB/s <br> depends on type of local SSD / job queue: <br> 520 MB/s @ single / multiple <br> 800 MB/s @ multiple_e <br> 6600 MB/s @ fat <br> 6500 MB/s @ gpu_4 <br> 6500 MB/s @ gpu_8<br />
| style="height=20px; text-align:left;padding:3px"| 400 MB/s - 500 MB/s<br> depends on type of local SSDs / job queue: <br> 500 MB/s @ multiple <br> 400 MB/s @ multiple_e<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Write perf./node<br />
| style="height=20px; text-align:left;padding:3px"| 500 MB/s - 4 GB/s <br> depends on type of local SSD / job queue: <br> 500 MB/s @ single / multiple <br> 650 MB/s @ multiple_e <br> 2900 MB/s @ fat <br> 2090 MB/s @ gpu_4 <br> 4060 MB/s @ gpu_8<br />
| style="height=20px; text-align:left;padding:3px"| 250 MB/s - 350 MB/s <br> depends on type of local SSDs / job queue: <br> 350 MB/s @ multiple <br> 250 MB/s @ multiple_e<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Total read perf.<br />
| style="height=20px; text-align:left;padding:3px"| n*500-6000 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| n*400-500 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| 18 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 54 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 45 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Total write perf.<br />
| style="height=20px; text-align:left;padding:3px"| n*500-4000 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| n*250-350 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| 18 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 54 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 38 GB/s<br />
|}<br />
---------------------------------------------------------------------------------------------------------<br />
global: all nodes of uc1 access the same file system;<br />
local: each node has its own file system;<br />
permanent: files are stored permanently;<br />
batch job: files are removed at end of the batch job.<br />
---------------------------------------------------------------------------------------------------------<br />
Table 2: Properties of the file systems<br />
<br />
== Selecting the appropriate file system ==<br />
<br />
In general, you should separate your data and store it on the appropriate file system.<br />
Permanently needed data like software or important results should be stored below $HOME<br />
but capacity restrictions (quotas) apply. In case you accidentally deleted data on $HOME<br />
you can usually restore it from backup. Permanent data which is not needed for months<br />
or exceeds the capacity restrictions should be sent to the LSDF Online Storage <br />
or to the archive and deleted from the file systems. Temporary data which is only needed on a single<br />
node and which does not exceed the disk space shown in the table above should be stored<br />
below $TMP. Data which is read many times on a single node, e.g. if you are doing AI training, <br />
should be copied to $TMP and read from there. Temporary data which is used from many nodes <br />
of your batch job and which is only needed during job runtime should be stored on a <br />
parallel on-demand file system. Temporary data which can be recomputed or which is the <br />
result of one job and input for another job should be stored in workspaces. The lifetime <br />
of data in workspaces is limited and depends on the lifetime of the workspace which can be <br />
several months.<br />
<br />
For further details please check the chapters below.<br />
<br />
== $HOME ==<br />
<br />
The home directories of bwUniCluster 2.0 (uc2) users are located in the parallel file system Lustre.<br />
You have access to your home directory from all nodes of uc2. A regular backup of these directories <br />
to tape archive is done automatically. The directory $HOME is used to hold those files that are<br />
permanently used like source codes, configuration files, executable programs etc. <br />
<br />
On uc2 there is a default user quota limit of 1 TiB and 10 million inodes (files and directories) per user. <br />
You can chek your current usage and limits with the command<br />
<pre><br />
$ lfs quota -uh $(whoami) $HOME<br />
</pre><br />
In addition to the user limit there is a limit of your organization (e.g. university) which depends on the financial share. This limit is enforced with so-called Lustre project quotas. You can show the current usage and limits of your organization with the following command:<br />
<pre><br />
lfs quota -ph $(grep $(echo $HOME | sed -e "s|/[^/]*/[^/]*$||") /pfs/data5/project_ids.txt | cut -f 1 -d\ ) $HOME<br />
</pre><br />
<br />
== Workspaces ==<br />
<br />
On uc2 workspaces can be used to store large non-permanent data sets, e.g. restart files or output<br />
data that has to be post-processed. The file system used for workspaces is also the parallel file system Lustre. This file system is especially designed for parallel access and for a high throughput to large<br />
files. It is able to provide high data transfer rates of up to 54 GB/s write and read performance when data access is parallel. <br />
<br />
Workspaces have a lifetime and the data on a workspace expires as a whole after a fixed period. The maximum lifetime of a workspace is 60 days, but it can be renewed at the end of that period 3 times to a total maximum of 240 days after workspace generation. If a workspace has inadvertently expired we can restore the data during a limited time (few weeks). In this case you should create a new workspace and report the name of the new and of the expired workspace in a ticket.<br />
<br />
Creating, deleting, finding and extending workspaces is explained on the [[workspace]] page.<br />
<br />
On uc2 there is a default user quota limit of 40 TiB and 30 million inodes (files and directories) per user. <br />
You can chek your current usage and limits with the command<br />
<pre><br />
$ lfs quota -uh $(whoami) /pfs/work7<br />
</pre><br />
Note that the quotas include data and inodes for all of your workspaces and all of your expired workspaces (as long as they are not yet completely removed).<br />
<br />
=== Reminder for workspace deletion ===<br />
<br />
Normally you will get an email every day starting 7 days before a workspace expires. You can send yourself a calender entry which reminds you when a workspace will be automatically deleted:<br />
<br />
$ ws_send_ical.sh <workspace> <email><br />
<br />
== Improving Performance on $HOME and workspaces ==<br />
<br />
The following recommendations might help to improve throughput and metadata<br />
performance on Lustre filesystems.<br />
<br />
=== '''Improving Throughput Performance''' ===<br />
<br />
Depending on your application some adaptations might be necessary if you want to reach<br />
the full bandwidth of the filesystems. Parallel filesystems typically stripe files over storage subsystems, i.e. large files are separated into stripes and distributed to different storage subsystems. In Lustre, the size of these stripes (sometimes also mentioned as chunks) is called stripe size and the number of used storage subsystems is called stripe count.<br />
<br />
When you are designing your application you should consider that the performance of<br />
parallel filesystems is generally better if data is transferred in large blocks and stored in<br />
few large files. In more detail, to increase throughput performance of a parallel application<br />
following aspects should be considered:<br />
<br />
* collect large chunks of data and write them sequentially at once,<br />
<br />
* to exploit complete filesystem bandwidth use several clients,<br />
<br />
* avoid competitive file access by different tasks or use blocks with boundaries at stripe size (default is 1MB),<br />
<br />
* if files are small enough for the SSDs and are only used by one process store them on $TMP.<br />
<br />
With previous Lustre versions adapting the Lustre stripe count was the most important optimization. However, for the workspaces of uc2 the new Lustre feature Progressive File Layouts has been used to define file striping parameters. This means that the stripe count is adapted if the file size is growing. In normal cases users no longer need to adapt file striping parameters in case they have very huge files or in order to reach better performance. <br />
<br />
If you know what you are doing you can still change striping parameters, e.g. the stripe count, of a directory and of newly created files. New files and directories inherit the stripe count from the parent directory. E.g. if you want to enhance throughput on a single very large file which is created in the directory $HOME/my_output_dir you can use the command<br />
<pre><br />
$ lfs setstripe -c-1 $HOME/my_output_dir<br />
</pre><br />
to change the stripe count to -1 which means that all storage subsystems of the file system are used to store that file. If you change the stripe count of a directory the stripe count of existing files inside this<br />
directory is not changed. If you want to change the stripe count of existing files, change<br />
the stripe count of the parent directory, copy the files to new files, remove the old files<br />
and move the new files back to the old name. In order to check the stripe setting of<br />
the file my_file use<br />
<pre><br />
$ lfs getstripe my_file<br />
</pre><br />
Also note that changes on the striping parameters (e.g. stripe count) are not saved in the<br />
backup, i.e. if directories have to be recreated this information is lost and the default stripe<br />
count will be used. Therefore, you should annotate for which directories you made changes<br />
to the striping parameters so that you can repeat these changes if required.<br />
<br />
=== '''Improving Metadata Performance''' ===<br />
<br />
Metadata performance on parallel file systems is usually not as good as with local<br />
filesystems. In addition, it is usually not scalable, i.e. a limited resource. Therefore,<br />
you should omit metadata operations whenever possible. For example, it is much better<br />
to have few large files than lots of small files. In more detail, to increase metadata<br />
performance of a parallel application following aspects should be considered:<br />
<br />
* avoid creating many small files,<br />
<br />
* avoid competitive directory access, e.g. by creating files in separate subdirectories for each task,<br />
<br />
* if many small files are only used within a batch job and accessed by one process store them on $TMP,<br />
<br />
* change the default colorization setting of the command ls (see below).<br />
<br />
On modern Linux systems, the GNU ls command often uses colorization by default to<br />
visually highlight the file type; this is especially true if the command is run within a terminal<br />
session. This is because the default shell profile initializations usually contain an alias<br />
directive similar to the following for the ls command:<br />
<pre><br />
$ alias ls="ls --color=tty"<br />
</pre><br />
However, running the ls command in this way for files on a Lustre file system requires<br />
a stat() call to be used to determine the file type. This can result in a performance<br />
overhead, because the stat() call always needs to determine the size of a file, and that<br />
in turn means that the client node must query the object size of all the backing objects<br />
that make up a file. As a result of the default colorization setting, running a simple<br />
ls command on a Lustre file system often takes as much time as running the ls command<br />
with the -l option (the same is true if the -F, -p, or the -classify option, or any other option <br />
that requires information from a stat() call, is used). To avoid this performance overhead<br />
when using ls commands, add an alias directive similar to the following<br />
to your shell startup script:<br />
<pre><br />
$ alias ls="ls --color=never"<br />
</pre><br />
<br />
== Workspaces on flash storage ==<br />
<br />
There is another workspace file system for special reqirements available. The file system is called ''full flash pfs'' and is based on the parallel file system Lustre. Access will be granted on special request.<br />
<br />
=== Advantages of this file system ===<br />
<br />
# All storage devices are based on flash (no hard disks) with low access times. Hence performance is better compared to other parallel file systems for read and write access with small blocks and with small files, i.e. IOPS rates are improved.<br />
# The file system is mounted on bwUniCluster 2.0 and HoreKa, i.e. it can be used to share data between these clusters.<br />
<br />
=== Access restrictions ===<br />
<br />
* Only HoreKa users or KIT users of bwUniCluster 2.0 can use this file system.<br />
* Access is granted on request and for appropriate requirements. In order to request access, please open a ticket with the subject ''full flash pfs access required'' and describe the following topics:<br />
** number of files you want to store on this file system<br />
** needed capacity<br />
** number of nodes used by your typical jobs<br />
** special I/O requirements of your jobs<br />
<br />
=== Using the file system ===<br />
<br />
After access is granted, you can use the file system in the same way as a normal workspace. You just have to specify the name of the flash-based workspace file system using the option ''-F'' to all the commands that manage workspaces. On bwUniCluster 2.0 it is called ''ffuc'', on HoreKa it is ''ffhk''. For example, to create a workspace with name myws and a lifetime of 60 days on bwUniCluster 2.0 execute:<br />
ws_allocate -F ffuc myws 60<br />
<br />
If you want to use the full flash pfs on bwUniCluster 2.0 '''and''' HoreKa at the same time, please note that you only have to manage a particular workspace on one of the clusters since the name of the workspace directory is different. However, the path to each workspace is visible and can be used on both clusters.<br />
<br />
Other features are similar to normal workspaces. For example, we are able to restore expired workspaces for few weeks and you have to open a ticket to request the restore. There are quota limits with a default limit of 5 TB capacity and 25 millions inodes per user. You can check your current usage with<br />
lfs quota -uh $(whoami) /pfs/work8<br />
<br />
== $TMP ==<br />
<br />
While all tasks of a parallel application access the same $HOME and workspace directory, the <br />
$TMP directory is local to each node on bwUniCluster 2.0. All nodes have fast SSDs <br />
local storage devices which are used to store data below $TMP. Different tasks of a parallel<br />
application use different $TMP directories when they do not utilize one node. This directory should<br />
be used for temporary files being accessed by single tasks. It should also be used if you read the <br />
same data many times from a single node, e.g. if you are doing AI training. In this case you should <br />
copy the data at the beginning of your batch job to $TMP and read the data from there.<br />
In addition, this directory should be used for the installation<br />
of software packages. This means that the software package to be installed should be<br />
unpacked, compiled and linked in a subdirectory of $TMP. The real installation of the<br />
package (e.g. make install) should be made in(to) the Lustre filesystem. <br />
<br />
Each time a batch job is started, a subdirectory is created on each node and assigned to the job. <br />
$TMP is newly set and the name of the subdirectory contains the Job-id so that the<br />
subdirectory name is unique for each job. This unique name is then assigned to the<br />
environment variable $TMP within the job. At the end of the job the subdirectory is removed.<br />
Although $TMP points to the same path name for different nodes of a job, the physical location <br />
on these nodes is different.<br />
<br />
== LSDF Online Storage==<br />
<br />
In some cases it is useful to have access to the LSDF Online Storage on the HPC-Clusters also. Therefore the LSDF Online Storage is mounted on the Login- and Datamover-Nodes.<br />
Furthermore it can be used on the compute nodes during the job runtime with the constraint flag "LSDF" ([[bwUniCluster_2.0_Slurm_common_Features|Slurm common features]]<br />
). There is also an example about the LSDF batch usage: [[bwUniCluster_2.0_Slurm_common_Features#LSDF_Online_Storage|Slurm LSDF example ]] .<br />
<pre><br />
#!/bin/bash<br />
#SBATCH ...<br />
#SBATCH --constraint=LSDF<br />
</pre><br />
<br><br />
For the usage of the LSDF Online Storage the following environment variables are available: $LSDF, $LSDFPROJECTS, $LSDFHOME.<br />
Please request storage projects in the LSDF Online Storage seperately:<br />
[https://www.lsdf.kit.edu/os/storagerequest/: LSDF Storage Request].<br />
<br />
==BeeOND (BeeGFS On-Demand)==<br />
<br />
Users of the HPC-Cluster have possibility to request a private BeeOND (BeeGFS) parallel filesystem for each job. The file system is created during job startup and purged after your job.<br />
<br />
'''IMPORTANT: All data on the private filesystem will be deleted after your job. Make sure you have copied your data back to the global filesystem (within job), e.g., $HOME or any workspace.'''<br />
<br />
BeeOND/BeeGFS can be used like any other parallel file system. Tools like cp or rsync can be used to copy data in and out. <br />
<br />
For detailed usage see here:[[BwUniCluster_2.0_Slurm_common_Features#BeeOND_.28BeeGFS_On-Demand.29|Request on-demand file system]]<br />
<br />
==Backup and Archiving==<br />
<br />
There are regular backups of all data of the home directories,whereas ACLs and extended attributes will<br />
not be backuped. <br />
<br />
Please open a ticket if you need backuped data.<br />
<br />
<br />
[[Category:bwUniCluster 2.0|File System]][[Category:Hardware_and_Architecture|bwUniCluster 2.0]]</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Hardware_and_Architecture&diff=11449
BwUniCluster2.0/Hardware and Architecture
2022-11-14T16:05:53Z
<p>P Weisbrod: /* Components of bwUniCluster */</p>
<hr />
<div>= Architecture of bwUniCluster 2.0 =<br />
<br />
The bwUniCluster 2.0 is a parallel computer with distributed memory. Each node of system consists of at least two Intel Xeon processor, local memory, disks, network adapters and optionally accelerators (NVIDIA Tesla V100). All nodes are connected by a fast InfiniBand interconnect. In addition the file system<br />
Lustre, that is connected by coupling the InfiniBand of the file server with the InfiniBand<br />
switch of the compute cluster, is added to bwUniCluster 2.0 to provide a fast and scalable<br />
parallel file system.<br />
<br />
The operating system on each node is Red Hat Enterprise Linux (RHEL) 8.4. A number of additional software packages like e.g. SLURM have been installed on top. Some of these components are of special interest to end users and are briefly<br />
discussed in this document. Others which are of greater importance to system<br />
administrators will not be covered by this document.<br />
<br />
The individual nodes of the system may act in different roles. According to the services supplied by the nodes, they are separated into disjoint groups. From an end users point of view the different groups of nodes are login nodes, compute nodes, file server nodes and administrative server nodes.<br />
<br />
'''Login Nodes'''<br />
<br />
The login nodes are the only nodes that are directly accessible by end users. These nodes<br />
are used for interactive login, file management, program development and interactive pre-<br />
and postprocessing. Two nodes are dedicated to this service but they are all accessible via<br />
one address and a DNS round-robin alias distributes the login sessions to the<br />
different login nodes.<br />
<br />
'''Compute Node'''<br />
<br />
The majority of nodes are compute nodes which are managed by a batch system. Users <br />
submit their jobs to the SLURM batch system and a job is executed when the required resources become available (depending on its fair-share priority).<br />
<br />
'''File Server Nodes'''<br />
<br />
The hardware of the parallel file system Lustre incorporates some file server nodes; the file<br />
system Lustre is connected by coupling the InfiniBand of the file server with the independent InfiniBand switch of the compute cluster. In addition to shared file space there is also local storage on the disks of each node (for details see chapter "File Systems").<br />
<br />
'''Administrative Server Nodes'''<br />
<br />
Some other nodes are delivering additional services like resource management, external<br />
network connection, administration etc. These nodes can be accessed directly by system administrators only.<br />
<br />
= Components of bwUniCluster =<br />
<br />
{| class="wikitable"<br />
|- <br />
! style="width:9%"|<br />
! style="width:13%"| Compute nodes "Thin"<br />
! style="width:13%"| Compute nodes "HPC"<br />
! style="width:13%"| Compute nodes "Fat"<br />
! style="width:13%"| GPU x4<br />
! style="width:13%"| GPU x8<br />
! style="width:13%"| Login<br />
|-<br />
!scope="column"| Number of nodes<br />
| 100 + 60<br />
| 360<br />
| 6<br />
| 14<br />
| 10<br />
| 4<br />
|-<br />
!scope="column"| Processors<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6248<br />
|-<br />
!scope="column"| Number of sockets<br />
| 2<br />
| 2<br />
| 4<br />
| 2<br />
| 2<br />
| 2<br />
|-<br />
!scope="column"| Processor frequency (GHz)<br />
| 2.1 Ghz<br />
| 2.1 Ghz<br />
| 2.1 Ghz<br />
| 2.1 Ghz<br />
| 2.5 Ghz<br />
| <br />
|-<br />
!scope="column"| Total number of cores<br />
| 40<br />
| 40<br />
| 80<br />
| 40<br />
| 40<br />
| 40<br />
|-<br />
!scope="column"| Main memory<br />
| 96 GB / 192 GB<br />
| 96 GB<br />
| 3 TB<br />
| 384 GB<br />
| 768 GB<br />
| 384 GB<br />
|-<br />
!scope="column"| Local SSD<br />
| 960 GB SATA<br />
| 960 GB SATA<br />
| 4,8 TB NVMe<br />
| 3,2 TB NVMe<br />
| 6,4 TB NVMe<br />
| <br />
|-<br />
!scope="column"| Accelerators<br />
| -<br />
| -<br />
| -<br />
| 4x NVIDIA Tesla V100<br />
| 8x NVIDIA Tesla V100<br />
| <br />
|-<br />
!scope="column"| Interconnect<br />
| IB HDR100 (blocking)<br />
| IB HDR100<br />
| IB HDR<br />
| IB HDR<br />
| IB HDR<br />
| IB HDR100 (blocking)<br />
|}<br />
Table 1: Properties of the nodes<br />
<br />
= File Systems =<br />
<br />
On bwUniCluster 2.0 the parallel file system Lustre is used for most globally visible user data. Lustre is open source and Lustre solutions and support are available from different vendors. Nowadays, most of the biggest HPC systems are using Lustre. An initial home directory on a Lustre file system is created automatically after account activation, and the environment variable $HOME holds its name. Users can create so-called workspaces on another Lustre file system for non-permanent data with temporary lifetime. There is another workspace file system based on flash storage for special requirements available.<br />
<br />
Within a batch job further file systems are available: <br />
* The directory $TMP is only available and visible on the local node. It is located on fast SSD storage devices.<br />
* On request a parallel on-demand (BeeOND) file system is created which uses the SSDs of the nodes which were allocated to the batch job.<br />
* On request the external LSDF Online Storage is mounted on the nodes which were allocated to the batch job. This file system is based on the parallel file system Spectrum Scale. <br />
<br />
Some of the characteristics of the file systems are shown in Table 2.<br />
<br />
{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"<br />
|- style="width:20%;height=20px; text-align:left;padding:3px"<br />
! style="background-color:#AAA;padding:3px"| Property<br />
! style="background-color:#AAA;padding:3px"| $TMP<br />
! style="background-color:#AAA;padding:3px"| BeeOND<br />
! style="background-color:#AAA;padding:3px"| $HOME<br />
! style="background-color:#AAA;padding:3px"| Workspace<br />
! style="background-color:#AAA;padding:3px"| Workspace <br> on flash<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Visibility <br />
| style="height=20px; text-align:left;padding:3px"| local node<br />
| style="height=20px; text-align:left;padding:3px"| nodes of batch job<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Lifetime <br />
| style="height=20px; text-align:left;padding:3px"| batch job runtime<br />
| style="height=20px; text-align:left;padding:3px"| batch job runtime<br />
| style="height=20px; text-align:left;padding:3px"| permanent<br />
| style="height=20px; text-align:left;padding:3px"| max. 240 days<br />
| style="height=20px; text-align:left;padding:3px"| max. 240 days<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Disk space <br />
| style="height=20px; text-align:left;padding:3px"| 960 GB - 6.4 TB <br> details see table 1<br />
| style="height=20px; text-align:left;padding:3px"| n*250 GB<br />
| style="height=20px; text-align:left;padding:3px"| 1.2 PiB<br />
| style="height=20px; text-align:left;padding:3px"| 4.1 PiB<br />
| style="height=20px; text-align:left;padding:3px"| 236 TiB<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Capacity Quotas<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 1 TiB per user <br> also per organization<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 40 TiB per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 5 TiB per user<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Inode Quotas<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 10 million per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 30 million per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 25 million per user<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Backup<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Read perf./node<br />
| style="height=20px; text-align:left;padding:3px"| 500 MB/s - 6 GB/s <br> depends on type of local SSD / job queue: <br> 520 MB/s @ single / multiple <br> 800 MB/s @ multiple_e <br> 6600 MB/s @ fat <br> 6500 MB/s @ gpu_4 <br> 6500 MB/s @ gpu_8<br />
| style="height=20px; text-align:left;padding:3px"| 400 MB/s - 500 MB/s<br> depends on type of local SSDs / job queue: <br> 500 MB/s @ multiple <br> 400 MB/s @ multiple_e<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Write perf./node<br />
| style="height=20px; text-align:left;padding:3px"| 500 MB/s - 4 GB/s <br> depends on type of local SSD / job queue: <br> 500 MB/s @ single / multiple <br> 650 MB/s @ multiple_e <br> 2900 MB/s @ fat <br> 2090 MB/s @ gpu_4 <br> 4060 MB/s @ gpu_8<br />
| style="height=20px; text-align:left;padding:3px"| 250 MB/s - 350 MB/s <br> depends on type of local SSDs / job queue: <br> 350 MB/s @ multiple <br> 250 MB/s @ multiple_e<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Total read perf.<br />
| style="height=20px; text-align:left;padding:3px"| n*500-6000 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| n*400-500 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| 18 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 54 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 45 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Total write perf.<br />
| style="height=20px; text-align:left;padding:3px"| n*500-4000 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| n*250-350 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| 18 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 54 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 38 GB/s<br />
|}<br />
---------------------------------------------------------------------------------------------------------<br />
global: all nodes of uc1 access the same file system;<br />
local: each node has its own file system;<br />
permanent: files are stored permanently;<br />
batch job: files are removed at end of the batch job.<br />
---------------------------------------------------------------------------------------------------------<br />
Table 2: Properties of the file systems<br />
<br />
== Selecting the appropriate file system ==<br />
<br />
In general, you should separate your data and store it on the appropriate file system.<br />
Permanently needed data like software or important results should be stored below $HOME<br />
but capacity restrictions (quotas) apply. In case you accidentally deleted data on $HOME<br />
you can usually restore it from backup. Permanent data which is not needed for months<br />
or exceeds the capacity restrictions should be sent to the LSDF Online Storage <br />
or to the archive and deleted from the file systems. Temporary data which is only needed on a single<br />
node and which does not exceed the disk space shown in the table above should be stored<br />
below $TMP. Data which is read many times on a single node, e.g. if you are doing AI training, <br />
should be copied to $TMP and read from there. Temporary data which is used from many nodes <br />
of your batch job and which is only needed during job runtime should be stored on a <br />
parallel on-demand file system. Temporary data which can be recomputed or which is the <br />
result of one job and input for another job should be stored in workspaces. The lifetime <br />
of data in workspaces is limited and depends on the lifetime of the workspace which can be <br />
several months.<br />
<br />
For further details please check the chapters below.<br />
<br />
== $HOME ==<br />
<br />
The home directories of bwUniCluster 2.0 (uc2) users are located in the parallel file system Lustre.<br />
You have access to your home directory from all nodes of uc2. A regular backup of these directories <br />
to tape archive is done automatically. The directory $HOME is used to hold those files that are<br />
permanently used like source codes, configuration files, executable programs etc. <br />
<br />
On uc2 there is a default user quota limit of 1 TiB and 10 million inodes (files and directories) per user. <br />
You can chek your current usage and limits with the command<br />
<pre><br />
$ lfs quota -uh $(whoami) $HOME<br />
</pre><br />
In addition to the user limit there is a limit of your organization (e.g. university) which depends on the financial share. This limit is enforced with so-called Lustre project quotas. You can show the current usage and limits of your organization with the following command:<br />
<pre><br />
lfs quota -ph $(grep $(echo $HOME | sed -e "s|/[^/]*/[^/]*$||") /pfs/data5/project_ids.txt | cut -f 1 -d\ ) $HOME<br />
</pre><br />
<br />
== Workspaces ==<br />
<br />
On uc2 workspaces can be used to store large non-permanent data sets, e.g. restart files or output<br />
data that has to be post-processed. The file system used for workspaces is also the parallel file system Lustre. This file system is especially designed for parallel access and for a high throughput to large<br />
files. It is able to provide high data transfer rates of up to 54 GB/s write and read performance when data access is parallel. <br />
<br />
Workspaces have a lifetime and the data on a workspace expires as a whole after a fixed period. The maximum lifetime of a workspace is 60 days, but it can be renewed at the end of that period 3 times to a total maximum of 240 days after workspace generation. If a workspace has inadvertently expired we can restore the data during a limited time (few weeks). In this case you should create a new workspace and report the name of the new and of the expired workspace in a ticket.<br />
<br />
Creating, deleting, finding and extending workspaces is explained on the [[workspace]] page.<br />
<br />
On uc2 there is a default user quota limit of 40 TiB and 30 million inodes (files and directories) per user. <br />
You can chek your current usage and limits with the command<br />
<pre><br />
$ lfs quota -uh $(whoami) /pfs/work7<br />
</pre><br />
Note that the quotas include data and inodes for all of your workspaces and all of your expired workspaces (as long as they are not yet completely removed).<br />
<br />
=== Reminder for workspace deletion ===<br />
<br />
Normally you will get an email every day starting 7 days before a workspace expires. You can send yourself a calender entry which reminds you when a workspace will be automatically deleted:<br />
<br />
$ ws_send_ical.sh <workspace> <email><br />
<br />
== Improving Performance on $HOME and workspaces ==<br />
<br />
The following recommendations might help to improve throughput and metadata<br />
performance on Lustre filesystems.<br />
<br />
=== '''Improving Throughput Performance''' ===<br />
<br />
Depending on your application some adaptations might be necessary if you want to reach<br />
the full bandwidth of the filesystems. Parallel filesystems typically stripe files over storage subsystems, i.e. large files are separated into stripes and distributed to different storage subsystems. In Lustre, the size of these stripes (sometimes also mentioned as chunks) is called stripe size and the number of used storage subsystems is called stripe count.<br />
<br />
When you are designing your application you should consider that the performance of<br />
parallel filesystems is generally better if data is transferred in large blocks and stored in<br />
few large files. In more detail, to increase throughput performance of a parallel application<br />
following aspects should be considered:<br />
<br />
* collect large chunks of data and write them sequentially at once,<br />
<br />
* to exploit complete filesystem bandwidth use several clients,<br />
<br />
* avoid competitive file access by different tasks or use blocks with boundaries at stripe size (default is 1MB),<br />
<br />
* if files are small enough for the SSDs and are only used by one process store them on $TMP.<br />
<br />
With previous Lustre versions adapting the Lustre stripe count was the most important optimization. However, for the workspaces of uc2 the new Lustre feature Progressive File Layouts has been used to define file striping parameters. This means that the stripe count is adapted if the file size is growing. In normal cases users no longer need to adapt file striping parameters in case they have very huge files or in order to reach better performance. <br />
<br />
If you know what you are doing you can still change striping parameters, e.g. the stripe count, of a directory and of newly created files. New files and directories inherit the stripe count from the parent directory. E.g. if you want to enhance throughput on a single very large file which is created in the directory $HOME/my_output_dir you can use the command<br />
<pre><br />
$ lfs setstripe -c-1 $HOME/my_output_dir<br />
</pre><br />
to change the stripe count to -1 which means that all storage subsystems of the file system are used to store that file. If you change the stripe count of a directory the stripe count of existing files inside this<br />
directory is not changed. If you want to change the stripe count of existing files, change<br />
the stripe count of the parent directory, copy the files to new files, remove the old files<br />
and move the new files back to the old name. In order to check the stripe setting of<br />
the file my_file use<br />
<pre><br />
$ lfs getstripe my_file<br />
</pre><br />
Also note that changes on the striping parameters (e.g. stripe count) are not saved in the<br />
backup, i.e. if directories have to be recreated this information is lost and the default stripe<br />
count will be used. Therefore, you should annotate for which directories you made changes<br />
to the striping parameters so that you can repeat these changes if required.<br />
<br />
=== '''Improving Metadata Performance''' ===<br />
<br />
Metadata performance on parallel file systems is usually not as good as with local<br />
filesystems. In addition, it is usually not scalable, i.e. a limited resource. Therefore,<br />
you should omit metadata operations whenever possible. For example, it is much better<br />
to have few large files than lots of small files. In more detail, to increase metadata<br />
performance of a parallel application following aspects should be considered:<br />
<br />
* avoid creating many small files,<br />
<br />
* avoid competitive directory access, e.g. by creating files in separate subdirectories for each task,<br />
<br />
* if many small files are only used within a batch job and accessed by one process store them on $TMP,<br />
<br />
* change the default colorization setting of the command ls (see below).<br />
<br />
On modern Linux systems, the GNU ls command often uses colorization by default to<br />
visually highlight the file type; this is especially true if the command is run within a terminal<br />
session. This is because the default shell profile initializations usually contain an alias<br />
directive similar to the following for the ls command:<br />
<pre><br />
$ alias ls="ls --color=tty"<br />
</pre><br />
However, running the ls command in this way for files on a Lustre file system requires<br />
a stat() call to be used to determine the file type. This can result in a performance<br />
overhead, because the stat() call always needs to determine the size of a file, and that<br />
in turn means that the client node must query the object size of all the backing objects<br />
that make up a file. As a result of the default colorization setting, running a simple<br />
ls command on a Lustre file system often takes as much time as running the ls command<br />
with the -l option (the same is true if the -F, -p, or the -classify option, or any other option <br />
that requires information from a stat() call, is used). To avoid this performance overhead<br />
when using ls commands, add an alias directive similar to the following<br />
to your shell startup script:<br />
<pre><br />
$ alias ls="ls --color=never"<br />
</pre><br />
<br />
== Workspaces on flash storage ==<br />
<br />
There is another workspace file system for special reqirements available. The file system is called ''full flash pfs'' and is based on the parallel file system Lustre. Access will be granted on special request.<br />
<br />
=== Advantages of this file system ===<br />
<br />
# All storage devices are based on flash (no hard disks) with low access times. Hence performance is better compared to other parallel file systems for read and write access with small blocks and with small files, i.e. IOPS rates are improved.<br />
# The file system is mounted on bwUniCluster 2.0 and HoreKa, i.e. it can be used to share data between these clusters.<br />
<br />
=== Access restrictions ===<br />
<br />
* Only HoreKa users or KIT users of bwUniCluster 2.0 can use this file system.<br />
* Access is granted on request and for appropriate requirements. In order to request access, please open a ticket with the subject ''full flash pfs access required'' and describe the following topics:<br />
** number of files you want to store on this file system<br />
** needed capacity<br />
** number of nodes used by your typical jobs<br />
** special I/O requirements of your jobs<br />
<br />
=== Using the file system ===<br />
<br />
After access is granted, you can use the file system in the same way as a normal workspace. You just have to specify the name of the flash-based workspace file system using the option ''-F'' to all the commands that manage workspaces. On bwUniCluster 2.0 it is called ''ffuc'', on HoreKa it is ''ffhk''. For example, to create a workspace with name myws and a lifetime of 60 days on bwUniCluster 2.0 execute:<br />
ws_allocate -F ffuc myws 60<br />
<br />
If you want to use the full flash pfs on bwUniCluster 2.0 '''and''' HoreKa at the same time, please note that you only have to manage a particular workspace on one of the clusters since the name of the workspace directory is different. However, the path to each workspace is visible and can be used on both clusters.<br />
<br />
Other features are similar to normal workspaces. For example, we are able to restore expired workspaces for few weeks and you have to open a ticket to request the restore. There are quota limits with a default limit of 5 TB capacity and 25 millions inodes per user. You can check your current usage with<br />
lfs quota -uh $(whoami) /pfs/work8<br />
<br />
== $TMP ==<br />
<br />
While all tasks of a parallel application access the same $HOME and workspace directory, the <br />
$TMP directory is local to each node on bwUniCluster 2.0. All nodes have fast SSDs <br />
local storage devices which are used to store data below $TMP. Different tasks of a parallel<br />
application use different $TMP directories when they do not utilize one node. This directory should<br />
be used for temporary files being accessed by single tasks. It should also be used if you read the <br />
same data many times from a single node, e.g. if you are doing AI training. In this case you should <br />
copy the data at the beginning of your batch job to $TMP and read the data from there.<br />
In addition, this directory should be used for the installation<br />
of software packages. This means that the software package to be installed should be<br />
unpacked, compiled and linked in a subdirectory of $TMP. The real installation of the<br />
package (e.g. make install) should be made in(to) the Lustre filesystem. <br />
<br />
Each time a batch job is started, a subdirectory is created on each node and assigned to the job. <br />
$TMP is newly set and the name of the subdirectory contains the Job-id so that the<br />
subdirectory name is unique for each job. This unique name is then assigned to the<br />
environment variable $TMP within the job. At the end of the job the subdirectory is removed.<br />
Although $TMP points to the same path name for different nodes of a job, the physical location <br />
on these nodes is different.<br />
<br />
== LSDF Online Storage==<br />
<br />
In some cases it is useful to have access to the LSDF Online Storage on the HPC-Clusters also. Therefore the LSDF Online Storage is mounted on the Login- and Datamover-Nodes.<br />
Furthermore it can be used on the compute nodes during the job runtime with the constraint flag "LSDF" ([[bwUniCluster_2.0_Slurm_common_Features|Slurm common features]]<br />
). There is also an example about the LSDF batch usage: [[bwUniCluster_2.0_Slurm_common_Features#LSDF_Online_Storage|Slurm LSDF example ]] .<br />
<pre><br />
#!/bin/bash<br />
#SBATCH ...<br />
#SBATCH --constraint=LSDF<br />
</pre><br />
<br><br />
For the usage of the LSDF Online Storage the following environment variables are available: $LSDF, $LSDFPROJECTS, $LSDFHOME.<br />
Please request storage projects in the LSDF Online Storage seperately:<br />
[https://www.lsdf.kit.edu/os/storagerequest/: LSDF Storage Request].<br />
<br />
==BeeOND (BeeGFS On-Demand)==<br />
<br />
Users of the HPC-Cluster have possibility to request a private BeeOND (BeeGFS) parallel filesystem for each job. The file system is created during job startup and purged after your job.<br />
<br />
'''IMPORTANT: All data on the private filesystem will be deleted after your job. Make sure you have copied your data back to the global filesystem (within job), e.g., $HOME or any workspace.'''<br />
<br />
BeeOND/BeeGFS can be used like any other parallel file system. Tools like cp or rsync can be used to copy data in and out. <br />
<br />
For detailed usage see here:[[BwUniCluster_2.0_Slurm_common_Features#BeeOND_.28BeeGFS_On-Demand.29|Request on-demand file system]]<br />
<br />
==Backup and Archiving==<br />
<br />
There are regular backups of all data of the home directories,whereas ACLs and extended attributes will<br />
not be backuped. <br />
<br />
Please open a ticket if you need backuped data.<br />
<br />
<br />
[[Category:bwUniCluster 2.0|File System]][[Category:Hardware_and_Architecture|bwUniCluster 2.0]]</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Hardware_and_Architecture&diff=11448
BwUniCluster2.0/Hardware and Architecture
2022-11-14T16:05:21Z
<p>P Weisbrod: /* Components of bwUniCluster */</p>
<hr />
<div>= Architecture of bwUniCluster 2.0 =<br />
<br />
The bwUniCluster 2.0 is a parallel computer with distributed memory. Each node of system consists of at least two Intel Xeon processor, local memory, disks, network adapters and optionally accelerators (NVIDIA Tesla V100). All nodes are connected by a fast InfiniBand interconnect. In addition the file system<br />
Lustre, that is connected by coupling the InfiniBand of the file server with the InfiniBand<br />
switch of the compute cluster, is added to bwUniCluster 2.0 to provide a fast and scalable<br />
parallel file system.<br />
<br />
The operating system on each node is Red Hat Enterprise Linux (RHEL) 8.4. A number of additional software packages like e.g. SLURM have been installed on top. Some of these components are of special interest to end users and are briefly<br />
discussed in this document. Others which are of greater importance to system<br />
administrators will not be covered by this document.<br />
<br />
The individual nodes of the system may act in different roles. According to the services supplied by the nodes, they are separated into disjoint groups. From an end users point of view the different groups of nodes are login nodes, compute nodes, file server nodes and administrative server nodes.<br />
<br />
'''Login Nodes'''<br />
<br />
The login nodes are the only nodes that are directly accessible by end users. These nodes<br />
are used for interactive login, file management, program development and interactive pre-<br />
and postprocessing. Two nodes are dedicated to this service but they are all accessible via<br />
one address and a DNS round-robin alias distributes the login sessions to the<br />
different login nodes.<br />
<br />
'''Compute Node'''<br />
<br />
The majority of nodes are compute nodes which are managed by a batch system. Users <br />
submit their jobs to the SLURM batch system and a job is executed when the required resources become available (depending on its fair-share priority).<br />
<br />
'''File Server Nodes'''<br />
<br />
The hardware of the parallel file system Lustre incorporates some file server nodes; the file<br />
system Lustre is connected by coupling the InfiniBand of the file server with the independent InfiniBand switch of the compute cluster. In addition to shared file space there is also local storage on the disks of each node (for details see chapter "File Systems").<br />
<br />
'''Administrative Server Nodes'''<br />
<br />
Some other nodes are delivering additional services like resource management, external<br />
network connection, administration etc. These nodes can be accessed directly by system administrators only.<br />
<br />
= Components of bwUniCluster =<br />
<br />
{| class="wikitable"<br />
|- <br />
! style="width:9%"|<br />
! style="width:13%"| Compute nodes "Thin"<br />
! style="width:13%"| Compute nodes "HPC"<br />
! style="width:13%"| Compute nodes "Fat"<br />
! style="width:13%"| GPU x4<br />
! style="width:13%"| GPU x8<br />
! style="width:13%"| Login<br />
|-<br />
!scope="column"| Number of nodes<br />
| 100 + 60<br />
| 360<br />
| 6<br />
| 14<br />
| 10<br />
| 4<br />
|-<br />
!scope="column"| Processors<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6248<br />
|-<br />
!scope="column"| Number of sockets<br />
| 2<br />
| 2<br />
| 4<br />
| 2<br />
| 2<br />
| 2<br />
|-<br />
!scope="column"| Processor frequency (GHz)<br />
| 2.1 Ghz<br />
| 2.1 Ghz<br />
| 2.1 Ghz<br />
| 2.1 Ghz<br />
| 2.5 Ghz<br />
| <br />
|-<br />
!scope="column"| Total number of cores<br />
| 40<br />
| 40<br />
| 80<br />
| 40<br />
| 40<br />
| 40 / 20 (Broadwell)<br />
|-<br />
!scope="column"| Main memory<br />
| 96 GB / 192 GB<br />
| 96 GB<br />
| 3 TB<br />
| 384 GB<br />
| 768 GB<br />
| 384 GB / 128 GB (Broadwell)<br />
|-<br />
!scope="column"| Local SSD<br />
| 960 GB SATA<br />
| 960 GB SATA<br />
| 4,8 TB NVMe<br />
| 3,2 TB NVMe<br />
| 6,4 TB NVMe<br />
| <br />
|-<br />
!scope="column"| Accelerators<br />
| -<br />
| -<br />
| -<br />
| 4x NVIDIA Tesla V100<br />
| 8x NVIDIA Tesla V100<br />
| <br />
|-<br />
!scope="column"| Interconnect<br />
| IB HDR100 (blocking)<br />
| IB HDR100<br />
| IB HDR<br />
| IB HDR<br />
| IB HDR<br />
| IB HDR100 (blocking)<br />
|}<br />
Table 1: Properties of the nodes<br />
<br />
= File Systems =<br />
<br />
On bwUniCluster 2.0 the parallel file system Lustre is used for most globally visible user data. Lustre is open source and Lustre solutions and support are available from different vendors. Nowadays, most of the biggest HPC systems are using Lustre. An initial home directory on a Lustre file system is created automatically after account activation, and the environment variable $HOME holds its name. Users can create so-called workspaces on another Lustre file system for non-permanent data with temporary lifetime. There is another workspace file system based on flash storage for special requirements available.<br />
<br />
Within a batch job further file systems are available: <br />
* The directory $TMP is only available and visible on the local node. It is located on fast SSD storage devices.<br />
* On request a parallel on-demand (BeeOND) file system is created which uses the SSDs of the nodes which were allocated to the batch job.<br />
* On request the external LSDF Online Storage is mounted on the nodes which were allocated to the batch job. This file system is based on the parallel file system Spectrum Scale. <br />
<br />
Some of the characteristics of the file systems are shown in Table 2.<br />
<br />
{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"<br />
|- style="width:20%;height=20px; text-align:left;padding:3px"<br />
! style="background-color:#AAA;padding:3px"| Property<br />
! style="background-color:#AAA;padding:3px"| $TMP<br />
! style="background-color:#AAA;padding:3px"| BeeOND<br />
! style="background-color:#AAA;padding:3px"| $HOME<br />
! style="background-color:#AAA;padding:3px"| Workspace<br />
! style="background-color:#AAA;padding:3px"| Workspace <br> on flash<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Visibility <br />
| style="height=20px; text-align:left;padding:3px"| local node<br />
| style="height=20px; text-align:left;padding:3px"| nodes of batch job<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Lifetime <br />
| style="height=20px; text-align:left;padding:3px"| batch job runtime<br />
| style="height=20px; text-align:left;padding:3px"| batch job runtime<br />
| style="height=20px; text-align:left;padding:3px"| permanent<br />
| style="height=20px; text-align:left;padding:3px"| max. 240 days<br />
| style="height=20px; text-align:left;padding:3px"| max. 240 days<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Disk space <br />
| style="height=20px; text-align:left;padding:3px"| 960 GB - 6.4 TB <br> details see table 1<br />
| style="height=20px; text-align:left;padding:3px"| n*250 GB<br />
| style="height=20px; text-align:left;padding:3px"| 1.2 PiB<br />
| style="height=20px; text-align:left;padding:3px"| 4.1 PiB<br />
| style="height=20px; text-align:left;padding:3px"| 236 TiB<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Capacity Quotas<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 1 TiB per user <br> also per organization<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 40 TiB per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 5 TiB per user<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Inode Quotas<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 10 million per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 30 million per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 25 million per user<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Backup<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Read perf./node<br />
| style="height=20px; text-align:left;padding:3px"| 500 MB/s - 6 GB/s <br> depends on type of local SSD / job queue: <br> 520 MB/s @ single / multiple <br> 800 MB/s @ multiple_e <br> 6600 MB/s @ fat <br> 6500 MB/s @ gpu_4 <br> 6500 MB/s @ gpu_8<br />
| style="height=20px; text-align:left;padding:3px"| 400 MB/s - 500 MB/s<br> depends on type of local SSDs / job queue: <br> 500 MB/s @ multiple <br> 400 MB/s @ multiple_e<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Write perf./node<br />
| style="height=20px; text-align:left;padding:3px"| 500 MB/s - 4 GB/s <br> depends on type of local SSD / job queue: <br> 500 MB/s @ single / multiple <br> 650 MB/s @ multiple_e <br> 2900 MB/s @ fat <br> 2090 MB/s @ gpu_4 <br> 4060 MB/s @ gpu_8<br />
| style="height=20px; text-align:left;padding:3px"| 250 MB/s - 350 MB/s <br> depends on type of local SSDs / job queue: <br> 350 MB/s @ multiple <br> 250 MB/s @ multiple_e<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Total read perf.<br />
| style="height=20px; text-align:left;padding:3px"| n*500-6000 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| n*400-500 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| 18 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 54 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 45 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Total write perf.<br />
| style="height=20px; text-align:left;padding:3px"| n*500-4000 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| n*250-350 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| 18 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 54 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 38 GB/s<br />
|}<br />
---------------------------------------------------------------------------------------------------------<br />
global: all nodes of uc1 access the same file system;<br />
local: each node has its own file system;<br />
permanent: files are stored permanently;<br />
batch job: files are removed at end of the batch job.<br />
---------------------------------------------------------------------------------------------------------<br />
Table 2: Properties of the file systems<br />
<br />
== Selecting the appropriate file system ==<br />
<br />
In general, you should separate your data and store it on the appropriate file system.<br />
Permanently needed data like software or important results should be stored below $HOME<br />
but capacity restrictions (quotas) apply. In case you accidentally deleted data on $HOME<br />
you can usually restore it from backup. Permanent data which is not needed for months<br />
or exceeds the capacity restrictions should be sent to the LSDF Online Storage <br />
or to the archive and deleted from the file systems. Temporary data which is only needed on a single<br />
node and which does not exceed the disk space shown in the table above should be stored<br />
below $TMP. Data which is read many times on a single node, e.g. if you are doing AI training, <br />
should be copied to $TMP and read from there. Temporary data which is used from many nodes <br />
of your batch job and which is only needed during job runtime should be stored on a <br />
parallel on-demand file system. Temporary data which can be recomputed or which is the <br />
result of one job and input for another job should be stored in workspaces. The lifetime <br />
of data in workspaces is limited and depends on the lifetime of the workspace which can be <br />
several months.<br />
<br />
For further details please check the chapters below.<br />
<br />
== $HOME ==<br />
<br />
The home directories of bwUniCluster 2.0 (uc2) users are located in the parallel file system Lustre.<br />
You have access to your home directory from all nodes of uc2. A regular backup of these directories <br />
to tape archive is done automatically. The directory $HOME is used to hold those files that are<br />
permanently used like source codes, configuration files, executable programs etc. <br />
<br />
On uc2 there is a default user quota limit of 1 TiB and 10 million inodes (files and directories) per user. <br />
You can chek your current usage and limits with the command<br />
<pre><br />
$ lfs quota -uh $(whoami) $HOME<br />
</pre><br />
In addition to the user limit there is a limit of your organization (e.g. university) which depends on the financial share. This limit is enforced with so-called Lustre project quotas. You can show the current usage and limits of your organization with the following command:<br />
<pre><br />
lfs quota -ph $(grep $(echo $HOME | sed -e "s|/[^/]*/[^/]*$||") /pfs/data5/project_ids.txt | cut -f 1 -d\ ) $HOME<br />
</pre><br />
<br />
== Workspaces ==<br />
<br />
On uc2 workspaces can be used to store large non-permanent data sets, e.g. restart files or output<br />
data that has to be post-processed. The file system used for workspaces is also the parallel file system Lustre. This file system is especially designed for parallel access and for a high throughput to large<br />
files. It is able to provide high data transfer rates of up to 54 GB/s write and read performance when data access is parallel. <br />
<br />
Workspaces have a lifetime and the data on a workspace expires as a whole after a fixed period. The maximum lifetime of a workspace is 60 days, but it can be renewed at the end of that period 3 times to a total maximum of 240 days after workspace generation. If a workspace has inadvertently expired we can restore the data during a limited time (few weeks). In this case you should create a new workspace and report the name of the new and of the expired workspace in a ticket.<br />
<br />
Creating, deleting, finding and extending workspaces is explained on the [[workspace]] page.<br />
<br />
On uc2 there is a default user quota limit of 40 TiB and 30 million inodes (files and directories) per user. <br />
You can chek your current usage and limits with the command<br />
<pre><br />
$ lfs quota -uh $(whoami) /pfs/work7<br />
</pre><br />
Note that the quotas include data and inodes for all of your workspaces and all of your expired workspaces (as long as they are not yet completely removed).<br />
<br />
=== Reminder for workspace deletion ===<br />
<br />
Normally you will get an email every day starting 7 days before a workspace expires. You can send yourself a calender entry which reminds you when a workspace will be automatically deleted:<br />
<br />
$ ws_send_ical.sh <workspace> <email><br />
<br />
== Improving Performance on $HOME and workspaces ==<br />
<br />
The following recommendations might help to improve throughput and metadata<br />
performance on Lustre filesystems.<br />
<br />
=== '''Improving Throughput Performance''' ===<br />
<br />
Depending on your application some adaptations might be necessary if you want to reach<br />
the full bandwidth of the filesystems. Parallel filesystems typically stripe files over storage subsystems, i.e. large files are separated into stripes and distributed to different storage subsystems. In Lustre, the size of these stripes (sometimes also mentioned as chunks) is called stripe size and the number of used storage subsystems is called stripe count.<br />
<br />
When you are designing your application you should consider that the performance of<br />
parallel filesystems is generally better if data is transferred in large blocks and stored in<br />
few large files. In more detail, to increase throughput performance of a parallel application<br />
following aspects should be considered:<br />
<br />
* collect large chunks of data and write them sequentially at once,<br />
<br />
* to exploit complete filesystem bandwidth use several clients,<br />
<br />
* avoid competitive file access by different tasks or use blocks with boundaries at stripe size (default is 1MB),<br />
<br />
* if files are small enough for the SSDs and are only used by one process store them on $TMP.<br />
<br />
With previous Lustre versions adapting the Lustre stripe count was the most important optimization. However, for the workspaces of uc2 the new Lustre feature Progressive File Layouts has been used to define file striping parameters. This means that the stripe count is adapted if the file size is growing. In normal cases users no longer need to adapt file striping parameters in case they have very huge files or in order to reach better performance. <br />
<br />
If you know what you are doing you can still change striping parameters, e.g. the stripe count, of a directory and of newly created files. New files and directories inherit the stripe count from the parent directory. E.g. if you want to enhance throughput on a single very large file which is created in the directory $HOME/my_output_dir you can use the command<br />
<pre><br />
$ lfs setstripe -c-1 $HOME/my_output_dir<br />
</pre><br />
to change the stripe count to -1 which means that all storage subsystems of the file system are used to store that file. If you change the stripe count of a directory the stripe count of existing files inside this<br />
directory is not changed. If you want to change the stripe count of existing files, change<br />
the stripe count of the parent directory, copy the files to new files, remove the old files<br />
and move the new files back to the old name. In order to check the stripe setting of<br />
the file my_file use<br />
<pre><br />
$ lfs getstripe my_file<br />
</pre><br />
Also note that changes on the striping parameters (e.g. stripe count) are not saved in the<br />
backup, i.e. if directories have to be recreated this information is lost and the default stripe<br />
count will be used. Therefore, you should annotate for which directories you made changes<br />
to the striping parameters so that you can repeat these changes if required.<br />
<br />
=== '''Improving Metadata Performance''' ===<br />
<br />
Metadata performance on parallel file systems is usually not as good as with local<br />
filesystems. In addition, it is usually not scalable, i.e. a limited resource. Therefore,<br />
you should omit metadata operations whenever possible. For example, it is much better<br />
to have few large files than lots of small files. In more detail, to increase metadata<br />
performance of a parallel application following aspects should be considered:<br />
<br />
* avoid creating many small files,<br />
<br />
* avoid competitive directory access, e.g. by creating files in separate subdirectories for each task,<br />
<br />
* if many small files are only used within a batch job and accessed by one process store them on $TMP,<br />
<br />
* change the default colorization setting of the command ls (see below).<br />
<br />
On modern Linux systems, the GNU ls command often uses colorization by default to<br />
visually highlight the file type; this is especially true if the command is run within a terminal<br />
session. This is because the default shell profile initializations usually contain an alias<br />
directive similar to the following for the ls command:<br />
<pre><br />
$ alias ls="ls --color=tty"<br />
</pre><br />
However, running the ls command in this way for files on a Lustre file system requires<br />
a stat() call to be used to determine the file type. This can result in a performance<br />
overhead, because the stat() call always needs to determine the size of a file, and that<br />
in turn means that the client node must query the object size of all the backing objects<br />
that make up a file. As a result of the default colorization setting, running a simple<br />
ls command on a Lustre file system often takes as much time as running the ls command<br />
with the -l option (the same is true if the -F, -p, or the -classify option, or any other option <br />
that requires information from a stat() call, is used). To avoid this performance overhead<br />
when using ls commands, add an alias directive similar to the following<br />
to your shell startup script:<br />
<pre><br />
$ alias ls="ls --color=never"<br />
</pre><br />
<br />
== Workspaces on flash storage ==<br />
<br />
There is another workspace file system for special reqirements available. The file system is called ''full flash pfs'' and is based on the parallel file system Lustre. Access will be granted on special request.<br />
<br />
=== Advantages of this file system ===<br />
<br />
# All storage devices are based on flash (no hard disks) with low access times. Hence performance is better compared to other parallel file systems for read and write access with small blocks and with small files, i.e. IOPS rates are improved.<br />
# The file system is mounted on bwUniCluster 2.0 and HoreKa, i.e. it can be used to share data between these clusters.<br />
<br />
=== Access restrictions ===<br />
<br />
* Only HoreKa users or KIT users of bwUniCluster 2.0 can use this file system.<br />
* Access is granted on request and for appropriate requirements. In order to request access, please open a ticket with the subject ''full flash pfs access required'' and describe the following topics:<br />
** number of files you want to store on this file system<br />
** needed capacity<br />
** number of nodes used by your typical jobs<br />
** special I/O requirements of your jobs<br />
<br />
=== Using the file system ===<br />
<br />
After access is granted, you can use the file system in the same way as a normal workspace. You just have to specify the name of the flash-based workspace file system using the option ''-F'' to all the commands that manage workspaces. On bwUniCluster 2.0 it is called ''ffuc'', on HoreKa it is ''ffhk''. For example, to create a workspace with name myws and a lifetime of 60 days on bwUniCluster 2.0 execute:<br />
ws_allocate -F ffuc myws 60<br />
<br />
If you want to use the full flash pfs on bwUniCluster 2.0 '''and''' HoreKa at the same time, please note that you only have to manage a particular workspace on one of the clusters since the name of the workspace directory is different. However, the path to each workspace is visible and can be used on both clusters.<br />
<br />
Other features are similar to normal workspaces. For example, we are able to restore expired workspaces for few weeks and you have to open a ticket to request the restore. There are quota limits with a default limit of 5 TB capacity and 25 millions inodes per user. You can check your current usage with<br />
lfs quota -uh $(whoami) /pfs/work8<br />
<br />
== $TMP ==<br />
<br />
While all tasks of a parallel application access the same $HOME and workspace directory, the <br />
$TMP directory is local to each node on bwUniCluster 2.0. All nodes have fast SSDs <br />
local storage devices which are used to store data below $TMP. Different tasks of a parallel<br />
application use different $TMP directories when they do not utilize one node. This directory should<br />
be used for temporary files being accessed by single tasks. It should also be used if you read the <br />
same data many times from a single node, e.g. if you are doing AI training. In this case you should <br />
copy the data at the beginning of your batch job to $TMP and read the data from there.<br />
In addition, this directory should be used for the installation<br />
of software packages. This means that the software package to be installed should be<br />
unpacked, compiled and linked in a subdirectory of $TMP. The real installation of the<br />
package (e.g. make install) should be made in(to) the Lustre filesystem. <br />
<br />
Each time a batch job is started, a subdirectory is created on each node and assigned to the job. <br />
$TMP is newly set and the name of the subdirectory contains the Job-id so that the<br />
subdirectory name is unique for each job. This unique name is then assigned to the<br />
environment variable $TMP within the job. At the end of the job the subdirectory is removed.<br />
Although $TMP points to the same path name for different nodes of a job, the physical location <br />
on these nodes is different.<br />
<br />
== LSDF Online Storage==<br />
<br />
In some cases it is useful to have access to the LSDF Online Storage on the HPC-Clusters also. Therefore the LSDF Online Storage is mounted on the Login- and Datamover-Nodes.<br />
Furthermore it can be used on the compute nodes during the job runtime with the constraint flag "LSDF" ([[bwUniCluster_2.0_Slurm_common_Features|Slurm common features]]<br />
). There is also an example about the LSDF batch usage: [[bwUniCluster_2.0_Slurm_common_Features#LSDF_Online_Storage|Slurm LSDF example ]] .<br />
<pre><br />
#!/bin/bash<br />
#SBATCH ...<br />
#SBATCH --constraint=LSDF<br />
</pre><br />
<br><br />
For the usage of the LSDF Online Storage the following environment variables are available: $LSDF, $LSDFPROJECTS, $LSDFHOME.<br />
Please request storage projects in the LSDF Online Storage seperately:<br />
[https://www.lsdf.kit.edu/os/storagerequest/: LSDF Storage Request].<br />
<br />
==BeeOND (BeeGFS On-Demand)==<br />
<br />
Users of the HPC-Cluster have possibility to request a private BeeOND (BeeGFS) parallel filesystem for each job. The file system is created during job startup and purged after your job.<br />
<br />
'''IMPORTANT: All data on the private filesystem will be deleted after your job. Make sure you have copied your data back to the global filesystem (within job), e.g., $HOME or any workspace.'''<br />
<br />
BeeOND/BeeGFS can be used like any other parallel file system. Tools like cp or rsync can be used to copy data in and out. <br />
<br />
For detailed usage see here:[[BwUniCluster_2.0_Slurm_common_Features#BeeOND_.28BeeGFS_On-Demand.29|Request on-demand file system]]<br />
<br />
==Backup and Archiving==<br />
<br />
There are regular backups of all data of the home directories,whereas ACLs and extended attributes will<br />
not be backuped. <br />
<br />
Please open a ticket if you need backuped data.<br />
<br />
<br />
[[Category:bwUniCluster 2.0|File System]][[Category:Hardware_and_Architecture|bwUniCluster 2.0]]</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Hardware_and_Architecture&diff=11447
BwUniCluster2.0/Hardware and Architecture
2022-11-14T16:04:43Z
<p>P Weisbrod: /* Components of bwUniCluster */</p>
<hr />
<div>= Architecture of bwUniCluster 2.0 =<br />
<br />
The bwUniCluster 2.0 is a parallel computer with distributed memory. Each node of system consists of at least two Intel Xeon processor, local memory, disks, network adapters and optionally accelerators (NVIDIA Tesla V100). All nodes are connected by a fast InfiniBand interconnect. In addition the file system<br />
Lustre, that is connected by coupling the InfiniBand of the file server with the InfiniBand<br />
switch of the compute cluster, is added to bwUniCluster 2.0 to provide a fast and scalable<br />
parallel file system.<br />
<br />
The operating system on each node is Red Hat Enterprise Linux (RHEL) 8.4. A number of additional software packages like e.g. SLURM have been installed on top. Some of these components are of special interest to end users and are briefly<br />
discussed in this document. Others which are of greater importance to system<br />
administrators will not be covered by this document.<br />
<br />
The individual nodes of the system may act in different roles. According to the services supplied by the nodes, they are separated into disjoint groups. From an end users point of view the different groups of nodes are login nodes, compute nodes, file server nodes and administrative server nodes.<br />
<br />
'''Login Nodes'''<br />
<br />
The login nodes are the only nodes that are directly accessible by end users. These nodes<br />
are used for interactive login, file management, program development and interactive pre-<br />
and postprocessing. Two nodes are dedicated to this service but they are all accessible via<br />
one address and a DNS round-robin alias distributes the login sessions to the<br />
different login nodes.<br />
<br />
'''Compute Node'''<br />
<br />
The majority of nodes are compute nodes which are managed by a batch system. Users <br />
submit their jobs to the SLURM batch system and a job is executed when the required resources become available (depending on its fair-share priority).<br />
<br />
'''File Server Nodes'''<br />
<br />
The hardware of the parallel file system Lustre incorporates some file server nodes; the file<br />
system Lustre is connected by coupling the InfiniBand of the file server with the independent InfiniBand switch of the compute cluster. In addition to shared file space there is also local storage on the disks of each node (for details see chapter "File Systems").<br />
<br />
'''Administrative Server Nodes'''<br />
<br />
Some other nodes are delivering additional services like resource management, external<br />
network connection, administration etc. These nodes can be accessed directly by system administrators only.<br />
<br />
= Components of bwUniCluster =<br />
<br />
{| class="wikitable"<br />
|- <br />
! style="width:9%"|<br />
! style="width:13%"| Compute nodes "Thin"<br />
! style="width:13%"| Compute nodes "HPC"<br />
! style="width:13%"| Compute nodes "Fat"<br />
! style="width:13%"| GPU x4<br />
! style="width:13%"| GPU x8<br />
! style="width:13%"| Login<br />
|-<br />
!scope="column"| Number of nodes<br />
| 100 + 60<br />
| 360<br />
| 6<br />
| 14<br />
| 10<br />
| 4<br />
|-<br />
!scope="column"| Processors<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6248<br />
|-<br />
!scope="column"| Number of sockets<br />
| 2<br />
| 2<br />
| 4<br />
| 2<br />
| 2<br />
| 2<br />
|-<br />
!scope="column"| Processor frequency (GHz)<br />
| 2.1 Ghz<br />
| 2.1 Ghz<br />
| 2.1 Ghz<br />
| 2.1 Ghz<br />
| 2.5 Ghz<br />
| <br />
|-<br />
!scope="column"| Total number of cores<br />
| 40<br />
| 40<br />
| 80<br />
| 40<br />
| 40<br />
| 40 / 20 (Broadwell)<br />
|-<br />
!scope="column"| Main memory<br />
| 96 GB / 192 GB<br />
| 96 GB<br />
| 3 TB<br />
| 384 GB<br />
| 768 GB<br />
| 384 GB / 128 GB (Broadwell)<br />
|-<br />
!scope="column"| Local SSD<br />
| 960 GB SATA<br />
| 960 GB SATA<br />
| 4,8 TB NVMe<br />
| 3,2 TB NVMe<br />
| 6,4 TB NVMe<br />
| <br />
|-<br />
!scope="column"| Accelerators<br />
| -<br />
| -<br />
| -<br />
| -<br />
| 4x NVIDIA Tesla V100<br />
| 8x NVIDIA Tesla V100<br />
| <br />
|-<br />
!scope="column"| Interconnect<br />
| IB HDR100 (blocking)<br />
| IB HDR100<br />
| IB FDR<br />
| IB HDR<br />
| IB HDR<br />
| IB HDR<br />
| IB HDR100 (blocking)<br />
|}<br />
Table 1: Properties of the nodes<br />
<br />
= File Systems =<br />
<br />
On bwUniCluster 2.0 the parallel file system Lustre is used for most globally visible user data. Lustre is open source and Lustre solutions and support are available from different vendors. Nowadays, most of the biggest HPC systems are using Lustre. An initial home directory on a Lustre file system is created automatically after account activation, and the environment variable $HOME holds its name. Users can create so-called workspaces on another Lustre file system for non-permanent data with temporary lifetime. There is another workspace file system based on flash storage for special requirements available.<br />
<br />
Within a batch job further file systems are available: <br />
* The directory $TMP is only available and visible on the local node. It is located on fast SSD storage devices.<br />
* On request a parallel on-demand (BeeOND) file system is created which uses the SSDs of the nodes which were allocated to the batch job.<br />
* On request the external LSDF Online Storage is mounted on the nodes which were allocated to the batch job. This file system is based on the parallel file system Spectrum Scale. <br />
<br />
Some of the characteristics of the file systems are shown in Table 2.<br />
<br />
{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"<br />
|- style="width:20%;height=20px; text-align:left;padding:3px"<br />
! style="background-color:#AAA;padding:3px"| Property<br />
! style="background-color:#AAA;padding:3px"| $TMP<br />
! style="background-color:#AAA;padding:3px"| BeeOND<br />
! style="background-color:#AAA;padding:3px"| $HOME<br />
! style="background-color:#AAA;padding:3px"| Workspace<br />
! style="background-color:#AAA;padding:3px"| Workspace <br> on flash<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Visibility <br />
| style="height=20px; text-align:left;padding:3px"| local node<br />
| style="height=20px; text-align:left;padding:3px"| nodes of batch job<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Lifetime <br />
| style="height=20px; text-align:left;padding:3px"| batch job runtime<br />
| style="height=20px; text-align:left;padding:3px"| batch job runtime<br />
| style="height=20px; text-align:left;padding:3px"| permanent<br />
| style="height=20px; text-align:left;padding:3px"| max. 240 days<br />
| style="height=20px; text-align:left;padding:3px"| max. 240 days<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Disk space <br />
| style="height=20px; text-align:left;padding:3px"| 960 GB - 6.4 TB <br> details see table 1<br />
| style="height=20px; text-align:left;padding:3px"| n*250 GB<br />
| style="height=20px; text-align:left;padding:3px"| 1.2 PiB<br />
| style="height=20px; text-align:left;padding:3px"| 4.1 PiB<br />
| style="height=20px; text-align:left;padding:3px"| 236 TiB<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Capacity Quotas<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 1 TiB per user <br> also per organization<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 40 TiB per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 5 TiB per user<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Inode Quotas<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 10 million per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 30 million per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 25 million per user<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Backup<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Read perf./node<br />
| style="height=20px; text-align:left;padding:3px"| 500 MB/s - 6 GB/s <br> depends on type of local SSD / job queue: <br> 520 MB/s @ single / multiple <br> 800 MB/s @ multiple_e <br> 6600 MB/s @ fat <br> 6500 MB/s @ gpu_4 <br> 6500 MB/s @ gpu_8<br />
| style="height=20px; text-align:left;padding:3px"| 400 MB/s - 500 MB/s<br> depends on type of local SSDs / job queue: <br> 500 MB/s @ multiple <br> 400 MB/s @ multiple_e<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Write perf./node<br />
| style="height=20px; text-align:left;padding:3px"| 500 MB/s - 4 GB/s <br> depends on type of local SSD / job queue: <br> 500 MB/s @ single / multiple <br> 650 MB/s @ multiple_e <br> 2900 MB/s @ fat <br> 2090 MB/s @ gpu_4 <br> 4060 MB/s @ gpu_8<br />
| style="height=20px; text-align:left;padding:3px"| 250 MB/s - 350 MB/s <br> depends on type of local SSDs / job queue: <br> 350 MB/s @ multiple <br> 250 MB/s @ multiple_e<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Total read perf.<br />
| style="height=20px; text-align:left;padding:3px"| n*500-6000 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| n*400-500 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| 18 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 54 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 45 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Total write perf.<br />
| style="height=20px; text-align:left;padding:3px"| n*500-4000 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| n*250-350 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| 18 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 54 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 38 GB/s<br />
|}<br />
---------------------------------------------------------------------------------------------------------<br />
global: all nodes of uc1 access the same file system;<br />
local: each node has its own file system;<br />
permanent: files are stored permanently;<br />
batch job: files are removed at end of the batch job.<br />
---------------------------------------------------------------------------------------------------------<br />
Table 2: Properties of the file systems<br />
<br />
== Selecting the appropriate file system ==<br />
<br />
In general, you should separate your data and store it on the appropriate file system.<br />
Permanently needed data like software or important results should be stored below $HOME<br />
but capacity restrictions (quotas) apply. In case you accidentally deleted data on $HOME<br />
you can usually restore it from backup. Permanent data which is not needed for months<br />
or exceeds the capacity restrictions should be sent to the LSDF Online Storage <br />
or to the archive and deleted from the file systems. Temporary data which is only needed on a single<br />
node and which does not exceed the disk space shown in the table above should be stored<br />
below $TMP. Data which is read many times on a single node, e.g. if you are doing AI training, <br />
should be copied to $TMP and read from there. Temporary data which is used from many nodes <br />
of your batch job and which is only needed during job runtime should be stored on a <br />
parallel on-demand file system. Temporary data which can be recomputed or which is the <br />
result of one job and input for another job should be stored in workspaces. The lifetime <br />
of data in workspaces is limited and depends on the lifetime of the workspace which can be <br />
several months.<br />
<br />
For further details please check the chapters below.<br />
<br />
== $HOME ==<br />
<br />
The home directories of bwUniCluster 2.0 (uc2) users are located in the parallel file system Lustre.<br />
You have access to your home directory from all nodes of uc2. A regular backup of these directories <br />
to tape archive is done automatically. The directory $HOME is used to hold those files that are<br />
permanently used like source codes, configuration files, executable programs etc. <br />
<br />
On uc2 there is a default user quota limit of 1 TiB and 10 million inodes (files and directories) per user. <br />
You can chek your current usage and limits with the command<br />
<pre><br />
$ lfs quota -uh $(whoami) $HOME<br />
</pre><br />
In addition to the user limit there is a limit of your organization (e.g. university) which depends on the financial share. This limit is enforced with so-called Lustre project quotas. You can show the current usage and limits of your organization with the following command:<br />
<pre><br />
lfs quota -ph $(grep $(echo $HOME | sed -e "s|/[^/]*/[^/]*$||") /pfs/data5/project_ids.txt | cut -f 1 -d\ ) $HOME<br />
</pre><br />
<br />
== Workspaces ==<br />
<br />
On uc2 workspaces can be used to store large non-permanent data sets, e.g. restart files or output<br />
data that has to be post-processed. The file system used for workspaces is also the parallel file system Lustre. This file system is especially designed for parallel access and for a high throughput to large<br />
files. It is able to provide high data transfer rates of up to 54 GB/s write and read performance when data access is parallel. <br />
<br />
Workspaces have a lifetime and the data on a workspace expires as a whole after a fixed period. The maximum lifetime of a workspace is 60 days, but it can be renewed at the end of that period 3 times to a total maximum of 240 days after workspace generation. If a workspace has inadvertently expired we can restore the data during a limited time (few weeks). In this case you should create a new workspace and report the name of the new and of the expired workspace in a ticket.<br />
<br />
Creating, deleting, finding and extending workspaces is explained on the [[workspace]] page.<br />
<br />
On uc2 there is a default user quota limit of 40 TiB and 30 million inodes (files and directories) per user. <br />
You can chek your current usage and limits with the command<br />
<pre><br />
$ lfs quota -uh $(whoami) /pfs/work7<br />
</pre><br />
Note that the quotas include data and inodes for all of your workspaces and all of your expired workspaces (as long as they are not yet completely removed).<br />
<br />
=== Reminder for workspace deletion ===<br />
<br />
Normally you will get an email every day starting 7 days before a workspace expires. You can send yourself a calender entry which reminds you when a workspace will be automatically deleted:<br />
<br />
$ ws_send_ical.sh <workspace> <email><br />
<br />
== Improving Performance on $HOME and workspaces ==<br />
<br />
The following recommendations might help to improve throughput and metadata<br />
performance on Lustre filesystems.<br />
<br />
=== '''Improving Throughput Performance''' ===<br />
<br />
Depending on your application some adaptations might be necessary if you want to reach<br />
the full bandwidth of the filesystems. Parallel filesystems typically stripe files over storage subsystems, i.e. large files are separated into stripes and distributed to different storage subsystems. In Lustre, the size of these stripes (sometimes also mentioned as chunks) is called stripe size and the number of used storage subsystems is called stripe count.<br />
<br />
When you are designing your application you should consider that the performance of<br />
parallel filesystems is generally better if data is transferred in large blocks and stored in<br />
few large files. In more detail, to increase throughput performance of a parallel application<br />
following aspects should be considered:<br />
<br />
* collect large chunks of data and write them sequentially at once,<br />
<br />
* to exploit complete filesystem bandwidth use several clients,<br />
<br />
* avoid competitive file access by different tasks or use blocks with boundaries at stripe size (default is 1MB),<br />
<br />
* if files are small enough for the SSDs and are only used by one process store them on $TMP.<br />
<br />
With previous Lustre versions adapting the Lustre stripe count was the most important optimization. However, for the workspaces of uc2 the new Lustre feature Progressive File Layouts has been used to define file striping parameters. This means that the stripe count is adapted if the file size is growing. In normal cases users no longer need to adapt file striping parameters in case they have very huge files or in order to reach better performance. <br />
<br />
If you know what you are doing you can still change striping parameters, e.g. the stripe count, of a directory and of newly created files. New files and directories inherit the stripe count from the parent directory. E.g. if you want to enhance throughput on a single very large file which is created in the directory $HOME/my_output_dir you can use the command<br />
<pre><br />
$ lfs setstripe -c-1 $HOME/my_output_dir<br />
</pre><br />
to change the stripe count to -1 which means that all storage subsystems of the file system are used to store that file. If you change the stripe count of a directory the stripe count of existing files inside this<br />
directory is not changed. If you want to change the stripe count of existing files, change<br />
the stripe count of the parent directory, copy the files to new files, remove the old files<br />
and move the new files back to the old name. In order to check the stripe setting of<br />
the file my_file use<br />
<pre><br />
$ lfs getstripe my_file<br />
</pre><br />
Also note that changes on the striping parameters (e.g. stripe count) are not saved in the<br />
backup, i.e. if directories have to be recreated this information is lost and the default stripe<br />
count will be used. Therefore, you should annotate for which directories you made changes<br />
to the striping parameters so that you can repeat these changes if required.<br />
<br />
=== '''Improving Metadata Performance''' ===<br />
<br />
Metadata performance on parallel file systems is usually not as good as with local<br />
filesystems. In addition, it is usually not scalable, i.e. a limited resource. Therefore,<br />
you should omit metadata operations whenever possible. For example, it is much better<br />
to have few large files than lots of small files. In more detail, to increase metadata<br />
performance of a parallel application following aspects should be considered:<br />
<br />
* avoid creating many small files,<br />
<br />
* avoid competitive directory access, e.g. by creating files in separate subdirectories for each task,<br />
<br />
* if many small files are only used within a batch job and accessed by one process store them on $TMP,<br />
<br />
* change the default colorization setting of the command ls (see below).<br />
<br />
On modern Linux systems, the GNU ls command often uses colorization by default to<br />
visually highlight the file type; this is especially true if the command is run within a terminal<br />
session. This is because the default shell profile initializations usually contain an alias<br />
directive similar to the following for the ls command:<br />
<pre><br />
$ alias ls="ls --color=tty"<br />
</pre><br />
However, running the ls command in this way for files on a Lustre file system requires<br />
a stat() call to be used to determine the file type. This can result in a performance<br />
overhead, because the stat() call always needs to determine the size of a file, and that<br />
in turn means that the client node must query the object size of all the backing objects<br />
that make up a file. As a result of the default colorization setting, running a simple<br />
ls command on a Lustre file system often takes as much time as running the ls command<br />
with the -l option (the same is true if the -F, -p, or the -classify option, or any other option <br />
that requires information from a stat() call, is used). To avoid this performance overhead<br />
when using ls commands, add an alias directive similar to the following<br />
to your shell startup script:<br />
<pre><br />
$ alias ls="ls --color=never"<br />
</pre><br />
<br />
== Workspaces on flash storage ==<br />
<br />
There is another workspace file system for special reqirements available. The file system is called ''full flash pfs'' and is based on the parallel file system Lustre. Access will be granted on special request.<br />
<br />
=== Advantages of this file system ===<br />
<br />
# All storage devices are based on flash (no hard disks) with low access times. Hence performance is better compared to other parallel file systems for read and write access with small blocks and with small files, i.e. IOPS rates are improved.<br />
# The file system is mounted on bwUniCluster 2.0 and HoreKa, i.e. it can be used to share data between these clusters.<br />
<br />
=== Access restrictions ===<br />
<br />
* Only HoreKa users or KIT users of bwUniCluster 2.0 can use this file system.<br />
* Access is granted on request and for appropriate requirements. In order to request access, please open a ticket with the subject ''full flash pfs access required'' and describe the following topics:<br />
** number of files you want to store on this file system<br />
** needed capacity<br />
** number of nodes used by your typical jobs<br />
** special I/O requirements of your jobs<br />
<br />
=== Using the file system ===<br />
<br />
After access is granted, you can use the file system in the same way as a normal workspace. You just have to specify the name of the flash-based workspace file system using the option ''-F'' to all the commands that manage workspaces. On bwUniCluster 2.0 it is called ''ffuc'', on HoreKa it is ''ffhk''. For example, to create a workspace with name myws and a lifetime of 60 days on bwUniCluster 2.0 execute:<br />
ws_allocate -F ffuc myws 60<br />
<br />
If you want to use the full flash pfs on bwUniCluster 2.0 '''and''' HoreKa at the same time, please note that you only have to manage a particular workspace on one of the clusters since the name of the workspace directory is different. However, the path to each workspace is visible and can be used on both clusters.<br />
<br />
Other features are similar to normal workspaces. For example, we are able to restore expired workspaces for few weeks and you have to open a ticket to request the restore. There are quota limits with a default limit of 5 TB capacity and 25 millions inodes per user. You can check your current usage with<br />
lfs quota -uh $(whoami) /pfs/work8<br />
<br />
== $TMP ==<br />
<br />
While all tasks of a parallel application access the same $HOME and workspace directory, the <br />
$TMP directory is local to each node on bwUniCluster 2.0. All nodes have fast SSDs <br />
local storage devices which are used to store data below $TMP. Different tasks of a parallel<br />
application use different $TMP directories when they do not utilize one node. This directory should<br />
be used for temporary files being accessed by single tasks. It should also be used if you read the <br />
same data many times from a single node, e.g. if you are doing AI training. In this case you should <br />
copy the data at the beginning of your batch job to $TMP and read the data from there.<br />
In addition, this directory should be used for the installation<br />
of software packages. This means that the software package to be installed should be<br />
unpacked, compiled and linked in a subdirectory of $TMP. The real installation of the<br />
package (e.g. make install) should be made in(to) the Lustre filesystem. <br />
<br />
Each time a batch job is started, a subdirectory is created on each node and assigned to the job. <br />
$TMP is newly set and the name of the subdirectory contains the Job-id so that the<br />
subdirectory name is unique for each job. This unique name is then assigned to the<br />
environment variable $TMP within the job. At the end of the job the subdirectory is removed.<br />
Although $TMP points to the same path name for different nodes of a job, the physical location <br />
on these nodes is different.<br />
<br />
== LSDF Online Storage==<br />
<br />
In some cases it is useful to have access to the LSDF Online Storage on the HPC-Clusters also. Therefore the LSDF Online Storage is mounted on the Login- and Datamover-Nodes.<br />
Furthermore it can be used on the compute nodes during the job runtime with the constraint flag "LSDF" ([[bwUniCluster_2.0_Slurm_common_Features|Slurm common features]]<br />
). There is also an example about the LSDF batch usage: [[bwUniCluster_2.0_Slurm_common_Features#LSDF_Online_Storage|Slurm LSDF example ]] .<br />
<pre><br />
#!/bin/bash<br />
#SBATCH ...<br />
#SBATCH --constraint=LSDF<br />
</pre><br />
<br><br />
For the usage of the LSDF Online Storage the following environment variables are available: $LSDF, $LSDFPROJECTS, $LSDFHOME.<br />
Please request storage projects in the LSDF Online Storage seperately:<br />
[https://www.lsdf.kit.edu/os/storagerequest/: LSDF Storage Request].<br />
<br />
==BeeOND (BeeGFS On-Demand)==<br />
<br />
Users of the HPC-Cluster have possibility to request a private BeeOND (BeeGFS) parallel filesystem for each job. The file system is created during job startup and purged after your job.<br />
<br />
'''IMPORTANT: All data on the private filesystem will be deleted after your job. Make sure you have copied your data back to the global filesystem (within job), e.g., $HOME or any workspace.'''<br />
<br />
BeeOND/BeeGFS can be used like any other parallel file system. Tools like cp or rsync can be used to copy data in and out. <br />
<br />
For detailed usage see here:[[BwUniCluster_2.0_Slurm_common_Features#BeeOND_.28BeeGFS_On-Demand.29|Request on-demand file system]]<br />
<br />
==Backup and Archiving==<br />
<br />
There are regular backups of all data of the home directories,whereas ACLs and extended attributes will<br />
not be backuped. <br />
<br />
Please open a ticket if you need backuped data.<br />
<br />
<br />
[[Category:bwUniCluster 2.0|File System]][[Category:Hardware_and_Architecture|bwUniCluster 2.0]]</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Hardware_and_Architecture&diff=11446
BwUniCluster2.0/Hardware and Architecture
2022-11-14T16:03:24Z
<p>P Weisbrod: </p>
<hr />
<div>= Architecture of bwUniCluster 2.0 =<br />
<br />
The bwUniCluster 2.0 is a parallel computer with distributed memory. Each node of system consists of at least two Intel Xeon processor, local memory, disks, network adapters and optionally accelerators (NVIDIA Tesla V100). All nodes are connected by a fast InfiniBand interconnect. In addition the file system<br />
Lustre, that is connected by coupling the InfiniBand of the file server with the InfiniBand<br />
switch of the compute cluster, is added to bwUniCluster 2.0 to provide a fast and scalable<br />
parallel file system.<br />
<br />
The operating system on each node is Red Hat Enterprise Linux (RHEL) 8.4. A number of additional software packages like e.g. SLURM have been installed on top. Some of these components are of special interest to end users and are briefly<br />
discussed in this document. Others which are of greater importance to system<br />
administrators will not be covered by this document.<br />
<br />
The individual nodes of the system may act in different roles. According to the services supplied by the nodes, they are separated into disjoint groups. From an end users point of view the different groups of nodes are login nodes, compute nodes, file server nodes and administrative server nodes.<br />
<br />
'''Login Nodes'''<br />
<br />
The login nodes are the only nodes that are directly accessible by end users. These nodes<br />
are used for interactive login, file management, program development and interactive pre-<br />
and postprocessing. Two nodes are dedicated to this service but they are all accessible via<br />
one address and a DNS round-robin alias distributes the login sessions to the<br />
different login nodes.<br />
<br />
'''Compute Node'''<br />
<br />
The majority of nodes are compute nodes which are managed by a batch system. Users <br />
submit their jobs to the SLURM batch system and a job is executed when the required resources become available (depending on its fair-share priority).<br />
<br />
'''File Server Nodes'''<br />
<br />
The hardware of the parallel file system Lustre incorporates some file server nodes; the file<br />
system Lustre is connected by coupling the InfiniBand of the file server with the independent InfiniBand switch of the compute cluster. In addition to shared file space there is also local storage on the disks of each node (for details see chapter "File Systems").<br />
<br />
'''Administrative Server Nodes'''<br />
<br />
Some other nodes are delivering additional services like resource management, external<br />
network connection, administration etc. These nodes can be accessed directly by system administrators only.<br />
<br />
= Components of bwUniCluster =<br />
<br />
{| class="wikitable"<br />
|- <br />
! style="width:9%"|<br />
! style="width:13%"| Compute nodes "Thin"<br />
! style="width:13%"| Compute nodes "HPC"<br />
! style="width:13%"| Compute nodes "HPC Broadwell"<br />
! style="width:13%"| Compute nodes "Fat"<br />
! style="width:13%"| GPU x4<br />
! style="width:13%"| GPU x8<br />
! style="width:13%"| Login<br />
|-<br />
!scope="column"| Number of nodes<br />
| 100 + 60<br />
| 360<br />
| 352<br />
| 6<br />
| 14<br />
| 10<br />
| 4<br />
|-<br />
!scope="column"| Processors<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon E5-2660 v4<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6230<br />
| Intel Xeon Gold 6248<br />
|-<br />
!scope="column"| Number of sockets<br />
| 2<br />
| 2<br />
| 2<br />
| 4<br />
| 2<br />
| 2<br />
| 2<br />
|-<br />
!scope="column"| Processor frequency (GHz)<br />
| 2.1 Ghz<br />
| 2.1 Ghz<br />
| 2.0 GHz<br />
| 2.1 Ghz<br />
| 2.1 Ghz<br />
| 2.5 Ghz<br />
| <br />
|-<br />
!scope="column"| Total number of cores<br />
| 40<br />
| 40<br />
| 28<br />
| 80<br />
| 40<br />
| 40<br />
| 40 / 20 (Broadwell)<br />
|-<br />
!scope="column"| Main memory<br />
| 96 GB / 192 GB<br />
| 96 GB<br />
| 128 GB<br />
| 3 TB<br />
| 384 GB<br />
| 768 GB<br />
| 384 GB / 128 GB (Broadwell)<br />
|-<br />
!scope="column"| Local SSD<br />
| 960 GB SATA<br />
| 960 GB SATA<br />
| 480 GB SATA<br />
| 4,8 TB NVMe<br />
| 3,2 TB NVMe<br />
| 6,4 TB NVMe<br />
| <br />
|-<br />
!scope="column"| Accelerators<br />
| -<br />
| -<br />
| -<br />
| -<br />
| 4x NVIDIA Tesla V100<br />
| 8x NVIDIA Tesla V100<br />
| <br />
|-<br />
!scope="column"| Interconnect<br />
| IB HDR100 (blocking)<br />
| IB HDR100<br />
| IB FDR<br />
| IB HDR<br />
| IB HDR<br />
| IB HDR<br />
| IB HDR100 (blocking)<br />
|}<br />
Table 1: Properties of the nodes<br />
<br />
= File Systems =<br />
<br />
On bwUniCluster 2.0 the parallel file system Lustre is used for most globally visible user data. Lustre is open source and Lustre solutions and support are available from different vendors. Nowadays, most of the biggest HPC systems are using Lustre. An initial home directory on a Lustre file system is created automatically after account activation, and the environment variable $HOME holds its name. Users can create so-called workspaces on another Lustre file system for non-permanent data with temporary lifetime. There is another workspace file system based on flash storage for special requirements available.<br />
<br />
Within a batch job further file systems are available: <br />
* The directory $TMP is only available and visible on the local node. It is located on fast SSD storage devices.<br />
* On request a parallel on-demand (BeeOND) file system is created which uses the SSDs of the nodes which were allocated to the batch job.<br />
* On request the external LSDF Online Storage is mounted on the nodes which were allocated to the batch job. This file system is based on the parallel file system Spectrum Scale. <br />
<br />
Some of the characteristics of the file systems are shown in Table 2.<br />
<br />
{| style="width:100%; vertical-align:top; background:#f5fffa;border:1px solid #000000;padding:1px"<br />
|- style="width:20%;height=20px; text-align:left;padding:3px"<br />
! style="background-color:#AAA;padding:3px"| Property<br />
! style="background-color:#AAA;padding:3px"| $TMP<br />
! style="background-color:#AAA;padding:3px"| BeeOND<br />
! style="background-color:#AAA;padding:3px"| $HOME<br />
! style="background-color:#AAA;padding:3px"| Workspace<br />
! style="background-color:#AAA;padding:3px"| Workspace <br> on flash<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Visibility <br />
| style="height=20px; text-align:left;padding:3px"| local node<br />
| style="height=20px; text-align:left;padding:3px"| nodes of batch job<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
| style="height=20px; text-align:left;padding:3px"| global<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Lifetime <br />
| style="height=20px; text-align:left;padding:3px"| batch job runtime<br />
| style="height=20px; text-align:left;padding:3px"| batch job runtime<br />
| style="height=20px; text-align:left;padding:3px"| permanent<br />
| style="height=20px; text-align:left;padding:3px"| max. 240 days<br />
| style="height=20px; text-align:left;padding:3px"| max. 240 days<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Disk space <br />
| style="height=20px; text-align:left;padding:3px"| 960 GB - 6.4 TB <br> details see table 1<br />
| style="height=20px; text-align:left;padding:3px"| n*250 GB<br />
| style="height=20px; text-align:left;padding:3px"| 1.2 PiB<br />
| style="height=20px; text-align:left;padding:3px"| 4.1 PiB<br />
| style="height=20px; text-align:left;padding:3px"| 236 TiB<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Capacity Quotas<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 1 TiB per user <br> also per organization<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 40 TiB per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 5 TiB per user<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Inode Quotas<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 10 million per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 30 million per user<br />
| style="height=20px; text-align:left;padding:3px"| yes <br> 25 million per user<br />
|- <br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Backup<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| yes<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
| style="height=20px; text-align:left;padding:3px"| no<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Read perf./node<br />
| style="height=20px; text-align:left;padding:3px"| 500 MB/s - 6 GB/s <br> depends on type of local SSD / job queue: <br> 520 MB/s @ single / multiple <br> 800 MB/s @ multiple_e <br> 6600 MB/s @ fat <br> 6500 MB/s @ gpu_4 <br> 6500 MB/s @ gpu_8<br />
| style="height=20px; text-align:left;padding:3px"| 400 MB/s - 500 MB/s<br> depends on type of local SSDs / job queue: <br> 500 MB/s @ multiple <br> 400 MB/s @ multiple_e<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Write perf./node<br />
| style="height=20px; text-align:left;padding:3px"| 500 MB/s - 4 GB/s <br> depends on type of local SSD / job queue: <br> 500 MB/s @ single / multiple <br> 650 MB/s @ multiple_e <br> 2900 MB/s @ fat <br> 2090 MB/s @ gpu_4 <br> 4060 MB/s @ gpu_8<br />
| style="height=20px; text-align:left;padding:3px"| 250 MB/s - 350 MB/s <br> depends on type of local SSDs / job queue: <br> 350 MB/s @ multiple <br> 250 MB/s @ multiple_e<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 1 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Total read perf.<br />
| style="height=20px; text-align:left;padding:3px"| n*500-6000 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| n*400-500 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| 18 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 54 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 45 GB/s<br />
|- style="vertical-align:top;"<br />
| style="background-color:#d3ddd8;height=20px; text-align:left;padding:3px"| Total write perf.<br />
| style="height=20px; text-align:left;padding:3px"| n*500-4000 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| n*250-350 MB/s<br />
| style="height=20px; text-align:left;padding:3px"| 18 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 54 GB/s<br />
| style="height=20px; text-align:left;padding:3px"| 38 GB/s<br />
|}<br />
---------------------------------------------------------------------------------------------------------<br />
global: all nodes of uc1 access the same file system;<br />
local: each node has its own file system;<br />
permanent: files are stored permanently;<br />
batch job: files are removed at end of the batch job.<br />
---------------------------------------------------------------------------------------------------------<br />
Table 2: Properties of the file systems<br />
<br />
== Selecting the appropriate file system ==<br />
<br />
In general, you should separate your data and store it on the appropriate file system.<br />
Permanently needed data like software or important results should be stored below $HOME<br />
but capacity restrictions (quotas) apply. In case you accidentally deleted data on $HOME<br />
you can usually restore it from backup. Permanent data which is not needed for months<br />
or exceeds the capacity restrictions should be sent to the LSDF Online Storage <br />
or to the archive and deleted from the file systems. Temporary data which is only needed on a single<br />
node and which does not exceed the disk space shown in the table above should be stored<br />
below $TMP. Data which is read many times on a single node, e.g. if you are doing AI training, <br />
should be copied to $TMP and read from there. Temporary data which is used from many nodes <br />
of your batch job and which is only needed during job runtime should be stored on a <br />
parallel on-demand file system. Temporary data which can be recomputed or which is the <br />
result of one job and input for another job should be stored in workspaces. The lifetime <br />
of data in workspaces is limited and depends on the lifetime of the workspace which can be <br />
several months.<br />
<br />
For further details please check the chapters below.<br />
<br />
== $HOME ==<br />
<br />
The home directories of bwUniCluster 2.0 (uc2) users are located in the parallel file system Lustre.<br />
You have access to your home directory from all nodes of uc2. A regular backup of these directories <br />
to tape archive is done automatically. The directory $HOME is used to hold those files that are<br />
permanently used like source codes, configuration files, executable programs etc. <br />
<br />
On uc2 there is a default user quota limit of 1 TiB and 10 million inodes (files and directories) per user. <br />
You can chek your current usage and limits with the command<br />
<pre><br />
$ lfs quota -uh $(whoami) $HOME<br />
</pre><br />
In addition to the user limit there is a limit of your organization (e.g. university) which depends on the financial share. This limit is enforced with so-called Lustre project quotas. You can show the current usage and limits of your organization with the following command:<br />
<pre><br />
lfs quota -ph $(grep $(echo $HOME | sed -e "s|/[^/]*/[^/]*$||") /pfs/data5/project_ids.txt | cut -f 1 -d\ ) $HOME<br />
</pre><br />
<br />
== Workspaces ==<br />
<br />
On uc2 workspaces can be used to store large non-permanent data sets, e.g. restart files or output<br />
data that has to be post-processed. The file system used for workspaces is also the parallel file system Lustre. This file system is especially designed for parallel access and for a high throughput to large<br />
files. It is able to provide high data transfer rates of up to 54 GB/s write and read performance when data access is parallel. <br />
<br />
Workspaces have a lifetime and the data on a workspace expires as a whole after a fixed period. The maximum lifetime of a workspace is 60 days, but it can be renewed at the end of that period 3 times to a total maximum of 240 days after workspace generation. If a workspace has inadvertently expired we can restore the data during a limited time (few weeks). In this case you should create a new workspace and report the name of the new and of the expired workspace in a ticket.<br />
<br />
Creating, deleting, finding and extending workspaces is explained on the [[workspace]] page.<br />
<br />
On uc2 there is a default user quota limit of 40 TiB and 30 million inodes (files and directories) per user. <br />
You can chek your current usage and limits with the command<br />
<pre><br />
$ lfs quota -uh $(whoami) /pfs/work7<br />
</pre><br />
Note that the quotas include data and inodes for all of your workspaces and all of your expired workspaces (as long as they are not yet completely removed).<br />
<br />
=== Reminder for workspace deletion ===<br />
<br />
Normally you will get an email every day starting 7 days before a workspace expires. You can send yourself a calender entry which reminds you when a workspace will be automatically deleted:<br />
<br />
$ ws_send_ical.sh <workspace> <email><br />
<br />
== Improving Performance on $HOME and workspaces ==<br />
<br />
The following recommendations might help to improve throughput and metadata<br />
performance on Lustre filesystems.<br />
<br />
=== '''Improving Throughput Performance''' ===<br />
<br />
Depending on your application some adaptations might be necessary if you want to reach<br />
the full bandwidth of the filesystems. Parallel filesystems typically stripe files over storage subsystems, i.e. large files are separated into stripes and distributed to different storage subsystems. In Lustre, the size of these stripes (sometimes also mentioned as chunks) is called stripe size and the number of used storage subsystems is called stripe count.<br />
<br />
When you are designing your application you should consider that the performance of<br />
parallel filesystems is generally better if data is transferred in large blocks and stored in<br />
few large files. In more detail, to increase throughput performance of a parallel application<br />
following aspects should be considered:<br />
<br />
* collect large chunks of data and write them sequentially at once,<br />
<br />
* to exploit complete filesystem bandwidth use several clients,<br />
<br />
* avoid competitive file access by different tasks or use blocks with boundaries at stripe size (default is 1MB),<br />
<br />
* if files are small enough for the SSDs and are only used by one process store them on $TMP.<br />
<br />
With previous Lustre versions adapting the Lustre stripe count was the most important optimization. However, for the workspaces of uc2 the new Lustre feature Progressive File Layouts has been used to define file striping parameters. This means that the stripe count is adapted if the file size is growing. In normal cases users no longer need to adapt file striping parameters in case they have very huge files or in order to reach better performance. <br />
<br />
If you know what you are doing you can still change striping parameters, e.g. the stripe count, of a directory and of newly created files. New files and directories inherit the stripe count from the parent directory. E.g. if you want to enhance throughput on a single very large file which is created in the directory $HOME/my_output_dir you can use the command<br />
<pre><br />
$ lfs setstripe -c-1 $HOME/my_output_dir<br />
</pre><br />
to change the stripe count to -1 which means that all storage subsystems of the file system are used to store that file. If you change the stripe count of a directory the stripe count of existing files inside this<br />
directory is not changed. If you want to change the stripe count of existing files, change<br />
the stripe count of the parent directory, copy the files to new files, remove the old files<br />
and move the new files back to the old name. In order to check the stripe setting of<br />
the file my_file use<br />
<pre><br />
$ lfs getstripe my_file<br />
</pre><br />
Also note that changes on the striping parameters (e.g. stripe count) are not saved in the<br />
backup, i.e. if directories have to be recreated this information is lost and the default stripe<br />
count will be used. Therefore, you should annotate for which directories you made changes<br />
to the striping parameters so that you can repeat these changes if required.<br />
<br />
=== '''Improving Metadata Performance''' ===<br />
<br />
Metadata performance on parallel file systems is usually not as good as with local<br />
filesystems. In addition, it is usually not scalable, i.e. a limited resource. Therefore,<br />
you should omit metadata operations whenever possible. For example, it is much better<br />
to have few large files than lots of small files. In more detail, to increase metadata<br />
performance of a parallel application following aspects should be considered:<br />
<br />
* avoid creating many small files,<br />
<br />
* avoid competitive directory access, e.g. by creating files in separate subdirectories for each task,<br />
<br />
* if many small files are only used within a batch job and accessed by one process store them on $TMP,<br />
<br />
* change the default colorization setting of the command ls (see below).<br />
<br />
On modern Linux systems, the GNU ls command often uses colorization by default to<br />
visually highlight the file type; this is especially true if the command is run within a terminal<br />
session. This is because the default shell profile initializations usually contain an alias<br />
directive similar to the following for the ls command:<br />
<pre><br />
$ alias ls="ls --color=tty"<br />
</pre><br />
However, running the ls command in this way for files on a Lustre file system requires<br />
a stat() call to be used to determine the file type. This can result in a performance<br />
overhead, because the stat() call always needs to determine the size of a file, and that<br />
in turn means that the client node must query the object size of all the backing objects<br />
that make up a file. As a result of the default colorization setting, running a simple<br />
ls command on a Lustre file system often takes as much time as running the ls command<br />
with the -l option (the same is true if the -F, -p, or the -classify option, or any other option <br />
that requires information from a stat() call, is used). To avoid this performance overhead<br />
when using ls commands, add an alias directive similar to the following<br />
to your shell startup script:<br />
<pre><br />
$ alias ls="ls --color=never"<br />
</pre><br />
<br />
== Workspaces on flash storage ==<br />
<br />
There is another workspace file system for special reqirements available. The file system is called ''full flash pfs'' and is based on the parallel file system Lustre. Access will be granted on special request.<br />
<br />
=== Advantages of this file system ===<br />
<br />
# All storage devices are based on flash (no hard disks) with low access times. Hence performance is better compared to other parallel file systems for read and write access with small blocks and with small files, i.e. IOPS rates are improved.<br />
# The file system is mounted on bwUniCluster 2.0 and HoreKa, i.e. it can be used to share data between these clusters.<br />
<br />
=== Access restrictions ===<br />
<br />
* Only HoreKa users or KIT users of bwUniCluster 2.0 can use this file system.<br />
* Access is granted on request and for appropriate requirements. In order to request access, please open a ticket with the subject ''full flash pfs access required'' and describe the following topics:<br />
** number of files you want to store on this file system<br />
** needed capacity<br />
** number of nodes used by your typical jobs<br />
** special I/O requirements of your jobs<br />
<br />
=== Using the file system ===<br />
<br />
After access is granted, you can use the file system in the same way as a normal workspace. You just have to specify the name of the flash-based workspace file system using the option ''-F'' to all the commands that manage workspaces. On bwUniCluster 2.0 it is called ''ffuc'', on HoreKa it is ''ffhk''. For example, to create a workspace with name myws and a lifetime of 60 days on bwUniCluster 2.0 execute:<br />
ws_allocate -F ffuc myws 60<br />
<br />
If you want to use the full flash pfs on bwUniCluster 2.0 '''and''' HoreKa at the same time, please note that you only have to manage a particular workspace on one of the clusters since the name of the workspace directory is different. However, the path to each workspace is visible and can be used on both clusters.<br />
<br />
Other features are similar to normal workspaces. For example, we are able to restore expired workspaces for few weeks and you have to open a ticket to request the restore. There are quota limits with a default limit of 5 TB capacity and 25 millions inodes per user. You can check your current usage with<br />
lfs quota -uh $(whoami) /pfs/work8<br />
<br />
== $TMP ==<br />
<br />
While all tasks of a parallel application access the same $HOME and workspace directory, the <br />
$TMP directory is local to each node on bwUniCluster 2.0. All nodes have fast SSDs <br />
local storage devices which are used to store data below $TMP. Different tasks of a parallel<br />
application use different $TMP directories when they do not utilize one node. This directory should<br />
be used for temporary files being accessed by single tasks. It should also be used if you read the <br />
same data many times from a single node, e.g. if you are doing AI training. In this case you should <br />
copy the data at the beginning of your batch job to $TMP and read the data from there.<br />
In addition, this directory should be used for the installation<br />
of software packages. This means that the software package to be installed should be<br />
unpacked, compiled and linked in a subdirectory of $TMP. The real installation of the<br />
package (e.g. make install) should be made in(to) the Lustre filesystem. <br />
<br />
Each time a batch job is started, a subdirectory is created on each node and assigned to the job. <br />
$TMP is newly set and the name of the subdirectory contains the Job-id so that the<br />
subdirectory name is unique for each job. This unique name is then assigned to the<br />
environment variable $TMP within the job. At the end of the job the subdirectory is removed.<br />
Although $TMP points to the same path name for different nodes of a job, the physical location <br />
on these nodes is different.<br />
<br />
== LSDF Online Storage==<br />
<br />
In some cases it is useful to have access to the LSDF Online Storage on the HPC-Clusters also. Therefore the LSDF Online Storage is mounted on the Login- and Datamover-Nodes.<br />
Furthermore it can be used on the compute nodes during the job runtime with the constraint flag "LSDF" ([[bwUniCluster_2.0_Slurm_common_Features|Slurm common features]]<br />
). There is also an example about the LSDF batch usage: [[bwUniCluster_2.0_Slurm_common_Features#LSDF_Online_Storage|Slurm LSDF example ]] .<br />
<pre><br />
#!/bin/bash<br />
#SBATCH ...<br />
#SBATCH --constraint=LSDF<br />
</pre><br />
<br><br />
For the usage of the LSDF Online Storage the following environment variables are available: $LSDF, $LSDFPROJECTS, $LSDFHOME.<br />
Please request storage projects in the LSDF Online Storage seperately:<br />
[https://www.lsdf.kit.edu/os/storagerequest/: LSDF Storage Request].<br />
<br />
==BeeOND (BeeGFS On-Demand)==<br />
<br />
Users of the HPC-Cluster have possibility to request a private BeeOND (BeeGFS) parallel filesystem for each job. The file system is created during job startup and purged after your job.<br />
<br />
'''IMPORTANT: All data on the private filesystem will be deleted after your job. Make sure you have copied your data back to the global filesystem (within job), e.g., $HOME or any workspace.'''<br />
<br />
BeeOND/BeeGFS can be used like any other parallel file system. Tools like cp or rsync can be used to copy data in and out. <br />
<br />
For detailed usage see here:[[BwUniCluster_2.0_Slurm_common_Features#BeeOND_.28BeeGFS_On-Demand.29|Request on-demand file system]]<br />
<br />
==Backup and Archiving==<br />
<br />
There are regular backups of all data of the home directories,whereas ACLs and extended attributes will<br />
not be backuped. <br />
<br />
Please open a ticket if you need backuped data.<br />
<br />
<br />
[[Category:bwUniCluster 2.0|File System]][[Category:Hardware_and_Architecture|bwUniCluster 2.0]]</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance/2022-11&diff=11435
BwUniCluster2.0/Maintenance/2022-11
2022-11-10T14:11:42Z
<p>P Weisbrod: /* Compilers, Libaries and Runtime Environments */</p>
<hr />
<div>The following changes have been introduced during the maintenance interval between on 07.11.2022 (Monday) 08:00 and 10.11.2022 (Thursday) 17:00.<br />
<br />
The host key of the system have not changed. You should not receive any warnings by your SSH client(s), but if there should be a warning or if you want to check that you are connecting to the correct system, you can verify the key hashes using the following list:<br />
<br />
{|class="wikitable"<br />
! Algorithm<br />
! Hash (SHA256)<br />
! Hash (MD5)<br />
|-<br />
|RSA<br />
|p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|59:2a:67:44:4a:d7:89:6c:c0:0d:74:ba:3c:c4:63:6d<br />
|-<br />
|ECDSA<br />
|k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|85:d4:d9:97:e0:f0:43:30:6e:66:8e:d0:b6:9b:85:d1<br />
|-<br />
|ED25519<br />
|yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|42:d2:0d:ab:87:48:fc:1d:5d:b3:7c:bf:22:c3:5f:b7<br />
|}<br />
<br />
= Hardware =<br />
<br />
* All firmware versions on all components were upgraded<br />
<br />
= Operating system =<br />
<br />
* The operating system remains at RHEL 8.4 EUS<br />
<br />
* The Mellanox OFED InfiniBand Stack was updated to version 5.5-2.1.7.0<br />
<br />
= Compilers, Libaries and Runtime Environments =<br />
* The Intel Parallel Studio (Compiler, MKL, MPI) version 2019 modules were removed during the maintenance.<br />
* clang 9 and llvm 10 modules were removed<br />
<br />
= Userspace tools =<br />
<br />
<br />
= Software Modules =<br />
<br />
<br />
= Batch system =<br />
<br />
* The Slurm version was upgraded to version 22.05.5<br />
* Pyxis Plugin version was upgraded to 0.14.0<br />
<br />
= Storage =<br />
* Lustre client, BeeGFS client and Spectrum Scale client were updated<br />
<br />
= Graphics stack =<br />
<br />
* The NVIDIA driver was upgraded to version 515.65.07<br />
* Cuda 11.7 was installed<br />
<br />
= Containers =<br />
<br />
<br />
<br />
= JupyterHub =</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance/2022-11&diff=11434
BwUniCluster2.0/Maintenance/2022-11
2022-11-10T14:11:32Z
<p>P Weisbrod: /* Graphics stack */</p>
<hr />
<div>The following changes have been introduced during the maintenance interval between on 07.11.2022 (Monday) 08:00 and 10.11.2022 (Thursday) 17:00.<br />
<br />
The host key of the system have not changed. You should not receive any warnings by your SSH client(s), but if there should be a warning or if you want to check that you are connecting to the correct system, you can verify the key hashes using the following list:<br />
<br />
{|class="wikitable"<br />
! Algorithm<br />
! Hash (SHA256)<br />
! Hash (MD5)<br />
|-<br />
|RSA<br />
|p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|59:2a:67:44:4a:d7:89:6c:c0:0d:74:ba:3c:c4:63:6d<br />
|-<br />
|ECDSA<br />
|k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|85:d4:d9:97:e0:f0:43:30:6e:66:8e:d0:b6:9b:85:d1<br />
|-<br />
|ED25519<br />
|yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|42:d2:0d:ab:87:48:fc:1d:5d:b3:7c:bf:22:c3:5f:b7<br />
|}<br />
<br />
= Hardware =<br />
<br />
* All firmware versions on all components were upgraded<br />
<br />
= Operating system =<br />
<br />
* The operating system remains at RHEL 8.4 EUS<br />
<br />
* The Mellanox OFED InfiniBand Stack was updated to version 5.5-2.1.7.0<br />
<br />
= Compilers, Libaries and Runtime Environments =<br />
* The Intel Parallel Studio (Compiler, MKL, MPI) version 2019 modules were removed during the maintenance.<br />
* clang 9 and llvm 10 modules were be removed<br />
<br />
= Userspace tools =<br />
<br />
<br />
= Software Modules =<br />
<br />
<br />
= Batch system =<br />
<br />
* The Slurm version was upgraded to version 22.05.5<br />
* Pyxis Plugin version was upgraded to 0.14.0<br />
<br />
= Storage =<br />
* Lustre client, BeeGFS client and Spectrum Scale client were updated<br />
<br />
= Graphics stack =<br />
<br />
* The NVIDIA driver was upgraded to version 515.65.07<br />
* Cuda 11.7 was installed<br />
<br />
= Containers =<br />
<br />
<br />
<br />
= JupyterHub =</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance/2022-11&diff=11433
BwUniCluster2.0/Maintenance/2022-11
2022-11-10T13:38:33Z
<p>P Weisbrod: </p>
<hr />
<div>The following changes have been introduced during the maintenance interval between on 07.11.2022 (Monday) 08:00 and 10.11.2022 (Thursday) 17:00.<br />
<br />
The host key of the system have not changed. You should not receive any warnings by your SSH client(s), but if there should be a warning or if you want to check that you are connecting to the correct system, you can verify the key hashes using the following list:<br />
<br />
{|class="wikitable"<br />
! Algorithm<br />
! Hash (SHA256)<br />
! Hash (MD5)<br />
|-<br />
|RSA<br />
|p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|59:2a:67:44:4a:d7:89:6c:c0:0d:74:ba:3c:c4:63:6d<br />
|-<br />
|ECDSA<br />
|k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|85:d4:d9:97:e0:f0:43:30:6e:66:8e:d0:b6:9b:85:d1<br />
|-<br />
|ED25519<br />
|yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|42:d2:0d:ab:87:48:fc:1d:5d:b3:7c:bf:22:c3:5f:b7<br />
|}<br />
<br />
= Hardware =<br />
<br />
* All firmware versions on all components were upgraded<br />
<br />
= Operating system =<br />
<br />
* The operating system remains at RHEL 8.4 EUS<br />
<br />
* The Mellanox OFED InfiniBand Stack was updated to version 5.5-2.1.7.0<br />
<br />
= Compilers, Libaries and Runtime Environments =<br />
* The Intel Parallel Studio (Compiler, MKL, MPI) version 2019 modules were removed during the maintenance.<br />
* clang 9 and llvm 10 modules were be removed<br />
<br />
= Userspace tools =<br />
<br />
<br />
= Software Modules =<br />
<br />
<br />
= Batch system =<br />
<br />
* The Slurm version was upgraded to version 22.05.5<br />
* Pyxis Plugin version was upgraded to 0.14.0<br />
<br />
= Storage =<br />
* Lustre client, BeeGFS client and Spectrum Scale client were updated<br />
<br />
= Graphics stack =<br />
<br />
* The NVIDIA driver was upgraded to version 515.65.07<br />
* Cuda 11.7 was be installed<br />
<br />
= Containers =<br />
<br />
<br />
<br />
= JupyterHub =</div>
P Weisbrod
https://wiki.bwhpc.de/wiki/index.php?title=BwUniCluster2.0/Maintenance/2022-11&diff=11316
BwUniCluster2.0/Maintenance/2022-11
2022-11-03T14:48:16Z
<p>P Weisbrod: </p>
<hr />
<div>The following changes have been introduced during the maintenance interval between on 07.11.2022 (Monday) 08:00 and 10.11.2022 (Thursday) 17:00.<br />
<br />
The host key of the system have not changed. You should not receive any warnings by your SSH client(s), but if there should be a warning or if you want to check that you are connecting to the correct system, you can verify the key hashes using the following list:<br />
<br />
{|class="wikitable"<br />
! Algorithm<br />
! Hash (SHA256)<br />
! Hash (MD5)<br />
|-<br />
|RSA<br />
|p6Ion2YKZr5cnzf6L6DS1xGnIwnC1BhLbOEmDdp7FA0<br />
|59:2a:67:44:4a:d7:89:6c:c0:0d:74:ba:3c:c4:63:6d<br />
|-<br />
|ECDSA<br />
|k8l1JnfLf1y1Qi55IQmo11+/NZx06Rbze7akT5R7tE8<br />
|85:d4:d9:97:e0:f0:43:30:6e:66:8e:d0:b6:9b:85:d1<br />
|-<br />
|ED25519<br />
|yEe5nJ5hZZ1YbgieWr+phqRZKYbrV7zRe8OR3X03cn0<br />
|42:d2:0d:ab:87:48:fc:1d:5d:b3:7c:bf:22:c3:5f:b7<br />
|}<br />
<br />
= Hardware =<br />
<br />
* All firmware versions on all components will be upgraded<br />
<br />
= Operating system =<br />
<br />
* The operating system version will not change, it will remain at RHEL 8.4 EUS<br />
<br />
* The Mellanox OFED InfiniBand Stack will be updated to version 5.5-2.1.7.0<br />
<br />
= Compilers, Libaries and Runtime Environments =<br />
* The Intel Parallel Studio (Compiler, MKL, MPI) version 2019 modules will be removed during the maintenance.<br />
* clang 9 and llvm 10 modules will be removed<br />
<br />
= Userspace tools =<br />
<br />
<br />
= Software Modules =<br />
<br />
<br />
= Batch system =<br />
<br />
* The Slurm version will be upgraded to version 22.05.5<br />
* Pyxis Plugin version will be upgraded to 0.14.0<br />
<br />
= Storage =<br />
* Lustre client, BeeGFS client and Spectrum Scale client will be updated<br />
<br />
= Graphics stack =<br />
<br />
* The NVIDIA driver will be upgraded to version 515.65.07<br />
* Cuda 11.7 will be installed<br />
<br />
= Containers =<br />
<br />
<br />
<br />
= JupyterHub =</div>
P Weisbrod