bwHPC Wiki - User contributions [en]

Helix/Slurm

2025-12-12T14:50:23Z

H Winkhardt: /* GPU Programs */

= General information about Slurm =
The bwForCluster Helix uses Slurm as batch system.
* Slurm documentation: https://slurm.schedmd.com/documentation.html
* Slurm cheat sheet: https://slurm.schedmd.com/pdfs/summary.pdf
* Slurm tutorials: https://slurm.schedmd.com/tutorials.html

= Slurm Command Overview =

{| width=750px class="wikitable"
! Slurm commands !! Brief explanation
|-
| [https://slurm.schedmd.com/sbatch.html sbatch] || Submits a job and queues it in an input queue
|-
| [https://slurm.schedmd.com/salloc.html salloc] || Request resources for an interactive job
|-
| [https://slurm.schedmd.com/squeue.html squeue] || Displays information about active, eligible, blocked, and/or recently completed jobs
|-
| [https://slurm.schedmd.com/scontrol.html scontrol] || Displays detailed job state information
|-
| [https://slurm.schedmd.com/scontrol.html sstat] || Displays status information about a running job
|-
| [https://slurm.schedmd.com/scancel.html scancel] || Cancels a job
|-
|}

= Job Submission =

Batch jobs are submitted with the command:

<syntaxhighlight lang="bash">$ sbatch <job-script> </syntaxhighlight>

A job script contains options for Slurm in lines beginning with #SBATCH as well as your commands which you want to execute on the compute nodes. For example:

<syntaxhighlight lang="slurm">#!/bin/bash
#SBATCH --partition=cpu-single
#SBATCH --ntasks=1
#SBATCH --time=00:20:00
#SBATCH --mem=1gb
#SBATCH --export=NONE
echo 'Hello world'
</syntaxhighlight>

This jobs requests one core (--ntasks=1) and 1 GB memory (--mem=1gb) for 20 minutes (--time=00:20:00) on nodes provided by the partition 'cpu-single'.

For the sake of a better reproducibility of jobs it is recommended to use the option --export=NONE to prevent the propagation of environment variables from the submit session into the job environment and to load required software modules in the job script.

== Partitions ==

On bwForCluster Helix it is necessary to request a partition with "--partition=<partition_name>" on job submission. Within a partition job allocations are routed automatically to the most suitable compute node(s) for the requested resources (e.g. amount of nodes and cores, memory, number of GPUs). The devel partition is the default partition, if no partition is requested.

The partitions devel, cpu-single and gpu-single are operated in shared mode, i.e. jobs from different users can run on the same node. Jobs can get exclusive access to compute nodes in these partitions with the "--exclusive" option. The partitions cpu-multi and gpu-multi are operated in exclusive mode. Jobs in these partitions automatically get exclusive access to the requested compute nodes.

{| class="wikitable"
|-
! style="width:20%"| Partition
! style="width:20%"| Node Access Policy
! style="width:20%"| [https://wiki.bwhpc.de/e/Helix/Hardware#Compute_Nodes Node Types]
! style="width:20%"| Default
! style="width:20%"| Limits
|-
| devel
| shared
| cpu, gpu4
| ntasks=1, time=00:10:00, mem-per-cpu=2gb
| nodes=2, time=00:30:00
|-
| cpu-single
| shared
| cpu, fat
| ntasks=1, time=00:30:00, mem-per-cpu=2gb
| nodes=1, time=120:00:00
|-
| gpu-single
| shared
| gpu4, gpu8
| ntasks=1, time=00:30:00, mem-per-cpu=2gb
| nodes=1, time=120:00:00
|-
| cpu-multi
| job exclusive
| cpu
| nodes=2, time=00:30:00
| nodes=32, time=48:00:00
|-
| gpu-multi
| job exclusive
| gpu4
| nodes=2, time=00:30:00
| nodes=8, time=48:00:00
|}

== GPU requests ==

For the partitions gpu-single and gpu-multi is it required to request GPU ressources.
* The number of GPUs is requested with the option "--gres=gpu:<number-of-gpus>".
* A specific GPU type can be requested with the option "--gres=gpu:<gpu-type>:<number-of-gpus>". Possible values for <gpu-type> are listed in the line 'GPU Type' of the [https://wiki.bwhpc.de/e/Helix/Hardware#Compute_Nodes Compute Nodes table].
* GPUs that are suitable for a specific GPU memory requirement can be requested with option "--gres=gpu:<number-of-gpus>,gpumem_per_gpu:<required-gpumem>GB". This only restricts the selection of possible GPU types. For the job the total GPU memory per GPU is available as listed in the line 'GPU memory per GPU' of the [https://wiki.bwhpc.de/e/Helix/Hardware#Compute_Nodes Compute Nodes table].

== Constraints ==

It is possible to refine the resource request for a job with the option "--constraint=<feature>".

{| class="wikitable"
|-
! style="width:20%"| Feature
! style="width:80%"| Meaning
|-
| fp64
| request GPU types with FP64 capability (double precision)
|}

== Examples ==

Here you can find some example scripts for batch jobs.

=== Serial Programs ===

<syntaxhighlight lang="slurm">
#!/bin/bash
#SBATCH --partition=cpu-single
#SBATCH --ntasks=1
#SBATCH --time=20:00:00
#SBATCH --mem=4gb
./my_serial_program
</syntaxhighlight>

'''Notes:'''
* Jobs with "--mem" up to 236gb can run on all node types associated with the cpu-single partition.

=== Multi-threaded Programs ===

<syntaxhighlight lang="slurm">
#!/bin/bash
#SBATCH --partition=cpu-single
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=16
#SBATCH --time=01:30:00
#SBATCH --mem=50gb
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
./my_multithreaded_program
</syntaxhighlight>

'''Notes:'''
* Jobs with "--ntasks-per-node" up to 64 and "--mem" up to 236gb can run on all node types associated with the cpu-single partition.
* With "export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}" you can set the number of threads according to the number of resources requested.

=== MPI Programs ===

<syntaxhighlight lang="slurm">
#!/bin/bash
#SBATCH --partition=cpu-multi
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=64
#SBATCH --time=12:00:00
module load compiler/gnu
module load mpi/openmpi
srun ./my_mpi_program
</syntaxhighlight>

'''Notes:'''
* "--mem" requests the memory per node. The maximum is 236gb.
* The Compiler and MPI modules used for the compilation must be loaded before the start of the program.
* It is recommended to start MPI programs with 'srun'.

=== GPU Programs ===

<syntaxhighlight lang="slurm">
#!/bin/bash
#SBATCH --partition=gpu-single
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --gres=gpu:A40:1
#SBATCH --time=12:00:00
#SBATCH --mem=16gb

module load devel/cuda
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
./my_cuda_program
</syntaxhighlight>

'''Notes:'''
* The number of GPUs per node is requested with the option "--gres=gpu:<number-of-gpus>"
* It is recommended to request a suitable GPU type for your application with the option "--gres=gpu:<gpu-type>:<number-of-gpus>". For <gpu-type> put the 'GPU Type' listed in the [https://wiki.bwhpc.de/e/Helix/Hardware#Compute_Nodes Compute Nodes table].
** Example for a request of two A40 GPUs: --gres=gpu:A40:2
** Example for a request of one A100 GPU: --gres=gpu:A100:1
* If you are unsure on which GPU type your code runs faster, please run a test case and compare the run times. In general the following applies:
** A40 GPUs are optimized for single precision computations.
** A100 and H200 GPUs offer better performance for double precision computations or if the code makes use of tensor cores.
* The CUDA module used for compilation must be loaded before the start of the program.

=== More examples ===

Further batch script examples are available on bwForCluster Helix in the directory: <code>/opt/bwhpc/common/system/slurm-examples</code>

= Interactive Jobs =

Interactive jobs must NOT run on the login nodes, however resources for interactive jobs can be requested using srun. The following example requests an interactive session on 1 core for 2 hours:

<syntaxhighlight lang="bash">$ salloc --partition=cpu-single --ntasks=1 --time=2:00:00 </syntaxhighlight>

After execution of this command wait until the queueing system has granted you the requested resources. Once granted you will be automatically logged on the allocated compute node.

If you use applications or tools which provide a GUI, enable X-forwarding for your interactive session with:

<syntaxhighlight lang="bash">$ salloc --partition=cpu-single --ntasks=1 --time=2:00:00 --x11 </syntaxhighlight>

Once the walltime limit has been reached you will be automatically logged out from the compute node.

For convenient access to specific GUI applications (JupyterLab, ...) on the cluster, we provide a web-based platform: [[Helix/bwVisu | bwVisu]]

= Job Monitoring =

== Information about submitted jobs ==

For an overview of your submitted jobs use the command:

<syntaxhighlight lang="bash">$ squeue</syntaxhighlight>

To get detailed information about a specific job use the command:

<syntaxhighlight lang="bash">$ scontrol show job <jobid></syntaxhighlight>

'''Notes:''' 
A job start may be delayed for various reasons:
* (QOSMaxCpuPerUserLimit) - There is a limit to how many CPU cores a user can use at the same time. The job exceeds this limit.
* (QOSMaxGRESPerUser) - There is a limit to how many GPUs a user can use at the same time. The job exceeds this limit.
* (QOSMinGRES) - The job was submitted to a gpu partition without requesting a GPU.
* (launch failed requeued held) - The job has failed to start. You may be able to resume it using scontrol. Alternatively you can cancel it and submit it again.
For further reasons please refer to: https://slurm.schedmd.com/job_reason_codes.html

== Information about resource usage of running jobs ==

You can monitor the resource usage of running jobs with the sstat command. For example:

<pre>
$ sstat --format=JobId,AveCPU,AveRSS,MaxRSS -j <jobid>
</pre>

This will show average CPU time, average and maximum memory consumption of all tasks in the running job.

'sstat -e' command shows a list of fields that can be specified with the '--format' option.

== Interactive access to running jobs ==

It is also possible to attach an interactive shell to a running job with command:

<syntaxhighlight lang="bash">$ srun --jobid=<jobid> --overlap --pty /bin/bash</syntaxhighlight>

Commands like 'top' show you the most busy processes on the node. To exit 'top' type 'q'.

To monitor your GPU processes use the command 'nvidia-smi'.

== Job Feedback ==
You get feedback on resource usage and job efficiency for completed jobs with the command:

<syntaxhighlight lang="bash">
$ seff <jobid>
</syntaxhighlight>

Example Output:

<syntaxhighlight lang="bash">
============================= JOB FEEDBACK =============================
Job ID: 12345678
Cluster: helix
User/Group: hd_ab123/hd_hd
State: COMPLETED (exit code 0)
Nodes: 2
Cores per node: 64
CPU Utilized: 3-04:11:46
CPU Efficiency: 97.90% of 3-05:49:52 core-walltime
Job Wall-clock time: 00:36:29
Memory Utilized: 432.74 GB (estimated maximum)
Memory Efficiency: 85.96% of 503.42 GB (251.71 GB/node)
</syntaxhighlight>

Explanation:
* Nodes: Number of allocated nodes for the job.
* Cores per node: Number of physical cores per node allocated for the job.
* CPU Utilized: Sum of utilized core time.
* CPU Efficiency: 'CPU Utilized' with respect to core-walltime (= 'Nodes' x 'Cores per node' x 'Job Wall-clock time') in percent.
* Job Wall-clock time: runtime of the job.
* Memory Utilized: Sum of memory used. For multi node MPI jobs the sum is only correct when srun is used instead of mpirun.
* Memory Efficiency: 'Memory Utilized' with respect to total allocated memory for the job.

== Job Monitoring Portal ==
For more detailed information about your jobs visit the job monitoring portal: https://helix-monitoring.bwservices.uni-heidelberg.de

= Accounting =

Jobs are billed for allocated CPU cores, memory and GPUs.

To see the accounting data of a specific job:

<syntaxhighlight lang="bash">$ sacct -j <jobid> --format=user,jobid,account,nnodes,ncpus,time,elapsed,AllocTRES%50</syntaxhighlight>

To retrive the job history for a specific user for a certain time frame:

<syntaxhighlight lang="bash">$ sacct -u <user> -S 2022-08-20 -E 2022-08-30 --format=user,jobid,account,nnodes,ncpus,time,elapsed,AllocTRES%50</syntaxhighlight>

= Overview about free resources =

On the login nodes the following command shows what resources are available for immediate use:

<syntaxhighlight lang="bash">$ sinfo_t_idle</syntaxhighlight>

Helix/Slurm

2025-12-12T14:49:53Z

H Winkhardt: GPU example script update for Helix

= General information about Slurm =
The bwForCluster Helix uses Slurm as batch system.
* Slurm documentation: https://slurm.schedmd.com/documentation.html
* Slurm cheat sheet: https://slurm.schedmd.com/pdfs/summary.pdf
* Slurm tutorials: https://slurm.schedmd.com/tutorials.html

= Slurm Command Overview =

{| width=750px class="wikitable"
! Slurm commands !! Brief explanation
|-
| [https://slurm.schedmd.com/sbatch.html sbatch] || Submits a job and queues it in an input queue
|-
| [https://slurm.schedmd.com/salloc.html salloc] || Request resources for an interactive job
|-
| [https://slurm.schedmd.com/squeue.html squeue] || Displays information about active, eligible, blocked, and/or recently completed jobs
|-
| [https://slurm.schedmd.com/scontrol.html scontrol] || Displays detailed job state information
|-
| [https://slurm.schedmd.com/scontrol.html sstat] || Displays status information about a running job
|-
| [https://slurm.schedmd.com/scancel.html scancel] || Cancels a job
|-
|}

= Job Submission =

Batch jobs are submitted with the command:

<syntaxhighlight lang="bash">$ sbatch <job-script> </syntaxhighlight>

A job script contains options for Slurm in lines beginning with #SBATCH as well as your commands which you want to execute on the compute nodes. For example:

<syntaxhighlight lang="slurm">#!/bin/bash
#SBATCH --partition=cpu-single
#SBATCH --ntasks=1
#SBATCH --time=00:20:00
#SBATCH --mem=1gb
#SBATCH --export=NONE
echo 'Hello world'
</syntaxhighlight>

This jobs requests one core (--ntasks=1) and 1 GB memory (--mem=1gb) for 20 minutes (--time=00:20:00) on nodes provided by the partition 'cpu-single'.

For the sake of a better reproducibility of jobs it is recommended to use the option --export=NONE to prevent the propagation of environment variables from the submit session into the job environment and to load required software modules in the job script.

== Partitions ==

On bwForCluster Helix it is necessary to request a partition with "--partition=<partition_name>" on job submission. Within a partition job allocations are routed automatically to the most suitable compute node(s) for the requested resources (e.g. amount of nodes and cores, memory, number of GPUs). The devel partition is the default partition, if no partition is requested.

The partitions devel, cpu-single and gpu-single are operated in shared mode, i.e. jobs from different users can run on the same node. Jobs can get exclusive access to compute nodes in these partitions with the "--exclusive" option. The partitions cpu-multi and gpu-multi are operated in exclusive mode. Jobs in these partitions automatically get exclusive access to the requested compute nodes.

{| class="wikitable"
|-
! style="width:20%"| Partition
! style="width:20%"| Node Access Policy
! style="width:20%"| [https://wiki.bwhpc.de/e/Helix/Hardware#Compute_Nodes Node Types]
! style="width:20%"| Default
! style="width:20%"| Limits
|-
| devel
| shared
| cpu, gpu4
| ntasks=1, time=00:10:00, mem-per-cpu=2gb
| nodes=2, time=00:30:00
|-
| cpu-single
| shared
| cpu, fat
| ntasks=1, time=00:30:00, mem-per-cpu=2gb
| nodes=1, time=120:00:00
|-
| gpu-single
| shared
| gpu4, gpu8
| ntasks=1, time=00:30:00, mem-per-cpu=2gb
| nodes=1, time=120:00:00
|-
| cpu-multi
| job exclusive
| cpu
| nodes=2, time=00:30:00
| nodes=32, time=48:00:00
|-
| gpu-multi
| job exclusive
| gpu4
| nodes=2, time=00:30:00
| nodes=8, time=48:00:00
|}

== GPU requests ==

For the partitions gpu-single and gpu-multi is it required to request GPU ressources.
* The number of GPUs is requested with the option "--gres=gpu:<number-of-gpus>".
* A specific GPU type can be requested with the option "--gres=gpu:<gpu-type>:<number-of-gpus>". Possible values for <gpu-type> are listed in the line 'GPU Type' of the [https://wiki.bwhpc.de/e/Helix/Hardware#Compute_Nodes Compute Nodes table].
* GPUs that are suitable for a specific GPU memory requirement can be requested with option "--gres=gpu:<number-of-gpus>,gpumem_per_gpu:<required-gpumem>GB". This only restricts the selection of possible GPU types. For the job the total GPU memory per GPU is available as listed in the line 'GPU memory per GPU' of the [https://wiki.bwhpc.de/e/Helix/Hardware#Compute_Nodes Compute Nodes table].

== Constraints ==

It is possible to refine the resource request for a job with the option "--constraint=<feature>".

{| class="wikitable"
|-
! style="width:20%"| Feature
! style="width:80%"| Meaning
|-
| fp64
| request GPU types with FP64 capability (double precision)
|}

== Examples ==

Here you can find some example scripts for batch jobs.

=== Serial Programs ===

<syntaxhighlight lang="slurm">
#!/bin/bash
#SBATCH --partition=cpu-single
#SBATCH --ntasks=1
#SBATCH --time=20:00:00
#SBATCH --mem=4gb
./my_serial_program
</syntaxhighlight>

'''Notes:'''
* Jobs with "--mem" up to 236gb can run on all node types associated with the cpu-single partition.

=== Multi-threaded Programs ===

<syntaxhighlight lang="slurm">
#!/bin/bash
#SBATCH --partition=cpu-single
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=16
#SBATCH --time=01:30:00
#SBATCH --mem=50gb
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
./my_multithreaded_program
</syntaxhighlight>

'''Notes:'''
* Jobs with "--ntasks-per-node" up to 64 and "--mem" up to 236gb can run on all node types associated with the cpu-single partition.
* With "export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}" you can set the number of threads according to the number of resources requested.

=== MPI Programs ===

<syntaxhighlight lang="slurm">
#!/bin/bash
#SBATCH --partition=cpu-multi
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=64
#SBATCH --time=12:00:00
module load compiler/gnu
module load mpi/openmpi
srun ./my_mpi_program
</syntaxhighlight>

'''Notes:'''
* "--mem" requests the memory per node. The maximum is 236gb.
* The Compiler and MPI modules used for the compilation must be loaded before the start of the program.
* It is recommended to start MPI programs with 'srun'.

=== GPU Programs ===

<syntaxhighlight lang="slurm">
#!/bin/bash
#SBATCH --partition=gpu-single
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --gres=gpu:A40:1
#SBATCH --time=12:00:00
#SBATCH --mem=59gb

module load devel/cuda
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
./my_cuda_program
</syntaxhighlight>

'''Notes:'''
* The number of GPUs per node is requested with the option "--gres=gpu:<number-of-gpus>"
* It is recommended to request a suitable GPU type for your application with the option "--gres=gpu:<gpu-type>:<number-of-gpus>". For <gpu-type> put the 'GPU Type' listed in the [https://wiki.bwhpc.de/e/Helix/Hardware#Compute_Nodes Compute Nodes table].
** Example for a request of two A40 GPUs: --gres=gpu:A40:2
** Example for a request of one A100 GPU: --gres=gpu:A100:1
* If you are unsure on which GPU type your code runs faster, please run a test case and compare the run times. In general the following applies:
** A40 GPUs are optimized for single precision computations.
** A100 and H200 GPUs offer better performance for double precision computations or if the code makes use of tensor cores.
* The CUDA module used for compilation must be loaded before the start of the program.

=== More examples ===

Further batch script examples are available on bwForCluster Helix in the directory: <code>/opt/bwhpc/common/system/slurm-examples</code>

= Interactive Jobs =

Interactive jobs must NOT run on the login nodes, however resources for interactive jobs can be requested using srun. The following example requests an interactive session on 1 core for 2 hours:

<syntaxhighlight lang="bash">$ salloc --partition=cpu-single --ntasks=1 --time=2:00:00 </syntaxhighlight>

After execution of this command wait until the queueing system has granted you the requested resources. Once granted you will be automatically logged on the allocated compute node.

If you use applications or tools which provide a GUI, enable X-forwarding for your interactive session with:

<syntaxhighlight lang="bash">$ salloc --partition=cpu-single --ntasks=1 --time=2:00:00 --x11 </syntaxhighlight>

Once the walltime limit has been reached you will be automatically logged out from the compute node.

For convenient access to specific GUI applications (JupyterLab, ...) on the cluster, we provide a web-based platform: [[Helix/bwVisu | bwVisu]]

= Job Monitoring =

== Information about submitted jobs ==

For an overview of your submitted jobs use the command:

<syntaxhighlight lang="bash">$ squeue</syntaxhighlight>

To get detailed information about a specific job use the command:

<syntaxhighlight lang="bash">$ scontrol show job <jobid></syntaxhighlight>

'''Notes:''' 
A job start may be delayed for various reasons:
* (QOSMaxCpuPerUserLimit) - There is a limit to how many CPU cores a user can use at the same time. The job exceeds this limit.
* (QOSMaxGRESPerUser) - There is a limit to how many GPUs a user can use at the same time. The job exceeds this limit.
* (QOSMinGRES) - The job was submitted to a gpu partition without requesting a GPU.
* (launch failed requeued held) - The job has failed to start. You may be able to resume it using scontrol. Alternatively you can cancel it and submit it again.
For further reasons please refer to: https://slurm.schedmd.com/job_reason_codes.html

== Information about resource usage of running jobs ==

You can monitor the resource usage of running jobs with the sstat command. For example:

<pre>
$ sstat --format=JobId,AveCPU,AveRSS,MaxRSS -j <jobid>
</pre>

This will show average CPU time, average and maximum memory consumption of all tasks in the running job.

'sstat -e' command shows a list of fields that can be specified with the '--format' option.

== Interactive access to running jobs ==

It is also possible to attach an interactive shell to a running job with command:

<syntaxhighlight lang="bash">$ srun --jobid=<jobid> --overlap --pty /bin/bash</syntaxhighlight>

Commands like 'top' show you the most busy processes on the node. To exit 'top' type 'q'.

To monitor your GPU processes use the command 'nvidia-smi'.

== Job Feedback ==
You get feedback on resource usage and job efficiency for completed jobs with the command:

<syntaxhighlight lang="bash">
$ seff <jobid>
</syntaxhighlight>

Example Output:

<syntaxhighlight lang="bash">
============================= JOB FEEDBACK =============================
Job ID: 12345678
Cluster: helix
User/Group: hd_ab123/hd_hd
State: COMPLETED (exit code 0)
Nodes: 2
Cores per node: 64
CPU Utilized: 3-04:11:46
CPU Efficiency: 97.90% of 3-05:49:52 core-walltime
Job Wall-clock time: 00:36:29
Memory Utilized: 432.74 GB (estimated maximum)
Memory Efficiency: 85.96% of 503.42 GB (251.71 GB/node)
</syntaxhighlight>

Explanation:
* Nodes: Number of allocated nodes for the job.
* Cores per node: Number of physical cores per node allocated for the job.
* CPU Utilized: Sum of utilized core time.
* CPU Efficiency: 'CPU Utilized' with respect to core-walltime (= 'Nodes' x 'Cores per node' x 'Job Wall-clock time') in percent.
* Job Wall-clock time: runtime of the job.
* Memory Utilized: Sum of memory used. For multi node MPI jobs the sum is only correct when srun is used instead of mpirun.
* Memory Efficiency: 'Memory Utilized' with respect to total allocated memory for the job.

== Job Monitoring Portal ==
For more detailed information about your jobs visit the job monitoring portal: https://helix-monitoring.bwservices.uni-heidelberg.de

= Accounting =

Jobs are billed for allocated CPU cores, memory and GPUs.

To see the accounting data of a specific job:

<syntaxhighlight lang="bash">$ sacct -j <jobid> --format=user,jobid,account,nnodes,ncpus,time,elapsed,AllocTRES%50</syntaxhighlight>

To retrive the job history for a specific user for a certain time frame:

<syntaxhighlight lang="bash">$ sacct -u <user> -S 2022-08-20 -E 2022-08-30 --format=user,jobid,account,nnodes,ncpus,time,elapsed,AllocTRES%50</syntaxhighlight>

= Overview about free resources =

On the login nodes the following command shows what resources are available for immediate use:

<syntaxhighlight lang="bash">$ sinfo_t_idle</syntaxhighlight>

Issueing Entitlements in bwHPC

2025-12-09T14:32:20Z

H Winkhardt: More links

The HPC systems provided by the bwHPC initiative are, as any supercomputer, considered dual-use commodities and are subjected to export controls as outlined in a [https://www.bafa.de/EN/Foreign_Trade/Export_Control/Export_Control_and_Academia/export_control_academia_node.html guide about "Export Control and Academia"] by the Federal Office for Economic Affairs and Export Control, Germany (BAFA).

Therefore, any principal investigator or any user seeking to use the bwUnicluster or one of the bwForClusters has to comply with [https://eur-lex.europa.eu/legal-content/EN-DE/TXT/?from=DE&uri=CELEX%3A32021R0821 European Community Council regulation No 2021/821] as well as with the [http://www.gesetze-im-internet.de/englisch_awg/index.html German Foreign Trade and Payments Act] and [http://www.gesetze-im-internet.de/englisch_awv/index.html German Foreign Trade and Payments Ordinance] which includes:
* no violation regarding embargoed countries and persons
* no violation regarding regulations of dual-use commodities.

 
Acknowledged compliance to EU and German regulations (e.g. in a signed statement) is part of the entitlement for the [[Registration/bwUniCluster/Entitlement|bwUniCluster]] and the [[Registration/bwForCluster/Entitlement|bwForClusters]] issued by the home organization of the principal investigator or user.

Registration/SSH/SSH-FIDO2-Quick-Start

2025-12-02T09:53:16Z

H Winkhardt: typo

= SSH with Yubikey - Quick Start Guide =

This guide shows you how to secure your SSH keys with a Yubikey using FIDO2.

<div style="border: 3px solid #28a745; padding: 15px; background-color: #d4edda; margin: 10px 0;">
'''Why FIDO2 SSH Keys?'''

FIDO2 SSH keys (ED25519-SK) offer significant advantages:
* '''No 2-factor unlock required''' - work immediately after registration
* '''Hardware-protected''' - private key never leaves the Yubikey
* '''Physical presence required''' - you must touch the key to authenticate

'''Note:''' ECDSA-SK keys are not supported on these clusters. Use ED25519-SK instead.

{| class="wikitable" style="text-align:center; margin-top:10px;"
|+ Cluster Support
|-
! Cluster !! FIDO2 Support
|-
| bwUniCluster 3.0 || style="background-color:#90EE90;" | ✓
|-
| bwForCluster BinAC 2 || style="background-color:#FFB6C1;" | ✗
|-
| bwForCluster Helix || style="background-color:#FFB6C1;" | ✗
|-
| bwForCluster JUSTUS 2 || style="background-color:#FFB6C1;" | ✗
|-
| bwForCluster NEMO 2 || style="background-color:#90EE90;" | ✓
|}
</div>

= Requirements =

* A Yubikey (5 series or newer)
* OpenSSH 8.2 or newer
* Linux system with <code>yubikey-manager</code> and <code>openssh-client</code> packages

'''Check your OpenSSH version:'''
<pre>
ssh -V
ssh -Q PubkeyAcceptedAlgorithms | grep sk-ssh-ed25519@openssh.com
</pre>

'''Need help with installation?''' See the [https://gitlab.rz.uni-freiburg.de/escience-public/yubikey#ssh-and-yubikeys detailed setup guide] for your operating system.

= Quick Start Guide =

== Step 1: Set up your Yubikey ==

'''What is a FIDO2 PIN?''' The PIN protects your Yubikey's FIDO2 functions. It's separate from any other PINs on your Yubikey (like the PIV PIN).

Set a PIN for your Yubikey's FIDO2 functionality:

<pre>
ykman fido access change-pin
</pre>

You'll be asked to enter a new PIN. Choose a PIN you can remember - you have 8 attempts before needing to reset.

'''Note:''' For GUI setup, you can use [https://www.yubico.com/products/yubico-authenticator/ Yubico Authenticator] instead.

'''Forgot your PIN or need to reset?''' See the [https://gitlab.rz.uni-freiburg.de/escience-public/yubikey#troubleshooting troubleshooting guide].

== Step 2: Create an SSH Key ==

'''What happens here?''' This creates a non-discoverable FIDO2 SSH key. The private key material is wrapped/encrypted with a secret stored on your Yubikey - the Yubikey is required for every authentication, but the actual key credentials are stored on your computer.

'''Note:''' For keys stored directly on the Yubikey (resident/discoverable keys), see the [https://gitlab.rz.uni-freiburg.de/escience-public/yubikey#creating-discoverable-ssh-keys resident keys documentation].

Connect your Yubikey and create a new SSH key:

<pre>
ssh-keygen -t ed25519-sk
</pre>

When prompted:
* '''Enter PIN:''' First, you'll be asked to enter your FIDO2 PIN (the one you set in Step 1)
* '''Filename:''' Choose a descriptive name like <code>id_ed25519_sk_bwunicluster</code> or <code>id_ed25519_sk_nemo2</code>
* '''Passphrase:''' Press Enter to skip - the key is protected by requiring physical Yubikey presence
* '''Touch your Yubikey:''' The key will blink - touch it to confirm

Two files are created:
* <code>~/.ssh/id_ed25519_sk_bwunicluster</code> - key handle (encrypted credentials that require the Yubikey)
* <code>~/.ssh/id_ed25519_sk_bwunicluster.pub</code> - public key (this is what you'll upload)

'''Want keys stored on the Yubikey itself?''' See [https://gitlab.rz.uni-freiburg.de/escience-public/yubikey#creating-discoverable-ssh-keys resident keys documentation] for discoverable keys that can be copied to any machine.

== Step 3: Register Your Public Key ==

<div style="border: 3px solid #17a2b8; padding: 15px; background-color: #d1ecf1; margin: 10px 0;">
'''FIDO2 SSH Keys work immediately without 2-factor authentication!'''

Unlike regular SSH keys, FIDO2 keys do not need to be "unlocked" with username/password.
They work as soon as you register them as '''Interactive Keys'''.
</div>

'''For bwUniCluster 3.0 and bwForCluster NEMO 2:'''

1. View your public key:
<pre>
cat ~/.ssh/id_ed25519_sk_bwunicluster.pub
</pre>

2. Copy the complete output (starts with <code>sk-ssh-ed25519@openssh.com</code>)

3. Follow the [[Registration/SSH#Adding_a_new_SSH_Key|SSH Key Registration Guide]] to:
* Add your public key in bwIDM/bwServices
* Register it as an '''Interactive Key''' for your cluster
* Connect immediately - no 2FA unlock needed!

== Step 4: Connect with Your Yubikey ==

After registering your key as an '''Interactive Key''' in bwIDM:

<pre>
ssh -i ~/.ssh/id_ed25519_sk_bwunicluster your_username@bwunicluster.scc.kit.edu
</pre>

'''What to expect:'''
1. Your Yubikey will blink
2. Touch it to authenticate
3. You're logged in - no password needed!

'''Tip:''' Add your key to <code>~/.ssh/config</code> to avoid typing the <code>-i</code> option every time:

<pre>
Host bwuni
HostName bwunicluster.scc.kit.edu
User fr_user1234
IdentityFile ~/.ssh/id_ed25519_sk_bwunicluster

Host nemo2
HostName login.nemo.uni-freiburg.de
User fr_user1234
IdentityFile ~/.ssh/id_ed25519_sk_nemo2
</pre>

Then simply connect with: <code>ssh bwuni</code> or <code>ssh nemo2</code>

See [https://gitlab.rz.uni-freiburg.de/escience-public/yubikey#ssh-configuration-for-fido2-secured-ssh-keys SSH config examples] for more advanced configurations.

= Advanced Topics =

== Using Multiple Yubikeys (Optional) ==

'''Why?''' Having backup Yubikeys adds convenience - you can switch between devices without reconfiguring.

'''Note:''' If you lose your Yubikey, you can still access the cluster using regular SSH keys (with 2FA unlock) or password login with TOTP.

'''Setup:'''

1. Create a key on your first Yubikey: <code>id_ed25519_sk_key1</code>
2. Create another key on your second Yubikey: <code>id_ed25519_sk_key2</code>
3. Register '''both public keys''' in bwIDM as Interactive Keys
4. Configure <code>~/.ssh/config</code> with multiple identities:

<pre>
Host bwuni nemo2
User fr_user1234
IdentityFile ~/.ssh/id_ed25519_sk_key1
IdentityFile ~/.ssh/id_ed25519_sk_key2
</pre>

SSH will automatically try each key until it finds one with a connected Yubikey.

= Troubleshooting =

'''Common Issues:'''

* '''Key not working?'''
** Make sure your Yubikey is plugged in
** Try unplugging and re-plugging the Yubikey
** Check if the key is blinking (requires touch)

* '''Command not found?'''
** Install <code>yubikey-manager</code>:
*** Debian/Ubuntu: <code>sudo apt install yubikey-manager</code>
*** RHEL/Rocky/Alma: <code>sudo dnf install yubikey-manager</code>
** Install <code>openssh-client</code> if missing

* '''Old OpenSSH?'''
** Check version: <code>ssh -V</code>
** Update to version 8.2 or newer
** On older systems, consider using a newer OpenSSH version or upgrade your OS

* '''"invalid format" or "unknown key type"?'''
** Your OpenSSH version is too old (needs 8.2+)
** The cluster doesn't support FIDO2 keys (only bwUniCluster 3.0 and NEMO 2)

* '''PIN locked after too many attempts?'''
** Reset FIDO2 PIN: See [https://gitlab.rz.uni-freiburg.de/escience-public/yubikey#configure-yubikey-for-fido2 PIN reset guide]

'''Need more help?''' See the [https://gitlab.rz.uni-freiburg.de/escience-public/yubikey#troubleshooting complete troubleshooting guide].

= Learn More =

'''Cluster-Specific Documentation:'''
* [[Registration/SSH|Complete SSH Key Registration Guide for bwUniCluster/bwForCluster]]

'''Yubikey & FIDO2 SSH Documentation:'''
* [https://gitlab.rz.uni-freiburg.de/escience-public/yubikey Comprehensive Yubikey Guide] - Setup, configuration, troubleshooting
** [https://gitlab.rz.uni-freiburg.de/escience-public/yubikey#creating-discoverable-ssh-keys Resident Keys] - Store keys directly on Yubikey
** [https://gitlab.rz.uni-freiburg.de/escience-public/yubikey#ssh-configuration-for-fido2-secured-ssh-keys SSH Config Examples] - Automate your SSH connections
** [https://gitlab.rz.uni-freiburg.de/escience-public/yubikey#yubikey-best-practices Backup Strategies] - Multi-key and recovery setups

'''Official Yubico Documentation:'''
* [https://developers.yubico.com/SSH/Securing_SSH_with_FIDO2.html Securing SSH with FIDO2] - Technical deep dive

Login Problems

2025-11-20T15:23:35Z

H Winkhardt: Fixed Redlinks

This page lists common causes for problems with logging in as well as listing information you should provide when writing a ticket

== Cannot Connect to Cluster ==

Please note that you need to come from internet hosts inside your university network. Either by connecting from a machine located on the campus or by using a VPN to virtually become part of the university network. Also note, that unauthenticated wireless connection options of a university may be considerd as out-of-network.

== Missing Entitlement ==

This can happen because

* you have not completed User Access Step A for the cluster you want to use:     → [[Registration/bwForCluster/Entitlement|bwForCluster]] or     → [[Registration/bwUniCluster/Entitlement|bwUniCluster]]
* you have left University
* there is a malfunction of the IDM of your University

You can check your Entitlements as shown on the Entitlement page:

    → [[Registration/bwForCluster/Entitlement#Check_your_Entitlements|Check your Entitlements]]

In short: go to https://login.bwidm.de/ and select Index(Übersicht) → Personal Data, then select the tab "Shibboleth" and search for Cluster on that page to find <pre>http://bwidm.de/entitlement/bwUniCluster</pre> or <pre>http://bwidm.de/entitlement/bwForCluster</pre> (these are the names of the Entitlement, not links).

In case your Entitlement is missing, please contact the support of the university you are a member of. The bw Clusters rely on your university to provide the correct information about your ongoing affiliation and cannot solve the problem.

=== Missing bw Group ===

You can also check if your compute project group still exists by selecting the tab "Groups" on the same page as described above.

== Lost TOTP Secret ==

Please avoid this problem by creating a backup TAN list and keep it in a safe place - e.g. in printed form in a folder in your office.

To create a backup tan list, use the registration server used by your cluster. You will be asked a current TOTP password from your device.

    → see [[Registration/2FA#Token_Management]] 

Please contact us via authenticated login in the [[bwSupportPortal|Support Portal]] and not via Email.

We will only be able to deactivate your current secret, which will enable you to create a new one without having to enter a TOTP.

For the next steps, follow the relevant wiki page: https://wiki.bwhpc.de/e/Registration/2FA

In short:

a) Delete the old token, reload the web page and verify that all tokens have been deleted.
b) Create a new smartphone token and test it (if the QR code is missing check your browser's add-ons/ad-blockers)
c) Create a backup Tan List, print it on paper and keep in a safe place for cases like this

Note: Users from KIT Karlsruhe who are using a cluster which uses https://login.bwidm.de/ to register the service will need to contact KIT support to have the TOTP reset

== TOTP Secret Disabled ==

After a certain number of unsuccessful TOTP attempts, your TOTP will have accumulated too many errors if no successful login resets this.

In this case, you will have to open a ticket at the [[bwSupportPortal|Support Portal]] for us to re-enable the TOTP.

Some programs used to open a ssh connection with some configuration options cause the client to keep trying to re-login in the background but fails because this mechanism cannot account for entering a TOTP. Please check configuration of your software for such behavior.

== Forgot Password ==

You can re-set the ssh password on the registration server used by your cluster

    → list or registration servers also at: [[Registration/2FA#Token_Management]] 

== ZAS Permission Missing (bwForClusters only) ==

This can have one of the following reasons:

* You have not yet successfully become a member of your compute project group
* Your account has been disabled by an RV owner or manager and no longer belongs to the compute project
* The compute project you are a member of has not been renewed and has expired
* There is a ZAS malfunction

You can see the status of your own compute project or a project membership at https://zas.bwhpc.de/en/zas_info_bwforcluster.php via the points "RV collaboration" or "My RVs".

== Writing a Ticket ==

Before writing a ticket to the [[bwSupportPortal|Support Portal]], please check if your Entitlement is missing - in which case you should contact your own university support.

Please provide the following information:

* which cluster you have tried to access
* your username
* your eppn

Login Problems

2025-11-20T15:22:37Z

H Winkhardt: Fixed Redlink

This page lists common causes for problems with logging in as well as listing information you should provide when writing a ticket

== Cannot Connect to Cluster ==

Please note that you need to come from internet hosts inside your university network. Either by connecting from a machine located on the campus or by using a VPN to virtually become part of the university network. Also note, that unauthenticated wireless connection options of a university may be considerd as out-of-network.

== Missing Entitlement ==

This can happen because

* you have not completed User Access Step A for the cluster you want to use:     → [[Registration/bwForCluster/Entitlement|bwForCluster]] or     → [[Registration/bwUniCluster/Entitlement|bwUniCluster]]
* you have left University
* there is a malfunction of the IDM of your University

You can check your Entitlements as shown on the Entitlement page:

    → [[Registration/bwForCluster/Entitlement#Check_your_Entitlements|Check your Entitlements]]

In short: go to https://login.bwidm.de/ and select Index(Übersicht) → Personal Data, then select the tab "Shibboleth" and search for Cluster on that page to find <pre>http://bwidm.de/entitlement/bwUniCluster</pre> or <pre>http://bwidm.de/entitlement/bwForCluster</pre> (these are the names of the Entitlement, not links).

In case your Entitlement is missing, please contact the support of the university you are a member of. The bw Clusters rely on your university to provide the correct information about your ongoing affiliation and cannot solve the problem.

=== Missing bw Group ===

You can also check if your compute project group still exists by selecting the tab "Groups" on the same page as described above.

== Lost TOTP Secret ==

Please avoid this problem by creating a backup TAN list and keep it in a safe place - e.g. in printed form in a folder in your office.

To create a backup tan list, use the registration server used by your cluster. You will be asked a current TOTP password from your device.

    → see [[Registration/2FA#Token_Management]] 

Please contact us via authenticated login in the [[Support Portal]] and not via Email.

We will only be able to deactivate your current secret, which will enable you to create a new one without having to enter a TOTP.

For the next steps, follow the relevant wiki page: https://wiki.bwhpc.de/e/Registration/2FA

In short:

a) Delete the old token, reload the web page and verify that all tokens have been deleted.
b) Create a new smartphone token and test it (if the QR code is missing check your browser's add-ons/ad-blockers)
c) Create a backup Tan List, print it on paper and keep in a safe place for cases like this

Note: Users from KIT Karlsruhe who are using a cluster which uses https://login.bwidm.de/ to register the service will need to contact KIT support to have the TOTP reset

== TOTP Secret Disabled ==

After a certain number of unsuccessful TOTP attempts, your TOTP will have accumulated too many errors if no successful login resets this.

In this case, you will have to open a ticket at the [[Support Portal]] for us to re-enable the TOTP.

Some programs used to open a ssh connection with some configuration options cause the client to keep trying to re-login in the background but fails because this mechanism cannot account for entering a TOTP. Please check configuration of your software for such behavior.

== Forgot Password ==

You can re-set the ssh password on the registration server used by your cluster

    → list or registration servers also at: [[Registration/2FA#Token_Management]] 

== ZAS Permission Missing (bwForClusters only) ==

This can have one of the following reasons:

* You have not yet successfully become a member of your compute project group
* Your account has been disabled by an RV owner or manager and no longer belongs to the compute project
* The compute project you are a member of has not been renewed and has expired
* There is a ZAS malfunction

You can see the status of your own compute project or a project membership at https://zas.bwhpc.de/en/zas_info_bwforcluster.php via the points "RV collaboration" or "My RVs".

== Writing a Ticket ==

Before writing a ticket to the [[bwSupportPortal]], please check if your Entitlement is missing - in which case you should contact your own university support.

Please provide the following information:

* which cluster you have tried to access
* your username
* your eppn

Development/Containers

2025-11-12T13:56:55Z

H Winkhardt: Nemo1 -> Nemo2

The bwHPC clusters mostly provide Singularity as a container runtime environment.

To learn more about the usage of Singularity, visit the corresponding cluster page.

* [[BwUniCluster3.0/Containers]]
* [[JUSTUS2/Software/Singularity]]
* [[Helix/Software/Singularity]]
* [[NEMO2/Software/Singularity_Containers]]

Development/Containers

2025-11-12T13:55:15Z

H Winkhardt: UC2 -> UC3

OpenFoam

2025-11-12T13:52:51Z

H Winkhardt: Shortened redirect chain

#REDIRECT [[BwUniCluster3.0/Software/OpenFoam]]

Registration/Login/Username

2025-11-12T13:51:11Z

H Winkhardt: Removed outdated references to UC2

__FORCETOC__

= Login Username =

All members of universities and colleges in Baden-Württemberg can use the bwHPC resources.
Prefixes are used to ensure that the usernames assigned by the home organization are unique within the state.
The username for the bwHPC clusters is the same as the one assigned by the university or college, but it is prefixed with two letters for the home institution.

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
Starting with '''bwUniCluster 3.0''', KIT users have the '''prefix ka''' on all systems.
|}

There are two ways to find out your username for the bwForCluster and the bwUniCluster:

* [[#Get_your_Username_by_adding_a_Prefix_to_your_local_Account_Name|Get your Username by adding a Prefix to your local Account Name]]
* [[#Find_out_your_Username_by_visiting_the_Registration_Service|Find out your Username by visiting the Registration Service]]

== Get your Username by adding a Prefix to your local Account Name ==

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
If your university is not listed in the [https://www.bwidm.de/hochschulen.php '''bwIDM Membership Table'''], please ask your university to join bwIDM or update the table.
|}

If you want to use a '''bwForCluster''' or the '''bwUniCluster''' you need to add a prefix to your local username.
The username is constructed as follows (see [[#Examples|Examples]] below):

<code><prefix>_<local username></code>

=== Prefix for Universities of Applied Sciences ===

For universities see table below, users from the [[Registration/HAW|HAW BW e.V.]] or a university of applied sciences should use this table updated by bwIDM as a guideline:
* [https://www.bwidm.de/hochschulen.php '''bwIDM Membership Table''']

=== Prefix for Universities ===

{| class="wikitable"
|-
!style="width:280px" | User from University
!style="width:60px" | Prefix
!style="width:430px" | Username
|-
| Universität Freiburg
| fr
| colspan="2" align="center"| fr_''<username>''
|-
| Universität Heidelberg
| hd
| colspan="2" align="center"| hd_''<username>''
|-
| Universität Hohenheim
| ho
| colspan="2" align="center"| ho_''<username>''
|-
| Karlsruhe Institute of Technology (KIT)
| ka
| align="center"| ka_''<username>''
|-
| Universität Konstanz
| kn
| colspan="2" align="center"| kn_''<username>''
|-
| Universität Mannheim
| ma
| colspan="2" align="center"| ma_''<username>''
|-
| Universität Stuttgart
| st
| colspan="2" align="center"| st_''<username>''
|-
| Universität Tübingen
| tu
| colspan="2" align="center"| tu_''<username>''
|-
| Universität Ulm
| ul
| colspan="2" align="center"| ul_''<username>''
|}

=== Examples ===

* If you are a member of the University of Konstanz and your local username is <code>ab1234</code>, your username on any bwHPC cluster is <code>kn_ab1234</code>.
* If your local username for the university is <code>vwxyz1234</code> and you are a user of the University of Freiburg, your username on any bwHPC cluster is <code>fr_vwxyz1234</code>.
* If you are from Aalen University and your username is <code>xyzs12342</code>, your username for any bwHPC cluster is <code>aa_xyzs12342</code>.
* If your KIT username is <code>ab1234</code>, your username on any bwHPC cluster is <code>ka_ab1234</code>.

== Find out your Username by visiting the Registration Service ==

Alternatively, you can log in to the registration service and verify your username online.
To do this, follow the next steps:

1. Select the cluster you want to know your username for: → [https://login.bwidm.de '''bwUniCluster 3.0'''] → [https://bwservices.uni-tuebingen.de '''BinAC'''] → [https://login.bwidm.de '''JUSTUS 2'''] → [https://bwservices.uni-heidelberg.de '''Helix'''] → [https://login.bwidm.de '''NEMO2''']

2. Select your home organization from the list and click '''Proceed'''.

3. Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button.

4. You will be redirected back to the registration website.

5. Find the cluster entry and select '''Registry Info'''.
[[File:BwIDM-pw.png|center|frame|Check Registry Info.]]

6. Depending on the registration service, you will see one of the following entries. See '''Username for login''' or '''localUid''' for your username.
{| style="width:50%;" align="center"
|-
!style="width:50%;" align="center"|[[File:BwIDM-user.png|center|thumb|300px|Username for login (newer registration services).]]
!style="width:50%;" align="center"|[[File:BwIDM-uid.png|center|thumb|300px|Username for login in ''localUid'' (older registration services).]]
|-
|}

Development/Python

2025-10-29T14:11:48Z

H Winkhardt: typo

== Introduction ==
Python is a versatile, easy-to-learn, interpreted programming language. It offers a wide range of libraries for scientific tasks and visualization. Python counts to the best languages for machine learning. Python can be used in particular as an open source alternative for tasks that have usually been used for Matlab.

== Installation and Versions ==
Python is available on all systems. With <code>python --version</code> you can see the currently active default python version. In general, you can choose from various types of python installations:
* '''System python:''' This python version comes together with the operating system and is available upon login to the cluster. Other python versions might be installed along with it. All versions can be seen with
*: <code>ls /usr/bin/python[0-9].*[0-9] | sort -V | cut -d"/" -f4 | xargs</code>
*: They can change over time. You can access a specific Python version by specifying the version in the Python command.
* '''[[Environment_Modules | Software module]]:''' Available versions can be identified via
*: <syntaxhighlight lang="bash">
module avail devel/python
</syntaxhighlight>
*: The provided software modules on the clusters are optimized for making efficient use of the cluster resources. Please use them whenever possible.
* '''Python distributions and virtual environments:''' By using python distributions such as Anaconda, you can easily install the needed python version into a virtual environment. For the use of conda on bwHPC clusters, please refer to [[Development/Conda|Conda]]. Alternatively, you can use more python specific tools for installing python. Some options are listed in [[#Virtual Environments and Package Management | Virtual Environments and Package Management]]
* '''[[Development/Containers | Container]]:''' Containers can contain their own python installation. Keep this in mind when you are working with containers provided by others.

== Running Python Code ==

There are three ways to run Python commands:

* Within a '''terminal''' by executing the comand <code>python</code>. This starts a Python shell, where all commands are evaluated by the Python interpreter.
* Within a '''script''' (file ends with ''.py'' and can be run with <code>python myProgram.py</code>)
* Within a '''notebook''' (file ends with .ipynb). You can use other programming languages and markdown within a notebook next to your python code. Besides software development itself, teaching, prototyping and visualization are good use cases for notebooks as well.

== Development Environments ==

Development Environments are usually more comfortable than running code directly from the shell. Some common options are:

* [[Jupyter]]
* [[Development/VS_Code | VS Code]]
* PyCharm

== Virtual Environments and Package Management ==

Packages contain a set of functions that offer additional functionality. A package can be installed by using a package manager. Virtual environments prevent conflicts between different Python packages by using separate installation directories.

At least one virtual environment should be defined per project. This way, it is clear which packages are needed by a specific project. All virtual environments allow to save the corresponding packages with their specific version numbers to a file. This allows to reinstall them in another place and therefore improves the reproducibility of projects. Furthermore, it makes finding and removing packages, that aren't needed anymore, easier.

=== Overview ===
The following table provides an overview of common tools in the field of virtual environments and package management. The main differences between the various options are highlighted. After deciding on a specific tool, it can be installed by following the given link.
To make it short, if you plan to use...
* ...Python only:
** venv is the most basic option. Other options build upon it.
** poetry is widely used and offers a broad set of functionality
** uv is the latest option and much faster than poetry while offering the same (or more) functionality.
* ...Python + Conda:
** We wouldn't recommend it but if you want to use conda only: Install the conda packages first into the conda environment and afterwards the python packages. Otherwise, problems might arise.
** For a faster and more up to date solution, choose pixi.

{| class="wikitable"
|- style="text-align:center;"
! Tool
! Description
! Can install python versions
! Installs packages from PyPI
! Installs packages from conda
! Dependency Resolver
! Dependency Management
! Creates Virtual Environments
! Supports building, packaging and publishing code (to PyPI)
|-
| pyenv
| Manages python versions on your system.
| yes
| no
| no
| no
| no
| no
| no
|-
| pip
| For installing python packages.
| no
| yes
| no
| yes
| no
| no
| no
|-
| venv
| For installing and managing python packages. Part of Python's standard library.
| no
| no
| no
| yes
| yes
| yes
| no
|-
| poetry
| For installing and managing python packages. Install it with pipx.
| no
| yes
| no
| yes
| yes
| yes
| yes
|-
| pipx
| For installing and running python applications (like poetry) globally while having them in isolated environments. It is useful for keeping applications globally available and at the same time separated in their own virtual environments. Use it when the installation instructions of an application offer you this way of installation.
| no
| no
| no
| yes
| yes
| yes (only for single applications)
| no
|-
| uv
| Replaces poetry, pyenv, pip etc. and is very fast (https://www.loopwerk.io/articles/2025/uv-keeps-getting-better/)
| yes
| yes
| no
| yes
| yes
| yes
| yes
|-
| pixi
| For installing and managing python as well as conda packages. Uses uv in the background
| yes
| yes
| yes
| yes
| yes
| yes
| yes
|}

=== Pip ===

The standard package manager under Python is <code>pip</code>. It can be used to install, update and delete packages. Pip can be called directly or via <code>python -m pip</code>. The standard repository from which packages are obtained is PyPI (https://pypi.org/). When a package depends on others, they are automatically installed as well.
In the following, the most common pip commands are shown exemplarily. Packages should always be installed within virtual environments to avoid conflicting installations. If you decide to not use a virtual environment, the install commands have to be accomplished by a <code>--user</code> flag or controlled via environment variables.

Installation of packages 
<syntaxhighlight lang="python">
pip install pandas # Installs the latest compatible version of pandas and its required dependencies
pip install pandas=1.5.3 # Installs exact version
pip install pandas>=1.5.3 # Installs version newer or equal to 1.5.3
</syntaxhighlight>

The packages from PyPI usually consist of precompiled libraries. However, <code>pip</code> is also capable of creating packages from source code. However, this may require the C/C++ compiler and other dependencies needed to build the libraries. In the example, pip obtains the source code of matplotlib from github.com, installs its dependencies, compiles the library and installs it:
<syntaxhighlight lang="python">
pip install git+https://github.com/matplotlib/matplotlib
</syntaxhighlight>
Upgrade packages 
<syntaxhighlight lang="python">
pip install --upgrade pandas # Updates the library if update is available
</syntaxhighlight>
Removing packages 
<syntaxhighlight lang="python">
pip uninstall pandas # Removes pandas
</syntaxhighlight>
Show packages 
<syntaxhighlight lang="python">
pip list # Shows the installed packages
pip freeze # Shows the installed packages and their versions
</syntaxhighlight>
Save State 
To allow for reproducibility it is important to provide information about the full list of packages and their exact versions [https://pip.pypa.io/en/stable/topics/repeatable-installs/ see version pinning].
<syntaxhighlight lang="python">
pip freeze > requirements.txt # redirect package and version information to a textfile
pip install -r requirements.txt # Installs all packages that are listed in the file
</syntaxhighlight>

=== Venv ===

The module <code>venv</code> enables the creation of a virtual environment and is a standard component of Python. Creating a <code>venv</code> means that a folder is created which contains a separate copy of the Python binary file as well as <code>pip</code> and <code>setuptools</code>. After activating the <code>venv</code>, the binary file in this folder is used when <code>python</code> or <code>pip</code> is called. This folder is also the installation target for other Python packages.

==== Creation ====

Create, activate, install software, deactivate:
<syntaxhighlight lang="bash">
python3.11 -m venv myEnv # Create venv
source myEnv/bin/activate # Activate venv
pip install --upgrade pip # Update of the venv-local pip
pip install <list of packages> # Install packages/modules
deactivate
</syntaxhighlight>

Install list of software:
<syntaxhighlight lang="bash">
pip install -r requirements.txt # Install packages/modules
</syntaxhighlight>

==== Usage ====
To use the virtual environment after all dependencies have been installed there, it is sufficient to simply activate it:
<syntaxhighlight lang="bash">
source myEnv/bin/activate # Activate venv
</syntaxhighlight>

With <code>venv</code> activated the terminal prompt will reflect that accordingly:
<syntaxhighlight lang="bash">
(myEnv) $
</syntaxhighlight>
It is no longer necessary to specify the Python version, the simple command <code>python</code> starts the Python version that was used to create the virtual environment. 
You can check, which Python version is in use via:
<syntaxhighlight lang="bash">
(myEnv) $ which python
</path/to/project>/myEnv/bin/python
</syntaxhighlight>

=== Poetry ===

==== Creation ====
When you want to create a virtual environment for an already existing project, you can go to the top directory and run
<syntaxhighlight lang="bash">
poetry init # Create virtual environment
</syntaxhighlight>
Otherwise start with the demo project:
<syntaxhighlight lang="bash">
poetry new poetry-demo
</syntaxhighlight>
You can set the allowed python versions in the pyproject.toml. To switch between python installations on your sytem, you can use
<syntaxhighlight lang="bash">
poetry env use python3.11
</syntaxhighlight>

==== Usage ====

Install and update packages
<syntaxhighlight lang="bash">
poetry install <package_name>
poetry update # Update to latest versions of packages
</syntaxhighlight>

To execute something within the virtual environment:
<syntaxhighlight lang="bash">
poetry run <command>
</syntaxhighlight>

Helpful links:
* [https://python-poetry.org/docs/managing-environments/#activating-the-environment Activate Environment]

Helpful commands
<syntaxhighlight lang="bash">
poetry env info # show environment information
poetry env list # list all virtual environments associated with the current project
poetry env list --full-path
poetry env remove # delete virtual environment
</syntaxhighlight>

=== uv ===

==== Creation ====

With uv you can create two types of new projects. Either an application (default) or a library.

→ '''[https://docs.astral.sh/uv/concepts/projects/ projects]'''
<syntaxhighlight lang="bash">
uv init <application_name>
# If you want to create a python package:
uv init --package <application_name>
</syntaxhighlight>

When you have an existing project, you can create a virtual environment at .venv with a specific python version as follows:
<syntaxhighlight lang="bash">
uv python list # get available versions
uv venv --python 3.13
</syntaxhighlight>
A specific name or path can be given instead of "venv" as well. If the python version is not available on the system, uv downloads it.

==== Usage ====
The currently active environment is saved in the environment variable <code>VIRTUAL_ENV</code>. The environment can be activated and deactivated as follows:
<syntaxhighlight lang="bash">
source .venv/bin/activate
deactivate
</syntaxhighlight>
If you activated the virtual environment, you can install packagages as usual. Otherwise, you need to add "uv" before the command to run it within the virtual environment:
<syntaxhighlight lang="bash">
uv add <package_name>
</syntaxhighlight>
Note: Additionally, there is a pip interface for uv but it should only be used in fitting edge cases.

To execute a python file within the virtual environment:
<syntaxhighlight lang="bash">
uv run <file.py>
</syntaxhighlight>

If one project has multiple different groups of packages that are used, it could make sense to look into uv workspaces:

→ '''[https://docs.astral.sh/uv/concepts/projects/workspaces/ workspaces]'''

=== Pixi ===

==== Creation ====

With uv you can create two types of new projects. Either an application (default) or a library.

→ '''[https://pixi.sh/latest/python/tutorial/ Python Tutorial]'''
<syntaxhighlight lang="bash">
pixi init <project_name> --format pyproject
</syntaxhighlight>

==== Usage ====
Add a new package to the environment:
<syntaxhighlight lang="bash">
# Get it from conda:
pixi add <package_name>
# Get it from PyPi
pixi add <package_name> --pypi
</syntaxhighlight>

To execute a command within the virtual environment:
<syntaxhighlight lang="bash">
pixi run <command>
</syntaxhighlight>

== Workflow Management Systems ==

There are a lot of different workflow management systems. One example is [https://snakemake.readthedocs.io/en/stable/index.html Snakemake] which is often used in the field of biology/medicine. Two use cases for snakemake workflows would be:
* Use an existing workflow to analyze data.
* Create your own workflow to make your research more reproducible.
Leaving the python world, a common choice for bioinformaticians would be Nextflow as there is a strong community that focuses on standardisation. Those more standardised pipelines can be found at the [https://nf-co.re/pipelines/ nf-core website].
In other research fields, other workflow management systems might be more common.

=== Snakemake ===

Some recommendations:
* Start the Snakemake pipeline within a batch job.
* Define rules that don't need much time as "localrule".
* Work with rule grouping as described here: https://hpc.nih.gov/apps/snakemake.html#group .
* Set the following snakemake profile options:
*: <syntaxhighlight lang="bash">
max-jobs-per-second: 1
max-status-checks-per-second: 1
jobs: 400
</syntaxhighlight>

== File Formats ==

While providing final results in a human readable format like csv it can be more beneficial to store intermediate results in file formats that are faster to read from and write to. Some common options are the following:
* '''Pickle'''
*: Python specific. Very fast in the case of storing numerical dataframes. Not recommended for longterm storage as changes in library versions etc. can lead to unreadable pickle files.
* '''Feather'''
*: Good option in regards to read/write time.
* '''Parquet'''
*: Good for big data files. Reduces storage need.
*: Used via libraries like PyArrow and FastParquet. The latter is a version with less features but therefore better performance when used with small to medium sized datasets. [See the comparison [https://dev.to/alexmercedcoder/all-about-parquet-part-08-reading-and-writing-parquet-files-in-python-338d here]]
*: Significantly slower when storing a mixed dataframe in contrast to a numerical dataframe.
* '''hdf5'''
* '''Database'''

== Best Practices ==
* Always use virtual environments! Use one environment per project.
* If a new or existing Python project is to be created or used, the following procedure is recommended:
# Version the Python source files and the <code>requirements.txt</code> file with a version control system, e.g. git. Exclude unnecessary folders and files like for example <code>venv</code> via an entry in the ignore file, for example <code>.gitignore</code>.
# Create and activate a virtual environment.
# Use specialized number crunching python modules (e.g. numpy and scipy), don't use plain python for serious caluclations
## check if optimized compiled modules are available on the cluster (numpy, scipy)
# Update pip <code>pip install --upgrade pip</code>.
# Install all required packages via the <code>requirements.txt</code> file in the case of venv. Or by using the corresponding command of your chosen tool.
* List or dictionary comprehensions are to be prefered over loops as they are in general faster.
* Be aware of the differences between references, shallow and deep copies.
* Do not parallelize by hand but use libraries were possible (Dask, ...).

Status

2025-10-15T10:38:23Z

H Winkhardt: ZAS wieder da

= bwHPC Cluster and Service Status Page =

== Current Status ==



{| style=" background:#B8FFB8; width:100%;"
| style="padding:8px; background:#8AFF8A; font-size:120%; font-weight:bold; text-align:left" | Status
|-
|
Normal operation of all systems.
|}







== Old Messages ==

15.10.2025: central application page (ZAS) [https://zas.bwhpc.de/] currently down
* Renewal and application of new projects (Rechenvorhaben/RV) and registration of new RV members not possible.
* Filling out the bwUniCluster3.0 questionnaire not possible.
* Login and compute activities are '''not''' affected

09.10.2025: central application page (ZAS) [https://zas.bwhpc.de/] currently down
* Renewal and application of new projects (Rechenvorhaben/RV) and registration of new RV members not possible.
* Filling out the bwUniCluster3.0 questionnaire not possible.
* 10.10.: ongoing issue
* Login and compute activities are '''not''' affected
* 2025-10-10T16: Some sites report normal operation. Identity providers need to update DFN information, expected within 24h.

Main Page

2025-10-15T10:37:22Z

H Winkhardt: ZAS wieder da

'''Welcome to the bwHPC Wiki.'''

bwHPC represents services and resources in the State of '''B'''aden-'''W'''ürttemberg, Germany, for High Performance Computing ('''HPC'''), Data Intensive Computing ('''DIC''') and Large Scale Scientific Data Management ('''LS2DM''').

The main bwHPC web page is at '''[https://www.bwhpc.de/ https://www.bwhpc.de/]'''.

Many topics depend on the cluster system you use.
First choose the cluster you use, then select the correct topic.




{| style=" background:#B8FFB8; width:100%;"
| style="padding:8px; background:#8AFF8A; font-size:120%; font-weight:bold; text-align:left" | [[Status|Maintenances and Outages]] (Currently: 0)
|}









{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Courses / eLearning
|-
|
* [https://training.bwhpc.de/ eLearning and Online Courses]
* [https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC Introduction to Linux in HPC (external resource)]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | Need Access to a Cluster?
|-
|
* [[When to use a HPC Cluster]]
* [[Running Calculations]]
* [[Registration]]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | HPC System Specific Documentation
|-
|
bwHPC encompasses several HPC compute clusters at different universities in Baden-Württemberg. Each cluster is dedicated to [https://www.bwhpc.de/bwhpc-domains.php specific research domains].

Documentation differs between compute clusters, please see cluster specific overview pages:
{|
|-
| style="padding:5px; width:30%" | [[BwUniCluster3.0|bwUniCluster 3.0]]
| style="padding-left:20px;" | General Purpose, Teaching
|-
| style="padding:5px; width:30%" | [[:JUSTUS2| bwForCluster JUSTUS 2]]
| style="padding-left:20px;" | Theoretical Chemistry, Condensed Matter Physics, and Quantum Sciences
|-
| style="padding:5px; width:30%" | [[Helix|bwForCluster Helix]]
| style="padding-left:20px;" | Structural and Systems Biology, Medical Science, Soft Matter, Computational Humanities, and Mathematics and Computer Science
|-
| style="padding:5px; width:30%" | [[NEMO2|bwForCluster NEMO 2]]
| style="padding-left:20px;" | Neurosciences, Particle Physics, Materials Science, and Microsystems Engineering
|-
| style="padding:5px; width:30%" | [[BinAC|bwForCluster BinAC]]
| style="padding-left:20px;" | Bioinformatics, Geosciences and Astrophysics
|-
| style="padding:5px; width:30%" | [[BinAC2|bwForCluster BinAC 2]]
| style="padding-left:20px;" | Bioinformatics, Astrophysics, Geosciences, Pharmacy, and Medical Informatics
|}
|-
|bwHPC Clusters: [https://www.bwhpc.de/cluster.php operational status]
Further Compute Clusters in Baden-Württemberg (separate access policies):
* Datenanalyse Cluster der Hochschulen (DACHS): [[DACHS | Datenanalyse Cluster der Hochschulen (DACHS)]]
* bwHPC tier 1: [https://kb.hlrs.de/platforms/index.php/Hunter_(HPE) Hunter] ([https://www.hlrs.de/apply-for-computing-time getting access])
* bwHPC tier 2: [https://www.nhr.kit.edu/userdocs/horeka HoreKa] ([https://www.nhr.kit.edu/userdocs/horeka/projects/ getting access])
|}

{| style="background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | Documentation valid for all Clusters
|-
|
* [[Environment Modules| Software Modules]] and software documentation explained
* [https://www.bwhpc.de/software.html List of Software] on all clusters
* [[Development| Software Development and Parallel Programming]]
* [[Energy Efficient Cluster Usage]]
* [[HPC Glossary]]

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | Scientific Data Storage
|-
|
For user guides of the scientific data storage services:
* [[SDS@hd]]
* [https://www.rda.kit.edu/english bwDataArchive]
* [https://uni-tuebingen.de/einrichtungen/zentrum-fuer-datenverarbeitung/projekte/laufende-projekte/bwsfs bwSFS]
Associated, but local scientific storage services are:
* [https://wiki.scc.kit.edu/lsdf/index.php/Category:LSDF_Online_Storage LSDF Online Storage] (only for KIT and KIT partners)
|}

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | Data Management
|-
|
* [[Data Transfer|Data Transfer]]
* [https://www.forschungsdaten.org/index.php/FDM-Kontakte#Deutschland Research Data Management (RDM)] contact persons
* [https://www.forschungsdaten.info Portal for Research Data Management] (Forschungsdaten.info)
|}
{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Support
|-
|
* [[BwSupportPortal|Submit a Ticket in our Support Portal]]
Support is provided by the [https://www.bwhpc.de/teams.php bwHPC Competence Centers]:
|}
{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Acknowledgement
|-
|
* Please [[Acknowledgement|acknowledge]] our resources in your publications.
|}

Main Page

2025-10-15T08:24:06Z

H Winkhardt: ZAS wieder down

'''Welcome to the bwHPC Wiki.'''

bwHPC represents services and resources in the State of '''B'''aden-'''W'''ürttemberg, Germany, for High Performance Computing ('''HPC'''), Data Intensive Computing ('''DIC''') and Large Scale Scientific Data Management ('''LS2DM''').

The main bwHPC web page is at '''[https://www.bwhpc.de/ https://www.bwhpc.de/]'''.

Many topics depend on the cluster system you use.
First choose the cluster you use, then select the correct topic.








{| style=" background:#FFD28A; width:100%;"
| style="padding:8px; background:#FFC05C; font-size:120%; font-weight:bold; text-align:left" | [[Status|Maintenances and Outages]] (Currently: 1)
|}






{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Courses / eLearning
|-
|
* [https://training.bwhpc.de/ eLearning and Online Courses]
* [https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC Introduction to Linux in HPC (external resource)]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | Need Access to a Cluster?
|-
|
* [[When to use a HPC Cluster]]
* [[Running Calculations]]
* [[Registration]]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | HPC System Specific Documentation
|-
|
bwHPC encompasses several HPC compute clusters at different universities in Baden-Württemberg. Each cluster is dedicated to [https://www.bwhpc.de/bwhpc-domains.php specific research domains].

Documentation differs between compute clusters, please see cluster specific overview pages:
{|
|-
| style="padding:5px; width:30%" | [[BwUniCluster3.0|bwUniCluster 3.0]]
| style="padding-left:20px;" | General Purpose, Teaching
|-
| style="padding:5px; width:30%" | [[:JUSTUS2| bwForCluster JUSTUS 2]]
| style="padding-left:20px;" | Theoretical Chemistry, Condensed Matter Physics, and Quantum Sciences
|-
| style="padding:5px; width:30%" | [[Helix|bwForCluster Helix]]
| style="padding-left:20px;" | Structural and Systems Biology, Medical Science, Soft Matter, Computational Humanities, and Mathematics and Computer Science
|-
| style="padding:5px; width:30%" | [[NEMO2|bwForCluster NEMO 2]]
| style="padding-left:20px;" | Neurosciences, Particle Physics, Materials Science, and Microsystems Engineering
|-
| style="padding:5px; width:30%" | [[BinAC|bwForCluster BinAC]]
| style="padding-left:20px;" | Bioinformatics, Geosciences and Astrophysics
|-
| style="padding:5px; width:30%" | [[BinAC2|bwForCluster BinAC 2]]
| style="padding-left:20px;" | Bioinformatics, Astrophysics, Geosciences, Pharmacy, and Medical Informatics
|}
|-
|bwHPC Clusters: [https://www.bwhpc.de/cluster.php operational status]
Further Compute Clusters in Baden-Württemberg (separate access policies):
* Datenanalyse Cluster der Hochschulen (DACHS): [[DACHS | Datenanalyse Cluster der Hochschulen (DACHS)]]
* bwHPC tier 1: [https://kb.hlrs.de/platforms/index.php/Hunter_(HPE) Hunter] ([https://www.hlrs.de/apply-for-computing-time getting access])
* bwHPC tier 2: [https://www.nhr.kit.edu/userdocs/horeka HoreKa] ([https://www.nhr.kit.edu/userdocs/horeka/projects/ getting access])
|}

{| style="background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | Documentation valid for all Clusters
|-
|
* [[Environment Modules| Software Modules]] and software documentation explained
* [https://www.bwhpc.de/software.html List of Software] on all clusters
* [[Development| Software Development and Parallel Programming]]
* [[Energy Efficient Cluster Usage]]
* [[HPC Glossary]]

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | Scientific Data Storage
|-
|
For user guides of the scientific data storage services:
* [[SDS@hd]]
* [https://www.rda.kit.edu/english bwDataArchive]
* [https://uni-tuebingen.de/einrichtungen/zentrum-fuer-datenverarbeitung/projekte/laufende-projekte/bwsfs bwSFS]
Associated, but local scientific storage services are:
* [https://wiki.scc.kit.edu/lsdf/index.php/Category:LSDF_Online_Storage LSDF Online Storage] (only for KIT and KIT partners)
|}

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | Data Management
|-
|
* [[Data Transfer|Data Transfer]]
* [https://www.forschungsdaten.org/index.php/FDM-Kontakte#Deutschland Research Data Management (RDM)] contact persons
* [https://www.forschungsdaten.info Portal for Research Data Management] (Forschungsdaten.info)
|}
{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Support
|-
|
* [[BwSupportPortal|Submit a Ticket in our Support Portal]]
Support is provided by the [https://www.bwhpc.de/teams.php bwHPC Competence Centers]:
|}
{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Acknowledgement
|-
|
* Please [[Acknowledgement|acknowledge]] our resources in your publications.
|}

Status

2025-10-15T08:24:01Z

H Winkhardt: ZAS wieder down

= bwHPC Cluster and Service Status Page =

== Current Status ==





{| style=" background:#FFD28A; width:100%;"
| style="padding:8px; background:#FFC05C; font-size:120%; font-weight:bold; text-align:left" | Status
|-
|
15.10.2025: central application page (ZAS) [https://zas.bwhpc.de/] currently down
* Renewal and application of new projects (Rechenvorhaben/RV) and registration of new RV members not possible.
* Filling out the bwUniCluster3.0 questionnaire not possible.
* Login and compute activities are '''not''' affected
|}




== Old Messages ==

09.10.2025: central application page (ZAS) [https://zas.bwhpc.de/] currently down
* Renewal and application of new projects (Rechenvorhaben/RV) and registration of new RV members not possible.
* Filling out the bwUniCluster3.0 questionnaire not possible.
* 10.10.: ongoing issue
* Login and compute activities are '''not''' affected
* 2025-10-10T16: Some sites report normal operation. Identity providers need to update DFN information, expected within 24h.

Status

2025-10-13T07:38:33Z

H Winkhardt: Update Status ZAS

= bwHPC Cluster and Service Status Page =

== Current Status ==



{| style=" background:#B8FFB8; width:100%;"
| style="padding:8px; background:#8AFF8A; font-size:120%; font-weight:bold; text-align:left" | Status
|-
|
Normal operation of all systems.
|}








== Old Messages ==

09.10.2025: central application page (ZAS) [https://zas.bwhpc.de/] currently down
* Renewal and application of new projects (Rechenvorhaben/RV) and registration of new RV members not possible.
* Filling out the bwUniCluster3.0 questionnaire not possible.
* 10.10.: ongoing issue
* Login and compute activities are '''not''' affected
* 2025-10-10T16: Some sites report normal operation. Identity providers need to update DFN information, expected within 24h.

Main Page

2025-10-13T07:36:03Z

H Winkhardt: Update Status ZAS

Status

2025-10-10T12:26:50Z

H Winkhardt: UC3 Fragebogen

{| style=" background:#ffa833; width:100%;"
| style="padding:8px; background:#f80; font-size:120%; font-weight:bold; text-align:left" | ZAS
|-
|
09.10.2025: central application page (ZAS) [https://zas.bwhpc.de/] currently down
* Renewal and application of new projects (Rechenvorhaben/RV) and registration of new RV members not possible.
* Filling out the bwUniCluster3.0 questionnaire not possible.
* 10.10.: ongoing issue
* Login and compute activities are '''not''' affected
|}

SDS@hd/Registration/Entitlement

2025-10-07T09:11:34Z

H Winkhardt: Mannheim

== Get SDS@hd Entitlement ==

New SVs can only be opened by bwIDM members who have the '''sds-hd-sv''' entitlement from their home university. This entitlement may have to be applied for individually at the home organisation.

If you are not sure if you already have an entitlement, please check it first with the [[Registration/bwForCluster/Entitlement#Check_your_Entitlements|'''Check your Entitlements''']] guide below.
If you need the entitlement, please follow the link for your institution or contact your local service desk if no information is provided:
* Hochschule Esslingen
* Universität Freiburg
* Universität Heidelberg:
** Employees of Heidelberg University receive this entitlement automatically.
* Universität Hohenheim
* Karlsruhe Institute of Technology (KIT)
* Universität Konstanz
* Universität Mannheim
** Employees of the University of Mannheim receive this entitlement automatically.
* Universität Stuttgart
* Universität Tübingen
* Universität Ulm
* HAW BW e.V.

== Check your Entitlements ==

To make sure you do not already have the entitlement, please log in to '''https://login.bwidm.de/user/index.xhtml'''.
To see the list of your entitlements, first select the '''Shibboleth''' tab at the top.
If the list below <code><nowiki>urn:oid:1.3.6.1.4.1.5923.1.1.1.7</nowiki></code> contains
<pre>http://bwidm.de/entitlement/bwForCluster</pre>
you already have the entitlement and can skip step A.
{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
<code><nowiki>http://bwidm.de/entitlement/bwForCluster</nowiki></code> is an attribute and not a link!
See [https://www.bwidm.de/dienste.php bwUniCluster und bwForCluster] for more information about needed attributes for this service.
|}
[[File:BwIDM-idp.png|center|600px|thumb|Verify Entitlement.]]

----

[[Registration/bwForCluster/RV|Go to step B]]

Helix/bwVisu/RStudio

2025-10-06T11:51:40Z

H Winkhardt: Java module path

__FORCETOC__
[https://posit.co/products/open-source/rstudio-server/ RStudio] is an integrated development environment (IDE) for programming with R. It can handle Python code, too.

== Change R Version==

By default, the default R version on Helix is used. You can view the current default version by executing <code>module -t -d avail 2>&1 | grep math/R</code> on Helix.

=== Change the R version to a version unavailable on Helix ===

# Open a shell on Helix by [https://wiki.bwhpc.de/e/Helix/Login sshing into Helix] or by starting the RStudio app on bwVisu and switching to the terminal tab.
# [https://cran.r-project.org/doc/FAQ/R-FAQ.html#How-can-R-be-obtained_003f-1 Download] your desired R version, [https://cran.r-project.org/doc/manuals/r-release/R-admin.html#Getting-and-unpacking-the-sources-1 unpack it] and change into the unpacked directory:
#: <syntaxhighlight lang="bash">tar -xf R-x.y.z.tar.gz
cd R-x.y.z.tar.gz</syntaxhighlight>
#: where <code>x.y.z</code> denotes the version of your downloaded R installation files.
# Patch the configuration file to make it compatible with your shell environment on Helix:
#: <syntaxhighlight lang="bash">sed '/date-time conversions do not work in 2020/s/^/: #/' -i configure</syntaxhighlight>
# Install R into a chosen location in your home directory:
#: <syntaxhighlight lang="bash">./configure prefix=$HOME/path/in/your/home/directory
module load devel/java_jdk
make
make install</syntaxhighlight>
# Make your installed R version the default by exporting the location of its binary to your <code>PATH</code>: Open or create the file <code>.bashrc</code> in your home directory and add the following line:
#: <syntaxhighlight lang="bash">export "PATH=/path/to/your/R/installation/directory/bin:$PATH"</syntaxhighlight>
#: Please make sure that the suffix <code>/bin</code> is added to the path to the installation location of your custom R version. This is the default location where the binaries of R will be usually installed.
# Test the installation: Reload your bash environment and obtain the version of the R binary in your path:
#: <syntaxhighlight lang="bash">source $HOME/.bashrc && R --version</syntaxhighlight>
#: If the displayed version number is equal to the one of your custom R installation, then the installation was successfull and your custom R version will be automatically loaded in RStudio when you choose <code>custom</code> in the dropdown menu when starting the job.
# (Re)start RStudio

== Add R packages ==

* Add a new package to RStudio
*# [[Helix/Login|Login to bwForCluster Helix]] or start the RStudio app on bwVisu and switch to the Terminal tab.
*# Follow the instructions from the [[Helix/Software/R#Installing_R-Packages_into_your_home_folder|R page]]
* '''Note''': For some packages, not all development dependencies are available through bwVisu. Some packages may have to be installed on the Login nodes after a ssh-login.
* To create dashboards, you can use RShiny with bwVisu RStudio as well.

Helix/bwVisu/RStudio

2025-10-06T11:46:46Z

H Winkhardt: /* Add R packages */

__FORCETOC__
[https://posit.co/products/open-source/rstudio-server/ RStudio] is an integrated development environment (IDE) for programming with R. It can handle Python code, too.

== Change R Version==

By default, the default R version on Helix is used. You can view the current default version by executing <code>module -t -d avail 2>&1 | grep math/R</code> on Helix.

=== Change the R version to a version unavailable on Helix ===

# Open a shell on Helix by [https://wiki.bwhpc.de/e/Helix/Login sshing into Helix] or by starting the RStudio app on bwVisu and switching to the terminal tab.
# [https://cran.r-project.org/doc/FAQ/R-FAQ.html#How-can-R-be-obtained_003f-1 Download] your desired R version, [https://cran.r-project.org/doc/manuals/r-release/R-admin.html#Getting-and-unpacking-the-sources-1 unpack it] and change into the unpacked directory:
#: <syntaxhighlight lang="bash">tar -xf R-x.y.z.tar.gz
cd R-x.y.z.tar.gz</syntaxhighlight>
#: where <code>x.y.z</code> denotes the version of your downloaded R installation files.
# Patch the configuration file to make it compatible with your shell environment on Helix:
#: <syntaxhighlight lang="bash">sed '/date-time conversions do not work in 2020/s/^/: #/' -i configure</syntaxhighlight>
# Install R into a chosen location in your home directory:
#: <syntaxhighlight lang="bash">./configure prefix=$HOME/path/in/your/home/directory
module load devel/java
make
make install</syntaxhighlight>
# Make your installed R version the default by exporting the location of its binary to your <code>PATH</code>: Open or create the file <code>.bashrc</code> in your home directory and add the following line:
#: <syntaxhighlight lang="bash">export "PATH=/path/to/your/R/installation/directory/bin:$PATH"</syntaxhighlight>
#: Please make sure that the suffix <code>/bin</code> is added to the path to the installation location of your custom R version. This is the default location where the binaries of R will be usually installed.
# Test the installation: Reload your bash environment and obtain the version of the R binary in your path:
#: <syntaxhighlight lang="bash">source $HOME/.bashrc && R --version</syntaxhighlight>
#: If the displayed version number is equal to the one of your custom R installation, then the installation was successfull and your custom R version will be automatically loaded in RStudio when you choose <code>custom</code> in the dropdown menu when starting the job.
# (Re)start RStudio

== Add R packages ==

* Add a new package to RStudio
*# [[Helix/Login|Login to bwForCluster Helix]] or start the RStudio app on bwVisu and switch to the Terminal tab.
*# Follow the instructions from the [[Helix/Software/R#Installing_R-Packages_into_your_home_folder|R page]]
* '''Note''': For some packages, not all development dependencies are available through bwVisu. Some packages may have to be installed on the Login nodes after a ssh-login.
* To create dashboards, you can use RShiny with bwVisu RStudio as well.

Batch Jobs - bwUniCluster Features

2025-10-02T13:36:02Z

H Winkhardt: Redirected page to BwUniCluster3.0/Running Jobs

#REDIRECT[[BwUniCluster3.0/Running_Jobs]]

BwUniCluster3.0/Software/Python Dask

2025-10-01T11:24:30Z

H Winkhardt: UC2 -> UC3

This guide explains how to use Python Dask and dask-jobqueue on bwUniCluster3.0.

== Installation and Usage ==
Please have a look at our [https://github.com/hpcraink/workshop-parallel-jupyter Workshop] on how to use Dask on bwUniCluster3.0 (2_Fundamentals: Creating Environments and 6_Dask).

BinAC2

2025-09-29T12:46:37Z

H Winkhardt: Formatting error

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:center; color:#000;vertical-align:middle;font-size:75%;" |
[[File:BinAC2_Logo_RGB_subtitel.svg|center|500px||]]
|-
|
|}

The '''bwForCluster BinAC 2''' supports researchers from the broader fields of Bioinformatics, Astrophysics, Geosciences, Pharmacy, and Medical Informatics.

{| style=" background:#ffa833; width:100%;"
| style="padding:8px; background:#f80; font-size:120%; font-weight:bold; text-align:left" | BinAC 1 -> BinAC 2 Migration
|-
|
* Still using BinAC 1: '''[[BinAC|Go To BinAC 1 Legacy Wiki]]'''
* '''Users of BinAC 1 must''' [[Registration/bwForCluster/BinAC2|'''re-register for BinAC 2''']] (only step C necessary). You keep your existing projects (Rechenvorhaben).
* '''[[BinAC2/Migrate BinAC 1 workspaces to BinAC 2 workspaces|Migrate BinAC 1 workspaces to BinAC 2 workspaces]]'''
* '''[[BinAC2/Migrate Moab to Slurm jobs|Migrate Moab/Torque to Slurm jobs]]'''
|}

{| style=" background:#FEF4AB; width:100%;"
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News and Events
|-
|
* [https://uni-tuebingen.de/de/274923 11th bwHPC Symposium - September 23rd, Tübingen]

|}

{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support
|-
|
* [[BinAC2/Getting_Started|Getting Started]]
* [https://training.bwhpc.de E-Learning Courses]
* [[BinAC2/Support|Contact and Support]]
* Send [[Feedback|Feedback]] about Wiki pages
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | User Documentation
|-
|
* [[Registration/bwForCluster|Registration]]

* [[BinAC2/Login|Login]]

* [[BinAC2/Hardware_and_Architecture|Hardware and Architecture]]
** [[BinAC2/Hardware_and_Architecture#Compute_Nodes|Node Specifications]]
** [[BinAC2/Hardware_and_Architecture#File_Systems|File Systems and Workspaces]]
* Usage of [[BinAC2/Software|Software]] on BinAC 2
** For available Software Modules see [https://www.bwhpc.de/software.php bwhpc.de Software Search] (select bwForCluster BinAC 2)
** Create Software Environments with [[Development/Conda|Conda]]
** Use [[BinAC2/Software/Nextflow|nf-core Nextflow pipelines]]
** See [[Development]] for info about compiler and parallelization

* [[BinAC2/Slurm|Batch System (SLURM)]]
** [[BinAC2/SLURM_Partitions|SLURM Partitions]]

|}
{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Cluster Funding
|-
|
* Please [[BinAC2/Acknowledgement|acknowledge]] the cluster in your publications.
|}

bwHPC Wiki:General disclaimer

2025-09-17T12:45:47Z

H Winkhardt: New SCC name

bwHPC Wiki is hosted by Scientific Computing Center (SCC) at Karlsruhe Institute of Technology. [http://www.scc.kit.edu/en/legals.php SCC disclaimer] is valid for all pages of bwHPC Wiki.

bwHPC Wiki:Privacy policy

2025-09-17T12:45:22Z

H Winkhardt: New SCC name

bwHPC Wiki is hosted by Scientific Computing Center (SCC) at Karlsruhe Institute of Technology. [http://www.scc.kit.edu/en/datenschutz.php SCC private policy] is valid for all pages of bwHPC Wiki.

R

2025-09-11T07:55:33Z

H Winkhardt:

'''R''' is a language and environment for statistical computing and graphics. Please refer to corresponding page:

* [[Development/R | General remarks]]
* Cluster-specific pages:
** [[BwUniCluster3.0/Software/R | R on bwUniCluster 3.0]]
** [[Helix/Software/R | R on Helix]]
* RStudio IDE
** [[Helix/bwVisu/RStudio | RStudio on Helix]] through [[Helix/bwVisu | bwVisu ]]

Python

2025-09-11T07:42:27Z

H Winkhardt: Create disambiguation to make the search a bit easier

Disambiguation page for the programming language '''Python''' and related tools and packages.

* [[Development/Python | General remarks on Python usage and setting up environments]]
* Additional cluster-specific pages
** [[JUSTUS2/Software/Python | Python on JUSTUS 2]]
* Specific related software packages
** [[BwUniCluster3.0/Software/Python Dask | Dask on the bwUniCluster3.0]]
** [[Development/ollama | ollama ]]
* Jupyter IDE and environments
** [[BwUniCluster3.0/Jupyter | Jupyterlab on the bwUniCluster3.0 ]]
** [[Helix/bwVisu/JupyterLab | Jupyterlab on Helix]] through [[Helix/bwVisu | bwVisu ]]
** [[BinAC2/Software/Jupyterlab | Jupyterlab module on BinAC2 ]]

External links:
* [https://github.com/hpcraink/workshop-parallel-jupyter Workshop] for parallel Python via Jupyter on the bwUniCluster3.0

R

2025-09-10T10:32:16Z

H Winkhardt:

R

2025-09-10T10:31:48Z

H Winkhardt:

'''R''' is a language and environment for statistical computing and graphics. Please refer to the corresponding cluster-specific page:

* [[Development/R | General remarks]]
* Cluster-specific pages:
** [[BwUniCluster3.0/Software/R | R on bwUniCluster 3.0]]
** [[Helix/Software/R | R on Helix]]

Development/Python

2025-09-09T08:30:42Z

H Winkhardt: typo

== Introduction ==
Python is a versatile, easy-to-learn, interpreted programming language. It offers a wide range of libraries for scientific tasks and visualization. Python counts to the best languages for machine learning. Python can be used in particular as an open source alternative for tasks that have usually been used for Matlab.

== Installation and Versions ==
Python is available on all systems. With <code>python --version</code> you can see the currently active default python version. In general, you can choose from various types of python installations:
* '''System python:''' This python version comes together with the operating system and is available upon login to the cluster. Other python versions might be installed along with it. All versions can be seen with
*: <code>ls /usr/bin/python[0-9].*[0-9] | sort -V | cut -d"/" -f4 | xargs</code>
*: They can change over time. You can access a specific Python version by specifying the version in the Python command.
* '''[[Environment_Modules | Software module]]:''' Available versions can be identified via
*: <syntaxhighlight lang="bash">
module avail devel/python
</syntaxhighlight>
* '''Python distributions and virtual environments:''' By using python distributions such as Anaconda, you can easily install the needed python version into a virtual environment. For the use of conda on bwHPC clusters, please refer to [[Development/Conda|Conda]]. Alternatively, you can use more python specific tools for installing python. Some options are listed in [[#Virtual Environments and Package Management | Virtual Environments and Package Management]]
* '''[[Development/Containers | Container]]:''' Containers can contain their own python installation. Keep this in mind when you are working with containers provided by others.

== Running Python Code ==

There are three ways to run Python commands:

* Within a '''terminal''' by executing the comand <code>python</code>. This starts a Python shell, where all commands are evaluated by the Python interpreter.
* Within a '''script''' (file ends with ''.py'' and can be run with <code>python myProgram.py</code>)
* Within a '''notebook''' (file ends with .ipynb). You can use other programming languages and markdown within a notebook next to your python code. Besides software development itself, teaching, prototyping and visualization are good use cases for notebooks as well.

== Development Environments ==

Development Environments are usually more comfortable than running code directly from the shell. Some common options are:

* [[Jupyter]]
* [[Development/VS_Code | VS Code]]
* PyCharm

== Virtual Environments and Package Management ==

Virtual environments allow Python packages to be installed in a separate installation directory. There should be at least one environment per project. This prevents version conflicts, promotes clarity as to which packages are required for which software and prevents the home directory from being cluttered with libraries that are not (or no longer) required.
A package can be installed via a package manager and contains an additional set of functions.

=== Overview ===
The following table provides an overview of both and explains the differences between the various options. After deciding on a specific tool, it can be installed by following the given link.
To make it short, if you plan to use...
* ...Python only: venv is the base for later solutions, poetry is widely used, uv is the latest and fastest option.
* ...Python + Conda: Installing conda packages first into a conda environment and then python packages with pip should work. For a faster and more up to date solution, choose pixi.

{| class="wikitable"
|- style="text-align:center;"
! Tool
! Description
! Can install python versions
! Installs packages from PyPI
! Installs packages from conda
! Dependency Resolver
! Dependency Management
! Creates Virtual Environments
! Supports building, packaging and publishing code (to PyPI)
|-
| pyenv
| Manages python versions on your system.
| yes
| no
| no
| no
| no
| no
| no
|-
| pip
| For installing python packages.
| no
| yes
| no
| yes
| no
| no
| no
|-
| venv
| For installing and managing python packages. Part of Python's standard library.
| no
| no
| no
| yes
| yes
| yes
| no
|-
| setuptools
| For packaging python projects.
| no
| no
| no
| no
| no
| no
| yes
|-
| poetry
| For installing and managing python packages. Install it with pipx.
| no
| yes
| no
| yes
| yes
| yes
| yes
|-
| pipx
| For installing and running python applications (like poetry) globally while having them in isolated environments. It is useful for keeping applications globally available and at the same time separated in their own virtual environments. Use it when the installation instructions of an application offer you this way of installation.
| no
| no
| no
| yes
| yes
| yes (only for single applications)
| yes
|-
| uv
| Replaces pixi, poetry, pyenv, pip etc. and is very fast (https://www.loopwerk.io/articles/2025/uv-keeps-getting-better/)
| yes
| yes
| no
| yes
| yes
| yes
| yes
|-
| pixi
| For installing and managing python as well as conda packages. Uses uv.
| yes
| yes
| yes
| yes
| yes
| yes
| yes
|}

=== Pip ===

The standard package manager under Python is <code>pip</code>. It can be used to install, update and delete packages. Pip can be called directly or via <code>python -m pip</code>. The standard repository from which packages are obtained is PyPI (https://pypi.org/). When a package depends on others, they are automatically installed as well.
In the following, the most common pip commands are shown exemplarily. Packages should always be installed within virtual environments to avoid conflicting installations. If you decide to not use a virtual environment, the install commands have to be accomplished by a <code>--user</code> flag or controlled via environment variables.

Installation of packages 
<syntaxhighlight lang="python">
pip install pandas # Installs the latest compatible version of pandas and its required dependencies
pip install pandas=1.5.3 # Installs exact version
pip install pandas>=1.5.3 # Installs version newer or equal to 1.5.3
</syntaxhighlight>

The packages from PyPI usually consist of precompiled libraries. However, <code>pip</code> is also capable of creating packages from source code. However, this may require the C/C++ compiler and other dependencies needed to build the libraries. In the example, pip obtains the source code of matplotlib from github.com, installs its dependencies, compiles the library and installs it:
<syntaxhighlight lang="python">
pip install git+https://github.com/matplotlib/matplotlib
</syntaxhighlight>
Upgrade packages 
<syntaxhighlight lang="python">
pip install --upgrade pandas # Updates the library if update is available
</syntaxhighlight>
Removing packages 
<syntaxhighlight lang="python">
pip uninstall pandas # Removes pandas
</syntaxhighlight>
Show packages 
<syntaxhighlight lang="python">
pip list # Shows the installed packages
pip freeze # Shows the installed packages and their versions
</syntaxhighlight>
Save State 
To allow for reproducibility it is important to provide information about the full list of packages and their exact versions [https://pip.pypa.io/en/stable/topics/repeatable-installs/ see version pinning].
<syntaxhighlight lang="python">
pip freeze > requirements.txt # redirect package and version information to a textfile
pip install -r requirements.txt # Installs all packages that are listed in the file
</syntaxhighlight>

=== Venv ===

The module <code>venv</code> enables the creation of a virtual environment and is a standard component of Python. Creating a <code>venv</code> means that a folder is created which contains a separate copy of the Python binary file as well as <code>pip</code> and <code>setuptools</code>. After activating the <code>venv</code>, the binary file in this folder is used when <code>python</code> or <code>pip</code> is called. This folder is also the installation target for other Python packages.

==== Creation ====

Create, activate, install software, deactivate:
<syntaxhighlight lang="bash">
python3.11 -m venv myEnv # Create venv
source myEnv/bin/activate # Activate venv
pip install --upgrade pip # Update of the venv-local pip
pip install <list of packages> # Install packages/modules
deactivate
</syntaxhighlight>

Install list of software:
<syntaxhighlight lang="bash">
pip install -r requirements.txt # Install packages/modules
</syntaxhighlight>

==== Usage ====
To use the virtual environment after all dependencies have been installed there, it is sufficient to simply activate it:
<syntaxhighlight lang="bash">
source myEnv/bin/activate # Activate venv
</syntaxhighlight>

With <code>venv</code> activated the terminal prompt will reflect that accordingly:
<syntaxhighlight lang="bash">
(myEnv) $
</syntaxhighlight>
It is no longer necessary to specify the Python version, the simple command <code>python</code> starts the Python version that was used to create the virtual environment. 
You can check, which Python version is in use via:
<syntaxhighlight lang="bash">
(myEnv) $ which python
</path/to/project>/myEnv/bin/python
</syntaxhighlight>

=== Poetry ===

==== Creation ====
When you want to create a virtual environment for an already existing project, you can go to the top directory and run
<syntaxhighlight lang="bash">
poetry init # Create virtual enviroment
</syntaxhighlight>
Otherwise start with the demo project:
<syntaxhighlight lang="bash">
poetry new poetry-demo
</syntaxhighlight>
You can set the allowed python versions in the pyproject.toml. To switch between python installations on your sytem, you can use
<syntaxhighlight lang="bash">
poetry env use python3.11
</syntaxhighlight>

==== Usage ====

Install and update packages
<syntaxhighlight lang="bash">
poetry install <package_name>
poetry update # Update to latest versions of packages
</syntaxhighlight>

To execute something within the virtual environment:
<syntaxhighlight lang="bash">
poetry run <command>
</syntaxhighlight>

Helpful links:
* [https://python-poetry.org/docs/managing-environments/#activating-the-environment Activate Environment]

Helpful commands
<syntaxhighlight lang="bash">
poetry env info # show environment information
poetry env list # list all virtual environments associated with the current project
poetry env list --full-path
poetry env remove # delete virtual environment
</syntaxhighlight>

== Best Practices ==
* Always use virtual environments! Use one environment per project.
* If a new or existing Python project is to be created or used, the following procedure is recommended:
# Version the Python source files and the <code>requirements.txt</code> file with a version control system, e.g. git. Exclude unnecessary folders and files like for example <code>venv</code> via an entry in the ignore file, for example <code>.gitignore</code>.
# Create and activate a virtual environment.
# Update pip <code>pip install --upgrade pip</code>.
# Install all required packages via the <code>requirements.txt</code> file in the case of venv. Or by using the corresponding command of your chosen tool.
* List or dictionary comprehensions are to be prefered over loops as they are in general faster.
* Be aware of the differences between references, shallow and deep copies.
* Do not parallelize by hand but use libraries were possible (Dask, ...).

Development/Python

2025-09-09T08:29:07Z

H Winkhardt: typo

== Introduction ==
Python is a versatile, easy-to-learn, interpreted programming language. It offers a wide range of libraries for scientific tasks and visualization. Python counts to the best languages for machine learning. Python can be used in particular as an open source alternative for tasks that have usually been used for Matlab.

== Installation and Versions ==
Python is available on all systems. With <code>python --version</code> you can see the currently active default python version. In general, you can choose from various types of python installations:
* '''System python:''' This python version comes together with the operating system and is available upon login to the cluster. Other python versions might be installed along with it. All versions can be seen with
*: <code>ls /usr/bin/python[0-9].*[0-9] | sort -V | cut -d"/" -f4 | xargs</code>
*: They can change over time. You can access a specific Python version by specifying the version in the Python command.
* '''[[Environment_Modules | Software module]]:''' Available versions can be identified via
*: <syntaxhighlight lang="bash">
module avail devel/python
</syntaxhighlight>
* '''Python distributions and virtual environments:''' By using python distributions such as Anaconda, you can easily install the needed python version into a virtual environment. For the use of conda on bwHPC clusters, please refer to [[Development/Conda|Conda]]. Alternatively, you can use more python specific tools for installing python. Some options are listed in [[#Virtual Environments and Package Management | Virtual Environments and Package Management]]
* '''[[Development/Containers | Container]]:''' Containers can contain their own python installation. Keep this in mind when you are working with containers provided by others.

== Running Python Code ==

There are three ways to run Python commands:

* Within a '''terminal''' by executing the comand <code>python</code>. This starts a Python shell, where all commands are evaluated by the Python interpreter.
* Within a '''script''' (file ends with ''.py'' and can be run with <code>python myProgram.py</code>)
* Within a '''notebook''' (file ends with .ipynb). You can use other programming languages and markdown within a notebook next to your python code. Besides software development itself, teaching, prototyping and visualization are good use cases for notebooks as well.

== Development Environments ==

Development Environments are usually more comfortable than running code directly from the shell. Some common options are:

* [[Jupyter]]
* [[Development/VS_Code | VS Code]]
* PyCharm

== Virtual Environments and Package Management ==

Virtual environments allow Python packages to be installed in a separate installation directory. There should be at least one environment per project. This prevents version conflicts, promotes clarity as to which packages are required for which software and prevents the home directory from being cluttered with libraries that are not (or no longer) required.
A package can be installed via a package manager and contains an additional set of functions.

=== Overview ===
The following table provides an overview of both and explains the differences between the various options. After deciding on a specific tool, it can be installed by following the given link.
To make it short, if you plan to use...
* ...Python only: venv is the base for later solutions, poetry is widely used, uv is the latest and fastest option.
* ...Python + Conda: Installing conda packages first into a conda environment and then python packages with pip should work. For a faster and more up to date solution, choose pixi.

{| class="wikitable"
|- style="text-align:center;"
! Tool
! Description
! Can install python versions
! Installs packages from PyPI
! Installs packages from conda
! Dependency Resolver
! Dependency Management
! Creates Virtual Environments
! Supports building, packaging and publishing code (to PyPI)
|-
| pyenv
| Manages python versions on your system.
| yes
| no
| no
| no
| no
| no
| no
|-
| pip
| For installing python packages.
| no
| yes
| no
| yes
| no
| no
| no
|-
| venv
| For installing and managing python packages. Part of Python's standard library.
| no
| no
| no
| yes
| yes
| yes
| no
|-
| setuptools
| For packaging python projects.
| no
| no
| no
| no
| no
| no
| yes
|-
| poetry
| For installing and managing python packages. Install it with pipx.
| no
| yes
| no
| yes
| yes
| yes
| yes
|-
| pipx
| For installing and running python applications (like poetry) globally while having them in isolated environments. It is useful for keeping applications globally available and at the same time separated in their own virtual environments. Use it when the installation instructions of an application offer you this way of installation.
| no
| no
| no
| yes
| yes
| yes (only for single applications)
| yes
|-
| uv
| Replaces pixi, poetry, pyenv, pip etc. dand is very fast (https://www.loopwerk.io/articles/2025/uv-keeps-getting-better/)
| yes
| yes
| no
| yes
| yes
| yes
| yes
|-
| pixi
| For installing and managing python as well as conda packages. Uses uv.
| yes
| yes
| yes
| yes
| yes
| yes
| yes
|}

=== Pip ===

The standard package manager under Python is <code>pip</code>. It can be used to install, update and delete packages. Pip can be called directly or via <code>python -m pip</code>. The standard repository from which packages are obtained is PyPI (https://pypi.org/). When a package depends on others, they are automatically installed as well.
In the following, the most common pip commands are shown exemplarily. Packages should always be installed within virtual environments to avoid conflicting installations. If you decide to not use a virtual environment, the install commands have to be accomplished by a <code>--user</code> flag or controlled via environment variables.

Installation of packages 
<syntaxhighlight lang="python">
pip install pandas # Installs the latest compatible version of pandas and its required dependencies
pip install pandas=1.5.3 # Installs exact version
pip install pandas>=1.5.3 # Installs version newer or equal to 1.5.3
</syntaxhighlight>

The packages from PyPI usually consist of precompiled libraries. However, <code>pip</code> is also capable of creating packages from source code. However, this may require the C/C++ compiler and other dependencies needed to build the libraries. In the example, pip obtains the source code of matplotlib from github.com, installs its dependencies, compiles the library and installs it:
<syntaxhighlight lang="python">
pip install git+https://github.com/matplotlib/matplotlib
</syntaxhighlight>
Upgrade packages 
<syntaxhighlight lang="python">
pip install --upgrade pandas # Updates the library if update is available
</syntaxhighlight>
Removing packages 
<syntaxhighlight lang="python">
pip uninstall pandas # Removes pandas
</syntaxhighlight>
Show packages 
<syntaxhighlight lang="python">
pip list # Shows the installed packages
pip freeze # Shows the installed packages and their versions
</syntaxhighlight>
Save State 
To allow for reproducibility it is important to provide information about the full list of packages and their exact versions [https://pip.pypa.io/en/stable/topics/repeatable-installs/ see version pinning].
<syntaxhighlight lang="python">
pip freeze > requirements.txt # redirect package and version information to a textfile
pip install -r requirements.txt # Installs all packages that are listed in the file
</syntaxhighlight>

=== Venv ===

The module <code>venv</code> enables the creation of a virtual environment and is a standard component of Python. Creating a <code>venv</code> means that a folder is created which contains a separate copy of the Python binary file as well as <code>pip</code> and <code>setuptools</code>. After activating the <code>venv</code>, the binary file in this folder is used when <code>python</code> or <code>pip</code> is called. This folder is also the installation target for other Python packages.

==== Creation ====

Create, activate, install software, deactivate:
<syntaxhighlight lang="bash">
python3.11 -m venv myEnv # Create venv
source myEnv/bin/activate # Activate venv
pip install --upgrade pip # Update of the venv-local pip
pip install <list of packages> # Install packages/modules
deactivate
</syntaxhighlight>

Install list of software:
<syntaxhighlight lang="bash">
pip install -r requirements.txt # Install packages/modules
</syntaxhighlight>

==== Usage ====
To use the virtual environment after all dependencies have been installed there, it is sufficient to simply activate it:
<syntaxhighlight lang="bash">
source myEnv/bin/activate # Activate venv
</syntaxhighlight>

With <code>venv</code> activated the terminal prompt will reflect that accordingly:
<syntaxhighlight lang="bash">
(myEnv) $
</syntaxhighlight>
It is no longer necessary to specify the Python version, the simple command <code>python</code> starts the Python version that was used to create the virtual environment. 
You can check, which Python version is in use via:
<syntaxhighlight lang="bash">
(myEnv) $ which python
</path/to/project>/myEnv/bin/python
</syntaxhighlight>

=== Poetry ===

==== Creation ====
When you want to create a virtual environment for an already existing project, you can go to the top directory and run
<syntaxhighlight lang="bash">
poetry init # Create virtual enviroment
</syntaxhighlight>
Otherwise start with the demo project:
<syntaxhighlight lang="bash">
poetry new poetry-demo
</syntaxhighlight>
You can set the allowed python versions in the pyproject.toml. To switch between python installations on your sytem, you can use
<syntaxhighlight lang="bash">
poetry env use python3.11
</syntaxhighlight>

==== Usage ====

Install and update packages
<syntaxhighlight lang="bash">
poetry install <package_name>
poetry update # Update to latest versions of packages
</syntaxhighlight>

To execute something within the virtual environment:
<syntaxhighlight lang="bash">
poetry run <command>
</syntaxhighlight>

Helpful links:
* [https://python-poetry.org/docs/managing-environments/#activating-the-environment Activate Environment]

Helpful commands
<syntaxhighlight lang="bash">
poetry env info # show environment information
poetry env list # list all virtual environments associated with the current project
poetry env list --full-path
poetry env remove # delete virtual environment
</syntaxhighlight>

== Best Practices ==
* Always use virtual environments! Use one environment per project.
* If a new or existing Python project is to be created or used, the following procedure is recommended:
# Version the Python source files and the <code>requirements.txt</code> file with a version control system, e.g. git. Exclude unnecessary folders and files like for example <code>venv</code> via an entry in the ignore file, for example <code>.gitignore</code>.
# Create and activate a virtual environment.
# Update pip <code>pip install --upgrade pip</code>.
# Install all required packages via the <code>requirements.txt</code> file in the case of venv. Or by using the corresponding command of your chosen tool.
* List or dictionary comprehensions are to be prefered over loops as they are in general faster.
* Be aware of the differences between references, shallow and deep copies.
* Do not parallelize by hand but use libraries were possible (Dask, ...).

VNC

2025-09-01T13:31:00Z

H Winkhardt:

'''VNC''' is short for '''V'''irtual '''N'''etwork '''C'''omputing. It is used to have a graphical user interface on a remote computer.

== Cluster Specific Information ==

* [[BwUniCluster3.0/Software/Start vnc desktop]]

* [[JUSTUS2/Visualization]]

Also see [[:Category:Visualization]]

Acknowledgement

2025-09-01T13:30:05Z

H Winkhardt: UC2 -> UC3

Remember to acknowledge our resources in your publications!

Such recognition is important for acquiring funding for the next generation of hardware, support services, data storage, and infrastructure.

The publications will be referenced on the bwHPC website: https://www.bwhpc.de/user_publications.html

== HPC Clusters ==

→[[BwUniCluster3.0/Acknowledgement| bwUniCluster Acknowledgement]]

→[[BinAC2/Acknowledgement| bwForCluster BinAC2 Acknowledgement]]

→[[Helix/Acknowledgement| bwForCluster Helix Acknowledgement]]

→[[JUSTUS2/Acknowledgement| bwForCluster JUSTUS2 Acknowledgement]]

→[[NEMO2/Acknowledgement| bwForCluster NEMO2 Acknowledgement]]

== Data Facilities ==

→[[SDS@hd/Acknowledgement| SDS@hd Acknowledgement]]

JUSTUS2/Software/Turbomole

2025-09-01T13:27:13Z

H Winkhardt: UC2 -> UC3

{{Softwarepage|chem/turbomole}}

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| chem/turbomole
|-
| License
| [https://www.turbomole.org/turbomole/order-turbomole/ Commerical]
|-
| Citing
| [https://www.turbomole.org/turbomole/turbomole-documentation/ See Turbomole manual]
|-
| Links
| [https://www.turbomole.com Homepage] | [https://www.turbomole-gmbh.com/turbomole-manuals.html Documentation]
|-
| User Forum
| [https://www.turbo-forum.com/ external]
|}
= Description =
'''Turbomole''' is a general purpose ''quantum chemistry'' software package for ''ab initio'' electronic structure calculations and provides:
* ground state calculations for methods such as Hartree-Fock, DFT, MP2, and CCSD(T);
* excited state calculations at different levels such as full RPA, TDDFT, CIS(D), CC2, an ADC(2);
* geometry optimizations, transition state searches, molecular dynamics calculations;
* property and spectra calculations such as IR, UV/VIS, Raman, and CD;
* approximations like resolution-of-the-identity (RI) to speed-up the calculations without introducing uncontrollable or unknown errors; as well as
* parallel versions for all kind of jobs.
For more information on Turbmole's features please visit [https://www.turbomole-gmbh.com/program-overview.html http://www.turbomole-gmbh.com/program-overview.html].
 
 

= Versions and Availability =

On the command line interface (CLI) of a particular bwHPC cluster a list of all available Turbomole versions can be inquired as followed
<pre>
$ module avail chem/turbomole
</pre>

A current list of the versions available on the bwUniCluster and bwForClusters can be found here:
https://www.bwhpc.de/software.php

== bwUniCluster 3.0 ==

* Turbomole 7.9

== bwForCluster JUSTUS 2 ==

* Turbomole 7.5
* Turbomole 7.4.1

== Parallel computing ==
The Turbomole ''Module'' subsumes all available parallel computing variants of Turbomole's binaries. Turbomole defines the following parallel computing variants:
* SMP = Shared-memory parallel computing based on OpenMP and Fork() with the latter using separated address spaces.
* MPI = Message passing interface protocol based parallel computing
However only one of the parallel variants or the sequential variant can be loaded at once and most Turbomole's binaries support only 1 or 2 of the parallelization variants. Like for Turbomole installations without the ''Module'' system of the bwHPC clusters, the variants have to be triggered by the environment variable $PARA_ARCH.
 
 

= Usage =
== Before loading the Module ==
Before loading the Turbomole ''Module'' the parallel computing variant has to be defined via the environment variable $PARA_ARCH using the abbreviations SMP or MPI, e.g.:
<pre>
$ export PARA_ARCH=MPI
</pre>
will [[#Loading the Module|later load]] the '''MPI binary variants'''. If the variable $PARA_ARCH is not defined or empty, the sequential binary variants will be active once the Turbomole Module is loaded.
 
== Loading the Module ==
You can load the default version of ''Turbomole'' with the command:
<pre>
$ module load chem/turbomole
</pre>
The Turbomole ''Module'' does not depend on any other ''Module'', but on the variable $PARA_ARCH. Moreover, Turbomole provides its own libraries regarding ''OpenMP'', ''Fork()'', and ''MPI'' based parallelization.
If you wish to load a specific (older) version you can do so by executing e.g.:
<pre>
$ module load chem/turbomole/7.4.1
</pre>
to load the version 7.4.1
 
== Switching between different parallel variants ==
To switch between the different parallel variants provided by the Turbomole ''Module'', simply define the new parallel variant via $PARA_ARCH and load the ''Module'' again. Note that for switching between the parallel variants unloading of the Turbomole ''Module'' is not required. For instance to change to the MPI variant, execute:
<pre>
$ export PARA_ARCH=MPI
$ module load chem/turbomole
</pre>
 
== Turbomole binaries ==
The '''Turbomole''' software package consists of a set of stand-alone program binaries providing different features and parallelization support:
{| width=750px class="wikitable"
|- style=text-align:left"
! Binary !! Features !! OpenMP !! Fork !! MPI
|- style="vertical-align:top;"
| define
| Interactive input generator
| no
| no
| no
|- style="vertical-align:top;"
| dscf
| Energy calculations
| yes
| yes
| yes
|- style="vertical-align:top;"
| grad
| Gradient calculations
| yes
| yes
| yes
|- style="vertical-align:top;"
| ridft
| Energy calc. with fast Coulomb approximation
| no
| yes
| yes
|- style="vertical-align:top;"
| rdgrad
| Gradient calc. with fast Coulomb approximation
| yes
| yes
| yes
|- style="vertical-align:top;"
| ricc2
| Electronic excitation energies, transition moments and properties of excited states
| yes
| no
| yes
|- style="vertical-align:top;"
| statpt
| Hessian and coordinate update for stationary point search
| no
| no
| no
|- style="vertical-align:top;"
| aoforce
| Analytic calculation of force constants, vibrational frequencies and IR intensities
| yes
| yes
| yes
|- style="vertical-align:top;"
| escf
| Calc. of time dependent and dielectric properties
| yes
| yes
| yes
|- style="vertical-align:top;"
| egrad
| gradients and first-order properties of excited states
| yes
| no
| yes
|- style="vertical-align:top;"
| odft
| Orbital-dependent energy calc.
| yes
| no
| no
|}
For the complete set of binaries and more detailed description of their features read [https://www.turbomole.org/turbomole/turbomole-documentation/ here].
 

== Turbomole tools ==
Turbomole's tool set contains scripts and binaries that help to prepare, execute workflows (such as geometry optimisation) and postprocess results.
{| width=750px class="wikitable"
|- style="text-align:left"
! Script !! Type !! Description
|- style="vertical-align:top;"
| x2t
| Preparation
| Converts XYZ coordinates into ''Turbomole'' coordinates.
|- style="vertical-align:top;"
| sdg
| Preparation
| Shows data group from ''control'' file: for example$ sdg coordshows current Turbomole coordinates used.
|- style="vertical-align:top;"
| jobex
| Optimization workflow
| Shell script that controls and executes automatic optimizations of molecular geometry parameters.

|- style="vertical-align:top;"
| tm2molden
| Postprocessing
| Creates a molden format input file for the [[Molden]] program.
|- style="vertical-align:top;"
| eiger
| Postprocessing
| Script front-end of program ''eigerf'' to display HOMO-LUMO gap and MO eigenvalues.
|}
For the complete set of tools and more detailed description of their features read [http://www.turbomole-gmbh.com/manuals/version_6_5/Documentation_html/node9.html here].
 
== Disk Usage ==
By default, scratch files of Turbomole binaries are placed in the directory of Turbomole binary execution. Please do not run your Turbomole calculations in your $HOME or $WORK directory.
 
 
= Examples =
== Single node jobs ==
=== Template provided by Turbomole Module ===
The Turbomole ''Module'' provides a simple job script example of Cubane (C8H8) that runs an energy relaxation via MPI parallel ''dscf'' using 4 cores on a single node and its local file system. To simply run the example do the following steps:
<pre>
$ module load chem/turbomole
$ mkdir -vp ~/Turbomole-example/
$ cd ~/Turbomole-examples/
$ cp -r $TURBOMOLE_EXA_DIR/* ~/Turbmole-example/
$ sbatch bwHPC_turbomole_single-node_tmpdir_example.sh
</pre>
The last step submits the job example script bwHPC_turbomole_single-node_example.sh to the queueing system. Once started on a compute node, all calculations will be done under an unique directory on the [[BwUniCluster_File_System#File_Systems|local file system]] of that particular compute node.
 
=== Geometry optimization ===
To do a geometry optimization of the [[#Single node job template provided by Turbomole Module|previous job example]] modify bwHPC_turbomole_single-node_tmpdir_example.sh by replacing the following line
<syntaxhighlight lang="bash">
time dscf > dscf.out 2>&1
</syntaxhighlight>
with
<syntaxhighlight lang="bash">
time jobex -dscf -keep 2>&1
</syntaxhighlight>
and submit the modified script to the queueing system on bwForCluster JUSTUS 2 (→[[JUSTUS2/Jobscripts: Running Your Calculations]] and bwUniCluster (→[[BwUniCluster3.0/Slurm]]) via sbatch.
The Turbomole command ''jobex'' controls the call of all the required Turbomole binaries for the geometry optimization.
 
 

= Turbomole-Specific Environment Variables =
To see a list of all Turbomole environments set by the 'module load'-command do the following:
<pre>
module show chem/turbomole 2>&1 | grep -E '(setenv|prepend-path)'
</pre>

= Version-Specific Information =
For specific information about the Turbomole version (e.g. 7.5) to be loaded do the following:
<pre>
module help chem/turbomole/7.5
</pre>

----

JUSTUS2/Software/Turbomole

2025-09-01T13:26:49Z

H Winkhardt: UC2 -> UC3

{{Softwarepage|chem/turbomole}}

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| chem/turbomole
|-
| License
| [https://www.turbomole.org/turbomole/order-turbomole/ Commerical]
|-
| Citing
| [https://www.turbomole.org/turbomole/turbomole-documentation/ See Turbomole manual]
|-
| Links
| [https://www.turbomole.com Homepage] | [https://www.turbomole-gmbh.com/turbomole-manuals.html Documentation]
|-
| User Forum
| [https://www.turbo-forum.com/ external]
|}
= Description =
'''Turbomole''' is a general purpose ''quantum chemistry'' software package for ''ab initio'' electronic structure calculations and provides:
* ground state calculations for methods such as Hartree-Fock, DFT, MP2, and CCSD(T);
* excited state calculations at different levels such as full RPA, TDDFT, CIS(D), CC2, an ADC(2);
* geometry optimizations, transition state searches, molecular dynamics calculations;
* property and spectra calculations such as IR, UV/VIS, Raman, and CD;
* approximations like resolution-of-the-identity (RI) to speed-up the calculations without introducing uncontrollable or unknown errors; as well as
* parallel versions for all kind of jobs.
For more information on Turbmole's features please visit [https://www.turbomole-gmbh.com/program-overview.html http://www.turbomole-gmbh.com/program-overview.html].
 
 

= Versions and Availability =

On the command line interface (CLI) of a particular bwHPC cluster a list of all available Turbomole versions can be inquired as followed
<pre>
$ module avail chem/turbomole
</pre>

A current list of the versions available on the bwUniCluster and bwForClusters can be found here:
https://www.bwhpc.de/software.php

== bwUniCluster 3.0 ==

* Turbomole 7.9

== bwForCluster JUSTUS 2 ==

* Turbomole 7.5
* Turbomole 7.4.1

== Parallel computing ==
The Turbomole ''Module'' subsumes all available parallel computing variants of Turbomole's binaries. Turbomole defines the following parallel computing variants:
* SMP = Shared-memory parallel computing based on OpenMP and Fork() with the latter using separated address spaces.
* MPI = Message passing interface protocol based parallel computing
However only one of the parallel variants or the sequential variant can be loaded at once and most Turbomole's binaries support only 1 or 2 of the parallelization variants. Like for Turbomole installations without the ''Module'' system of the bwHPC clusters, the variants have to be triggered by the environment variable $PARA_ARCH.
 
 

= Usage =
== Before loading the Module ==
Before loading the Turbomole ''Module'' the parallel computing variant has to be defined via the environment variable $PARA_ARCH using the abbreviations SMP or MPI, e.g.:
<pre>
$ export PARA_ARCH=MPI
</pre>
will [[#Loading the Module|later load]] the '''MPI binary variants'''. If the variable $PARA_ARCH is not defined or empty, the sequential binary variants will be active once the Turbomole Module is loaded.
 
== Loading the Module ==
You can load the default version of ''Turbomole'' with the command:
<pre>
$ module load chem/turbomole
</pre>
The Turbomole ''Module'' does not depend on any other ''Module'', but on the variable $PARA_ARCH. Moreover, Turbomole provides its own libraries regarding ''OpenMP'', ''Fork()'', and ''MPI'' based parallelization.
If you wish to load a specific (older) version you can do so by executing e.g.:
<pre>
$ module load chem/turbomole/7.4.1
</pre>
to load the version 7.4.1
 
== Switching between different parallel variants ==
To switch between the different parallel variants provided by the Turbomole ''Module'', simply define the new parallel variant via $PARA_ARCH and load the ''Module'' again. Note that for switching between the parallel variants unloading of the Turbomole ''Module'' is not required. For instance to change to the MPI variant, execute:
<pre>
$ export PARA_ARCH=MPI
$ module load chem/turbomole
</pre>
 
== Turbomole binaries ==
The '''Turbomole''' software package consists of a set of stand-alone program binaries providing different features and parallelization support:
{| width=750px class="wikitable"
|- style=text-align:left"
! Binary !! Features !! OpenMP !! Fork !! MPI
|- style="vertical-align:top;"
| define
| Interactive input generator
| no
| no
| no
|- style="vertical-align:top;"
| dscf
| Energy calculations
| yes
| yes
| yes
|- style="vertical-align:top;"
| grad
| Gradient calculations
| yes
| yes
| yes
|- style="vertical-align:top;"
| ridft
| Energy calc. with fast Coulomb approximation
| no
| yes
| yes
|- style="vertical-align:top;"
| rdgrad
| Gradient calc. with fast Coulomb approximation
| yes
| yes
| yes
|- style="vertical-align:top;"
| ricc2
| Electronic excitation energies, transition moments and properties of excited states
| yes
| no
| yes
|- style="vertical-align:top;"
| statpt
| Hessian and coordinate update for stationary point search
| no
| no
| no
|- style="vertical-align:top;"
| aoforce
| Analytic calculation of force constants, vibrational frequencies and IR intensities
| yes
| yes
| yes
|- style="vertical-align:top;"
| escf
| Calc. of time dependent and dielectric properties
| yes
| yes
| yes
|- style="vertical-align:top;"
| egrad
| gradients and first-order properties of excited states
| yes
| no
| yes
|- style="vertical-align:top;"
| odft
| Orbital-dependent energy calc.
| yes
| no
| no
|}
For the complete set of binaries and more detailed description of their features read [https://www.turbomole.org/turbomole/turbomole-documentation/ here].
 

== Turbomole tools ==
Turbomole's tool set contains scripts and binaries that help to prepare, execute workflows (such as geometry optimisation) and postprocess results.
{| width=750px class="wikitable"
|- style="text-align:left"
! Script !! Type !! Description
|- style="vertical-align:top;"
| x2t
| Preparation
| Converts XYZ coordinates into ''Turbomole'' coordinates.
|- style="vertical-align:top;"
| sdg
| Preparation
| Shows data group from ''control'' file: for example$ sdg coordshows current Turbomole coordinates used.
|- style="vertical-align:top;"
| jobex
| Optimization workflow
| Shell script that controls and executes automatic optimizations of molecular geometry parameters.

|- style="vertical-align:top;"
| tm2molden
| Postprocessing
| Creates a molden format input file for the [[Molden]] program.
|- style="vertical-align:top;"
| eiger
| Postprocessing
| Script front-end of program ''eigerf'' to display HOMO-LUMO gap and MO eigenvalues.
|}
For the complete set of tools and more detailed description of their features read [http://www.turbomole-gmbh.com/manuals/version_6_5/Documentation_html/node9.html here].
 
== Disk Usage ==
By default, scratch files of Turbomole binaries are placed in the directory of Turbomole binary execution. Please do not run your Turbomole calculations in your $HOME or $WORK directory.
 
 
= Examples =
== Single node jobs ==
=== Template provided by Turbomole Module ===
The Turbomole ''Module'' provides a simple job script example of Cubane (C8H8) that runs an energy relaxation via MPI parallel ''dscf'' using 4 cores on a single node and its local file system. To simply run the example do the following steps:
<pre>
$ module load chem/turbomole
$ mkdir -vp ~/Turbomole-example/
$ cd ~/Turbomole-examples/
$ cp -r $TURBOMOLE_EXA_DIR/* ~/Turbmole-example/
$ sbatch bwHPC_turbomole_single-node_tmpdir_example.sh
</pre>
The last step submits the job example script bwHPC_turbomole_single-node_example.sh to the queueing system. Once started on a compute node, all calculations will be done under an unique directory on the [[BwUniCluster_File_System#File_Systems|local file system]] of that particular compute node.
 
=== Geometry optimization ===
To do a geometry optimization of the [[#Single node job template provided by Turbomole Module|previous job example]] modify bwHPC_turbomole_single-node_tmpdir_example.sh by replacing the following line
<syntaxhighlight lang="bash">
time dscf > dscf.out 2>&1
</syntaxhighlight>
with
<syntaxhighlight lang="bash">
time jobex -dscf -keep 2>&1
</syntaxhighlight>
and submit the modified script to the queueing system on bwForCluster JUSTUS 2 (→[[JUSTUS2/Jobscripts: Running Your Calculations]] and bwUniCluster (→[[BwUniCluster2.0/Slurm]]) via sbatch.
The Turbomole command ''jobex'' controls the call of all the required Turbomole binaries for the geometry optimization.
 
 
= Turbomole-Specific Environment Variables =
To see a list of all Turbomole environments set by the 'module load'-command do the following:
<pre>
module show chem/turbomole 2>&1 | grep -E '(setenv|prepend-path)'
</pre>

= Version-Specific Information =
For specific information about the Turbomole version (e.g. 7.5) to be loaded do the following:
<pre>
module help chem/turbomole/7.5
</pre>

----

Development/ollama

2025-09-01T13:24:51Z

H Winkhardt: UC2 -> UC3

Using LLMs even for inferencing requires large computational resources - currently at best a powerful GPU -
as provided by the bwHPC clusters.
This page explains to how to make usage of bwHPC resources, using Ollama as an example to show best practices at work.

== Introduction ==
Ollama is an inferencing framework that provides access to a multitude of powerful, large models and allows
performant access to a variety of accelerators, e.g. from CPUs using AVX-512 to APUs like the AMD MI-300A,
as well as GPUs like multiple NVIDIA H100.

Installing the inference server Ollama by default assumes you have root permission to install the server globally for all users
into the directory <code>/usr/local/bin</code>. Of course, this is '''not''' sensible.
Therefore the clusters provide the [[Environment_Modules|Environment Modules]] including binaries and libraries for CPU (if available AVX-512), AMD ROCm (if available) and NVIDIA CUDA using:
module load cs/ollama

More information is available in [https://github.com/ollama/ollama/tree/main/docs Ollamas Github documentation] page.

The inference server Ollama opens the well-known port 11434. The compute node's IP is on the internal network, e.g. 10.1.0.101,
which is not visible to any outside computer like Your laptop.
Therefore we need a way to forward this port on an IP visible to the outside, aka the login nodes.

{|style="background:#FEF4AB; width:100%;"
|style="padding:5px; background:#FEF4AB; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#FEF4AB; text-align:left"|
Please note: this module started off in the Category <code>devel</code>, but has been moved to the correct category computer science, or short <code>cs</code>.
|}

== Preparation ==

Prior to starting and pulling models, it is a '''good idea''' to allocate a proper [[Workspace]] for the (multi-gigabyte) models
and create a soft-link into this directory for Ollama:
ws_allocate ollama_models 60
ln -s `ws_find ollama_models`/ ~/.ollama

Now we may allocate a compute node using [[BwUniCluster3.0/Slurm|Slurm]].
At first You may start with interactively checking out the method in one terminal:
srun --time=00:30:00 --gres=gpu:1 --pty /bin/bash
Please note that on bwUniCluster, You need to provide a partition, here containing a GPU, e.g. for this 30 minute run, we may select <code>--partition=dev_gpu_4</code>, on DACHS <code>--partition=gpu1</code>.

Your Shell's prompt will list the node's name, e.g. on bwUniCluster node <code>uc2n520</code>:
[USERNAME@uc2n520 ~]$

Now You may load the Ollama module and start the server on the compute node and make sure using <code>OLLAMA_HOST</code> that it serves to the external IP address:
module load cs/ollama
export OLLAMA_HOST=0.0.0.0:11434
ollama serve

You should be able to see the usage of the accelerator:

[[File:ollama_gpus.png|850x329px]]

== Accessing from login nodes ==
From another terminal You may log into the Cluster's login node a second time and pull a LLM (please check [https://ollama.com/search] for available models):
module load cs/ollama
export OLLAMA_HOST=uc2n520
ollama pull deepseek-r1

On the previous terminal on the compute node, You should see the model being downloaded and installed into the workspace.
Of course developing on the login nodes is not viable, therefore You may want to forward the ports.

{|style="background:#dedefe; width:100%;"
|style="padding:5px; background:#dedefe; text-align:left"|
[[Image:Info.svg|center]]
|style="padding:5px; background:#dedefe; text-align:left"|
On GPUs with 48GB VRAM like NVIDIA L40S, you may want to use the 70b model of Deepseek, i.e. <code>ollama pull deepseek-r1:70b</code> and amend the below commands accordingly.
|}

== Port forwarding ==

The login nodes of course have externally visible IP addresses, e.g. <code>bwunicluster.scc.kit.edu</code> which get to resolved to one of the multiple login nodes.
Using the Secure shell <code>ssh</code> one may forward a port from the login node to the compute node.

Of course, You may want to '''locally on Your laptop'''.
Open another terminal and start the Secure shell using the port forwarding:
ssh -L 11434:uc2n520:11434 USERNAME@bwunicluster.scc.kit.edu
Your OTP: 123456
Password:

You may check using whether this worked using Your local browser on Your Laptop:
[[File:firefox_ollama.png|645x325px]]

== Local programming ==
Now that You made sure You have access to the compute nodes GPU, you may develop on your local system:
python -m venv ollama_test
source ollama_test/bin/activate
python -m pip install ollama
export OLLAMA_HOST=localhost

and call <code>python</code> to run the following code:
import ollama
response = ollama.chat(model='deepseek-r1', messages=[ { 'role': 'user', 'content': 'why is the sky blue?'},])
print(response)

You should now see DeepSeek's response regarding Rayleigh Scattering.

On the compute node, You will see the computation:

[[File:ollama_gpus_computing.png|850x329px]]

'''Enjoy!'''

== Best Practice ==
Running interactively is generally '''not''' a good idea, especially not with very large models. Better submit Your job with mail notification, here file <code>ollama.slurm</code>:
#!/bin/bash
#SBATCH --partition=gpu_h100 # bwUniCluster3, for DACHS: gpu8
#SBATCH --gres=gpu:h100:4 # bwUniCluster3, for DACHS: gpu:h100:8
#SBATCH --ntasks-per-node=96 # considering bwUniCluster3 AMD EPYC9454, same on DACHS
#SBATCH --mem=500G # considering bwUniCluster3 768GB, enough on DACHS
#SBATCH --time=2:00:00 # Please be courteous to other users
#SBATCH --mail-type=BEGIN # Email when the job starts
#SBATCH --mail-user=my@mail.de # Your email address

module load cs/ollama # Load the the
export OLLAMA_HOST=0.0.0.0 # Serve on global interface
export OLLAMA_KEEP_ALIVE=-1 # Do not unload model (default is 5 minutes)
ollama serve

After starting the SSH for portforwarding or on the login-node after setting <code>export OLLAMA_HOST=</code> to the allocated node (see output of <code>squeue</code>):
ollama run deepseek-r1:671b
>>> /? # Shows the help
>>> /? shortcuts # Shows the keyboard shortcuts
>>> /show # Show information regarding model, prompt
>>> What is log(e)? # Returns explanation of logarithm under the assumption of base 10 and the natural logarithm including LaTeX Math notation.

Registration/SSH/rrsync

2025-09-01T13:22:28Z

H Winkhardt: UC2 -> UC3

= RSYNC for SSH Command Keys =

If you want to use rsync for SSH command keys, you normally need one SSH key per directory that you want to synchronize.
With rrsync you can use one SSH key per main directory.

== Usage ==

You can find all the options in the [https://github.com/WayneD/rsync/blob/master/support/rrsync.1.md man page] for rrsync.
Below we provide some useful examples for daily use on the bwHPC clusters.

If you want to copy files from your home directory but only want to allow read operations, you can use the following SSH command:
<pre>
/usr/local/bin/rrsync -ro /home/aa/aa_bb/aa_abc1/
</pre>
[[File:Ssh-com.png|center|600px|thumb|Add command SSH key to service.]]

Then you can run the following rsync command (replace login.cluster.bw with the node you want to use):
<pre>
rsync -av login.cluster.bw:homesubdir/ localsubdir/
</pre>

For read-only access to your workspaces, you could use the following command:
<pre>
rsync -av localsubdir/ login.cluster.bw:homesubdir/
/usr/local/bin/rrsync error: sending to read-only server is not allowed
</pre>

For write-only access to your workspaces, you could use the following command:
<pre>
/usr/local/bin/rrsync -wo /workspace/aa_abc1-abc-0
</pre>
[[File:Ssh-com.png|center|600px|thumb|Add command SSH key to service.]]

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
With rrsync, you must use relative paths starting from the configured path in your SSH command.
|}

Then you could run the following rsync command (replace login.cluster.bw with the node you want to use and your user home):
<pre>
rsync -av localsubdir/ login.cluster.bw:workspacesubdir/
</pre>

Reading is not permitted with read-only access.
<pre>
rsync -av login.cluster.bw:workspacesubdir/ localsubdir/
/usr/local/bin/rrsync error: reading from write-only server is not allowed
</pre>

If you do not want to restrict read or write access, simply omit '-ro' or '-wo' from your SSH command:
<pre>
/usr/local/bin/rrsync -ro /home/aa/aa_bb/aa_abc1/
</pre>

== rrsync Path on the Clusters ==

If you want to use the rrysnc command, you must specify the following paths on the clusters:
* bwUniCluster3.0:
<pre>/usr/bin/rrsync</pre>
* HELIX/NEMO:
<pre>/usr/local/bin/rrsync</pre>

Rjags

2025-09-01T13:19:44Z

H Winkhardt: UC2 -> UC3

#REDIRECT [[BwUniCluster3.0/Software/R/Rjags]]

BwUniCluster

2025-09-01T13:17:06Z

H Winkhardt: UC2 -> UC3

#REDIRECT [[BwUniCluster3.0]]

MATLAB

2025-09-01T13:16:00Z

H Winkhardt: UC2 -> UC3

'''MATLAB''' (MATrix LABoratory) is a high-level programming language and interactive computing environment for numerical calculation and data visualization.

Each cluster provides its own Matlab installation and documentation.

To learn more about Matlab installations, visit the appropriate cluster site.

* [[bwUniCluster3.0/Software/Matlab]]
* [[Helix/Software/Matlab]]
* [[NEMO2/Software/Matlab]]

Registration/Login

2025-09-01T13:06:44Z

H Winkhardt: Fixed Redlink to Account

= Login to the Clusters =

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
Access to the clusters (bwUniCluster/bwForCluster) is restricted to IP addresses of universities/colleges from Baden-Württemberg [https://bgpview.io/asn/553#prefixes-v4 (BelWü network)].
If you are outside the BelWü network (e.g. at home), you must first establish a VPN connection to your home university or a connection to an SSH jump host at your home university.
|}

== SSH Clients ==

After completing the [[Registration|web registration]] and [[Registration/Password|setting a service password]] the HPC cluster is ready for your '''SSH''' based login.
* [[Registration/Login/Client| What Client to Use]]

== Cluster Specific Information ==

Every cluster has its own documentation for login.
* If you want to '''login''' to the bwUniCluster, please refer to 
→ '''[[bwUniCluster3.0/Login|bwUniCluster 3.0]]''' 
* If you want to '''login''' to one of the bwForClusters, please refer to 
→ '''[[BinAC/Login|BINAC]]''' / '''[[BinAC2/Login|BINAC2]]''' 
→ '''[[JUSTUS2/Login|JUSTUS 2]]''' 
→ '''[[Helix/Login|Helix]]''' 
→ '''[[NEMO2/Login|NEMO2]]'''

= Allowed Activities on Login Nodes =

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
To guarantee usability for all the users of clusters you must not run your compute jobs on the login nodes.
Compute jobs must be submitted to the queuing system.
Any compute job running on the login nodes will be terminated without any notice.
Any long-running compilation or any long-running pre- or post-processing of batch jobs must also be submitted to the queuing system.
|}

The login nodes of the bwHPC clusters are the access point to the compute system, your <code>$HOME</code> directory and your workspaces.
These nodes are shared with all the users therefore, your activities on the login nodes are limited to primarily set up your batch jobs.
Your activities may also be:
* '''short''' compilation of your program code and
* '''short''' pre- and post-processing of your batch jobs.

= Additional Login Information =

Following sub-pages go into deeper detail about the following topics:
* [[Registration/Login/Username|How do I find out my cluster username?]]
* [[Registration/Login/Hostname|What are the hostnames of the login nodes of a cluster?]]
* [[Registration/Account|Is my account on the cluster still valid?]]
* Configuring your shell: [[.bashrc Do's and Don'ts]]
These pages are also referenced in the cluster-specific login documentations.

Registration/Login

2025-09-01T13:05:21Z

H Winkhardt: Removed UC2 and added BINAC2

= Login to the Clusters =

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
Access to the clusters (bwUniCluster/bwForCluster) is restricted to IP addresses of universities/colleges from Baden-Württemberg [https://bgpview.io/asn/553#prefixes-v4 (BelWü network)].
If you are outside the BelWü network (e.g. at home), you must first establish a VPN connection to your home university or a connection to an SSH jump host at your home university.
|}

== SSH Clients ==

After completing the [[Registration|web registration]] and [[Registration/Password|setting a service password]] the HPC cluster is ready for your '''SSH''' based login.
* [[Registration/Login/Client| What Client to Use]]

== Cluster Specific Information ==

Every cluster has its own documentation for login.
* If you want to '''login''' to the bwUniCluster, please refer to 
→ '''[[bwUniCluster3.0/Login|bwUniCluster 3.0]]''' 
* If you want to '''login''' to one of the bwForClusters, please refer to 
→ '''[[BinAC/Login|BINAC]]''' / '''[[BinAC2/Login|BINAC2]]''' 
→ '''[[JUSTUS2/Login|JUSTUS 2]]''' 
→ '''[[Helix/Login|Helix]]''' 
→ '''[[NEMO2/Login|NEMO2]]'''

= Allowed Activities on Login Nodes =

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
To guarantee usability for all the users of clusters you must not run your compute jobs on the login nodes.
Compute jobs must be submitted to the queuing system.
Any compute job running on the login nodes will be terminated without any notice.
Any long-running compilation or any long-running pre- or post-processing of batch jobs must also be submitted to the queuing system.
|}

The login nodes of the bwHPC clusters are the access point to the compute system, your <code>$HOME</code> directory and your workspaces.
These nodes are shared with all the users therefore, your activities on the login nodes are limited to primarily set up your batch jobs.
Your activities may also be:
* '''short''' compilation of your program code and
* '''short''' pre- and post-processing of your batch jobs.

= Additional Login Information =

Following sub-pages go into deeper detail about the following topics:
* [[Registration/Login/Username|How do I find out my cluster username?]]
* [[Registration/Login/Hostname|What are the hostnames of the login nodes of a cluster?]]
* [[Registration/Login/Account|Is my account on the cluster still valid?]]
* Configuring your shell: [[.bashrc Do's and Don'ts]]
These pages are also referenced in the cluster-specific login documentations.

Registration/Login/Hostname

2025-09-01T12:59:33Z

H Winkhardt: Fixed no of Helix Login nodes

= Login Hostnames of the bwHPC Clusters =

From outside the clusters users can only use the login nodes to submit jobs.

Please go to the section of the cluster you want to login:

* '''[[Registration/Login/Hostname#bwUniCluster_3.0_Hostnames|bwUniCluster 3.0 Hostnames]]'''
* '''[[Registration/Login/Hostname#BINAC Hostnames|BINAC Hostnames]]'''
* '''[[Registration/Login/Hostname#BINAC2 Hostnames|BINAC2 Hostnames]]'''
* '''[[Registration/Login/Hostname#JUSTUS2 Hostnames|JUSTUS2 Hostnames]]'''
* '''[[Registration/Login/Hostname#Helix Hostnames|Helix Hostnames]]'''
* '''[[Registration/Login/Hostname#NEMO2 Hostnames|NEMO2 Hostnames]]'''

== bwUniCluster 3.0 Hostnames ==

The system has two login nodes.
The selection of the login node is done automatically.
If you are logging in multiple times, different sessions might run on different login nodes.

Login to bwUniCluster 3.0:

{| class="wikitable"
! Hostname !! Node type
|-
| '''uc3.scc.kit.edu''' || login to one of the two login nodes
|-
| '''bwunicluster.scc.kit.edu''' || login to one of the two login nodes
|-
|}

In general, you should use automatic selection to allow us to balance the load over the four login nodes.
If you need to connect to specific login nodes, you can use the following hostnames:

{| class="wikitable"
! Hostname !! Node type
|-
| '''uc3-login1.scc.kit.edu''' || bwUniCluster 3.0 first login node
|-
| '''uc3-login2.scc.kit.edu''' || bwUniCluster 3.0 second login node
|-
|}

== BINAC Hostnames ==

The system has three login nodes.
You have to select the login node yourself.

Login to BINAC:

{| class="wikitable"
! Hostname !! Node type
|-
| '''login01.binac.uni-tuebingen.de''' || BINAC first login node
|-
| '''login02.binac.uni-tuebingen.de''' || BINAC second login node
|-
| '''login03.binac.uni-tuebingen.de''' || BINAC third login node
|-
|}

== BINAC2 Hostnames ==

BinAC 2 has one login node serving as a load balancer. We use DNS round-robin scheduling to load-balance the incoming connections between the actual three login nodes. If you are logging in multiple times, different sessions might run on different login nodes and hence programs started in one session might not be visible in another sessions.

{| class="wikitable"
! Hostname !! Destination
|-
| '''login.binac2.uni-tuebingen.de''' || one of the three login nodes
|-
|}

== JUSTUS2 Hostnames ==

The system has four login nodes.
The selection of the login node is done automatically.
If you are logging in multiple times, different sessions might run on different login nodes.

Login to JUSTUS2:

{| class="wikitable"
! Hostname !! Node type
|-
| '''justus2.uni-ulm.de''' || login to one of the four login nodes
|-
|}

In general, you should use automatic selection to allow us to balance the load over the four login nodes.
If you need to connect to specific login nodes, you can use the following hostnames:

{| class="wikitable"
! Hostname !! Node type
|-
| '''justus2-login01.rz.uni-ulm.de''' || JUSTUS2 first login node
|-
| '''justus2-login02.rz.uni-ulm.de''' || JUSTUS2 second login node
|-
| '''justus2-login03.rz.uni-ulm.de''' || JUSTUS2 third login node
|-
| '''justus2-login04.rz.uni-ulm.de''' || JUSTUS2 fourth login node
|-
|}

== Helix Hostnames ==

The system has two login nodes.
The selection of the login node is done automatically.
If you are logging in multiple times, different sessions might run on different login nodes.

Login to Helix:

{| class="wikitable"
! Hostname !! Node type
|-
| '''helix.bwservices.uni-heidelberg.de''' || login to one of two login nodes
|-
|}

In general, you should use automatic selection to allow us to balance the load over the login nodes.

== NEMO2 Hostnames ==

The system has two login nodes.
You have to select the login node yourself.

Login to NEMO2:

{| class="wikitable"
! Hostname !! Node type
|-
| '''login.nemo.uni-freiburg.de''' || NEMO2 first or second login node
|-
| '''login1.nemo.uni-freiburg.de''' || NEMO2 first login node
|-
| '''login2.nemo.uni-freiburg.de''' || NEMO2 second login node
|-
|}

Registration/Login/Hostname

2025-09-01T12:58:34Z

H Winkhardt: Added BINAC2

= Login Hostnames of the bwHPC Clusters =

From outside the clusters users can only use the login nodes to submit jobs.

Please go to the section of the cluster you want to login:

* '''[[Registration/Login/Hostname#bwUniCluster_3.0_Hostnames|bwUniCluster 3.0 Hostnames]]'''
* '''[[Registration/Login/Hostname#BINAC Hostnames|BINAC Hostnames]]'''
* '''[[Registration/Login/Hostname#BINAC2 Hostnames|BINAC2 Hostnames]]'''
* '''[[Registration/Login/Hostname#JUSTUS2 Hostnames|JUSTUS2 Hostnames]]'''
* '''[[Registration/Login/Hostname#Helix Hostnames|Helix Hostnames]]'''
* '''[[Registration/Login/Hostname#NEMO2 Hostnames|NEMO2 Hostnames]]'''

== bwUniCluster 3.0 Hostnames ==

The system has two login nodes.
The selection of the login node is done automatically.
If you are logging in multiple times, different sessions might run on different login nodes.

Login to bwUniCluster 3.0:

{| class="wikitable"
! Hostname !! Node type
|-
| '''uc3.scc.kit.edu''' || login to one of the two login nodes
|-
| '''bwunicluster.scc.kit.edu''' || login to one of the two login nodes
|-
|}

In general, you should use automatic selection to allow us to balance the load over the four login nodes.
If you need to connect to specific login nodes, you can use the following hostnames:

{| class="wikitable"
! Hostname !! Node type
|-
| '''uc3-login1.scc.kit.edu''' || bwUniCluster 3.0 first login node
|-
| '''uc3-login2.scc.kit.edu''' || bwUniCluster 3.0 second login node
|-
|}

== BINAC Hostnames ==

The system has three login nodes.
You have to select the login node yourself.

Login to BINAC:

{| class="wikitable"
! Hostname !! Node type
|-
| '''login01.binac.uni-tuebingen.de''' || BINAC first login node
|-
| '''login02.binac.uni-tuebingen.de''' || BINAC second login node
|-
| '''login03.binac.uni-tuebingen.de''' || BINAC third login node
|-
|}

== BINAC2 Hostnames ==

BinAC 2 has one login node serving as a load balancer. We use DNS round-robin scheduling to load-balance the incoming connections between the actual three login nodes. If you are logging in multiple times, different sessions might run on different login nodes and hence programs started in one session might not be visible in another sessions.

{| class="wikitable"
! Hostname !! Destination
|-
| '''login.binac2.uni-tuebingen.de''' || one of the three login nodes
|-
|}

== JUSTUS2 Hostnames ==

The system has four login nodes.
The selection of the login node is done automatically.
If you are logging in multiple times, different sessions might run on different login nodes.

Login to JUSTUS2:

{| class="wikitable"
! Hostname !! Node type
|-
| '''justus2.uni-ulm.de''' || login to one of the four login nodes
|-
|}

In general, you should use automatic selection to allow us to balance the load over the four login nodes.
If you need to connect to specific login nodes, you can use the following hostnames:

{| class="wikitable"
! Hostname !! Node type
|-
| '''justus2-login01.rz.uni-ulm.de''' || JUSTUS2 first login node
|-
| '''justus2-login02.rz.uni-ulm.de''' || JUSTUS2 second login node
|-
| '''justus2-login03.rz.uni-ulm.de''' || JUSTUS2 third login node
|-
| '''justus2-login04.rz.uni-ulm.de''' || JUSTUS2 fourth login node
|-
|}

== Helix Hostnames ==

The system has four login nodes.
The selection of the login node is done automatically.
If you are logging in multiple times, different sessions might run on different login nodes.

Login to Helix:

{| class="wikitable"
! Hostname !! Node type
|-
| '''helix.bwservices.uni-heidelberg.de''' || login to one of two login nodes
|-
|}

In general, you should use automatic selection to allow us to balance the load over the login nodes.

== NEMO2 Hostnames ==

The system has two login nodes.
You have to select the login node yourself.

Login to NEMO2:

{| class="wikitable"
! Hostname !! Node type
|-
| '''login.nemo.uni-freiburg.de''' || NEMO2 first or second login node
|-
| '''login1.nemo.uni-freiburg.de''' || NEMO2 first login node
|-
| '''login2.nemo.uni-freiburg.de''' || NEMO2 second login node
|-
|}

Registration/Login/Hostname

2025-09-01T12:53:53Z

H Winkhardt: Remove UC2

= Login Hostnames of the bwHPC Clusters =

From outside the clusters users can only use the login nodes to submit jobs.

Please go to the section of the cluster you want to login:

* '''[[Login/Hostname#bwUniCluster_3.0_Hostnames|bwUniCluster 3.0 Hostnames]]'''
* '''[[Login/Hostname#BINAC Hostnames|BINAC Hostnames]]'''
* '''[[Login/Hostname#JUSTUS2 Hostnames|JUSTUS2 Hostnames]]'''
* '''[[Login/Hostname#Helix Hostnames|Helix Hostnames]]'''
* '''[[Login/Hostname#NEMO2 Hostnames|NEMO2 Hostnames]]'''

== bwUniCluster 3.0 Hostnames ==

The system has two login nodes.
The selection of the login node is done automatically.
If you are logging in multiple times, different sessions might run on different login nodes.

Login to bwUniCluster 3.0:

{| class="wikitable"
! Hostname !! Node type
|-
| '''uc3.scc.kit.edu''' || login to one of the two login nodes
|-
| '''bwunicluster.scc.kit.edu''' || login to one of the two login nodes
|-
|}

In general, you should use automatic selection to allow us to balance the load over the four login nodes.
If you need to connect to specific login nodes, you can use the following hostnames:

{| class="wikitable"
! Hostname !! Node type
|-
| '''uc3-login1.scc.kit.edu''' || bwUniCluster 3.0 first login node
|-
| '''uc3-login2.scc.kit.edu''' || bwUniCluster 3.0 second login node
|-
|}

== BINAC Hostnames ==

The system has three login nodes.
You have to select the login node yourself.

Login to BINAC:

{| class="wikitable"
! Hostname !! Node type
|-
| '''login01.binac.uni-tuebingen.de''' || BINAC first login node
|-
| '''login02.binac.uni-tuebingen.de''' || BINAC second login node
|-
| '''login03.binac.uni-tuebingen.de''' || BINAC third login node
|-
|}

== JUSTUS2 Hostnames ==

The system has four login nodes.
The selection of the login node is done automatically.
If you are logging in multiple times, different sessions might run on different login nodes.

Login to JUSTUS2:

{| class="wikitable"
! Hostname !! Node type
|-
| '''justus2.uni-ulm.de''' || login to one of the four login nodes
|-
|}

In general, you should use automatic selection to allow us to balance the load over the four login nodes.
If you need to connect to specific login nodes, you can use the following hostnames:

{| class="wikitable"
! Hostname !! Node type
|-
| '''justus2-login01.rz.uni-ulm.de''' || JUSTUS2 first login node
|-
| '''justus2-login02.rz.uni-ulm.de''' || JUSTUS2 second login node
|-
| '''justus2-login03.rz.uni-ulm.de''' || JUSTUS2 third login node
|-
| '''justus2-login04.rz.uni-ulm.de''' || JUSTUS2 fourth login node
|-
|}

== Helix Hostnames ==

The system has four login nodes.
The selection of the login node is done automatically.
If you are logging in multiple times, different sessions might run on different login nodes.

Login to Helix:

{| class="wikitable"
! Hostname !! Node type
|-
| '''helix.bwservices.uni-heidelberg.de''' || login to one of two login nodes
|-
|}

In general, you should use automatic selection to allow us to balance the load over the login nodes.

== NEMO2 Hostnames ==

The system has two login nodes.
You have to select the login node yourself.

Login to NEMO2:

{| class="wikitable"
! Hostname !! Node type
|-
| '''login.nemo.uni-freiburg.de''' || NEMO2 first or second login node
|-
| '''login1.nemo.uni-freiburg.de''' || NEMO2 first login node
|-
| '''login2.nemo.uni-freiburg.de''' || NEMO2 second login node
|-
|}

Acknowledgement

2025-08-21T15:09:07Z

H Winkhardt: Update NEMO2 und BinAC2

Remember to acknowledge our resources in your publications!

Such recognition is important for acquiring funding for the next generation of hardware, support services, data storage, and infrastructure.

The publications will be referenced on the bwHPC website: https://www.bwhpc.de/user_publications.html

== HPC Clusters ==

→[[BwUniCluster2.0/Acknowledgement| bwUniCluster Acknowledgement]]

→[[BinAC2/Acknowledgement| bwForCluster BinAC2 Acknowledgement]]

→[[Helix/Acknowledgement| bwForCluster Helix Acknowledgement]]

→[[JUSTUS2/Acknowledgement| bwForCluster JUSTUS2 Acknowledgement]]

→[[NEMO2/Acknowledgement| bwForCluster NEMO2 Acknowledgement]]

== Data Facilities ==

→[[SDS@hd/Acknowledgement| SDS@hd Acknowledgement]]

BWUniCluster User Access Members Uni Mannheim

2025-08-21T13:07:35Z

H Winkhardt: Update to UC3

= Entitlement for bwUniCluster @ Uni Mannheim =

Members of the University of Mannheim need to apply for an entitlement with a valid university-account to use bwHPC resources.
Please use this web registration service:

[https://entitlements.uni-mannheim.de https://entitlements.uni-mannheim.de]

The entitlement service asks for a project description, which is used for internal documentation only. Please indicate that you want access to the '''bwUniCluster'''.

With the approval of your request, you will be able to [[Registration/bwUniCluster/Service|register for the '''bwUniCluster''']].

If you have any questions, please contact [mailto:hpc-support@mailman.uni-mannheim.de hpc-support@mailman.uni-mannheim.de] .

== Terms of Use ==

As with any other UNIT-service, users of the bwUniCluster must strictly adhere to the [https://www.uni-mannheim.de/media/Einrichtungen/it/Benutzerordnung_und_wichtige_Dokumente/Benutzungsordnung_Informationssysteme_Universitaet_Mannheim.pdf terms of use of Mannheim University].

== Acknowledgement ==

Publications written with the help of the bwUniCluster must [[BwUniCluster3.0/Acknowledgement | acknowledge the support given by the bwHPC-project]].

= Technical details =

Users from Mannheim have a special quota in the [[BwUniCluster3.0/Hardware_and_Architecture/Filesystem_Details#$HOME|HOME file system]] on bwUniCluster3.0, 250 GiB and 2.5 million inodes. Workspaces are unchanged from the default.

Helix/Software/Matlab

2025-08-14T09:13:10Z

H Winkhardt: -r deprecated in favor of -batch

{{Softwarepage|math/matlab}}

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| math/matlab
|-
| License
| [https://de.mathworks.com/pricing-licensing/index.html?intendeduse=edu&prodcode=ML Academic License/Commercial]
|-
| Citing
| n/a
|-
| Links
| [https://de.mathworks.com/products/matlab/ MATLAB Homepage] | [https://de.mathworks.com/index.html?s_tid=gn_logo MathWorks Homepage] | [https://de.mathworks.com/support/?s_tid=gn_supp Support and more]
|-
| Graphical Interface
| No
|}

= Description =

'''MATLAB''' (MATrix LABoratory) is a high-level programming language and interactive computing environment for numerical calculation and data visualization.

= Loading MATLAB =

The preferable way is to run the MATLAB command line interface without GUI:
<pre>$ matlab -nodisplay</pre>

An interactive MATLAB session with graphical user interface (GUI) can be started with the command (requires X11 forwarding enabled for your ssh login):
<pre>$ matlab</pre>

Note: Do not start a long-duration interactive MATLAB session on a login node of the cluster. Submit an [[Helix/Slurm#Interactive_Jobs | interactive job]] and start MATLAB from within the dedicated compute node assigned to you by the queueing system.

The following generic command will execute a MATLAB script or function named "example":
<pre>$ matlab -nodisplay -batch example > result.out 2>&1</pre>

The output of this session will be redirected to the file result.out. The option <syntaxhighlight style="border:0px" inline=1>-batch</syntaxhighlight> executes the MATLAB statement non-interactively.

= Parallel Computing Using MATLAB =

Parallelization of MATLAB jobs is realized via the built-in multi-threading provided by MATLAB's BLAS and FFT implementation and the parallel computing functionality of MATLAB's Parallel Computing Toolbox (PCT).

== Implicit Threading ==

A large number of built-in MATLAB functions may utilize multiple cores automatically without any code modifications required. This is referred to as implicit multi-threading and must be strictly distinguished from explicit parallelism provided by the Parallel Computing Toolbox (PCT) which requires specific commands in your code in order to create threads.

Implicit threading particularly takes place for linear algebra operations (such as the solution to a linear system A\b or matrix products A*B) and FFT operations. Many other high-level MATLAB functions do also benefit from multi-threading capabilities of their underlying routines. If multi-threading is not desired, single-threading can be enforced by adding the command line option <syntaxhighlight style="border:0px" inline=1>-singleCompThread</syntaxhighlight>.

Whenever implicit threading takes place, MATLAB will detect the total number of cores that exist on a machine and by default makes use of all of them. This has very important implications for MATLAB jobs in HPC environments with shared-node job scheduling policy (i.e. with multiple users sharing one compute node). Due to this behaviour, a MATLAB job may take over more compute resources than assigned by the queueing system of the cluster (and thereby taking away these resources from all other users with running jobs on the same node - including your own jobs).

Therefore, when running in multi-threaded mode, MATLAB always requires the user's intervention to not allocate all cores of the machine (unless requested so from the queueing system). The number of threads must be controlled from within the code by means of the <syntaxhighlight style="border:0px" inline=1>maxNumCompThreads(N)</syntaxhighlight> function or, alternatively, with the <syntaxhighlight style="border:0px" inline=1>feature('numThreads', N)</syntaxhighlight> function (which is undocumented).

== Using the Parallel Computing Toolbox (PCT) ==

By using the PCT one can make explicit use of several cores on multicore processors to parallelize MATLAB applications without MPI programming. Under MATLAB version 8.4 and earlier, this toolbox provides 12 workers (MATLAB computational engines) to execute applications locally on a single multicore node. Under MATLAB version 8.5 and later, the number of workers available is equal to the number of cores on a single node (up to a maximum of 512).

If multiple PCT jobs are running at the same time, they all write temporary MATLAB job information to the same location. This race condition can cause one or more of the parallel MATLAB jobs fail to use the parallel functionality of the toolbox.

To solve this issue, each MATLAB job should explicitly set a unique location where these files are created. This can be accomplished by the following snippet of code added to your MATLAB script.

<pre>
% create a local cluster object
pc = parcluster('local')

% get the number of dedicated cores from environment
nprocs = str2num(getenv('SLURM_NPROCS'))

% you may explicitly set the JobStorageLocation to the tmp directory that is unique to each cluster job (and is on local, fast scratch)
parpool_tmpdir = [getenv('TMP'),'/.matlab/local_cluster_jobs/slurm_jobID_',getenv('SLURM_JOB_ID')]
mkdir(parpool_tmpdir)
pc.JobStorageLocation = parpool_tmpdir

% start the parallel pool
parpool(pc, nprocs)
</pre>

If a large number of MATLAB-jobs are run in parallel, they can also conflict when writing generic information to <syntaxhighlight style="border:0px" inline=1>~/.matlab</syntaxhighlight>. This can be circumvented by setting <syntaxhighlight style="border:0px" inline=1>$MATLAB_PREFDIR</syntaxhighlight> to different directories in your Batch-script, e.g.

<pre>export MATLAB_PREFDIR=$TMP</pre>

== Using a different implementation of BLAS/LAPACK ==

By default, Matlab uses a version of Intel MKL as its BLAS/LAPACK library. It is possible to manually change this to different libraries by setting the <syntaxhighlight style="border:0px" inline=1>BLAS_VERSION</syntaxhighlight> and <syntaxhighlight style="border:0px" inline=1>LAPACK_VERSION</syntaxhighlight> environment variables. The following lines can be added to the batch-script to change it, in this example to BLIS and Flame, which are optimized for AMD processors:

<pre>
module load numlib/aocl/3.2.0

export BLAS_VERSION=$AOCL_LIB_DIR/libblis-mt.so
export LAPACK_VERSION=$AOCL_LIB_DIR/libflame.so

export BLIS_NUM_THREADS=$SLURM_NTASKS
</pre>

This can increase performance depending on the task, for example large matrix multiplications, but caution is advised.

= General Performance Tips for MATLAB =

MATLAB data structures (arrays or matrices) are dynamic in size, i.e. MATLAB will automatically resize the structure on demand. Although this seems to be convenient, MATLAB continually needs to allocate a new chunk of memory and copy over the data to the new block of memory as the array or matrix grows in a loop. This may take a significant amount of extra time during execution of the program.

Code performance can often be drastically improved by pre-allocating memory for the final expected size of the array or matrix before actually starting the processing loop. In order to pre-allocate an array of strings, you can use MATLAB's build-in cell function. In order to pre-allocate an array or matrix of numbers, you can use MATLAB's build-in zeros function.

The performance benefit of pre-allocation is illustrated with the following example code.

<pre>
% prealloc.m

clear all;

num=10000000;

disp('Without pre-allocation:')
tic
for i=1:num
a(i)=i;
end
toc

disp('With pre-allocation:')
tic
b=zeros(1,num);
for i=1:num
b(i)=i;
end
toc
</pre>

On a compute node, the result may look like this:

<pre>
Without pre-allocation:
Elapsed time is 2.879446 seconds.
With pre-allocation:
Elapsed time is 0.097557 seconds.
</pre>

Please recognize that the code runs almost 30 times faster with pre-allocation.

= Compile MATLAB binaries with mcc =

MATLAB on Helix comes with a compiler, <code>mcc</code>, that can be used to create binaries from MATLAB-code.
Stand-alone MATLAB programs compiled with <code>mcc</code> do not require any license tokens at runtime and you can start jobs in parallel without any risk of running out of licences.

HPC Glossary/Batch system

2025-07-15T09:03:36Z

H Winkhardt: Moab raus

When we speak of a '''batch system''' on compute clusters, we mean the system that knows which compute nodes are used by whom and when they will become available. It also knows about all waiting jobs and determines which job are going to start next on which node whenever a node bekomes available.

== Why do we need a Resource Management System? ==

An HPC cluster is a multi-user system. Users have compute jobs with different demands on number of processor cores, memory, disk space and run-time. Some users run a program only occasionally for a big task, other users must run many simulations to finish their projects.

The cluster only provides a limited number of compute resources with certain features. Free access for all users to all compute nodes without time limit will not work. Therefore we need a resource management system (batch system) for the scheduling and the distribution of compute jobs on suitable compute resources.
The use of a resource management system pursues several objectives:

* Fair distribution of resources among users
* Compute jobs should start as soon as possible
* Full load and efficient usage of all resources
[[image:distributing_jobs1.svg]]

== How does a Resource Management System work? ==
A resource management system or batch system manages the compute nodes, jobs and queues and basically consists of two components:

* A resource manager which is responsible for the node status and for the distribution of jobs over the compute nodes.
* workload manager (scheduler) which is in charge of job scheduling, job managing, job monitoring and job reporting.

A resource management system works as follows:

* The user creates a job script containing requests for compute resources and submits the script to the resource management system.
* The scheduler parses the job script for resource requests and determines where to run the job and how to schedule it.
* The scheduler delegates the job to the resource manager.
* The resource manager executes the job and communicates the status information to the scheduler.

== How does a Job Scheduler work?==

The job scheduling process is influenced by many and sometimes contrasting parameters which are used as metrics for the scheduling algorithm. The objectives of a resource management system are approached in the following way:

* Fair distribution of resources among users: Ensure that all users get a fair share of processing time for their jobs.
* Compute jobs should start as soon as possible: Minimize the time jobs have to wait until they start.
* Full load and efficient usage of all resources: Aim for the highest possible utilization with the available jobs, because cluster resources are very expensive

The following simple example illustrates how a scheduler works.
Let us consider a system with 4 nodes. Each node has 4 processor cores.

Here the cluster is empty. No jobs are scheduled:

[[image:Scheduling-workload0.svg]]

Now jobs with different resource requests are scheduled:

* one job which needs two nodes (blue)
* one job which needs one node (red)
* multiple jobs, which need only one core (yellow, orange)
* some jobs are short running, others are long running (see time axis)

[[image:Scheduling-workload1.svg]]

Up to now enough resources are available so that all jobs can start instantly.

More jobs are submitted. Now not all job can start immediately on the available hardware resources. Some jobs are scheduled to run at a later time.

* Big jobs (in terms of hardware resources) often have to wait longer, because they cannot be scheduled flexibly.
* Long running jobs can delay the scheduling of big jobs, because they block needed hardware resources.
* Small and short jobs can be scheduled to fill gaps.
* If a job stops prematurely, short jobs can be rescheduled to start earlier (back filling).

[[image:Scheduling-workload2.svg]]

In addition to cores and time a scheduler has to consider many more metrics like memory, co-processors, fair share, and priority. Scheduling is a multi-dimensional optimization problem.

== Resource Management Systems on bwHPC Clusters ==

Slurm: complete resource management system with integrated resource manager and scheduler
All bwHPC cluster follow a fairshare policy. The waiting time of jobs depends on

* your job's resource requests: Jobs with large resource requests wait longer for free resources.
* your usage history: High resource usage in a short time leads to a lower job priority.
* your university's share (bwUniCluster only): If the usage exceeds the university's share, the job priority decreases.

bwHPC clusters have specific configurations for the resource management system concerning:

* Job submission and monitoring commands (via Slurm)
* Queues and limits for the available hardware
* Node access policy:
** shared: compute nodes can be shared by jobs of different users
** single user: compute nodes can be shared by jobs of a single user
** single job: compute nodes are allocated exclusively for a single job

FAQ: How many jobs can run on a single node at the same time?
This depends on the node access policy:

* On "shared nodes", more than one job can run simultaneously if the requested resources are available, and these jobs may have been submitted by different users.
* On "single user nodes", more than one job can run simultaneously, but the jobs have to be submitted by one and the same user
* On "single job nodes", the access is exclusive for one job, irrespective of the number of requested cores.