bwHPC Wiki - User contributions [en]

NEMO2/Workspaces

2026-02-11T08:58:49Z

F Bartusch: Group workspaces works with BinAC 2

<div style="border: 3px solid #ffc107; padding: 15px; background-color: #fff3cd; margin: 10px 0;">
'''Note:''' This is the updated Workspaces guide. The old wiki: [[Workspace]].
</div>

'''Workspace tools''' provide temporary storage spaces called '''workspaces''' for your calculations. They are meant for data that needs to persist longer than a single job, but not permanently.

== What are Workspaces? ==

Workspaces give you access to the cluster's fast parallel filesystems (like Lustre or Weka). '''You cannot write directly to these parallel filesystems''' - workspaces provide your designated area.

'''Use workspaces for:'''
* Jobs generating intermediate data
* Data shared between multiple compute nodes
* Multi-step workflows
* Temporary scratch space during calculations

'''Don't use workspaces for:'''
* Permanent storage (use HOME or project directories)
* Single-node temporary files (use <tt>$TMPDIR</tt> instead)

== Important - Read First ==

* '''No Backup:''' Data is '''not backed up''' and will be '''automatically deleted''' after expiration
* '''Time-limited:''' Lifetime typically 30-100 days (cluster-specific). See [[Workspaces/Advanced_Features/Quotas#Cluster-Specific_Workspace_Limits|Cluster Limits]]
* '''Automatic Reminders:''' Email notifications before expiration
* '''Backup Important Data:''' Copy results to permanent storage before expiration

'''For advanced features and detailed options:''' [[Workspaces/Advanced Features]]

== Command Overview ==

Main commands:

* <tt>ws_allocate</tt> - Create or extend workspace
* <tt>ws_list</tt> - List your workspaces
* <tt>ws_find</tt> - Find workspace path (for scripts)
* <tt>ws_extend</tt> - Extend workspace lifetime
* <tt>ws_release</tt> - Release (delete) workspace
* <tt>ws_restore</tt> - Restore expired/released workspace
* <tt>ws_register</tt> - Create symbolic links

All commands support <tt>-h</tt> for help.

== Quick Start ==

{| class="wikitable"
|-
!style="width:40%" | Task
!style="width:60%" | Command
|-
|Create workspace (30 days)
|<tt>ws_allocate myWs 30</tt>
|-
|Create group workspace
|<tt>ws_allocate -G groupname myWs 30</tt>
|-
|List all workspaces
|<tt>ws_list</tt>
|-
|See what expires soon
|<tt>ws_list -Rr</tt> (<tt>-R</tt>=by time, <tt>-r</tt>=reverse)
|-
|Find path (for scripts)
|<tt>ws_find myWs</tt>
|-
|Extend by 30 days
|<tt>ws_extend myWs 30</tt>
|-
|Delete workspace
|<tt>ws_release myWs</tt>
|-
|Restore workspace
|<tt>ws_restore -l</tt> then <tt>ws_restore oldname newname</tt>
|}

== Create Workspace ==

Create a workspace with a '''name''' and '''lifetime''' in days:

$ ws_allocate myWs 30 # Create for 30 days

Returns:

Workspace created. Duration is 720 hours.
Further extensions available: 3
/work/workspace/scratch/username-myWs-0

'''Common options:'''

$ ws_allocate -G groupname myWs 30 # Group-writable (for teams)
$ ws_allocate -g myWs 30 # Group-readable
$ ws_allocate -F ffuc myWs 30 # bwUniCluster 3.0: Flash filesystem

'''Capture path in variable:'''

$ WORKSPACE=$(ws_allocate myWs 30)
$ cd "$WORKSPACE"

'''Important:''' Running the same command again is safe - returns the existing workspace path.

'''Details:''' [[Workspaces/Advanced_Features/ws_allocate|Advanced Features guide]]

== List Your Workspaces ==

$ ws_list # List all workspaces

Shows workspace ID, location, extensions, creation date, and remaining time.

'''Common options:'''

$ ws_list -Rr # Sort by remaining time, reverse (last to expire first)
$ ws_list -s # Short format (names only, for scripts)
$ ws_list -g # Show group workspaces

'''Note:''' To list expired workspaces for restore, use <tt>ws_restore -l</tt>. See [[#Restore_Workspace|Restore]].

'''Details:''' [[Workspaces/Advanced_Features/ws_list|Advanced Features guide]]

== Extend Workspace Lifetime ==

Extend before workspace expires:

$ ws_extend myWs 30 # Extend by 30 days from now

'''Alternative:'''

$ ws_allocate -x myWs 30 # Same result

'''Note:''' Each extension consumes one available extension. See [[Workspaces/Advanced_Features/Quotas#Cluster-Specific_Workspace_Limits|Cluster Limits]].

'''Group workspaces:''' See [[#Extend_Group_Workspace|Extend Group Workspace]] for extending workspaces created by colleagues.

'''Details:''' [[Workspaces/Advanced_Features/ws_extend|Advanced Features guide]]

== Release (Delete) Workspace ==

When no longer needed:

$ ws_release myWs # Release workspace

'''What happens:'''
* Workspace becomes inaccessible immediately
* Data kept for short grace period (typically until next cleanup)
* Can be restored with <tt>ws_restore</tt> during grace period
* May still count toward quota until final deletion

'''Details:''' [[Workspaces/Advanced_Features/ws_release|Advanced Features guide]]

== Restore Workspace ==

{| class="wikitable"
|-
!style="width:40%" | Works on cluster
!style="width:10%" | bwUC 3.0
!style="width:10%" | BinAC2
!style="width:10%" | Helix
!style="width:10%" | JUSTUS 2
!style="width:10%" | NEMO2
!style="width:10%" | DACHS
|-
|<tt>ws_restore</tt>
|style="background-color:#90EE90; text-align:center;" | ✓
|style="background-color:#90EE90; text-align:center;" | ✓
|style="background-color:#90EE90; text-align:center;" | ✓
|style="background-color:#90EE90; text-align:center;" | ✓
|style="background-color:#90EE90; text-align:center;" | ✓
|style="background-color:#90EE90; text-align:center;" | ✓
|}

Recover expired or released workspaces within grace period:

'''Restore procedure:'''

$ ws_restore -l # (1) List restorable workspaces
$ ws_allocate restored 60 # (2) Create target workspace
$ ws_restore username-myWs-0 restored # (3) Restore

'''Important:''' Use the '''full name''' from <tt>ws_restore -l</tt> (with username and timestamp), not the short name.

'''Details:''' [[Workspaces/Advanced_Features/ws_restore|Advanced Features guide]]

== Work with Groups (Share Workspaces) ==

{| class="wikitable"
|-
!style="width:40%" | Works on cluster
!style="width:10%" | bwUC 3.0
!style="width:10%" | BinAC2
!style="width:10%" | Helix
!style="width:10%" | JUSTUS 2
!style="width:10%" | NEMO2
|-
|<tt>-g</tt> (group-readable)
| style="text-align:center;" |
| style="background-color:#90EE90; text-align:center;" | ✓
| style="text-align:center;" |
| style="text-align:center;" |
|style="background-color:#90EE90; text-align:center;" | ✓
|-
|<tt>-G</tt> (group-writable)
| style="text-align:center;" |
| style="background-color:#90EE90; text-align:center;" | ✓
| style="text-align:center;" |
| style="text-align:center;" |
|style="background-color:#90EE90; text-align:center;" | ✓
|}

Simple team collaboration with group workspaces:

=== Create Group Workspace ===

'''Group-readable (read-only):'''

$ ws_allocate -g myWs 30

'''Group-writable (recommended):'''

$ ws_allocate -G projectgroup myWs 30 # Replace 'projectgroup' with your group

'''Tip:''' Set default in <tt>~/.ws_user.conf</tt>:

<pre>
groupname: projectgroup
</pre>

Then simply: <tt>ws_allocate myWs 30</tt>

=== List Group Workspaces ===

$ ws_list -g # Show all group workspaces

=== Extend Group Workspace ===

Anyone in the group can extend group-writable workspaces (<tt>-G</tt>):

$ ws_extend myWs 30 # If you created it
$ ws_allocate -x -u alice myWs 30 # If colleague created it

=== Manage Reminders ===

Take over reminder responsibility:

$ ws_allocate -r 7 -u alice -x myWs 0 # Update timing and take over reminders

Changes reminder to 7 days before expiration and redirects emails to you.

=== Why Use Group Workspaces? ===

* Simple collaboration - everyone accesses same data
* No permission problems - automatic group permissions
* Independent extensions - team can extend without creator
* Easy discovery - <tt>ws_list -g</tt> shows all team workspaces

'''Advanced sharing:''' [[Workspaces/Advanced_Features/Sharing|Sharing guide]] for ACLs and ws_share

BinAC2/Hardware and Architecture

2026-02-11T08:57:04Z

F Bartusch: Change link to new workspace documentation

= Hardware and Architecture =

The bwForCluster BinAC 2 supports researchers from the broader fields of Bioinformatics, Medical Informatics, Astrophysics, Geosciences and Pharmacy.

[[File:Binac2 schema.png|600px|thumb|center|Overview on the BinAC 2 hardware architecture.]]

== Operating System and Software ==

* Operating System: Rocky Linux 9.6
* Queuing System: [https://slurm.schedmd.com/documentation.html Slurm] (see [[BinAC2/Slurm]] for help)
* (Scientific) Libraries and Software: [[Environment Modules]]

== Compute Nodes ==

BinAC 2 offers compute nodes, high-mem nodes, and three types of GPU nodes.
* 180 compute nodes
* 16 SMP node
* 32 GPU nodes (2xA30)
* 8 GPU nodes (4xA100)
* 4 GPU nodes (4xH200)
* plus several special purpose nodes for login, interactive jobs, etc.

Compute node specification:
{| class="wikitable"
|-
! style="width:10%"|
! style="width:10%"| Standard
! style="width:10%"| High-Mem
! style="width:10%"| GPU (A30)
! style="width:10%"| GPU (A100)
! style="width:10%"| GPU (H200)
|-
!scope="column"| Quantity
| 168 / 12
| 14 / 2
| 32
| 8
| 4
|-
!scope="column" | Processors
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7543.html AMD EPYC Milan 7543] / 2 x [https://www.amd.com/en/products/processors/server/epyc/7003-series/amd-epyc-75f3.html AMD EPYC Milan 75F3]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7443.html AMD EPYC Milan 7443] / 2 x [https://www.amd.com/en/products/processors/server/epyc/7003-series/amd-epyc-75f3.html AMD EPYC Milan 75F3]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7543.html AMD EPYC Milan 7543]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7543.html AMD EPYC Milan 7543]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/9005-series/amd-epyc-9555.html AMD EPYC Milan 9555]
|-
!scope="column" | Processor Base Frequency (GHz)
| 2.80 / 2.95
| 2.85 / 2.95
| 2.80
| 2.80
| 3.20
|-
!scope="column" | Number of Physical Cores / Hypertreads
| 64 / 128
| 48 / 96 // 64 / 128
| 64 / 128
| 64 / 128
| 128 / 256
|-
!scope="column" | Working Memory (GB)
| 512
| 2048
| 512
| 512
| 1536
|-
!scope="column" | Local Disk (GiB)
| 450 (NVMe-SSD)
| 14000 (NVMe-SSD)
| 450 (NVMe-SSD)
| 14000 (NVMe-SSD)
| 28000 (NVMe-SSD)
|-
!scope="column" | Interconnect
| HDR 100 IB (84 nodes) / 100GbE (96 nodes)
| 100GbE
| 100GbE
| 100GbE
| HDR 200 IB + 100GbE
|-
!scope="column" | Coprocessors
| -
| -
| 2 x [https://www.nvidia.com/de-de/data-center/products/a30-gpu/ NVIDIA A30 (24 GB ECC HBM2, NVLink)]
| 4 x [https://www.nvidia.com/de-de/data-center/a100/ NVIDIA A100 (80 GB ECC HBM2e)]
| 4 x [https://www.nvidia.com/de-de/data-center/h200/ NVIDIA H200 NVL (141 GB ECC HBM3e, NVLink)]
|}

= Network =

The compute nodes and the parallel file system are connected via 100GbE ethernet
In contrast to BinAC 1 not all compute nodes are connected via Infiniband, but there are 84 standard compute nodes connected via HDR Infiniband (100 GbE). In order to get your jobs onto the Infiniband nodes, submit your job with <code>--constraint=ib</code>.

'''Question:'''
OpenMPI throws the following warning:
<pre>
--------------------------------------------------------------------------
No OpenFabrics connection schemes reported that they were able to be
used on a specific port. As such, the openib BTL (OpenFabrics
support) will be disabled for this port.
Local host: node1-083
Local device: mlx5_0
Local port: 1
CPCs attempted: rdmacm, udcm
--------------------------------------------------------------------------
[node1-083:2137377] 3 more processes have sent help message help-mpi-btl-openib-cpc-base.txt / no cpcs for port
[node1-083:2137377] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
</pre>
What should i do?

'''Answer:'''
BinAC2 has two (almost) separate networks, a 100GbE network and and InfiniBand network, both connecting a subset of the nodes. Both networks require different cables and switches.
Concerning the network cards for the nodes, however, there exist VPI network cards which can be configured to work in either mode (https://docs.nvidia.com/networking/display/connectx6vpi/specifications#src-2487215234_Specifications-MCX653105A-ECATSpecifications).
OpenMPI can use a number of layers for transferring data and messages between processes. When it ramps up, it will test all means of communication that were configured during compilation and then tries to figure out the fastest path between all processes.
If OpenMPI encounters such a VPI card, it will first try to establish a Remote Direct Memory Access communication (RDMA) channel using the OpenFabrics (OFI) layer.
On nodes with 100Gb ethernet, this fails as there is no RDMA protocol configured. OpenMPI will fall back to TCP transport but not without complaints.

'''Workaround:'''
For single-node jobs or on regular compute nodes, A30 and A100 GPU nodes: Add the lines
<pre>
export OMPI_MCA_btl="^ofi,openib"
export OMPI_MCA_mtl="^ofi"
</pre>
to your job script to disable the OFI transport layer. If you need high-bandwidth, low-latency transport between all processes on all nodes, switch to the Infiniband partition (<code>#SBATCH --constraint=ib</code>). ''Do not turn off the OFI layer on Infiniband nodes as this will be the best choice between nodes!''

= File Systems =

The bwForCluster BinAC 2 consists of two separate storage systems, one for the user's home directory $HOME and one serving as a project/work space.
The home directory is limited in space and parallel access but offers snapshots of your files and backup.

The project/work is a parallel file system (PFS) which offers fast and parallel file access and a bigger capacity than the home directory. It is mounted at <code>/pfs/10</code> on the login and compute nodes. This storage is based on Lustre and can be accessed in parallel from many nodes. The PFS contains the project and the work directory. Each compute project has its own directory at <code>/pfs/10/project</code> that is accessible for all members of the compute project.
Each user can create workspaces under <code>/pfs/10/work</code> using the workspace tools. These directories are only accessible for the user who created the workspace.

Additionally, each compute node provides high-speed temporary storage (SSD) on the node-local solid state disk via the $TMPDIR environment variable.

{| class="wikitable"
|-
! style="width:10%"|
! style="width:10%"| <tt>$HOME</tt>
! style="width:10%"| project
! style="width:10%"| work
! style="width:10%"| <tt>$TMPDIR</tt>
|-
!scope="column" | Visibility
| global
| global
| global
| node local
|-
!scope="column" | Lifetime
| permanent
| permanent
| work space lifetime (max. 30 days, max. 5 extensions)
| batch job walltime
|-
!scope="column" | Capacity
| -
| 8.1 PB
| 1000 TB
| 480 GB (compute nodes); 7.7 TB (GPU-A30 nodes); 16 TB (GPU-A100 and SMP nodes); 31 TB (GPU-H200 nodes)
|-
!scope="column" | File System Type
| NFS
| Lustre
| Lustre
| XFS
|-
!scope="column" | Speed (read)
| ≈ 1 GB/s, shared by all nodes
| max. 12 GB/s
| ≈ 145 GB/s peak, aggregated over 56 nodes, ideal striping
| ≈ 3 GB/s (compute)/ ≈5 GB/S (GPUA-30)/ ≈ 26 GB/s (GPU-A100 + SMP)/ ≈ 42 GB/s (GPU-H200) per node
|-
!scope="column" | [https://en.wikipedia.org/wiki/Disk_quota#Quotas Quotas]
| 40 GB per user
| not yet, maybe in the future
| none
| none
|-
!scope="column" | Backup
| yes (nightly)
| '''no'''
| '''no'''
| '''no'''
|}

global : all nodes access the same file system
local : each node has its own file system
permanent : files are stored permanently
batch job walltime : files are removed at end of the batch job

{| class="wikitable" style="color:red; background-color:#ffffcc;" cellpadding="10"
|
Please note that due to the large capacity of '''work''' and '''project''' and due to frequent file changes on these file systems, no backup can be provided.
Backing up these file systems would require a redundant storage facility with multiple times the capacity of '''project'''. Furthermore, regular backups would significantly degrade the performance.
Data is stored redundantly, i.e. immune against disk failures but not immune against catastrophic incidents like cyber attacks or a fire in the server room.
Please consider to use on of the remote storage facilities like [https://wiki.bwhpc.de/e/SDS@hd SDS@hd], [https://uni-tuebingen.de/einrichtungen/zentrum-fuer-datenverarbeitung/projekte/laufende-projekte/bwsfs bwSFS], [https://www.scc.kit.edu/en/services/lsdf.php LSFD Online Storage] or the [https://www.rda.kit.edu/english/ bwDataArchive] to back up your valuable data.
|}

=== Home ===

Home directories are meant for permanent file storage of files that are keep being used like source codes, configuration files, executable programs etc.; the content of home directories will be backed up on a regular basis.
Because the backup space is limited we enforce a quota of 40GB on the home directories.

'''NOTE:'''
Compute jobs on nodes must not write temporary data to $HOME.
Instead they should use the local $TMPDIR directory for I/O-heavy use cases
and work spaces for less I/O intense multinode-jobs.


=== Project ===

The data is stored on HDDs. The primary focus of <code>/pfs/10/project</code> is pure capacity, not speed.

Every project gets a dedicated directory located at:

<syntaxhighlight>
/pfs/10/project/<project_id>/
</syntaxhighlight>

You can check the project(s) you are member of via:

<syntaxhighlight>
# id $USER | grep -o 'bw[^)]*'
bw16f003
</syntaxhighlight>

In this case, your project directory would be:
<code>
/pfs/10/project/bw16f003/
</code>

Check our [[BinAC2/Project_Data_Organization | data organization guide ]] for methods to organize data inside the project directory.

=== Workspaces ===

Data on the fast storage pool at <code>/pfs/10/work</code> is stored on SSDs.
The primary focus of this filesystem is speed, not capacity. 
Data on <code>/pfs/10/work</code> is organized into so-called ''workspaces''.
A workspace is a directory that you create and manage using the workspace tools.

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
Please always remember, that workspaces are intended solely for temporary work data, and there is no backup of data in the workspaces. 
Of you don't extend your workspaces they will be deleted at some point.
|}

==== Workspace tools documentation ====

You can find more information about the workspace tools in the general documentation:
:: → '''[[Workspaces | General workspace tools documentation]]'''

==== BinAC 2 workspace policies and limits ====

Some values and behaviors described in the general workspace documentation do not apply to BinAC 2. 
On BinAC 2, workspaces:

* have a '''lifetime of 30 days'''
* can be extended up to '''five times'''

Because the capacity of <code>/pfs/10/work</code> is limited, workspace expiration is enforced.
You will receive automated email reminders one week before a workspace expires.

If a workspace’s lifetime expires (i.e. is not extended), it is moved to the special directory
<code>/pfs/10/work/.removed</code>.
Expired workspaces can be restored for a keep time of 14 days.
After this period, expired workspaces are moved to a special location in the project filesystem.
After at least six months, a workspace is considered abandoned and is permanently deleted.

In addition to the standard workspace tools, we provide the script
<syntaxhighlight inline>ws_shelve</syntaxhighlight> for copying a workspace to a project directory.
<syntaxhighlight inline>ws_shelve</syntaxhighlight> takes two arguments:

* the name of the workspace to copy
* the acronym of your project

This command creates and submits a Slurm job that uses <code>rsync</code> to copy the workspace
from <code>/pfs/10/work</code> to <code>/pfs/10/project</code>.
A new directory is created with a timestamp suffix to prevent name collisions.

<syntaxhighlight>
# Copy the workspace /pfs/10/work/tu_xyz01-test to /pfs/10/project/bw10a001

$ ws_shelve test bw10a001
Workspace shelving submitted.
Job ID: 2058525
Logs will be written to:
/pfs/10/project/bw10a001/.ws_shelve/tu_xyz01/logs
</syntaxhighlight>

In this example, the workspace is copied to
<syntaxhighlight inline>/pfs/10/project/bw10a001/tu_xyz01/test-<timestamp></syntaxhighlight>.

=== SDS@hd ===

SDS@hd is mounted via NFS on login and compute nodes at <syntaxhighlight inline>/mnt/sds-hd</syntaxhighlight>.

To access your Speichervorhaben, the export to BinAC 2 must first be enabled by the SDS@hd-Team. Please contact [mailto:sds-hd-support@urz.uni-heidelberg.de SDS@hd support] and provide the acronym of your Speichervorhaben, along with a request to enable the export to BinAC 2.

Once this has been done, you can access your Speichervorhaben as described in the [https://wiki.bwhpc.de/e/SDS@hd/Access/NFS#Access_your_data SDS documentation].

<syntaxhighlight>
$ kinit $USER
Password for <user>@BWSERVICES.UNI-HEIDELBERG.DE:
</syntaxhighlight>

The Kerberos ticket store is shared across all nodes. Creating a single ticket is sufficient to access your Speichervorhaben on all nodes.

=== More Details on the Lustre File System ===
[https://www.lustre.org/ Lustre] is a distributed parallel file system.
* The entire logical volume as presented to the user is formed by multiple physical or local drives. Data is distributed over more than one physical or logical volume/hard drive, single files can be larger than the capacity of a single hard drive.
* The file system can be mounted from all nodes ("clients") in parallel at the same time for reading and writing. This also means that technically you can write to the same file from two different compute nodes! Usually, this will create an unpredictable mess! Never ever do this unless you know exactly what you are doing!
* On a single server or client, the bandwidth of multiple network interfaces can be aggregated to increase the throughput ("multi-rail").

Lustre works by chopping files into many small parts ("stripes", file objects) which are then stored on the object storage servers. The information which part of the file is stored where on which object storage server, when it was changed last etc. and the entire directory structure is stored on the metadata servers. Think of the entries on the metadata server as being pointers pointing to the actual file objects on the object storage servers.
A Lustre file system can consist of many metadata servers (MDS) and object storage servers (OSS).
Each MDS or OSS can again hold one or more so-called object storage targets (OST) or metadata targets (MDT) which can e.g. be simply multiple hard drives.
The capacity of a Lustre file system can hence be easily scaled by adding more servers.

==== Useful Lustre Comamnds ====
Commands specific to the Lustre file system are divided into user commands (<code>lfs ...</code>) and administrative commands (<code>lctl ...</code>). On BinAC2, users may only execute user commands, and also not all of them.
* <code>lfs help <command></code>: Print built-in help for command; Alternative: <code>man lfs <command></code>
* <code>lfs find</code>: Drop-in replacement for the <code>find</code> command, much faster on Lustre filesystems as it directly talks to the metadata sever
* <code>lfs --list-commands</code>: Print a list of available commands

==== Moving data between WORK and PROJECT ====
!! IMPORTANT !! Calling <code>mv</code> on files will not physically move them between the fast and the slow pool of the file system. Instead, the file metadata, i.e. the path to the file in the directory tree will be modified (i.e. data stored on the MDS). The stripes of the file on the OSS, however, will remain exactly were they were. The only result will be the confusing situation that you now have metadata entries under <code>/pfs/10/project</code> that still point to WORK OSTs. This may sound confusing at first. When using <code>mv</code> on the same file system, Lustre only renames the files and makes them available from a different path. The pointers to the file objects on the OSS stay identical. This will only change if you either create a copy of the file at a different path (with <code>cp</code> or <code>rsync</code>, e.g.) or if you explicitly instruct Lustre to move the actual file objects to another storage location, e.g. another pool of the same file system.

Proper ways of moving data between the pools
* Copy the data - which will create new files -, then delete the old files. Example:
<pre>
$> cp -r /pfs/10/work/tu_abcde01-my-precious-ws/* /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/.
$> rm -rf /pfs/10/work/tu_abcde01-my-precious-ws/*
$> ws_release --delete-data my-precious-ws
</pre>
* Alternative to copy: use <code>rsync</code> to copy data between the workspace and the project directories. Example:
<pre>
$> rsync -av /pfs/10/work/tu_abcde01-my-precious-ws/simulation/output /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/simulation25/
$> rm -rf /pfs/10/work/tu_abcde01-my-precious-ws/*
$> ws_release --delete-data my-precious-ws
</pre>
* If there are many subfolders with similar size, you can use <code>xargs</code> to copy them in parallel:
<pre>
$> find . -maxdepth 1 -mindepth 1 -type d -print | xargs -P4 -I{} rsync -aHAXW --inplace --update {} /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/simulation25/
</pre>
will launch four parallel <code>rsync</code> processes at a time, each will copy one of the subdirectories.
* First move the metadata with <code>mv</code>, then use <code>lfs migrate</code> or the wrapper <code>lfs_migrate</code> to actually migrate the file stripes. This is also a possible resolution if you already <code>mv</code>ed data from <code>work</code> to <code>project</code> or vice versa.
** <code>lfs migrate</code> is the raw lustre command. It can only operate on one file at a time, but offers access to all options.
** <code>lfs_migrate</code> is a versatile wrapper script that can work on single files or recursively on entire directories. If available, it will try to use <code>lfs migrate</code>, otherwise it will fall back to <code>rsync</code> (see <code>lfs_migrate --help</code> for all options.)
Example with <code>lfs migrate</code>:
<pre>
$> mv /pfs/10/work/tu_abcde01-my-precious-ws/* /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/.
$> cd /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/.
$> lfs find . -type f --pool work -0 | xargs -0 lfs migrate --pool project # find all files whose file objects are on the work pool and migrate the objects to the project pool
$> ws_release --delete-data my-precious-ws
</pre>
Example with <code>lfs_migrate</code>:
<pre>
$> mv /pfs/10/work/tu_abcde01-my-precious-ws/* /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/.
$> cd /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/.
$> lfs_migrate --yes -q -p project * # migrate all file objects in the current directory to the project pool, be quiet (-q) and do not ask for confirmation (--yes)
$> ws_release --delete-data my-precious-ws
</pre>
Both migration commands can also be combined with options to restripe the files during migration, i.e. you can also change the number of OSTs the file is striped over, the size of a single strip etc.
Attention! Both <code>lfs migrate</code> and <code>lfs_migrate</code> will not change the path of the file(s), you must also <code>mv</code> them! If used without <code>mv</code>, the files will still belong to the workspace although their file object stripes are now on the <code>project</code> pool and a subsequent <code>rm</code> in the workspace will wipe them.

All of the above procedures may take a considerable amount of time depending on the amount of data, so it might be advisable to execute them in a terminal multiplexer like <code>screen</code> or <code>tmux</code> or wrap them into small SLURM jobs with <code>sbatch --wrap="<command>"</code>.

Question: I totally lost overview, how do i find out where my files are located?

Answer:
* Use <code>lfs find</code> to find files on a specific pool. Example:
<pre>
$> lfs find . --pool project # recursively find all files in the current directory whose file objects are on the "project" pool
</pre>
* Use <code>lfs getstripe</code> to query the striping pattern and the pool (also works recursively if called with a directory). Example:
<pre>
$> lfs getstripe parameter.h
parameter.h
lmm_stripe_count: 1
lmm_stripe_size: 1048576
lmm_pattern: raid0
lmm_layout_gen: 1
lmm_stripe_offset: 44
lmm_pool: project
obdidx objid objid group
44 7991938 0x79f282 0xd80000400
</pre>
shows that the file is striped over OST 44 (obdidx) which belongs to pool project (lmm_pool).

Why pathes and storage pools should match:
There are four different possible scenarios with two subdirectories and two pools:
* File path in <code>/pfs/10/work</code>, file objects on pool <code>work</code>: good.
* File path in <code>/pfs/10/project</code>, file objects on pool <code>project</code>: good.
* File path in <code>/pfs/10/project</code>, file objects on pool <code>work</code>: bad. This will "leak" storage from the fast pool, making it unavailable for workspaces.
* File path in <code>/pfs/10/work</code>, file objects on pool <code>project</code>: bad. Access will be slow, and if (volatile) workspaces are purged, data residing on <code>project</code> will (voluntarily or involuntarily) be deleted.
The latter two situations may arise from <code>mv</code>ing data between workspaces and project folders.

==== More on data striping and how to influence it ====
!! The default striping patterns on BinAC2 are set for good reasons and should not light-heartedly be changed! Doing so wrongly will in the best case only hurt your performance. In the worst case, it will also hurt all other users and endanger the stability of the cluster. Please talk to the admins first if you think that you need a non-default pattern.
* Reading striping patterns with <code>lfs getstripe</code>
* Setting striping patterns with <code>lfs setstripe</code> for new files and directories
* Restriping files with <code>lfs migrate</code>
* Progressive File Layout

==== Architecture of BinAC2's Lustre File System ====
Metadata Servers:
* 2 metadata servers
* 1 MDT per server
* MDT Capacity: 31TB, hardware RAID6 on NVMe drives (flash memory/SSD)
* Networking: 2x 100 GbE, 2x HDR-100 InfiniBand

Object storage servers:
* 8 object storage servers
* 2 fast OSTs per server
** 70 TB per OST, software RAID (raid-z2, 10+2 reduncancy)
** NVMe drives, directly attached to the PCIe bus
* 8 slow OSTs per server
** 143 TB per OST, hardware RAID (RAID6, 8+2 redundancy)
** externally attached via SAS
* Networking: 2x 100 GbE, 2x HDR-100 InfiniBand

* All fast OSTs are assigned to the pool <code>work</code>
* All slow OSTs are assigned to the pool <code>project</code>
* All files that are created under <code>/pfs/10/work</code> are by default stored on the fast pool
* All files that are created under <code>/pfs/10/project</code> are by default stored on the slow pool
* Metadata is distributed over both MDTs. All subdirectories of a directory (workspace or project folder) are typically on the same MDT. Directory striping/placement on MDTs can not be influenced by users.
* Default OST striping: Stripes have size 1 MiB. Files are striped over one OST if possible, i.e. all stripes of a file are on the same OST. New files are created on the most empty OST.
Internally, the slow and the fast pool belong to the same Lustre file system and namespace.

More reading:
* [https://doc.lustre.org/lustre_manual.xhtml The Lustre 2.X Manual] ([http://doc.lustre.org/lustre_manual.pdf PDF])
* [https://wiki.lustre.org/Main_Page The Lustre Wiki]

BinAC2/Hardware and Architecture

2026-01-21T10:45:31Z

F Bartusch:

= Hardware and Architecture =

The bwForCluster BinAC 2 supports researchers from the broader fields of Bioinformatics, Medical Informatics, Astrophysics, Geosciences and Pharmacy.

[[File:Binac2 schema.png|600px|thumb|center|Overview on the BinAC 2 hardware architecture.]]

== Operating System and Software ==

* Operating System: Rocky Linux 9.6
* Queuing System: [https://slurm.schedmd.com/documentation.html Slurm] (see [[BinAC2/Slurm]] for help)
* (Scientific) Libraries and Software: [[Environment Modules]]

== Compute Nodes ==

BinAC 2 offers compute nodes, high-mem nodes, and three types of GPU nodes.
* 180 compute nodes
* 16 SMP node
* 32 GPU nodes (2xA30)
* 8 GPU nodes (4xA100)
* 4 GPU nodes (4xH200)
* plus several special purpose nodes for login, interactive jobs, etc.

Compute node specification:
{| class="wikitable"
|-
! style="width:10%"|
! style="width:10%"| Standard
! style="width:10%"| High-Mem
! style="width:10%"| GPU (A30)
! style="width:10%"| GPU (A100)
! style="width:10%"| GPU (H200)
|-
!scope="column"| Quantity
| 168 / 12
| 14 / 2
| 32
| 8
| 4
|-
!scope="column" | Processors
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7543.html AMD EPYC Milan 7543] / 2 x [https://www.amd.com/en/products/processors/server/epyc/7003-series/amd-epyc-75f3.html AMD EPYC Milan 75F3]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7443.html AMD EPYC Milan 7443] / 2 x [https://www.amd.com/en/products/processors/server/epyc/7003-series/amd-epyc-75f3.html AMD EPYC Milan 75F3]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7543.html AMD EPYC Milan 7543]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7543.html AMD EPYC Milan 7543]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/9005-series/amd-epyc-9555.html AMD EPYC Milan 9555]
|-
!scope="column" | Processor Base Frequency (GHz)
| 2.80 / 2.95
| 2.85 / 2.95
| 2.80
| 2.80
| 3.20
|-
!scope="column" | Number of Physical Cores / Hypertreads
| 64 / 128
| 48 / 96 // 64 / 128
| 64 / 128
| 64 / 128
| 128 / 256
|-
!scope="column" | Working Memory (GB)
| 512
| 2048
| 512
| 512
| 1536
|-
!scope="column" | Local Disk (GiB)
| 450 (NVMe-SSD)
| 14000 (NVMe-SSD)
| 450 (NVMe-SSD)
| 14000 (NVMe-SSD)
| 28000 (NVMe-SSD)
|-
!scope="column" | Interconnect
| HDR 100 IB (84 nodes) / 100GbE (96 nodes)
| 100GbE
| 100GbE
| 100GbE
| HDR 200 IB + 100GbE
|-
!scope="column" | Coprocessors
| -
| -
| 2 x [https://www.nvidia.com/de-de/data-center/products/a30-gpu/ NVIDIA A30 (24 GB ECC HBM2, NVLink)]
| 4 x [https://www.nvidia.com/de-de/data-center/a100/ NVIDIA A100 (80 GB ECC HBM2e)]
| 4 x [https://www.nvidia.com/de-de/data-center/h200/ NVIDIA H200 NVL (141 GB ECC HBM3e, NVLink)]
|}

= Network =

The compute nodes and the parallel file system are connected via 100GbE ethernet
In contrast to BinAC 1 not all compute nodes are connected via Infiniband, but there are 84 standard compute nodes connected via HDR Infiniband (100 GbE). In order to get your jobs onto the Infiniband nodes, submit your job with <code>--constraint=ib</code>.

'''Question:'''
OpenMPI throws the following warning:
<pre>
--------------------------------------------------------------------------
No OpenFabrics connection schemes reported that they were able to be
used on a specific port. As such, the openib BTL (OpenFabrics
support) will be disabled for this port.
Local host: node1-083
Local device: mlx5_0
Local port: 1
CPCs attempted: rdmacm, udcm
--------------------------------------------------------------------------
[node1-083:2137377] 3 more processes have sent help message help-mpi-btl-openib-cpc-base.txt / no cpcs for port
[node1-083:2137377] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
</pre>
What should i do?

'''Answer:'''
BinAC2 has two (almost) separate networks, a 100GbE network and and InfiniBand network, both connecting a subset of the nodes. Both networks require different cables and switches.
Concerning the network cards for the nodes, however, there exist VPI network cards which can be configured to work in either mode (https://docs.nvidia.com/networking/display/connectx6vpi/specifications#src-2487215234_Specifications-MCX653105A-ECATSpecifications).
OpenMPI can use a number of layers for transferring data and messages between processes. When it ramps up, it will test all means of communication that were configured during compilation and then tries to figure out the fastest path between all processes.
If OpenMPI encounters such a VPI card, it will first try to establish a Remote Direct Memory Access communication (RDMA) channel using the OpenFabrics (OFI) layer.
On nodes with 100Gb ethernet, this fails as there is no RDMA protocol configured. OpenMPI will fall back to TCP transport but not without complaints.

'''Workaround:'''
For single-node jobs or on regular compute nodes, A30 and A100 GPU nodes: Add the lines
<pre>
export OMPI_MCA_btl="^ofi,openib"
export OMPI_MCA_mtl="^ofi"
</pre>
to your job script to disable the OFI transport layer. If you need high-bandwidth, low-latency transport between all processes on all nodes, switch to the Infiniband partition (<code>#SBATCH --constraint=ib</code>). ''Do not turn off the OFI layer on Infiniband nodes as this will be the best choice between nodes!''

= File Systems =

The bwForCluster BinAC 2 consists of two separate storage systems, one for the user's home directory $HOME and one serving as a project/work space.
The home directory is limited in space and parallel access but offers snapshots of your files and backup.

The project/work is a parallel file system (PFS) which offers fast and parallel file access and a bigger capacity than the home directory. It is mounted at <code>/pfs/10</code> on the login and compute nodes. This storage is based on Lustre and can be accessed in parallel from many nodes. The PFS contains the project and the work directory. Each compute project has its own directory at <code>/pfs/10/project</code> that is accessible for all members of the compute project.
Each user can create workspaces under <code>/pfs/10/work</code> using the workspace tools. These directories are only accessible for the user who created the workspace.

Additionally, each compute node provides high-speed temporary storage (SSD) on the node-local solid state disk via the $TMPDIR environment variable.

{| class="wikitable"
|-
! style="width:10%"|
! style="width:10%"| <tt>$HOME</tt>
! style="width:10%"| project
! style="width:10%"| work
! style="width:10%"| <tt>$TMPDIR</tt>
|-
!scope="column" | Visibility
| global
| global
| global
| node local
|-
!scope="column" | Lifetime
| permanent
| permanent
| work space lifetime (max. 30 days, max. 5 extensions)
| batch job walltime
|-
!scope="column" | Capacity
| -
| 8.1 PB
| 1000 TB
| 480 GB (compute nodes); 7.7 TB (GPU-A30 nodes); 16 TB (GPU-A100 and SMP nodes); 31 TB (GPU-H200 nodes)
|-
!scope="column" | File System Type
| NFS
| Lustre
| Lustre
| XFS
|-
!scope="column" | Speed (read)
| ≈ 1 GB/s, shared by all nodes
| max. 12 GB/s
| ≈ 145 GB/s peak, aggregated over 56 nodes, ideal striping
| ≈ 3 GB/s (compute)/ ≈5 GB/S (GPUA-30)/ ≈ 26 GB/s (GPU-A100 + SMP)/ ≈ 42 GB/s (GPU-H200) per node
|-
!scope="column" | [https://en.wikipedia.org/wiki/Disk_quota#Quotas Quotas]
| 40 GB per user
| not yet, maybe in the future
| none
| none
|-
!scope="column" | Backup
| yes (nightly)
| '''no'''
| '''no'''
| '''no'''
|}

global : all nodes access the same file system
local : each node has its own file system
permanent : files are stored permanently
batch job walltime : files are removed at end of the batch job

{| class="wikitable" style="color:red; background-color:#ffffcc;" cellpadding="10"
|
Please note that due to the large capacity of '''work''' and '''project''' and due to frequent file changes on these file systems, no backup can be provided.
Backing up these file systems would require a redundant storage facility with multiple times the capacity of '''project'''. Furthermore, regular backups would significantly degrade the performance.
Data is stored redundantly, i.e. immune against disk failures but not immune against catastrophic incidents like cyber attacks or a fire in the server room.
Please consider to use on of the remote storage facilities like [https://wiki.bwhpc.de/e/SDS@hd SDS@hd], [https://uni-tuebingen.de/einrichtungen/zentrum-fuer-datenverarbeitung/projekte/laufende-projekte/bwsfs bwSFS], [https://www.scc.kit.edu/en/services/lsdf.php LSFD Online Storage] or the [https://www.rda.kit.edu/english/ bwDataArchive] to back up your valuable data.
|}

=== Home ===

Home directories are meant for permanent file storage of files that are keep being used like source codes, configuration files, executable programs etc.; the content of home directories will be backed up on a regular basis.
Because the backup space is limited we enforce a quota of 40GB on the home directories.

'''NOTE:'''
Compute jobs on nodes must not write temporary data to $HOME.
Instead they should use the local $TMPDIR directory for I/O-heavy use cases
and work spaces for less I/O intense multinode-jobs.


=== Project ===

The data is stored on HDDs. The primary focus of <code>/pfs/10/project</code> is pure capacity, not speed.

Every project gets a dedicated directory located at:

<syntaxhighlight>
/pfs/10/project/<project_id>/
</syntaxhighlight>

You can check the project(s) you are member of via:

<syntaxhighlight>
# id $USER | grep -o 'bw[^)]*'
bw16f003
</syntaxhighlight>

In this case, your project directory would be:
<code>
/pfs/10/project/bw16f003/
</code>

Check our [[BinAC2/Project_Data_Organization | data organization guide ]] for methods to organize data inside the project directory.

=== Workspaces ===

Data on the fast storage pool at <code>/pfs/10/work</code> is stored on SSDs.
The primary focus of this filesystem is speed, not capacity. 
Data on <code>/pfs/10/work</code> is organized into so-called ''workspaces''.
A workspace is a directory that you create and manage using the workspace tools.

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
Please always remember, that workspaces are intended solely for temporary work data, and there is no backup of data in the workspaces. 
Of you don't extend your workspaces they will be deleted at some point.
|}

==== Workspace tools documentation ====

You can find more information about the workspace tools in the general documentation:
:: → '''[[Workspace | General workspace tools documentation]]'''

==== BinAC 2 workspace policies and limits ====

Some values and behaviors described in the general workspace documentation do not apply to BinAC 2. 
On BinAC 2, workspaces:

* have a '''lifetime of 30 days'''
* can be extended up to '''five times'''

Because the capacity of <code>/pfs/10/work</code> is limited, workspace expiration is enforced.
You will receive automated email reminders one week before a workspace expires.

If a workspace’s lifetime expires (i.e. is not extended), it is moved to the special directory
<code>/pfs/10/work/.removed</code>.
Expired workspaces can be restored for a keep time of 14 days.
After this period, expired workspaces are moved to a special location in the project filesystem.
After at least six months, a workspace is considered abandoned and is permanently deleted.

In addition to the standard workspace tools, we provide the script
<syntaxhighlight inline>ws_shelve</syntaxhighlight> for copying a workspace to a project directory.
<syntaxhighlight inline>ws_shelve</syntaxhighlight> takes two arguments:

* the name of the workspace to copy
* the acronym of your project

This command creates and submits a Slurm job that uses <code>rsync</code> to copy the workspace
from <code>/pfs/10/work</code> to <code>/pfs/10/project</code>.
A new directory is created with a timestamp suffix to prevent name collisions.

<syntaxhighlight>
# Copy the workspace /pfs/10/work/tu_xyz01-test to /pfs/10/project/bw10a001

$ ws_shelve test bw10a001
Workspace shelving submitted.
Job ID: 2058525
Logs will be written to:
/pfs/10/project/bw10a001/.ws_shelve/tu_xyz01/logs
</syntaxhighlight>

In this example, the workspace is copied to
<syntaxhighlight inline>/pfs/10/project/bw10a001/tu_xyz01/test-<timestamp></syntaxhighlight>.

=== SDS@hd ===

SDS@hd is mounted via NFS on login and compute nodes at <syntaxhighlight inline>/mnt/sds-hd</syntaxhighlight>.

To access your Speichervorhaben, the export to BinAC 2 must first be enabled by the SDS@hd-Team. Please contact [mailto:sds-hd-support@urz.uni-heidelberg.de SDS@hd support] and provide the acronym of your Speichervorhaben, along with a request to enable the export to BinAC 2.

Once this has been done, you can access your Speichervorhaben as described in the [https://wiki.bwhpc.de/e/SDS@hd/Access/NFS#Access_your_data SDS documentation].

<syntaxhighlight>
$ kinit $USER
Password for <user>@BWSERVICES.UNI-HEIDELBERG.DE:
</syntaxhighlight>

The Kerberos ticket store is shared across all nodes. Creating a single ticket is sufficient to access your Speichervorhaben on all nodes.

=== More Details on the Lustre File System ===
[https://www.lustre.org/ Lustre] is a distributed parallel file system.
* The entire logical volume as presented to the user is formed by multiple physical or local drives. Data is distributed over more than one physical or logical volume/hard drive, single files can be larger than the capacity of a single hard drive.
* The file system can be mounted from all nodes ("clients") in parallel at the same time for reading and writing. This also means that technically you can write to the same file from two different compute nodes! Usually, this will create an unpredictable mess! Never ever do this unless you know exactly what you are doing!
* On a single server or client, the bandwidth of multiple network interfaces can be aggregated to increase the throughput ("multi-rail").

Lustre works by chopping files into many small parts ("stripes", file objects) which are then stored on the object storage servers. The information which part of the file is stored where on which object storage server, when it was changed last etc. and the entire directory structure is stored on the metadata servers. Think of the entries on the metadata server as being pointers pointing to the actual file objects on the object storage servers.
A Lustre file system can consist of many metadata servers (MDS) and object storage servers (OSS).
Each MDS or OSS can again hold one or more so-called object storage targets (OST) or metadata targets (MDT) which can e.g. be simply multiple hard drives.
The capacity of a Lustre file system can hence be easily scaled by adding more servers.

==== Useful Lustre Comamnds ====
Commands specific to the Lustre file system are divided into user commands (<code>lfs ...</code>) and administrative commands (<code>lctl ...</code>). On BinAC2, users may only execute user commands, and also not all of them.
* <code>lfs help <command></code>: Print built-in help for command; Alternative: <code>man lfs <command></code>
* <code>lfs find</code>: Drop-in replacement for the <code>find</code> command, much faster on Lustre filesystems as it directly talks to the metadata sever
* <code>lfs --list-commands</code>: Print a list of available commands

==== Moving data between WORK and PROJECT ====
!! IMPORTANT !! Calling <code>mv</code> on files will not physically move them between the fast and the slow pool of the file system. Instead, the file metadata, i.e. the path to the file in the directory tree will be modified (i.e. data stored on the MDS). The stripes of the file on the OSS, however, will remain exactly were they were. The only result will be the confusing situation that you now have metadata entries under <code>/pfs/10/project</code> that still point to WORK OSTs. This may sound confusing at first. When using <code>mv</code> on the same file system, Lustre only renames the files and makes them available from a different path. The pointers to the file objects on the OSS stay identical. This will only change if you either create a copy of the file at a different path (with <code>cp</code> or <code>rsync</code>, e.g.) or if you explicitly instruct Lustre to move the actual file objects to another storage location, e.g. another pool of the same file system.

Proper ways of moving data between the pools
* Copy the data - which will create new files -, then delete the old files. Example:
<pre>
$> cp -r /pfs/10/work/tu_abcde01-my-precious-ws/* /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/.
$> rm -rf /pfs/10/work/tu_abcde01-my-precious-ws/*
$> ws_release --delete-data my-precious-ws
</pre>
* Alternative to copy: use <code>rsync</code> to copy data between the workspace and the project directories. Example:
<pre>
$> rsync -av /pfs/10/work/tu_abcde01-my-precious-ws/simulation/output /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/simulation25/
$> rm -rf /pfs/10/work/tu_abcde01-my-precious-ws/*
$> ws_release --delete-data my-precious-ws
</pre>
* If there are many subfolders with similar size, you can use <code>xargs</code> to copy them in parallel:
<pre>
$> find . -maxdepth 1 -mindepth 1 -type d -print | xargs -P4 -I{} rsync -aHAXW --inplace --update {} /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/simulation25/
</pre>
will launch four parallel <code>rsync</code> processes at a time, each will copy one of the subdirectories.
* First move the metadata with <code>mv</code>, then use <code>lfs migrate</code> or the wrapper <code>lfs_migrate</code> to actually migrate the file stripes. This is also a possible resolution if you already <code>mv</code>ed data from <code>work</code> to <code>project</code> or vice versa.
** <code>lfs migrate</code> is the raw lustre command. It can only operate on one file at a time, but offers access to all options.
** <code>lfs_migrate</code> is a versatile wrapper script that can work on single files or recursively on entire directories. If available, it will try to use <code>lfs migrate</code>, otherwise it will fall back to <code>rsync</code> (see <code>lfs_migrate --help</code> for all options.)
Example with <code>lfs migrate</code>:
<pre>
$> mv /pfs/10/work/tu_abcde01-my-precious-ws/* /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/.
$> cd /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/.
$> lfs find . -type f --pool work -0 | xargs -0 lfs migrate --pool project # find all files whose file objects are on the work pool and migrate the objects to the project pool
$> ws_release --delete-data my-precious-ws
</pre>
Example with <code>lfs_migrate</code>:
<pre>
$> mv /pfs/10/work/tu_abcde01-my-precious-ws/* /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/.
$> cd /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/.
$> lfs_migrate --yes -q -p project * # migrate all file objects in the current directory to the project pool, be quiet (-q) and do not ask for confirmation (--yes)
$> ws_release --delete-data my-precious-ws
</pre>
Both migration commands can also be combined with options to restripe the files during migration, i.e. you can also change the number of OSTs the file is striped over, the size of a single strip etc.
Attention! Both <code>lfs migrate</code> and <code>lfs_migrate</code> will not change the path of the file(s), you must also <code>mv</code> them! If used without <code>mv</code>, the files will still belong to the workspace although their file object stripes are now on the <code>project</code> pool and a subsequent <code>rm</code> in the workspace will wipe them.

All of the above procedures may take a considerable amount of time depending on the amount of data, so it might be advisable to execute them in a terminal multiplexer like <code>screen</code> or <code>tmux</code> or wrap them into small SLURM jobs with <code>sbatch --wrap="<command>"</code>.

Question: I totally lost overview, how do i find out where my files are located?

Answer:
* Use <code>lfs find</code> to find files on a specific pool. Example:
<pre>
$> lfs find . --pool project # recursively find all files in the current directory whose file objects are on the "project" pool
</pre>
* Use <code>lfs getstripe</code> to query the striping pattern and the pool (also works recursively if called with a directory). Example:
<pre>
$> lfs getstripe parameter.h
parameter.h
lmm_stripe_count: 1
lmm_stripe_size: 1048576
lmm_pattern: raid0
lmm_layout_gen: 1
lmm_stripe_offset: 44
lmm_pool: project
obdidx objid objid group
44 7991938 0x79f282 0xd80000400
</pre>
shows that the file is striped over OST 44 (obdidx) which belongs to pool project (lmm_pool).

Why pathes and storage pools should match:
There are four different possible scenarios with two subdirectories and two pools:
* File path in <code>/pfs/10/work</code>, file objects on pool <code>work</code>: good.
* File path in <code>/pfs/10/project</code>, file objects on pool <code>project</code>: good.
* File path in <code>/pfs/10/project</code>, file objects on pool <code>work</code>: bad. This will "leak" storage from the fast pool, making it unavailable for workspaces.
* File path in <code>/pfs/10/work</code>, file objects on pool <code>project</code>: bad. Access will be slow, and if (volatile) workspaces are purged, data residing on <code>project</code> will (voluntarily or involuntarily) be deleted.
The latter two situations may arise from <code>mv</code>ing data between workspaces and project folders.

==== More on data striping and how to influence it ====
!! The default striping patterns on BinAC2 are set for good reasons and should not light-heartedly be changed! Doing so wrongly will in the best case only hurt your performance. In the worst case, it will also hurt all other users and endanger the stability of the cluster. Please talk to the admins first if you think that you need a non-default pattern.
* Reading striping patterns with <code>lfs getstripe</code>
* Setting striping patterns with <code>lfs setstripe</code> for new files and directories
* Restriping files with <code>lfs migrate</code>
* Progressive File Layout

==== Architecture of BinAC2's Lustre File System ====
Metadata Servers:
* 2 metadata servers
* 1 MDT per server
* MDT Capacity: 31TB, hardware RAID6 on NVMe drives (flash memory/SSD)
* Networking: 2x 100 GbE, 2x HDR-100 InfiniBand

Object storage servers:
* 8 object storage servers
* 2 fast OSTs per server
** 70 TB per OST, software RAID (raid-z2, 10+2 reduncancy)
** NVMe drives, directly attached to the PCIe bus
* 8 slow OSTs per server
** 143 TB per OST, hardware RAID (RAID6, 8+2 redundancy)
** externally attached via SAS
* Networking: 2x 100 GbE, 2x HDR-100 InfiniBand

* All fast OSTs are assigned to the pool <code>work</code>
* All slow OSTs are assigned to the pool <code>project</code>
* All files that are created under <code>/pfs/10/work</code> are by default stored on the fast pool
* All files that are created under <code>/pfs/10/project</code> are by default stored on the slow pool
* Metadata is distributed over both MDTs. All subdirectories of a directory (workspace or project folder) are typically on the same MDT. Directory striping/placement on MDTs can not be influenced by users.
* Default OST striping: Stripes have size 1 MiB. Files are striped over one OST if possible, i.e. all stripes of a file are on the same OST. New files are created on the most empty OST.
Internally, the slow and the fast pool belong to the same Lustre file system and namespace.

More reading:
* [https://doc.lustre.org/lustre_manual.xhtml The Lustre 2.X Manual] ([http://doc.lustre.org/lustre_manual.pdf PDF])
* [https://wiki.lustre.org/Main_Page The Lustre Wiki]

BinAC2/Hardware and Architecture

2026-01-19T10:55:46Z

F Bartusch:

= Hardware and Architecture =

The bwForCluster BinAC 2 supports researchers from the broader fields of Bioinformatics, Medical Informatics, Astrophysics, Geosciences and Pharmacy.

[[File:Binac2 schema.png|600px|thumb|center|Overview on the BinAC 2 hardware architecture.]]

== Operating System and Software ==

* Operating System: Rocky Linux 9.6
* Queuing System: [https://slurm.schedmd.com/documentation.html Slurm] (see [[BinAC2/Slurm]] for help)
* (Scientific) Libraries and Software: [[Environment Modules]]

== Compute Nodes ==

BinAC 2 offers compute nodes, high-mem nodes, and three types of GPU nodes.
* 180 compute nodes
* 16 SMP node
* 32 GPU nodes (2xA30)
* 8 GPU nodes (4xA100)
* 4 GPU nodes (4xH200)
* plus several special purpose nodes for login, interactive jobs, etc.

Compute node specification:
{| class="wikitable"
|-
! style="width:10%"|
! style="width:10%"| Standard
! style="width:10%"| High-Mem
! style="width:10%"| GPU (A30)
! style="width:10%"| GPU (A100)
! style="width:10%"| GPU (H200)
|-
!scope="column"| Quantity
| 168 / 12
| 14 / 2
| 32
| 8
| 4
|-
!scope="column" | Processors
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7543.html AMD EPYC Milan 7543] / 2 x [https://www.amd.com/en/products/processors/server/epyc/7003-series/amd-epyc-75f3.html AMD EPYC Milan 75F3]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7443.html AMD EPYC Milan 7443] / 2 x [https://www.amd.com/en/products/processors/server/epyc/7003-series/amd-epyc-75f3.html AMD EPYC Milan 75F3]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7543.html AMD EPYC Milan 7543]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7543.html AMD EPYC Milan 7543]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/9005-series/amd-epyc-9555.html AMD EPYC Milan 9555]
|-
!scope="column" | Processor Base Frequency (GHz)
| 2.80 / 2.95
| 2.85 / 2.95
| 2.80
| 2.80
| 3.20
|-
!scope="column" | Number of Physical Cores / Hypertreads
| 64 / 128
| 48 / 96 // 64 / 128
| 64 / 128
| 64 / 128
| 128 / 256
|-
!scope="column" | Working Memory (GB)
| 512
| 2048
| 512
| 512
| 1536
|-
!scope="column" | Local Disk (GiB)
| 450 (NVMe-SSD)
| 14000 (NVMe-SSD)
| 450 (NVMe-SSD)
| 14000 (NVMe-SSD)
| 28000 (NVMe-SSD)
|-
!scope="column" | Interconnect
| HDR 100 IB (84 nodes) / 100GbE (96 nodes)
| 100GbE
| 100GbE
| 100GbE
| HDR 200 IB + 100GbE
|-
!scope="column" | Coprocessors
| -
| -
| 2 x [https://www.nvidia.com/de-de/data-center/products/a30-gpu/ NVIDIA A30 (24 GB ECC HBM2, NVLink)]
| 4 x [https://www.nvidia.com/de-de/data-center/a100/ NVIDIA A100 (80 GB ECC HBM2e)]
| 4 x [https://www.nvidia.com/de-de/data-center/h200/ NVIDIA H200 NVL (141 GB ECC HBM3e, NVLink)]
|}

= Network =

The compute nodes and the parallel file system are connected via 100GbE ethernet
In contrast to BinAC 1 not all compute nodes are connected via Infiniband, but there are 84 standard compute nodes connected via HDR Infiniband (100 GbE). In order to get your jobs onto the Infiniband nodes, submit your job with <code>--constraint=ib</code>.

'''Question:'''
OpenMPI throws the following warning:
<pre>
--------------------------------------------------------------------------
No OpenFabrics connection schemes reported that they were able to be
used on a specific port. As such, the openib BTL (OpenFabrics
support) will be disabled for this port.
Local host: node1-083
Local device: mlx5_0
Local port: 1
CPCs attempted: rdmacm, udcm
--------------------------------------------------------------------------
[node1-083:2137377] 3 more processes have sent help message help-mpi-btl-openib-cpc-base.txt / no cpcs for port
[node1-083:2137377] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
</pre>
What should i do?

'''Answer:'''
BinAC2 has two (almost) separate networks, a 100GbE network and and InfiniBand network, both connecting a subset of the nodes. Both networks require different cables and switches.
Concerning the network cards for the nodes, however, there exist VPI network cards which can be configured to work in either mode (https://docs.nvidia.com/networking/display/connectx6vpi/specifications#src-2487215234_Specifications-MCX653105A-ECATSpecifications).
OpenMPI can use a number of layers for transferring data and messages between processes. When it ramps up, it will test all means of communication that were configured during compilation and then tries to figure out the fastest path between all processes.
If OpenMPI encounters such a VPI card, it will first try to establish a Remote Direct Memory Access communication (RDMA) channel using the OpenFabrics (OFI) layer.
On nodes with 100Gb ethernet, this fails as there is no RDMA protocol configured. OpenMPI will fall back to TCP transport but not without complaints.

'''Workaround:'''
For single-node jobs or on regular compute nodes, A30 and A100 GPU nodes: Add the lines
<pre>
export OMPI_MCA_btl="^ofi,openib"
export OMPI_MCA_mtl="^ofi"
</pre>
to your job script to disable the OFI transport layer. If you need high-bandwidth, low-latency transport between all processes on all nodes, switch to the Infiniband partition (<code>#SBATCH --constraint=ib</code>). ''Do not turn off the OFI layer on Infiniband nodes as this will be the best choice between nodes!''

= File Systems =

The bwForCluster BinAC 2 consists of two separate storage systems, one for the user's home directory $HOME and one serving as a project/work space.
The home directory is limited in space and parallel access but offers snapshots of your files and backup.

The project/work is a parallel file system (PFS) which offers fast and parallel file access and a bigger capacity than the home directory. It is mounted at <code>/pfs/10</code> on the login and compute nodes. This storage is based on Lustre and can be accessed in parallel from many nodes. The PFS contains the project and the work directory. Each compute project has its own directory at <code>/pfs/10/project</code> that is accessible for all members of the compute project.
Each user can create workspaces under <code>/pfs/10/work</code> using the workspace tools. These directories are only accessible for the user who created the workspace.

Additionally, each compute node provides high-speed temporary storage (SSD) on the node-local solid state disk via the $TMPDIR environment variable.

{| class="wikitable"
|-
! style="width:10%"|
! style="width:10%"| <tt>$HOME</tt>
! style="width:10%"| project
! style="width:10%"| work
! style="width:10%"| <tt>$TMPDIR</tt>
|-
!scope="column" | Visibility
| global
| global
| global
| node local
|-
!scope="column" | Lifetime
| permanent
| permanent
| work space lifetime (max. 30 days, max. 5 extensions)
| batch job walltime
|-
!scope="column" | Capacity
| -
| 8.1 PB
| 1000 TB
| 480 GB (compute nodes); 7.7 TB (GPU-A30 nodes); 16 TB (GPU-A100 and SMP nodes); 31 TB (GPU-H200 nodes)
|-
!scope="column" | File System Type
| NFS
| Lustre
| Lustre
| XFS
|-
!scope="column" | Speed (read)
| ≈ 1 GB/s, shared by all nodes
| max. 12 GB/s
| ≈ 145 GB/s peak, aggregated over 56 nodes, ideal striping
| ≈ 3 GB/s (compute)/ ≈5 GB/S (GPUA-30)/ ≈ 26 GB/s (GPU-A100 + SMP)/ ≈ 42 GB/s (GPU-H200) per node
|-
!scope="column" | [https://en.wikipedia.org/wiki/Disk_quota#Quotas Quotas]
| 40 GB per user
| not yet, maybe in the future
| none
| none
|-
!scope="column" | Backup
| yes (nightly)
| '''no'''
| '''no'''
| '''no'''
|}

global : all nodes access the same file system
local : each node has its own file system
permanent : files are stored permanently
batch job walltime : files are removed at end of the batch job

{| class="wikitable" style="color:red; background-color:#ffffcc;" cellpadding="10"
|
Please note that due to the large capacity of '''work''' and '''project''' and due to frequent file changes on these file systems, no backup can be provided.
Backing up these file systems would require a redundant storage facility with multiple times the capacity of '''project'''. Furthermore, regular backups would significantly degrade the performance.
Data is stored redundantly, i.e. immune against disk failures but not immune against catastrophic incidents like cyber attacks or a fire in the server room.
Please consider to use on of the remote storage facilities like [https://wiki.bwhpc.de/e/SDS@hd SDS@hd], [https://uni-tuebingen.de/einrichtungen/zentrum-fuer-datenverarbeitung/projekte/laufende-projekte/bwsfs bwSFS], [https://www.scc.kit.edu/en/services/lsdf.php LSFD Online Storage] or the [https://www.rda.kit.edu/english/ bwDataArchive] to back up your valuable data.
|}

=== Home ===

Home directories are meant for permanent file storage of files that are keep being used like source codes, configuration files, executable programs etc.; the content of home directories will be backed up on a regular basis.
Because the backup space is limited we enforce a quota of 40GB on the home directories.

'''NOTE:'''
Compute jobs on nodes must not write temporary data to $HOME.
Instead they should use the local $TMPDIR directory for I/O-heavy use cases
and work spaces for less I/O intense multinode-jobs.


=== Project ===

The data is stored on HDDs. The primary focus of <code>/pfs/10/project</code> is pure capacity, not speed.

Every project gets a dedicated directory located at:

<syntaxhighlight>
/pfs/10/project/<project_id>/
</syntaxhighlight>

You can check the project(s) you are member of via:

<syntaxhighlight>
# id $USER | grep -o 'bw[^)]*'
bw16f003
</syntaxhighlight>

In this case, your project directory would be:
<code>
/pfs/10/project/bw16f003/
</code>

Check our [[BinAC2/Project_Data_Organization | data organization guide ]] for methods to organize data inside the project directory.

=== Workspaces ===

Data on the fast storage pool at <code>/pfs/10/work</code> is stored on SSDs.
The primary focus of this filesystem is speed, not capacity. 
Data on <code>/pfs/10/work</code> is organized into so-called ''workspaces''.
A workspace is a directory that you create and manage using the workspace tools.

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
Please always remember, that workspaces are intended solely for temporary work data, and there is no backup of data in the workspaces. 
Of you don't extend your workspaces they will be deleted at some point.
|}

==== Workspace tools documentation ====

You can find more information about the workspace tools in the general documentation:
:: → '''[[Workspace | General workspace tools documentation]]'''

==== BinAC 2 workspace policies and limits ====

Some values and behaviors described in the general workspace documentation do not apply to BinAC 2. 
On BinAC 2, workspaces:

* have a '''lifetime of 30 days'''
* can be extended up to '''five times'''

Because the capacity of <code>/pfs/10/work</code> is limited, workspace expiration is enforced.
You will receive automated email reminders one week before a workspace expires.

If a workspace’s lifetime expires (i.e. is not extended), it is moved to the special directory
<code>/pfs/10/work/.removed</code>.
Expired workspaces can be restored for a keep time of 14 days.
After this period, expired workspaces are moved to a special location in the project filesystem.
After at least six months, a workspace is considered abandoned and is permanently deleted.

In addition to the standard workspace tools, we provide the script
<syntaxhighlight inline>ws_shelve</syntaxhighlight> for copying a workspace to a project directory.
<syntaxhighlight inline>ws_shelve</syntaxhighlight> takes two arguments:

* the name of the workspace to copy
* the acronym of your project

This command creates and submits a Slurm job that uses <code>rsync</code> to copy the workspace
from <code>/pfs/10/work</code> to <code>/pfs/10/project</code>.
A new directory is created with a timestamp suffix to prevent name collisions.

<syntaxhighlight>
# Copy the workspace /pfs/10/work/tu_xyz01-test to /pfs/10/project/bw10a001

$ ws_shelve test bw10a001
Workspace shelving submitted.
Job ID: 2058525
Logs will be written to:
/pfs/10/project/bw10a001/.ws_shelve/tu_xyz01/logs
</syntaxhighlight>

In this example, the workspace is copied to
<syntaxhighlight inline>/pfs/10/project/bw10a001/tu_xyz01/test-<timestamp></syntaxhighlight>.

=== SDS@hd ===

SDS@hd is mounted via NFS on login and compute nodes at <syntaxhighlight inline>/mnt/sds-hd</syntaxhighlight>.

To access your Speichervorhaben, the export to BinAC 2 must first be enabled by the SDS@hd-Team. Please contact [mailto:sds-hd-support@urz.uni-heidelberg.de SDS@hd support] and provide the acronym of your Speichervorhaben, along with a request to enable the export to BinAC 2.

Once this has been done, you can access your Speichervorhaben as described in the [https://wiki.bwhpc.de/e/SDS@hd/Access/NFS#Access_your_data SDS documentation].

<syntaxhighlight>
$ kinit $USER
Password for <user>@BWSERVICES.UNI-HEIDELBERG.DE:
</syntaxhighlight>

The Kerberos ticket store is shared across all nodes. Creating a single ticket is sufficient to access your Speichervorhaben on all nodes.

=== More Details on the Lustre File System ===
[https://www.lustre.org/ Lustre] is a distributed parallel file system.
* The entire logical volume as presented to the user is formed by multiple physical or local drives. Data is distributed over more than one physical or logical volume/hard drive, single files can be larger than the capacity of a single hard drive.
* The file system can be mounted from all nodes ("clients") in parallel at the same time for reading and writing. This also means that technically you can write to the same file from two different compute nodes! Usually, this will create an unpredictable mess! Never ever do this unless you know exactly what you are doing!
* On a single server or client, the bandwidth of multiple network interfaces can be aggregated to increase the throughput ("multi-rail").

Lustre works by chopping files into many small parts ("stripes", file objects) which are then stored on the object storage servers. The information which part of the file is stored where on which object storage server, when it was changed last etc. and the entire directory structure is stored on the metadata servers. Think of the entries on the metadata server as being pointers pointing to the actual file objects on the object storage servers.
A Lustre file system can consist of many metadata servers (MDS) and object storage servers (OSS).
Each MDS or OSS can again hold one or more so-called object storage targets (OST) or metadata targets (MDT) which can e.g. be simply multiple hard drives.
The capacity of a Lustre file system can hence be easily scaled by adding more servers.

==== Useful Lustre Comamnds ====
Commands specific to the Lustre file system are divided into user commands (<code>lfs ...</code>) and administrative commands (<code>lctl ...</code>). On BinAC2, users may only execute user commands, and also not all of them.
* <code>lfs help <command></code>: Print built-in help for command; Alternative: <code>man lfs <command></code>
* <code>lfs find</code>: Drop-in replacement for the <code>find</code> command, much faster on Lustre filesystems as it directly talks to the metadata sever
* <code>lfs --list-commands</code>: Print a list of available commands

==== Moving data between WORK and PROJECT ====
!! IMPORTANT !! Calling <code>mv</code> on files will not physically move them between the fast and the slow pool of the file system. Instead, the file metadata, i.e. the path to the file in the directory tree will be modified (i.e. data stored on the MDS). The stripes of the file on the OSS, however, will remain exactly were they were. The only result will be the confusing situation that you now have metadata entries under <code>/pfs/10/project</code> that still point to WORK OSTs. This may sound confusing at first. When using <code>mv</code> on the same file system, Lustre only renames the files and makes them available from a different path. The pointers to the file objects on the OSS stay identical. This will only change if you either create a copy of the file at a different path (with <code>cp</code> or <code>rsync</code>, e.g.) or if you explicitly instruct Lustre to move the actual file objects to another storage location, e.g. another pool of the same file system.

Proper ways of moving data between the pools
* Copy the data - which will create new files -, then delete the old files. Example:
<pre>
$> cp -ar /pfs/10/work/tu_abcde01-my-precious-ws/* /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/.
$> rm -rf /pfs/10/work/tu_abcde01-my-precious-ws/*
$> ws_release --delete-data my-precious-ws
</pre>
* Alternative to copy: use <code>rsync</code> to copy data between the workspace and the project directories. Example:
<pre>
$> rsync -av /pfs/10/work/tu_abcde01-my-precious-ws/simulation/output /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/simulation25/
$> rm -rf /pfs/10/work/tu_abcde01-my-precious-ws/*
$> ws_release --delete-data my-precious-ws
</pre>
* If there are many subfolders with similar size, you can use <code>xargs</code> to copy them in parallel:
<pre>
$> find . -maxdepth 1 -mindepth 1 -type d -print | xargs -P4 -I{} rsync -aHAXW --inplace --update {} /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/simulation25/
</pre>
will launch four parallel <code>rsync</code> processes at a time, each will copy one of the subdirectories.
* First move the metadata with <code>mv</code>, then use <code>lfs migrate</code> or the wrapper <code>lfs_migrate</code> to actually migrate the file stripes. This is also a possible resolution if you already <code>mv</code>ed data from <code>work</code> to <code>project</code> or vice versa.
** <code>lfs migrate</code> is the raw lustre command. It can only operate on one file at a time, but offers access to all options.
** <code>lfs_migrate</code> is a versatile wrapper script that can work on single files or recursively on entire directories. If available, it will try to use <code>lfs migrate</code>, otherwise it will fall back to <code>rsync</code> (see <code>lfs_migrate --help</code> for all options.)
Example with <code>lfs migrate</code>:
<pre>
$> mv /pfs/10/work/tu_abcde01-my-precious-ws/* /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/.
$> cd /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/.
$> lfs find . -type f --pool work -0 | xargs -0 lfs migrate --pool project # find all files whose file objects are on the work pool and migrate the objects to the project pool
$> ws_release --delete-data my-precious-ws
</pre>
Example with <code>lfs_migrate</code>:
<pre>
$> mv /pfs/10/work/tu_abcde01-my-precious-ws/* /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/.
$> cd /pfs/10/project/bw10a001/tu_abcde01/my-precious-research/.
$> lfs_migrate --yes -q -p project * # migrate all file objects in the current directory to the project pool, be quiet (-q) and do not ask for confirmation (--yes)
$> ws_release --delete-data my-precious-ws
</pre>
Both migration commands can also be combined with options to restripe the files during migration, i.e. you can also change the number of OSTs the file is striped over, the size of a single strip etc.
Attention! Both <code>lfs migrate</code> and <code>lfs_migrate</code> will not change the path of the file(s), you must also <code>mv</code> them! If used without <code>mv</code>, the files will still belong to the workspace although their file object stripes are now on the <code>project</code> pool and a subsequent <code>rm</code> in the workspace will wipe them.

All of the above procedures may take a considerable amount of time depending on the amount of data, so it might be advisable to execute them in a terminal multiplexer like <code>screen</code> or <code>tmux</code> or wrap them into small SLURM jobs with <code>sbatch --wrap="<command>"</code>.

Question: I totally lost overview, how do i find out where my files are located?

Answer:
* Use <code>lfs find</code> to find files on a specific pool. Example:
<pre>
$> lfs find . --pool project # recursively find all files in the current directory whose file objects are on the "project" pool
</pre>
* Use <code>lfs getstripe</code> to query the striping pattern and the pool (also works recursively if called with a directory). Example:
<pre>
$> lfs getstripe parameter.h
parameter.h
lmm_stripe_count: 1
lmm_stripe_size: 1048576
lmm_pattern: raid0
lmm_layout_gen: 1
lmm_stripe_offset: 44
lmm_pool: project
obdidx objid objid group
44 7991938 0x79f282 0xd80000400
</pre>
shows that the file is striped over OST 44 (obdidx) which belongs to pool project (lmm_pool).

Why pathes and storage pools should match:
There are four different possible scenarios with two subdirectories and two pools:
* File path in <code>/pfs/10/work</code>, file objects on pool <code>work</code>: good.
* File path in <code>/pfs/10/project</code>, file objects on pool <code>project</code>: good.
* File path in <code>/pfs/10/project</code>, file objects on pool <code>work</code>: bad. This will "leak" storage from the fast pool, making it unavailable for workspaces.
* File path in <code>/pfs/10/work</code>, file objects on pool <code>project</code>: bad. Access will be slow, and if (volatile) workspaces are purged, data residing on <code>project</code> will (voluntarily or involuntarily) be deleted.
The latter two situations may arise from <code>mv</code>ing data between workspaces and project folders.

==== More on data striping and how to influence it ====
!! The default striping patterns on BinAC2 are set for good reasons and should not light-heartedly be changed! Doing so wrongly will in the best case only hurt your performance. In the worst case, it will also hurt all other users and endanger the stability of the cluster. Please talk to the admins first if you think that you need a non-default pattern.
* Reading striping patterns with <code>lfs getstripe</code>
* Setting striping patterns with <code>lfs setstripe</code> for new files and directories
* Restriping files with <code>lfs migrate</code>
* Progressive File Layout

==== Architecture of BinAC2's Lustre File System ====
Metadata Servers:
* 2 metadata servers
* 1 MDT per server
* MDT Capacity: 31TB, hardware RAID6 on NVMe drives (flash memory/SSD)
* Networking: 2x 100 GbE, 2x HDR-100 InfiniBand

Object storage servers:
* 8 object storage servers
* 2 fast OSTs per server
** 70 TB per OST, software RAID (raid-z2, 10+2 reduncancy)
** NVMe drives, directly attached to the PCIe bus
* 8 slow OSTs per server
** 143 TB per OST, hardware RAID (RAID6, 8+2 redundancy)
** externally attached via SAS
* Networking: 2x 100 GbE, 2x HDR-100 InfiniBand

* All fast OSTs are assigned to the pool <code>work</code>
* All slow OSTs are assigned to the pool <code>project</code>
* All files that are created under <code>/pfs/10/work</code> are by default stored on the fast pool
* All files that are created under <code>/pfs/10/project</code> are by default stored on the slow pool
* Metadata is distributed over both MDTs. All subdirectories of a directory (workspace or project folder) are typically on the same MDT. Directory striping/placement on MDTs can not be influenced by users.
* Default OST striping: Stripes have size 1 MiB. Files are striped over one OST if possible, i.e. all stripes of a file are on the same OST. New files are created on the most empty OST.
Internally, the slow and the fast pool belong to the same Lustre file system and namespace.

More reading:
* [https://doc.lustre.org/lustre_manual.xhtml The Lustre 2.X Manual] ([http://doc.lustre.org/lustre_manual.pdf PDF])
* [https://wiki.lustre.org/Main_Page The Lustre Wiki]

BinAC2/Software

2025-12-19T13:49:16Z

F Bartusch:

== Environment Modules ==
Most software is provided as Modules.

Required reading to use: [[Environment Modules]]

== Available Software ==

* Web: Visit [https://www.bwhpc.de/software.php https://www.bwhpc.de/software.php], select <code>Cluster → bwForCluster BinAC 2</code>

* On the cluster: <code>module avail</code>


== Documentation ==

Documentation for environment modules available on the cluster:

* with command <code>module help</code>
* examples in <code>$SOFTNAME_EXA_DIR</code>

== Documentation in the Wiki ==

For some applications additional documentation is provided here.

* [[BinAC2/Software/Alphafold | Alphafold 3 ]]
* [[BinAC2/Software/AMDock | AMDock ]]


* [[BinAC/Software/Cellranger | Cell Ranger]]
* [[Development/Conda | Conda / Mamba]]

* [[BinAC2/Software/Jupyterlab | JupyterLab]]
* [[BinAC2/Software/Nextflow | Nextflow and nf-core]]
* [[BinAC2/Software/Snakemake | Snakemake]]
* [[BinAC2/Software/TigerVNC | TigerVNC: Remote visualization using VNC]]

BinAC2/Software

2025-12-19T13:37:48Z

F Bartusch:

== Environment Modules ==
Most software is provided as Modules.

Required reading to use: [[Environment Modules]]

== Available Software ==

* Web: Visit [https://www.bwhpc.de/software.php https://www.bwhpc.de/software.php], select <code>Cluster → bwForCluster BinAC 2</code>

* On the cluster: <code>module avail</code>


== Documentation ==

Documentation for environment modules available on the cluster:

* with command <code>module help</code>
* examples in <code>$SOFTNAME_EXA_DIR</code>

== Documentation in the Wiki ==

For some applications additional documentation is provided here.

* [[BinAC2/Software/Alphafold | Alphafold 3 ]]
* [[BinAC2/Software/AMDock | AMDock ]]


* [[BinAC/Software/Cellranger | Cell Ranger]]
* [[Development/Conda | Conda]]

* [[BinAC2/Software/Jupyterlab | JupyterLab]]
* [[BinAC2/Software/Nextflow | Nextflow and nf-core]]
* [[BinAC2/Software/Snakemake | Snakemake]]
* [[BinAC2/Software/TigerVNC | TigerVNC: Remote visualization using VNC]]

BinAC2/Software/Alphafold

2025-12-16T10:57:55Z

F Bartusch: Created page with "{{Softwarepage|bio/alphafold}} {| width=600px class="wikitable" |- ! Description !! Content |- | module load | bio/alphafold |- | License | ACC BY-NC-SA 4.0 - see [https://github.com/google-deepmind/alphafold3/blob/main/LICENSE] |- | Citing | See [https://github.com/google-deepmind/alphafold3?tab=readme-ov-file#citing-this-work] |- | Links | DeepMind AlphaFold Website: [https://deepmind.google/technologies/alphafold/] Alphafold 3 Repository: [https://github.com/goo..."

{{Softwarepage|bio/alphafold}}

{| width=600px class="wikitable"
|-
! Description !! Content
|-
| module load
| bio/alphafold
|-
| License
| ACC BY-NC-SA 4.0 - see [https://github.com/google-deepmind/alphafold3/blob/main/LICENSE]
|-
| Citing
| See [https://github.com/google-deepmind/alphafold3?tab=readme-ov-file#citing-this-work]
|-
| Links
| DeepMind AlphaFold Website: [https://deepmind.google/technologies/alphafold/] 
Alphafold 3 Repository: [https://github.com/google-deepmind/alphafold3]
|}

= Description =

AlphaFold 3 developed by DeepMind predicts the 3D structure and interactions of biological molecules: proteins, DNA, RNA, ligands, and other small molecules.

BinAC 2 provides almost everything you need for working with Alphafold 3:
* Alphafold 3 installed in an Apptainer image
* Alphafold 3 database

'''BUT''' you have to bring your own parameter file! Due to license issues we are not allowed to share them publicly. You can find information on how to obtain them [https://github.com/google-deepmind/alphafold3?tab=readme-ov-file#obtaining-model-parameters here].

= Interpreting Predictions =

When using AlphaFold3, users should treat predicted structures as hypotheses rather than definitive representations of molecular reality.
While the method often produces highly accurate models, confidence can vary substantially across regions, especially for flexible loops, disordered segments, or novel interactions not well represented in training data. Users should therefore critically assess confidence metrics provided by the model and, where possible, validate predictions against experimental data or independent computational approaches. Biological plausibility, consistency with known functional or biochemical evidence, and sensitivity to alternative inputs should also be considered. Overall, AlphaFold3 outputs are most powerful when used as guidance for downstream analysis and experimental design, not as final ground truth.

AlphaFold 3 supplies multiple confidence metrics to help you critically assess its predictions:

* Predicted LDDT (pLDDT): predicted atomic coordinates are accompanied by pLDDT scores. These reflect AlphaFold 3’s local confidence in the prediction of the position of that particular atom.
* Predicted Aligned Error (PAE) scores and a PAE plot: an indication of AlphaFold’s confidence in the packing and relative positions of domains, molecular chains such as proteins and DNA, and other entities like ligands and ions.
* Predicted TM (pTM) score: a single-value metric reflecting the accuracy of the overall predicted structure.
* Interface-predicted TM (ipTM) score: measures the accuracy of predictions of one component of the complex relative to the other components of the complex.
* Per chain pTM and per-chain pair ipTM: confidence in individual chains or pairs of chains.

Further Reading:

* [https://www.ebi.ac.uk/training/online/courses/alphafold/alphafold-3-and-alphafold-server/introducing-alphafold-3/how-have-alphafold-3s-predictions-been-validated/ AlphaFold3 validation overview]

* [https://www.ebi.ac.uk/training/online/courses/alphafold/alphafold-3-and-alphafold-server/how-to-assess-the-quality-of-alphafold-3-predictions/ How to assess the quality of AlphaFold3 predictions]

= Usage =

AlphaFold's algorithm can be devided into two steps:

# CPU-only part: Computation of several multiple sequence alignments (MSA)
# GPU-part: MSA used as input for neural network for infering structure

This results in two optimal resource profiles regarding the number of CPU cores and GPUs.
Therefore we provide two template jobscripts for both steps at <code>/opt/bwhpc/common/bio/alphafold/3.0.1/bwhpc-examples</code>.

== Input ==

You can provide inputs to <code>run_alphafold.py</code> in one of two ways:

* Single input file: Use the <code>--json_path</code> flag followed by the path to a single JSON file.
* Multiple input files: Use the <code>--input_dir</code> flag followed by the path to a directory of JSON files.

Please see the official [https://github.com/google-deepmind/alphafold3/blob/main/docs/input.md#top-level-structure AlphaFold 3 documentation] for more information regarding JSON file structure.

The example jobscript uses a very simple input JSON:

<syntaxhighlight>
{
"name": "2PV7",
"sequences": [
{
"protein": {
"id": ["A", "B"],
"sequence": "GMRESYANENQFGFKTINSDIHKIVIVGGYGKLGGLFARYLRASGYPISILDREDWAVAESILANADVVIVSVPINLTLETIERLKPYLTENMLLADLTSVKREPLAKMLEVHTGAVLGLHPMFGADIASMAKQVVVRCDGRFPERYEWLLEQIQIWGAKIYQTNATEHDHNMTYIQALRHFSTFANGLHLSKQPINLANLLALSSPIYRLELAMIGRLFAQDAELYADIIMDKSENLAVIETLKQTYDEALTFFENNDRQGFIDAFHKVRDWFGDYSEQFLKESRQLLQQANDLKQG"
}
}
],
"modelSeeds": [1],
"dialect": "alphafold3",
"version": 1
}
</syntaxhighlight>

== AlphaFold 3 database ==

Don't change <code>--db_dir=$ALPHAFOLD_DATABASES</code> without a good reason. Upon loading the AlphaFold 3 module, the environment variable <code>$ALPHAFOLD_DATABASES</code> is set to the centrally stored database at <code>/pfs/10/project/db/alphafold/3.0.1/databases/</code>.

== AlphaFold 3 model parameters ==

Set the path to your AlphaFold 3 model parameters you received from DeepMind after applying for it. If you place the file <code>af3.bin.zst</code> in the directory <code>$HOME/af3-models</code> the example scripts will work out of the box.

Please note: You can also store the parameters in your project directory or a workspace for slightly better performance. The template uses <code>$HOME/af3-models</code> because this environment variable works for everyone.

== Output ==

You can set the output directory via the <code>--output_dir=$ALPHAFOLD_RESULTS_DIR</code> option. The template jobscript creates a workspace called <code>alphafold</code> as output directory.

For every input job, AlphaFold 3 writes all its outputs in a directory called by the sanitized version of the job name. E.g. for job name "My first fold (TEST)", AlphaFold 3 will write its outputs in a directory called <code>My_first_fold_TEST</code> (the case is respected). If such directory already exists, AlphaFold 3 will append a timestamp to the directory name to avoid overwriting existing data unless <code>--force_output_dir</code> is passed.

Please see the official [https://github.com/google-deepmind/alphafold3/blob/main/docs/output.md AlphaFold 3 documentation] for more information regarding output directory structure.

= CPU-only: Multiple Sequence Alignment =

In the beginning, AlphaFold 3 computes three multiple sequence alignments (MSA).
These MSAs are computed on the CPU sequentially and the number of threads are hard-coded:

* jackhmmer on UniRef90 using 8 threads
* jackhmmer on MGnify using 8 threads
* HHblits on BFD + Uniclust30 using 4 threads

We provide a template jobscript at: <code>/opt/bwhpc/common/bio/alphafold/3.0.1/bwhpc-examples/binac2-af3-alignment.slurm
</code>

You can change <code>--time</code> and <code>--mem</code>, although these are sensible defaults.

<syntaxhighlight>
#!/bin/bash
#SBATCH --partition=compute
#SBATCH --nodes=1

# Alphafold creates alignments sequentially, using at max 8 cores.
#SBATCH --ntasks-per-node=8

#SBATCH --time=10:00:00
#SBATCH --mem=100gb
</syntaxhighlight>

The MSAs are stored in the directory specified by <code>--output_dir</code>.

= GPU-part: Model Inference =

Aftere computing the MSAs, AlphaFold performs model inference on the GPU. Only one GPU is used.
This use case has this optimal resource profile:

<syntaxhighlight>
#!/bin/bash
#SBATCH --partition=gpu
#SBATCH --nodes=1
#SBATCH --cpus-per-task=8
#SBATCH --gres=gpu:a100:1
#SBATCH --time=06:00:00
#SBATCH --mem=100gb
</syntaxhighlight>

The template jobscript only differs to CPU-only part in the option <code>--norun_data_pipeline</code>. This means the MSAs aren't recomputed, but are taken from the previous CPU-only job.

= Chaining CPU- and GPU-Jobs =

To run the data pipeline first and then start the inference job as soon as the first one is finished, you can chain them like this:

<syntaxhighlight>
JOBID=$(sbatch --parsable binac2-af3-alignment.slurm)
sbatch --dependency=afterok:$JOBID binac2-af3-inference.slurm
</syntaxhighlight>

BinAC2/Acknowledgement

2025-12-05T11:30:59Z

F Bartusch:

When preparing a publication describing work that involved the usage of a bwForCluster, e.g. BinAC 2, please ensure that you reference the bwHPC initiative, the bwHPC-S5 project and – if appropriate – also the bwHPC facility itself. The following sample text is suggested as a starting point.

'''Acknowledgement:'''
<blockquote>
'''The authors acknowledge support by the High Performance and Cloud Computing Group at the Zentrum für Datenverarbeitung of the University of Tübingen, the state of Baden-Württemberg through bwHPC and the German Research Foundation (DFG) for funding under [https://gepris.dfg.de/gepris/projekt/455787709 "Project number 455787709"] (bwForCluster BinAC 2)'''
</blockquote>

In addition, we kindly ask you to notify us of any reports, conference papers, journal articles, theses, posters, talks which contain results obtained on any bwHPC resource by sending an email to
[mailto:publications@bwhpc.de publications@bwhpc.de] stating:
* cluster facility (e.g. bwForCluster BinAC 2)
* RV acronym (e.g. bw16A000)
* author(s)
* title ''or'' booktitle
* journal, volume, pages ''or'' editors, address, publisher
* year.

Such recognition is highly important for acquiring funding for the next generation of hardware, support services, data storage and infrastructure.

The publications will be referenced on the bwHPC website:
https://www.bwhpc.de/user_publications.html

BinAC2/Acknowledgement

2025-12-05T11:29:29Z

F Bartusch: Use project number in acknowledgement

When preparing a publication describing work that involved the usage of a bwForCluster, e.g. BinAC 2, please ensure that you reference the bwHPC initiative, the bwHPC-S5 project and – if appropriate – also the bwHPC facility itself. The following sample text is suggested as a starting point.

<blockquote>
The authors acknowledge support by the High Performance and Cloud Computing Group at the Zentrum für Datenverarbeitung of the University of Tübingen, the state of Baden-Württemberg through bwHPC and the German Research Foundation (DFG) for funding under [https://gepris.dfg.de/gepris/projekt/455787709 "Project number 455787709"] (bwForCluster BinAC 2)
</blockquote>

In addition, we kindly ask you to notify us of any reports, conference papers, journal articles, theses, posters, talks which contain results obtained on any bwHPC resource by sending an email to
[mailto:publications@bwhpc.de publications@bwhpc.de] stating:
* cluster facility (e.g. bwForCluster BinAC 2)
* RV acronym (e.g. bw16A000)
* author(s)
* title ''or'' booktitle
* journal, volume, pages ''or'' editors, address, publisher
* year.

Such recognition is highly important for acquiring funding for the next generation of hardware, support services, data storage and infrastructure.

The publications will be referenced on the bwHPC website:
https://www.bwhpc.de/user_publications.html

Registration/Login/Username

2025-12-02T08:46:44Z

F Bartusch: BinAC -> BinAC 2

__FORCETOC__

= Login Username =

All members of universities and colleges in Baden-Württemberg can use the bwHPC resources.
Prefixes are used to ensure that the usernames assigned by the home organization are unique within the state.
The username for the bwHPC clusters is the same as the one assigned by the university or college, but it is prefixed with two letters for the home institution.

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
Starting with '''bwUniCluster 3.0''', KIT users have the '''prefix ka''' on all systems.
|}

There are two ways to find out your username for the bwForCluster and the bwUniCluster:

* [[#Get_your_Username_by_adding_a_Prefix_to_your_local_Account_Name|Get your Username by adding a Prefix to your local Account Name]]
* [[#Find_out_your_Username_by_visiting_the_Registration_Service|Find out your Username by visiting the Registration Service]]

== Get your Username by adding a Prefix to your local Account Name ==

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
If your university is not listed in the [https://www.bwidm.de/hochschulen.php '''bwIDM Membership Table'''], please ask your university to join bwIDM or update the table.
|}

If you want to use a '''bwForCluster''' or the '''bwUniCluster''' you need to add a prefix to your local username.
The username is constructed as follows (see [[#Examples|Examples]] below):

<code><prefix>_<local username></code>

=== Prefix for Universities of Applied Sciences ===

For universities see table below, users from the [[Registration/HAW|HAW BW e.V.]] or a university of applied sciences should use this table updated by bwIDM as a guideline:
* [https://www.bwidm.de/hochschulen.php '''bwIDM Membership Table''']

=== Prefix for Universities ===

{| class="wikitable"
|-
!style="width:280px" | User from University
!style="width:60px" | Prefix
!style="width:430px" | Username
|-
| Universität Freiburg
| fr
| colspan="2" align="center"| fr_''<username>''
|-
| Universität Heidelberg
| hd
| colspan="2" align="center"| hd_''<username>''
|-
| Universität Hohenheim
| ho
| colspan="2" align="center"| ho_''<username>''
|-
| Karlsruhe Institute of Technology (KIT)
| ka
| align="center"| ka_''<username>''
|-
| Universität Konstanz
| kn
| colspan="2" align="center"| kn_''<username>''
|-
| Universität Mannheim
| ma
| colspan="2" align="center"| ma_''<username>''
|-
| Universität Stuttgart
| st
| colspan="2" align="center"| st_''<username>''
|-
| Universität Tübingen
| tu
| colspan="2" align="center"| tu_''<username>''
|-
| Universität Ulm
| ul
| colspan="2" align="center"| ul_''<username>''
|}

=== Examples ===

* If you are a member of the University of Konstanz and your local username is <code>ab1234</code>, your username on any bwHPC cluster is <code>kn_ab1234</code>.
* If your local username for the university is <code>vwxyz1234</code> and you are a user of the University of Freiburg, your username on any bwHPC cluster is <code>fr_vwxyz1234</code>.
* If you are from Aalen University and your username is <code>xyzs12342</code>, your username for any bwHPC cluster is <code>aa_xyzs12342</code>.
* If your KIT username is <code>ab1234</code>, your username on any bwHPC cluster is <code>ka_ab1234</code>.

== Find out your Username by visiting the Registration Service ==

Alternatively, you can log in to the registration service and verify your username online.
To do this, follow the next steps:

1. Select the cluster you want to know your username for: → [https://login.bwidm.de '''bwUniCluster 3.0'''] → [https://bwservices.uni-tuebingen.de '''BinAC 2'''] → [https://login.bwidm.de '''JUSTUS 2'''] → [https://bwservices.uni-heidelberg.de '''Helix'''] → [https://login.bwidm.de '''NEMO2''']

2. Select your home organization from the list and click '''Proceed'''.

3. Enter your home-organisational user ID / username and your home-organisational password and click '''Login''' button.

4. You will be redirected back to the registration website.

5. Find the cluster entry and select '''Registry Info'''.
[[File:BwIDM-pw.png|center|frame|Check Registry Info.]]

6. Your cluster username will be after the string '''Username for login'''.
{| style="width:50%;" align="center"
|-
!style="width:50%;" align="center"|[[File:BwIDM-user.png|center|thumb|300px|Username for login (newer registration services).]]
|-
|}

Registration/Account

2025-12-02T08:45:13Z

F Bartusch: BinAC -> BinAC 2

If you have trouble login to your account, on the cluster, you can try the following to verify your account:

1. Select the cluster you want to check your account and select your home organization: → [https://login.bwidm.de '''bwUniCluster 3.0'''] → [https://bwservices.uni-tuebingen.de '''BinAC 2'''] → [https://login.bwidm.de '''JUSTUS 2'''] → [https://bwservices.uni-heidelberg.de '''Helix'''] → [https://login.bwidm.de '''NEMO2''']

2. Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.
[[File:BwIDM-login.png|center|600px|thumb|Select your home organization]]

3. Authenticate yourself via the user id / username and password provided by your home institution.

4. You will be redirected back to the registration website.

5. Find the cluster entry and select '''Registry info'''
[[File:Reg_info.png|center|frame|Registry info.]]

6. Verify that your account is '''ACTIVE'''.
[[File:Reg_check_account.png|center|800px|thumb|Registry info.]]

Registration/Deregistration

2025-12-02T08:44:41Z

F Bartusch: BinAC -> BinAC 2

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
Note that de-registering automatically unsubscribes you from the cluster mailing list.
|}

If you plan to permanently leave a bwUniCluster/bwForCluster, follow the de-register checklist:

1. Transfer all your data in $HOME and your workspaces to your local computer/storage and after that clear off all your data.

2. Select the cluster you want to de-register: 
→ [https://login.bwidm.de '''bwUniCluster 3.0'''] 

→ [https://bwservices.uni-tuebingen.de '''BinAC 2'''] 
→ [https://login.bwidm.de '''JUSTUS 2'''] 
→ [https://bwservices.uni-heidelberg.de '''Helix'''] 
→ [https://login.bwidm.de '''NEMO2''']

3. Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.
[[File:BwIDM-login.png|center|600px|thumb|Select your home organization]]

4. Authenticate yourself via the user id / username and password provided by your home institution.

5. You will be redirected back to the registration website.

6. Find the cluster entry and select '''Registry Info'''.
[[File:BwIDM-pw.png|center|frame|De-register from a bwForCluster.]]

7. Click '''Deregister'''.

8. You will be redirected to a confirmation page. Click '''DEREGISTER'''.
[[File:BwIDM-dereg.png|center|400px|thumb|Confirm de-registeristration.]]

9. If the de-registration was successful, you will be redirected back to the main page.

Registration/Password

2025-12-02T08:43:58Z

F Bartusch: BinAC -> BinAC 2

Your bwUniCluster/bwForCluster '''password''' is the service password you set during the registration for [[Registration/bwUniCluster/Service|bwUniCluster]] or [[Registration/bwForCluster/Service|bwForCluster]].
At any time, you can set a new bwUniCluster/bwForCluster password via the registration websites by carrying out the following steps:

1. Select the cluster you want change your password and select your home organization: → [https://login.bwidm.de '''bwUniCluster 3.0'''] → [https://bwservices.uni-tuebingen.de '''BINAC 2'''] → [https://login.bwidm.de '''JUSTUS 2'''] → [https://bwservices.uni-heidelberg.de '''Helix'''] → [https://login.bwidm.de '''NEMO2''']

2. Select your home organization from the list on the main page and click '''Proceed''' or '''Fortfahren'''.
[[File:BwIDM-login.png|center|600px|thumb|Select your home organization]]

3. Authenticate yourself via the user id / username and password provided by your home institution.

4. You will be redirected back to the registration website.

5. Find the cluster entry and select '''Set Service Password'''
[[File:BwIDM-pw.png|center|frame|Set password for desired cluster.]]

6. Enter the new password, repeat it and click '''SAVE''' button.
Be sure to use a secure password that is different from any other passwords you currently use or have used on other systems.
[[File:BwIDM-passwd.png|center|600px|thumb|Set service password]]

7. If the change was successful, the message "Das Passwort wurde bei dem Dienst geändert/Password has been changed" will be shown.

SDS@hd/Access

2025-12-02T08:41:34Z

F Bartusch: BinAC -> BinAC 2

This page provides an overview on how to access data served by SDS@hd. To get an introduction to data transfer in general, see [[Data_Transfer|data transfer]].

== Prerequisites ==

* You need to be [[SDS@hd/Registration|registered]].
* You need to be in the belwue-Network. This means you have to use the VPN Service of your HomeOrganization, if you want to access SDS@hd from outside the bwHPC-Clusters (e.g. via eduroam or from your personal notebook).

== Needed Information, independent of the chosen tool ==

* [[Registration/Login/Username| Username]]: Same as for the bwHPC Clusters
* Password: The Service Password that you set at bwServices in the [[SDS@hd/Registration|registration step]].
* SV-Acronym: Use the lower case version of the acronym for all access options.
* Hostname: The hostname depends on the chosen network protocol:
** For [[Data_Transfer/SSHFS|SSHFS]] and [[Data_Transfer/SFTP|SFTP]]: ''lsdf02-sshfs.urz.uni-heidelberg.de''
** For [[SDS@hd/Access/SMB|SMB]] and [[SDS@hd/Access/NFS|NFS]]: ''lsdf02.urz.uni-heidelberg.de''
** For [[Data_Transfer/WebDAV|WebDAV]] the url is: ''https://lsdf02-webdav.urz.uni-heidelberg.de''

== Recommended Setup ==
The following graphic shows the recommended way for accessing SDS@hd via Windows/Mac/Linux. The table provides an overview of the most important access options and links to the related pages. 
If you have various use cases, it is recommended to use [[Data_Transfer/Rclone|Rclone]]. You can copy, sync and mount with it. Thanks to its multithreading capability Rclone is a good fit for transferring big data. 
For an overview of all connection possibilities, please have a look at [[Data_Transfer/All_Data_Transfer_Routes|all data transfer routes]].

[[File:Data_transfer_diagram_simple.jpg|center|500px]]
Figure 1: SDS@hd main transfer routes

{| class="wikitable"
|- style="font-weight:bold; text-align:center; vertical-align:middle;"
!
! Use Case
! Windows
! Mac
! Linux
! Possible Bandwith
! Firewall Ports
|-
| [[Data_Transfer/Rclone|Rclone]] + <protocol>
| copy, sync and mount, multithreading
| ✓
| ✓
| ✓
| depends on used protocol
| depends on used protocol
|-
| [[SDS@hd/Access/SMB|SMB]]
| mount as network drive in file explorer or usage via Rclone
| [[SDS@hd/Access/SMB#Windows|✓]]
| [[SDS@hd/Access/SMB#Mac|✓]]
| [[SDS@hd/Access/SMB#Linux|✓]]
| up to 40 Gbit/sec
| 139 (netbios), 135 (rpc), 445 (smb)
|-
| [[Data_Transfer/WebDAV|WebDAV]]
| go to solution for restricted networks
| [✓]
| ✓
| ✓
| up to 100GBit/sec
| 80 (http), 443 (https)
|- style="vertical-align:middle;"
| [[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]]
| Graphical User Interface (GUI)
| [[Data_Transfer/Graphical_Clients#MobaXterm|✓]]
| ☓
| ☓
| see sftp
| see sftp
|- style="vertical-align:middle;"
| [[SDS@hd/Access/NFS|NFS]]
| mount for multi-user environments
| ☓
| ☓
| [[SDS@hd/Access/NFS|✓]]
| up to 40 Gbit/sec
| -
|- style="vertical-align:middle;"
| [[Data_Transfer/SSHFS|SSHFS]]
| mount, needs stable internet connection
| ☓
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| see sftp
| see sftp
|- style="vertical-align:middle;"
| [[Data_Transfer/SFTP|SFTP]]
| interactive shell, better usability when used together with Rclone
| [[Data_Transfer/SFTP#Windows|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| up to 40 Gbit/sec
| 22 (ssh)
|}
Table 1: SDS@hd transfer routes

=== Access from a bwHPC Cluster ===

'''bwUniCluster''' 
You can't mount to your $HOME directory but you can create a mount under $TMPDIR by following the instructions for [[Data_Transfer/Rclone#Usage_Rclone_Mount | Rclone mount]]. It is advised to wait a couple of seconds (<code>sleep 5</code>) before trying to use the mounted directory.

'''bwForCluster Helix''' 
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes.

'''bwForCluster BinAC 2''' 
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes. The prerequisites are:
* The SV responsible has enabled the SV on BinAC 2 once by writing to [mailto:sds-hd-support@urz.uni-heidelberg.de sds-hd-support@urz.uni-heidelberg.de]
* You have a valid kerberos ticket, which can be fetched with <code>kinit <userID></code>

'''Other''' 
You can mount your SDS@hd SV on the cluster yourself by using [[Data_Transfer/Rclone#Usage_Rclone_Mount | Rclone mount]]. As transfer protocol you can use WebDAV or sftp. For a full overview please have a look at [[Data_Transfer/All_Data_Transfer_Routes | All Data Transfer Routes]].

=== Access via Webbrowser (read-only) ===

Visit [https://lsdf02-webdav.urz.uni-heidelberg.de/ lsdf02-webdav.urz.uni-heidelberg.de] and login with your SDS@hd username and service password. Here you can get an overview of the data in your "Speichervorhaben" and download single files. To be able to do more, like moving data, uploading new files, or downloading complete folders, a suitable client is needed as described above.

== Best Practices ==

* '''Managing access rights with ACLs''' -> Please set ACLs either via the [https://www.urz.uni-heidelberg.de/de/service-katalog/desktop-und-arbeitsplatz/windows-terminalserver Windows terminal server] or via bwForCluster Helix. ACL changes won't work when used locally on a mounted directory.
* '''Multiuser environment''' - -> Use [[SDS@hd/Access/NFS|NFS]]

== Troubleshooting ==

* '''Issue:''' The credentials aren't accepted
*: → '''[[Registration/Login/Username| Check your Username]]'''
*: → '''[[Registration/bwForCluster/Helix#Troubleshooting_with_the_Help_of_bwServices | Troubleshooting with bwServices]]'''

Registration/bwForCluster/Service

2025-12-02T08:38:42Z

F Bartusch: Remove BinAC 1

= Prerequisites for Step C =

Prerequisites for successful account creation:
* '''[[Registration/bwForCluster/Entitlement|Step A: bwForCluster Entitlement]]'''
* '''[[Registration/bwForCluster/RV|Step B: Apply for a Rechenvorhaben/project]].

Once you have registered your own RV (''Rechenvorhaben'') or a membership in an RV, you will receive an email with a website to create an account for yourself on that cluster.

= Step C: bwForCluster Registration (Account Creation) =

To finish the registration procedure select the cluster your RV was assigned to:

* '''[[Registration/bwForCluster/BinAC2|bwForCluster BinAC 2]]'''
* '''[[Registration/bwForCluster/JUSTUS2|bwForCluster JUSTUS 2]]'''
* '''[[Registration/bwForCluster/Helix|bwForCluster Helix]]'''
* '''[[Registration/bwForCluster/NEMO2|bwForCluster NEMO 2]]'''
----

[[Registration/bwForCluster| Go back to bwForCluster Registration Home]]

Registration/Login/Hostname

2025-12-02T08:36:44Z

F Bartusch: Remove BinAC 1

= Login Hostnames of the bwHPC Clusters =

From outside the clusters users can only use the login nodes to submit jobs.

Please go to the section of the cluster you want to login:

* '''[[Registration/Login/Hostname#bwUniCluster_3.0_Hostnames|bwUniCluster 3.0 Hostnames]]'''
* '''[[Registration/Login/Hostname#BINAC2 Hostnames|BINAC2 Hostnames]]'''
* '''[[Registration/Login/Hostname#JUSTUS2 Hostnames|JUSTUS2 Hostnames]]'''
* '''[[Registration/Login/Hostname#Helix Hostnames|Helix Hostnames]]'''
* '''[[Registration/Login/Hostname#NEMO2 Hostnames|NEMO2 Hostnames]]'''

== bwUniCluster 3.0 Hostnames ==

The system has two login nodes.
The selection of the login node is done automatically.
If you are logging in multiple times, different sessions might run on different login nodes.

Login to bwUniCluster 3.0:

{| class="wikitable"
! Hostname !! Node type
|-
| '''uc3.scc.kit.edu''' || login to one of the two login nodes
|-
| '''bwunicluster.scc.kit.edu''' || login to one of the two login nodes
|-
|}

In general, you should use automatic selection to allow us to balance the load over the four login nodes.
If you need to connect to specific login nodes, you can use the following hostnames:

{| class="wikitable"
! Hostname !! Node type
|-
| '''uc3-login1.scc.kit.edu''' || bwUniCluster 3.0 first login node
|-
| '''uc3-login2.scc.kit.edu''' || bwUniCluster 3.0 second login node
|-
|}

== BINAC2 Hostnames ==

BinAC 2 has one login node serving as a load balancer. We use DNS round-robin scheduling to load-balance the incoming connections between the actual three login nodes. If you are logging in multiple times, different sessions might run on different login nodes and hence programs started in one session might not be visible in another sessions.

{| class="wikitable"
! Hostname !! Destination
|-
| '''login.binac2.uni-tuebingen.de''' || one of the three login nodes
|-
|}

== JUSTUS2 Hostnames ==

The system has four login nodes.
The selection of the login node is done automatically.
If you are logging in multiple times, different sessions might run on different login nodes.

Login to JUSTUS2:

{| class="wikitable"
! Hostname !! Node type
|-
| '''justus2.uni-ulm.de''' || login to one of the four login nodes
|-
|}

In general, you should use automatic selection to allow us to balance the load over the four login nodes.
If you need to connect to specific login nodes, you can use the following hostnames:

{| class="wikitable"
! Hostname !! Node type
|-
| '''justus2-login01.rz.uni-ulm.de''' || JUSTUS2 first login node
|-
| '''justus2-login02.rz.uni-ulm.de''' || JUSTUS2 second login node
|-
| '''justus2-login03.rz.uni-ulm.de''' || JUSTUS2 third login node
|-
| '''justus2-login04.rz.uni-ulm.de''' || JUSTUS2 fourth login node
|-
|}

== Helix Hostnames ==

The system has two login nodes.
The selection of the login node is done automatically.
If you are logging in multiple times, different sessions might run on different login nodes.

Login to Helix:

{| class="wikitable"
! Hostname !! Node type
|-
| '''helix.bwservices.uni-heidelberg.de''' || login to one of two login nodes
|-
|}

In general, you should use automatic selection to allow us to balance the load over the login nodes.

== NEMO2 Hostnames ==

The system has two login nodes.
You have to select the login node yourself.

Login to NEMO2:

{| class="wikitable"
! Hostname !! Node type
|-
| '''login.nemo.uni-freiburg.de''' || NEMO2 first or second login node
|-
| '''login1.nemo.uni-freiburg.de''' || NEMO2 first login node
|-
| '''login2.nemo.uni-freiburg.de''' || NEMO2 second login node
|-
|}

Registration/Login

2025-12-02T08:33:35Z

F Bartusch:

= Login to the Clusters =

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
Access to the clusters (bwUniCluster/bwForCluster) is restricted to IP addresses of universities/colleges from Baden-Württemberg [https://bgpview.io/asn/553#prefixes-v4 (BelWü network)].
If you are outside the BelWü network (e.g. at home), you must first establish a VPN connection to your home university or a connection to an SSH jump host at your home university.
|}

== SSH Clients ==

After completing the [[Registration|web registration]] and [[Registration/Password|setting a service password]] the HPC cluster is ready for your '''SSH''' based login.
* [[Registration/Login/Client| What Client to Use]]

== Cluster Specific Information ==

Every cluster has its own documentation for login.
* If you want to '''login''' to the bwUniCluster, please refer to 
→ '''[[bwUniCluster3.0/Login|bwUniCluster 3.0]]''' 
* If you want to '''login''' to one of the bwForClusters, please refer to 
→ '''[[BinAC2/Login|BinAC 2]]''' 
→ '''[[JUSTUS2/Login|JUSTUS 2]]''' 
→ '''[[Helix/Login|Helix]]''' 
→ '''[[NEMO2/Login|NEMO2]]'''

= Allowed Activities on Login Nodes =

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
To guarantee usability for all the users of clusters you must not run your compute jobs on the login nodes.
Compute jobs must be submitted to the queuing system.
Any compute job running on the login nodes will be terminated without any notice.
Any long-running compilation or any long-running pre- or post-processing of batch jobs must also be submitted to the queuing system.
|}

The login nodes of the bwHPC clusters are the access point to the compute system, your <code>$HOME</code> directory and your workspaces.
These nodes are shared with all the users therefore, your activities on the login nodes are limited to primarily set up your batch jobs.
Your activities may also be:
* '''short''' compilation of your program code and
* '''short''' pre- and post-processing of your batch jobs.

= Additional Login Information =

Following sub-pages go into deeper detail about the following topics:
* [[Registration/Login/Username|How do I find out my cluster username?]]
* [[Registration/Login/Hostname|What are the hostnames of the login nodes of a cluster?]]
* [[Registration/Account|Is my account on the cluster still valid?]]
* Configuring your shell: [[.bashrc Do's and Don'ts]]
These pages are also referenced in the cluster-specific login documentations.

Main Page

2025-12-01T11:06:02Z

F Bartusch:

'''Welcome to the bwHPC Wiki.'''

bwHPC represents services and resources in the State of '''B'''aden-'''W'''ürttemberg, Germany, for High Performance Computing ('''HPC'''), Data Intensive Computing ('''DIC''') and Large Scale Scientific Data Management ('''LS2DM''').

The main bwHPC web page is at '''[https://www.bwhpc.de/ https://www.bwhpc.de/]'''.

Many topics depend on the cluster system you use.
First choose the cluster you use, then select the correct topic.




{| style=" background:#B8FFB8; width:100%;"
| style="padding:8px; background:#8AFF8A; font-size:120%; text-align:left" | [[Status|'''Status:''' Maintenances [0] & Outages [0]]]
|}









{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Courses / eLearning
|-
|
* [https://training.bwhpc.de/ eLearning and Online Courses]
* [https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC Introduction to Linux in HPC (external resource)]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | Need Access to a Cluster?
|-
|
* [[When to use a HPC Cluster]]
* [[Running Calculations]]
* [[Registration]]
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | HPC System Specific Documentation
|-
|
bwHPC encompasses several HPC compute clusters at different universities in Baden-Württemberg. Each cluster is dedicated to [https://www.bwhpc.de/bwhpc-domains.php specific research domains].

Documentation differs between compute clusters, please see cluster specific overview pages:
{|
|-
| style="padding:5px; width:30%" | [[BwUniCluster3.0|bwUniCluster 3.0]]
| style="padding-left:20px;" | General Purpose, Teaching
|-
| style="padding:5px; width:30%" | [[:JUSTUS2| bwForCluster JUSTUS 2]]
| style="padding-left:20px;" | Theoretical Chemistry, Condensed Matter Physics, and Quantum Sciences
|-
| style="padding:5px; width:30%" | [[Helix|bwForCluster Helix]]
| style="padding-left:20px;" | Structural and Systems Biology, Medical Science, Soft Matter, Computational Humanities, and Mathematics and Computer Science
|-
| style="padding:5px; width:30%" | [[NEMO2|bwForCluster NEMO 2]]
| style="padding-left:20px;" | Neurosciences, Particle Physics, Materials Science, and Microsystems Engineering
|-
| style="padding:5px; width:30%" | [[BinAC2|bwForCluster BinAC 2]]
| style="padding-left:20px;" | Bioinformatics, Astrophysics, Geosciences, Pharmacy, and Medical Informatics
|}
|-
|bwHPC Clusters: [https://www.bwhpc.de/cluster.php operational status]
Further Compute Clusters in Baden-Württemberg (separate access policies):
* Datenanalyse Cluster der Hochschulen (DACHS): [[DACHS | Datenanalyse Cluster der Hochschulen (DACHS)]]
* bwHPC tier 1: [https://kb.hlrs.de/platforms/index.php/Hunter_(HPE) Hunter] ([https://www.hlrs.de/apply-for-computing-time getting access])
* bwHPC tier 2: [https://www.nhr.kit.edu/userdocs/horeka HoreKa] ([https://www.nhr.kit.edu/userdocs/horeka/projects/ getting access])
|}

{| style="background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | Documentation valid for all Clusters
|-
|
* [[Environment Modules| Software Modules]] and software documentation explained
* [https://www.bwhpc.de/software.html List of Software] on all clusters
* [[Development| Software Development and Parallel Programming]]
* [[Energy Efficient Cluster Usage]]
* [[HPC Glossary]]

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | Scientific Data Storage
|-
|
For user guides of the scientific data storage services:
* [[SDS@hd]]: Everyone can join existing storage projects, entitlement needed for creating your own, [https://sds-hd.urz.uni-heidelberg.de/management/shib/sds_costs.php cost sheet]
* [https://uni-tuebingen.de/einrichtungen/zentrum-fuer-datenverarbeitung/projekte/laufende-projekte/bwsfs bwSFS]
Associated, but local scientific storage services are:
* [https://wiki.scc.kit.edu/lsdf/index.php/Category:LSDF_Online_Storage LSDF Online Storage] (only for KIT and KIT partners)
|}

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | Scientific Data Archiving
|-
|
For user guides of the scientific data archiving services:
* [https://www.rda.kit.edu/english bwDataArchive]: Open to all scientists at KIT and institutions in Baden-Württemberg that have signed a service agreement with KIT
Associated, but local scientific storage services are:
* [https://www.urz.uni-heidelberg.de/de/service-katalog/speicher/heiarchive heiArchive] (only for members of the University of Heidelberg)
For instructions on how to move the data, depending on the offered connections, can be found under [[Data Transfer | Data Transfer]].
|}

{| style="height:100%; background:#ffeaef; width:100%"
| style="padding:8px; background:#f5dfdf; font-size:120%; font-weight:bold; text-align:left" | Data Management
|-
|
* [[Data Transfer|Data Transfer]]
* [https://www.forschungsdaten.org/index.php/FDM-Kontakte#Deutschland Research Data Management (RDM)] contact persons
* [https://www.forschungsdaten.info Portal for Research Data Management] (Forschungsdaten.info)
|}
{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Support
|-
|
* [[BwSupportPortal|Submit a Ticket in our Support Portal]]
Support is provided by the [https://www.bwhpc.de/teams.php bwHPC Competence Centers]:
|}
{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Acknowledgement
|-
|
* Please [[Acknowledgement|acknowledge]] our resources in your publications.
|}

BinAC2/Software

2025-11-12T13:41:43Z

F Bartusch:

== Environment Modules ==
Most software is provided as Modules.

Required reading to use: [[Environment Modules]]

== Available Software ==

* Web: Visit [https://www.bwhpc.de/software.php https://www.bwhpc.de/software.php], select <code>Cluster → bwForCluster BinAC 2</code>

* On the cluster: <code>module avail</code>


== Documentation ==

Documentation for environment modules available on the cluster:

* with command <code>module help</code>
* examples in <code>$SOFTNAME_EXA_DIR</code>

== Documentation in the Wiki ==

For some applications additional documentation is provided here.

* [[BinAC2/Software/Alphafold | Alphafold 3 ]]
* [[BinAC2/Software/AMDock | AMDock ]]


* [[BinAC/Software/Cellranger | Cell Ranger]]
* [[Development/Conda | Conda]]

* [[BinAC2/Software/Jupyterlab | JupyterLab]]
* [[BinAC2/Software/Nextflow | Nextflow and nf-core]]
* [[BinAC2/Software/TigerVNC | TigerVNC: Remote visualization using VNC]]

BinAC2

2025-11-12T13:41:00Z

F Bartusch:

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:center; color:#000;vertical-align:middle;font-size:75%;" |
[[File:BinAC2_Logo_RGB_subtitel.svg|center|500px||]]
|-
|
|}

The '''bwForCluster BinAC 2''' supports researchers from the broader fields of Bioinformatics, Astrophysics, Geosciences, Pharmacy, and Medical Informatics.

{| style=" background:#FEF4AB; width:100%;"
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News and Events
|-
|


|}

{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support
|-
|
* [[BinAC2/Getting_Started|Getting Started]]
* [https://training.bwhpc.de E-Learning Courses]
* [[BinAC2/Support|Contact and Support]]
* Send [[Feedback|Feedback]] about Wiki pages
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | User Documentation
|-
|
* [[Registration/bwForCluster|Registration]]

* [[BinAC2/Login|Login]]

* [[BinAC2/Hardware_and_Architecture|Hardware and Architecture]]
** [[BinAC2/Hardware_and_Architecture#Compute_Nodes|Node Specifications]]
** [[BinAC2/Hardware_and_Architecture#File_Systems|File Systems and Workspaces]]
* Usage of [[BinAC2/Software|Software]] on BinAC 2
** For available Software Modules see [https://www.bwhpc.de/software.php bwhpc.de Software Search] (select bwForCluster BinAC 2)
** Create Software Environments with [[Development/Conda|Conda]]
** Use [[BinAC2/Software/Nextflow|nf-core Nextflow pipelines]]
** See [[Development]] for info about compiler and parallelization

* [[BinAC2/Slurm|Batch System (SLURM)]]
** [[BinAC2/SLURM_Partitions|SLURM Partitions]]

|}
{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Cluster Funding
|-
|
* Please [[BinAC2/Acknowledgement|acknowledge]] the cluster in your publications.
|}

BinAC2/Software

2025-11-12T13:09:27Z

F Bartusch:

== Environment Modules ==
Most software is provided as Modules.

Required reading to use: [[Environment Modules]]

== Available Software ==

* Web: Visit [https://www.bwhpc.de/software.php https://www.bwhpc.de/software.php], select <code>Cluster → bwForCluster BinAC 2</code>

* On the cluster: <code>module avail</code>


== Documentation ==

Documentation for environment modules available on the cluster:

* with command <code>module help</code>
* examples in <code>$SOFTNAME_EXA_DIR</code>

== Documentation in the Wiki ==

For some applications additional documentation is provided here.

* [[BinAC2/Software/Alphafold | Alphafold ]]
* [[BinAC2/Software/AMDock | AMDock ]]


* [[BinAC/Software/Cellranger | Cell Ranger]]
* [[Development/Conda | Conda]]

* [[BinAC2/Software/Jupyterlab | JupyterLab]]
* [[BinAC2/Software/Nextflow | Nextflow and nf-core]]
* [[BinAC2/Software/TigerVNC | TigerVNC: Remote visualization using VNC]]

BinAC2/Login

2025-10-22T09:16:01Z

F Bartusch:

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
Access to bwForCluster BinAC 2 is only possible from IP addresses within the [https://www.belwue.de BelWü] network which connects universities and other scientific institutions in Baden-Württemberg.
If your computer is in your University network (e.g. at your office), you should be able to connect to bwForCluster BinAC 2 without restrictions.
If you are outside the BelWü network (e.g. at home), a VPN (virtual private network) connection to your University network must be established first. Please consult the VPN documentation of your University.
|}

'''Prerequisites for successful login:'''

You need to have
* completed the 3-step [[registration/bwForCluster|bwForCluster registration]] procedure.
* [[Registration/Password|set a service password]] for bwForCluster BinAC 2.


= Login to bwForCluster BinAC 2 =

Login to bwForCluster BinAC 2 is only possible with a Secure Shell (SSH) client for which you must know your [[BinAC2/Login#Username|username]] on the cluster and the [[BinAC2/Login#Hostname|hostname]] of the BinAC 2 login node.

For more gneral information on SSH clients, visit the [[Registration/Login/Client|SSH clients Guide]].

== TOTP Second Factor ==

At the moment no second factor is needed. We are currently implementing a new TOTP procedure.

== Username ==

Your <code><username></code> on BinAC 2 consists of a prefix and your local username.
For prefixes please refer to the [[Registration/Login/Username|Username Guide]].

Example: If your local username at your University is <code>ab123</code> and you are a user from Tübingen University, your username on the cluster is: <code>tu_ab123</code>.

== Hostnames ==

BinAC 2 has one login node serving as a load balancer. We use DNS round-robin scheduling to load-balance the incoming connections between the actual three login nodes. If you are logging in multiple times, different sessions might run on different login nodes and hence programs started in one session might not be visible in another sessions.

{| class="wikitable"
! Hostname !! Destination
|-
| login.binac2.uni-tuebingen.de || one of the three login nodes
|-
|}

You can choose a specific login node by using specific ports on the load balancer. Please only do this if there is a real reason for that (e.g. connecting to a running tmux/screen session).

{| class="wikitable"
! Port !! Destination
|-
| 2221 || login01
|-
| 2222 || login02
|-
| 2223 || login03
|-
|}
Usage: <code>ssh -p <port> [other options] <username>@login.binac2.uni-tuebingen.de</code>

== Login with SSH command (Linux, Mac, Windows) ==

Most Unix and Unix-like operating systems like Linux or MacOS come with a built-in SSH client provided by the OpenSSH project.
Windows 10 and Windows also come with a built-in OpenSSH client.

For login use one of the following ssh commands:

ssh <username>@login.binac2.uni-tuebingen.de

To run graphical applications on the cluster, you need to enable X11 forwarding with the <code>-X</code> flag:

ssh -X <username>@login.binac2.uni-tuebingen.de

For login to a specific login node (here: login03):

ssh -p 2223 <username>@login.binac2.uni-tuebingen.de

== Login with graphical SSH client (Windows) ==

For Windows we suggest using MobaXterm for login and file transfer.

Start MobaXterm and fill in the following fields:
<pre>
Remote name : login.binac2.uni-tuebingen.de
Specify user name : <username>
Port : 22
</pre>

After that click on 'ok'. Then a terminal will open where you can enter your credentials.

BinAC/Acknowledgement

2025-10-10T09:23:08Z

F Bartusch:

When preparing a publication describing work that involved the usage of a bwForCluster, e.g. BinAC, please ensure that you reference the bwHPC initiative, the bwHPC-S5 project and – if appropriate – also the bwHPC facility itself. The following sample text is suggested as a starting point.

Acknowledgement:
The authors acknowledge support by the High Performance and Cloud Computing Group at the Zentrum für Datenverarbeitung of the University of Tübingen, the state of Baden-Württemberg through bwHPC
and the German Research Foundation (DFG) through grant no INST 37/935-1 FUGG.

In addition, we kindly ask you to notify us of any reports, conference papers, journal articles, theses, posters, talks which contain results obtained on any bwHPC resource by sending an email to
[mailto:publications@bwhpc.de publications@bwhpc.de] stating:
* cluster facility (e.g. bwForCluster BinAC)
* RV acronym (e.g. bw16A000)
* author(s)
* title ''or'' booktitle
* journal, volume, pages ''or'' editors, address, publisher
* year.

Such recognition is highly important for acquiring funding for the next generation of hardware, support services, data storage and infrastructure.

The publications will be referenced on the bwHPC website:
https://www.bwhpc.de/user_publications.html

BinAC2/Acknowledgement

2025-10-10T09:22:54Z

F Bartusch:

When preparing a publication describing work that involved the usage of a bwForCluster, e.g. BinAC2, please ensure that you reference the bwHPC initiative, the bwHPC-S5 project and – if appropriate – also the bwHPC facility itself. The following sample text is suggested as a starting point.

Acknowledgement:
The authors acknowledge support by the High Performance and Cloud Computing Group at the Zentrum für Datenverarbeitung of the University of Tübingen, the state of Baden-Württemberg through bwHPC
and the German Research Foundation (DFG) through grant no INST 37/1159-1 FUGG.

In addition, we kindly ask you to notify us of any reports, conference papers, journal articles, theses, posters, talks which contain results obtained on any bwHPC resource by sending an email to
[mailto:publications@bwhpc.de publications@bwhpc.de] stating:
* cluster facility (e.g. bwForCluster BinAC2)
* RV acronym (e.g. bw16A000)
* author(s)
* title ''or'' booktitle
* journal, volume, pages ''or'' editors, address, publisher
* year.

Such recognition is highly important for acquiring funding for the next generation of hardware, support services, data storage and infrastructure.

The publications will be referenced on the bwHPC website:
https://www.bwhpc.de/user_publications.html

BinAC2/Getting Started

2025-09-17T07:19:23Z

F Bartusch:

== Purpose and Goals ==

The Getting Started guide is designed for users who are new to HPC systems in general and to BinAC 2 specifically. After reading this guide, you should have a basic understanding of how to use BinAC 2 for your research.

Please note that this guide does not cover basic Linux command-line skills. If you're unfamiliar with commands such as listing directory contents or using a text editor, we recommend first exploring the Linux module on the [https://training.bwhpc.de bwHPC training platform].

This guide also doesn't cover every feature of the system but aims to provide a broad overview. For more detailed information about specific features, please refer to the dedicated Wiki pages on topics like the batch system, storage, and more.

Some terms in this guide may be unfamiliar. You can look them up in the [[HPC_Glossary|HPC Glossary]].

== General Workflow of Running a Calculation ==

On an '''HPC Cluster''', you do not simply log in and run your software. Instead, you write a '''Batch Script''' that contains all the commands needed to run and process your job, then submit it to a waiting queue to be executed on one of several hundred '''Compute Nodes'''.

== Get Access to the Cluster ==

Follow the registration process for the bwForCluster. → [[Registration/bwForCluster|How to Register for a bwForCluster]]

== Login to the Cluster ==

Set up your service password and 2FA token, then log in to BinAC 2. → [[BinAC2/Login|Login BinAC]]

== Using the Linux command line ==

It is expected that you have at least basic Linux and command-line knowledge before using bwForCluster BinAC 2.
There are numerous resources available online for learning fundamental concepts and commands.
Here are two:

* bwHPC Linux Training course → [https://training.bwhpc.de/ Linux course on training.bwhpc.de]
* HPC Wiki (external site) → [https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC/The_Command_Line Introduction to the Linux command line]

Also see: [[.bashrc Do's and Don'ts]]

= File System Basics =

BinAC 2 offers several file systems for your data, each serving different needs.
These are explained here in a short and simple form. For more detailed documentation, visit: [https://wiki.bwhpc.de/e/BwForCluster_BinAC_Hardware_and_Architecture#Storage_Architecture here].

== Home File System ==

Home directories are intended for the permanent storage of frequently used files, such as like source codes, configuration files, executable programs, conda environments, etc.
The home file system is backed up daily and has a quota.
If that quota is reached, you may experience issues when working with BinAC 2.

Here are some useful command line and bash tips for accessing the Home File system.

<source lang="bash">
# For changing to your home directory, simply run:
cd

# To access files in your home directory within your job script, you can use one of these:
~/myFile # or
$HOME/myFile
</source>

== Project File System ==

BinAC 2 has a <code>project</code> file system intended for data that:
* is shared between members of a compute project
* is not actively used for computations in near future

The data is stored on HDDs. The primary focus of <code>project</code> is pure capacity, not speed.

Every project gets a dedicated directory located at:

<syntaxhighlight>
/pfs/10/project/<project_id>/
</syntaxhighlight>

You can check the project you're member of:

<syntaxhighlight>
# id $USER | grep -o 'bw[^)]*'
bw16f003
</syntaxhighlight>

In this case, your project directory would be:
```
/pfs/10/project/bw16f003/
```

Check our [[BinAC2/Project_Data_Organization | data organization guide ]] for methods to organize data inside the project directory.

== Work File System ==

BinAC 2 has a <code>work</code> file system on SSDs intended for data that is actively used and produced by compute jobs.
Each user creates workspaces on their own via the [[BinAC2/Hardware_and_Architecture#Work | workspace tools]].

The project file system is available at <code>/pfs/10/work</code>

<source lang="bash">
$ ll /pfs/10/work/
total 1822
drwxr-xr-x. 3 root root 33280 Feb 12 14:56 db
drwx------. 5 tu_iioba01 tu_tu 25600 Jan 8 14:42 tu_iioba01-alphafold3
[..]
</source>

As you can see from the file permissions, the resulting workspace can only be accessed by you, not by other group members or other users.

== Scratch ==

Each compute node provides local storage, which is much faster than accessing <code>project</code> and <code>work</code> file systems.
When you execute a job, a dedicated temporary directory will be assigned to it on the compute node. This is often referred to as the <code>scratch</code> directory.
Programs frequently generate temporary data only needed during execution. If the program you are using offers an option for setting a temporary directory,
please configure it to use the <code>scratch</code> directory.
You can use the environment variable <code>$TMPDIR</code>, which will point to your job's <code>scratch</code> directory.

= Batch System Basics =

On HPC clusters like BinAC 2, you don't run analyses directly on the login node.
Instead, you write a script and submit it as a job to the batch system.
BinAC 2 uses SLURM as its batch system.
The system then schedules the job to run on one of the available compute nodes, where the actual computation takes place.

The cluster consists of compute nodes with different [[BinAC2/Hardware_and_Architecture#Compute_Nodes | hardware features]].
These hardware features are only available when submitting the jobs to the correct [[BinAC2/SLURM_Partitions | partitions]].

The getting started guide only provides very basic SLURM information.
Please read the extensive [[BinAC2/Slurm | SLURM documentation]].

== Simple Script Job ==

You will have to write job scripts in order to conduct your computations on BinAC 2.
Use your favourite text editor to create simple job script called 'myjob.sh'.

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
Please note that there are differences between Windows and Linux line endings.
Make sure that your editor uses Linux line endings when you are using Windows.
You can check your line endings with <code>vim -b <your script></code>. Windows line endings will be displayed as <code>^M</code>.
|}

<source lang="bash">
#SBATCH --ntasks=1
#SBATCH --time=10:00
#SBATCH --mem=5000m
#SBATCH --job-name=simple

echo "Scratch directory: $TMPDIR"
echo "Date:"
date

echo "My job is running on node:"
hostname
uname -a

sleep 240
</source>

== Basic SLURM commands ==

Submit the job script you wrote with <code>sbatch</code>.

<source lang="bash">
$ sbatch myjob.sh
Submitted batch job 75441
</source>

Take a note of your <code>jobID</code>. The scheduler will reserve one core and 5000MB memory for 5 minutes on a compute node for your job.
The job should be scheduled within seconds if BinAC 2 is not fully busy.
The output will be stored in a file called <code>slurm-<JobID>.out</code>

<source lang="bash">
$ cat slurm-75441.out
Scratch directory: /scratch/75441
Date:
Thu Feb 13 09:56:41 AM CET 2025
My job is running on node:
node1-083
Linux node1-083 5.14.0-503.14.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Nov 15 12:04:32 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
</source>

There are tons of options, details and caveats for SLURM job script.
Most of them are explained in the [[BinAC2/Slurm | SLURM documentation]].
If you encounter any problems, just send a mail to hpcmaster@uni-tuebingen.de.

You can get an overview of your queued and running jobs with <code>squeue</code>

<source lang="bash">
[tu_iioba01@login01 ~]$ squeue --user=$USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
75441 compute simple tu_iioba R 0:03 1 node1-083
</source>

Let's assume you pulled a Homer and want to stop/kill/remove a running job.

<source lang="bash">
scancel <JobID>
</source>



= Software =

There are several mechanisms how software can be installed on BinAC 2.
If you need software that is not installed on BinAC 2, open a ticket and we can find a way to provide the software on the cluster.

== Environment Modules ==

Environment modules is the 'classic' way for providing software on clusters.
A module consists of a specific software version and can be loaded.
The module system then manipulates the PATH and other environment variables such that the software can be used.

<source lang="bash">
# Show available modules
$ module avail

# Load a module
$ module load bio/samtools/1.21

# Show the module's help
$ module help bio/samtools/1.21
</source>

A more detailed description of module environments can be found [https://wiki.bwhpc.de/e/Environment_Modules on this wiki page]

Sometimes software packages have so many dependencies or the user wants a combination of tools, so that environment modules cannot be used in a meaningful way.
Then other solutions like Conda environments or Singularity containers (see below) can be used.

== Conda Environments ==

Conda environments are a nice possibility for creating custom environments on the cluster, as a majority of the scientific software is available in the meantime as conda packages.
BinAC 2 already provides Conda via Miniforge.
You can find a general documtation for using Conda on [[Development/Conda | on this wiki page]].

== Apptainer (formerly Singularity) ==

Sometimes software is also available in a software container format.
Apptainer (formerly called Singularity) is installed on all BinAC 2 nodes. You can pull Apptainer containers or Docker images from registries onto BinAC 2 and use them.
You can also build new Apptainer containers on your own machine and copy them to BinAC.

Please note that Apptainer containers should be stored in the <code>project</code> file system.
We configured Apptainer such that containers stored in your home directory do not work.

BinAC2/Project Data Organization

2025-09-17T07:16:39Z

F Bartusch: Created page with "= Data Organization on BinAC 2 project directories = This guide explains how to organize and manage your data in the project directory at <code>/pfs/10/project</code> BinAC 2. Each project has its own dedicated directory with flexible permission systems to help you control access to your files and collaborate effectively with team members. '''Important''' Use workspaces for actual computations. Data in workspaces are stored on '''much''' faster storage! == Project Di..."

= Data Organization on BinAC 2 project directories =

This guide explains how to organize and manage your data in the project directory at <code>/pfs/10/project</code> BinAC 2.
Each project has its own dedicated directory with flexible permission systems to help you control access to your files and collaborate effectively with team members.

'''Important''' Use workspaces for actual computations. Data in workspaces are stored on '''much''' faster storage!

== Project Directory Structure ==

Every project gets a dedicated directory located at:
<pre>/pfs/10/project/<project_id>/</pre>

For example, if your project ID is <code>bw16f003</code>, your project directory would be:
<pre>/pfs/10/project/bw16f003/</pre>

=== Understanding Your Project Directory ===

When you list your project directory, you'll see something like this:
<pre>$ ls -ld /pfs/10/project/bw16f003/
drwxrwx---. 5 root bw16f003 33280 Jul 25 14:13 /pfs/10/project/bw16f003/</pre>

Let's break down what this means:
* '''<code>d</code>''': This is a directory (not a file)
* '''<code>rwxrwx---</code>''': The permissions (explained in detail below)
* '''<code>root</code>''': The owner of the directory
* '''<code>bw16f003</code>''': The group that owns the directory (your project group)
* '''<code>33280</code>''': The size in bytes
* '''<code>Jul 25 14:13</code>''': Last modification date
* '''<code>/pfs/10/project/bw16f003/</code>''': The full path

== Understanding Unix Permissions ==

Unix permissions control who can read, write, or execute files and directories. They are displayed as a 10-character string like <code>drwxrwx---</code>.

=== Permission Characters ===

* '''<code>r</code>''' (read): Can view file contents or list directory contents
* '''<code>w</code>''' (write): Can modify files or create/delete files in directories
* '''<code>x</code>''' (execute): Can run files as programs or enter directories

=== Permission Groups ===

Permissions are shown for three groups of users:

# '''Owner''' (positions 2-4): The user who owns the file/directory
# '''Group''' (positions 5-7): Users who belong to the file's group
# '''Others''' (positions 8-10): Everyone else

=== Example Breakdown ===

For <code>drwxrwx---</code>:
* '''<code>d</code>''': Directory
* '''<code>rwx</code>''' (owner): Owner can read, write, and execute
* '''<code>rwx</code>''' (group): Group members can read, write, and execute
* '''<code>---</code>''' (others): Others have no permissions

=== Numeric Permission Values ===

Permissions on files and directories are changed with the tool <code>chmod</code>, which uses numerical values for describing permissions.

{| class="wikitable"
|-
! Number !! Binary !! Permissions !! Description
|-
| 0 || 000 || --- || No permissions
|-
| 1 || 001 || --x || Execute only
|-
| 2 || 010 || -w- || Write only
|-
| 3 || 011 || -wx || Write and execute
|-
| 4 || 100 || r-- || Read only
|-
| 5 || 101 || r-x || Read and execute
|-
| 6 || 110 || rw- || Read and write
|-
| 7 || 111 || rwx || Read, write, and execute
|}

=== Common Permission Patterns ===

{| class="wikitable"
|-
! Pattern !! Use Case !! Description
|-
| <code>700</code> || Private files/directories || Only owner can access
|-
| <code>750</code> || Shared read-only || Owner full access, group read-only
|-
| <code>770</code> || Shared read-write || Owner and group full access
|-
| <code>644</code> || Public readable files || Owner can edit, others can read
|-
| <code>755</code> || Public executable directories || Owner can edit, others can read/execute
|}

== Data Organization Strategies ==

=== Private Directories ===

Create directories that '''only you''' can access:

<pre># Create a private directory
mkdir /pfs/10/project/<project_id>/$USER

# Set permissions so only you can access it
chmod 700 /pfs/10/project/<project_id>/$USER</pre>

'''Permission breakdown for <code>700</code>:'''
* '''Owner''': <code>rwx</code> (7 = 4+2+1) - full access
* '''Group''': <code>---</code> (0) - no access
* '''Others''': <code>---</code> (0) - no access

=== Shared Project Directories ===

Create directories that '''all''' project members can access:

<pre># Create a shared directory
mkdir /pfs/10/project/<project_id>/<your directory name>

# Set permissions for group access
chmod 770 /pfs/10/project/<project_id>/<your directory name></pre>

'''Permission breakdown for <code>770</code>:'''
* '''Owner''': <code>rwx</code> (7) - full access
* '''Group''': <code>rwx</code> (7) - full access for project members
* '''Others''': <code>---</code> (0) - no access

=== Read-Only Shared Directories ===

Create directories where project members can read but not modify:

<pre># Create a read-only shared directory
mkdir /pfs/10/project/<project_id>/<your directory name>

# Set read-only permissions for group
chmod 750 /pfs/10/project/<project_id>/<your directory name></pre>

'''Permission breakdown for <code>750</code>:'''
* '''Owner''': <code>rwx</code> (7) - full access
* '''Group''': <code>r-x</code> (5 = 4+1) - can read and enter, but not write
* '''Others''': <code>---</code> (0) - no access

== Advanced Access Control with ACLs ==

For more fine-grained control beyond basic Unix permissions, you can use Access Control Lists (ACLs). ACLs allow you to set permissions for specific users.

=== Checking Current ACLs ===

<pre># Check if a directory has ACLs
getfacl /pfs/10/project/<project_id>/<your directory name></pre>

=== Setting User-Specific Permissions ===

<pre># Give user 'alice' read and execute permissions
setfacl -m u:alice:rx /pfs/10/project/<project_id>/<your directory name>

# Give user 'bob' full permissions
setfacl -m u:bob:rwx /pfs/10/project/<project_id>/<your directory name></pre>

=== Removing ACLs ===

<pre># Remove ACL for specific user
setfacl -x u:alice /pfs/10/project/<project_id>/<your directory name>

# Remove all ACLs and revert to basic permissions
setfacl -b /pfs/10/project/<project_id>/<your directory name></pre>

=== Default ACLs for New Files ===

Set default ACLs that will be applied to new files created in a directory:

<pre># Set default ACL for new files in directory
setfacl -d -m u:alice:rw /pfs/10/project/<project_id>/<your directory name></pre>

== Practical Examples ==

=== Example 1: Personal Workspace with Shared Results ===

<pre># Create your personal workspace
mkdir /pfs/10/project/<project_id>/user_alice
chmod 700 /pfs/10/project/<project_id>/user_alice

# Create a results directory that others can read
mkdir /pfs/10/project/<project_id>/user_alice/results
chmod 755 /pfs/10/project/<project_id>/user_alice/results</pre>

=== Example 2: Collaborative Analysis Directory ===

<pre># Create collaborative space
mkdir /pfs/10/project/<project_id>/collaborative_analysis
chmod 770 /pfs/10/project/<project_id>/collaborative_analysis

# Create subdirectories for different types of work
mkdir /pfs/10/project/<project_id>/collaborative_analysis/{scripts,data,results}
chmod 770 /pfs/10/project/<project_id>/collaborative_analysis/*</pre>

=== Example 3: Mixed Access with ACLs ===

<pre># Create a directory with complex permissions
mkdir /pfs/10/project/<project_id>/complex_project
chmod 750 /pfs/10/project/<project_id>/complex_project

# Give specific users different levels of access
setfacl -m u:alice:rwx /pfs/10/project/<project_id>/complex_project
setfacl -m u:bob:rx /pfs/10/project/<project_id>/complex_project
setfacl -m g:external_collaborators:r /pfs/10/project/<project_id>/complex_project</pre>

== Best Practices ==

=== Plan Your Directory Structure ===

Before creating directories, plan your organization:
* '''Personal directories''': For work-in-progress and private files
* '''Shared directories''': For collaboration and shared resources

=== Use Descriptive Names ===

Choose clear, descriptive directory names:
* ✗ <code>protein_folding_analysis</code>
* ✗ <code>shared_reference_genomes</code>
* ✗ <code>alice_optimization_scripts</code>
* ✗ <code>stuff</code>
* ✗ <code>temp</code>
* ✗ <code>dir1</code>

=== Set Permissions Appropriately ===

* Start with restrictive permissions and open up as needed
* Use <code>750</code> for directories you want to share read-only
* Use <code>770</code> for full collaboration
* Use ACLs for complex permission requirements

=== Regular Backups ===

'''Important''' The cluster does not provide automatic backups for project directories. You are responsible for backing up your important data.

'''Why Backups Matter:'''

Hardware failures can result in permanent data loss
Human errors (accidental deletion, wrong commands) happen
Corrupted files from failed jobs or system issues
No recovery possible once data is lost from the cluster

=== Document Your Organization ===

Create a README file in your project directory explaining the structure:

<pre># Create documentation
cat > /pfs/10/project/<project_id>/README.md << 'EOF'
# Project <project_id> Directory Structure

## Overview
This project focuses on protein folding simulation and analysis.

## Directory Structure
- `shared_data/`: Reference datasets (read-only for group)
- `collaborative_analysis/`: Shared analysis scripts and results
- `user_alice/`: Alice's personal workspace
- `user_bob/`: Bob's personal workspace
- `archive/`: Completed analysis and backups

## Permissions
- Group members have read access to shared_data/
- All group members can contribute to collaborative_analysis/
- Personal directories are private to each user
EOF</pre>

== Troubleshooting ==

=== Permission Denied Errors ===

If you get "Permission denied" errors:

# Check the permissions:
<pre>ls -la /pfs/10/project/<project_id>/problematic_directory</pre>

# Check if you're in the correct group:
<pre>groups</pre>

# Check ACLs if basic permissions look correct:
<pre>getfacl /pfs/10/project/<project_id>/problematic_directory</pre>

=== Cannot Create Files in Directory ===

This usually means the directory lacks write permissions for your user/group:

<pre># Check directory permissions
ls -ld /pfs/10/project/<project_id>/target_directory

# Fix if you own the directory
chmod g+w /pfs/10/project/<project_id>/target_directory</pre>

=== Accidentally Locked Yourself Out ===

If you accidentally removed your own permissions:

<pre># As the owner, you can always restore permissions
chmod 755 /pfs/10/project/<project_id>/locked_directory</pre>

BinAC2/Hardware and Architecture

2025-09-17T07:16:26Z

F Bartusch:

= Hardware and Architecture =

The bwForCluster BinAC 2 supports researchers from the broader fields of Bioinformatics, Medical Informatics, Astrophysics, Geosciences and Pharmacy.

== Operating System and Software ==

* Operating System: Rocky Linux 9.5
* Queuing System: [https://slurm.schedmd.com/documentation.html Slurm] (see [[BinAC2/Slurm]] for help)
* (Scientific) Libraries and Software: [[Environment Modules]]

== Compute Nodes ==

BinAC 2 offers compute nodes, high-mem nodes, and three types of GPU nodes.
* 180 compute nodes
* 16 SMP node
* 32 GPU nodes (2xA30)
* 8 GPU nodes (4xA100)
* 4 GPU nodes (4xH200)
* plus several special purpose nodes for login, interactive jobs, etc.

Compute node specification:
{| class="wikitable"
|-
! style="width:10%"|
! style="width:10%"| Standard
! style="width:10%"| High-Mem
! style="width:10%"| GPU (A30)
! style="width:10%"| GPU (A100)
! style="width:10%"| GPU (H200)
|-
!scope="column"| Quantity
| 168 / 12
| 14 / 2
| 32
| 8
| 4
|-
!scope="column" | Processors
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7543.html AMD EPYC Milan 7543] / 2 x [https://www.amd.com/en/products/processors/server/epyc/7003-series/amd-epyc-75f3.html AMD EPYC Milan 75F3]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7443.html AMD EPYC Milan 7443] / 2 x [https://www.amd.com/en/products/processors/server/epyc/7003-series/amd-epyc-75f3.html AMD EPYC Milan 75F3]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7543.html AMD EPYC Milan 7543]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7543.html AMD EPYC Milan 7543]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/9005-series/amd-epyc-9555.html AMD EPYC Milan 9555]
|-
!scope="column" | Processor Base Frequency (GHz)
| 2.80 / 2.95
| 2.85 / 2.95
| 2.80
| 2.80
| 3.20
|-
!scope="column" | Number of Physical Cores / Hypertreads
| 64 / 128
| 48 / 96 // 64 / 128
| 64 / 128
| 64 / 128
| 128 / 256
|-
!scope="column" | Working Memory (GB)
| 512
| 2048
| 512
| 512
| 1536
|-
!scope="column" | Local Disk (GiB)
| 450 (NVMe-SSD)
| 14000 (NVMe-SSD)
| 450 (NVMe-SSD)
| 14000 (NVMe-SSD)
| 28000 (NVMe-SSD)
|-
!scope="column" | Interconnect
| HDR 100 IB (84 nodes) / 100GbE (96 nodes)
| 100GbE
| 100GbE
| 100GbE
| HDR 200 IB + 100GbE
|-
!scope="column" | Coprocessors
| -
| -
| 2 x [https://www.nvidia.com/de-de/data-center/products/a30-gpu/ NVIDIA A30 (24 GB ECC HBM2, NVLink)]
| 4 x [https://www.nvidia.com/de-de/data-center/a100/ NVIDIA A100 (80 GB ECC HBM2e)]
| 4 x [https://www.nvidia.com/de-de/data-center/h200/ NVIDIA H200 NVL (141 GB ECC HBM3e, NVLink)]
|}

= Network =

The compute nodes and the parallel file system are connected via 100GbE ethernet
In contrast to BinAC 1 not all compute nodes are connected via Infiniband, but there are 84 standard compute nodes connected via HDR Infiniband (100 GbE). In order to get your jobs onto the Infiniband nodes, submit your job with <code>--constraint=ib</code>.

= File Systems =

The bwForCluster BinAC 2 consists of two separate storage systems, one for the user's home directory $HOME and one serving as a project/work space.
The home directory is limited in space and parallel access but offers snapshots of your files and backup.

The project/work is a parallel file system (PFS) which offers fast and parallel file access and a bigger capacity than the home directory. It is mounted at <code>/pfs/10</code> on the login and compute nodes. This storage is based on Lustre and can be accessed parallel from many nodes. The PFS contains the project and the work directory. Each compute project has its own directory at <code>/pfs/10/project</code> that is accessible for all members of the compute project.
Each user can create workspaces under <code>/pfs/10/work</code> using the workspace tools. These directories are only accessible for the user who created the workspace.

Additionally, each compute node provides high-speed temporary storage (SSD) on the node-local solid state disk via the $TMPDIR environment variable.

{| class="wikitable"
|-
! style="width:10%"|
! style="width:10%"| <tt>$HOME</tt>
! style="width:10%"| project
! style="width:10%"| work
! style="width:10%"| <tt>$TMPDIR</tt>
|-
!scope="column" | Visibility
| global
| global
| global
| node local
|-
!scope="column" | Lifetime
| permanent
| permanent
| work space lifetime (max. 30 days, max. 5 extensions)
| batch job walltime
|-
!scope="column" | Capacity
| -
| 8.1 PB
| 1000 TB
| 480 GB (compute nodes); 7.7 TB (GPU-A30 nodes); 16 TB (GPU-A100 and SMP nodes); 31 TB (GPU-H200 nodes)
|-
!scope="column" | Speed (read)
| ≈ 1 GB/s, shared by all nodes
| max. 12 GB/s
| ≈ 145 GB/s peak, aggregated over 56 nodes, ideal striping
| ≈ 3 GB/s (compute)/ ≈5 GB/S (GPUA-30)/ ≈ 26 GB/s (GPU-A100 + SMP)/ ≈ 42 GB/s (GPU-H200) per node
|-
!scope="column" | [https://en.wikipedia.org/wiki/Disk_quota#Quotas Quotas]
| 40 GB per user
| not yet, maybe in the future
| none
| none
|-
!scope="column" | Backup
| yes (nightly)
| '''no'''
| '''no'''
| '''no'''
|}

global : all nodes access the same file system
local : each node has its own file system
permanent : files are stored permanently
batch job walltime : files are removed at end of the batch job

{| class="wikitable" style="color:red; background-color:#ffffcc;" cellpadding="10"
|
Please note that due to the large capacity of '''work''' and '''project''' and due to frequent file changes on these file systems, no backup can be provided.
Backing up these file systems would require a redundant storage facility with multiple times the capacity of '''project'''. Furthermore, regular backups would significantly degrade the performance.
Data is stored redundantly, i.e. immune against disk failures but not immune against catastrophic incidents like cyber attacks or a fire in the server room.
Please consider to use on of the remote storage facilities like [https://wiki.bwhpc.de/e/SDS@hd SDS@hd], [https://uni-tuebingen.de/einrichtungen/zentrum-fuer-datenverarbeitung/projekte/laufende-projekte/bwsfs bwSFS], [https://www.scc.kit.edu/en/services/lsdf.php LSFD Online Storage] or the [https://www.rda.kit.edu/english/ bwDataArchive] to back up your valuable data.
|}

=== Home ===

Home directories are meant for permanent file storage of files that are keep being used like source codes, configuration files, executable programs etc.; the content of home directories will be backed up on a regular basis.
Because the backup space is limited we enforce a quota of 40GB on the home directories.

'''NOTE:'''
Compute jobs on nodes must not write temporary data to $HOME.
Instead they should use the local $TMPDIR directory for I/O-heavy use cases
and work spaces for less I/O intense multinode-jobs.


=== Project ===

The data is stored on HDDs. The primary focus of <code>/pfs/10/project</code> is pure capacity, not speed.

Every project gets a dedicated directory located at:

<syntaxhighlight>
/pfs/10/project/<project_id>/
</syntaxhighlight>

You can check the project you're member of:

<syntaxhighlight>
# id $USER | grep -o 'bw[^)]*'
bw16f003
</syntaxhighlight>

In this case, your project directory would be:
```
/pfs/10/project/bw16f003/
```

Check our [[BinAC2/Project_Data_Organization | data organization guide ]] for methods to organize data inside the project directory.

=== Workspaces ===

Data on the fast storage pool at <code>/pfs/10/work</code> is stored on SSDs.
The primary focus is speed, not capacity.

In contrast to BinAC 1 we will enforce work space lifetime, as the capacity is limited.
We ask you to only store data you actively use for computations on <code>/pfs/10/work</code>.
Please move data to <code>/pfs/10/project</code> when you don't need it on the fast storage any more.

Each user should create workspaces at <code>/pfs/10/work</code> through the workspace tools

You can find more info on workspace tools on our general page:

:: → '''[[Workspace]]s'''

To create a work space you'll need to supply a name for your work space area and a lifetime in days.
For more information read the corresponding help, e.g: <code>ws_allocate -h.</code>
{| class="wikitable"
|-
!style="width:30%" | Command
!style="width:70%" | Action
|-
|<code>ws_allocate mywork 30</code>
|Allocate a work space named "mywork" for 30 days.
|-
|<code>ws_allocate myotherwork</code>
|Allocate a work space named "myotherwork" with maximum lifetime.
|-
|<code>ws_list -a</code>
|List all your work spaces.
|-
|<code>ws_find mywork</code>
|Get absolute path of work space "mywork".
|-
|<code>ws_extend mywork 30</code>
|Extend life me of work space mywork by 30 days from now.
|-
|<code>ws_release mywork</code>
|Manually erase your work space "mywork". Please remove directory content first.
|-
|}

=== Scratch ===

Please use the fast local scratch space for storing temporary data during your jobs.

For each job a scratch directory will be created on the compute nodes. It is available via the environment variable <code>$TMPDIR</code>, which points to <code>/scratch/<jobID></code>.

Especially the SMP nodes and the GPU nodes are equipped with large and fast local disks that should be used for temporary data, scratch data or data staging for ML model training.
The Lustre file system (<code>WORK</code> and <code>PROJECT</code>) is unsuited for repetitive random I/O, I/O sizes smaller than the Lustre and ZFS block size (1M) or I/O patterns where files are opened and closed in rapid succession. The XFS file system of the local scratch drives is better suited for typical scratch workloads and access patterns. Moreover, the local scratch drives offer a lower latency and a higher bandwidth than <code>WORK</code>.

=== SDS@hd ===

SDS@hd is mounted via NFS on login and compute nodes at <syntaxhighlight inline>/mnt/sds-hd</syntaxhighlight>.

To access your Speichervorhaben, the export to BinAC 2 must first be enabled by the SDS@hd-Team. Please contact [mailto:sds-hd-support@urz.uni-heidelberg.de SDS@hd support] and provide the acronym of your Speichervorhaben, along with a request to enable the export to BinAC 2.

Once this has been done, you can access your Speichervorhaben as described in the [https://wiki.bwhpc.de/e/SDS@hd/Access/NFS#Access_your_data SDS documentation].

<syntaxhighlight>
$ kinit $USER
Password for <user>@BWSERVICES.UNI-HEIDELBERG.DE:
</syntaxhighlight>

The Kerberos ticket store is shared across all nodes. Creating a single ticket is sufficient to access your Speichervorhaben on all nodes.

BinAC2/Hardware and Architecture

2025-09-17T06:27:45Z

F Bartusch:

= Hardware and Architecture =

The bwForCluster BinAC 2 supports researchers from the broader fields of Bioinformatics, Medical Informatics, Astrophysics, Geosciences and Pharmacy.

== Operating System and Software ==

* Operating System: Rocky Linux 9.5
* Queuing System: [https://slurm.schedmd.com/documentation.html Slurm] (see [[BinAC2/Slurm]] for help)
* (Scientific) Libraries and Software: [[Environment Modules]]

== Compute Nodes ==

BinAC 2 offers compute nodes, high-mem nodes, and three types of GPU nodes.
* 180 compute nodes
* 16 SMP node
* 32 GPU nodes (2xA30)
* 8 GPU nodes (4xA100)
* 4 GPU nodes (4xH200)
* plus several special purpose nodes for login, interactive jobs, etc.

Compute node specification:
{| class="wikitable"
|-
! style="width:10%"|
! style="width:10%"| Standard
! style="width:10%"| High-Mem
! style="width:10%"| GPU (A30)
! style="width:10%"| GPU (A100)
! style="width:10%"| GPU (H200)
|-
!scope="column"| Quantity
| 168 / 12
| 14 / 2
| 32
| 8
| 4
|-
!scope="column" | Processors
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7543.html AMD EPYC Milan 7543] / 2 x [https://www.amd.com/en/products/processors/server/epyc/7003-series/amd-epyc-75f3.html AMD EPYC Milan 75F3]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7443.html AMD EPYC Milan 7443] / 2 x [https://www.amd.com/en/products/processors/server/epyc/7003-series/amd-epyc-75f3.html AMD EPYC Milan 75F3]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7543.html AMD EPYC Milan 7543]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7543.html AMD EPYC Milan 7543]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/9005-series/amd-epyc-9555.html AMD EPYC Milan 9555]
|-
!scope="column" | Processor Base Frequency (GHz)
| 2.80 / 2.95
| 2.85 / 2.95
| 2.80
| 2.80
| 3.20
|-
!scope="column" | Number of Physical Cores / Hypertreads
| 64 / 128
| 48 / 96 // 64 / 128
| 64 / 128
| 64 / 128
| 128 / 256
|-
!scope="column" | Working Memory (GB)
| 512
| 2048
| 512
| 512
| 1536
|-
!scope="column" | Local Disk (GiB)
| 450 (NVMe-SSD)
| 14000 (NVMe-SSD)
| 450 (NVMe-SSD)
| 14000 (NVMe-SSD)
| 28000 (NVMe-SSD)
|-
!scope="column" | Interconnect
| HDR 100 IB (84 nodes) / 100GbE (96 nodes)
| 100GbE
| 100GbE
| 100GbE
| HDR 200 IB + 100GbE
|-
!scope="column" | Coprocessors
| -
| -
| 2 x [https://www.nvidia.com/de-de/data-center/products/a30-gpu/ NVIDIA A30 (24 GB ECC HBM2, NVLink)]
| 4 x [https://www.nvidia.com/de-de/data-center/a100/ NVIDIA A100 (80 GB ECC HBM2e)]
| 4 x [https://www.nvidia.com/de-de/data-center/h200/ NVIDIA H200 NVL (141 GB ECC HBM3e, NVLink)]
|}

= Network =

The compute nodes and the parallel file system are connected via 100GbE ethernet
In contrast to BinAC 1 not all compute nodes are connected via Infiniband, but there are 84 standard compute nodes connected via HDR Infiniband (100 GbE). In order to get your jobs onto the Infiniband nodes, submit your job with <code>--constraint=ib</code>.

= File Systems =

The bwForCluster BinAC 2 consists of two separate storage systems, one for the user's home directory $HOME and one serving as a project/work space.
The home directory is limited in space and parallel access but offers snapshots of your files and backup.

The project/work is a parallel file system (PFS) which offers fast and parallel file access and a bigger capacity than the home directory. It is mounted at <code>/pfs/10</code> on the login and compute nodes. This storage is based on Lustre and can be accessed parallel from many nodes. The PFS contains the project and the work directory. Each compute project has its own directory at <code>/pfs/10/project</code> that is accessible for all members of the compute project.
Each user can create workspaces under <code>/pfs/10/work</code> using the workspace tools. These directories are only accessible for the user who created the workspace.

Additionally, each compute node provides high-speed temporary storage (SSD) on the node-local solid state disk via the $TMPDIR environment variable.

{| class="wikitable"
|-
! style="width:10%"|
! style="width:10%"| <tt>$HOME</tt>
! style="width:10%"| project
! style="width:10%"| work
! style="width:10%"| <tt>$TMPDIR</tt>
|-
!scope="column" | Visibility
| global
| global
| global
| node local
|-
!scope="column" | Lifetime
| permanent
| permanent
| work space lifetime (max. 30 days, max. 5 extensions)
| batch job walltime
|-
!scope="column" | Capacity
| -
| 8.1 PB
| 1000 TB
| 480 GB (compute nodes); 7.7 TB (GPU-A30 nodes); 16 TB (GPU-A100 and SMP nodes); 31 TB (GPU-H200 nodes)
|-
!scope="column" | Speed (read)
| ≈ 1 GB/s, shared by all nodes
| max. 12 GB/s
| ≈ 145 GB/s peak, aggregated over 56 nodes, ideal striping
| ≈ 3 GB/s (compute)/ ≈5 GB/S (GPUA-30)/ ≈ 26 GB/s (GPU-A100 + SMP)/ ≈ 42 GB/s (GPU-H200) per node
|-
!scope="column" | [https://en.wikipedia.org/wiki/Disk_quota#Quotas Quotas]
| 40 GB per user
| not yet, maybe in the future
| none
| none
|-
!scope="column" | Backup
| yes (nightly)
| '''no'''
| '''no'''
| '''no'''
|}

global : all nodes access the same file system
local : each node has its own file system
permanent : files are stored permanently
batch job walltime : files are removed at end of the batch job

{| class="wikitable" style="color:red; background-color:#ffffcc;" cellpadding="10"
|
Please note that due to the large capacity of '''work''' and '''project''' and due to frequent file changes on these file systems, no backup can be provided.
Backing up these file systems would require a redundant storage facility with multiple times the capacity of '''project'''. Furthermore, regular backups would significantly degrade the performance.
Data is stored redundantly, i.e. immune against disk failures but not immune against catastrophic incidents like cyber attacks or a fire in the server room.
Please consider to use on of the remote storage facilities like [https://wiki.bwhpc.de/e/SDS@hd SDS@hd], [https://uni-tuebingen.de/einrichtungen/zentrum-fuer-datenverarbeitung/projekte/laufende-projekte/bwsfs bwSFS], [https://www.scc.kit.edu/en/services/lsdf.php LSFD Online Storage] or the [https://www.rda.kit.edu/english/ bwDataArchive] to back up your valuable data.
|}

=== Home ===

Home directories are meant for permanent file storage of files that are keep being used like source codes, configuration files, executable programs etc.; the content of home directories will be backed up on a regular basis.
Because the backup space is limited we enforce a quota of 40GB on the home directories.

'''NOTE:'''
Compute jobs on nodes must not write temporary data to $HOME.
Instead they should use the local $TMPDIR directory for I/O-heavy use cases
and work spaces for less I/O intense multinode-jobs.


=== Project ===

The data is stored on HDDs. The primary focus of <code>/pfs/10/project</code> is pure capacity, not speed.

Every project gets a dedicated directory located at:

<syntaxhighlight>
/pfs/10/project/<project_id>/
</syntaxhighlight>

You can check the project you're member of:

<syntaxhighlight>
# id $USER | grep -o 'bw[^)]*'
bw16f003
</syntaxhighlight>

In this case, your project directory would be:
```
/pfs/10/project/bw16f003/
```

Check our [[BinAC2/Data_Organization | data organization guide ]] for methods to organize data inside the project directory.

=== Workspaces ===

Data on the fast storage pool at <code>/pfs/10/work</code> is stored on SSDs.
The primary focus is speed, not capacity.

In contrast to BinAC 1 we will enforce work space lifetime, as the capacity is limited.
We ask you to only store data you actively use for computations on <code>/pfs/10/work</code>.
Please move data to <code>/pfs/10/project</code> when you don't need it on the fast storage any more.

Each user should create workspaces at <code>/pfs/10/work</code> through the workspace tools

You can find more info on workspace tools on our general page:

:: → '''[[Workspace]]s'''

To create a work space you'll need to supply a name for your work space area and a lifetime in days.
For more information read the corresponding help, e.g: <code>ws_allocate -h.</code>
{| class="wikitable"
|-
!style="width:30%" | Command
!style="width:70%" | Action
|-
|<code>ws_allocate mywork 30</code>
|Allocate a work space named "mywork" for 30 days.
|-
|<code>ws_allocate myotherwork</code>
|Allocate a work space named "myotherwork" with maximum lifetime.
|-
|<code>ws_list -a</code>
|List all your work spaces.
|-
|<code>ws_find mywork</code>
|Get absolute path of work space "mywork".
|-
|<code>ws_extend mywork 30</code>
|Extend life me of work space mywork by 30 days from now.
|-
|<code>ws_release mywork</code>
|Manually erase your work space "mywork". Please remove directory content first.
|-
|}

=== Scratch ===

Please use the fast local scratch space for storing temporary data during your jobs.

For each job a scratch directory will be created on the compute nodes. It is available via the environment variable <code>$TMPDIR</code>, which points to <code>/scratch/<jobID></code>.

Especially the SMP nodes and the GPU nodes are equipped with large and fast local disks that should be used for temporary data, scratch data or data staging for ML model training.
The Lustre file system (<code>WORK</code> and <code>PROJECT</code>) is unsuited for repetitive random I/O, I/O sizes smaller than the Lustre and ZFS block size (1M) or I/O patterns where files are opened and closed in rapid succession. The XFS file system of the local scratch drives is better suited for typical scratch workloads and access patterns. Moreover, the local scratch drives offer a lower latency and a higher bandwidth than <code>WORK</code>.

=== SDS@hd ===

SDS@hd is mounted via NFS on login and compute nodes at <syntaxhighlight inline>/mnt/sds-hd</syntaxhighlight>.

To access your Speichervorhaben, the export to BinAC 2 must first be enabled by the SDS@hd-Team. Please contact [mailto:sds-hd-support@urz.uni-heidelberg.de SDS@hd support] and provide the acronym of your Speichervorhaben, along with a request to enable the export to BinAC 2.

Once this has been done, you can access your Speichervorhaben as described in the [https://wiki.bwhpc.de/e/SDS@hd/Access/NFS#Access_your_data SDS documentation].

<syntaxhighlight>
$ kinit $USER
Password for <user>@BWSERVICES.UNI-HEIDELBERG.DE:
</syntaxhighlight>

The Kerberos ticket store is shared across all nodes. Creating a single ticket is sufficient to access your Speichervorhaben on all nodes.

BinAC2/Hardware and Architecture

2025-09-09T09:22:47Z

F Bartusch:

= Hardware and Architecture =

The bwForCluster BinAC 2 supports researchers from the broader fields of Bioinformatics, Medical Informatics, Astrophysics, Geosciences and Pharmacy.

== Operating System and Software ==

* Operating System: Rocky Linux 9.5
* Queuing System: [https://slurm.schedmd.com/documentation.html Slurm] (see [[BinAC2/Slurm]] for help)
* (Scientific) Libraries and Software: [[Environment Modules]]

== Compute Nodes ==

BinAC 2 offers compute nodes, high-mem nodes, and three types of GPU nodes.
* 180 compute nodes
* 16 SMP node
* 32 GPU nodes (2xA30)
* 8 GPU nodes (4xA100)
* 4 GPU nodes (4xH200)
* plus several special purpose nodes for login, interactive jobs, etc.

Compute node specification:
{| class="wikitable"
|-
! style="width:10%"|
! style="width:10%"| Standard
! style="width:10%"| High-Mem
! style="width:10%"| GPU (A30)
! style="width:10%"| GPU (A100)
! style="width:10%"| GPU (H200)
|-
!scope="column"| Quantity
| 180
| 14 / 2
| 32
| 8
| 4
|-
!scope="column" | Processors
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7543.html AMD EPYC Milan 7543]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7443.html AMD EPYC Milan 7443] / 2 x [https://www.amd.com/en/products/processors/server/epyc/7003-series/amd-epyc-75f3.html AMD EPYC Milan 75F3]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7543.html AMD EPYC Milan 7543]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/7003-series/amd-epyc-7543.html AMD EPYC Milan 7543]
| 2 x [https://www.amd.com/de/products/processors/server/epyc/9005-series/amd-epyc-9555.html AMD EPYC Milan 9555]
|-
!scope="column" | Processor Base Frequency (GHz)
| 2.80
| 2.85 / 2.95
| 2.80
| 2.80
| 3.20
|-
!scope="column" | Number of Physical Cores / Hypertreads
| 64 / 128
| 48 / 96 // 64 / 128
| 64 / 128
| 64 / 128
| 128 / 256
|-
!scope="column" | Working Memory (GB)
| 512
| 2048
| 512
| 512
| 1536
|-
!scope="column" | Local Disk (GiB)
| 450 (NVMe-SSD)
| 14000 (NVMe-SSD)
| 450 (NVMe-SSD)
| 14000 (NVMe-SSD)
| 28000 (NVMe-SSD)
|-
!scope="column" | Interconnect
| HDR 100 IB (84 nodes) / 100GbE (96 nodes)
| 100GbE
| 100GbE
| 100GbE
| HDR 200 IB + 100GbE
|-
!scope="column" | Coprocessors
| -
| -
| 2 x [https://www.nvidia.com/de-de/data-center/products/a30-gpu/ NVIDIA A30 (24 GB ECC HBM2, NVLink)]
| 4 x [https://www.nvidia.com/de-de/data-center/a100/ NVIDIA A100 (80 GB ECC HBM2e)]
| 4 x [https://www.nvidia.com/de-de/data-center/h200/ NVIDIA H200 NVL (141 GB ECC HBM3e, NVLink)]
|}

= Network =

The compute nodes and the parallel file system are connected via 100GbE ethernet
In contrast to BinAC 1 not all compute nodes are connected via Infiniband, but there are 84 standard compute nodes connected via HDR Infiniband (100 GbE). In order to get your jobs onto the Infiniband nodes, submit your job with <code>--constraint=ib</code>.

= File Systems =

The bwForCluster BinAC 2 consists of two separate storage systems, one for the user's home directory $HOME and one serving as a project/work space.
The home directory is limited in space and parallel access but offers snapshots of your files and backup.

The project/work is a parallel file system (PFS) which offers fast and parallel file access and a bigger capacity than the home directory. It is mounted at <code>/pfs/10</code> on the login and compute nodes. This storage is based on Lustre and can be accessed parallel from many nodes. The PFS contains the project and the work directory. Each compute project has its own directory at <code>/pfs/10/project</code> that is accessible for all members of the compute project.
Each user can create workspaces under <code>/pfs/10/work</code> using the workspace tools. These directories are only accessible for the user who created the workspace.

Additionally, each compute node provides high-speed temporary storage (SSD) on the node-local solid state disk via the $TMPDIR environment variable.

{| class="wikitable"
|-
! style="width:10%"|
! style="width:10%"| <tt>$HOME</tt>
! style="width:10%"| project
! style="width:10%"| work
! style="width:10%"| <tt>$TMPDIR</tt>
|-
!scope="column" | Visibility
| global
| global
| global
| node local
|-
!scope="column" | Lifetime
| permanent
| permanent
| work space lifetime (max. 30 days, max. 5 extensions)
| batch job walltime
|-
!scope="column" | Capacity
| -
| 8.1 PB
| 1000 TB
| 480 GB (compute nodes); 7.7 TB (GPU-A30 nodes); 16 TB (GPU-A100 and SMP nodes); 31 TB (GPU-H200 nodes)
|-
!scope="column" | Speed (read)
| ≈ 1 GB/s, shared by all nodes
| max. 12 GB/s
| ≈ 145 GB/s peak, aggregated over 56 nodes, ideal striping
| ≈ 3 GB/s (compute)/ ≈5 GB/S (GPUA-30)/ ≈ 26 GB/s (GPU-A100 + SMP)/ ≈ 42 GB/s (GPU-H200) per node
|-
!scope="column" | [https://en.wikipedia.org/wiki/Disk_quota#Quotas Quotas]
| 40 GB per user
| not yet, maybe in the future
| none
| none
|-
!scope="column" | Backup
| yes (nightly)
| '''no'''
| '''no'''
| '''no'''
|}

global : all nodes access the same file system
local : each node has its own file system
permanent : files are stored permanently
batch job walltime : files are removed at end of the batch job

{| class="wikitable" style="color:red; background-color:#ffffcc;" cellpadding="10"
|
Please note that due to the large capacity of '''work''' and '''project''' and due to frequent file changes on these file systems, no backup can be provided.
Backing up these file systems would require a redundant storage facility with multiple times the capacity of '''project'''. Furthermore, regular backups would significantly degrade the performance.
Data is stored redundantly, i.e. immune against disk failures but not immune against catastrophic incidents like cyber attacks or a fire in the server room.
Please consider to use on of the remote storage facilities like [https://wiki.bwhpc.de/e/SDS@hd SDS@hd], [https://uni-tuebingen.de/einrichtungen/zentrum-fuer-datenverarbeitung/projekte/laufende-projekte/bwsfs bwSFS], [https://www.scc.kit.edu/en/services/lsdf.php LSFD Online Storage] or the [https://www.rda.kit.edu/english/ bwDataArchive] to back up your valuable data.
|}

=== Home ===

Home directories are meant for permanent file storage of files that are keep being used like source codes, configuration files, executable programs etc.; the content of home directories will be backed up on a regular basis.
Because the backup space is limited we enforce a quota of 40GB on the home directories.

'''NOTE:'''
Compute jobs on nodes must not write temporary data to $HOME.
Instead they should use the local $TMPDIR directory for I/O-heavy use cases
and work spaces for less I/O intense multinode-jobs.


=== Project ===

Each compute project has its own project directory at <code>/pfs/10/project</code>.

<pre>
$ ls -lh /pfs/10/project/
drwxrwx---. 2 root bw16f003 33K Dec 12 16:46 bw16f003
[...]
</pre>

As you can see the directory is owned by a group representing your compute project (here bw16f003) and the directory is accessible by all group members. It is upon your group to decide how to use the space inside this directory: shared data folders, personal directories for each project member, software containers, etc.

The data is stored on HDDs. The primary focus of <code>/pfs/10/project</code> is pure capacity, not speed.

=== Workspaces ===

Data on the fast storage pool at <code>/pfs/10/work</code> is stored on SSDs.
The primary focus is speed, not capacity.

In contrast to BinAC 1 we will enforce work space lifetime, as the capacity is limited.
We ask you to only store data you actively use for computations on <code>/pfs/10/work</code>.
Please move data to <code>/pfs/10/project</code> when you don't need it on the fast storage any more.

Each user should create workspaces at <code>/pfs/10/work</code> through the workspace tools

You can find more info on workspace tools on our general page:

:: → '''[[Workspace]]s'''

To create a work space you'll need to supply a name for your work space area and a lifetime in days.
For more information read the corresponding help, e.g: <code>ws_allocate -h.</code>
{| class="wikitable"
|-
!style="width:30%" | Command
!style="width:70%" | Action
|-
|<code>ws_allocate mywork 30</code>
|Allocate a work space named "mywork" for 30 days.
|-
|<code>ws_allocate myotherwork</code>
|Allocate a work space named "myotherwork" with maximum lifetime.
|-
|<code>ws_list -a</code>
|List all your work spaces.
|-
|<code>ws_find mywork</code>
|Get absolute path of work space "mywork".
|-
|<code>ws_extend mywork 30</code>
|Extend life me of work space mywork by 30 days from now.
|-
|<code>ws_release mywork</code>
|Manually erase your work space "mywork". Please remove directory content first.
|-
|}

=== Scratch ===

Please use the fast local scratch space for storing temporary data during your jobs.

For each job a scratch directory will be created on the compute nodes. It is available via the environment variable <code>$TMPDIR</code>, which points to <code>/scratch/<jobID></code>.

Especially the SMP nodes and the GPU nodes are equipped with large and fast local disks that should be used for temporary data, scratch data or data staging for ML model training.
The Lustre file system (<code>WORK</code> and <code>PROJECT</code>) is unsuited for repetitive random I/O, I/O sizes smaller than the Lustre and ZFS block size (1M) or I/O patterns where files are opened and closed in rapid succession. The XFS file system of the local scratch drives is better suited for typical scratch workloads and access patterns. Moreover, the local scratch drives offer a lower latency and a higher bandwidth than <code>WORK</code>.

=== SDS@hd ===

SDS@hd is mounted via NFS on login and compute nodes at <syntaxhighlight inline>/mnt/sds-hd</syntaxhighlight>.

To access your Speichervorhaben, the export to BinAC 2 must first be enabled by the SDS@hd-Team. Please contact [mailto:sds-hd-support@urz.uni-heidelberg.de SDS@hd support] and provide the acronym of your Speichervorhaben, along with a request to enable the export to BinAC 2.

Once this has been done, you can access your Speichervorhaben as described in the [https://wiki.bwhpc.de/e/SDS@hd/Access/NFS#Access_your_data SDS documentation].

<syntaxhighlight>
$ kinit $USER
Password for <user>@BWSERVICES.UNI-HEIDELBERG.DE:
</syntaxhighlight>

The Kerberos ticket store is shared across all nodes. Creating a single ticket is sufficient to access your Speichervorhaben on all nodes.

BinAC2

2025-09-09T09:17:41Z

F Bartusch:

{| style="width: 100%; border-spacing: 5px;"
| style="text-align:center; color:#000;vertical-align:middle;font-size:75%;" |
[[File:BinAC2_Logo_RGB_subtitel.svg|center|500px||]]
|-
|
|}

The '''bwForCluster BinAC 2''' supports researchers from the broader fields of Bioinformatics, Astrophysics, Geosciences, Pharmacy, and Medical Informatics.

{| style=" background:#ffa833; width:100%;"
| style="padding:8px; background:#f80; font-size:120%; font-weight:bold; text-align:left" | BinAC 1 -> BinAC 2 Migration
|-
|
* Still using BinAC 1: '''[[BinAC|Go To BinAC 1 Legacy Wiki]]'''
* '''Users of BinAC 1 must''' [[Registration/bwForCluster/BinAC2|'''re-register for BinAC 2''']] (only step C necessary). You keep your existing projects (Rechenvorhaben).
* '''[[BinAC2/Migrate BinAC 1 workspaces to BinAC 2 workspaces|Migrate BinAC 1 workspaces to BinAC 2 workspaces]]'''
* '''[[BinAC2/Migrate Moab to Slurm jobs|Migrate Moab/Torque to Slurm jobs]]'''
|}

{| style=" background:#FEF4AB; width:100%;"
| style="padding:8px; background:#FFE856; font-size:120%; font-weight:bold; text-align:left" | News and Events
|-
|
* [https://uni-tuebingen.de/de/274923|11th bwHPC Symposium - September 23rd, Tübingen]

|}

{| style=" background:#eeeefe; width:100%;"
| style="padding:8px; background:#dedefe; font-size:120%; font-weight:bold; text-align:left" | Training & Support
|-
|
* [[BinAC2/Getting_Started|Getting Started]]
* [https://training.bwhpc.de E-Learning Courses]
* [[BinAC2/Support|Contact and Support]]
* Send [[Feedback|Feedback]] about Wiki pages
|}

{| style=" background:#deffee; width:100%;"
| style="padding:8px; background:#cef2e0; font-size:120%; font-weight:bold; text-align:left" | User Documentation
|-
|
* [[Registration/bwForCluster|Registration]]

* [[BinAC2/Login|Login]]

* [[BinAC2/Hardware_and_Architecture|Hardware and Architecture]]
** [[BinAC2/Hardware_and_Architecture#Compute_Nodes|Node Specifications]]
** [[BinAC2/Hardware_and_Architecture#File_Systems|File Systems and Workspaces]]
* Usage of [[BinAC2/Software|Software]] on BinAC 2
** For available Software Modules see [https://www.bwhpc.de/software.php bwhpc.de Software Search] (select bwForCluster BinAC 2)
** Create Software Environments with [[Development/Conda|Conda]]
** Use [[BinAC2/Software/Nextflow|nf-core Nextflow pipelines]]
** See [[Development]] for info about compiler and parallelization

* [[BinAC2/Slurm|Batch System (SLURM)]]
** [[BinAC2/SLURM_Partitions|SLURM Partitions]]

|}
{| style=" background:#e6e9eb; width:100%;"
| style="padding:8px; background:#d1dadf; font-size:120%; font-weight:bold; text-align:left" | Cluster Funding
|-
|
* Please [[BinAC2/Acknowledgement|acknowledge]] the cluster in your publications.
|}

BinAC2/Slurm

2025-08-26T12:40:05Z

F Bartusch: /* Interactive Jobs : srun / salloc */

= General information about Slurm =

Any kind of calculation on the compute nodes of bwForCluster BinAC 2 requires the user to define calculations as a sequence of commands or single command together with required run time, number of CPU cores and main memory and submit all, i.e., the '''batch job''', to a resource and workload managing software. BinAC 2 has installed the workload managing software Slurm. Therefore any job submission by the user is to be executed by commands of the Slurm software. Slurm queues and runs user jobs based on fair sharing policies.

= External Slurm documentation =

You can find the official Slurm configuration and some other material here:

* Slurm documentation: https://slurm.schedmd.com/documentation.html
* Slurm cheat sheet: https://slurm.schedmd.com/pdfs/summary.pdf
* Slurm tutorials: https://slurm.schedmd.com/tutorials.html

= SLURM terminology =

SLURM knows and mirrors the division of the cluster into '''nodes''' with several '''cores'''. When queuing '''jobs''', there are several ways of requesting resources and it is important to know which term means what in SLURM. Here are some basic SLURM terms:

;Job
: A job is a self-contained computation that may encompass multiple tasks and is given specific resources like individual CPUs/GPUs, a specific amount of RAM or entire nodes. These resources are said to have been allocated for the job.

;Task
: A task is a single run of a single process. By default, one task is run per node and one CPU is assigned per task.

;Partition
: A partition (usually called queue outside SLURM) is a waiting line in which jobs are put by users.

;Socket
: Receptacle on the motherboard for one physically packaged processor (each of which can contain one or more cores).

;Core
: A complete private set of registers, execution units, and retirement queues needed to execute programs.

;Thread
: One or more hardware contexts withing a single core. Each thread has attributes of one core, managed & scheduled as a single logical processor by the OS.

;CPU
: A '''CPU''' in Slurm means a '''single core'''. This is different from the more common terminology, where a CPU (a microprocessor chip) consists of multiple cores. Slurm uses the term '''sockets''' when talking about CPU chips. Depending upon system configuration, a CPU can be either a '''core''' or a '''thread'''. On '''BinAC 2 Hyperthreading is activated on every machine'''. This means that the operating system and Slurm sees each physical core as two logical cores.

= Slurm Commands =

{| width=750px class="wikitable"
! Slurm commands !! Brief explanation
|-
| [https://slurm.schedmd.com/sbatch.html sbatch] || Submits a job and queues it in an input queue
|-
| [https://slurm.schedmd.com/salloc.html saclloc] || Request resources for an interactive job
|-
| [https://slurm.schedmd.com/squeue.html squeue] || Displays information about active, eligible, blocked, and/or recently completed jobs
|-
| [https://slurm.schedmd.com/scontrol.html scontrol] || Displays detailed job state information
|-
| [https://slurm.schedmd.com/scontrol.html sstat] || Displays status information about a running job
|-
| [https://slurm.schedmd.com/scancel.html scancel] || Cancels a job
|-
|}

== Interactive Jobs ==

You can run interactive jobs for testing and developing your job scripts.
Several nodes are reserved for interactive work, so your jobs should start right away.
You can only submit one job to this partition at a time. A job can run for up to 10 hours (about one workday).

This example command gives you 16 cores and 128 GB of memory for four hours on one of the reserved nodes:

<pre>
salloc --partition=interactive --time=4:00:00 --cpus-per-task=16 --mem=128gb
</pre>

You can also use srun to request the same resources:

<pre>
srun --partition=interactive --time=4:00:00 --cpus-per-task=16 --mem=128gb --pty bash
</pre>

== Job Submission : sbatch ==

Batch jobs are submitted by using the command '''sbatch'''. The main purpose of the '''sbatch''' command is to specify the resources that are needed to run the job. '''sbatch''' will then queue the batch job. However, starting of batch job depends on the availability of the requested resources and the fair sharing value.
 
=== sbatch Command Parameters ===
The syntax and use of '''sbatch''' can be displayed via:
<pre>
$ man sbatch
</pre>
'''sbatch''' options can be used from the command line or in your job script. The following table shows the syntax and provides examples for each option.

{| class="wikitable"
! colspan="5" | sbatch Options
|-
! Command line
! Job Script
! Purpose
! Example
! Default value
|- style="vertical-align:top;"
| <code>-t ''time''</code> or <code>--time=''time''</code>
| #SBATCH --time=''time''
| Wall clock time limit. 
| <code>-t 2:30:00</code> Limits run time to 2h 30 min.<code>-t 2-12</code> Limits run time to 2 days and 12 hours.
| Depends on Slurm partition.
|-
|- style="vertical-align:top;"
| -N ''count'' or --nodes=''count''
| #SBATCH --nodes=''count''
| Number of nodes to be used.
| <code>-N 1</code> Run job on one node.<code>-N 2</code> Run job on two nodes (have to use MPI!)
|
|- style="vertical-align:top;"
| -n ''count'' or --ntasks=''count''
| #SBATCH --ntasks=''count''
| Number of tasks to be launched.
| <code>-n 2</code> launch two tasks in the job.
| One task per node
|-
|- style="vertical-align:top;"
| --ntasks-per-node=''count''
| #SBATCH --ntasks-per-node=''count''
| Maximum count of tasks per node. (Replaces the option <code>ppn</code> of MOAB.)
| <code>--ntasks-per-node=2</code> Run 2 tasks per node
| 1 task per node
|-
|- style="vertical-align:top;"
| -c ''count'' or --cpus-per-task=''count''
| #SBATCH --cpus-per-task=''count''
| Number of CPUs required per (MPI-)task.
| <code>-c 2</code> Request two CPUs per (MPI-)task.
| 1 CPU per (MPI-)task
|-
|- style="vertical-align:top;"
| <code>--mem=<size>[units]</code>
| #SBATCH --mem=''value_in_MB''
| Memory in MegaByte per node.<code>[units]</code> can be one of <code>[K<nowiki>|</nowiki>M<nowiki>|</nowiki>G<nowiki>|</nowiki>T]</code>.
| <code>--mem=10g</code> Request 10GB RAM per node <code>--mem=0</code> Request all memory on node
| Depends on Slurm configuration.It is better to specify <code>--mem</code> in every case.
|-
|- style="vertical-align:top;"
| --mem-per-cpu=''value_in_MB''
| #SBATCH --mem-per-cpu=''value_in_MB''
| Minimum Memory required per allocated CPU. (Replaces the option pmem of MOAB. You should omit the setting of this option.)
|-
|- style="vertical-align:top;"
| --mail-type=''type''
| #SBATCH --mail-type=''type''
| Notify user by email when certain event types occur. Valid type values are NONE, BEGIN, END, FAIL, REQUEUE, ALL.
|-
|- style="vertical-align:top;"
| --mail-user=''mail-address''
| #SBATCH --mail-user=''mail-address''
| The specified mail-address receives email notification of state changes as defined by --mail-type.
|-
|- style="vertical-align:top;"
| --output=''name''
| #SBATCH --output=''name''
| File in which job output is stored.
|-
|- style="vertical-align:top;"
| --error=''name''
| #SBATCH --error=''name''
| File in which job error messages are stored.
|-
|- style="vertical-align:top;"
| -J ''name'' or --job-name=''name''
| #SBATCH --job-name=''name''
| Job name.
|-
|- style="vertical-align:top;"
| --export=[ALL,] ''env-variables''
| #SBATCH --export=[ALL,] ''env-variables''
| Identifies which environment variables from the submission environment are propagated to the launched application. Default is ALL. If adding an environment variable to the submission environment is intended, the argument ALL must be added.
|-
|- style="vertical-align:top;"
| -A ''group-name'' or --account=''group-name''
| #SBATCH --account=''group-name''
| Change resources used by this job to specified group. You may need this option if your account is assigned to more than one group. By command "scontrol show job" the project group the job is accounted on can be seen behind "Account=".
|-
|- style="vertical-align:top;"
| -p ''queue-name'' or --partition=''queue-name''
| #SBATCH --partition=''queue-name''
| Request a specific queue for the resource allocation.
|-
|- style="vertical-align:top;"
| --reservation=''reservation-name''
| #SBATCH --reservation=''reservation-name''
| Use a specific reservation for the resource allocation.
|-
|- style="vertical-align:top;"
| -C ''LSDF'' or --constraint=''LSDF''
| #SBATCH --constraint=LSDF
| Job constraint LSDF Filesystems.
|-
|}
 

==== sbatch --partition ''queues'' ====
Queue classes define maximum resources such as walltime, nodes and processes per node and queue of the compute system. Details can be found here:
* [[BinAC2/SLURM_Partitions|BinAC 2 partitions]]
 

=== sbatch Examples ===

If you are coming from Moab/Torque on BinAC 1 or you are new to HPC/Slurm the <code>sbatch</code> options may confuse you. The following examples give an orientation how to run typical workloads on BinAC 2.

You can find every file mentioned on this Wiki page on BinAC 2 at: <code>/pfs/10/project/examples</code>

==== Serial Programs ====
When you use serial programs that use only one process, you can omit most of the <code>sbatch</code> parameters, as the default values are sufficient.

To submit a serial job that runs the script <code>serial_job.sh</code> and requires 5000 MB of main memory and 10 minutes of wall clock time, Slurm will allocate one '''physical''' core to your job.

a) execute:
<pre>
$ sbatch -p compute -t 10:00 --mem=5000m serial_job.sh
</pre>
or
b) add after the initial line of your script '''serial_job.sh''' the lines:
<source lang="bash">
#SBATCH --time=10:00
#SBATCH --mem=5000m
#SBATCH --job-name=simple-serial-job
</source>
and execute the modified script with the command line option ''--partition=compute''
<pre>
$ sbatch -p=compute serial_job.sh
</pre>
Note, that sbatch command line options overrule script options.
 
 

==== Multithreaded Programs ====

Multithreaded programs run their processes on multiple threads and share resources such as memory. 
You may use a program that includes a built-in option for multithreading (e.g., options like <code>--threads</code>). 
For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) a number of threads is defined by the environment variable <code>OMP_NUM_THREADS</code>. By default, this variable is set to 1 (<code>OMP_NUM_THREADS=1</code>).

'''Important:''' Hyperthreading is activated on bwForCluster BinAC 2. Hyperthreading can be beneficial for some applications and codes, but it can also degrade performance in other cases. We therefore recommend to run a small test job with and without hyperthreading to determine the best choice. '''

'''a) Program with built-in multithreading option'''

The example uses the common Bioinformatics software called <code>samtools</code> as example for using built-in multithreading.

The module <code>bio/samtools/1.21</code> provides an example jobscript that requests 4 CPUs and runs <code>samtools sort</code> with 4 threads.

<pre>
#!/bin/bash

#SBATCH --time=19:00
#SBATCH --nodes=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=5000m
#SBATCH --partition compute
[...]
samtools sort -@ 4 sample.bam -o sample.sorted.bam
</pre>

You can use the example jobscript with this command

<pre>
sbatch /opt/bwhpc/common/bio/samtools/1.21/bwhpc-examples/binac2-samtools-1.21-bwhpc-examples.slurm
</pre>

'''b) OpenMP'''

We will run an exaple OpenMP Hello-World program. The jobscript looks like this:

<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --cpus-per-task=4
#SBATCH --time=1:00
#SBATCH --mem=5000m
#SBATCH -J OpenMP-Hello-World

export OMP_NUM_THREADS=$(${SLURM_JOB_CPUS_PER_NODE}/2)

echo "Executable running on ${SLURM_JOB_CPUS_PER_NODE} cores with ${OMP_NUM_THREADS} threads"

# Run parallel Hello World
/pfs/10/project/examples/openmp_hello_world
</pre>

Submit the job to the <code>compute</code> partition and get the output (in the stdout-file)

<pre>
sbatch --partition=compute /pfs/10/project/examples/openmp_hello_world.sh

Executable running on 4 cores with 4 threads
Hello from process: 0
Hello from process: 2
Hello from process: 1
Hello from process: 3
</pre>

==== OpenMPI ====

If you want to run MPI-jobs on batch nodes, generate a wrapper script <code>mpi_hello_world.sh</code> for '''OpenMPI''' containing the following lines:

<source lang="bash">
#!/bin/bash

#SBATCH --partition compute
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=2
#SBATCH --mem-per-cpu=2000
#SBATCH --time=05:00

# Load the MPI implementation of your choice
module load mpi/openmpi/4.1-gnu-14.2

# Run your MPI program
mpirun --bind-to core --map-by core --report-bindings mpi_hello_world
</source>

'''Attention:''' Do '''NOT''' add mpirun options <code>-n <number_of_processes></code> or any other option defining processes or nodes, since Slurm instructs mpirun about number of processes and node hostnames.

Use '''ALWAYS''' the MPI options <code>--bind-to core</code> and <code>--map-by core|socket|node</code>.
Please type <code>man mpirun</code> for an explanation of the meaning of the different options of mpirun option <code>--map-by</code>.

The above jobscript runs four OpenMPI tasks, distributed between two nodes. Because of hyperthreading you have to set <code>--cpus-per-task=2</code>. This means each MPI-task will get one physical core. If you omit <code>--cpus-per-task=2</code> MPI will fail.

'''Attention:''' Not all compute nodes are connected via Infiniband. Tell Slurm you want Infiniband via <code>--constraint=ib</code> when submitting or add <code>#SBATCH --constraint=ib</code> to your jobscript.

<pre>
$ sbatch --constraint=ib /pfs/10/project/examples/mpi_hello_world.sh
</pre>

This will run a simple Hello World program:

<pre>
[...]
Hello world from processor node2-031, rank 3 out of 4 processors
Hello world from processor node2-031, rank 2 out of 4 processors
Hello world from processor node2-030, rank 1 out of 4 processors
Hello world from processor node2-030, rank 0 out of 4 processors

</pre>

==== Multithreaded + MPI parallel Programs ====

Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes. '''Because hyperthreading is switched on BinaC 2, the option --cpus-per-task (-c) must be set to 2*n, if you want to use n threads.'''
 
 
===== OpenMPI with Multithreading =====
Multiple MPI tasks using '''OpenMPI''' must be launched by the MPI parallel program '''mpirun'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).
 
'''For OpenMPI''' a job-script to submit a batch job called ''job_ompi_omp.sh'' that runs a MPI program with 4 tasks and a 28-fold threaded program ''ompi_omp_program'' requiring 3000 MByte of physical memory per thread (using 28 threads per MPI task you will get 28*3000 MByte = 84000 MByte per MPI task) and total wall clock time of 3 hours looks like:

<source lang="bash">
#!/bin/bash
#SBATCH --ntasks=4
#SBATCH --cpus-per-task=56
#SBATCH --time=03:00:00
#SBATCH --mem=83gb # 84000 MB = 84000/1024 GB = 82.1 GB
#SBATCH --export=ALL,MPI_MODULE=mpi/openmpi/3.1,EXECUTABLE=./ompi_omp_program
#SBATCH --output="parprog_hybrid_%j.out"

# Use when a defined module environment related to OpenMPI is wished
module load ${MPI_MODULE}
export OMP_NUM_THREADS=$((${SLURM_CPUS_PER_TASK}/2))
export MPIRUN_OPTIONS="--bind-to core --map-by socket:PE=${OMP_NUM_THREADS} -report-bindings"
export NUM_CORES=${SLURM_NTASKS}*${OMP_NUM_THREADS}
echo "${EXECUTABLE} running on ${NUM_CORES} cores with ${SLURM_NTASKS} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpirun -n ${SLURM_NTASKS} ${MPIRUN_OPTIONS} ${EXECUTABLE}"
echo $startexe
exec $startexe
</source>
Execute the script '''job_ompi_omp.sh''' by command sbatch:
<pre>
$ sbatch -p compute ./job_ompi_omp.sh
</pre>
* With the mpirun option ''--bind-to core'' MPI tasks and OpenMP threads are bound to physical cores.
* With the option ''--map-by node:PE=<value>'' (neighbored) MPI tasks will be attached to different nodes and each MPI task is bound to the first core of a node. <value> must be set to ${OMP_NUM_THREADS}.
* The option ''-report-bindings'' shows the bindings between MPI tasks and physical cores.
* The mpirun-options '''--bind-to core''', '''--map-by socket|...|node:PE=<value>''' should always be used when running a multithreaded MPI program.
 

==== GPU jobs ====

The nodes in the gpu_4 and gpu_8 queues have 4 or 8 NVIDIA Tesla V100 GPUs. Just submitting a job to these queues is not enough to also allocate one or more GPUs, you have to do so using the "--gres=gpu" parameter. You have to specifiy how many GPUs your job needs, e.g. "--gres=gpu:2" will request two GPUs.

The GPU nodes are shared between multiple jobs if the jobs don't request all the GPUs in a node and there are enough ressources to run more than one job. The individual GPUs are always bound to a single job and will not be shared between different jobs.

a) add after the initial line of your script job.sh the line including the
information about the GPU usage: #SBATCH --gres=gpu:2
<pre>
#!/bin/bash
#SBATCH --ntasks=40
#SBATCH --time=02:00:00
#SBATCH --mem=4000
#SBATCH --gres=gpu:2
</pre>

or b) execute:
<pre>
$ sbatch -p <queue> -n 40 -t 02:00:00 --mem 4000 --gres=gpu:2 job.sh
</pre>
 
If you start an interactive session on of the GPU nodes, you can use the "nvidia-smi" command to list the GPUs allocated to your job:
<pre>
$ nvidia-smi
Sun Mar 29 15:20:05 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2... Off | 00000000:3A:00.0 Off | 0 |
| N/A 29C P0 39W / 300W | 9MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla V100-SXM2... Off | 00000000:3B:00.0 Off | 0 |
| N/A 30C P0 41W / 300W | 8MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 14228 G /usr/bin/X 8MiB |
| 1 14228 G /usr/bin/X 8MiB |
+-----------------------------------------------------------------------------+
</pre>

 
In case of using Open MPI, the underlying communication infrastructure (UCX and Open MPI's BTL) is CUDA-aware.
However, there may be warnings, e.g. when running
<pre>
$ module load compiler/gnu/10.3 mpi/openmpi devel/cuad
$ mpirun mpirun -np 2 ./mpi_cuda_app
--------------------------------------
WARNING: There are more than one active ports on host 'uc2n520', but the
default subnet GID prefix was detected on more than one of these
ports. If these ports are connected to different physical IB
networks, this configuration will fail in Open MPI. This version of
Open MPI requires that every physically separate IB subnet that is
used between connected MPI processes must have different subnet ID
values.

Please see this FAQ entry for more details:

http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid

NOTE: You can turn off this warning by setting the MCA parameter
btl_openib_warn_default_gid_prefix to 0.
--------------------------------------------------------------------------
</pre>

Please run Open MPI's mpirun using the following command:
<pre>
$ mpirun --mca pml ucx --mca btl_openib_warn_default_gid_prefix 0 -np 2 ./mpi_cuda_app
</pre>
or disabling the (older) communication layer Bit-Transfer-Layer (short BTL) alltogether:
<pre>
$ mpirun --mca pml ucx --mca btl ^openib -np 2 ./mpi_cuda_app
</pre>

(Please note, that CUDA per v11.4 is only available with up to GCC-10)
 
 

== Start time of job or resources : squeue --start ==
The command can be used by any user to displays the estimated start time of a job based a number of analysis types based on historical usage, earliest available reservable resources, and priority based backlog. The command squeue is explained in detail on the webpage https://slurm.schedmd.com/squeue.html or via manpage (man squeue).
 
 
=== Access ===
By default, this command can be run by '''any user'''.
 
 

== List of your submitted jobs : squeue ==
Displays information about YOUR active, pending and/or recently completed jobs. The command displays all own active and pending jobs. The command squeue is explained in detail on the webpage https://slurm.schedmd.com/squeue.html or via manpage (man squeue).
 
 
=== Access ===
By default, this command can be run by any user.
 
 

=== Flags ===
{| width=750px class="wikitable"
|-
! Flag !! Description
|-
| -l, --long
| Report more of the available information for the selected jobs or job steps, subject to any constraints specified.
|}
 

=== Examples ===
''squeue'' example on BinaC 2 (Only your own jobs are displayed!).
<pre>
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
18088744 single CPV.sbat ab1234 PD 0:00 1 (Priority)
18098414 multiple CPV.sbat ab1234 PD 0:00 2 (Priority)
18090089 multiple CPV.sbat ab1234 R 2:27 2 uc2n[127-128]
$ squeue -l
JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON)
18088654 single CPV.sbat ab1234 COMPLETI 4:29 2:00:00 1 uc2n374
18088785 single CPV.sbat ab1234 PENDING 0:00 2:00:00 1 (Priority)
18098414 multiple CPV.sbat ab1234 PENDING 0:00 2:00:00 2 (Priority)
18088683 single CPV.sbat ab1234 RUNNING 0:14 2:00:00 1 uc2n413
</pre>
* The output of ''squeue'' shows how many jobs of yours are running or pending and how many nodes are in use by your jobs.
 

== Shows free resources : sinfo_t_idle ==
The Slurm command sinfo is used to view partition and node information for a system running Slurm. It incorporates down time, reservations, and node state information in determining the available backfill window. The sinfo command can only be used by the administrator.
 
SCC has prepared a special script (sinfo_t_idle) to find out how many processors are available for immediate use on the system. It is anticipated that users will use this information to submit jobs that meet these criteria and thus obtain quick job turnaround times.
 
 
=== Access ===
By default, this command can be used by any user or administrator.
 
 
=== Example ===
* The following command displays what resources are available for immediate use for the whole partition.
<pre>$ sinfo_t_idle
Partition dev_multiple : 8 nodes idle
Partition multiple : 332 nodes idle
Partition dev_single : 4 nodes idle
Partition single : 76 nodes idle
Partition long : 80 nodes idle
Partition fat : 5 nodes idle
Partition dev_special : 342 nodes idle
Partition special : 342 nodes idle
Partition dev_multiple_e: 7 nodes idle
Partition multiple_e : 335 nodes idle
Partition gpu_4 : 12 nodes idle
Partition gpu_8 : 6 nodes idle
</pre>
* For the above example jobs in all partitions can be run immediately.
 

== Detailed job information : scontrol show job ==
scontrol show job displays detailed job state information and diagnostic output for all or a specified job of yours. Detailed information is available for active, pending and recently completed jobs. The command scontrol is explained in detail on the webpage https://slurm.schedmd.com/scontrol.html or via manpage (man scontrol).
 
Display the state of all your jobs in normal mode: scontrol show job
 
Display the state of a job with <jobid> in normal mode: scontrol show job <jobid>
 
 
=== Access ===
* End users can use scontrol show job to view the status of their '''own jobs''' only.
 

=== Arguments ===
{| width=750px class="wikitable"
|-
! Option !! Default !! Description !! Example
|- style="vertical-align:top;"
|- style="width:12%;"
| -d
| (n/a)
| Detailed mode
| Example: Display the state with jobid 18089884 in detailed mode. <pre>scontrol -d show job 18089884</pre>
|}
 
 

=== Scontrol show job Example ===
Here is an example from BinAC 2.
<pre>
squeue # show my own jobs (here the userid is replaced!)
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
18089884 multiple CPV.sbat bq0742 R 33:44 2 uc2n[165-166]

$
$ # now, see what's up with my pending job with jobid 18089884
$
$ scontrol show job 18089884

JobId=18089884 JobName=CPV.sbatch
UserId=bq0742(8946) GroupId=scc(12345) MCS_label=N/A
Priority=3 Nice=0 Account=kit QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:35:06 TimeLimit=02:00:00 TimeMin=N/A
SubmitTime=2020-03-16T14:14:54 EligibleTime=2020-03-16T14:14:54
AccrueTime=2020-03-16T14:14:54
StartTime=2020-03-16T15:12:51 EndTime=2020-03-16T17:12:51 Deadline=N/A
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2020-03-16T15:12:51
Partition=multiple AllocNode:Sid=uc2n995:5064
ReqNodeList=(null) ExcNodeList=(null)
NodeList=uc2n[165-166]
BatchHost=uc2n165
NumNodes=2 NumCPUs=160 NumTasks=80 CPUs/Task=1 ReqB:S:C:T=0:0:*:1
TRES=cpu=160,mem=96320M,node=2,billing=160
Socks/Node=* NtasksPerN:B:S:C=40:0:*:1 CoreSpec=*
MinCPUsNode=40 MinMemoryCPU=1204M MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
Command=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/CPV.sbatch
WorkDir=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin
StdErr=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/slurm-18089884.out
StdIn=/dev/null
StdOut=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/slurm-18089884.out
Power=
MailUser=(null) MailType=NONE
</pre>
 
You can use standard Linux pipe commands to filter the very detailed scontrol show job output.
* In which state the job is?
<pre>$ scontrol show job 18089884 | grep -i State
JobState=COMPLETED Reason=None Dependency=(null)
</pre>
 

== Cancel Slurm Jobs ==
The scancel command is used to cancel jobs. The command scancel is explained in detail on the webpage https://slurm.schedmd.com/scancel.html or via manpage (man scancel).
 
 
=== Canceling own jobs : scancel ===
scancel is used to signal or cancel jobs, job arrays or job steps. The command is:
<pre>
$ scancel [-i] <job-id>
$ scancel -t <job_state_name>
</pre>

 
{| width=750px class="wikitable"
! Flag !! Default !! Description !! Example
|- style="vertical-align:top;"
| -i, --interactive
| (n/a)
| Interactive mode.
| Cancel the job 987654 interactively. <pre> scancel -i 987654 </pre>
|-
| -t, --state
| (n/a)
| Restrict the scancel operation to jobs in a certain state. "job_state_name" may have a value of either "PENDING", "RUNNING" or "SUSPENDED".
| Cancel all jobs in state "PENDING". <pre> scancel -t "PENDING" </pre>
|}
 

= Resource Managers =
=== Batch Job (Slurm) Variables ===
The following environment variables of Slurm are added to your environment once your job has started
(only an excerpt of the most important ones).
{| width=750px class="wikitable"
! Environment !! Brief explanation
|-
| SLURM_JOB_CPUS_PER_NODE
| Number of processes per node dedicated to the job
|-
| SLURM_JOB_NODELIST
| List of nodes dedicated to the job
|-
| SLURM_JOB_NUM_NODES
| Number of nodes dedicated to the job
|-
| SLURM_MEM_PER_NODE
| Memory per node dedicated to the job
|-
| SLURM_NPROCS
| Total number of processes dedicated to the job
|-
| SLURM_CLUSTER_NAME
| Name of the cluster executing the job
|-
| SLURM_CPUS_PER_TASK
| Number of CPUs requested per task
|-
| SLURM_JOB_ACCOUNT
| Account name
|-
| SLURM_JOB_ID
| Job ID
|-
| SLURM_JOB_NAME
| Job Name
|-
| SLURM_JOB_PARTITION
| Partition/queue running the job
|-
| SLURM_JOB_UID
| User ID of the job's owner
|-
| SLURM_SUBMIT_DIR
| Job submit folder. The directory from which sbatch was invoked.
|-
| SLURM_JOB_USER
| User name of the job's owner
|-
| SLURM_RESTART_COUNT
| Number of times job has restarted
|-
| SLURM_PROCID
| Task ID (MPI rank)
|-
| SLURM_NTASKS
| The total number of tasks available for the job
|-
| SLURM_STEP_ID
| Job step ID
|-
| SLURM_STEP_NUM_TASKS
| Task count (number of MPI ranks)
|-
| SLURM_JOB_CONSTRAINT
| Job constraints
|}
See also:
* [https://slurm.schedmd.com/sbatch.html#lbAI Slurm input and output environment variables]
 

=== Job Exit Codes ===
A job's exit code (also known as exit status, return code and completion code) is captured by SLURM and saved as part of the job record.
 
Any non-zero exit code will be assumed to be a job failure and will result in a Job State of FAILED with a reason of "NonZeroExitCode".
 
The exit code is an 8 bit unsigned number ranging between 0 and 255. While it is possible for a job to return a negative exit code, SLURM will display it as an unsigned value in the 0 - 255 range.
 
 
==== Displaying Exit Codes and Signals ====
SLURM displays a job's exit code in the output of the '''scontrol show job''' and the sview utility.
 
When a signal was responsible for a job or step's termination, the signal number will be displayed after the exit code, delineated by a colon(:).
 
 
==== Submitting Termination Signal ====
Here is an example, how to 'save' a Slurm termination signal in a typical jobscript.
<source lang="bash">
[...]
exit_code=$?
mpirun -np <#cores> <EXE_BIN_DIR>/<executable> ... (options) 2>&1
[ "$exit_code" -eq 0 ] && echo "all clean..." || \
echo "Executable <EXE_BIN_DIR>/<executable> finished with exit code ${$exit_code}"
[...]
</source>
* Do not use ''''time'''' mpirun! The exit code will be the one submitted by the first (time) program.
* You do not need an '''exit $exit_code''' in the scripts.
 
 
----
[[#top|Back to top]]

BinAC2/Slurm

2025-08-26T11:15:42Z

F Bartusch:

= General information about Slurm =

Any kind of calculation on the compute nodes of bwForCluster BinAC 2 requires the user to define calculations as a sequence of commands or single command together with required run time, number of CPU cores and main memory and submit all, i.e., the '''batch job''', to a resource and workload managing software. BinAC 2 has installed the workload managing software Slurm. Therefore any job submission by the user is to be executed by commands of the Slurm software. Slurm queues and runs user jobs based on fair sharing policies.

= External Slurm documentation =

You can find the official Slurm configuration and some other material here:

* Slurm documentation: https://slurm.schedmd.com/documentation.html
* Slurm cheat sheet: https://slurm.schedmd.com/pdfs/summary.pdf
* Slurm tutorials: https://slurm.schedmd.com/tutorials.html

= SLURM terminology =

SLURM knows and mirrors the division of the cluster into '''nodes''' with several '''cores'''. When queuing '''jobs''', there are several ways of requesting resources and it is important to know which term means what in SLURM. Here are some basic SLURM terms:

;Job
: A job is a self-contained computation that may encompass multiple tasks and is given specific resources like individual CPUs/GPUs, a specific amount of RAM or entire nodes. These resources are said to have been allocated for the job.

;Task
: A task is a single run of a single process. By default, one task is run per node and one CPU is assigned per task.

;Partition
: A partition (usually called queue outside SLURM) is a waiting line in which jobs are put by users.

;Socket
: Receptacle on the motherboard for one physically packaged processor (each of which can contain one or more cores).

;Core
: A complete private set of registers, execution units, and retirement queues needed to execute programs.

;Thread
: One or more hardware contexts withing a single core. Each thread has attributes of one core, managed & scheduled as a single logical processor by the OS.

;CPU
: A '''CPU''' in Slurm means a '''single core'''. This is different from the more common terminology, where a CPU (a microprocessor chip) consists of multiple cores. Slurm uses the term '''sockets''' when talking about CPU chips. Depending upon system configuration, a CPU can be either a '''core''' or a '''thread'''. On '''BinAC 2 Hyperthreading is activated on every machine'''. This means that the operating system and Slurm sees each physical core as two logical cores.

= Slurm Commands =

{| width=750px class="wikitable"
! Slurm commands !! Brief explanation
|-
| [https://slurm.schedmd.com/sbatch.html sbatch] || Submits a job and queues it in an input queue
|-
| [https://slurm.schedmd.com/salloc.html saclloc] || Request resources for an interactive job
|-
| [https://slurm.schedmd.com/squeue.html squeue] || Displays information about active, eligible, blocked, and/or recently completed jobs
|-
| [https://slurm.schedmd.com/scontrol.html scontrol] || Displays detailed job state information
|-
| [https://slurm.schedmd.com/scontrol.html sstat] || Displays status information about a running job
|-
| [https://slurm.schedmd.com/scancel.html scancel] || Cancels a job
|-
|}

== Interactive Jobs : srun / salloc ==

You can run interactive jobs for testing and developing your job scripts.
Several nodes are reserved for interactive work, so your jobs should start right away.
You can only submit one job to this partition at a time. A job can run for up to 10 hours (about one workday).

This example command gives you 16 cores and 128 GB of memory for four hours on one of the reserved nodes:

<pre>
salloc --partition=interactive --time=4:00:00 --cpus-per-task=16 --mem=128gb
</pre>

You can also use srun to request the same resources:

<pre>
srun --partition=interactive --time=4:00:00 --cpus-per-task=16 --mem=128gb --pty bash
</pre>

== Job Submission : sbatch ==

Batch jobs are submitted by using the command '''sbatch'''. The main purpose of the '''sbatch''' command is to specify the resources that are needed to run the job. '''sbatch''' will then queue the batch job. However, starting of batch job depends on the availability of the requested resources and the fair sharing value.
 
=== sbatch Command Parameters ===
The syntax and use of '''sbatch''' can be displayed via:
<pre>
$ man sbatch
</pre>
'''sbatch''' options can be used from the command line or in your job script. The following table shows the syntax and provides examples for each option.

{| class="wikitable"
! colspan="5" | sbatch Options
|-
! Command line
! Job Script
! Purpose
! Example
! Default value
|- style="vertical-align:top;"
| <code>-t ''time''</code> or <code>--time=''time''</code>
| #SBATCH --time=''time''
| Wall clock time limit. 
| <code>-t 2:30:00</code> Limits run time to 2h 30 min.<code>-t 2-12</code> Limits run time to 2 days and 12 hours.
| Depends on Slurm partition.
|-
|- style="vertical-align:top;"
| -N ''count'' or --nodes=''count''
| #SBATCH --nodes=''count''
| Number of nodes to be used.
| <code>-N 1</code> Run job on one node.<code>-N 2</code> Run job on two nodes (have to use MPI!)
|
|- style="vertical-align:top;"
| -n ''count'' or --ntasks=''count''
| #SBATCH --ntasks=''count''
| Number of tasks to be launched.
| <code>-n 2</code> launch two tasks in the job.
| One task per node
|-
|- style="vertical-align:top;"
| --ntasks-per-node=''count''
| #SBATCH --ntasks-per-node=''count''
| Maximum count of tasks per node. (Replaces the option <code>ppn</code> of MOAB.)
| <code>--ntasks-per-node=2</code> Run 2 tasks per node
| 1 task per node
|-
|- style="vertical-align:top;"
| -c ''count'' or --cpus-per-task=''count''
| #SBATCH --cpus-per-task=''count''
| Number of CPUs required per (MPI-)task.
| <code>-c 2</code> Request two CPUs per (MPI-)task.
| 1 CPU per (MPI-)task
|-
|- style="vertical-align:top;"
| <code>--mem=<size>[units]</code>
| #SBATCH --mem=''value_in_MB''
| Memory in MegaByte per node.<code>[units]</code> can be one of <code>[K<nowiki>|</nowiki>M<nowiki>|</nowiki>G<nowiki>|</nowiki>T]</code>.
| <code>--mem=10g</code> Request 10GB RAM per node <code>--mem=0</code> Request all memory on node
| Depends on Slurm configuration.It is better to specify <code>--mem</code> in every case.
|-
|- style="vertical-align:top;"
| --mem-per-cpu=''value_in_MB''
| #SBATCH --mem-per-cpu=''value_in_MB''
| Minimum Memory required per allocated CPU. (Replaces the option pmem of MOAB. You should omit the setting of this option.)
|-
|- style="vertical-align:top;"
| --mail-type=''type''
| #SBATCH --mail-type=''type''
| Notify user by email when certain event types occur. Valid type values are NONE, BEGIN, END, FAIL, REQUEUE, ALL.
|-
|- style="vertical-align:top;"
| --mail-user=''mail-address''
| #SBATCH --mail-user=''mail-address''
| The specified mail-address receives email notification of state changes as defined by --mail-type.
|-
|- style="vertical-align:top;"
| --output=''name''
| #SBATCH --output=''name''
| File in which job output is stored.
|-
|- style="vertical-align:top;"
| --error=''name''
| #SBATCH --error=''name''
| File in which job error messages are stored.
|-
|- style="vertical-align:top;"
| -J ''name'' or --job-name=''name''
| #SBATCH --job-name=''name''
| Job name.
|-
|- style="vertical-align:top;"
| --export=[ALL,] ''env-variables''
| #SBATCH --export=[ALL,] ''env-variables''
| Identifies which environment variables from the submission environment are propagated to the launched application. Default is ALL. If adding an environment variable to the submission environment is intended, the argument ALL must be added.
|-
|- style="vertical-align:top;"
| -A ''group-name'' or --account=''group-name''
| #SBATCH --account=''group-name''
| Change resources used by this job to specified group. You may need this option if your account is assigned to more than one group. By command "scontrol show job" the project group the job is accounted on can be seen behind "Account=".
|-
|- style="vertical-align:top;"
| -p ''queue-name'' or --partition=''queue-name''
| #SBATCH --partition=''queue-name''
| Request a specific queue for the resource allocation.
|-
|- style="vertical-align:top;"
| --reservation=''reservation-name''
| #SBATCH --reservation=''reservation-name''
| Use a specific reservation for the resource allocation.
|-
|- style="vertical-align:top;"
| -C ''LSDF'' or --constraint=''LSDF''
| #SBATCH --constraint=LSDF
| Job constraint LSDF Filesystems.
|-
|}
 

==== sbatch --partition ''queues'' ====
Queue classes define maximum resources such as walltime, nodes and processes per node and queue of the compute system. Details can be found here:
* [[BinAC2/SLURM_Partitions|BinAC 2 partitions]]
 

=== sbatch Examples ===

If you are coming from Moab/Torque on BinAC 1 or you are new to HPC/Slurm the <code>sbatch</code> options may confuse you. The following examples give an orientation how to run typical workloads on BinAC 2.

You can find every file mentioned on this Wiki page on BinAC 2 at: <code>/pfs/10/project/examples</code>

==== Serial Programs ====
When you use serial programs that use only one process, you can omit most of the <code>sbatch</code> parameters, as the default values are sufficient.

To submit a serial job that runs the script <code>serial_job.sh</code> and requires 5000 MB of main memory and 10 minutes of wall clock time, Slurm will allocate one '''physical''' core to your job.

a) execute:
<pre>
$ sbatch -p compute -t 10:00 --mem=5000m serial_job.sh
</pre>
or
b) add after the initial line of your script '''serial_job.sh''' the lines:
<source lang="bash">
#SBATCH --time=10:00
#SBATCH --mem=5000m
#SBATCH --job-name=simple-serial-job
</source>
and execute the modified script with the command line option ''--partition=compute''
<pre>
$ sbatch -p=compute serial_job.sh
</pre>
Note, that sbatch command line options overrule script options.
 
 

==== Multithreaded Programs ====

Multithreaded programs run their processes on multiple threads and share resources such as memory. 
You may use a program that includes a built-in option for multithreading (e.g., options like <code>--threads</code>). 
For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) a number of threads is defined by the environment variable <code>OMP_NUM_THREADS</code>. By default, this variable is set to 1 (<code>OMP_NUM_THREADS=1</code>).

'''Important:''' Hyperthreading is activated on bwForCluster BinAC 2. Hyperthreading can be beneficial for some applications and codes, but it can also degrade performance in other cases. We therefore recommend to run a small test job with and without hyperthreading to determine the best choice. '''

'''a) Program with built-in multithreading option'''

The example uses the common Bioinformatics software called <code>samtools</code> as example for using built-in multithreading.

The module <code>bio/samtools/1.21</code> provides an example jobscript that requests 4 CPUs and runs <code>samtools sort</code> with 4 threads.

<pre>
#!/bin/bash

#SBATCH --time=19:00
#SBATCH --nodes=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=5000m
#SBATCH --partition compute
[...]
samtools sort -@ 4 sample.bam -o sample.sorted.bam
</pre>

You can use the example jobscript with this command

<pre>
sbatch /opt/bwhpc/common/bio/samtools/1.21/bwhpc-examples/binac2-samtools-1.21-bwhpc-examples.slurm
</pre>

'''b) OpenMP'''

We will run an exaple OpenMP Hello-World program. The jobscript looks like this:

<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --cpus-per-task=4
#SBATCH --time=1:00
#SBATCH --mem=5000m
#SBATCH -J OpenMP-Hello-World

export OMP_NUM_THREADS=$(${SLURM_JOB_CPUS_PER_NODE}/2)

echo "Executable running on ${SLURM_JOB_CPUS_PER_NODE} cores with ${OMP_NUM_THREADS} threads"

# Run parallel Hello World
/pfs/10/project/examples/openmp_hello_world
</pre>

Submit the job to the <code>compute</code> partition and get the output (in the stdout-file)

<pre>
sbatch --partition=compute /pfs/10/project/examples/openmp_hello_world.sh

Executable running on 4 cores with 4 threads
Hello from process: 0
Hello from process: 2
Hello from process: 1
Hello from process: 3
</pre>

==== OpenMPI ====

If you want to run MPI-jobs on batch nodes, generate a wrapper script <code>mpi_hello_world.sh</code> for '''OpenMPI''' containing the following lines:

<source lang="bash">
#!/bin/bash

#SBATCH --partition compute
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=2
#SBATCH --mem-per-cpu=2000
#SBATCH --time=05:00

# Load the MPI implementation of your choice
module load mpi/openmpi/4.1-gnu-14.2

# Run your MPI program
mpirun --bind-to core --map-by core --report-bindings mpi_hello_world
</source>

'''Attention:''' Do '''NOT''' add mpirun options <code>-n <number_of_processes></code> or any other option defining processes or nodes, since Slurm instructs mpirun about number of processes and node hostnames.

Use '''ALWAYS''' the MPI options <code>--bind-to core</code> and <code>--map-by core|socket|node</code>.
Please type <code>man mpirun</code> for an explanation of the meaning of the different options of mpirun option <code>--map-by</code>.

The above jobscript runs four OpenMPI tasks, distributed between two nodes. Because of hyperthreading you have to set <code>--cpus-per-task=2</code>. This means each MPI-task will get one physical core. If you omit <code>--cpus-per-task=2</code> MPI will fail.

'''Attention:''' Not all compute nodes are connected via Infiniband. Tell Slurm you want Infiniband via <code>--constraint=ib</code> when submitting or add <code>#SBATCH --constraint=ib</code> to your jobscript.

<pre>
$ sbatch --constraint=ib /pfs/10/project/examples/mpi_hello_world.sh
</pre>

This will run a simple Hello World program:

<pre>
[...]
Hello world from processor node2-031, rank 3 out of 4 processors
Hello world from processor node2-031, rank 2 out of 4 processors
Hello world from processor node2-030, rank 1 out of 4 processors
Hello world from processor node2-030, rank 0 out of 4 processors

</pre>

==== Multithreaded + MPI parallel Programs ====

Multithreaded + MPI parallel programs operate faster than serial programs on multi CPUs with multiple cores. All threads of one process share resources such as memory. On the contrary MPI tasks do not share memory but can be spawned over different nodes. '''Because hyperthreading is switched on BinaC 2, the option --cpus-per-task (-c) must be set to 2*n, if you want to use n threads.'''
 
 
===== OpenMPI with Multithreading =====
Multiple MPI tasks using '''OpenMPI''' must be launched by the MPI parallel program '''mpirun'''. For multithreaded programs based on '''Open''' '''M'''ulti-'''P'''rocessing (OpenMP) number of threads are defined by the environment variable OMP_NUM_THREADS. By default this variable is set to 1 (OMP_NUM_THREADS=1).
 
'''For OpenMPI''' a job-script to submit a batch job called ''job_ompi_omp.sh'' that runs a MPI program with 4 tasks and a 28-fold threaded program ''ompi_omp_program'' requiring 3000 MByte of physical memory per thread (using 28 threads per MPI task you will get 28*3000 MByte = 84000 MByte per MPI task) and total wall clock time of 3 hours looks like:

<source lang="bash">
#!/bin/bash
#SBATCH --ntasks=4
#SBATCH --cpus-per-task=56
#SBATCH --time=03:00:00
#SBATCH --mem=83gb # 84000 MB = 84000/1024 GB = 82.1 GB
#SBATCH --export=ALL,MPI_MODULE=mpi/openmpi/3.1,EXECUTABLE=./ompi_omp_program
#SBATCH --output="parprog_hybrid_%j.out"

# Use when a defined module environment related to OpenMPI is wished
module load ${MPI_MODULE}
export OMP_NUM_THREADS=$((${SLURM_CPUS_PER_TASK}/2))
export MPIRUN_OPTIONS="--bind-to core --map-by socket:PE=${OMP_NUM_THREADS} -report-bindings"
export NUM_CORES=${SLURM_NTASKS}*${OMP_NUM_THREADS}
echo "${EXECUTABLE} running on ${NUM_CORES} cores with ${SLURM_NTASKS} MPI-tasks and ${OMP_NUM_THREADS} threads"
startexe="mpirun -n ${SLURM_NTASKS} ${MPIRUN_OPTIONS} ${EXECUTABLE}"
echo $startexe
exec $startexe
</source>
Execute the script '''job_ompi_omp.sh''' by command sbatch:
<pre>
$ sbatch -p compute ./job_ompi_omp.sh
</pre>
* With the mpirun option ''--bind-to core'' MPI tasks and OpenMP threads are bound to physical cores.
* With the option ''--map-by node:PE=<value>'' (neighbored) MPI tasks will be attached to different nodes and each MPI task is bound to the first core of a node. <value> must be set to ${OMP_NUM_THREADS}.
* The option ''-report-bindings'' shows the bindings between MPI tasks and physical cores.
* The mpirun-options '''--bind-to core''', '''--map-by socket|...|node:PE=<value>''' should always be used when running a multithreaded MPI program.
 

==== GPU jobs ====

The nodes in the gpu_4 and gpu_8 queues have 4 or 8 NVIDIA Tesla V100 GPUs. Just submitting a job to these queues is not enough to also allocate one or more GPUs, you have to do so using the "--gres=gpu" parameter. You have to specifiy how many GPUs your job needs, e.g. "--gres=gpu:2" will request two GPUs.

The GPU nodes are shared between multiple jobs if the jobs don't request all the GPUs in a node and there are enough ressources to run more than one job. The individual GPUs are always bound to a single job and will not be shared between different jobs.

a) add after the initial line of your script job.sh the line including the
information about the GPU usage: #SBATCH --gres=gpu:2
<pre>
#!/bin/bash
#SBATCH --ntasks=40
#SBATCH --time=02:00:00
#SBATCH --mem=4000
#SBATCH --gres=gpu:2
</pre>

or b) execute:
<pre>
$ sbatch -p <queue> -n 40 -t 02:00:00 --mem 4000 --gres=gpu:2 job.sh
</pre>
 
If you start an interactive session on of the GPU nodes, you can use the "nvidia-smi" command to list the GPUs allocated to your job:
<pre>
$ nvidia-smi
Sun Mar 29 15:20:05 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2... Off | 00000000:3A:00.0 Off | 0 |
| N/A 29C P0 39W / 300W | 9MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla V100-SXM2... Off | 00000000:3B:00.0 Off | 0 |
| N/A 30C P0 41W / 300W | 8MiB / 32510MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 14228 G /usr/bin/X 8MiB |
| 1 14228 G /usr/bin/X 8MiB |
+-----------------------------------------------------------------------------+
</pre>

 
In case of using Open MPI, the underlying communication infrastructure (UCX and Open MPI's BTL) is CUDA-aware.
However, there may be warnings, e.g. when running
<pre>
$ module load compiler/gnu/10.3 mpi/openmpi devel/cuad
$ mpirun mpirun -np 2 ./mpi_cuda_app
--------------------------------------
WARNING: There are more than one active ports on host 'uc2n520', but the
default subnet GID prefix was detected on more than one of these
ports. If these ports are connected to different physical IB
networks, this configuration will fail in Open MPI. This version of
Open MPI requires that every physically separate IB subnet that is
used between connected MPI processes must have different subnet ID
values.

Please see this FAQ entry for more details:

http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid

NOTE: You can turn off this warning by setting the MCA parameter
btl_openib_warn_default_gid_prefix to 0.
--------------------------------------------------------------------------
</pre>

Please run Open MPI's mpirun using the following command:
<pre>
$ mpirun --mca pml ucx --mca btl_openib_warn_default_gid_prefix 0 -np 2 ./mpi_cuda_app
</pre>
or disabling the (older) communication layer Bit-Transfer-Layer (short BTL) alltogether:
<pre>
$ mpirun --mca pml ucx --mca btl ^openib -np 2 ./mpi_cuda_app
</pre>

(Please note, that CUDA per v11.4 is only available with up to GCC-10)
 
 

== Start time of job or resources : squeue --start ==
The command can be used by any user to displays the estimated start time of a job based a number of analysis types based on historical usage, earliest available reservable resources, and priority based backlog. The command squeue is explained in detail on the webpage https://slurm.schedmd.com/squeue.html or via manpage (man squeue).
 
 
=== Access ===
By default, this command can be run by '''any user'''.
 
 

== List of your submitted jobs : squeue ==
Displays information about YOUR active, pending and/or recently completed jobs. The command displays all own active and pending jobs. The command squeue is explained in detail on the webpage https://slurm.schedmd.com/squeue.html or via manpage (man squeue).
 
 
=== Access ===
By default, this command can be run by any user.
 
 

=== Flags ===
{| width=750px class="wikitable"
|-
! Flag !! Description
|-
| -l, --long
| Report more of the available information for the selected jobs or job steps, subject to any constraints specified.
|}
 

=== Examples ===
''squeue'' example on BinaC 2 (Only your own jobs are displayed!).
<pre>
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
18088744 single CPV.sbat ab1234 PD 0:00 1 (Priority)
18098414 multiple CPV.sbat ab1234 PD 0:00 2 (Priority)
18090089 multiple CPV.sbat ab1234 R 2:27 2 uc2n[127-128]
$ squeue -l
JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON)
18088654 single CPV.sbat ab1234 COMPLETI 4:29 2:00:00 1 uc2n374
18088785 single CPV.sbat ab1234 PENDING 0:00 2:00:00 1 (Priority)
18098414 multiple CPV.sbat ab1234 PENDING 0:00 2:00:00 2 (Priority)
18088683 single CPV.sbat ab1234 RUNNING 0:14 2:00:00 1 uc2n413
</pre>
* The output of ''squeue'' shows how many jobs of yours are running or pending and how many nodes are in use by your jobs.
 

== Shows free resources : sinfo_t_idle ==
The Slurm command sinfo is used to view partition and node information for a system running Slurm. It incorporates down time, reservations, and node state information in determining the available backfill window. The sinfo command can only be used by the administrator.
 
SCC has prepared a special script (sinfo_t_idle) to find out how many processors are available for immediate use on the system. It is anticipated that users will use this information to submit jobs that meet these criteria and thus obtain quick job turnaround times.
 
 
=== Access ===
By default, this command can be used by any user or administrator.
 
 
=== Example ===
* The following command displays what resources are available for immediate use for the whole partition.
<pre>$ sinfo_t_idle
Partition dev_multiple : 8 nodes idle
Partition multiple : 332 nodes idle
Partition dev_single : 4 nodes idle
Partition single : 76 nodes idle
Partition long : 80 nodes idle
Partition fat : 5 nodes idle
Partition dev_special : 342 nodes idle
Partition special : 342 nodes idle
Partition dev_multiple_e: 7 nodes idle
Partition multiple_e : 335 nodes idle
Partition gpu_4 : 12 nodes idle
Partition gpu_8 : 6 nodes idle
</pre>
* For the above example jobs in all partitions can be run immediately.
 

== Detailed job information : scontrol show job ==
scontrol show job displays detailed job state information and diagnostic output for all or a specified job of yours. Detailed information is available for active, pending and recently completed jobs. The command scontrol is explained in detail on the webpage https://slurm.schedmd.com/scontrol.html or via manpage (man scontrol).
 
Display the state of all your jobs in normal mode: scontrol show job
 
Display the state of a job with <jobid> in normal mode: scontrol show job <jobid>
 
 
=== Access ===
* End users can use scontrol show job to view the status of their '''own jobs''' only.
 

=== Arguments ===
{| width=750px class="wikitable"
|-
! Option !! Default !! Description !! Example
|- style="vertical-align:top;"
|- style="width:12%;"
| -d
| (n/a)
| Detailed mode
| Example: Display the state with jobid 18089884 in detailed mode. <pre>scontrol -d show job 18089884</pre>
|}
 
 

=== Scontrol show job Example ===
Here is an example from BinAC 2.
<pre>
squeue # show my own jobs (here the userid is replaced!)
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
18089884 multiple CPV.sbat bq0742 R 33:44 2 uc2n[165-166]

$
$ # now, see what's up with my pending job with jobid 18089884
$
$ scontrol show job 18089884

JobId=18089884 JobName=CPV.sbatch
UserId=bq0742(8946) GroupId=scc(12345) MCS_label=N/A
Priority=3 Nice=0 Account=kit QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:35:06 TimeLimit=02:00:00 TimeMin=N/A
SubmitTime=2020-03-16T14:14:54 EligibleTime=2020-03-16T14:14:54
AccrueTime=2020-03-16T14:14:54
StartTime=2020-03-16T15:12:51 EndTime=2020-03-16T17:12:51 Deadline=N/A
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2020-03-16T15:12:51
Partition=multiple AllocNode:Sid=uc2n995:5064
ReqNodeList=(null) ExcNodeList=(null)
NodeList=uc2n[165-166]
BatchHost=uc2n165
NumNodes=2 NumCPUs=160 NumTasks=80 CPUs/Task=1 ReqB:S:C:T=0:0:*:1
TRES=cpu=160,mem=96320M,node=2,billing=160
Socks/Node=* NtasksPerN:B:S:C=40:0:*:1 CoreSpec=*
MinCPUsNode=40 MinMemoryCPU=1204M MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
OverSubscribe=NO Contiguous=0 Licenses=(null) Network=(null)
Command=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/CPV.sbatch
WorkDir=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin
StdErr=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/slurm-18089884.out
StdIn=/dev/null
StdOut=/pfs/data5/home/kit/scc/bq0742/git/CPV/bin/slurm-18089884.out
Power=
MailUser=(null) MailType=NONE
</pre>
 
You can use standard Linux pipe commands to filter the very detailed scontrol show job output.
* In which state the job is?
<pre>$ scontrol show job 18089884 | grep -i State
JobState=COMPLETED Reason=None Dependency=(null)
</pre>
 

== Cancel Slurm Jobs ==
The scancel command is used to cancel jobs. The command scancel is explained in detail on the webpage https://slurm.schedmd.com/scancel.html or via manpage (man scancel).
 
 
=== Canceling own jobs : scancel ===
scancel is used to signal or cancel jobs, job arrays or job steps. The command is:
<pre>
$ scancel [-i] <job-id>
$ scancel -t <job_state_name>
</pre>

 
{| width=750px class="wikitable"
! Flag !! Default !! Description !! Example
|- style="vertical-align:top;"
| -i, --interactive
| (n/a)
| Interactive mode.
| Cancel the job 987654 interactively. <pre> scancel -i 987654 </pre>
|-
| -t, --state
| (n/a)
| Restrict the scancel operation to jobs in a certain state. "job_state_name" may have a value of either "PENDING", "RUNNING" or "SUSPENDED".
| Cancel all jobs in state "PENDING". <pre> scancel -t "PENDING" </pre>
|}
 

= Resource Managers =
=== Batch Job (Slurm) Variables ===
The following environment variables of Slurm are added to your environment once your job has started
(only an excerpt of the most important ones).
{| width=750px class="wikitable"
! Environment !! Brief explanation
|-
| SLURM_JOB_CPUS_PER_NODE
| Number of processes per node dedicated to the job
|-
| SLURM_JOB_NODELIST
| List of nodes dedicated to the job
|-
| SLURM_JOB_NUM_NODES
| Number of nodes dedicated to the job
|-
| SLURM_MEM_PER_NODE
| Memory per node dedicated to the job
|-
| SLURM_NPROCS
| Total number of processes dedicated to the job
|-
| SLURM_CLUSTER_NAME
| Name of the cluster executing the job
|-
| SLURM_CPUS_PER_TASK
| Number of CPUs requested per task
|-
| SLURM_JOB_ACCOUNT
| Account name
|-
| SLURM_JOB_ID
| Job ID
|-
| SLURM_JOB_NAME
| Job Name
|-
| SLURM_JOB_PARTITION
| Partition/queue running the job
|-
| SLURM_JOB_UID
| User ID of the job's owner
|-
| SLURM_SUBMIT_DIR
| Job submit folder. The directory from which sbatch was invoked.
|-
| SLURM_JOB_USER
| User name of the job's owner
|-
| SLURM_RESTART_COUNT
| Number of times job has restarted
|-
| SLURM_PROCID
| Task ID (MPI rank)
|-
| SLURM_NTASKS
| The total number of tasks available for the job
|-
| SLURM_STEP_ID
| Job step ID
|-
| SLURM_STEP_NUM_TASKS
| Task count (number of MPI ranks)
|-
| SLURM_JOB_CONSTRAINT
| Job constraints
|}
See also:
* [https://slurm.schedmd.com/sbatch.html#lbAI Slurm input and output environment variables]
 

=== Job Exit Codes ===
A job's exit code (also known as exit status, return code and completion code) is captured by SLURM and saved as part of the job record.
 
Any non-zero exit code will be assumed to be a job failure and will result in a Job State of FAILED with a reason of "NonZeroExitCode".
 
The exit code is an 8 bit unsigned number ranging between 0 and 255. While it is possible for a job to return a negative exit code, SLURM will display it as an unsigned value in the 0 - 255 range.
 
 
==== Displaying Exit Codes and Signals ====
SLURM displays a job's exit code in the output of the '''scontrol show job''' and the sview utility.
 
When a signal was responsible for a job or step's termination, the signal number will be displayed after the exit code, delineated by a colon(:).
 
 
==== Submitting Termination Signal ====
Here is an example, how to 'save' a Slurm termination signal in a typical jobscript.
<source lang="bash">
[...]
exit_code=$?
mpirun -np <#cores> <EXE_BIN_DIR>/<executable> ... (options) 2>&1
[ "$exit_code" -eq 0 ] && echo "all clean..." || \
echo "Executable <EXE_BIN_DIR>/<executable> finished with exit code ${$exit_code}"
[...]
</source>
* Do not use ''''time'''' mpirun! The exit code will be the one submitted by the first (time) program.
* You do not need an '''exit $exit_code''' in the scripts.
 
 
----
[[#top|Back to top]]

BinAC2/Software/AMDock

2025-08-20T16:04:34Z

F Bartusch:

{| width=700px class="wikitable"
|-
! Description !! Content
|-
| module load
| chem/amdock
|-
| License
| [https://github.com/Valdes-Tresanco-MS/AMDock?tab=GPL-3.0-1-ov-file GPL-3.0]
|-
| Links
| [https://github.com/Valdes-Tresanco-MS/AMDock GitHub]
|-
| Graphical Interface
| Yes
|}

= Description =

AMDock (Assisted Molecular Docking) is a user-friendly graphical tool to assist in the docking of protein-ligand complexes using Autodock-Vina or AutoDock4. This tool integrates several external programs for processing docking input files, define the search space (box) and perform docking under user’s supervision.

You can use the AMDock graphical user interface on BinAC 2 via TigerVNC. The AMDock module provides an ready to use example jobscript.

= Usage =

== VNC viewer ==

First, you will need an VNC viewer on your local workstation. Please refer to the [https://wiki.bwhpc.de/e/BinAC2/Software/TigerVNC#VNC_viewer TigerVNC documentation] for installing the VNC viewer.

== Start AMDock ==

The AMDock module provides a ready to use example jobscript.

<pre>
# Load AMDock module
module load chem/amdock/1.6.2

# Copy the jobscript into your current working directory
cp $AMDOCK_EXA_DIR/binac2-amdock-1.6.2-example.slurm .
</pre>

Adjust needed resources (cpus, memory, time) as needed. Please don't touch anything else.

<pre>
#!/bin/bash

# Adjust these values as needed
#SBATCH --cpus-per-task=10
#SBATCH --mem=20gb
#SBATCH --time=6:00:00

# Don't change these settings
#SBATCH --nodes=1
#SBATCH --partition=interactive
#SBATCH --gres=display:1
#SBATCH --signal=B:SIGUSR1@60
#SBATCH --job-name=amdock-1.6.2

module load chem/amdock/1.6.2

# AMDock is used via a graphical user interface (GUI)
# We need to start a VNC server in order to work with
# AMDock
# This wrapper script will start the VNC server and prints
# connection details to the job's stdout file
amdock_wrapper
</pre>

Submit the AMDock job.

<pre>
jobid=$(sbatch --parsable binac2-amdock-1.6.2-example.slurm)
</pre>

== Create SSH tunnel ==

The compute node on which AMDock is running is not directly reachable from your local workstation. Therefore, you need to create an SSH tunnel from your workstation to the compute node via the BinAC 2 login node.

The job's standard output file (<code>slurm-${jobid}.out</code>) contains all the information you need. Please note that details such as the IP address, port number, and access URL will differ in your case.

<pre>
$ cat slurm-${jobid}.out
[...]

Paste this ssh command in a terminal on local host (i.e., laptop)
-----------------------------------------------------------------
ssh -N -L 5901:172.0.0.1:5901 tu_iioba01@login.binac2.uni-tuebingen.de

Then you can connect to the session with your local vncviewer:
-----------------------------------------------------------------
vncviewer localhost:5901
[...]
</pre>

Linux users can simply copy&paste the <code>ssh -N -L ...</code> command into their terminal.
Windows users, please refer to [https://wiki.bwhpc.de/e/BinAC2/Software/Jupyterlab#Windows_Users this documentation]. It's about JupyterLab, but the SSH-tunnel is created in a similar way.

== Connect to VNC Server ==

Now you can use your local VNC viewer and access the VNC session. Please use the correct port you got when starting the VNC server. At some point you will need to enter the password you create earlier.

=== Windows ===

TODO

=== Linux ===

<pre>
vncviewer localhost:5901
</pre>

You should be greeted with the virtual desktop running AMDock.

[[File:Binac2-amdock.png | 800px | center | Xterm opened in VNC viewer]]

Attention: If you close the AMDock window the VNCserver, as well as the interactive job, will be stopped.

BinAC2/Software

2025-08-20T16:02:39Z

F Bartusch:

== Environment Modules ==
Most software is provided as Modules.

Required reading to use: [[Environment Modules]]

== Available Software ==

* Web: Visit [https://www.bwhpc.de/software.php https://www.bwhpc.de/software.php], select <code>Cluster → bwForCluster BinAC 2</code>

* On the cluster: <code>module avail</code>


== Documentation ==

Documentation for environment modules available on the cluster:

* with command <code>module help</code>
* examples in <code>$SOFTNAME_EXA_DIR</code>

== Documentation in the Wiki ==

For some applications additional documentation is provided here.


* [[BinAC2/Software/AMDock | AMDock ]]


* [[BinAC/Software/Cellranger | Cell Ranger]]
* [[Development/Conda | Conda]]

* [[BinAC2/Software/Jupyterlab | JupyterLab]]
* [[BinAC2/Software/Nextflow | Nextflow and nf-core]]
* [[BinAC2/Software/TigerVNC | TigerVNC: Remote visualization using VNC]]

BinAC2/Software

2025-08-20T16:02:23Z

F Bartusch:

== Environment Modules ==
Most software is provided as Modules.

Required reading to use: [[Environment Modules]]

== Available Software ==

* Web: Visit [https://www.bwhpc.de/software.php https://www.bwhpc.de/software.php], select <code>Cluster → bwForCluster BinAC 2</code>

* On the cluster: <code>module avail</code>


== Documentation ==

Documentation for environment modules available on the cluster:

* with command <code>module help</code>
* examples in <code>$SOFTNAME_EXA_DIR</code>

== Documentation in the Wiki ==

For some applications additional documentation is provided here.


* [[BinAC2/Software/AMDock | AMDock ]]


* [[BinAC/Software/Cellranger | Cell Ranger]]
* [[Development/Conda | Conda]]

* [[BinAC2/Software/Jupyterlab | JupyterLab]]
* [[BinAC/Software/Nextflow | Nextflow and nf-core]]
* [[BinAC2/Software/TigerVNC | TigerVNC: Remote visualization using VNC]]

BinAC2/Software/AMDock

2025-08-20T16:02:00Z

F Bartusch: F Bartusch moved page BinAC/Software/AMDock to BinAC2/Software/AMDock without leaving a redirect: Aus Versehen im BinAC 1 Baum angelegt ...

{| width=700px class="wikitable"
|-
! Description !! Content
|-
| module load
| chem/amdock
|-
| License
| [https://github.com/Valdes-Tresanco-MS/AMDock?tab=GPL-3.0-1-ov-file GPL-3.0]
|-
| Links
| [https://github.com/Valdes-Tresanco-MS/AMDock GitHub]
|-
| Graphical Interface
| Yes
|}

= Description =

AMDock (Assisted Molecular Docking) is a user-friendly graphical tool to assist in the docking of protein-ligand complexes using Autodock-Vina or AutoDock4. This tool integrates several external programs for processing docking input files, define the search space (box) and perform docking under user’s supervision.

You can use the AMDock graphical user interface on BinAC 2 via TigerVNC. The AMDock provides an ready to use example jobscript.

= Usage =

== VNC viewer ==

First, you will need an VNC viewer on your local workstation. Please refer to the [https://wiki.bwhpc.de/e/BinAC2/Software/TigerVNC#VNC_viewer TigerVNC documentation] for installing the VNC viewer.

== Start AMDock ==

The AMDock module provides a ready to use example jobscript.

<pre>
# Load AMDock module
module load chem/amdock/1.6.2

# Copy the jobscript into your current working directory
cp $AMDOCK_EXA_DIR/binac2-amdock-1.6.2-example.slurm .
</pre>

Adjust needed resources (cpus, memory, time) as needed. Please don't touch anything else.

<pre>
#!/bin/bash

# Adjust these values as needed
#SBATCH --cpus-per-task=10
#SBATCH --mem=20gb
#SBATCH --time=6:00:00

# Don't change these settings
#SBATCH --nodes=1
#SBATCH --partition=interactive
#SBATCH --gres=display:1
#SBATCH --signal=B:SIGUSR1@60
#SBATCH --job-name=amdock-1.6.2

module load chem/amdock/1.6.2

# AMDock is used via a graphical user interface (GUI)
# We need to start a VNC server in order to work with
# AMDock
# This wrapper script will start the VNC server and prints
# connection details to the job's stdout file
amdock_wrapper
</pre>

Submit the AMDock job.

<pre>
jobid=$(sbatch --parsable binac2-amdock-1.6.2-example.slurm)
</pre>

== Create SSH tunnel ==

The compute node on which AMDock is running is not directly reachable from your local workstation. Therefore, you need to create an SSH tunnel from your workstation to the compute node via the BinAC 2 login node.

The job's standard output file (<code>slurm-${jobid}.out</code>) contains all the information you need. Please note that details such as the IP address, port number, and access URL will differ in your case.

<pre>
$ cat slurm-${jobid}.out
[...]

Paste this ssh command in a terminal on local host (i.e., laptop)
-----------------------------------------------------------------
ssh -N -L 5901:172.0.0.1:5901 tu_iioba01@login.binac2.uni-tuebingen.de

Then you can connect to the session with your local vncviewer:
-----------------------------------------------------------------
vncviewer localhost:5901
[...]
</pre>

Linux users can simply copy&paste the <code>ssh -N -L ...</code> command into their terminal.
Windows users, please refer to [https://wiki.bwhpc.de/e/BinAC2/Software/Jupyterlab#Windows_Users this documentation]. It's about JupyterLab, but the SSH-tunnel is created in a similar way.

== Connect to VNC Server ==

Now you can use your local VNC viewer and access the VNC session. Please use the correct port you got when starting the VNC server. At some point you will need to enter the password you create earlier.

=== Windows ===

TODO

=== Linux ===

<pre>
vncviewer localhost:5901
</pre>

You should be greeted with the virtual desktop running AMDock.

[[File:Binac2-amdock.png | 800px | center | Xterm opened in VNC viewer]]

Attention: If you close the AMDock window the VNCserver, as well as the interactive job, will be stopped.

BinAC2/Software/AMDock

2025-08-20T15:50:04Z

F Bartusch:

{| width=700px class="wikitable"
|-
! Description !! Content
|-
| module load
| chem/amdock
|-
| License
| [https://github.com/Valdes-Tresanco-MS/AMDock?tab=GPL-3.0-1-ov-file GPL-3.0]
|-
| Links
| [https://github.com/Valdes-Tresanco-MS/AMDock GitHub]
|-
| Graphical Interface
| Yes
|}

= Description =

AMDock (Assisted Molecular Docking) is a user-friendly graphical tool to assist in the docking of protein-ligand complexes using Autodock-Vina or AutoDock4. This tool integrates several external programs for processing docking input files, define the search space (box) and perform docking under user’s supervision.

You can use the AMDock graphical user interface on BinAC 2 via TigerVNC. The AMDock provides an ready to use example jobscript.

= Usage =

== VNC viewer ==

First, you will need an VNC viewer on your local workstation. Please refer to the [https://wiki.bwhpc.de/e/BinAC2/Software/TigerVNC#VNC_viewer TigerVNC documentation] for installing the VNC viewer.

== Start AMDock ==

The AMDock module provides a ready to use example jobscript.

<pre>
# Load AMDock module
module load chem/amdock/1.6.2

# Copy the jobscript into your current working directory
cp $AMDOCK_EXA_DIR/binac2-amdock-1.6.2-example.slurm .
</pre>

Adjust needed resources (cpus, memory, time) as needed. Please don't touch anything else.

<pre>
#!/bin/bash

# Adjust these values as needed
#SBATCH --cpus-per-task=10
#SBATCH --mem=20gb
#SBATCH --time=6:00:00

# Don't change these settings
#SBATCH --nodes=1
#SBATCH --partition=interactive
#SBATCH --gres=display:1
#SBATCH --signal=B:SIGUSR1@60
#SBATCH --job-name=amdock-1.6.2

module load chem/amdock/1.6.2

# AMDock is used via a graphical user interface (GUI)
# We need to start a VNC server in order to work with
# AMDock
# This wrapper script will start the VNC server and prints
# connection details to the job's stdout file
amdock_wrapper
</pre>

Submit the AMDock job.

<pre>
jobid=$(sbatch --parsable binac2-amdock-1.6.2-example.slurm)
</pre>

== Create SSH tunnel ==

The compute node on which AMDock is running is not directly reachable from your local workstation. Therefore, you need to create an SSH tunnel from your workstation to the compute node via the BinAC 2 login node.

The job's standard output file (<code>slurm-${jobid}.out</code>) contains all the information you need. Please note that details such as the IP address, port number, and access URL will differ in your case.

<pre>
$ cat slurm-${jobid}.out
[...]

Paste this ssh command in a terminal on local host (i.e., laptop)
-----------------------------------------------------------------
ssh -N -L 5901:172.0.0.1:5901 tu_iioba01@login.binac2.uni-tuebingen.de

Then you can connect to the session with your local vncviewer:
-----------------------------------------------------------------
vncviewer localhost:5901
[...]
</pre>

Linux users can simply copy&paste the <code>ssh -N -L ...</code> command into their terminal.
Windows users, please refer to [https://wiki.bwhpc.de/e/BinAC2/Software/Jupyterlab#Windows_Users this documentation]. It's about JupyterLab, but the SSH-tunnel is created in a similar way.

== Connect to VNC Server ==

Now you can use your local VNC viewer and access the VNC session. Please use the correct port you got when starting the VNC server. At some point you will need to enter the password you create earlier.

=== Windows ===

TODO

=== Linux ===

<pre>
vncviewer localhost:5901
</pre>

You should be greeted with the virtual desktop running AMDock.

[[File:Binac2-amdock.png | 800px | center | Xterm opened in VNC viewer]]

Attention: If you close the AMDock window the VNCserver, as well as the interactive job, will be stopped.

File:Binac2-amdock.png

2025-08-20T15:48:13Z

F Bartusch:

BinAC2/Software/TigerVNC

2025-08-20T14:38:15Z

F Bartusch:

{| width=700px class="wikitable"
|-
! Description !! Content
|-
| module load
| vis/tigervnc
|-
| License
| [https://github.com/TigerVNC/tigervnc?tab=GPL-2.0-1-ov-file GPL-2.0]
|-
| Links
| [https://tigervnc.org/ Homepage]
|-
|
| [https://github.com/TigerVNC/tigervnc GitHub]
|-
| Graphical Interface
| Yes
|}

= Description =

TigerVNC is a high-performance, platform-neutral implementation of VNC (Virtual Network Computing), a client/server application that allows users to launch and interact with graphical applications on remote machines. TigerVNC provides the levels of performance necessary to run 3D and video applications, and it attempts to maintain a common look and feel and re-use components, where possible, across the various platforms that it supports.

In simple terms: TigerVNC lets you run graphical programs on compute nodes while seeing and controlling them from your own computer.

= Technical Background =

This is some technical background. You don't really need it in order to use TigerVNC, but maybe you are interested how it works.

== Why Use VNC on HPC? ==

BinAC 2 doesn't have monitors or keyboards attached to the compute nodes. VNC allows you to:

* Run graphical programs (like data visualization tools, editors with GUIs, or analysis software)
* See the program windows on your local computer
* Interact with these programs using your mouse and keyboard

== X11 - The Graphics System ==

TigerVNC will provide a virtual display for your programs via an X11 server.

Think of X11 as the "graphics engine" that makes windows, buttons, and visual interfaces work on Linux systems. Just like Windows has its own way of showing programs on screen, Linux uses X11 to display graphical applications.
When you run a program with a graphical interface on BinAC 2, X11 is what creates the windows and handles mouse clicks and keyboard input.

== VNC - Remote Desktop Access ==

VNC (Virtual Network Computing) is like having a "remote control" for a computer's desktop. It lets you see and control a computer's screen from anywhere over the network.
Imagine you're at home but want to use a computer at work - VNC lets you see that computer's desktop in a window on your home computer and control it as if you were sitting right in front of it.

= Usage =

You will need a VNC viewer on your local workstation.

For programs that are installed as modules on BinAC 2 we might already have ready to use example jobscripts, so you don't need to start the VNC server yourself. Please check the <code>module help</code> of the module or ask us via hpcmaster@uni-tuebingen.de.

== VNC viewer ==

You will need to install a VNC viewer on your computer in order to connect to the VNC server on BinAC. In principel every VNC viewer should work, but by installing the corresponding TigerVNC viewer one eliminates some problems from the start.

=== Windows ===

There is an official TigerVNC client for Windows:

* [https://sourceforge.net/projects/tigervnc/files/stable/1.15.0/vncviewer64-1.15.0.exe/download Download on Sourceforge]

=== Linux ===

Many Linux distributions have TigerVNC available in their repository. Usually this will install not the newest TigerVNC version.

<pre>
# Ubuntu
sudo apt-get install tigervnc-viewer

# RHEL/Rocky/Fedora
dnf install tigervnc
</pre>

You can also install the newest version provided by TigerVNC on [https://sourceforge.net/projects/tigervnc/files/stable/1.15.0/ Sourceforge].

== Start VNC Server on BinAC 2 ==

First, submit an interactive job. This command will start a 4-hour interactive job and tells the system it will need one display.

<pre>
srun --partition=interactive -t 4:00:00 --gres=display:1 --pty bash
</pre>

When the job has started, load the TigerVNC module

<pre>
module load vis/tigervnc/1.15.0
</pre>

This will add some helper scripts into your <code>PATH</code>

You will need to set a password for the VNC server if you use TigerVNC for the first time. You don't need a view-only password.
Your password will be stored at <code>~/.config/tigervnc/passwd</code>

<pre>
vncpasswd

Creating VNC password using TigerVNC container...
Password:
Verify:
Would you like to enter a view-only password (y/n)? n
A view-only password is not used
</pre>

Now get an available display and start the VNC server, connection details are printed to the console.

<pre>
display_num=$(get_display)

start_vncserver ${display_num}
INFO: Instance stats will not be available - requires cgroups v2 with systemd as manager.
INFO: instance started successfully

New 'node1-001:1 (tu_iioba01)' desktop is node1-001:1

Starting applications specified in /opt/bwhpc/common/vis/tigervnc/1.15.0/bin/xstartup
Log file is /home/tu/tu_tu/tu_iioba01/.vnc/node1-001:1.log

Paste this ssh command in a terminal on local host (i.e., laptop)
-----------------------------------------------------------------
ssh -N -L 5901:172.0.0.1:5901 tu_iioba01@login.binac2.uni-tuebingen.de

Then you can connect to the session with your local vncviewer:
-----------------------------------------------------------------
vncviewer localhost:5901
</pre>

You succesfully started

== Connect to VNC Server ==

You will need to create an SSH-tunnel from your local workstation to the compute node on which the VNC server is running.
Please check the already existing documentation in the [https://wiki.bwhpc.de/e/BinAC2/Software/Jupyterlab#Create_SSH_tunnel JupyterLab page ].

Now you can use your local VNC viewer and access the VNC session. Please use the correct port you got when starting the VNC server. At some point you will need to enter the password you create earlier.

=== Windows ===
TODO

=== Linux ===

<pre>
vncviewer localhost:5901
</pre>

You should be greeted with the virtual desktop.

[[File:Binac2-tigervnc-xfce.png | 800px | center | VNC viewer with Xfce window manager]]

You can now start your program. In this example we are starting a simple X terminal window. The terminal should then appear in your vncviewer.

<pre>
DISPLAY=:${display_num} xterm
</pre>

[[File:Binac2-tigervnc-xfce-xterm.png | 800px | center | Xterm opened in VNC viewer]]

== Stop VNC Server ==

It's important to stop the VNC server when you're done.

<pre>
stop_vncserver ${display_num}
Killing Xvnc process ID 3515575
INFO: Stopping tigervnc_1 instance of /opt/bwhpc/common/vis/tigervnc/1.15.0/tigervnc-1.15.0.sif (PID=3515518)
</pre>

BinAC2/Software/TigerVNC

2025-08-20T14:37:47Z

F Bartusch:

{| width=700px class="wikitable"
|-
! Description !! Content
|-
| module load
| vis/tigervnc
|-
| License
| [https://github.com/TigerVNC/tigervnc?tab=GPL-2.0-1-ov-file GPL-2.0]
|-
| Links
| [https://tigervnc.org/ Homepage]
|-
|
| [https://github.com/TigerVNC/tigervnc GitHub]
|-
| Graphical Interface
| Yes
|}

= Description =

TigerVNC is a high-performance, platform-neutral implementation of VNC (Virtual Network Computing), a client/server application that allows users to launch and interact with graphical applications on remote machines. TigerVNC provides the levels of performance necessary to run 3D and video applications, and it attempts to maintain a common look and feel and re-use components, where possible, across the various platforms that it supports.

In simple terms: TigerVNC lets you run graphical programs on compute nodes while seeing and controlling them from your own computer.

= Technical Background =

This is some technical background. You don't really need it in order to use TigerVNC, but maybe you are interested how it works.

== Why Use VNC on HPC? ==

BinAC 2 doesn't have monitors or keyboards attached to the compute nodes. VNC allows you to:

* Run graphical programs (like data visualization tools, editors with GUIs, or analysis software)
* See the program windows on your local computer
* Interact with these programs using your mouse and keyboard

== X11 - The Graphics System ==

TigerVNC will provide a virtual display for your programs via an X11 server.

Think of X11 as the "graphics engine" that makes windows, buttons, and visual interfaces work on Linux systems. Just like Windows has its own way of showing programs on screen, Linux uses X11 to display graphical applications.
When you run a program with a graphical interface on BinAC 2, X11 is what creates the windows and handles mouse clicks and keyboard input.

== VNC - Remote Desktop Access ==

VNC (Virtual Network Computing) is like having a "remote control" for a computer's desktop. It lets you see and control a computer's screen from anywhere over the network.
Imagine you're at home but want to use a computer at work - VNC lets you see that computer's desktop in a window on your home computer and control it as if you were sitting right in front of it.

= Usage =

You will need a VNC viewer on your local workstation.

For programs that are installed as modules on BinAC 2 we might already haveready to use example jobscripts, so you don't need to start the VNC server yourself. Please check the <code>module help</code> of the module or ask us via hpcmaster@uni-tuebingen.de.

== VNC viewer ==

You will need to install a VNC viewer on your computer in order to connect to the VNC server on BinAC. In principel every VNC viewer should work, but by installing the corresponding TigerVNC viewer one eliminates some problems from the start.

=== Windows ===

There is an official TigerVNC client for Windows:

* [https://sourceforge.net/projects/tigervnc/files/stable/1.15.0/vncviewer64-1.15.0.exe/download Download on Sourceforge]

=== Linux ===

Many Linux distributions have TigerVNC available in their repository. Usually this will install not the newest TigerVNC version.

<pre>
# Ubuntu
sudo apt-get install tigervnc-viewer

# RHEL/Rocky/Fedora
dnf install tigervnc
</pre>

You can also install the newest version provided by TigerVNC on [https://sourceforge.net/projects/tigervnc/files/stable/1.15.0/ Sourceforge].

== Start VNC Server on BinAC 2 ==

First, submit an interactive job. This command will start a 4-hour interactive job and tells the system it will need one display.

<pre>
srun --partition=interactive -t 4:00:00 --gres=display:1 --pty bash
</pre>

When the job has started, load the TigerVNC module

<pre>
module load vis/tigervnc/1.15.0
</pre>

This will add some helper scripts into your <code>PATH</code>

You will need to set a password for the VNC server if you use TigerVNC for the first time. You don't need a view-only password.
Your password will be stored at <code>~/.config/tigervnc/passwd</code>

<pre>
vncpasswd

Creating VNC password using TigerVNC container...
Password:
Verify:
Would you like to enter a view-only password (y/n)? n
A view-only password is not used
</pre>

Now get an available display and start the VNC server, connection details are printed to the console.

<pre>
display_num=$(get_display)

start_vncserver ${display_num}
INFO: Instance stats will not be available - requires cgroups v2 with systemd as manager.
INFO: instance started successfully

New 'node1-001:1 (tu_iioba01)' desktop is node1-001:1

Starting applications specified in /opt/bwhpc/common/vis/tigervnc/1.15.0/bin/xstartup
Log file is /home/tu/tu_tu/tu_iioba01/.vnc/node1-001:1.log

Paste this ssh command in a terminal on local host (i.e., laptop)
-----------------------------------------------------------------
ssh -N -L 5901:172.0.0.1:5901 tu_iioba01@login.binac2.uni-tuebingen.de

Then you can connect to the session with your local vncviewer:
-----------------------------------------------------------------
vncviewer localhost:5901
</pre>

You succesfully started

== Connect to VNC Server ==

You will need to create an SSH-tunnel from your local workstation to the compute node on which the VNC server is running.
Please check the already existing documentation in the [https://wiki.bwhpc.de/e/BinAC2/Software/Jupyterlab#Create_SSH_tunnel JupyterLab page ].

Now you can use your local VNC viewer and access the VNC session. Please use the correct port you got when starting the VNC server. At some point you will need to enter the password you create earlier.

=== Windows ===
TODO

=== Linux ===

<pre>
vncviewer localhost:5901
</pre>

You should be greeted with the virtual desktop.

[[File:Binac2-tigervnc-xfce.png | 800px | center | VNC viewer with Xfce window manager]]

You can now start your program. In this example we are starting a simple X terminal window. The terminal should then appear in your vncviewer.

<pre>
DISPLAY=:${display_num} xterm
</pre>

[[File:Binac2-tigervnc-xfce-xterm.png | 800px | center | Xterm opened in VNC viewer]]

== Stop VNC Server ==

It's important to stop the VNC server when you're done.

<pre>
stop_vncserver ${display_num}
Killing Xvnc process ID 3515575
INFO: Stopping tigervnc_1 instance of /opt/bwhpc/common/vis/tigervnc/1.15.0/tigervnc-1.15.0.sif (PID=3515518)
</pre>

BinAC2/Software/TigerVNC

2025-08-20T14:33:47Z

F Bartusch:

{| width=700px class="wikitable"
|-
! Description !! Content
|-
| module load
| vis/tigervnc
|-
| License
| [https://github.com/TigerVNC/tigervnc?tab=GPL-2.0-1-ov-file GPL-2.0]
|-
| Links
| [https://tigervnc.org/ Homepage]
|-
|
| [https://github.com/TigerVNC/tigervnc GitHub]
|-
| Graphical Interface
| Yes
|}

= Description =

TigerVNC is a high-performance, platform-neutral implementation of VNC (Virtual Network Computing), a client/server application that allows users to launch and interact with graphical applications on remote machines. TigerVNC provides the levels of performance necessary to run 3D and video applications, and it attempts to maintain a common look and feel and re-use components, where possible, across the various platforms that it supports.

In simple terms: TigerVNC lets you run graphical programs on compute nodes while seeing and controlling them from your own computer.

= Technical Background =

This is some technical background. You don't really need it in order to use TigerVNC, but maybe you are interested how it works.

== Why Use VNC on HPC? ==

BinAC 2 doesn't have monitors or keyboards attached to the compute nodes. VNC allows you to:

* Run graphical programs (like data visualization tools, editors with GUIs, or analysis software)
* See the program windows on your local computer
* Interact with these programs using your mouse and keyboard

== X11 - The Graphics System ==

TigerVNC will provide a virtual display for your programs via an X11 server.

Think of X11 as the "graphics engine" that makes windows, buttons, and visual interfaces work on Linux systems. Just like Windows has its own way of showing programs on screen, Linux uses X11 to display graphical applications.
When you run a program with a graphical interface on BinAC 2, X11 is what creates the windows and handles mouse clicks and keyboard input.

== VNC - Remote Desktop Access ==

VNC (Virtual Network Computing) is like having a "remote control" for a computer's desktop. It lets you see and control a computer's screen from anywhere over the network.
Imagine you're at home but want to use a computer at work - VNC lets you see that computer's desktop in a window on your home computer and control it as if you were sitting right in front of it.

= Usage =

== VNC viewer ==

You will need to install a VNC viewer on your computer in order to connect to the VNC server on BinAC. In principel every VNC viewer should work, but by installing the corresponding TigerVNC viewer one eliminates some problems from the start.

=== Windows ===

There is an official TigerVNC client for Windows:

* [https://sourceforge.net/projects/tigervnc/files/stable/1.15.0/vncviewer64-1.15.0.exe/download Download on Sourceforge]

=== Linux ===

Many Linux distributions have TigerVNC available in their repository. Usually this will install not the newest TigerVNC version.

<pre>
# Ubuntu
sudo apt-get install tigervnc-viewer

# RHEL/Rocky/Fedora
dnf install tigervnc
</pre>

You can also install the newest version provided by TigerVNC on [https://sourceforge.net/projects/tigervnc/files/stable/1.15.0/ Sourceforge].

== Start VNC Server on BinAC 2 ==

First, submit an interactive job. This command will start a 4-hour interactive job and tells the system it will need one display.

<pre>
srun --partition=interactive -t 4:00:00 --gres=display:1 --pty bash
</pre>

When the job has started, load the TigerVNC module

<pre>
module load vis/tigervnc/1.15.0
</pre>

This will add some helper scripts into your <code>PATH</code>

You will need to set a password for the VNC server if you use TigerVNC for the first time. You don't need a view-only password.
Your password will be stored at <code>~/.config/tigervnc/passwd</code>

<pre>
vncpasswd

Creating VNC password using TigerVNC container...
Password:
Verify:
Would you like to enter a view-only password (y/n)? n
A view-only password is not used
</pre>

Now get an available display and start the VNC server, connection details are printed to the console.

<pre>
display_num=$(get_display)

start_vncserver ${display_num}
INFO: Instance stats will not be available - requires cgroups v2 with systemd as manager.
INFO: instance started successfully

New 'node1-001:1 (tu_iioba01)' desktop is node1-001:1

Starting applications specified in /opt/bwhpc/common/vis/tigervnc/1.15.0/bin/xstartup
Log file is /home/tu/tu_tu/tu_iioba01/.vnc/node1-001:1.log

Paste this ssh command in a terminal on local host (i.e., laptop)
-----------------------------------------------------------------
ssh -N -L 5901:172.0.0.1:5901 tu_iioba01@login.binac2.uni-tuebingen.de

Then you can connect to the session with your local vncviewer:
-----------------------------------------------------------------
vncviewer localhost:5901
</pre>

You succesfully started

== Connect to VNC Server ==

You will need to create an SSH-tunnel from your local workstation to the compute node on which the VNC server is running.
Please check the already existing documentation in the [https://wiki.bwhpc.de/e/BinAC2/Software/Jupyterlab#Create_SSH_tunnel JupyterLab page ].

Now you can use your local VNC viewer and access the VNC session. Please use the correct port you got when starting the VNC server. At some point you will need to enter the password you create earlier.

=== Windows ===
TODO

=== Linux ===

<pre>
vncviewer localhost:5901
</pre>

You should be greeted with the virtual desktop.

[[File:Binac2-tigervnc-xfce.png | 800px | center | VNC viewer with Xfce window manager]]

You can now start your program. In this example we are starting a simple X terminal window. The terminal should then appear in your vncviewer.

<pre>
DISPLAY=:${display_num} xterm
</pre>

[[File:Binac2-tigervnc-xfce-xterm.png | 800px | center | Xterm opened in VNC viewer]]

== Stop VNC Server ==

It's important to stop the VNC server when you're done.

<pre>
stop_vncserver ${display_num}
Killing Xvnc process ID 3515575
INFO: Stopping tigervnc_1 instance of /opt/bwhpc/common/vis/tigervnc/1.15.0/tigervnc-1.15.0.sif (PID=3515518)
</pre>

File:Binac2-tigervnc-xfce-xterm.png

2025-08-20T14:30:25Z

F Bartusch:

File:Binac2-tigervnc-xfce.png

2025-08-20T14:29:53Z

F Bartusch:

BinAC2/Software/Jupyterlab

2025-08-20T12:39:45Z

F Bartusch:

{{Softwarepage|devel/jupyterlab}}

{| width=700px class="wikitable"
|-
! Description !! Content
|-
| module load
| devel/jupyterlab
|-
| License
| [https://github.com/jupyterlab/jupyterlab/blob/main/LICENSE JupyterLab License]
|-
| Links
| [https://jupyter.org/ Homepage]
|-
| Graphical Interface
| Yes
|}

= Description =

JupyterLab is a web-based interactive development environment for notebooks, code, and data.

Currently, BinAC 2 provides the following [https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-minimal-notebook JupyterLab Docker images] via Apptainer:

* <syntaxhighlight inline>minimal-notebook</syntaxhighlight>
* <syntaxhighlight inline>r-notebook</syntaxhighlight>
* <syntaxhighlight inline>julia-notebook</syntaxhighlight>
* <syntaxhighlight inline>scipy-notebook</syntaxhighlight>

= Usage =

This guide is valid for <syntaxhighlight inline>minimal-notebook</syntaxhighlight>. You can also follow it for the other notebook flavors — just replace <syntaxhighlight inline>minimal-notebook</syntaxhighlight> with one of the other notebooks listed above.

You will start a job on the cluster as usual, create an SSH tunnel, and connect to the running JupyterLab instance via your browser.

== Start JupyterLab ==

The module <syntaxhighlight inline>devel/jupyterlab</syntaxhighlight> provides a job script for starting a JupyterLab instance on BinAC 2. Load the module and copy the template job script into your workspace:

<syntaxhighlight>
$ module load devel/jupyterlab

</syntaxhighlight>

Upon loading the module, you can list the available template job scripts and copy the one you need into your workspace:

<syntaxhighlight>
$ ls $JUPYTERLAB_EXA_DIR
binac2-julia-notebook.slurm binac2-minimal-notebook.slurm binac2-r-notebook.slurm binac2-scipy-notebook.slurm

$ cp $JUPYTERLAB_EXA_DIR/binac2-minimal-notebook.slurm <your workspace>
</syntaxhighlight>

The job script is very simple; you only need to adjust the hardware resources according to your needs:

<syntaxhighlight lang="bash" line>
#!/bin/bash

# Adjust these values as needed
#SBATCH --cpus-per-task=1
#SBATCH --mem=2gb
#SBATCH --time=6:00:00

# Don't change these settings
#SBATCH --nodes=1
#SBATCH --partition=interactive
#SBATCH --job-name=minimal-notebook-7.4.1
#PBS -j oe

# Load Jupyterlab module
module load devel/jupyterlab/7.4.1

# Start jupyterlab
$ ${JUPYTERLAB_BIN_DIR}/minimal-notebook.sh
</syntaxhighlight>

Now submit the job to SLURM. This command will store the job ID in the variable <syntaxhighlight inline>jobid</syntaxhighlight>

<syntaxhighlight>
$ jobid=$(sbatch --parsable binac2-minimal-notebook.slurm)
</syntaxhighlight>

{|style="background:#deffee; width:100%;"
|style="padding:5px; background:#cef2e0; text-align:left"|
[[Image:Attention.svg|center|25px]]
|style="padding:5px; background:#cef2e0; text-align:left"|
Depending on the current load on BinAC 2 and the resources you requested in your job script, it may take some time for the job to start.
|}

== Create SSH tunnel ==

The compute node on which JupyterLab is running is not directly reachable from your local workstation. Therefore, you need to create an SSH tunnel from your workstation to the compute node via the BinAC 2 login node.

The job's standard output file (<syntaxhighlight inline>slurm-${jobid}.out</syntaxhighlight>) contains all the information you need. Please note that details such as the IP address, port number, and access URL will differ in your case.

<syntaxhighlight lang="bash">
$ cat slurm-${jobid}.out
</syntaxhighlight>

[[File:Binac2_jupyterlab_connection_details.png | 800px | center | JupyterLab connection info]]

=== Linux Users ===

Copy the <syntaxhighlight inline>ssh -N -L ... </syntaxhighlight> command and execute it in a shell on your workstation. After successful authentication, the SSH tunnel will be ready to use. The ssh command does not return any output. If there are no error messages, everything should be fine:

[[File:Binac2_jupyterlab_ssh_tunnel_linux.png | 800px | center | SSH tunnel creation with Linux]]

=== Windows Users ===

If you are using Windows, you will need to create the SSH tunnel using an SSH client of your choice (e.g. MobaXTerm, PuTTY, etc.). Her, we will show how to create an SSH tunnel with MobaXTerm.

Select <syntaxhighlight inline>Tunneling</syntaxhighlight> in the top ribbon. Then click <syntaxhighlight inline>New SSH tunnel</syntaxhighlight>.
Configure the SSH tunnel with the correct values taken from the SSH tunnel information provided above.
For the example in this tutorial it looks as follows:

[[File:Binac2_jupyterlab_mobaxterm.png | 800px | center | SSH tunnel creation with MobaXTerm]]

== Access JupyterLab ==

JupyterLab is now running on a BinAC 2 compute node, and you have created an SSH tunnel from your workstation to that compute node. Open a browser and paste the URL with the access token into the address bar:

[[File:Binac2_jupyterlab_browser_url.png | 800px | center | JupyterLab URL in browser ]]

Your browser should now display the JupyterLab interface:

[[File:Binac2_jupyterlab_browser_lab.png | 800px | center ]]

== Shut Down JupyterLab ==

You can shut down JupyterLab via <syntaxhighlight inline>File -> Shut Down</syntaxhighlight>.
Please note that this will also terminate your compute job on BinAC 2!

[[ File:Binac2_jupyterlab_browser_shutdown.png | 800px | center | JupyterLab user interface]]

== Tips & Tricks ==

=== Managing Notebooks ===

JupyterLab's root directory will be your home directory. Since your home directory is backed up daily, you may want to store your notebooks there.
It may also be a good idea to place your notebooks under proper version control using Git.

=== Access <syntaxhighlight inline>work</syntaxhighlight> and <syntaxhighlight inline>project</syntaxhighlight> ===

The notebooks will always be able to access your data stored in <syntaxhighlight inline>work</syntaxhighlight> and <syntaxhighlight inline>project</syntaxhighlight>. However, the file browser shows the content in your home directory and you won’t be able to access <syntaxhighlight inline>work</syntaxhighlight> and <syntaxhighlight inline>project</syntaxhighlight> initially.
You can create symbolic links in your home directory to work and project, making these two partitions available in JupyterLab's file browser.

<syntaxhighlight lang="bash">
$ cd ~
$ ln -s /pfs/10/work/ $HOME/work
$ ln -s /pfs/10/project $HOME/project
</syntaxhighlight>

Through that link in your home directory you can navigate to your research data in Jupyterlab's file browser.
Here is an example how I linked to my directory:

[[ File:Binac2_jupyterlab_link.png | 800px | center | Access work and project storage in JupyterLab file browser ]]

=== Managing Kernels ===

Depending on the notebook you started, the available kernels will differ.
The Python notebook, for example, only has one Python kernel, whereas the R notebook has only one R kernel, and so on.
Given the nearly endless combinations of programming languages and packages they provide, we suggest creating your own kernels if needed.
The kernels are stored in your hone directory on BinAC 2: <syntaxhighlight inline>$HOME/.local/share/jupyter/kernels/</syntaxhighlight>
You cannot install new kernels directly from within JupyterLab; this must be done from the command line in your usual BinAC 2 shell session.

=== Add a new Kernel ===

We will show you how to create new kernels for Python and R.

==== Python ====

<syntaxhighlight lang="bash">
module load devel/miniforge
conda create --name kernel_env python=3.13 pandas numpy matplotlib ipykernel # 1
conda activate kernel_env # 2
python -m ipykernel install --user --name pandas --display-name="Python 3.13 (pandas)" # 3
# Installed kernelspec pandas in $HOME/.local/share/jupyter/kernels/pandas
</syntaxhighlight>

The first command creates a new Conda environment called <syntaxhighlight inline>kernel_env</syntaxhighlight> and installs a specific Python packages , along with a few additional Python packages.
It's important that you also install <syntaxhighlight inline>ipykernel</syntaxhighlight>, as we will need it later to create the JupyterLab kernel.
The second command activates the <syntaxhighlight inline>kernel_env</syntaxhighlight> Conda environment.
The third command creates the new JupyterLab kernel.

[[ File:Binac2_jupyterlab_new_kernel.png | 800px | center | New Python kernel in JupyterLab]]

==== R ====

The instructions for creating new R kernels are slightly different.

<syntaxhighlight lang="bash">
module load devel/miniforge
conda create --name r_kernel_env r-base=4.4.3 jupyter r-irkernel
conda activate r_kernel_env
R
# In the R-Session
install.packages(...)
IRkernel::installspec(name = 'ir44', displayname = 'R 4.4.3')
</syntaxhighlight>

The first command creates a new Conda environment called <syntaxhighlight inline>r_kernel_env</syntaxhighlight> and installs a specific R version.
It's important to also install <syntaxhighlight inline>r-irkernel</syntaxhighlight>, as we will need it later to create the JupyterLab kernel.
The second command activates the <syntaxhighlight inline>r_kernel_env</syntaxhighlight> Conda environment and open an R session.
In this session, you can install any R packages you need for your kernel.
Finally, create the new kernel with the <syntaxhighlight inline>installspec</syntaxhighlight> command.

=== Remove a Kernel ===

To remove a kernel from JupyterLab, simply delete the corresponding directory at <syntaxhighlight inline>$HOME/.local/share/jupyter/kernels/</syntaxhighlight>:

<syntaxhighlight lang="bash">
# Remove the JupyterLab kernel installed in the previous example
rm -rf $HOME/.local/share/jupyter/kernels/pandas
</syntaxhighlight>

Also, remove the corresponding Conda environment if you do not need it any more:

<syntaxhighlight lang="bash">
conda env remove --name kernel_env
</syntaxhighlight>

BinAC2/SLURM Partitions

2025-08-20T12:38:09Z

F Bartusch:

== Partitions ==

The bwForCluster BinAC 2 provides two partitions for job submission.
Within a partition job allocations are routed automatically to the most suitable compute node(s) for the requested resources (e.g. amount of nodes and cores, memory, number and type of GPUs).

The <code>gpu</code> partition will only run 8 jobs per user at the same time. A user can only use 4 A100 and 8 A30 GPUs at the same time.

The <code>interactive</code> will only run 1 job per user at the same time.
This partition is reserved is dedicated for testing things and using tools via a graphical user interfae.
The four nodes <code>node1-00[1-4]</code> are exclusively reserved for this partition.
You can run a VNC server in this partition. Please use <code>#SBATCH --gres=display:1</code> in your jobscript or <code>--gres=display:1</code> on the command line if you need a display. This ensures that your job starts on a node with "free" displays, because each of the four nodes only provide 20 possible virtual displays.


{| class="wikitable"
|-
! style="width:10%"| Partition
! style="width:10%"| Node Access Policy
! style="width:10%"| Node Types
! style="width:20%"| Default
! style="width:20%"| Limits
|-
| compute (default)
| shared
| cpu
| ntasks=1, time=00:10:00, mem-per-cpu=1gb
| nodes=2, time=14-00:00:00
|-
| gpu
| shared
| gpu
| ntasks=1, time=00:10:00, mem-per-cpu=1gb
| time=14-00:00:00MaxJobsPerUser: 8MaxTRESPerUser:<pre>gres/gpu:a100=4,
gres/gpu:a30=8,
gres/gpu:h200=4</pre>
|-
| interactive
| shared
| cpu
| ntasks=1, time=00:10:00, mem-per-cpu=1gb
| time=10:00:00MaxJobsPerUser: 1
|-
|}

=== Parallel Jobs ===

In order to submit parallel jobs to the InfiniBand part of the cluster, i.e., for fast inter-node communication, please select the appropriate nodes via the <code>--constraint=ib</code> option in your job script. For less demanding parallel jobs, you may try the <code>--constraint=eth</code> option, which utilizes 100Gb/s Ethernet instead of the low-latency 100Gb/s InfiniBand.

=== GPU Jobs ===

BinAC 2 provides different GPU models for computations. Please select the appropriate GPU type and the amount of GPUs with the <code>--gres=aXX:N</code> option in your job script

{| class="wikitable"
|-
! style="width:20%"| GPU
! style="width:20%"| GPU Memory
! style="width:20%"| # GPUs per Node [N]
! style="width:20%"| Submit Option
|-
| Nvidia A30
| 24GB
| 2
| <code>--gres=gpu:a30:N</code>
|-
| Nvidia A100
| 80GB
| 4
| <code>--gres=gpu:a100:N</code>
|-
| Nvidia H200
| 141GB
| 4
| <code>--gres=gpu:h200:N</code>
|-
|}

BinAC2/SLURM Partitions

2025-08-20T12:34:03Z

F Bartusch:

== Partitions ==

The bwForCluster BinAC 2 provides two partitions for job submission.
Within a partition job allocations are routed automatically to the most suitable compute node(s) for the requested resources (e.g. amount of nodes and cores, memory, number and type of GPUs).

The <code>gpu</code> partition will only run 8 jobs per user at the same time. A user can only use 4 A100 and 8 A30 GPUs at the same time.

The <code>interactive</code> will only run 1 job per user at the same time.
This partition is reserved is dedicated for testing things and using tools via a graphical user interfae.
The four nodes <code>node1-00[1-4]</code> are exclusively reserved for this partition.


{| class="wikitable"
|-
! style="width:10%"| Partition
! style="width:10%"| Node Access Policy
! style="width:10%"| Node Types
! style="width:20%"| Default
! style="width:20%"| Limits
|-
| compute (default)
| shared
| cpu
| ntasks=1, time=00:10:00, mem-per-cpu=1gb
| nodes=2, time=14-00:00:00
|-
| gpu
| shared
| gpu
| ntasks=1, time=00:10:00, mem-per-cpu=1gb
| time=14-00:00:00MaxJobsPerUser: 8MaxTRESPerUser:<pre>gres/gpu:a100=4,
gres/gpu:a30=8,
gres/gpu:h200=4</pre>
|-
| interactive
| shared
| cpu
| ntasks=1, time=00:10:00, mem-per-cpu=1gb
| time=10:00:00MaxJobsPerUser: 1
|-
|}

=== Parallel Jobs ===

In order to submit parallel jobs to the InfiniBand part of the cluster, i.e., for fast inter-node communication, please select the appropriate nodes via the <code>--constraint=ib</code> option in your job script. For less demanding parallel jobs, you may try the <code>--constraint=eth</code> option, which utilizes 100Gb/s Ethernet instead of the low-latency 100Gb/s InfiniBand.

=== GPU Jobs ===

BinAC 2 provides different GPU models for computations. Please select the appropriate GPU type and the amount of GPUs with the <code>--gres=aXX:N</code> option in your job script

{| class="wikitable"
|-
! style="width:20%"| GPU
! style="width:20%"| GPU Memory
! style="width:20%"| # GPUs per Node [N]
! style="width:20%"| Submit Option
|-
| Nvidia A30
| 24GB
| 2
| <code>--gres=gpu:a30:N</code>
|-
| Nvidia A100
| 80GB
| 4
| <code>--gres=gpu:a100:N</code>
|-
| Nvidia H200
| 141GB
| 4
| <code>--gres=gpu:h200:N</code>
|-
|}

BinAC2/Software/AMDock

2025-08-19T15:22:56Z

F Bartusch: Created page with "stub"

stub

BinAC2/Software

2025-08-19T15:22:50Z

F Bartusch:

== Environment Modules ==
Most software is provided as Modules.

Required reading to use: [[Environment Modules]]

== Available Software ==

* Web: Visit [https://www.bwhpc.de/software.php https://www.bwhpc.de/software.php], select <code>Cluster → bwForCluster BinAC 2</code>

* On the cluster: <code>module avail</code>


== Documentation ==

Documentation for environment modules available on the cluster:

* with command <code>module help</code>
* examples in <code>$SOFTNAME_EXA_DIR</code>

== Documentation in the Wiki ==

For some applications additional documentation is provided here.


* [[BinAC/Software/AMDock | AMDock ]]


* [[BinAC/Software/Cellranger | Cell Ranger]]
* [[Development/Conda | Conda]]

* [[BinAC2/Software/Jupyterlab | JupyterLab]]
* [[BinAC/Software/Nextflow | Nextflow and nf-core]]
* [[BinAC2/Software/TigerVNC | TigerVNC: Remote visualization using VNC]]

BinAC2/Software/TigerVNC

2025-08-19T13:15:03Z

F Bartusch: Created page with "stub"

stub

BinAC2/Software

2025-08-19T13:14:38Z

F Bartusch:

== Environment Modules ==
Most software is provided as Modules.

Required reading to use: [[Environment Modules]]

== Available Software ==

* Web: Visit [https://www.bwhpc.de/software.php https://www.bwhpc.de/software.php], select <code>Cluster → bwForCluster BinAC 2</code>

* On the cluster: <code>module avail</code>


== Documentation ==

Documentation for environment modules available on the cluster:

* with command <code>module help</code>
* examples in <code>$SOFTNAME_EXA_DIR</code>

== Documentation in the Wiki ==

For some applications additional documentation is provided here.




* [[BinAC/Software/Cellranger | Cell Ranger]]
* [[Development/Conda | Conda]]

* [[BinAC2/Software/Jupyterlab | JupyterLab]]
* [[BinAC/Software/Nextflow | Nextflow and nf-core]]
* [[BinAC2/Software/TigerVNC | TigerVNC: Remote visualization using VNC]]