BinAC2/Getting Started

Purpose and Goals

The Getting Started guide is designed for users who are new to HPC systems in general and to BinAC 2 specifically. After reading this guide, you should have a basic understanding of how to use BinAC 2 for your research.

Please note that this guide does not cover basic Linux command-line skills. If you're unfamiliar with commands such as listing directory contents or using a text editor, we recommend first exploring the Linux module on the [training platform].

This guide also doesn't cover every feature of the system but aims to provide a broad overview. For more detailed information about specific features, please refer to the dedicated Wiki pages on topics like the batch system, storage, and more.

Some terms in this guide may be unfamiliar. You can look them up in the HPC Glossary.

General Workflow of Running a Calculation

On an HPC Cluster, you do not simply log in and run your software. Instead, you write a Batch Script that contains all the commands needed to run and process your job, then submit it to a waiting queue to be executed on one of several hundred Compute Nodes.

Get Access to the Cluster

Follow the registration process for the bwForCluster. → How to Register for a bwForCluster

Login to the Cluster

Set up your service password and 2FA token, then log in to BinAC 2. → Login BinAC

Using the Linux command line

It is expected that you have at least basic Linux and command-line knowledge before using bwForCluster BinAC 2. There are numerous resources available online for learning fundamental concepts and commands. Here are two:

• bwHPC Linux Training course → Linux course on training.bwhpc.de
• HPC Wiki (external site) → Introduction to the Linux command line

Also see: .bashrc Do's and Don'ts

File System Basics

BinAC 2 offers several file systems for your data, each serving different needs. These are explained here in a short and simple form. For more detailed documentation, visit: here.

Home File System

Home directories are intended for the permanent storage of frequently used files, such as like source codes, configuration files, executable programs, conda environments, etc. The home file system is backed up daily and has a quota. If that quota is reached, you may experience issues when working with BinAC 2.

Here are some useful command line and bash tips for accessing the Home File system.

# For changing to your home directory, simply run:
cd

# To access files in your home directory within your job script, you can use one of these:
~/myFile   # or
$HOME/myFile

Project File System

BinAC 2 has a project file system intended for data that:

• is shared between members of a compute project
• is not actively used for computations in near future

The project file system is available at /pfs/10/project/:

$ ll /pfs/10/project/
total 333
drwxrwx---.  3 root       bw16f003 33280 Dec 19 17:16 bw16f003
drwxrwx---.  3 root       bw16g002 25600 Dec 17 15:23 bw16g002
[..]

As shown by the file permissions the project directories are only accessible to users belonging to the specific compute project.

Work File System

BinAC 2 has a work file system on SSDs intended for data that is actively used and produced by compute jobs. Each user creates workspaces on their own via the workspace tools.

The project file system is available at /pfs/10/work

$ ll /pfs/10/work/
total 1822
drwxr-xr-x.  3 root        root          33280 Feb 12 14:56 db
drwx------.  5 tu_iioba01  tu_tu         25600 Jan  8 14:42 tu_iioba01-alphafold3
[..]

As you can see from the file permissions, the resulting workspace can only be accessed by you, not by other group members or other users.

Temporary Data

If your job creates temporary data, you can use the fast SSD with a capacity of 211 GB on the compute nodes. The temporary directory for your job is available via the $TMPDIR environment variable.

Batch System Basics

On cluster systems like BinAC you do not run your analysis by hand on the login node. Instead, you write a script and submit it to the batch system, this is called a job. The batch systems then tries to schedule the jobs on the available compute nodes.

Queue/Job Basics

The cluster consists of compute nodes with different hardware features. These hardware features (e.g. high-mem or GPUs) are only available when submitting the jobs to the specific queue. Also, each queue has different settings regarding maximal walltime. The most recent queue settings are displayed on login as message of the day on the terminal.

Get an overview of the number of running and queued jobs:

$ qstat -q

Queue            Memory CPU Time Walltime Node  Run Que Lm  State
---------------- ------ -------- -------- ----  --- --- --  -----
tiny               --      --       --      --    0   0 --   E R
long               --      --       --      --  850   0 --   E R
gpu                --      --       --      --   66   0 --   E R
smp                --      --       --      --    4   1 --   E R
short              --      --       --      --  131  90 --   E R
                                               ----- -----
                                                1051    91

To check all running and queued jobs:

qstat

Just your own jobs.

qstat -u <username>

Interactive Jobs

Interactive jobs are a good method for testing if/how software works with your data.

To start a 1 core job on a compute node providing a remote shell.

qsub -q short -l nodes=1:ppn=1 -I

The same but requesting the whole node.

qsub -q short -l nodes=1:ppn=28 -I

Standard Unix commands are directly available, for everything else use the modules.

module avail

Be aware that we allow node sharing. Do not disturb the calculations of other users.

Simple Script Job

Use your favourite text editor to create a script called 'script.sh'.

Please note that there are differences between Windows and Linux line endings. Make sure that your editor uses Linux line endings when you are using Windows. You can check your line endings with vim -b <your script>. Windows line endings will be displayed as ^M.

#PBS -l nodes=1:ppn=1
#PBS -l walltime=00:05:00
#PBS -l mem=1gb
#PBS -S /bin/bash
#PBS -N Simple_Script_Job
#PBS -j oe
#PBS -o LOG
cd $PBS_O_WORKDIR
echo "my Username is:"
whoami
echo "My job is running on node:"
uname -a

Submit the job using

qsub -q tiny script.sh

Take a note of your jobID. The scheduler will reserve one core and 1 gigabyte of memory for 5 minutes on a compute node for your job. The job should be scheduled within minute if the tiny queue is empty and write your username and the execution node into the output file.

There are tons of options, details and caveats. Most of the options are explained on this page, but be aware that there are some differences on BinAC.

If your job needs GPUs, you have to specify how many GPUs you want. Just submitting the job to the GPU queue does not work:

#PBS -l nodes=1:ppn=1:gpus=1
#PBS -q gpu

If you encounter any problems, just send a mail to hpcmaster@uni-tuebingen.de.

Killing a Job

Let's assume you build a Homer and want to stop/kill/remove a running job.

qdel <jobID>

Best Practices

The scheduler will reserve computational resources (nodes, cores, gpus, memory) for a specified period for you. By following some best practices, you can avoid some common problems beforehand.

Specify memory for your job

Often we get tickets with question like "Why did the system kill my job?". Most often the user did not specify the required memory resources for the job. Then the following happens:

The job is started on a compute node, where it shares the resources other jobs. Let us assume that the other jobs on this node occupy already 100 gigabyte of memory. Now your job tries to allocate 40 gigabyte of memory. As the compute node has only 128 gigabyte, your job crashes because it cannot allocate that much memory.

You can make your life easier by specifying the required memory in your job script with:

#PBS -l mem=xxgb

Then you have the guarantee that your job can allocate xx gigabyte of memory.

If you do not know how much memory your job will need, look into the documentation of the tools you use or ask us. We also started a wiki page on which we will document some guidelines and pitfalls for specific tools.

Use the reserved resources

Reserved resources (nodes, cores, gpus, memory) are not available to other users and their jobs. You have the responsibility that your programs utilize the reserved resources.

An extreme example: You request a whole node (node=1:ppn=28), but your job uses just one core. The other 27 cores are idling. This is bad practice, so take care that the used programs really use the requested resources.

Another example are tools that do not benefit from a increasing number of cores. Please check the documentation of your tools and also check the feedback files that report the CPU efficiency of your job.

[...]
CPU efficiency, 0-100%                      | 25.00
[...]

This job for example used only 25% of the available CPU resources.

Software

There are several mechanisms how software can be installed on BinAC. If you need software that is not installed on BinAC you can open a ticket and we can find a way to provide the software on the cluster.

Environment Modules

Environment modules is the 'classic' way for providing software on clusters. A module consists of a specific software version and can be loaded. The module system then manipulates the PATH and other environment variables such that the software can be used.

# Show available modules
$ module avail

# Load a module
$ module load bio/bowtie2/2.4.1

# Show the module's help
$ module help bio/bowtie2/2.4.1

A more detailed description of module environments can be found on this wiki page

Sometimes software packages have so many dependencies or the user wants a combination of tools, so that environment modules cannot be used in a meaningful way. Then other solutions like conda environments or Singularity container (see below) can be used.

Conda Environments

Conda environments is a nice possibility for creating custom environments on the cluster, as a majority of the scientific software is available in the meantime as conda packages. First, you have to install Miniconda in your home directory.

# Download installer
$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
$ sh Miniconda3-latest-Linux-x86_64.sh
$ source ~/.bashrc

Then you can create your first environment and install software into it:

# Create an environment
$ conda create --name my_first_conda_environment
# Activate this environment
conda activate my_first_conda_environment
# Install software into this environment
$ conda install scipy=1.5.2

You will need to add this line to your jobscripts such that the environments are available on the compute nodes:

source $HOME/miniconda3/etc/profile.d/conda.sh
conda activate <env_name>

When installing software conda will solve dependencies on the fly. But it is not guaranteed that conda will use the exact same package versions in the future. For the sake of reproducibility, you can write a file containing all conda packages together with their versions:

# Export packages installed in the active environment
$ conda list --explicit > spec-file.txt
# Create a new environment with the exact same conda packages
$ conda create --name myenv --file spec-file.txt

Singularity Container

Sometimes software is also available in a software container format. Singularity is installed on all BinAC nodes. You can pull Singularity or Docker containers from registries onto BinAC and use them. You can also build new Singularity containers on your own machine and copy them to BinAC.

Please note that Singularity containers should be stored in the work file system. We configured Singularity such that containers stored in your home directory do not work.