BinAC/Quickstart Guide

File System Basics

The details of the file systems are explained here.

Home File System

Home directories are meant for permanent file storage of files that are keep being used like source codes, configuration files, executable programs, conda environments, etc. It is backuped daily and has a quota. If that quota is reached, you will usually experience problems when working on the cluster.

Work File System

Use the work file system and not your home directory for your calculations and data. Create a work directory, usually users use their username:

cd /beegfs/work/
mkdir <username>
cd <username>

Do not use the login nodes to carry out any calculations or heavy load file transfers.

How to share data with coworkers?

You can share data with coworkers of your compute project my modifying file ownership and permissions on a work directory. Every compute project has its own group on BinAC, namely the project's acronym. In this example, the group for user tu_iioba01 is bw16f003.

$ id tu_iioba01
uid=900102(tu_iioba01) gid=500001(tu_tu) groups=500001(tu_tu),500002(bw16f003)

In order to share data with coworkers in the same compute project change the group owner of a directory you want to share to this acronym. You can also set the so-called SGID-Bit, then new files and and subdirectories will automatically belong to the group.

# Change ownership. This command changes the ownwership for ALL files and subdirectories.
$ chown -R <username>:<acronym> /beegfs/work/<your directory>
# Set SGID-Bit
$ chmod g+s /beegfs/work/<your directory>

Now you can set the file permissions for your coworkers in this directory as wanted by granting read, write, and execute permissions to files and subdirectories.

# Some examples
# Coworkers can read,write,delete,execute files in the directory
$ chmod 770 /beegfs/work/<your directory>
# Coworkers can read, owner still can write/delete files
$ chmod 640 /beegfs/work/<your directory>/<important_dataset>

Temporary Data

If your job creates temporary data, you can use the fast SSD with a capacity of 211 GB on the compute nodes. The temporary directory for your job is available via the $TMPDIR environment variable.

Batch System Basics

On cluster systems like BinAC you do not run your analysis by hand on the login node. Instead, you write a script and submit it to the batch system, this is called a job. The batch systems then tries to schedule the jobs on the available compute nodes.

Queue/Job Basics

The cluster consists of compute nodes with different hardware features. These hardware features (e.g. high-mem or GPUs) are only available when submitting the jobs to the specific queue. Also, each queue has different settings regarding maximal walltime. The most recent queue settings are displayed on login as message of the day on the terminal.

Get an overview of the number of running and queued jobs:

$ qstat -q

Queue            Memory CPU Time Walltime Node  Run Que Lm  State
---------------- ------ -------- -------- ----  --- --- --  -----
tiny               --      --       --      --    0   0 --   E R
long               --      --       --      --  850   0 --   E R
gpu                --      --       --      --   66   0 --   E R
smp                --      --       --      --    4   1 --   E R
short              --      --       --      --  131  90 --   E R
                                               ----- -----
                                                1051    91

To check all running and queued jobs:

qstat

Just your own jobs.

qstat -u <username>

Interactive Jobs

Interactive jobs are a good method for testing if/how software works with your data.

To start a 1 core job on a compute node providing a remote shell.

qsub -q short -l nodes=1:ppn=1 -I

The same but requesting the whole node.

qsub -q short -l nodes=1:ppn=28 -I

Standard Unix commands are directly available, for everything else use the modules.

module avail

Be aware that we allow node sharing. Do not disturb the calculations of other users.

Simple Script Job

Use your favourite text editor to create a script calles 'script.sh'.

#PBS -l nodes=1:ppn=1
#PBS -l walltime=00:05:00
#PBS -l mem=1gb
#PBS -S /bin/bash
#PBS -N Simple_Script_Job
#PBS -j oe
#PBS -o LOG
cd $PBS_O_WORKDIR
echo "my Username is:"
whoami
echo "My job is running on node:"
uname -a

Submit the job using

qsub -q tiny script.sh

Take a note of your jobID. The scheduler will reserve one core and 1 gigabyte of memory for 5 minutes on a compute node for your job. The job should be scheduled within minute if the tiny queue is empty and write your username and the execution node into the output file.

There are tons of options, details and caveats. Most of the options are explained on this page, but be aware that there are some differences on BinAC.

If there is anything not working, as you like, send an email to hpcmaster@uni-tuebingen.de.

Killing a Job

Let's assume you build a Homer and want to stop/kill/remove a running job.

qdel <jobID>

Best Practices

The scheduler will reserve computational resources (nodes, cores, gpus, memory) for a specified period for you. By following some best practices, you can avoid some common problems beforehand.

Specify memory for your job

Often we get tickets with question like "Why did the system kill my job?". Most often the user did not specify the required memory resources for the job. Then the following happens:

The job is started on a compute node, where it shares the resources other jobs. Let us assume that the other jobs on this node occupy already 100 gigabyte of memory. Now your job tries to allocate 40 gigabyte of memory. As the compute node has only 128 gigabyte, your job crashes because it cannot allocate that much memory.

You can make your life easier by specifying the required memory in your job script with:

#PBS -l mem=xxgb

Then you have the guarantee that your job can allocate xx gigabyte of memory.

If you do not know how much memory your job will need, look into the documentation of the tools you use or ask us.

Use the reserved resources

Reserved resources (nodes, cores, gpus, memory) are not available to other users and their jobs. You have the responsibility that your programs utilize the reserved resources.

An extreme example: You request a whole node (node=1:ppn=28), but your job uses just one core. The other 27 cores are idling. This is bad practice, so take care that the used programs really use the requested resources.

Another example are tools that do not benefit from a increasing number of cores. Please check the documentation of your tools and also check the feedback files that report the CPU efficiency of your job.

[...]
CPU efficiency, 0-100%                      | 25.00
[...]

This job for example used only 25% of the available CPU resources.

Software

There are several mechanisms how software can be installed on BinAC. If you need software that is not installed on BinAC you can open a ticket and we can find a way to provide the software on the cluster.

Environment Modules

Environment modules is the 'classic' way for providing software on clusters. A module consists of a specific software version and can be loaded. The module system then manipulates the PATH and other environment variables such that the software can be used.

# Show available modules
$ module avail

# Load a module
$ module load bio/bowtie2/2.4.1

# Show the module's help
$ module help bio/bowtie2/2.4.1

A more detailed description of module environments can be found on this wiki page

Sometimes software packages have so many dependencies or the user wants a combination of tools, so that environment modules cannot be used in a meaningful way. Then other solutions like conda environments or Singularity container (see below) can be used.

Conda Environments

Conda environments is a nice possibility for creating custom environments on the cluster, as a majority of the scientific software is available in the meantime as conda packages. First, you have to install Miniconda in your home directory.

# Download installer
$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
$ sh Miniconda3-latest-Linux-x86_64.sh
$ source ~/.bashrc

Then you can create your first environment and install software into it:

# Create an environment
$ conda create --name my_first_conda_environment
# Activate this environment
conda activate my_first_conda_environment
# Install software into this environment
$ conda install scipy=1.5.2

You will need to add this line to your jobscripts such that the environments are available on the compute nodes:

source $HOME/miniconda3/etc/profile.d/conda.sh
conda activate <env_name>

When installing software conda will solve dependencies on the fly. But it is not guaranteed that conda will use the exact same package versions in the future. For the sake of reproducibility, you can write a file containing all conda packages together with their versions:

# Export packages installed in the active environment
$ conda list --explicit > spec-file.txt
# Create a new environment with the exact same conda packages
$ conda create --name myenv --file spec-file.txt

Singularity Container

Sometimes software is also available in a software container format. Singularity is installed on all BinAC nodes. You can pull Singularity or Docker containers from registries onto BinAC and use them. You can also build new Singularity containers on your own machine and copy them to BinAC.