BinAC/Quickstart Guide
File System Basics
The details of the file systems are explained here.
Home File System
Home directories are meant for permanent file storage of files that are keep being used like source codes, configuration files, executable programs, conda environments, etc. It is backuped daily and has a quota. If that quota is reached, you will usually experience problems when working on the cluster.
Work File System
Use the work file system and not your home directory for your calculations and data. Create a work directory, usually users use their username:
cd /beegfs/work/
mkdir <username>
cd <username>
Do not use the login nodes to carry out any calculations or heavy load file transfers.
You can share data with coworkers who are members in the same compute project by modifying file ownership and permissions on a directory in the work filesystem. Every compute project has its own group on BinAC, namely the project's acronym. In this example, the group (e.g. the project's acronym) for user tu_iioba01 is bw16f003.
$ id tu_iioba01
uid=900102(tu_iioba01) gid=500001(tu_tu) groups=500001(tu_tu),500002(bw16f003)
In order to share data with coworkers in the same compute project, change the group of the directory you want to share to the project acronym. You can also set the so-called SGID-Bit. Then new files and subdirectories will automatically belong to this group.
# Change ownership. This command changes the ownwership for ALL files and subdirectories.
$ chown -R <username>:<acronym> /beegfs/work/<your directory>
# Set SGID-Bit
$ chmod g+s /beegfs/work/<your directory>
# Show the directory's permissions and owner
$ ls -ld /beegfs/work/<your directory>
drwxr-sr-x 3 tu_iioba01 bw16f003 1 Apr 16 15:06 /beegfs/work/<your directory>
The group of the directory is now bw16f003. The permissions for the group is 'r-s'. This means members of the group can access the directory content and newly created subdirectories/files automatically belong to the group bw16f003.
Now you can set the file permissions for your coworkers in this directory as wanted by granting read, write, and execute permissions to files and subdirectories.
# Some examples
# Coworkers can read,write,delete,execute files in the directory
$ chmod 770 /beegfs/work/<your directory>
# Coworkers can read, owner still can write/delete files
$ chmod 740 /beegfs/work/<your directory>/<important_dataset>
For a more detailed explanation of file permissions, please refer to this tutorial
Temporary Data
If your job creates temporary data, you can use the fast SSD with a capacity of 211 GB on the compute nodes. The temporary directory for your job is available via the $TMPDIR environment variable.
Batch System Basics
On cluster systems like BinAC you do not run your analysis by hand on the login node. Instead, you write a script and submit it to the batch system, this is called a job. The batch systems then tries to schedule the jobs on the available compute nodes.
Queue/Job Basics
The cluster consists of compute nodes with different hardware features. These hardware features (e.g. high-mem or GPUs) are only available when submitting the jobs to the specific queue. Also, each queue has different settings regarding maximal walltime. The most recent queue settings are displayed on login as message of the day on the terminal.
Get an overview of the number of running and queued jobs:
$ qstat -q
Queue Memory CPU Time Walltime Node Run Que Lm State
---------------- ------ -------- -------- ---- --- --- -- -----
tiny -- -- -- -- 0 0 -- E R
long -- -- -- -- 850 0 -- E R
gpu -- -- -- -- 66 0 -- E R
smp -- -- -- -- 4 1 -- E R
short -- -- -- -- 131 90 -- E R
----- -----
1051 91
To check all running and queued jobs:
qstat
Just your own jobs.
qstat -u <username>
Interactive Jobs
Interactive jobs are a good method for testing if/how software works with your data.
To start a 1 core job on a compute node providing a remote shell.
qsub -q short -l nodes=1:ppn=1 -I
The same but requesting the whole node.
qsub -q short -l nodes=1:ppn=28 -I
Standard Unix commands are directly available, for everything else use the modules.
module avail
Be aware that we allow node sharing. Do not disturb the calculations of other users.
Simple Script Job
Use your favourite text editor to create a script called 'script.sh'.
 |
Please note that there are differences between Windows and Linux line endings.
Make sure that your editor uses Linux line endings when you are using Windows.
You can check your line endings with |
#PBS -l nodes=1:ppn=1
#PBS -l walltime=00:05:00
#PBS -l mem=1gb
#PBS -S /bin/bash
#PBS -N Simple_Script_Job
#PBS -j oe
#PBS -o LOG
cd $PBS_O_WORKDIR
echo "my Username is:"
whoami
echo "My job is running on node:"
uname -a
Submit the job using
qsub -q tiny script.sh
Take a note of your jobID. The scheduler will reserve one core and 1 gigabyte of memory for 5 minutes on a compute node for your job. The job should be scheduled within minute if the tiny queue is empty and write your username and the execution node into the output file.
There are tons of options, details and caveats. Most of the options are explained on this page, but be aware that there are some differences on BinAC.
If your job needs GPUs, you have to specify how many GPUs you want. Just submitting the job to the GPU queue does not work:
#PBS -l nodes=1:ppn=1:gpus=1
#PBS -q gpu
If you encounter any problems, just send a mail to hpcmaster@uni-tuebingen.de.
Killing a Job
Let's assume you build a Homer and want to stop/kill/remove a running job.
qdel <jobID>
Best Practices
The scheduler will reserve computational resources (nodes, cores, gpus, memory) for a specified period for you. By following some best practices, you can avoid some common problems beforehand.
Specify memory for your job
Often we get tickets with question like "Why did the system kill my job?". Most often the user did not specify the required memory resources for the job. Then the following happens:
The job is started on a compute node, where it shares the resources other jobs. Let us assume that the other jobs on this node occupy already 100 gigabyte of memory. Now your job tries to allocate 40 gigabyte of memory. As the compute node has only 128 gigabyte, your job crashes because it cannot allocate that much memory.
You can make your life easier by specifying the required memory in your job script with:
#PBS -l mem=xxgb
Then you have the guarantee that your job can allocate xx gigabyte of memory.
If you do not know how much memory your job will need, look into the documentation of the tools you use or ask us. We also started a wiki page on which we will document some guidelines and pitfalls for specific tools.
Use the reserved resources
Reserved resources (nodes, cores, gpus, memory) are not available to other users and their jobs. You have the responsibility that your programs utilize the reserved resources.
An extreme example: You request a whole node (node=1:ppn=28), but your job uses just one core. The other 27 cores are idling. This is bad practice, so take care that the used programs really use the requested resources.
Another example are tools that do not benefit from a increasing number of cores. Please check the documentation of your tools and also check the feedback files that report the CPU efficiency of your job.
[...]
CPU efficiency, 0-100% | 25.00
[...]
This job for example used only 25% of the available CPU resources.
Software
There are several mechanisms how software can be installed on BinAC. If you need software that is not installed on BinAC you can open a ticket and we can find a way to provide the software on the cluster.
Environment Modules
Environment modules is the 'classic' way for providing software on clusters. A module consists of a specific software version and can be loaded. The module system then manipulates the PATH and other environment variables such that the software can be used.
# Show available modules
$ module avail
# Load a module
$ module load bio/bowtie2/2.4.1
# Show the module's help
$ module help bio/bowtie2/2.4.1
A more detailed description of module environments can be found on this wiki page
Sometimes software packages have so many dependencies or the user wants a combination of tools, so that environment modules cannot be used in a meaningful way. Then other solutions like conda environments or Singularity container (see below) can be used.
Conda Environments
Conda environments is a nice possibility for creating custom environments on the cluster, as a majority of the scientific software is available in the meantime as conda packages. First, you have to install Miniconda in your home directory.
# Download installer
$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
$ sh Miniconda3-latest-Linux-x86_64.sh
$ source ~/.bashrc
Then you can create your first environment and install software into it:
# Create an environment
$ conda create --name my_first_conda_environment
# Activate this environment
conda activate my_first_conda_environment
# Install software into this environment
$ conda install scipy=1.5.2
You will need to add this line to your jobscripts such that the environments are available on the compute nodes:
source $HOME/miniconda3/etc/profile.d/conda.sh
conda activate <env_name>
When installing software conda will solve dependencies on the fly. But it is not guaranteed that conda will use the exact same package versions in the future. For the sake of reproducibility, you can write a file containing all conda packages together with their versions:
# Export packages installed in the active environment
$ conda list --explicit > spec-file.txt
# Create a new environment with the exact same conda packages
$ conda create --name myenv --file spec-file.txt
Singularity Container
Sometimes software is also available in a software container format. Singularity is installed on all BinAC nodes. You can pull Singularity or Docker containers from registries onto BinAC and use them. You can also build new Singularity containers on your own machine and copy them to BinAC.
Please note that Singularity containers should be stored in the work file system. We configured Singularity such that containers stored in your home directory do not work.