JUSTUS2/Software/Singularity

Description	Content
module load	---
Availability	BwForCluster_JUSTUS_2
License	Open-source software, distributed under the 3-clause BSD License. More...
Citing	---
Links	Homepage | Documentation
Graphical Interface	No

1 Description

Singularity is a container platform.

2 License

Singularity is free, open-source software released under the 3-clause BSD license. Please read the license for additional information about Singularity.

3 Usage

3.1 Loading the module

You can load the default version of Singularity with the following command:

$ module load devel/singularity

If you wish to load another (older) version of Singularity, you can do so using

$ module load devel/singularity/<version>

with <version> specifying the desired version.

Important: On JUSTUS 2, Singularity is automatically available on all compute nodes. You do not have to load a module.

3.2 Batch jobs with containers

Batch jobs utilizing Singularity containers are generally built the same way as all other batch jobs, where the job script contains singularity commands. For example:

#!/bin/bash
# Allocate one node
#SBATCH --nodes=1
# Number of program instances to be executed
#SBATCH --tasks-per-node=4
# 8 GB memory required per node
#SBATCH --mem=16G
# Maximum run time of job
#SBATCH --time=1:00:00
# Give job a reasonable name
#SBATCH --job-name=Singularity
# File name for standard output (%j will be replaced by job id)
#SBATCH --output=singularity_job-%j.out
# File name for error output
#SBATCH --error=singularity_job-%j.err

module load your/singularity/version #(not needed on JUSTUS 2, but could be necessary on other system)

cd your/workspace

# Run container (two options to start a container)
singularity run [options] <container>
singularity exec [options] <container> <command>

3.2.1 Using GPUs

#!/bin/bash
[…]
# Allocate one GPU per node
#SBATCH --partition=gpu
#SBATCH --gres=gpu:1
[…]

module load your/singularity/version #(not needed on JUSTUS 2, but could be necessary on other system)

cd your/workspace

# Run container (two options to start a container)
singularity run --nv [options] <container>
singularity exec --nv [options] <container> <command>

Using the flag is advisable, but may be omitted if the correct GPU- and driver-APIs are available on the container.

4 Examples

4.1 Run your first container on JUSTUS 2

Build a TensorFlow container with Singularity and execute a Python command:

# request interactive node with GPUs
$ srun --nodes=1 --exclusive --gres=gpu:2 --pty bash

# create workspace and navigate into it
$ WORKSPACE=`ws_allocate tensorflow 3`
$ cd $WORKSPACE

# build container
$ singularity build tensorflow-20.11-tf2-py3.sif docker://nvcr.io/nvidia/tensorflow:20.11-tf2-py3

# execute Python command
$ singularity exec --nv tensorflow-20.11-tf2-py3.sif python -c 'import tensorflow as tf; \
  print("Num GPUs Available: ",len(tf.config.experimental.list_physical_devices("GPU")))'

Note: Ready-to-use containers can be pulled from the NVIDIA GPU CLOUD (NGC) catalog.

4.2 NGC environment modules on JUSTUS 2

1) Prepare a workspace to store the container

$ srun --nodes=1 --exclusive --gres=gpu:2 --pty bash
$ WORKSPACE=`ws_allocate npc 3`
$ cd $WORKSPACE
$ export NGC_IMAGE_DIR=$(pwd)

Important: Containers can only run in workspaces.

2) PyTorch container

$ module load ngc/.numlib
$ module load 20.12-py3
$ python3
>>> import torch
>>> x = torch.randn(2,3)
>>> print(x)
>>> quit()
$ module unload 20.12-py3
$ module unload ngc/.numlib

Note: Use the container in the same manner as an interactive shell.

3) LAMMPS container

$ module load ngc/.chem
$ module load 29Oct2020
$ wget https://lammps.sandia.gov/inputs/in.lj.txt
$ export SINGULARITY_BINDPATH=$(pwd)
$ mpirun -n 2 lmp -in in.lj.txt -var x 8 -var y 8 -var z 8 -k on g 2 -sf kk -pk kokkos cuda/aware on neigh full \
  comm device binsize 2.8
$ module unload 29Oct2020
$ module unload ngc/.chem

Note: Use SINGULARITY_BINDPATH=<PATH> to mount the directory with the input file.

Currently, module load ngc/.numlib for numeric libraries and module load ngc/.chem for chemistry programs can be selected.

4.3 Batch jobs with containers on JUSTUS 2

Run a GROMACS container with Singularity as a batch job:

$ WORKSPACE=`ws_allocate gromacs 3`                                         # allocate workspace
$ cd $WORKSPACE                                                             # change to workspace
$ singularity pull gromacs-2020_2.sif docker://nvcr.io/hpc/gromacs:2020.2   # pull container from NGC
$ cp -r /opt/bwhpc/common/chem/ngc/gromacs/ ./bwhpc-examples/               # copy example to workspace
$ cd ./bwhpc-examples                                                       # change to example directory
$ sbatch gromacs-2020.2_gpu.slurm                                           # submit job
$ squeue                                                                    # obtain JOBID
$ scontrol show job <JOBID>                                                 # check state of job

More batch job examples are located at /opt/bwhpc/common/chem/ngc.

5 Useful links

• Documentation (english)
• Wikipedia article (german)
• Wikipedia article (english)

---