View source for NEMO/Software/Singularity Containers

= Introduction =

Singularity is an open-source software for container-virtualization. Because not every different software configuration can be provided as [[Environment Modules|Modules]] on the clusters, containers offer a way to use a pre-built scientific software in a closed and reproducible space, independent from the environment. A Singularity container contains its own operating system, the intended software and all required dependencies, except for Kernel components (e.g. drivers). This also means that you can use software that isn't available for RHEL/CentOS, but is offered for other Linux systems. Singularity containers are easily movable between systems and do not require root-rights for execution (different to Docker). <br>

For example, a user may build a software package from a library or from source on their own computer, move it to the server (for example with scp) and execute it. This works as long as Singularity is installed on both systems, without having to deal with the environment on the cluster. <br>

The container generally works as its own closed-off environment. While you can access data and files stored on the host, you generally do not have access to software or modules running there. This means that you usually have to provide software that could otherwise be found in a module inside the container.<br>

The following tutorial gives a brief introduction to the program to create and run a container on a cluster. <br>

'''Building containers can be tricky. If you need help, please don't hesitate to contact [https://wiki.bwhpc.de/e/Category:Support us]. [https://wiki.bwhpc.de/e/Category:Support We can help you].'''

=Containers=

== Requirements to build a Container ==

Singularity requires a Linux-system. If no Linux computer is available, a virtual machine with a Linux-OS can be used. Singularity does ''not'' work on the Windows Subsystem for Linux (WSL). <br>

First, install Singularity 3 from source by following the instructions on the [https://sylabs.io/guides/3.0/user-guide/installation.html# official page]. Singularity 3 also requires the installation of the programming language Go. <br>

Singularity has to be installed on all systems trying to use the container, and has to be loaded with an appropriate module first on the cluster. <br>

== Building a Container ==

The command to build a new container is ''singularity build''. A container that should remain writable after building can be created with ''singularity build --sandbox''. A specific home directory can be defined by using ''singularity build --home /your/home/path/''.  <br>

Most manual building operations require root-rights, while importing existing containers can often be done without them. If a container has to be made up from scratch, it can be built manually in a writable shell or through a definition file. <br>

Singularity 3 supports two formats of containers: Singularity Image Files (''.sif''), the default option, and as a Directory.  <br>

If you are pulling a pre-made container directly on the cluster, use a workspace instead of your home-directory. Aside from being preferable in general, the configuration of the cluster might not allow a container to be built in your home to begin with.<br>

=== Building an existing container ===

If an existing container from Docker (''docker://'') or a container library (''library://'' or ''shub://'') should be built using Singularity, it can simply be imported as is. A writable sandbox container can be built by flagging --sandbox: <br>

<pre>
$ singularity build <containername> library://path/to/file
$ singularity build --sandbox <containername> library://path/to/file
</pre>

<br>

=== Building a new container with a definition file ===

A definition file, or recipe in older guides, provides the program with a script to build the container from. The same definition-file always reliably produces the same container, so it is encouraged to use definition-files for reproducibility if possible. <br>

The file has to contain a '''header''', specifying the operating system and its source. If one wants to use CentOS, for example: <br>

<pre>
Bootstrap: library
From: centos
</pre>

<br>

After this, various sections with specifications can follow in any order. The most important are '''%post''', '''%files''' and '''%environment'''. <br>

'''%post''' constitutes the main part of the file, with a set of bash-instructions used to build the container in order. For example, the process might start with an update and the installation of standard tools required later on: <br>

<pre>
%post
	yum -y install update
	yum -y install wget tar
	cd /
	mkdir example
</pre>

<br>

Keep in mind that the container contains almost no pre-installed packages, depending on the base image that is used. <br>

For package-managers and installations, make sure that prompts are automatically answered with "yes" (**-y** in the example), otherwise the automatic building will be aborted.<br>

'''%files''' can be used to access files on the home system and copy them inside the container at the start of building process. The syntax is simply ''filepath targetpath''. <br>

<pre>
%files
	your/file/path.f /dir
</pre>

<br>

'''%environment''' can be used to set environment-variables such as paths, for example: <br>

<pre>
%environment
	export PKG_CONFIG_PATH=/usr/bin
</pre>

<br>

Inputs into '''%environment''' from the definition-file are written to ''/.singularity.d/env/90-environment.sh'', which is sourced at runtime. In ''%post'', commands for starting the container can be written to that file or to ''/.singularity.d/env/91-environment.sh''. This can be useful if no '~/.bashrc' or similar is available. Note that information provided through ''%environment'' will ''not'' be available for the build-process, only for the completed container.  <br>

Some other options for the definition-file are '''%runtime''', which can include a script to be executed at the start of the ''run'' command, and '''%labels''' and '''%help''' to provide the user of the container with credentials and a metadata help-file, respectively. More options can be found on the [https://sylabs.io/guides/3.0/user-guide/definition_files.html Singularity homepage]. <br>

The file is invoked with <br>

<pre>
$ sudo singularity build <containername> definition-file.def
</pre>

<br>

The container is built in a temporary directory. If this needs to be changed from the default, ''--tmpdir your/tmp/dir'' can be used to specify a temporary directory.

<br>

=== Building a new container in the Shell ===

If no definition-file is a available, a container can be built by executing the same commands that would be put in the definition file manually in a shell. Effectively, this approach is the same as building an existing container, just with the intention to write into it immidiately. To do so, a container with the desired OS has to be built as a sandbox and opened as a writable shell: <br>

<pre>
$ sudo singularity build --sandbox <containername> library://centos
$ sudo singularity shell --writable <containername>
</pre>

<br>

This shell already behaves like its own OS. Note that if you took a base-container with only an OS, it contains almost no pre-installed packages, and that some paths (e.g. ''~/'') may still reference the host-system. In general, ''cd /'' gets you to the highest parent-folder within singularity. <br>

<pre>
$ yum -y install update
$ yum -y install wget tar
$ cd /
$ mkdir example
</pre>

<br>

== Using a container ==

To use a container on a cluster, an available Singularity-module has to be loaded. For example: <br>

<pre>
$ module load your/singularity/version
</pre>

<br>

Keep in mind that other modules you may have loaded (that aren't needed for running the container itself) will ''not'' be available inside the container. <br>

There are three key ways to run services on a regular container, the already known ''shell'', as well as ''exec'' and ''run''.  By default, all Singularity containers are read-only. To make them writable, ''--writable'' has to be specified in the command. <br>

Again, the container can be opened as a shell: <br>

<pre>
$ singularity shell <containername>
</pre>

<br>

On the cluster, the easiest way to run one-line commands (including commands to run other files) is passing them to the container through ''exec'': <br>

<pre>
$ singularity exec --writable <containername> yum -y install update
$ singularity exec <containername> script.sh
</pre>

<br>

''Exec'' is generally the easiest way to execute scripts. Without further specification, it runs the container in the current directory, making it a good option to run on workspaces, which Singularity might otherwise struggle with if run from ''$HOME''.

The third option, ''run'', allows to execute a '''%runscript''' that was provided in the definition file during the build-process: <br>

<pre>
$ singularity run <containername>
</pre>

This is for example useful to start an installed program.

<br>

=== Containers and Batch Jobs ===
[[Batch Jobs]] utilizing Singularity containers are generally built the same way as all other batch jobs, where the job script contains a ''singularity exec'' command. For example: <br>

<pre>
#!/bin/bash
#MSUB -l nodes=1:gpu
#MSUB -l walltime=3:00:00
#MSUB -l mem=5000mb
#MSUB -N Singularity

module load your/singularity/version
cd your/workspace
singularity exec --nv <containername> script1.sh
singularity exec --nv <containername> python script2.py
</pre>

<br>

=== Using GPUS ===

Containers can run on nvidia-GPUs as well. First, make sure that all necessary libraries (for example CUDA) are installed on the container itself and all appropriate drivers are available on the host system. To load the requirements from the host system, both ''exec'' and ''shell'' need another specification to be able to interact with the GPUs - the flag ''--nv'' after the command. <br>

<pre>
$ singularity exec --nv <containername> script.sh
</pre>

<br>

= Concluding notes =

* Keep track of your inputs when using a (writable) shell.
* Modules loaded outside the container don't work on the inside.
* Use ''singularity exec'' for Batch-Jobs.
* Use ''singularity exec --nv'' for (nvidia-)GPUs.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

----
[[Category:Scientific_Applications]]