M Janczyk: Created page with "'''Enroot''' is a container runtime developed by NVIDIA that runs OCI/Docker containers without root privileges. On NEMO it is the '''recommended''' container solution and is integrated with Slurm via the '''Pyxis''' SPANK plugin. == How it works == Enroot converts a Docker image into a SquashFS file (`.sqsh`), unpacks it on demand into an overlay filesystem, and runs your workload inside that environment. The Pyxis plugin adds `--container-*` options to..."

2026-05-07T17:05:11Z

Created page with "'''Enroot''' is a container runtime developed by NVIDIA that runs OCI/Docker containers without root privileges. On NEMO it is the '''recommended''' container solution and is integrated with Slurm via the '''Pyxis''' SPANK plugin. == How it works == Enroot converts a Docker image into a SquashFS file (<tt>.sqsh</tt>), unpacks it on demand into an overlay filesystem, and runs your workload inside that environment. The Pyxis plugin adds <tt>--container-*</tt> options to..."

New page

'''Enroot''' is a container runtime developed by NVIDIA that runs OCI/Docker containers without root privileges.
On NEMO it is the '''recommended''' container solution and is integrated with Slurm via the '''Pyxis''' SPANK plugin.

== How it works ==

Enroot converts a Docker image into a SquashFS file (<tt>.sqsh</tt>), unpacks it on demand into an overlay filesystem, and runs your workload inside that environment.
The Pyxis plugin adds <tt>--container-*</tt> options to <tt>srun</tt>/<tt>salloc</tt>/<tt>sbatch</tt> so containers become first-class Slurm jobs.

== Default mounts ==

The following paths are '''automatically mounted''' into every container when launched via Slurm/Pyxis:

{| class="wikitable"
|-
! Host path !! Notes
|-
| <tt>/home</tt> || all home directories, read-write
|-
| <tt>/work</tt> || all workspace filesystems, read-write
|-
| <tt>/etc/slurm</tt> || Slurm configuration (read-only)
|-
| <tt>/usr/lib64/slurm</tt> || Slurm plug-in libraries (read-only)
|}

You do '''not''' need to pass <tt>--container-mount-home</tt> or <tt>--container-mounts=/work</tt> manually — they are already there.
The paths inside the container are '''identical''' to the paths on the host system, so scripts and config files referencing <tt>$HOME</tt> or workspace paths work without modification.

Note that <tt>ws_*</tt> tools (e.g. <tt>ws_find</tt>, <tt>ws_list</tt>) are '''not available inside the container'''.
Determine your workspace path on the login node before submitting the job and pass it as an environment variable or hard-code it in your script.

== Container image storage ==

Unpacked container images are stored in:

<pre>
~/.local/share/enroot/
</pre>

Images are SquashFS files and can be several GB.
To avoid filling your home quota, store images in a [[Workspaces|workspace]] and symlink the default path:

<pre>
# create a workspace (100 days)
ws_allocate enroot 100

# replace the default enroot directory with a symlink to the workspace
mkdir -p ~/.local/share
ln -s $(ws_find enroot) ~/.local/share/enroot
</pre>

Enroot (and Pyxis) will now transparently use the workspace path for all image storage.

== Usage without Slurm (interactive) ==

=== Import an image ===

<pre>
# from Docker Hub
enroot import docker://ubuntu:24.04

# from quay.io
enroot import docker://quay.io#rockylinux/rockylinux:9

# from NVIDIA NGC
enroot import docker://nvcr.io#nvidia/pytorch:24.01-py3
</pre>

This creates a <tt>.sqsh</tt> file in the current directory.

=== Create a container ===

The container name must be prefixed with <tt>pyxis_</tt> for Slurm/Pyxis to find it later (omit the prefix when passing <tt>--container-name</tt> to Slurm).

<pre>
enroot create --name pyxis_ubuntu ubuntu+24.04.sqsh
</pre>

=== Start a container ===

<pre>
# interactive shell, read-write
enroot start --rw pyxis_ubuntu bash

# get root inside the container (to install packages)
enroot start --root --rw pyxis_ubuntu bash

# mount an extra directory (e.g. from outside /home and /work)
enroot start --rw -m /tmp/mydata:/data pyxis_ubuntu bash
</pre>

=== List and remove containers ===

<pre>
enroot list --fancy
enroot remove pyxis_ubuntu
</pre>

== Usage via Slurm / Pyxis ==

=== Interactive allocation ===

<pre>
# use an already-created container
salloc -p cpu --container-name=ubuntu

# pull, create and start in one step (container is created under ~/.local/share/enroot/)
salloc -p cpu --container-image=ubuntu:24.04 --container-name=ubuntu
salloc -p cpu --container-image="quay.io#rockylinux/rockylinux:9" --container-name=rocky
salloc -p l40s --gres=gpu:1 --container-image="nvcr.io#nvidia/pytorch:24.01-py3" --container-name=pytorch

# start with a specific working directory inside the container
# $(ws_find enroot) is evaluated on the login node before the job starts
salloc -p cpu --container-name=ubuntu --container-workdir=$(ws_find enroot)
</pre>

=== Batch job ===

<pre lang="bash">
#!/bin/bash
#SBATCH -p cpu
#SBATCH --container-name=ubuntu

python3 /work/classic/myWs/train.py
</pre>

=== Useful Pyxis options ===

All options are listed in <tt>srun --help</tt> under ''Options provided by plugins''.

{| class="wikitable"
|-
! Option !! Effect
|-
| <tt>--container-name=NAME</tt> || Use existing enroot container (omit the <tt>pyxis_</tt> prefix)
|-
| <tt>--container-image=IMAGE</tt> || Pull image from registry and create container on the fly
|-
| <tt>--container-mount-home</tt> || Mount <tt>$HOME</tt> into container (already in defaults, but explicit flag also works)
|-
| <tt>--container-mounts=SRC:DST[,…]</tt> || Bind-mount additional paths
|-
| <tt>--container-writable</tt> || Make the container overlay writable
|-
| <tt>--container-remap-root</tt> || Become root inside the container (no real root on host)
|-
| <tt>--container-workdir=PATH</tt> || Set the working directory inside the container
|}

== GPU access ==

GPU passthrough is '''automatic''' — no extra flags are needed.
Enroot/Pyxis detect the allocated GPUs via Slurm's GRES mechanism and make them available inside the container.

<pre>
salloc -p l40s --gres=gpu:1 --container-name=pytorch
# nvidia-smi works inside the container out of the box
</pre>

== Tips ==

* Install extra packages interactively with <tt>enroot start --root --rw</tt> before submitting batch jobs.
* Use <tt>--container-writable</tt> in batch jobs only if your script modifies the container filesystem; otherwise the default overlay is discarded after the job anyway.
* Store large datasets in workspaces (<tt>/work</tt>), not in <tt>$HOME</tt>, to avoid filling your home quota with data files.

NEMO2/Containers/Enroot - Revision history