M Janczyk: Created page with "'''CernVM-FS''' (CVMFS) is a read-only, HTTP-based file system used to distribute scientific software and reference data worldwide. On NEMO the client is preconfigured and available on all login and compute nodes — no setup or module loading is required. == How it works == All repositories live under `/cvmfs`. Mounting is handled by '''autofs''': * At first glance `/cvmfs` appears '''empty''' — nothing is mounted initially. * A repository is mounted..."

2026-10-20T13:01:38Z

Created page with "'''CernVM-FS''' (CVMFS) is a read-only, HTTP-based file system used to distribute scientific software and reference data worldwide. On NEMO the client is preconfigured and available on all login and compute nodes — no setup or module loading is required. == How it works == All repositories live under <tt>/cvmfs</tt>. Mounting is handled by '''autofs''': * At first glance <tt>/cvmfs</tt> appears '''empty''' — nothing is mounted initially. * A repository is mounted..."

New page

'''CernVM-FS''' (CVMFS) is a read-only, HTTP-based file system used to distribute scientific software and reference data worldwide. On NEMO the client is preconfigured and available on all login and compute nodes — no setup or module loading is required.

== How it works ==

All repositories live under <tt>/cvmfs</tt>. Mounting is handled by '''autofs''':

* At first glance <tt>/cvmfs</tt> appears '''empty''' — nothing is mounted initially.
* A repository is mounted '''on demand''', the first time you access it (e.g. <tt>ls /cvmfs/sft.cern.ch</tt>).
* The first access can take a few seconds while the mount is established; afterwards it behaves like a normal directory.
* Unused repositories are unmounted automatically after an idle timeout.

Files are not pre-downloaded: CVMFS fetches data on demand from Stratum-1 mirrors and caches it locally. The cache is shared between all repositories on a node and does '''not''' count against your home or workspace quota.

== Available repositories ==

{| class="wikitable"
|-
! Repository !! Content !! Relevant for
|-
| <tt>sft.cern.ch</tt> || CERN SFT: LCG software stacks and releases (HEP software, <tt>lcg/views</tt> setup scripts) || HEP analyses; also useful for anyone needing a complete, prebuilt scientific stack (ROOT, Python, Geant4, …)
|-
| <tt>atlas.cern.ch</tt> || ATLAS experiment software || ATLAS members
|-
| <tt>atlas-condb.cern.ch</tt> || ATLAS conditions databases || ATLAS members
|-
| <tt>atlas-nightlies.cern.ch</tt> || ATLAS nightly builds || ATLAS developers
|-
| <tt>cms.cern.ch</tt> || CMS experiment software (CMSSW) || CMS members
|-
| <tt>belle.cern.ch</tt> || Belle II software || Belle II members
|-
| <tt>belle.kek.jp</tt> || Belle II software (KEK mirror) || Belle II members
|-
| <tt>ilc.desy.de</tt> || ILC software (DESY) || ILC members
|-
| <tt>etp.kit.edu</tt> || Software from ETP, Karlsruhe Institute of Technology || ETP collaborations
|-
| <tt>data.galaxyproject.org</tt> || Galaxy Project reference data: genome sequences, tool indexes, <tt>.loc</tt> files || Mainly jobs from [https://usegalaxy.eu usegalaxy.eu]; the data itself (genomes, indexes) is usable by anyone
|-
| <tt>idc.galaxyproject.org</tt> || Simon's Data Club: community-maintained reference data for pathogens and additional genomes || Mainly jobs from [https://usegalaxy.eu usegalaxy.eu]; the data itself is usable by anyone
|-
| <tt>singularity.galaxyproject.org</tt> || Singularity/Apptainer containers for BioContainers tools (see [[NEMO2/Containers/Apptainer|Apptainer]]) || '''Everyone''' — prebuilt bioinformatics containers, independent of Galaxy
|-
| <tt>unpacked.cern.ch</tt> || Unpacked container images from various registries (Docker Hub, CERN GitLab, NGC, quay.io, …), startable directly with Apptainer || '''Everyone''' — HEP software stacks and other images without pulling
|}

== Usage examples ==

=== LCG software stack (sft.cern.ch) ===

The LCG (LCG = "LHC Computing Grid") stack is CERN's common software distribution for particle physics. It bundles hundreds of packages — ROOT, Geant4, Python with scientific libraries, compilers, and more — built consistently against each other for several platform combinations (OS, compiler, optimization level).

The stack is organized in two layers:

* <tt>/cvmfs/sft.cern.ch/lcg/releases/</tt> — the individual packages, each built for a specific platform (e.g. <tt>x86_64-el9-gcc13-opt</tt>: AlmaLinux 9, GCC 13, optimized).
* <tt>/cvmfs/sft.cern.ch/lcg/views/</tt> — ready-to-use "views" of a release. A view is a curated, consistent set of packages with a <tt>setup.sh</tt> that sets all environment variables (PATH, LD_LIBRARY_PATH, PYTHONPATH, …). Sourcing a view gives you a complete working environment in one step — comparable to loading a module.

For a first-time user this means: pick a release number and a platform, source its view, and you have the full HEP software environment:

<pre>
ls /cvmfs/sft.cern.ch/lcg/views

# e.g. LCG 110 with x86_64-el9-gcc16-opt:
source /cvmfs/sft.cern.ch/lcg/views/LCG_110/x86_64-el9-gcc16-opt/setup.sh
</pre>

Which releases and platforms exist is listed at [https://lcginfo.cern.ch/ LCGinfo] (releases, packages, platforms). The platform string encodes OS, compiler and build type — choose the one matching the NEMO OS (AlmaLinux/Rocky 9 → <tt>el9</tt>).

You can also use single packages from <tt>releases/</tt> without sourcing a full view, e.g. ROOT. Note that the same version can exist in several builds, each with multiple platform directories — list them first:

<pre>
ls /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.40.04-866e3/
# x86_64-el9-gcc16-opt/ ...

source /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.40.04-866e3/x86_64-el9-gcc16-opt/bin/thisroot.sh
</pre>

=== CMS (CMSSW) ===

<pre>
source /cvmfs/cms.cern.ch/cmsset_default.sh
</pre>

=== Galaxy reference data ===

Genome sequences and prebuilt tool indexes:

<pre>
/cvmfs/data.galaxyproject.org/byhand/ # manually curated data, organized by genome build
/cvmfs/data.galaxyproject.org/managed/ # data manager generated, organized by index type
</pre>

This data is primarily used by jobs from [https://usegalaxy.eu usegalaxy.eu] (the European Galaxy server, whose jobs run on NEMO). But the data itself is plain files — genome FASTAs, Bowtie2/BWA/HISAT2 indexes, Kraken2 databases, BLAST databases, etc. — and can be used directly by anyone, e.g.:

<pre>
ls /cvmfs/data.galaxyproject.org/managed/bwa_mem_index/
ls /cvmfs/data.galaxyproject.org/managed/kraken2_databases/
</pre>

Additional community-maintained data (pathogen databases, Vertebrate Genomes Project assemblies) are in <tt>/cvmfs/idc.galaxyproject.org/</tt>.

=== Apptainer containers from CVMFS ===

The <tt>singularity.galaxyproject.org</tt> repository hosts pre-built Apptainer/Singularity images for the [https://biocontainers.pro/ BioContainers] collection — bioinformatics tools packaged from [https://bioconda.github.io/ Bioconda] (the bioinformatics channel of conda-forge). Unlike the reference data above, these containers are '''not''' tied to Galaxy: anyone can run them directly with [[NEMO2/Containers/Apptainer|Apptainer]] — no pull, no build, no registry rate limits:

<pre>
# find a container, e.g. for bwa:
ls /cvmfs/singularity.galaxyproject.org/b/w/

apptainer exec /cvmfs/singularity.galaxyproject.org/b/w/bwa:0.7.17--he4a0461_10 bwa
</pre>

Container paths are organized in two levels by the first two characters of the tool name (<tt>b/w/</tt> for <tt>bwa</tt>, <tt>z/o/</tt> for <tt>zol</tt>). Each file is named <tt><tool>:<version>--<build></tt>, mirroring the Bioconda package naming: the build string is the conda build identifier (e.g. <tt>he4a0461_10</tt>, or <tt>py312h9b2995c_0</tt> for Python packages, where <tt>py312</tt> indicates the Python version). Many versions and builds exist side by side, so you can pin an exact version for reproducibility.

To find the right container, look up the tool on [https://bioconda.github.io/ Bioconda] or [https://biocontainers.pro/ BioContainers] to see available versions, then list the matching files in the corresponding bucket directory.

=== HEP container images (unpacked.cern.ch) ===

CVMFS can also serve '''container images themselves'''. The repository <tt>unpacked.cern.ch</tt> (managed at CERN, publicly available) hosts unpacked container images — e.g. experiment software stacks from Docker Hub or CERN GitLab registries. Because the images are stored as unpacked root file systems, Apptainer can start them directly from CVMFS, without pulling or converting:

<pre>
apptainer exec /cvmfs/unpacked.cern.ch/registry.hub.docker.com/library/centos\:centos7 /bin/bash
</pre>

Advantages over pulling images from a registry: no registry rate limits, automatic deduplication of files shared between images, and automatic updates (images are re-published within ~15 minutes of an upstream change).

The image catalogue is community-driven: to get an image added, open a pull request against the [https://github.com/cvmfs/images-unpacked.cern.ch/blob/master/recipe.yaml wishlist on GitHub]. See the [https://cvmfs.readthedocs.io/en/stable/cpt-containers/ CVMFS containers documentation] for details.

Images are organized by source registry, e.g. <tt>registry.hub.docker.com/</tt>, <tt>gitlab-registry.cern.ch/</tt>, <tt>ghcr.io/</tt>, <tt>nvcr.io/</tt>, <tt>quay.io/</tt>:

<pre>
ls /cvmfs/unpacked.cern.ch/registry.hub.docker.com/

apptainer exec /cvmfs/unpacked.cern.ch/registry.hub.docker.com/library/centos\:centos7 /bin/bash
</pre>

== Notes and limitations ==

* <tt>/cvmfs</tt> is '''read-only''' — you cannot write to it.
* The local cache is limited (30 GB per node, shared by all repositories). Very large datasets are fetched on demand; repeated access is fast because of the cache.
* The cache lives under <tt>/tmp</tt> on each node and is node-local: a file fetched on one compute node is not cached on another.
* Do not use <tt>/cvmfs</tt> paths as output locations — write results to your [[NEMO2/Workspaces|workspace]] or home directory.
* If a repository fails to mount or <tt>ls /cvmfs/<repo></tt> hangs, retry after a moment; if the problem persists, contact [[NEMO2/Support]].

== Further documentation ==

* [https://cvmfs.readthedocs.io/ CernVM-FS official documentation]
* [https://cvmfs.readthedocs.io/en/stable/cpt-quickstart.html CVMFS Quick Start]
* [https://galaxyproject.org/admin/reference-data-repo/ Galaxy Project reference data]
* [https://github.com/galaxyproject/idc Simon's Data Club (idc)]

NEMO2/CVMFS - Revision history