NEMO2/CVMFS
CernVM-FS (CVMFS) is a read-only, HTTP-based file system used to distribute scientific software and reference data worldwide. On NEMO the client is preconfigured and available on all login and compute nodes — no setup or module loading is required.
How it works
All repositories live under /cvmfs. Mounting is handled by autofs:
- At first glance /cvmfs appears empty — nothing is mounted initially.
- A repository is mounted on demand, the first time you access it (e.g. ls /cvmfs/sft.cern.ch).
- The first access can take a few seconds while the mount is established; afterwards it behaves like a normal directory.
- Unused repositories are unmounted automatically after an idle timeout.
Files are not pre-downloaded: CVMFS fetches data on demand from Stratum-1 mirrors and caches it locally. The cache is shared between all repositories on a node and does not count against your home or workspace quota.
Available repositories
| Repository | Content | Relevant for |
|---|---|---|
| sft.cern.ch | CERN SFT: LCG software stacks and releases (HEP software, lcg/views setup scripts) | HEP analyses; also useful for anyone needing a complete, prebuilt scientific stack (ROOT, Python, Geant4, …) |
| atlas.cern.ch | ATLAS experiment software | ATLAS members |
| atlas-condb.cern.ch | ATLAS conditions databases | ATLAS members |
| atlas-nightlies.cern.ch | ATLAS nightly builds | ATLAS developers |
| cms.cern.ch | CMS experiment software (CMSSW) | CMS members |
| belle.cern.ch | Belle II software | Belle II members |
| belle.kek.jp | Belle II software (KEK mirror) | Belle II members |
| ilc.desy.de | ILC software (DESY) | ILC members |
| etp.kit.edu | Software from ETP, Karlsruhe Institute of Technology | ETP collaborations |
| data.galaxyproject.org | Galaxy Project reference data: genome sequences, tool indexes, .loc files | Mainly jobs from usegalaxy.eu; the data itself (genomes, indexes) is usable by anyone |
| idc.galaxyproject.org | Simon's Data Club: community-maintained reference data for pathogens and additional genomes | Mainly jobs from usegalaxy.eu; the data itself is usable by anyone |
| singularity.galaxyproject.org | Singularity/Apptainer containers for BioContainers tools (see Apptainer) | Everyone — prebuilt bioinformatics containers, independent of Galaxy |
| unpacked.cern.ch | Unpacked container images from various registries (Docker Hub, CERN GitLab, NGC, quay.io, …), startable directly with Apptainer | Everyone — HEP software stacks and other images without pulling |
Usage examples
LCG software stack (sft.cern.ch)
The LCG (LCG = "LHC Computing Grid") stack is CERN's common software distribution for particle physics. It bundles hundreds of packages — ROOT, Geant4, Python with scientific libraries, compilers, and more — built consistently against each other for several platform combinations (OS, compiler, optimization level).
The stack is organized in two layers:
- /cvmfs/sft.cern.ch/lcg/releases/ — the individual packages, each built for a specific platform (e.g. x86_64-el9-gcc13-opt: AlmaLinux 9, GCC 13, optimized).
- /cvmfs/sft.cern.ch/lcg/views/ — ready-to-use "views" of a release. A view is a curated, consistent set of packages with a setup.sh that sets all environment variables (PATH, LD_LIBRARY_PATH, PYTHONPATH, …). Sourcing a view gives you a complete working environment in one step — comparable to loading a module.
For a first-time user this means: pick a release number and a platform, source its view, and you have the full HEP software environment:
ls /cvmfs/sft.cern.ch/lcg/views # e.g. LCG 110 with x86_64-el9-gcc16-opt: source /cvmfs/sft.cern.ch/lcg/views/LCG_110/x86_64-el9-gcc16-opt/setup.sh
Which releases and platforms exist is listed at LCGinfo (releases, packages, platforms). The platform string encodes OS, compiler and build type — choose the one matching the NEMO OS (AlmaLinux/Rocky 9 → el9).
You can also use single packages from releases/ without sourcing a full view, e.g. ROOT. Note that the same version can exist in several builds, each with multiple platform directories — list them first:
ls /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.40.04-866e3/ # x86_64-el9-gcc16-opt/ ... source /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.40.04-866e3/x86_64-el9-gcc16-opt/bin/thisroot.sh
CMS (CMSSW)
source /cvmfs/cms.cern.ch/cmsset_default.sh
Galaxy reference data
Genome sequences and prebuilt tool indexes:
/cvmfs/data.galaxyproject.org/byhand/ # manually curated data, organized by genome build /cvmfs/data.galaxyproject.org/managed/ # data manager generated, organized by index type
This data is primarily used by jobs from usegalaxy.eu (the European Galaxy server, whose jobs run on NEMO). But the data itself is plain files — genome FASTAs, Bowtie2/BWA/HISAT2 indexes, Kraken2 databases, BLAST databases, etc. — and can be used directly by anyone, e.g.:
ls /cvmfs/data.galaxyproject.org/managed/bwa_mem_index/ ls /cvmfs/data.galaxyproject.org/managed/kraken2_databases/
Additional community-maintained data (pathogen databases, Vertebrate Genomes Project assemblies) are in /cvmfs/idc.galaxyproject.org/.
Apptainer containers from CVMFS
The singularity.galaxyproject.org repository hosts pre-built Apptainer/Singularity images for the BioContainers collection — bioinformatics tools packaged from Bioconda (the bioinformatics channel of conda-forge). Unlike the reference data above, these containers are not tied to Galaxy: anyone can run them directly with Apptainer — no pull, no build, no registry rate limits:
# find a container, e.g. for bwa: ls /cvmfs/singularity.galaxyproject.org/b/w/ apptainer exec /cvmfs/singularity.galaxyproject.org/b/w/bwa:0.7.17--he4a0461_10 bwa
Container paths are organized in two levels by the first two characters of the tool name (b/w/ for bwa, z/o/ for zol). Each file is named <tool>:<version>--<build>, mirroring the Bioconda package naming: the build string is the conda build identifier (e.g. he4a0461_10, or py312h9b2995c_0 for Python packages, where py312 indicates the Python version). Many versions and builds exist side by side, so you can pin an exact version for reproducibility.
To find the right container, look up the tool on Bioconda or BioContainers to see available versions, then list the matching files in the corresponding bucket directory.
HEP container images (unpacked.cern.ch)
CVMFS can also serve container images themselves. The repository unpacked.cern.ch (managed at CERN, publicly available) hosts unpacked container images — e.g. experiment software stacks from Docker Hub or CERN GitLab registries. Because the images are stored as unpacked root file systems, Apptainer can start them directly from CVMFS, without pulling or converting:
apptainer exec /cvmfs/unpacked.cern.ch/registry.hub.docker.com/library/centos\:centos7 /bin/bash
Advantages over pulling images from a registry: no registry rate limits, automatic deduplication of files shared between images, and automatic updates (images are re-published within ~15 minutes of an upstream change).
The image catalogue is community-driven: to get an image added, open a pull request against the wishlist on GitHub. See the CVMFS containers documentation for details.
Images are organized by source registry, e.g. registry.hub.docker.com/, gitlab-registry.cern.ch/, ghcr.io/, nvcr.io/, quay.io/:
ls /cvmfs/unpacked.cern.ch/registry.hub.docker.com/ apptainer exec /cvmfs/unpacked.cern.ch/registry.hub.docker.com/library/centos\:centos7 /bin/bash
Notes and limitations
- /cvmfs is read-only — you cannot write to it.
- The local cache is limited (30 GB per node, shared by all repositories). Very large datasets are fetched on demand; repeated access is fast because of the cache.
- The cache lives under /tmp on each node and is node-local: a file fetched on one compute node is not cached on another.
- Do not use /cvmfs paths as output locations — write results to your workspace or home directory.
- If a repository fails to mount or ls /cvmfs/<repo> hangs, retry after a moment; if the problem persists, contact NEMO2/Support.