NEMO2/Easybuild Modules

EasyBuild on NEMO2

EasyBuild is a software build and installation framework designed to manage scientific software on High-Performance Computing (HPC) systems. It automates the process of building and installing software, ensuring reproducibility and consistency across environments. EasyBuild uses "easyconfig" files (with .eb extension) to define build parameters, dependencies, and configurations, simplifying the management of complex software stacks.

NEMO2 uses EasyBuild exclusively to manage its software modules. All available software on NEMO2 is provided as EasyBuild modules.

Resources

• Building your own modules: NEMO2/Easybuild_Modules/EB_Build_Module - Complete guide for the eb-build-module.sh script
• Available modules: NEMO EasyBuild Modules List
• EasyBuild official website: https://easybuild.io/
• EasyBuild documentation: https://docs.easybuild.io/
• EasyBuild GitHub: https://github.com/easybuilders/easybuild

Understanding Architectures on NEMO2

NEMO2 has different CPU and GPU hardware architectures, and software modules are compiled specifically for each architecture to achieve optimal performance (see NEMO2/Hardware for hardware details).

Available Architecture Modules

You can see all available architectures by running:

module avail arch

This will show five architecture options:

• arch/milan - AMD EPYC Milan CPUs
• arch/genoa - AMD EPYC Genoa CPUs
• arch/mi300a - AMD MI300A APU (Accelerated Processing Unit - combined CPU+GPU)
• arch/l40s - NVIDIA L40S GPUs (Intel-based nodes)
• arch/h200 - NVIDIA H200 GPUs

Physical vs. Symbolic Architectures

Important: While five architecture modules are available, there are only three physical architectures on NEMO2:

Architecture Module	Physical Hardware	Type	Notes
milan	AMD EPYC Milan CPUs	Physical	Separate installation directory
genoa	AMD EPYC Genoa CPUs	Physical	Separate installation directory
mi300a	AMD MI300A APU (CPU+GPU)	Symbolic link → genoa	Uses genoa installation directory
l40s	Intel CPUs + NVIDIA L40S GPUs	Physical	Separate installation directory
h200	AMD Genoa CPUs + NVIDIA H200 GPUs	Symbolic link → genoa	Uses genoa installation directory

What this means: When you build or use modules for mi300a or h200, they are actually stored in the genoa directory. This is because these GPU nodes use AMD Genoa CPUs as their host processors.

Viewing Modules for Specific Architectures

By default, the login nodes use Genoa CPUs and display only Genoa modules. To see modules for a different architecture:

# Load a specific architecture
module load arch/milan

# Now module avail will show milan modules
module avail

The $ARCH environment variable is automatically set when you load an architecture:

module load arch/genoa
echo $ARCH  # Output: genoa

Architecture Compatibility

Important for beginners: Different architectures may not be compatible with each other. Software compiled for one architecture may not work optimally (or at all) on another.

General compatibility rules:

• milan → genoa: Software compiled for Milan will usually work on Genoa (but may not be optimally tuned)
• genoa → milan: Software compiled for Genoa may not work on Milan
• GPU architectures: Always use architecture-specific builds for GPU software

When in doubt, build separate versions for each architecture you plan to use.

Building Your Own EasyBuild Modules

If you need software that is not yet available as a module on NEMO2, you can build your own EasyBuild modules. NEMO2 provides the eb-build-module.sh script that makes this process straightforward.

For complete documentation, see: NEMO2/Easybuild_Modules/EB_Build_Module

Why Build Your Own Modules?

• Software you need is not available in the system-wide modules
• You need a different version than what's provided
• You want to customize build options or dependencies
• You're testing software before requesting a system-wide installation

Quick Start Guide

Step 1: Search for available easyconfig files (these define how to build software):

eb-build-module.sh -s Python

Step 2: Preview what will be built (dry-run - highly recommended for beginners):

eb-build-module.sh -d -m Python-3.11.3-GCCcore-12.3.0

This shows all dependencies that will be built without actually building anything.

Step 3: Build the module with default settings (20 cores, 12 hours, genoa architecture):

eb-build-module.sh -m Python-3.11.3-GCCcore-12.3.0

Step 4: Use your newly built module:

module load arch/genoa  # If not already loaded
module avail python     # Your module will appear here (lowercase)
module load lang/python/3.11.3-gcccore-12.3.0

Note: Module names are lowercase due to NEMO2's naming scheme, even though you build with Python-3.11.3-GCCcore-12.3.0.eb.

Building for Different Architectures

To build for a specific architecture (important if you'll run jobs on specific hardware):

# For Genoa CPUs
eb-build-module.sh -a genoa -m Python-3.11.3-GCCcore-12.3.0

# For MI300A APUs (uses genoa as base, stored in genoa directory)
eb-build-module.sh -a mi300a -m PyTorch-2.1.2-foss-2023a

# For L40S GPUs (Intel-based, uses CUDA)
eb-build-module.sh -a l40s -m CUDA-12.0.0

# For H200 GPUs (uses genoa as base, stored in genoa directory)
eb-build-module.sh -a h200 -m CUDA-12.0.0

Tip: If you plan to use software on multiple architectures, you need to build it separately for each one.

Key Script Features

• Architecture support: Automatically configures builds for milan, genoa, mi300a, l40s, or h200
• GPU detection: Allocates GPUs automatically when building GPU software
• Flexible options: Customize cores (-c), walltime (-w), and installation prefix (-p)
• Build modes:
  • Standard build (default)
  • Rebuild (-r) - force rebuild even if already exists
  • Module-only (-o) - generate module file without building
  • Dry-run (-d) - preview without building
• Search function: Use -s to find available easyconfig files
• Pass-through options: Use -- to pass additional EasyBuild options directly

Where Are Modules Installed?

By default, modules are installed to:

~/.local/easybuild/<arch>/

For example:

• ~/.local/easybuild/milan/ - Milan-specific modules
• ~/.local/easybuild/genoa/ - Genoa, MI300A, and H200 modules
• ~/.local/easybuild/l40s/ - L40S-specific modules

Module files location: The actual module files that you load with module load are placed in:

~/.local/easybuild/modules/all/          # Architecture-independent modules
~/.local/easybuild/<arch>/modules/all/   # Architecture-specific modules

Example module paths:

~/.local/easybuild/modules/all/lang/python/3.11.3-gcccore-12.3.0.lua
~/.local/easybuild/genoa/modules/all/lang/python/3.11.3-gcccore-12.3.0.lua

Important: On NEMO2, the module naming scheme converts all names to lowercase. While EasyBuild uses files like Python-3.11.3-GCCcore-12.3.0.eb, the resulting modules are named in lowercase: lang/python/3.11.3-gcccore-12.3.0. This applies to all modules including compilers (compiler/gcc), libraries, etc.

These paths are automatically added to your MODULEPATH when you load an architecture with module load arch/<arch>, making your custom modules visible with module avail.

You can change the installation location with the -p option or by setting the PREFIX environment variable.

Using Your Custom Modules

After building, your modules are automatically available when you load the corresponding architecture:

# Load the architecture you built for
module load arch/genoa

# Your custom modules now appear alongside system modules
module avail

# Load your custom module (remember: lowercase names)
module load category/softwarename/version-toolchain

How it works: When you load arch/genoa, the following paths are added to your MODULEPATH:

• ~/.local/easybuild/modules/all/ - for architecture-independent modules
• ~/.local/easybuild/genoa/modules/all/ - for genoa-specific modules

This makes your custom-built modules visible alongside system modules when you run module avail.

Note: Module names appear in lowercase (e.g., lang/python/3.11.3-gcccore-12.3.0) due to NEMO2's naming scheme, even though the easyconfig files use capitalized names.

Common Build Options

# Build with more cores for faster compilation (large packages)
eb-build-module.sh -c 64 -w 24 -m GCC-13.2.0

# Rebuild an existing module
eb-build-module.sh -r -m Python-3.11.3-GCCcore-12.3.0

# Pass additional EasyBuild options
eb-build-module.sh -m Python-3.11.3-GCCcore-12.3.0 -- --force --debug

Need Help?

• Full script documentation: NEMO2/Easybuild_Modules/EB_Build_Module
• EasyBuild documentation: https://docs.easybuild.io/
• Available easyconfigs: Use eb-build-module.sh -s pattern to search

Compiling Your Own Software (Non-EasyBuild)

If you compile your own applications outside of EasyBuild (e.g., custom C/C++/Fortran code), you should create architecture-specific builds using the $ARCH environment variable.

Using $ARCH in Your Workflows

The $ARCH variable is automatically set when you load an architecture module:

module load arch/milan
echo $ARCH  # Output: milan

Use this in your directory structure:

# Organize binaries by architecture
$HOME/software/$ARCH/myapp/bin/myprogram

# Example paths for different architectures:
# ~/.local/software/milan/myapp/bin/myprogram
# ~/.local/software/genoa/myapp/bin/myprogram
# ~/.local/software/l40s/myapp/bin/myprogram

Compiling for Different Architectures

# Compile for Milan
module load arch/milan
gcc -O3 -march=znver3 mycode.c -o $HOME/software/$ARCH/bin/myprogram

# Compile for Genoa
module load arch/genoa
gcc -O3 -march=znver4 mycode.c -o $HOME/software/$ARCH/bin/myprogram

Running Cross-Architecture Binaries

If you compile for one architecture but run on another, you must load the correct architecture module:

# Request a Genoa node but run Milan-compiled code
salloc -n1 -p genoa

# Load the architecture the code was compiled for
module load arch/milan

# Now run your Milan-compiled binary on the Genoa node
$HOME/software/milan/myapp/bin/myprogram

Warning: This should only be done when Milan code is compatible with Genoa. Genoa code will likely not work on Milan nodes.

Testing Modules and Software

Interactive Testing

You can test modules and software interactively using srun or salloc (see Interactive Jobs and Partitions).

Quick tests on login nodes (up to 15 minutes):

srun -p login <command>
# or
salloc -p login

Testing on specific architecture:

# Interactive session on Milan partition
salloc -n1 -p milan
module load arch/milan
module load YourModule
<your-command>
exit

# Interactive session with AMD MI300A APU
salloc -n1 -p mi300a --gres=gpu:1
module load arch/mi300a
module load system/rocm  # ROCm for AMD GPUs
<your-gpu-command>
exit

# Interactive session with NVIDIA GPU (L40S example)
salloc -n1 -p l40s --gres=gpu:1
module load arch/l40s
module load system/cuda  # CUDA for NVIDIA GPUs
<your-gpu-command>
exit

Tip: If you get MPI or memory errors, try using srun to launch your program: srun <program>

Working with Module Listings

Viewing Available Modules

Basic module commands:

# List all available modules
module avail

# List modules for a specific category
module avail compiler
module avail math
module avail chem

# Search for a specific module
module avail python
module avail gcc

Managing Long Module Lists

If module avail produces a very long list, use these techniques:

1. Get an overview:

module overview

This shows a condensed summary of available modules.

2. Filter by category or name:

module avail compiler   # Show only compiler modules
module avail llvm       # Show only LLVM-related modules
module avail python     # Show only Python-related modules

3. Hide module extensions:

Many modules have extensions (additional packages/libraries). To hide these:

# Hide extensions for one command
module --nx avail
# or
module --no_extensions avail

# Hide extensions permanently
export LMOD_AVAIL_EXTENSIONS=no

Add the export command to your ~/.bashrc to make it permanent.

Understanding Module Names

EasyBuild module names on NEMO2 follow a specific format:

category/softwarename/version-toolchain

All components are lowercase due to NEMO2's naming scheme.

Example: lang/python/3.11.3-gcccore-12.3.0

• Category: lang (programming language)
• Software: python (lowercase)
• Version: 3.11.3
• Toolchain: gcccore-12.3.0 (the compiler/libraries used to build it, also lowercase)

Important: When building, you use the EasyBuild file with capitalized names (e.g., Python-3.11.3-GCCcore-12.3.0.eb), but the resulting module will be in lowercase (e.g., lang/python/3.11.3-gcccore-12.3.0). This applies to all parts: software names (python, not Python), toolchain components (gcccore, not GCCcore), and everything else.

Common categories include:

• lang/ - Programming languages (python, r, julia, etc.)
• compiler/ - Compilers (gcc, intel, llvm, etc.)
• system/ - System software (cuda, mpi, etc.)
• bio/ - Biology/Bioinformatics software
• chem/ - Chemistry software
• math/ - Mathematical software
• ai/ - Artificial Intelligence/Machine Learning
• lib/ - Libraries
• devel/ - Development tools

When building your own modules, use the same naming convention for consistency.