Modules

We use the Lmod module system to provide compilers, MPI stacks, libraries, and tools.

The legacy module tree (elysium/2024) is currently the default view. To switch to the current tree, load:

module load elysium/2026

The elysium/202x modules modify the MODULEPATH. They are sticky, so they are not removed by module purge. elysium/2024 and elysium/2026 are mutually exclusive: loading one unloads the other. The legacy tree (elysium/2024) is frozen and no longer updated by admins. The 2026 tree is built with gcc@13.4.0, which provides better optimization support for Elysium’s Zen 4 CPUs than gcc@11.

Known Issues

• Intel MPI in elysium/2024 (legacy tree) is known to be unstable in this version. It can cause random MPI deadlocks where applications stop progressing while the Slurm job is still running.
• In the current tree (elysium/2026), Intel MPI is stable and recommended.
• MPICH versions before 5.0.0 are not suitable for multi-node runs. Use 5.0.0 or newer for multi-node jobs.
• If you must use MPICH < 5.0.0, restrict jobs to single-node execution.

Common module commands

ml is a short alias for module.

• ml av: list available modules in the current module path.
• ml load <module>: load a module. Example: ml load openmpi/5.0.9.
• ml list: show currently loaded modules.
• ml unload <module>: unload a module. Example: ml unload openmpi/5.0.9.
• ml purge: unload all currently loaded modules except sticky ones.
• ml help <module>: show module help text. Example: ml help openmpi/5.0.9.
• ml show <module>: print the contents of the modulefile.
• ml whatis <module>: show a short module description.
• ml spider <name>: search all known versions. Use ml spider -A <name> to also show hidden entries.
• ml use <path>: add a directory to your module search path.
• ml unuse <path>: remove a directory from your module search path.

Using your own modulefiles

You can keep your own modulefiles in a personal directory and add it with ml use.

mkdir -p "$HOME/modules/mytool"
cat > "$HOME/modules/mytool/1.0.lua" <<'LUA'
help([[MyTool 1.0]])
whatis("Name: MyTool")
whatis("Version: 1.0")
depends_on("openmpi/5.0.9")
prepend_path("PATH", "/path/to/mytool/bin")
LUA

ml use "$HOME/modules"
ml load mytool/1.0

In this example, loading mytool/1.0 also loads openmpi/5.0.9 as a dependency.

To make your personal modules always visible, add this to your ~/.bashrc:

ml use "$HOME/modules"

If you need additional software or module versions, please contact support.

Spack

We provide a central Spack 1.1.1 installation in parallel to the existing 0.23.0 stack.

This separation is necessary because a database migration would render the 0.23.0 installation unusable. The new 2026 module tree is not compatible with the legacy tree, and self-built packages from the old setup must be rebuilt.

By default, you will still see the legacy module tree. The new stack is opt-in via:

module load spack/2026

If you already used the legacy setup, use this dedicated migration guide: Migration Guide for Existing Users.

Legacy documentation remains available here: Spack (Legacy 0.23.0).

Table of Contents

1. Quick Setup (New Users)
2. Architecture Overview
3. Guide to Using Spack
4. What Is New in Modern Spack
5. Legacy Setup

Quick Setup

For a fresh setup:

rub-deploy-spack-configs-2026
module load spack/2026

After that, you can install your own packages into your home Spack tree and generate personal modulefiles. Add module load spack/2026 to your ~/.bashrc if you want this active in ever login shell.

Architecture Overview

We use a central Spack installation combined with per-user overlays:

Central Spack (read-only, maintained by HPC)
    ├─ pre-built packages
    ├─ MPI configurations
    └─ cluster-wide defaults
            ↓ (upstream)
User Spack ($HOME)
    ├─ personal installs
    └─ personal modules
            ↓
Lmod module system

This means:

• You automatically use centrally installed packages when available.
• Cluster-wide configurations (e.g. MPI defaults, compiler settings) are inherited.
• Your own installations go into your personal $HOME Spack tree.
• Only missing packages are built locally.

Guide to Using Spack

This section covers the most common Spack workflows on Elysium.

Searching and Inspecting Packages

Search for packages:

spack list <keyword>
spack list openfoam

Show details (versions, variants, dependencies):

spack info hdf5

Preview what will actually be installed:

spack spec hdf5 +mpi

Always check spack spec before installing complex packages.

Installing Packages

Basic installation:

spack install hdf5

Enable or disable variants:

spack install hdf5 +mpi +cxx ~fortran

Specify a compiler:

spack install hdf5 %gcc@13.4.0

Force rebuilding dependencies with the same compiler:

spack install --fresh hdf5 %gcc@13.4.0

Specify dependencies explicitly:

spack install hdf5 ^openmpi@5.0.9

Everything combined:

spack install hdf5@1.14.6 +mpi %gcc@13.4.0 ^openmpi@5.0.9

Virtual Providers (BLAS, MPI, FFT, etc.)

Some packages depend on virtual interfaces instead of concrete libraries. Examples include:

• blas
• lapack
• mpi
• fftw-api

To see which virtual packages exist:

spack providers

To list available providers for a specific interface:

spack providers blas
spack providers fftw-api

Example output:

Providers for blas:
    amdblis
    openblas
    intel-oneapi-mkl

You can select a specific provider during installation:

spack install gromacs ^blas=openblas

This is often preferred over manually selecting a specific concrete library, because it keeps the dependency graph clean and compatible.

Inspecting and Comparing Installations

List installed packages with variants and hashes:

spack find -vl hdf5

Inspect a specific installation:

spack spec /<hash>

Compare two installations:

spack diff /hash1 /hash2

Removing Packages

Remove a specific installation by hash:

spack uninstall /<hash>

Overriding Package Definitions

On Elysium, the central builtin repository is provided by the HPC team. If you want to override a package definition (e.g. to test changes), you can create a local repository on top of it.

Create a local repo

mkdir -p $HOME/spack/var/spack/repos/packages
cat > $HOME/spack/var/spack/repos/repo.yaml <<'EOF'
repo:
  namespace: overrides
EOF

Register it in ~/.spack/repos.yaml (place it above builtin to override):

repos:
  overrides: $HOME/spack/var/spack/repos
  builtin:
    destination: /cluster/spack/spack-packages

Check:

spack repo list

Override a package (example: ffmpeg)

mkdir -p $HOME/spack/var/spack/repos/packages/ffmpeg
cp /cluster/spack/spack-packages/repos/spack_repo/builtin/packages/ffmpeg/package.py \
   $HOME/spack/var/spack/repos/packages/ffmpeg/

Edit the copied package.py and adjust versions, dependencies, variants, etc.

Install explicitly from your namespace:

spack install overrides.ffmpeg

Verify which repository is used

For a spec (not yet installed):

spack spec -N ffmpeg

For installed packages:

spack find -N ffmpeg

The -N option shows the namespace (overrides or builtin).

Custom Changes to Packages using spack develop

If you want to modify the source code of a package (e.g. openfoam) and rebuild it locally, you can use spack develop. This allows you to work directly on a source checkout without creating tarballs or calculating checksums.

Create and activate a development environment

mkdir -p ~/openfoam-dev
cd ~/openfoam-dev
spack env create -d .
spacktivate . #shortcut for `spack env activate .`

Using a dedicated environment keeps your development work isolated from your normal Spack setup.

Add and install the package

spack add openfoam
spack install

This performs a normal installation and ensures all dependencies are available.

Switch to development mode

spack develop openfoam

This checks out the source code into the environment directory (e.g. ~/openfoam-dev/openfoam/) and registers it as the active development source.

You can verify this with:

spack find -cv openfoam

Look for dev_path=.../openfoam in the output.

Modify the source and rebuild

cd ~/openfoam-dev/openfoam
# edit source files here (e.g. with vim)

cd ~/openfoam-dev
spack install

Spack will now build openfoam from your modified local sources.

If compilation fails, Spack will print the relevant error messages and the path to the full build log.

Legacy Setup

If you need to reference the old 0.23.0 setup, use: Spack (Legacy 0.23.0).

MATLAB

This page describes how to configure MATLAB to submit jobs to the Elysium HPC cluster, retrieve results, and debug errors.

Initial Configuration

Running MATLAB on the HPC Cluster

This setup is intended for job submission when you are logged directly into the cluster via the command line. This process only needs to be completed once per cluster.

After logging into the cluster, start MATLAB and run:

configCluster

Jobs will run across multiple nodes on the cluster rather than on the host machine.

Running MATLAB on the Desktop

This setup is intended for job submission when MATLAB is installed on your machine and jobs are run remotely on the cluster. This setup needs to be done once per cluster, per version of MATLAB installed on your machine.

Start MATLAB and run:

userpath

Download the Integration Scripts (RUB.Desktop.zip) directly from this page. Extract the ZIP contents into the folder returned by userpath.

Create a new cluster profile:

configCluster

Submission to the cluster requires SSH credentials. You will be prompted for your cluster username (LoginID).

Jobs will now run on the cluster rather than on the local machine. Before submitting jobs, log in to login002 via SSH on the command line so the two-factor authentication login is cached.

Note To run jobs on the local machine instead of the cluster, use the Processes profile.

% Get a handle to the local resources
c = parcluster('Processes');

Configuring Jobs

Prior to submitting a job, you can assign scheduler flags such as queue, email, wall time, and more. The following properties are mandatory and must be set before you can submit a job:

• AccountName
• Nodes
• Partition
• WallTime

% Get a handle to the cluster
c = parcluster;

% REQUIRED

% Specify an account
c.AdditionalProperties.AccountName = 'account-name';

% Specify number of nodes
c.AdditionalProperties.Nodes = 1;

% Specify the partition
c.AdditionalProperties.Partition = 'partition-name';

% Specify the wall time (e.g. 1 day, 5 hours, 30 minutes)
c.AdditionalProperties.WallTime = '1-05:30';

% OPTIONAL

% Specify a constraint
c.AdditionalProperties.Constraint = 'feature-name';

% Request email notification of job status
c.AdditionalProperties.EmailAddress = 'firstname.familyname@ruhr-uni-bochum.de';

% Specify number of GPUs (default: 0)
c.AdditionalProperties.GPUsPerNode = 1;

% Specify the number of CPUs per GPU
c.AdditionalProperties.CPUsPerGPU = 1;

% Specify memory to use, per core (default: 4GB)
c.AdditionalProperties.MemPerCPU = '6GB';

% Specify cores per node (default: 0)
c.AdditionalProperties.ProcsPerNode = 4;

% Set node exclusivity (default: false)
% Note that this will automatically be set to true if using more
% than one node.
c.AdditionalProperties.RequireExclusiveNode = true;

% Specify a reservation
c.AdditionalProperties.Reservation = 'reservation-name';

To persist changes made to AdditionalProperties between MATLAB sessions, save the profile:

c.saveProfile

To see the values of the current configuration options, display AdditionalProperties:

c.AdditionalProperties

Unset a value when it is no longer needed:

% Turn off email notifications
c.AdditionalProperties.EmailAddress = '';

% Do not request an entire node
c.AdditionalProperties.RequireExclusiveNode = false;

Note The instructions above cover the basics of configuring and running jobs on the cluster. For a more in-depth walkthrough of the job submission workflow, see the demo script ScalingToTheClusterDemoRemote.mlx.

Independent Batch Job - MATLAB on the HPC Cluster or Desktop

Use the batch command to submit asynchronous jobs to the cluster. The batch command returns a job object, which is used to access the output of the submitted job. See the MATLAB documentation for batch for more details.

% Get a handle to the cluster
c = parcluster;

% Submit job to query where MATLAB is running on the cluster
job = c.batch(@pwd, 1, {}, 'CurrentFolder', '.', 'AutoAddClientPath', false);

% Query job for state
job.State

% If job is finished, fetch the results
job.fetchOutputs{1}

% Delete the job after results are no longer needed
job.delete

To retrieve a list of running or completed jobs, call parcluster to return the cluster object. The cluster object stores an array of jobs that are listed as queued, running, finished, or failed.

c = parcluster;
jobs = c.Jobs

% Get a handle to the second job in the list
job2 = c.Jobs(2);

Once the job has been selected, fetch the results as previously shown.

fetchOutputs is used to retrieve function output arguments. If you call batch with a script, use load instead. Data written to disk on the cluster must be retrieved directly from the file system, for example via SFTP.

% Fetch all results from the second job in the list
job2.fetchOutputs{:}

% Alternate: load results if the job was a script instead of a function
job2.load

Parallel Batch Job - MATLAB on the HPC Cluster or Desktop

The batch command also supports parallel workflows. Save the following example as parallel_example.m.

function [sim_t, A] = parallel_example(iter)

if nargin == 0
    iter = 8;
end

disp('Start sim')

A = nan(iter, 1);
t0 = tic;
parfor idx = 1:iter
    A(idx) = idx;
    pause(2)
    idx
end
sim_t = toc(t0);

disp('Sim completed')

save RESULTS A

end

When using the batch command, specify a Pool argument:

% Get a handle to the cluster
c = parcluster;

% Submit a batch pool job using 4 workers for 16 simulations
job = c.batch(@parallel_example, 1, {16}, 'CurrentFolder', '.', 'Pool', 4, 'AutoAddClientPath', false);

% View current job status
job.State

% Fetch the results after a finished state is retrieved
job.fetchOutputs{1}

Example output:

ans =
    8.1678

The job ran in 8.17 seconds using four workers. Note that these jobs always request N + 1 CPU cores, since one worker is required to manage the batch job and pool of workers. For example, a job that needs eight workers will require nine CPU cores.

Run the same simulation again but increase the pool size. This time, to retrieve the results later, keep track of the job ID.

Note For some applications, there will be diminishing returns when allocating too many workers, as the overhead may exceed computation time.

% Get a handle to the cluster
c = parcluster;

% Submit a batch pool job using 8 workers for 16 simulations
job = c.batch(@parallel_example, 1, {16}, 'CurrentFolder', '.', 'Pool', 8, 'AutoAddClientPath', false);

% Get the job ID
id = job.ID

Example output:

id =
    4

% Clear job from workspace (as though MATLAB exited)
clear job

With a handle to the cluster, the findJob method searches for the job with the specified job ID:

% Get a handle to the cluster
c = parcluster;

% Find the old job
job = c.findJob('ID', 4);

% Retrieve the state of the job
job.State

Example output:

ans =
    finished

% Fetch the results
job.fetchOutputs{1};

Example output:

ans =
    4.1503

The job now runs in 4.15 seconds using eight workers. Run code with different numbers of workers to determine the ideal number to use. Alternatively, to retrieve job results via a graphical user interface, use the Job Monitor (Parallel > Monitor Jobs). It will take some time until the list is shown.

Debugging

If a serial job produces an error, call the getDebugLog method to view the error log file.

When submitting an independent job, specify the task:

c.getDebugLog(job.Tasks)

For pool jobs, specify only the job object:

c.getDebugLog(job)

When troubleshooting a job, the cluster administrators may request the scheduler ID of the job. You can derive it by calling getTaskSchedulerIDs:

job.getTaskSchedulerIDs()

Example output:

ans =
    4911680

Helper Functions

To Learn More

To learn more about the MATLAB Parallel Computing Toolbox, see:

• Parallel Computing Overview
• Parallel Computing Documentation
• Parallel Computing Coding Examples
• Parallel Computing Tutorials
• Parallel Computing Videos
• Parallel Computing Webinars

Vampir

Vampir is a framework for analyzing program behavior of serial and parallel software by utilizing function instrumentation via Score-p. Vampir is licensed by HPC.nrw and can be used freely on the Elysium Cluster. This site merely shows a small test case to show how Score-p can be used to generate profiling data and how Vampir can be started on Elysium. For information how to use Vampir to analyze your application, extract useful performance metrics, and identify bottlenecks, please refer to the Score-p Cheat Sheet and the official Vampir Documentation.

Compilation with Instrumented Functions

In order to generate profiling data the function calls in the application need to be instrumented. This means inserting additional special function calls that record the time, current call stack, and much more. Fortunately, this is not done manually, but can easily achieved by using the Score-p compiler wrapper. To follow along you can use this MPI Example Code.

To use the Score-p compiler wrapper, all that is needed is to prepend the compiler by the scorep command:

module load openmpi/5.0.5-d3ii4pq
module load scorep/8.4-openmpi-5.0.5-6mtx3p6
scorep mpicc -o mpi-test.x mpi-test.c

In the case of a Makefile, or other build systems, the compiler variable has to be adjusted accordingly.

Generating Profiling Data

Profiling data is created by running the application. Note that the profiling files can grow to enormous sizes. Thus, it is advisable to choose a small representative test case for your application and not a full production run. In its default mode Score-p collects profiling data by sampling the applications call-stack from time to time. In order to generate an accurate profile tracing needs to be enabled in your job script:

module load openmpi/5.0.5-d3ii4pq
module load scorep/8.4-openmpi-5.0.5-6mtx3p6

export SCOREP_ENABLE_TRACING=true
mpirun -np 4 ./mpi-test.x

Here is a full job script for the example:

#!/bin/bash
#SBATCH --partition=cpu
#SBATCH --ntasks=4
#SBATCH --nodes=1
#SBATCH --account=<Account>
#SBATCH --time=00-00:05:00

module purge
module load openmpi/5.0.5-d3ii4pq
module load scorep/8.4-openmpi-5.0.5-6mtx3p6

export SCOREP_ENABLE_TRACING=true
mpirun -np 4 ./mpi-test.x

The execution of the instrumented application will take significantly longer than usual. Thus, it should never be used for production runs, but merely for profiling. After the application is finished a new directory was created, containing the time stamp and some other information in its name e.g.: scorep-20251222_0912_1523094386395226 The file traces.otf2 contains the profiling data required by Vampir.

Visualizing With Vampir

In order to visualize the profiling data a Visualization Session has to be established. Vampir can be started with

module load vampir
vglrun +pr -fps 20 vampir ./traces.otf2

This will open the Vampir graphical user interface:

VASP

Build configuration (MKL)

On Elysium, VASP can be built with Spack using:

spack install vasp@6.4.3 +openmp +fftlib ^openmpi@5.0.5 ^fftw@3+openmp ^intel-oneapi-mkl threads=openmp +ilp64

This configuration uses:

• Intel oneAPI MKL (ILP64) for BLAS, LAPACK and ScaLAPACK,
• VASP’s internal FFTLIB to avoid MKL CDFT issues on AMD,
• OpenMPI 5.0.5 as MPI implementation,
• OpenMP enabled for hybrid parallelisation.

We choose MKL as baseline because it is the de-facto HPC standard and performs well on AMD EPYC when AVX512 code paths are enabled.

Activating AVX512

Intel’s MKL only enables AVX512 optimisations on Intel CPUs. On AMD, MKL defaults to AVX2/SSE code paths.

To unlock the faster AVX512 kernels on AMD EPYC we provide libfakeintel, which fakes Intel CPUID flags.

⚠ Intel gives no guarantee that all AVX512 instructions work on AMD CPUs. In practice, the community has shown that not every kernel uses full AVX512 width, but the overall speed-up is still substantial.

Activate AVX512 by preloading the library in your job:

export LD_PRELOAD=/lib64/libfakeintel2025.so:${LD_PRELOAD}

Test case 1 – Si256 (DFT / Hybrid HSE06)

This benchmark uses a 256-atom silicon supercell (Si256) with the HSE06 hybrid functional. Hybrid DFT combines FFT-heavy parts with dense BLAS/LAPACK operations and is therefore a good proxy for most large-scale electronic-structure workloads.

Baseline: MPI-only, 1 node

→ Always enable AVX512. The baseline DFT case runs 17 % faster with libfakeintel,

Build configuration (AOCL)

AOCL (AMD Optimized Libraries) is AMD’s analogue to MKL, providing:

• AMDBLIS (BLAS implementation)
• AMDlibFLAME (LAPACK)
• AMDScaLAPACK, AMDFFTW optimised for AMD EPYC
• built with AOCC compiler

Build example:

spack install vasp@6.4.3 +openmp +fftlib %aocc ^amdfftw@5 ^amdblis@5 threads=openmp ^amdlibflame@5 ^amdscalapack@5 ^openmpi

AOCL detects AMD micro-architecture automatically and therefore does not require libfakeintel.

Baseline: MPI-only, 1 node

The AOCl build is another 5% faster than MKL with AVX512 enabled.

Hybrid parallelisation and NUMA domains

Each compute node has two EPYC 9254 CPUs with 24 cores each (48 total). Each CPU is subdivided into 4 NUMA domains with separate L3 caches and memory controllers.

• MPI-only: 48 ranks per node (1 per core).
• Hybrid L3: 8 MPI ranks × 6 OpenMP threads each, bound to individual L3 domains.

This L3-hybrid layout increases memory locality, because each rank mainly uses its own local memory and avoids cross-socket traffic.

Single-node hybrid results (Si256)

Hybrid L3 adds a modest 4-5 % speed-up.

Multi-node scaling (Si256)

Interpretation

AOCL shows the strongest scaling across nodes; MKL’s hybrid variant catches up in scaling compared to its MPI-only counterpart. The L3-hybrid layout maintains efficiency even in the multi-node regime.

Recommendations for DFT / Hybrid-DFT workloads

• AOCL generally outperforms MKL (+AVX512) on AMD EPYC.
• Prefer L3-Hybrid (8×6) on single-node and even multi-node jobs for FFT-heavy hybrid-DFT cases.
• For pure MPI runs, both MKL (+AVX512) and AOCL scale well; AOCL slightly better.
• Always preload libfakeintel2025.so if MKL is used.

Jobscript examples

AOCL – Hybrid L3 (8×6)

#!/bin/bash
#SBATCH -J vasp_aocl_l3hyb
#SBATCH -N 1
#SBATCH --ntasks=8
#SBATCH --cpus-per-task=6
#SBATCH -p cpu
#SBATCH -t 48:00:00
#SBATCH --exclusive

module purge
module load vasp-aocl

export OMP_NUM_THREADS=6
export OMP_PLACES=cores
export OMP_PROC_BIND=close
export BLIS_NUM_THREADS=6

mpirun -np 8 --bind-to l3 --report-bindings vasp_std

MKL (+AVX512) – Hybrid L3 (8×6)

#!/bin/bash
#SBATCH -J vasp_mkl_avx512_l3hyb
#SBATCH -N 1
#SBATCH --ntasks=8
#SBATCH --cpus-per-task=6
#SBATCH -p cpu
#SBATCH -t 48:00:00
#SBATCH --exclusive

module purge
module load vasp-mkl
export LD_PRELOAD=/lib64/libfakeintel2025.so:${LD_PRELOAD}

export OMP_NUM_THREADS=6
export OMP_PLACES=cores
export OMP_PROC_BIND=close
export MKL_NUM_THREADS=6
export MKL_DYNAMIC=FALSE

mpirun -np 8 --bind-to l3 --report-bindings vasp_std

Test case 2 – XAS (Core-level excitation)

The XAS Mn-in-ZnO case models a core-level excitation (X-ray Absorption Spectroscopy). These workloads are not FFT-dominated; instead they involve many unoccupied bands and projector evaluations.

Single-node results (XAS)

Multi-node scaling (XAS)

Interpretation

For core-level / XAS calculations, hybrid OpenMP parallelisation is counter-productive, and scaling beyond one node deteriorates performance due to load imbalance and communication overhead.

Recommendations for XAS and similar workloads

• Use MPI-only and single-node configuration.
• MKL and AOCL perform identically within margin of error.
• Hybrid modes reduce efficiency and should be avoided.
• Set OMP_NUM_THREADS=1 to avoid unwanted OpenMP activity.

General guidance

For optimal performance on Elysium with AMD EPYC processors, we recommend using the AOCL build as the default choice for all VASP workloads. AOCL consistently outperforms or matches MKL (+AVX512) across tested scenarios (e.g., 5 % faster for Si256 single-node, up to 1.89× speedup for multi-node scaling) and does not require additional configuration like libfakeintel. However, MKL remains a robust alternative, especially for users requiring compatibility with existing workflows.

Recommendations:

• Default to AOCL: Use the AOCL build for all workloads unless specific constraints (e.g., compatibility with Intel-based tools) require MKL.
• AVX512 for MKL: If using MKL, always preload libfakeintel2025.so to enable AVX512 optimizations.
• Benchmark if unsure: Test both MPI-only and L3 Hybrid on one node to determine the optimal configuration for your specific system.