Overlays

Table of Contents

1. Motivation
2. Overlay stacks
3. Limitations
4. Overlay configuration
- 4.1. Definition
- 4.2. Overlay definition files
5. Overlay types
6. Variant files
- 6.1. YAML format of variant files

This is work in progress!!!

1. Motivation

Idea: seamless stacking of multiple module hierarchies, use cases:

develop/build new modules without exposing them to other users
transparent/seamless support for modules optimised for certain CPUs
modules for HPC systems like slurm and required software to support HPC hardware.
modules on another filesystem or location than the default (@PSI: /opt/psi)
adding new groups. Examples
- system specific software
- software used only by certain groups
- ..
..

2. Overlay stacks

Overlays can be stacked. The first overlay - the overlay at the bottom of the stack - is called "base overlay". Multiple loaded overlays are organised as stack. Loading an overlay pushes it on top of the stack. It is only possible to unload the overlay on top of the stack.

The 'overlay' at the bottom of the stack is called the base overlay and cannot be unloaded.

3. Limitations

The relative directory structure of a hierarchical group must be identical to the base overlay. It is not possible to define a new hierarchical group.

Overlays must be loaded before the first module.

Unloading an overlay is only possible if no modules are loaded.

Only the overlay on top of the stack can be unloaded.

4. Overlay configuration

4.1. Definition

An overlay is defined by

a root directory for the software installation
a root directory for the modulefiles
a name

In most cases both root directories are identical.

An overlay is defined by the two root directories. Different names can be used for the same overlay.

If only the root directory for the software installation is specified, this is also used as root for the modulefiles.

4.2. Overlay definition files

Configuration files are in YAML format.

The system wide configuration file is $PMODULES_ROOT/config/overlays.yaml.

User can define their own overlays in $HOME/.Pmodules/overlays.yaml or specify one in the environment variable

All overlays must be defined in the system or user configuration file.

The user configuration file is queried first and can be used to overwrite the definition of an overlay in the system configuration file.

Format:

name_of_overlay:
  root_dir: <DIR>
  modulefiles_root_dir: <DIR>
  type: normal|hidding|replacing
...

name_of_overlay: Is the name of the overlay.
root_dir: Is the root directory for software installation. The specification of root_dir is mandatory.
modulefiles_root_dir: Is the root directory for the installation of modulefiles. If it is not specified, root_dir is used. This is the most usual case.
type: Specifies the type of the overlay. The default is normal. See below for the type specifications.

5. Overlay types

normal

In an overlay of this types the modules in the overlay are added to the already available modules provided by the overlays on the stack. If modules with identical name/version exist in multiple overlays, the module from the overlay which is higher on the stack is used. In other words modules with identical name/version are replaced by overlays loaded later.

Example: Assume we have the overlays base and private. In the base overlay are the modules git/2.33.1, gnuplot/5.4.0 and gnuplot/5.4.2 in the group Tools. In private overlay are the modules gnuplot/5.4.2 and gnuplot/5.4.3 (maybe compiled with a different configuration).

After loading the private overlay the following Gnuplot modules are available:

git/2.33.1`and `gnuplot/5.4.0 from the base overlay
gnuplot/5.4.2 and `gnuplot/5.4.3`from the private overlay

Note: the module gnuplot/5.4.2 from the base overlay has been replaced by the module with the same name from the private overlay.

hiding

An overlay of this types hides all modules with the same name in previous loaded overlays.

Example: Assume we have the same overlays and Gnuplot modules as in the example above.

After loading the private overlay with type hiding the following Gnuplot modules are available:

git/2.33.1 from the base overlay
gnuplot/5.4.2 and `gnuplot/5.4.3`from the private overlay

Note: the Gnuplot modules from the base overlay are hidden!

replacing

A overlay of this types replaces hole groups.

Example: If you load the private overlay from the examples above as replacing, the following modules are available in the group Tools:

gnuplot/5.4.2 and `gnuplot/5.4.3`from the private overlay

Note: the Git module from the base overlay is not available any more.

6. Variant files

With the implementation of overlays a new format for the so called variant files is necessary. In the old format extensions are almost impossible. Nowadays a common format used for configuration files is YAML.

6.1. YAML format of variant files

---
overlay: <overlay-name>
module_name/module_version:
  variants:
  - relstage: <release-stage>
    modulefiles_root_dir: <modulefile-root-dir>
    dependencies:
    - dependency_1
    - dependency_2
    - ...
...

overlay

Name of the overlay the module will be installed in. The specified overlay must be defined in one of the overlay definition files. If no overlay is specified the base overlay is used.

module_name/module_version

Specification of the module with this name and version. For each version a specification should be added.

variants

Array of variants.

relstage

Release stage, either unstable, stable or deprecated. If not specified the release stage defaults to unstable.

modulefiles_root_dir

This key can be used to overwrite the modulefiles_root_dir specified in the overlay definition file.

dependencies

Array of dependencies. Dependencies can have one of the following prefixes:

b: build-dependency: A build-dependency is required at build time and must be loaded to compile the module. But it is not required at runtime.
i: install-dependency: A install-dependency is required at runtime, but must not be loaded at runtime of the module. An example is the Intel compiler. It requires GCC but GCC must not be loaded. A install-dependency is loaded at build-time.
no prefix: dependencies without prefix must be loaded at build- and runtime.