Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
S src
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 7
    • Issues 7
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
    • Iterations
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Operations
    • Operations
    • Incidents
    • Environments
  • Analytics
    • Analytics
    • CI/CD
    • Code Review
    • Issue
    • Repository
    • Value Stream
  • Wiki
    • Wiki
  • Members
    • Members
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Pmodules
  • src
  • Wiki
  • Home

Last edited by gsell Mar 08, 2022
Page history

Home

Pmodules Wiki

Table of Contents
  • 1. Overlays
    • 1.1. Overview
    • 1.2. Overlay stacks
    • 1.3. Limitations
    • 1.4. Overlay configuration
    • 1.5. Overlay types
    • 1.6. Variant files

1. Overlays

This is work in progress!!!

1.1. Overview

Idea: seamless stacking of multiple module hierarchies, use cases:

  • develop/build new modules without exposing them to other users

  • transparent/seamless support for modules optimised for certain CPUs

  • modules for HPC systems like slurm and required software to support HPC hardware.

  • modules on another filesystem or location than the default (@PSI: /opt/psi)

  • …​

1.2. Overlay stacks

Overlays can be stacked. The first overlay - the overlay at the bottom of the stack - is called "base overlay". Multiple loaded overlays are organised as stack. Loading an overlay pushes it on top of the stack. It is only possible to unload the overlay on top of the stack.

The 'overlay' at the bottom of the stack is called the base overlay and cannot be unloaded.

1.3. Limitations

The relative directory structure of a hierarchy group must be identical to the base overlay. It is also not possible to define a new hierarchical group.

Overlays must be loaded before the first module.

Unloading an overlay is only possible if no modules are loaded.

Note: an overlay can add new non-hierarchical groups.

1.4. Overlay configuration

1.4.1. Definition

An overlay is defined by * a root directory for the software installation * a root directory for the modulefiles * a name

In most cases both root directories are identical.

An overlay is defined by the two root directories. Different names can be used for the same overlay.

If only the root directory for the software installation is specified, this is also used as root for the modulefiles.

1.4.2. Overlay definition files

Configuration files are in YAML format.

The system wide configuration file is $PMODULES_ROOT/config/overlays.yaml.

User can define their own overlays in $HOME/.Pmodules/overlays.yaml.

All overlays must be defined in the system or user configuration file.

The user configuration file is queried first and can be used to overwrite the definition of an overlay in the system configuration file.

Format:

---
overlays:
  name_of_overlay:
    root_dir: <DIR>
    modulefiles_root_dir: <DIR>
    type: normal|hidding|replacing
  ...
name_of_overlay

Is the name of the overlay.

root_dir

Is the root directory for software installation. The specification of root_dir is mandatory.

modulefiles_root_dir

Is the root directory for the installation of modulefiles. If it is not specified, root_dir is used. This is the most usual case.

type

Specifies the type of the overlay. The default is normal. See below for the type specifications.

1.5. Overlay types

normal

In an overlay of this types the modules in the overlay are added to the already available modules provided by the overlays on the stack. If modules with identical name/version exist in multiple overlays, the module from the overlay which is higher on the stack is used. In other words modules with identical name/version are replaced by overlays loaded later.

Example: Assume we have the overlays base and private. In the base overlay are the modules git/2.33.1, gnuplot/5.4.0 and gnuplot/5.4.2 in the group Tools. In private overlay are the modules gnuplot/5.4.2 and gnuplot/5.4.3 (maybe compiled with a different configuration).

After loading the private overlay the following Gnuplot modules are available:

  • git/2.33.1`and `gnuplot/5.4.0 from the base overlay

  • gnuplot/5.4.2 and gnuplot/5.4.3 from the private overlay

    Note: the module gnuplot/5.4.2 from the base overlay has been replaced by the module with the same name from the private overlay.

hiding

An overlay of this types hides all modules with the same name in previous loaded overlays.

Example: Assume we have the same overlays and Gnuplot modules as in the example above.

After loading the private overlay with type hiding the following Gnuplot modules are available:

  • git/2.33.1 from the base overlay

  • gnuplot/5.4.2 and gnuplot/5.4.3 from the private overlay

    Note: the Gnuplot modules from the base overlay are hidden!

replacing

A overlay of this types replaces hole groups.

Example: If you load the private overlay from the examples above as replacing, the following modules are available in the group Tools:

  • gnuplot/5.4.2 and gnuplot/5.4.3 from the private overlay

    Note: the Git module from the base overlay is not available any more.

1.6. Variant files

With the implementation of overlays a new format for the so called variant files is necessary. In the old format extensions are almost impossible. Nowadays a common format used for configuration files is YAML.

1.6.1. YAML format of variant files

---
defaults:
  overlay: <overlay-name>
module_name/module_version:
  variants:
  - relstage: <release-stage>
    use_flags: <flags>
    overlay: <overlay-name>
    dependencies:
    - dependency_1
    - dependency_2
    - ...
...
module_name:
  rel_stage: <release-stage>
  use_flags: <use-flags>
  overlay: <overlay>
  versions:
    <id-1>:                           #
      rel_stage: <release-stage>
      use_flags: <use-flags>
      overlay: <overlay>
      version: <version>
      dependencies:
        - dependency1
        - ...
    <id-2>:
   ...
overlay

Name of the overlay the module will be installed in. The specified overlay must be defined in one of the overlay definition files. If no overlay is specified the base overlay is used.

module_name/module_version

Specification of the module with this name and version. For each version a specification should be added.

variants

Array of variants.

relstage

Release stage, either unstable, stable or deprecated. If not specified the release stage defaults to unstable.

modulefiles_root_dir

This key can be used to overwrite the modulefiles_root_dir specified in the overlay definition file.

dependencies

Array of dependencies. Dependencies can have one of the following prefixes:

  • b: build-dependency: A build-dependency is required at build time and must be loaded to compile the module. But it is not required at runtime.

  • i: install-dependency: A install-dependency is required at runtime, but must not be loaded at runtime of the module. An example is the Intel compiler. It requires GCC but GCC must not be loaded. A install-dependency is loaded at build-time.

  • no prefix: dependencies without prefix must be loaded at build- and runtime.

1.6.2. Variants file and overlays

Clone repository
  • Overlays
  • Home