Skip to content

GitLab

S
src

Compile OPAL

Last edited by Achim Gsell
Page history

Compiling OPAL

1. Introduction

Compiling OPAL can be quite challenging due to all the required libraries. Even if some libraries are available on the system, this doesn’t mean that they can be used for compiling OPAL due to missing features. In this document we explain how to compile OPAL from scratch - including the recommended compiler and all libraries.

2. Requirements

2.1. System requirements

  • Operating system

    • Linux

    • Windows with Windows Services for Linux (WSL)

    • Mac with macos > 10.12

  • Required development tools/packages

    • macOS

      • Xcode

      • Command line tools

      • optional: Macports, Homebrew, Fink or another package manager

    • Linux (including WSL)

      • C/C compiler supporting C11 or better

      • optional: Fortran 95 compiler or better

      • make

      • bison or compatible

      • flex or compatible

      • autotools, cmake

      • standard tools like: bzip2, gzip, xz, tar

 

2.2. Required software

Before compiling everything yourself, you might want to try the versions available on your system. Check the requirements below before trying. The list below shows the required software, the minimal and the recommended version.

Name minimum Version recommended Versions Notes

C/C++ Compiler

must support C++ 11 or better. For further instructions see section below.

Fortran Compiler

optional, must support Fortan 95 or better.

cmake

3.0.0

3.11.4 or newer

1.With some version prior to 3.11.4 we had problems configuring HDF5 and Boost.
2. For compiling CMake version 3.11.4 and newer a C++ compiler supporting C++11 is required.

Open MPI

3.1.2

3.1.6

as alternative MPICH can be used

zlib

1.2.11

1.2.11

use system zlib if available (libz.a or libz.so)

hdf5

1.8.18

1.10.6

A parallel version is required.

GNU Scientific Library

2.5

2.6

VTK

7.1.1

8.2.0

optional: required if you want to compile H5hut with the VTK to H5hut mesh converter.

H5hut

2.0.0rc6

2.0.0rc6

A parallel version is required.

boost

1.66.0

1.73.0

The following boost libraries are required: chrono, filesystem, iostreams, regex, serialization, system

ParMETIS

4.0.0

4.0.3

optional: required for OPAL compiled with SAAMG solver.

OpenBLAS

0.2.9

0.3.10 or newer

optional: required for OPAL compiled with SAAMG solver.

trilinos

12.10.1

12.12.1 (macOS)
12.18.1 (Linux)

1. optional: required for OPAL compiled with SAAMG solver.
2. Version 12.18.1 does not compile on macOS.

AMReX

18.07

18.07

optional: required for OPAL compiled with AMR solvers.

google-test

1.7.0

1.10.0 or newer

optional: Required to run the unit-tests

2.3. Compiler

2.3.1. GNU Compiler Collection

The GNU Compiler Collection (GCC) is the recommended compiler for OPAL on a Linux system and on Windows with the Windows Services for Linux. If you want to use GCC on macOS, it is strongly recommended to use GCC provided by Macports, Homebrew, Fink or another software package manager.

NOTE: If you do not want to use the compilers provided by the system (Linux distribution, Xcode,…​), make sure that the commands cc and c++ are calling the right binary. At least with GCC from Macports cc and c++ are not set with port select.

Name minimum Version recommended Version Notes

GNU Compiler Collection

7.3.0

7.5.x,8.4.x, 9.3.x

For the time being GCC 10.1 is not recommended. GCC 10.x fails to compile AMReX.

If you want to compile GCC, you have to compile the following libraries first with an already available GCC:

Name minimum Version recommended Version Notes

GNU Multiple Precision Arithmetic Library (GMP)

6.2.0

6.2.0 or newer

GNU MPFR Library

4.0.1

4.0.2

GNU MPC Library

1.1.0

1.1.0

2.3.2. Clang/LLVM (macOS only)

Clang is the recommended compiler for OPAL on macOS - as long as you do not need AMR. If you want to compile OPAL with AMR, a Fortran compiler is required. In this case you can either use the new Flang front-end of LLVM or GCC from one of the package managers.

Note: The LLVM Fortran front-end Flang is not included in Xcode. You have to compile it yourself.

Name minimum Version recommended Version Notes

Clang (LLVM)

9.0.0

11.0.3

1. Xcode 9 (macOS 10.12)
2. Xcode 11 (macOS 10.15)

3. Compile OPAL on a Linux system @PSI

The easiest and recommended solution is to use the Pmodules environment available at PSI. Please read this wiki-page if you are not using a PSI Linux distribution or you do not know whether the Pmodules are installed on your system.

Instead of loading each single module, you can use the meta-modules available in the directory /afs/psi.ch/project/amas/modulefiles/OPAL. To make these meta-modules available, run the command:

module use /afs/psi.ch/project/amas/modulefiles/OPAL

3.1. Compiling on Merlin6

On Merlin6 special version of the modules should be used. These modules provide a better integration with the SLURM batch system. The recommended modules are:

cmake/3.15.5
gcc/8.4.0
openmpi/3.1.6_slurm
hdf5/1.10.6_slurm
gsl/2.6
H5hut/2.0.0rc6_slurm
boost/1.70.0_slurm
amrex/18.07_3d_slurm
OpenBLAS/0.3.10
parmetis/4.0.3_slurm
trilinos/12.18.1_slurm
gtest/1.10.0

To load these modules, load the toolchain master_slurm:

module load toolchain/master_slurm

Note: This toolchain can be used to compile OPAL 2.2.x, OPAL 2.4.x and the current master branch.

3.2. Other Linux Systems

To compile OPAL 2.0.x or 2.2.x

Use the modules:

cmake/3.10.3
gcc/7.3.0
openmpi/3.1.3
boost/1.68.0
hdf5/1.10.4
H5hut/2.0.0rc5
gsl/2.5
trilinos/12.12.1
OpenBLAS/0.2.20

To load these modules, load the toolchain 2.0 or 2.2:

module load toolchain/2.0

or

module load toolchain/2.2

To compile OPAL 2.4.x or master

Use the modules:

cmake/3.15.5
gcc/8.4.0
openmpi/3.1.6
hdf5/1.10.6
gsl/2.6
H5hut/2.0.0rc6
boost/1.70.0
amrex/18.07_3d
OpenBLAS/0.3.10
parmetis/4.0.3
trilinos/12.18.1
gtest/1.10.0

To load these modules, load the toolchain 2.4 or master:

module load toolchain/2.4

or

module load toolchain/master

4. Build the OPAL tool-chain from scratch

If you do not have access to the required software in binary form - either via a package manager or a module environment - you have to compile the missing pieces or everything yourself.

We provide recipes to build all required software. These recipes can be used on Linux, macOS and Windows Services for Linux. They are tested with, several Linux distributions including Redhat Enterprise Linux 6 & 7, Ubuntu 18 & 20, openSUE leaf 15, macOS 10.12 - 10.15 as well as with the Windows Services for Linux.

4.1. The OPAL Tool-chain Builder

The OPAL Tool-chain Builder (OTB) is a collection of recipes to build all required software you need to compile OPAL.

Run the following command to download the recipes:

git clone git@gitlab.psi.ch:OPAL/build-recipes.git

or

git clone https://gitlab.psi.ch/OPAL/build-recipes.git

4.2. Setting up the environment

First change into the directory you used in the above Git command. In the following we assume that you are in this directory.

To setup the build environment, you have to source the file setup.sh. The syntax is:

source setup [--prefix OTB_PREFIX] [CONFIG_FILE]

If you source setup.sh without arguments,

  • ${HOME}/OPAL will be used as installation directory.

  • Open MPI will be used

  • the default versions defined in the build recipes will be used.

  • GCC will be used on Linux (and WSL)

  • Clang will be used on macOS

With the option --prefix OTB_PREFIX a root directory for all to be installed files can be defined.will be used.

The configuration file can be used * to override the default toolchain for the OS. * to override the default versions of the software packages * to select a MPI implementation (open-mpi or MPICH) * to define a list of recipies which must be executed * to define a list of symbolic links in $OTB_PREFIX, which are created by setup.sh.

For more details please see the configurarion files in the config sub-directory.

After sourcing setup.sh the following environment variable are set (among others):

OTB_PREFIX

Installation prefix for all software we have to compile and install. In the following instructions we use ${HOME}/OPAL.

OTB_DOWNLOAD_DIR

This is the directory where we store downloaded files. In this instruction we use ${PREFIX}/tmp/Downloads.

OTB_SRC_DIR

This is the directory where we unpack and compile software. Here we use ${PREFIX}/tmp/src.

NJOBS

The number of parallel make jobs. This number depends on the number of core on your system. We use getconf _NPROCESSORS_ONLN to get the number of cores (including hyperthreading). The number of parallel make jobs is set to 10, if we have more cores.

OTB_RECIPES (optional)

Array with recipes to be executed. This might be set in a configuration file. This environment variable is set in the configuration files we provide. It can be used to compile everything in one step with the following command:

for recipe in "${OTB_RECIPES[@]}"; do ${recipe} || break; done

4.3. Compiling the tool-chain

Compiling GCC

Compile GCC only if really required!

On macOS use Clang provided by Xcode are a GCC provided by a package manager!

./010-build-gmp
./020-build-mpfr
./030-build-mpc
./040-build-gcc

Compiling CMake

Before compiling CMake, check whether it is installed on your system and if whether the version fulfil the requirements.

./050-build-cmake

Everything else

./060-build-openmpi
./070-build-zlib
./080-build-hdf5
./090-build-gsl
./100-build-h5hut
./110-build-boost
./200-build-parmetis
./210-build-openblas
./220-build-trilinos
./230-build-amrex
./300-build-gtest

4.4. macOS: Using the tool-chain and Macports, Homebrew or …​

Most of the required packages are available via Macports, Homebrew or another package manager. If you want to use them, you should know what you are doing.

If you are using one of this package managers, it is recommended to use a GCC provided by it - in case you do not want to use Clang.

Other packages which are uncritical are

cmake
openmpi
mpich

5. Building OPAL

5.1. Get the OPAL sources

Clone the OPAL Git repository

mkdir -p "${SRC_DIR}/OPAL"
cd "${SRC_DIR}/OPAL"
git clone https://gitlab.psi.ch/OPAL/src.git

Select the OPAL branch you want to compile. The current stable branch is OPAL-2.0:

cd "${SRC_DIR}/OPAL/src"
git checkout OPAL-2.0

To checkout the (unstable) development branch run:

cd "${SRC_DIR}/OPAL/src"
git checkout master

To get a list of all available branches, you can run the command

git branch -a

from inside your clone.

5.2. Setup OPAL build environment

The environment must be set up as described above for building the dependencies.

5.3. Configure OPAL

OPAL uses CMake to configure the build process. You can either pass options to cmake or you can run the command ccmake in the build directory after calling cmake.

mkdir -p "${SRC_DIR}/OPAL/build" && cd "$_"
CC=mpicc CXX=mpicxx cmake \
        --prefix="${PREFIX}" \
        "${SRC_DIR}/OPAL/src"

The following table shows the most important options:

CMAKE_BUILD_TYPE

build type can be either Debug, Release, RelWithDebInfo or MinSizeRel

CMAKE_INSTALL_PREFIX

prefix for installation

ENABLE_AMR

enable AMR solver

requires AMReX (release 18.07)

ENABLE_AMR_MG_SOLVER

enable AMR multigrid solver

requires trilinos >= 12.x and AMReX (release 18.07)

ENABLE_SAAMG_SOLVER

enable SAAMG solver

requires trilinos >= 12.x and parmetis >= 4.0.3

5.4. Compile and install OPAL

cd "${SRC_DIR}/OPAL/build"
make -j ${NJOBS}
make install

5.5. Compile OPAL with the SAAMG solver

To compile OPAL with the SAAMG follow the steps in the section "Compile OPAL", but replace the configuration step with:

mkdir -p "${SRC_DIR}/OPAL/build"
cd "${SRC_DIR}/OPAL/build"
CC=mpicc CXX=mpicxx cmake \
        --prefix="${PREFIX}" \
        -D ENABLE_SAAMG_SOLVER=TRUE \
        "${SRC_DIR}/OPAL/src"