Compiling OPAL
1. Introduction
Compiling OPAL can be quite challenging due to all the required libraries. Even if some libraries are available on the system, this doesn’t mean that they can be used for compiling OPAL due to missing features. In this document we explain how to compile OPAL from scratch - including the recommended compiler and all libraries.
2. Requirements
2.1. System requirements
-
Operating system
-
Linux
-
Windows with Windows Services for Linux (WSL)
-
Mac with macos > 10.12
-
-
Required development tools/packages
-
macOS
-
Xcode
-
Command line tools
-
optional: Macports, Homebrew, Fink or another package manager
-
-
Linux (including WSL)
-
C/C compiler supporting C11 or better
-
optional: Fortran 95 compiler or better
-
make
-
bison or compatible
-
flex or compatible
-
autotools, cmake
-
standard tools like: bzip2, gzip, xz, tar
-
-
2.2. Required software
Before compiling everything yourself, you might want to try the versions available on your system. Check the requirements below before trying. The list below shows the required software, the minimal and the recommended version.
Name | minimum Version | recommended Versions | Notes |
---|---|---|---|
C/C++ Compiler |
must support C++ 11 or better. For further instructions see section below. |
||
Fortran Compiler |
optional, must support Fortan 95 or better. |
||
cmake |
3.0.0 |
3.11.4 or newer |
1.With some version prior to 3.11.4 we had problems configuring HDF5 and Boost. |
Open MPI |
3.1.2 |
3.1.6 |
as alternative MPICH can be used |
zlib |
1.2.11 |
1.2.11 |
use system zlib if available ( |
hdf5 |
1.8.18 |
1.10.6 |
A parallel version is required. |
GNU Scientific Library |
2.5 |
2.6 |
|
VTK |
7.1.1 |
8.2.0 |
optional: required if you want to compile H5hut with the VTK to H5hut mesh converter. |
H5hut |
2.0.0rc6 |
2.0.0rc6 |
A parallel version is required. |
boost |
1.66.0 |
1.73.0 |
The following boost libraries are required: |
ParMETIS |
4.0.0 |
4.0.3 |
optional: required for OPAL compiled with SAAMG solver. |
OpenBLAS |
0.2.9 |
0.3.10 or newer |
optional: required for OPAL compiled with SAAMG solver. |
trilinos |
12.10.1 |
12.12.1 (macOS) |
1. optional: required for OPAL compiled with SAAMG solver. |
AMReX |
18.07 |
18.07 |
optional: required for OPAL compiled with AMR solvers. |
google-test |
1.7.0 |
1.10.0 or newer |
optional: Required to run the unit-tests |
2.3. Compiler
2.3.1. GNU Compiler Collection
The GNU Compiler Collection (GCC) is the recommended compiler for OPAL on a Linux system and on Windows with the Windows Services for Linux. If you want to use GCC on macOS, it is strongly recommended to use GCC provided by Macports, Homebrew, Fink or another software package manager.
NOTE: If you do not want to use the compilers provided by the system (Linux distribution, Xcode,…), make sure that the commands
cc
andc++
are calling the right binary. At least with GCC from Macportscc
andc++
are not set withport select
.
Name | minimum Version | recommended Version | Notes |
---|---|---|---|
GNU Compiler Collection |
7.3.0 |
7.5.x,8.4.x, 9.3.x |
For the time being GCC 10.1 is not recommended. GCC 10.x fails to compile AMReX. |
If you want to compile GCC, you have to compile the following libraries first with an already available GCC:
Name | minimum Version | recommended Version | Notes |
---|---|---|---|
GNU Multiple Precision Arithmetic Library (GMP) |
6.2.0 |
6.2.0 or newer |
|
GNU MPFR Library |
4.0.1 |
4.0.2 |
|
GNU MPC Library |
1.1.0 |
1.1.0 |
2.3.2. Clang/LLVM (macOS only)
Clang is the recommended compiler for OPAL on macOS - as long as you do not need AMR. If you want to compile OPAL with AMR, a Fortran compiler is required. In this case you can either use the new Flang front-end of LLVM or GCC from one of the package managers.
Note: The LLVM Fortran front-end Flang is not included in Xcode. You have to compile it yourself.
Name | minimum Version | recommended Version | Notes |
---|---|---|---|
Clang (LLVM) |
9.0.0 |
11.0.3 |
1. Xcode 9 (macOS 10.12) |
3. Compile OPAL on a Linux system @PSI
The easiest and recommended solution is to use the Pmodules environment available at PSI. Please read this wiki-page if you are not using a PSI Linux distribution or you do not know whether the Pmodules are installed on your system.
Instead of loading each single module, you can use the meta-modules
available in the directory
/afs/psi.ch/project/amas/modulefiles/OPAL
. To make these
meta-modules available, run the command:
module use /afs/psi.ch/project/amas/modulefiles/OPAL
3.1. Compiling on Merlin6
On Merlin6 special version of the modules should be used. These modules provide a better integration with the SLURM batch system. The recommended modules are:
cmake/3.15.5 gcc/8.4.0 openmpi/3.1.6_slurm hdf5/1.10.6_slurm gsl/2.6 H5hut/2.0.0rc6_slurm boost/1.70.0_slurm amrex/18.07_3d_slurm OpenBLAS/0.3.10 parmetis/4.0.3_slurm trilinos/12.18.1_slurm gtest/1.10.0
To load these modules, load the toolchain master_slurm
:
module load toolchain/master_slurm
Note: This toolchain can be used to compile OPAL 2.2.x, OPAL 2.4.x and the current master branch.
3.2. Other Linux Systems
To compile OPAL 2.0.x or 2.2.x
Use the modules:
cmake/3.10.3 gcc/7.3.0 openmpi/3.1.3 boost/1.68.0 hdf5/1.10.4 H5hut/2.0.0rc5 gsl/2.5 trilinos/12.12.1 OpenBLAS/0.2.20
To load these modules, load the toolchain 2.0
or 2.2
:
module load toolchain/2.0
or
module load toolchain/2.2
To compile OPAL 2.4.x or master
Use the modules:
cmake/3.15.5 gcc/8.4.0 openmpi/3.1.6 hdf5/1.10.6 gsl/2.6 H5hut/2.0.0rc6 boost/1.70.0 amrex/18.07_3d OpenBLAS/0.3.10 parmetis/4.0.3 trilinos/12.18.1 gtest/1.10.0
To load these modules, load the toolchain 2.4
or master
:
module load toolchain/2.4
or
module load toolchain/master
4. Build the OPAL tool-chain from scratch
If you do not have access to the required software in binary form - either via a package manager or a module environment - you have to compile the missing pieces or everything yourself.
We provide recipes to build all required software. These recipes can be used on Linux, macOS and Windows Services for Linux. They are tested with, several Linux distributions including Redhat Enterprise Linux 6 & 7, Ubuntu 18 & 20, openSUE leaf 15, macOS 10.12 - 10.15 as well as with the Windows Services for Linux.
4.1. The OPAL Tool-chain Builder
The OPAL Tool-chain Builder (OTB) is a collection of recipes to build all required software you need to compile OPAL.
Run the following command to download the recipes:
git clone git@gitlab.psi.ch:OPAL/build-recipes.git
or
git clone https://gitlab.psi.ch/OPAL/build-recipes.git
4.2. Setting up the environment
First change into the directory you used in the above Git command. In the following we assume that you are in this directory.
To setup the build environment, you have to source the file
setup.sh
. The syntax is:
source setup [--prefix OTB_PREFIX] [CONFIG_FILE]
If you source setup.sh
without arguments,
-
${HOME}/OPAL
will be used as installation directory. -
Open MPI will be used
-
the default versions defined in the build recipes will be used.
-
GCC will be used on Linux (and WSL)
-
Clang will be used on macOS
With the option --prefix OTB_PREFIX
a root directory for all to be
installed files can be defined.will be used.
The configuration file can be used
* to override the default toolchain for the OS.
* to override the default versions of the software packages
* to select a MPI implementation (open-mpi or MPICH)
* to define a list of recipies which must be executed
* to define a list of symbolic links in $OTB_PREFIX
, which are
created by setup.sh
.
For more details please see the configurarion files in the config
sub-directory.
After sourcing setup.sh
the following environment variable are set (among others):
OTB_PREFIX
-
Installation prefix for all software we have to compile and install. In the following instructions we use
${HOME}/OPAL
. OTB_DOWNLOAD_DIR
-
This is the directory where we store downloaded files. In this instruction we use
${PREFIX}/tmp/Downloads
. OTB_SRC_DIR
-
This is the directory where we unpack and compile software. Here we use
${PREFIX}/tmp/src
. NJOBS
-
The number of parallel
make
jobs. This number depends on the number of core on your system. We usegetconf _NPROCESSORS_ONLN
to get the number of cores (including hyperthreading). The number of parallelmake
jobs is set to 10, if we have more cores. -
OTB_RECIPES
(optional) -
Array with recipes to be executed. This might be set in a configuration file. This environment variable is set in the configuration files we provide. It can be used to compile everything in one step with the following command:
for recipe in "${OTB_RECIPES[@]}"; do ${recipe} || break; done
4.3. Compiling the tool-chain
Compiling GCC
Compile GCC only if really required!
On macOS use Clang provided by Xcode are a GCC provided by a package manager!
./010-build-gmp ./020-build-mpfr ./030-build-mpc ./040-build-gcc
Compiling CMake
Before compiling CMake, check whether it is installed on your system and if whether the version fulfil the requirements.
./050-build-cmake
Everything else
./060-build-openmpi ./070-build-zlib ./080-build-hdf5 ./090-build-gsl ./100-build-h5hut ./110-build-boost ./200-build-parmetis ./210-build-openblas ./220-build-trilinos ./230-build-amrex ./300-build-gtest
4.4. macOS: Using the tool-chain and Macports, Homebrew or …
Most of the required packages are available via Macports, Homebrew or another package manager. If you want to use them, you should know what you are doing.
If you are using one of this package managers, it is recommended to use a GCC provided by it - in case you do not want to use Clang.
Other packages which are uncritical are
cmake openmpi mpich
5. Building OPAL
5.1. Get the OPAL sources
Clone the OPAL Git repository
mkdir -p "${SRC_DIR}/OPAL" cd "${SRC_DIR}/OPAL" git clone https://gitlab.psi.ch/OPAL/src.git
Select the OPAL branch you want to compile. The current stable branch is OPAL-2.0
:
cd "${SRC_DIR}/OPAL/src" git checkout OPAL-2.0
To checkout the (unstable) development branch run:
cd "${SRC_DIR}/OPAL/src" git checkout master
To get a list of all available branches, you can run the command
git branch -a
from inside your clone.
5.2. Setup OPAL build environment
The environment must be set up as described above for building the dependencies.
5.3. Configure OPAL
OPAL uses CMake to configure the build process. You can either pass options to cmake
or you can run the command ccmake
in the build directory after calling cmake
.
mkdir -p "${SRC_DIR}/OPAL/build" && cd "$_" CC=mpicc CXX=mpicxx cmake \ --prefix="${PREFIX}" \ "${SRC_DIR}/OPAL/src"
The following table shows the most important options:
|
build type can be either |
|
|
prefix for installation |
|
|
enable AMR solver |
requires AMReX (release 18.07) |
|
enable AMR multigrid solver |
requires trilinos >= 12.x and AMReX (release 18.07) |
|
enable SAAMG solver |
requires trilinos >= 12.x and parmetis >= 4.0.3 |
5.5. Compile OPAL with the SAAMG solver
To compile OPAL with the SAAMG follow the steps in the section "Compile OPAL", but replace the configuration step with:
mkdir -p "${SRC_DIR}/OPAL/build" cd "${SRC_DIR}/OPAL/build" CC=mpicc CXX=mpicxx cmake \ --prefix="${PREFIX}" \ -D ENABLE_SAAMG_SOLVER=TRUE \ "${SRC_DIR}/OPAL/src"