Code indexing in gitaly is broken and leads to code not being visible to the user. We work on the issue with highest priority.

Skip to content
Snippets Groups Projects
Select Git revision
  • master default protected
1 result
Compare
Forked from OPAL / OPAL-X / src
24 commits behind the upstream repository.
Name Last commit Last update
.gitlab
CMakeModules
LICENSES
cmake
hooks
input-files
ipynb
src
tests
tools
.clang-format
.dir-locals.el
.gitignore
.local.vimrc
C++-Readme
CMakeLists.txt
Doxyfile.in
LICENSE
OPAL-202x.code-workspace
README.md
gen_OPALrevision
README.md

This is OPAL-X

BUILDING OPAL-X on Merlin

Modules needed OPENMP build

cmake/3.25.2

openmpi/4.1.5_slurm

fftw/3.3.10_merlin6    

gsl/2.7                

H5hut/2.0.0rc6_slurm

gcc/12.3.0             

boost/1.82.0_slurm     

gtest/1.13.0-1         

hdf5/1.10.8_slurm     

gnutls/3.5.19

Clone repo and build opal-x with OPENMP

$ git clone git@gitlab.psi.ch:OPAL/opal-x/src.git opal-x

$ cd opal-x

$ ./gen_OPALrevision

To compile for OPENMP:

$ mkdir build_openmp && cd build_openmp
$ cmake .. -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_STANDARD=20  -DENABLE_SOLVERS=ON  -DENABLE_FFT=ON -DIPPL_PLATFORMS=openmp

To compile for GPU, for example Amper80 on Gwendolen

$ mkdir build_cuda && cd build_cuda

in debug mode:

$ cmake .. -DCMAKE_BUILD_TYPE=Debug -DIPPL_PLATFORMS=CUDA -DKokkos_ARCH_AMPERE80=ON -DCMAKE_CXX_STANDARD=20 -DENABLE_FFT=ON  -DENABLE_SOLVERS=ON

and release (optimized) mode:

$ cmake .. -DCMAKE_BUILD_TYPE=Release -DIPPL_PLATFORMS=CUDA -DKokkos_ARCH_AMPERE80=ON -DCMAKE_CXX_STANDARD=20 -DENABLE_FFT=ON  -DENABLE_SOLVERS=ON

To compile for other GPU architecture, like Pascal on the Merlin's login node

$ mkdir build_cuda_login && cd build_cuda_login

in debug mode:

$ cmake .. -DCMAKE_BUILD_TYPE=Debug -DIPPL_PLATFORMS=CUDA -DKokkos_ARCH_PASCAL61=ON -DCMAKE_CXX_STANDARD=20 -DENABLE_FFT=ON  -DENABLE_SOLVERS=ON

and release (optimized) mode:

$ cmake .. -DCMAKE_BUILD_TYPE=Release -DIPPL_PLATFORMS=CUDA -DKokkos_ARCH_PASCAL61=ON -DCMAKE_CXX_STANDARD=20 -DENABLE_FFT=ON  -DENABLE_SOLVERS=ON

Submitting jobs on Gwendolen and Merlin GPUs

To execute opal-x on merlin's gpus (compile for PASCAL61), the job script should looks like

#!/bin/bash
#SBATCH --error=merlin.error
#SBATCH --output=merlin.out
#SBATCH --time=00:10:00
#SBATCH --nodes=1
#SBATCH --ntasks=2
#SBATCH --cluster=gmerlin6
#SBATCH --partition=gpu-short
#SBATCH --account=merlin
##SBATCH --exclusive
#SBATCH --gpus=2
#SBATCH --nodelist=merlin-g-001

##unlink core
ulimit -c unlimited

srun ./opalx DriftTest-1.in  --info 10 --kokkos-map-device-id-by=mpi_rank

and for Gwendolen (compile for AMPERE80)

#!/bin/bash
#SBATCH --error=gwendolen.error
#SBATCH --output=gwendolen.out
#SBATCH --time=00:02:00
#SBATCH --nodes=1
#SBATCH --ntasks=2
#SBATCH --clusters=gmerlin6
#SBATCH --partition=gwendolen # Mandatory, as gwendolen is not the default partition
#SBATCH --account=gwendolen   # Mandatory, as gwendolen is not the default account
##SBATCH --exclusive
#SBATCH --gpus=2

##unlink core
ulimit -c unlimited

srun ./opalx DriftTest-1.in  --info 10 --kokkos-map-device-id-by=mpi_rank

The documentation has been moved to the Wiki.