Optimizer stops to write to disk after a couple of configurations

I'm having problems with the configurations in this repo, in the directory optim-quads.

Steps to reproduce

• Run the batch job provided in the repo (change the path to OPAL to your path)

• The job runs normally, but after 2min or so, nothing is written to the disk anymore, even if you wait 20min

• Problems do not arise if --ntasks=8 is set in the Slurm file

• The same job runs fine with OPAL 2.0, so it is not a bad configuration

added Bug OPAL-T + 1 deleted label

added Optimiser label

This happens with the version that Matthias provided, correct?

@frey_m did you check if the directory already exists?

Yes, but also with the latest Master (commit 03bf3d6b).

@adelmann As @bellotti_r mentioned, this issue has nothing to do with my changes.

I applied the changes also to another branch derived from the OPAL-2.0 branch such that @bellotti_r is able to work until this issue is figured out.

Just a small observation. When running with master, KEEP is not an attribute of OPTIMIZE and the line needs to be commented out.

@snuverink_j Yes, that's right. The KEEP flag I added for @bellotti_r in a branch.

There are "just" 56 commits that changed the optimizer. Here's the list of all commits.

I'm trying to bisect this list of commits to reduce the number of commits that have to be checked.

@kraus Thanks.

I've tried with 695cf3a9 (28. on the list of commits), 8c0a18cc (14.), c5d47d25 (7.), d1af3752 (4.), d4e3216c (2.) and the head of the master. All ran smoothly, i.e. I can't reproduce this issue...

Thanks for your effort. Which versions of the libraries are you using? My setup is as follows:

gcc/7.3.0
cmake/3.9.6
gsl/2.5
boost/1.68.0
openmpi/3.1.3
hdf5/1.10.4
H5hut/2.0.0rc5

I use the module opal-toolchains/gcc-7.3.0-openmpi-3.1.3-hdf5-1.10.4 which provides the same versions except for cmake (3.10.3).

I don't think that changes much.

I suspect the number of cores is the problem. When I run the configuration with 8 cores, it works. When I change to 128, it doesn't.

How many cores have you been using?

172 as in the slurm file in your repository.

You could try my binary under ~kraus/build/opal/src on merlin.

I don't have sufficient permissions to read your binary.

I changed the permissions on my home directory. You could try again.

Your binary seems to work. I'll try to recompile OPAL with the module you're using.

I can't seem to reproduce this either on master with either 128 or 172 cores.

It is working now. So it's really just the version of cmake that made the difference.

Thanks for all your help! :)

closed

Why should the cmake version have an impact on that? The only thing I can imagine is the issue of linking with a wrong library.

I don't know. My first htought was that I might've made a mistake during my first compilation, but you've experienced the same issue, right? Therefore the only difference is the cmake version...

That's true, I experienced the same but I just do not understand why it should be an issue of cmake. Could you compare the cmake logs?

Another thing. It works with OPAL-2.0, right. There you used cmake/3.9.6, too. Is this correct?

Yes, I've used the same modules for both.

Here are the cmake logs. I couldn't find any difference apart from the cmake version. Very strange...

CMakeOutput_provided_modules.log

CMakeOutput_manual_modules.log

The BOOST version is different and in one there's no Trilinos. Could you also send the linking output, i.e. make VERBOSE=1

make_output_manual_modules.txt

make_output_automatic_modules.txt

@frey_m The problem persists in your branch (OPAL 2.1)