Many failed regression tests

Today there were many failed regression tests: http://amas.web.psi.ch/opal/regressionTests/master/results_2019-05-10.xml

There were four merge requests (issue assigned to the authors of these):

1. !82 (merged)
2. !87 (merged)
3. !90 (merged)
4. !92 (merged)

I had a quick look and I suspect this change in the PoissonSolver https://gitlab.psi.ch/OPAL/src/blob/master/src/Solvers/FFTPoissonSolver.cpp:

@@ -388,7 +388,7 @@ void FFTPoissonSolver::computePotentialDKS(Field_t &rho) {
    if (Ippl::myNode() == 0) {
        IpplTimings::startTimer(GreensFunctionTimer_m);
-       integratedGreensFunction();
+       integratedGreensFunctionDKS();
        IpplTimings::stopTimer(GreensFunctionTimer_m);
        //transform the greens function
        int dimsize[3] = {2*nr_m[0], 2*nr_m[1], 2*nr_m[2]};

I don't think that the DKS stuff is the cause, rather one of my commits (most of the failed tests use OPAL-T.

This should not be the cause, since this part of the code will be discarded by preprocessor if OPAL is compiled without enabling DKS (and I think that is the case for running regression tests).

My changes mostly affect AMR and the multi-bunch mode in the Cyc-tracker. I think the only cyclotron test that failed is RestartTest-2.

@locans_u : ah yes that is a good point.

It's most likely !82 (merged). I'm working on it.

unassigned @frey_m and @locans_u

mentioned in merge request !95 (merged)

After !95 (merged) the situation is much better, but not fully recovered: http://amas.web.psi.ch/opal/regressionTests/master/results_2019-05-13.xml

This is interesting because I've checked before I commited the changes and all tests using ParallelTTracker (except Degrader-1) passed. And I've just checked again (on merlin) and even the ones that are supposed to fail according to todays results passed. I used the master branch without any changes and there are also no changes in the regression-tests repository.

cc: @gsell

I've ran the failing tests on opalrunner and got the same results as on merlin: all OPAL-T tests (except Degrader-1) passed.

I've just noticed that someone, presumably @gsell, ran the tests again, and the published results now show the same result as I got.

unassigned @kraus

ok strange, @gsell Do we have an explanation?

Also Restart-2 is still failing where it wasn't before.

@snuverink_j Maybe the dumping of the azimuth to the stat file causes Restart-2 to fail.

I was logged in into opalrunner yesterday afternoon and noticed that there are 30 opal processes (opalrunner has 16 threads). It could just be that the some of the tests ran out of time (the regression tests are stopped after 300 seconds).

It seems that Restart-2 finished on time.

Is OPAL compiled with AMR for the regression tests? If yes, in case of AMR I also dump the reference particle. However, I do not hope that this affects the rms that much.

@frey_m OPAL is compiled on opalrunner with the following options:

-DCMAKE_BUILD_TYPE=RELEASE
-DENABLE_SAAMG_SOLVER=TRUE
-DENABLE_AMR=TRUE
-DENABLE_AMR_MG_SOLVER=TRUE
-DBUILD_OPAL_UNIT_TESTS=TRUE

Then we should check if it works with

-DENABLE_AMR=FALSE
-DENABLE_AMR_MG_SOLVER=FALSE

@frey_m : The plots show a small discrepancy after about 35 meters, e.g.

Yes, I saw that when checking. I assume that the restart starts at around 35 meters.

@kraus you are right, I run the tests again. But had no time to check the results and write a comment. Some of the failures can be explained with the high load. Still have to check why these tests were still running.

Then we should check if it works with

-DENABLE_AMR=FALSE
-DENABLE_AMR_MG_SOLVER=FALSE

Presumably @kraus checked this?

I haven't checked the Restart-2 test. But I can do that, the binary is already there.

It passed with

-DENABLE_AMR=FALSE
-DENABLE_AMR_MG_SOLVER=FALSE

         Checksum for reference RestartTest-2.stat OK 
         Checksum for reference RestartTest-2.out OK 
         Checksum for reference RestartTest-2.lbal OK 
         run simulation
Test rms_x(last) passed: 4.3059479593043903e-13 (eps=1e-08) 
Test rms_y(last) passed: 8.803721640582296e-17 (eps=1e-08) 
Test rms_s(last) passed: 9.974659986866641e-18 (eps=1e-08) 

It might be the guard in the PartBunchBase class at line 2062 ... however, I would be astonished about the effect. Sry, I have currently no time to test ... I have other debugging issues.

@frey_m No worries, it is not so urgent (imho), but good that we have cornered the . Assigning to you.

assigned to @frey_m

When disabling the #ifndef ENABLE_AMR in PartBunchBase at the aforementioned line, the solution agrees with the reference solution.

rms_x_comparison.pdf

mentioned in commit 1ab56acb

mentioned in merge request !97 (merged)

mentioned in commit 0a1ee1b9

closed via merge request !97 (merged)

mentioned in commit 7130ecd0