Ten fold increase in compute time for external field evaluation
Summary
When running some simulations a drastic increase in overall compute time is noticed. This seems to originate from the evaluation of external fields.
Steps to reproduce
Run the same simulation with Opal 2.2 and 2.4 and compare the timings.
What is the current bug behavior?
The time to evaluate the external fields has regressed from Opal 2.2.
What is the expected correct behavior?
The evaluation of the external fields shouldn't take longer as in Opal 2.2
Relevant logs and/or screenshots
Timing for Opal 2.4
Timings{0}> -----------------------------------------------------------------
Timings{0}> Timing results for 16 nodes:
Timings{0}> -----------------------------------------------------------------
Timings{0}> mainTimer........... Wall tot = 1333.67, CPU tot = 1304.62
Timings{0}>
Timings{0}> Binaryrepart........ Wall max = 0.0760612, CPU max = 0.12
Timings{0}> Wall avg = 0.0748878, CPU avg = 0.076875
Timings{0}> Wall min = 0.0710163, CPU min = 0.04
Timings{0}>
Timings{0}> Boundingbox......... Wall max = 21.1333, CPU max = 21.51
Timings{0}> Wall avg = 20.492, CPU avg = 20.3581
Timings{0}> Wall min = 19.5912, CPU min = 18.98
Timings{0}>
Timings{0}> Boundingbox-bounds.. Wall max = 3.31469, CPU max = 3.25
Timings{0}> Wall avg = 2.33188, CPU avg = 2.32125
Timings{0}> Wall min = 1.45284, CPU min = 1.43
Timings{0}>
Timings{0}> Boundingbox-update.. Wall max = 12.3236, CPU max = 12.66
Timings{0}> Wall avg = 12.2843, CPU avg = 12.1619
Timings{0}> Wall min = 12.2186, CPU min = 11.72
Timings{0}>
Timings{0}> Compute Statistics.. Wall max = 24.9249, CPU max = 25.18
Timings{0}> Wall avg = 23.5228, CPU avg = 23.6244
Timings{0}> Wall min = 21.6156, CPU min = 21.35
Timings{0}>
Timings{0}> ComputePotential.... Wall max = 212.61, CPU max = 213.74
Timings{0}> Wall avg = 211.029, CPU avg = 210.689
Timings{0}> Wall min = 207.543, CPU min = 207.01
Timings{0}>
Timings{0}> Create Distr........ Wall max = 0.0197351, CPU max = 0.03
Timings{0}> Wall avg = 0.0186866, CPU avg = 0.020625
Timings{0}> Wall min = 0.0185006, CPU min = 0.01
Timings{0}>
Timings{0}> External field eval. Wall max = 711.153, CPU max = 711.74
Timings{0}> Wall avg = 361.201, CPU avg = 361.191
Timings{0}> Wall min = 2.25931, CPU min = 2.35
Timings{0}>
Timings{0}> Fast inside test.... Wall max = 0, CPU max = 0
Timings{0}> Wall avg = 0, CPU avg = 0
Timings{0}> Wall min = 0, CPU min = 0
Timings{0}>
Timings{0}> Histogram........... Wall max = 0, CPU max = 0
Timings{0}> Wall avg = 0, CPU avg = 0
Timings{0}> Wall min = 0, CPU min = 0
Timings{0}>
Timings{0}> Initialize geometry. Wall max = 0, CPU max = 0
Timings{0}> Wall avg = 0, CPU avg = 0
Timings{0}> Wall min = 0, CPU min = 0
Timings{0}>
Timings{0}> Inside test......... Wall max = 0, CPU max = 0
Timings{0}> Wall avg = 0, CPU avg = 0
Timings{0}> Wall min = 0, CPU min = 0
Timings{0}>
Timings{0}> Load Distr.......... Wall max = 0, CPU max = 0
Timings{0}> Wall avg = 0, CPU avg = 0
Timings{0}> Wall min = 0, CPU min = 0
Timings{0}>
Timings{0}> Particle Inside..... Wall max = 0, CPU max = 0
Timings{0}> Wall avg = 0, CPU avg = 0
Timings{0}> Wall min = 0, CPU min = 0
Timings{0}>
Timings{0}> Ray tracing......... Wall max = 0, CPU max = 0
Timings{0}> Wall avg = 0, CPU avg = 0
Timings{0}> Wall min = 0, CPU min = 0
Timings{0}>
Timings{0}> SelfField total..... Wall max = 380.273, CPU max = 380.01
Timings{0}> Wall avg = 344.384, CPU avg = 344.401
Timings{0}> Wall min = 317.067, CPU min = 318.26
Timings{0}>
Timings{0}> SF: GreensFTotal.... Wall max = 134.448, CPU max = 134.95
Timings{0}> Wall avg = 133.307, CPU avg = 132.716
Timings{0}> Wall min = 131.767, CPU min = 130.46
Timings{0}>
Timings{0}> TIntegration1....... Wall max = 20.5137, CPU max = 20.53
Timings{0}> Wall avg = 19.1446, CPU avg = 19.1488
Timings{0}> Wall min = 17.3008, CPU min = 17.47
Timings{0}>
Timings{0}> TIntegration2....... Wall max = 38.3021, CPU max = 38.52
Timings{0}> Wall avg = 36.8959, CPU avg = 36.8775
Timings{0}> Wall min = 35.997, CPU min = 35.53
Timings{0}>
Timings{0}> WakeField........... Wall max = 0, CPU max = 0
Timings{0}> Wall avg = 0, CPU avg = 0
Timings{0}> Wall min = 0, CPU min = 0
Timings{0}>
Timings{0}> Write H5-File....... Wall max = 49.9556, CPU max = 21.73
Timings{0}> Wall avg = 49.9353, CPU avg = 20.2519
Timings{0}> Wall min = 49.8026, CPU min = 17.74
Timings{0}>
Timings{0}> Write Stat.......... Wall max = 9.01406, CPU max = 2.07
Timings{0}> Wall avg = 0.833675, CPU avg = 0.405
Timings{0}> Wall min = 0.281822, CPU min = 0.22
Timings{0}>
Timings{0}> -----------------------------------------------------------------Timing for Opal 2.2 and the same input file
Timings{0}> -----------------------------------------------------------------
Timings{0}> Timing results for 16 nodes:
Timings{0}> -----------------------------------------------------------------
Timings{0}> mainTimer........... Wall tot = 698.587, CPU tot = 672.33
Timings{0}>
Timings{0}> Binaryrepart........ Wall max = 0.0804251, CPU max = 0.13
Timings{0}> Wall avg = 0.0792938, CPU avg = 0.078125
Timings{0}> Wall min = 0.0762691, CPU min = 0.04
Timings{0}>
Timings{0}> Boundingbox......... Wall max = 21.9074, CPU max = 22.05
Timings{0}> Wall avg = 21.2645, CPU avg = 21.2381
Timings{0}> Wall min = 20.2972, CPU min = 19.86
Timings{0}>
Timings{0}> Boundingbox-bounds.. Wall max = 3.23519, CPU max = 3.3
Timings{0}> Wall avg = 2.29564, CPU avg = 2.33812
Timings{0}> Wall min = 1.43132, CPU min = 1.19
Timings{0}>
Timings{0}> Boundingbox-update.. Wall max = 13.0062, CPU max = 13.74
Timings{0}> Wall avg = 12.9693, CPU avg = 12.9756
Timings{0}> Wall min = 12.8177, CPU min = 12.44
Timings{0}>
Timings{0}> Compute Statistics.. Wall max = 26.1269, CPU max = 26.62
Timings{0}> Wall avg = 24.6571, CPU avg = 24.815
Timings{0}> Wall min = 22.6815, CPU min = 22.65
Timings{0}>
Timings{0}> ComputePotential.... Wall max = 221.744, CPU max = 222.57
Timings{0}> Wall avg = 219.925, CPU avg = 219.121
Timings{0}> Wall min = 215.744, CPU min = 214.95
Timings{0}>
Timings{0}> Create Distr........ Wall max = 0.0196037, CPU max = 0.03
Timings{0}> Wall avg = 0.01854, CPU avg = 0.019375
Timings{0}> Wall min = 0.0183704, CPU min = 0.01
Timings{0}>
Timings{0}> External field eval. Wall max = 83.0337, CPU max = 83.09
Timings{0}> Wall avg = 37.3202, CPU avg = 37.67
Timings{0}> Wall min = 2.13352, CPU min = 2.24
Timings{0}>
Timings{0}> Fast inside test.... Wall max = 0, CPU max = 0
Timings{0}> Wall avg = 0, CPU avg = 0
Timings{0}> Wall min = 0, CPU min = 0
Timings{0}>
Timings{0}> Histogram........... Wall max = 0, CPU max = 0
Timings{0}> Wall avg = 0, CPU avg = 0
Timings{0}> Wall min = 0, CPU min = 0
Timings{0}>
Timings{0}> Initialize geometry. Wall max = 0, CPU max = 0
Timings{0}> Wall avg = 0, CPU avg = 0
Timings{0}> Wall min = 0, CPU min = 0
Timings{0}>
Timings{0}> Inside test......... Wall max = 0, CPU max = 0
Timings{0}> Wall avg = 0, CPU avg = 0
Timings{0}> Wall min = 0, CPU min = 0
Timings{0}>
Timings{0}> Load Distr.......... Wall max = 0, CPU max = 0
Timings{0}> Wall avg = 0, CPU avg = 0
Timings{0}> Wall min = 0, CPU min = 0
Timings{0}>
Timings{0}> Particle Inside..... Wall max = 0, CPU max = 0
Timings{0}> Wall avg = 0, CPU avg = 0
Timings{0}> Wall min = 0, CPU min = 0
Timings{0}>
Timings{0}> Ray tracing......... Wall max = 0, CPU max = 0
Timings{0}> Wall avg = 0, CPU avg = 0
Timings{0}> Wall min = 0, CPU min = 0
Timings{0}>
Timings{0}> Secondary emission.. Wall max = 0, CPU max = 0
Timings{0}> Wall avg = 0, CPU avg = 0
Timings{0}> Wall min = 0, CPU min = 0
Timings{0}>
Timings{0}> SelfField total..... Wall max = 398.178, CPU max = 399.5
Timings{0}> Wall avg = 359.377, CPU avg = 358.998
Timings{0}> Wall min = 330.56, CPU min = 330.17
Timings{0}>
Timings{0}> SF: GreensFTotal.... Wall max = 140.186, CPU max = 141.31
Timings{0}> Wall avg = 139.146, CPU avg = 138.486
Timings{0}> Wall min = 137.841, CPU min = 135.62
Timings{0}>
Timings{0}> TIntegration1....... Wall max = 21.4125, CPU max = 21.77
Timings{0}> Wall avg = 20.1085, CPU avg = 19.935
Timings{0}> Wall min = 18.3236, CPU min = 18.12
Timings{0}>
Timings{0}> TIntegration2....... Wall max = 40.1375, CPU max = 40.82
Timings{0}> Wall avg = 38.7925, CPU avg = 38.8013
Timings{0}> Wall min = 37.9342, CPU min = 37.21
Timings{0}>
Timings{0}> WakeField........... Wall max = 0, CPU max = 0
Timings{0}> Wall avg = 0, CPU avg = 0
Timings{0}> Wall min = 0, CPU min = 0
Timings{0}>
Timings{0}> Write H5-File....... Wall max = 47.1853, CPU max = 21.09
Timings{0}> Wall avg = 47.1643, CPU avg = 18.93
Timings{0}> Wall min = 47.0338, CPU min = 15.95
Timings{0}>
Timings{0}> Write Stat.......... Wall max = 9.99641, CPU max = 3.93
Timings{0}> Wall avg = 2.74196, CPU avg = 2.36437
Timings{0}> Wall min = 2.17362, CPU min = 2.09
Timings{0}>
Timings{0}> -----------------------------------------------------------------
To upload designs, you'll need to enable LFS. More information