src issueshttps://gitlab.psi.ch/OPAL/src/-/issues2018-12-06T11:13:03+01:00https://gitlab.psi.ch/OPAL/src/-/issues/249Objectives for Sampler2018-12-06T11:13:03+01:00krausObjectives for SamplerThe user should be able to define quantities of interest (a la optimizer) which are written to the json file. This avoids lengthy post processing of the stat files of all runs.The user should be able to define quantities of interest (a la optimizer) which are written to the json file. This avoids lengthy post processing of the stat files of all runs.krauskraushttps://gitlab.psi.ch/OPAL/src/-/issues/644Sampler cleanup with KEEP option not working when directory is present2021-06-11T10:53:01+02:00snuverink_jjochem.snuverink@psi.chSampler cleanup with KEEP option not working when directory is present### Summary
Reported by @ext-piot_p on the OPAL mailing list. With the option `SAMPLE, KEEP` non-empty, non-wanted files should be deleted, but an error message is shown instead when a non-empty subdirectory is present in the simulation...### Summary
Reported by @ext-piot_p on the OPAL mailing list. With the option `SAMPLE, KEEP` non-empty, non-wanted files should be deleted, but an error message is shown instead when a non-empty subdirectory is present in the simulation directory.
### Steps to reproduce
An OPAL Sampler simulation whereby also files in the auxiliary directory (`/data`) are produced together with the `KEEP` option of the `SAMPLER`.
### What is the current *bug* behavior?
An error message is shown and the unwanted files are not (all) deleted.
### What is the expected *correct* behavior?
No error message and all but the files with extensions in the `KEEP` option should be deleted.
### Relevant logs and/or screenshots
```
OPAL{0}> opal varyTDCphase.in --inputfile=varyTDCphase.tmpl --outfile=phiscan --outdir=phiscan --num-masters=1 --num-coworkers=2 --restartstep=-2147483648 --jsonDumpFreq=1 --nsamples=11 --simtmpdir=/home/piot/OpalSandBox/scan_example/phiscan --templates=/home/piot/OpalSandBox/scan_example/template
7 (PID: 214335) ▶ Worker
1 (PID: 214329) ▶ Sampler
2 (PID: 214330) ▶ Worker
3 (PID: 214331) ▶ Worker
4 (PID: 214332) ▶ Worker
5 (PID: 214333) ▶ Worker
6 (PID: 214334) ▶ Worker
✔ 40 objectives
✔ 1 dvars
0 (PID: 214328) ▶ Pilot
Warning: argument "one-pilot-converge" not found! Using default value (0).
Warning: argument "restartfile" not found! Using default value ().
Warning: argument "restartfile" not found! Using default value ().
Ippl> CommMPI: Started job 1 on host `localhost.localdomain'.
Ippl> CommMPI: Parent process waiting for children ...
Ippl> CommMPI: Child 1 ready.
Ippl> CommMPI: Initialization complete.
Warning: argument "restartfile" not found! Using default value ().
Warning: argument "restartfile" not found! Using default value ().
Ippl> CommMPI: Started job 1 on host `localhost.localdomain'.
Ippl> CommMPI: Parent process waiting for children ...
Ippl> CommMPI: Child 1 ready.
Ippl> CommMPI: Initialization complete.
Warning: argument "restartfile" not found! Using default value ().
Warning: argument "restartfile" not found! Using default value ().
Ippl> CommMPI: Started job 1 on host `localhost.localdomain'.
Ippl> CommMPI: Parent process waiting for children ...
Ippl> CommMPI: Child 1 ready.
Ippl> CommMPI: Initialization complete.
Can't remove file in directory '/home/piot/OpalSandBox/scan_example/phiscan/1', (boost::filesystem::remove: Directory not empty: "/home/piot/OpalSandBox/scan_example/phiscan/1/data")
```
### Possible fixes
The function [`OpalSimulation::Cleanup(const std::vector<std::string>& keep)`](https://gitlab.psi.ch/OPAL/src/-/blob/f3d1060a910e9f46f67bfb6807db1ea9e992c57a/src/Optimize/OpalSimulation.cpp#L632)
does not take into account the possibility that a subdirectory might be present. It could check if the path is a directory and if so use `boost::filesystem::remove_all` (or [the c++17 equivalent](https://en.cppreference.com/w/cpp/filesystem/remove)) instead of `boost::filesystem::remove` to remove these non-empty subdirectories. Possibly even better would be if the subdirectories are traversed and files with proper extensions are kept.OPAL 2021.1krauskraushttps://gitlab.psi.ch/OPAL/src/-/issues/614Make sampler more robust2020-10-09T10:33:26+02:00bellotti_rMake sampler more robust**Current state**<br />
If one machine configuration crashes (e. g. segfault), the entire random sample dies.
**Why is it bad**<br />
I'm referring to a situation where the sampler reads from a TSV file instead of generating configurati...**Current state**<br />
If one machine configuration crashes (e. g. segfault), the entire random sample dies.
**Why is it bad**<br />
I'm referring to a situation where the sampler reads from a TSV file instead of generating configurations.<br />
- The more machines are run in parallel, the more configurations have to be restarted. This is a cumbersome manual process, and therefore error prone.
- All progress of simulations in progress is lost. (waste of resources)
- The unfinished machine settings have to be identified, copied to a new TSV file, and another RS has to be started, which is again a manual process.
Long story short, this bug imposes unnecessary work and introduces unnecessary error sources for OPAL users.https://gitlab.psi.ch/OPAL/src/-/issues/557Optimizer gets stuck2021-06-10T19:16:54+02:00snuverink_jjochem.snuverink@psi.chOptimizer gets stuckAs reported by Finn O'Shea: the optimizer get stuck from time to time with the following example:
[fdopt.in](/uploads/e8ef7cb47cc60dcfbb8daab723b461df/fdopt.in)
[fdopt.data](/uploads/2f965a6b27a71ba36ab64519a5d8c74d/fdopt.data)
[fdopt.t...As reported by Finn O'Shea: the optimizer get stuck from time to time with the following example:
[fdopt.in](/uploads/e8ef7cb47cc60dcfbb8daab723b461df/fdopt.in)
[fdopt.data](/uploads/2f965a6b27a71ba36ab64519a5d8c74d/fdopt.data)
[fdopt.tmpl](/uploads/6e7dc37c7fe54832e01c0d7a41755e2f/fdopt.tmpl)snuverink_jjochem.snuverink@psi.chsnuverink_jjochem.snuverink@psi.chhttps://gitlab.psi.ch/OPAL/src/-/issues/498File header for Sampler classes2020-04-07T16:40:00+02:00frey_mFile header for Sampler classesThis issue adds the file header according to #482 to the sampler classes.This issue adds the file header according to #482 to the sampler classes.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/src/-/issues/465Segmentation fault in FromFile due to RASTER=TRUE2020-02-22T22:21:22+01:00frey_mSegmentation fault in FromFile due to RASTER=TRUE### Summary
Bug report through OPAL mailing list:
```bash
Dear all,
I’m trying to use the FROMFILE option for scanning parameters in the sampler.
The files work fine for uniform distributions, but when I introduce FROMFILE,
I get the...### Summary
Bug report through OPAL mailing list:
```bash
Dear all,
I’m trying to use the FROMFILE option for scanning parameters in the sampler.
The files work fine for uniform distributions, but when I introduce FROMFILE,
I get the message, that
Error{1}> *** User error detected by function "FromFile()"
Error{1}> Couldn't find the dvar 'VQ1B' in the file './samplefile_corV3.dat'
Error{1}> Couldn't find the dvar 'VQ1B' in the file './samplefile_corV3.dat'
That happens, when I write ‘VQ1B’ – which is not in the manual.
Using the manual and writing VQ1B I get a segmentation fault.
[dinux9:59198] *** Process received signal ***
[dinux9:59198] Signal: Segmentation fault (11)
[dinux9:59198] Signal code: Address not mapped (1)
[dinux9:59198] Failing at address: (nil)
[dinux9:59198] [ 0] /lib64/libpthread.so.0(+0x10c10)[0x7fe6e78c4c10]
[dinux9:59198] [ 1] opal(_ZN8FromFile8allocateERKN5boost10shared_ptrI12CmdArgumentsEERKN4Comm8Bundle_tE+0x1190)[0x898ab0]
[dinux9:59198] [ 2] opal(_ZN9SampleCmd7executeEv+0x2d41)[0x8a16d1]
[dinux9:59198] [ 3] opal(_ZNK10OpalParser7executeEP6ObjectRKSs+0x35)[0x7b0e25]
[dinux9:59198] [ 4] opal(_ZNK10OpalParser11parseActionER9Statement+0x120)[0x7b4bc0]
[dinux9:59198] [ 5] opal(_ZNK10OpalParser5parseER9Statement+0x1cf)[0x7b541f]
[dinux9:59198] [ 6] opal(_ZNK10OpalParser3runEv+0x2c)[0x7b0aec]
[dinux9:59198] [ 7] opal(_ZNK10OpalParser3runEP11TokenStream+0x70)[0x7b5d40]
[dinux9:59198] [ 8] opal(main+0xf75)[0x6ff155]
[dinux9:59198] [ 9] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fe6e5c2c725]
[dinux9:59198] [10] opal(_start+0x29)[0x70b039]
[dinux9:59198] *** End of error message ***
```
[2019_diag_EMF_corV3_sampler.in](/uploads/4c52ba86ecd3afbcca5ebefa8c52e095/2019_diag_EMF_corV3_sampler.in)
[samplefile_corV3.dat](/uploads/1d030ccb2d1c172171793f325963d805/samplefile_corV3.dat)
### What is the current *bug* behavior?
Segmentation fault.
### What is the expected *correct* behavior?
No segmentation fault.
### Possible fixes
The reason is the `RASTER=TRUE` option which increases the number of required samples in the `FROMFILE` file.
See in [line 335](https://gitlab.psi.ch/OPAL/src/blob/master/src/Sample/SampleCmd.cpp#L335) in `SampleCmd.cpp`. Hence `nSamples > globalSize_m` (`globalSize_m` is the number of lines in the provided FROMFILE file).
It can be fixed by
```C++
nSamples = std::min(nSamples, globalSize_m);
```frey_mfrey_mhttps://gitlab.psi.ch/OPAL/src/-/issues/337Sampler fails due to static variable in MueLuBottomSolver2020-07-14T10:29:45+02:00frey_mSampler fails due to static variable in MueLuBottomSolver### Summary
As in #336 the sampler crashes when dispatching new individuals after some finished.
### Steps to reproduce
Run the sampler with AMR Trilinos and the smoothed aggregation algebraic multigrid solver for the linear system o...### Summary
As in #336 the sampler crashes when dispatching new individuals after some finished.
### Steps to reproduce
Run the sampler with AMR Trilinos and the smoothed aggregation algebraic multigrid solver for the linear system of
equations.
### What is the current *bug* behavior?
The sampler crashes.
### What is the expected *correct* behavior?
The sampler runs smoothly.
### Possible fixes
Remove the static variable.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/src/-/issues/336Sampler fails due to static variable in SDDSWriters2020-07-14T10:41:25+02:00frey_mSampler fails due to static variable in SDDSWriters### Summary
Sampler crashed due to SDDSWriters.
### Steps to reproduce
Run the sampler.
### What is the current *bug* behavior?
The static variable `isFirst` of the new SDDSWriter interface (e.g.
[GridLBalWriter](https://gitlab.ps...### Summary
Sampler crashed due to SDDSWriters.
### Steps to reproduce
Run the sampler.
### What is the current *bug* behavior?
The static variable `isFirst` of the new SDDSWriter interface (e.g.
[GridLBalWriter](https://gitlab.psi.ch/OPAL/src/blob/master/src/Structure/GridLBalWriter.cpp#L16)) causes the sampler to crash with
```
Error{0}>
Error{0}> *** User error detected by function "SDDSColumnSet::addColumnValue"
Error{0}> column name 't' doesn't exists
Error{0}> column name 't' doesn't exists
```
### What is the expected *correct* behavior?
The sampler runs smoothly.
### Possible fixes
Making the variable a non-static `isFirst` a member of `SDDSWriter.h` fixes the problem.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/src/-/issues/269Halo evaluation as objective2018-12-11T09:14:40+01:00frey_mHalo evaluation as objectiveIt would be nice having a halo evaluation as an objective for the sampler and optimizer. I also think about multi-bunch simulations. We need either the halo written to the stat-file or evaluate it using the H5 file. In case of multi-bunc...It would be nice having a halo evaluation as an objective for the sampler and optimizer. I also think about multi-bunch simulations. We need either the halo written to the stat-file or evaluate it using the H5 file. In case of multi-bunch simulations where particles are assigned to energy bins, we'd need a stat file for every energy bin. Therefore, I think, it's easier to add new objective funtion.
I suggest something like this
```c++
halo(dir, step, bin)
```
where ```dir=x,y,z``` and ```bin``` the energy bin (default: 0).frey_mfrey_mhttps://gitlab.psi.ch/OPAL/src/-/issues/266Sampler: Allow multiple masters2018-12-06T10:01:43+01:00frey_mSampler: Allow multiple mastersFor small simulations (e.g. a few seconds), we cannot use the full set of cores efficiently since one master is not able to keep up.For small simulations (e.g. a few seconds), we cannot use the full set of cores efficiently since one master is not able to keep up.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/src/-/issues/265Sampler: Delete files that are not used2018-12-02T17:06:44+01:00frey_mSampler: Delete files that are not usedWhen running the SAMPLER, it might run thounds of OPAL simulations. In each directory we will then have files like *.lbal, *.out, etc., that might not be needed. Since on clusters you sometimes / normally have quotas on number of files, ...When running the SAMPLER, it might run thounds of OPAL simulations. In each directory we will then have files like *.lbal, *.out, etc., that might not be needed. Since on clusters you sometimes / normally have quotas on number of files, I think it would make sense that we add an option to the SAMPLER command where we can specify which files to delete directly after one simulation.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/src/-/issues/264Sampler: Memory reduction (especially for LatinHyperCube)2018-11-30T15:20:36+01:00frey_mSampler: Memory reduction (especially for LatinHyperCube)Every process allocates memory for the sampling methods which is actually not needed. In case of Latin Hypercube every process has then a container with all indices. This may cause computing nodes being out of memory.Every process allocates memory for the sampling methods which is actually not needed. In case of Latin Hypercube every process has then a container with all indices. This may cause computing nodes being out of memory.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/src/-/issues/259Structure of output JSON file2018-12-02T20:27:00+01:00huber_yStructure of output JSON fileRight now the output JSON file is structured in a way such that when read by python using the json module it is converted to an array of dictionaries where each directionary contains a key "ID" and the corresponding ID is the value, plus...Right now the output JSON file is structured in a way such that when read by python using the json module it is converted to an array of dictionaries where each directionary contains a key "ID" and the corresponding ID is the value, plus additonal key such as dvars containing the design variables.
For finding an ID it would in my eyes be easier if the file was structured such that the resulting array contains only one dictionary, where the keys are something like ID+"dvars" where ID is the actual ID, since this allows to easily access elements.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/src/-/issues/251Allow optimizer and sampler runs to restart from given h5 file2019-02-25T08:43:02+01:00krausAllow optimizer and sampler runs to restart from given h5 fileRestart currently doesn't work for runs when launched from the optimizer / sampler. This feature should be relatively easy to implement. Reported by M. Abo-Bakr.Restart currently doesn't work for runs when launched from the optimizer / sampler. This feature should be relatively easy to implement. Reported by M. Abo-Bakr.krausfrey_mkraushttps://gitlab.psi.ch/OPAL/src/-/issues/250Store stat files of Sampler (and Optimizer?) in a single file2021-06-10T17:58:47+02:00krausStore stat files of Sampler (and Optimizer?) in a single fileThe stat files should be stored into a single file such that all data from all runs can be evaluated in post processing. The data should be supplemented with the values of the design variables. File formats that come into considerations ...The stat files should be stored into a single file such that all data from all runs can be evaluated in post processing. The data should be supplemented with the values of the design variables. File formats that come into considerations are HDF5, SDDS, a NoSQL database, zip/tar gunzip etc.krausadelmanngsellkraushttps://gitlab.psi.ch/OPAL/src/-/issues/239Remove requirement of data file for SAMPLER and OPTIMIZER2020-06-22T09:42:35+02:00adelmannRemove requirement of data file for SAMPLER and OPTIMIZERWe still use the runOPAL.py scheme with template / data file. However the design variables could be easily looked up in the dictionary or we invent new types: DvarReal etc.
[Edit: this has been moved to a separate issue, see #249! If we...We still use the runOPAL.py scheme with template / data file. However the design variables could be easily looked up in the dictionary or we invent new types: DvarReal etc.
[Edit: this has been moved to a separate issue, see #249! If we could define quantities of interest (a la optimizer) and write them to the json file lengthly post processing would be avoided.]
Edit:
- [x] remove the need for a data file (!288)
- [x] write a json file containing the values of the DVARs and objectives (#249)OPAL 2.4.0frey_mfrey_m