pyOPALTools issueshttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues2020-02-19T09:09:56+01:00https://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/73SDDSParser has difficulties with OPAL 1.6 stat files2020-02-19T09:09:56+01:00snuverink_jjochem.snuverink@psi.chSDDSParser has difficulties with OPAL 1.6 stat filesThe SDDSParser seems to have problems reading version 1.6 stat files, like e.g. https://gitlab.psi.ch/OPAL/pyOPALTools/blob/da35eef62da7f618b6a51194a7710c8dc9c3a823/opal/test/Gantry2/70MeV_Gantry2_track16.stat
Minimal example (with pand...The SDDSParser seems to have problems reading version 1.6 stat files, like e.g. https://gitlab.psi.ch/OPAL/pyOPALTools/blob/da35eef62da7f618b6a51194a7710c8dc9c3a823/opal/test/Gantry2/70MeV_Gantry2_track16.stat
Minimal example (with pandas 0.25.1):
```python
import pandas as pd
keys = ['t', 's', 'numParticles', 'charge', 'energy', 'rms_x', 'rms_y', 'rms_s', 'rms_px', 'rms_py', 'rms_ps', 'emit_x', 'emit_y', 'emit_s', 'mean_x', 'mean_y', 'mean_s', 'max_x', 'max_y', 'max_s', 'xpx', 'ypy', 'zpz', 'notused1', 'notused2', 'Dx', 'DDx', 'Dy', 'DDy', 'Bx_head', 'By_head', 'Bz_head', 'Ex_head', 'Ey_head', 'Ez_head', 'Bx_ref', 'By_ref', 'Bz_ref', 'Ex_ref', 'Ey_ref', 'Ez_ref', 'Bx_tail', 'By_tail', 'Bz_tail', 'Ex_tail', 'Ey_tail', 'Ez_tail', 'dE', 'partsOutside']
pd.read_csv('Gantry2/70MeV_Gantry2_track16.stat', skiprows=56,
sep='\s+', names=keys, index_col=False)
```
```
t s numParticles charge energy rms_x rms_y rms_s rms_px rms_py ... Ey_ref Ez_ref Bx_tail By_tail Bz_tail Ex_tail Ey_tail Ez_tail dE partsOutside
0 rows × 49 columns
```
I think this really should have worked, but it seems the trailing space in the header string causes problems (even though skiprows specifies these should be skipped!).
Minimal file that has the same issue:
```ascii
description="26 Dispersion in x "
1.000000000000000e-12 3.159271238650963e+01 1000
```
```python
df = pd.read_csv("Gantry2/example.txt", skiprows=1, sep='\s+', names=['t','s','numParticles'])
df.size
0
```
It can be fixed by forcing to use the "python" engine:
```python
pd.read_csv("Gantry2/example.txt", skiprows=1, sep='\s+\t', names=['t','s','numParticles'])
df.size
3
```snuverink_jjochem.snuverink@psi.chsnuverink_jjochem.snuverink@psi.chhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/75H5Parser method to close files2020-03-26T17:43:21+01:00snuverink_jjochem.snuverink@psi.chH5Parser method to close filesFrom Sebastian Thomas:
> I have a question concerning the H5Parser https://gitlab.psi.ch/OPAL/pyOPALTools/blob/master/opal/parser/H5Parser.py
>
> I'm not an extremely experienced coder, but shouldn't the file opened in H5Parser.parse("f...From Sebastian Thomas:
> I have a question concerning the H5Parser https://gitlab.psi.ch/OPAL/pyOPALTools/blob/master/opal/parser/H5Parser.py
>
> I'm not an extremely experienced coder, but shouldn't the file opened in H5Parser.parse("file") via
>
> `self.__h5f = h5py.File(fname, 'r')`
>
> be closed again? Wenn I use the parser on an H5 file in a juyter notebook, I run into some problems because for example OPAL tries to delete the monitor output, but is not allowed to, due to the file "beeing in use". I ask mainly out of curiosity.
Since h5py 2.4 opened file will automatically close:
http://docs.h5py.org/en/stable/whatsnew/2.4.html#files-will-now-auto-close
However, if the H5Parser still exists, this will not work, and it would be nice to have a manual option to close the file.
Proposal is to add a specific close method to H5Parser, as discussed with @frey\_m.snuverink_jjochem.snuverink@psi.chsnuverink_jjochem.snuverink@psi.chhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/70Follow-up from "Resolve "Multiple steps in probe""2020-03-27T07:44:00+01:00frey_mFollow-up from "Resolve "Multiple steps in probe""The following discussions from !45 should be addressed:
- [x] @snuverink_j started a [discussion](https://gitlab.psi.ch/OPAL/pyOPALTools/merge_requests/45#note_14524): (+1 comment)
> sorry for being late with commenting. You could...The following discussions from !45 should be addressed:
- [x] @snuverink_j started a [discussion](https://gitlab.psi.ch/OPAL/pyOPALTools/merge_requests/45#note_14524): (+1 comment)
> sorry for being late with commenting. You could use the `hypot` function as well: https://docs.scipy.org/doc/numpy/reference/generated/numpy.hypot.html
- [x] @snuverink_j started a [discussion](https://gitlab.psi.ch/OPAL/pyOPALTools/merge_requests/45#note_14527): (+2 comments)
> Can the range become very large? In that case append might not be cpu efficient, and it is better to preallocate the array (e.g. https://stackoverflow.com/questions/2641691/building-up-an-array-in-numpy-scipy-by-iteration-in-python):
>
> ```python
> x = np.zeros(1,self.ds.size)
> for s in range(self.ds.size):
> x[i] = self.ds.getData('x', step=s)
> ```frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/74Selection of bunch in plot_probe_histogram2020-03-27T07:44:00+01:00frey_mSelection of bunch in plot_probe_histogramWe need to be able to select a single bunch in case of multi-bunch simulations.We need to be able to select a single bunch in case of multi-bunch simulations.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/72README: point to wiki2020-03-27T07:44:00+01:00frey_mREADME: point to wikifrey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/71Use extend instead of append2020-03-27T07:44:00+01:00frey_mUse extend instead of appendThe list append of https://gitlab.psi.ch/OPAL/pyOPALTools/merge_requests/46 causes wrong behaviour. We need to use list extend instead.The list append of https://gitlab.psi.ch/OPAL/pyOPALTools/merge_requests/46 causes wrong behaviour. We need to use list extend instead.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/68H5 analysis: select last step if negative step number2020-03-27T07:44:00+01:00frey_mH5 analysis: select last step if negative step numberfrey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/69Multiple steps in probe2020-04-07T10:41:36+02:00frey_mMultiple steps in probeWe need to update the `ProbePlotter` class since probe H5 files can now store multiple steps.We need to update the `ProbePlotter` class since probe H5 files can now store multiple steps.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/58update of probe plotting utilities2020-04-07T10:46:34+02:00frey_mupdate of probe plotting utilitiesDue to the changes made in https://gitlab.psi.ch/OPAL/src/merge_requests/161 the plotting functions of the probe dataset (H5) are not valid anymore if the probe is saved with `OPTION,PSDUMPEACHTURN=TRUE`. However, we need to guarantee bo...Due to the changes made in https://gitlab.psi.ch/OPAL/src/merge_requests/161 the plotting functions of the probe dataset (H5) are not valid anymore if the probe is saved with `OPTION,PSDUMPEACHTURN=TRUE`. However, we need to guarantee both versions.https://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/77Add licenses2020-04-08T17:05:46+02:00frey_mAdd licensesWe need to add the licenses of the different distributions to pyOPALTools.We need to add the licenses of the different distributions to pyOPALTools.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/78File header2020-04-09T08:32:13+02:00frey_mFile headerWe should add file headers like in OPAL, i.e., OPAL/src#482.We should add file headers like in OPAL, i.e., OPAL/src#482.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/80module uq : chaospy (3.2.9) chaospy.dot does not exist2020-04-14T10:44:11+02:00snuverink_jjochem.snuverink@psi.chmodule uq : chaospy (3.2.9) chaospy.dot does not existWhen I run the confidence_interval notebook I get the following error:
```
AttributeError Traceback (most recent call last)
<ipython-input-2-637cb2ef6afe> in <module>
16
17 y_train, y_pred, y_lo, y_...When I run the confidence_interval notebook I get the following error:
```
AttributeError Traceback (most recent call last)
<ipython-input-2-637cb2ef6afe> in <module>
16
17 y_train, y_pred, y_lo, y_up = uq.confidence_interval(x_train, y_train, alpha=alpha,
---> 18 n_boot=n_boot, sample=sample)
19
20 plt.figure(figsize=(16, 9))
~/Documents/OPAL/pyOPALTools/surrogate/uq.py in confidence_interval(self, x, y, alpha, **kwargs)
129
130 lo, up = bs.confidence_interval(alpha=alpha)
--> 131 lo_pce = cp.dot(poly, np.asarray(lo))
132 up_pce = cp.dot(poly, np.asarray(up))
133
AttributeError: module 'chaospy' has no attribute 'dot'
```
For this I had to add the following fix to bootstrap.py:
```diff
- return 2.0 * fitted - qlo, 2.0 * fitted - qhi
+ return 2.0 * np.asarray(fitted) - qlo, 2.0 * np.asarray(fitted) - qhi
```frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/76Additional arguments for the H5 version of plot_probe_histogram2020-04-20T08:09:39+02:00frey_mAdditional arguments for the H5 version of plot_probe_histogramBegin and end of steps to read by
```Python
begin = kwargs.pop('begin', 0)
end = kwargs.pop('end', self.ds.size)
```Begin and end of steps to read by
```Python
begin = kwargs.pop('begin', 0)
end = kwargs.pop('end', self.ds.size)
```frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/32Documentation style Sphinx2020-04-20T11:08:56+02:00frey_mDocumentation style Sphinx@bellotti\_r changed the documentation in some classes using [Sphinx](http://www.sphinx-doc.org/en/master/). It's similar to Doxygen. We should use a consistent format. I'm fine with the new proposed documentation.@bellotti\_r changed the documentation in some classes using [Sphinx](http://www.sphinx-doc.org/en/master/). It's similar to Doxygen. We should use a consistent format. I'm fine with the new proposed documentation.snuverink_jjochem.snuverink@psi.chsnuverink_jjochem.snuverink@psi.chhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/79Add Jupyter Notebooks to Sphinx documentation2020-04-20T11:08:56+02:00snuverink_jjochem.snuverink@psi.chAdd Jupyter Notebooks to Sphinx documentationAs discussed in https://gitlab.psi.ch/OPAL/pyOPALTools/issues/32#note_19574, it would be good to include the jupyter notebooks in the sphinx documentation.As discussed in https://gitlab.psi.ch/OPAL/pyOPALTools/issues/32#note_19574, it would be good to include the jupyter notebooks in the sphinx documentation.snuverink_jjochem.snuverink@psi.chsnuverink_jjochem.snuverink@psi.chhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/82Sampler and Optimizer files of OPAL-2.2.0 cannot be loaded2020-04-22T16:05:30+02:00frey_mSampler and Optimizer files of OPAL-2.2.0 cannot be loadedSampler and optimizer files that are generated with OPAL-2.2.0 cannot be loaded due to the version number. Adding the line
```Python
self.__version_support = {
'2.1.0': self.__parse_version_2_1_0,
'2.2.0': self.__parse_version_2_...Sampler and optimizer files that are generated with OPAL-2.2.0 cannot be loaded due to the version number. Adding the line
```Python
self.__version_support = {
'2.1.0': self.__parse_version_2_1_0,
'2.2.0': self.__parse_version_2_1_0 # OPAL-2.2.0 support
}
```
would fix the problem. However, we should do a more general approach in order to avoid this error in future.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/81Auto deploy sphinx documentation2020-05-28T09:45:23+02:00snuverink_jjochem.snuverink@psi.chAuto deploy sphinx documentationSimilar to the OPAL manual, the sphinx documentation (#32) for pyOPALTools should be generated automatically on http://amas.web.psi.ch/docs/pyOPALTools/index.html
Note that the jupyter notebooks are now included (#79 !57) in the documen...Similar to the OPAL manual, the sphinx documentation (#32) for pyOPALTools should be generated automatically on http://amas.web.psi.ch/docs/pyOPALTools/index.html
Note that the jupyter notebooks are now included (#79 !57) in the documentation. The python package `nbsphinx` is needed, as well as `pandoc` and `latex`. There are some issues with the latex labels with matplotlib, therefore not all notebooks might build without problems, but this will then be fixed later.gsellgsellhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/86Add new dataset for field data on grid2020-06-08T16:50:33+02:00frey_mAdd new dataset for field data on gridOPAL writes the field data on the grid to file if `DBG_SCALARFIELD` is enabled. We should provide a parser, dataset and plotting functions.OPAL writes the field data on the grid to file if `DBG_SCALARFIELD` is enabled. We should provide a parser, dataset and plotting functions.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/90Follow-up from "Resolve "H5Parser method to close files""2020-06-09T15:31:26+02:00frey_mFollow-up from "Resolve "H5Parser method to close files""The following discussion from !51 should be addressed:
- [ ] @frey_m started a [discussion](https://gitlab.psi.ch/OPAL/pyOPALTools/merge_requests/51#note_22727): (+12 comments)
> @snuverink\_j I get an error when `H5Parser.close` ...The following discussion from !51 should be addressed:
- [ ] @frey_m started a [discussion](https://gitlab.psi.ch/OPAL/pyOPALTools/merge_requests/51#note_22727): (+12 comments)
> @snuverink\_j I get an error when `H5Parser.close` is called. My script only loads a H5 file and calls the print function. I use the h5py version 2.10.0. Have you experienced any error like this?
>
> ```bash
> Exception ignored in: <bound method H5Dataset.__del__ of <opal.datasets.H5Dataset.H5Dataset object at 0x7f08da008f98>>
> Traceback (most recent call last):
> File "/home/matthias/Documents/projects/pyOPALTools/opal/datasets/H5Dataset.py", line 98, in __del__
> File "/home/matthias/Documents/projects/pyOPALTools/opal/parser/H5Parser.py", line 59, in close
> File "/home/matthias/Documents/projects/pyOPALTools/pyOPALTools.venv/lib/python3.5/site-packages/h5py/_hl/files.py", line 432, in close
> File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
> File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
> File "h5py/h5f.pyx", line 267, in h5py.h5f.get_obj_ids
> File "h5py/h5i.pyx", line 37, in h5py.h5i.wrap_identifier
> SystemError: Parent module 'h5py' not loaded, cannot perform relative import
> ```
**Possible fix:**
Replacing the line `self.__h5f.close()` in `H5Parser.close` by `del self.__h5f`.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/93Update on FieldDataset2020-06-18T13:29:01+02:00frey_mUpdate on FieldDatasetThis issue should
- add the file FieldAnalysis.py
- provide the line plot functionality to FieldPlotter
- do some changes on FieldParser and FieldDatasetThis issue should
- add the file FieldAnalysis.py
- provide the line plot functionality to FieldPlotter
- do some changes on FieldParser and FieldDatasetfrey_mfrey_m