pyOPALTools issueshttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues2020-04-09T08:32:13+02:00https://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/78File header2020-04-09T08:32:13+02:00frey_mFile headerWe should add file headers like in OPAL, i.e., OPAL/src#482.We should add file headers like in OPAL, i.e., OPAL/src#482.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/77Add licenses2020-04-08T17:05:46+02:00frey_mAdd licensesWe need to add the licenses of the different distributions to pyOPALTools.We need to add the licenses of the different distributions to pyOPALTools.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/76Additional arguments for the H5 version of plot_probe_histogram2020-04-20T08:09:39+02:00frey_mAdditional arguments for the H5 version of plot_probe_histogramBegin and end of steps to read by
```Python
begin = kwargs.pop('begin', 0)
end = kwargs.pop('end', self.ds.size)
```Begin and end of steps to read by
```Python
begin = kwargs.pop('begin', 0)
end = kwargs.pop('end', self.ds.size)
```frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/75H5Parser method to close files2020-03-26T17:43:21+01:00snuverink_jjochem.snuverink@psi.chH5Parser method to close filesFrom Sebastian Thomas:
> I have a question concerning the H5Parser https://gitlab.psi.ch/OPAL/pyOPALTools/blob/master/opal/parser/H5Parser.py
>
> I'm not an extremely experienced coder, but shouldn't the file opened in H5Parser.parse("f...From Sebastian Thomas:
> I have a question concerning the H5Parser https://gitlab.psi.ch/OPAL/pyOPALTools/blob/master/opal/parser/H5Parser.py
>
> I'm not an extremely experienced coder, but shouldn't the file opened in H5Parser.parse("file") via
>
> `self.__h5f = h5py.File(fname, 'r')`
>
> be closed again? Wenn I use the parser on an H5 file in a juyter notebook, I run into some problems because for example OPAL tries to delete the monitor output, but is not allowed to, due to the file "beeing in use". I ask mainly out of curiosity.
Since h5py 2.4 opened file will automatically close:
http://docs.h5py.org/en/stable/whatsnew/2.4.html#files-will-now-auto-close
However, if the H5Parser still exists, this will not work, and it would be nice to have a manual option to close the file.
Proposal is to add a specific close method to H5Parser, as discussed with @frey\_m.snuverink_jjochem.snuverink@psi.chsnuverink_jjochem.snuverink@psi.chhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/74Selection of bunch in plot_probe_histogram2020-03-27T07:44:00+01:00frey_mSelection of bunch in plot_probe_histogramWe need to be able to select a single bunch in case of multi-bunch simulations.We need to be able to select a single bunch in case of multi-bunch simulations.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/73SDDSParser has difficulties with OPAL 1.6 stat files2020-02-19T09:09:56+01:00snuverink_jjochem.snuverink@psi.chSDDSParser has difficulties with OPAL 1.6 stat filesThe SDDSParser seems to have problems reading version 1.6 stat files, like e.g. https://gitlab.psi.ch/OPAL/pyOPALTools/blob/da35eef62da7f618b6a51194a7710c8dc9c3a823/opal/test/Gantry2/70MeV_Gantry2_track16.stat
Minimal example (with pand...The SDDSParser seems to have problems reading version 1.6 stat files, like e.g. https://gitlab.psi.ch/OPAL/pyOPALTools/blob/da35eef62da7f618b6a51194a7710c8dc9c3a823/opal/test/Gantry2/70MeV_Gantry2_track16.stat
Minimal example (with pandas 0.25.1):
```python
import pandas as pd
keys = ['t', 's', 'numParticles', 'charge', 'energy', 'rms_x', 'rms_y', 'rms_s', 'rms_px', 'rms_py', 'rms_ps', 'emit_x', 'emit_y', 'emit_s', 'mean_x', 'mean_y', 'mean_s', 'max_x', 'max_y', 'max_s', 'xpx', 'ypy', 'zpz', 'notused1', 'notused2', 'Dx', 'DDx', 'Dy', 'DDy', 'Bx_head', 'By_head', 'Bz_head', 'Ex_head', 'Ey_head', 'Ez_head', 'Bx_ref', 'By_ref', 'Bz_ref', 'Ex_ref', 'Ey_ref', 'Ez_ref', 'Bx_tail', 'By_tail', 'Bz_tail', 'Ex_tail', 'Ey_tail', 'Ez_tail', 'dE', 'partsOutside']
pd.read_csv('Gantry2/70MeV_Gantry2_track16.stat', skiprows=56,
sep='\s+', names=keys, index_col=False)
```
```
t s numParticles charge energy rms_x rms_y rms_s rms_px rms_py ... Ey_ref Ez_ref Bx_tail By_tail Bz_tail Ex_tail Ey_tail Ez_tail dE partsOutside
0 rows × 49 columns
```
I think this really should have worked, but it seems the trailing space in the header string causes problems (even though skiprows specifies these should be skipped!).
Minimal file that has the same issue:
```ascii
description="26 Dispersion in x "
1.000000000000000e-12 3.159271238650963e+01 1000
```
```python
df = pd.read_csv("Gantry2/example.txt", skiprows=1, sep='\s+', names=['t','s','numParticles'])
df.size
0
```
It can be fixed by forcing to use the "python" engine:
```python
pd.read_csv("Gantry2/example.txt", skiprows=1, sep='\s+\t', names=['t','s','numParticles'])
df.size
3
```snuverink_jjochem.snuverink@psi.chsnuverink_jjochem.snuverink@psi.chhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/72README: point to wiki2020-03-27T07:44:00+01:00frey_mREADME: point to wikifrey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/71Use extend instead of append2020-03-27T07:44:00+01:00frey_mUse extend instead of appendThe list append of https://gitlab.psi.ch/OPAL/pyOPALTools/merge_requests/46 causes wrong behaviour. We need to use list extend instead.The list append of https://gitlab.psi.ch/OPAL/pyOPALTools/merge_requests/46 causes wrong behaviour. We need to use list extend instead.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/70Follow-up from "Resolve "Multiple steps in probe""2020-03-27T07:44:00+01:00frey_mFollow-up from "Resolve "Multiple steps in probe""The following discussions from !45 should be addressed:
- [x] @snuverink_j started a [discussion](https://gitlab.psi.ch/OPAL/pyOPALTools/merge_requests/45#note_14524): (+1 comment)
> sorry for being late with commenting. You could...The following discussions from !45 should be addressed:
- [x] @snuverink_j started a [discussion](https://gitlab.psi.ch/OPAL/pyOPALTools/merge_requests/45#note_14524): (+1 comment)
> sorry for being late with commenting. You could use the `hypot` function as well: https://docs.scipy.org/doc/numpy/reference/generated/numpy.hypot.html
- [x] @snuverink_j started a [discussion](https://gitlab.psi.ch/OPAL/pyOPALTools/merge_requests/45#note_14527): (+2 comments)
> Can the range become very large? In that case append might not be cpu efficient, and it is better to preallocate the array (e.g. https://stackoverflow.com/questions/2641691/building-up-an-array-in-numpy-scipy-by-iteration-in-python):
>
> ```python
> x = np.zeros(1,self.ds.size)
> for s in range(self.ds.size):
> x[i] = self.ds.getData('x', step=s)
> ```frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/69Multiple steps in probe2020-04-07T10:41:36+02:00frey_mMultiple steps in probeWe need to update the `ProbePlotter` class since probe H5 files can now store multiple steps.We need to update the `ProbePlotter` class since probe H5 files can now store multiple steps.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/68H5 analysis: select last step if negative step number2020-03-27T07:44:00+01:00frey_mH5 analysis: select last step if negative step numberfrey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/67make YAML optional2019-10-14T15:44:24+02:00frey_mmake YAML optionalfrey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/66setup.py update?2019-10-14T15:48:36+02:00ext-neveu_nsetup.py update?While using pyOPALTools with a Steve Hudson at ANL, he got an import/path error related to yaml:
```
File "deap_libe_lcls_gun.py", line 37, in <module>
from fitness import fitness_opal
File "/lcrc/project/MCS-SLAC/shudson/emitta...While using pyOPALTools with a Steve Hudson at ANL, he got an import/path error related to yaml:
```
File "deap_libe_lcls_gun.py", line 37, in <module>
from fitness import fitness_opal
File "/lcrc/project/MCS-SLAC/shudson/emittance_minimization/code/fitness.py", line 17, in <module>
from libEopal import run_opal_nowatch
File "/lcrc/project/MCS-SLAC/shudson/emittance_minimization/code/libEopal.py", line 9, in <module>
from opal.datasets.StatDataset import *
File "/lcrc/project/MCS-SLAC/software/pyOPALTools/opal/__init__.py", line 1, in <module>
from .opal import load_dataset
File "/lcrc/project/MCS-SLAC/software/pyOPALTools/opal/opal.py", line 6, in <module>
from .datasets.filetype import FileType
File "/lcrc/project/MCS-SLAC/software/pyOPALTools/opal/datasets/__init__.py", line 1, in <module>
import opal.datasets.filetype
File "/lcrc/project/MCS-SLAC/software/pyOPALTools/opal/datasets/filetype.py", line 9, in <module>
from opal.utilities.logger import opal_logger
File "/lcrc/project/MCS-SLAC/software/pyOPALTools/opal/utilities/__init__.py", line 1, in <module>
import opal.utilities.logger
File "/lcrc/project/MCS-SLAC/software/pyOPALTools/opal/utilities/logger.py", line 28, in <module>
with open(path, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'opal/utilities/log.yaml'
```
He suggested: "adding a package_data clause to the setup.py, that may make the import pick up yaml files.
For example in libensemble I use this:
```package_data={'libensemble.sim_funcs.branin': ['known_minima_and_func_values']},```
to pick up the data file known_minima_and_func_values.
In any case, the PYTHONPATH approach seems to work too."
Does this make sense to do here? Thanks!frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/65update phase space plotter2019-10-14T15:48:35+02:00frey_mupdate phase space plotterfrey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/64SDDS dataframe with mixed types2019-10-14T15:48:36+02:00frey_mSDDS dataframe with mixed typesThe Pandas data frame returns only strings if we have mixed datatype. We need to distinguish types.
edit: [minimal stat file without header](/uploads/4cba5b475cf51e80097e9b18fa9de72e/headerremoved.stat)The Pandas data frame returns only strings if we have mixed datatype. We need to distinguish types.
edit: [minimal stat file without header](/uploads/4cba5b475cf51e80097e9b18fa9de72e/headerremoved.stat)frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/63JSON parsing of design variable bounds2019-10-14T15:48:35+02:00frey_mJSON parsing of design variable boundsThe bounds of the design variables are returned as a string, e.g. `'[0, 1]'`, instead of an array.The bounds of the design variables are returned as a string, e.g. `'[0, 1]'`, instead of an array.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/62SDDS data set: Return a dataframe rather than a raw array2019-09-01T16:08:18+02:00frey_mSDDS data set: Return a dataframe rather than a raw arrayRelated to #60Related to #60frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/61Fix data selection in H5Statistics2019-10-14T15:48:35+02:00frey_mFix data selection in H5StatisticsWhen selecting a turn of a probe H5 file, the full data set is taken instead of just bunch specific if the argument `bunch > -1`.When selecting a turn of a probe H5 file, the full data set is taken instead of just bunch specific if the argument `bunch > -1`.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/60Access monitor in file2019-09-05T10:32:14+02:00frey_mAccess monitor in fileThe monitor stat file writes a row per monitor to a SDDS. We should add a routine that allows to return a full row of a stat file.The monitor stat file writes a row per monitor to a SDDS. We should add a routine that allows to return a full row of a stat file.frey_mfrey_mhttps://gitlab.psi.ch/OPAL/pyOPALTools/-/issues/59Matplotlib styles update2019-08-15T14:14:07+02:00frey_mMatplotlib styles update`mpl.rcParams['text.latex.unicode']` is deprecated in version > 2.1.`mpl.rcParams['text.latex.unicode']` is deprecated in version > 2.1.frey_mfrey_m