Integration of Blissdata into BEC

Description

This MR settles ground to have Blissdata within BEC, and demonstrates basic functionality.

Testing it

Installation

Here are instructions to be able to test Blissdata and ESRF tools like Flint with BEC.

1. install RedisSearch and RedisJSON extensions
   • there are conda packages: conda install esrf-bcu::redisjson, conda install esrf-bcu::redisearch
2. update ophyd_devices and BEC server
   • BEC server dependencies update to the right Ophyd branch ; so, updating bec_server should be enough to update everything
     • Ophyd Devices needs to be on a specific branch: https://gitlab.psi.ch/bec/ophyd_devices/-/commits/signal_valueinfo
     • Blissdata will be installed ; required version is 2.0.0rc2 (as of 12/23/24)
3. if you want to use the BLISS Nexus Writer to save data you can install it with pip install blisswriter
4. if you want to try Flint, there is no package in PyPI yet (12/23/24) but it can easily be installed from source
   • clone Bliss: git clone https://gitlab.esrf.fr/bliss/bliss -b master
   • cd bliss/flint
   • pip install [-e] .

BEC configuration

Blissdata is meant to take "ownership" of a Redis database, indexing it with RedisSearch and having the Blissdata memory tracker evicting data to free memory if needed. Also, Redis configuration has to be tweaked depending on the beamline and the host computer capacity:

• Maximum memory has to be enough for at least one complete demanding scan (depends on the beamline)
  • the more the better...
• Blissdata can be used to write and read quite a lot of data of all sorts (very small or relatively bigger)
  • it is best to ensure maximum performance of Redis by configuring multiple I/O threads (one for each Redis client)
• RDB file saving has to be deactivated
• Redis eviction policy has to be set to "no eviction", because Blissdata memory tracker will deal with it

So, the recommended way is to dedicate a Redis process for Blissdata, to separate what is classic Redis operations (storing keys, pub/sub, etc) from data processing since the needs are different.

The following information has to be specified in BEC configuration YAML file:

redis:
  host: 127.0.0.1
  port: 24878
################ NEW
redis_data:
  host: 127.0.0.1
  port: 24879
####################
...
service_config:
  abort_on_ctrl_c: False
  enforce_ACLs: False
  ############### NEW
  redis_data:
    memory_limit: 2gb
  ###################
  file_writer:
    ...
    ############### NEW (optional)
    enable_bliss_writer: True
    ###############

If enable_bliss_writer is set to True in file_writer service configuration section, keep in mind that the BEC File Writer will not be started by bec-server ; BLISS Writer will be started instead.

Starting BEC servers

In order to make things simpler, bec-server start has a new --start-redis option, to automatically start Redis processes including the dedicated Redis process for Blissdata + memory tracker, properly configured:

bec-server start --start-redis --config ~/dev/bec/test_config.yaml

(--config is optional but it may be useful to have alternative config for BEC, like if the Bliss Writer is enabled or not)

This will start the 2 Redis, the memory tracker, and BEC servers. It makes a new tmux window called bec-redis to be able to check output of the processes:

(output of tmux attach -t bec-redis)

Starting BEC command line

BEC command line can be started like always. Do not forget to add --config with the path to the configuration file, though, if needed.

If Flint is installed there is one caveat: Flint is based on PyQt5 ; BEC Widgets is using Pyside6 - both can coexist without any problem, libraries are different, however you may have to set QT_API=pyside6 in the environment when running bec or anything from BEC Widgets, to tell which Qt binding qtpy has to use (surprisingly, it does not automatically goes to the highest version available).

Flint can be started on demand at startup, with the --flint flag on the bec command line.