bec_service: service_info and metrics are constantly sending messages to Redis
Bug report (well, not really)
Summary
Examination of redis commands with the MONITOR
mode is obfuscated by constantly exchanged messages
to update bec_service
service info and metrics.
Expected Behavior vs Actual Behavior
No constant communication with Redis for updating service info and metrics - no communication with Redis at all when "nothing" is happening.
Steps to Reproduce
Just put redis in MONITOR mode, you see:
(bec) [matias@kashyyyk bec]$ redis-cli
127.0.0.1:6379> MONITOR
OK
1705591083.886344 [0 [::1]:60322] "PUBLISH" "internal/services/metrics/907f5fb0-43bb-4a92-909a-602f99ee0304" "MSGVERSION_1.2_80_334_EOH_{\"msg_type\": \"service_metric_message\", \"version\": 1.2, \"compression\": \"msgpack\"}\x82\xa7content\x82\xa4name\xa6SciHub\xa7metrics\x8b\xa3pid\xcdU\xd6\xa7cmdline\x92\xd9+/home/matias/miniconda3/envs/bec/bin/python\xd9//home/matias/miniconda3/envs/bec/bin/bec-scihub\xacprocess_name\xaabec-scihub\xa8username\xa6matias\xa8hostname\xa8kashyyyk\xabcpu_percent\xcb?\xf0\x00\x00\x00\x00\x00\x00\xa9cpu_times\xcb?\xed\x1e\xb8Q\xeb\x85\x1f\xabnum_threads\x0e\xacmemory_in_mb\xcb@P\xe7\x80\x00\x00\x00\x00\xb3memory_used_percent\xcb?\xdbA\x03)@\xb8\xcd\xabcreate_time\xcbA\xd9jPH\xc3\xd7\n\xa8metadata\x80"
1705591083.967919 [0 [::1]:60372] "PUBLISH" "internal/services/metrics/d413dadb-aa70-47ea-95d4-37e28ce67555" "MSGVERSION_1.2_80_331_EOH_{\"msg_type\": \"service_metric_message\", \"version\": 1.2, \"compression\": \"msgpack\"}\x82\xa7content\x82\xa4name\xa9DAPServer\xa7metrics\x8b\xa3pid\xcdU\xe5\xa7cmdline\x92\xd9+/home/matias/miniconda3/envs/bec/bin/python\xd9,/home/matias/miniconda3/envs/bec/bin/bec-dap\xacprocess_name\xa7bec-dap\xa8username\xa6matias\xa8hostname\xa8kashyyyk\xabcpu_percent\xcb?\xf0\x00\x00\x00\x00\x00\x00\xa9cpu_times\xcb?\xf1\xeb\x85\x1e\xb8Q\xec\xabnum_threads\r\xacmemory_in_mb\xcb@Z\xae\xc0\x00\x00\x00\x00\xb3memory_used_percent\xcb?\xe5\x82pO\xc5\xa9\xaa\xabcreate_time\xcbA\xd9jPH\xe4z\xe1\xa8metadata\x80"
1705591083.994687 [0 [::1]:60236] "PUBLISH" "internal/services/metrics/87b92d36-3f22-4f30-a475-2cfba48a14c3" "MSGVERSION_1.2_80_348_EOH_{\"msg_type\": \"service_metric_message\", \"version\": 1.2, \"compression\": \"msgpack\"}\x82\xa7content\x82\xa4name\xaaScanServer\xa7metrics\x8b\xa3pid\xcdU\xc0\xa7cmdline\x92\xd9+/home/matias/miniconda3/envs/bec/bin/python\xd94/home/matias/miniconda3/envs/bec/bin/bec-scan-server\xacprocess_name\xafbec-scan-server\xa8username\xa6matias\xa8hostname\xa8kashyyyk\xabcpu_percent\xcb?\xf0\x00\x00\x00\x00\x00\x00\xa9cpu_times\xcb?\xf0\x00\x00\x00\x00\x00\x00\xabnum_threads\x0f\xacmemory_in_mb\xcb@Px\x80\x00\x00\x00\x00\xb3memory_used_percent\xcb?\xda\x8e\rvH\x06$\xabcreate_time\xcbA\xd9jPH\x99\x99\x9a\xa8metadata\x80"
1705591084.081926 [0 [::1]:60278] "PUBLISH" "internal/services/metrics/5c549217-efd6-42a1-966b-af9aaf5029db" "MSGVERSION_1.2_80_350_EOH_{\"msg_type\": \"service_metric_message\", \"version\": 1.2, \"compression\": \"msgpack\"}\x82\xa7content\x82\xa4name\xabScanBundler\xa7metrics\x8b\xa3pid\xcdU\xc8\xa7cmdline\x92\xd9+/home/matias/miniconda3/envs/bec/bin/python\xd95/home/matias/miniconda3/envs/bec/bin/bec-scan-bundler\xacprocess_name\xafbec-scan-bundle\xa8username\xa6matias\xa8hostname\xa8kashyyyk\xabcpu_percent\xcb?\xf0\x00\x00\x00\x00\x00\x00\xa9cpu_times\xcb?\xee\xb8Q\xeb\x85\x1e\xb8\xabnum_threads\r\xacmemory_in_mb\xcb@P*\xc0\x00\x00\x00\x00\xb3memory_used_percent\xcb?\xda\x10\xb3<aQ\xa7\xabcreate_time\xcbA\xd9jPH\xa7\n=\xa8metadata\x80"
1705591084.244966 [0 [::1]:60494] "PUBLISH" "internal/services/metrics/1db27f47-da41-45ea-a476-29ffa492416c" "MSGVERSION_1.2_80_352_EOH_{\"msg_type\": \"service_metric_message\", \"version\": 1.2, \"compression\": \"msgpack\"}\x82\xa7content\x82\xa4name\xacDeviceServer\xa7metrics\x8b\xa3pid\xcdU\xd7\xa7cmdline\x92\xd9+/home/matias/miniconda3/envs/bec/bin/python\xd96/home/matias/miniconda3/envs/bec/bin/bec-device-server\xacprocess_name\xafbec-device-serv\xa8username\xa6matias\xa8hostname\xa8kashyyyk\xabcpu_percent\xcb@\x00\x00\x00\x00\x00\x00\x00\xa9cpu_times\xcb?\xfa\x8f\\(\xf5\xc2\x8f\xabnum_threads\x17\xacmemory_in_mb\xcb@YI\x80\x00\x00\x00\x00\xb3memory_used_percent\xcb?\xe4bsV\xaf\x1bt\xabcreate_time\xcbA\xd9jPH\xc6ff\xa8metadata\x80"
1705591084.474037 [0 [::1]:60612] "MULTI"
1705591084.474105 [0 [::1]:60612] "PUBLISH" "internal/services/status/98ef07ed-38e1-462c-a4d8-a5e52c6e6bf2" "MSGVERSION_1.2_72_106_EOH_{\"msg_type\": \"status_message\", \"version\": 1.2, \"compression\": \"msgpack\"}\x82\xa7content\x83\xa4name\xb1FileWriterManager\xa6status\x01\xa4info\x83\xa4user\xa6matias\xa8hostname\xa8kashyyyk\xa9timestamp\xcbA\xd9jPK\x1e<m\xa8metadata\x80"
...
Proposal
What about using tmux
to get the state of services instead of using Redis ?
Presumably if a service is properly running in a tmux pane, it is up.
For metrics: I am not sure what those are about, but maybe there is another way to get them ?
Alternative proposal
Let's use a secondary Redis DB to exchange those messages, so it does not disturb the main DB (there can be 256 DBs in one single Redis instance, identified by number)
But, it would still put some stress on the redis server, which can be avoided IMO.