Code indexing in gitaly is broken and leads to code not being visible to the user. We work on the issue with highest priority.

Skip to content

Resolve "investigate slow startup of bec_lib"

guijar_m requested to merge 283-investigate-slow-startup-of-bec_lib into main

Description

Thanks to profiling of module import time, it was determined which modules were taking time when doing import bec_lib.

The strategy to reduce import time is to replace the module with a placeholder, which will become the real module at the moment it is really used ("proxy" pattern applied to do lazy module loading). Like this, no change is done in the logic of the code and objects are still imported at top of the file etc. Of course, the import call itself changes to use the lazy import feature.

Changes:

  • new proxy.py module with Proxy class
    • code taken from BLISS (which was already taken from ActiveState recipe), this allows to have a proxy for any Python object
  • new lazy_import and lazy_import_from functions
    • substitute real import with a proxy object, calls importlib.import_module only when needed
  • reorganization of bec_lib.utils as a set of modules within a directory instead of a single file
  • tracked what was taking time, and replaced with lazy imports
    • some little changes to prevent early init of the proxies with real objects
    • 3 bigger changes:
      • all modules loaded in bec_lib.__init__.py have been cleared except bec_logger (because it makes kind of sense? It does not increase startup time thanks for other changes in bec_lib.logger)
      • threadlocked decorator usage is removed (not the decorator itself, but now it is for nothing hopefully...)
      • logger.__init__ not longer set level, this is done in configure() (otherwise IPython is called since the beginning)

I decided to keep numpy import because it does not add too much overhead and I didn't want to patch many modules with a numpy proxy just for a few milliseconds.

Result: bec IPython client startup passes from 1.5 second to 0.6

improved_import_time2

Closes #283 (closed)

Edited by guijar_m

Merge request reports