ci, build: transition to pyproject.toml

Description

Transition from setup.py, setup.cfg to pyproject.toml

Related Issues

Fix #151 (closed)

Type of Change

• use pyproject.toml for build configuration
• use build (https://pypa-build.readthedocs.io/en/latest/) as build frontend
• use hatchling (https://hatch.pypa.io/latest/why/#build-backend) as build backend

Additional Comments

Notes:

• editable mode with pyproject.toml and setuptools has quite a few limitations and associated "bugs", so trying a different build frontend/backend might be a way to go
• in fact, setup.py itself is not deprecated (https://packaging.python.org/en/latest/discussions/setup-py-deprecated/#setup-py-deprecated), here, we only avoid using deprecated python setup.py sdist bdist_wheel and python setup.py develop invocations
• other build backends don't support non-python setuptools' scripts entry points
• trying to keep editable mode propagation through e.g. bec-server to other bec deps doesn't work anymore (pip install -e ... invocation in python subprocess fails)
• pip show displays location of editable installs differently (instead of "Location" one should grep for "Editable project location"), and earlier versions of pip have a bug: https://github.com/pypa/pip/issues/11638 (see changes in bec_client/bin/bec), fixed in pip 23.0 (2023-01-30) https://pip.pypa.io/en/stable/news/#id115 bec_client/bin/bec is rewritten in python, and the file locations are extracted from package metadata
• editable installs do not work when python run from bec folder https://github.com/pypa/setuptools/issues/3557 e.g. importing bec_lib will cause python to import bec/bec_lib folder, which doesn't have __init__.py (namespace package). For now, it's solved with --config-settings editable_mode=strict (test with setuptools>=61,<65) wrong search path was added to sys.path, which is fixed after transitioning to hatchling
• conda doesn't recognize editable pip installs (with pyproject.toml) as local, instead they are shown as pypi, like any other package installed from pypi index (in general, pyproject.toml support in conda is not fully there: https://github.com/conda/conda/issues/12462, however, the project is already using hatch as a build system: https://github.com/conda/conda/issues/12508)
• pip install -e ./bec_server won't automatically install other bec dependencies (i.e. scan_server, device_server, file_writer, etc.) in development mode. This customized installation step (available only through "overloading" setup.py), while being handy, introduces unexpected and implicit behavior to pip install -e. IMHO, an explicit installation via a script (or, potentially, hatch env) would be a cleaner solution, keeping pip install -e semantically predictable (relevant: https://github.com/pypa/pip/issues/11881)
• there is an open issue with hatch to add support for editable dependencies: https://github.com/pypa/hatch/issues/588, which should provide an alternative solution to the previous point
• With poetry it's possible to specify dependencies to be installed in editable mode, e.g.

  [tool.poetry.dependencies]
  bec-lib = {path = "../bec_lib", develop = true}

  and it works with their build frontend poetry install bec_server. However, this functionality doesn't seem to be supported with pip install -e bec_server. Furthermore, currently poetry uses non-standard format for pyproject.toml configs (https://github.com/python-poetry/poetry/issues/3332), but it shouldn't be too difficult to work with it.

TODOs for this MR:

• fix bec-server start in conda end-2-end tests (Seems like this was caused by the presence of --config-settings editable_mode=strict, which is not present anymore after adopting hatchling. Still worth noting that something is wrong with conda end-2-end test on main branch anyways, as despite installing bec_lib[dev] on e.g. https://gitlab.psi.ch/bec/bec/-/jobs/82377#L162, the pandas dependency is not present on path: https://gitlab.psi.ch/bec/bec/-/jobs/82377#L234, but it should)
• verify semantic-release publish works (no access to v7 docs? but semantic-release version -D version_variable=... -D version_toml=... does produce the desired result)
• in bin/install_bec_dev.sh support installation of bec packages into separate venvs
• check if wheel is no longer needed with pip>23.2 (relevant: https://github.com/pypa/pip/issues/12094)
• clarify a need for bec/semantic_release/__init__.py (it's still used in docs as a "representative" version for all packages)

Further ideas:

• other tools can read their configurations from pyproject.toml too:
  • https://black.readthedocs.io/en/stable/usage_and_configuration/the_basics.html#what-on-earth-is-a-pyproject-toml-file
  • https://pylint.readthedocs.io/en/stable/user_guide/usage/run.html#command-line-options
  • https://python-semantic-release.readthedocs.io/en/latest/configuration.html

etc. Possibly, vscode/pycharm could be configured to use pyproject.toml files too. For python-semantic-release it might require adopting v8 (https://github.com/python-semantic-release/python-semantic-release/issues/572).

• introduce src layout(?) (https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/)
• explore hatch as build frontend