Plux
====
<p>
<a href="https://github.com/localstack/plux/actions/workflows/build.yml"><img alt="CI badge" src="https://github.com/localstack/plux/actions/workflows/build.yml/badge.svg"></img></a>
<a href="https://pypi.org/project/plux/"><img alt="PyPI Version" src="https://img.shields.io/pypi/v/plux?color=blue"></a>
<a href="https://img.shields.io/pypi/l/plux.svg"><img alt="PyPI License" src="https://img.shields.io/pypi/l/plux.svg"></a>
<a href="https://github.com/psf/black"><img alt="Code style: black" src="https://img.shields.io/badge/code%20style-black-000000.svg"></a>
</p>
plux is the dynamic code loading framework used in [LocalStack](https://github.com/localstack/localstack).
Overview
--------
Plux builds a higher-level plugin mechanism around [Python's entry point mechanism](https://packaging.python.org/specifications/entry-points/).
It provides tools to load plugins from entry points at run time, and to discover entry points from plugins at build time (so you don't have to declare entry points statically in your `setup.py`).
### Core concepts
* `PluginSpec`: describes a `Plugin`. Each plugin has a namespace, a unique name in that namespace, and a `PluginFactory` (something that creates `Plugin` the spec is describing.
In the simplest case, that can just be the Plugin's class).
* `Plugin`: an object that exposes a `should_load` and `load` method.
Note that it does not function as a domain object (it does not hold the plugins lifecycle state, like initialized, loaded, etc..., or other metadata of the Plugin)
* `PluginFinder`: finds plugins, either at build time (by scanning the modules using `pkgutil` and `setuptools`) or at run time (reading entrypoints of the distribution using [importlib](https://docs.python.org/3/library/importlib.metadata.html#entry-points))
* `PluginManager`: manages the run time lifecycle of a Plugin, which has three states:
* resolved: the entrypoint pointing to the PluginSpec was imported and the `PluginSpec` instance was created
* init: the `PluginFactory` of the `PluginSpec` was successfully invoked
* loaded: the `load` method of the `Plugin` was successfully invoked
### Loading Plugins
At run time, a `PluginManager` uses a `PluginFinder` that in turn uses importlib to scan the available entrypoints for things that look like a `PluginSpec`.
With `PluginManager.load(name: str)` or `PluginManager.load_all()`, plugins within the namespace that are discoverable in entrypoints can be loaded.
If an error occurs at any state of the lifecycle, the `PluginManager` informs the `PluginLifecycleListener` about it, but continues operating.
### Discovering entrypoints
To build a source distribution and a wheel of your code with your plugins as entrypoints, simply run `python setup.py plugins sdist bdist_wheel`.
If you don't have a `setup.py`, you can use the plux build frontend and run `python -m plux entrypoints`.
How it works:
For discovering plugins at build time, plux provides a custom setuptools command `plugins`, invoked via `python setup.py plugins`.
The command uses a special `PluginFinder` that collects from the codebase anything that can be interpreted as a `PluginSpec`, and creates from it a plugin index file `plux.json`, that is placed into the `.egg-info` distribution metadata directory.
When a setuptools command is used to create the distribution (e.g., `python setup.py sdist/bdist_wheel/...`), plux finds the `plux.json` plugin index and extends automatically the list of entry points (collected into `.egg-info/entry_points.txt`).
The `plux.json` file becomes a part of the distribution, s.t., the plugins do not have to be discovered every time your distribution is installed elsewhere.
Discovering at build time also works when using `python -m build`, since it calls registered setuptools scripts.
Examples
--------
To build something using the plugin framework, you will first want to introduce a Plugin that does something when it is loaded.
And then, at runtime, you need a component that uses the `PluginManager` to get those plugins.
### One class per plugin
This is the way we went with `LocalstackCliPlugin`. Every plugin class (e.g., `ProCliPlugin`) is essentially a singleton.
This is easy, as the classes are discoverable as plugins.
Simply create a Plugin class with a name and namespace and it will be discovered by the build time `PluginFinder`.
```python
from plux import Plugin
# abstract case (not discovered at build time, missing name)
class CliPlugin(Plugin):
namespace = "my.plugins.cli"
def load(self, cli):
self.attach(cli)
def attach(self, cli):
raise NotImplementedError
# discovered at build time (has a namespace, name, and is a Plugin)
class MyCliPlugin(CliPlugin):
name = "my"
def attach(self, cli):
# ... attach commands to cli object
```
now we need a `PluginManager` (which has a generic type) to load the plugins for us:
```python
cli = # ... needs to come from somewhere
manager: PluginManager[CliPlugin] = PluginManager("my.plugins.cli", load_args=(cli,))
plugins: List[CliPlugin] = manager.load_all()
# todo: do stuff with the plugins, if you want/need
# in this example, we simply use the plugin mechanism to run a one-shot function (attach) on a load argument
```
### Re-usable plugins
When you have lots of plugins that are structured in a similar way, we may not want to create a separate Plugin class
for each plugin. Instead we want to use the same `Plugin` class to do the same thing, but use several instances of it.
The `PluginFactory`, and the fact that `PluginSpec` instances defined at module level are discoverable (inpired
by [pluggy](https://github.com/pytest-dev/pluggy)), can be used to achieve that.
```python
from plux import Plugin, PluginFactory, PluginSpec
import importlib
class ServicePlugin(Plugin):
def __init__(self, service_name):
self.service_name = service_name
self.service = None
def should_load(self):
return self.service_name in config.SERVICES
def load(self):
module = importlib.import_module("localstack.services.%s" % self.service_name)
# suppose we define a convention that each service module has a Service class, like moto's `Backend`
self.service = module.Service()
def service_plugin_factory(name) -> PluginFactory:
def create():
return ServicePlugin(name)
return create
# discoverable
s3 = PluginSpec("localstack.plugins.services", "s3", service_plugin_factory("s3"))
# discoverable
dynamodb = PluginSpec("localstack.plugins.services", "dynamodb", service_plugin_factory("dynamodb"))
# ... could be simplified with convenience framework code, but the principle will stay the same
```
Then we could use the `PluginManager` to build a Supervisor
```python
from plux import PluginManager
class Supervisor:
manager: PluginManager[ServicePlugin]
def start(self, service_name):
plugin = self.manager.load(service_name)
service = plugin.service
service.start()
```
### Functions as plugins
with the `@plugin` decorator, you can expose functions as plugins. They will be wrapped by the framework
into `FunctionPlugin` instances, which satisfy both the contract of a Plugin, and that of the function.
```python
from plugin import plugin
@plugin(namespace="localstack.configurators")
def configure_logging(runtime):
logging.basicConfig(level=runtime.config.loglevel)
@plugin(namespace="localstack.configurators")
def configure_somethingelse(runtime):
# do other stuff with the runtime object
pass
```
With a PluginManager via `load_all`, you receive the `FunctionPlugin` instances, that you can call like the functions
```python
runtime = LocalstackRuntime()
for configurator in PluginManager("localstack.configurators").load_all():
configurator(runtime)
```
Configuring your distribution
-----------------------------
If you are building a python distribution that exposes plugins discovered by plux, you need to configure your projects build system so other dependencies creates the `entry_points.txt` file when installing your distribution.
For a [`pyproject.toml`](https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/) template this involves adding the `build-system` section:
```toml
[build-system]
requires = ['setuptools', 'wheel', 'plux>=1.3.1']
build-backend = "setuptools.build_meta"
# ...
```
Install
-------
pip install plux
Develop
-------
Create the virtual environment, install dependencies, and run tests
make venv
make test
Run the code formatter
make format
Upload the pypi package using twine
make upload
Raw data
{
"_id": null,
"home_page": "https://github.com/localstack/plux",
"name": "plux",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": null,
"author": "Thomas Rausch",
"author_email": "thomas@localstack.cloud",
"download_url": "https://files.pythonhosted.org/packages/dd/aa/d40ba10368d566c8e4bf62ac7ce0dcea2e6b7cc1102c1b39e07495b2d347/plux-1.12.1.tar.gz",
"platform": null,
"description": "Plux\n====\n\n<p>\n <a href=\"https://github.com/localstack/plux/actions/workflows/build.yml\"><img alt=\"CI badge\" src=\"https://github.com/localstack/plux/actions/workflows/build.yml/badge.svg\"></img></a>\n <a href=\"https://pypi.org/project/plux/\"><img alt=\"PyPI Version\" src=\"https://img.shields.io/pypi/v/plux?color=blue\"></a>\n <a href=\"https://img.shields.io/pypi/l/plux.svg\"><img alt=\"PyPI License\" src=\"https://img.shields.io/pypi/l/plux.svg\"></a>\n <a href=\"https://github.com/psf/black\"><img alt=\"Code style: black\" src=\"https://img.shields.io/badge/code%20style-black-000000.svg\"></a>\n</p>\n\nplux is the dynamic code loading framework used in [LocalStack](https://github.com/localstack/localstack).\n\n\nOverview\n--------\n\nPlux builds a higher-level plugin mechanism around [Python's entry point mechanism](https://packaging.python.org/specifications/entry-points/).\nIt provides tools to load plugins from entry points at run time, and to discover entry points from plugins at build time (so you don't have to declare entry points statically in your `setup.py`).\n\n### Core concepts\n\n* `PluginSpec`: describes a `Plugin`. Each plugin has a namespace, a unique name in that namespace, and a `PluginFactory` (something that creates `Plugin` the spec is describing.\n In the simplest case, that can just be the Plugin's class).\n* `Plugin`: an object that exposes a `should_load` and `load` method.\n Note that it does not function as a domain object (it does not hold the plugins lifecycle state, like initialized, loaded, etc..., or other metadata of the Plugin)\n* `PluginFinder`: finds plugins, either at build time (by scanning the modules using `pkgutil` and `setuptools`) or at run time (reading entrypoints of the distribution using [importlib](https://docs.python.org/3/library/importlib.metadata.html#entry-points))\n* `PluginManager`: manages the run time lifecycle of a Plugin, which has three states:\n * resolved: the entrypoint pointing to the PluginSpec was imported and the `PluginSpec` instance was created\n * init: the `PluginFactory` of the `PluginSpec` was successfully invoked\n * loaded: the `load` method of the `Plugin` was successfully invoked\n\n\n\n### Loading Plugins\n\nAt run time, a `PluginManager` uses a `PluginFinder` that in turn uses importlib to scan the available entrypoints for things that look like a `PluginSpec`.\nWith `PluginManager.load(name: str)` or `PluginManager.load_all()`, plugins within the namespace that are discoverable in entrypoints can be loaded.\nIf an error occurs at any state of the lifecycle, the `PluginManager` informs the `PluginLifecycleListener` about it, but continues operating.\n\n### Discovering entrypoints\n\nTo build a source distribution and a wheel of your code with your plugins as entrypoints, simply run `python setup.py plugins sdist bdist_wheel`.\nIf you don't have a `setup.py`, you can use the plux build frontend and run `python -m plux entrypoints`.\n\nHow it works:\nFor discovering plugins at build time, plux provides a custom setuptools command `plugins`, invoked via `python setup.py plugins`.\nThe command uses a special `PluginFinder` that collects from the codebase anything that can be interpreted as a `PluginSpec`, and creates from it a plugin index file `plux.json`, that is placed into the `.egg-info` distribution metadata directory.\nWhen a setuptools command is used to create the distribution (e.g., `python setup.py sdist/bdist_wheel/...`), plux finds the `plux.json` plugin index and extends automatically the list of entry points (collected into `.egg-info/entry_points.txt`).\nThe `plux.json` file becomes a part of the distribution, s.t., the plugins do not have to be discovered every time your distribution is installed elsewhere.\nDiscovering at build time also works when using `python -m build`, since it calls registered setuptools scripts.\n\n\nExamples\n--------\n\nTo build something using the plugin framework, you will first want to introduce a Plugin that does something when it is loaded.\nAnd then, at runtime, you need a component that uses the `PluginManager` to get those plugins.\n\n### One class per plugin\n\nThis is the way we went with `LocalstackCliPlugin`. Every plugin class (e.g., `ProCliPlugin`) is essentially a singleton.\nThis is easy, as the classes are discoverable as plugins.\nSimply create a Plugin class with a name and namespace and it will be discovered by the build time `PluginFinder`.\n\n```python\nfrom plux import Plugin\n\n# abstract case (not discovered at build time, missing name)\nclass CliPlugin(Plugin):\n namespace = \"my.plugins.cli\"\n\n def load(self, cli):\n self.attach(cli)\n\n def attach(self, cli):\n raise NotImplementedError\n\n# discovered at build time (has a namespace, name, and is a Plugin)\nclass MyCliPlugin(CliPlugin):\n name = \"my\"\n\n def attach(self, cli):\n # ... attach commands to cli object\n\n```\n\nnow we need a `PluginManager` (which has a generic type) to load the plugins for us:\n\n```python\ncli = # ... needs to come from somewhere\n\nmanager: PluginManager[CliPlugin] = PluginManager(\"my.plugins.cli\", load_args=(cli,))\n\nplugins: List[CliPlugin] = manager.load_all()\n\n# todo: do stuff with the plugins, if you want/need\n# in this example, we simply use the plugin mechanism to run a one-shot function (attach) on a load argument\n\n```\n\n### Re-usable plugins\n\nWhen you have lots of plugins that are structured in a similar way, we may not want to create a separate Plugin class\nfor each plugin. Instead we want to use the same `Plugin` class to do the same thing, but use several instances of it.\nThe `PluginFactory`, and the fact that `PluginSpec` instances defined at module level are discoverable (inpired\nby [pluggy](https://github.com/pytest-dev/pluggy)), can be used to achieve that.\n\n```python\nfrom plux import Plugin, PluginFactory, PluginSpec\nimport importlib\n\nclass ServicePlugin(Plugin):\n\n def __init__(self, service_name):\n self.service_name = service_name\n self.service = None\n\n def should_load(self):\n return self.service_name in config.SERVICES\n\n def load(self):\n module = importlib.import_module(\"localstack.services.%s\" % self.service_name)\n # suppose we define a convention that each service module has a Service class, like moto's `Backend`\n self.service = module.Service()\n\ndef service_plugin_factory(name) -> PluginFactory:\n def create():\n return ServicePlugin(name)\n\n return create\n\n# discoverable\ns3 = PluginSpec(\"localstack.plugins.services\", \"s3\", service_plugin_factory(\"s3\"))\n\n# discoverable\ndynamodb = PluginSpec(\"localstack.plugins.services\", \"dynamodb\", service_plugin_factory(\"dynamodb\"))\n\n# ... could be simplified with convenience framework code, but the principle will stay the same\n\n```\n\nThen we could use the `PluginManager` to build a Supervisor\n\n```python\nfrom plux import PluginManager\n\nclass Supervisor:\n manager: PluginManager[ServicePlugin]\n\n def start(self, service_name):\n plugin = self.manager.load(service_name)\n service = plugin.service\n service.start()\n\n```\n\n### Functions as plugins\n\nwith the `@plugin` decorator, you can expose functions as plugins. They will be wrapped by the framework\ninto `FunctionPlugin` instances, which satisfy both the contract of a Plugin, and that of the function.\n\n```python\nfrom plugin import plugin\n\n@plugin(namespace=\"localstack.configurators\")\ndef configure_logging(runtime):\n logging.basicConfig(level=runtime.config.loglevel)\n\n \n@plugin(namespace=\"localstack.configurators\")\ndef configure_somethingelse(runtime):\n # do other stuff with the runtime object\n pass\n```\n\nWith a PluginManager via `load_all`, you receive the `FunctionPlugin` instances, that you can call like the functions\n\n```python\n\nruntime = LocalstackRuntime()\n\nfor configurator in PluginManager(\"localstack.configurators\").load_all():\n configurator(runtime)\n```\n\nConfiguring your distribution\n-----------------------------\n\nIf you are building a python distribution that exposes plugins discovered by plux, you need to configure your projects build system so other dependencies creates the `entry_points.txt` file when installing your distribution.\n\nFor a [`pyproject.toml`](https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/) template this involves adding the `build-system` section:\n\n```toml\n[build-system]\nrequires = ['setuptools', 'wheel', 'plux>=1.3.1']\nbuild-backend = \"setuptools.build_meta\"\n\n# ...\n```\n\nInstall\n-------\n\n pip install plux\n\nDevelop\n-------\n\nCreate the virtual environment, install dependencies, and run tests\n\n make venv\n make test\n\nRun the code formatter\n\n make format\n\nUpload the pypi package using twine\n\n make upload\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "A dynamic code loading framework for building pluggable Python distributions",
"version": "1.12.1",
"project_urls": {
"Homepage": "https://github.com/localstack/plux"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "7cc328e7eb46b3de177e276409bc6e225893c4884cd30c557c292686a0fcbbdf",
"md5": "8594866703d8c5cd445b3d5ef1b8f011",
"sha256": "b4aa4e67329f2fcd73fb28096a8a9304f5912ee6cce39994eac567e8eec65488"
},
"downloads": -1,
"filename": "plux-1.12.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8594866703d8c5cd445b3d5ef1b8f011",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 33922,
"upload_time": "2024-10-24T16:59:46",
"upload_time_iso_8601": "2024-10-24T16:59:46.390435Z",
"url": "https://files.pythonhosted.org/packages/7c/c3/28e7eb46b3de177e276409bc6e225893c4884cd30c557c292686a0fcbbdf/plux-1.12.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "ddaad40ba10368d566c8e4bf62ac7ce0dcea2e6b7cc1102c1b39e07495b2d347",
"md5": "8c0b09a9f033c76775531e99c81bfce1",
"sha256": "1ed44a6edbb7343f4711ff75ddaf87bed066c53625d41b63a0b4edd3f77792ba"
},
"downloads": -1,
"filename": "plux-1.12.1.tar.gz",
"has_sig": false,
"md5_digest": "8c0b09a9f033c76775531e99c81bfce1",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 31566,
"upload_time": "2024-10-24T16:59:48",
"upload_time_iso_8601": "2024-10-24T16:59:48.197738Z",
"url": "https://files.pythonhosted.org/packages/dd/aa/d40ba10368d566c8e4bf62ac7ce0dcea2e6b7cc1102c1b39e07495b2d347/plux-1.12.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-24 16:59:48",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "localstack",
"github_project": "plux",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "plux"
}
JSON
