selinon

Name	selinon JSON
Version	1.3.0.post0 JSON
	download
home_page	https://github.com/selinon/selinon
Summary	an advanced dynamic task flow management on top of Celery
upload_time	2023-01-27 13:17:59
maintainer	Fridolin Pokorny
docs_url	None
author	Fridolin Pokorny
requires_python
license	BSD
keywords	selinon celery yaml flow distributed-computing
VCS
bugtrack_url
requirements	No requirements were recorded.
Travis-CI
coveralls test coverage

Selinon
=======

An advanced task flow management on top of
`Celery <https://www.celeryproject.org/>`__.

Is this project helpful? `Send me a simple warm
message <https://saythanks.io/to/fridex>`__!

Crossroad
---------

- `PyPI <https://pypi.python.org/pypi/selinon>`__
- `Documentation <https://selinon.readthedocs.io/en/latest/>`__
- `Developers
documentation <https://selinon.readthedocs.io/en/latest/development.html>`__
- `Travis CI <https://travis-ci.org/selinon/selinon>`__

Last stable release: `Selinon 1.3.0 <https://pypi.org/project/selinon/1.3.0/>`_

TLDR;
-----

An advanced flow management above Celery (an asynchronous distributed
task queue) written in Python3, that allows you to:

- Dynamically schedule tasks based on results of previous tasks
- Group tasks into flows in simple YAML configuration files
- Schedule flows from other flows (even recursively)
- Store results of tasks in your storages and databases transparently,
validate results against defined JSON schemas
- Do redeployment respecting changes in the YAML configuration files
without purging queues (migrations)
- Track flow progress via the build-in tracing mechanism
- Complex per-task or per-flow failure handling with fallback tasks or
fallback flows
- No DAG limitation in your flows
- Selectively pick tasks in your flow graphs that should be executed
respecting task dependencies
- Make your deployment easy to orchestrate using orchestration tools
such as `Kubernetes <https://kubernetes.io>`__
- Highly scalable Turing complete solution for big data processing pipelines
- And (of course) much more... check
`docs <https://selinon.readthedocs.io>`__

YouTube Video
-------------

Let's explain Selinon using a `YouTube video (click to redirect to YouTube) <https://www.youtube.com/watch?v=Wwwi8e2wI1w>`_.

.. figure:: https://raw.githubusercontent.com/selinon/selinon/master/example/ironwood.jpeg
:alt: Ironwoodland
:align: center

About
-----

This tool is an implementation above Celery that enables you to define
flows and dependencies in flows, schedule tasks based on results of
Celery workers, their success or any external events. If you are not
familiar with Celery, check out its homepage
`www.celeryproject.org <http://www.celeryproject.org>`__ or `this nice
tutorial <https://tests4geeks.com/distribute-tasks-python-celery-rabbitmq/>`__.

Selinon was originally designed to take care of advanced flows in one of
Red Hat products, where it already served thousands of flows and tasks.
Its main aim is to simplify specifying group of tasks, grouping tasks
into flows, handle data and execution dependencies between tasks and
flows, easily reuse tasks and flows, model advanced execution units in
YAML configuration files and make the whole system easy to model, easy
to maintain and easy to debug.

By placing declarative configuration of the whole system into YAML files
you can keep tasks as simple as needed. Storing results of tasks in
databases, modeling dependencies or executing fallback tasks/flows on
failures are separated from task logic. This gives you a power to
dynamically change task and flow dependencies on demand, optimize data
retrieval and data storage from databases per task bases or even track
progress based on events traced in the system.

Selinon was designed to serve millions of tasks in clusters or data
centers orchestrated by `Kubernetes <https://kubernetes.io>`__,
`OpenShift <https://openshift.com>`__ or any other orchestration tool,
but can simplify even small systems. Moreover, Selinon can make them
easily scalable in the future and make developer's life much easier.

A Quick First Overview
----------------------

Selinon is serving recipes in a distributed environment, so let's make a
dinner!

If we want to make a dinner, we need to buy ingredients. These
ingredients are bought in ``buyIngredientsFlow``. This flow consists of
multiple tasks, but let's focus on our main flow. Once all ingredients
are bought, we can start preparing our dinner in ``prepareFlow``. Again,
this flow consists of some additional steps that need to be done in
order to accomplish our future needs. As you can see, if anything goes
wrong in mentioned flows (see red arrows), we make a fallback to pizza
with beer which we order. To make beer cool, we place it to our
``Fridge`` storage. If we successfully finished ``prepareFlow`` after
successful shopping, we can proceed to ``serveDinnerFlow``.

Just to point out - grey nodes represent flows (which can be made of
other flows or tasks) and white (rounded) nodes are tasks. Conditions
are represented in hexagons (see bellow). Black arrows represent time or
data dependencies between our nodes, grey arrows pinpoint where results
of tasks are stored.

.. figure:: https://raw.githubusercontent.com/selinon/selinon/master/example/graph/dinnerFlow.png
:alt: Main dinner flow
:align: center

For our dinner we need eggs, flour and some additional ingredients.
Moreover, we conditionally buy a flower based on our condition. Our task
``BuyFlowerTask`` will not be scheduled (or executed) if our condition
is ``False``. Conditions are made of predicates and these predicates can
be grouped as desired with logical operators. You can define your own
predicates if you want (default are available in
``selinon.predicates``). Everything that is bought is stored in
``Basket`` storage transparently.

Let's visualise our ``buyIngredientsFlow``:

.. figure:: https://raw.githubusercontent.com/selinon/selinon/master/example/graph/buyIngredientsFlow.png
:alt: Buy ingredients flow
:align: center

As stated in our main flow after buying ingredients, we proceed to
dinner preparation but first we need to check our recipe that is hosted
at ``http://recipes.lan/how-to-bake-pie.html``. Any ingredients we
bought are transparently retrieved from defined storage as defined in
our YAML configuration file. We warm up our oven to expected temperature
and once the temperature is reached and we have finished with dough, we
can proceed to baking.

Based on the description above, our ``prepareFlow`` will look like the
following graph:

.. figure:: https://raw.githubusercontent.com/selinon/selinon/master/example/graph/prepareFlow.png
:alt: Prepare flow
:align: center

Once everything is done we serve plates. As we want to serve plates for
all guests we need to make sure we schedule N tasks of type
``ServePlateTask``. Each time we run our whole dinner flow, number of
guests may vary so make sure no guest stays hungry. Our
``serveDinnerFlow`` would look like the following graph:

.. figure:: https://raw.githubusercontent.com/selinon/selinon/master/example/graph/serveDinnerFlow.png
:alt: Serve dinner flow
:align: center

This example demonstrates very simple flows. The whole configuration can
be found `here </example/dinner.yaml>`__. Just check it out how you can
easily define your flows! You can find a script that visualises graphs
based on the YAML configuration in `this repo </example/>`__ as well.

More info
---------

The example was intentionally simplified. You can also parametrize your
flows, schedule N tasks (where N is a run-time evaluated variable), do
result caching, placing tasks on separate queues in order to be capable
of doing fluent system updates, throttle execution of certain tasks in
time, propagate results of tasks to sub-flows etc. Just check
`documentation <https://selinon.github.io/selinon>`__ for more info.

Live Demo
---------

A live demo with few examples can be found
`here <https://github.com/selinon/demo>`__. Feel free to check it out.

Installation
------------

$ pip3 install selinon

Available extras:

* celery - needed if you use Celery
* mongodb - needed for MongoDB `storage adapter <http://selinon.readthedocs.io/en/latest/storage.html>`_
* postgresql - needed for PostgreSQL `storage adapter <http://selinon.readthedocs.io/en/latest/storage.html>`_
* redis - needed for Redis `storage adapter <http://selinon.readthedocs.io/en/latest/storage.html>`_
* s3 - needed for S3 `storage adapter <http://selinon.readthedocs.io/en/latest/storage.html>`_
* sentry - needed for `Sentry support <http://selinon.readthedocs.io/en/latest/trace.html#sentry-integration>`_

Extras can be installed via:

$ pip3 install selinon[celery,mongodb,postgresql,redis,s3,sentry]

Feel free to select only needed extras for your setup.

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/selinon/selinon",
    "name": "selinon",
    "maintainer": "Fridolin Pokorny",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "fridolin.pokorny@gmail.com",
    "keywords": "selinon celery yaml flow distributed-computing",
    "author": "Fridolin Pokorny",
    "author_email": "fridolin.pokorny@gmail.com",
    "download_url": "https://files.pythonhosted.org/packages/32/0a/a1fc9640ab9598800d2ed964e01e157897818efcb9fa8f1d93dd74c31e5b/selinon-1.3.0.post0.tar.gz",
    "platform": null,
    "description": "Selinon\n=======\n\nAn advanced task flow management on top of\n`Celery <https://www.celeryproject.org/>`__.\n\n|codecov| |PyPI Current Version| |PyPI Implementation| |PyPI Wheel|\n|Travis CI| |Documentation Status| |GitHub stars| |GitHub license|\n|Twitter|\n\nIs this project helpful? `Send me a simple warm\nmessage <https://saythanks.io/to/fridex>`__!\n\nCrossroad\n---------\n\n-  `PyPI <https://pypi.python.org/pypi/selinon>`__\n-  `Documentation <https://selinon.readthedocs.io/en/latest/>`__\n-  `Developers\n   documentation <https://selinon.readthedocs.io/en/latest/development.html>`__\n-  `Travis CI <https://travis-ci.org/selinon/selinon>`__\n\n\nLast stable release: `Selinon 1.3.0 <https://pypi.org/project/selinon/1.3.0/>`_\n\nTLDR;\n-----\n\nAn advanced flow management above Celery (an asynchronous distributed\ntask queue) written in Python3, that allows you to:\n\n-  Dynamically schedule tasks based on results of previous tasks\n-  Group tasks into flows in simple YAML configuration files\n-  Schedule flows from other flows (even recursively)\n-  Store results of tasks in your storages and databases transparently,\n   validate results against defined JSON schemas\n-  Do redeployment respecting changes in the YAML configuration files\n   without purging queues (migrations)\n-  Track flow progress via the build-in tracing mechanism\n-  Complex per-task or per-flow failure handling with fallback tasks or\n   fallback flows\n-  No DAG limitation in your flows\n-  Selectively pick tasks in your flow graphs that should be executed\n   respecting task dependencies\n-  Make your deployment easy to orchestrate using orchestration tools\n   such as `Kubernetes <https://kubernetes.io>`__\n-  Highly scalable Turing complete solution for big data processing pipelines\n-  And (of course) much more... check\n   `docs <https://selinon.readthedocs.io>`__\n\nYouTube Video\n-------------\n\nLet's explain Selinon using a `YouTube video (click to redirect to YouTube) <https://www.youtube.com/watch?v=Wwwi8e2wI1w>`_.\n\n.. figure:: https://raw.githubusercontent.com/selinon/selinon/master/example/ironwood.jpeg\n   :alt: Ironwoodland\n   :align: center\n\nAbout\n-----\n\nThis tool is an implementation above Celery that enables you to define\nflows and dependencies in flows, schedule tasks based on results of\nCelery workers, their success or any external events. If you are not\nfamiliar with Celery, check out its homepage\n`www.celeryproject.org <http://www.celeryproject.org>`__ or `this nice\ntutorial <https://tests4geeks.com/distribute-tasks-python-celery-rabbitmq/>`__.\n\nSelinon was originally designed to take care of advanced flows in one of\nRed Hat products, where it already served thousands of flows and tasks.\nIts main aim is to simplify specifying group of tasks, grouping tasks\ninto flows, handle data and execution dependencies between tasks and\nflows, easily reuse tasks and flows, model advanced execution units in\nYAML configuration files and make the whole system easy to model, easy\nto maintain and easy to debug.\n\nBy placing declarative configuration of the whole system into YAML files\nyou can keep tasks as simple as needed. Storing results of tasks in\ndatabases, modeling dependencies or executing fallback tasks/flows on\nfailures are separated from task logic. This gives you a power to\ndynamically change task and flow dependencies on demand, optimize data\nretrieval and data storage from databases per task bases or even track\nprogress based on events traced in the system.\n\nSelinon was designed to serve millions of tasks in clusters or data\ncenters orchestrated by `Kubernetes <https://kubernetes.io>`__,\n`OpenShift <https://openshift.com>`__ or any other orchestration tool,\nbut can simplify even small systems. Moreover, Selinon can make them\neasily scalable in the future and make developer's life much easier.\n\nA Quick First Overview\n----------------------\n\nSelinon is serving recipes in a distributed environment, so let's make a\ndinner!\n\nIf we want to make a dinner, we need to buy ingredients. These\ningredients are bought in ``buyIngredientsFlow``. This flow consists of\nmultiple tasks, but let's focus on our main flow. Once all ingredients\nare bought, we can start preparing our dinner in ``prepareFlow``. Again,\nthis flow consists of some additional steps that need to be done in\norder to accomplish our future needs. As you can see, if anything goes\nwrong in mentioned flows (see red arrows), we make a fallback to pizza\nwith beer which we order. To make beer cool, we place it to our\n``Fridge`` storage. If we successfully finished ``prepareFlow`` after\nsuccessful shopping, we can proceed to ``serveDinnerFlow``.\n\nJust to point out - grey nodes represent flows (which can be made of\nother flows or tasks) and white (rounded) nodes are tasks. Conditions\nare represented in hexagons (see bellow). Black arrows represent time or\ndata dependencies between our nodes, grey arrows pinpoint where results\nof tasks are stored.\n\n.. figure:: https://raw.githubusercontent.com/selinon/selinon/master/example/graph/dinnerFlow.png\n   :alt: Main dinner flow\n   :align: center\n\nFor our dinner we need eggs, flour and some additional ingredients.\nMoreover, we conditionally buy a flower based on our condition. Our task\n``BuyFlowerTask`` will not be scheduled (or executed) if our condition\nis ``False``. Conditions are made of predicates and these predicates can\nbe grouped as desired with logical operators. You can define your own\npredicates if you want (default are available in\n``selinon.predicates``). Everything that is bought is stored in\n``Basket`` storage transparently.\n\nLet's visualise our ``buyIngredientsFlow``:\n\n.. figure:: https://raw.githubusercontent.com/selinon/selinon/master/example/graph/buyIngredientsFlow.png\n   :alt: Buy ingredients flow\n   :align: center\n\nAs stated in our main flow after buying ingredients, we proceed to\ndinner preparation but first we need to check our recipe that is hosted\nat ``http://recipes.lan/how-to-bake-pie.html``. Any ingredients we\nbought are transparently retrieved from defined storage as defined in\nour YAML configuration file. We warm up our oven to expected temperature\nand once the temperature is reached and we have finished with dough, we\ncan proceed to baking.\n\nBased on the description above, our ``prepareFlow`` will look like the\nfollowing graph:\n\n.. figure:: https://raw.githubusercontent.com/selinon/selinon/master/example/graph/prepareFlow.png\n   :alt: Prepare flow\n   :align: center\n\nOnce everything is done we serve plates. As we want to serve plates for\nall guests we need to make sure we schedule N tasks of type\n``ServePlateTask``. Each time we run our whole dinner flow, number of\nguests may vary so make sure no guest stays hungry. Our\n``serveDinnerFlow`` would look like the following graph:\n\n.. figure:: https://raw.githubusercontent.com/selinon/selinon/master/example/graph/serveDinnerFlow.png\n   :alt: Serve dinner flow\n   :align: center\n\nThis example demonstrates very simple flows. The whole configuration can\nbe found `here </example/dinner.yaml>`__. Just check it out how you can\neasily define your flows! You can find a script that visualises graphs\nbased on the YAML configuration in `this repo </example/>`__ as well.\n\nMore info\n---------\n\nThe example was intentionally simplified. You can also parametrize your\nflows, schedule N tasks (where N is a run-time evaluated variable), do\nresult caching, placing tasks on separate queues in order to be capable\nof doing fluent system updates, throttle execution of certain tasks in\ntime, propagate results of tasks to sub-flows etc. Just check\n`documentation <https://selinon.github.io/selinon>`__ for more info.\n\nLive Demo\n---------\n\nA live demo with few examples can be found\n`here <https://github.com/selinon/demo>`__. Feel free to check it out.\n\nInstallation\n------------\n\n::\n\n    $ pip3 install selinon\n\nAvailable extras:\n\n* celery - needed if you use Celery\n* mongodb - needed for MongoDB `storage adapter <http://selinon.readthedocs.io/en/latest/storage.html>`_\n* postgresql - needed for PostgreSQL `storage adapter <http://selinon.readthedocs.io/en/latest/storage.html>`_\n* redis - needed for Redis `storage adapter <http://selinon.readthedocs.io/en/latest/storage.html>`_\n* s3 - needed for S3 `storage adapter <http://selinon.readthedocs.io/en/latest/storage.html>`_\n* sentry - needed for `Sentry support <http://selinon.readthedocs.io/en/latest/trace.html#sentry-integration>`_\n\nExtras can be installed via:\n\n::\n\n    $ pip3 install selinon[celery,mongodb,postgresql,redis,s3,sentry]\n\nFeel free to select only needed extras for your setup.\n\n\n.. |codecov| image:: https://codecov.io/gh/selinon/selinon/branch/master/graph/badge.svg\n.. |PyPI Current Version| image:: https://img.shields.io/pypi/v/selinon.svg\n.. |PyPI Implementation| image:: https://img.shields.io/pypi/implementation/selinon.svg\n.. |PyPI Wheel| image:: https://img.shields.io/pypi/wheel/selinon.svg\n.. |Travis CI| image:: https://travis-ci.org/selinon/selinon.svg?branch=master\n.. |Documentation Status| image:: https://readthedocs.org/projects/selinon/badge/?version=latest\n.. |GitHub stars| image:: https://img.shields.io/github/stars/selinon/selinon.svg\n.. |GitHub license| image:: https://img.shields.io/badge/license-BSD-blue.svg\n.. |Twitter| image:: https://img.shields.io/twitter/url/http/github.com/selinon/selinon.svg?style=social\n\n",
    "bugtrack_url": null,
    "license": "BSD",
    "summary": "an advanced dynamic task flow management on top of Celery",
    "version": "1.3.0.post0",
    "split_keywords": [
        "selinon",
        "celery",
        "yaml",
        "flow",
        "distributed-computing"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "985aead623118ccd4efeb4c9b8750afdf7796d48afbd4ae971a1c35d066a6d3e",
                "md5": "a7fc3ec50d0857d369dcab32e3880b34",
                "sha256": "7fc703ed791b55cacf6440d7cc5c192bf938971d6176512fb6b202ca68a1ba99"
            },
            "downloads": -1,
            "filename": "selinon-1.3.0.post0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "a7fc3ec50d0857d369dcab32e3880b34",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 324456,
            "upload_time": "2023-01-27T13:17:57",
            "upload_time_iso_8601": "2023-01-27T13:17:57.662708Z",
            "url": "https://files.pythonhosted.org/packages/98/5a/ead623118ccd4efeb4c9b8750afdf7796d48afbd4ae971a1c35d066a6d3e/selinon-1.3.0.post0-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "320aa1fc9640ab9598800d2ed964e01e157897818efcb9fa8f1d93dd74c31e5b",
                "md5": "2c8de57dc1c887829abfb500ca0ff991",
                "sha256": "926010ccf6853800addd2dcbeb367ad39f97e8f1c3c451d850427239d6f05f53"
            },
            "downloads": -1,
            "filename": "selinon-1.3.0.post0.tar.gz",
            "has_sig": false,
            "md5_digest": "2c8de57dc1c887829abfb500ca0ff991",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 215499,
            "upload_time": "2023-01-27T13:17:59",
            "upload_time_iso_8601": "2023-01-27T13:17:59.998396Z",
            "url": "https://files.pythonhosted.org/packages/32/0a/a1fc9640ab9598800d2ed964e01e157897818efcb9fa8f1d93dd74c31e5b/selinon-1.3.0.post0.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2023-01-27 13:17:59",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "github_user": "selinon",
    "github_project": "selinon",
    "travis_ci": true,
    "coveralls": true,
    "github_actions": false,
    "requirements": [],
    "tox": true,
    "lcname": "selinon"
}

Fridolin Pokorny