cls-luigi

Name	cls-luigi JSON
Version	0.1.1 JSON
	download
home_page	https://github.com/cls-python/cls-luigi
Summary	CLS-Luigi is an innovative pipeline tool designed to streamline the creation and execution of algorithmic pipelines by harnessing the power of combinatory logic.
upload_time	2024-02-29 13:15:55
maintainer	Daniel Scholtyssek
docs_url	None
author	Jan Bessai
requires_python	>=3.10
license	Apache License (2.0)
keywords	cls-luigi
VCS
bugtrack_url
requirements	No requirements were recorded.
Travis-CI	No Travis.
coveralls test coverage	No coveralls.

CLS-Luigi
=========
.. .. image:: https://raw.githubusercontent.com/cls-python/cls-luigi/master/doc/_static/images/logo.png
.. :target: https://github.com/cls-python/cls-luigi/
.. :width: 90%
.. :align: center

.. image:: https://img.shields.io/pypi/v/cls-luigi
:target: https://pypi.python.org/pypi/cls-luigi

.. image:: https://img.shields.io/pypi/pyversions/cls-luigi
:target: https://pypi.python.org/pypi/cls-luigi

.. image:: https://img.shields.io/pypi/l/cls-luigi?color=blue
:target: https://github.com/cls-python/cls-luigi/blob/main/LICENSE

.. image:: https://img.shields.io/github/issues/cls-python/cls-luigi
:target: https://github.com/cls-python/cls-luigi/issues

.. image:: https://github.com/cls-python/cls-luigi/actions/workflows/test-build-release.yaml/badge.svg
:target: https://github.com/cls-python/cls-luigi/actions/workflows/test-build-release.yaml

.. image:: https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/DanielScholtyssek/d2c0e9041ce417fd42c710572e02c2aa/raw/covbadge.json
:target: https://github.com/cls-python/cls-luigi/actions/workflows/test-build-release.yaml

.. image:: https://img.shields.io/badge/docs-online-green
:target: https://cls-python.github.io/cls-luigi
:alt: Documentation Status
..
.. image:: https://pyup.io/repos/github/cls-python/cls-luigi/shield.svg
:target: https://pyup.io/repos/github/cls-python/cls-luigi/
:alt: Updates

**************

TL:DR
-----

A framework for automated synthesis and execution of decision pipelines.

* Free software: Apache Software License 2.0
* Documentation: https://cls-python.github.io/cls-luigi/

What is it?
-----------

**In short:** our goal is to automatically create decision pipelines based on domain specific algorithmic repositories and depending on the available data!

In order to streamline decision-making processes, it is common practice to construct decision pipelines comprising various algorithms. These pipelines encompass tasks such as basic data pre-processing, statistical analysis, machine learning, and optimization. Consequently, building an effective pipeline necessitates expertise in the domains of machine learning, optimization, and the specific field at hand.

Moreover, it is important to note that there is no universally optimal pipeline for every problem in any domain. There is no single pipeline that represents the best choice, nor is there an ultimate model or configuration for machine learning approches. Likewise, the ideal model and solution method for optimization problems may vary. One of the main reasons for this can be the available data, which can change over time.

**CLS-Luigi** is an innovative pipeline tool designed to streamline the creation and execution of algorithmic pipelines by harnessing the power of combinatory logic.
It combines CLS-Python_, a type-theoretic framework for software component synthesis, with Luigi_, a tool created by Spotify to build and execute pipelines for batch jobs. At its core, **CLS-Luigi** aims to integrate elements from *AutoML*, *Algorithm selection and configuration*, and *DevOps* to provide a comprehensive solution. However, it is important to note that **CLS-Luigi** is currently in its early stages of development, undergoing ongoing refinement and enhancement.

Main Features
-------------

Here are just a few of the things that **CLS-Luigi** does well:

- Allows natural modeling by specifying the necessary input to a component based on the types of the components it depends on.
- It's easy to define templates for algorithmic components and templates for pipeline structures.
- Good for batch-type pipelines where previous pipeline executions have no/little influence on the current run.
- Consistent creation of (all) pipeline variants based on a user defined repository of components, guaranteeing soundness and completeness by leveraging the CLS-Python_ framework.
- Efficient execution and resource optimization using features of Luigi_. Luigi optimizes resource usage through caching mechanisms, avoiding redundant computations by rerunning identical sub-pipelines only when necessary.
- The framework places significant importance on componentization and promotes a structured approach that considers the data flow, actions on the data, and data propagation to subsequent computational steps when implementing components. This enables the reuse of components in different domain-specific repositories and their smooth integration into diverse pipelines, as long as they align with the pipeline structure. Moreover, the framework offers flexibility to Python programmers, allowing them to expand existing domain-specific repositories by inheriting from a already existing component and implementing the necessary methods to create a new problem-specific or problem-agnostic component, thereby fostering customization and adaptability.
- Integrated visualizer that can display the repository and the scheduled pipelines.
- Python based framework: offers a extensive collection of specialized libraries and tools that offer pre-built algorithms, statistical functions, and visualization capabilities tailored for optimization, machine learning, and data analytics tasks. Moreover, Python's seamless integration with other technologies enhances its appeal and makes it a valuable tool for our projects.

Authors
-------

* Jan Bessai <jan.bessai@tu-dortmund.de>
* Anne Meyer <anne2.meyer@tu-dortmund.de>
* Hadi Kutabi <hadi.kutabi@tu-dortmund.de>
* Daniel Scholtyssek <daniel.scholtyssek@tu-dortmund.de>

.. _CLS-Python: https://github.com/cls-python/cls-python
.. _Luigi: https://github.com/spotify/luigi

Changelog
=========

0.1.1 (2024-02-29)
------------------

* Commented TaskProcessingTime mixin to supress error messages.
* Fixing wrapped_task arg in the decoder.

0.1.0 (2023-06-07)
------------------

* First release on PyPI.

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/cls-python/cls-luigi",
    "name": "cls-luigi",
    "maintainer": "Daniel Scholtyssek",
    "docs_url": null,
    "requires_python": ">=3.10",
    "maintainer_email": "daniel.scholtyssek@tu-dortmund.de",
    "keywords": "cls-luigi",
    "author": "Jan Bessai",
    "author_email": "jan.bessai@tu-dortmund.de",
    "download_url": "https://files.pythonhosted.org/packages/21/f8/3609620254f1fab0a1fc1e9ad9a246d6543ef259e67690a41e5e6fc66084/cls-luigi-0.1.1.tar.gz",
    "platform": null,
    "description": "CLS-Luigi\n=========\n.. .. image:: https://raw.githubusercontent.com/cls-python/cls-luigi/master/doc/_static/images/logo.png\n..   :target: https://github.com/cls-python/cls-luigi/\n..   :width: 90%\n..   :align: center\n\n.. image:: https://img.shields.io/pypi/v/cls-luigi\n        :target: https://pypi.python.org/pypi/cls-luigi\n\n.. image:: https://img.shields.io/pypi/pyversions/cls-luigi\n        :target: https://pypi.python.org/pypi/cls-luigi\n\n.. image:: https://img.shields.io/pypi/l/cls-luigi?color=blue\n        :target: https://github.com/cls-python/cls-luigi/blob/main/LICENSE\n\n.. image:: https://img.shields.io/github/issues/cls-python/cls-luigi\n        :target: https://github.com/cls-python/cls-luigi/issues\n\n.. image:: https://github.com/cls-python/cls-luigi/actions/workflows/test-build-release.yaml/badge.svg\n        :target: https://github.com/cls-python/cls-luigi/actions/workflows/test-build-release.yaml\n\n.. image:: https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/DanielScholtyssek/d2c0e9041ce417fd42c710572e02c2aa/raw/covbadge.json\n        :target: https://github.com/cls-python/cls-luigi/actions/workflows/test-build-release.yaml\n\n.. image:: https://img.shields.io/badge/docs-online-green\n        :target: https://cls-python.github.io/cls-luigi\n        :alt: Documentation Status\n..\n  .. image:: https://pyup.io/repos/github/cls-python/cls-luigi/shield.svg\n     :target: https://pyup.io/repos/github/cls-python/cls-luigi/\n     :alt: Updates\n\n**************\n\nTL:DR\n-----\n\nA framework for automated synthesis and execution of decision pipelines.\n\n* Free software: Apache Software License 2.0\n* Documentation: https://cls-python.github.io/cls-luigi/\n\nWhat is it?\n-----------\n\n**In short:** our goal is to automatically create decision pipelines based on domain specific algorithmic repositories and depending on the available data!\n\nIn order to streamline decision-making processes, it is common practice to construct decision pipelines comprising various algorithms. These pipelines encompass tasks such as basic data pre-processing, statistical analysis, machine learning, and optimization. Consequently, building an effective pipeline necessitates expertise in the domains of machine learning, optimization, and the specific field at hand.\n\nMoreover, it is important to note that there is no universally optimal pipeline for every problem in any domain. There is no single pipeline that represents the best choice, nor is there an ultimate model or configuration for machine learning approches. Likewise, the ideal model and solution method for optimization problems may vary. One of the main reasons for this can be the available data, which can change over time.\n\n**CLS-Luigi** is an innovative pipeline tool designed to streamline the creation and execution of algorithmic pipelines by harnessing the power of combinatory logic.\nIt combines CLS-Python_, a type-theoretic framework for software component synthesis,  with Luigi_, a tool created by Spotify to build and execute pipelines for batch jobs. At its core, **CLS-Luigi** aims to integrate elements from *AutoML*, *Algorithm selection and configuration*, and *DevOps* to provide a comprehensive solution. However, it is important to note that **CLS-Luigi** is currently in its early stages of development, undergoing ongoing refinement and enhancement.\n\n\nMain Features\n-------------\n\nHere are just a few of the things that **CLS-Luigi** does well:\n\n- Allows natural modeling by specifying the necessary input to a component based on the types of the components it depends on.\n- It's easy to define templates for algorithmic components and templates for pipeline structures.\n- Good for batch-type pipelines where previous pipeline executions have no/little influence on the current run.\n- Consistent creation of (all) pipeline variants based on a user defined repository of components, guaranteeing soundness and completeness by leveraging the CLS-Python_ framework.\n- Efficient execution and resource optimization using features of Luigi_. Luigi optimizes resource usage through caching mechanisms, avoiding redundant computations by rerunning identical sub-pipelines only when necessary.\n- The framework places significant importance on componentization and promotes a structured approach that considers the data flow, actions on the data, and data propagation to subsequent computational steps when implementing components. This enables the reuse of components in different domain-specific repositories and their smooth integration into diverse pipelines, as long as they align with the pipeline structure. Moreover, the framework offers flexibility to Python programmers, allowing them to expand existing domain-specific repositories by inheriting from a already existing component and implementing the necessary methods to create a new problem-specific or problem-agnostic component, thereby fostering customization and adaptability.\n- Integrated visualizer that can display the repository and the scheduled pipelines.\n- Python based framework: offers a extensive collection of specialized libraries and tools that offer pre-built algorithms, statistical functions, and visualization capabilities tailored for optimization, machine learning, and data analytics tasks. Moreover, Python's seamless integration with other technologies enhances its appeal and makes it a valuable tool for our projects.\n\nAuthors\n-------\n\n* Jan Bessai <jan.bessai@tu-dortmund.de>\n* Anne Meyer <anne2.meyer@tu-dortmund.de>\n* Hadi Kutabi <hadi.kutabi@tu-dortmund.de>\n* Daniel Scholtyssek <daniel.scholtyssek@tu-dortmund.de>\n\n\n.. _CLS-Python: https://github.com/cls-python/cls-python\n.. _Luigi: https://github.com/spotify/luigi\n\n\nChangelog\n=========\n\n0.1.1 (2024-02-29)\n------------------\n\n* Commented TaskProcessingTime mixin to supress error messages.\n* Fixing wrapped_task arg in the decoder.\n\n0.1.0 (2023-06-07)\n------------------\n\n* First release on PyPI.\n\n",
    "bugtrack_url": null,
    "license": "Apache License (2.0)",
    "summary": "CLS-Luigi is an innovative pipeline tool designed to streamline the creation and execution of algorithmic pipelines by harnessing the power of combinatory logic.",
    "version": "0.1.1",
    "project_urls": {
        "Bug Tracker": "https://github.com/cls-python/cls-luigi/issues",
        "Documentation": "https://cls-python.github.io/cls-luigi",
        "Homepage": "https://github.com/cls-python/cls-luigi",
        "Source Code": "https://github.com/cls-python/cls-luigi"
    },
    "split_keywords": [
        "cls-luigi"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "00f75bebb72740107298e248b605b9923ad892de6687324f5f1b4ccf981e813e",
                "md5": "417b30d9bcdc5e56417573608d5bda78",
                "sha256": "910e3df541f4d2f7e4990c3c9561d8e4cd4ffee3ff345829d43e98f31883903a"
            },
            "downloads": -1,
            "filename": "cls_luigi-0.1.1-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "417b30d9bcdc5e56417573608d5bda78",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.10",
            "size": 359466,
            "upload_time": "2024-02-29T13:15:53",
            "upload_time_iso_8601": "2024-02-29T13:15:53.589949Z",
            "url": "https://files.pythonhosted.org/packages/00/f7/5bebb72740107298e248b605b9923ad892de6687324f5f1b4ccf981e813e/cls_luigi-0.1.1-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "21f83609620254f1fab0a1fc1e9ad9a246d6543ef259e67690a41e5e6fc66084",
                "md5": "369b845b05b35f096580bc5f531f4a0e",
                "sha256": "b215360455bfe35dd232b841ca828e44131af3f7040227e7d08cd127735add14"
            },
            "downloads": -1,
            "filename": "cls-luigi-0.1.1.tar.gz",
            "has_sig": false,
            "md5_digest": "369b845b05b35f096580bc5f531f4a0e",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.10",
            "size": 354698,
            "upload_time": "2024-02-29T13:15:55",
            "upload_time_iso_8601": "2024-02-29T13:15:55.026590Z",
            "url": "https://files.pythonhosted.org/packages/21/f8/3609620254f1fab0a1fc1e9ad9a246d6543ef259e67690a41e5e6fc66084/cls-luigi-0.1.1.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-02-29 13:15:55",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "cls-python",
    "github_project": "cls-luigi",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "tox": true,
    "lcname": "cls-luigi"
}

Jan Bessai