soopervisor


Namesoopervisor JSON
Version 0.5.1 PyPI version JSON
download
home_pagehttps://github.com/ploomber/soopervisor
Summary
upload_time2021-07-26 13:34:24
maintainer
docs_urlNone
author
requires_python
license
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls.
            Soopervisor
===========

.. image:: https://github.com/ploomber/soopervisor/workflows/CI/badge.svg
   :target: https://github.com/ploomber/soopervisor/workflows/CI/badge.svg
   :alt: CI badge


Soopervisor runs `Ploomber <github.com/ploomber/ploomber>`_ pipelines
for batch processing (large-scale training or batch serving) or online
inference.

.. code-block:: sh

   pip install soopervisor


Check out the `documentation <https://soopervisor.readthedocs.io/>`_ to learn more.

Supported platforms
===================

* Batch serving and large-scale training:

  * Kubernetes / Argo Workflows
  * AWS Batch

* Online inference:

  * AWS Lambda

Usage
=====

Say that you want to train multiple models in a Kubernetes
cluster, you may create a new target environment to execute your pipeline
using Argo Workflows:

.. code-block:: sh

   soopervisor add training --backend argo-workflows

After filling in some basic configuration settings, export the pipeline with:

.. code-block:: sh

   soopervisor export training


Depending on the selected backend (Argo, Airflow, AWS Batch or AWS Lambda),
configuration details will change but the API remains the same:
``soopervisor add``, then ``soopervisor export``.
CHANGELOG
=========

0.5.1 (2021-07-26)
------------------
* Better error message when lock files do not exist
* Documentation note on when using shared disks (must pass ``--skip-tests``)
* Adds ``build`` as a dependency
* Check for lock files before creating ``soopervisor.yaml``
* Fixes ``docker.build`` issue that caused a ``pipeline.yaml`` to be used even when the environment required one with another name
* Fixes error that caused AWS batch args to be passed as a single str

0.5 (2021-07-09)
----------------
* load_tasks tries to initialize a spec matching the target env name (e.g., ``training`` target looks for ``pipeline.train.yaml``
* Compatibility fixes with Ploomber (requires >=0.12.1)
* Fixes an error that caused Dockerfile to include a line to install project as a package even if there was no ``setup.py`` file

0.4.2 (2021-06-04)
------------------
* Adds ``exclude`` to ignore files/directories from docker image
* Adds user guide section to documentation

0.4.1 (2021-05-31)
------------------
* Adds ``--mode`` option to ``soopervisor export``
* Batch export stops if there are no tasks to execute
* Adds ``--skip-tests`` option to skip tests before submitting

0.4 (2021-05-22)
----------------

**Important**: Soopervisor was re-written. Some modules were deprecated and the
API changed. This new architecture allows us to greatly simplify user experience
and easily incorporate more platforms in the future.

* New CLI
* New documentation
* New (simplified) ``soopervisor.yaml`` configuration schema
* Support for non-packaged projects (i.e., the ones without a ``setup.py`` file)
* Support for AWS Batch
* Support for AWS Lambda
* Argo Workflows integration builds a docker image
* Airflow integration produces a DAG with ``DockerOperator`` tasks
* Deprecates ``build`` module
* Deprecates ``script`` module
* Deprecates Box integration


0.3.4 (2021-04-18)
------------------
* Export projects compatible with `ploomber.OnlineModel` to AWS Lambda
* Allow initialization from empty `soopervisor.yaml`

0.3.3 (2021-03-07)
------------------
* Support to pass extra cli args to ``ploomber task`` (via ``args`` in ``soopervisor.yaml``) when running in Argo and Airflow

0.3.2 (2021-02-13)
------------------
* Adds ``--root`` arg to ``soopervisor export-airflow`` to select an alternative project's root
* Determines default entry point using Ploomber's API to allow automated discovery of ``pipeline.yaml`` in package layouts (e.g. ``src/package/pipeline.yaml``)


0.3.1 (2021-02-11)
------------------
* Changes to the Airflow generated DAG
* Fixes a bug when initializing configuration from projects whose root is not the current directory

0.3 (2021-01-24)
----------------
* ``env.airflow.yaml`` optional when exporting to Airflow (#17)
* Validating exported argo YAML spec
* Output argo YAML spec displays script in literal mode to make it readable
* Fixed extra whitespace in generated script
* Refactors ``ArgoMountedVolume`` to provide flexibility for different types of k8s volumes
* Adds section in the documentation to run examples using minikube
* Adds a few ``echo`` statements to generated script to provide better status feedback


0.2.2 (2020-11-21)
------------------
* Adds ability to skip dag loading during project validation
* Box uploader imported only if needed
* Exposes option to skip dag loading from the CLI


0.2.1 (2020-11-20)
------------------
* Adds Airflow DAG export
* Adds Argo/Kubernetes DAG export
* Support for uploading products to Box


0.2 (2020-10-15)
----------------
* Adds ``DockerExecutor``
* Products are saved in a folder with the name of the current commit by default
* Conda environments are created locally in a `.soopervisor/` folder
* Conda environments are cached by default
* Ability to customize arguments to ``ploomber build``

0.1 (2020-08-09)
-----------------

* First release



            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/ploomber/soopervisor",
    "name": "soopervisor",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "",
    "keywords": "",
    "author": "",
    "author_email": "",
    "download_url": "https://files.pythonhosted.org/packages/b9/3d/71a7cbed29171c8ae613948b8adf745fe0266faef32272974da649e0f815/soopervisor-0.5.1.tar.gz",
    "platform": "",
    "description": "Soopervisor\n===========\n\n.. image:: https://github.com/ploomber/soopervisor/workflows/CI/badge.svg\n   :target: https://github.com/ploomber/soopervisor/workflows/CI/badge.svg\n   :alt: CI badge\n\n\nSoopervisor runs `Ploomber <github.com/ploomber/ploomber>`_ pipelines\nfor batch processing (large-scale training or batch serving) or online\ninference.\n\n.. code-block:: sh\n\n   pip install soopervisor\n\n\nCheck out the `documentation <https://soopervisor.readthedocs.io/>`_ to learn more.\n\nSupported platforms\n===================\n\n* Batch serving and large-scale training:\n\n  * Kubernetes / Argo Workflows\n  * AWS Batch\n\n* Online inference:\n\n  * AWS Lambda\n\nUsage\n=====\n\nSay that you want to train multiple models in a Kubernetes\ncluster, you may create a new target environment to execute your pipeline\nusing Argo Workflows:\n\n.. code-block:: sh\n\n   soopervisor add training --backend argo-workflows\n\nAfter filling in some basic configuration settings, export the pipeline with:\n\n.. code-block:: sh\n\n   soopervisor export training\n\n\nDepending on the selected backend (Argo, Airflow, AWS Batch or AWS Lambda),\nconfiguration details will change but the API remains the same:\n``soopervisor add``, then ``soopervisor export``.\nCHANGELOG\n=========\n\n0.5.1 (2021-07-26)\n------------------\n* Better error message when lock files do not exist\n* Documentation note on when using shared disks (must pass ``--skip-tests``)\n* Adds ``build`` as a dependency\n* Check for lock files before creating ``soopervisor.yaml``\n* Fixes ``docker.build`` issue that caused a ``pipeline.yaml`` to be used even when the environment required one with another name\n* Fixes error that caused AWS batch args to be passed as a single str\n\n0.5 (2021-07-09)\n----------------\n* load_tasks tries to initialize a spec matching the target env name (e.g., ``training`` target looks for ``pipeline.train.yaml``\n* Compatibility fixes with Ploomber (requires >=0.12.1)\n* Fixes an error that caused Dockerfile to include a line to install project as a package even if there was no ``setup.py`` file\n\n0.4.2 (2021-06-04)\n------------------\n* Adds ``exclude`` to ignore files/directories from docker image\n* Adds user guide section to documentation\n\n0.4.1 (2021-05-31)\n------------------\n* Adds ``--mode`` option to ``soopervisor export``\n* Batch export stops if there are no tasks to execute\n* Adds ``--skip-tests`` option to skip tests before submitting\n\n0.4 (2021-05-22)\n----------------\n\n**Important**: Soopervisor was re-written. Some modules were deprecated and the\nAPI changed. This new architecture allows us to greatly simplify user experience\nand easily incorporate more platforms in the future.\n\n* New CLI\n* New documentation\n* New (simplified) ``soopervisor.yaml`` configuration schema\n* Support for non-packaged projects (i.e., the ones without a ``setup.py`` file)\n* Support for AWS Batch\n* Support for AWS Lambda\n* Argo Workflows integration builds a docker image\n* Airflow integration produces a DAG with ``DockerOperator`` tasks\n* Deprecates ``build`` module\n* Deprecates ``script`` module\n* Deprecates Box integration\n\n\n0.3.4 (2021-04-18)\n------------------\n* Export projects compatible with `ploomber.OnlineModel` to AWS Lambda\n* Allow initialization from empty `soopervisor.yaml`\n\n0.3.3 (2021-03-07)\n------------------\n* Support to pass extra cli args to ``ploomber task`` (via ``args`` in ``soopervisor.yaml``) when running in Argo and Airflow\n\n0.3.2 (2021-02-13)\n------------------\n* Adds ``--root`` arg to ``soopervisor export-airflow`` to select an alternative project's root\n* Determines default entry point using Ploomber's API to allow automated discovery of ``pipeline.yaml`` in package layouts (e.g. ``src/package/pipeline.yaml``)\n\n\n0.3.1 (2021-02-11)\n------------------\n* Changes to the Airflow generated DAG\n* Fixes a bug when initializing configuration from projects whose root is not the current directory\n\n0.3 (2021-01-24)\n----------------\n* ``env.airflow.yaml`` optional when exporting to Airflow (#17)\n* Validating exported argo YAML spec\n* Output argo YAML spec displays script in literal mode to make it readable\n* Fixed extra whitespace in generated script\n* Refactors ``ArgoMountedVolume`` to provide flexibility for different types of k8s volumes\n* Adds section in the documentation to run examples using minikube\n* Adds a few ``echo`` statements to generated script to provide better status feedback\n\n\n0.2.2 (2020-11-21)\n------------------\n* Adds ability to skip dag loading during project validation\n* Box uploader imported only if needed\n* Exposes option to skip dag loading from the CLI\n\n\n0.2.1 (2020-11-20)\n------------------\n* Adds Airflow DAG export\n* Adds Argo/Kubernetes DAG export\n* Support for uploading products to Box\n\n\n0.2 (2020-10-15)\n----------------\n* Adds ``DockerExecutor``\n* Products are saved in a folder with the name of the current commit by default\n* Conda environments are created locally in a `.soopervisor/` folder\n* Conda environments are cached by default\n* Ability to customize arguments to ``ploomber build``\n\n0.1 (2020-08-09)\n-----------------\n\n* First release\n\n\n",
    "bugtrack_url": null,
    "license": "",
    "summary": "",
    "version": "0.5.1",
    "split_keywords": [],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "b1bb66f881e3316317aae2a68a332a7d",
                "sha256": "6838b5fd8fbf90055d82fac22f638275f94c066fee39a7ce8475fe19de5407d3"
            },
            "downloads": -1,
            "filename": "soopervisor-0.5.1-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "b1bb66f881e3316317aae2a68a332a7d",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 31170,
            "upload_time": "2021-07-26T13:34:23",
            "upload_time_iso_8601": "2021-07-26T13:34:23.157519Z",
            "url": "https://files.pythonhosted.org/packages/e7/ad/82a7cbd7905aa3b958514fd010025304bd4fa25d6dbd61f9e1bc239e2cb5/soopervisor-0.5.1-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "7beb73fc1366f1a34c68a70d5dfac7ae",
                "sha256": "0a46d2ff5b642a5e38377b53dac4c2b7bc0355a5048ab639a461f6094f681c89"
            },
            "downloads": -1,
            "filename": "soopervisor-0.5.1.tar.gz",
            "has_sig": false,
            "md5_digest": "7beb73fc1366f1a34c68a70d5dfac7ae",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 25687,
            "upload_time": "2021-07-26T13:34:24",
            "upload_time_iso_8601": "2021-07-26T13:34:24.466461Z",
            "url": "https://files.pythonhosted.org/packages/b9/3d/71a7cbed29171c8ae613948b8adf745fe0266faef32272974da649e0f815/soopervisor-0.5.1.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2021-07-26 13:34:24",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "github_user": "ploomber",
    "github_project": "soopervisor",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "lcname": "soopervisor"
}
        
Elapsed time: 0.29128s