Soopervisor
===========
.. image:: https://github.com/ploomber/soopervisor/workflows/CI/badge.svg
:target: https://github.com/ploomber/soopervisor/workflows/CI/badge.svg
:alt: CI badge
.. image:: https://github.com/ploomber/soopervisor/workflows/CI%20macOS/badge.svg
:target: https://github.com/ploomber/soopervisor/workflows/CI%20macOS/badge.svg
:alt: CI macOS badge
.. image:: https://github.com/ploomber/soopervisor/workflows/CI%20Windows/badge.svg
:target: https://github.com/ploomber/soopervisor/workflows/CI%20Windows/badge.svg
:alt: CI Windows badge
.. image:: https://img.shields.io/badge/code%20style-black-000000.svg
:target: https://github.com/psf/black
Soopervisor runs `Ploomber <https://github.com/ploomber/ploomber>`_ pipelines
for batch processing (large-scale training or batch serving) or online
inference.
.. code-block:: sh
pip install soopervisor
Check out the `documentation <https://soopervisor.readthedocs.io/>`_ to learn more.
*Compatible with Python 3.7 and higher.*
Supported platforms
===================
* Batch serving and large-scale training:
* `Airflow <https://soopervisor.readthedocs.io/en/latest/tutorials/airflow.html>`_
* `Argo/Kubernetes <https://soopervisor.readthedocs.io/en/latest/tutorials/kubernetes.html>`_
* `AWS Batch <https://soopervisor.readthedocs.io/en/latest/tutorials/aws-batch.html>`_
* `Kubeflow <https://soopervisor.readthedocs.io/en/latest/tutorials/kubeflow.html>`_
* `SLURM <https://soopervisor.readthedocs.io/en/latest/tutorials/slurm.html>`_
* Online inference:
* `AWS Lambda <https://soopervisor.readthedocs.io/en/latest/tutorials/aws-lambda.html>`_
From notebook to a production pipeline
======================================
We also have `an example <https://soopervisor.readthedocs.io/en/latest/tutorials/workflow.html>`_ that shows how to use our ecosystem of tools to
go **from a monolithic notebook to a pipeline deployed in Kubernetes.**
Usage
=====
Say that you want to train multiple models in a Kubernetes
cluster, you may create a new target environment to execute your pipeline
using Argo Workflows:
.. code-block:: sh
soopervisor add training --backend argo-workflows
After filling in some basic configuration settings, export the pipeline with:
.. code-block:: sh
soopervisor export training
Depending on the selected backend (Argo, Airflow, AWS Batch, or AWS Lambda),
configuration details will change, but the API remains the same:
``soopervisor add``, then ``soopervisor export``.
About Ploomber
==============
Ploomber is a big community of data enthusiasts pushing the boundaries of Data Science and Machine Learning tooling.
Whatever your skillset is, you can contribute to our mission. So whether you're a beginner or an experienced professional, you're welcome to join us on this journey!
`Click here to know how you can contribute to Ploomber. <https://github.com/ploomber/contributing/blob/main/README.md>`_
CHANGELOG
=========
0.9.2 (2023-07-18)
------------------
* Replaces `@requires` import to use `ploomber_core` instead of `ploomber`
0.9.1 (2022-10-24)
------------------
* Adds `--skip-docker` argument in `soopervisor export` to skip docker build (#103)
* Optimizes the generated AWS Batch ``Dockerfile`` so dependencies are only installed when the requirements are modified
* Allows using multiple ``requirements.txt`` files, generating one Docker image for each one (#86)
0.9 (2022-10-03)
----------------
* Allows execution of single tasks via ``soopervisor export --task {task-name}``
* Allowing to bundle custom libraries via ``lib/`` in docker image (#87)
* Fixes DAG loading when passing the ``--lazy`` argument in pipelines with a ``File`` client (#105)
* Display warning if source code contains file over 10MB (#81)
* Drop support for Python 3.6
0.8 (2022-06-09)
----------------
* Dropping support for Python 3.6
* Adds ``task_resources`` to AWS Batch
* AWS Batch exporter now generates a unique job definition name on each submission
* Adds ``--lazy`` option to ``soopervisor export``
* Pass custom arguments to ``docker build`` with environment variable ``DOCKER_ARGS``
0.7.2 (2022-02-14)
------------------
* Fixes an error that caused ``soopervisor export`` to fail due to non-serializable object
0.7.1 (2022-02-13)
------------------
**Important:** We detected a problem with this release when running ``soopervisor export``. Please use ``0.7.2``
* Fixes error that caused docker image building to fail if the repository didn't have a version
0.7 (2022-01-31)
----------------
**Important:** We detected a problem with this release when running ``soopervisor export``. Please use ``0.7.2``
* Improves CLI documentation
* Adds ``--git-ignore`` to documentation
* Various documentation improvements
* Changes short version of ``--until-build`` to ``-u``
* Adds "Task Communication" user guide
* Display warnings if passing CLI options that do not apply to SLURM
* SLURM exporter raises error if ``sbatch`` isn't installed
* Showing a warning if source dist is >5MB (#53)
* ``soopervisor add`` adds a default ``exclude`` value by extracting product paths from ``pipeline.yaml``
* Copying user settings when generating the Docker image
* Experimental Kubeflow integration
* Airflow integration allows to choose between ``BashOperator``, ``KubernetesPodOperator``, and ``DockerOperator`` using ``--preset``
* Many modules and test cases re-written for better code quality and maintainability
0.6.1 (2022-01-16)
------------------
* Fixes output message after exporting to Argo
* Adds flag to ``source.copy`` to ignore git
* ``soopervisor export`` raises and error if ``git`` isn't tracking any files
* Adds ``--git-ignore`` to ``soopervisor export``
0.6 (2022-01-04)
----------------
* Adds support for SLURM
* ``AirflowExporter`` uses ``KubernetesPodOperator`` by default (#33)
* Simplified Airflow and Argo/k8s tutorials
0.5.2 (2022-01-02)
------------------
* Clearer error message when pending ``git commit``
* Clearer error message when the user does not change docker repository default value (#29)
* Argo spec sets ``imagePullPolicy`` to ``Never`` if repository is ``null``
* Documents Kubernetes/Argo configuration schema
* General documentation improvements
0.5.1 (2021-07-26)
------------------
* Better error message when lock files do not exist
* Documentation note on when using shared disks (must pass ``--skip-tests``)
* Adds ``build`` as a dependency
* Check for lock files before creating ``soopervisor.yaml``
* Fixes ``docker.build`` issue that caused a ``pipeline.yaml`` to be used even when the environment required one with another name
* Fixes error that caused AWS batch args to be passed as a single str
0.5 (2021-07-09)
----------------
* load_tasks tries to initialize a spec matching the target env name (e.g., ``training`` target looks for ``pipeline.train.yaml``
* Compatibility fixes with Ploomber (requires >=0.12.1)
* Fixes an error that caused Dockerfile to include a line to install project as a package even if there was no ``setup.py`` file
0.4.2 (2021-06-04)
------------------
* Adds ``exclude`` to ignore files/directories from docker image
* Adds user guide section to documentation
0.4.1 (2021-05-31)
------------------
* Adds ``--mode`` option to ``soopervisor export``
* Batch export stops if there are no tasks to execute
* Adds ``--skip-tests`` option to skip tests before submitting
0.4 (2021-05-22)
----------------
**Important**: Soopervisor was re-written. Some modules were deprecated and the
API changed. This new architecture allows us to greatly simplify user experience
and easily incorporate more platforms in the future.
* New CLI
* New documentation
* New (simplified) ``soopervisor.yaml`` configuration schema
* Support for non-packaged projects (i.e., the ones without a ``setup.py`` file)
* Support for AWS Batch
* Support for AWS Lambda
* Argo Workflows integration builds a docker image
* Airflow integration produces a DAG with ``DockerOperator`` tasks
* Deprecates ``build`` module
* Deprecates ``script`` module
* Deprecates Box integration
0.3.4 (2021-04-18)
------------------
* Export projects compatible with `ploomber.OnlineModel` to AWS Lambda
* Allow initialization from empty `soopervisor.yaml`
0.3.3 (2021-03-07)
------------------
* Support to pass extra cli args to ``ploomber task`` (via ``args`` in ``soopervisor.yaml``) when running in Argo and Airflow
0.3.2 (2021-02-13)
------------------
* Adds ``--root`` arg to ``soopervisor export-airflow`` to select an alternative project's root
* Determines default entry point using Ploomber's API to allow automated discovery of ``pipeline.yaml`` in package layouts (e.g. ``src/package/pipeline.yaml``)
0.3.1 (2021-02-11)
------------------
* Changes to the Airflow generated DAG
* Fixes a bug when initializing configuration from projects whose root is not the current directory
0.3 (2021-01-24)
----------------
* ``env.airflow.yaml`` optional when exporting to Airflow (#17)
* Validating exported argo YAML spec
* Output argo YAML spec displays script in literal mode to make it readable
* Fixed extra whitespace in generated script
* Refactors ``ArgoMountedVolume`` to provide flexibility for different types of k8s volumes
* Adds section in the documentation to run examples using minikube
* Adds a few ``echo`` statements to generated script to provide better status feedback
0.2.2 (2020-11-21)
------------------
* Adds ability to skip dag loading during project validation
* Box uploader imported only if needed
* Exposes option to skip dag loading from the CLI
0.2.1 (2020-11-20)
------------------
* Adds Airflow DAG export
* Adds Argo/Kubernetes DAG export
* Support for uploading products to Box
0.2 (2020-10-15)
----------------
* Adds ``DockerExecutor``
* Products are saved in a folder with the name of the current commit by default
* Conda environments are created locally in a `.soopervisor/` folder
* Conda environments are cached by default
* Ability to customize arguments to ``ploomber build``
0.1 (2020-08-09)
-----------------
* First release
Raw data
{
"_id": null,
"home_page": "https://github.com/ploomber/soopervisor",
"name": "soopervisor",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/3d/82/444f37dbfc21dc001ae16ae4c75d219f385ed111fbfc2f417f227b049174/soopervisor-0.9.2.tar.gz",
"platform": null,
"description": "Soopervisor\n===========\n\n.. image:: https://github.com/ploomber/soopervisor/workflows/CI/badge.svg\n :target: https://github.com/ploomber/soopervisor/workflows/CI/badge.svg\n :alt: CI badge\n\n.. image:: https://github.com/ploomber/soopervisor/workflows/CI%20macOS/badge.svg\n :target: https://github.com/ploomber/soopervisor/workflows/CI%20macOS/badge.svg\n :alt: CI macOS badge\n\n.. image:: https://github.com/ploomber/soopervisor/workflows/CI%20Windows/badge.svg\n :target: https://github.com/ploomber/soopervisor/workflows/CI%20Windows/badge.svg\n :alt: CI Windows badge\n\n.. image:: https://img.shields.io/badge/code%20style-black-000000.svg\n :target: https://github.com/psf/black\n\nSoopervisor runs `Ploomber <https://github.com/ploomber/ploomber>`_ pipelines\nfor batch processing (large-scale training or batch serving) or online\ninference.\n\n.. code-block:: sh\n\n pip install soopervisor\n\n\nCheck out the `documentation <https://soopervisor.readthedocs.io/>`_ to learn more.\n\n*Compatible with Python 3.7 and higher.*\n\nSupported platforms\n===================\n\n* Batch serving and large-scale training:\n\n * `Airflow <https://soopervisor.readthedocs.io/en/latest/tutorials/airflow.html>`_\n * `Argo/Kubernetes <https://soopervisor.readthedocs.io/en/latest/tutorials/kubernetes.html>`_\n * `AWS Batch <https://soopervisor.readthedocs.io/en/latest/tutorials/aws-batch.html>`_\n * `Kubeflow <https://soopervisor.readthedocs.io/en/latest/tutorials/kubeflow.html>`_\n * `SLURM <https://soopervisor.readthedocs.io/en/latest/tutorials/slurm.html>`_\n\n* Online inference:\n\n * `AWS Lambda <https://soopervisor.readthedocs.io/en/latest/tutorials/aws-lambda.html>`_\n\n\nFrom notebook to a production pipeline\n======================================\n\nWe also have `an example <https://soopervisor.readthedocs.io/en/latest/tutorials/workflow.html>`_ that shows how to use our ecosystem of tools to\ngo **from a monolithic notebook to a pipeline deployed in Kubernetes.**\n\nUsage\n=====\n\nSay that you want to train multiple models in a Kubernetes\ncluster, you may create a new target environment to execute your pipeline\nusing Argo Workflows:\n\n.. code-block:: sh\n\n soopervisor add training --backend argo-workflows\n\nAfter filling in some basic configuration settings, export the pipeline with:\n\n.. code-block:: sh\n\n soopervisor export training\n\n\nDepending on the selected backend (Argo, Airflow, AWS Batch, or AWS Lambda),\nconfiguration details will change, but the API remains the same:\n``soopervisor add``, then ``soopervisor export``.\n\n\nAbout Ploomber\n==============\n\nPloomber is a big community of data enthusiasts pushing the boundaries of Data Science and Machine Learning tooling.\n\nWhatever your skillset is, you can contribute to our mission. So whether you're a beginner or an experienced professional, you're welcome to join us on this journey!\n\n`Click here to know how you can contribute to Ploomber. <https://github.com/ploomber/contributing/blob/main/README.md>`_\n\nCHANGELOG\n=========\n\n0.9.2 (2023-07-18)\n------------------\n* Replaces `@requires` import to use `ploomber_core` instead of `ploomber`\n\n0.9.1 (2022-10-24)\n------------------\n* Adds `--skip-docker` argument in `soopervisor export` to skip docker build (#103)\n* Optimizes the generated AWS Batch ``Dockerfile`` so dependencies are only installed when the requirements are modified\n* Allows using multiple ``requirements.txt`` files, generating one Docker image for each one (#86)\n\n0.9 (2022-10-03)\n----------------\n* Allows execution of single tasks via ``soopervisor export --task {task-name}``\n* Allowing to bundle custom libraries via ``lib/`` in docker image (#87)\n* Fixes DAG loading when passing the ``--lazy`` argument in pipelines with a ``File`` client (#105)\n* Display warning if source code contains file over 10MB (#81)\n* Drop support for Python 3.6\n\n0.8 (2022-06-09)\n----------------\n* Dropping support for Python 3.6\n* Adds ``task_resources`` to AWS Batch\n* AWS Batch exporter now generates a unique job definition name on each submission\n* Adds ``--lazy`` option to ``soopervisor export``\n* Pass custom arguments to ``docker build`` with environment variable ``DOCKER_ARGS``\n\n0.7.2 (2022-02-14)\n------------------\n* Fixes an error that caused ``soopervisor export`` to fail due to non-serializable object\n\n0.7.1 (2022-02-13)\n------------------\n**Important:** We detected a problem with this release when running ``soopervisor export``. Please use ``0.7.2``\n\n* Fixes error that caused docker image building to fail if the repository didn't have a version\n\n0.7 (2022-01-31)\n----------------\n\n**Important:** We detected a problem with this release when running ``soopervisor export``. Please use ``0.7.2``\n\n* Improves CLI documentation\n* Adds ``--git-ignore`` to documentation\n* Various documentation improvements\n* Changes short version of ``--until-build`` to ``-u``\n* Adds \"Task Communication\" user guide\n* Display warnings if passing CLI options that do not apply to SLURM\n* SLURM exporter raises error if ``sbatch`` isn't installed\n* Showing a warning if source dist is >5MB (#53)\n* ``soopervisor add`` adds a default ``exclude`` value by extracting product paths from ``pipeline.yaml``\n* Copying user settings when generating the Docker image\n* Experimental Kubeflow integration\n* Airflow integration allows to choose between ``BashOperator``, ``KubernetesPodOperator``, and ``DockerOperator`` using ``--preset``\n* Many modules and test cases re-written for better code quality and maintainability\n\n0.6.1 (2022-01-16)\n------------------\n* Fixes output message after exporting to Argo\n* Adds flag to ``source.copy`` to ignore git\n* ``soopervisor export`` raises and error if ``git`` isn't tracking any files\n* Adds ``--git-ignore`` to ``soopervisor export``\n\n0.6 (2022-01-04)\n----------------\n* Adds support for SLURM\n* ``AirflowExporter`` uses ``KubernetesPodOperator`` by default (#33)\n* Simplified Airflow and Argo/k8s tutorials\n\n0.5.2 (2022-01-02)\n------------------\n* Clearer error message when pending ``git commit``\n* Clearer error message when the user does not change docker repository default value (#29)\n* Argo spec sets ``imagePullPolicy`` to ``Never`` if repository is ``null``\n* Documents Kubernetes/Argo configuration schema\n* General documentation improvements\n\n0.5.1 (2021-07-26)\n------------------\n* Better error message when lock files do not exist\n* Documentation note on when using shared disks (must pass ``--skip-tests``)\n* Adds ``build`` as a dependency\n* Check for lock files before creating ``soopervisor.yaml``\n* Fixes ``docker.build`` issue that caused a ``pipeline.yaml`` to be used even when the environment required one with another name\n* Fixes error that caused AWS batch args to be passed as a single str\n\n0.5 (2021-07-09)\n----------------\n* load_tasks tries to initialize a spec matching the target env name (e.g., ``training`` target looks for ``pipeline.train.yaml``\n* Compatibility fixes with Ploomber (requires >=0.12.1)\n* Fixes an error that caused Dockerfile to include a line to install project as a package even if there was no ``setup.py`` file\n\n0.4.2 (2021-06-04)\n------------------\n* Adds ``exclude`` to ignore files/directories from docker image\n* Adds user guide section to documentation\n\n0.4.1 (2021-05-31)\n------------------\n* Adds ``--mode`` option to ``soopervisor export``\n* Batch export stops if there are no tasks to execute\n* Adds ``--skip-tests`` option to skip tests before submitting\n\n0.4 (2021-05-22)\n----------------\n\n**Important**: Soopervisor was re-written. Some modules were deprecated and the\nAPI changed. This new architecture allows us to greatly simplify user experience\nand easily incorporate more platforms in the future.\n\n* New CLI\n* New documentation\n* New (simplified) ``soopervisor.yaml`` configuration schema\n* Support for non-packaged projects (i.e., the ones without a ``setup.py`` file)\n* Support for AWS Batch\n* Support for AWS Lambda\n* Argo Workflows integration builds a docker image\n* Airflow integration produces a DAG with ``DockerOperator`` tasks\n* Deprecates ``build`` module\n* Deprecates ``script`` module\n* Deprecates Box integration\n\n\n0.3.4 (2021-04-18)\n------------------\n* Export projects compatible with `ploomber.OnlineModel` to AWS Lambda\n* Allow initialization from empty `soopervisor.yaml`\n\n0.3.3 (2021-03-07)\n------------------\n* Support to pass extra cli args to ``ploomber task`` (via ``args`` in ``soopervisor.yaml``) when running in Argo and Airflow\n\n0.3.2 (2021-02-13)\n------------------\n* Adds ``--root`` arg to ``soopervisor export-airflow`` to select an alternative project's root\n* Determines default entry point using Ploomber's API to allow automated discovery of ``pipeline.yaml`` in package layouts (e.g. ``src/package/pipeline.yaml``)\n\n\n0.3.1 (2021-02-11)\n------------------\n* Changes to the Airflow generated DAG\n* Fixes a bug when initializing configuration from projects whose root is not the current directory\n\n0.3 (2021-01-24)\n----------------\n* ``env.airflow.yaml`` optional when exporting to Airflow (#17)\n* Validating exported argo YAML spec\n* Output argo YAML spec displays script in literal mode to make it readable\n* Fixed extra whitespace in generated script\n* Refactors ``ArgoMountedVolume`` to provide flexibility for different types of k8s volumes\n* Adds section in the documentation to run examples using minikube\n* Adds a few ``echo`` statements to generated script to provide better status feedback\n\n\n0.2.2 (2020-11-21)\n------------------\n* Adds ability to skip dag loading during project validation\n* Box uploader imported only if needed\n* Exposes option to skip dag loading from the CLI\n\n\n0.2.1 (2020-11-20)\n------------------\n* Adds Airflow DAG export\n* Adds Argo/Kubernetes DAG export\n* Support for uploading products to Box\n\n\n0.2 (2020-10-15)\n----------------\n* Adds ``DockerExecutor``\n* Products are saved in a folder with the name of the current commit by default\n* Conda environments are created locally in a `.soopervisor/` folder\n* Conda environments are cached by default\n* Ability to customize arguments to ``ploomber build``\n\n0.1 (2020-08-09)\n-----------------\n\n* First release\n",
"bugtrack_url": null,
"license": "",
"summary": "",
"version": "0.9.2",
"project_urls": {
"Homepage": "https://github.com/ploomber/soopervisor"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b06208a19fac1e07d333943369c9f13e749dcf6c82feb32aa3e72d35ec44c5dc",
"md5": "43c2d97be5e20882b34934cbd51421c4",
"sha256": "e724b58976ab75921c44a5b5eeab83cd5bf09730b647742690e904338107e4a3"
},
"downloads": -1,
"filename": "soopervisor-0.9.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "43c2d97be5e20882b34934cbd51421c4",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 56512,
"upload_time": "2023-07-18T17:46:28",
"upload_time_iso_8601": "2023-07-18T17:46:28.708958Z",
"url": "https://files.pythonhosted.org/packages/b0/62/08a19fac1e07d333943369c9f13e749dcf6c82feb32aa3e72d35ec44c5dc/soopervisor-0.9.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3d82444f37dbfc21dc001ae16ae4c75d219f385ed111fbfc2f417f227b049174",
"md5": "62e6721972be04d37f6aa7dd657a9e55",
"sha256": "83c8f5082d4017ca4a79dd072bc7068a2ea3496b9550937df5bc186211c714fb"
},
"downloads": -1,
"filename": "soopervisor-0.9.2.tar.gz",
"has_sig": false,
"md5_digest": "62e6721972be04d37f6aa7dd657a9e55",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 47582,
"upload_time": "2023-07-18T17:46:30",
"upload_time_iso_8601": "2023-07-18T17:46:30.189958Z",
"url": "https://files.pythonhosted.org/packages/3d/82/444f37dbfc21dc001ae16ae4c75d219f385ed111fbfc2f417f227b049174/soopervisor-0.9.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-18 17:46:30",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ploomber",
"github_project": "soopervisor",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "soopervisor"
}
JSON
