<p align="center">
<img src="images/logo-no-background.png" alt="canarypy" style="width:200px;"/>
</p>
<p align="center">
<em>CanaryPy - A light and powerful canary release for Data Pipelines</em>
</p>
<p align="center">
<a href="https://github.com/thcidale0808/canarypy/actions?query=workflow%3ATest+event%3Apush+branch%3Amain" target="_blank">
<img src="https://github.com/thcidale0808/canarypy/actions/workflows/test.yml/badge.svg?branch=main" alt="Test">
</a>
<a href="https://pypi.org/project/canarypy/" target="_blank">
<img src="https://img.shields.io/pypi/v/canarypy?color=%2334D058&label=pypi%20package" alt="Package version">
</a>
<a href="https://pypi.org/project/canarypy" target="_blank">
<img src="https://img.shields.io/pypi/pyversions/canarypy.svg?color=%2334D058" alt="Supported Python versions">
</a>
</p>
## Introduction
CanaryPy is a lightweight yet potent solution developed for orchestrating canary releases in your data pipelines. Its primary aim is to facilitate incremental deployment of changes in your software, thereby minimizing the impact of any unanticipated issues. Should you encounter any hiccups during the process, CanaryPy enables a swift and seamless rollback to maintain stability.
It integrates a FastAPI application for managing APIs, a Streamlit application for interactive web dashboards, a command-line interface (CLI) for enabling user interaction through terminal commands, and a Python client for incorporating CanaryPy functionalities in your Python applications.
With CanaryPy, you gain the ability to test the waters with new features or updates, ensuring they function as expected in a live environment but affecting only a small subset of data pipelines. CanaryPy's canary release strategy promotes safer deployments, contributing to overall enhanced performance and robustness of your data platform.
## How it works
CanaryPy has three main entities:
* `Product`: It is an application that will have releases.
* `Release`: It is any version of your Product. It can be of two types: `active` and `canary`. A release is created as canary which will be tested against the `active` using a linear growth rollout strategy. If the canary performs well during all the phases, it'll be promoted to `active`. The release performance is measure based on its signals.
* `Signal`: It the execution result of a release. It can have a `success` or `failed` state. Typically, is the result of your job.
CanaryPy also includes a CLI to manage the products, releases, and signals. The CLI is built using the Click library.
The CLI uses the FastAPI application as the backend and therefore the following environment variables need to be set to run the CLI:
* `CANARYPY_URL`: The base URL of the FastAPI application. Defaults to `http://localhost:8080`.
### Features available
#### Products
Products are the fist step in the CanaryPy system. A product is a software application that is being released using the CanaryPy system. The CLI provides the following commands to manage products:
`canapyry product create`
This will prompt the user to enter the details of the product, and will create the product in the system.
#### Releases
Releases are the second step in the CanaryPy system. A release is a version of a product that is being released using the CanaryPy system. The CLI provides the following commands to manage releases:
`canarypy release create --semver_version <semver> --artifact_url <product>`
This will create a new release for a product. The semver version and the artifact URL are required parameters.
it's possible to fetch the latest stable release of a product using the python client as shown below:
```python
from canarypy.client import CanaryPyClient
client = CanaryPyClient(base_url="http://localhost:8000")
client.get_latest_stable_version(product_name)
```
#### Signals
Signals are the third step in the CanaryPy system. A signal is the result of an execution of a product with a release. The CLI provides the following commands to manage signals:
`canarypy signal create --status <status> --description <description> --instance-id <instance-id> -- semver-version <version> --artifact-url <artifact-url>`
It's also possible to send signals to the CanaryPy system using the Python client as shown below:
```python
from canarypy.client import CanaryPyClient
client = CanaryPyClient(base_url="http://localhost:8000")
client.send_signal_to_canary(artifact_url, version, instance_id, description, status)
```
# Airflow Integration
`CanaryPy` provides a plugin to integrate your Airflow tasks with CanaryPy backend with a minimal effort. The plugin can be installed by executing:
`pip install canarypy-airflow-plugin`
Check this [tutorial](examples/airflow-tutorial.md) to understand how this integration works. The plugin code can be found [here](integrations/airflow).
# CanaryPy Deployment
CanaryPy is built around a modular architecture composed by an FastAPI application that host the rest APIs and a Streamlist application that show metrics about release performance. Both applications use a shared PostgreSQL database to store the data.
Setting up CanaryPy for a production environment involves tuning up these two components and making sure the shared PostgreSQL database they both depend on is ready to go. Let's get into the nuts and bolts of how to do that.
## FastAPI Application
The FastAPI application serves as the backend for CanaryPy, providing RESTful APIs for managing products, releases, and signals.
### Summary
The application provides endpoints for creating and retrieving products and their respective releases. It also includes endpoints for creating and retrieving signals related to each release.
The FastAPI application uses a PostgreSQL database to store the data.
### Endpoints
The CanaryPy provides endpoints to manage products, releases, and signals. More details about the endpoints can be found in the `/docs` endpoint.
### How to run
1. The following environment variable can be set to start the FastAPI server:
* `CANARYPY_API_PORT`: The base URL of the FastAPI application. Defaults to `8080`.
* `CANARYPY_API_HOST`: The base URL of the FastAPI application. Defaults to `0.0.0.0`.
* `CANARYPY_API_RELOAD`: Whether to reload the server when code changes are detected. Defaults to `True`.
* `CANARYPY_API_DEBUG`: Whether to run the server in debug mode. Defaults to `True`.
* `CANARYPY_API_LOG_LEVEL`: The log level for the server. Defaults to `info`.
* `CANARYPY_DB_CONN_STRING`: The connection string for the database. Alternatively, you can set the connection details in separated environment variables:
* `CANARYPY_DB_USER`: The username for the database.
* `CANARYPY_DB_PASSWORD`: The password for the database.
* `CANARYPY_DB_HOST`: The host for the database.
* `CANARYPY_DB_PORT`: The port for the database.
* `CANARYPY_DB_NAME`: The name of the database.
2. Run the FastAPI application using the following command `canarypy api start`
## Streamlit Application
The Streamlit application serves as a web-based user interface for visualizing the release metrics.
### Summary
The application fetches the metrics data from the FastAPI backend and displays it in a user-friendly format, helping you understand the release trends and signal patterns.
The Streamlit application uses a PostgreSQL database to fetch the data for the dashboard.
### How to run
1. The following environment variable can be set to start the FastAPI server:
* `CANARYPY_DB_CONN_STRING`: The connection string for the database. Alternatively, you can set the connection details in separated environment variables:
* `CANARYPY_DB_USER`: The username for the database.
* `CANARYPY_DB_PASSWORD`: The password for the database.
* `CANARYPY_DB_HOST`: The host for the database.
* `CANARYPY_DB_PORT`: The port for the database.
* `CANARYPY_DB_NAME`: The name of the database.
2. Run the Streamlit application: `canarypy web start`
## Database Migrations
This application uses Alembic for database schema migrations. Alembic is a database migration tool for SQLAlchemy and allows us to automate changes to our database schema.
To perform the database migrations, we have encapsulated the necessary Alembic commands within a custom Click command. This simplifies the process, allowing you to upgrade your database by executing a single Python command.
Execute the following command to upgrade your database:
1. The following environment variable can be set to start the FastAPI server:
* `CANARYPY_DB_CONN_STRING`: The connection string for the database. Alternatively, you can set the connection details in separated environment variables:
* `CANARYPY_DB_USER`: The username for the database.
* `CANARYPY_DB_PASSWORD`: The password for the database.
* `CANARYPY_DB_HOST`: The host for the database.
* `CANARYPY_DB_PORT`: The port for the database.
* `CANARYPY_DB_NAME`: The name of the database.
2. Execute the following command to upgrade your database: `canarypy db upgrade`
Raw data
{
"_id": null,
"home_page": "https://github.com/thcidale0808/canarypy",
"name": "canarypy",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "",
"author": "Thiago Assumpcao",
"author_email": "thcidale@gmail.com",
"download_url": "",
"platform": null,
"description": "<p align=\"center\">\n <img src=\"images/logo-no-background.png\" alt=\"canarypy\" style=\"width:200px;\"/>\n</p>\n<p align=\"center\">\n <em>CanaryPy - A light and powerful canary release for Data Pipelines</em>\n</p>\n<p align=\"center\">\n<a href=\"https://github.com/thcidale0808/canarypy/actions?query=workflow%3ATest+event%3Apush+branch%3Amain\" target=\"_blank\">\n <img src=\"https://github.com/thcidale0808/canarypy/actions/workflows/test.yml/badge.svg?branch=main\" alt=\"Test\">\n</a>\n<a href=\"https://pypi.org/project/canarypy/\" target=\"_blank\">\n <img src=\"https://img.shields.io/pypi/v/canarypy?color=%2334D058&label=pypi%20package\" alt=\"Package version\">\n</a>\n<a href=\"https://pypi.org/project/canarypy\" target=\"_blank\">\n <img src=\"https://img.shields.io/pypi/pyversions/canarypy.svg?color=%2334D058\" alt=\"Supported Python versions\">\n</a>\n</p>\n\n## Introduction\n\nCanaryPy is a lightweight yet potent solution developed for orchestrating canary releases in your data pipelines. Its primary aim is to facilitate incremental deployment of changes in your software, thereby minimizing the impact of any unanticipated issues. Should you encounter any hiccups during the process, CanaryPy enables a swift and seamless rollback to maintain stability.\n\nIt integrates a FastAPI application for managing APIs, a Streamlit application for interactive web dashboards, a command-line interface (CLI) for enabling user interaction through terminal commands, and a Python client for incorporating CanaryPy functionalities in your Python applications.\n\nWith CanaryPy, you gain the ability to test the waters with new features or updates, ensuring they function as expected in a live environment but affecting only a small subset of data pipelines. CanaryPy's canary release strategy promotes safer deployments, contributing to overall enhanced performance and robustness of your data platform.\n\n## How it works\n\nCanaryPy has three main entities: \n* `Product`: It is an application that will have releases. \n* `Release`: It is any version of your Product. It can be of two types: `active` and `canary`. A release is created as canary which will be tested against the `active` using a linear growth rollout strategy. If the canary performs well during all the phases, it'll be promoted to `active`. The release performance is measure based on its signals.\n* `Signal`: It the execution result of a release. It can have a `success` or `failed` state. Typically, is the result of your job.\n\nCanaryPy also includes a CLI to manage the products, releases, and signals. The CLI is built using the Click library.\n\nThe CLI uses the FastAPI application as the backend and therefore the following environment variables need to be set to run the CLI:\n\n* `CANARYPY_URL`: The base URL of the FastAPI application. Defaults to `http://localhost:8080`.\n\n### Features available\n\n#### Products\nProducts are the fist step in the CanaryPy system. A product is a software application that is being released using the CanaryPy system. The CLI provides the following commands to manage products:\n\n`canapyry product create`\n\nThis will prompt the user to enter the details of the product, and will create the product in the system.\n\n#### Releases\nReleases are the second step in the CanaryPy system. A release is a version of a product that is being released using the CanaryPy system. The CLI provides the following commands to manage releases:\n\n`canarypy release create --semver_version <semver> --artifact_url <product>`\n\nThis will create a new release for a product. The semver version and the artifact URL are required parameters.\n\nit's possible to fetch the latest stable release of a product using the python client as shown below:\n\n```python\nfrom canarypy.client import CanaryPyClient\nclient = CanaryPyClient(base_url=\"http://localhost:8000\")\nclient.get_latest_stable_version(product_name)\n```\n\n#### Signals\nSignals are the third step in the CanaryPy system. A signal is the result of an execution of a product with a release. The CLI provides the following commands to manage signals:\n\n`canarypy signal create --status <status> --description <description> --instance-id <instance-id> -- semver-version <version> --artifact-url <artifact-url>`\n\nIt's also possible to send signals to the CanaryPy system using the Python client as shown below:\n\n```python\nfrom canarypy.client import CanaryPyClient\nclient = CanaryPyClient(base_url=\"http://localhost:8000\")\nclient.send_signal_to_canary(artifact_url, version, instance_id, description, status)\n```\n\n# Airflow Integration\n\n`CanaryPy` provides a plugin to integrate your Airflow tasks with CanaryPy backend with a minimal effort. The plugin can be installed by executing:\n\n`pip install canarypy-airflow-plugin`\n\nCheck this [tutorial](examples/airflow-tutorial.md) to understand how this integration works. The plugin code can be found [here](integrations/airflow).\n\n# CanaryPy Deployment\n\nCanaryPy is built around a modular architecture composed by an FastAPI application that host the rest APIs and a Streamlist application that show metrics about release performance. Both applications use a shared PostgreSQL database to store the data.\n\nSetting up CanaryPy for a production environment involves tuning up these two components and making sure the shared PostgreSQL database they both depend on is ready to go. Let's get into the nuts and bolts of how to do that.\n\n## FastAPI Application\n\nThe FastAPI application serves as the backend for CanaryPy, providing RESTful APIs for managing products, releases, and signals.\n\n### Summary\n\nThe application provides endpoints for creating and retrieving products and their respective releases. It also includes endpoints for creating and retrieving signals related to each release.\n\nThe FastAPI application uses a PostgreSQL database to store the data. \n\n### Endpoints\n\nThe CanaryPy provides endpoints to manage products, releases, and signals. More details about the endpoints can be found in the `/docs` endpoint.\n\n### How to run\n\n1. The following environment variable can be set to start the FastAPI server:\n * `CANARYPY_API_PORT`: The base URL of the FastAPI application. Defaults to `8080`.\n * `CANARYPY_API_HOST`: The base URL of the FastAPI application. Defaults to `0.0.0.0`.\n * `CANARYPY_API_RELOAD`: Whether to reload the server when code changes are detected. Defaults to `True`.\n * `CANARYPY_API_DEBUG`: Whether to run the server in debug mode. Defaults to `True`.\n * `CANARYPY_API_LOG_LEVEL`: The log level for the server. Defaults to `info`.\n * `CANARYPY_DB_CONN_STRING`: The connection string for the database. Alternatively, you can set the connection details in separated environment variables:\n * `CANARYPY_DB_USER`: The username for the database.\n * `CANARYPY_DB_PASSWORD`: The password for the database.\n * `CANARYPY_DB_HOST`: The host for the database.\n * `CANARYPY_DB_PORT`: The port for the database.\n * `CANARYPY_DB_NAME`: The name of the database.\n2. Run the FastAPI application using the following command `canarypy api start`\n\n## Streamlit Application\n\nThe Streamlit application serves as a web-based user interface for visualizing the release metrics.\n\n### Summary\n\nThe application fetches the metrics data from the FastAPI backend and displays it in a user-friendly format, helping you understand the release trends and signal patterns.\n\nThe Streamlit application uses a PostgreSQL database to fetch the data for the dashboard. \n\n### How to run\n\n1. The following environment variable can be set to start the FastAPI server:\n * `CANARYPY_DB_CONN_STRING`: The connection string for the database. Alternatively, you can set the connection details in separated environment variables:\n * `CANARYPY_DB_USER`: The username for the database.\n * `CANARYPY_DB_PASSWORD`: The password for the database.\n * `CANARYPY_DB_HOST`: The host for the database.\n * `CANARYPY_DB_PORT`: The port for the database.\n * `CANARYPY_DB_NAME`: The name of the database.\n2. Run the Streamlit application: `canarypy web start`\n\n## Database Migrations\n\nThis application uses Alembic for database schema migrations. Alembic is a database migration tool for SQLAlchemy and allows us to automate changes to our database schema.\n\nTo perform the database migrations, we have encapsulated the necessary Alembic commands within a custom Click command. This simplifies the process, allowing you to upgrade your database by executing a single Python command.\n\nExecute the following command to upgrade your database:\n1. The following environment variable can be set to start the FastAPI server:\n * `CANARYPY_DB_CONN_STRING`: The connection string for the database. Alternatively, you can set the connection details in separated environment variables:\n * `CANARYPY_DB_USER`: The username for the database.\n * `CANARYPY_DB_PASSWORD`: The password for the database.\n * `CANARYPY_DB_HOST`: The host for the database.\n * `CANARYPY_DB_PORT`: The port for the database.\n * `CANARYPY_DB_NAME`: The name of the database.\n2. Execute the following command to upgrade your database: `canarypy db upgrade`\n\n\n",
"bugtrack_url": null,
"license": "",
"summary": "CanaryPy - A light and powerful canary release for Data Pipelines",
"version": "0.0.14",
"project_urls": {
"Bug Reports": "https://github.com/thcidale0808/canarypy/issues",
"Homepage": "https://github.com/thcidale0808/canarypy",
"Source": "https://github.com/thcidale0808/canarypy/"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "43538f29d56da886df9735e58473ca6d7efee4c0683ac3a5c0f8c53f9c2f550c",
"md5": "741fc6827d5b2934664075903a90419f",
"sha256": "56b34b607e963840025efc051de019845087dcf8de839f72fbb0d01268e007a6"
},
"downloads": -1,
"filename": "canarypy-0.0.14-py3-none-any.whl",
"has_sig": false,
"md5_digest": "741fc6827d5b2934664075903a90419f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 38544,
"upload_time": "2023-07-09T22:33:33",
"upload_time_iso_8601": "2023-07-09T22:33:33.614093Z",
"url": "https://files.pythonhosted.org/packages/43/53/8f29d56da886df9735e58473ca6d7efee4c0683ac3a5c0f8c53f9c2f550c/canarypy-0.0.14-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-09 22:33:33",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "thcidale0808",
"github_project": "canarypy",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "fastapi",
"specs": [
[
"~=",
"0.75.2"
]
]
},
{
"name": "uvicorn",
"specs": [
[
"~=",
"0.17.6"
]
]
},
{
"name": "psycopg2-binary",
"specs": [
[
"~=",
"2.9.0"
]
]
},
{
"name": "SQLAlchemy",
"specs": [
[
">=",
"1.4.0"
],
[
"<",
"2.0.0"
]
]
},
{
"name": "python-jose",
"specs": [
[
">=",
"3.3.0"
],
[
"<",
"3.4.0"
]
]
},
{
"name": "requests",
"specs": [
[
">=",
"2.27.0"
],
[
"<",
"2.28.0"
]
]
},
{
"name": "aiofiles",
"specs": [
[
">=",
"0.8.0"
],
[
"<",
"0.9.0"
]
]
},
{
"name": "click",
"specs": [
[
"~=",
"8.1.2"
]
]
},
{
"name": "alembic",
"specs": []
},
{
"name": "InquirerPy",
"specs": []
},
{
"name": "pywebio",
"specs": []
},
{
"name": "plotly",
"specs": []
},
{
"name": "pandas",
"specs": []
},
{
"name": "streamlit",
"specs": []
}
],
"test_requirements": [],
"lcname": "canarypy"
}
JSON
