<dl>
<h1>
<div align=center><img src="https://storage.googleapis.com/vertflow/logo.png" alt="logo"/></div>
<div align=center>VertFlow</div>
</h1>
<p align="center"><i>Run Docker containers on Airflow using green energy</i></p>
<p align="center"><a href="https://drive.google.com/file/d/15XDWTu4kZfxE-SHAQcyMokr52gE_zGhv/view"><img src="https://storage.googleapis.com/vertflow/video_screenshot.png" width="358" height="201" alt="Video Demo"/></a></p>
</dl>
## đ About
**VertFlow is an [Airflow](https://airflow.apache.org/) operator for
running [Cloud Run Jobs](https://cloud.google.com/run/docs/create-jobs) on Google Cloud Platform in green data
centres.**
Cloud Run is a serverless container runtime, meaning you BYO Docker image and emit carbon only when the job is running.
This is *easier, cheaper and greener* than managing a Kubernetes cluster spinning 24/7.
**Not all data centres are created equal.**
Data centres run on electricity generated from various sources, including fossil fuels which emit harmful carbon
emissions. Some data centres are greener than others, using electricity from renewable sources such as wind and hydro.
When you deploy a container on Airflow using the VertFlow operator, it will run your container in the greenest GCP data
centre possible.
> âšī¸ Use VertFlow on [Cloud Composer 2](https://cloud.google.com/composer/docs/composer-2/composer-versioning-overview)
> to save even
> more money and CO2.
## đ§ How to install
1. `pip install VertFlow` on your Airflow instance.
2. Ensure your Airflow scheduler has outbound access to the public internet and the `roles/run.developer` Cloud IAM
role.
3. Get an [API Key for CO2 Signal](https://www.co2signal.com/), free for non-commercial use. Store in an Airflow variable called `VERTFLOW_API_KEY`.
> âšī¸ If you're using Cloud Composer, these instructions may be helpful:
> * [Installing PyPI packages](https://cloud.google.com/composer/docs/how-to/using/installing-python-dependencies#install-package)
> * [Setting up internet access](https://cloud.google.com/composer/docs/concepts/private-ip#public_internet_access_for_your_workflows)
> * [About service accounts for Cloud Composer](https://cloud.google.com/composer/docs/composer-2/access-control#about-service)
## đą How to use
Use the [`VertFlowOperator`](https://github.com/ovotech/VertFlow/blob/main/src/operator.py#L30) to instantiate a task in your DAG.
Provide:
* The address of the Docker image to run.
* A runtime specification, e.g. timeout and memory limits.
* A set of allowed regions to run the job in, based on your latency, data governance and other considerations. VertFlow
picks the greenest one.
```python
from VertFlow.operator import VertFlowOperator
from airflow import DAG
with DAG(
dag_id="hourly_dag_in_green_region",
schedule_interval="@hourly"
) as dag:
task = VertFlowOperator(
image_address="us-docker.pkg.dev/cloudrun/container/job:latest",
name="hello-world",
allowed_regions=["europe-west1", "europe-west4"],
command="echo",
arguments=["Hello World"],
service_account_email_address="my-service-account@embroidered-elephant-739.iam.gserviceaccount.com",
...
)
```
## đđē Shout out to CO2 Signal
VertFlow works thanks to real-time global carbon intensity data, gifted to the world for non-commercial use
by [CO2 Signal](https://www.co2signal.com/).
## đ¤ How to contribute
Found a bug or fancy resolving an issue? We welcome Pull Requests!
Raw data
{
"_id": null,
"home_page": "https://github.com/ovotech/VertFlow",
"name": "VertFlow",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "airflow,gcp,google,cloud run,cloud composer,green,environment,sustainability",
"author": "OVO Energy",
"author_email": "trading.dl@ovoenergy.com",
"download_url": "https://files.pythonhosted.org/packages/b0/9c/cd5c631f375f8ccab8add3f126e9dcd5eebe317364b6d823679235529497/VertFlow-0.4.1.tar.gz",
"platform": null,
"description": "<dl>\n <h1>\n <div align=center><img src=\"https://storage.googleapis.com/vertflow/logo.png\" alt=\"logo\"/></div>\n <div align=center>VertFlow</div>\n </h1>\n <p align=\"center\"><i>Run Docker containers on Airflow using green energy</i></p>\n <p align=\"center\"><a href=\"https://drive.google.com/file/d/15XDWTu4kZfxE-SHAQcyMokr52gE_zGhv/view\"><img src=\"https://storage.googleapis.com/vertflow/video_screenshot.png\" width=\"358\" height=\"201\" alt=\"Video Demo\"/></a></p>\n</dl>\n\n## \ud83d\udcd6 About\n\n**VertFlow is an [Airflow](https://airflow.apache.org/) operator for\nrunning [Cloud Run Jobs](https://cloud.google.com/run/docs/create-jobs) on Google Cloud Platform in green data\ncentres.** \nCloud Run is a serverless container runtime, meaning you BYO Docker image and emit carbon only when the job is running.\nThis is *easier, cheaper and greener* than managing a Kubernetes cluster spinning 24/7.\n\n**Not all data centres are created equal.** \nData centres run on electricity generated from various sources, including fossil fuels which emit harmful carbon\nemissions. Some data centres are greener than others, using electricity from renewable sources such as wind and hydro. \nWhen you deploy a container on Airflow using the VertFlow operator, it will run your container in the greenest GCP data\ncentre possible.\n\n> \u2139\ufe0f Use VertFlow on [Cloud Composer 2](https://cloud.google.com/composer/docs/composer-2/composer-versioning-overview)\n> to save even\n> more money and CO2.\n\n## \ud83d\udd27 How to install\n\n1. `pip install VertFlow` on your Airflow instance.\n2. Ensure your Airflow scheduler has outbound access to the public internet and the `roles/run.developer` Cloud IAM\n role.\n3. Get an [API Key for CO2 Signal](https://www.co2signal.com/), free for non-commercial use. Store in an Airflow variable called `VERTFLOW_API_KEY`.\n\n> \u2139\ufe0f If you're using Cloud Composer, these instructions may be helpful:\n> * [Installing PyPI packages](https://cloud.google.com/composer/docs/how-to/using/installing-python-dependencies#install-package)\n> * [Setting up internet access](https://cloud.google.com/composer/docs/concepts/private-ip#public_internet_access_for_your_workflows)\n> * [About service accounts for Cloud Composer](https://cloud.google.com/composer/docs/composer-2/access-control#about-service)\n\n## \ud83d\uddb1 How to use\n\nUse the [`VertFlowOperator`](https://github.com/ovotech/VertFlow/blob/main/src/operator.py#L30) to instantiate a task in your DAG.\nProvide:\n\n* The address of the Docker image to run.\n* A runtime specification, e.g. timeout and memory limits.\n* A set of allowed regions to run the job in, based on your latency, data governance and other considerations. VertFlow\n picks the greenest one.\n\n```python\nfrom VertFlow.operator import VertFlowOperator\nfrom airflow import DAG\n\nwith DAG(\n dag_id=\"hourly_dag_in_green_region\",\n schedule_interval=\"@hourly\"\n) as dag:\n task = VertFlowOperator(\n image_address=\"us-docker.pkg.dev/cloudrun/container/job:latest\",\n name=\"hello-world\",\n allowed_regions=[\"europe-west1\", \"europe-west4\"],\n command=\"echo\",\n arguments=[\"Hello World\"],\n service_account_email_address=\"my-service-account@embroidered-elephant-739.iam.gserviceaccount.com\",\n ...\n )\n```\n\n## \ud83d\udd0c\ud83d\uddfa Shout out to CO2 Signal\n\nVertFlow works thanks to real-time global carbon intensity data, gifted to the world for non-commercial use\nby [CO2 Signal](https://www.co2signal.com/).\n\n## \ud83e\udd1d How to contribute\n\nFound a bug or fancy resolving an issue? We welcome Pull Requests!\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "Apache Airflow operator for running Google Cloud Run Jobs using green energy",
"version": "0.4.1",
"project_urls": {
"Homepage": "https://github.com/ovotech/VertFlow"
},
"split_keywords": [
"airflow",
"gcp",
"google",
"cloud run",
"cloud composer",
"green",
"environment",
"sustainability"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "091c540578712d57c17b46ab1cd66a2e038db2c7ccaf08502c9cf3952241c3cf",
"md5": "5380297b1fd99ce43babee97f479e0aa",
"sha256": "52135efe1dc2e3e377cd376f9f5c313e51ed999148311c3d927ddc3798b3778d"
},
"downloads": -1,
"filename": "VertFlow-0.4.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "5380297b1fd99ce43babee97f479e0aa",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 19893,
"upload_time": "2023-07-28T13:25:42",
"upload_time_iso_8601": "2023-07-28T13:25:42.003699Z",
"url": "https://files.pythonhosted.org/packages/09/1c/540578712d57c17b46ab1cd66a2e038db2c7ccaf08502c9cf3952241c3cf/VertFlow-0.4.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b09ccd5c631f375f8ccab8add3f126e9dcd5eebe317364b6d823679235529497",
"md5": "d05c4d94cdc3ce4b6614b7631e2fc22c",
"sha256": "7af593db9d960ede74cd70e70146a0bc67cac98f6d151fc5511848dca8a98f1e"
},
"downloads": -1,
"filename": "VertFlow-0.4.1.tar.gz",
"has_sig": false,
"md5_digest": "d05c4d94cdc3ce4b6614b7631e2fc22c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 20885,
"upload_time": "2023-07-28T13:25:43",
"upload_time_iso_8601": "2023-07-28T13:25:43.042875Z",
"url": "https://files.pythonhosted.org/packages/b0/9c/cd5c631f375f8ccab8add3f126e9dcd5eebe317364b6d823679235529497/VertFlow-0.4.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-07-28 13:25:43",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ovotech",
"github_project": "VertFlow",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "vertflow"
}
JSON
