| Name | turbine-py JSON |
| Version |
2.1.4
JSON |
| download |
| home_page | https://meroxa.io/ |
| Summary | Meroxa Turbine data application framework |
| upload_time | 2023-08-15 20:40:18 |
| maintainer | |
| docs_url | None |
| author | Eric Cheatham |
| requires_python | >=3.9 |
| license | MIT |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Turbine
<p align="center" style="text-align:center;">
<img alt="turbine logo" src="https://github.com/meroxa/turbine-py/blob/main/docs/turbine-outline.svg?raw=true" width="500" />
</p>
Turbine is a data application framework for building server-side applications that are event-driven, respond to data in real-time, and scale using cloud-native best practices.
The benefits of using Turbine include:
* **Native Developer Tooling:** Turbine doesn't come with any bespoke DSL or patterns. Write software like you normally would!
* **Fits into Existing DevOps Workflows:** Build, test, and deploy. Turbine encourages best practices from the start. Don't test your data app in production ever again.
* **Local Development mirrors Production:** When running locally, you'll immediately see how your app reacts to data. What you get there will be exactly what happens in production but with _scale_ and _speed_.
* **Available in many different programming langauages:** Turbine started out in Go but is available in other languages too:
* [Go](https://github.com/meroxa/turbine-go)
* [Javascript](https://github.com/meroxa/turbine-js)
* [Python](https://github.com/meroxa/turbine-py)
## Getting Started
To get started, you'll need to install the `turbine-py` dependency via `pip`:
```bash
pip3 install turbine-py
```
Next, you'll need to [download the Meroxa CLI](https://github.com/meroxa/cli#installation-guide). Once downloaded and installed, initialize a new project using the following CLI command:
```bash
$ meroxa apps init testapp --lang py
```
The CLI will create a new folder called `testapp` located in the directory where the command was issued. If you want to initialize the app somewhere else, you can append the `--path` flag to the command (`meroxa apps init testapp --lang py --path ~/anotherdir`). Once you enter the `testapp` directory, the contents will look like this:
```bash
$ tree testapp/
testapp
├── README.md
├── main.py
├── app.json
├── __init__.py
└── fixtures
├── demo-cdc.json
└── demo-no-cdc.json
```
This will be a full-fledged Turbine app that can run. You can even run the tests using the command `meroxa apps run` in the root of the app directory. It provides just enough to show you what you need to get started.
### `main.py`
This configuration file is where you begin your Turbine journey. Any time a Turbine app runs, this is the entry point for the entire application. When the project is created, the file will look like this:
```python
# Dependencies of the example data app
import hashlib
import sys
from turbine.src.turbine_app import RecordList, TurbineApp
def anonymize(records: RecordList) -> RecordList:
for record in records:
try:
payload = record.value["payload"]
# Hash the email
payload["customer_email"] = hashlib.sha256(
payload["customer_email"].encode("utf-8")
).hexdigest()
except Exception as e:
print("Error occurred while parsing records: " + str(e))
return records
class App:
@staticmethod
async def run(turbine: TurbineApp):
try:
source = await turbine.resources("source_name")
records = await source.records("collection_name")
anonymized = await turbine.process(records, anonymize)
destination_db = await turbine.resources("destination_name")
await destination_db.write(anonymized, "collection_archive")
except Exception as e:
print(e, file=sys.stderr)
```
Let's talk about the important parts of this code. Turbine apps have five functions that comprise the entire DSL. Outside of these functions, you can write whatever code you want to accomplish your tasks:
```python
async def run(turbine: TurbineApp):
```
`run` is the main entry point for the application. This is where you can initialize the Turbine framework. This is also the place where, when you deploy your Turbine app to Meroxa, Meroxa will use this as the place to boot up the application.
```python
source = await turbine.resources("source_name")
```
The `resources` function identifies the upstream or downstream system that you want your code to work with. The `source_name` is the string identifier of the particular system. The string should map to an associated identifier in your `app.json` to configure what's being connected to. For more details, see the `app.json` section.
```python
records = await source.records("collection_name")
```
Once you've got `resources` set up, you can now stream records from it, but you need to identify what records you want. The `records` function identifies the records or events that you want to stream into your data app.
```python
anonymized = await turbine.process(records, anonymize)
```
The `process` function is Turbine's way of saying, for the records that are coming in, I want you to process these records against a function. Once your app is deployed on Meroxa, Meroxa will do the work to take each record or event that does get streamed to your app and then run your code against it. This allows Meroxa to scale out your processing relative to the velocity of the records streaming in.
```python
await destination_db.write(anonymized, "collection_archive")
```
The `write` function is optional. It takes any records given to it and streams them to the downstream system. In many cases, you might not need to stream data to another system, but this gives you an easy way to do so.
### `app.json`
This file contains all of the options for configuring a Turbine app. Upon initialization of an app, the CLI will scaffold the file for you with available options:
```json
{
"name": "testapp",
"language": "python",
"environment": "common",
"resources": {
"source_name": "fixtures/demo-cdc.json"
}
}
```
* `name` - The name of your application. This should not change after app initialization.
* `language` - Tells Meroxa what language the app is upon deployment.
* `environment` - "common" is the only available environment. Meroxa does have the ability to create isolated environments but this feature is currently in beta.
* `resources` - These are the named integrations that you'll use in your application. The `source_name` needs to match the name of the resource that you'll set up in Meroxa using the `meroxa resources create` command or via the Dashboard. You can point to the path in the fixtures that'll be used to mock the resource when you run `meroxa apps run`.
### Fixtures
Fixtures are JSON-formatted samples of data records you can use while locally developing your Turbine app. Whether CDC or non-CDC-formatted data records, fixtures adhere to the following structure:
```json
{
"collection_name": [
{
"key": "1",
"value": {
"schema": {
/* ... */
},
"payload": {
/* ... */
}
}
}
]
}
```
* `collection_name` — Identifies the name of the records or events you are streaming to your data app.
* `key` — Denotes one or more sample records within a fixture file. `key` is always a string.
* `value` — Holds the `schema` and `payload` of the sample data record.
* `schema` — Comes as part of your sample data record. `schema` describes the record or event structure.
* `payload` — Comes as part of your sample data record. `payload` describes what about the record or event changed.
Your newly created data app should have a `demo-cdc.json` and `demo-non-cdc.json` in the `/fixtures` directory as examples to follow.
### Testing
Testing should follow standard Python development practices.
## Documentation && Reference
The most comprehensive documentation for Turbine and how to work with Turbine apps is on the Meroxa site: [https://docs.meroxa.com/](https://docs.meroxa.com)
## Contributing
For information on how to contribute to Turbine-py, checkout our contribution guide [HERE](CONTRIBUTING.md)
### Example apps
[See what a sample python data app looks like using our framework](https://github.com/meroxa/turbine-py-examples)
Raw data
{
"_id": null,
"home_page": "https://meroxa.io/",
"name": "turbine-py",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": "",
"keywords": "",
"author": "Eric Cheatham",
"author_email": "eric@meroxa.io",
"download_url": "https://files.pythonhosted.org/packages/55/ea/084882addebf6945beb7fb835dfd53770be64251fe30419ab578c0de0832/turbine-py-2.1.4.tar.gz",
"platform": null,
"description": "# Turbine\n\n<p align=\"center\" style=\"text-align:center;\">\n <img alt=\"turbine logo\" src=\"https://github.com/meroxa/turbine-py/blob/main/docs/turbine-outline.svg?raw=true\" width=\"500\" />\n</p>\n\n\n\nTurbine is a data application framework for building server-side applications that are event-driven, respond to data in real-time, and scale using cloud-native best practices.\n\nThe benefits of using Turbine include:\n\n* **Native Developer Tooling:** Turbine doesn't come with any bespoke DSL or patterns. Write software like you normally would!\n\n* **Fits into Existing DevOps Workflows:** Build, test, and deploy. Turbine encourages best practices from the start. Don't test your data app in production ever again.\n\n* **Local Development mirrors Production:** When running locally, you'll immediately see how your app reacts to data. What you get there will be exactly what happens in production but with _scale_ and _speed_.\n\n* **Available in many different programming langauages:** Turbine started out in Go but is available in other languages too:\n * [Go](https://github.com/meroxa/turbine-go)\n * [Javascript](https://github.com/meroxa/turbine-js)\n * [Python](https://github.com/meroxa/turbine-py)\n\n\n## Getting Started\n\nTo get started, you'll need to install the `turbine-py` dependency via `pip`:\n\n```bash\npip3 install turbine-py\n```\n\nNext, you'll need to [download the Meroxa CLI](https://github.com/meroxa/cli#installation-guide). Once downloaded and installed, initialize a new project using the following CLI command:\n\n```bash\n$ meroxa apps init testapp --lang py\n```\n\nThe CLI will create a new folder called `testapp` located in the directory where the command was issued. If you want to initialize the app somewhere else, you can append the `--path` flag to the command (`meroxa apps init testapp --lang py --path ~/anotherdir`). Once you enter the `testapp` directory, the contents will look like this:\n\n```bash\n$ tree testapp/\ntestapp\n\u251c\u2500\u2500 README.md\n\u251c\u2500\u2500 main.py\n\u251c\u2500\u2500 app.json\n\u251c\u2500\u2500 __init__.py\n\u2514\u2500\u2500 fixtures\n \u251c\u2500\u2500 demo-cdc.json\n \u2514\u2500\u2500 demo-no-cdc.json\n```\n\nThis will be a full-fledged Turbine app that can run. You can even run the tests using the command `meroxa apps run` in the root of the app directory. It provides just enough to show you what you need to get started.\n\n### `main.py`\n\nThis configuration file is where you begin your Turbine journey. Any time a Turbine app runs, this is the entry point for the entire application. When the project is created, the file will look like this:\n\n```python\n# Dependencies of the example data app\nimport hashlib\nimport sys\n\nfrom turbine.src.turbine_app import RecordList, TurbineApp\n\ndef anonymize(records: RecordList) -> RecordList:\n for record in records:\n try:\n payload = record.value[\"payload\"]\n\n # Hash the email\n payload[\"customer_email\"] = hashlib.sha256(\n payload[\"customer_email\"].encode(\"utf-8\")\n ).hexdigest()\n\n except Exception as e:\n print(\"Error occurred while parsing records: \" + str(e))\n return records\n\n\nclass App:\n @staticmethod\n async def run(turbine: TurbineApp):\n try:\n source = await turbine.resources(\"source_name\")\n\n records = await source.records(\"collection_name\")\n\n anonymized = await turbine.process(records, anonymize)\n\n destination_db = await turbine.resources(\"destination_name\")\n\n await destination_db.write(anonymized, \"collection_archive\")\n except Exception as e:\n print(e, file=sys.stderr)\n```\n\nLet's talk about the important parts of this code. Turbine apps have five functions that comprise the entire DSL. Outside of these functions, you can write whatever code you want to accomplish your tasks:\n\n```python\nasync def run(turbine: TurbineApp):\n```\n\n`run` is the main entry point for the application. This is where you can initialize the Turbine framework. This is also the place where, when you deploy your Turbine app to Meroxa, Meroxa will use this as the place to boot up the application.\n\n```python\nsource = await turbine.resources(\"source_name\")\n```\n\nThe `resources` function identifies the upstream or downstream system that you want your code to work with. The `source_name` is the string identifier of the particular system. The string should map to an associated identifier in your `app.json` to configure what's being connected to. For more details, see the `app.json` section.\n\n```python\nrecords = await source.records(\"collection_name\")\n```\n\nOnce you've got `resources` set up, you can now stream records from it, but you need to identify what records you want. The `records` function identifies the records or events that you want to stream into your data app.\n\n```python\nanonymized = await turbine.process(records, anonymize)\n```\n\nThe `process` function is Turbine's way of saying, for the records that are coming in, I want you to process these records against a function. Once your app is deployed on Meroxa, Meroxa will do the work to take each record or event that does get streamed to your app and then run your code against it. This allows Meroxa to scale out your processing relative to the velocity of the records streaming in.\n\n```python\nawait destination_db.write(anonymized, \"collection_archive\")\n```\n\nThe `write` function is optional. It takes any records given to it and streams them to the downstream system. In many cases, you might not need to stream data to another system, but this gives you an easy way to do so.\n\n\n### `app.json`\n\nThis file contains all of the options for configuring a Turbine app. Upon initialization of an app, the CLI will scaffold the file for you with available options:\n\n```json\n{\n \"name\": \"testapp\",\n \"language\": \"python\",\n \"environment\": \"common\",\n \"resources\": {\n \"source_name\": \"fixtures/demo-cdc.json\"\n }\n}\n```\n\n* `name` - The name of your application. This should not change after app initialization.\n* `language` - Tells Meroxa what language the app is upon deployment.\n* `environment` - \"common\" is the only available environment. Meroxa does have the ability to create isolated environments but this feature is currently in beta.\n* `resources` - These are the named integrations that you'll use in your application. The `source_name` needs to match the name of the resource that you'll set up in Meroxa using the `meroxa resources create` command or via the Dashboard. You can point to the path in the fixtures that'll be used to mock the resource when you run `meroxa apps run`.\n\n### Fixtures\n\nFixtures are JSON-formatted samples of data records you can use while locally developing your Turbine app. Whether CDC or non-CDC-formatted data records, fixtures adhere to the following structure:\n\n```json\n{\n \"collection_name\": [\n {\n \"key\": \"1\",\n \"value\": {\n \t\t \"schema\": {\n \t\t\t /* ... */\n \t\t },\n \t\t \"payload\": {\n \t\t\t /* ... */\n \t\t }\n }\n }\n ]\n}\n```\n* `collection_name` \u2014 Identifies the name of the records or events you are streaming to your data app.\n* `key` \u2014 Denotes one or more sample records within a fixture file. `key` is always a string.\n* `value` \u2014 Holds the `schema` and `payload` of the sample data record.\n* `schema` \u2014 Comes as part of your sample data record. `schema` describes the record or event structure.\n* `payload` \u2014 Comes as part of your sample data record. `payload` describes what about the record or event changed.\n\nYour newly created data app should have a `demo-cdc.json` and `demo-non-cdc.json` in the `/fixtures` directory as examples to follow.\n\n### Testing\n\nTesting should follow standard Python development practices.\n\n## Documentation && Reference\n\nThe most comprehensive documentation for Turbine and how to work with Turbine apps is on the Meroxa site: [https://docs.meroxa.com/](https://docs.meroxa.com)\n\n## Contributing\n\nFor information on how to contribute to Turbine-py, checkout our contribution guide [HERE](CONTRIBUTING.md)\n\n### Example apps\n\n[See what a sample python data app looks like using our framework](https://github.com/meroxa/turbine-py-examples)\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Meroxa Turbine data application framework",
"version": "2.1.4",
"project_urls": {
"Homepage": "https://meroxa.io/",
"Issue Tracker": "https://github.com/meroxa/turbine-py/issues",
"Source Code": "https://github.com/meroxa/turbine-py/"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "d755890c4b8e48776d93a30bee848a1d6ebb53518d376ba4026521e2618ea601",
"md5": "895244ac195caabe4db0fced873c36f2",
"sha256": "660f5ab0ef8c5383e9505192dd34035c86125a3af8d50492bc2041e3605d32a4"
},
"downloads": -1,
"filename": "turbine_py-2.1.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "895244ac195caabe4db0fced873c36f2",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 25019,
"upload_time": "2023-08-15T20:40:16",
"upload_time_iso_8601": "2023-08-15T20:40:16.199946Z",
"url": "https://files.pythonhosted.org/packages/d7/55/890c4b8e48776d93a30bee848a1d6ebb53518d376ba4026521e2618ea601/turbine_py-2.1.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "55ea084882addebf6945beb7fb835dfd53770be64251fe30419ab578c0de0832",
"md5": "1646980d6701960953f2d317bf69455d",
"sha256": "1b3a2ab8fa8f191060003db4b6e39075b4ec303a3969deb875ca8f53dc864553"
},
"downloads": -1,
"filename": "turbine-py-2.1.4.tar.gz",
"has_sig": false,
"md5_digest": "1646980d6701960953f2d317bf69455d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 22393,
"upload_time": "2023-08-15T20:40:18",
"upload_time_iso_8601": "2023-08-15T20:40:18.137478Z",
"url": "https://files.pythonhosted.org/packages/55/ea/084882addebf6945beb7fb835dfd53770be64251fe30419ab578c0de0832/turbine-py-2.1.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-15 20:40:18",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "meroxa",
"github_project": "turbine-py",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"tox": true,
"lcname": "turbine-py"
}
