| Name | nodestream-plugin-k8s JSON |
| Version |
0.13.0
JSON |
| download |
| home_page | None |
| Summary | A plugin for nodestream that manages k8s resources. |
| upload_time | 2024-08-09 17:34:23 |
| maintainer | None |
| docs_url | None |
| author | Zach Probst |
| requires_python | <4.0,>=3.10 |
| license | Apache 2.0 |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Nodestream Kubernetes Plugin
NOTE: This plugin is still in development and is not ready for production use.
## Overview
This plugin allows you to deploy nodestream on a Kubernete cluster. As your project evolves by adding more data pipelines, `nodstream-plugin-k8s` will be able to manage your kubernetes resources for you.
## Installation
```bash
pip install nodestream-plugin-k8s
```
## Usage
### 1. Create a `nodestream.yaml` file and project.
```yaml
# nodestream.yaml
scopes:
crons:
pipelines:
- path: pipelines/crons/my_scheduled_pipeline.yaml
perpetual:
pipelines:
- path: pipelines/perpetual/my_kafka_pipeline.yaml
```
### 2. Annotate your pipeline with `nodestream-plugin-k8s` annotations.
```yaml
# nodestream.yaml
scopes:
crons:
pipelines:
- path: pipelines/crons/my_scheduled_pipeline.yaml
annotations:
nodestream_plugin_k8s_schedule: "0 0 * * *"
perpetual:
pipelines:
- path: pipelines/perpetual/my_kafka_pipeline.yaml
annotations:
nodestream_plugin_k8s_conccurency: 1
```
### 3. Run `nodestream-plugin-k8s` to deploy your pipelines.
```bash
nodestream k8s sync --namespace my-nodestream-app-namespace
```
### 4. Check your kubernetes cluster for your deployed pipelines.
```bash
kubectl get cronjobs -n my-nodestream-app-namespace
kubectl get deployments -n my-nodestream-app-namespace
```
### 5. Update your `nodestream.yaml` file and re-run `nodestream-plugin-k8s` to update your pipelines.
```yaml
# nodestream.yaml
scopes:
crons:
pipelines:
- path: pipelines/crons/my_scheduled_pipeline.yaml
annotations:
nodestream_plugin_k8s_schedule: "30 0 * * *"
perpetual:
pipelines:
- path: pipelines/perpetual/my_kafka_pipeline.yaml
annotations:
nodestream_plugin_k8s_conccurency: 100
```
```bash
nodestream k8s sync --namespace my-nodestream-app-namespace
```
### 6. Check your kubernetes cluster for your updated pipelines.
```bash
kubectl get cronjobs -n my-nodestream-app-namespace
kubectl get deployments -n my-nodestream-app-namespace
```
### 7. Delete your pipelines.
```bash
nodestream k8s destroy --namespace my-nodestream-app-namespace
```
## Annotations
### `nodestream_plugin_k8s_schedule`
This annotation is used to schedule a cronjob. The value of this annotation should be a cron expression.
### `nodestream_plugin_k8s_concurrency`
This annotation is used to set the concurrency of a deployment. The value of this annotation should be an integer.
### `nodestream_plugin_k8s_image`
This annotation is used to set the image of a deployment or job. The value of this annotation should be a valid docker image.
Raw data
{
"_id": null,
"home_page": null,
"name": "nodestream-plugin-k8s",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.10",
"maintainer_email": null,
"keywords": null,
"author": "Zach Probst",
"author_email": "Zach_Probst@intuit.com",
"download_url": "https://files.pythonhosted.org/packages/fc/34/d478e4c22019259b6b8b75da8a1b85f109f5acbdfd0e3e4a800d95bfd71f/nodestream_plugin_k8s-0.13.0.tar.gz",
"platform": null,
"description": "# Nodestream Kubernetes Plugin \n\nNOTE: This plugin is still in development and is not ready for production use.\n\n## Overview\n\nThis plugin allows you to deploy nodestream on a Kubernete cluster. As your project evolves by adding more data pipelines, `nodstream-plugin-k8s` will be able to manage your kubernetes resources for you. \n\n## Installation\n\n```bash\npip install nodestream-plugin-k8s\n```\n\n## Usage\n\n### 1. Create a `nodestream.yaml` file and project.\n\n```yaml\n# nodestream.yaml\nscopes:\n crons: \n pipelines:\n - path: pipelines/crons/my_scheduled_pipeline.yaml\n perpetual:\n pipelines:\n - path: pipelines/perpetual/my_kafka_pipeline.yaml\n```\n\n### 2. Annotate your pipeline with `nodestream-plugin-k8s` annotations.\n\n```yaml\n# nodestream.yaml\nscopes:\n crons:\n pipelines:\n - path: pipelines/crons/my_scheduled_pipeline.yaml\n annotations:\n nodestream_plugin_k8s_schedule: \"0 0 * * *\"\n perpetual:\n pipelines:\n - path: pipelines/perpetual/my_kafka_pipeline.yaml\n annotations:\n nodestream_plugin_k8s_conccurency: 1\n```\n\n### 3. Run `nodestream-plugin-k8s` to deploy your pipelines.\n\n```bash\nnodestream k8s sync --namespace my-nodestream-app-namespace \n```\n\n\n### 4. Check your kubernetes cluster for your deployed pipelines.\n\n```bash\nkubectl get cronjobs -n my-nodestream-app-namespace\nkubectl get deployments -n my-nodestream-app-namespace\n```\n\n### 5. Update your `nodestream.yaml` file and re-run `nodestream-plugin-k8s` to update your pipelines.\n\n```yaml\n# nodestream.yaml\nscopes:\n crons:\n pipelines:\n - path: pipelines/crons/my_scheduled_pipeline.yaml\n annotations:\n nodestream_plugin_k8s_schedule: \"30 0 * * *\"\n perpetual:\n pipelines:\n - path: pipelines/perpetual/my_kafka_pipeline.yaml\n annotations:\n nodestream_plugin_k8s_conccurency: 100\n```\n\n```bash\nnodestream k8s sync --namespace my-nodestream-app-namespace \n```\n\n### 6. Check your kubernetes cluster for your updated pipelines.\n\n```bash\nkubectl get cronjobs -n my-nodestream-app-namespace\nkubectl get deployments -n my-nodestream-app-namespace\n```\n\n### 7. Delete your pipelines.\n\n```bash\nnodestream k8s destroy --namespace my-nodestream-app-namespace \n```\n\n## Annotations\n\n### `nodestream_plugin_k8s_schedule`\n\nThis annotation is used to schedule a cronjob. The value of this annotation should be a cron expression. \n\n### `nodestream_plugin_k8s_concurrency`\n\nThis annotation is used to set the concurrency of a deployment. The value of this annotation should be an integer.\n\n### `nodestream_plugin_k8s_image`\n\nThis annotation is used to set the image of a deployment or job. The value of this annotation should be a valid docker image.\n\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "A plugin for nodestream that manages k8s resources.",
"version": "0.13.0",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "6244bb8b2be289e459ae08a00741faf373f8c7624ba5552852c12ef4e25f0299",
"md5": "42c9ce0ccf2220820fd1c19a2e8adcf4",
"sha256": "34946e04c184a1a5d381925a821d66275753414913013923823354834375e428"
},
"downloads": -1,
"filename": "nodestream_plugin_k8s-0.13.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "42c9ce0ccf2220820fd1c19a2e8adcf4",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.10",
"size": 9497,
"upload_time": "2024-08-09T17:34:21",
"upload_time_iso_8601": "2024-08-09T17:34:21.517938Z",
"url": "https://files.pythonhosted.org/packages/62/44/bb8b2be289e459ae08a00741faf373f8c7624ba5552852c12ef4e25f0299/nodestream_plugin_k8s-0.13.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "fc34d478e4c22019259b6b8b75da8a1b85f109f5acbdfd0e3e4a800d95bfd71f",
"md5": "7c93a803e51ccae286334768dd1b9c0d",
"sha256": "5042bcfbca812f03ff99fa37319b7978cd6fce9097e7fc75a2415bdcbd43a1ea"
},
"downloads": -1,
"filename": "nodestream_plugin_k8s-0.13.0.tar.gz",
"has_sig": false,
"md5_digest": "7c93a803e51ccae286334768dd1b9c0d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.10",
"size": 7040,
"upload_time": "2024-08-09T17:34:23",
"upload_time_iso_8601": "2024-08-09T17:34:23.593528Z",
"url": "https://files.pythonhosted.org/packages/fc/34/d478e4c22019259b6b8b75da8a1b85f109f5acbdfd0e3e4a800d95bfd71f/nodestream_plugin_k8s-0.13.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-09 17:34:23",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "nodestream-plugin-k8s"
}
