| Name | nodestream-plugin-k8s JSON |
| Version |
0.12.0
JSON |
| download |
| home_page | None |
| Summary | A plugin for nodestream that manages k8s resources. |
| upload_time | 2024-04-05 17:35:45 |
| maintainer | None |
| docs_url | None |
| author | Zach Probst |
| requires_python | <4.0,>=3.10 |
| license | Apache 2.0 |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Nodestream Kubernetes Plugin
NOTE: This plugin is still in development and is not ready for production use.
## Overview
This plugin allows you to deploy nodestream on a Kubernete cluster. As your project evolves by adding more data pipelines, `nodstream-plugin-k8s` will be able to manage your kubernetes resources for you.
## Installation
```bash
pip install nodestream-plugin-k8s
```
## Usage
### 1. Create a `nodestream.yaml` file and project.
```yaml
# nodestream.yaml
scopes:
crons:
pipelines:
- path: pipelines/crons/my_scheduled_pipeline.yaml
perpetual:
pipelines:
- path: pipelines/perpetual/my_kafka_pipeline.yaml
```
### 2. Annotate your pipeline with `nodestream-plugin-k8s` annotations.
```yaml
# nodestream.yaml
scopes:
crons:
pipelines:
- path: pipelines/crons/my_scheduled_pipeline.yaml
annotations:
nodestream_plugin_k8s_schedule: "0 0 * * *"
perpetual:
pipelines:
- path: pipelines/perpetual/my_kafka_pipeline.yaml
annotations:
nodestream_plugin_k8s_conccurency: 1
```
### 3. Run `nodestream-plugin-k8s` to deploy your pipelines.
```bash
nodestream k8s sync --namespace my-nodestream-app-namespace
```
### 4. Check your kubernetes cluster for your deployed pipelines.
```bash
kubectl get cronjobs -n my-nodestream-app-namespace
kubectl get deployments -n my-nodestream-app-namespace
```
### 5. Update your `nodestream.yaml` file and re-run `nodestream-plugin-k8s` to update your pipelines.
```yaml
# nodestream.yaml
scopes:
crons:
pipelines:
- path: pipelines/crons/my_scheduled_pipeline.yaml
annotations:
nodestream_plugin_k8s_schedule: "30 0 * * *"
perpetual:
pipelines:
- path: pipelines/perpetual/my_kafka_pipeline.yaml
annotations:
nodestream_plugin_k8s_conccurency: 100
```
```bash
nodestream k8s sync --namespace my-nodestream-app-namespace
```
### 6. Check your kubernetes cluster for your updated pipelines.
```bash
kubectl get cronjobs -n my-nodestream-app-namespace
kubectl get deployments -n my-nodestream-app-namespace
```
### 7. Delete your pipelines.
```bash
nodestream k8s destroy --namespace my-nodestream-app-namespace
```
## Annotations
### `nodestream_plugin_k8s_schedule`
This annotation is used to schedule a cronjob. The value of this annotation should be a cron expression.
### `nodestream_plugin_k8s_concurrency`
This annotation is used to set the concurrency of a deployment. The value of this annotation should be an integer.
### `nodestream_plugin_k8s_image`
This annotation is used to set the image of a deployment or job. The value of this annotation should be a valid docker image.
Raw data
{
"_id": null,
"home_page": null,
"name": "nodestream-plugin-k8s",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.10",
"maintainer_email": null,
"keywords": null,
"author": "Zach Probst",
"author_email": "Zach_Probst@intuit.com",
"download_url": "https://files.pythonhosted.org/packages/2c/1b/4d116dfabcc72588f572cc95f8a5e90f901dfc7709b2e30f74a19e751e2d/nodestream_plugin_k8s-0.12.0.tar.gz",
"platform": null,
"description": "# Nodestream Kubernetes Plugin \n\nNOTE: This plugin is still in development and is not ready for production use.\n\n## Overview\n\nThis plugin allows you to deploy nodestream on a Kubernete cluster. As your project evolves by adding more data pipelines, `nodstream-plugin-k8s` will be able to manage your kubernetes resources for you. \n\n## Installation\n\n```bash\npip install nodestream-plugin-k8s\n```\n\n## Usage\n\n### 1. Create a `nodestream.yaml` file and project.\n\n```yaml\n# nodestream.yaml\nscopes:\n crons: \n pipelines:\n - path: pipelines/crons/my_scheduled_pipeline.yaml\n perpetual:\n pipelines:\n - path: pipelines/perpetual/my_kafka_pipeline.yaml\n```\n\n### 2. Annotate your pipeline with `nodestream-plugin-k8s` annotations.\n\n```yaml\n# nodestream.yaml\nscopes:\n crons:\n pipelines:\n - path: pipelines/crons/my_scheduled_pipeline.yaml\n annotations:\n nodestream_plugin_k8s_schedule: \"0 0 * * *\"\n perpetual:\n pipelines:\n - path: pipelines/perpetual/my_kafka_pipeline.yaml\n annotations:\n nodestream_plugin_k8s_conccurency: 1\n```\n\n### 3. Run `nodestream-plugin-k8s` to deploy your pipelines.\n\n```bash\nnodestream k8s sync --namespace my-nodestream-app-namespace \n```\n\n\n### 4. Check your kubernetes cluster for your deployed pipelines.\n\n```bash\nkubectl get cronjobs -n my-nodestream-app-namespace\nkubectl get deployments -n my-nodestream-app-namespace\n```\n\n### 5. Update your `nodestream.yaml` file and re-run `nodestream-plugin-k8s` to update your pipelines.\n\n```yaml\n# nodestream.yaml\nscopes:\n crons:\n pipelines:\n - path: pipelines/crons/my_scheduled_pipeline.yaml\n annotations:\n nodestream_plugin_k8s_schedule: \"30 0 * * *\"\n perpetual:\n pipelines:\n - path: pipelines/perpetual/my_kafka_pipeline.yaml\n annotations:\n nodestream_plugin_k8s_conccurency: 100\n```\n\n```bash\nnodestream k8s sync --namespace my-nodestream-app-namespace \n```\n\n### 6. Check your kubernetes cluster for your updated pipelines.\n\n```bash\nkubectl get cronjobs -n my-nodestream-app-namespace\nkubectl get deployments -n my-nodestream-app-namespace\n```\n\n### 7. Delete your pipelines.\n\n```bash\nnodestream k8s destroy --namespace my-nodestream-app-namespace \n```\n\n## Annotations\n\n### `nodestream_plugin_k8s_schedule`\n\nThis annotation is used to schedule a cronjob. The value of this annotation should be a cron expression. \n\n### `nodestream_plugin_k8s_concurrency`\n\nThis annotation is used to set the concurrency of a deployment. The value of this annotation should be an integer.\n\n### `nodestream_plugin_k8s_image`\n\nThis annotation is used to set the image of a deployment or job. The value of this annotation should be a valid docker image.\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "A plugin for nodestream that manages k8s resources.",
"version": "0.12.0",
"project_urls": null,
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "1b89814c6ff7e64b1d2da5dced31f7fff0a5d194993b08b65239057f71f329e3",
"md5": "8d870315347e755b7214838795db795e",
"sha256": "dd8551e3ccc8f85d8ed0f1964a7ec6b14cab699753f129f1e24bc3f0000156c8"
},
"downloads": -1,
"filename": "nodestream_plugin_k8s-0.12.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8d870315347e755b7214838795db795e",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.10",
"size": 9496,
"upload_time": "2024-04-05T17:35:44",
"upload_time_iso_8601": "2024-04-05T17:35:44.222066Z",
"url": "https://files.pythonhosted.org/packages/1b/89/814c6ff7e64b1d2da5dced31f7fff0a5d194993b08b65239057f71f329e3/nodestream_plugin_k8s-0.12.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "2c1b4d116dfabcc72588f572cc95f8a5e90f901dfc7709b2e30f74a19e751e2d",
"md5": "a96c404c8291ae534f94adf8fd35cb63",
"sha256": "e6f95fdbf8004852b3142a618e9b48090d0423069c3c636314b1bf12179a7b15"
},
"downloads": -1,
"filename": "nodestream_plugin_k8s-0.12.0.tar.gz",
"has_sig": false,
"md5_digest": "a96c404c8291ae534f94adf8fd35cb63",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.10",
"size": 7227,
"upload_time": "2024-04-05T17:35:45",
"upload_time_iso_8601": "2024-04-05T17:35:45.825860Z",
"url": "https://files.pythonhosted.org/packages/2c/1b/4d116dfabcc72588f572cc95f8a5e90f901dfc7709b2e30f74a19e751e2d/nodestream_plugin_k8s-0.12.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-05 17:35:45",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "nodestream-plugin-k8s"
}
