| Name | prefect-shell JSON |
| Version |
0.2.3
JSON |
| download |
| home_page | None |
| Summary | Prefect integrations for interacting with shell commands. |
| upload_time | 2024-04-25 15:16:10 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.8 |
| license | Apache License 2.0 |
| keywords |
prefect
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Integrating shell commands into your dataflow with `prefect-shell`
<p align="center">
<a href="https://pypi.python.org/pypi/prefect-shell/" alt="PyPI version">
<img alt="PyPI" src="https://img.shields.io/pypi/v/prefect-shell?color=0052FF&labelColor=090422"></a>
<a href="https://pepy.tech/badge/prefect-shell/" alt="Downloads">
<img src="https://img.shields.io/pypi/dm/prefect-shell?color=0052FF&labelColor=090422" /></a>
</p>
Visit the full docs [here](https://PrefectHQ.github.io/prefect-shell) to see additional examples and the API reference.
The prefect-shell collection makes it easy to execute shell commands in your Prefect flows. Check out the examples below to get started!
## Getting Started
### Integrate with Prefect flows
With prefect-shell, you can bring your trusty shell commands (and/or scripts) straight into the Prefect flow party, complete with awesome Prefect logging.
No more separate logs, just seamless integration. Let's get the shell-abration started!
```python
from prefect import flow
from datetime import datetime
from prefect_shell import ShellOperation
@flow
def download_data():
today = datetime.today().strftime("%Y%m%d")
# for short running operations, you can use the `run` method
# which automatically manages the context
ShellOperation(
commands=[
"mkdir -p data",
"mkdir -p data/${today}"
],
env={"today": today}
).run()
# for long running operations, you can use a context manager
with ShellOperation(
commands=[
"curl -O https://masie_web.apps.nsidc.org/pub/DATASETS/NOAA/G02135/north/daily/data/N_seaice_extent_daily_v3.0.csv",
],
working_dir=f"data/{today}",
) as download_csv_operation:
# trigger runs the process in the background
download_csv_process = download_csv_operation.trigger()
# then do other things here in the meantime, like download another file
...
# when you're ready, wait for the process to finish
download_csv_process.wait_for_completion()
# if you'd like to get the output lines, you can use the `fetch_result` method
output_lines = download_csv_process.fetch_result()
download_data()
```
Outputs:
```bash
14:48:16.550 | INFO | prefect.engine - Created flow run 'tentacled-chachalaca' for flow 'download-data'
14:48:17.977 | INFO | Flow run 'tentacled-chachalaca' - PID 19360 triggered with 2 commands running inside the '.' directory.
14:48:17.987 | INFO | Flow run 'tentacled-chachalaca' - PID 19360 completed with return code 0.
14:48:17.994 | INFO | Flow run 'tentacled-chachalaca' - PID 19363 triggered with 1 commands running inside the PosixPath('data/20230201') directory.
14:48:18.009 | INFO | Flow run 'tentacled-chachalaca' - PID 19363 stream output:
% Total % Received % Xferd Average Speed Time Time Time Current
Dl
14:48:18.010 | INFO | Flow run 'tentacled-chachalaca' - PID 19363 stream output:
oad Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
14:48:18.840 | INFO | Flow run 'tentacled-chachalaca' - PID 19363 stream output:
11 1630k 11 192k 0 0 229k 0 0:00:07 --:--:-- 0:00:07 231k
14:48:19.839 | INFO | Flow run 'tentacled-chachalaca' - PID 19363 stream output:
83 1630k 83 1368k 0 0 745k 0 0:00:02 0:00:01 0:00:01 747k
14:48:19.993 | INFO | Flow run 'tentacled-chachalaca' - PID 19363 stream output:
100 1630k 100 1630k 0 0 819k 0 0
14:48:19.994 | INFO | Flow run 'tentacled-chachalaca' - PID 19363 stream output:
:00:01 0:00:01 --:--:-- 821k
14:48:19.996 | INFO | Flow run 'tentacled-chachalaca' - PID 19363 completed with return code 0.
14:48:19.998 | INFO | Flow run 'tentacled-chachalaca' - Successfully closed all open processes.
14:48:20.203 | INFO | Flow run 'tentacled-chachalaca' - Finished in state Completed()
```
!!! info "Utilize Previously Saved Blocks"
You can save commands within a `ShellOperation` block, then reuse them across multiple flows, or even plain Python scripts.
Save the block with desired commands:
```python
from prefect_shell import ShellOperation
ping_op = ShellOperation(commands=["ping -t 1 prefect.io"])
ping_op.save("block-name")
```
Load the saved block:
```python
from prefect_shell import ShellOperation
ping_op = ShellOperation.load("block-name")
```
To [view and edit the blocks](https://orion-docs.prefect.io/ui/blocks/) on Prefect UI:
```bash
prefect block register -m prefect_shell
```
## Resources
For more tips on how to use tasks and flows in a Collection, check out [Using Collections](https://orion-docs.prefect.io/collections/usage/)!
### Installation
Install `prefect-shell` with `pip`:
```bash
pip install -U prefect-shell
```
A list of available blocks in `prefect-shell` and their setup instructions can be found [here](https://PrefectHQ.github.io/prefect-shell/blocks_catalog).
Requires an installation of Python 3.7+.
We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.
These tasks are designed to work with Prefect 2. For more information about how to use Prefect, please refer to the [Prefect documentation](https://orion-docs.prefect.io/).
### Feedback
If you encounter any bugs while using `prefect-shell`, feel free to open an issue in the [prefect-shell](https://github.com/PrefectHQ/prefect-shell) repository.
If you have any questions or issues while using `prefect-shell`, you can find help in either the [Prefect Discourse forum](https://discourse.prefect.io/) or the [Prefect Slack community](https://prefect.io/slack).
Feel free to star or watch [`prefect-shell`](https://github.com/PrefectHQ/prefect-shell) for updates too!
### Contributing
If you'd like to help contribute to fix an issue or add a feature to `prefect-shell`, please [propose changes through a pull request from a fork of the repository](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork).
Here are the steps:
1. [Fork the repository](https://docs.github.com/en/get-started/quickstart/fork-a-repo#forking-a-repository)
2. [Clone the forked repository](https://docs.github.com/en/get-started/quickstart/fork-a-repo#cloning-your-forked-repository)
3. Install the repository and its dependencies:
```
pip install -e ".[dev]"
```
4. Make desired changes
5. Add tests
6. Insert an entry to [CHANGELOG.md](https://github.com/PrefectHQ/prefect-shell/blob/main/CHANGELOG.md)
7. Install `pre-commit` to perform quality checks prior to commit:
```
pre-commit install
```
8. `git commit`, `git push`, and create a pull request
Raw data
{
"_id": null,
"home_page": null,
"name": "prefect-shell",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "prefect",
"author": null,
"author_email": "\"Prefect Technologies, Inc.\" <help@prefect.io>",
"download_url": "https://files.pythonhosted.org/packages/91/7d/3e886e2f4a1a9dbd09633edf861dc77bbd524aca83b8a63a141439e16e5e/prefect_shell-0.2.3.tar.gz",
"platform": null,
"description": "# Integrating shell commands into your dataflow with `prefect-shell`\n\n<p align=\"center\">\n <a href=\"https://pypi.python.org/pypi/prefect-shell/\" alt=\"PyPI version\">\n <img alt=\"PyPI\" src=\"https://img.shields.io/pypi/v/prefect-shell?color=0052FF&labelColor=090422\"></a>\n <a href=\"https://pepy.tech/badge/prefect-shell/\" alt=\"Downloads\">\n <img src=\"https://img.shields.io/pypi/dm/prefect-shell?color=0052FF&labelColor=090422\" /></a>\n</p>\n\nVisit the full docs [here](https://PrefectHQ.github.io/prefect-shell) to see additional examples and the API reference.\n\nThe prefect-shell collection makes it easy to execute shell commands in your Prefect flows. Check out the examples below to get started!\n\n## Getting Started\n\n### Integrate with Prefect flows\n\nWith prefect-shell, you can bring your trusty shell commands (and/or scripts) straight into the Prefect flow party, complete with awesome Prefect logging.\n\nNo more separate logs, just seamless integration. Let's get the shell-abration started!\n\n```python\nfrom prefect import flow\nfrom datetime import datetime\nfrom prefect_shell import ShellOperation\n\n@flow\ndef download_data():\n today = datetime.today().strftime(\"%Y%m%d\")\n\n # for short running operations, you can use the `run` method\n # which automatically manages the context\n ShellOperation(\n commands=[\n \"mkdir -p data\",\n \"mkdir -p data/${today}\"\n ],\n env={\"today\": today}\n ).run()\n\n # for long running operations, you can use a context manager\n with ShellOperation(\n commands=[\n \"curl -O https://masie_web.apps.nsidc.org/pub/DATASETS/NOAA/G02135/north/daily/data/N_seaice_extent_daily_v3.0.csv\",\n ],\n working_dir=f\"data/{today}\",\n ) as download_csv_operation:\n\n # trigger runs the process in the background\n download_csv_process = download_csv_operation.trigger()\n\n # then do other things here in the meantime, like download another file\n ...\n\n # when you're ready, wait for the process to finish\n download_csv_process.wait_for_completion()\n\n # if you'd like to get the output lines, you can use the `fetch_result` method\n output_lines = download_csv_process.fetch_result()\n\ndownload_data()\n```\n\nOutputs:\n```bash\n14:48:16.550 | INFO | prefect.engine - Created flow run 'tentacled-chachalaca' for flow 'download-data'\n14:48:17.977 | INFO | Flow run 'tentacled-chachalaca' - PID 19360 triggered with 2 commands running inside the '.' directory.\n14:48:17.987 | INFO | Flow run 'tentacled-chachalaca' - PID 19360 completed with return code 0.\n14:48:17.994 | INFO | Flow run 'tentacled-chachalaca' - PID 19363 triggered with 1 commands running inside the PosixPath('data/20230201') directory.\n14:48:18.009 | INFO | Flow run 'tentacled-chachalaca' - PID 19363 stream output:\n % Total % Received % Xferd Average Speed Time Time Time Current\n Dl\n14:48:18.010 | INFO | Flow run 'tentacled-chachalaca' - PID 19363 stream output:\noad Upload Total Spent Left Speed\n 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\n14:48:18.840 | INFO | Flow run 'tentacled-chachalaca' - PID 19363 stream output:\n 11 1630k 11 192k 0 0 229k 0 0:00:07 --:--:-- 0:00:07 231k\n14:48:19.839 | INFO | Flow run 'tentacled-chachalaca' - PID 19363 stream output:\n 83 1630k 83 1368k 0 0 745k 0 0:00:02 0:00:01 0:00:01 747k\n14:48:19.993 | INFO | Flow run 'tentacled-chachalaca' - PID 19363 stream output:\n100 1630k 100 1630k 0 0 819k 0 0\n14:48:19.994 | INFO | Flow run 'tentacled-chachalaca' - PID 19363 stream output:\n:00:01 0:00:01 --:--:-- 821k\n14:48:19.996 | INFO | Flow run 'tentacled-chachalaca' - PID 19363 completed with return code 0.\n14:48:19.998 | INFO | Flow run 'tentacled-chachalaca' - Successfully closed all open processes.\n14:48:20.203 | INFO | Flow run 'tentacled-chachalaca' - Finished in state Completed()\n```\n\n!!! info \"Utilize Previously Saved Blocks\"\n\n You can save commands within a `ShellOperation` block, then reuse them across multiple flows, or even plain Python scripts.\n \n Save the block with desired commands:\n\n ```python\n from prefect_shell import ShellOperation\n\n ping_op = ShellOperation(commands=[\"ping -t 1 prefect.io\"])\n ping_op.save(\"block-name\")\n ```\n\n Load the saved block:\n\n ```python\n from prefect_shell import ShellOperation\n\n ping_op = ShellOperation.load(\"block-name\")\n ```\n\n To [view and edit the blocks](https://orion-docs.prefect.io/ui/blocks/) on Prefect UI:\n\n ```bash\n prefect block register -m prefect_shell\n ```\n\n## Resources\n\nFor more tips on how to use tasks and flows in a Collection, check out [Using Collections](https://orion-docs.prefect.io/collections/usage/)!\n\n### Installation\n\nInstall `prefect-shell` with `pip`:\n\n```bash\npip install -U prefect-shell\n```\n\nA list of available blocks in `prefect-shell` and their setup instructions can be found [here](https://PrefectHQ.github.io/prefect-shell/blocks_catalog).\n\nRequires an installation of Python 3.7+.\n\nWe recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.\n\nThese tasks are designed to work with Prefect 2. For more information about how to use Prefect, please refer to the [Prefect documentation](https://orion-docs.prefect.io/).\n\n### Feedback\n\nIf you encounter any bugs while using `prefect-shell`, feel free to open an issue in the [prefect-shell](https://github.com/PrefectHQ/prefect-shell) repository.\n\nIf you have any questions or issues while using `prefect-shell`, you can find help in either the [Prefect Discourse forum](https://discourse.prefect.io/) or the [Prefect Slack community](https://prefect.io/slack).\n\nFeel free to star or watch [`prefect-shell`](https://github.com/PrefectHQ/prefect-shell) for updates too!\n \n### Contributing\n \nIf you'd like to help contribute to fix an issue or add a feature to `prefect-shell`, please [propose changes through a pull request from a fork of the repository](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork).\n \nHere are the steps:\n\n1. [Fork the repository](https://docs.github.com/en/get-started/quickstart/fork-a-repo#forking-a-repository)\n2. [Clone the forked repository](https://docs.github.com/en/get-started/quickstart/fork-a-repo#cloning-your-forked-repository)\n3. Install the repository and its dependencies:\n```\npip install -e \".[dev]\"\n```\n4. Make desired changes\n5. Add tests\n6. Insert an entry to [CHANGELOG.md](https://github.com/PrefectHQ/prefect-shell/blob/main/CHANGELOG.md)\n7. Install `pre-commit` to perform quality checks prior to commit:\n```\npre-commit install\n```\n8. `git commit`, `git push`, and create a pull request\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "Prefect integrations for interacting with shell commands.",
"version": "0.2.3",
"project_urls": {
"Homepage": "https://github.com/PrefectHQ/prefect/tree/main/src/integrations/prefect-shell"
},
"split_keywords": [
"prefect"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f84206c6f335e471074802ed2277ffd6b26b441f019cbcf2743eb011721d431e",
"md5": "1a73d1a2ea60c6d75b911eee29e72039",
"sha256": "da4e62bc209e769ddc62e6f23f11eeacc99e8d8dadd5b150348d8cdca6ec2256"
},
"downloads": -1,
"filename": "prefect_shell-0.2.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1a73d1a2ea60c6d75b911eee29e72039",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 9267,
"upload_time": "2024-04-25T15:16:07",
"upload_time_iso_8601": "2024-04-25T15:16:07.361128Z",
"url": "https://files.pythonhosted.org/packages/f8/42/06c6f335e471074802ed2277ffd6b26b441f019cbcf2743eb011721d431e/prefect_shell-0.2.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "917d3e886e2f4a1a9dbd09633edf861dc77bbd524aca83b8a63a141439e16e5e",
"md5": "e8a3e1046840bef6fb8e6867fbc1319a",
"sha256": "7bbddb85a8ab7e5b5b6c6f5e29bc0bafbc64c75a5bd14cff90a75b8421900b90"
},
"downloads": -1,
"filename": "prefect_shell-0.2.3.tar.gz",
"has_sig": false,
"md5_digest": "e8a3e1046840bef6fb8e6867fbc1319a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 14728,
"upload_time": "2024-04-25T15:16:10",
"upload_time_iso_8601": "2024-04-25T15:16:10.467293Z",
"url": "https://files.pythonhosted.org/packages/91/7d/3e886e2f4a1a9dbd09633edf861dc77bbd524aca83b8a63a141439e16e5e/prefect_shell-0.2.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-25 15:16:10",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "PrefectHQ",
"github_project": "prefect",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"circle": true,
"requirements": [],
"lcname": "prefect-shell"
}
