| Name | py-flowgrid JSON |
| Version |
0.1.0
JSON |
| download |
| home_page | https://github.com/fernaper/flowgrid |
| Summary | A simplified, powerful interface for distributed task management in Python, built on Celery. |
| upload_time | 2024-09-03 16:41:03 |
| maintainer | None |
| docs_url | None |
| author | Fernando Pérez |
| requires_python | None |
| license | MIT |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# FlowGrid
**FlowGrid** is a Python library designed to improve parallelization across multiple machines with a powerful and user-friendly interface, allowing you to focus on your application logic without worrying about the complexities of task distribution and management.
## Motivation
The primary goal of **FlowGrid** is to provide an easy-to-use interface for parallelizing Python tasks across multiple workers.
In Python, a common design pattern is to launch asynchronous tasks while instantly responding to the user—*FastAPI*, for instance, addresses this with `background_tasks`, though it acknowledges this method may not be the most reliable, especially for long-running tasks.
**FlowGrid** abstracts the complexity of task management, progress tracking, and task cancellation, making it easier to build scalable and robust distributed systems. This library is also designed to seamlessly integrate with popular frameworks like *FastAPI*, *Flask*, and *Django*, solving common concurrency challenges effortlessly.
While other solutions like **Celery** (which FlowGrid uses under the hood) are available, they often require more configuration and are more challenging to use. FlowGrid simplifies the process while maintaining the full power of Celery, with additional enhancements such as native task state checking and improved handling of task cancellations, even for tasks in progress.
## Installation
> **Note:** FlowGrid will soon be available on PyPI. You will be able to install it using pip:
> ```bash
> pip install flowgrid
> ```
> FlowGrid requires Python 3.7 or higher.
## Basic Usage
### Defining and Launching Tasks
FlowGrid simplifies task management in Celery by allowing you to define tasks using decorators and manage them seamlessly. Here's an example:
```python
import time
from flowgrid import FlowGrid
fg = FlowGrid()
@fg.task
def add(x: float, y: float) -> float:
# Simulate a long running task
time.sleep(10)
return x + y
def main():
task = add(1, 2)
# Task id is none because it is not launched
print('TASK:', task.task_id)
# You can explicitly launch the task or let the
# wait function do it for you
# task = fg.launch(task) # Can be uncommented
# At this point the task id is available
# print('TASK:', task.task_id) # Can be uncommented
response = fg.wait(task)
print('RESPONSE:', response)
if __name__ == '__main__':
main()
```
### Real-Time Progress Updates
FlowGrid supports real-time progress tracking for your tasks. You can update and monitor the progress easily:
```python
import time
from flowgrid import FlowGrid
fg = FlowGrid()
@fg.task
def add_multiple(x: float, y: float, times: int = 10) -> float:
response = x
for i in range(times):
time.sleep(1)
fg.update(progress=i, total=times, percent=100*i/times)
response += y
return response
def main():
task = add_multiple(10, 5, times=5)
response = fg.wait(task) # Expected: 10 + 5*5 = 35
print('RESPONSE:', response)
if __name__ == '__main__':
main()
```
### Task Cancellation
FlowGrid allows you to cancel tasks, either forcefully or gracefully:
#### Forceful Cancellation
```python
import time
from flowgrid import FlowGrid
fg = FlowGrid()
@fg.task
def add_multiple(x: float, y: float, times: int = 10) -> float:
response = x
for i in range(times):
time.sleep(1)
fg.update(progress=i, total=times, percent=100*i/times)
response += y
return response
def main():
task = fg.launch(add_multiple(10, 5, times=5))
time.sleep(3)
# Simulating a user cancelling the task
print('CANCELLING TASK')
# force=True will terminate the task immediately
fg.revoke(task, force=True)
if __name__ == '__main__':
main()
```
#### Graceful Cancellation
```python
import time
from flowgrid import FlowGrid
fg = FlowGrid()
@fg.task
def add_multiple(x: float, y: float, times: int = 10) -> float:
response = x
for i in range(times):
# Check for revocation and stop if needed
if fg.is_revoked():
print('CANCELLED')
return
time.sleep(1)
fg.update(progress=i, total=times, percent=100*i/times)
response += y
return response
def main():
task = fg.launch(add_multiple(10, 5, times=5))
print('TASK:', task.task_id)
time.sleep(3)
# Simulating a user cancelling the task
print('CANCELLING TASK')
# Graceful cancellation
fg.revoke(task)
if __name__ == '__main__':
main()
```
## Worker Management
FlowGrid workers can be launched with the command:
```bash
flowgrid worker --app <path-to-your-flowgrid-class>
```
You can also use the `-A` shorthand for `--app`:
```bash
flowgrid worker -A <path-to-your-flowgrid-class>
```
For example in order to prepare workers to launch the initial example from this repository, you can run:
```bash
flowgrid worker -A examples.01-base.fg
```
The `.fg` must be included because it is the variable name of my FlowGrid instance in the file `examples/01-base.py`.
To view all available options, use:
```bash
flowgrid worker -h
```
Some useful options include:
- **Concurrency**: Control the number of worker processes with `--concurrency` or `-c`.
- **Log Level**: Set the logging level with `--loglevel` or `-l`.
## Example
Here's a basic example that demonstrates how to define and launch tasks using FlowGrid:
```python
import time
from flowgrid import FlowGrid
fg = FlowGrid()
@fg.task
def add(x: float, y: float) -> float:
time.sleep(10)
return x + y
def main():
task = add(1, 2)
print('TASK:', task.task_id)
response = fg.wait(task)
print('RESPONSE:', response)
if __name__ == '__main__':
main()
```
## Future Enhancements
- **PyPI Release**: Soon, FlowGrid will be available for installation via pip.
- **Extended Documentation**: More detailed documentation and examples will be added as the project evolves.
- **Support chaining tasks without waiting**: Currently, you have to wait for a task to finish before chaining another task. You can check the example `examples/06-chaining.py` to see how to chain tasks. With the next solution it will look in the exact same way but if you use `fg.launch` the response will be instant and you can return the task to the user from the beginning.
- **Support task sequence without relationship**: Right now you can only chain tasks just by including them as paramteres, but what if the response of a task is not needed for the next one. We will create an interface called: `fg.sequence` that will allow you to define a sequence of tasks that will be executed in order.
## License
FlowGrid is licensed under the MIT License. See `LICENSE` for more information.
## Contributing
Contributions are welcome! Please fork the repository and submit a pull request to contribute to FlowGrid.
## Contact
For any questions or inquiries, please contact the maintainers via [your email/contact information].
Raw data
{
"_id": null,
"home_page": "https://github.com/fernaper/flowgrid",
"name": "py-flowgrid",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": null,
"author": "Fernando P\u00e9rez",
"author_email": "fernaperg@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/e9/41/0eccca26095fca89af58b5f0a61181a85cba4e52d86cb735b78343437aeb/py-flowgrid-0.1.0.tar.gz",
"platform": null,
"description": "\n# FlowGrid\n\n**FlowGrid** is a Python library designed to improve parallelization across multiple machines with a powerful and user-friendly interface, allowing you to focus on your application logic without worrying about the complexities of task distribution and management.\n\n## Motivation\n\nThe primary goal of **FlowGrid** is to provide an easy-to-use interface for parallelizing Python tasks across multiple workers.\n\nIn Python, a common design pattern is to launch asynchronous tasks while instantly responding to the user\u2014*FastAPI*, for instance, addresses this with `background_tasks`, though it acknowledges this method may not be the most reliable, especially for long-running tasks.\n\n**FlowGrid** abstracts the complexity of task management, progress tracking, and task cancellation, making it easier to build scalable and robust distributed systems. This library is also designed to seamlessly integrate with popular frameworks like *FastAPI*, *Flask*, and *Django*, solving common concurrency challenges effortlessly.\n\nWhile other solutions like **Celery** (which FlowGrid uses under the hood) are available, they often require more configuration and are more challenging to use. FlowGrid simplifies the process while maintaining the full power of Celery, with additional enhancements such as native task state checking and improved handling of task cancellations, even for tasks in progress.\n\n## Installation\n\n> **Note:** FlowGrid will soon be available on PyPI. You will be able to install it using pip:\n> ```bash\n> pip install flowgrid\n> ```\n> FlowGrid requires Python 3.7 or higher.\n\n## Basic Usage\n\n### Defining and Launching Tasks\n\nFlowGrid simplifies task management in Celery by allowing you to define tasks using decorators and manage them seamlessly. Here's an example:\n\n```python\nimport time\nfrom flowgrid import FlowGrid\n\nfg = FlowGrid()\n\n@fg.task\ndef add(x: float, y: float) -> float:\n # Simulate a long running task\n time.sleep(10)\n return x + y\n\ndef main():\n task = add(1, 2)\n # Task id is none because it is not launched\n print('TASK:', task.task_id)\n\n # You can explicitly launch the task or let the\n # wait function do it for you\n # task = fg.launch(task) # Can be uncommented\n\n # At this point the task id is available\n # print('TASK:', task.task_id) # Can be uncommented\n response = fg.wait(task)\n print('RESPONSE:', response)\n\nif __name__ == '__main__':\n main()\n```\n\n### Real-Time Progress Updates\n\nFlowGrid supports real-time progress tracking for your tasks. You can update and monitor the progress easily:\n\n```python\nimport time\nfrom flowgrid import FlowGrid\n\nfg = FlowGrid()\n\n@fg.task\ndef add_multiple(x: float, y: float, times: int = 10) -> float:\n response = x\n for i in range(times):\n time.sleep(1)\n fg.update(progress=i, total=times, percent=100*i/times)\n response += y\n return response\n\ndef main():\n task = add_multiple(10, 5, times=5)\n response = fg.wait(task) # Expected: 10 + 5*5 = 35\n print('RESPONSE:', response)\n\nif __name__ == '__main__':\n main()\n```\n\n### Task Cancellation\n\nFlowGrid allows you to cancel tasks, either forcefully or gracefully:\n\n#### Forceful Cancellation\n\n```python\nimport time\nfrom flowgrid import FlowGrid\n\nfg = FlowGrid()\n\n@fg.task\ndef add_multiple(x: float, y: float, times: int = 10) -> float:\n response = x\n for i in range(times):\n time.sleep(1)\n fg.update(progress=i, total=times, percent=100*i/times)\n response += y\n return response\n\ndef main():\n task = fg.launch(add_multiple(10, 5, times=5))\n\n time.sleep(3)\n\n # Simulating a user cancelling the task\n print('CANCELLING TASK')\n # force=True will terminate the task immediately\n fg.revoke(task, force=True)\n\nif __name__ == '__main__':\n main()\n```\n\n#### Graceful Cancellation\n\n```python\nimport time\nfrom flowgrid import FlowGrid\n\nfg = FlowGrid()\n\n@fg.task\ndef add_multiple(x: float, y: float, times: int = 10) -> float:\n response = x\n for i in range(times):\n # Check for revocation and stop if needed\n if fg.is_revoked():\n print('CANCELLED')\n return\n time.sleep(1)\n fg.update(progress=i, total=times, percent=100*i/times)\n response += y\n return response\n\ndef main():\n task = fg.launch(add_multiple(10, 5, times=5))\n print('TASK:', task.task_id)\n\n time.sleep(3)\n\n # Simulating a user cancelling the task\n print('CANCELLING TASK')\n # Graceful cancellation\n fg.revoke(task)\n\nif __name__ == '__main__':\n main()\n```\n\n## Worker Management\n\nFlowGrid workers can be launched with the command:\n\n```bash\nflowgrid worker --app <path-to-your-flowgrid-class>\n```\n\nYou can also use the `-A` shorthand for `--app`:\n\n```bash\nflowgrid worker -A <path-to-your-flowgrid-class>\n```\n\nFor example in order to prepare workers to launch the initial example from this repository, you can run:\n\n```bash\nflowgrid worker -A examples.01-base.fg\n```\n\nThe `.fg` must be included because it is the variable name of my FlowGrid instance in the file `examples/01-base.py`.\n\n\nTo view all available options, use:\n\n```bash\nflowgrid worker -h\n```\n\nSome useful options include:\n\n- **Concurrency**: Control the number of worker processes with `--concurrency` or `-c`.\n- **Log Level**: Set the logging level with `--loglevel` or `-l`.\n\n## Example\n\nHere's a basic example that demonstrates how to define and launch tasks using FlowGrid:\n\n```python\nimport time\nfrom flowgrid import FlowGrid\n\nfg = FlowGrid()\n\n@fg.task\ndef add(x: float, y: float) -> float:\n time.sleep(10)\n return x + y\n\ndef main():\n task = add(1, 2)\n print('TASK:', task.task_id)\n response = fg.wait(task)\n print('RESPONSE:', response)\n\nif __name__ == '__main__':\n main()\n```\n\n## Future Enhancements\n\n- **PyPI Release**: Soon, FlowGrid will be available for installation via pip.\n- **Extended Documentation**: More detailed documentation and examples will be added as the project evolves.\n- **Support chaining tasks without waiting**: Currently, you have to wait for a task to finish before chaining another task. You can check the example `examples/06-chaining.py` to see how to chain tasks. With the next solution it will look in the exact same way but if you use `fg.launch` the response will be instant and you can return the task to the user from the beginning.\n- **Support task sequence without relationship**: Right now you can only chain tasks just by including them as paramteres, but what if the response of a task is not needed for the next one. We will create an interface called: `fg.sequence` that will allow you to define a sequence of tasks that will be executed in order.\n\n## License\n\nFlowGrid is licensed under the MIT License. See `LICENSE` for more information.\n\n## Contributing\n\nContributions are welcome! Please fork the repository and submit a pull request to contribute to FlowGrid.\n\n## Contact\n\nFor any questions or inquiries, please contact the maintainers via [your email/contact information].\n\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "A simplified, powerful interface for distributed task management in Python, built on Celery.",
"version": "0.1.0",
"project_urls": {
"Homepage": "https://github.com/fernaper/flowgrid"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "73ec20429c5dca3969c711dc58aa0cef6a8e72ce58e12e776549f8b9fc493096",
"md5": "05b822ffbe7dfa8a268e5818bd0c605c",
"sha256": "cba8823ec5102a3933d2736f99a71b1a8177020184e81b4c2a662f1aa568108e"
},
"downloads": -1,
"filename": "py_flowgrid-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "05b822ffbe7dfa8a268e5818bd0c605c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 13488,
"upload_time": "2024-09-03T16:41:01",
"upload_time_iso_8601": "2024-09-03T16:41:01.660349Z",
"url": "https://files.pythonhosted.org/packages/73/ec/20429c5dca3969c711dc58aa0cef6a8e72ce58e12e776549f8b9fc493096/py_flowgrid-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e9410eccca26095fca89af58b5f0a61181a85cba4e52d86cb735b78343437aeb",
"md5": "7225ff578c1cbbf638ed0865d3f98a75",
"sha256": "160ac2f83227a15d50684a93b7415412bbaf15faed947139681f280732912c14"
},
"downloads": -1,
"filename": "py-flowgrid-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "7225ff578c1cbbf638ed0865d3f98a75",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 12070,
"upload_time": "2024-09-03T16:41:03",
"upload_time_iso_8601": "2024-09-03T16:41:03.074657Z",
"url": "https://files.pythonhosted.org/packages/e9/41/0eccca26095fca89af58b5f0a61181a85cba4e52d86cb735b78343437aeb/py-flowgrid-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-03 16:41:03",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "fernaper",
"github_project": "flowgrid",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "py-flowgrid"
}
