| Name | CroFlow JSON |
| Version |
0.0.1
JSON |
| download |
| home_page | None |
| Summary | CroFlow is a Python library for running asynchronous coroutines concurrently, yielding results as soon as they are available. It supports both single-threaded and multi-threaded execution, offering flexibility for efficientlymanaging complex asynchronous tasks. handling exceptions and timeouts to optimize performance and manage complex workflows. |
| upload_time | 2024-08-05 02:49:06 |
| maintainer | None |
| docs_url | None |
| author | Curtidor |
| requires_python | None |
| license | None |
| keywords |
croflow
asynchronous
threading
parallel execution
coroutines
task management
performance optimization
python 3
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# CroFlow
CroFlow is a Python library for running asynchronous coroutines concurrently, yielding results as soon as they are available. It supports both single-threaded and multi-threaded execution, offering flexibility for efficiently managing complex asynchronous tasks.
## Why Use CroFlow?
* **Instant Results:** CroFlow delivers results as soon as they become available, allowing your application to process and respond to data without waiting for all tasks to complete. This can significantly improve the responsiveness and user experience of applications that rely on real-time data processing.
* **Ideal for High-Performance Applications:** Perfect for scenarios where rapid execution and result retrieval are crucial, such as in real-time data processing, streaming applications, or services requiring quick response times.
* **Versatile Use Cases:** Whether you're building web servers to web scrappers, data pipelines, or other concurrent systems, CroFlow helps manage complex asynchronous tasks efficiently. Its flexibility makes it suitable for both small-scale projects and large, distributed systems where timely results are essential.
## Features:
* **Concurrent Execution:** Run multiple coroutines concurrently in the same thread or across various threads.
* **Timeout Handling:** Specify a timeout for coroutine execution to prevent tasks from running indefinitely.
* **Exception Handling:** Choose whether to return exceptions as results or handle them silently.
* **Debugging Support:** Option to print stack traces for exceptions that occur during coroutine execution.
## Basic Usage
Here's a simple example of how to use CroFlow to run multiple coroutines concurrently:
```py
import asyncio
from CroFlow import run_coros
async def task_one():
await asyncio.sleep(3)
return "Task One Completed"
async def task_two():
await asyncio.sleep(1)
return "Task Two Completed"
async def main():
coros = [task_one, task_two]
async for result in run_coros(coros):
print(result)
asyncio.run(main())
```
## Multi-threaded Execution
To run coroutines across multiple threads, use threading_run_coros:
```py
import asyncio
from CroFlow import threading_run_coros
async def task_one():
await asyncio.sleep(2.9)
return "Task One Completed"
async def task_two():
await asyncio.sleep(1.3)
return "Task Two Completed"
async def main():
coros = [task_one, task_two]
# max_process_groups is the number of threads to use
async for result in threading_run_coros(coros, max_process_groups=2):
print(result)
asyncio.run(main())
```
## Handling Exceptions
you can configure CroFlow to return exceptions as part of the results (exceptions are returned by default):
```py
import asyncio
from CroFlow import run_coros
async def task_one():
raise ValueError("An error occurred in Task One")
async def task_two():
return "Task Two Completed"
async def main():
coros = [task_one, task_two]
async for result in run_coros(coros, return_exceptions=True):
print(result, type(result))
asyncio.run(main())
```
### Output
```
>>> An error occurred in Task One <class 'ValueError'>
>>> Task Two Completed
```
## Setting Timeouts
Specify a timeout to limit the execution time of each coroutine:
```py
import asyncio
from CroFlow import run_coros
async def task_one():
await asyncio.sleep(5)
return "Task One Completed"
async def task_two():
return "Task Two Completed"
async def main():
coros = [task_one, task_two]
async for result in run_coros(coros, timeout=2):
print(result, type(result))
asyncio.run(main())
```
### Output
```
>>> Task Two Completed <class 'str'>
>>> <class 'TimeoutError'>
```
# API Reference
**`run_coros(coros, *args, timeout=0, return_exceptions=True, debug=False, loop=None, **kwargs)`**
Runs a list of coroutines concurrently.
* **coros:** List of coroutines to run.
* **timeout:** Maximum time in seconds to allow each coroutine to run. Defaults to 0 (no timeout).
* **return_exceptions:** Whether to return exceptions as part of the results. Defaults to True.
* **debug:** If True, prints stack traces for exceptions. Defaults to False.
* **loop:** Event loop to use. If None or closed, a new loop is created.
**`threading_run_coros(coros, *args, max_process_groups=4, timeout=0, return_exceptions=True, debug=False, **kwargs)`**
Runs a list of coroutines concurrently using multiple threads.
* **coros:** List of coroutines to run.
* **max_process_groups:** Maximum number of process groups to create concurrently. Defaults to 4.
* **timeout:** Maximum time in seconds to allow each coroutine to run. Defaults to 0 (no timeout).
* **return_exceptions:** Whether to return exceptions as part of the results. Defaults to True.
* **debug:** If True, prints stack traces for exceptions. Defaults to False.
# Comparison: `asyncio.as_completed` vs. `CroFlow`
When working with asynchronous coroutines in Python, you have different tools to handle concurrent execution. Below is a comparison between `asyncio.as_completed` and the custom `cro_flow` function to help you decide which is best suited for your needs.
### `asyncio.as_completed`
**Purpose:**
- Provides an iterator that yields coroutines as they complete, in the order they finish.
**Key Features:**
- **Concurrency:** Runs coroutines concurrently within the asyncio event loop.
- **Simplicity:** Straightforward usage for processing coroutines as they complete.
- **Error Handling:** Does not handle exceptions or provide debugging capabilities.
- **Timeout Handling:** Does not support per-coroutine timeouts.
- **Event Loop:** Operates within the existing asyncio event loop.
**Example Usage:**
```python
coros = [coro1(), coro2(), coro3()]
async for task in asyncio.as_completed(coros):
result = await task
print(result)
```
**Best For:**
* Simple use cases where you need to handle coroutines as they finish without additional error handling or timeout features.
### `CroFlow`
**Purpose:**
* Yields the results from a list of coroutines concurrently using threading (`optionally`) and asyncio, with features for error handling, timeout control, and event loop management.
**Key Features:**
* **Concurrency:** Offers flexibility by allowing coroutines to be executed either within the asyncio event loop or using threading (via `ThreadPoolExecutor`). When using threading, coroutines are divided into groups, and each group is processed concurrently in different threads. This approach helps manage the load on the main event loop and can efficiently handle large numbers of coroutines by distributing the workload across multiple threads and event loops.
* **Error Handling:** Provides options for handling and returning exceptions, with optional debugging and error logging.
* **Timeout Handling:** Allows specifying a timeout for each coroutine.
* **Event Loop Management:** Can use an existing event loop or create a new one if necessary.
* **Grouping:** `(If using the threading method)` Divides coroutines into smaller groups for concurrent execution.
Example Usage:
```py
async for result in run_coros([coro1, coro2, coro3], timeout=5, debug=True):
print(result)
```
**Best For:**
* Scenarios requiring detailed control over coroutine execution, including error handling, timeout management, and concurrent execution using threads.
## Summary
| Feature | `asyncio.as_completed` | `CroFlow` |
|---------------------|---------------------------------------------------|--------------------------------------------------|
| **Purpose** | Yields coroutines as they complete | Yields the results of coroutines as they become available |
| **Concurrency** | Uses asyncio event loop only | Uses asyncio event loop, with the additional choice of threading |
| **Error Handling** | No built-in error handling | Handles exceptions with optional logging |
| **Timeout Handling**| No Supports per-coroutine timeouts | Supports per-coroutine timeouts |
| **Event Loop** | Uses existing event loop | Can create or use an existing event loop |
| **Simplicity** | Simple and straightforward | More complex but offers extensive control |
## Contributing
Contributions are welcome! Please open an issue or submit a pull request with your changes.
## License
This project is licensed under the MIT License.
Raw data
{
"_id": null,
"home_page": null,
"name": "CroFlow",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "CroFlow, asynchronous, threading, parallel execution, coroutines, task management, performance optimization, Python 3",
"author": "Curtidor",
"author_email": "tannermatos18@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/70/26/7a26bd474f999b8d8ba2fca004e9afe4fb028062a0f8a74a07b648f80211/CroFlow-0.0.1.tar.gz",
"platform": null,
"description": "# CroFlow\r\n\r\nCroFlow is a Python library for running asynchronous coroutines concurrently, yielding results as soon as they are available. It supports both single-threaded and multi-threaded execution, offering flexibility for efficiently managing complex asynchronous tasks.\r\n\r\n ## Why Use CroFlow?\r\n* **Instant Results:** CroFlow delivers results as soon as they become available, allowing your application to process and respond to data without waiting for all tasks to complete. This can significantly improve the responsiveness and user experience of applications that rely on real-time data processing.\r\n\r\n* **Ideal for High-Performance Applications:** Perfect for scenarios where rapid execution and result retrieval are crucial, such as in real-time data processing, streaming applications, or services requiring quick response times.\r\n\r\n* **Versatile Use Cases:** Whether you're building web servers to web scrappers, data pipelines, or other concurrent systems, CroFlow helps manage complex asynchronous tasks efficiently. Its flexibility makes it suitable for both small-scale projects and large, distributed systems where timely results are essential.\r\n\r\n## Features:\r\n* **Concurrent Execution:** Run multiple coroutines concurrently in the same thread or across various threads.\r\n\r\n* **Timeout Handling:** Specify a timeout for coroutine execution to prevent tasks from running indefinitely.\r\n\r\n* **Exception Handling:** Choose whether to return exceptions as results or handle them silently.\r\n\r\n* **Debugging Support:** Option to print stack traces for exceptions that occur during coroutine execution.\r\n\r\n## Basic Usage\r\n\r\nHere's a simple example of how to use CroFlow to run multiple coroutines concurrently:\r\n\r\n```py\r\nimport asyncio\r\nfrom CroFlow import run_coros\r\n\r\n\r\nasync def task_one():\r\n await asyncio.sleep(3)\r\n return \"Task One Completed\"\r\n\r\n\r\nasync def task_two():\r\n await asyncio.sleep(1)\r\n return \"Task Two Completed\"\r\n\r\n\r\nasync def main():\r\n coros = [task_one, task_two]\r\n\r\n async for result in run_coros(coros):\r\n print(result)\r\n\r\n\r\nasyncio.run(main())\r\n```\r\n\r\n## Multi-threaded Execution\r\nTo run coroutines across multiple threads, use threading_run_coros:\r\n\r\n```py\r\nimport asyncio\r\nfrom CroFlow import threading_run_coros\r\n\r\n\r\nasync def task_one():\r\n await asyncio.sleep(2.9)\r\n return \"Task One Completed\"\r\n\r\n\r\nasync def task_two():\r\n await asyncio.sleep(1.3)\r\n return \"Task Two Completed\"\r\n\r\n\r\nasync def main():\r\n coros = [task_one, task_two]\r\n\r\n # max_process_groups is the number of threads to use\r\n async for result in threading_run_coros(coros, max_process_groups=2):\r\n print(result)\r\n\r\n\r\nasyncio.run(main())\r\n```\r\n\r\n## Handling Exceptions\r\nyou can configure CroFlow to return exceptions as part of the results (exceptions are returned by default):\r\n\r\n```py\r\nimport asyncio\r\nfrom CroFlow import run_coros\r\n\r\n\r\nasync def task_one():\r\n raise ValueError(\"An error occurred in Task One\")\r\n\r\n\r\nasync def task_two():\r\n return \"Task Two Completed\"\r\n\r\n\r\nasync def main():\r\n coros = [task_one, task_two]\r\n\r\n async for result in run_coros(coros, return_exceptions=True):\r\n print(result, type(result))\r\n\r\n asyncio.run(main())\r\n```\r\n### Output\r\n```\r\n>>> An error occurred in Task One <class 'ValueError'>\r\n>>> Task Two Completed\r\n```\r\n\r\n## Setting Timeouts\r\n\r\nSpecify a timeout to limit the execution time of each coroutine:\r\n\r\n```py\r\nimport asyncio\r\nfrom CroFlow import run_coros\r\n\r\n\r\nasync def task_one():\r\n await asyncio.sleep(5)\r\n return \"Task One Completed\"\r\n\r\n\r\nasync def task_two():\r\n return \"Task Two Completed\"\r\n\r\n\r\nasync def main():\r\n coros = [task_one, task_two]\r\n\r\n async for result in run_coros(coros, timeout=2):\r\n print(result, type(result))\r\n\r\n asyncio.run(main())\r\n\r\n```\r\n\r\n### Output\r\n```\r\n>>> Task Two Completed <class 'str'>\r\n>>> <class 'TimeoutError'>\r\n```\r\n\r\n# API Reference\r\n\r\n**`run_coros(coros, *args, timeout=0, return_exceptions=True, debug=False, loop=None, **kwargs)`**\r\n\r\nRuns a list of coroutines concurrently.\r\n\r\n* **coros:** List of coroutines to run.\r\n* **timeout:** Maximum time in seconds to allow each coroutine to run. Defaults to 0 (no timeout).\r\n* **return_exceptions:** Whether to return exceptions as part of the results. Defaults to True.\r\n* **debug:** If True, prints stack traces for exceptions. Defaults to False.\r\n* **loop:** Event loop to use. If None or closed, a new loop is created.\r\n\r\n**`threading_run_coros(coros, *args, max_process_groups=4, timeout=0, return_exceptions=True, debug=False, **kwargs)`**\r\n\r\nRuns a list of coroutines concurrently using multiple threads.\r\n\r\n* **coros:** List of coroutines to run.\r\n* **max_process_groups:** Maximum number of process groups to create concurrently. Defaults to 4.\r\n* **timeout:** Maximum time in seconds to allow each coroutine to run. Defaults to 0 (no timeout).\r\n* **return_exceptions:** Whether to return exceptions as part of the results. Defaults to True.\r\n* **debug:** If True, prints stack traces for exceptions. Defaults to False.\r\n\r\n# Comparison: `asyncio.as_completed` vs. `CroFlow`\r\n\r\nWhen working with asynchronous coroutines in Python, you have different tools to handle concurrent execution. Below is a comparison between `asyncio.as_completed` and the custom `cro_flow` function to help you decide which is best suited for your needs.\r\n\r\n### `asyncio.as_completed`\r\n\r\n**Purpose:**\r\n- Provides an iterator that yields coroutines as they complete, in the order they finish.\r\n\r\n**Key Features:**\r\n- **Concurrency:** Runs coroutines concurrently within the asyncio event loop.\r\n- **Simplicity:** Straightforward usage for processing coroutines as they complete.\r\n- **Error Handling:** Does not handle exceptions or provide debugging capabilities.\r\n- **Timeout Handling:** Does not support per-coroutine timeouts.\r\n- **Event Loop:** Operates within the existing asyncio event loop.\r\n\r\n**Example Usage:**\r\n```python\r\ncoros = [coro1(), coro2(), coro3()]\r\nasync for task in asyncio.as_completed(coros):\r\n result = await task\r\n print(result)\r\n```\r\n\r\n**Best For:**\r\n\r\n* Simple use cases where you need to handle coroutines as they finish without additional error handling or timeout features.\r\n\r\n### `CroFlow`\r\n**Purpose:**\r\n\r\n* Yields the results from a list of coroutines concurrently using threading (`optionally`) and asyncio, with features for error handling, timeout control, and event loop management.\r\n\r\n**Key Features:**\r\n\r\n* **Concurrency:** Offers flexibility by allowing coroutines to be executed either within the asyncio event loop or using threading (via `ThreadPoolExecutor`). When using threading, coroutines are divided into groups, and each group is processed concurrently in different threads. This approach helps manage the load on the main event loop and can efficiently handle large numbers of coroutines by distributing the workload across multiple threads and event loops.\r\n* **Error Handling:** Provides options for handling and returning exceptions, with optional debugging and error logging.\r\n* **Timeout Handling:** Allows specifying a timeout for each coroutine.\r\n* **Event Loop Management:** Can use an existing event loop or create a new one if necessary.\r\n* **Grouping:** `(If using the threading method)` Divides coroutines into smaller groups for concurrent execution.\r\n \r\nExample Usage:\r\n\r\n```py\r\nasync for result in run_coros([coro1, coro2, coro3], timeout=5, debug=True):\r\n print(result)\r\n```\r\n**Best For:**\r\n\r\n* Scenarios requiring detailed control over coroutine execution, including error handling, timeout management, and concurrent execution using threads.\r\n\r\n## Summary\r\n\r\n| Feature | `asyncio.as_completed` | `CroFlow` |\r\n|---------------------|---------------------------------------------------|--------------------------------------------------|\r\n| **Purpose** | Yields coroutines as they complete | Yields the results of coroutines as they become available |\r\n| **Concurrency** | Uses asyncio event loop only | Uses asyncio event loop, with the additional choice of threading |\r\n| **Error Handling** | No built-in error handling | Handles exceptions with optional logging |\r\n| **Timeout Handling**| No Supports per-coroutine timeouts | Supports per-coroutine timeouts |\r\n| **Event Loop** | Uses existing event loop | Can create or use an existing event loop |\r\n| **Simplicity** | Simple and straightforward | More complex but offers extensive control |\r\n\r\n\r\n## Contributing\r\nContributions are welcome! Please open an issue or submit a pull request with your changes.\r\n\r\n## License\r\nThis project is licensed under the MIT License.\r\n",
"bugtrack_url": null,
"license": null,
"summary": "CroFlow is a Python library for running asynchronous coroutines concurrently, yielding results as soon as they are available. It supports both single-threaded and multi-threaded execution, offering flexibility for efficientlymanaging complex asynchronous tasks. handling exceptions and timeouts to optimize performance and manage complex workflows.",
"version": "0.0.1",
"project_urls": null,
"split_keywords": [
"croflow",
" asynchronous",
" threading",
" parallel execution",
" coroutines",
" task management",
" performance optimization",
" python 3"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8517d4f876c859df608bd03355d03ad7a350f086a86dfe1823d06027e35b0bee",
"md5": "fabcd9a546a2b158ed36cf7f0b8fc259",
"sha256": "6f4e89a7d9bd3fe5bf29d6b27cc9d477501a655b01e0405fe936950c74487a66"
},
"downloads": -1,
"filename": "CroFlow-0.0.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "fabcd9a546a2b158ed36cf7f0b8fc259",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 8267,
"upload_time": "2024-08-05T02:49:04",
"upload_time_iso_8601": "2024-08-05T02:49:04.932413Z",
"url": "https://files.pythonhosted.org/packages/85/17/d4f876c859df608bd03355d03ad7a350f086a86dfe1823d06027e35b0bee/CroFlow-0.0.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "70267a26bd474f999b8d8ba2fca004e9afe4fb028062a0f8a74a07b648f80211",
"md5": "7533cd53f54ce45bd23baa499737964d",
"sha256": "4c1ab0cb6945784701a52b97a80a698b6dd5cf78adb3ac3520f669a6a78a59b1"
},
"downloads": -1,
"filename": "CroFlow-0.0.1.tar.gz",
"has_sig": false,
"md5_digest": "7533cd53f54ce45bd23baa499737964d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 7689,
"upload_time": "2024-08-05T02:49:06",
"upload_time_iso_8601": "2024-08-05T02:49:06.514653Z",
"url": "https://files.pythonhosted.org/packages/70/26/7a26bd474f999b8d8ba2fca004e9afe4fb028062a0f8a74a07b648f80211/CroFlow-0.0.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-05 02:49:06",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "croflow"
}
