| Name | asyncio-pypes JSON |
| Version |
0.0.4
JSON |
| download |
| home_page | |
| Summary | Scalable async pipelines in Python, made easy |
| upload_time | 2023-09-17 20:58:52 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.5.9 |
| license | MIT License Copyright (c) [2023] [Michael Roumanos] Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
asyncio
pipeline
pipe
stream
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
<h3 align="center">(pronounced "pipes")</h3>
<div align="center">
[]()
[](https://github.com/mroumanos/pypes/issues)
[](https://github.com/mroumanos/pypes/pulls)
[](/LICENSE)
</div>
---
<p align="center"> Scalable async pipelines in Python, made easy.
<br>
</p>
## 📝 Table of Contents
- [About](#about)
- [Getting Started](#getting_started)
- [Usage](#usage)
- [Built Using](#built_using)
- [TODO](#todo)
- [Contributing](../CONTRIBUTING.md)
- [Authors](#authors)
- [Acknowledgments](#acknowledgement)
## 🧐 About <a name = "about"></a>
This package is designed to make building asynchronous streams that balance and
scale automatically *easy*. Built on pure Python -- no dependencies -- this
framework can be used for variable, decoupled, task-based workloads like
web scraping, database management operations, and more. Scale this out-of-the-box,
with minimal hardware and coding, to process 10k+/s on production loads.
Simple pipelines
```
import pypes
app = pypes.App()
@app.task(interval=1.0)
async def every_second():
return datetime.utcnow()
@app.task()
async def task1(stream):
async for s in stream:
print(f"streaming from task1: {s}")
yield obj
if __name__ == '__main__':
pipeline = every_second \
.map(task1)
pipeline.run()
```
To scaled pipelines
```
import pypes
import aiohttp
app = pypes.App()
@app.task(interval=0.1)
async def every_second():
return "http://www.google.com"
@app.task(scaler=pypes.scale.TanhTaskScaler()) # this scales workers automatically to consume incoming requests
async def task1(stream):
async for s in stream:
yield await aiohttp.get(s)
async def task2(stream):
async for s in stream:
if s.response_code != 200:
print("failed request: {s}")
yield
if __name__ == '__main__':
pipeline = every_second \
.map(task1)
.reduce(task2)
pipeline.run()
```
## 🏁 Getting Started <a name = "getting_started"></a>
Start with a simple pipeline, and build out from there!
```
import pypes
app = pypes.App()
@app.task(interval=1.0)
async def every_second():
return datetime.utcnow()
@app.task()
async def task1(stream):
async for s in stream:
print(f"streaming from task1: {s}")
yield obj
if __name__ == '__main__':
pipeline = every_second \
.map(task1)
pipeline.run()
```
For more, see [readthedocs](https://async-pypes.readthedocs.io)
### Prerequisites
`pypes` is based on pure Python (3.5+) and does not require other dependencies.
### Installing
```
pip install pypes
```
## 🔧 Running the tests <a name = "tests"></a>
To be created!
### Break down into end to end tests
To be created!
### And coding style tests
To be created!
## 🎈 Usage <a name="usage"></a>
Import the library
```
import pypes
```
Create an App object
```
app = pypes.App()
```
Create a trigger task
```
@app.task(interval=1.0)
async def every_second():
return 1
```
Create downtream tasks
```
@app.task()
async def task_n(stream):
async for s in stream:
# process "s" here
yield
```
Create + configure the pipeline
```
pipeline = every_second \
.map(task_n)
```
Run the pipeline
```
pipeline.run()
```
This will run continuously until interrupted.
## ⛏️ Built Using <a name = "built_using"></a>
- [Python](https://www.python.org/)
- [asyncio](https://docs.python.org/3/library/asyncio.html)
## ✔️ TODO <a name = "todo"></a>
- [ ] Extend to multithreads
- [ ] Extend to multiprocess
- [ ] Build visualization server
- [ ] Add pipeline pipe functions (join, head, ...)
## ✍️ Authors <a name = "authors"></a>
- [@mroumanos](https://github.com/mroumanos)
- Contributors: you?
## 🎉 Acknowledgements <a name = "acknowledgement"></a>
- [faust streaming](https://github.com/faust-streaming/faust)
Raw data
{
"_id": null,
"home_page": "",
"name": "asyncio-pypes",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.5.9",
"maintainer_email": "",
"keywords": "asyncio,pipeline,pipe,stream",
"author": "",
"author_email": "mroumanos <mroumanos@pm.me>",
"download_url": "https://files.pythonhosted.org/packages/e3/f7/f065d4ad8eb6e38fbdb7727b0d5d02f1e6481b0f103510768c4884c9271e/asyncio-pypes-0.0.4.tar.gz",
"platform": null,
"description": "\n\n<h3 align=\"center\">(pronounced \"pipes\")</h3>\n\n<div align=\"center\">\n\n[]()\n[](https://github.com/mroumanos/pypes/issues)\n[](https://github.com/mroumanos/pypes/pulls)\n[](/LICENSE)\n\n</div>\n\n---\n\n<p align=\"center\"> Scalable async pipelines in Python, made easy.\n <br> \n</p>\n\n## \ud83d\udcdd Table of Contents\n\n- [About](#about)\n- [Getting Started](#getting_started)\n- [Usage](#usage)\n- [Built Using](#built_using)\n- [TODO](#todo)\n- [Contributing](../CONTRIBUTING.md)\n- [Authors](#authors)\n- [Acknowledgments](#acknowledgement)\n\n## \ud83e\uddd0 About <a name = \"about\"></a>\n\nThis package is designed to make building asynchronous streams that balance and \nscale automatically *easy*. Built on pure Python -- no dependencies -- this \nframework can be used for variable, decoupled, task-based workloads like \nweb scraping, database management operations, and more. Scale this out-of-the-box, \nwith minimal hardware and coding, to process 10k+/s on production loads.\n\nSimple pipelines\n```\nimport pypes\n\napp = pypes.App()\n\n@app.task(interval=1.0)\nasync def every_second():\n return datetime.utcnow()\n\n@app.task()\nasync def task1(stream):\n async for s in stream:\n print(f\"streaming from task1: {s}\")\n yield obj\n\nif __name__ == '__main__':\n\n pipeline = every_second \\\n .map(task1)\n\n pipeline.run()\n```\n\nTo scaled pipelines\n```\nimport pypes\nimport aiohttp\n\napp = pypes.App()\n\n@app.task(interval=0.1)\nasync def every_second():\n return \"http://www.google.com\"\n\n@app.task(scaler=pypes.scale.TanhTaskScaler()) # this scales workers automatically to consume incoming requests\nasync def task1(stream):\n async for s in stream:\n yield await aiohttp.get(s)\n\nasync def task2(stream):\n async for s in stream:\n if s.response_code != 200:\n print(\"failed request: {s}\")\n yield\n\nif __name__ == '__main__':\n\n pipeline = every_second \\\n .map(task1)\n .reduce(task2)\n\n pipeline.run()\n```\n\n## \ud83c\udfc1 Getting Started <a name = \"getting_started\"></a>\n\nStart with a simple pipeline, and build out from there!\n```\nimport pypes\n\napp = pypes.App()\n\n@app.task(interval=1.0)\nasync def every_second():\n return datetime.utcnow()\n\n@app.task()\nasync def task1(stream):\n async for s in stream:\n print(f\"streaming from task1: {s}\")\n yield obj\n\nif __name__ == '__main__':\n\n pipeline = every_second \\\n .map(task1)\n\n pipeline.run()\n```\nFor more, see [readthedocs](https://async-pypes.readthedocs.io)\n\n### Prerequisites\n\n`pypes` is based on pure Python (3.5+) and does not require other dependencies.\n\n### Installing\n\n```\npip install pypes\n```\n\n## \ud83d\udd27 Running the tests <a name = \"tests\"></a>\n\nTo be created!\n\n### Break down into end to end tests\n\nTo be created!\n\n### And coding style tests\n\nTo be created!\n\n## \ud83c\udf88 Usage <a name=\"usage\"></a>\n\nImport the library\n```\nimport pypes\n```\n\nCreate an App object\n```\napp = pypes.App()\n```\n\nCreate a trigger task\n```\n@app.task(interval=1.0)\nasync def every_second():\n return 1\n```\n\nCreate downtream tasks\n```\n@app.task()\nasync def task_n(stream):\n async for s in stream:\n # process \"s\" here\n yield\n```\n\nCreate + configure the pipeline\n```\npipeline = every_second \\\n .map(task_n)\n```\n\nRun the pipeline\n```\npipeline.run()\n```\n\nThis will run continuously until interrupted.\n\n## \u26cf\ufe0f Built Using <a name = \"built_using\"></a>\n\n- [Python](https://www.python.org/)\n- [asyncio](https://docs.python.org/3/library/asyncio.html)\n\n## \u2714\ufe0f TODO <a name = \"todo\"></a>\n\n- [ ] Extend to multithreads\n- [ ] Extend to multiprocess\n- [ ] Build visualization server\n- [ ] Add pipeline pipe functions (join, head, ...)\n\n## \u270d\ufe0f Authors <a name = \"authors\"></a>\n\n- [@mroumanos](https://github.com/mroumanos)\n- Contributors: you?\n\n## \ud83c\udf89 Acknowledgements <a name = \"acknowledgement\"></a>\n\n- [faust streaming](https://github.com/faust-streaming/faust)\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) [2023] [Michael Roumanos] Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.",
"summary": "Scalable async pipelines in Python, made easy",
"version": "0.0.4",
"project_urls": {
"Documentation": "https://async-pypes.readthedocs.io",
"Homepage": "https://github.com/mroumanos/pypes",
"Repository": "https://github.com/mroumanos/pypes.git"
},
"split_keywords": [
"asyncio",
"pipeline",
"pipe",
"stream"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "d6d19884e65304074217107c778cc50c1e9fd71902565f22801bef74026fc11b",
"md5": "c2cb0a9346b5c576fc0ccb87f8717839",
"sha256": "42dc8b55ec15c5f57b667aac92e28cf2ed376cfcf867bf64d56f1497ce3b783d"
},
"downloads": -1,
"filename": "asyncio_pypes-0.0.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c2cb0a9346b5c576fc0ccb87f8717839",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.5.9",
"size": 14290,
"upload_time": "2023-09-17T20:58:50",
"upload_time_iso_8601": "2023-09-17T20:58:50.382129Z",
"url": "https://files.pythonhosted.org/packages/d6/d1/9884e65304074217107c778cc50c1e9fd71902565f22801bef74026fc11b/asyncio_pypes-0.0.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e3f7f065d4ad8eb6e38fbdb7727b0d5d02f1e6481b0f103510768c4884c9271e",
"md5": "cfb382d5859fd1c10b1ae9d81db60a53",
"sha256": "87dc6d9228b6ea2090dbe455d2c9ecf9e5d421b567f6ef1a187e508520a76f79"
},
"downloads": -1,
"filename": "asyncio-pypes-0.0.4.tar.gz",
"has_sig": false,
"md5_digest": "cfb382d5859fd1c10b1ae9d81db60a53",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.5.9",
"size": 12470,
"upload_time": "2023-09-17T20:58:52",
"upload_time_iso_8601": "2023-09-17T20:58:52.502880Z",
"url": "https://files.pythonhosted.org/packages/e3/f7/f065d4ad8eb6e38fbdb7727b0d5d02f1e6481b0f103510768c4884c9271e/asyncio-pypes-0.0.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-09-17 20:58:52",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mroumanos",
"github_project": "pypes",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "asyncio-pypes"
}
