| Name | open-fleet JSON |
| Version |
0.0.7
JSON |
| download |
| home_page | None |
| Summary | distributed task distribution framework |
| upload_time | 2024-08-09 12:31:36 |
| maintainer | None |
| docs_url | None |
| author | Shiyu Huang |
| requires_python | >=3.8 |
| license | None |
| keywords |
distributed
framework
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Fleet
Fleet is a generic distributed task distribution framework based on a distributed file system. Task distribution frameworks like Ray and Celery require network connections for communication, which makes them difficult to use in clusters with poor network conditions. Fleet is a distributed framework based on a shared file system, independent of any network communication, allowing for task distribution among nodes without any network connections.
## Features
- Distributed task distribution based on a shared file system
- Supports dynamic scaling
- Supports worker node heartbeat
- Supports manager node restart
- Supports set timeout for each task
- Supports set max_job and max_work_time for worker
- Pure Python implementation
## Install
```bash
pip install open-fleet
```
## Usage
See more examples in [./examples](./examples).
```python
# run_manager.py
from fleet.manager import Manager
from fleet.config.config import get_args
base_dir = "./share_dir"
def main():
job_list = [1,2,3,4]
args = get_args(f"--base_dir {base_dir}")
manager = Manager(args=args, job_list=job_list)
manager.run()
if __name__ == '__main__':
main()
```
```python
# run_worker.py
from fleet.worker import Worker
from fleet.config.config import get_args
from run_manager import base_dir
def add_one(x, info):
return {"status": "success", "result": x + 1}
def main():
args = get_args(f"--base_dir {base_dir}")
worker = Worker(args=args, job_func=add_one)
worker.run()
if __name__ == '__main__':
main()
```
Raw data
{
"_id": null,
"home_page": null,
"name": "open-fleet",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "distributed framework",
"author": "Shiyu Huang",
"author_email": "huangsy1314@163.com",
"download_url": "https://files.pythonhosted.org/packages/f5/0a/0528fcd722c02b1977a7024eebc76847090670514ec28184114c38e9120c/open_fleet-0.0.7.tar.gz",
"platform": null,
"description": "# Fleet\n\nFleet is a generic distributed task distribution framework based on a distributed file system. Task distribution frameworks like Ray and Celery require network connections for communication, which makes them difficult to use in clusters with poor network conditions. Fleet is a distributed framework based on a shared file system, independent of any network communication, allowing for task distribution among nodes without any network connections.\n\n## Features\n- Distributed task distribution based on a shared file system\n- Supports dynamic scaling\n- Supports worker node heartbeat\n- Supports manager node restart\n- Supports set timeout for each task\n- Supports set max_job and max_work_time for worker\n- Pure Python implementation\n\n## Install\n\n```bash\npip install open-fleet\n```\n\n## Usage\n\nSee more examples in [./examples](./examples).\n\n```python\n# run_manager.py\nfrom fleet.manager import Manager\nfrom fleet.config.config import get_args\n\nbase_dir = \"./share_dir\"\n\ndef main():\n job_list = [1,2,3,4]\n args = get_args(f\"--base_dir {base_dir}\")\n manager = Manager(args=args, job_list=job_list)\n manager.run()\n\nif __name__ == '__main__':\n main()\n```\n \n```python\n# run_worker.py\nfrom fleet.worker import Worker\nfrom fleet.config.config import get_args\n\nfrom run_manager import base_dir\n\ndef add_one(x, info):\n return {\"status\": \"success\", \"result\": x + 1}\n\ndef main():\n args = get_args(f\"--base_dir {base_dir}\")\n worker = Worker(args=args, job_func=add_one)\n worker.run()\n\nif __name__ == '__main__':\n main()\n```\n",
"bugtrack_url": null,
"license": null,
"summary": "distributed task distribution framework",
"version": "0.0.7",
"project_urls": {
"Documentation": "https://github.com/huangshiyu13/Fleet"
},
"split_keywords": [
"distributed",
"framework"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c89953b66b27e42cff9b1076a94108e978b31c431b3eb364bab7f8f5315f24ca",
"md5": "8b709cbe042b670a9e611023b0307914",
"sha256": "90598cfc90a38b1b306b11110fffec1f7e1b7f3b11473dfce2917fda68ddf002"
},
"downloads": -1,
"filename": "open_fleet-0.0.7-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8b709cbe042b670a9e611023b0307914",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 15720,
"upload_time": "2024-08-09T12:31:31",
"upload_time_iso_8601": "2024-08-09T12:31:31.292212Z",
"url": "https://files.pythonhosted.org/packages/c8/99/53b66b27e42cff9b1076a94108e978b31c431b3eb364bab7f8f5315f24ca/open_fleet-0.0.7-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "f50a0528fcd722c02b1977a7024eebc76847090670514ec28184114c38e9120c",
"md5": "c4d5d428b86be0fb558c53ca705803c2",
"sha256": "3c66ff6909c28bfa019eb16a1eda80553a7a49acfc9f47eaccacf26a9be65b6b"
},
"downloads": -1,
"filename": "open_fleet-0.0.7.tar.gz",
"has_sig": false,
"md5_digest": "c4d5d428b86be0fb558c53ca705803c2",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 15102,
"upload_time": "2024-08-09T12:31:36",
"upload_time_iso_8601": "2024-08-09T12:31:36.895044Z",
"url": "https://files.pythonhosted.org/packages/f5/0a/0528fcd722c02b1977a7024eebc76847090670514ec28184114c38e9120c/open_fleet-0.0.7.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-09 12:31:36",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "huangshiyu13",
"github_project": "Fleet",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "open-fleet"
}
