| Name | open-fleet JSON |
| Version |
0.0.3
JSON |
| download |
| home_page | None |
| Summary | distributed task distribution framework |
| upload_time | 2024-04-19 10:23:14 |
| maintainer | None |
| docs_url | None |
| author | Shiyu Huang |
| requires_python | >=3.8 |
| license | None |
| keywords |
distributed
framework
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Fleet
Fleet is a generic distributed task distribution framework based on a distributed file system. Task distribution frameworks like Ray and Celery require network connections for communication, which makes them difficult to use in clusters with poor network conditions. Fleet is a distributed framework based on a shared file system, independent of any network communication, allowing for task distribution among nodes without any network connections.
## Features
- Distributed task distribution based on a shared file system
- Supports dynamic scaling
- Supports worker node heartbeat
- Supports manager node restart
- Supports set timeout for each task
- Pure Python implementation
## Install
```bash
pip install open-fleet
```
## Usage
See more examples in [./examples](./examples).
```python
# run_manager.py
from fleet.manager import Manager
from fleet.config.config import get_args
base_dir = "./share_dir"
def main():
job_list = [1,2,3,4]
args = get_args(f"--base_dir {base_dir}")
manager = Manager(args=args, job_list=job_list)
manager.run()
if __name__ == '__main__':
main()
```
```python
# run_worker.py
from fleet.worker import Worker
from fleet.config.config import get_args
from run_manager import base_dir
def add_one(x, info):
return {"status": "success", "result": x + 1}
def main():
args = get_args(f"--base_dir {base_dir}")
worker = Worker(args=args, job_func=add_one)
worker.run()
if __name__ == '__main__':
main()
```
Raw data
{
"_id": null,
"home_page": null,
"name": "open-fleet",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "distributed framework",
"author": "Shiyu Huang",
"author_email": "huangsy1314@163.com",
"download_url": "https://files.pythonhosted.org/packages/48/ff/0489def51dfafef87655152d18077c936a10430296ab2db2d0fe29c8d8de/open_fleet-0.0.3.tar.gz",
"platform": null,
"description": "# Fleet\n\nFleet is a generic distributed task distribution framework based on a distributed file system. Task distribution frameworks like Ray and Celery require network connections for communication, which makes them difficult to use in clusters with poor network conditions. Fleet is a distributed framework based on a shared file system, independent of any network communication, allowing for task distribution among nodes without any network connections.\n\n## Features\n- Distributed task distribution based on a shared file system\n- Supports dynamic scaling\n- Supports worker node heartbeat\n- Supports manager node restart\n- Supports set timeout for each task\n- Pure Python implementation\n\n## Install\n\n```bash\npip install open-fleet\n```\n\n## Usage\n\nSee more examples in [./examples](./examples).\n\n```python\n# run_manager.py\nfrom fleet.manager import Manager\nfrom fleet.config.config import get_args\n\nbase_dir = \"./share_dir\"\n\ndef main():\n job_list = [1,2,3,4]\n args = get_args(f\"--base_dir {base_dir}\")\n manager = Manager(args=args, job_list=job_list)\n manager.run()\n\nif __name__ == '__main__':\n main()\n```\n \n```python\n# run_worker.py\nfrom fleet.worker import Worker\nfrom fleet.config.config import get_args\n\nfrom run_manager import base_dir\n\ndef add_one(x, info):\n return {\"status\": \"success\", \"result\": x + 1}\n\ndef main():\n args = get_args(f\"--base_dir {base_dir}\")\n worker = Worker(args=args, job_func=add_one)\n worker.run()\n\nif __name__ == '__main__':\n main()\n```\n",
"bugtrack_url": null,
"license": null,
"summary": "distributed task distribution framework",
"version": "0.0.3",
"project_urls": {
"Documentation": "https://github.com/huangshiyu13/Fleet"
},
"split_keywords": [
"distributed",
"framework"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a666a3e9c48d222e00a78340340ccdd3c50ea544e36dd79457cdbeb27288d63f",
"md5": "716901b76f832154c2b7f56565a4ea85",
"sha256": "4f4617ec50fc5db9ed4c022dd02c3f9e794cda25860fc623b1efb58da25aa252"
},
"downloads": -1,
"filename": "open_fleet-0.0.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "716901b76f832154c2b7f56565a4ea85",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 14882,
"upload_time": "2024-04-19T10:23:13",
"upload_time_iso_8601": "2024-04-19T10:23:13.005460Z",
"url": "https://files.pythonhosted.org/packages/a6/66/a3e9c48d222e00a78340340ccdd3c50ea544e36dd79457cdbeb27288d63f/open_fleet-0.0.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "48ff0489def51dfafef87655152d18077c936a10430296ab2db2d0fe29c8d8de",
"md5": "47b2219d372fc0125a5462b77e4d7fa4",
"sha256": "a12739eeb741aaafc4cdfa0c5ac6f1e55a34485f31e130f3887f31f23e084f6d"
},
"downloads": -1,
"filename": "open_fleet-0.0.3.tar.gz",
"has_sig": false,
"md5_digest": "47b2219d372fc0125a5462b77e4d7fa4",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 14199,
"upload_time": "2024-04-19T10:23:14",
"upload_time_iso_8601": "2024-04-19T10:23:14.175462Z",
"url": "https://files.pythonhosted.org/packages/48/ff/0489def51dfafef87655152d18077c936a10430296ab2db2d0fe29c8d8de/open_fleet-0.0.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-19 10:23:14",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "huangshiyu13",
"github_project": "Fleet",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "open-fleet"
}
