Thư viện scheduler multiple partitions của Profiling
<b>nop</b>: maximum partitions now support is 1000
<br>
<b>delays</b>: maximum time delays now support is 3600 seconds (1 hour)
sample code:
```python
from mobio.libs.m_scheduler_partitioning.m_scheduler import MobioScheduler
from mobio.libs.m_scheduler_partitioning.scheduler_models.scheduler_state_model import SchedulerStateModel
class SampleScheduler(MobioScheduler):
def process(self):
if self.url_connection:
SchedulerStateModel(self.url_connection).set_busy(
worker_id=self.node_id
)
print("Hi there ! :)")
if __name__ == "__main__":
SampleScheduler(root_node="test-scheduler", nop=100, delays=1, url_connection="mongodb://test_user:test_password@0.0.0.0:27017/test_db", zookeeper_uri="127.0.0.1:2181")
```
# Change logs
* 0.1.2
* log state of worker
* get free worker
* Để không bị mất 50k cho anh Lợi, thêm 2 index này:
* db.scheduler_state.createIndex({"expiry_time": 1}, {expireAfterSeconds: 5, name: "expiry_time_1"})
* db.scheduler_state.createIndex({"root_node": 1, "state":1}, {name: "root_node_1_state_1"})
* 0.1.3
* fix issue khi worker rebalance không tự động cập nhật danh sách partitions.
* 0.1.4
* fix issue register worker
* cơ chế đảm bảo việc register worker với hệ thống csdl
* 0.1.5
* Refix issue register worker
* Thử nghiệm cơ chế đảm bảo 1 partition chỉ nằm trên 1 worker.
* *NOTE*: phần này chưa đảm bảo được việc đủ partitions trên các workers
* 0.1.6
* missing version do nâng cấp CICD
* 0.1.7
* Sử dụng threading để quản lý heart_beat và expiry_time
* Kiểm tra trạng thái subscribe của worker mỗi khi chuẩn bị process data (Đảm bảo rằng việc subscribe phải diễn ra thành công tránh 2 worker cùng xử lý 1 partition)
* 0.1.8
* Cho phép truyền zookeeper_uri vào param khi khởi tạo Schedule, nếu không truyền thì lấy mặc định từ ENV: ZOOKEEPER_CLUSTER
* 0.1.9
* Tự động quản lý và cập nhật state cho worker.
* fix lỗi tự động release worker ở bản 0.1.7
* 0.1.10
* Fix lỗi không release khi set time delay giữa các lần chạy nhỏ hơn 15 seconds :(
* 0.1.11
* Fix lỗi list partitions = [] dẫn đến việc không truy vấn được data trong db
* Bỏ 1 số function validate không cần thiết.
* 0.1.12
* Chuyển luồng update state vào main thread. Do ko reproduce được case trung state vẫn update trên k8s. :'(
* 0.1.13
* reformat code.
* 0.1.14
* Support mongo >= 4
Raw data
{
"_id": null,
"home_page": "https://github.com/mobiovn",
"name": "m-scheduler-partitioning",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3",
"maintainer_email": null,
"keywords": "mobio, scheduler, partitioning",
"author": "MOBIO",
"author_email": "contact@mobio.vn",
"download_url": "https://files.pythonhosted.org/packages/7b/49/dfffd50f07cfa0bc2254692c05161f7ee7e37ed408ee7d4b0d28b99ea625/m_scheduler_partitioning-0.1.14.tar.gz",
"platform": null,
"description": "Th\u01b0 vi\u1ec7n scheduler multiple partitions c\u1ee7a Profiling\n\n<b>nop</b>: maximum partitions now support is 1000 \n<br>\n<b>delays</b>: maximum time delays now support is 3600 seconds (1 hour)\n\nsample code:\n\n```python\nfrom mobio.libs.m_scheduler_partitioning.m_scheduler import MobioScheduler\nfrom mobio.libs.m_scheduler_partitioning.scheduler_models.scheduler_state_model import SchedulerStateModel\n\n\nclass SampleScheduler(MobioScheduler):\n def process(self):\n if self.url_connection:\n SchedulerStateModel(self.url_connection).set_busy(\n worker_id=self.node_id\n )\n print(\"Hi there ! :)\")\n\n\nif __name__ == \"__main__\":\n SampleScheduler(root_node=\"test-scheduler\", nop=100, delays=1, url_connection=\"mongodb://test_user:test_password@0.0.0.0:27017/test_db\", zookeeper_uri=\"127.0.0.1:2181\")\n\n```\n# Change logs\n* 0.1.2\n * log state of worker\n * get free worker\n * \u0110\u1ec3 kh\u00f4ng b\u1ecb m\u1ea5t 50k cho anh L\u1ee3i, th\u00eam 2 index n\u00e0y:\n\n * db.scheduler_state.createIndex({\"expiry_time\": 1}, {expireAfterSeconds: 5, name: \"expiry_time_1\"})\n * db.scheduler_state.createIndex({\"root_node\": 1, \"state\":1}, {name: \"root_node_1_state_1\"})\n \n* 0.1.3\n * fix issue khi worker rebalance kh\u00f4ng t\u1ef1 \u0111\u1ed9ng c\u1eadp nh\u1eadt danh s\u00e1ch partitions.\n \n* 0.1.4\n * fix issue register worker\n * c\u01a1 ch\u1ebf \u0111\u1ea3m b\u1ea3o vi\u1ec7c register worker v\u1edbi h\u1ec7 th\u1ed1ng csdl\n* 0.1.5\n * Refix issue register worker\n * Th\u1eed nghi\u1ec7m c\u01a1 ch\u1ebf \u0111\u1ea3m b\u1ea3o 1 partition ch\u1ec9 n\u1eb1m tr\u00ean 1 worker. \n * *NOTE*: ph\u1ea7n n\u00e0y ch\u01b0a \u0111\u1ea3m b\u1ea3o \u0111\u01b0\u1ee3c vi\u1ec7c \u0111\u1ee7 partitions tr\u00ean c\u00e1c workers\n\n* 0.1.6\n * missing version do n\u00e2ng c\u1ea5p CICD\n\n* 0.1.7\n * S\u1eed d\u1ee5ng threading \u0111\u1ec3 qu\u1ea3n l\u00fd heart_beat v\u00e0 expiry_time\n * Ki\u1ec3m tra tr\u1ea1ng th\u00e1i subscribe c\u1ee7a worker m\u1ed7i khi chu\u1ea9n b\u1ecb process data (\u0110\u1ea3m b\u1ea3o r\u1eb1ng vi\u1ec7c subscribe ph\u1ea3i di\u1ec5n ra th\u00e0nh c\u00f4ng tr\u00e1nh 2 worker c\u00f9ng x\u1eed l\u00fd 1 partition)\n\n* 0.1.8\n * Cho ph\u00e9p truy\u1ec1n zookeeper_uri v\u00e0o param khi kh\u1edfi t\u1ea1o Schedule, n\u1ebfu kh\u00f4ng truy\u1ec1n th\u00ec l\u1ea5y m\u1eb7c \u0111\u1ecbnh t\u1eeb ENV: ZOOKEEPER_CLUSTER\n\n* 0.1.9\n * T\u1ef1 \u0111\u1ed9ng qu\u1ea3n l\u00fd v\u00e0 c\u1eadp nh\u1eadt state cho worker.\n * fix l\u1ed7i t\u1ef1 \u0111\u1ed9ng release worker \u1edf b\u1ea3n 0.1.7\n\n* 0.1.10\n * Fix l\u1ed7i kh\u00f4ng release khi set time delay gi\u1eefa c\u00e1c l\u1ea7n ch\u1ea1y nh\u1ecf h\u01a1n 15 seconds :(\n\n* 0.1.11\n * Fix l\u1ed7i list partitions = [] d\u1eabn \u0111\u1ebfn vi\u1ec7c kh\u00f4ng truy v\u1ea5n \u0111\u01b0\u1ee3c data trong db\n * B\u1ecf 1 s\u1ed1 function validate kh\u00f4ng c\u1ea7n thi\u1ebft.\n\n* 0.1.12\n * Chuy\u1ec3n lu\u1ed3ng update state v\u00e0o main thread. Do ko reproduce \u0111\u01b0\u1ee3c case trung state v\u1eabn update tr\u00ean k8s. :'(\n\n* 0.1.13\n * reformat code.\n\n* 0.1.14\n * Support mongo >= 4\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Mobio scheduler partitioning libs",
"version": "0.1.14",
"project_urls": {
"Homepage": "https://github.com/mobiovn",
"Source": "https://github.com/mobiovn"
},
"split_keywords": [
"mobio",
" scheduler",
" partitioning"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "7b49dfffd50f07cfa0bc2254692c05161f7ee7e37ed408ee7d4b0d28b99ea625",
"md5": "406f58c414e2800dd174ba6b84390461",
"sha256": "63a2f2d6c0288e170e45ec703592202aa7f147c017cfd5bf7c4d7db775b704ca"
},
"downloads": -1,
"filename": "m_scheduler_partitioning-0.1.14.tar.gz",
"has_sig": false,
"md5_digest": "406f58c414e2800dd174ba6b84390461",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3",
"size": 10921,
"upload_time": "2024-04-22T04:47:30",
"upload_time_iso_8601": "2024-04-22T04:47:30.587482Z",
"url": "https://files.pythonhosted.org/packages/7b/49/dfffd50f07cfa0bc2254692c05161f7ee7e37ed408ee7d4b0d28b99ea625/m_scheduler_partitioning-0.1.14.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-22 04:47:30",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "m-scheduler-partitioning"
}
JSON