| Name | sdk-tmp-1207 JSON |
| Version |
0.1.1
JSON |
| download |
| home_page | |
| Summary | |
| upload_time | 2022-12-08 08:45:16 |
| maintainer | |
| docs_url | None |
| author | Xiaodong Yang |
| requires_python | >=3.8,<4.0 |
| license | |
| keywords |
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# pyraisdk
## Dynamic Batching
### Description
When we deploy a model in AML with GPU instance to provide inference service, if it occupies GPU for inferencing in each request separately, it will be quite inefficient. This is a shared module helping to collect data items from different requests, and inferencing batchly in a backend thread. This will considerably improve usage efficiency of GPU.
### Usage Examples
Build `YourModel` class inherited from `pyraisdk.dynbatch.BaseModel`.
```python
from typing import List
from pyraisdk.dynbatch import BaseModel
class SimpleModel(BaseModel):
def predict(self, items: List[str]) -> List[int]:
rs = []
for item in items:
rs.append(len(item))
return rs
def preprocess(self, items: List[str]) -> List[str]:
rs = []
for item in items:
rs.append(f'[{item}]')
return rs
```
Initialize a `pyraisdk.dynbatch.DynamicBatchModel` with `YourModel` instance, and call `predict / predict_one` for inferencing.
```python
from pyraisdk.dynbatch import DynamicBatchModel
# prepare model
simple_model = SimpleModel()
batch_model = DynamicBatchModel(simple_model)
# predict
items = ['abc', '123456', 'xyzcccffaffaaa']
predictions = batch_model.predict(items)
assert predictions == [5, 8, 16]
# predict_one
item = 'abc'
prediction = batch_model.predict_one(item)
assert prediction == 5
```
Concurrent requests to `predict / predict_one`, in different threads.
```python
from threading import Thread
from pyraisdk.dynbatch import DynamicBatchModel
# prepare model
simple_model = SimpleModel()
batch_model = DynamicBatchModel(simple_model)
# thread run function
def run(name, num):
for step in range(num):
item = f'{name}-{step}'
prediction = batch_model.predict_one(item)
assert prediction == len(item) + 2
# start concurrent inference
threads = [Thread(target=run, args=(f'{tid}', 100)) for tid in range(20)]
for t in threads:
t.start()
for t in threads:
t.join()
```
## Loging & Events
### Description
This module is for logging and event tracing.
### interface
```python
def initialize(
eh_hostname: Optional[str] = None,
client_id: Optional[str] = None,
eh_conn_str: Optional[str] = None,
eh_structured: Optional[str] = None,
eh_unstructured: Optional[str] = None,
role: Optional[str] = None,
instance: Optional[str] = None,
)
```
Parameter description for `initialize`:
- **eh_hostname**: Fully Qualified Namespace aka EH Endpoint URL (*.servicebus.windows.net). Default, read $EVENTHUB_NAMESPACE
- **client_id**: client_id of service principal. Default, read $UAI_CLIENT_ID
- **eh_conn_str**: connection string of eventhub namespace. Default, read $EVENTHUB_CONN_STRING
- **eh_structured**: structured eventhub name. Default, read $EVENTHUB_AUX_STRUCTURED
- **eh_unstructured**: unstructured eventhub name. Default, read $EVENTHUB_AUX_UNSTRUCTURED
- **role**: role, Default: RemoteModel_${ENDPOINT_NAME}
- **instance**: instance, Default: "${ENDPOINT_VERSION}|{os.uname()[1]}" or "${ENDPOINT_VERSION}|{_probably_unique_id()}"
```python
def event(self, key: str, code: str, numeric: float, detail: str='', corr_id: str='', elem: int=-1)
def infof(self, format: str, *args: Any)
def infocf(self, corr_id: str, elem: int, format: str, *args: Any)
def warnf(self, format: str, *args: Any)
def warncf(self, corr_id: str, elem: int, format: str, *args: Any)
def errorf(self, format: str, *args: Any)
def errorcf(self, corr_id: str, elem: int, ex: Optional[Exception], format: str, *args: Any)
def fatalf(self, format: str, *args: Any)
def fatalcf(self, corr_id: str, elem: int, ex: Optional[Exception], format: str, *args: Any)
```
### examples
```python
# export EVENTHUB_AUX_UNSTRUCTURED='ehunstruct'
# export EVENTHUB_AUX_STRUCTURED='ehstruct'
# export UAI_CLIENT_ID='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
# export EVENTHUB_NAMESPACE='xxx.servicebus.windows.net'
from pyraisdk import rlog
rlog.initialize()
rlog.infof('this is a info message %s', 123)
rlog.event('LifetimeEvent', 'STOP_GRACEFUL_SIGNAL', 0, 'detail info')
```
```python
# export EVENTHUB_AUX_UNSTRUCTURED='ehunstruct'
# export EVENTHUB_AUX_STRUCTURED='ehstruct'
# export EVENTHUB_CONN_STRING='<connection string>'
from pyraisdk import rlog
rlog.initialize()
rlog.infocf('corrid', -1, 'this is a info message: %s', 123)
rlog.event('RequestDuration', '200', 0.01, 'this is duration in seconds')
```
```python
from pyraisdk import rlog
rlog.initialize(eh_structured='ehstruct', eh_unstructured='ehunstruct', eh_conn_str='<eventhub-conn-str>')
rlog.errorcf('corrid', -1, Exception('error msg'), 'error message: %s %s', 1,2)
rlog.event('CpuUsage', '', 0.314, detail='cpu usage', corr_id='corrid', elem=-1)
```
Raw data
{
"_id": null,
"home_page": "",
"name": "sdk-tmp-1207",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8,<4.0",
"maintainer_email": "",
"keywords": "",
"author": "Xiaodong Yang",
"author_email": "xiaoyan@microsoft.com",
"download_url": "https://files.pythonhosted.org/packages/ad/8e/ce7e769c9123d7d1f36ebdaeab374e1a6869aa272dafaaf275e5f92edca4/sdk_tmp_1207-0.1.1.tar.gz",
"platform": null,
"description": "# pyraisdk\n\n## Dynamic Batching\n\n### Description\nWhen we deploy a model in AML with GPU instance to provide inference service, if it occupies GPU for inferencing in each request separately, it will be quite inefficient. This is a shared module helping to collect data items from different requests, and inferencing batchly in a backend thread. This will considerably improve usage efficiency of GPU.\n\n### Usage Examples\n\nBuild `YourModel` class inherited from `pyraisdk.dynbatch.BaseModel`.\n\n```python\nfrom typing import List\nfrom pyraisdk.dynbatch import BaseModel\n\nclass SimpleModel(BaseModel):\n def predict(self, items: List[str]) -> List[int]:\n rs = []\n for item in items:\n rs.append(len(item))\n return rs\n \n def preprocess(self, items: List[str]) -> List[str]:\n rs = []\n for item in items:\n rs.append(f'[{item}]')\n return rs\n```\n\nInitialize a `pyraisdk.dynbatch.DynamicBatchModel` with `YourModel` instance, and call `predict / predict_one` for inferencing.\n\n```python\nfrom pyraisdk.dynbatch import DynamicBatchModel\n\n# prepare model\nsimple_model = SimpleModel()\nbatch_model = DynamicBatchModel(simple_model)\n\n# predict\nitems = ['abc', '123456', 'xyzcccffaffaaa']\npredictions = batch_model.predict(items)\nassert predictions == [5, 8, 16]\n\n# predict_one\nitem = 'abc'\nprediction = batch_model.predict_one(item)\nassert prediction == 5\n```\n\nConcurrent requests to `predict / predict_one`, in different threads.\n\n```python\nfrom threading import Thread\nfrom pyraisdk.dynbatch import DynamicBatchModel\n\n# prepare model\nsimple_model = SimpleModel()\nbatch_model = DynamicBatchModel(simple_model)\n\n# thread run function\ndef run(name, num):\n for step in range(num):\n item = f'{name}-{step}'\n prediction = batch_model.predict_one(item)\n assert prediction == len(item) + 2\n\n# start concurrent inference\nthreads = [Thread(target=run, args=(f'{tid}', 100)) for tid in range(20)]\nfor t in threads:\n t.start()\nfor t in threads:\n t.join()\n```\n\n\n## Loging & Events\n\n### Description\nThis module is for logging and event tracing.\n\n### interface\n\n```python\n\ndef initialize(\n eh_hostname: Optional[str] = None,\n client_id: Optional[str] = None,\n eh_conn_str: Optional[str] = None,\n eh_structured: Optional[str] = None,\n eh_unstructured: Optional[str] = None,\n role: Optional[str] = None,\n instance: Optional[str] = None,\n)\n```\n\nParameter description for `initialize`:\n- **eh_hostname**: Fully Qualified Namespace aka EH Endpoint URL (*.servicebus.windows.net). Default, read $EVENTHUB_NAMESPACE\n- **client_id**: client_id of service principal. Default, read $UAI_CLIENT_ID\n- **eh_conn_str**: connection string of eventhub namespace. Default, read $EVENTHUB_CONN_STRING\n- **eh_structured**: structured eventhub name. Default, read $EVENTHUB_AUX_STRUCTURED\n- **eh_unstructured**: unstructured eventhub name. Default, read $EVENTHUB_AUX_UNSTRUCTURED\n- **role**: role, Default: RemoteModel_${ENDPOINT_NAME}\n- **instance**: instance, Default: \"${ENDPOINT_VERSION}|{os.uname()[1]}\" or \"${ENDPOINT_VERSION}|{_probably_unique_id()}\"\n\n\n```python\n\ndef event(self, key: str, code: str, numeric: float, detail: str='', corr_id: str='', elem: int=-1)\ndef infof(self, format: str, *args: Any)\ndef infocf(self, corr_id: str, elem: int, format: str, *args: Any)\ndef warnf(self, format: str, *args: Any)\ndef warncf(self, corr_id: str, elem: int, format: str, *args: Any)\ndef errorf(self, format: str, *args: Any)\ndef errorcf(self, corr_id: str, elem: int, ex: Optional[Exception], format: str, *args: Any)\ndef fatalf(self, format: str, *args: Any)\ndef fatalcf(self, corr_id: str, elem: int, ex: Optional[Exception], format: str, *args: Any)\n\n```\n\n### examples\n\n```python\n# export EVENTHUB_AUX_UNSTRUCTURED='ehunstruct'\n# export EVENTHUB_AUX_STRUCTURED='ehstruct'\n# export UAI_CLIENT_ID='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'\n# export EVENTHUB_NAMESPACE='xxx.servicebus.windows.net'\n\nfrom pyraisdk import rlog\nrlog.initialize()\n\nrlog.infof('this is a info message %s', 123)\nrlog.event('LifetimeEvent', 'STOP_GRACEFUL_SIGNAL', 0, 'detail info')\n\n```\n\n```python\n# export EVENTHUB_AUX_UNSTRUCTURED='ehunstruct'\n# export EVENTHUB_AUX_STRUCTURED='ehstruct'\n# export EVENTHUB_CONN_STRING='<connection string>'\n\nfrom pyraisdk import rlog\nrlog.initialize()\n\nrlog.infocf('corrid', -1, 'this is a info message: %s', 123)\nrlog.event('RequestDuration', '200', 0.01, 'this is duration in seconds')\n\n```\n\n```python\nfrom pyraisdk import rlog\nrlog.initialize(eh_structured='ehstruct', eh_unstructured='ehunstruct', eh_conn_str='<eventhub-conn-str>')\n\nrlog.errorcf('corrid', -1, Exception('error msg'), 'error message: %s %s', 1,2)\nrlog.event('CpuUsage', '', 0.314, detail='cpu usage', corr_id='corrid', elem=-1)\n\n```\n",
"bugtrack_url": null,
"license": "",
"summary": "",
"version": "0.1.1",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "8711aa8a4e40f4cb42b576c35d426f73",
"sha256": "35d6b26e3ccd8cfb3c5b56a279244496efdd443e8c118569900951a20f781ece"
},
"downloads": -1,
"filename": "sdk_tmp_1207-0.1.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "8711aa8a4e40f4cb42b576c35d426f73",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8,<4.0",
"size": 14164,
"upload_time": "2022-12-08T08:44:53",
"upload_time_iso_8601": "2022-12-08T08:44:53.053703Z",
"url": "https://files.pythonhosted.org/packages/73/2b/44e04589d2b669d981dadfe260921cf2a82ba345f9a0d4aec3477f3429b8/sdk_tmp_1207-0.1.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "243af0cfe1c27440c9de8c1db0429342",
"sha256": "31384f0f91a751b076db12cc72fea99e6bec29d7e7146f4408babb13dd05f7fe"
},
"downloads": -1,
"filename": "sdk_tmp_1207-0.1.1.tar.gz",
"has_sig": false,
"md5_digest": "243af0cfe1c27440c9de8c1db0429342",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8,<4.0",
"size": 12881,
"upload_time": "2022-12-08T08:45:16",
"upload_time_iso_8601": "2022-12-08T08:45:16.720347Z",
"url": "https://files.pythonhosted.org/packages/ad/8e/ce7e769c9123d7d1f36ebdaeab374e1a6869aa272dafaaf275e5f92edca4/sdk_tmp_1207-0.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-12-08 08:45:16",
"github": false,
"gitlab": false,
"bitbucket": false,
"lcname": "sdk-tmp-1207"
}
