# Overview
This package provides a MLServer runtime compatible with Openvino. This package has couple features:
1. If server detect that model file is onnx format script will auto convert to openvino format (xml, bin) with dynamic batch size for openvino.
2. Openvino dynamic batch size
3. Grpc Ready
4. V2 Inference Protocol
5. Models metrics
### Why MLserver?
For serving Openvino I choose MLServer because this framework has V2 Inference Protocol (https://kserve.github.io/website/modelserving/inference_api/), grpc and metrics out of the box.
# Install
```sh
pip install mlserver mlserver-openvino
```
# Content Types
If no content type is present on the request or metadata,
the Openvino runtime will try to decode the payload as a NumPy Array.
To avoid this, either send a different content type explicitly, or define the correct one as part of your model’s [metadata](https://mlserver.readthedocs.io/en/latest/reference/model-settings.html).
# Models repository
Your models add to models folder.
Accepted files: ["model.xml", "model.onnx"]
```sh
/example
/models/your-model-name/
/tests
setup.py
README.md
```
Training and serve example: https://mlserver.readthedocs.io/en/latest/examples/sklearn/README.html
# Metrics
For download metrics (prometheus) use below links
```sh
GET http://<your-endpoint>/metrics
GET http://0.0.0.0:8080/metrics
```
# Start docker server
```sh
# Build docker image
mlserver build . -t test
# Start server and pass mlserevr_models_dir
docker run -it --rm -e MLSERVER_MODELS_DIR=/opt/mlserver/models/ -p 8080:8080 -p 8081:8081 test
```
# Example queries:
For example script see below files:
```sh
/example/grpc-example.py
/example/rest-example.py
```
# Kserve usage
1. First create one time kserve runtime from file: kserve/cluster-runtime.yaml
2. Create InferenceService from template:
```yaml
apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
name: "my-openvino-model"
spec:
predictor:
model:
modelFormat:
name: openvino
runtime: kserve-mlserver-openvino
#storageUri: "gs://kfserving-examples/models/xgboost/iris"
storageUri: https://github.com/myrepo/models/mymodel.joblib?raw=true
```
## Example model-settings.json
```json
{
"name": "mnist-onnx-openvino",
"implementation": "mlserver_openvino.OpenvinoRuntime",
"parameters": {
"uri": "./model.onnx",
"version": "v0.1.0",
"extra": {
"transform": [
{
"name": "Prepare Metadata",
"pipeline_file_path": "./pipeline.cloudpickle",
"input_index": 0
}
]
}
},
"inputs": [
{
"name": "input-0",
"datatype": "FP32",
"shape": [28,28,1]
}
],
"outputs": [
{
"name": "output",
"datatype": "FP32",
"shape": [10]
}
]
}
```
## Transformers
If you add transformer pipeline in extra properties you should dump code in same python version as execute mlserver
## Tests
```sh
make test
```
Raw data
{
"_id": null,
"home_page": "https://github.com/gawsoftpl/mlserver-openvino",
"name": "mlserver-openvino",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "",
"author": "Gawsoft",
"author_email": "biuro@gawsoft.pl",
"download_url": "https://files.pythonhosted.org/packages/90/a0/5e9b79602a9111fb4cccd49ccdd69b31907d80e0d8ffe66c102b35931b3a/mlserver_openvino-0.4.10.tar.gz",
"platform": null,
"description": "# Overview\nThis package provides a MLServer runtime compatible with Openvino. This package has couple features:\n1. If server detect that model file is onnx format script will auto convert to openvino format (xml, bin) with dynamic batch size for openvino.\n2. Openvino dynamic batch size\n3. Grpc Ready\n4. V2 Inference Protocol\n5. Models metrics\n\n### Why MLserver?\nFor serving Openvino I choose MLServer because this framework has V2 Inference Protocol (https://kserve.github.io/website/modelserving/inference_api/), grpc and metrics out of the box.\n\n# Install\n```sh\npip install mlserver mlserver-openvino\n```\n\n# Content Types\nIf no content type is present on the request or metadata, \nthe Openvino runtime will try to decode the payload as a NumPy Array. \nTo avoid this, either send a different content type explicitly, or define the correct one as part of your model\u2019s [metadata](https://mlserver.readthedocs.io/en/latest/reference/model-settings.html).\n\n# Models repository\nYour models add to models folder.\nAccepted files: [\"model.xml\", \"model.onnx\"]\n```sh\n/example\n/models/your-model-name/\n/tests\nsetup.py\nREADME.md\n```\nTraining and serve example: https://mlserver.readthedocs.io/en/latest/examples/sklearn/README.html\n\n# Metrics\nFor download metrics (prometheus) use below links\n```sh\nGET http://<your-endpoint>/metrics\nGET http://0.0.0.0:8080/metrics\n```\n\n# Start docker server\n```sh\n# Build docker image\nmlserver build . -t test\n\n# Start server and pass mlserevr_models_dir\ndocker run -it --rm -e MLSERVER_MODELS_DIR=/opt/mlserver/models/ -p 8080:8080 -p 8081:8081 test\n```\n\n# Example queries:\nFor example script see below files:\n```sh\n/example/grpc-example.py\n/example/rest-example.py\n```\n\n# Kserve usage\n1. First create one time kserve runtime from file: kserve/cluster-runtime.yaml\n2. Create InferenceService from template:\n```yaml\napiVersion: \"serving.kserve.io/v1beta1\"\nkind: \"InferenceService\"\nmetadata:\n name: \"my-openvino-model\"\nspec:\n predictor:\n model:\n modelFormat:\n name: openvino\n runtime: kserve-mlserver-openvino\n #storageUri: \"gs://kfserving-examples/models/xgboost/iris\"\n storageUri: https://github.com/myrepo/models/mymodel.joblib?raw=true\n\n```\n\n## Example model-settings.json\n```json\n{\n \"name\": \"mnist-onnx-openvino\",\n \"implementation\": \"mlserver_openvino.OpenvinoRuntime\",\n \"parameters\": {\n \"uri\": \"./model.onnx\",\n \"version\": \"v0.1.0\",\n \"extra\": {\n \"transform\": [\n {\n \"name\": \"Prepare Metadata\",\n \"pipeline_file_path\": \"./pipeline.cloudpickle\",\n \"input_index\": 0\n }\n ]\n }\n },\n \"inputs\": [\n {\n \"name\": \"input-0\",\n \"datatype\": \"FP32\",\n \"shape\": [28,28,1]\n }\n ],\n \"outputs\": [\n {\n \"name\": \"output\",\n \"datatype\": \"FP32\",\n \"shape\": [10]\n }\n ]\n}\n\n```\n\n## Transformers\nIf you add transformer pipeline in extra properties you should dump code in same python version as execute mlserver\n\n\n## Tests\n```sh\nmake test\n```\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "Openvino runtime for MLServer",
"version": "0.4.10",
"project_urls": {
"Homepage": "https://github.com/gawsoftpl/mlserver-openvino"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c642eb11b5fff61ffcb5c4c922caed6314d0d2f1d4c4063d086671bd733fbae4",
"md5": "1860effb37ce0f4109dfcf9da73a63ff",
"sha256": "b386e51db0ae13517693d6ee686e8277e8e468cc2db531817749040f552819a5"
},
"downloads": -1,
"filename": "mlserver_openvino-0.4.10-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1860effb37ce0f4109dfcf9da73a63ff",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 13962,
"upload_time": "2023-09-24T16:26:39",
"upload_time_iso_8601": "2023-09-24T16:26:39.126216Z",
"url": "https://files.pythonhosted.org/packages/c6/42/eb11b5fff61ffcb5c4c922caed6314d0d2f1d4c4063d086671bd733fbae4/mlserver_openvino-0.4.10-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "90a05e9b79602a9111fb4cccd49ccdd69b31907d80e0d8ffe66c102b35931b3a",
"md5": "4559ae58f9d01af9e38ed0253245487f",
"sha256": "128e82b0ab9e93348a70bce8a8f2c186a0e14f3318d454a15bc831bc42bf9dd8"
},
"downloads": -1,
"filename": "mlserver_openvino-0.4.10.tar.gz",
"has_sig": false,
"md5_digest": "4559ae58f9d01af9e38ed0253245487f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 12542,
"upload_time": "2023-09-24T16:26:40",
"upload_time_iso_8601": "2023-09-24T16:26:40.305788Z",
"url": "https://files.pythonhosted.org/packages/90/a0/5e9b79602a9111fb4cccd49ccdd69b31907d80e0d8ffe66c102b35931b3a/mlserver_openvino-0.4.10.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-09-24 16:26:40",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "gawsoftpl",
"github_project": "mlserver-openvino",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "mlserver",
"specs": [
[
">=",
"1.3.5"
]
]
},
{
"name": "openvino",
"specs": [
[
">=",
"2023.1.0"
]
]
},
{
"name": "openvino-dev",
"specs": [
[
">=",
"2023.1.0"
]
]
},
{
"name": "openvino-dev",
"specs": [
[
">=",
"2023.1.0"
]
]
},
{
"name": "numpy",
"specs": [
[
"<=",
"1.23.4"
]
]
},
{
"name": "grpcio",
"specs": [
[
">=",
"1.47.0"
]
]
},
{
"name": "joblib",
"specs": [
[
">=",
"1.2.0"
]
]
},
{
"name": "cloudpickle",
"specs": [
[
">=",
"2.2.0"
]
]
},
{
"name": "scikit-learn",
"specs": [
[
">=",
"1.2.0"
]
]
},
{
"name": "Keras-Preprocessing",
"specs": [
[
">=",
"1.1.2"
]
]
}
],
"lcname": "mlserver-openvino"
}
JSON
