<p align="center">
<img src="https://user-images.githubusercontent.com/38581401/134487662-49733d45-2ba0-4c19-aa07-1f43fd35c453.png" height="230" alt="MOSEC" />
</p>
<p align="center">
<a href="https://discord.gg/JTWcRNGQyQ">
<img alt="discord invitation link" src="https://img.shields.io/discord/916177932236521533">
</a>
<a href="https://pypi.org/project/mosec/">
<img src="https://badge.fury.io/py/mosec.svg" alt="PyPI version" height="20">
</a>
<a href="https://pepy.tech/project/mosec">
<img src="https://pepy.tech/badge/mosec/month" alt="PyPi Downloads" height="20">
</a>
<a href="https://tldrlegal.com/license/apache-license-2.0-(apache-2.0)">
<img src="https://img.shields.io/github/license/mosecorg/mosec" alt="License" height="20">
</a>
<a href="https://github.com/mosecorg/mosec/actions/workflows/check.yml?query=workflow%3A%22lint+and+test%22+branch%3Amain">
<img src="https://github.com/mosecorg/mosec/actions/workflows/check.yml/badge.svg?branch=main" alt="Check status" height="20">
</a>
</p>
<p align="center">
<i>Model Serving made Efficient in the Cloud.</i>
</p>
## Introduction
Mosec is a high-performance and flexible model serving framework for building ML model-enabled backend and microservices. It bridges the gap between any machine learning models you just trained and the efficient online service API.
- **Highly performant**: web layer and task coordination built with Rust 🦀, which offers blazing speed in addition to efficient CPU utilization powered by async I/O
- **Ease of use**: user interface purely in Python 🐍, by which users can serve their models in an ML framework-agnostic manner using the same code as they do for offline testing
- **Dynamic batching**: aggregate requests from different users for batched inference and distribute results back
- **Pipelined stages**: spawn multiple processes for pipelined stages to handle CPU/GPU/IO mixed workloads
- **Cloud friendly**: designed to run in the cloud, with the model warmup, graceful shutdown, and Prometheus monitoring metrics, easily managed by Kubernetes or any container orchestration systems
- **Do one thing well**: focus on the online serving part, users can pay attention to the model performance and business logic
## Installation
Mosec requires Python 3.7 or above. Install the latest [PyPI package](https://pypi.org/project/mosec/) with:
```shell
> pip install -U mosec
```
## Usage
### Write the server
Import the libraries and set up a basic logger to better observe what happens.
```python
import logging
from mosec import Server, Worker
from mosec.errors import ValidationError
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter(
"%(asctime)s - %(process)d - %(levelname)s - %(filename)s:%(lineno)s - %(message)s"
)
sh = logging.StreamHandler()
sh.setFormatter(formatter)
logger.addHandler(sh)
```
Then, we **build an API** to calculate the exponential with base **e** for a given number. To achieve that, we simply inherit the `Worker` class and override the `forward` method. Note that the input `req` is by default a JSON-decoded object, e.g., a dictionary here (wishfully it receives data like `{"x": 1}`). We also enclose the input parsing part with a `try...except...` block to reject invalid input (e.g., no key named `"x"` or field `"x"` cannot be converted to `float`).
```python
import math
class CalculateExp(Worker):
def forward(self, req: dict) -> dict:
try:
x = float(req["x"])
except KeyError:
raise ValidationError("cannot find key 'x'")
except ValueError:
raise ValidationError("cannot convert 'x' value to float")
y = math.exp(x) # f(x) = e ^ x
logger.debug(f"e ^ {x} = {y}")
return {"y": y}
```
Finally, we append the worker to the server to construct a `single-stage workflow`, and we specify the number of processes we want it to run in parallel. Then we run the server.
```python
if __name__ == "__main__":
server = Server()
server.append_worker(
CalculateExp, num=2
) # we spawn two processes for parallel computing
server.run()
```
### Run the server
After merging the snippets above into a file named `server.py`, we can first have a look at the command line arguments:
```shell
> python server.py --help
```
Then let's start the server...
```shell
> python server.py
```
and in another terminal, test it:
```console
> curl -X POST http://127.0.0.1:8000/inference -d '{"x": 2}'
{
"y": 7.38905609893065
}
> curl -X POST http://127.0.0.1:8000/inference -d '{"input": 2}' # wrong schema
validation error: cannot find key 'x'
```
or check the metrics:
```shell
> curl http://127.0.0.1:8000/metrics
```
For more debug logs, you can enable it by changing the Python & Rust log level:
```python
logger.setLevel(logging.DEBUG)
```
```shell
> RUST_LOG=debug python server.py
```
That's it! You have just hosted your **_exponential-computing model_** as a server! 😉
## Example
More ready-to-use examples can be found in the [Example](https://mosecorg.github.io/mosec/example) section. It includes:
- Multi-stage workflow
- Batch processing worker
- Shared memory IPC
- Customized GPU allocation
- PyTorch deep learning models:
- sentiment analysis
- image recognition
## Qualitative Comparison<sup>\*</sup>
| | Batcher | Pipeline | Parallel | I/O Format<sup>(1)</sup> | Framework<sup>(2)</sup> | Backend | Activity |
| ----------------------------------------------------------- | :-----: | :------: | :------: | ------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | ------- | ----------------------------------------------------------------------------- |
| [TF Serving](https://github.com/tensorflow/serving) | ✅ | ✅ | ✅ | Limited<a href="https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/api_rest.md#request-format-1"><sup>(a)</sup></a> | Heavily TF | C++ |  |
| [Triton](https://github.com/triton-inference-server/server) | ✅ | ✅ | ✅ | Limited | Multiple | C++ |  |
| [MMS](https://github.com/awslabs/multi-model-server) | ✅ | ❌ | ✅ | Limited | Heavily MX | Java |  |
| [BentoML](https://github.com/bentoml/BentoML) | ✅ | ❌ | ❌ | Limited<a href="https://docs.bentoml.org/en/latest/concepts.html#api-function-return-value"><sup>(b)</sup></a> | Multiple | Python |  |
| [Streamer](https://github.com/ShannonAI/service-streamer) | ✅ | ❌ | ✅ | Customizable | Agnostic | Python |  |
| [Flask](https://github.com/pallets/flask)<sup>(3)</sup> | ❌ | ❌ | ❌ | Customizable | Agnostic | Python |  |
| **[Mosec](https://github.com/mosecorg/mosec)** | ✅ | ✅ | ✅ | Customizable | Agnostic | Rust |  |
<sup>\*As accessed on 08 Oct 2021. By no means is this comparison showing that other frameworks are inferior, but rather it is used to illustrate the trade-off. The information is not guaranteed to be absolutely accurate. Please let us know if you find anything that may be incorrect.</sup>
<sup>**(1)**: Data format of the service's request and response. "Limited" in the sense that the framework has pre-defined requirements on the format.</sup>
<sup>**(2)**: Supported machine learning frameworks. "Heavily" means the serving framework is designed towards a specific ML framework. Thus it is hard, if not impossible, to adapt to others. "Multiple" means the serving framework provides adaptation to several existing ML frameworks. "Agnostic" means the serving framework does not necessarily care about the ML framework. Hence it supports all ML frameworks (in Python).</sup>
<sup>**(3)**: Flask is a representative of general purpose web frameworks to host ML models.</sup>
## Contributing
We welcome any kind of contribution. Please give us feedback by [raising issues](https://github.com/mosecorg/mosec/issues/new/choose) or discussing on [Discord](https://discord.gg/JTWcRNGQyQ). You could also directly [contribute](https://mosecorg.github.io/mosec/contributing) your code and pull request!
Raw data
{
"_id": null,
"home_page": "https://github.com/mosecorg/mosec",
"name": "mosec-tiinfer",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "",
"keywords": "",
"author": "Keming Yang",
"author_email": "kemingy94@gmail.com",
"download_url": "",
"platform": null,
"description": "<p align=\"center\">\n <img src=\"https://user-images.githubusercontent.com/38581401/134487662-49733d45-2ba0-4c19-aa07-1f43fd35c453.png\" height=\"230\" alt=\"MOSEC\" />\n</p>\n\n<p align=\"center\">\n <a href=\"https://discord.gg/JTWcRNGQyQ\">\n <img alt=\"discord invitation link\" src=\"https://img.shields.io/discord/916177932236521533\">\n </a>\n <a href=\"https://pypi.org/project/mosec/\">\n <img src=\"https://badge.fury.io/py/mosec.svg\" alt=\"PyPI version\" height=\"20\">\n </a>\n <a href=\"https://pepy.tech/project/mosec\">\n <img src=\"https://pepy.tech/badge/mosec/month\" alt=\"PyPi Downloads\" height=\"20\">\n </a>\n <a href=\"https://tldrlegal.com/license/apache-license-2.0-(apache-2.0)\">\n <img src=\"https://img.shields.io/github/license/mosecorg/mosec\" alt=\"License\" height=\"20\">\n </a>\n <a href=\"https://github.com/mosecorg/mosec/actions/workflows/check.yml?query=workflow%3A%22lint+and+test%22+branch%3Amain\">\n <img src=\"https://github.com/mosecorg/mosec/actions/workflows/check.yml/badge.svg?branch=main\" alt=\"Check status\" height=\"20\">\n </a>\n</p>\n\n<p align=\"center\">\n <i>Model Serving made Efficient in the Cloud.</i>\n</p>\n\n## Introduction\n\nMosec is a high-performance and flexible model serving framework for building ML model-enabled backend and microservices. It bridges the gap between any machine learning models you just trained and the efficient online service API.\n\n- **Highly performant**: web layer and task coordination built with Rust \ud83e\udd80, which offers blazing speed in addition to efficient CPU utilization powered by async I/O\n- **Ease of use**: user interface purely in Python \ud83d\udc0d, by which users can serve their models in an ML framework-agnostic manner using the same code as they do for offline testing\n- **Dynamic batching**: aggregate requests from different users for batched inference and distribute results back\n- **Pipelined stages**: spawn multiple processes for pipelined stages to handle CPU/GPU/IO mixed workloads\n- **Cloud friendly**: designed to run in the cloud, with the model warmup, graceful shutdown, and Prometheus monitoring metrics, easily managed by Kubernetes or any container orchestration systems\n- **Do one thing well**: focus on the online serving part, users can pay attention to the model performance and business logic\n\n## Installation\n\nMosec requires Python 3.7 or above. Install the latest [PyPI package](https://pypi.org/project/mosec/) with:\n\n```shell\n> pip install -U mosec\n```\n\n## Usage\n\n### Write the server\n\nImport the libraries and set up a basic logger to better observe what happens.\n\n```python\nimport logging\n\nfrom mosec import Server, Worker\nfrom mosec.errors import ValidationError\n\nlogger = logging.getLogger()\nlogger.setLevel(logging.DEBUG)\nformatter = logging.Formatter(\n \"%(asctime)s - %(process)d - %(levelname)s - %(filename)s:%(lineno)s - %(message)s\"\n)\nsh = logging.StreamHandler()\nsh.setFormatter(formatter)\nlogger.addHandler(sh)\n```\n\nThen, we **build an API** to calculate the exponential with base **e** for a given number. To achieve that, we simply inherit the `Worker` class and override the `forward` method. Note that the input `req` is by default a JSON-decoded object, e.g., a dictionary here (wishfully it receives data like `{\"x\": 1}`). We also enclose the input parsing part with a `try...except...` block to reject invalid input (e.g., no key named `\"x\"` or field `\"x\"` cannot be converted to `float`).\n\n```python\nimport math\n\n\nclass CalculateExp(Worker):\n def forward(self, req: dict) -> dict:\n try:\n x = float(req[\"x\"])\n except KeyError:\n raise ValidationError(\"cannot find key 'x'\")\n except ValueError:\n raise ValidationError(\"cannot convert 'x' value to float\")\n y = math.exp(x) # f(x) = e ^ x\n logger.debug(f\"e ^ {x} = {y}\")\n return {\"y\": y}\n```\n\nFinally, we append the worker to the server to construct a `single-stage workflow`, and we specify the number of processes we want it to run in parallel. Then we run the server.\n\n```python\nif __name__ == \"__main__\":\n server = Server()\n server.append_worker(\n CalculateExp, num=2\n ) # we spawn two processes for parallel computing\n server.run()\n\n```\n\n### Run the server\n\nAfter merging the snippets above into a file named `server.py`, we can first have a look at the command line arguments:\n\n```shell\n> python server.py --help\n```\n\nThen let's start the server...\n\n```shell\n> python server.py\n```\n\nand in another terminal, test it:\n\n```console\n> curl -X POST http://127.0.0.1:8000/inference -d '{\"x\": 2}'\n{\n \"y\": 7.38905609893065\n}\n\n> curl -X POST http://127.0.0.1:8000/inference -d '{\"input\": 2}' # wrong schema\nvalidation error: cannot find key 'x'\n```\n\nor check the metrics:\n\n```shell\n> curl http://127.0.0.1:8000/metrics\n```\n\nFor more debug logs, you can enable it by changing the Python & Rust log level:\n\n```python\nlogger.setLevel(logging.DEBUG)\n```\n\n```shell\n> RUST_LOG=debug python server.py\n```\n\nThat's it! You have just hosted your **_exponential-computing model_** as a server! \ud83d\ude09\n\n## Example\n\nMore ready-to-use examples can be found in the [Example](https://mosecorg.github.io/mosec/example) section. It includes:\n\n- Multi-stage workflow\n- Batch processing worker\n- Shared memory IPC\n- Customized GPU allocation\n- PyTorch deep learning models:\n - sentiment analysis\n - image recognition\n\n## Qualitative Comparison<sup>\\*</sup>\n\n| | Batcher | Pipeline | Parallel | I/O Format<sup>(1)</sup> | Framework<sup>(2)</sup> | Backend | Activity |\n| ----------------------------------------------------------- | :-----: | :------: | :------: | ------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | ------- | ----------------------------------------------------------------------------- |\n| [TF Serving](https://github.com/tensorflow/serving) | \u2705 | \u2705 | \u2705 | Limited<a href=\"https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/api_rest.md#request-format-1\"><sup>(a)</sup></a> | Heavily TF | C++ |  |\n| [Triton](https://github.com/triton-inference-server/server) | \u2705 | \u2705 | \u2705 | Limited | Multiple | C++ |  |\n| [MMS](https://github.com/awslabs/multi-model-server) | \u2705 | \u274c | \u2705 | Limited | Heavily MX | Java |  |\n| [BentoML](https://github.com/bentoml/BentoML) | \u2705 | \u274c | \u274c | Limited<a href=\"https://docs.bentoml.org/en/latest/concepts.html#api-function-return-value\"><sup>(b)</sup></a> | Multiple | Python |  |\n| [Streamer](https://github.com/ShannonAI/service-streamer) | \u2705 | \u274c | \u2705 | Customizable | Agnostic | Python |  |\n| [Flask](https://github.com/pallets/flask)<sup>(3)</sup> | \u274c | \u274c | \u274c | Customizable | Agnostic | Python |  |\n| **[Mosec](https://github.com/mosecorg/mosec)** | \u2705 | \u2705 | \u2705 | Customizable | Agnostic | Rust |  |\n\n\n<sup>\\*As accessed on 08 Oct 2021. By no means is this comparison showing that other frameworks are inferior, but rather it is used to illustrate the trade-off. The information is not guaranteed to be absolutely accurate. Please let us know if you find anything that may be incorrect.</sup>\n\n<sup>**(1)**: Data format of the service's request and response. \"Limited\" in the sense that the framework has pre-defined requirements on the format.</sup>\n<sup>**(2)**: Supported machine learning frameworks. \"Heavily\" means the serving framework is designed towards a specific ML framework. Thus it is hard, if not impossible, to adapt to others. \"Multiple\" means the serving framework provides adaptation to several existing ML frameworks. \"Agnostic\" means the serving framework does not necessarily care about the ML framework. Hence it supports all ML frameworks (in Python).</sup>\n<sup>**(3)**: Flask is a representative of general purpose web frameworks to host ML models.</sup>\n\n## Contributing\n\nWe welcome any kind of contribution. Please give us feedback by [raising issues](https://github.com/mosecorg/mosec/issues/new/choose) or discussing on [Discord](https://discord.gg/JTWcRNGQyQ). You could also directly [contribute](https://mosecorg.github.io/mosec/contributing) your code and pull request!\n\n\n",
"bugtrack_url": null,
"license": "Apache License 2.0",
"summary": "Model Serving made Efficient in the Cloud.",
"version": "0.0.7",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "9ed22ec04601f472b232ef51db0b3bdb41cf0a76f1b89b50c6336cd2a9e96074",
"md5": "581636071253b22edbbc19f5528f2cd7",
"sha256": "c744cc75ca07ae26c903eb9bee0b581766fced55e228582441291f8d4936c0b6"
},
"downloads": -1,
"filename": "mosec_tiinfer-0.0.7-cp37-cp37m-manylinux1_x86_64.whl",
"has_sig": false,
"md5_digest": "581636071253b22edbbc19f5528f2cd7",
"packagetype": "bdist_wheel",
"python_version": "cp37",
"requires_python": ">=3.6",
"size": 3039570,
"upload_time": "2023-03-22T12:37:23",
"upload_time_iso_8601": "2023-03-22T12:37:23.974737Z",
"url": "https://files.pythonhosted.org/packages/9e/d2/2ec04601f472b232ef51db0b3bdb41cf0a76f1b89b50c6336cd2a9e96074/mosec_tiinfer-0.0.7-cp37-cp37m-manylinux1_x86_64.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "34f72c7ba43d4974208643b25d9cc59f06dbd5c06603f1365b8cc3e73c096599",
"md5": "cab3054cf3ff11fd0eb02b1177d72b2a",
"sha256": "4e8f61c3fe9bbd721e37f9f07780da05351a36dfae7542f527dcdec8b9954104"
},
"downloads": -1,
"filename": "mosec_tiinfer-0.0.7-cp38-cp38-manylinux1_x86_64.whl",
"has_sig": false,
"md5_digest": "cab3054cf3ff11fd0eb02b1177d72b2a",
"packagetype": "bdist_wheel",
"python_version": "cp38",
"requires_python": ">=3.6",
"size": 3039566,
"upload_time": "2023-03-22T12:35:44",
"upload_time_iso_8601": "2023-03-22T12:35:44.866434Z",
"url": "https://files.pythonhosted.org/packages/34/f7/2c7ba43d4974208643b25d9cc59f06dbd5c06603f1365b8cc3e73c096599/mosec_tiinfer-0.0.7-cp38-cp38-manylinux1_x86_64.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-03-22 12:37:23",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "mosecorg",
"github_project": "mosec",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "mosec-tiinfer"
}
JSON
